Repository: dask/dask

Issue:
Much slower (100 x) xarray.open_mfdataset  + compute after upgrading to dask > 2022.1.0
We're  opening 3-5 netcdf files, each 200-300mb.

Running on Azure AKS,  python 3.8
xarray==0.21.1
dask==2022.1.1

```
  def _get_forecast_dataframe(self, request: Any, lat: float, lon: float) -> Union[pd.DataFrame, None]:
        """Get a DataFrame with four day forecast"""
        ds = xr.open_mfdataset(self.local_paths)
        times = ds.time.data
        lats = ds.lat.data
        lons = ds.lon.data
        vars = [ds[v] for v in ds.variables if v not in ["lat", "lon", "time", "crs"]]
        forcecast= [v for v in vars if v.long_name.startswith("forecast")]
        tss = {v.name: v.sel(lat=lat, lon=lon, method="nearest").compute() for v in forcecast}
        df = pd.DataFrame(data=tss, index=times)
        return df
```

This call executes in 300-400ms with dask <= 2022.1.0
After upgrading, automatic if package version not pinned, the call takes 20-30 sec 

**What happened**:

After upgrading (automatically), our processes had close to 100x longer call times

**What you expected to happen**:

Upgrading having no negative performance impact

**Minimal Complete Verifiable Example**:

```
import xarray as xr

# weather files from Norwegian Met Service
# if specified not available , download from https://thredds.met.no/thredds/catalog/metpplatest/catalog.xml
# download to /mnt/data, or update path below

url= "https://thredds.met.no/thredds/fileServer/metpplatest/met_forecast_1_0km_nordic_20220225T11Z.nc"

def compute(lat: float, lon: float):
    ds = xr.open_mfdataset("/mnt/data/met*.nc")
    lats = ds.x.data
    lons = ds.y.data
    if lat < lats[0] or lat > lats[-1]:
        print("lat out-of-bounds")
    if lon < lons[0] or lon > lons[-1]:
        print("lon out-of-bounds")
    vars = [ds[v] for v in ds.variables if v  in ["air_pressure_at_sea_level"]]
    tss = {v.name: v.sel(x=lat, y=lon, method="nearest").compute() for v in vars}
    print(tss)

if __name__ == "__main__":
    compute(0, 0)
```

Timing with dask==2202.2:

root@d93a10d88800:/workspace/airmine# time python test-dask.py

real    0m3.689s
user    0m3.045s
sys     0m1.761s

Timing with dask==2202.1
root@d93a10d88800:/workspace/airmine# time python test-dask.py

real    0m0.972s
user    0m0.962s
sys     0m1.165s

This is with opening a single file, we typically open an array

**Anything else we need to know?**:

**Environment**:

VERSION="20.04.2 LTS (Focal Fossa)
Python 3.8.5 [GCC 9.3.0] on linux
xarray==0.21.1
dask==2022.1.1
Installed with pip -r requirements


Much slower (100 x) xarray.open_mfdataset  + compute after upgrading to dask > 2022.1.0
We're  opening 3-5 netcdf files, each 200-300mb.

Running on Azure AKS,  python 3.8
xarray==0.21.1
dask==2022.1.1

```
  def _get_forecast_dataframe(self, request: Any, lat: float, lon: float) -> Union[pd.DataFrame, None]:
        """Get a DataFrame with four day forecast"""
        ds = xr.open_mfdataset(self.local_paths)
        times = ds.time.data
        lats = ds.lat.data
        lons = ds.lon.data
        vars = [ds[v] for v in ds.variables if v not in ["lat", "lon", "time", "crs"]]
        forcecast= [v for v in vars if v.long_name.startswith("forecast")]
        tss = {v.name: v.sel(lat=lat, lon=lon, method="nearest").compute() for v in forcecast}
        df = pd.DataFrame(data=tss, index=times)
        return df
```

This call executes in 300-400ms with dask <= 2022.1.0
After upgrading, automatic if package version not pinned, the call takes 20-30 sec 

**What happened**:

After upgrading (automatically), our processes had close to 100x longer call times

**What you expected to happen**:

Upgrading having no negative performance impact

**Minimal Complete Verifiable Example**:

```
import xarray as xr

# weather files from Norwegian Met Service
# if specified not available , download from https://thredds.met.no/thredds/catalog/metpplatest/catalog.xml
# download to /mnt/data, or update path below

url= "https://thredds.met.no/thredds/fileServer/metpplatest/met_forecast_1_0km_nordic_20220225T11Z.nc"

def compute(lat: float, lon: float):
    ds = xr.open_mfdataset("/mnt/data/met*.nc")
    lats = ds.x.data
    lons = ds.y.data
    if lat < lats[0] or lat > lats[-1]:
        print("lat out-of-bounds")
    if lon < lons[0] or lon > lons[-1]:
        print("lon out-of-bounds")
    vars = [ds[v] for v in ds.variables if v  in ["air_pressure_at_sea_level"]]
    tss = {v.name: v.sel(x=lat, y=lon, method="nearest").compute() for v in vars}
    print(tss)

if __name__ == "__main__":
    compute(0, 0)
```

Timing with dask==2202.2:

root@d93a10d88800:/workspace/airmine# time python test-dask.py

real    0m3.689s
user    0m3.045s
sys     0m1.761s

Timing with dask==2202.1
root@d93a10d88800:/workspace/airmine# time python test-dask.py

real    0m0.972s
user    0m0.962s
sys     0m1.165s

This is with opening a single file, we typically open an array

**Anything else we need to know?**:

**Environment**:

VERSION="20.04.2 LTS (Focal Fossa)
Python 3.8.5 [GCC 9.3.0] on linux
xarray==0.21.1
dask==2022.1.1
Installed with pip -r requirements




Discussion:
Thanks for reporting @espenairmine. Are you able to provide a minimal reproducer others can run (see https://blog.dask.org/2018/02/28/minimal-bug-reports for more details)? 
@jrbourbeau  - hopefully reproduceable with updated code above
It would help if you can check the code above. Particularly the path in the `compute` function is not right. 

If I naively run:

```python
import xarray as xr

# weather files from Norwegian Met Service
# if specified not available , download from https://thredds.met.no/thredds/catalog/metpplatest/catalog.xml
# download to /mnt/data, or update path below

url= "https://thredds.met.no/thredds/fileServer/metpplatest/met_forecast_1_0km_nordic_20220225T11Z.nc"

def compute(lat: float, lon: float):
    ds = xr.open_mfdataset("/mnt/data/met*.nc")  # what should this be?
    lats = ds.x.data
    lons = ds.y.data
    if lat < lats[0] or lat > lats[-1]:
        print("lat out-of-bounds")
    if lon < lons[0] or lon > lons[-1]:
        print("lon out-of-bounds")
    vars = [ds[v] for v in ds.variables if v  in ["air_pressure_at_sea_level"]]
    tss = {v.name: v.sel(x=lat, y=lon, method="nearest").compute() for v in vars}
    print(tss)

if __name__ == "__main__":
    compute(0, 0)
```

I get:

```python-traceback
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Input In [1], in <module>
     20     print(tss)
     23 if __name__ == "__main__":
---> 24     compute(0, 0)

Input In [1], in compute(lat, lon)
     10 def compute(lat: float, lon: float):
---> 11     ds = xr.open_mfdataset("/mnt/data/met*.nc")  # what should this be?
     12     lats = ds.x.data
     13     lons = ds.y.data

File ~/conda/envs/dask-dev/lib/python3.9/site-packages/xarray/backends/api.py:873, in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, data_vars, coords, combine, parallel, join, attrs_file, combine_attrs, **kwargs)
    870     paths = [os.fspath(p) if isinstance(p, os.PathLike) else p for p in paths]
    872 if not paths:
--> 873     raise OSError("no files to open")
    875 if combine == "nested":
    876     if isinstance(concat_dim, (str, DataArray)) or concat_dim is None:

OSError: no files to open
```

@jsignell -  sorry, could have included code to download the files, but it was in the comment above:
```
# weather files from Norwegian Met Service
# if specified not available , download from https://thredds.met.no/thredds/catalog/metpplatest/catalog.xml
# download to /mnt/data, or update path below

url= "https://thredds.met.no/thredds/fileServer/metpplatest/met_forecast_1_0km_nordic_20220225T11Z.nc"
```
I think you can technically open the url directly in the open_mfdataset call, but that circumvents the problem - much slower processing of local files
Ah sorry it's been a while since I was downloading from a thredds server. 

I downloaded a datafile from here: https://thredds.met.no/thredds/catalog/metpplatest/catalog.html?dataset=metpplatest/met_forecast_1_0km_nordic_20220303T13Z.nc 

And ran:

```python
import xarray as xr

def compute(lat: float, lon: float):
    ds = xr.open_mfdataset(
        [
            "data/met_forecast_1_0km_nordic_20220303T13Z.nc",
        ]
    ) 
    lats = ds.x.data
    lons = ds.y.data
    if lat < lats[0] or lat > lats[-1]:
        print("lat out-of-bounds")
    if lon < lons[0] or lon > lons[-1]:
        print("lon out-of-bounds")
    vars = [ds[v] for v in ds.variables if v in ["air_pressure_at_sea_level"]]
    tss = {v.name: v.sel(x=lat, y=lon, method="nearest").compute() for v in vars}
    print(tss)

%time compute(0, 0)

```
with latest dask I got:

```
# CPU times: user 1.64 s, sys: 189 ms, total: 1.83 s
# Wall time: 1.91 s
```

with dask 2021.10.0 I got: 
```
CPU times: user 46.1 ms, sys: 4.32 ms, total: 50.5 ms
Wall time: 47.7 ms
```

when I use `open_dataset` instead I get much more similar times:

latest
```
CPU times: user 36.7 ms, sys: 3 ms, total: 39.7 ms
Wall time: 39.3 ms
```

2021.10.0
```
CPU times: user 41.2 ms, sys: 3.7 ms, total: 44.9 ms
Wall time: 45.1 ms
```

I'm investigating more now.
Ok I took a look at the dask graphs for these arrays and the main difference I am seeing is there is an extra layer in the new graph. I think the layer is coming from here: 

https://github.com/dask/dask/blob/f5b42d4b90d45f9c9bf42fcae99e4a4589246f18/dask/array/core.py#L299-L319

It looks like it was added in https://github.com/dask/dask/pull/7417 @ian-r-rose I'm hoping that you might have some insight.

I simplified down the MRE:

```python
import xarray as xr

ds = xr.open_mfdataset(["data/met_forecast_1_0km_nordic_20220303T13Z.nc"])
var = next(ds[v] for v in ds.variables if v in ["air_pressure_at_sea_level"])
result = var.sel(x=0, y=0, method="nearest")
result.data.dask
```

latest dask:
```
HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7fd48a7dee80>
 0. original-open_dataset-a5d44fce2ee529890053648591a49defair_pressure_at_sea_level-c0e41894600ddf6e227b3db4b107fbe5
 1. open_dataset-a5d44fce2ee529890053648591a49defair_pressure_at_sea_level-c0e41894600ddf6e227b3db4b107fbe5
 2. getitem-6a48430f3a080c3080e74c001bf3ad4d
```

dask 2021.10.0
```
HighLevelGraph with 2 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7fad69a62fd0>
 0. open_dataset-a465c8db28098227714fcb124311e413air_pressure_at_sea_level-03b0f79a193fef70fa71ac8beb31f839
 1. getitem-6b50c5c17667e0cd242d1d602d06bb72
```
Let me also ping @dcherian from xarray for visibility.
Thanks for the ping @jsignell, sorry to be slow in responding. I'm having a tough time reproducing the behavior here. I tried this on xarray==0.21.1 and dask==2021.10.0/dask==2021.2.1, and did the following steps:

1. Downloaded a ~100MiB netcdf file from the above thredds server ([here](https://thredds.met.no/thredds/catalog/metpplatest/catalog.html?dataset=metpplatest/met_analysis_1_0km_nordic_latest.nc)].
2. Ran the snippets from both [this comment](https://github.com/dask/dask/issues/8753#issuecomment-1058158881) and [this comment](https://github.com/dask/dask/issues/8753#issuecomment-1058133329).
3. Tried the above with both `xr.open_dataset` and `xr.open_mfdataset`.

In no case did I see anything close to 20-30 seconds to compute, everything was in the tens of ms range. Is there perhaps something I've missed in trying to reproduce this?

There *was* a change in how `da.from_array` constructed it's graphs recently (#7417), so it wouldn't be hugely surprising to find some downstream regressions we need to address. The difference in number of HLG layers that @jsignell found is expected, but the final low-level graph should be fairly similar. But without being able to reproduce, it's hard for me to say for sure.

I think the 20-30 seconds was for more than one file. I was seeing ~2 seconds and ~50ms for one file with dask main and dask 2021.10.0 respectively. 
With both versions I'm seeing ~50-100 ms. What scheduler are you using? If it's multiprocessing or distributed, there may be some undesirable inter-process communication of the array that's being triggered.
I didn't do any scheduler setup, so just using the default.
A quick update based on some investigations that @jsignell and I carried out earlier today:

We are indeed able to reproduce a performance regression for this example, and it is related to #7417. That PR changed `from_array` to use `Blockwise`, which in theory allows for higher-level optimizations and cheaper graph communication and construction. But the introduction of `Blockwise` prevented one type of low-level optimization: slicing fusion.

This is because fusing adjacent slices relies on checking whether each task is "get-ish": https://github.com/dask/dask/blob/02f38865939ac4775ba3f34789bfd3c5e915eebe/dask/array/optimization.py#L130-L138 

With the old low-level-graph, the `getter` callable was directly available on the task, and this comparison was easy. But in blockwise, the callables are obfuscated within a `SubgraphCallable`, and the optimizer was no longer able to identify that task as get-ish, and the optimization did not take place.

I'm still cooking up a reproducer that does not rely on xarray, and will post it here when I have it.

There are (at least) two fixes that could be made here:
1. The long term, correct fix is to convert slicing to be a HighLevelGraph Layer (see some early work in #7655), and make sure that it can talk well with `Blockwise`. It may even make sense to convert `da.from_array` from blockwise to whatever slicing layer we come up with. But that design discussion is not particularly close right now.
2. A short term, hacky fix is to make sure that the slicing fusion can recognize a blockwise subgraph callable as a getter. This would be ugly, and involve introspecting `SubgraphCallable`s, making it a very leaky abstraction. But I think it would be fairly easy to do, as long as we put big blinking disclaimers that it should be removed as soon as high-level-slicing is done.
 
Okay, here is a reproducer with just dask:

```python
import numpy
import dask.array as da

# Use an ndarray subclass to avoid special case for
# numpy arrays. Lazy arrays like xarray datasets
# already satisfy this
class MyArray(numpy.ndarray):
    pass

x = numpy.random.uniform(size=(1000, 1000)).view(MyArray)



xx = da.from_array(x, chunks=100)
yy = xx[23:24, 65:66]  # Slice from one chunk, which can be fused with from_array
dsk = yy.__dask_optimize__(yy.dask, yy.__dask_keys__())
print(dsk)
```

On `dask==2021.10.0` this produces
```python
{('getitem-482c4f7d051bab02a4e5de435b6635d3',
  0,
  0): ('array-getitem-482c4f7d051bab02a4e5de435b6635d3', 0, 0),
 'array-original-0043e9339edbf7b18e0dffa2354ca805': MyArray([[0.75496145, 0.71040058, 0.26402173, ..., 0.89002368,
           0.47585329, 0.96497852],
          [0.5823307 , 0.07175467, 0.14595467, ..., 0.63779591,
           0.67730445, 0.71338149],
          [0.42915523, 0.4993591 , 0.29309595, ..., 0.60710493,
           0.43123075, 0.52111913],
          ...,
          [0.0385207 , 0.20286531, 0.14720955, ..., 0.45732608,
           0.5083299 , 0.76006257],
          [0.72467882, 0.9119804 , 0.60712482, ..., 0.9274443 ,
           0.05040048, 0.96039124],
          [0.12749118, 0.38615043, 0.72269677, ..., 0.29804572,
           0.18775866, 0.3715634 ]]),
 ('array-getitem-482c4f7d051bab02a4e5de435b6635d3',
  0,
  0): (<function dask.array.core.getter(a, b, asarray=True, lock=None)>, 'array-original-0043e9339edbf7b18e0dffa2354ca805', (slice(23, 24, None),
   slice(65, 66, None)), False, False)}
```
whereas on `main` this produces
```python
{('getitem-8565cad6f854187d2bcb8c57560c230a',
  0,
  0): ('array-getitem-8565cad6f854187d2bcb8c57560c230a', 0, 0),
 'original-array-7600483ec382d2244bea4a0da58d8596': MyArray([[0.57031275, 0.07244231, 0.35438451, ..., 0.23643452,
           0.2705955 , 0.23255325],
          [0.53274312, 0.50732558, 0.79660538, ..., 0.1799429 ,
           0.63057926, 0.41924305],
          [0.98659251, 0.06321354, 0.29775424, ..., 0.660547  ,
           0.072433  , 0.73981265],
          ...,
          [0.16659373, 0.51461264, 0.74210757, ..., 0.40877502,
           0.44473356, 0.76993016],
          [0.33193714, 0.37566295, 0.84591651, ..., 0.21424494,
           0.91305219, 0.6167193 ],
          [0.64256872, 0.41688069, 0.3727772 , ..., 0.55530654,
           0.28894569, 0.38069073]]),
 ('array-getitem-8565cad6f854187d2bcb8c57560c230a',
  0,
  0): (<function dask.array.chunk.getitem(obj, index)>, (subgraph_callable-4b804397-1337-48c0-835e-73857e94bb0f,
   'original-array-7600483ec382d2244bea4a0da58d8596',
   (slice(0, 100, None), slice(0, 100, None))), (slice(23, 24, 1),
   slice(65, 66, 1)))}
```

Notice how in the first case the slice operation is fused with the original read from disk (effectively pushing it into the storage layer), but in the second case the slice operation is kept separate.
Thanks for reporting @espenairmine. Are you able to provide a minimal reproducer others can run (see https://blog.dask.org/2018/02/28/minimal-bug-reports for more details)? 
@jrbourbeau  - hopefully reproduceable with updated code above
It would help if you can check the code above. Particularly the path in the `compute` function is not right. 

If I naively run:

```python
import xarray as xr

# weather files from Norwegian Met Service
# if specified not available , download from https://thredds.met.no/thredds/catalog/metpplatest/catalog.xml
# download to /mnt/data, or update path below

url= "https://thredds.met.no/thredds/fileServer/metpplatest/met_forecast_1_0km_nordic_20220225T11Z.nc"

def compute(lat: float, lon: float):
    ds = xr.open_mfdataset("/mnt/data/met*.nc")  # what should this be?
    lats = ds.x.data
    lons = ds.y.data
    if lat < lats[0] or lat > lats[-1]:
        print("lat out-of-bounds")
    if lon < lons[0] or lon > lons[-1]:
        print("lon out-of-bounds")
    vars = [ds[v] for v in ds.variables if v  in ["air_pressure_at_sea_level"]]
    tss = {v.name: v.sel(x=lat, y=lon, method="nearest").compute() for v in vars}
    print(tss)

if __name__ == "__main__":
    compute(0, 0)
```

I get:

```python-traceback
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Input In [1], in <module>
     20     print(tss)
     23 if __name__ == "__main__":
---> 24     compute(0, 0)

Input In [1], in compute(lat, lon)
     10 def compute(lat: float, lon: float):
---> 11     ds = xr.open_mfdataset("/mnt/data/met*.nc")  # what should this be?
     12     lats = ds.x.data
     13     lons = ds.y.data

File ~/conda/envs/dask-dev/lib/python3.9/site-packages/xarray/backends/api.py:873, in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, data_vars, coords, combine, parallel, join, attrs_file, combine_attrs, **kwargs)
    870     paths = [os.fspath(p) if isinstance(p, os.PathLike) else p for p in paths]
    872 if not paths:
--> 873     raise OSError("no files to open")
    875 if combine == "nested":
    876     if isinstance(concat_dim, (str, DataArray)) or concat_dim is None:

OSError: no files to open
```

@jsignell -  sorry, could have included code to download the files, but it was in the comment above:
```
# weather files from Norwegian Met Service
# if specified not available , download from https://thredds.met.no/thredds/catalog/metpplatest/catalog.xml
# download to /mnt/data, or update path below

url= "https://thredds.met.no/thredds/fileServer/metpplatest/met_forecast_1_0km_nordic_20220225T11Z.nc"
```
I think you can technically open the url directly in the open_mfdataset call, but that circumvents the problem - much slower processing of local files
Ah sorry it's been a while since I was downloading from a thredds server. 

I downloaded a datafile from here: https://thredds.met.no/thredds/catalog/metpplatest/catalog.html?dataset=metpplatest/met_forecast_1_0km_nordic_20220303T13Z.nc 

And ran:

```python
import xarray as xr

def compute(lat: float, lon: float):
    ds = xr.open_mfdataset(
        [
            "data/met_forecast_1_0km_nordic_20220303T13Z.nc",
        ]
    ) 
    lats = ds.x.data
    lons = ds.y.data
    if lat < lats[0] or lat > lats[-1]:
        print("lat out-of-bounds")
    if lon < lons[0] or lon > lons[-1]:
        print("lon out-of-bounds")
    vars = [ds[v] for v in ds.variables if v in ["air_pressure_at_sea_level"]]
    tss = {v.name: v.sel(x=lat, y=lon, method="nearest").compute() for v in vars}
    print(tss)

%time compute(0, 0)

```
with latest dask I got:

```
# CPU times: user 1.64 s, sys: 189 ms, total: 1.83 s
# Wall time: 1.91 s
```

with dask 2021.10.0 I got: 
```
CPU times: user 46.1 ms, sys: 4.32 ms, total: 50.5 ms
Wall time: 47.7 ms
```

when I use `open_dataset` instead I get much more similar times:

latest
```
CPU times: user 36.7 ms, sys: 3 ms, total: 39.7 ms
Wall time: 39.3 ms
```

2021.10.0
```
CPU times: user 41.2 ms, sys: 3.7 ms, total: 44.9 ms
Wall time: 45.1 ms
```

I'm investigating more now.
Ok I took a look at the dask graphs for these arrays and the main difference I am seeing is there is an extra layer in the new graph. I think the layer is coming from here: 

https://github.com/dask/dask/blob/f5b42d4b90d45f9c9bf42fcae99e4a4589246f18/dask/array/core.py#L299-L319

It looks like it was added in https://github.com/dask/dask/pull/7417 @ian-r-rose I'm hoping that you might have some insight.

I simplified down the MRE:

```python
import xarray as xr

ds = xr.open_mfdataset(["data/met_forecast_1_0km_nordic_20220303T13Z.nc"])
var = next(ds[v] for v in ds.variables if v in ["air_pressure_at_sea_level"])
result = var.sel(x=0, y=0, method="nearest")
result.data.dask
```

latest dask:
```
HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7fd48a7dee80>
 0. original-open_dataset-a5d44fce2ee529890053648591a49defair_pressure_at_sea_level-c0e41894600ddf6e227b3db4b107fbe5
 1. open_dataset-a5d44fce2ee529890053648591a49defair_pressure_at_sea_level-c0e41894600ddf6e227b3db4b107fbe5
 2. getitem-6a48430f3a080c3080e74c001bf3ad4d
```

dask 2021.10.0
```
HighLevelGraph with 2 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x7fad69a62fd0>
 0. open_dataset-a465c8db28098227714fcb124311e413air_pressure_at_sea_level-03b0f79a193fef70fa71ac8beb31f839
 1. getitem-6b50c5c17667e0cd242d1d602d06bb72
```
Let me also ping @dcherian from xarray for visibility.
Thanks for the ping @jsignell, sorry to be slow in responding. I'm having a tough time reproducing the behavior here. I tried this on xarray==0.21.1 and dask==2021.10.0/dask==2021.2.1, and did the following steps:

1. Downloaded a ~100MiB netcdf file from the above thredds server ([here](https://thredds.met.no/thredds/catalog/metpplatest/catalog.html?dataset=metpplatest/met_analysis_1_0km_nordic_latest.nc)].
2. Ran the snippets from both [this comment](https://github.com/dask/dask/issues/8753#issuecomment-1058158881) and [this comment](https://github.com/dask/dask/issues/8753#issuecomment-1058133329).
3. Tried the above with both `xr.open_dataset` and `xr.open_mfdataset`.

In no case did I see anything close to 20-30 seconds to compute, everything was in the tens of ms range. Is there perhaps something I've missed in trying to reproduce this?

There *was* a change in how `da.from_array` constructed it's graphs recently (#7417), so it wouldn't be hugely surprising to find some downstream regressions we need to address. The difference in number of HLG layers that @jsignell found is expected, but the final low-level graph should be fairly similar. But without being able to reproduce, it's hard for me to say for sure.

I think the 20-30 seconds was for more than one file. I was seeing ~2 seconds and ~50ms for one file with dask main and dask 2021.10.0 respectively. 
With both versions I'm seeing ~50-100 ms. What scheduler are you using? If it's multiprocessing or distributed, there may be some undesirable inter-process communication of the array that's being triggered.
I didn't do any scheduler setup, so just using the default.
A quick update based on some investigations that @jsignell and I carried out earlier today:

We are indeed able to reproduce a performance regression for this example, and it is related to #7417. That PR changed `from_array` to use `Blockwise`, which in theory allows for higher-level optimizations and cheaper graph communication and construction. But the introduction of `Blockwise` prevented one type of low-level optimization: slicing fusion.

This is because fusing adjacent slices relies on checking whether each task is "get-ish": https://github.com/dask/dask/blob/02f38865939ac4775ba3f34789bfd3c5e915eebe/dask/array/optimization.py#L130-L138 

With the old low-level-graph, the `getter` callable was directly available on the task, and this comparison was easy. But in blockwise, the callables are obfuscated within a `SubgraphCallable`, and the optimizer was no longer able to identify that task as get-ish, and the optimization did not take place.

I'm still cooking up a reproducer that does not rely on xarray, and will post it here when I have it.

There are (at least) two fixes that could be made here:
1. The long term, correct fix is to convert slicing to be a HighLevelGraph Layer (see some early work in #7655), and make sure that it can talk well with `Blockwise`. It may even make sense to convert `da.from_array` from blockwise to whatever slicing layer we come up with. But that design discussion is not particularly close right now.
2. A short term, hacky fix is to make sure that the slicing fusion can recognize a blockwise subgraph callable as a getter. This would be ugly, and involve introspecting `SubgraphCallable`s, making it a very leaky abstraction. But I think it would be fairly easy to do, as long as we put big blinking disclaimers that it should be removed as soon as high-level-slicing is done.
 
Okay, here is a reproducer with just dask:

```python
import numpy
import dask.array as da

# Use an ndarray subclass to avoid special case for
# numpy arrays. Lazy arrays like xarray datasets
# already satisfy this
class MyArray(numpy.ndarray):
    pass

x = numpy.random.uniform(size=(1000, 1000)).view(MyArray)



xx = da.from_array(x, chunks=100)
yy = xx[23:24, 65:66]  # Slice from one chunk, which can be fused with from_array
dsk = yy.__dask_optimize__(yy.dask, yy.__dask_keys__())
print(dsk)
```

On `dask==2021.10.0` this produces
```python
{('getitem-482c4f7d051bab02a4e5de435b6635d3',
  0,
  0): ('array-getitem-482c4f7d051bab02a4e5de435b6635d3', 0, 0),
 'array-original-0043e9339edbf7b18e0dffa2354ca805': MyArray([[0.75496145, 0.71040058, 0.26402173, ..., 0.89002368,
           0.47585329, 0.96497852],
          [0.5823307 , 0.07175467, 0.14595467, ..., 0.63779591,
           0.67730445, 0.71338149],
          [0.42915523, 0.4993591 , 0.29309595, ..., 0.60710493,
           0.43123075, 0.52111913],
          ...,
          [0.0385207 , 0.20286531, 0.14720955, ..., 0.45732608,
           0.5083299 , 0.76006257],
          [0.72467882, 0.9119804 , 0.60712482, ..., 0.9274443 ,
           0.05040048, 0.96039124],
          [0.12749118, 0.38615043, 0.72269677, ..., 0.29804572,
           0.18775866, 0.3715634 ]]),
 ('array-getitem-482c4f7d051bab02a4e5de435b6635d3',
  0,
  0): (<function dask.array.core.getter(a, b, asarray=True, lock=None)>, 'array-original-0043e9339edbf7b18e0dffa2354ca805', (slice(23, 24, None),
   slice(65, 66, None)), False, False)}
```
whereas on `main` this produces
```python
{('getitem-8565cad6f854187d2bcb8c57560c230a',
  0,
  0): ('array-getitem-8565cad6f854187d2bcb8c57560c230a', 0, 0),
 'original-array-7600483ec382d2244bea4a0da58d8596': MyArray([[0.57031275, 0.07244231, 0.35438451, ..., 0.23643452,
           0.2705955 , 0.23255325],
          [0.53274312, 0.50732558, 0.79660538, ..., 0.1799429 ,
           0.63057926, 0.41924305],
          [0.98659251, 0.06321354, 0.29775424, ..., 0.660547  ,
           0.072433  , 0.73981265],
          ...,
          [0.16659373, 0.51461264, 0.74210757, ..., 0.40877502,
           0.44473356, 0.76993016],
          [0.33193714, 0.37566295, 0.84591651, ..., 0.21424494,
           0.91305219, 0.6167193 ],
          [0.64256872, 0.41688069, 0.3727772 , ..., 0.55530654,
           0.28894569, 0.38069073]]),
 ('array-getitem-8565cad6f854187d2bcb8c57560c230a',
  0,
  0): (<function dask.array.chunk.getitem(obj, index)>, (subgraph_callable-4b804397-1337-48c0-835e-73857e94bb0f,
   'original-array-7600483ec382d2244bea4a0da58d8596',
   (slice(0, 100, None), slice(0, 100, None))), (slice(23, 24, 1),
   slice(65, 66, 1)))}
```

Notice how in the first case the slice operation is fused with the original read from disk (effectively pushing it into the storage layer), but in the second case the slice operation is kept separate.

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -283,14 +283,14 @@ def graph_from_arraylike(
         and has_keyword(getitem, "lock")
         and (not asarray or lock)
     ):
-        getter = partial(getitem, asarray=asarray, lock=lock)
+        kwargs = {"asarray": asarray, "lock": lock}
     else:
         # Common case, drop extra parameters
-        getter = getitem
+        kwargs = {}
 
     if inline_array:
         layer = core_blockwise(
-            getter,
+            getitem,
             name,
             out_ind,
             arr,
@@ -298,6 +298,7 @@ def graph_from_arraylike(
             ArraySliceDep(chunks),
             out_ind,
             numblocks={},
+            **kwargs,
         )
         return HighLevelGraph.from_collections(name, layer)
     else:
@@ -314,6 +315,7 @@ def graph_from_arraylike(
             ArraySliceDep(chunks),
             out_ind,
             numblocks={},
+            **kwargs,
         )
 
         deps = {
diff --git a/dask/array/optimization.py b/dask/array/optimization.py
--- a/dask/array/optimization.py
+++ b/dask/array/optimization.py
@@ -1,5 +1,8 @@
+from __future__ import annotations
+
 from itertools import zip_longest
 from numbers import Integral
+from typing import Any, Callable
 
 import numpy as np
 
@@ -9,7 +12,7 @@
 from dask.blockwise import fuse_roots, optimize_blockwise
 from dask.core import flatten, reverse_dict
 from dask.highlevelgraph import HighLevelGraph
-from dask.optimization import fuse, inline_functions
+from dask.optimization import SubgraphCallable, fuse, inline_functions
 from dask.utils import ensure_dict
 
 # All get* functions the optimizations know about
@@ -98,14 +101,14 @@ def hold_keys(dsk, dependencies):
             # when there's either more than one dependent, or the dependent is
             # no longer a get* function or an alias. We then add the final
             # key to the list of keys not to fuse.
-            if type(task) is tuple and task and task[0] in GETTERS:
+            if _is_getter_task(task):
                 try:
                     while len(dependents[dep]) == 1:
                         new_dep = next(iter(dependents[dep]))
                         new_task = dsk[new_dep]
                         # If the task is a get* or an alias, continue up the
                         # linear chain
-                        if new_task[0] in GETTERS or new_task in dsk:
+                        if _is_getter_task(new_task) or new_task in dsk:
                             dep = new_dep
                         else:
                             break
@@ -115,6 +118,49 @@ def hold_keys(dsk, dependencies):
     return hold_keys
 
 
+def _is_getter_task(
+    value,
+) -> tuple[Callable, Any, Any, bool, bool | None] | None:
+    """Check if a value in a Dask graph looks like a getter.
+
+    1. Is it a tuple with the first element a known getter.
+    2. Is it a SubgraphCallable with a single element in its
+       dsk which is a known getter.
+
+    If a getter is found, it returns a tuple with (getter, array, index, asarray, lock).
+    Otherwise it returns ``None``.
+
+    TODO: the second check is a hack to allow for slice fusion between tasks produced
+    from blockwise layers and slicing operations. Once slicing operations have
+    HighLevelGraph layers which can talk to Blockwise layers this check *should* be
+    removed, and we should not have to introspect SubgraphCallables.
+    """
+    if type(value) is not tuple:
+        return None
+    first = value[0]
+    get = None
+    if first in GETTERS:
+        get = first
+    # We only accept SubgraphCallables with a single sub-task right now as it's
+    # not clear which task to inspect if there is more than one, or how to resolve
+    # conflicts if they occur.
+    elif isinstance(first, SubgraphCallable) and len(first.dsk) == 1:
+        v = next(iter(first.dsk.values()))
+        if type(v) is tuple and len(v) > 1 and v[0] in GETTERS:
+            get = v[0]
+    if get is None:  # Didn't find a getter
+        return None
+
+    length = len(value)
+    if length == 3:
+        # getter defaults to asarray=True, getitem is semantically False
+        return get, value[1], value[2], get is not getitem, None
+    elif length == 5:
+        return get, *value[1:]
+
+    return None
+
+
 def optimize_slices(dsk):
     """Optimize slices
 
@@ -127,21 +173,11 @@ def optimize_slices(dsk):
     fancy_ind_types = (list, np.ndarray)
     dsk = dsk.copy()
     for k, v in dsk.items():
-        if type(v) is tuple and v[0] in GETTERS and len(v) in (3, 5):
-            if len(v) == 3:
-                get, a, a_index = v
-                # getter defaults to asarray=True, getitem is semantically False
-                a_asarray = get is not getitem
-                a_lock = None
-            else:
-                get, a, a_index, a_asarray, a_lock = v
-            while type(a) is tuple and a[0] in GETTERS and len(a) in (3, 5):
-                if len(a) == 3:
-                    f2, b, b_index = a
-                    b_asarray = f2 is not getitem
-                    b_lock = None
-                else:
-                    f2, b, b_index, b_asarray, b_lock = a
+        if a_task := _is_getter_task(v):
+            get, a, a_index, a_asarray, a_lock = a_task
+
+            while b_task := _is_getter_task(a):
+                f2, b, b_index, b_asarray, b_lock = b_task
 
                 if a_lock and a_lock is not b_lock:
                     break


Test patch:
diff --git a/dask/array/tests/test_optimization.py b/dask/array/tests/test_optimization.py
--- a/dask/array/tests/test_optimization.py
+++ b/dask/array/tests/test_optimization.py
@@ -6,21 +6,107 @@
 
 import dask
 import dask.array as da
-from dask.array.core import getter, getter_nofancy
+from dask.array.chunk import getitem as da_getitem
+from dask.array.core import getter as da_getter
+from dask.array.core import getter_nofancy as da_getter_nofancy
 from dask.array.optimization import (
+    _is_getter_task,
     fuse_slice,
-    getitem,
     optimize,
     optimize_blockwise,
     optimize_slices,
 )
 from dask.array.utils import assert_eq
 from dask.highlevelgraph import HighLevelGraph
-from dask.optimization import fuse
+from dask.optimization import SubgraphCallable, fuse
 from dask.utils import SerializableLock
 
 
-def test_fuse_getitem():
+def _wrap_getter(func, wrap):
+    """
+    Getters generated from a Blockwise layer might be wrapped in a SubgraphCallable.
+    Make sure that the optimization functions can still work if that is the case.
+    """
+    if wrap:
+        return SubgraphCallable({"key": (func, "index")}, outkey="key", inkeys="index")
+    else:
+        return func
+
+
+@pytest.fixture(params=[True, False])
+def getter(request):
+    """
+    Parameterized fixture for dask.array.core.getter both alone (False)
+    and wrapped in a SubgraphCallable (True).
+    """
+    yield _wrap_getter(da_getter, request.param)
+
+
+@pytest.fixture(params=[True, False])
+def getitem(request):
+    """
+    Parameterized fixture for dask.array.chunk.getitem both alone (False)
+    and wrapped in a SubgraphCallable (True).
+    """
+    yield _wrap_getter(da_getitem, request.param)
+
+
+@pytest.fixture(params=[True, False])
+def getter_nofancy(request):
+    """
+    Parameterized fixture for dask.array.chunk.getter_nofancy both alone (False)
+    and wrapped in a SubgraphCallable (True).
+    """
+    yield _wrap_getter(da_getter_nofancy, request.param)
+
+
+def _check_get_task_eq(a, b) -> bool:
+    """
+    Check that two tasks (possibly containing nested tasks) are equal, where
+    equality is lax by allowing the callable in a SubgraphCallable to be the same
+    as a non-wrapped task.
+    """
+    if len(a) < 1 or len(a) != len(b):
+        return False
+
+    a_callable = (
+        list(a[0].dsk.values())[0][0] if isinstance(a[0], SubgraphCallable) else a[0]
+    )
+    b_callable = (
+        list(b[0].dsk.values())[0][0] if isinstance(b[0], SubgraphCallable) else b[0]
+    )
+    if a_callable != b_callable:
+        return False
+
+    for ae, be in zip(a[1:], b[1:]):
+        if dask.core.istask(ae):
+            if not _check_get_task_eq(ae, be):
+                return False
+        elif ae != be:
+            return False
+    return True
+
+
+def _assert_getter_dsk_eq(a, b):
+    """
+    Compare two getter dsks.
+
+    TODO: this is here to support the fact that low-level array slice fusion needs to be
+    able to introspect slicing tasks. But some slicing tasks (e.g. `from_array`) could
+    be hidden within SubgraphCallables. This and _check_get_task_eq should be removed
+    when high-level slicing lands, and replaced with basic equality checks.
+    """
+    assert a.keys() == b.keys()
+    for k, av in a.items():
+        bv = b[k]
+        if dask.core.istask(av):
+            assert _check_get_task_eq(av, bv)
+        else:
+            assert av == bv
+
+
+def test_fuse_getitem(getter, getter_nofancy, getitem):
+
     pairs = [
         (
             (getter, (getter, "x", slice(1000, 2000)), slice(15, 20)),
@@ -105,10 +191,10 @@ def test_fuse_getitem():
 
     for inp, expected in pairs:
         result = optimize_slices({"y": inp})
-        assert result == {"y": expected}
+        _assert_getter_dsk_eq(result, {"y": expected})
 
 
-def test_fuse_getitem_lock():
+def test_fuse_getitem_lock(getter, getter_nofancy, getitem):
     lock1 = SerializableLock()
     lock2 = SerializableLock()
 
@@ -159,10 +245,10 @@ def test_fuse_getitem_lock():
 
     for inp, expected in pairs:
         result = optimize_slices({"y": inp})
-        assert result == {"y": expected}
+        _assert_getter_dsk_eq(result, {"y": expected})
 
 
-def test_optimize_with_getitem_fusion():
+def test_optimize_with_getitem_fusion(getter):
     dsk = {
         "a": "some-array",
         "b": (getter, "a", (slice(10, 20), slice(100, 200))),
@@ -171,11 +257,11 @@ def test_optimize_with_getitem_fusion():
 
     result = optimize(dsk, ["c"])
     expected_task = (getter, "some-array", (15, slice(150, 160)))
-    assert any(v == expected_task for v in result.values())
+    assert any(_check_get_task_eq(v, expected_task) for v in result.values())
     assert len(result) < len(dsk)
 
 
-def test_optimize_slicing():
+def test_optimize_slicing(getter):
     dsk = {
         "a": (range, 10),
         "b": (getter, "a", (slice(None, None, None),)),
@@ -186,7 +272,7 @@ def test_optimize_slicing():
 
     expected = {"e": (getter, (range, 10), (slice(0, 5, None),))}
     result = optimize_slices(fuse(dsk, [], rename_keys=False)[0])
-    assert result == expected
+    _assert_getter_dsk_eq(result, expected)
 
     # protect output keys
     expected = {
@@ -196,7 +282,7 @@ def test_optimize_slicing():
     }
     result = optimize_slices(fuse(dsk, ["c", "d", "e"], rename_keys=False)[0])
 
-    assert result == expected
+    _assert_getter_dsk_eq(result, expected)
 
 
 def test_fuse_slice():
@@ -255,11 +341,11 @@ def test_nonfusible_fancy_indexing():
             fuse_slice(a, b)
 
 
-def test_hard_fuse_slice_cases():
+def test_hard_fuse_slice_cases(getter):
     dsk = {
         "x": (getter, (getter, "x", (None, slice(None, None))), (slice(None, None), 5))
     }
-    assert optimize_slices(dsk) == {"x": (getter, "x", (None, 5))}
+    _assert_getter_dsk_eq(optimize_slices(dsk), {"x": (getter, "x", (None, 5))})
 
 
 def test_dont_fuse_numpy_arrays():
@@ -271,7 +357,7 @@ def test_dont_fuse_numpy_arrays():
         assert sum(isinstance(v, np.ndarray) for v in dsk.values()) == 1
 
 
-def test_fuse_slices_with_alias():
+def test_fuse_slices_with_alias(getter, getitem):
     dsk = {
         "x": np.arange(16).reshape((4, 4)),
         ("dx", 0, 0): (getter, "x", (slice(0, 4), slice(0, 4))),
@@ -282,10 +368,10 @@ def test_fuse_slices_with_alias():
     dsk2 = optimize(dsk, keys)
     assert len(dsk2) == 3
     fused_key = (dsk2.keys() - {"x", ("dx2", 0)}).pop()
-    assert dsk2[fused_key] == (getter, "x", (slice(0, 4), 0))
+    assert _check_get_task_eq(dsk2[fused_key], (getter, "x", (slice(0, 4), 0)))
 
 
-def test_dont_fuse_fancy_indexing_in_getter_nofancy():
+def test_dont_fuse_fancy_indexing_in_getter_nofancy(getitem, getter_nofancy):
     dsk = {
         "a": (
             getitem,
@@ -293,10 +379,10 @@ def test_dont_fuse_fancy_indexing_in_getter_nofancy():
             ([1, 3], slice(50, 60, None)),
         )
     }
-    assert optimize_slices(dsk) == dsk
+    _assert_getter_dsk_eq(optimize_slices(dsk), dsk)
 
     dsk = {"a": (getitem, (getter_nofancy, "x", [1, 2, 3]), 0)}
-    assert optimize_slices(dsk) == dsk
+    _assert_getter_dsk_eq(optimize_slices(dsk), dsk)
 
 
 @pytest.mark.parametrize("chunks", [10, 5, 3])
@@ -311,7 +397,7 @@ def test_fuse_getter_with_asarray(chunks):
         assert s.count("getitem") + s.count("getter") <= 1
         if v is not x:
             assert "1234567890" not in s
-    n_getters = len([v for v in dsk.values() if v[0] in (getitem, getter)])
+    n_getters = len([v for v in dsk.values() if _is_getter_task(v)])
     if y.npartitions > 1:
         assert n_getters == y.npartitions
     else:
@@ -320,30 +406,48 @@ def test_fuse_getter_with_asarray(chunks):
     assert_eq(z, x + 1)
 
 
-@pytest.mark.parametrize(
-    "get,remove", [(getter, False), (getter_nofancy, False), (getitem, True)]
-)
-def test_remove_no_op_slices_if_get_is_not_getter_or_getter_nofancy(get, remove):
-    # Test that no-op slices are removed as long as get is not getter or
+def test_remove_no_op_slices_for_getitem(getitem):
+    null = slice(0, None)
+    opts = [
+        ((getitem, "x", null, False, False), "x"),
+        ((getitem, (getitem, "x", null, False, False), null), "x"),
+        ((getitem, (getitem, "x", (null, null), False, False), ()), "x"),
+    ]
+    for orig, final in opts:
+        _assert_getter_dsk_eq(optimize_slices({"a": orig}), {"a": final})
+
+
+@pytest.mark.parametrize("which", ["getter", "getter_nofancy"])
+def test_dont_remove_no_op_slices_for_getter_or_getter_nofancy(
+    which, getitem, getter, getter_nofancy
+):
+    # Test that no-op slices are *not* removed when using getter or
     # getter_nofancy. This ensures that `get` calls are always made in all
     # tasks created by `from_array`, even after optimization
+
+    # Pytest doesn't make it easy to parameterize over parameterized fixtures
+    if which == "getter":
+        get = getter
+    else:
+        get = getter_nofancy
+
     null = slice(0, None)
     opts = [
         (
             (get, "x", null, False, False),
-            "x" if remove else (get, "x", null, False, False),
+            (get, "x", null, False, False),
         ),
         (
             (getitem, (get, "x", null, False, False), null),
-            "x" if remove else (get, "x", null, False, False),
+            (get, "x", null, False, False),
         ),
         (
             (getitem, (get, "x", (null, null), False, False), ()),
-            "x" if remove else (get, "x", (null, null), False, False),
+            (get, "x", (null, null), False, False),
         ),
     ]
     for orig, final in opts:
-        assert optimize_slices({"a": orig}) == {"a": final}
+        _assert_getter_dsk_eq(optimize_slices({"a": orig}), {"a": final})
 
 
 @pytest.mark.xfail(reason="blockwise fusion does not respect this, which is ok")