Repository: pydata/xarray

Issue:
Implicit use of dask feature
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples: https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
I tried to use the `to_netcdf` function to store a dataset into a NetCDF file, but the following exception was raised
```
Traceback (most recent call last):
  File "dask-error.py", line 27, in <module>
    ds.to_netcdf("test.nc")
  File "/home/sam/dev/xarray-test/.venv/lib/python3.8/site-packages/xarray/core/dataset.py", line 1544, in to_netcdf
    return to_netcdf(
  File "/home/sam/dev/xarray-test/.venv/lib/python3.8/site-packages/xarray/backends/api.py", line 1051, in to_netcdf
    scheduler = _get_scheduler()
  File "/home/sam/dev/xarray-test/.venv/lib/python3.8/site-packages/xarray/backends/locks.py", line 79, in _get_scheduler
    actual_get = dask.base.get_scheduler(get, collection)
AttributeError: module 'dask' has no attribute 'base'
```
This code sample works perfectly as expected when the dask package is not installed in the environment, and the method works as expected. However, we dask is installed the `_get_scheduler` function is called and produces the error (this can be found here) https://github.com/pydata/xarray/blob/b9e6a36ff7a0ca3593165cf191f4152666fa4a66/xarray/backends/locks.py#L79

After a little digging through, the problem is that the `base` module in the dask package depends on the toolz package, which is not a default dependency of dask and so causes a silent import failure when dask initialises its namespace (https://github.com/dask/dask/blob/416d348f7174a302815758cb87dbf6983226ddc5/dask/__init__.py#L10). As a result, the base package is not importable form the dask top level, and importing it separately gives as follows
```
from dask import base
```
raises a ModuleNotFoundError.
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sam/dev/xarray-test/.venv/lib/python3.8/site-packages/dask/base.py", line 13, in <module>
    from tlz import merge, groupby, curry, identity
ModuleNotFoundError: No module named 'tlz'
```
I recommend the following fix. At the following line in the `_get_scheduler` function
https://github.com/pydata/xarray/blob/b9e6a36ff7a0ca3593165cf191f4152666fa4a66/xarray/backends/locks.py#L75
replace the import with the following
```
from dask.base import get_scheduler
```
and remove `dask.base` from the later call.

I should, however, point out that `get_scheduler` does not appear to be part of the Dask public API.





**What you expected to happen**:
The `to_netcdf` method should have exited silently and created a new file in the working directory with the contents of the data set.

**Minimal Complete Verifiable Example**:
This code is basically the "Toy weather data" example from the documentation, except for the last line.
```python
import numpy as np
import pandas as pd

import xarray as xr

np.random.seed(123)

xr.set_options(display_style="html")

times = pd.date_range("2000-01-01", "2001-12-31", name="time")
annual_cycle = np.sin(2 * np.pi * (times.dayofyear.values / 365.25 - 0.28))

base = 10 + 15 * annual_cycle.reshape(-1, 1)
tmin_values = base + 3 * np.random.randn(annual_cycle.size, 3)
tmax_values = base + 10 + 3 * np.random.randn(annual_cycle.size, 3)

ds = xr.Dataset(
    {
        "tmin": (("time", "location"), tmin_values),
        "tmax": (("time", "location"), tmax_values),
    },
    {"time": times, "location": ["IA", "IN", "IL"]},
)

ds.to_netcdf("test.nc") ## error here
```

**Anything else we need to know?**:
As mentioned above, the error on manifests when the dask package with no extras installed is present in the environment. (Many of the extras require the toolz package, at which time the import error goes away.)

**Environment**:
In a clean virtual environment, install the following packages.
```
pip install xarray netCDF4 dask
```
The package versions installed are as followed (generated by `pip freeze`):
```
cftime==1.1.3
dask==2.18.1
netCDF4==1.5.3
numpy==1.18.5
pandas==1.0.5
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
six==1.15.0
xarray==0.15.1
```
(Also running python3.8.2 on Debian Linux, not that I suppose this matters.)

<details><summary>Output of <tt>xr.show_versions()</tt></summary>
INSTALLED VERSIONS
------------------
commit: None
python: 3.8.2+ (heads/3.8:882a7f44da, Apr 26 2020, 19:31:38) 
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 5.4.0-37-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.3

xarray: 0.15.1
pandas: 1.0.5
numpy: 1.18.5
scipy: None
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.1.3
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.18.1
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
setuptools: 41.2.0
pip: 19.2.3
conda: None
pytest: None
IPython: None
sphinx: None
<!-- Paste the output here xr.show_versions() here -->
</details>




Discussion:

Patch:
diff --git a/doc/internals.rst b/doc/internals.rst
index 46c117e3..b158f12a 100644
--- a/doc/internals.rst
+++ b/doc/internals.rst
@@ -42,6 +42,38 @@ xarray objects via the (readonly) :py:attr:`Dataset.variables
 <xarray.Dataset.variables>` and
 :py:attr:`DataArray.variable <xarray.DataArray.variable>` attributes.
 
+Duck arrays
+-----------
+
+.. warning::
+
+    This is a experimental feature.
+
+xarray can wrap custom `duck array`_ objects as long as they define numpy's
+``shape``, ``dtype`` and ``ndim`` properties and the ``__array__``,
+``__array_ufunc__`` and ``__array_function__`` methods.
+
+In certain situations (e.g. when printing the collapsed preview of
+variables of a ``Dataset``), xarray will display the repr of a `duck array`_
+in a single line, truncating it to a certain number of characters. If that
+would drop too much information, the `duck array`_ may define a
+``_repr_inline_`` method that takes ``max_width`` (number of characters) as an
+argument:
+
+.. code:: python
+
+    class MyDuckArray:
+        ...
+
+        def _repr_inline_(self, max_width):
+            """ format to a single line with at most max_width characters """
+            ...
+
+        ...
+
+.. _duck array: https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html
+
+
 Extending xarray
 ----------------
 
diff --git a/doc/whats-new.rst b/doc/whats-new.rst
index 81ef2cd9..c4cf931b 100644
--- a/doc/whats-new.rst
+++ b/doc/whats-new.rst
@@ -29,10 +29,16 @@ New Features
   property for :py:class:`CFTimeIndex` and show ``calendar`` and ``length`` in
   :py:meth:`CFTimeIndex.__repr__` (:issue:`2416`, :pull:`4092`)
   `Aaron Spring <https://github.com/aaronspring>`_.
+- Use a wrapped array's ``_repr_inline_`` method to construct the collapsed ``repr``
+  of :py:class:`DataArray` and :py:class:`Dataset` objects and
+  document the new method in :doc:`internals`. (:pull:`4248`).
+  By `Justus Magin <https://github.com/keewis>`_.
 
 
 Bug fixes
 ~~~~~~~~~
+- Fixed a bug in backend caused by basic installation of Dask (:issue:`4164`, :pull:`4318`)
+  `Sam Morley <https://github.com/inakleinbottle>`_.
 
 
 Documentation
diff --git a/xarray/backends/locks.py b/xarray/backends/locks.py
index 435690f2..bb876a43 100644
--- a/xarray/backends/locks.py
+++ b/xarray/backends/locks.py
@@ -72,12 +72,15 @@ def _get_scheduler(get=None, collection=None) -> Optional[str]:
     dask.base.get_scheduler
     """
     try:
-        import dask  # noqa: F401
+        # Fix for bug caused by dask installation that doesn't involve the toolz library
+        # Issue: 4164
+        import dask
+        from dask.base import get_scheduler  # noqa: F401
+
+        actual_get = get_scheduler(get, collection)
     except ImportError:
         return None
 
-    actual_get = dask.base.get_scheduler(get, collection)
-
     try:
         from dask.distributed import Client
 
diff --git a/xarray/core/formatting.py b/xarray/core/formatting.py
index 28eaae5f..9aa20f2b 100644
--- a/xarray/core/formatting.py
+++ b/xarray/core/formatting.py
@@ -261,6 +261,8 @@ def inline_variable_array_repr(var, max_width):
         return inline_dask_repr(var.data)
     elif isinstance(var._data, sparse_array_type):
         return inline_sparse_repr(var.data)
+    elif hasattr(var._data, "_repr_inline_"):
+        return var._data._repr_inline_(max_width)
     elif hasattr(var._data, "__array_function__"):
         return maybe_truncate(repr(var._data).replace("\n", " "), max_width)
     else:


Test patch:
diff --git a/xarray/tests/test_formatting.py b/xarray/tests/test_formatting.py
index 82de8080..1cc91266 100644
--- a/xarray/tests/test_formatting.py
+++ b/xarray/tests/test_formatting.py
@@ -7,6 +7,7 @@ import pytest
 
 import xarray as xr
 from xarray.core import formatting
+from xarray.core.npcompat import IS_NEP18_ACTIVE
 
 from . import raises_regex
 
@@ -391,6 +392,44 @@ class TestFormatting:
         assert actual == expected
 
 
+@pytest.mark.skipif(not IS_NEP18_ACTIVE, reason="requires __array_function__")
+def test_inline_variable_array_repr_custom_repr():
+    class CustomArray:
+        def __init__(self, value, attr):
+            self.value = value
+            self.attr = attr
+
+        def _repr_inline_(self, width):
+            formatted = f"({self.attr}) {self.value}"
+            if len(formatted) > width:
+                formatted = f"({self.attr}) ..."
+
+            return formatted
+
+        def __array_function__(self, *args, **kwargs):
+            return NotImplemented
+
+        @property
+        def shape(self):
+            return self.value.shape
+
+        @property
+        def dtype(self):
+            return self.value.dtype
+
+        @property
+        def ndim(self):
+            return self.value.ndim
+
+    value = CustomArray(np.array([20, 40]), "m")
+    variable = xr.Variable("x", value)
+
+    max_width = 10
+    actual = formatting.inline_variable_array_repr(variable, max_width=10)
+
+    assert actual == value._repr_inline_(max_width)
+
+
 def test_set_numpy_options():
     original_options = np.get_printoptions()
     with formatting.set_numpy_options(threshold=10):