Repository: napari/napari

Issue:
When viewer.add_image() is called on a very large float array without specifying contrast_limits, the data range calculation overwhelms system resources.
## 🐛 Bug

When viewer.add_image() is called on a very large float array without specifying contrast_limits, the data range calculation overwhelms system resources.

## To Reproduce

Steps to reproduce the behavior:

The following script lazily generates a very large array. Wrapper function can be used to count the number of array-generating calls.
```
import dask.array as da
from dask import delayed
import numpy as np
import napari
import itertools

def counter(func, counter_dict={}):
    counter_dict[func]=0
    def wrap(*args,**kwargs):
        counter_dict[func] += 1
        print("Number of times {} has been called so far {}".format(func.__name__, counter_dict[func]))
        return func(*args,**kwargs)
    return wrap

@counter
def get_random_array():
    return np.random.randint(256, size = (1080,1920), dtype = np.uint8).astype(np.float64)

sample = get_random_array()
lazy_random_array = delayed(get_random_array)
lazy_arrays = [lazy_random_array() for i in range(2000)]
dask_arrays = [da.from_delayed(lazy_array, shape = sample.shape, dtype = sample.dtype) for lazy_array in lazy_arrays]
dask_stack = da.stack(dask_arrays, axis = 0)

viewer = napari.Viewer()
layer = viewer.add_image(data = dask_stack)

napari.run()
```
The above code results in many hundreds of calls to get_random_array() and overwhelms system memory/CPU.
The likely culprit is the layer_utils calc_data_range() function, which nominally checks for array size before performing the min/max calculation. However, in the case of script above the entire array seems to be evaluated when deriving contrast limits.

## Expected behavior

Correctly prevent evaluation of excessively large arrays for contrast limits check.

## Environment

napari: 0.4.9
Platform: Windows-10-10.0.19041-SP0
Python: 3.8.10 | packaged by conda-forge | (default, May 11 2021, 06:25:23) [MSC v.1916 64 bit (AMD64)]
Qt: 5.12.9
PyQt5: 5.12.3
NumPy: 1.20.2
SciPy: 1.6.3
Dask: 2021.05.0
VisPy: 0.6.6

OpenGL:
- GL version: 4.6.0 NVIDIA 462.31
- MAX_TEXTURE_SIZE: 32768

Screens:
- screen 1: resolution 2560x1440, scale 1.0
- screen 2: resolution 1440x2560, scale 1.0

Plugins:
- console: 0.0.3
- scikit-image: 0.4.9.dev8+g43a604be
- svg: 0.1.5



Discussion:
This issue has been mentioned on **Image.sc Forum**. There might be relevant details there:

https://forum.image.sc/t/failing-to-implement-lazy-dask-delayed-image-processing-pipeline-from-tutorial/54007/5

Thanks @Remote-Radical ... I think there's been an issue elsewhere that we shouldn't call `calc_data_range` when the provided data is a dask array.  in any case, I agree.  should be an easy fix.
Yeah, this function used to give us more problems, i thought we had it improved now, but i guess not.

For anyone else coming across this, as a temporary work around you can just explicitly pass the contrast limits to avoid the computation, but we should fix this too.

```python
layer = viewer.add_image(data = dask_stack, contrast_limits=[0, 255])
```

Maybe setting the default contrast limit to the maximum values allowed by the datatype for large images/dask arrays would be a possible solution?
> Maybe setting the default contrast limit to the maximum values allowed by the datatype for large images/dask arrays would be a possible solution?

Thanks for the suggestion @jo-mueller! We already do this for uint8 data. For float data it is clearly problematic. For uint16, it's also frequently a problem because 12-bit data in [0, 4096) is often packed into a uint16, resulting in vastly reduced contrast. But yes, we discussed this at the dev meeting and we are going to turn off auto-computation of contrast limits for anything that's not a NumPy array. Instead we will make it easier to reset the contrast limits based on in-memory data by adding a "Reset" button to the contrast limits slider.
Sorry for hijacking this, but since this seems to be w.i.p. already: IMHO, it would be good to mention in the docs whether the auto-computation is performed on, e.g., a per-channel basis or for the full (T/Z/C) image stack (thus sharing the same contrast limits for among all channels).
@jwindhager agreed - and just in case anyone coming here is wondering what the behaviour is, it is performed when each layer is added separately, and so if you make three calls to

```python
viewer.add_image(data_c1)
viewer.add_image(data_c2)
viewer.add_image(data_c3)
```
you'll get 3 layers and it will happen 3 times. Or

```python
viewer.add_image(data_all_c, channel_axis=0)
```
you'll get 3 layers and it will happen 3 times. But
```python
viewer.add_image(data_all_c)
```
you'll only get one layer and it will just happen once, but now all three channels will use same colormap and so be forced to have some contrast_limits.

Resolving patch:
diff --git a/napari/layers/image/image.py b/napari/layers/image/image.py
--- a/napari/layers/image/image.py
+++ b/napari/layers/image/image.py
@@ -10,6 +10,7 @@
 from scipy import ndimage as ndi
 
 from ...utils import config
+from ...utils._dtype import get_dtype_limits
 from ...utils.colormaps import AVAILABLE_COLORMAPS
 from ...utils.events import Event
 from ...utils.translations import trans
@@ -292,7 +293,14 @@ def __init__(
         )
         self._experimental_clipping_planes = ClippingPlaneList()
         if contrast_limits is None:
-            self.contrast_limits_range = self._calc_data_range()
+            if not isinstance(data, np.ndarray):
+                dtype = getattr(data, 'dtype', None)
+                if np.issubdtype(dtype, np.integer):
+                    self.contrast_limits_range = get_dtype_limits(dtype)
+                else:
+                    self.contrast_limits_range = (0, 1)
+            else:
+                self.contrast_limits_range = self._calc_data_range()
         else:
             self.contrast_limits_range = contrast_limits
         self._contrast_limits = tuple(self.contrast_limits_range)


Test patch:
diff --git a/napari/_qt/_tests/test_qt_notifications.py b/napari/_qt/_tests/test_qt_notifications.py
--- a/napari/_qt/_tests/test_qt_notifications.py
+++ b/napari/_qt/_tests/test_qt_notifications.py
@@ -154,6 +154,7 @@ def test_notification_error(mock_show, monkeypatch, clean_current):
     mock_show.assert_called_once()
 
 
+@pytest.mark.sync_only
 @pytest.mark.skipif(PY37_OR_LOWER, reason="Fails on py37")
 def test_notifications_error_with_threading(make_napari_viewer):
     """Test notifications of `threading` threads, using a dask example."""
diff --git a/napari/layers/_tests/test_dask_layers.py b/napari/layers/_tests/test_dask_layers.py
--- a/napari/layers/_tests/test_dask_layers.py
+++ b/napari/layers/_tests/test_dask_layers.py
@@ -1,3 +1,4 @@
+import os
 from contextlib import contextmanager
 from distutils.version import LooseVersion
 
@@ -9,6 +10,39 @@
 from napari import layers, utils, viewer
 
 
+@pytest.mark.sync_only
+def test_dask_not_greedy():
+    """Make sure that we don't immediately calculate dask arrays."""
+
+    FETCH_COUNT = 0
+    # the min requirements seems to double the number of fetches
+    MINREQ = int(bool(os.environ.get('MIN_REQ'))) + 1
+
+    def get_plane(block_id):
+        if block_id:
+            nonlocal FETCH_COUNT
+            FETCH_COUNT += 1
+        return np.random.rand(1, 1, 1, 10, 10)
+
+    arr = da.map_blocks(
+        get_plane,
+        chunks=((1,) * 4, (1,) * 2, (1,) * 8, (10,), (10,)),
+        dtype=float,
+    )
+    layer = layers.Image(arr)
+    assert FETCH_COUNT == 1 * MINREQ
+    assert tuple(layer.contrast_limits) == (0, 1)
+
+    arr2 = da.map_blocks(
+        get_plane,
+        chunks=((1,) * 4, (1,) * 4, (1,) * 4, (10,), (10,)),
+        dtype='uint8',
+    )
+    layer = layers.Image(arr2)
+    assert FETCH_COUNT == 2 * MINREQ
+    assert tuple(layer.contrast_limits) == (0, 2 ** 8 - 1)
+
+
 def test_dask_array_doesnt_create_cache():
     """Test that dask arrays don't create cache but turns off fusion."""
     # by default we have no dask_cache and task fusion is active