Repository: napari/napari

Issue:
pyramid checking seems to be breaking lazy loading
## 🐛 Bug
I was just trying to familiarize myself with napari's handling of zarr datasets using one of my big light sheet datasets... but it would hang indefinitely each time I added it.

I went back to the example in `zarr_nD_image.py`, but there too, as i increase the size of the zarr store, napari starts to hang upon `napari.view_image(data)`... regardless of whether I provide the `contrast_limits`, `rgb`  or  `is_pyramid=False` arguments.  Digging a bit more, it seems that, as of #556, it is impossible to avoid the callchain from `Image.__init__()` -> `get_pyramid_and_rgb()` -> `is_pyramid()` ... and that `is_pyramid()` call takes forever on a big zarr store.

Is this a regression or is `zarr_nD_image.py` now invoking `napari.view_image` incorrectly?


Discussion:
@tlambert03 this is a known issue, in that @sofroniewn and I did know about it, but it appears I neglected to create an issue for it! :grimacing: Maybe it was on slack, or a comment on a different issue. Sorry about that!

Anyway, the problem is this line:

https://github.com/napari/napari/blob/8ce944a1dff680f4d06b238f7556211550ba1d4b/napari/util/misc.py#L68

`[... for d in data]` actually instantiates each `d` when `data` is a zarr array.

The workaround is to load the zarr file not with zarr.open, but with `da.from_zarr`.

The fix is to change that expression to be lazier. Additionally, `is_pyramid=False` should probably skip the check, but I'm not sure... 
Ok thanks
yeah @tlambert03 this is not great ... any suggestions / improvements would be appreciated. I think it's pretty natural for people to do what you did with `zarr` instead of `da.from_zarr` so we should do better than we do right now
I need to get much more familiar with the pyramid handling (and other general array-handling) parts of napari before I'd be able to make any sort of useful suggestions.  But in the short term, would it make sense to change the zarr example to use `da.from_zarr`?
https://github.com/napari/napari/blob/15deef08f1aea5e6a6951f7a83d4713f16070ed4/examples/zarr_nD_image.py#L14-L21

I also need to look back through #556 to understand why the `if pyramid:` check was removed in [`get_pyramid_and_rgb`](https://github.com/napari/napari/blob/15deef08f1aea5e6a6951f7a83d4713f16070ed4/napari/util/misc.py#L111).  If the user explicitly passes `pyramid=False`, why does `is_pyramid` still need to be called in [this line](https://github.com/napari/napari/blob/15deef08f1aea5e6a6951f7a83d4713f16070ed4/napari/util/misc.py#L138)?
I guess the calling of `is_pyramid` was more of an input check, because it leads to this error if things don't match - https://github.com/napari/napari/blob/15deef08f1aea5e6a6951f7a83d4713f16070ed4/napari/util/misc.py#L167 but maybe that's a bad idea

Agree changing the zarr example to use `da.from_zarr` is a good idea.
I ran into this issue today as well and this also breaks lazy loading for other lazy input sources like `h5py` or `z5py` datasets. 
Maybe leave out this check if `is_pyramid is not None`? This might cause some downstream errors for inputs that are not consistent with the passed pyramid flag though. Or just perform the check for `tuple` and `list` inputs? This would cause issues if someone passes a pyramid format as np.object though.

Resolving patch:
diff --git a/napari/util/misc.py b/napari/util/misc.py
--- a/napari/util/misc.py
+++ b/napari/util/misc.py
@@ -65,7 +65,11 @@ def is_rgb(shape):
 def is_pyramid(data):
     """If shape of arrays along first axis is strictly decreasing.
     """
-    size = np.array([np.prod(d.shape) for d in data])
+    # If the data has ndim and is not one-dimensional then cannot be pyramid
+    if hasattr(data, 'ndim') and data.ndim > 1:
+        return False
+
+    size = np.array([np.prod(d.shape, dtype=np.uint64) for d in data])
     if len(size) > 1:
         return np.all(size[:-1] > size[1:])
     else:


Test patch:
diff --git a/napari/layers/image/tests/test_big_image_timing.py b/napari/layers/image/tests/test_big_image_timing.py
new file mode 100644
--- /dev/null
+++ b/napari/layers/image/tests/test_big_image_timing.py
@@ -0,0 +1,40 @@
+import dask.array as da
+import zarr
+from napari.layers import Image
+import pytest
+
+
+data_dask = da.random.random(
+    size=(100_000, 1000, 1000), chunks=(1, 1000, 1000)
+)
+data_zarr = zarr.zeros((100_000, 1000, 1000))
+
+
+@pytest.mark.timeout(2)
+@pytest.mark.parametrize('data', [data_dask, data_zarr])
+def test_timing_fast_big_dask_all_specified_(data):
+    layer = Image(data, is_pyramid=False, contrast_limits=[0, 1])
+    assert layer.data.shape == data.shape
+
+
+@pytest.mark.timeout(2)
+@pytest.mark.parametrize('data', [data_dask, data_zarr])
+def test_timing_fast_big_dask_is_pyramid_specified(data):
+    layer = Image(data, is_pyramid=False)
+    assert layer.data.shape == data.shape
+
+
+@pytest.mark.skip(reason="currently fails as we try and generate pyramid")
+@pytest.mark.timeout(2)
+@pytest.mark.parametrize('data', [data_dask, data_zarr])
+def test_timing_fast_big_dask_contrast_limits_specified(data):
+    layer = Image(data, contrast_limits=[0, 1])
+    assert layer.data.shape == data.shape
+
+
+@pytest.mark.skip(reason="currently fails as we try and generate pyramid")
+@pytest.mark.timeout(2)
+@pytest.mark.parametrize('data', [data_dask, data_zarr])
+def test_timing_fast_big_dask_nothing_specified(data):
+    layer = Image(data)
+    assert layer.data.shape == data.shape
diff --git a/napari/util/tests/test_misc.py b/napari/util/tests/test_misc.py
--- a/napari/util/tests/test_misc.py
+++ b/napari/util/tests/test_misc.py
@@ -1,4 +1,4 @@
-import time
+import pytest
 import numpy as np
 import dask.array as da
 from skimage.transform import pyramid_gaussian
@@ -14,6 +14,11 @@
 )
 
 
+data_dask = da.random.random(
+    size=(100_000, 1000, 1000), chunks=(1, 1000, 1000)
+)
+
+
 def test_is_rgb():
     shape = (10, 15)
     assert not is_rgb(shape)
@@ -79,12 +84,10 @@ def test_calc_data_fast_uint8():
     assert calc_data_range(data) == [0, 255]
 
 
+@pytest.mark.timeout(2)
 def test_calc_data_range_fast_big():
-    data = da.random.random(size=(100_000, 1000, 1000), chunks=(1, 1000, 1000))
-    t0 = time.time()
-    _ = calc_data_range(data)
-    t1 = time.time()
-    assert t1 - t0 < 2
+    val = calc_data_range(data_dask)
+    assert len(val) > 0
 
 
 def test_is_pyramid():
@@ -119,6 +122,16 @@ def test_is_pyramid():
     )
     assert is_pyramid(data)
 
+    # Check for integer overflow with big data
+    s = 8192
+    data = [da.ones((s,) * 3), da.ones((s // 2,) * 3), da.ones((s // 4,) * 3)]
+    assert is_pyramid(data)
+
+
+@pytest.mark.timeout(2)
+def test_timing_is_pyramid_big():
+    assert not is_pyramid(data_dask)
+
 
 def test_trim_pyramid():
 
diff --git a/requirements/test.txt b/requirements/test.txt
--- a/requirements/test.txt
+++ b/requirements/test.txt
@@ -2,4 +2,5 @@ pytest
 pytest-faulthandler
 pytest-qt
 pytest-ordering
+pytest-timeout
 xarray