Repository: napari/napari

Issue:
Overflow when calculating multiscale sizes
## 🐛 Bug

I was playing with really large multiscale images using `viewer.add_image` and found some overflows. 

I have a fix for this one incoming.

## To Reproduce

My multiscale image:

> ```[dask.array<from-zarr, shape=(17179869184, 17179869184), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(8589934592, 8589934592), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(4294967296, 4294967296), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(2147483648, 2147483648), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(1073741824, 1073741824), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(536870912, 536870912), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(268435456, 268435456), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(134217728, 134217728), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(67108864, 67108864), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(33554432, 33554432), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(16777216, 16777216), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(8388608, 8388608), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(4194304, 4194304), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(2097152, 2097152), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(1048576, 1048576), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(524288, 524288), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(262144, 262144), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(131072, 131072), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(65536, 65536), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(32768, 32768), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(16384, 16384), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(8192, 8192), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(4096, 4096), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(2048, 2048), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>, dask.array<from-zarr, shape=(1024, 1024), dtype=uint8, chunksize=(1024, 1024), chunktype=numpy.ndarray>]```

When I try to add that image to the viewer I get this error:

```
File "/Users/kharrington/git/kephale/nesoi/repos/napari/napari/layers/image/_image_utils.py", line 86, in guess_multiscale
    raise ValueError(
```
> ```ValueError: Input data should be an array-like object, or a sequence of arrays of decreasing size. Got arrays in incorrect order, shapes: [(17179869184, 17179869184), (8589934592, 8589934592), (4294967296, 4294967296), (2147483648, 2147483648), (1073741824, 1073741824), (536870912, 536870912), (268435456, 268435456), (134217728, 134217728), (67108864, 67108864), (33554432, 33554432), (16777216, 16777216), (8388608, 8388608), (4194304, 4194304), (2097152, 2097152), (1048576, 1048576), (524288, 524288), (262144, 262144), (131072, 131072), (65536, 65536), (32768, 32768), (16384, 16384), (8192, 8192), (4096, 4096), (2048, 2048), (1024, 1024)]```

This is very confusing because the ordering of the arrays *is* correct. 

Using a breakpoint at the end of this snippet https://github.com/napari/napari/blob/ac8266032d3aa8e64cc2e33fb297269a8c1983ae/napari/layers/image/_image_utils.py#L67-L84

reveals the overflow

```
(Pdb) sizes
array([                  0,                   0,                   0,
       4611686018427387904, 1152921504606846976,  288230376151711744,
         72057594037927936,   18014398509481984,    4503599627370496,
          1125899906842624,     281474976710656,      70368744177664,
            17592186044416,       4398046511104,       1099511627776,
              274877906944,         68719476736,         17179869184,
                4294967296,          1073741824,           268435456,
                  67108864,            16777216,             4194304,
                   1048576], dtype=uint64)

(Pdb) shapes
[(17179869184, 17179869184), (8589934592, 8589934592), (4294967296, 4294967296), (2147483648, 2147483648), (1073741824, 1073741824), (536870912, 536870912), (268435456, 268435456), (134217728, 134217728), (67108864, 67108864), (33554432, 33554432), (16777216, 16777216), (8388608, 8388608), (4194304, 4194304), (2097152, 2097152), (1048576, 1048576), (524288, 524288), (262144, 262144), (131072, 131072), (65536, 65536), (32768, 32768), (16384, 16384), (8192, 8192), (4096, 4096), (2048, 2048), (1024, 1024)]
```

## Expected behavior

I do not necessarily expect this size of image to work, but it shouldn't fail like this.

## Environment

napari version is `7d3dd01e5a280552ea53199656cf9f9009b6f778 (HEAD -> poor-mans-octree, origin/poor-mans-octree, kephale/poor-mans-octree)` (link: https://github.com/kephale/napari/tree/poor-mans-octree) which is pretty close to current `main`



Discussion:

Resolving patch:
diff --git a/napari/layers/image/_image_utils.py b/napari/layers/image/_image_utils.py
--- a/napari/layers/image/_image_utils.py
+++ b/napari/layers/image/_image_utils.py
@@ -64,28 +64,27 @@ def guess_multiscale(data) -> Tuple[bool, LayerDataProtocol]:
         # pyramid with only one level, unwrap
         return False, data[0]
 
-    shapes = [d.shape for d in data]
-    sizes = np.array([np.prod(shape, dtype=np.uint64) for shape in shapes])
+    sizes = [d.size for d in data]
     if len(sizes) <= 1:
         return False, data
 
-    consistent = bool(np.all(sizes[:-1] > sizes[1:]))
-    if np.all(sizes == sizes[0]):
+    consistent = all(s1 > s2 for s1, s2 in zip(sizes[:-1], sizes[1:]))
+    if all(s == sizes[0] for s in sizes):
         # note: the individual array case should be caught by the first
         # code line in this function, hasattr(ndim) and ndim > 1.
         raise ValueError(
             trans._(
-                'Input data should be an array-like object, or a sequence of arrays of decreasing size. Got arrays of single shape: {shape}',
+                'Input data should be an array-like object, or a sequence of arrays of decreasing size. Got arrays of single size: {size}',
                 deferred=True,
-                shape=shapes[0],
+                size=sizes[0],
             )
         )
     if not consistent:
         raise ValueError(
             trans._(
-                'Input data should be an array-like object, or a sequence of arrays of decreasing size. Got arrays in incorrect order, shapes: {shapes}',
+                'Input data should be an array-like object, or a sequence of arrays of decreasing size. Got arrays in incorrect order, sizes: {sizes}',
                 deferred=True,
-                shapes=shapes,
+                sizes=sizes,
             )
         )
 


Test patch:
diff --git a/napari/layers/image/_tests/test_image_utils.py b/napari/layers/image/_tests/test_image_utils.py
--- a/napari/layers/image/_tests/test_image_utils.py
+++ b/napari/layers/image/_tests/test_image_utils.py
@@ -1,5 +1,6 @@
 import time
 
+import dask
 import dask.array as da
 import numpy as np
 import pytest
@@ -74,6 +75,18 @@ def test_guess_multiscale():
     data = [da.ones((s,) * 3), da.ones((s // 2,) * 3), da.ones((s // 4,) * 3)]
     assert guess_multiscale(data)[0]
 
+    # Test for overflow in calculating array sizes
+    s = 17179869184
+    data = [
+        da.from_delayed(
+            dask.delayed(lambda: None), shape=(s,) * 2, dtype=np.float64
+        ),
+        da.from_delayed(
+            dask.delayed(lambda: None), shape=(s // 2,) * 2, dtype=np.float64
+        ),
+    ]
+    assert guess_multiscale(data)[0]
+
 
 def test_guess_multiscale_strip_single_scale():
     data = [np.empty((10, 10))]