Repository: napari/napari

Issue:
[test-bot] Benchmark tests failing
The Benchmarks workflow failed on 2022-10-16 10:08 UTC

The most recent failing test was on macos-latest py3.9 Qt
with commit: e90c7f882bd6cb4b280fc8834189ccfc1f4ee4a0

Full run: https://github.com/napari/napari/actions/runs/3258476073

(This post will be updated if another test fails, as long as this issue remains open.)



Discussion:
@andy-sweet I'm wondering if your recent work on time_set_view_slice may have affected this. Not sure if it will ultimately matter since that will all be reworked soon?

Separately, I need to check to see if the speedups would trigger a failure since they are being reported as well. 

Qt benchmarks report:
```
       before           after         ratio
     [36c94477]       [e90c7f88]
     <v0.4.16rc7^0>       <latest^0>
-      1.76±0.02s         984±10ms     0.56  benchmark_qt_slicing.QtViewerAsyncPointsAndImage2DSuite.time_z_scroll(65536, 0.0, 512)
-      1.69±0.02s          927±6ms     0.55  benchmark_qt_slicing.QtViewerAsyncPointsAndImage2DSuite.time_z_scroll(65536, 0.0, 1024)
-      1.36±0.03s          659±2ms     0.48  benchmark_qt_viewer_vectors.QtViewerViewVectorSuite.time_vectors_multi_refresh(65536)
-         455±8ms        218±0.7ms     0.48  benchmark_qt_viewer_vectors.QtViewerViewVectorSuite.time_vectors_refresh(65536)
-      20.6±0.02s        8.30±0.1s     0.40  benchmark_qt_slicing.QtViewerAsyncPointsSuite.time_z_scroll(131072)
       before           after         ratio
     [36c94477]       [e90c7f88]
     <v0.4.16rc7^0>       <latest^0>
+        378±10μs      1.68±0.05ms     4.44  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(16)
+         380±5μs      1.68±0.01ms     4.42  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(2048)
+         379±5μs      1.63±0.03ms     4.29  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(256)
+         379±7μs      1.61±0.05ms     4.26  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(128)
+         386±3μs      1.63±0.03ms     4.24  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(32)
+        381±10μs      1.60±0.02ms     4.20  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(64)
+         387±4μs      1.62±0.04ms     4.19  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(4096)
+        392±10μs      1.62±0.01ms     4.14  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(1024)
+         389±9μs      1.56±0.04ms     4.00  benchmark_qt_viewer_image.QtViewerImageSuite.time_set_view_slice(512)
+        818±30μs      2.03±0.05ms     2.48  benchmark_qt_viewer_labels.QtViewerSingleLabelsSuite.time_set_view_slice
+        885±20μs      2.11±0.06ms     2.39  benchmark_qt_slicing.AsyncImage2DSuite.time_set_view_slice(0.0, 'skin_data')
+      1.56±0.2ms      2.80±0.09ms     1.79  benchmark_qt_slicing.AsyncImage2DSuite.time_refresh(0.0, 'skin_data')
```
@kcpevey : do you mean #5003? Agreed that definitely looks the most likely culprit in the last week. These checks run once a week right?

Is there a command I can run on my machine to compare that commit against the previous one?
Looks like the base here (36c94477) is from 2022/05/31 - am I reading that right? If that's true, our list of potential culprits will probably grow quite a bit!
You can run a manual dispatch comparing two commits. Go to the Action page for the Benchmark and select `Run workflow`. You'll get a popup with spaces from the commits hashes. https://github.com/napari/napari/actions/workflows/benchmarks.yml

This particular check is from the weekly schedule (every sunday). You can also run manually, as mentioned, or by adding the `run-benchmarks` label to a PR. 

The base here is indeed [36c9447](https://github.com/napari/napari/commit/36c94477b5b63e160f75cf8c4aef5524e2a6af19) which is the latest release (its also associated with rc7)
Also, to not be misleading. This is the report for the non-Qt benchmarks:
```
before           after         ratio
     [36c94477]       [e90c7f88]
     <v0.4.16rc7^0>       <latest^0>
-     2.07±0.01ms      1.28±0.02ms     0.62  benchmark_vectors_layer.Vectors3DSuite.time_set_view_slice(256)
-        1.39±0ms         848±10μs     0.61  benchmark_points_layer.Points3DSuite.time_set_view_slice(4096)
-       132±0.9ms       77.1±0.5ms     0.58  benchmark_vectors_layer.Vectors3DSuite.time_refresh(16384)
-      27.7±0.3ms       15.8±0.2ms     0.57  benchmark_points_layer.Points3DSuite.time_create_layer(16384)
-      82.2±0.7ms       43.5±0.8ms     0.53  benchmark_points_layer.Points2DSuite.time_add(16384)
-     6.25±0.04ms      3.26±0.04ms     0.52  benchmark_points_layer.Points3DSuite.time_refresh(16384)
-         328±3ms        158±0.4ms     0.48  benchmark_vectors_layer.Vectors3DSuite.time_create_layer(16384)
-       169±0.8ms       80.7±0.3ms     0.48  benchmark_vectors_layer.Vectors3DSuite.time_length(16384)
-         341±3ms          162±4ms     0.47  benchmark_points_layer.Points2DSuite.time_add(65536)
-        87.1±3ms         37.2±2ms     0.43  benchmark_points_layer.Points3DSuite.time_create_layer(65536)
-      21.7±0.2ms       8.80±0.1ms     0.41  benchmark_points_layer.Points3DSuite.time_refresh(65536)
-         227±2ms         91.0±1ms     0.40  benchmark_vectors_layer.Vectors2DSuite.time_length(65536)
-         449±3ms        177±0.9ms     0.39  benchmark_vectors_layer.Vectors2DSuite.time_create_layer(65536)
-         307±3ms       90.1±0.5ms     0.29  benchmark_vectors_layer.Vectors3DSuite.time_refresh(65536)
-     4.68±0.04ms      1.30±0.02ms     0.28  benchmark_vectors_layer.Vectors3DSuite.time_set_view_slice(1024)
-     3.86±0.03ms      1.07±0.01ms     0.28  benchmark_points_layer.Points3DSuite.time_set_view_slice(16384)
-         881±2ms          186±1ms     0.21  benchmark_vectors_layer.Vectors3DSuite.time_create_layer(65536)
-         447±1ms       93.9±0.3ms     0.21  benchmark_vectors_layer.Vectors3DSuite.time_length(65536)
-      14.6±0.1ms      1.89±0.01ms     0.13  benchmark_points_layer.Points3DSuite.time_set_view_slice(65536)
-      15.3±0.3ms      1.57±0.01ms     0.10  benchmark_vectors_layer.Vectors3DSuite.time_set_view_slice(4096)
-         3.45±0s          255±4ms     0.07  benchmark_points_layer.PointsSlicingSuite.time_slice_points(False)
-      3.46±0.01s          255±5ms     0.07  benchmark_points_layer.PointsSlicingSuite.time_slice_points(True)
-      55.5±0.3ms      2.36±0.02ms     0.04  benchmark_vectors_layer.Vectors3DSuite.time_set_view_slice(16384)
-         220±2ms      5.18±0.08ms     0.02  benchmark_vectors_layer.Vectors3DSuite.time_set_view_slice(65536)
       before           after         ratio
     [36c94477]       [e90c7f88]
     <v0.4.16rc7^0>       <latest^0>
+     8.35±0.04μs          519±3μs    62.16  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(16)
+      8.47±0.1μs          516±5μs    60.90  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(64)
+      8.50±0.3μs          513±7μs    60.35  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(256)
+      9.69±0.1μs          515±4μs    53.16  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(1024)
+      13.6±0.1μs          524±2μs    38.57  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(4096)
+      15.7±0.4μs          524±2μs    33.43  benchmark_surface_layer.Surface2DSuite.time_refresh(16)
+     16.3±0.05μs          528±9μs    32.45  benchmark_surface_layer.Surface2DSuite.time_refresh(256)
+      16.4±0.2μs         531±10μs    32.36  benchmark_surface_layer.Surface2DSuite.time_refresh(64)
+        18.1±1μs          531±6μs    29.37  benchmark_surface_layer.Surface2DSuite.time_refresh(1024)
+      21.4±0.3μs         549±10μs    25.69  benchmark_surface_layer.Surface2DSuite.time_refresh(4096)
+     21.2±0.09μs          543±2μs    25.56  benchmark_shapes_layer.Shapes2DSuite.time_set_view_slice(128)
+      21.3±0.1μs          539±2μs    25.32  benchmark_shapes_layer.Shapes2DSuite.time_set_view_slice(256)
+      21.3±0.1μs          539±1μs    25.25  benchmark_shapes_layer.Shapes2DSuite.time_set_view_slice(32)
+      21.4±0.1μs          538±3μs    25.20  benchmark_shapes_layer.Shapes2DSuite.time_set_view_slice(64)
+      21.7±0.2μs        538±0.6μs    24.72  benchmark_shapes_layer.Shapes2DSuite.time_set_view_slice(16)
+      29.6±0.5μs          569±4μs    19.22  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(16384)
+      32.1±0.2μs          554±1μs    17.23  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(16)
+     32.7±0.09μs          557±2μs    17.06  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(64)
+      34.2±0.5μs          565±4μs    16.51  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(256)
+     37.8±0.03μs          581±2μs    15.36  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(1024)
+      39.4±0.6μs          579±3μs    14.70  benchmark_surface_layer.Surface2DSuite.time_refresh(16384)
+      80.6±0.5μs      1.15±0.02ms    14.29  benchmark_image_layer.Image2DSuite.time_set_view_slice(1024)
+      80.3±0.2μs      1.14±0.02ms    14.23  benchmark_image_layer.Image2DSuite.time_set_view_slice(2048)
+      80.9±0.9μs      1.15±0.04ms    14.23  benchmark_image_layer.Image2DSuite.time_set_view_slice(512)
+      80.0±0.2μs      1.13±0.01ms    14.17  benchmark_image_layer.Image2DSuite.time_set_view_slice(4096)
+        79.4±1μs      1.13±0.01ms    14.17  benchmark_image_layer.Image2DSuite.time_set_view_slice(32)
+      80.3±0.1μs         1.12±0ms    13.98  benchmark_image_layer.Image2DSuite.time_set_view_slice(256)
+        80.9±1μs         1.12±0ms    13.90  benchmark_image_layer.Image2DSuite.time_set_view_slice(16)
+        81.3±1μs         1.12±0ms    13.81  benchmark_image_layer.Image2DSuite.time_set_view_slice(128)
+      81.8±0.5μs         1.12±0ms    13.74  benchmark_image_layer.Image2DSuite.time_set_view_slice(64)
+      85.5±0.2μs      1.13±0.01ms    13.21  benchmark_labels_layer.Labels2DSuite.time_set_view_slice(16)
+      86.6±0.2μs         1.14±0ms    13.15  benchmark_labels_layer.Labels2DSuite.time_set_view_slice(32)
+      92.6±0.2μs      1.14±0.03ms    12.29  benchmark_labels_layer.Labels2DSuite.time_set_view_slice(64)
+      51.9±0.4μs          618±1μs    11.91  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(4096)
+         116±3μs         1.18±0ms    10.19  benchmark_labels_layer.Labels2DSuite.time_set_view_slice(128)
+        98.3±2μs          647±5μs     6.58  benchmark_points_layer.Points2DSuite.time_set_view_slice(16)
+         101±2μs          650±6μs     6.41  benchmark_points_layer.Points2DSuite.time_set_view_slice(64)
+       105±0.8μs         653±10μs     6.24  benchmark_points_layer.Points2DSuite.time_set_view_slice(256)
+        228±10μs      1.30±0.01ms     5.70  benchmark_labels_layer.Labels2DSuite.time_set_view_slice(256)
+       120±0.3μs          675±1μs     5.65  benchmark_points_layer.Points2DSuite.time_set_view_slice(1024)
+         137±2μs          734±4μs     5.37  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(16384)
+       150±0.7μs          702±4μs     4.67  benchmark_surface_layer.Surface2DSuite.time_set_view_slice(65536)
+         165±1μs          732±7μs     4.43  benchmark_surface_layer.Surface2DSuite.time_refresh(65536)
+         174±2μs          743±1μs     4.27  benchmark_points_layer.Points2DSuite.time_set_view_slice(4096)
+        587±50μs      1.65±0.01ms     2.81  benchmark_labels_layer.Labels2DSuite.time_set_view_slice(512)
+        792±20μs      1.95±0.01ms     2.47  benchmark_labels_layer.Labels2DSuite.time_refresh(16)
+         402±8μs         975±20μs     2.43  benchmark_points_layer.Points2DSuite.time_set_view_slice(16384)
+        796±40μs      1.92±0.02ms     2.42  benchmark_labels_layer.Labels2DSuite.time_refresh(64)
+         822±9μs      1.93±0.04ms     2.35  benchmark_labels_layer.Labels2DSuite.time_refresh(32)
+         522±9μs      1.23±0.02ms     2.35  benchmark_vectors_layer.Vectors2DSuite.time_set_view_slice(65536)
+        868±10μs      1.94±0.08ms     2.23  benchmark_labels_layer.Labels2DSuite.time_refresh(128)
+        910±20μs      2.02±0.02ms     2.22  benchmark_image_layer.Image2DSuite.time_set_data(16)
+     1.03±0.07ms      2.27±0.04ms     2.19  benchmark_image_layer.Image2DSuite.time_set_data(32)
+     1.01±0.05ms      2.20±0.02ms     2.19  benchmark_labels_layer.Labels2DSuite.time_paint_circle(128)
+       380±0.7μs         830±10μs     2.18  benchmark_text_manager.TextManagerSuite.time_create(16, {'constant': 'test'})
+        894±30μs      1.94±0.03ms     2.17  benchmark_image_layer.Image2DSuite.time_refresh(16)
+        380±20μs          812±8μs     2.14  benchmark_text_manager.TextManagerSuite.time_create(64, {'constant': 'test'})
+     1.08±0.04ms      2.30±0.02ms     2.13  benchmark_image_layer.Image2DSuite.time_refresh(64)
+     1.02±0.01ms      2.16±0.01ms     2.12  benchmark_labels_layer.Labels2DSuite.time_paint_circle(64)
+         385±2μs         816±10μs     2.12  benchmark_text_manager.TextManagerSuite.time_create(1024, {'constant': 'test'})
+        984±10μs      2.09±0.02ms     2.12  benchmark_labels_layer.Labels2DSuite.time_paint_circle(16)
+        988±20μs      2.09±0.03ms     2.11  benchmark_labels_layer.Labels2DSuite.time_refresh(256)
+         384±8μs          807±6μs     2.10  benchmark_text_manager.TextManagerSuite.time_create(256, {'constant': 'test'})
+     1.08±0.06ms      2.27±0.06ms     2.10  benchmark_image_layer.Image2DSuite.time_refresh(128)
+         392±4μs          816±4μs     2.08  benchmark_text_manager.TextManagerSuite.time_create(4096, {'constant': 'test'})
+         408±5μs          843±7μs     2.07  benchmark_text_manager.TextManagerSuite.time_create(16384, {'constant': 'test'})
+     1.10±0.03ms       2.28±0.1ms     2.06  benchmark_image_layer.Image2DSuite.time_refresh(512)
+     1.03±0.01ms      2.10±0.01ms     2.05  benchmark_labels_layer.Labels2DSuite.time_paint_circle(32)
+     1.09±0.01ms      2.22±0.02ms     2.03  benchmark_image_layer.Image2DSuite.time_refresh(32)
+     1.11±0.02ms      2.25±0.02ms     2.02  benchmark_image_layer.Image2DSuite.time_refresh(256)
+        1.11±0ms      2.24±0.06ms     2.02  benchmark_image_layer.Image2DSuite.time_set_data(512)
+     1.10±0.01ms      2.21±0.03ms     2.01  benchmark_image_layer.Image2DSuite.time_set_data(64)
+     1.14±0.02ms      2.29±0.03ms     2.01  benchmark_image_layer.Image2DSuite.time_set_data(2048)
+     1.13±0.02ms      2.27±0.01ms     2.00  benchmark_image_layer.Image2DSuite.time_set_data(4096)
+     1.12±0.02ms      2.25±0.03ms     2.00  benchmark_image_layer.Image2DSuite.time_set_data(256)
+     1.14±0.03ms      2.26±0.04ms     1.99  benchmark_image_layer.Image2DSuite.time_refresh(2048)
+     1.13±0.02ms      2.24±0.02ms     1.98  benchmark_image_layer.Image2DSuite.time_refresh(4096)
+     1.13±0.03ms      2.22±0.02ms     1.97  benchmark_image_layer.Image2DSuite.time_refresh(1024)
+     1.13±0.06ms      2.22±0.05ms     1.97  benchmark_image_layer.Image2DSuite.time_set_data(1024)
+     1.17±0.02ms      2.29±0.03ms     1.96  benchmark_labels_layer.Labels2DSuite.time_paint_circle(256)
+     1.13±0.02ms      2.20±0.08ms     1.95  benchmark_image_layer.Image2DSuite.time_set_data(128)
+        707±10μs      1.32±0.03ms     1.87  benchmark_points_layer.Points2DSuite.time_refresh(64)
+        490±10μs          915±9μs     1.87  benchmark_text_manager.TextManagerSuite.time_create(6[5536](https://github.com/napari/napari/actions/runs/3258476073/jobs/5350591390#step:7:5537), {'constant': 'test'})
+        759±20μs      1.41±0.02ms     1.85  benchmark_points_layer.Points2DSuite.time_refresh(256)
+     1.50±0.02ms       2.64±0.1ms     1.77  benchmark_labels_layer.Labels2DSuite.time_paint_circle(512)
+        707±20μs      1.24±0.04ms     1.76  benchmark_points_layer.Points2DSuite.time_refresh(16)
+     1.36±0.03ms      2.35±0.01ms     1.72  benchmark_labels_layer.Labels2DSuite.time_refresh(512)
+        610±10μs      1.05±0.01ms     1.72  benchmark_text_manager.TextManagerSuite.time_create(16, 'float_property')
+         644±8μs      1.08±0.02ms     1.68  benchmark_text_manager.TextManagerSuite.time_create(16, 'string_property')
+         914±4μs      1.53±0.02ms     1.68  benchmark_points_layer.Points2DSuite.time_refresh(1024)
+         647±1μs         1.07±0ms     1.65  benchmark_text_manager.TextManagerSuite.time_create(64, 'string_property')
+     5.59±0.08ms       9.22±0.1ms     1.65  benchmark_points_layer.Points2DSuite.time_create_layer(256)
+         663±6μs         1.08±0ms     1.63  benchmark_text_manager.TextManagerSuite.time_create(64, 'float_property')
+     7.67±0.03ms       12.5±0.1ms     1.63  benchmark_labels_layer.Labels2DSuite.time_create_layer(128)
+        682±10μs      1.10±0.01ms     1.62  benchmark_text_manager.TextManagerSuite.time_create(256, 'string_property')
+     7.52±0.09ms      11.8±0.05ms     1.57  benchmark_labels_layer.Labels2DSuite.time_create_layer(64)
+     7.55±0.05ms       11.7±0.3ms     1.56  benchmark_labels_layer.Labels2DSuite.time_create_layer(32)
+        799±10μs      1.23±0.01ms     1.54  benchmark_text_manager.TextManagerSuite.time_create(1024, 'string_property')
+        829±20μs      1.27±0.02ms     1.53  benchmark_text_manager.TextManagerSuite.time_create(256, 'float_property')
+      5.56±0.1ms       8.46±0.3ms     1.52  benchmark_points_layer.Points2DSuite.time_create_layer(16)
+     5.77±0.06ms       8.76±0.2ms     1.52  benchmark_points_layer.Points2DSuite.time_create_layer(64)
+      7.35±0.1ms       11.1±0.1ms     1.51  benchmark_labels_layer.Labels2DSuite.time_create_layer(16)
```
Thanks for the info. I was looking for something more like a `git bisect` that I could run with one fast benchmark on my local machine, so I could more easily pinpoint the commit that may have caused the regression. I think [asv find](https://asv.readthedocs.io/en/stable/commands.html#asv-find) sounds promising, so I temporarily edited `QtViewerImageSuite.params` to be `[16]` and ran

```
asv find --python=same 36c94477..e90c7f88 "QtViewerImageSuite.time_set_view_slice"
```

That identified 88ca786f as the greatest regression, but the regression is not substantial at all (i.e. 1.07±0.07ms  vs.  987±50μs), and the code change is clearly benign. So maybe I'm misusing that.

Instead, I just manually ran `asv run --bench "QtViewerImageSuite.time_set_view_slice" --python=same` at 36c9447 and at the commit associated with #5003 saw a similarly large difference. I think I've identified the regression - should be pretty easy to fix.

Resolving patch:
diff --git a/napari/layers/base/base.py b/napari/layers/base/base.py
--- a/napari/layers/base/base.py
+++ b/napari/layers/base/base.py
@@ -722,6 +722,9 @@ def extent(self) -> Extent:
     @property
     def _slice_indices(self):
         """(D, ) array: Slice indices in data coordinates."""
+        if len(self._slice_input.not_displayed) == 0:
+            # All dims are displayed dimensions
+            return (slice(None),) * self.ndim
         return self._slice_input.data_indices(
             self._data_to_world.inverse,
             getattr(self, '_round_index', True),


Test patch: