Repository: napari/napari

Issue:
Deleting points raises IndexError (MacOS)
## 🐛 Bug

Removing points from a points layer via "Delete selected points" sometimes results in an`IndexError: index is out of bounds`. This happens only for certain points and in specific situations - git bisects indicates commit 5dd37eccb84e3291123 to be the culprit (maybe some events race condition?)


## To Reproduce


The following script generates 3 points `P1--P2--P3`. Directly selecting the middle one (P2) and quickly deleting it via the keyboard results in the above error in ~30% of all cases
 
```python
import numpy as np
import napari
w = napari.Viewer()
points=np.stack([np.ones(3), 10*np.arange(3)], axis=-1)
points_layer = w.add_points(points, size=2)
points_layer.mode="select"
napari.run()
```

```python
IndexError: index 2 is out of bounds for axis 0 with size 2
```
## Expected behavior

Point being removed without throwing a tantrum

## Environment

```
napari: 0.4.10 (3bd58bde6858eb4d9f)
Platform: macOS-10.15.7-x86_64-i386-64bit
System: MacOS 10.15.7
Python: 3.8.8 (default, Feb 24 2021, 13:46:16) [Clang 10.0.0 ]
Qt: 5.15.2
PyQt5: 5.15.4
NumPy: 1.19.5
SciPy: 1.7.0
Dask: 2021.04.1
VisPy: 0.6.6

OpenGL:
- GL version: 2.1 ATI-3.10.18
- MAX_TEXTURE_SIZE: 16384

```
## Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Maybe there should be a limit on the number of terrible bugs one person can report in a 24h period hey @maweigert? 😂 

(Joking, just thought it was funny that I literally *just* finished responding to your other bug, hit "archive" on the GH notification, and this was the next top notification! 😂)

Thanks for the git-bisect! Should really help narrow down what is happening!
Ok so this one I can't seem to reproduce... Perhaps it has to do with mouse sensitivity and whether the point thinks it's moving while it's being deleted, and my mouse is on a less sensitive setting than yours... (Tried with both trackpad and mouse.) Would you mind copy-pasting the full traceback @maweigert? You might want to use a gist.github.com to paste into if it's too long. Thank you!
> Would you mind copy-pasting the full traceback @maweigert?

Sure, here it is https://gist.github.com/maweigert/034ecccab08b02cf4aface706f5fb732
@maweigert thanks for the report. I couldn't reproduce on ubuntu but I've managed to reproduce now on Mac and am looking into it.
Finally managed to figure out exactly what's happening here.

@jni was right that this happens when a slight mouse drag is detected on click. Since #2992 this point move also emits a `data` event, causing `_on_layers_change` to fire [on the viewer](https://github.com/napari/napari/blob/9dd9fa6390ad4dfe9a08194de8c84e7e87bfeaa0/napari/components/viewer_model.py#L335). This resets the cursor position to 0 (not sure that this is strictly necessary?), which eventually causes `get_value` to be called on the `points` layer and [this line](https://github.com/napari/napari/blob/9dd9fa6390ad4dfe9a08194de8c84e7e87bfeaa0/napari/layers/base/base.py#L1007) sets `self._value` to 0 (the new cursor position). 

Because previously we weren't emitting the `data` event, `self._value` was always holding the point being selected for deletion, which was causing `self._value` to be reset to `None` on the points layer [here](https://github.com/napari/napari/blob/9dd9fa6390ad4dfe9a08194de8c84e7e87bfeaa0/napari/layers/points/points.py#L1398). Now that `self._value` isn't `None` but a real point index, we run into issues with setting the highlight on the point (we still try to highlight a point that may not exist) and from then on with any cursor position change because we're still trying to access a point whose properties have been deleted - errors fire as you move the cursor around.

It looks like the comment on [this line](https://github.com/napari/napari/blob/9dd9fa6390ad4dfe9a08194de8c84e7e87bfeaa0/napari/layers/base/base.py#L1005) has come of age and maybe now is the time to consider decoupling `base.get_value` and `points._set_value`, although this would be a much larger change than just fixing this one bug and seems like it would also affect `shapes`.

Two quick alternatives if we wanted to fix this ASAP and then look at the `self._value` issue separately would be
- not setting cursor position to 0 `on_layers_change` [here](https://github.com/napari/napari/blob/9dd9fa6390ad4dfe9a08194de8c84e7e87bfeaa0/napari/components/viewer_model.py#L347) - not sure exactly what the implications of that would be. I've tried to discern changes in Viewer behaviour without it but couldn't really, and I've also run tests without it and all seems okay, but probably someone else should weigh in here.
- always setting `self._value` to `None` in `remove_selected` [here](https://github.com/napari/napari/blob/9dd9fa6390ad4dfe9a08194de8c84e7e87bfeaa0/napari/layers/points/points.py#L1398) which I think should be a minimal impact change for users because as soon as they move their cursor they will still get valid information in their status bar.

The last option would be my preferred solution if we want to look at `self._value` separately after this bug is fixed, but not sure if that's what we want @sofroniewn @jni?
 
@DragaDoncila Honestly I like *both* options, and think we should probably do both. I don't think it should make any difference setting the cursor position to (0, 0, ...) when changing the layer list, as it's not a particularly special value. But I agree with you that setting `self._value` to None feels more "proximal" to the fix. 😅 So if @sofroniewn agrees I would suggest make both changes in a PR.

A bigger structural fix can come later, this is a pretty significant bug so I'd like the fix in the next release.
> not setting cursor position to 0 on_layers_change

I think you might need to either pad/ drop dimensions so that cursor.position has correct dimensionality - i.e. the significance is not about it going to `0` but going to the right `n` in our `nD` representation. That's a shame no test is currently catching that behaviour, but you might be able to write a test which changes dimensionality of layers and moves cursor to find bug

> always setting self._value to None in remove_selected

seems like a good idea

Resolving patch:
diff --git a/napari/components/viewer_model.py b/napari/components/viewer_model.py
--- a/napari/components/viewer_model.py
+++ b/napari/components/viewer_model.py
@@ -344,7 +344,15 @@ def _on_layers_change(self, event):
             self.dims.ndim = ndim
             for i in range(ndim):
                 self.dims.set_range(i, (world[0, i], world[1, i], ss[i]))
-        self.cursor.position = (0,) * self.dims.ndim
+
+        new_dim = self.dims.ndim
+        dim_diff = new_dim - len(self.cursor.position)
+        if dim_diff < 0:
+            self.cursor.position = self.cursor.position[:new_dim]
+        elif dim_diff > 0:
+            self.cursor.position = tuple(
+                list(self.cursor.position) + [0] * dim_diff
+            )
         self.events.layers_change()
 
     def _update_interactive(self, event):
diff --git a/napari/layers/points/points.py b/napari/layers/points/points.py
--- a/napari/layers/points/points.py
+++ b/napari/layers/points/points.py
@@ -1395,8 +1395,7 @@ def remove_selected(self):
                     self.properties[k], index, axis=0
                 )
             self.text.remove(index)
-            if self._value in self.selected_data:
-                self._value = None
+            self._value = None
             self.selected_data = set()
             self.data = np.delete(self.data, index, axis=0)
 


Test patch:
diff --git a/napari/_tests/test_viewer.py b/napari/_tests/test_viewer.py
--- a/napari/_tests/test_viewer.py
+++ b/napari/_tests/test_viewer.py
@@ -302,3 +302,35 @@ class NewLabels(layers.Labels):
     # Make a viewer and add the custom layer
     viewer = make_napari_viewer(show=True)
     viewer.add_layer(NewLabels(np.zeros((10, 10, 10), dtype=np.uint8)))
+
+
+def test_emitting_data_doesnt_change_points_value(make_napari_viewer):
+    """Test emitting data with no change doesn't change the layer _value."""
+    viewer = make_napari_viewer()
+
+    data = np.array([[0, 0], [10, 10], [20, 20]])
+    layer = viewer.add_points(data, size=2)
+    viewer.layers.selection.active = layer
+
+    assert layer._value is None
+    viewer.cursor.position = tuple(layer.data[1])
+    assert layer._value == 1
+
+    layer.events.data(value=layer.data)
+    assert layer._value == 1
+
+
+@pytest.mark.parametrize('layer_class, data, ndim', layer_test_data)
+def test_emitting_data_doesnt_change_cursor_position(
+    make_napari_viewer, layer_class, data, ndim
+):
+    """Test emitting data event from layer doesn't change cursor position"""
+    viewer = make_napari_viewer()
+    layer = layer_class(data)
+    viewer.add_layer(layer)
+
+    new_position = (5,) * ndim
+    viewer.cursor.position = new_position
+    layer.events.data(value=layer.data)
+
+    assert viewer.cursor.position == new_position
diff --git a/napari/components/_tests/test_viewer_model.py b/napari/components/_tests/test_viewer_model.py
--- a/napari/components/_tests/test_viewer_model.py
+++ b/napari/components/_tests/test_viewer_model.py
@@ -562,6 +562,23 @@ def test_active_layer_cursor_size():
     assert viewer.cursor.size == 10
 
 
+def test_cursor_ndim_matches_layer():
+    """Test cursor position ndim matches viewer ndim after update."""
+    viewer = ViewerModel()
+    np.random.seed(0)
+    im = viewer.add_image(np.random.random((10, 10)))
+    assert viewer.dims.ndim == 2
+    assert len(viewer.cursor.position) == 2
+
+    im.data = np.random.random((10, 10, 10))
+    assert viewer.dims.ndim == 3
+    assert len(viewer.cursor.position) == 3
+
+    im.data = np.random.random((10, 10))
+    assert viewer.dims.ndim == 2
+    assert len(viewer.cursor.position) == 2
+
+
 def test_sliced_world_extent():
     """Test world extent after adding layers and slicing."""
     np.random.seed(0)
diff --git a/napari/layers/points/_tests/test_points.py b/napari/layers/points/_tests/test_points.py
--- a/napari/layers/points/_tests/test_points.py
+++ b/napari/layers/points/_tests/test_points.py
@@ -330,6 +330,7 @@ def test_removing_selected_points():
     assert len(layer.selected_data) == 0
     keep = [1, 2] + list(range(4, 10))
     assert np.all(layer.data == data[keep])
+    assert layer._value is None
 
     # Select another point and remove it
     layer.selected_data = {4}