Repository: napari/napari

Issue:
Move layer triggers weird bug in viewer
## 🐛 Bug

When I want to move a layer, a strange bug is happening in the viewer. I don't know if it's related to my environment, if I am doing something wrong. But here is what happens: when I move a layer in the viewer, I get this error:

```
> ---------------------------------------------------------------------------
> AttributeError                            Traceback (most recent call last)
> /usr/local/lib/python3.10/site-packages/napari/_qt/containers/_layer_delegate.py in paint(self=<napari._qt.containers._layer_delegate.LayerDelegate object>, painter=<PyQt5.QtGui.QPainter object>, option=<PyQt5.QtWidgets.QStyleOptionViewItem object>, index=<PyQt5.QtCore.QModelIndex object>)
>      84         # update the icon based on layer type
>      85 
> ---> 86         self.get_layer_icon(option, index)
>         self.get_layer_icon = <bound method LayerDelegate.get_layer_icon of <napari._qt.containers._layer_delegate.LayerDelegate object at 0x7f193fc9b7f0>>
>         option = <PyQt5.QtWidgets.QStyleOptionViewItem object at 0x7f1901960040>
>         index = <PyQt5.QtCore.QModelIndex object at 0x7f18ffd2f450>
>      87         # paint the standard itemView (includes name, icon, and vis. checkbox)
>      88         super().paint(painter, option, index)
> 
> /usr/local/lib/python3.10/site-packages/napari/_qt/containers/_layer_delegate.py in get_layer_icon(self=<napari._qt.containers._layer_delegate.LayerDelegate object>, option=<PyQt5.QtWidgets.QStyleOptionViewItem object>, index=<PyQt5.QtCore.QModelIndex object>)
>      97             icon_name = 'folder-open' if expanded else 'folder'
>      98         else:
> ---> 99             icon_name = f'new_{layer._type_string}'
>         icon_name = undefined
>     100 
>     101         try:
> 
> AttributeError: 'NoneType' object has no attribute '_type_string'
```
In addition, there is the same error as #4420: the layer remains in the foreground, even if it has been moved from the last layer to a lower layer.

## To Reproduce

Steps to reproduce the behavior:

1. Move layer from on old index to a new index
`viewer.layers.move(old_index, new_index)`


## Expected behavior
This error  `AttributeError: 'NoneType' object has no attribute '_type_string' ` should not be raised...
And

## Environment
napari: 0.4.15
Platform: Linux-5.17.4-200.fc35.x86_64-x86_64-with-glibc2.34
System: Fedora Linux 35 (Workstation Edition)
Python: 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)]
Qt: 5.15.2
PyQt5: 5.15.6
NumPy: 1.21.5
SciPy: 1.8.0
Dask: 2022.04.0
VisPy: 0.9.6

OpenGL:
- GL version: 4.6.0 NVIDIA 510.47.03
- MAX_TEXTURE_SIZE: 16384

Screens:
- screen 1: resolution 1920x1200, scale 1.0

## Additional context
This bug appears in the context of the development of a new plugin (this logic is triggered in a widget). I have another bug here #4420 which might be related.


Discussion:
Sounds annoying, thanks for reporting! What happens if you do this from the ipython console?:

```python
import napari
import numpy as np
v = napari.Viewer()
il = v.add_image(np.random.rand(10, 10), colormap='reds')
il2 = v.add_image(np.random.rand(10, 10), colormap='blues')
v.layers.move(1, 0)
```

For me, this works fine. Can you find an isolated piece of code that you can share and reproduces your issue?
> For me, this works fine. Can you find an isolated piece of code that you can share and reproduces your issue?

This works fine... I guess that issue has probably to do with my environment/plugin. I will try to investigate further and give you my feedback.
I am using the`viewer.layers.events.inserted` event to trigger a move inside the callback method (a move on the just inserted layer).
I think that this is causing the issue, but I need to figure out a way for it to work: maybe trigger another function after the callback that makes the move, because as of now is seems that the inserted layer is "really" inserted only after the callback function has been entirely executed.
But it seems to me, that there is a bug in the inserted callback function handling. The point of having an inserting and inserted event is specifically to target those 2 specific scenarios...
Yes, this sounds familiar... I had a similar issue in the past, I think.

... Found it, #3337. Not sure if it's related, but it might be. 

I was able to reproduce *some kind of* error; it does not seem to be exactly the same as your message, however. Try this:

```python
import napari
import numpy as np
v = napari.Viewer()

il = v.add_image(np.random.rand(10, 10), colormap='reds')

def move():
    v.layers.move(1, 0)

v.layers.events.inserted.connect(move)

il2 = v.add_image(np.random.rand(10, 10), colormap='blues')
```

maybe the event wizards @Czaki and @tlambert03 have some ideas :)
> Yes, this sounds familiar... I had a similar issue in the past, I think.
> 
> ... Found it, #3337. Not sure if it's related, but it might be.
> 
> I was able to reproduce _some kind of_ error; it does not seem to be exactly the same as your message, however. Try this:
> 
> ```python
> import napari
> import numpy as np
> v = napari.Viewer()
> 
> il = v.add_image(np.random.rand(10, 10), colormap='reds')
> 
> def move():
>     v.layers.move(1, 0)
> 
> v.layers.events.inserted.connect(move)
> 
> il2 = v.add_image(np.random.rand(10, 10), colormap='blues')
> ```
> 
> maybe the event wizards @Czaki and @tlambert03 have some ideas :)

Thanks a lot for your help, and for pointing me to similar issues :)

The error I can trigger with your code is slightly different than the one I posted originally, but they seem close to me.
Ok. I think that the source of problems is that `connect()`inserts the connected function in the first position so your functions are called before all napari internal hooks. Then You modify the state of the layer list and all other callbacks are triggered with the wrong state. 

Maybe simplest solution could be use `position` keyword argument, like this: `v.layers.events.inserted.connect(move, position="last")`

But if it is not important to make movement immediately I prefer to use [`Timer` ](https://docs.python.org/3/library/threading.html#timer-objects) and [`ensure_main_thread`](https://github.com/napari/superqt/blob/main/docs/decorators.md#move-to-thread-decorators) to be sure that all callbacks will be processed first.

To be more accurate. The error in @brisvag code is caused by triggering a move before vispy layer was created and looking on @cnstt traceback it looks like vispy layer is created but the entry in layerlist is not added yet. 
Thanks a lot @Czaki for your reply :pray: 

> Maybe simplest solution could be use `position` keyword argument, like this: `v.layers.events.inserted.connect(move, position="last")`

What does the position keyword changes here exactly?


> But if it is not important to make movement immediately I prefer to use [`Timer` ](https://docs.python.org/3/library/threading.html#timer-objects) and [`ensure_main_thread`](https://github.com/napari/superqt/blob/main/docs/decorators.md#move-to-thread-decorators) to be sure that all callbacks will be processed first.

In my case the move (reordering of layers) could happen after, that's not a big issue. I just don't want it to be too visible for the user and I don't want the user to mess around before it has been moved. But right after the callback would be fine!
Default value for `position` is `"first"`. So added callback is executed before earlier added. 
Changing to `"last"` make callback called after all earlier added callbacks. 

The problem could happen, if someone added next callback on the end of callback list. 

So if you set short time in `Timer` this should do job for you. Feel free to ask questions if documentation is not clear for you or you do not meet some concepts ealier. 
With `position="last"` it works now ! I spent 1 day trying to figure out how it works, and it finally works now! Thanks a lot!
I must admit I don't really understand the magic behind the position keyword, is it something like : "the callback function that is added is executed "after" the finishing of the inserting layer process?
Anyway it works now, maybe it would be great to add this "position" argument in the documentation, because haven't seen it anywhere (or I missed it).
Here is the documentation: https://napari.org/api/napari.utils.events.EventEmitter.html#napari.utils.events.EventEmitter.connect

But it may not be clear where it is for someone who does not know the napari code. 

A little more explanation. In napari, callback functions are stored on a list (you could connect multiple functions to one event) and executed in order from this list.  So connect effectively is adding the callback to the list. And position controls if it is inserted on begin or the end. 

I agree that it should be better implemented/documented. 
Your explanation is very clear, thank you! I just add only one callback function, so it should work fine for me :)
Should I close the issue?
Thanks @Czaki, I also didn't know of the `position` keyword!

Actually, before we close: should we consider defaulting to `last` for this kind of callback (`inserted` rather than `insert`)? I think @cnstt's assumption that this callback should fire *after* the viewer reaches a consistent state is very reasonable.
> Actually, before we close: should we consider defaulting to `last` for this kind of callback (`inserted` rather than `insert`)? I think @cnstt's assumption that this callback should fire _after_ the viewer reaches a consistent state is very reasonable.

I open the #4425 issue for this. Even changing the default position to `"last"` is not solving this problem. Only lower probability of meeting such error. So anyone who could meet it may have more problems understanding what happened. 
Fwiw, I've always disliked the position argument in general.  I don't include it in psygnal, and I think that, internally, napari shouldn't care _what_ has been connected or in what order.  So @Czaki #4425 seems to be the more important issue here.  (That is, it's up to us to make sure that the event is only emitted after all internal state has stabilized)
I have expected the same kind of behavior for the event callback `viewer.layers.selection.events.changed.connect()`, the addition of position='last' solved my issue (infinite loop).

Resolving patch:
diff --git a/napari/utils/events/event.py b/napari/utils/events/event.py
--- a/napari/utils/events/event.py
+++ b/napari/utils/events/event.py
@@ -369,6 +369,34 @@ def source(self) -> Any:
     def source(self, s):
         self._source = None if s is None else weakref.ref(s)
 
+    def _is_core_callback(
+        self, callback: Union[CallbackRef, Callback], core: str
+    ):
+        """
+        Check if the callback is a core callback
+
+        Parameters
+        ----------
+        callback : Union[CallbackRef, Callback]
+            The callback to check. Callback could be function or
+            weak reference to object method coded using weakreference
+            to object and method name stored in tuple.
+        core : str
+            Name of core module, for example 'napari'.
+        """
+        try:
+            if isinstance(callback, partial):
+                callback = callback.func
+            if not isinstance(callback, tuple):
+                return callback.__module__.startswith(core + '.')
+            obj = callback[0]()  # get object behind weakref
+            if obj is None:  # object is dead
+                return False
+            return obj.__module__.startswith(core + '.')
+
+        except AttributeError:
+            return False
+
     def connect(
         self,
         callback: Union[Callback, CallbackRef, CallbackStr, 'EventEmitter'],
@@ -473,12 +501,26 @@ def connect(
                     position=position,
                 )
             )
+        core_callbacks_indexes = [
+            i
+            for i, c in enumerate(self._callbacks)
+            if self._is_core_callback(c, 'napari')
+        ]
+        core_callbacks_count = (
+            max(core_callbacks_indexes) + 1 if core_callbacks_indexes else 0
+        )
+        if self._is_core_callback(callback, 'napari'):
+            callback_bounds = (0, core_callbacks_count)
+        else:
+            callback_bounds = (core_callbacks_count, len(callback_refs))
 
         # bounds: upper & lower bnds (inclusive) of possible cb locs
-        bounds: List[int] = list()
+        bounds: List[int] = []
         for ri, criteria in enumerate((before, after)):
             if criteria is None or criteria == []:
-                bounds.append(len(callback_refs) if ri == 0 else 0)
+                bounds.append(
+                    callback_bounds[1] if ri == 0 else callback_bounds[0]
+                )
             else:
                 if not isinstance(criteria, list):
                     criteria = [criteria]


Test patch:
diff --git a/napari/utils/_tests/test_notification_manager.py b/napari/utils/_tests/test_notification_manager.py
--- a/napari/utils/_tests/test_notification_manager.py
+++ b/napari/utils/_tests/test_notification_manager.py
@@ -14,8 +14,6 @@
     show_warning,
 )
 
-PY38_OR_HIGHER = bool(getattr(threading, 'excepthook', None))
-
 
 # capsys fixture comes from pytest
 # https://docs.pytest.org/en/stable/logging.html#caplog-fixture
@@ -66,6 +64,7 @@ def test_notification_manager_no_gui(monkeypatch):
             'This is another information message'
         )
         assert len(notification_manager.records) == 2
+        assert len(store) == 2
         assert store[-1].type == 'info'
 
         # test that exceptions that go through sys.excepthook are catalogued
@@ -75,8 +74,12 @@ def test_notification_manager_no_gui(monkeypatch):
 
         # pytest intercepts the error, so we can manually call sys.excepthook
         assert sys.excepthook == notification_manager.receive_error
-        sys.excepthook(*sys.exc_info())
+        try:
+            raise ValueError("a")
+        except ValueError:
+            sys.excepthook(*sys.exc_info())
         assert len(notification_manager.records) == 3
+        assert len(store) == 3
         assert store[-1].type == 'error'
 
         # test that warnings that go through showwarning are catalogued
@@ -117,8 +120,7 @@ def _raise():
         with pytest.raises(PurposefulException):
             raise PurposefulException("this is an exception")
 
-    if PY38_OR_HIGHER:
-        previous_threading_exhook = threading.excepthook
+    previous_threading_exhook = threading.excepthook
 
     with notification_manager:
         notification_manager.records.clear()
@@ -127,21 +129,18 @@ def _raise():
         notification_manager.notification_ready.connect(store.append)
 
         # Test exception inside threads
-        if PY38_OR_HIGHER:
-            # `threading.excepthook` available only for Python >= 3.8
-            assert (
-                threading.excepthook
-                == notification_manager.receive_thread_error
-            )
+        assert (
+            threading.excepthook == notification_manager.receive_thread_error
+        )
 
         exception_thread = threading.Thread(target=_raise)
         exception_thread.start()
         time.sleep(0.02)
 
-        if PY38_OR_HIGHER:
+        try:
+            raise ValueError("a")
+        except ValueError:
             threading.excepthook(sys.exc_info())
-        else:
-            sys.excepthook(*sys.exc_info())
 
         assert len(notification_manager.records) == 1
         assert store[-1].type == 'error'
@@ -162,7 +161,6 @@ def _raise():
             raise AssertionError("Thread notification not received in time")
 
     # make sure we've restored the threading except hook
-    if PY38_OR_HIGHER:
-        assert threading.excepthook == previous_threading_exhook
+    assert threading.excepthook == previous_threading_exhook
 
     assert all(isinstance(x, Notification) for x in store)
diff --git a/napari/utils/events/_tests/test_event_emitter.py b/napari/utils/events/_tests/test_event_emitter.py
--- a/napari/utils/events/_tests/test_event_emitter.py
+++ b/napari/utils/events/_tests/test_event_emitter.py
@@ -1,4 +1,5 @@
 import weakref
+from functools import partial
 
 import pytest
 
@@ -148,6 +149,88 @@ def meth4(self, _event):
     assert t.m4 == 1
 
 
+def test_event_order_func():
+    res_li = []
+
+    def fun1():
+        res_li.append(1)
+
+    def fun2(val):
+        res_li.append(val)
+
+    def fun3():
+        res_li.append(3)
+
+    def fun4():
+        res_li.append(4)
+
+    def fun5(val):
+        res_li.append(val)
+
+    def fun6(val):
+        res_li.append(val)
+
+    fun1.__module__ = "napari.test.sample"
+    fun3.__module__ = "napari.test.sample"
+    fun5.__module__ = "napari.test.sample"
+
+    e = EventEmitter(type="test")
+    e.connect(fun1)
+    e.connect(partial(fun2, val=2))
+    e()
+    assert res_li == [1, 2]
+    res_li = []
+    e.connect(fun3)
+    e()
+    assert res_li == [3, 1, 2]
+    res_li = []
+    e.connect(fun4)
+    e()
+    assert res_li == [3, 1, 4, 2]
+    res_li = []
+    e.connect(partial(fun5, val=5), position="last")
+    e()
+    assert res_li == [3, 1, 5, 4, 2]
+    res_li = []
+    e.connect(partial(fun6, val=6), position="last")
+    e()
+    assert res_li == [3, 1, 5, 4, 2, 6]
+
+
+def test_event_order_methods():
+    res_li = []
+
+    class Test:
+        def fun1(self):
+            res_li.append(1)
+
+        def fun2(self):
+            res_li.append(2)
+
+    class Test2:
+        def fun3(self):
+            res_li.append(3)
+
+        def fun4(self):
+            res_li.append(4)
+
+    Test.__module__ = "napari.test.sample"
+
+    t1 = Test()
+    t2 = Test2()
+
+    e = EventEmitter(type="test")
+    e.connect(t1.fun1)
+    e.connect(t2.fun3)
+    e()
+    assert res_li == [1, 3]
+    res_li = []
+    e.connect(t1.fun2)
+    e.connect(t2.fun4)
+    e()
+    assert res_li == [2, 1, 4, 3]
+
+
 def test_no_event_arg():
     class TestOb:
         def __init__(self):