Repository: napari/napari

Issue:
Creating Node() objects in on_draw() causes crash
# Description
This is a bug report but with a PR that shows the problem. I want to see if anyone else can reproduce this or if it's just my machine. If people can reproduce it I'm looking for debugging ideas.

I was getting crashes like this with the Octree work, but with a lot of debugging, I was able to come up with this simplified situation that still causes the crash. No async/octree code running. No layers or any data loaded. Nothing being drawn in the main window.

# Steps

1. With the code in this PR run `napari` with no arguments, no data.
2. Click and drag the mouse in the main window, keep dragging it around.
3. I get a segfault after 101 iterations. Anyone else?

# Results

    VispyCamera.on_draw: 95
    VispyCamera.on_draw: 96
    VispyCamera.on_draw: 97
    VispyCamera.on_draw: 98
    VispyCamera.on_draw: 99
    VispyCamera.on_draw: 100
    VispyCamera.on_draw: 101
    /Users/pbw/miniconda3/envs/new-base/bin/pythonw: line 3: 38446 Segmentation fault: 11
    /Users/pbw/miniconda3/envs/new-base/python.app/Contents/MacOS/python "$@"
    /Users/pbw/miniconda3/envs/new-base/lib/python3.8/multiprocessing/resource_tracker.py:216:
    UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at
    shutdown warnings.warn('resource_tracker: There appear to be %d '

# Comments

* We create a `Node` object but we do not even hold a reference to it.
* If you change it to create 100 `Node` objects in a tight loop, it still fails on the 101st iteration
    * But this time it created created 10,100 nodes.
    * So it must be something we are doing 101 times that causes the problem
    * But it only causes the problem if at least one node was created.
* Are we not allowed to create nodes here for some reason? During a draw?
* Could we create a test that shows this crash?
    * Hooks into something and creates Nodes at the right time?


Discussion:
> I get a segfault after 101 iterations. Anyone else?

Ok very intriguing I can get a seg fault on the 71st draw every time launching with `napari` only with `pyqt5`, i.e. doesn't happen with `pyside2`.

Also if I run `python examples/add_image.py` I don't get the seg fault

If I get rid of the welcome visual (set `welcome=False`) I get the seg fault on the 90th draw
I thought maybe if you just created one `Node` that would be enough, and then N iterations later it would crash. But I tested that just now. You have to create them all, or at least you have to create more than one.

This seems contrived but this came up organically with Octree. Basically adding `ImageVisuals` as you panned around was crashing. I somewhat worked around it by creating a pool of ImageVisuals up front, but even that sometimes crashes if the pool is too big, etc.

I wonder if it's illegal to create new nodes during the draw, if that has to be scheduled to happen in an event when no drawing is in progress?  But I think I tried that with `QTimer.singleShot` and it didn't help. But I should try it again maybe I was still creating nodes somewhere.

If you step through it creating a `Node` it surprisingly gets into shader code related to transforms. But I didn't see it actually doing OpenGL stuff. It seems to be just getting the shader code organized and ready.

But if it does making OpenGL calls on construction of a `Node`, that could explain how creating a `Node` and not even keeping a reference to it could have some non-local effect.
@pwinston do you now know enough to create a napari independent crashing example?
Early on I created a vispy example that added and removed 400 `ImageVisuals` on a key press. And I could hit the key all day. So that's creating Nodes and it worked fine.

But I could go back and try some different permutations. But to date I've only seen it crash in napari.

But that is a good question. Can we get at even smaller example that shows it. Starting from a vispy example or chopping down napari even more.


> But that is a good question. Can we get at even smaller example that shows it. Starting from a vispy example or chopping down napari even more.

Yeah, I quickly tried what I saw as the minimal vispy only example

```python
from vispy import app, scene
from vispy.scene.node import Node
app.use_app('pyqt5')

# Create canvas and view
canvas = scene.SceneCanvas(keys='interactive', size=(600, 600), show=True)
view = canvas.central_widget.add_view()
view.camera = scene.cameras.PanZoomCamera()

node_index = 0
def on_draw(event):
    global node_index
    print(f"VispyCamera.on_draw: {node_index}")
    Node()
    node_index += 1

canvas.connect(on_draw)

# Run example
app.run()
```

But this ran fine. I wonder if the example would need to be embedded in a parent Qt window, or something like that? The fact that our current example doesn't crash with PySide2 is making me wonder that.

Do you see the octree crashes with pyside2? Would be interesting to know that too
I hadn't run it with PySide2 before. Trying it now it seems stable. I have not seen a crash. I created 4096 nodes as a test. Took many minutes but they showed up. And with 256 nodes I can zoom in/out and they will be created/destroyed 30-50 at a time and it works.

With PyQt5 if I don't use a pool I can't even create 10 nodes. And with a pool can only do 30 or so.

For the quadtree we'll mostly only need to draw 20-40 chunks/tiles/nodes at one time. But we might have many layers each with their own tree. And there may be special cases where we want to draw more. So I think this is a must-fix bug. Or must-work-around. 

We might need to draw lots of chunks to solve #845. That's a single large image without chunks but all already in RAM. We will probably want to create chunks but not an octree, because downsampling the images is too slow. Chunk creation is almost instant because it's just new views onto the same numpy data. Basically we would only have level 0 of the octree. Basically a grid. 

So we might need to render a lot of chunks in that case. We'd basically page in the image in from RAM into VRAM one chunk at a time. Not sure what it will really look like. But it should come over pretty fast.
I found a weird workaround for this specific case. But it does not help the Octree. So it must be just nudging things slightly.

But if you set `AVOID_PYTQ5_CRASH` to `True` in `event_loop.py` that prevents the crash in the PR for me. I let it to go up to 10,000 and saw no crash, where it crashes around 100 for me normally.

But again this isn't helpful as a real workaround. But I think it does show how precarious things are.
> But if you set AVOID_PYTQ5_CRASH to True in event_loop.py that prevents the crash in the PR for me. I let it to go up to 10,000 and saw no crash, where it crashes around 100 for me normally.

Interesting. I wonder if @Czaki might have any guesses at what could be happening here too
@sofroniewn pointed me to #1029.  The example Tony points as “one such report” is:
https://riverbankcomputing.com/pipermail/pyqt/2011-August/030376.html

That thread ends with Phil Thompson, the creator of PyQt, implementing a fix in SIP to correct the problem. Basically to check which thread the object is in and call `deleteLater` when needed. And he made the fix in 2011. This seems consistent with there being no ongoing outcry that PyQt simply doesn't work with threading. If that were the case, I think that issue would be coming up left and right.

I tried the QTimer GC snippet from that thread and it made it last longer but then still crashed. I tried just turning off GC entirely and it doesn't crash. That does imply the solution is turn off GC though. Any sort of memory issues with Qt or OpenGL could lead to a crash when GC is run.

So this is good information, but not sure what it implies yet. Basically if there's a memory problem, running GC might crash, but ideally, the solution is to fix that problem, not don't run GC.

Although in Raveler we created a "disable GC" context object and used it in like a dozen places. Basically to toggle GC off for certain big operations. Basically, we found running GC at the end instead of while the operating was happening sped things up a lot. As long as you can afford the high watermark of memory use, that's valid.
No longer create nodes on draw (waving the mouse). We now create nodes automatically using a `QTimer` see `CanvasUpdater` class in `qt_viewer.py`. So just run and it will count up and crash.

Also added `napari.utils.config` option `config.ALLOW_QT` that makes the UI look like:

<img width="824" alt="napari-minimal" src="https://user-images.githubusercontent.com/4163446/97928239-9594e480-1d34-11eb-9a50-09612b6cc049.png">

That seems to prevent the crash but still experimenting. Sometimes the behavior changes when I don't think I've changed anything. Which is possible if there are some truly random elements.



> > But if you set AVOID_PYTQ5_CRASH to True in event_loop.py that prevents the crash in the PR for me. I let it to go up to 10,000 and saw no crash, where it crashes around 100 for me normally.
> 
> Interesting. I wonder if @Czaki might have any guesses at what could be happening here too

I think it could be similar to the termination problem (ctrl+c in terminal). When python code is not called, then it cannot do some job (maybe related to gc). And this code force to go to python code on each `notify` call. 
If this PR still allows reproducing error? 
I just updated the PR. Yes as checked in it causes the crash for me. Just run `napari` with pyqt5. This is what I get:

    CanvasUpdater: 69
    CanvasUpdater: 70
    CanvasUpdater: 71
    CanvasUpdater: 72
    CanvasUpdater: 73
    18:15:12 ERROR Unhandled exception:
    Traceback (most recent call last):
      File "/Users/pbw/dev/vispy/vispy/app/backends/_qt.py", line 825, in paintGL
        self._vispy_canvas.events.draw(region=None)
      File "/Users/pbw/dev/vispy/vispy/util/event.py", line 455, in __call__
        self._invoke_callback(cb, event)
      File "/Users/pbw/dev/vispy/vispy/util/event.py", line 473, in _invoke_callback
        _handle_exception(self.ignore_callback_errors,
      File "/Users/pbw/dev/vispy/vispy/util/event.py", line 471, in _invoke_callback
        cb(event)
      File "/Users/pbw/dev/vispy/vispy/app/canvas.py", line 427, in swap_buffers
        self._backend._vispy_swap_buffers()
      File "/Users/pbw/dev/vispy/vispy/app/backends/_qt.py", line 791, in _vispy_swap_buffers
        ctx = self.context()
    RuntimeError: wrapped C/C++ object of type CanvasBackendDesktop has been deleted

    /Users/pbw/miniconda3/envs/new-base/bin/pythonw: line 3: 72930 Segmentation fault: 11  
    /Users/pbw/miniconda3/envs/new-base/python.app/Contents/MacOS/python "$@"
    /Users/pbw/miniconda3/envs/new-base/lib/python3.8/multiprocessing/resource_tracker.py:216: 
    UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
     warnings.warn('resource_tracker: There appear to be %d '

But if you set `napari.utils.config.ALLOW_QT` to False then it doesn't seem to crash for me. I'll look into that.


> I just updated the PR. Yes as checked in it causes the crash for me. Just run `napari` with pyqt5.

which system and qt version, and python version?
My machine is macOS Catalina 10.15.6 with:
    PyQt5                         5.14.2             
    PyQt5-sip                     12.7.2   
    Python 3.8.1 
> I just updated the PR. Yes as checked in it causes the crash for me. Just run napari with pyqt5. This is what I get:

I'm no longer getting a crash - just passed `CanvasUpdater: 1648`, using pyqt5, launching with napari

PyQt5: 5.14.1
Python: 3.7.0
I wonder if my config is off though as I still see welcome visual
After my last update `ALLOW_QT` is `True` so you should see normal UI. And it crashes for me.
If you set `ALLOW_QT` to `False` you see minimal UI and it doesn't crash for me.
I have to run now, but I'm hoping I can add back some QT stuff slowly and see what crashes.
But we could be chasing wild geeses here. Memory stuff is tricky.
I think that I miss something in the description. I see no debug messages in the terminal when trying to reproduce this. Could anyone show gif with this bug? 
The steps for me with this PR are:
1) Run `napari`
2) The terminal prints messages up to around `CanvasUpdater: 73`
3) It crashes with this message:

     /Users/pbw/miniconda3/envs/new-base/bin/pythonw: line 3: 74133 Segmentation fault: 11
     /Users/pbw/miniconda3/envs/new-base/python.app/Contents/MacOS/python "$@"
     /Users/pbw/miniconda3/envs/new-base/lib/python3.8/multiprocessing/resource_tracker.py:216: 
     UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown

No interaction with the napari GUI is needed. It just does the count itself when it starts.

Here is a GIF:

![napari-1811](https://user-images.githubusercontent.com/4163446/97937144-cf251a00-1d4b-11eb-8575-fc89257cee1f.gif)

For more context, this crash was first seen with the Octree work. There we dynamically create and remove a lot of `ImageVisuals`. With PySide2 it works fine. With PyQt5 there are crashes. This PR I think demonstrates the same crash I'm seeing with the Octree. And this crash is PyQt5 only as well. It does not crash with PySide2.
I checked in a new attempt at a fix that seems promising.

As before if I run the PR as-is I get a crash with PyQt5 after less than 100 iterations.

However if I set `napari.utils.config.DELAY_LOAD_VISUALS` to `True` it does not crash. 

For example running it like this:
`DELAY_LOAD_VISUALS=False` -> Ran 10 times got 10 crashes @73
`DELAY_LOAD_VISUALS=True` -> Stopped @ 11,000 no crash.
`DELAY_LOAD_VISUALS=False` -> Ran 10 times got 10 crashes @73
`DELAY_LOAD_VISUALS=True` -> Stopped @ 11,000 no crash.
`DELAY_LOAD_VISUALS=True` -> Stopped @ 11,000 no crash.

Still TBD but certainly seem to have an effect.

The change is just factoring out all Visual creation from `QtViewer.__init__()` into a method `QtViewer.add_visuals` and calling `add_visuals()` immediately after calling init.

Does NOT fix the octree case though...
A shame this doesn't fix octree. But it must be telling us something about when it's okay to create visuals and when it's not okay. But it's not obvious when it is okay. And of course there could be something else going on.
Using `Qtimer.singleShot` so all the octree visuals stuff happens in it's own event does not work. I tried that long ago and it didn't work then either. It seems like it should, you are starting fresh with basically no stack. Nothing should be stopping you from making any changes you want.

But it fails then too. Maybe that means you need some certain context. You need to be inside some specific Vispy event or Qt event or something that means it's okay to muck with visuals? Guessing.

Nothing about the docs or API suggests any of this is necessary, that I know of. But just going by trying to make PyQt5 not crash here. Just trying to guess what it wants.

The other angle would be we've done something wrong unrelated to all this that's put us in this fragile state. And nothing we do in this state will work very well. We have to fix THAT thing.
Using what we learned above I tried every which way to make Octree visuals work with PyQt5 and it always crashes. I create a pool of invisible visuals up front, at the same time we create the Axes/Welcome which seems to be a safe time now.

I add the entire pool as a child of my VispyTiledImageLayer parent node. And I just toggle the visibility of the nodes. Still crashes. ALSO I do all this from a vispy `app.Timer`, which how I see all the vispy examples changing things. We were not using `app.Timer`, since all the examples do surely that's safe? It's not, at least how we are doing things.

I've not seen any vispy example that changes the scene graph through a timer or through any other means. The examples always setup "the scene" and then it only sets transforms or calls `set_data` on things to make changes. They never actually change the scenegraph, that I saw, I didn't check them all.

But nothing in the API or docs that I've seen implies you can't change the scene graph dynamically. And it works for PySide2. There might be some simple `allow_changes = True` type flag. Like "blocking" events or suspending processing. That allows to safely make changes. But I have not found it.

**Basically we need to find a way to change the scene graph without crashing under PyQt5.**

As for Octree I think it's probably time to just write the `TiledImageVisual`. Even under PySide adding new visuals, even 64x64 tiles, is really slow. And since it also crashes under PyQt5, that's two big strikes against that approach. And since David Hoese strongly recommended the approach, that's really a third big strike against the other way.

I think with a nice `TiledImageVisual` and with the `DELAY_LOAD_VISUALS` change, we should be stable under PyQt5. Although we will not have discovered how to actually make changes safely. And `TiledImageVisual` once working as no known downside. It should just be faster and more stable, better in every way than juggling ImageVisuals.

I feel like there "must be" some simple thing we are missing. Something seemingly harmless that we are doing wrong. Something that's putting things into this fragile/broken state. I can't believe vispy or PyQt5 are this fragile. If that were true we'd be hearing about it right and left. Unless simply no one changes the scene graph dynamically, which seems unlikely. 
> I think with a nice TiledImageVisual and with the DELAY_LOAD_VISUALS change, we should be stable under PyQt5. Although we will not have discovered how to actually make changes safely. And TiledImageVisual once working as no known downside. It should just be faster and more stable, better in every way than juggling ImageVisuals.

Ok this seems like a good way forward. Thanks for all the debugging and ideas collected here.

> I feel like there "must be" some simple things we are missing. So thing we are doing wrong. That's putting things into this fragile/broken state. I can't believe vispy or PyQt5 are this fragile. Or we'd hear about it left and right. It might be something we are doing (or not doing) that's unusual.

I do feel like your right too. At times we weren't so good about doing things like assigning parents to widgets etc and that caused mysterious bugs. It really could end up being something like that, but that can be so hard to chase down
I still have a big problem with using `vispy` code. 

But when I try to reproduce I got such stacktrace: 
```
CanvasUpdater: 648
CanvasUpdater: 649
CanvasUpdater: 650
WARNING: Traceback (most recent call last):
  File "/home/czaki/.pyenv/versions/3.8.3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/czaki/.pyenv/versions/3.8.3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/czaki/Projekty/napari/napari/__main__.py", line 254, in <module>
    sys.exit(main())
  File "/home/czaki/Projekty/napari/napari/__main__.py", line 250, in main
    _run()
  File "/home/czaki/Projekty/napari/napari/__main__.py", line 186, in _run
    view_path(
  File "/home/czaki/.pyenv/versions/3.8.3/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/czaki/Projekty/napari/napari/_qt/event_loop.py", line 89, in gui_qt
    app.exec_()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/backends/_qt.py", line 508, in event
    out = super(QtBaseCanvasBackend, self).event(ev)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/backends/_qt.py", line 473, in mouseMoveEvent
    self._vispy_mouse_move(
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/base.py", line 213, in _vispy_mouse_move
    ev = self._vispy_canvas.events.mouse_move(**kwargs)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/Projekty/napari/napari/_vispy/vispy_canvas.py", line 39, in _process_mouse_event
    super()._process_mouse_event(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/canvas.py", line 359, in _process_mouse_event
    getattr(picked.events, event.type)(scene_event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 228, in viewbox_mouse_event
    self.pan(p1s - p2s)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 128, in pan
    self.rect = self.rect + pan
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 150, in rect
    self.view_changed()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/base_camera.py", line 380, in view_changed
    self._update_transform()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 287, in _update_transform
    self.tf_mat.matrix = self.transform.as_matrix().matrix
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/linear.py", line 401, in matrix
    self.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/base_transform.py", line 153, in update
    self.changed(*args)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/chain.py", line 212, in _subtr_changed
    self.update(ev)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/base_transform.py", line 153, in update
    self.changed(*args)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/chain.py", line 212, in _subtr_changed
    self.update(ev)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/base_transform.py", line 153, in update
    self.changed(*args)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/visual.py", line 265, in _transform_changed
    self.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/node.py", line 330, in update
    self.events.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 473, in _invoke_callback
    _handle_exception(self.ignore_callback_errors,
  << caught exception here: >>
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/visual.py", line 578, in _subv_update
    self.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/node.py", line 333, in update
    c.update(node=self)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/canvas.py", line 200, in update
    super(SceneCanvas, self).update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/canvas.py", line 442, in update
    self._backend._vispy_update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/backends/_qt.py", line 427, in _vispy_update
    self.update()
RuntimeError: wrapped C/C++ object of type CanvasBackendDesktop has been deleted
19:00:00 WARNING Traceback (most recent call last):
  File "/home/czaki/.pyenv/versions/3.8.3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/czaki/.pyenv/versions/3.8.3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/czaki/Projekty/napari/napari/__main__.py", line 254, in <module>
    sys.exit(main())
  File "/home/czaki/Projekty/napari/napari/__main__.py", line 250, in main
    _run()
  File "/home/czaki/Projekty/napari/napari/__main__.py", line 186, in _run
    view_path(
  File "/home/czaki/.pyenv/versions/3.8.3/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/czaki/Projekty/napari/napari/_qt/event_loop.py", line 89, in gui_qt
    app.exec_()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/backends/_qt.py", line 508, in event
    out = super(QtBaseCanvasBackend, self).event(ev)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/backends/_qt.py", line 473, in mouseMoveEvent
    self._vispy_mouse_move(
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/base.py", line 213, in _vispy_mouse_move
    ev = self._vispy_canvas.events.mouse_move(**kwargs)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/Projekty/napari/napari/_vispy/vispy_canvas.py", line 39, in _process_mouse_event
    super()._process_mouse_event(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/canvas.py", line 359, in _process_mouse_event
    getattr(picked.events, event.type)(scene_event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 228, in viewbox_mouse_event
    self.pan(p1s - p2s)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 128, in pan
    self.rect = self.rect + pan
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 150, in rect
    self.view_changed()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/base_camera.py", line 380, in view_changed
    self._update_transform()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/cameras/panzoom.py", line 287, in _update_transform
    self.tf_mat.matrix = self.transform.as_matrix().matrix
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/linear.py", line 401, in matrix
    self.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/base_transform.py", line 153, in update
    self.changed(*args)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/chain.py", line 212, in _subtr_changed
    self.update(ev)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/base_transform.py", line 153, in update
    self.changed(*args)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/chain.py", line 212, in _subtr_changed
    self.update(ev)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/transforms/base_transform.py", line 153, in update
    self.changed(*args)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/visual.py", line 265, in _transform_changed
    self.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/node.py", line 330, in update
    self.events.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 455, in __call__
    self._invoke_callback(cb, event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 473, in _invoke_callback
    _handle_exception(self.ignore_callback_errors,
  << caught exception here: >>
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/util/event.py", line 471, in _invoke_callback
    cb(event)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/visuals/visual.py", line 578, in _subv_update
    self.update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/node.py", line 333, in update
    c.update(node=self)
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/scene/canvas.py", line 200, in update
    super(SceneCanvas, self).update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/canvas.py", line 442, in update
    self._backend._vispy_update()
  File "/home/czaki/.pyenv/versions/napari/lib/python3.8/site-packages/vispy/app/backends/_qt.py", line 427, in _vispy_update
    self.update()
RuntimeError: wrapped C/C++ object of type CanvasBackendDesktop has been deleted
ERROR: Invoking <bound method CompoundVisual._subv_update of <Compound at 0x7f9f903fd6a0>> for Event
19:00:00 ERROR Invoking <bound method CompoundVisual._subv_update of <Compound at 0x7f9f903fd6a0>> for Event
```
Hmm looks like some stuff related to transforms in there too, I wonder if what we're doing here is a bit suspicious https://github.com/napari/napari/blob/master/napari/_vispy/vispy_base_layer.py#L66-L76.
Yeah "wrapped C/C++ object of type CanvasBackendDesktop has been deleted" was the most common error. I took to just mean memory corruption or something. It happens so randomly and so weirdly, I don't think it's a logic error.
The function `SceneCanvas._update_scenegraph` in vispy is:

    def _update_scenegraph(self, event):
        """Called when topology of scenegraph has changed.
        """
        self._draw_order.clear()
        self.update()

Stepping through I think I felt like that was getting called over and over, when probably there is a way to suspend/block events/updates while making changes. But really my understanding of vispy is not strong enough.

It seems like there is an underlying bug in Vispy and/or PyQt here. Ideally, it wouldn't outright crash. But the question is are we hitting the bug because we are doing something wrong? And how wrong? Are we doing something crazy? Or is it more like there is a narrow path you have to walk perfectly to avoid crashes? I'm not sure.

Does the fact the there is no crash with PySide2 mean it's definitely a PyQt5 bug? Or is PySide2 just being overly nice in tolerating our bad behavior?
> Yeah "wrapped C/C++ object of type CanvasBackendDesktop has been deleted" was the most common error. I took to just mean memory corruption or something. It happens so randomly and so weirdly, I don't think it's a logic error.

This is a very big logic error. It means that this object in Qt was deleted, but we still keep its reference from python. 

Also, segfault means try to access memory which is not assigned to process. 

> Does the fact the there is no crash with PySide2 mean it's definitely a PyQt5 bug? Or is PySide2 just being overly nice in tolerating our bad behavior?

It may mean that PyQt5 faster clean memory for example. 
> This is a very big logic error.

I'm distinguishing a logic error vs. memory corruption. By logic error I mean we did something wrong, and that wrong thing led PyQt5 to purposely delete the C++ object backing `CanvasBackendDesktop`. As in, some code called operator `delete` on the object. It did it on purpose.

By memory corruption, I mean someone somewhere wrote out of the bounds of some array, or wrote into some wrong location, which is trivial to do in C++. And they wrote on top of the memory used by that C++ object. No one called delete, but the object's memory was destroyed.

To me, the way the bug comes and goes seemingly at random suggests memory corruption. However, that's just a guess or a hunch. Until we find and fix the problem, it could be anything.

Whatever it is, we really need to fix it. Without a fix we are very constrained in what we can do in napari as far as creating nodes after startup. The other goal would be create a minimal small program that shows the same problem. That would help too, it would take the bulk of napari out of the equation.
Also, we should consider making a PR for the "delay load visuals" change from this PR. I can create a separate PR for that if we want to do it.

You don't need to call it separately like I did in this PR. You can have the last line of `QtViewer.__init__()` call `QtViewer._add_visuals()`.  This totally fixes the crashing problem in this PR. It just didn't fix Octree. 

I suspect it fixes things because it means we do all the Qt stuff first, and then the Vispy stuff. Instead of interleaving them. But that's just a guess.

Even if this didn't fix anything, however, it's cleaning things up. The `QtViewer.__init__()` is over 150 lines, and this is just extracting some very like-minded lines related to visuals.

Looking at those 150 lines I'd ask myself "is this just an undifferentiated soup of code with no patterns or no structure?" or are there in fact sections of this `__init__` doing things which are thematically different than other sections?

Even just squinting you can see the Vispy/visual stuff is different looking from the Qt/Widget/event stuff. Qt is mostly 2D GUI and Vispy is mostly OpenGL, so they are fundamentally different. So doing [extract method](https://refactoring.com/catalog/extractFunction.html) to pull out the vispy stuff makes sense here, even if it didn't fix a crash.
> Also, we should consider making a PR for the "delay load visuals" change from this PR. I can create a separate PR for that if we want to do it.
>
> You don't need to call it separately like I did in this PR. You can have the last line of QtViewer.__init__() call QtViewer._add_visuals(). This totally fixes the crashing problem in this PR. It just didn't fix Octree.

I liked this PR for both the fixing reasons and the organizational reasons that you mention above. If you submit as a standalone I'd happily review @pwinston 

Resolving patch:
diff --git a/napari/_qt/qt_viewer.py b/napari/_qt/qt_viewer.py
--- a/napari/_qt/qt_viewer.py
+++ b/napari/_qt/qt_viewer.py
@@ -157,6 +157,57 @@ def __init__(self, viewer, welcome=True):
             self.toggle_console_visibility
         )
 
+        self._create_canvas()
+
+        main_widget = QWidget()
+        main_layout = QVBoxLayout()
+        main_layout.setContentsMargins(10, 22, 10, 2)
+        main_layout.addWidget(self.canvas.native)
+        main_layout.addWidget(self.dims)
+        main_layout.setSpacing(10)
+        main_widget.setLayout(main_layout)
+
+        self.setOrientation(Qt.Vertical)
+        self.addWidget(main_widget)
+
+        self._last_visited_dir = str(Path.home())
+
+        self._cursors = {
+            'cross': Qt.CrossCursor,
+            'forbidden': Qt.ForbiddenCursor,
+            'pointing': Qt.PointingHandCursor,
+            'standard': QCursor(),
+        }
+
+        self._update_palette()
+
+        self.viewer.events.interactive.connect(self._on_interactive)
+        self.viewer.cursor.events.style.connect(self._on_cursor)
+        self.viewer.cursor.events.size.connect(self._on_cursor)
+        self.viewer.events.palette.connect(self._update_palette)
+        self.viewer.layers.events.reordered.connect(self._reorder_layers)
+        self.viewer.layers.events.inserted.connect(self._on_add_layer_change)
+        self.viewer.layers.events.removed.connect(self._remove_layer)
+
+        # stop any animations whenever the layers change
+        self.viewer.events.layers_change.connect(lambda x: self.dims.stop())
+
+        self.setAcceptDrops(True)
+
+        for layer in self.viewer.layers:
+            self._add_layer(layer)
+
+        self.view = self.canvas.central_widget.add_view()
+        self.camera = VispyCamera(
+            self.view, self.viewer.camera, self.viewer.dims
+        )
+        self.canvas.connect(self.camera.on_draw)
+
+        # Add axes, scale bar and welcome visuals.
+        self._add_visuals(welcome)
+
+    def _create_canvas(self) -> None:
+        """Create the canvas and hook up events."""
         self.canvas = VispyCanvas(
             keys=None,
             vsync=True,
@@ -177,11 +228,14 @@ def __init__(self, viewer, welcome=True):
         self.canvas.connect(self.on_draw)
         self.canvas.connect(self.on_resize)
 
-        self.view = self.canvas.central_widget.add_view()
-        self.camera = VispyCamera(
-            self.view, self.viewer.camera, self.viewer.dims
-        )
-        self.canvas.connect(self.camera.on_draw)
+    def _add_visuals(self, welcome: bool) -> None:
+        """Add visuals for axes, scale bar, and welcome text.
+
+        Parameters
+        ----------
+        welcome : bool
+            Show the welcome visual.
+        """
 
         self.axes = VispyAxesVisual(
             self.viewer.axes,
@@ -209,43 +263,6 @@ def __init__(self, viewer, welcome=True):
             self.viewer.events.palette.connect(self.welcome._on_palette_change)
             self.canvas.events.resize.connect(self.welcome._on_canvas_change)
 
-        main_widget = QWidget()
-        main_layout = QVBoxLayout()
-        main_layout.setContentsMargins(10, 22, 10, 2)
-        main_layout.addWidget(self.canvas.native)
-        main_layout.addWidget(self.dims)
-        main_layout.setSpacing(10)
-        main_widget.setLayout(main_layout)
-
-        self.setOrientation(Qt.Vertical)
-        self.addWidget(main_widget)
-
-        self._last_visited_dir = str(Path.home())
-
-        self._cursors = {
-            'cross': Qt.CrossCursor,
-            'forbidden': Qt.ForbiddenCursor,
-            'pointing': Qt.PointingHandCursor,
-            'standard': QCursor(),
-        }
-
-        self._update_palette()
-
-        self.viewer.events.interactive.connect(self._on_interactive)
-        self.viewer.cursor.events.style.connect(self._on_cursor)
-        self.viewer.cursor.events.size.connect(self._on_cursor)
-        self.viewer.events.palette.connect(self._update_palette)
-        self.viewer.layers.events.reordered.connect(self._reorder_layers)
-        self.viewer.layers.events.inserted.connect(self._on_add_layer_change)
-        self.viewer.layers.events.removed.connect(self._remove_layer)
-        # stop any animations whenever the layers change
-        self.viewer.events.layers_change.connect(lambda x: self.dims.stop())
-
-        self.setAcceptDrops(True)
-
-        for layer in self.viewer.layers:
-            self._add_layer(layer)
-
     def _create_performance_dock_widget(self):
         """Create the dock widget that shows performance metrics.
         """


Test patch: