Repository: vispy/vispy

Issue:
Implement a server-side GLIR cache
This is backend-independent. The GL context needs to store a list of (almost) all GLIR commands emitted since initialization. Unnecessary GLIR commands (old commands superseded by new commands) are automatically discarded (pruning), so that this list should never be very long.
- [ ] implement the cache (easy)
- [ ] dynamically prune old GLIR commands that are superseded by new ones: `DRAW`, `UNIFORM`, `FUNC` (slightly less straightforward)
- completely get rid of `on_initialize()` (can be done in another PR)
- fix the `ipynb_webgl` backend so that the latest state is automatically reinitialized when the context is lost (can be done in another PR)



Discussion:
Can we enumerate situations where the cache will not be able to completely reconstruct a dropped context? I only have one so far:  shaders that modify their textures. Anything else?

Sorry for chiming in so late. But I don't like the idea of such a cache so much. I understand that it has some advantages, but this cache can potentially take a whole lot of memory ...

Simple example that will fill bring your system into swapping in seconds: texture that gets random regions updated at each frame.

The idea was to intelligently merge requests like that. 
There also might not be a better solution..

If this is really all brought about by the IPython NB issue, an alternative solution is to tell people they can only (successfully) re-run a cell with a `.show()` if the `Canvas` instance is created in that cell. This seems like a much simpler solution, even if it does restrict the IPython use cases.

> The idea was to intelligently merge requests like that. 

That's very complex to do when random regions are updated. You'd have to keep track of smallest contiguous data etc. 

Can we get a summary of what the problem is exactly? I read it in the other issue, but it's not entirely clear to me.

This is not only about ipynb issue, as Luke pointed out. Limiting the ipynb framework is not an option for me, I really need this backend to work properly for my project.

@almarklein Only the last state will be saved, so there's really no way this could end up taking a lot of space. We can also add a hard limit (something like no more than 100 commands or something) if it proves to be a concern. But let me add that the current constant recompilation issue is much more dramatic in this respect compared to this potential future issue. ;)

> That's very complex to do when random regions are updated. You'd have to keep track of smallest contiguous data etc.

I don't think we should go there: we could just keep the last `DATA` call. We don't need this cached state to be perfectly representative. If there are very rare cases where this doesn't give exactly the expected result, that's no big deal. We'll tackle the problem when we'll see it. And at least the backend will not just crash...

Alternatively we could have an user option specifying how many `DRAW` calls should be saved in the cache.

Is it really that hard? For each buffer, we cache any creation commands and a single data command. Whenever a new data command is issued, just update the array cached with the original command.

Ah, that could work yes. 
Btw if we have an agreement I volunteer to implement this.

As a matter of fact, I did implement such "caching" mechanism by coaslescing updates into a single continuous block of memory to be uploaded to GPU. But this requires CPU storage.

So basically I guess this cache idea means taking your implementation of GPUBuffer with CPU-GPU synchronization and adapt it to the new GLIR architecture...

> Is it really that hard? For each buffer, we cache any creation commands and a single data command. 

But remember that you can update _part_ of a texture/buffer. So you cannot simply ignore all DATA commands but the last.

Even if we do manage to get this right, it means that we effectively all data on the CPU, which is something that we tried to avoid in the past... I'm just wondering whether there is another solution...

You can avoid it to some extent but it will be a bit more tricky. The main idea is to compute the largest contiguous block of any update operation:
1. If it is included in the current recorded update, just apply the update on the saved block (no extra CPU).
2. If it is larger (start_new < start_old or end_new  > end_old or both), the new block to be uploaded is a combination of old and new. This part might be tricky.

Yes, but even then, you are storing on the CPU _at least_ as much memory as there is on the GPU. 

The original problem was that in the ipython notebook, calling `canvas.show()` creates a new canvas. This seems weird, I would expect `canvas.show()` on a canvas that is already shown to be a no-op. I'd much rather fix _that_ instead.

@almarklein that's not how the IPython notebook works, unfortunately. There's no way around that.

I propose we go ahead with this cache, with an option to enable it or disable it. When using an ipynb backend, it will be enabled by default. When using a desktop backend, it will be disabled by default. It could also be enabled or disabled depending on the user's choice, and it would be documented. In particular, it would let users save a WYSIWYG version of a complex visualization, so it would not only be used for the ipynb backend.

> that's not how the IPython notebook works, unfortunately. There's no way around that.

I agree that the best overall solution would be to ensure GL contexts can't just be randomly dropped. Can you clarify why this is not possible? It seems like you have some custom javascript code that manages the webgl widget; why is it not possible to ensure this widget is safely stored before a cell is closed? This also seems like a simple enough problem that the ipython folks would consider fixing it, and possibly even simple enough that vispy could patch an existing ipython at runtime..

It's not a problem in IPython, it's not a bug either. It's just the way IPython works. :) When you have an IPython widget, it represents one view of a model. These views can be easily created and closed. There can be multiple views of a model. Models can also be deleted.

So the normal workflow in the IPython notebook is that widgets can be created and deleted lightly and easily. It is expected that the state will be saved in the model (Python-side); here, what would make the most sense would be to keep a cache of the GLIR commands, as you pointed out yourself.

The other solution is to force using `on_initialize()`, but I agree it is a less ideal solution than the cache.

The cache seems really elegant to me (and probably much simpler than "patching" IPython). Can you please precise why you don't like it?

In any case I hope we can find an agreement soon because this is rather urgent to me. If we don't, I'd have to use a separate branch or a fork to go on with my project.

On most other backends, the behavior is that you can hide and show the window without the GL context disappearing. This behavior of context dropping is unique to the webgl backend, and as you can see requires a lot of effort to address in gloo or elsewhere (and even the proposed cache solution is not perfect; it is impossible to completely reconstruct the GL state in all cases). So the best approach is just to ensure, on the client side, that the webgl context is tied to the canvas itself, rather than to the cell that appears when you call `show()`. It's fine if the widget appears in the `show()` cell, but it really should not be destroyed when the cell is closed; just hidden.

Would it be possible to create a hidden webgl widget with the Canvas, and then simply display it within the `show()` cell?

AFAIK that's impossible. The WebGL context is tied to the `<canvas>`. The user is free to close the canvas at any time, then both the `<canvas>` and the associated WebGL context are gone forever. There's really no way around that.

Even if the cache solution is not perfect, it will have the advantage of working...

Somewhere in ipython is a callback that gets invoked when the user closes or re-executes the cell, and this callback is responsible for destroying the contents of the cell. If we can hook into that callback, then it should be possible to remove the webgl canvas from the cell before it is destroyed, and subsequently to re-insert it to the new cell when it becomes available. 

Here's another way of stating the problem: the model/view architecture cannot accurately represent a vispy canvas because we do so much work on the GPU--a large part of our model is actually stored inside the view itself. In that respect it does not make much sense to support the destruction and creation of multiple views on the model. So the options are:
- Store all of the model on the CPU, outside the view  (the cache approach). 
- Make the webgl context conceptually part of the model, rather than part of the view (by preventing ipython from destroying it with the cell).

I'd be surprised if that could work. And even if it could, it looks like an ugly hack... The proper way to integrate Vispy in the notebook is to have state consistency between the client and the server: that's how the whole widget model works in IPython. Given that we do the rendering in the client, we've got to have a sort of copy of the GL context in the server. That's not necessarily easy; however, the recent GLIR changes make this much easier than before. The more I think about it, the more I'm convinced that's the right way to do it.

We'd appreciate the perspective of an IPython dev (@minrk @Carreau @takluyver @jdfreder) at this point. To sum up the situation:

We have a WebGL widget in the notebook. A client-side OpenGL context is tied to the `<canvas>` in the cell's widget area. This widget is displayed when our `canvas.show()` function is called; this function just calls `display_html()`.

When the user re-executes the cell, or when the cell is deleted, or when the user closes the widget, the canvas is destroyed, and the GL context is lost. It is therefore impossible to recreate the canvas in the state it was.

(I just see Luke's latest message above which completes this summary)

As a reminder, yet another alternative is to implement a mandatory initialization logic in Vispy, which we are not currently doing. The idea is that as soon as the context is lost for some reason, we recreate the widget from scratch to its initial state.

I hope we can fix this in IPython. I think the cache option is not elegant at all; it's nearly impossible to make it work in all cases, and has the side effect of consuming significant amounts of memory. The on_initialize option seems a much easier solution in that respect.

But on showing/closing the canvas: programatically, we can simply add `if self._visible: return` somewhere in `Canvas.show()`, so I see no problem there. This leaves us with the problem of closing the widget. Well, if you close a Qt canvas, we cannot re-use it either. So there seems to be no problem either. Or am I missing a point?

> I hope we can fix this in IPython

Again, there's nothing to "fix" in IPython...

> it's nearly impossible to make it work in all cases

It is no problem at all if the result is not 100% accurate in 100% of the cases.

> and has the side effect of consuming significant amounts of memory

It will consume peanuts. We'll just have a copy of the buffers on the CPU, as before, and only when using the IPython notebook. Nothing will change on the desktop backend since it can be disabled by default there!

> The on_initialize option seems a much easier solution in that respect.

Well, I'd be happy if we could agree on this! If you prefer this option rather than the cache, then let's go with that. @lcampagn @Eric89GXL ? The problem I see is with the visuals.

> But on showing/closing the canvas: programatically, we can simply add if self._visible: return somewhere in Canvas.show(), so I see no problem there

??

> Well, if you close a Qt canvas, we cannot re-use it either. So there seems to be no problem either. Or am I missing a point?

In most cases on the desktop, the application closes when you close the canvas. This is not the case in IPython.

Would you accept a PR with the GLIR cache option if it's strictly restricted to the IPython backend? There might be a way to confine the implementation to `_ipynb_webgl.py` (see [this](https://github.com/vispy/vispy/blob/master/vispy/app/backends/_ipynb_webgl.py#L108))

> The on_initialize option seems a much easier solution in that respect.

Unless I've misunderstood it, the `on_initialize` approach is essentially the same as the cache approach except that, instead of handling caching and context reset at the glir stage, we would push that responsibility out to the canvases, visuals, and users. The memory overhead would be the same, but the solution would be a much larger mess because the problem becomes a part of the public-facing API, rather than being handled internally at the glir stage (or better yet, at the ipython stage where the problem originates). 

> But on showing/closing the canvas: programatically, we can simply add if self._visible: return somewhere in Canvas.show()

The problem is that when you re-execute a cell in ipython, it is immediately removed and re-created. The context is dropped before we even get to show().

> The problem is that when you re-execute a cell in ipython, it is immediately removed and re-created. The context is dropped before we even get to show().

Ok, so what about a context is shown by default when it is instantiated? No need to do `c.show()`. Re-executing that cell will recreate the actual canvas. All should work nicely.

> Ok, so what about a context is shown by default when it is instantiated? No need to do c.show(). Re-executing that cell will recreate the actual canvas. All should work nicely.

We already ruled out this option. There are many situations where it'll fail, for example if you initialize gloo objects in a different cell.

If I manage to confine the implementation to the backend I think I'll just go ahead with the cache: it won't impact the other backends at all so there's no reason why you couldn't live with that! :)

> When the user re-executes the cell, or when the cell is deleted, or when the user closes the widget, the canvas is destroyed, and the GL context is lost. 

I'm not familiar with GL. What information is there in the 'context' that the frontend modifies after it is displayed?

I am in favor of the ipython-only cache since @rossant is in a hurry, I am rather strongly opposed to the `on_initialize` approach, and it is not yet clear whether it will be possible to prevent context loss in ipython. When we have more time to look into it, perhaps we can switch to the latter solution.

> I'm not familiar with GL. What information is there in the 'context' that the frontend modifies after it is displayed?

The context contains information about all data being drawn (vertices, textures, etc.), the program code that is being executed on the GPU, and all of the global state variables required by GL that set up the viewport and rendering modes. There are a few cases where this data can exist on the GPU in a state that was never present on the CPU (for example, if you upload a full texture, and then later upload new data to a sub-region of that texture, then the complete texture only ever exists on the GPU). 

More importantly, there are many cases where the user-specified data is sent directly to the GPU without any local storage within vispy. If the context is dropped, then all of this data / code / state must be sent again to the GPU, hence the need for some kind of cache that would handle this (or some way to prevent context loss in the first place).

OK, so you can recreate the state perfectly, but it might be slow? I don't think that's something IPython can immediately help with, although if we can do things that make life easier, we'd be interested to know about that.

> We already ruled out this option. There are many situations where it'll fail, for example if you initialize gloo objects in a different cell.

That should still work as expected. (Come to think of it, I think we need to disable the `gloo.Context` when a canvas is closed), but if the context mechanism works as it should, gloo objects are created in the current context, or are bound to the next context that _will_ be created.

> OK, so you can recreate the state perfectly, but it might be slow? I don't think that's something IPython can immediately help with, although if we can do things that make life easier, we'd be interested to know about that.

With a bit of work, we can recreate the state accurately in most cases. I don't think it would be particularly slow. That's the solution I'd prefer, but I have yet to convince the other devs. :)

They'd rather solve the context loss problem in IPython directly. But AFAIK there's no way around that: when you close a widget, the HTML elements are completely deleted, right? What they have in mind is a solution where we'd save somehow the GL context in the browser when the canvas is deleted, and put it back once it's recreated. I think it would be very hard and clunky to do that in IPython, and it might actually be something that is impossible with WebGL anyway.

The HTML elements are deleted, but it may be possible to attach it to the model. If you display the same widget twice, do the views share a GL context, or do they have one each?

They have one each: there is no context sharing in WebGL (contrary to desktop OpenGL). That being said, in the way things are currently implemented, the contexts are _effectively_ identical (in other ways, there is some sort of CPU-based context sharing rather than GPU-based). It might be because comm messages are received by both widgets in the notebook?

In our architecture, comm messages go to the model; if you have multiple views, they should stay in sync. But if you can't share a context, I'm not sure it makes sense to store it in the model.

@takluyver, where we hope the IPython devs can help is in deciding how we can keep the context from disappearing when the cell is cleared. Basically we have two statements:

```
# cell 1
canvas = vispy.app.Canvas()
```

```
# cell 2
canvas.show()
```

The second cell is the one that creates the `<canvas>` HTML element, and thus contains the WebGL context. If that cell is re-executed, then the `<canvas>` disappears and the GL context goes with it. 

So: would it be possible to create a hidden `<canvas>` in the first cell that is subsequently shown in the second cell? And more importantly, would it be possible to add a callback that allows us to detect when the second cell is about to be destroyed so that we can preserve the `<canvas>`?

If you can create a hidden canvas with an attached GL context, and then display it later, why can you not then display it a second time, so you have two views with the same context? If that were possible, it would make sense to construct the GL context along with the model.

I'll have to make some tests with WebGL but I'm pretty sure that it's not
possible. The context is tied to an individual canvas.

Le vendredi 14 novembre 2014, Thomas Kluyver notifications@github.com a
écrit :

> If you can create a hidden canvas with an attached GL context, and then
> display it later, why can you not then display it a second time, so you
> have two views with the same context? If that were possible, it would make
> sense to construct the GL context along with the model.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/vispy/vispy/issues/615#issuecomment-62986600.

Remember when you proposed to just recreate the canvas in order to work around the problem, and I said it didn't work? Well, apparently it's a bug with glir/context that occurs even outside the notebook. In interactive mode (ipython notebook, console, or python terminal), create and show a canvas once, it works, recreate it and reshow it, it is empty. If this bug is fixed, there might be an easy fix to the notebook problem.

It looks like the relevant GLIR commands (that create the program, the buffers, etc.) do not get emitted at all the second time the canvas is created.

For the Qt backend, there is a line in app/backends/_qt.py ~L323 that shuts down the GL context when the window is closed (the line should not be there, but that can be discussed elsewhere). If I remove that line, then I am able to close and reopen a canvas window without trouble. The ipython_webgl backend has not implemented `_vispy_close()` at all, so I do not think this is the problem.

No, the problem is still there in the Qt backend if I remove that line. Note that the problem happens when I _recreate_ a new Canvas instance and show it again.

How do you "recreate" a canvas? Can you post your python session? This is what I did:

```
>>> import vispy.scene as vs
>>> c = vs.SceneCanvas(show=True)
>>> r = vs.visuals.Rectangle(pos=(10, 10), height=100, width=100, color=(1, 0, 0, 1), parent=c.scene)
[ now close the window]
>>> c.show()
```

Ah, I do not use the scene stuff.

My script `_test.py`:

``` python
import numpy as np
from vispy import app
from vispy import gloo

VS = '''
attribute vec2 a_position;
void main() {
    gl_Position = vec4(a_position, 0., 1.);
    gl_PointSize = 3.;
}
'''

FS = '''
void main() {
    gl_FragColor = vec4(1., 0., 0., 1.);
}
'''

class MyCanvas(app.Canvas):
    def __init__(self):
        super(MyCanvas, self).__init__(keys='interactive')
        self.program = gloo.Program(VS, FS)
        self.program['a_position'] = .25 * np.random.randn(100000, 2).astype(np.float32)

    def on_draw(self, event):
        gloo.clear('black')
        gloo.set_viewport((0, 0,) + self.size)
        self.program.draw('points')

def _show():
    c = MyCanvas()
    c.show()
```

then

```
python -i _test.py
>>> _show()  # OK, close the window
>>> _show()  # empty
```

There's a mystery. Maybe this needs to be resolved as a separate issue before we continue here.

I agree. If this is solved, I think the ipynb issue can be partly solved very easily, and we can put back the whole cache discussion to later. The temporary solution (that should be good enough) would be to just recreate the canvas from scratch when the context is lost. Let me add a new issue.

Sorry for the slow response. +1 for avoiding the cache if possible, so if we can just re-show the canvas by fixing the bug, that is a _much_ simpler solution and probably prone to far fewer problems.


Resolving patch:
diff --git a/vispy/app/backends/_ipynb_webgl.py b/vispy/app/backends/_ipynb_webgl.py
--- a/vispy/app/backends/_ipynb_webgl.py
+++ b/vispy/app/backends/_ipynb_webgl.py
@@ -14,6 +14,7 @@
 from ...util import logger, keys
 from ...ext import six
 from vispy.gloo.glir import BaseGlirParser
+from vispy.app import Timer
 
 # Import for displaying Javascript on notebook
 import os.path as op
@@ -60,7 +61,7 @@
 
 
 # ------------------------------------------------------------- application ---
-def _prepare_js():
+def _prepare_js(force=False):
     pkgdir = op.dirname(__file__)
     jsdir = op.join(pkgdir, '../../html/static/js/')
     # Make sure the JS files are installed to user directory (new argument
@@ -69,8 +70,7 @@ def _prepare_js():
         kwargs = {'user': True}
     else:
         kwargs = {}
-    install_nbextension([op.join(jsdir, 'vispy.min.js'),
-                         op.join(jsdir, 'jquery.mousewheel.min.js')],
+    install_nbextension(op.join(jsdir, 'vispy.min.js'), overwrite=force,
                         **kwargs)
     backend_path = op.join(jsdir, 'webgl-backend.js')
     with open(backend_path, 'r') as f:
@@ -139,7 +139,7 @@ def __init__(self, *args, **kwargs):
         self._create_widget(size=size)
 
     def _create_widget(self, size=None):
-        self._widget = VispyWidget(self._gen_event, size=size)
+        self._widget = VispyWidget(self, size=size)
         # Set glir parser on context and context.shared
         context = self._vispy_canvas.context
         context.shared.parser = WebGLGlirParser(self._widget)
@@ -312,18 +312,32 @@ def _vispy_stop(self):
 
 
 # ---------------------------------------------------------- IPython Widget ---
+def _stop_timers(canvas):
+    """Stop all timers in a canvas."""
+    for attr in dir(canvas):
+        try:
+            attr_obj = getattr(canvas, attr)
+        except NotImplementedError:
+            # This try/except is needed because canvas.position raises
+            # an error (it is not implemented in this backend).
+            attr_obj = None
+        if isinstance(attr_obj, Timer):
+            attr_obj.stop()
+
+
 class VispyWidget(DOMWidget):
     _view_name = Unicode("VispyView", sync=True)
 
     width = Int(sync=True)
     height = Int(sync=True)
 
-    def __init__(self, gen_event, **kwargs):
+    def __init__(self, canvas_backend, **kwargs):
         super(VispyWidget, self).__init__(**kwargs)
         w, h = kwargs.get('size', (500, 200))
         self.width = w
         self.height = h
-        self.gen_event = gen_event
+        self.canvas_backend = canvas_backend
+        self.gen_event = canvas_backend._gen_event
         self.on_msg(self.events_received)
 
     def events_received(self, _, msg):
@@ -331,6 +345,10 @@ def events_received(self, _, msg):
             events = msg['contents']
             for ev in events:
                 self.gen_event(ev)
+        elif msg['msg_type'] == 'status':
+            if msg['contents'] == 'removed':
+                # Stop all timers associated to the widget.
+                _stop_timers(self.canvas_backend._vispy_canvas)
 
     def send_glir_commands(self, commands):
         # TODO: check whether binary websocket is available (ipython >= 3)


Test patch: