Repository: bokeh/bokeh

Issue:
Delay between autoload.js and websocket request
# READ AND FOLLOW THESE INSTRUCTIONS CAREFULLY

*ISSUES THAT DO NOT CONTAIN NECESSARY INFORMATION MAY BE CLOSED, IMMEDIATELY*

The issue tracker is NOT the place for general support. For questions and
technical assistance, come ask the [Bokeh mailing list](https://groups.google.com/a/continuum.io/forum/#!forum/bokeh) or join the chat on [Gitter](https://gitter.im/bokeh/bokeh). For feature requests, please provide a detailed description or proposal of the new capability or behavior.

For defects or deficiencies, please provide ALL OF THE FOLLOWING:

#### ALL software version info (bokeh, python, notebook, OS, browser, any other relevant packages)

bokeh 1.0.2
python 3.6.5
OS CentOS-7.5.1804

#### Description of expected behavior and the observed behavior

For whatever reason, it appears that on some requests, there can be a significant delay between the autoload.js request and the subsequent websocket connection. Normally, this process takes no more than 1-2 seconds:

```
doc.session_context.request.arguments: {'bokeh-autoload-element': [b'1088'], 'bokeh-app-path': [b'/graphs/enviz_graphs'], 'bokeh-absolute-url': [b'https://*redacted*/graphs/enviz_graphs'], 'processor_id': [b'83,187,196,1114,206,335,536,212,214,1173,217,250,252,256,265,876,268,298,999']}
2019-01-18 22:44:45,794 root_url should end with a /, adding one
2019-01-18 22:44:45,797 200 GET /graphs/enviz_graphs/autoload.js?bokeh-autoload-element=1089&bokeh-app-path=/graphs/enviz_graphs&bokeh-absolute-url=https://*redacted*/graphs/enviz_graphs&processor_id=83%2C187%2C196%2C1114%2C206%2C335%2C536%2C212%2C214%2C1173%2C217%2C250%2C252%2C256%2C265%2C876%2C268%2C298%2C999 (10.50.1.159) 398.52ms
2019-01-18 22:44:47,291 101 GET /graphs/enviz_graphs/ws?bokeh-protocol-version=1.0&bokeh-session-id=ImqIQZ1sbiZS4KsAOocVHGFgUGfJJLwHxG44Irv9Xls9&pid=83,187,196,1114,206,335,536,212,214,1173,217,250,252,256,265,876,268,298,999 (10.50.1.159) 0.56ms
2019-01-18 22:44:47,291 WebSocket connection opened
2019-01-18 22:44:47,291 Receiver created for Protocol('1.0')
2019-01-18 22:44:47,291 ProtocolHandler created for Protocol('1.0')
2019-01-18 22:44:47,291 ServerConnection created
2019-01-18 22:44:47,350 Sending pull-doc-reply from session 'ImqIQZ1sbiZS4KsAOocVHGFgUGfJJLwHxG44Irv9Xls9'
```

Notice the autoload request at 22:44:45 and the ws request at 22:44:47. (2 seconds)

However, sometimes the ws request can arrive nearly a minute later:

```
doc.session_context.request.arguments: {'bokeh-autoload-element': [b'1090'], 'bokeh-app-path': [b'/graphs/enviz_graphs'], 'bokeh-absolute-url': [b'https://*redacted*/graphs/enviz_graphs'], 'processor_id': [b'83,187,196,1114,206,335,536,212,214,1173,217,250,252,256,265,876,268,298,300,1347,1350,1352,284,307,1115,1229,999,92,']}
2019-01-18 22:45:10,741 root_url should end with a /, adding one
2019-01-18 22:45:10,745 200 GET /graphs/enviz_graphs/autoload.js?bokeh-autoload-element=1090&bokeh-app-path=/graphs/enviz_graphs&bokeh-absolute-url=https://*redacted*/graphs/enviz_graphs&processor_id=83%2C187%2C196%2C1114%2C206%2C335%2C536%2C212%2C214%2C1173%2C217%2C250%2C252%2C256%2C265%2C876%2C268%2C298%2C300%2C1347%2C1350%2C1352%2C284%2C307%2C1115%2C1229%2C999%2C92%2C (10.50.1.159) 392.75ms
2019-01-18 22:45:35,357 Scheduling 1 sessions to discard
2019-01-18 22:45:35,357 Discarding session '1fz6E0KuyuaCaCscdyKLyI2YJze38csKNckNQkotkrE8' last in use 24616.089113235474 milliseconds ago
2019-01-18 22:45:35,358 Deleting 1 modules for <bokeh.document.document.Document object at 0x7f8bb89f8438>
2019-01-18 22:45:50,352 [pid 11775] 1 clients connected
2019-01-18 22:45:50,352 [pid 11775]   /enviz_graphs has 1 sessions with 0 unused
2019-01-18 22:46:05,562 101 GET /graphs/enviz_graphs/ws?bokeh-protocol-version=1.0&bokeh-session-id=1fz6E0KuyuaCaCscdyKLyI2YJze38csKNckNQkotkrE8&pid=83,187,196,1114,206,335,536,212,214,1173,217,250,252,256,265,876,268,298,300,1347,1350,1352,284,307,1115,1229,999,92, (10.50.1.159) 0.58ms
2019-01-18 22:46:05,562 WebSocket connection opened
doc.session_context.request.arguments: {'pid': [b'83,187,196,1114,206,335,536,212,214,1173,217,250,252,256,265,876,268,298,300,1347,1350,1352,284,307,1115,1229,999,92,']}
2019-01-18 22:46:05,563 Error running application handler <bokeh.application.handlers.directory.DirectoryHandler object at 0x7f8bb8f75cf8>: local variable 'current_pids' referenced before assignment
File "env_frontend.py", line 30, in modify_doc:
if len(current_pids) < 1: Traceback (most recent call last):
File "/*redacted*/enviz/venv/lib64/python3.6/site-packages/bokeh/application/handlers/code_runner.py", line 180, in run
exec(self._code, module.__dict__)
File "/*redacted*/enviz/venv/lib/python3.6/site-packages/enviz_graphs/main.py", line 7, in <module>
modify_doc(doc)
File "/*redacted*/enviz/venv/lib/python3.6/site-packages/enviz_graphs/env_frontend.py", line 30, in modify_doc
if len(current_pids) < 1:
UnboundLocalError: local variable 'current_pids' referenced before assignment
2019-01-18 22:46:05,563 Receiver created for Protocol('1.0')
2019-01-18 22:46:05,563 ProtocolHandler created for Protocol('1.0')
2019-01-18 22:46:05,563 ServerConnection created
2019-01-18 22:46:05,631 Sending pull-doc-reply from session '1fz6E0KuyuaCaCscdyKLyI2YJze38csKNckNQkotkrE8'
```

Notice the autoload request at 22:45:10 and the ws request at 22:46:05. (55 seconds)

In that gap, it appears that the session created by the autoload request was discarded as being unused at 22:46:05. (we have the default 15000 ms timeout for that.)

In both cases, the request for autoload.js takes less than 400 ms, so the slowdown seems like it would be in the browser, though I don't yet have any browser profiling that caught it.

Then, when the ws request comes in, it tries to create a new session, but fails to run our module, as the correct keys aren't in doc.session_context.request.arguments.

After this, every request to the bokeh server fails at requesting autoload until we restart the server, as it appears that doc.session_context.request.arguments is always None after that.

#### Complete, minimal, self-contained example code that reproduces the issue

N/A

#### Stack traceback and/or browser JavaScript console output

N/A

#### Screenshots or screencasts of the bug in action

N/A



Discussion:
@gotyaoi I have never encountered anything like this, or heard of anything like it before, either. Without some lead to reproduce this there is essentially zero chance we will be able to investigate anything, and this will have to be closed as `norepro`. The only thing I can possibly suggest offhand is that your script evidently has logic/syntax errors (see the reported tracebacks), and it it possible that these are implicated. 
Sorry, it's been a long day of even getting this far in debugging this. I will continue debugging it, including trying to trace a request in the browser and providing repro code, after the long weekend.

Regarding the stack trace, it only occurs in this case because the script is expecting a key in the request parameters that isn't there when the session is created the second time at the ws request.
Ok, so after more investigation, in our case the delay between the autoload.js request and the ws request was caused by unreliable wifi. However, I believe I have a reliable way to reproduce the behavior we saw afterwards. Interestingly, it appears to exhibit different behavior when run from a single file, as opposed to when run from a directory.

### flask_app.py
```
import flask
import bokeh.embed

app = flask.Flask('testapp')

template = '''<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <link rel="stylesheet" href="https://cdn.pydata.org/bokeh/release/bokeh-0.12.13.min.css" type="text/css" />
        <script type="text/javascript" src="https://cdn.pydata.org/bokeh/release/bokeh-0.12.13.min.js"></script>
        <title>testapp</title>
    </head>
    <body>
        {{ script|safe }}
    </body>
</html>'''

@app.route('/')
def home():
    fig = flask.request.args.get('type')
    script = bokeh.embed.server_document('http://localhost:5006/bokeh_app', arguments={"fig": fig})
    return flask.render_template_string(template, script=script)
```

### main.py (lifted mostly from https://github.com/bokeh/bokeh/blob/master/examples/app/sliders.py)
```
import numpy as np

from bokeh.plotting import curdoc
from bokeh.layouts import row, widgetbox
from bokeh.models import ColumnDataSource
from bokeh.models.widgets import Slider, TextInput
from bokeh.plotting import figure

doc = curdoc()

# Set up data
N = 200
x = np.linspace(0, 4*np.pi, N)
fig = doc.session_context.request.arguments['fig'][0].decode('utf-8')
if fig == 'cos':
    func = np.cos
elif fig == 'tan':
    func = np.tan
else:
    fig = 'sin'
    func = np.sin
y = func(x)
source = ColumnDataSource(data=dict(x=x, y=y))

# Set up plot
plot = figure(plot_height=400, plot_width=400, title="my sine wave",
              tools="crosshair,pan,reset,save,wheel_zoom",
              x_range=[0, 4*np.pi], y_range=[-2.5, 2.5])

plot.line('x', 'y', source=source, line_width=3, line_alpha=0.6)

# Set up widgets
text = TextInput(title="title", value='my {0} wave'.format(fig))
offset = Slider(title="offset", value=0.0, start=-5.0, end=5.0, step=0.1)
amplitude = Slider(title="amplitude", value=1.0, start=-5.0, end=5.0, step=0.1)
phase = Slider(title="phase", value=0.0, start=0.0, end=2*np.pi)
freq = Slider(title="frequency", value=1.0, start=0.1, end=5.1, step=0.1)

# Set up callbacks
def update_title(attrname, old, new):
    plot.title.text = text.value

text.on_change('value', update_title)

def update_data(attrname, old, new):

    # Get the current slider values
    a = amplitude.value
    b = offset.value
    w = phase.value
    k = freq.value

    # Generate the new curve
    x = np.linspace(0, 4*np.pi, N)
    y = a*func(k*x + w) + b

    source.data = dict(x=x, y=y)

for w in [offset, amplitude, phase, freq]:
    w.on_change('value', update_data)

# Set up layouts and add to document
inputs = widgetbox(text, offset, amplitude, phase, freq)

doc.add_root(row(inputs, plot, width=800))
doc.title = "Sliders"
```

### in first terminal
```
mkdir test
cd test
python3 -m venv venv
./venv/bin/pip install bokeh flask numpy
(write flask_app.py into this directory)
mkdir bokeh_app
cd bokeh_app
(write main.py into this directory)
cd ..
./venv/bin/bokeh serve --allow-websocket-origin localhost:5000 --log-level=debug bokeh_app
```

### in second terminal
```
cd test
export FLASK_APP=flask_app.py
./venv/bin/flask run
```

### edit venv/lib/python3.?/site-packages/bokeh/core/_templates/autoload_js.js
add `alert("please wait 30 seconds, then click ok.")` below `(function(root) {`. This will simulate the unreliable wifi by blocking the execution of sutoload.js in the browser until you dismiss the modal.

### in browser
visit http://localhost:5000?type=tan and dismiss the modal immediately. tangent function graph should load.
visit http://localhost:5000?type=tan and wait 30 seconds before dismissing the modal (you will see the `Discarding session '...' last in use ... milliseconds ago` message in the logs. tangent function graph should fail to load, as the session created by the autoload request was garbage collected before the ws request came through and the session created by the ws request didn't have the correct parameters.
all subsequent requests to the server should fail, no matter how quickly the modal is dismissed, until the bokeh is restarted.

now, assuming that instead of a bokeh_app directory with a main.py, you had a bokeh_app.py file with the contents of main.py, the request where you wait 30 seconds would fail, but subsequent requests would succeed without the need to restart bokeh.
For the problem that occurs in both scenarios, I think the solution is not to create the session at all on the autoload.js request, and also to have autoload.js call the websocket endpoint with the same parameters it was called with.

As for why the server breaks after the exception when operating in directory mode, I have no idea.
@gotyaoi I think that this:

> the session created by the autoload request was garbage collected before the ws request came through 

could be mitigated by setting different values for session timeout and collection? If that's the case I would say I'd be more inclined to reduce risk and work and simply leave this issue as documentation. Can you try adjusting those parameters and reporting back?

Alternatively, if you can make a PR that has the changes you are proposing we would be happy to consider it. 
I mean, that might work until the really bad case comes and a request takes like... a 5 minute delay or something.

I will work on a pull request next week, and also try to investigate the difference between directory mode and single file mode.
@gotyaoi did you still intend to do any work on this issue? If not I am inclined to close, as I think extremely long (minutes) request times are by far the exception, and if a user is in that situation they will just have to up the timeout/collection appropriately for their specialized use case.
That may be the case. I'm currently looking for why it breaks until restart in directory mode and not single file mode, as that's something that seems to be the case with all exceptions, not just this case. It would probably be fine if just the one request failed in the bad case.
Sorry this took so long. I've traced the problem to these two lines in the Directory Handler: https://github.com/bokeh/bokeh/blob/master/bokeh/application/handlers/directory.py#L179-L180. Basically, it checks if the embedded ScriptHandler has encountered a failure and if so, it bails out. Allowing it to invoke the embedded ScriptHandler's modify_document method clears the error flag first thing.

Commenting those out allows the server to recover from an exception similarly to a bare ScriptHandler. I don't see any negative consequences from removing those lines in superficial testing, but I wanted to ask before opening a PR.
@gotyaoi I think the issue is that the `failed` property encompasses too much (both script and lifecycle handler failure). I think that at the top of `modify_doc` there, we *do* want to bail if `self._lifecycle_handler` has failed (but should not check the script handler since it has not even been run this time around yet). Can you test out that change, and if it works, a PR would be great.
Yeah, switching `self.failed` to `self._lifecycle_handler.failed` also seems to work. I will submit a PR in a little bit.

Resolving patch:
diff --git a/bokeh/application/handlers/directory.py b/bokeh/application/handlers/directory.py
--- a/bokeh/application/handlers/directory.py
+++ b/bokeh/application/handlers/directory.py
@@ -176,7 +176,7 @@ def modify_document(self, doc):
         if they are found.
 
         '''
-        if self.failed:
+        if self._lifecycle_handler.failed:
             return
         # Note: we do NOT copy self._theme, which assumes the Theme
         # class is immutable (has no setters)


Test patch: