Repository: bokeh/bokeh

Issue:
Unhandled exception in application causes all later sessions to fail
## The problem

In a "bokeh serve" application, if there is an unhandled exception in the main code, all new sessions after the exception do not work.

Background: https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/BZpiJJ4TO2M

## Expected behaviour

The preferred behaviour would be that an unhandled exception would only affect the session that caused the exception, and sessions started later would continue to work. Other frameworks typically isolate exceptions so that if a request causes an exception, only that request is affected.

## Versions

Bokeh: 0.13.0
Python: 3.5.2

## Example application

The below example application is copied from https://bokeh.pydata.org/en/latest/docs/user_guide/server.html#single-module-format, except it raises an exception in 33% of new sessions.

To duplicate the issue, do:
1. Run "bokeh serve myapp.py"
2. Open a browser to http://localhost:5006/boom. If you don't see "Exception: boom!" in the console, you can click the "Press Me" button to see numbers appear.
3. Refresh the page until you see "Exception: boom!" in the console. The Bokeh app won't appear in the browser.
4. Refresh the page again. The app still doesn't appear. This demonstrates that later sessions are affected.
5. Restart the server and refresh the page. If the exception isn't raised, the app will load. It will work until a new session triggers the exception.

```
# myapp.py

from random import random

from bokeh.layouts import column
from bokeh.models import Button
from bokeh.palettes import RdYlBu3
from bokeh.plotting import figure, curdoc

# create a plot and style its properties
p = figure(x_range=(0, 100), y_range=(0, 100), toolbar_location=None)
p.border_fill_color = 'black'
p.background_fill_color = 'black'
p.outline_line_color = None
p.grid.grid_line_color = None

# add a text renderer to our plot (no data yet)
r = p.text(x=[], y=[], text=[], text_color=[], text_font_size="20pt",
           text_baseline="middle", text_align="center")

i = 0

ds = r.data_source

# create a callback that will add a number in a random location
def callback():
    global i
    
    # BEST PRACTICE --- update .data in one step with a new dict
    new_data = dict()
    new_data['x'] = ds.data['x'] + [random()*70 + 15]
    new_data['y'] = ds.data['y'] + [random()*70 + 15]
    new_data['text_color'] = ds.data['text_color'] + [RdYlBu3[i%3]]
    new_data['text'] = ds.data['text'] + [str(i)]
    ds.data = new_data

    i = i + 1

# add a button widget and configure with the call back
button = Button(label="Press Me")
button.on_click(callback)

boom = random()
print(boom)
if boom < 0.3333:
  raise Exception('boom!')

# put the button and plot in a layout and add to the document
curdoc().add_root(column(button, p))
```

Example console output:

```
(venv) $ bokeh serve myapp.py 
2018-06-27 12:59:45,343 Starting Bokeh server version 0.13.0 (running on Tornado 5.0.1)
2018-06-27 12:59:45,345 Bokeh app running at: http://localhost:5006/myapp
2018-06-27 12:59:45,345 Starting Bokeh server with process id: 13979
0.5604607908584933
2018-06-27 13:00:06,274 200 GET /myapp (::1) 96.25ms
2018-06-27 13:00:06,437 101 GET /myapp/ws?bokeh-protocol-version=1.0&bokeh-session-id=OxhSy81ZBM8URWfk6wp7OTH7bTictF12u6pI8o7fkIUs (::1) 0.64ms
2018-06-27 13:00:06,437 WebSocket connection opened
2018-06-27 13:00:06,438 ServerConnection created
0.6452850662545722
0.2762922805351794
2018-06-27 13:00:11,120 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x7feae4861a58>: boom!
File "myapp.py", line 46, in <module>:
raise Exception('boom!') Traceback (most recent call last):
  File "/home/matt/Clients/QEye/venv/lib/python3.5/site-packages/bokeh/application/handlers/code_runner.py", line 163, in run
    exec(self._code, module.__dict__)
  File "/home/matt/Clients/QEye/isolate/myapp.py", line 46, in <module>
    raise Exception('boom!')
Exception: boom!
 
2018-06-27 13:00:11,125 200 GET /myapp (::1) 25.45ms
2018-06-27 13:00:11,132 WebSocket connection closed: code=1001, reason=None
2018-06-27 13:00:11,207 101 GET /myapp/ws?bokeh-protocol-version=1.0&bokeh-session-id=Y5IlTpoIxVs42drth0tKZonyuwGnUNM3zKK6OAG5eg6n (::1) 0.60ms
2018-06-27 13:00:11,208 WebSocket connection opened
2018-06-27 13:00:11,208 ServerConnection created
```


Discussion:
Offhand, catching exceptions around here is probably what is necessary

https://github.com/bokeh/bokeh/blob/70f111deb85ee8a9b5b2de845b24b439d64c93a0/bokeh/application/application.py#L153-L174
Actually issue is that once a hander is marked "failed" it is never reset before the next use. 
Here is a minimal diff that fixes things, but more work is needed, there may be similar changes needed elsewhwere, as well as module cleanup or other explicit work that needs to happen to avoid memory leaks, etc. 
```
(base) ❯ git diff
diff --git a/bokeh/application/handlers/code.py b/bokeh/application/handlers/code.py
index 8b96158f5..5a40eb90e 100644
--- a/bokeh/application/handlers/code.py
+++ b/bokeh/application/handlers/code.py
@@ -135,9 +135,6 @@ class CodeHandler(Handler):
         '''

         '''
-        if self.failed:
-            return
-
         module = self._runner.new_module()

         # One reason modules are stored is to prevent the module
@@ -151,7 +148,6 @@ class CodeHandler(Handler):
         old_doc = curdoc()
         set_curdoc(doc)
         old_io = self._monkeypatch_io()
-
         try:
             def post_check():
                 newdoc = curdoc()
diff --git a/bokeh/application/handlers/code_runner.py b/bokeh/application/handlers/code_runner.py
index f2135875e..62a13e4fc 100644
--- a/bokeh/application/handlers/code_runner.py
+++ b/bokeh/application/handlers/code_runner.py
@@ -130,9 +130,6 @@ class CodeRunner(object):
             Module

         '''
-        if self.failed:
-            return None
-
         module_name = 'bk_script_' + make_id().replace('-', '')
         module = ModuleType(str(module_name)) # str needed for py2.7
         module.__dict__['__file__'] = os.path.abspath(self._path)
@@ -150,6 +147,9 @@ class CodeRunner(object):
                 if expected post-conditions are not met after code execution.

         '''
+        self._failed = False
+        self._error = None
+        self._error_detail = None
         try:
             # Simulate the sys.path behaviour decribed here:
             #
```
Ok, more specifically, this was the way it was because `CodeRunner` may detect and fail on bad code during initialization. So we can't just "reset" the failed state every time, at least not entirely. Basically the code is not currently distinguishing between:

* permanent failure ("the source file has a syntax error") 
* transient failure ("one call to `modify_doc` raised, but others might not") 

I think adding a second internal flag to `CodeRunner` will alleviate the issue, though I still recommend guarding against exceptions escaping in the first place as the best practice for users to take. 

Resolving patch:
diff --git a/bokeh/application/handlers/code.py b/bokeh/application/handlers/code.py
--- a/bokeh/application/handlers/code.py
+++ b/bokeh/application/handlers/code.py
@@ -135,11 +135,14 @@ def modify_document(self, doc):
         '''
 
         '''
-        if self.failed:
-            return
 
         module = self._runner.new_module()
 
+        # If no module was returned it means the code runner has some permanent
+        # unfixable problem, e.g. the configured source code has a syntax error
+        if module is None:
+            return
+
         # One reason modules are stored is to prevent the module
         # from being gc'd before the document is. A symptom of a
         # gc'd module is that its globals become None. Additionally
diff --git a/bokeh/application/handlers/code_runner.py b/bokeh/application/handlers/code_runner.py
--- a/bokeh/application/handlers/code_runner.py
+++ b/bokeh/application/handlers/code_runner.py
@@ -62,9 +62,9 @@ def __init__(self, source, path, argv):
                 as ``sys.argv`` when the code executes
 
         '''
-        self._failed = False
-        self._error = None
-        self._error_detail = None
+        self._permanent_error = None
+        self._permanent_error_detail = None
+        self.reset_run_errors()
 
         import ast
         self._code = None
@@ -73,10 +73,10 @@ def __init__(self, source, path, argv):
             nodes = ast.parse(source, path)
             self._code = compile(nodes, filename=path, mode='exec', dont_inherit=True)
         except SyntaxError as e:
-            self._failed = True
-            self._error = ("Invalid syntax in \"%s\" on line %d:\n%s" % (os.path.basename(e.filename), e.lineno, e.text))
             import traceback
-            self._error_detail = traceback.format_exc()
+            self._code = None
+            self._permanent_error = ("Invalid syntax in \"%s\" on line %d:\n%s" % (os.path.basename(e.filename), e.lineno, e.text))
+            self._permanent_error_detail = traceback.format_exc()
 
         self._path = path
         self._source = source
@@ -90,21 +90,21 @@ def error(self):
         ''' If code execution fails, may contain a related error message.
 
         '''
-        return self._error
+        return self._error if self._permanent_error is None else self._permanent_error
 
     @property
     def error_detail(self):
         ''' If code execution fails, may contain a traceback or other details.
 
         '''
-        return self._error_detail
+        return self._error_detail if self._permanent_error_detail is None else self._permanent_error_detail
 
     @property
     def failed(self):
         ''' ``True`` if code execution failed
 
         '''
-        return self._failed
+        return self._failed or self._code is None
 
     @property
     def path(self):
@@ -130,7 +130,9 @@ def new_module(self):
             Module
 
         '''
-        if self.failed:
+        self.reset_run_errors()
+
+        if self._code is None:
             return None
 
         module_name = 'bk_script_' + make_id().replace('-', '')
@@ -139,6 +141,17 @@ def new_module(self):
 
         return module
 
+    def reset_run_errors(self):
+        ''' Clears any transient error conditions from a previous run.
+
+        Returns
+            None
+
+        '''
+        self._failed = False
+        self._error = None
+        self._error_detail = None
+
     def run(self, module, post_check):
         ''' Execute the configured source code in a module and run any post
         checks.
@@ -167,7 +180,7 @@ def run(self, module, post_check):
             self._failed = True
             self._error_detail = traceback.format_exc()
 
-            exc_type, exc_value, exc_traceback = sys.exc_info()
+            _exc_type, _exc_value, exc_traceback = sys.exc_info()
             filename, line_number, func, txt = traceback.extract_tb(exc_traceback)[-1]
 
             self._error = "%s\nFile \"%s\", line %d, in %s:\n%s" % (str(e), os.path.basename(filename), line_number, func, txt)
diff --git a/bokeh/application/handlers/notebook.py b/bokeh/application/handlers/notebook.py
--- a/bokeh/application/handlers/notebook.py
+++ b/bokeh/application/handlers/notebook.py
@@ -72,7 +72,7 @@ def __init__(self, *args, **kwargs):
         with open(filename) as f:
             nb = nbformat.read(f, nbformat.NO_CONVERT)
             exporter = nbconvert.PythonExporter()
-            source, meta = exporter.from_notebook_node(nb)
+            source, _ = exporter.from_notebook_node(nb)
             kwargs['source'] = source
 
         super(NotebookHandler, self).__init__(*args, **kwargs)


Test patch:
diff --git a/bokeh/application/handlers/tests/test_code_runner.py b/bokeh/application/handlers/tests/test_code_runner.py
--- a/bokeh/application/handlers/tests/test_code_runner.py
+++ b/bokeh/application/handlers/tests/test_code_runner.py
@@ -18,13 +18,17 @@
 #-----------------------------------------------------------------------------
 
 # Standard library imports
+import os
+from os.path import abspath, dirname
+import sys
+from types import ModuleType
 
 # External imports
 
 # Bokeh imports
 
 # Module under test
-#import bokeh.application.handlers.code_runner as bahc
+import bokeh.application.handlers.code_runner as bahc
 
 #-----------------------------------------------------------------------------
 # Setup
@@ -34,6 +38,119 @@
 # General API
 #-----------------------------------------------------------------------------
 
+class TestCodeRunner(object):
+
+    def test_init(self):
+        cr = bahc.CodeRunner("# test", "path", [])
+        assert cr.failed is False
+        assert cr.error is None
+        assert cr.error_detail is None
+        assert cr.ran is False
+        assert cr.source == "# test"
+        assert cr.path == "path"
+
+    def test_syntax_error_init(self):
+        cr = bahc.CodeRunner("This is a syntax error", "path", [])
+        assert cr.failed is True
+        assert "Invalid syntax in" in cr.error
+        assert cr.error_detail is not None
+
+    def test_new_module_success(self):
+        cr = bahc.CodeRunner("# test", "path", [])
+        m = cr.new_module()
+        assert isinstance(m, ModuleType)
+        assert m.__dict__['__name__'].startswith('bk_script_')
+        assert m.__dict__['__file__'] == abspath("path")
+
+    def test_new_module_resets_run_errors(self):
+        cr = bahc.CodeRunner("# test", "path", [])
+        cr._failed = True
+        m = cr.new_module()
+        assert isinstance(m, ModuleType)
+        assert m.__dict__['__name__'].startswith('bk_script_')
+        assert m.__dict__['__file__'] == abspath("path")
+
+    def test_new_module_returns_None_for_permanent_errors(self):
+        cr = bahc.CodeRunner("This is a syntax error", "path", [])
+        assert cr.failed is True
+        m = cr.new_module()
+        assert m is None
+
+    def test_reset_run_errors(self):
+        cr = bahc.CodeRunner("# test", "path", [])
+        cr._failed = True
+        cr._error = "error"
+        cr._error_detail = "detail"
+        cr.reset_run_errors()
+        assert cr.failed is False
+        assert cr.error is None
+        assert cr.error_detail is None
+
+    def test_reset_run_errors_leaves_permanent_errors(self):
+        cr = bahc.CodeRunner("This is a syntax error", "path", [])
+        cr._failed = True
+        cr.reset_run_errors()
+        assert cr.failed is True
+        assert cr.error is not None
+        assert cr.error_detail is not None
+
+    def test_run_sets_ran(self):
+        cr = bahc.CodeRunner("# test", "path", [])
+        m = cr.new_module()
+        assert not cr.ran
+        cr.run(m, lambda: None)
+        assert cr.ran
+
+    def test_run_runs_post_check(self):
+        cr = bahc.CodeRunner("# test", "path", [])
+        m = cr.new_module()
+        assert not cr.ran
+        result = {}
+        def post_check():
+            result['ran'] = True
+        cr.run(m, post_check)
+        assert cr.ran
+        assert result == dict(ran=True)
+
+    def test_run_fixups_argv(self):
+        cr = bahc.CodeRunner("import sys; argv = list(sys.argv)", "path", ["foo", 10])
+        assert not cr.ran
+        m = cr.new_module()
+        cr.run(m, lambda: None)
+        assert m.__dict__['argv'] == ["path", "foo", 10]
+
+    def test_run_fixups_path(self):
+        cr = bahc.CodeRunner("import sys; path = list(sys.path)", "/dir/to/path", ["foo", 10])
+        assert not cr.ran
+        m = cr.new_module()
+        cr.run(m, lambda: None)
+        assert m.__dict__['path'][0] == dirname("/dir/to/path")
+        assert m.__dict__['path'][1:] == sys.path
+
+    def test_run_restores_cwd(self):
+        old_cwd = os.getcwd()
+        cr = bahc.CodeRunner("import os; os.chdir('/')", "path", ["foo", 10])
+        assert not cr.ran
+        m = cr.new_module()
+        cr.run(m, lambda: None)
+        assert os.getcwd() == old_cwd
+
+    def test_run_restores_argv(self):
+        old_argv = list(sys.argv)
+        cr = bahc.CodeRunner("# test", "path", ["foo", 10])
+        assert not cr.ran
+        m = cr.new_module()
+        cr.run(m, lambda: None)
+        assert sys.argv == old_argv
+
+    def test_run_restores_path(self):
+        old_path = list(sys.path)
+        cr = bahc.CodeRunner("# test", "path", ["foo", 10])
+        assert not cr.ran
+        m = cr.new_module()
+        cr.run(m, lambda: None)
+        assert sys.path == old_path
+
 #-----------------------------------------------------------------------------
 # Dev API
 #-----------------------------------------------------------------------------