Repository: bokeh/bokeh

Issue:
Add support for globs to `bokeh serve`
I have been working on a way to make it easier to use `panel serve` to launch deployments at a proxied port on `jupyter notebook` calls, so that your deployment shows up at `localhost:8888/panel`. I've got this working for cases where you have a dir with one notebook in it called main.ipynb, but I'd like to have it look at all the notebooks and generate an index. This would be equivalent to calling `panel serve *.ipynb` but using a flag instead. The reason for the flag is that it isn't possible to pass the * in the [proxy code](https://github.com/pyviz/jupyter-panel-proxy/blob/b0d483d2774d159a065eae3432854d1196c8b5e0/panel_server/__init__.py#L3):

```
'command': ["panel", "serve", ".", "--allow-websocket-origin=*", "--port", "{port}", "--prefix", "{base_url}panel"],
```

For context this functionality would make it very easy to deploy apps on binder. The user would just need to include jupyter-panel-proxy in their env and stick `?urlpath=panel` at the end of the binder url.

These are the relevant PRs:
  - https://github.com/pyviz/jupyter-panel-proxy/pull/1
  - https://github.com/pyviz-demos/glaciers/pull/11

If we were able to add the `--all-notebooks` flag, then I'd be happy to write a jupyter-bokeh-proxy as well :)


Discussion:
I see that @bryevdv, you've been touching the serve code recently, so just wanted to check that this would be a reasonable thing to add from your perspective. 
@jsignell I am still missing some context. Why can't you call `bokeh serve` with all the individual notebooks on the command line on your end? (i.e. you figure out what all those need to be, and the pass them explicitly). Currently the notion of "directory style app" is very clear and delineated with no "except in this case"es or "almost always behaves this way"s. I'd really prefer to try and not muddy the waters. 

Ot alternatively if we must add a flag it should just be a simple flag that turns off directory mode altogether for everything (not one that turns off directory mode for some things but leaves it on for others). 
> The reason for the flag is that it isn't possible to pass the * in the proxy code:

I think that it's not the case that it isn't possible to pass `*`, but that there's a difference between:
```
bokeh serve *.ipynb
```
and
```
bokeh serve '*.ipynb'
```
The former one uses shell expansion, whereas the later pass on a verbatim glob. This gets worse when spawning a subprocess, e.g. with `Popen`, because then you would drop those quotes (assuming not using shell mode).

So, if I read this feature request correctly, all you actually need is support for globs in `bokeh serve`, and (I suppose) a `GlobHandler`, that would accompany `DirectoryHandler`, without altering its current semantics. If that's the case, I think it would be a nice addition.
Ok now I actually understand the context. I agree it it probably nicer to make 
 ```
 bokeh serve '*.ipynb'
```
work. I'm not sure about a new application handler, though. I think I would instead suggest confining changes to `bokeh.command`, i.e. detect `*` in the command line args as early as possibly and emulate shell expansion there, before passing the expanded list on to downstream `bokeh.sever`. That is, basically right in before this line:

https://github.com/bokeh/bokeh/blob/05be3eb0133bf5fac82484536c71500c30900dad/bokeh/command/subcommands/serve.py#L592
Thinking more, if we are going to do our own globbing internally, then *that* should probably be configurable, e.g. via `--no-glob` or similar. Otherwise we leave no avenue for users to run files that do actually have literal asterisks in them. But `--no-glob` (or whatever) is easy to explain and document and does not have interactions with anything else. 
I think that the default should be no glob (or no magic in general), so the command would be e.g. `bokeh serve --glob notebooks/**/*.ipynb`.
I'm fine with the suggestion to do our own globbing and agree that it is preferable to an `--all-notebooks` flag. I think having the ability to turn off the globbing also seems reasonable.

> the default should be no glob

@mattpap that changes the default behavior on a feature that we are using in long-lived projects. Is that really preferable?
> that changes the default behavior on a feature that we are using in long-lived projects. Is that really preferable?

I don't think it does? Currently we always get shell globbing for 
```
bokeh serve *.py
```
And we always will&mdash;we have no ability to affect or turn off shell globbing. My understanding of this issue is that it is desirable that (effectively) this *also* work:
```
bokeh serve '*.py'  
```
for cases when this might arise (e.g. calling a subprocess without `shell=True`) But that it does not currently work. So the proposal (as I understand it) it to allow 
```
bokeh serve --glob '*.py'  
```
which would do globbing manually when:

* the `--glob` flag is passed
* the arg that is passed has a literal `*` in it

This would not affect normal command line usage AFAICT, but would allow globs passed in subprocess calls to behave as expected. 
Ah sorry yes that is true. Ok I am good with the suggestion. 

Resolving patch:
diff --git a/bokeh/command/subcommands/serve.py b/bokeh/command/subcommands/serve.py
--- a/bokeh/command/subcommands/serve.py
+++ b/bokeh/command/subcommands/serve.py
@@ -83,6 +83,22 @@
 This index can be disabled with the ``--disable-index`` option, and the redirect
 behavior can be disabled with the ``--disable-index-redirect`` option.
 
+Another way to run multiple applications is using glob notation to indicate
+that all the files matching a particular pattern should be served.
+
+.. code-block:: sh
+
+    bokeh serve *.py
+
+Command line shells will normally expand the ``*.py`` automatically. However,
+if you are starting a Bokeh server programmatically, then filename arguments
+with globs may not be expanded by the shell. In situations like this, the
+``--glob`` flag may be used to make the Bokeh server explicitly perform globbing:
+
+.. code-block:: python
+
+    subprocess.call(["bokeh", "serve", "--glob", "*.py"])
+
 Application Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -327,6 +343,7 @@
 import argparse
 from fnmatch import fnmatch
 import os
+from glob import glob
 
 # External imports
 from tornado.autoreload import watch
@@ -564,6 +581,11 @@ class Serve(Subcommand):
             default=DEFAULT_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES,
             type=int,
         )),
+
+        ('--glob', dict(
+            action='store_true',
+            help='Process all filenames as globs',
+        )),
     )
 
     def invoke(self, args):
@@ -588,8 +610,15 @@ def invoke(self, args):
         # even if Tornado is not installed
         from bokeh.server.server import Server
 
-        argvs = { f : args.args for f in args.files}
-        applications = build_single_handler_applications(args.files, argvs)
+        files = []
+        for f in args.files:
+            if args.glob:
+                files.extend(glob(f))
+            else:
+                files.append(f)
+
+        argvs = { f : args.args for f in files}
+        applications = build_single_handler_applications(files, argvs)
 
         if len(applications) == 0:
             # create an empty application by default


Test patch:
diff --git a/bokeh/command/subcommands/tests/apps/apply_theme.py b/bokeh/command/subcommands/tests/apps/apply_theme.py
new file mode 100644
--- /dev/null
+++ b/bokeh/command/subcommands/tests/apps/apply_theme.py
@@ -0,0 +1,18 @@
+""" Example demonstrating how to apply a theme
+
+"""
+
+import numpy as np
+
+from bokeh.io import curdoc
+from bokeh.plotting import figure
+
+p = figure(width=800)
+props = dict(line_width=4, line_alpha=0.7)
+
+x = np.arange(-np.pi, np.pi, np.pi / 16)
+l0 = p.line(x, np.sin(x), color='yellow', legend='sin', **props)
+l2 = p.line(x, np.cos(x), color='red', legend='cos', **props)
+
+curdoc().add_root(p)
+curdoc().theme = 'dark_minimal'
diff --git a/bokeh/command/subcommands/tests/apps/line_on_off.py b/bokeh/command/subcommands/tests/apps/line_on_off.py
new file mode 100644
--- /dev/null
+++ b/bokeh/command/subcommands/tests/apps/line_on_off.py
@@ -0,0 +1,30 @@
+""" Example demonstrating turning lines on and off - with bokeh server
+
+"""
+
+import numpy as np
+
+from bokeh.io import curdoc
+from bokeh.layouts import row
+from bokeh.palettes import Viridis3
+from bokeh.plotting import figure
+from bokeh.models import CheckboxGroup
+
+p = figure()
+props = dict(line_width=4, line_alpha=0.7)
+x = np.linspace(0, 4 * np.pi, 100)
+l0 = p.line(x, np.sin(x), color=Viridis3[0], legend="Line 0", **props)
+l1 = p.line(x, 4 * np.cos(x), color=Viridis3[1], legend="Line 1", **props)
+l2 = p.line(x, np.tan(x), color=Viridis3[2], legend="Line 2", **props)
+
+checkbox = CheckboxGroup(labels=["Line 0", "Line 1", "Line 2"], active=[0, 1, 2], width=100)
+
+def update(attr, old, new):
+    l0.visible = 0 in checkbox.active
+    l1.visible = 1 in checkbox.active
+    l2.visible = 2 in checkbox.active
+
+checkbox.on_change('active', update)
+
+layout = row(checkbox, p)
+curdoc().add_root(layout)
diff --git a/bokeh/command/subcommands/tests/test_serve.py b/bokeh/command/subcommands/tests/test_serve.py
--- a/bokeh/command/subcommands/tests/test_serve.py
+++ b/bokeh/command/subcommands/tests/test_serve.py
@@ -258,6 +258,11 @@ def test_args():
             default=20*1024*1024,
             type=int,
         )),
+
+        ('--glob', dict(
+            action='store_true',
+            help='Process all filenames as globs',
+        )),
     )
 
 
@@ -310,6 +315,40 @@ def test_port_not_available():
     finally:
         sock.close()
 
+def test_no_glob_by_default_on_filename_if_wildcard_in_quotes():
+    import os.path
+
+    here = os.path.abspath(os.path.dirname(__file__))
+    path = os.path.join(here, 'apps', '*.py')
+    out = check_error([path])
+    expected = "ERROR: Path for Bokeh server application does not exist:"
+    assert expected in out
+    assert '*' in out
+
+@pytest.mark.skipif(six.PY2, reason="Travis bug causes bad file descriptor")
+def test_glob_flag_on_filename_if_wildcard_in_quotes():
+    from fcntl import fcntl, F_GETFL, F_SETFL
+    from os import O_NONBLOCK, read
+    import os.path
+
+    pat = re.compile(r'Bokeh app running at: http://localhost:(\d+)/line_on_off')
+    m = None
+    here = os.path.abspath(os.path.dirname(__file__))
+    path = os.path.join(here, 'apps', '*.py')
+
+    with run_bokeh_serve(["--port", "0", "--glob", path]) as p:
+        flags = fcntl(p.stdout, F_GETFL)
+        fcntl(p.stdout, F_SETFL, flags | O_NONBLOCK)
+        sleep(2)
+        o = read(p.stdout.fileno(), 100*1024)
+        m = pat.search(o.decode())
+        if m is None:
+            pytest.fail("no matching log line in process output")
+        port = int(m.group(1))
+        assert port > 0
+        r = requests.get("http://localhost:%d/apply_theme" % (port,))
+        assert r.status_code == 200
+
 @pytest.mark.skipif(six.PY2, reason="Travis bug causes bad file descriptor")
 def test_actual_port_printed_out():
     from fcntl import fcntl, F_GETFL, F_SETFL