Repository: dask/distributed

Issue:
No such file or directory "bokeh" - in release 1.9.2 (conda install)
Get this error in my logs:

```
scheduler_1 | distributed.scheduler - INFO -            http at:           172.17.0.4:9786
scheduler_1 | distributed.scheduler - WARNING - Could not start Bokeh web UI
scheduler_1 | Traceback (most recent call last):
scheduler_1 |   File "/opt/conda/envs/fire/lib/python3.5/site-packages/distributed/cli/dscheduler.py", line scheduler_1 |     bokeh_proc[0] = subprocess.Popen(args)
scheduler_1 |   File "/opt/conda/envs/fire/lib/python3.5/subprocess.py", line 950, in __init__
scheduler_1 |     restore_signals, start_new_session)
scheduler_1 |   File "/opt/conda/envs/fire/lib/python3.5/subprocess.py", line 1544, in _execute_child
scheduler_1 |     raise child_exception_type(errno_num, err_msg)
scheduler_1 | FileNotFoundError: [Errno 2] No such file or directory: 'bokeh'
scheduler_1 | distributed.core - INFO - Connection from 172.17.0.6:60119 to Scheduler
scheduler_1 | distributed.scheduler - INFO - Register 172.17.0.6:5700
scheduler_1 | distributed.scheduler - INFO - Starting worker compute stream, 172.17.0.6:5700
```



Discussion:
Hrm, that's odd.  Everything should still run fine, you're just not getting access to the web UI.  Will take a look, perhaps something isn't being packaged up correctly.

This was my experience from a clean conda install on linux-64.  Can you try this to see if maybe it's an environmental issue?  Also, what is the result of `import distributed.bokeh`

```
mrocklin@notebook:~/workspace/distributed$ conda create -n test python=3.5 distributed bokeh ipython 
Fetching package metadata: ......
Solving package specifications: .........

Package plan for installation in environment /home/mrocklin/Software/anaconda/envs/test:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    click-6.6                  |           py35_0         100 KB
    docutils-0.12              |           py35_0         665 KB
    jmespath-0.9.0             |           py35_0          28 KB
    msgpack-python-0.4.6       |           py35_0         207 KB
    numpy-1.11.0               |           py35_0         6.1 MB
    path.py-8.2                |           py35_0          47 KB
    psutil-4.1.0               |           py35_0         218 KB
    pytz-2016.3                |           py35_0         176 KB
    setuptools-20.7.0          |           py35_0         457 KB
    tblib-1.3.0                |           py35_0          12 KB
    pexpect-4.0.1              |           py35_0          66 KB
    pip-8.1.1                  |           py35_1         1.6 MB
    python-dateutil-2.5.2      |           py35_0         237 KB
    traitlets-4.2.1            |           py35_0         112 KB
    botocore-1.4.10            |           py35_0         1.7 MB
    ipython-4.1.2              |           py35_1         988 KB
    pandas-0.18.0              |      np111py35_0        12.9 MB
    boto3-1.3.0                |           py35_0          96 KB
    dask-0.8.2                 |           py35_0         303 KB
    s3fs-0.0.2                 |           py35_0          17 KB
    distributed-1.9.2          |           py35_0         151 KB
    ------------------------------------------------------------
                                           Total:        26.1 MB

The following NEW packages will be INSTALLED:

    bokeh:            0.11.1-py35_0     
    boto3:            1.3.0-py35_0      
    botocore:         1.4.10-py35_0     
    chest:            0.2.3-py35_0      
    click:            6.6-py35_0        
    cloudpickle:      0.1.1-py35_0      
    dask:             0.8.2-py35_0      
    decorator:        4.0.9-py35_0      
    distributed:      1.9.2-py35_0      
    docutils:         0.12-py35_0       
    heapdict:         1.0.0-py35_0      
    ipython:          4.1.2-py35_1      
    ipython_genutils: 0.1.0-py35_0      
    jinja2:           2.8-py35_0        
    jmespath:         0.9.0-py35_0      
    locket:           0.2.0-py35_0      
    markupsafe:       0.23-py35_0       
    mkl:              11.3.1-0          
    msgpack-python:   0.4.6-py35_0      
    numpy:            1.11.0-py35_0     
    openssl:          1.0.2g-0          
    pandas:           0.18.0-np111py35_0
    partd:            0.3.2-py35_1      
    path.py:          8.2-py35_0        
    pexpect:          4.0.1-py35_0      
    pickleshare:      0.5-py35_0        
    pip:              8.1.1-py35_1      
    psutil:           4.1.0-py35_0      
    ptyprocess:       0.5-py35_0        
    python:           3.5.1-0           
    python-dateutil:  2.5.2-py35_0      
    pytz:             2016.3-py35_0     
    pyyaml:           3.11-py35_1       
    readline:         6.2-2             
    requests:         2.9.1-py35_0      
    s3fs:             0.0.2-py35_0      
    setuptools:       20.7.0-py35_0     
    simplegeneric:    0.8.1-py35_0      
    six:              1.10.0-py35_0     
    sqlite:           3.9.2-0           
    tblib:            1.3.0-py35_0      
    tk:               8.5.18-0          
    toolz:            0.7.4-py35_0      
    tornado:          4.3-py35_0        
    traitlets:        4.2.1-py35_0      
    wheel:            0.29.0-py35_0     
    xz:               5.0.5-1           
    yaml:             0.1.6-0           
    zlib:             1.2.8-0           

Fetching packages ...
click-6.6-py35 100% |################################| Time: 0:00:00 280.94 kB/s
docutils-0.12- 100% |################################| Time: 0:00:01 345.37 kB/s
jmespath-0.9.0 100% |################################| Time: 0:00:00 179.97 kB/s
msgpack-python 100% |################################| Time: 0:00:01 175.07 kB/s
numpy-1.11.0-p 100% |################################| Time: 0:00:12 497.68 kB/s
path.py-8.2-py 100% |################################| Time: 0:00:00 258.32 kB/s
psutil-4.1.0-p 100% |################################| Time: 0:00:00 315.70 kB/s
pytz-2016.3-py 100% |################################| Time: 0:00:00 197.81 kB/s
setuptools-20. 100% |################################| Time: 0:00:02 201.90 kB/s
tblib-1.3.0-py 100% |################################| Time: 0:00:00   4.68 MB/s
pexpect-4.0.1- 100% |################################| Time: 0:00:00 223.46 kB/s
pip-8.1.1-py35 100% |################################| Time: 0:00:01 920.17 kB/s
python-dateuti 100% |################################| Time: 0:00:00 364.72 kB/s
traitlets-4.2. 100% |################################| Time: 0:00:00 293.40 kB/s
botocore-1.4.1 100% |################################| Time: 0:00:07 257.29 kB/s
ipython-4.1.2- 100% |################################| Time: 0:00:02 481.29 kB/s
pandas-0.18.0- 100% |################################| Time: 0:00:14 939.39 kB/s
boto3-1.3.0-py 100% |################################| Time: 0:00:00 344.08 kB/s
dask-0.8.2-py3 100% |################################| Time: 0:00:00 374.19 kB/s
s3fs-0.0.2-py3 100% |################################| Time: 0:00:00  63.09 kB/s
distributed-1. 100% |################################| Time: 0:00:00 337.18 kB/s
Extracting packages ...
[      COMPLETE      ]|###########################################################| 100%
Linking packages ...
[      COMPLETE      ]|###########################################################| 100%
#
# To activate this environment, use:
# $ source activate test
#
# To deactivate this environment, use:
# $ source deactivate
#
mrocklin@notebook:~/workspace/distributed$ source activate test
discarding /home/mrocklin/Software/anaconda/bin from PATH
prepending /home/mrocklin/Software/anaconda/envs/test/bin to PATH
(test)mrocklin@notebook:~/workspace/distributed$ dscheduler 
distributed.scheduler - INFO - Start Scheduler at:        192.168.1.107:8786
distributed.scheduler - INFO -            http at:        192.168.1.107:9786
distributed.scheduler - INFO -  Start Bokeh UI at:        http://192.168.1.107:8787/status/
bokeh.command.subcommands.serve - INFO - Check for unused sessions every 50 milliseconds
bokeh.command.subcommands.serve - INFO - Unused sessions last for 1 milliseconds
bokeh.command.subcommands.serve - INFO - Starting Bokeh server on port 8787 with applications at paths ['/status', '/tasks']
distributed.core - INFO - Connection from 127.0.0.1:45967 to Scheduler
distributed.core - INFO - Connection from 127.0.0.1:45968 to Scheduler
```

Roughly the same result with python 2.7

Hi - just testing now:

```
import distributed.bokeh 
```

Worked fine... I am using this environment.yaml on a miniconda install on centos:

```
name: fire
dependencies:
  - python=3.5
  - matplotlib
  - ipywidgets
  - dask
  - distributed
  - bokeh
  - pandas
  - pyproj
  - shapely
  - psycopg2
  - fiona
  - seaborn
  - notebook
  - numpy
  - scipy
  - pip
```

OK -- still get the error, here is a dockerfile I am using:

```
FROM frolvlad/alpine-glibc

RUN apk --update add \
    bash \
    git \
    curl \
    ca-certificates \
    bzip2 \
    unzip \
    sudo \
    libstdc++ \
    glib \
    libxext \
    libxrender \
    && rm -rf glibc*apk /var/cache/apk/*

# Configure environment
ENV CONDA_DIR=/opt/conda \
  PATH=$CONDA_DIR/bin:$PATH \
  SHELL=/bin/bash \
  LANG=C.UTF-8

# Install conda
RUN mkdir -p $CONDA_DIR && \
    echo export PATH=$CONDA_DIR/bin:'$PATH' > /etc/profile.d/conda.sh && \
    curl https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  -o mconda.sh && \
    /bin/bash mconda.sh -f -b -p $CONDA_DIR && \
    rm mconda.sh && \
    $CONDA_DIR/bin/conda install --yes conda

RUN /opt/conda/bin/conda install --yes distributed bokeh ipython
```

Then I follow up with the command:

```
[nfaggian@firefly scheduler]$ docker build -t alpine . 
[nfaggian@firefly scheduler]$ docker run -t -i alpine /opt/conda/bin/dscheduler
distributed.scheduler - INFO - Start Scheduler at:           172.17.0.2:8786
distributed.scheduler - INFO -            http at:           172.17.0.2:9786
distributed.scheduler - WARNING - Could not start Bokeh web UI
Traceback (most recent call last):
  File "/opt/conda/lib/python3.5/site-packages/distributed/cli/dscheduler.py", line 73, in main
    bokeh_proc[0] = subprocess.Popen(args)
  File "/opt/conda/lib/python3.5/subprocess.py", line 950, in __init__
    restore_signals, start_new_session)
  File "/opt/conda/lib/python3.5/subprocess.py", line 1544, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'bokeh'

```

Do you see the same output? am I doing something wrong -- I know Alpine linux isn't Centos but it is a smaller distro to test on (assuming the glibc stuff isn't an issue).

Nath

It looks like setting `$PATH` at the same time as `$CONDA_DIR` in your Dockerfile is not working and thus your `PATH` doesn't include the conda bin directory, which is why the `bokeh` command is failing. If I use the following in your Dockerfile it seems to work:

```
# Configure environment
ENV CONDA_DIR=/opt/conda \
  SHELL=/bin/bash \
  LANG=C.UTF-8
ENV PATH=$CONDA_DIR/bin:$PATH
```

OK - **sorry** it was an environment variable issue! I just discovered this now as @koverholt posted!

Thanks for the follow up.

Thanks for posting your Dockerfile, this is a neat setup.

Your conda environment name `fire` caught my eye, and the work in your GH repos reminded me of similar work that I lightly participated in last year: https://github.com/sallamander/forest-fires.

Wow excellent, I will have a read! I am looking at fire simulators in the Australian region, Matthews work is helping me to form a distributed simulator execution environment...

It seems like we shouldn't trust that `bokeh` is on the path.  I'll see if I can start up the server just from Python.


Resolving patch:
diff --git a/distributed/cli/dscheduler.py b/distributed/cli/dscheduler.py
--- a/distributed/cli/dscheduler.py
+++ b/distributed/cli/dscheduler.py
@@ -1,9 +1,9 @@
 from __future__ import print_function, division, absolute_import
 
 import logging
+import multiprocessing
 import os
 import socket
-import subprocess
 from sys import argv, exit
 from time import sleep
 
@@ -22,12 +22,8 @@
 
 import signal
 
-bokeh_proc = [False]
-
 
 def handle_signal(sig, frame):
-    if bokeh_proc[0]:
-        bokeh_proc[0].terminate()
     IOLoop.instance().add_callback(IOLoop.instance().stop)
 
 signal.signal(signal.SIGINT, handle_signal)
@@ -70,7 +66,10 @@ def main(center, host, port, http_port, bokeh_port, show, _bokeh):
                      sum([['--host', host] for host in hosts], []))
             if show:
                 args.append('--show')
-            bokeh_proc[0] = subprocess.Popen(args)
+            from bokeh.command.bootstrap import main
+            proc = multiprocessing.Process(target=main, args=(args,))
+            proc.daemon = True
+            proc.start()
 
             logger.info(" Start Bokeh UI at:        http://%s:%d/status/"
                         % (ip, bokeh_port))
@@ -82,7 +81,7 @@ def main(center, host, port, http_port, bokeh_port, show, _bokeh):
     loop.start()
     loop.close()
     scheduler.stop()
-    bokeh_proc[0].terminate()
+    proc.terminate()
 
     logger.info("End scheduler at %s:%d", ip, port)
 


Test patch: