Repository: jupyterhub/jupyterhub

Issue:
calling a java app via subprocess and then jupyterhub hangs
I have these scripts 

script.py:

``` python
import os.path,subprocess
from subprocess import STDOUT,PIPE

def execute_imagej():
    cmd = ["../apps/Fiji.app/ImageJ-linux64","--headless","./testfijis.py"]
    proc = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    stdout,stderr = proc.communicate()
    print ('This was "' + stdout + '"')

execute_imagej()
```

testfijis.py:

``` python
print(1)
```

when i run from the command line :

python script.py
or
ipython scriipt.py

it works fine, i got a reply.

when i run from jupyter notebook the script.py in a cell, it does not work when the jupyter server is instantiated via jupyterhub. Otherwise, it works fine.

It freezes and after a while, jupyterhub emits the following error message :
22:01:17.414 - error: [ConfigProxy] Proxy error:  code=ECONNRESET

Some ideas ?



Discussion:
The ECONNRESET suggests that starting this subprocess of a subprocess is somehow bringing down the notebook **server**, which is quite strange. How are you spawning the single-user servers?

When you run the notebook outside JupyterHub and it works, is it using the same machine, user, and Python install/env?

**machine**
- jupyterhub and jupyter are launched from the same server, a virtualbox one

**user**
- jupyterhub is launched with this cmd (vagrant user with root permissions)
  - sudo jupyterhub
- jupyter is launced with this cmd (vagrant user)
  - jupyter notebook --ip 0.0.0.0

**python environment**
- jupyterhub is triggered with python 3.4.3. Corresponding running process :
  - /usr/bin/python3 /usr/local/bin/jupyterhub
- jupyter standalone is triggered with python 2.7.6 through a virtual environment. Corresponding running process :
  - /home/vagrant/.virtualenvs/venvpy2/bin/python /home/vagrant/.virtualenvs/venvpy2/bin/jupyter-notebook --ip 0.0.0.0
- jupyterhub-singleuser is triggered with python 3.4.3. Corresponding running process : 
  - /usr/bin/python3 /usr/local/bin/jupyterhub-singleuser --user=vagrant --port=35448 --cookie-name=jupyter-hub-token-vagrant --base-url=/user/vagrant --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost

**notebooks / kernels**
- via jupyterhub, the following ipython kernel process is triggered :
  /home/vagrant/.virtualenvs/venvpy2/bin/python -m IPython.kernel -f /home/vagrant/.local/share/jupyter/runtime/kernel-59864d70-4013-47b2-9847-f73d5209eff6.json

with the corresponding kernel:
{
  "stdin_port": 54611, 
  "ip": "127.0.0.1", 
  "control_port": 54599, 
  "hb_port": 60651, 
  "signature_scheme": "hmac-sha256", 
  "key": "153ef843-518f-46c5-9338-a6b9c8c34c1a", 
  "shell_port": 47891, 
  "transport": "tcp", 
  "iopub_port": 47071
}
- via jupyter, the following ipython kernel process is triggered :
  /home/vagrant/.virtualenvs/venvpy2/bin/python -m IPython.kernel -f /run/user/1000/jupyter/kernel-3c15d751-c126-4958-b92d-d56356fe6d3d.json

with the corresponding kernel:
{
  "stdin_port": 40861, 
  "ip": "127.0.0.1", 
  "control_port": 46217, 
  "hb_port": 56458, 
  "signature_scheme": "hmac-sha256", 
  "key": "d0ea6a94-d0b3-4b48-a375-b279b50b7794", 
  "shell_port": 58589, 
  "transport": "tcp", 
  "iopub_port": 33713
}

let me know if you need something else

Can you do `pip list` (or `conda list`) in each env? Can you run the standalone notebook server with the same Python as the one used in the JupyterHub case to test?

```
/usr/bin/python3 -m notebook [args]
```

Just have done the test before reading your reply :)
when jupyter standalone run with python 3, i got the same problem, the subprocess of the subprocess freezes... so the problem does not look to be on jupyterhub, but rather on jupyter when running with python3  

here are the "pip list" 

**pip list**
**root environment**
apt-xapian-index (0.45)
argparse (1.2.1)
backports.ssl-match-hostname (3.4.0.2)
certifi (2015.9.6.2)
chardet (2.0.1)
Cheetah (2.4.4)
cloud-init (0.7.5)
colorama (0.2.5)
configobj (4.7.2)
decorator (4.0.4)
functools32 (3.2.3-2)
GDAL (1.11.2)
gyp (0.1)
html5lib (0.999)
httplib2 (0.8)
ipykernel (4.1.1)
ipython (4.0.0)
ipython-genutils (0.1.0)
Jinja2 (2.8)
jsonpatch (1.3)
jsonpointer (1.0)
jsonschema (2.5.1)
jupyter-client (4.1.1)
jupyter-core (4.0.6)
jupyterdrive (1.1.0, /home/vagrant/jupyter-drive)
keyring (3.5)
Landscape-Client (14.12)
launchpadlib (1.10.2)
lazr.restfulclient (0.13.3)
lazr.uri (1.0.3)
MarkupSafe (0.23)
mistune (0.7.1)
nbconvert (4.1.0)
nbformat (4.0.1)
notebook (4.1.0.dev)
numpy (1.8.2)
oauth (1.0.1)
PAM (0.4.2)
path.py (8.1.2)
pbr (1.8.0)
pexpect (4.0.1)
pickleshare (0.5)
pip (1.5.4)
prettytable (0.7.2)
ptyprocess (0.5)
pycrypto (2.6.1)
pycups (1.9.66)
pycurl (7.19.3)
Pygments (2.0.2)
pygobject (3.12.0)
pyOpenSSL (0.13)
pyserial (2.6)
pysmbc (1.0.14.1)
python-apt (0.9.3.5ubuntu1)
python-debian (0.1.21-nmu2ubuntu2)
pyxdg (0.25)
PyYAML (3.10)
pyzmq (14.7.0)
requests (2.2.1)
SecretStorage (2.0.0)
setuptools (3.3)
simplegeneric (0.8.1)
simplejson (3.3.1)
six (1.9.0)
ssh-import-id (3.21)
stevedore (1.8.0)
system-service (0.1.6)
tornado (4.2.1)
traitlets (4.0.0)
Twisted-Core (13.2.0)
Twisted-Names (13.2.0)
Twisted-Web (13.2.0)
urllib3 (1.7.1)
virtualenv (13.1.2)
virtualenv-clone (0.2.6)
virtualenvwrapper (4.7.1)
wadllib (1.3.2)
wheel (0.24.0)
wsgiref (0.1.2)
zope.interface (4.0.5)

**venvpy2**
affine (1.2.0)
backports.ssl-match-hostname (3.4.0.2)
beautifulsoup4 (4.4.1)
blaze (0.9.0)
blinker (1.4)
build (1.0.2)
certifi (2015.9.6.2)
click (5.1)
click-plugins (1.0.2)
cligj (0.3.0)
cloudpickle (0.1.1)
configparser (3.3.0.post2)
dask (0.7.6)
datashape (0.5.0)
decorator (4.0.4)
descartes (1.0.1)
docopt (0.6.2)
docutils (0.12)
doit (0.29.0)
enum34 (1.0.4)
et-xmlfile (1.0.1)
Fiona (1.6.2)
folium (0.1.6)
funcsigs (0.4)
functools32 (3.2.3.post2)
GDAL (1.11.2)
gdbn (0.1)
geographiclib (1.44)
geopy (1.11.0)
ghp-import (0.4.1)
gnumpy (0.2)
gplearn (0.1.0)
graphviz (0.4.7)
husl (4.0.3)
ipdb (0.8.1)
ipykernel (4.0.3)
ipython (4.0.0)
ipython-genutils (0.1.0)
ipywidgets (4.0.3)
javabridge (1.0.11)
jdcal (1.2)
Jinja2 (2.8)
joblib (0.8.4)
jsonschema (2.5.1)
jupyter (1.0.0)
jupyter-client (4.0.0)
jupyter-console (4.0.2)
jupyter-core (4.0.6)
Lasagne (0.2.dev1)
Logbook (0.11.2)
lxml (3.4.4)
Mako (1.0.2)
Markdown (2.6.2)
MarkupSafe (0.23)
matplotlib (1.4.3)
mistune (0.7.1)
mock (1.3.0)
multipledispatch (0.4.8)
natsort (4.0.3)
nbconvert (4.0.0)
nbformat (4.0.0)
neo4jrestclient (2.1.0)
networkx (1.10)
Nikola (7.7.2)
nolearn (0.6a0.dev0)
nose (1.3.7)
notebook (4.0.5)
numexpr (2.4.6)
numpy (1.10.4)
odo (0.4.0)
openpyxl (2.3.2)
Optunity (1.1.1)
Orange (2.7.8)
pandas (0.17.1)
path.py (8.1.1)
patsy (0.4.0)
pbr (1.8.0)
pexpect (3.3)
pickleshare (0.5)
Pillow (3.0.0)
pip (7.1.2)
psutil (3.3.0)
ptyprocess (0.5)
pygdal (1.11.2.1)
Pygments (2.0.2)
pyinotify (0.9.6)
pymongo (3.0.3)
pyparsing (2.0.3)
pyproj (1.9.4)
PyRSS2Gen (1.1)
python-dateutil (2.4.2)
pytz (2015.7)
pyzmq (14.7.0)
qtconsole (4.0.1)
rasterio (0.26.0)
requests (2.8.1)
rpy2 (2.7.0)
scikit-learn (0.15.2)
scipy (0.16.0)
seaborn (0.6.0)
setuptools (18.2)
Shapely (1.5.12)
simplegeneric (0.8.1)
singledispatch (3.4.0.3)
six (1.10.0)
sklearn (0.0)
snuggs (1.3.1)
SQLAlchemy (1.0.11)
statsmodels (0.6.1)
tables (3.2.2)
tabulate (0.7.5)
terminado (0.5)
Theano (0.7.0)
toolz (0.7.4)
tornado (4.2.1)
tqdm (3.4.0)
traitlets (4.0.0)
twitter (1.17.1)
Unidecode (0.4.18)
webassets (0.11.1)
wheel (0.24.0)
xgboost (0.4)
xlrd (0.9.4)
Yapsy (1.11.223)
yolk3k (0.8.7)
zmq (0.0.0)

**venvpy3**
backports-abc (0.4)
beautifulsoup4 (4.4.1)
decorator (4.0.4)
hide-code (0.1.4)
ipykernel (4.2.1)
ipython (4.0.1)
ipython-genutils (0.1.0)
ipywidgets (4.1.1)
Jinja2 (2.8)
jsonschema (2.5.1)
jupyter (1.0.0)
jupyter-client (4.1.1)
jupyter-console (4.0.3)
jupyter-core (4.0.6)
MarkupSafe (0.23)
mistune (0.7.1)
nbconvert (4.1.0)
nbformat (4.0.1)
notebook (4.0.6)
path.py (8.1.2)
pexpect (4.0.1)
pickleshare (0.5)
pip (7.1.2)
ptyprocess (0.5)
Pygments (2.0.2)
pyzmq (15.1.0)
qtconsole (4.1.1)
requests (2.9.1)
setuptools (18.2)
simplegeneric (0.8.1)
terminado (0.5)
tornado (4.3)
traitlets (4.0.0)
wheel (0.24.0)

If you do

```
pip install --upgrade jupyter-client
```

in pyvenv2, do you start to see the issue in Python2, as well?

good point !

Now it is not working neither in python 2 :(

In that case, It's the issue we are working on at jupyter/jupyter_client#109

If you pin jupyter_client to less than 4.1 while we work on a fix, you should be okay while we work on a fix

```
pip install 'jupyter-client<4.1'
```

Hi,

I installed "jupyter-client" back to version 4.0.0 on the three environment (not virtual, virtual venvpy2, virtual venvpy3).

Now I can use indeed jupyter standalone, either with python 2 or python 3 kernel. 

However, through jupyterhub, it keeps freezing ... and still emitting the same error message

Should I re-open the issue ? 

That would be really strange. Are you sure here are no versions of jupyter_client 4.1 installed anywhere on your system that might be in use?


Resolving patch:
diff --git a/jupyterhub/spawner.py b/jupyterhub/spawner.py
--- a/jupyterhub/spawner.py
+++ b/jupyterhub/spawner.py
@@ -318,9 +318,6 @@ def set_user_setuid(username):
     gids = [ g.gr_gid for g in grp.getgrall() if username in g.gr_mem ]
     
     def preexec():
-        # don't forward signals
-        os.setpgrp()
-        
         # set the user and group
         os.setgid(gid)
         try:
@@ -405,6 +402,7 @@ def start(self):
         self.log.info("Spawning %s", ' '.join(pipes.quote(s) for s in cmd))
         self.proc = Popen(cmd, env=env,
             preexec_fn=self.make_preexec_fn(self.user.name),
+            start_new_session=True, # don't forward signals
         )
         self.pid = self.proc.pid
     


Test patch: