Repository: jupyterhub/jupyterhub

Issue:
JupyterHub restarts proxy during cleanup if SIGTERM received
Originally encountered this issue when running jupyterhub as a supervisor program and stopping the program via ``supervisorctl stop jupyterhub``.  Configuration file:
```
[program:jupyterhub]
command = /usr/local/bin/jupyterhub_start
user = root
stdout_logfile = /var/log/jupyterhub.log
redirect_stderr = true
startretries=1
exitcodes=0,2
stopsignal=TERM
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
```
Was able to reproduce it running jupyterhub from shell and using kill on the jupyterhub process from another shell after looking up the jupyterhub process id using ``ps aux | grep jupyterhub``.  Sending SIGINT or using Ctl+C caused JupyterHub to shut down the proxy normally.

Using Python 3.5, Configurable-HTTP-Proxy 3.1.0, Jupyterhub 0.8.1, nodejs 8.9.1  

**How to reproduce the issue**

Send a termination signal to the jupyterhub process. 

**What you expected to happen**

Jupyterhub to stop the configurable-http-proxy

**What actually happens**

Jupyterhub tries to clean up the proxy, starts the proxy again, and stops.

```
[I 2017-12-01 22:27:02.115 JupyterHub proxy:458] Starting proxy @ http://*:8000/
22:27:02.217 - info: [ConfigProxy] Proxying http://*:8000 to (no default)
22:27:02.219 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[W 2017-12-01 22:27:02.319 JupyterHub proxy:304] Adding missing default route
[I 2017-12-01 22:27:02.320 JupyterHub proxy:370] Adding default route for Hub: / => http://127.0.0.1:8081
22:27:02.325 - info: [ConfigProxy] 200 GET /api/routes 
22:27:02.328 - info: [ConfigProxy] Adding route / -> http://127.0.0.1:8081
22:27:02.329 - info: [ConfigProxy] 201 POST /api/routes/ 
[I 2017-12-01 22:27:02.330 JupyterHub app:1581] JupyterHub is now running at http://:8000/
[C 2017-12-01 22:27:30.316 JupyterHub app:1590] Received SIGTERM, shutting down
[I 2017-12-01 22:27:30.320 JupyterHub app:1399] Cleaning up single-user servers...
[I 2017-12-01 22:27:30.323 JupyterHub proxy:496] Cleaning up proxy[4204]...
[I 2017-12-01 22:27:30.339 JupyterHub app:1431] ...done
[E 2017-12-01 22:27:32.289 JupyterHub proxy:509] Proxy stopped with exit code -15
[W 2017-12-01 22:27:32.303 JupyterHub proxy:456] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2017-12-01 22:27:32.305 JupyterHub proxy:458] Starting proxy @ http://*:8000/
```


Discussion:
Thanks for the report @rwedge.

This seems like a bug and/or timing issue to me. @minrk this looks like the [`check_running`](https://github.com/jupyterhub/jupyterhub/blame/master/jupyterhub/proxy.py#L511) is restarting the proxy. This [line](https://github.com/jupyterhub/jupyterhub/blame/master/jupyterhub/proxy.py#L497) in the proxy `stop` looks a bit off with the `poll()`.

Resolving patch:
diff --git a/jupyterhub/proxy.py b/jupyterhub/proxy.py
--- a/jupyterhub/proxy.py
+++ b/jupyterhub/proxy.py
@@ -424,6 +424,8 @@ def _auth_token_default(self):
                       help="""The command to start the proxy"""
                       )
 
+    _check_running_callback = Any(help="PeriodicCallback to check if the proxy is running")
+
     @gen.coroutine
     def start(self):
         public_server = Server.from_url(self.public_url)
@@ -490,10 +492,13 @@ def _check_process():
         _check_process()
         self.log.debug("Proxy started and appears to be up")
         pc = PeriodicCallback(self.check_running, 1e3 * self.check_running_interval)
+        self._check_running_callback = pc
         pc.start()
 
     def stop(self):
         self.log.info("Cleaning up proxy[%i]...", self.proxy_process.pid)
+        if self._check_running_callback is not None:
+            self._check_running_callback.stop()
         if self.proxy_process.poll() is None:
             try:
                 self.proxy_process.terminate()


Test patch: