Repository: jupyterhub/jupyterhub

Issue:
HTTP 403: Forbidden error when starting jupyterhub
Hi, 
I have been trying to use jupyterhub for a while now but I am struggling with a 403 error which I cannot sort out.

I am using jupyterhub version 0.8.0.b2 on fedora 25. I have installed all dependencies as described in the the installation instructions and configured jupyterhub to run on port 443. Apart from the IP of my server and the user admin name, all other options are set to the default ones in the jupyterhub_config.py file. 

I can connect to the server through ssh and there is a webpage working at port 80. I have set selinux as permissive and all relevant ports should be properly set up in the firewall. 

Now, whenever I run jupyterhub I get the following messages:
```
[root@psat jupyterhub]# jupyterhub -f jupyterhub_config.py --debug
[D 2017-08-29 17:27:42.508 JupyterHub application:555] Looking for jupyterhub_config in /etc/jupyterhub
[D 2017-08-29 17:27:42.509 JupyterHub application:577] Loaded config file: /etc/jupyterhub/jupyterhub_config.py
[I 2017-08-29 17:27:42.521 JupyterHub app:843] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2017-08-29 17:27:42.521 JupyterHub app:894] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2017-08-29 17:27:42.542 JupyterHub orm:503] database schema version found: 3ec6993fe20c
[D 2017-08-29 17:27:42.600 JupyterHub app:1216] Loading state for fmilano from db
[D 2017-08-29 17:27:42.605 JupyterHub app:1244] Loaded users: 
     fmilano admin
[I 2017-08-29 17:27:42.611 JupyterHub app:1531] Hub API listening on http://127.0.0.1:8081/hub/
[I 2017-08-29 17:27:42.612 JupyterHub proxy:457] Starting proxy @ https://137.43.92.162:443/
[D 2017-08-29 17:27:42.612 JupyterHub proxy:458] Proxy cmd: ['configurable-http-proxy', '--ip', '137.43.92.162', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error', '--ssl-key', '/etc/letsencrypt/live/psat.ucd.ie/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/psat.ucd.ie/fullchain.pem']
17:27:42.891 - info: [ConfigProxy] Proxying https://137.43.92.162:443 to (no default)
17:27:42.894 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[D 2017-08-29 17:27:43.001 JupyterHub proxy:490] Proxy started and appears to be up
[D 2017-08-29 17:27:43.003 JupyterHub proxy:551] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
17:27:43.041 - warn: [ConfigProxy] 403 GET /api/routes 
[E 2017-08-29 17:27:43.040 JupyterHub app:1626]
    Traceback (most recent call last):
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1624, in launch_instance_async
        yield self.start()
      File "/home/fmilano/jupyterhub/jupyterhub/app.py", line 1572, in start
        yield self.proxy.check_routes(self.users, self._service_map)
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 293, in check_routes
        routes = yield self.get_all_routes()
      File "/home/fmilano/jupyterhub/jupyterhub/proxy.py", line 588, in get_all_routes
        resp = yield self.api_request('', client=client)
    tornado.httpclient.HTTPError: HTTP 403: Forbidden
```
I have been looking for a solution for a week now (I have tried different ports, different configuration of apache, ssh, and the firewall), but I am always getting the same message and the jupyterhub exits as indicated above.

Any hint?

Thanks in advance for your help!



Discussion:
Hi @fmilano1975, There is now a 0.8.0b3 release. Would you mind giving that version a try? Thanks.

cc/ @minrk Do you see anything with the 443 port usage or permissions/SSL cert that would prevent connection? Or possibly an apache setting?
This probably means that there is a `configurable-http-proxy` instance still running with an older API token. Try shutting down the hub and check if there are any:

    ps aux | grep configurable-http-proxy

If there are, shut them down and try launching jupyterhub again.
It would be nice if the service was doing that automatically on shutdown.
Is there a permanent solution for this? Our Jupyterhub restarts every night and fails to start again.
Hi,
Since I run Jupyterhub through systemd, I have a 

    ExecStartPre=/var/local/anaconda3/jupyter/home/kill-configurable-http-proxy.sh

that will kill any remaining process:

```
#!/bin/bash
ps aux | grep "/var/local/anaconda3/bin/configurable-http-proxy" |grep -v grep|awk '{print "kill -9 " $2}'|sh
```

It is just a dirty fix , but it works for me.

I ran up against a stale `configurable-http-proxy`  when running tests yesterday. @minrk, is there a straightforward way to do the cleanup automatically or should I document the solution you mention above to check for stale processes?
If jupyterhub wrote a pid file for the proxy, it could find the previous process and clean it up on startup.

@Adam-Origamiiris how is your jupyterhub restarting every night? Do you have logs? Even if JupyterHub crashes it should shutdown CHP unless it's dying of a pretty severe error

Resolving patch:
diff --git a/jupyterhub/proxy.py b/jupyterhub/proxy.py
--- a/jupyterhub/proxy.py
+++ b/jupyterhub/proxy.py
@@ -22,7 +22,9 @@
 from functools import wraps
 import json
 import os
+import signal
 from subprocess import Popen
+import time
 from urllib.parse import quote
 
 from tornado import gen
@@ -437,6 +439,12 @@ def _auth_token_default(self):
                       help="""The command to start the proxy"""
                       )
 
+    pid_file = Unicode(
+        "jupyterhub-proxy.pid",
+        config=True,
+        help="File in which to write the PID of the proxy process.",
+    )
+
     _check_running_callback = Any(help="PeriodicCallback to check if the proxy is running")
 
     def __init__(self, **kwargs):
@@ -448,9 +456,72 @@ def __init__(self, **kwargs):
                 " if Proxy.should_start is False" % self.__class__.__name__
             )
 
+    def _check_previous_process(self):
+        """Check if there's a process leftover and shut it down if so"""
+        if not self.pid_file or not os.path.exists(self.pid_file):
+            return
+        pid_file = os.path.abspath(self.pid_file)
+        self.log.warning("Found proxy pid file: %s", pid_file)
+        try:
+            with open(pid_file, "r") as f:
+                pid = int(f.read().strip())
+        except ValueError:
+            self.log.warning("%s did not appear to contain a pid", pid_file)
+            self._remove_pid_file()
+            return
+
+        try:
+            os.kill(pid, 0)
+        except ProcessLookupError:
+            self.log.warning("Proxy no longer running at pid=%s", pid)
+            self._remove_pid_file()
+            return
+
+        # if we got here, CHP is still running
+        self.log.warning("Proxy still running at pid=%s", pid)
+        for i, sig in enumerate([signal.SIGTERM] * 2 + [signal.SIGKILL]):
+            try:
+                os.kill(pid, signal.SIGTERM)
+            except ProcessLookupError:
+                break
+            time.sleep(1)
+            try:
+                os.kill(pid, 0)
+            except ProcessLookupError:
+                break
+
+        try:
+            os.kill(pid, 0)
+        except ProcessLookupError:
+            self.log.warning("Stopped proxy at pid=%s", pid)
+            self._remove_pid_file()
+            return
+        else:
+            raise RuntimeError("Failed to stop proxy at pid=%s", pid)
 
+    def _write_pid_file(self):
+        """write pid for proxy to a file"""
+        self.log.debug("Writing proxy pid file: %s", self.pid_file)
+        with open(self.pid_file, "w") as f:
+            f.write(str(self.proxy_process.pid))
+
+    def _remove_pid_file(self):
+        """Cleanup pid file for proxy after stopping"""
+        if not self.pid_file:
+            return
+        self.log.debug("Removing proxy pid file %s", self.pid_file)
+        try:
+            os.remove(self.pid_file)
+        except FileNotFoundError:
+            self.log.debug("PID file %s already removed", self.pid_file)
+            pass
 
     async def start(self):
+        """Start the proxy process"""
+        # check if there is a previous instance still around
+        self._check_previous_process()
+
+        # build the command to launch
         public_server = Server.from_url(self.public_url)
         api_server = Server.from_url(self.api_url)
         env = os.environ.copy()
@@ -496,6 +567,8 @@ async def start(self):
             )
             raise
 
+        self._write_pid_file()
+
         def _check_process():
             status = self.proxy_process.poll()
             if status is not None:
@@ -519,13 +592,20 @@ def _check_process():
         self._check_running_callback = pc
         pc.start()
 
-    def _kill_proc_tree(self, pid):
-        import psutil
-        parent = psutil.Process(pid)
-        children = parent.children(recursive=True)
-        for child in children:
-            child.kill()
-        psutil.wait_procs(children, timeout=5)
+    def _terminate(self):
+        """Terminate our process"""
+        if os.name == 'nt':
+            # On Windows we spawned a shell on Popen, so we need to
+            # terminate all child processes as well
+            import psutil
+
+            parent = psutil.Process(self.proxy_process.pid)
+            children = parent.children(recursive=True)
+            for child in children:
+                child.kill()
+            psutil.wait_procs(children, timeout=5)
+        else:
+            self.proxy_process.terminate()
 
     def stop(self):
         self.log.info("Cleaning up proxy[%i]...", self.proxy_process.pid)
@@ -533,14 +613,10 @@ def stop(self):
             self._check_running_callback.stop()
         if self.proxy_process.poll() is None:
             try:
-                if os.name == 'nt':
-                     # On Windows we spawned a shell on Popen, so we need to
-                    # terminate all child processes as well
-                    self._kill_proc_tree(self.proxy_process.pid)
-                else:
-                    self.proxy_process.terminate()
+                self._terminate()
             except Exception as e:
                 self.log.error("Failed to terminate proxy process: %s", e)
+        self._remove_pid_file()
 
     async def check_running(self):
         """Check if the proxy is still running"""
@@ -549,6 +625,7 @@ async def check_running(self):
         self.log.error("Proxy stopped with exit code %r",
                        'unknown' if self.proxy_process is None else self.proxy_process.poll()
                        )
+        self._remove_pid_file()
         await self.start()
         await self.restore_routes()
 


Test patch: