Repository: jupyterhub/jupyterhub

Issue:
JupyterHub fails on startup, when there is a pid file, but the actual pid is an unrelated process
**Describe the bug**

Given: a Linux server running JupyterHub among other things. Proxy is running with pid 1234
When: the server is restarted, without properly shutting down JH. A process comes up before JH with pid 1234. Then JH is started.
Then: as the pid file is there, it will try to stop the process with pid 1234, either successfully (which can cause unpredictable result) or unsuccessfully (e.g. permission denied to kill that process).

**Expected behavior**

It should check the user and the command of the running service with pid "1234". If the user is the same and the command is something related to JupyterHub then is it is okay to try to kill that process. If not then it should remove the pid file and start up.

**Compute Information**

 - JupyterHub Version 1.0.0



Discussion:
it should indeed check that it's jupyterhub! Workaround until this has occurred is to delete the pid file manually.

This situation should only occur if jupyterhub crashes hard or the machine is shutdown in an ungraceful way. Are you seeing it happen more often than that?
Yes, that workaround is working, this is that same what I have done to solve it. 
And yes, It "only occurs if jupyterhub crashes hard or the machine is shutdown in an ungraceful way".
Just ran into this. We're running the hub on Windows, and we launch/shutdown a number of processes alongside the hub. After a restart of our system, one of the other process we launched had assumed the configurable proxy PID.

I'm happy to dig into a fix. It looks like the `ConfigurableHTTPProxy` launcher knows the expected name and command line argument to launch the proxy. We could use that to verify that the process behind that PID is actually the proxy when checking for an existing process. Does that seem like a viable approach @minrk ?

Resolving patch:
diff --git a/jupyterhub/proxy.py b/jupyterhub/proxy.py
--- a/jupyterhub/proxy.py
+++ b/jupyterhub/proxy.py
@@ -497,6 +497,19 @@ def _check_pid(self, pid):
 
             if not psutil.pid_exists(pid):
                 raise ProcessLookupError
+
+            try:
+                process = psutil.Process(pid)
+                if self.command and self.command[0]:
+                    process_cmd = process.cmdline()
+                    if process_cmd and not any(
+                        self.command[0] in clause for clause in process_cmd
+                    ):
+                        raise ProcessLookupError
+            except (psutil.AccessDenied, psutil.NoSuchProcess):
+                # If there is a process at the proxy's PID but we don't have permissions to see it,
+                # then it is unlikely to actually be the proxy.
+                raise ProcessLookupError
         else:
             os.kill(pid, 0)
 
@@ -692,8 +705,17 @@ def _terminate_win(self, pid):
         parent = psutil.Process(pid)
         children = parent.children(recursive=True)
         for child in children:
-            child.kill()
-        psutil.wait_procs(children, timeout=5)
+            child.terminate()
+        gone, alive = psutil.wait_procs(children, timeout=5)
+        for p in alive:
+            p.kill()
+        # Clear the shell, too, if it still exists.
+        try:
+            parent.terminate()
+            parent.wait(timeout=5)
+            parent.kill()
+        except psutil.NoSuchProcess:
+            pass
 
     def _terminate(self):
         """Terminate our process"""


Test patch: