Repository: benoitc/gunicorn

Issue:
gunicorn.pidfile.validate crashes gunicorn when PID exists but is from a different user
```
Traceback (most recent call last):
  File "/opt/python2.7/bin/gunicorn", line 11, in <module>
    sys.exit(run())
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/app/base.py", line 189, in run
    super(Application, self).run()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/arbiter.py", line 171, in run
    self.start()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/arbiter.py", line 125, in start
    self.pidfile.create(self.pid)
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/pidfile.py", line 23, in create
    oldpid = self.validate()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/pidfile.py", line 75, in validate
    os.kill(wpid, 0)
OSError: [Errno 1] Operation not permitted
```

This happens because the process identified by the pid-file exists, but belongs to a different user than the one starting gunicorn.

(This is with gunicorn 19.3.0)



Discussion:
Thanks for the report :) How did you start gunicorn? Is the process still associated to the arbiter? does the user starting gunicorn have the rights to kill its children?

We run gunicorn with supervisord, nothing really fancy.

I believe what happened was that the server was rebooted, and a few gunicorns did not shutdown in time, so when the server came up again their old pid-files were still there, but now the PIDs belonged to gunicorns from other users (the server runs a bunch of projects as different users.)

@ski-csis but each pidfile is created per projetcs right ? So if the same suer is launching the project each time it shouldn't happen. Can you share your supervisor conf?

There is a pid-file per gunicorn, yes. That's not the problem.

The problem is that the pid in the pid-file no longer belongs to the user running the gunicorn, because gunicorn was not shutdown cleanly before a reboot.

If you want to simulate similar conditions, just pust `1` in a pid-file, and see how you get the same exception, when you try to start gunicorn.

```
vagrant@vagrant:~$ echo 1 > some.pid
vagrant@vagrant:~$ gunicorn --pid some.pid app:app
[2015-08-04 07:28:27 +0000] [2698] [INFO] Starting gunicorn 19.3.0
Traceback (most recent call last):
  File "/opt/python2.7/bin/gunicorn", line 11, in <module>
    sys.exit(run())
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/app/base.py", line 189, in run
    super(Application, self).run()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/arbiter.py", line 171, in run
    self.start()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/arbiter.py", line 125, in start
    self.pidfile.create(self.pid)
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/pidfile.py", line 23, in create
    oldpid = self.validate()
  File "/opt/python2.7/lib/python2.7/site-packages/gunicorn/pidfile.py", line 75, in validate
    os.kill(wpid, 0)
OSError: [Errno 1] Operation not permitted
vagrant@vagrant:~$ gunicorn --version
gunicorn (version 19.3.0)
```

Such error is expected but looking at that issue I think we can have a better answer to it. Ie instead of just crashing we could log the error and have an option to explicitly overwrite the pidfile if it exists. Another option would be not touching on that pidfile and instead create a new pidfile incrementing the name:

```
 /tmp/gunicorn.pid --> /tmp/gunicorn.pid.1
```

What do others think about it? 

bump.

I'd be fine to just catch errors and print an error.

What do other daemons do?

Both exists. Maybe it's safer to just print catch/display error and exist indeed.

Here's another idea that I think mixes robustness and safety:
- If there's a pid file, check if there's a process with that PID.
- If not, remove the pid file and continue.
- If so, abort and log.

This way, the common case of a dead pidfile will be cleaned up. It's possible that a new process has that pid, but less likely. If there's a pidfile that can't be remove, we act paranoid.

I think this is better than creating a suffixed pidfile because that might trick an init system that expects gunicorn to write its pidfile at the configured location only.

Any feedback on my proposal?

@tilgovi sorry  I missed that response. I think it's fine doing that way. Go ahead :)

bump

Can any one suggest some workaround or the configuration that should be used for gunicorn with supervisord

I think this is all we need to do here:

```diff
diff --git a/gunicorn/pidfile.py b/gunicorn/pidfile.py
index 7f7f844..6831512 100644
--- a/gunicorn/pidfile.py
+++ b/gunicorn/pidfile.py
@@ -73,8 +73,9 @@ class Pidfile(object):
 
                 try:
                     os.kill(wpid, 0)
-                    return wpid
                 except OSError as e:
+                    if e.args[0] == errno.EPERM:
+                        return wpid
                     if e.args[0] == errno.ESRCH:
                         return
                     raise
```
Oops, I accidentally deleted the first `return wpid`. Here's the proper diff:

```diff
diff --git a/gunicorn/pidfile.py b/gunicorn/pidfile.py
index 7f7f844..a6e085f 100644
--- a/gunicorn/pidfile.py
+++ b/gunicorn/pidfile.py
@@ -75,6 +75,8 @@ class Pidfile(object):
                     os.kill(wpid, 0)
                     return wpid
                 except OSError as e:
+                    if e.args[0] == errno.EPERM:
+                        return wpid
                     if e.args[0] == errno.ESRCH:
                         return
                     raise
```
Randall's inline patch looks good to me. I think we can easily unit test the new behavior by using a mock with a side effect.

Resolving patch:
diff --git a/gunicorn/pidfile.py b/gunicorn/pidfile.py
--- a/gunicorn/pidfile.py
+++ b/gunicorn/pidfile.py
@@ -75,6 +75,8 @@ def validate(self):
                     os.kill(wpid, 0)
                     return wpid
                 except OSError as e:
+                    if e.args[0] == errno.EPERM:
+                        return wpid
                     if e.args[0] == errno.ESRCH:
                         return
                     raise


Test patch:
diff --git a/tests/test_pidfile.py b/tests/test_pidfile.py
new file mode 100644
--- /dev/null
+++ b/tests/test_pidfile.py
@@ -0,0 +1,59 @@
+# -*- coding: utf-8 -
+#
+# This file is part of gunicorn released under the MIT license.
+# See the NOTICE for more information.
+
+import errno
+import sys
+
+try:
+    import unittest.mock as mock
+except ImportError:
+    import mock
+
+import gunicorn.pidfile
+
+
+def builtin(name):
+    if sys.version_info >= (3, 0):
+        module = 'builtins'
+    else:
+        module = '__builtin__'
+
+    return '{0}.{1}'.format(module, name)
+
+
+@mock.patch(builtin('open'), new_callable=mock.mock_open)
+def test_validate_no_file(_open):
+    pidfile = gunicorn.pidfile.Pidfile('test.pid')
+    _open.side_effect = IOError(errno.ENOENT)
+    assert pidfile.validate() is None
+
+
+@mock.patch(builtin('open'), new_callable=mock.mock_open, read_data='1')
+@mock.patch('os.kill')
+def test_validate_file_pid_exists(kill, _open):
+    pidfile = gunicorn.pidfile.Pidfile('test.pid')
+    assert pidfile.validate() == 1
+
+
+@mock.patch(builtin('open'), new_callable=mock.mock_open, read_data='a')
+def test_validate_file_pid_malformed(_open):
+    pidfile = gunicorn.pidfile.Pidfile('test.pid')
+    assert pidfile.validate() is None
+
+
+@mock.patch(builtin('open'), new_callable=mock.mock_open, read_data='1')
+@mock.patch('os.kill')
+def test_validate_file_pid_exists_kill_exception(kill, _open):
+    pidfile = gunicorn.pidfile.Pidfile('test.pid')
+    kill.side_effect = OSError(errno.EPERM)
+    assert pidfile.validate() == 1
+
+
+@mock.patch(builtin('open'), new_callable=mock.mock_open, read_data='1')
+@mock.patch('os.kill')
+def test_validate_file_pid_does_not_exist(kill, _open):
+    pidfile = gunicorn.pidfile.Pidfile('test.pid')
+    kill.side_effect = OSError(errno.ESRCH)
+    assert pidfile.validate() is None