Repository: benoitc/gunicorn

Issue:
unix socket gets deleted on worker restart
We have a django app deployed on heroku with gunicorn 19.6.0
It is binding to a unix socket that's used by Nginx.

These are the options that we're using:

```
  --max-requests 500 \
  --max-requests-jitter 25 \
  --preload \
  --threads 2 \
  --timeout 27 \
  --workers 3 \
  --bind unix:/tmp/nginx.socket
```

What we're seeing is that the socket file disappears after a while.
Downgraded to gunicorn 19.4.5 and things are working fine.

Is the nginx socket file being deleted accidentally?



Discussion:
@diwu1989 do you have any gunicorn logs during th time int happen? It would help to figure what's happening. I can't reproduce it for now.

yes i realize my initial context doesn't provide enough, Im going to try to create a repro steps to post

here's one snippet of log I collected while rebooting a dyno

```
2016-06-26T05:24:48.691401+00:00 app[web.1]: [2016-06-26 05:24:48 +0000] [23] [ERROR] Exception in worker process
2016-06-26T05:24:48.691403+00:00 app[web.1]: Traceback (most recent call last):
2016-06-26T05:24:48.691404+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
2016-06-26T05:24:48.691405+00:00 app[web.1]:     worker.init_process()
2016-06-26T05:24:48.691406+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 109, in init_process
2016-06-26T05:24:48.691406+00:00 app[web.1]:     super(ThreadWorker, self).init_process()
2016-06-26T05:24:48.691407+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in init_process
2016-06-26T05:24:48.691408+00:00 app[web.1]:     self.run()
2016-06-26T05:24:48.691408+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 240, in run
2016-06-26T05:24:48.691409+00:00 app[web.1]:     s.close()
2016-06-26T05:24:48.691410+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/sock.py", line 123, in close
2016-06-26T05:24:48.691411+00:00 app[web.1]:     os.unlink(self.cfg_addr)
2016-06-26T05:24:48.691411+00:00 app[web.1]: OSError: [Errno 2] No such file or directory: '/tmp/nginx.socket'
```

this seems to be what's causing the socket to disappear

```
2016-06-26T05:35:11.069314+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [24] [DEBUG] GET /
2016-06-26T05:35:11.070223+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [24] [DEBUG] Closing connection.
2016-06-26T05:35:11.443232+00:00 heroku[router]: at=info method=GET path="/" host=gunicorn.herokuapp.com request_id=7cd8420c-53d5-4b27-9d17-e7793fb5b4d7 fwd="52.33.79.242" dyno=web.1 connect=0ms service=3ms status=200 bytes=162
2016-06-26T05:35:11.409075+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [23] [DEBUG] GET /
2016-06-26T05:35:11.409329+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [23] [INFO] Autorestarting worker after current request.
2016-06-26T05:35:11.410194+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [23] [DEBUG] Closing connection.
2016-06-26T05:35:11.787756+00:00 heroku[router]: at=info method=GET path="/" host=gunicorn.herokuapp.com request_id=9899ce58-1450-4427-8d2a-9201949eae72 fwd="52.33.79.242" dyno=web.1 connect=0ms service=3ms status=200 bytes=162
2016-06-26T05:35:11.774464+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [23] [DEBUG] GET /
2016-06-26T05:35:11.775451+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [23] [DEBUG] Closing connection.
2016-06-26T05:35:11.775798+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [23] [INFO] Worker exiting (pid: 23)
2016-06-26T05:35:11.800646+00:00 app[web.1]: [2016-06-26 05:35:11 +0000] [35] [INFO] Booting worker with pid: 35
2016-06-26T05:35:12.140399+00:00 heroku[router]: at=info method=GET path="/" host=gunicorn.herokuapp.com request_id=fecd3f83-46f4-46f9-aee1-eae85f9ae626 fwd="52.33.79.242" dyno=web.1 connect=0ms service=1ms status=502 bytes=311
2016-06-26T05:35:12.138120+00:00 app[web.1]: 2016/06/26 05:35:12 [crit] 29#0: *442 connect() to unix:/tmp/nginx.socket failed (2: No such file or directory) while connecting to upstream, client: 10.93.192.30, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/tmp/nginx.socket:/", host: "gunicorn.herokuapp.com"
2016-06-26T05:35:12.522873+00:00 heroku[router]: at=info method=GET path="/" host=gunicorn.herokuapp.com request_id=bf01f7fd-fcd1-4678-b882-c5ed0b21fa23 fwd="52.33.79.242" dyno=web.1 connect=0ms service=1ms status=502 bytes=311
2016-06-26T05:35:12.488939+00:00 app[web.1]: 2016/06/26 05:35:12 [crit] 29#0: *444 connect() to unix:/tmp/nginx.socket failed (2: No such file or directory) while connecting to upstream, client: 10.123.140.213, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/tmp/nginx.socket:/", host: "gunicorn.herokuapp.com"
2016-06-26T05:35:12.843974+00:00 heroku[router]: at=info method=GET path="/" host=gunicorn.herokuapp.com request_id=bc8cc4ba-bb4b-4326-9181-742b84e3a9e6 fwd="52.33.79.242" dyno=web.1 connect=0ms service=1ms status=502 bytes=311
2016-06-26T05:35:12.840446+00:00 app[web.1]: 2016/06/26 05:35:12 [crit] 29#0: *446 connect() to unix:/tmp/nginx.socket failed (2: No such file or directory) while connecting to upstream, client: 10.179.144.75, server: _, request: "GET / HTTP/1.1", upstream: "http://unix:/tmp/nginx.socket:/", host: "gunicorn.herokuapp.com"
```

Since im running with multiple workers, it seems that whenever one of them is autorestarting, as part of the worker exiting, the unix socket would get wiped
`Autorestarting worker after current request.`
`Worker exiting (pid: 23)`
`Booting worker with pid: 35`
`*442 connect() to unix:/tmp/nginx.socket failed (2: No such file or directory) while connecting to upstream`

commit 118668c8d6a04e15f748acd89d142fd2a1996fc6 seems to be fine
after commit c62cf2f5009a1d18b3ccda58e42b88b47dffbaa1 is bad
![image](https://cloud.githubusercontent.com/assets/380950/16360789/1c2a8dee-3b29-11e6-8449-25694d9fc36e.png)

to reproduce, clone this repo https://github.com/diwu1989/gunicorn-socket-bug
deploy to heroku app, then run apache bench on it to trigger the behavior

my guess is that the bug is related to this change
![image](https://cloud.githubusercontent.com/assets/380950/16360808/4f9fcdc8-3b2a-11e6-944f-d92a17a5c3d7.png)

@diwu1989 The code you reference in the old version was not present in a stable release. Anyway in the current version the only reason the socket can disappear is when gunicorn has quit or has been terminated (ie when the arbiter die). Do you have any production logs that shows a restart or termination of gunicorn?

 I've no available account on heroku right now, but will try to create one to test it on monday.

Sorry I missed the first log. I think there is a bug in the thread worker. it shouldn't close the socket there. I will have a closer look ASAP on it.

i dont think this is a heroku specific problem, it looks like gthread worker will end up deleting the unix socket when they reach their lifespan limit (if the max request setting is set)

@benoitc it should close the socket there

I may have made a mistake in reviewing c62cf2f5009a1d18b3ccda58e42b88b47dffbaa1. We made the change so that only one arbiter will close the socket, and removed the protections in the socket's `close` method. However, workers still close the socket.

The workers _should_ close the socket, though, because otherwise the OS still thinks there is a listener during graceful shutdown when what we want is immediate rejection of new connections so that load balancers and reverse proxies can fail over.

Maybe we should just make `unlink` explicit and not have it implied by `close`. Then, the top arbiter can explicitly unlike those sockets that should be unlinked.

when do you guys expect to have this fixed?

I'm still waiting for a response to my last comment, @diwu1989. What do you think?

@tilgovi before that we only have a barrier checking if the parent was actually unlinking the socket. Maybe we should just keep that? Ie. we keep the parent pid when creating the socket. If the pid is different we don't unlink. Thoughts?

@diwu1989 we will try to make a release during the coming week. 

Yes, we need to keep the barrier for `unlink`. What I am saying is we _also_ need to make `unlink` explicit so when the workers close the socket they don't unlink it.

So then within the conditional in the arbiter only we would explicitly call `unlink`.

why making it explicit? The socket is only created on the arbiter, so there is no chance that a worker could unlink it. Something like:

``` python
def close(self):
   if (os.pid() == self.parent_pid): os.unlink(self.fd)
   ...
```

if we make it explicit, then there is no need to have the barrier I described. What did I miss?

I think you may be forgetting again why the workers must explicitly close the socket. They do this for graceful shutdown. If the socket is closed by all processes the operating system will not hold new, half-open connections. It will reject them, letting downstream load balancers shift traffic instantly.

If the workers always exited right after they closed the socket, they could simply exit. Instead, the workers need to explicitly close the socket sometimes, but they should not unlink it.

A worker is not the only place the socket is closed before Gunicorn shuts down. The other place is where an old arbiter exits. The barrier is there to protect an exiting arbiter from closing the socket when another arbiter still exists and Gunicorn is not exiting.

The original report was about arbiter causing unlink. We fixed that, but in the meantime I fixed every worker type to explicit close as soon as graceful shutdown is triggered. Now we have a new case where unlink is called.

I'm suggesting that we make unlink explicit. You're sort of correct that the barrier is not necessary around the close operation. It is only necessary around the unlink.

If unlink were explicit, the workers could close without worrying about unlink, the arbiters could close without unlink, and the current conditional check could ensure that only one arbiter performs an explicit unlink, separate from close.

I don't follow... The unlink in the given snippet onlyt unlink if it's closed by the parent (pid) that created the socket. 

I'm not against having an explicit unlink, so let's do that, it's simpler.

please review #1309 for that :)

Sometimes the PID that unlinks it is not the PID that created it, such as after a USR2. Just checking the current and previous PID is not sufficient to cover all cases. That's why we introduced the explicit tracking of which arbiter is responsible to unlink.

@tilgovi well the parent pid would have been set each time the socket is initialised :) but anyway the patch above use explicit unlink. Let me know

Thanks for writing the patch. I'm mobile but will review in some hours.


Resolving patch:
diff --git a/gunicorn/arbiter.py b/gunicorn/arbiter.py
--- a/gunicorn/arbiter.py
+++ b/gunicorn/arbiter.py
@@ -15,8 +15,7 @@
 
 from gunicorn.errors import HaltServer, AppImportError
 from gunicorn.pidfile import Pidfile
-from gunicorn.sock import create_sockets
-from gunicorn import util
+from gunicorn import sock, systemd, util
 
 from gunicorn import __version__, SERVER_SOFTWARE
 
@@ -61,6 +60,7 @@ def __init__(self, app):
         self.setup(app)
 
         self.pidfile = None
+        self.systemd = False
         self.worker_age = 0
         self.reexec_pid = 0
         self.master_pid = 0
@@ -140,8 +140,21 @@ def start(self):
         self.cfg.on_starting(self)
 
         self.init_signals()
+
         if not self.LISTENERS:
-            self.LISTENERS = create_sockets(self.cfg, self.log)
+            fds = []
+
+            listen_fds = systemd.listen_fds()
+            if listen_fds:
+                self.systemd = True
+                fds = range(systemd.SD_LISTEN_FDS_START,
+                            systemd.SD_LISTEN_FDS_START + listen_fds)
+
+            elif self.master_pid:
+                for fd in os.environ.pop('GUNICORN_FD').split(','):
+                    fds.append(int(fd))
+
+            self.LISTENERS = sock.create_sockets(self.cfg, self.log, fds)
 
         listeners_str = ",".join([str(l) for l in self.LISTENERS])
         self.log.debug("Arbiter booted")
@@ -365,9 +378,8 @@ def stop(self, graceful=True):
         killed gracefully  (ie. trying to wait for the current connection)
         """
 
-        if self.reexec_pid == 0 and self.master_pid == 0:
-            for l in self.LISTENERS:
-                l.close()
+        unlink = self.reexec_pid == self.master_pid == 0 and not self.systemd
+        sock.close_sockets(self.LISTENERS, unlink)
 
         self.LISTENERS = []
         sig = signal.SIGTERM
@@ -402,10 +414,15 @@ def reexec(self):
         self.cfg.pre_exec(self)
 
         environ = self.cfg.env_orig.copy()
-        fds = [l.fileno() for l in self.LISTENERS]
-        environ['GUNICORN_FD'] = ",".join([str(fd) for fd in fds])
         environ['GUNICORN_PID'] = str(master_pid)
 
+        if self.systemd:
+            environ['LISTEN_PID'] = str(os.getpid())
+            environ['LISTEN_FDS'] = str(len(self.LISTENERS))
+        else:
+            environ['GUNICORN_FD'] = ','.join(
+                str(l.fileno()) for l in self.LISTENERS)
+
         os.chdir(self.START_CTX['cwd'])
 
         # exec the process using the original environment
@@ -439,7 +456,7 @@ def reload(self):
             # close all listeners
             [l.close() for l in self.LISTENERS]
             # init new listeners
-            self.LISTENERS = create_sockets(self.cfg, self.log)
+            self.LISTENERS = sock.create_sockets(self.cfg, self.log)
             listeners_str = ",".join([str(l) for l in self.LISTENERS])
             self.log.info("Listening at: %s", listeners_str)
 
diff --git a/gunicorn/sock.py b/gunicorn/sock.py
--- a/gunicorn/sock.py
+++ b/gunicorn/sock.py
@@ -13,8 +13,6 @@
 from gunicorn import util
 from gunicorn.six import string_types
 
-SD_LISTEN_FDS_START = 3
-
 
 class BaseSocket(object):
 
@@ -123,10 +121,6 @@ def bind(self, sock):
         util.chown(self.cfg_addr, self.conf.uid, self.conf.gid)
         os.umask(old_umask)
 
-    def close(self):
-        os.unlink(self.cfg_addr)
-        super(UnixSocket, self).close()
-
 
 def _sock_type(addr):
     if isinstance(addr, tuple):
@@ -141,41 +135,15 @@ def _sock_type(addr):
     return sock_type
 
 
-def create_sockets(conf, log):
-    """
-    Create a new socket for the given address. If the
-    address is a tuple, a TCP socket is created. If it
-    is a string, a Unix socket is created. Otherwise
-    a TypeError is raised.
+def create_sockets(conf, log, fds=None):
     """
+    Create a new socket for the configured addresses or file descriptors.
 
-    # Systemd support, use the sockets managed by systemd and passed to
-    # gunicorn.
-    # http://www.freedesktop.org/software/systemd/man/systemd.socket.html
+    If a configured address is a tuple then a TCP socket is created.
+    If it is a string, a Unix socket is created. Otherwise, a TypeError is
+    raised.
+    """
     listeners = []
-    if ('LISTEN_PID' in os.environ
-            and int(os.environ.get('LISTEN_PID')) == os.getpid()):
-        for i in range(int(os.environ.get('LISTEN_FDS', 0))):
-            fd = i + SD_LISTEN_FDS_START
-            try:
-                sock = socket.fromfd(fd, socket.AF_UNIX, socket.SOCK_STREAM)
-                sockname = sock.getsockname()
-                if isinstance(sockname, str) and sockname.startswith('/'):
-                    listeners.append(UnixSocket(sockname, conf, log, fd=fd))
-                elif len(sockname) == 2 and '.' in sockname[0]:
-                    listeners.append(TCPSocket("%s:%s" % sockname, conf, log,
-                        fd=fd))
-                elif len(sockname) == 4 and ':' in sockname[0]:
-                    listeners.append(TCP6Socket("[%s]:%s" % sockname[:2], conf,
-                        log, fd=fd))
-            except socket.error:
-                pass
-        del os.environ['LISTEN_PID'], os.environ['LISTEN_FDS']
-
-        if listeners:
-            log.debug('Socket activation sockets: %s',
-                    ",".join([str(l) for l in listeners]))
-            return listeners
 
     # get it only once
     laddr = conf.address
@@ -189,28 +157,19 @@ def create_sockets(conf, log):
         raise ValueError('keyfile "%s" does not exist' % conf.keyfile)
 
     # sockets are already bound
-    if 'GUNICORN_FD' in os.environ:
-        fds = os.environ.pop('GUNICORN_FD').split(',')
-        for i, fd in enumerate(fds):
-            fd = int(fd)
-            addr = laddr[i]
-            sock_type = _sock_type(addr)
+    if fds is not None:
+        for fd in fds:
+            sock = socket.fromfd(fd, socket.AF_UNIX, socket.SOCK_STREAM)
+            sock_name = sock.getsockname()
+            sock_type = _sock_type(sock_name)
+            listener = sock_type(sock_name, conf, log, fd=fd)
+            listeners.append(listener)
 
-            try:
-                listeners.append(sock_type(addr, conf, log, fd=fd))
-            except socket.error as e:
-                if e.args[0] == errno.ENOTCONN:
-                    log.error("GUNICORN_FD should refer to an open socket.")
-                else:
-                    raise
         return listeners
 
     # no sockets is bound, first initialization of gunicorn in this env.
     for addr in laddr:
         sock_type = _sock_type(addr)
-        # If we fail to create a socket from GUNICORN_FD
-        # we fall through and try and open the socket
-        # normally.
         sock = None
         for i in range(5):
             try:
@@ -235,3 +194,11 @@ def create_sockets(conf, log):
         listeners.append(sock)
 
     return listeners
+
+
+def close_sockets(listeners, unlink=True):
+    for sock in listeners:
+        sock_name = sock.getsockname()
+        sock.close()
+        if unlink and _sock_type(sock_name) is UnixSocket:
+            os.unlink(sock_name)
diff --git a/gunicorn/systemd.py b/gunicorn/systemd.py
new file mode 100644
--- /dev/null
+++ b/gunicorn/systemd.py
@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -
+#
+# This file is part of gunicorn released under the MIT license.
+# See the NOTICE for more information.
+
+import os
+
+SD_LISTEN_FDS_START = 3
+
+
+def listen_fds(unset_environment=True):
+    """
+    Get the number of sockets inherited from systemd socket activation.
+
+    :param unset_environment: clear systemd environment variables unless False
+    :type unset_environment: bool
+    :return: the number of sockets to inherit from systemd socket activation
+    :rtype: int
+
+    Returns zero immediately if $LISTEN_PID is not set to the current pid.
+    Otherwise, returns the number of systemd activation sockets specified by
+    $LISTEN_FDS.
+
+    When $LISTEN_PID matches the current pid, unsets the environment variables
+    unless the ``unset_environment`` flag is ``False``.
+
+    .. note::
+        Unlike the sd_listen_fds C function, this implementation does not set
+        the FD_CLOEXEC flag because the gunicorn arbiter never needs to do this.
+
+    .. seealso::
+        `<https://www.freedesktop.org/software/systemd/man/sd_listen_fds.html>`_
+
+    """
+    fds = int(os.environ.get('LISTEN_FDS', 0))
+    listen_pid = int(os.environ.get('LISTEN_PID', 0))
+
+    if listen_pid != os.getpid():
+        return 0
+
+    if unset_environment:
+        os.environ.pop('LISTEN_PID', None)
+        os.environ.pop('LISTEN_FDS', None)
+
+    return fds


Test patch:
diff --git a/tests/test_arbiter.py b/tests/test_arbiter.py
--- a/tests/test_arbiter.py
+++ b/tests/test_arbiter.py
@@ -29,14 +29,89 @@ def load_config(self):
         """No-op"""
 
 
-def test_arbiter_shutdown_closes_listeners():
+@mock.patch('gunicorn.sock.close_sockets')
+def test_arbiter_stop_closes_listeners(close_sockets):
     arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
     listener1 = mock.Mock()
     listener2 = mock.Mock()
-    arbiter.LISTENERS = [listener1, listener2]
+    listeners = [listener1, listener2]
+    arbiter.LISTENERS = listeners
+    arbiter.stop()
+    close_sockets.assert_called_with(listeners, True)
+
+
+@mock.patch('gunicorn.sock.close_sockets')
+def test_arbiter_stop_child_does_not_unlink_listeners(close_sockets):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    arbiter.reexec_pid = os.getpid()
+    arbiter.stop()
+    close_sockets.assert_called_with([], False)
+
+
+@mock.patch('gunicorn.sock.close_sockets')
+def test_arbiter_stop_parent_does_not_unlink_listeners(close_sockets):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    arbiter.master_pid = os.getppid()
+    arbiter.stop()
+    close_sockets.assert_called_with([], False)
+
+
+@mock.patch('gunicorn.sock.close_sockets')
+def test_arbiter_stop_does_not_unlink_systemd_listeners(close_sockets):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    arbiter.systemd = True
     arbiter.stop()
-    listener1.close.assert_called_with()
-    listener2.close.assert_called_with()
+    close_sockets.assert_called_with([], False)
+
+
+@mock.patch('os.getpid')
+@mock.patch('os.fork')
+@mock.patch('os.execvpe')
+def test_arbiter_reexec_passing_systemd_sockets(execvpe, fork, getpid):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    arbiter.LISTENERS = [mock.Mock(), mock.Mock()]
+    arbiter.systemd = True
+    fork.return_value = 0
+    getpid.side_effect = [2, 3]
+    arbiter.reexec()
+    environ = execvpe.call_args[0][2]
+    assert environ['GUNICORN_PID'] == '2'
+    assert environ['LISTEN_FDS'] == '2'
+    assert environ['LISTEN_PID'] == '3'
+
+
+@mock.patch('os.getpid')
+@mock.patch('os.fork')
+@mock.patch('os.execvpe')
+def test_arbiter_reexec_passing_gunicorn_sockets(execvpe, fork, getpid):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    listener1 = mock.Mock()
+    listener2 = mock.Mock()
+    listener1.fileno.return_value = 4
+    listener2.fileno.return_value = 5
+    arbiter.LISTENERS = [listener1, listener2]
+    fork.return_value = 0
+    getpid.side_effect = [2, 3]
+    arbiter.reexec()
+    environ = execvpe.call_args[0][2]
+    assert environ['GUNICORN_FD'] == '4,5'
+    assert environ['GUNICORN_PID'] == '2'
+
+
+@mock.patch('os.fork')
+def test_arbiter_reexec_limit_parent(fork):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    arbiter.reexec_pid = ~os.getpid()
+    arbiter.reexec()
+    assert fork.called is False, "should not fork when there is already a child"
+
+
+@mock.patch('os.fork')
+def test_arbiter_reexec_limit_child(fork):
+    arbiter = gunicorn.arbiter.Arbiter(DummyApplication())
+    arbiter.master_pid = ~os.getpid()
+    arbiter.reexec()
+    assert fork.called is False, "should not fork when arbiter is a child"
 
 
 class PreloadedAppWithEnvSettings(DummyApplication):
diff --git a/tests/test_sock.py b/tests/test_sock.py
--- a/tests/test_sock.py
+++ b/tests/test_sock.py
@@ -1,3 +1,8 @@
+# -*- coding: utf-8 -
+#
+# This file is part of gunicorn released under the MIT license.
+# See the NOTICE for more information.
+
 try:
     import unittest.mock as mock
 except ImportError:
@@ -6,13 +11,29 @@
 from gunicorn import sock
 
 
-@mock.patch('os.close')
-@mock.patch('os.getpid')
+def test_socket_close():
+    listener1 = mock.Mock()
+    listener1.getsockname.return_value = ('127.0.0.1', '80')
+    listener2 = mock.Mock()
+    listener2.getsockname.return_value = ('192.168.2.5', '80')
+    sock.close_sockets([listener1, listener2])
+    listener1.close.assert_called_with()
+    listener2.close.assert_called_with()
+
+
+@mock.patch('os.unlink')
+def test_unix_socket_close_unlink(unlink):
+    listener = mock.Mock()
+    listener.getsockname.return_value = '/var/run/test.sock'
+    sock.close_sockets([listener])
+    listener.close.assert_called_with()
+    unlink.assert_called_once_with('/var/run/test.sock')
+
+
 @mock.patch('os.unlink')
-@mock.patch('socket.fromfd')
-def test_unix_socket_close_unlink(fromfd, unlink, getpid, close):
-    fd = 42
-    gsock = sock.UnixSocket('test.sock', mock.Mock(), mock.Mock(), fd=fd)
-    gsock.close()
-    unlink.assert_called_with("test.sock")
-    close.assert_called_with(fd)
+def test_unix_socket_close_without_unlink(unlink):
+    listener = mock.Mock()
+    listener.getsockname.return_value = '/var/run/test.sock'
+    sock.close_sockets([listener], False)
+    listener.close.assert_called_with()
+    assert not unlink.called, 'unlink should not have been called'
diff --git a/tests/test_systemd.py b/tests/test_systemd.py
new file mode 100644
--- /dev/null
+++ b/tests/test_systemd.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -
+#
+# This file is part of gunicorn released under the MIT license.
+# See the NOTICE for more information.
+
+from contextlib import contextmanager
+import os
+
+try:
+    import unittest.mock as mock
+except ImportError:
+    import mock
+
+import pytest
+
+from gunicorn import systemd
+
+
+@contextmanager
+def check_environ(unset=True):
+    """
+    A context manager that asserts post-conditions of ``listen_fds`` at exit.
+
+    This helper is used to ease checking of the test post-conditions for the
+    systemd socket activation tests that parametrize the call argument.
+    """
+
+    with mock.patch.dict(os.environ):
+        old_fds = os.environ.get('LISTEN_FDS', None)
+        old_pid = os.environ.get('LISTEN_PID', None)
+
+        yield
+
+        if unset:
+            assert 'LISTEN_FDS' not in os.environ, \
+                "LISTEN_FDS should have been unset"
+            assert 'LISTEN_PID' not in os.environ, \
+                "LISTEN_PID should have been unset"
+        else:
+            new_fds = os.environ.get('LISTEN_FDS', None)
+            new_pid = os.environ.get('LISTEN_PID', None)
+            assert new_fds == old_fds, \
+                "LISTEN_FDS should not have been changed"
+            assert new_pid == old_pid, \
+                "LISTEN_PID should not have been changed"
+
+
+@pytest.mark.parametrize("unset", [True, False])
+def test_listen_fds_ignores_wrong_pid(unset):
+    with mock.patch.dict(os.environ):
+        os.environ['LISTEN_FDS'] = str(5)
+        os.environ['LISTEN_PID'] = str(1)
+        with check_environ(False):  # early exit — never changes the environment
+            assert systemd.listen_fds(unset) == 0, \
+                "should ignore listen fds not intended for this pid"
+
+
+@pytest.mark.parametrize("unset", [True, False])
+def test_listen_fds_returns_count(unset):
+    with mock.patch.dict(os.environ):
+        os.environ['LISTEN_FDS'] = str(5)
+        os.environ['LISTEN_PID'] = str(os.getpid())
+        with check_environ(unset):
+            assert systemd.listen_fds(unset) == 5, \
+                "should return the correct count of fds"