Repository: benoitc/gunicorn

Issue:
(deleted) tmp files left around 
Env:
gunicorn==19.6.0
OS = CentOS
Running under systemd.

A bunch of temp files are still owned by my gunicorn worker processes. The number of temp files is varying across workers, that worker has 45 left around, another one has 67, some seems to be shared. The naming of those temp files seems to be mean that they are used/owned by gunicorn. I ran into resource problems (max number of opened files > 1024) and one of our Ops guy noticed that gunicorn had many opened files.

Thanks for any hints on debugging this !!

I start/stop the service through systemd, and I have not configured how systemd should stop and reload the app. My upgrade process make it stop and start. For the record I tried to implement custom restart by sending USR2, then WINCH and TERM to the old process, but systemd was ignoring that, detecting it as non handled workflow and would restart gunicorn on its own.

```
gunicorn: 3378 crashzone    7u      REG                8,3         0 269227083 /tmp/wgunicorn-WUef6V (deleted)
gunicorn: 3378 crashzone    8u      REG                8,3         0 269227084 /tmp/wgunicorn-z78nQi (deleted)
gunicorn: 3378 crashzone    9u      REG                8,3         0 269227085 /tmp/wgunicorn-yBPZFy (deleted)
gunicorn: 3378 crashzone   10u      REG                8,3         0 269227116 /tmp/wgunicorn-IZ8yLv (deleted)
gunicorn: 3378 crashzone   11u      REG                8,3         0 269227118 /tmp/wgunicorn-b7RadI (deleted)
gunicorn: 3378 crashzone   12u      REG                8,3         0 269231113 /tmp/wgunicorn-QTigaF (deleted)
gunicorn: 3378 crashzone   13u      REG                8,3         0 269231114 /tmp/wgunicorn-2Vthud (deleted)
gunicorn: 3378 crashzone   14u      REG                8,3         0 269231116 /tmp/wgunicorn-8CHx_z (deleted)
gunicorn: 3378 crashzone   15u      REG                8,3         0 269231144 /tmp/wgunicorn-OIXCeX (deleted)
gunicorn: 3378 crashzone   16u      REG                8,3         0 269231145 /tmp/wgunicorn-saa7Vq (deleted)
gunicorn: 3378 crashzone   17u      REG                8,3         0 269231146 /tmp/wgunicorn-jX9a4S (deleted)
gunicorn: 3378 crashzone   18u      REG                8,3         0 269231154 /tmp/wgunicorn-Dm1KAE (deleted)
gunicorn: 3378 crashzone   19u      REG                8,3         0 269231155 /tmp/wgunicorn-RBV92D (deleted)
gunicorn: 3378 crashzone   20u      REG                8,3         0 269231157 /tmp/wgunicorn-VUD025 (deleted)
gunicorn: 3378 crashzone   21u      REG                8,3         0 269231158 /tmp/wgunicorn-VxoyId (deleted)
gunicorn: 3378 crashzone   22u      REG                8,3         0 269231159 /tmp/wgunicorn-h65JSk (deleted)
gunicorn: 3378 crashzone   23u      REG                8,3         0 269231160 /tmp/wgunicorn-yPNANx (deleted)
gunicorn: 3378 crashzone   24u      REG                8,3         0 269231161 /tmp/wgunicorn-P0YYso (deleted)
gunicorn: 3378 crashzone   25u      REG                8,3         0 269231162 /tmp/wgunicorn-ykwUxS (deleted)
gunicorn: 3378 crashzone   26u      REG                8,3         0 269231163 /tmp/wgunicorn-GhL7Yl (deleted)
gunicorn: 3378 crashzone   27u      REG                8,3         0 269231164 /tmp/wgunicorn-gV2KLt (deleted)
```

I have a wrapper for running gunicorn, which I ran entirely through python:

```
'''Start the gunicorn HTTP server
'''

import gunicorn.app.base
from gunicorn.six import iteritems


class StandaloneApplication(gunicorn.app.base.BaseApplication):
    '''A gunicorn application'''

    def __init__(self, app, options=None):
        '''Construct a gunicorn standalone app'''

        self.options = options or {}
        self.application = app
        super(StandaloneApplication, self).__init__()

    def init(self, parser, opts, args):
        '''init the Gunicorn app (the base class method is not implemented)'''

        super(StandaloneApplication, self).init(parser, opts, args)

    def load_config(self):
        '''Load our config file'''

        config = dict([(key, value) for key, value in iteritems(self.options)
                       if key in self.cfg.settings and value is not None])
        for key, value in iteritems(config):
            self.cfg.set(key.lower(), value)

    def load(self):
        '''Load our application'''

        return self.application
```

I run the application like that:

```
StandaloneApplication(app, options).run()
```



Discussion:
Right now i don't see how a worker can have more than one tempfile. It's only created when the worker instance is created and deleted when the worker crash. SO the pid in theory disappeared. Can you provide me some kind of scenario that would help me to reproduce that issue?

I am having this same issue. I don't see any updates on this. 

I did a very poor job of trying to come up with repro steps for this issue ... @xxwolfsrainxx / do you have any idea on how to answer @benoitc questions ?

I too am running a CustomApplication that extends gunicorn.app.base.BaseApplication that is started via a systemd service.  The application will just continue to grow the number of open files it has due to an increasing amount of tmp/wgunicorn\* files that are created. The files show "deleted" under lsof however it still shows open by many processes. I believe this is because the file is consistently opened by new gunicorn processes. For example a tmp file /tmp/wgunicorn-_fzofg23 is created, and then owned by many gunicorn worker processes (69129, 69130, 69131, etc). I think this is preventing the OS from cleaning up the actual file. Maybe my understanding is incorrect, please correct me if so. Is it possible that newly spawned processes re-use other processes temp files ?

In my case, I don't get that many requests per second so I have set this: 'max_requests': 100 
At some point I ran into very very high virtual memory usage (my wsgi handler writes files to disk), but now this is not the case anymore.

But that's for a different problem ... I think that creating a repro app would help a lot to figure out what's going on here. And maybe see if running outside of systemd changes something.

```
(running lsof on a gunicorn worker)
gunicorn: 27263 crashzone   22u      REG                8,3         0  271906700 /tmp/wgunicorn-nTV8qF (deleted)
gunicorn: 27263 crashzone   23u      REG                8,3         0  270268620 /tmp/wgunicorn-_hCfWy (deleted)
gunicorn: 27263 crashzone   24u      REG                8,3         0  270299152 /tmp/wgunicorn-zxYa4E (deleted)
gunicorn: 27263 crashzone   25u      REG                8,3         0  270284860 /tmp/wgunicorn-PSCD3b (deleted)
gunicorn: 27263 crashzone   26u      REG                8,3         0  270295094 /tmp/wgunicorn-SyAm4x (deleted)
gunicorn: 27263 crashzone   27u      REG                8,3         0  270299147 /tmp/wgunicorn-7R0GAq (deleted)
$ lsof -p 27263 | grep tmp | wc -l
21
```

Now that you mention it, my process is using an insane amount of virtual memory as well. I thought this was because of the way the OS collects the memory back from python. It seems you and I have similar issues. What did you do for the virtual memory issue? I will implement your max_requests 100 as well.

How can I reproduce it? 
#1310 refactors the way socket inheritance and activation works. Since that might affect systemd, can anyone try it?
Is #1310 part of a released version (as in available from PyPI) of gunicorn ?
@bsergean no, but I would like to merge it at some point and it helps to have people using systemd test it. I also hope that maybe it fixes your issue. Is it possible for you to test?
@bsergean actually, I apologize. I don't think #1310 is related to this issue at all. This issue is some problem with worker temp files, not socket inheritance. Please forgive the noise.
I easily reproduce this with SIGHUP. I think the workers are inheriting the arbiter's copy of old workers' tmp files.
Here's a patch to fix it. I will try to write a test and open a PR as soon as possible:

```diff
diff --git a/gunicorn/arbiter.py b/gunicorn/arbiter.py
index ff5d698..eb5c0dc 100644
--- a/gunicorn/arbiter.py
+++ b/gunicorn/arbiter.py
@@ -548,6 +548,10 @@ class Arbiter(object):
             self.WORKERS[pid] = worker
             return pid
 
+        # Do not inherit the temporary files of other workers
+        for sibling in self.WORKERS.values():
+            sibling.tmp.close()
+
         # Process Child
         worker_pid = os.getpid()
         try:
```
oh oh ... great findings :) !
Hey @tilgovi / I just tried your small patch and it works like a charm. I only see one temp file now (but I just restarted my web service).

Resolving patch:
diff --git a/gunicorn/arbiter.py b/gunicorn/arbiter.py
--- a/gunicorn/arbiter.py
+++ b/gunicorn/arbiter.py
@@ -565,6 +565,10 @@ def spawn_worker(self):
             self.WORKERS[pid] = worker
             return pid
 
+        # Do not inherit the temporary files of other workers
+        for sibling in self.WORKERS.values():
+            sibling.tmp.close()
+        
         # Process Child
         worker_pid = os.getpid()
         try:


Test patch: