Title: Using a global Popen object from a Flask route
Tags: python,flask,subprocess,gunicorn

Question: I have a small Flask app, running with Gunicorn configured with one worker. At the app's initialization, it starts up a subprocess.Popen and hangs onto it in a global variable.
import subprocess
def my_init():
   ...
   global MY_SUBPROCESS
   MY_SUBPROCESS = subprocess.Popen(["my_process", "foo", "bar"])

I'd like to have a route that reports on whether that global subprocess is still running.
@app.route("/poll-subprocess")
def my_poll():
   # expected None if the subprocess is still running, however it's always returning 0
   return MY_SUBPROCESS.poll()

The route can see the global Popen object just fine, however the .poll call always returns 0, even if the process is still running.
I haven't found anything in the subprocess documentation that can explain this. I suppose there is a threading issue playing into this somehow?

edit: here is a reproducible example:
from subprocess import Popen

from flask import Flask, jsonify
import gunicorn.app.base

# (copied from example https://docs.gunicorn.org/en/stable/custom.html)
class MyApp(gunicorn.app.base.BaseApplication):
    def __init__(self, app, options=None):
        self.options = options or {}
        self.application = app
        super().__init__()

    def load_config(self):
        config = {key: value for key, value in self.options.items()
                  if key in self.cfg.settings and value is not None}
        for key, value in config.items():
            self.cfg.set(key.lower(), value)

    def load(self):
        return self.application

# Spawn our global subprocess
MY_PROCESS = Popen(["sleep", "999"])

# Create the app
app = Flask(__name__)
@app.route("/")
def hello():
    """Return some info about our global subprocess"""
    return jsonify({
        "obj": str(MY_PROCESS),
        "pid": MY_PROCESS.pid,
        "poll": MY_PROCESS.poll()
    })

# Preview the same info we'll have exposed on a route
print({
    "obj": str(MY_PROCESS),
    "pid": MY_PROCESS.pid,
    "poll": MY_PROCESS.poll()
})

# Launch the app
MyApp(app).run()

Example output running it:
$ python ./demo.py
{'obj': '<subprocess.Popen object at 0x7fe2f87ed0f0>', 'pid': 20135, 'poll': None}
[2023-04-20 12:13:51 -0400] [20134] [INFO] Starting gunicorn 20.0.4
[2023-04-20 12:13:51 -0400] [20134] [INFO] Listening at: http://127.0.0.1:8000 (20134)
[2023-04-20 12:13:51 -0400] [20134] [INFO] Using worker: sync
[2023-04-20 12:13:51 -0400] [20138] [INFO] Booting worker with pid: 20138

$ curl http://127.0.0.1:8000/
{"obj":"<subprocess.Popen object at 0x7fe2f87ed0f0>","pid":20135,"poll":0}

Note it's reporting the same python object and pid as the main thread did when we started the app, however now poll() looks like the process terminated. According to ps, it's still running:
$ ps -p 20135 -o '%p %a'
  PID COMMAND
20135 sleep 999

Answer: This isn't a flask issue; it's a gunicorn issue. Gunicorn forks off workers, which then are responsible for handling requests, as children.
This means that when gunicorn runs waitpid(), it's doing so from a sibling of the sleep process, not from its parent. waitpid() only works correctly when run from a parent.
If you need to be able to start and monitor child processes, gunicorn is probably the wrong tool for the job.