Repository: pallets/werkzeug

Issue:
ResourceWarning for unclosed sockets with development server
When running e.g. a trivial Flask application with a Python 3.11b1 build configured with `--with-pydebug`, I get:

```
 * Serving Flask app 'werkzeugtest' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
.../lib/python3.11/site-packages/werkzeug/serving.py:716: ResourceWarning: unclosed <socket.socket fd=4, family=2, type=1, proto=0, laddr=('0.0.0.0', 0)>
  self.socket = socket.fromfd(fd, address_family, socket.SOCK_STREAM)
Object allocated at (most recent call last):
  File ".../proj/cpython/Lib/socketserver.py", lineno 452
    self.socket = socket.socket(self.address_family,
 * Running on http://127.0.0.1:5000 (Press CTRL+C to quit)

# pressing Ctrl-C

.../lib/python3.11/site-packages/flask/app.py:920: ResourceWarning: unclosed <socket.socket fd=3, family=2, type=1, proto=0, laddr=('127.0.0.1', 5000)>
  run_simple(t.cast(str, host), port, self, **options)
Object allocated at (most recent call last):
  File ".../lib/python3.11/site-packages/werkzeug/serving.py", lineno 894
    s = socket.socket(address_family, socket.SOCK_STREAM)
```

Reproducer:

```python
import flask

app = flask.Flask(__name__)

if __name__ == '__main__':
    app.run()
```

I'm not quite sure what's happening there, but I'd expect no `ResourceWarning` to be shown.

Note that to test with a Python 3.11.0b1 build with `--with-pydebug`, a faulty assertion needs to be commented out first in CPython:

https://github.com/python/cpython/issues/92597#issuecomment-1137418138

Fixed by commenting out:

https://github.com/python/cpython/blob/8d32a5c8c4e9c90b0a21e8b2fa0cd567465ced2b/Python/compile.c#L7532

Environment:

- Python version: 3.11.0b1
- Werkzeug version: 2.1.2



Discussion:
Happy to review a PR, but this is not something I will address on my own.
I found this a while ago but you beat me to writing up the problem. It's nothing new: I found it with Python 3.7 with a unknown version of `werkzeug<2.1.0` (I didn't keep great notes). The problem is that a socket is declared but is not always closed:

<https://github.com/pallets/werkzeug/blob/edef71c243e1e1396092f7b5f82ddad6c6f766cf/src/werkzeug/serving.py#L894>

I've been able to reliably reproduce it by running `python -Wd -X tracemalloc -c 'from flask import Flask; app = Flask("foo"); app.run(debug=True)'`.

I tried fixing it myself, but from a quick look, it seemed like it would require a non-trivial change. It'd be nice to be fixed, but it's probably not something I'll ever get to fixing. I figured I could at least share what I know to help the next person.
I'm going to close this. As far as I can tell this isn't possible to address, and it's not actually an error.

We need to open the socket, then reopen it with `fromfd` so that it's reused across reloads. One of the two warnings can be addressed by calling `s.detach()` after `prepare_socket()`, but for some reason the other warning points to the `socket.fromfd` line and that one can't be detached, it's the object used by the server.

Ctrl-C kills the process, so the server may not have time to close its socket. For some reason, this _doesn't_ happen with the base `http.server`, even though it doesn't do anything in particular to ensure the socket is closed. Maybe it has something to do with using `socket.fromfd`. If anyone wants to look into this, identifying what's different there is probably where I'd start.

Resolving patch:
diff --git a/src/werkzeug/serving.py b/src/werkzeug/serving.py
--- a/src/werkzeug/serving.py
+++ b/src/werkzeug/serving.py
@@ -657,6 +657,7 @@ class BaseWSGIServer(HTTPServer):
     multithread = False
     multiprocess = False
     request_queue_size = LISTEN_QUEUE
+    allow_reuse_address = True
 
     def __init__(
         self,
@@ -708,10 +709,36 @@ def __init__(
             try:
                 self.server_bind()
                 self.server_activate()
+            except OSError as e:
+                # Catch connection issues and show them without the traceback. Show
+                # extra instructions for address not found, and for macOS.
+                self.server_close()
+                print(e.strerror, file=sys.stderr)
+
+                if e.errno == errno.EADDRINUSE:
+                    print(
+                        f"Port {port} is in use by another program. Either identify and"
+                        " stop that program, or start the server with a different"
+                        " port.",
+                        file=sys.stderr,
+                    )
+
+                    if sys.platform == "darwin" and port == 5000:
+                        print(
+                            "On macOS, try disabling the 'AirPlay Receiver' service"
+                            " from System Preferences -> Sharing.",
+                            file=sys.stderr,
+                        )
+
+                sys.exit(1)
             except BaseException:
                 self.server_close()
                 raise
         else:
+            # TCPServer automatically opens a socket even if bind_and_activate is False.
+            # Close it to silence a ResourceWarning.
+            self.server_close()
+
             # Use the passed in socket directly.
             self.socket = socket.fromfd(fd, address_family, socket.SOCK_STREAM)
             self.server_address = self.socket.getsockname()
@@ -877,60 +904,6 @@ def is_running_from_reloader() -> bool:
     return os.environ.get("WERKZEUG_RUN_MAIN") == "true"
 
 
-def prepare_socket(hostname: str, port: int) -> socket.socket:
-    """Prepare a socket for use by the WSGI server and reloader.
-
-    The socket is marked inheritable so that it can be kept across
-    reloads instead of breaking connections.
-
-    Catch errors during bind and show simpler error messages. For
-    "address already in use", show instructions for resolving the issue,
-    with special instructions for macOS.
-
-    This is called from :func:`run_simple`, but can be used separately
-    to control server creation with :func:`make_server`.
-    """
-    address_family = select_address_family(hostname, port)
-    server_address = get_sockaddr(hostname, port, address_family)
-    s = socket.socket(address_family, socket.SOCK_STREAM)
-    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-    s.set_inheritable(True)
-
-    # Remove the socket file if it already exists.
-    if address_family == af_unix:
-        server_address = t.cast(str, server_address)
-
-        if os.path.exists(server_address):
-            os.unlink(server_address)
-
-    # Catch connection issues and show them without the traceback. Show
-    # extra instructions for address not found, and for macOS.
-    try:
-        s.bind(server_address)
-    except OSError as e:
-        print(e.strerror, file=sys.stderr)
-
-        if e.errno == errno.EADDRINUSE:
-            print(
-                f"Port {port} is in use by another program. Either"
-                " identify and stop that program, or start the"
-                " server with a different port.",
-                file=sys.stderr,
-            )
-
-            if sys.platform == "darwin" and port == 5000:
-                print(
-                    "On macOS, try disabling the 'AirPlay Receiver'"
-                    " service from System Preferences -> Sharing.",
-                    file=sys.stderr,
-                )
-
-        sys.exit(1)
-
-    s.listen(LISTEN_QUEUE)
-    return s
-
-
 def run_simple(
     hostname: str,
     port: int,
@@ -1057,12 +1030,7 @@ def run_simple(
         application = DebuggedApplication(application, evalex=use_evalex)
 
     if not is_running_from_reloader():
-        s = prepare_socket(hostname, port)
-        fd = s.fileno()
-        # Silence a ResourceWarning about an unclosed socket. This object is no longer
-        # used, the server will create another with fromfd.
-        s.detach()
-        os.environ["WERKZEUG_SERVER_FD"] = str(fd)
+        fd = None
     else:
         fd = int(os.environ["WERKZEUG_SERVER_FD"])
 
@@ -1077,6 +1045,8 @@ def run_simple(
         ssl_context,
         fd=fd,
     )
+    srv.socket.set_inheritable(True)
+    os.environ["WERKZEUG_SERVER_FD"] = str(srv.fileno())
 
     if not is_running_from_reloader():
         srv.log_startup()
@@ -1085,12 +1055,15 @@ def run_simple(
     if use_reloader:
         from ._reloader import run_with_reloader
 
-        run_with_reloader(
-            srv.serve_forever,
-            extra_files=extra_files,
-            exclude_patterns=exclude_patterns,
-            interval=reloader_interval,
-            reloader_type=reloader_type,
-        )
+        try:
+            run_with_reloader(
+                srv.serve_forever,
+                extra_files=extra_files,
+                exclude_patterns=exclude_patterns,
+                interval=reloader_interval,
+                reloader_type=reloader_type,
+            )
+        finally:
+            srv.server_close()
     else:
         srv.serve_forever()


Test patch: