Repository: benoitc/gunicorn

Issue:
Access Log line missing when client times out
I noticed that when running gunicorn with access log enabled, if the client times out and disconnects gunicorn doesn't log anything to the access or error logs. This seems like a bug, the request code typically still runs normally (e.g. there may be new data in the database from this request) so it seems like there should be a record of it in the access log. Other web servers (nginx is the one I tested it on just now) still log to the access log when the client times out, nginx uses the status code 499 for this case.

A simple way to demonstrate -  in a request route in a wsgi app like flask, put in `time.sleep(5)`.  Then hit the route, say it's running locally on port 8000, with `curl --max-time 2 localhost:8000/`. The access log won't show any record of the request.



Discussion:
We should look into this. From the description provided by @dcosson curl is timing out, not the gunicorn worker.

Right. It would also be nice if a worker timeout could log all the access log fields instead of just the process id when it gets restarted. But I was talking here about when the client times out or otherwise disconnects, not the gunicorn worker.

Hrmm not sure to see the issue there. Isn't the access log about **logging processed requests** by the server? 

If the client timeout then the requests isn't fully processed and the error is then logged in the error logs (depending on the level). Are there any server that log requests that client timeout in the access log?

Nginx.

499, as described. It's helpful to log these because if clients are timing out it might be because the requests are taking too long.

And to address the only logging actually processed requests point: 

If a client times out, the request is still fully processed, the client just didn't get any data back. Even if it was a GET request, it still would be nice to know as it can be very helpful for triaging bugs, if e.g. a user reported a page was blank and you see a 499 status in the request log for the request to that page, you found the problem. If there's no sign of it in the logs, it could just as easily be a bug with your client code not making the request at all.

Similarly if a worker times out, the request was processed arbitrarily far so it would be nice to see all the info about it, with a status code indicating that the worker was killed after the timeout was reached.

@dcosson do you think you could make a PR for this?

In all of the modules in `gunicorn/workers` you'll find calls to `log.access` toward the end of a `try`. I think you would need to move that to a finally clause and set the status code in an `except` clause that catches the client disconnect.

Cool, thanks for the pointers. I'll give it a shot 

Thanks! Let us know if you run into problems. I'd be happy to help.

@tilgovi fair enough then . And for the record: 

> 499 / CLIENT CLOSED REQUEST
> “An Nginx HTTP server extension. This code is introduced to log the case when the connection is closed by client while HTTP server is processing its request, making server unable to send the HTTP header back.“ — wikipedia

http://httpstatus.es/499 

So it should be done on all requests that fail to return to the client because its socket is down. 

@dcosson thanks!

Huh. I wonder if that means this should really be changed in `gunicorn.http.wsgi:send_headers`. :-/

@tilgovi you mean "only" there ?

@dcosson about the other case. If a worker timeout there is no way to know which request was handled at that time since these information is not shared with other workers or the arbiter. In that case you should rather log the events you need inside the application imo.

@benoitc that makes sense, thanks for clarifying

@tilgovi can you expand what was your idea ? Happy to implement it for the next release. Eventually we could also return a 499.

I mean that in `send_headers` we can catch and error and set the status to 499, maybe? I was trying to decide if it makes sense to keep the response header once it's been sent to the client, but if sending the header fails then set it to 499.

Or maybe it makes sense to 499 no matter what if the response doesn't finish cleanly?

Hr if the client timeout, we should probably just close indeed.... There is no point to send a reason if no-one will receive it. So yes let just catch it and end the request cleanly.


Resolving patch:
diff --git a/gunicorn/workers/base_async.py b/gunicorn/workers/base_async.py
--- a/gunicorn/workers/base_async.py
+++ b/gunicorn/workers/base_async.py
@@ -115,9 +115,9 @@ def handle_request(self, listener_name, req, sock, addr):
                     for item in respiter:
                         resp.write(item)
                 resp.close()
+            finally:
                 request_time = datetime.now() - request_start
                 self.log.access(resp, req, environ, request_time)
-            finally:
                 if hasattr(respiter, "close"):
                     respiter.close()
             if resp.should_close():
diff --git a/gunicorn/workers/gthread.py b/gunicorn/workers/gthread.py
--- a/gunicorn/workers/gthread.py
+++ b/gunicorn/workers/gthread.py
@@ -340,9 +340,9 @@ def handle_request(self, req, conn):
                         resp.write(item)
 
                 resp.close()
+            finally:
                 request_time = datetime.now() - request_start
                 self.log.access(resp, req, environ, request_time)
-            finally:
                 if hasattr(respiter, "close"):
                     respiter.close()
 
diff --git a/gunicorn/workers/sync.py b/gunicorn/workers/sync.py
--- a/gunicorn/workers/sync.py
+++ b/gunicorn/workers/sync.py
@@ -183,9 +183,9 @@ def handle_request(self, listener, req, client, addr):
                     for item in respiter:
                         resp.write(item)
                 resp.close()
+            finally:
                 request_time = datetime.now() - request_start
                 self.log.access(resp, req, environ, request_time)
-            finally:
                 if hasattr(respiter, "close"):
                     respiter.close()
         except EnvironmentError:


Test patch: