Repository: benoitc/gunicorn

Issue:
Hitting a RecursionError with gevent
I'm hitting the following `RecursionError` when a part of an application ([pywb](https://pypi.org/project/pywb/) delivering static files) tries to respond (the more complex functionalities work. :-)). i think that exception should be handled / propagated since i can't deduce where something went wrong in the app from this output.

```
Traceback (most recent call last):
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
    resp.write_file(respiter)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 392, in write_file
    if not self.sendfile(respiter):
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/http/wsgi.py", line 382, in sendfile
    sent += os.sendfile(sockno, fileno, offset + sent, count)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/usr/local/share/virtualenv/pywb/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 37, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  [Previous line repeated 923 more times]
RecursionError: maximum recursion depth exceeded
```
RecursionError with eventlet workers and sendfile
When using gunicorn with eventlet workers to serve static files with Flask, there is an infinite recursion error. Looking at `workers/geventlet.py`, it appears to be caused by patching `os.sendfile` to refer to `_eventlet_sendfile`, whereas `_eventlet_sendfile` just turns around and calls `os.sendfile()`:

```python
def _eventlet_sendfile(fdout, fdin, offset, nbytes):
    while True:
        try:
            return os.sendfile(fdout, fdin, offset, nbytes)
        except OSError as e:
            if e.args[0] == errno.EAGAIN:
                trampoline(fdout, write=True)
            else:
                raise
…
def patch_sendfile():
    setattr(os, "sendfile", _eventlet_sendfile)
```

A simple test case follows:

`requirements.txt`:
```
gunicorn[eventlet]==20.0
Flask==1.1
```

`sendfile_bug.py`:
```python
from flask import Flask
application = Flask(__name__)

```

`static/hello`:
```
hello
```

start command:
```shell
% gunicorn -b 0.0.0.0:5000 --log-level=debug -k eventlet sendfile_bug
```

request:
```shell
% curl http://localhost:5000/static/hello
```

Output from `gunicorn` process:
```
[2019-11-12 14:11:45 -0700] [62286] [INFO] Starting gunicorn 20.0.0
[2019-11-12 14:11:45 -0700] [62286] [DEBUG] Arbiter booted
[2019-11-12 14:11:45 -0700] [62286] [INFO] Listening at: http://0.0.0.0:5000 (62286)
[2019-11-12 14:11:45 -0700] [62286] [INFO] Using worker: eventlet
[2019-11-12 14:11:45 -0700] [62289] [INFO] Booting worker with pid: 62289
[2019-11-12 14:11:45 -0700] [62286] [DEBUG] 1 workers
[2019-11-12 14:11:49 -0700] [62289] [DEBUG] GET /static/hello
[2019-11-12 14:11:49 -0700] [62289] [ERROR] Error handling request
Traceback (most recent call last):
  File "venv/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
    resp.write_file(respiter)
  File "venv/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 397, in write_file
    if not self.sendfile(respiter):
  File "venv/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 387, in sendfile
    sent += os.sendfile(sockno, fileno, offset + sent, count)
  File "venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 32, in _eventlet_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 32, in _eventlet_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "venv/lib/python3.7/site-packages/gunicorn/workers/geventlet.py", line 32, in _eventlet_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  [Previous line repeated 968 more times]
RecursionError: maximum recursion depth exceeded
[2019-11-12 14:11:49 -0700] [62289] [DEBUG] Closing connection.
```


Discussion:
What versions of gunicorn and gevent are you using?
oh, sorry for missing that information. thanks for reminding me. it's `gunicorn 19.9.09` and `gevent 1.3.7`.
@funkyfuture looking at the traceback, I think you are using the master version of Gunicorn :)

The following line

```py
return os.sendfile(fdout, fdin, offset, nbytes)
```

should read

```py
return o_sendfile(fdout, fdin, offset, nbytes)
```

in Gunicorn 19.9. We've changed it to call `os.sendfile()` directly in https://github.com/benoitc/gunicorn/commit/e974f30517261b2bc95cfb2017a8688f367c8bf3#diff-24baff6269faf10cb820888842174a8bR37.

@jamadden do we still need to monkeypatch `os.sendfile()` manually? It looks like we could rely on Gevent: http://www.gevent.org/api/gevent._socket3.html#gevent._socket3.socket.sendfile
you're right, i'm relying on #1897. i just referred to what `pip` told me.
> do we still need to monkeypatch os.sendfile() manually? It looks like we could rely on Gevent:

gevent does provide `socket.sendfile` on all versions of Python 3. The version gevent provides is cooperative with the gevent loop, which `os.sendfile` is not. The purpose of your patch at this point is to execute `os.sendfile` against a non-blocking socket, using a loop, making it basically cooperative. That may or may not be more efficient than what gevent is doing; gevent's version may improve in the future as we incorporate more libuv support, or compile it with cython.

Unless there are fears or reports of substantial performance degradations, I would suggest that you can drop your monkey-patch and just use `socket.sendfile`.
@benoitc @javabrett bump here. If we switched to Python 3.5 as a minimum, we could use `socket.sendfile`.
Check #1559?
I am also seeing this after updating from Gunicorn 19.9.0 to 20.0.0 (with gevent 1.4.0, Django 2.2.6, Python 3.7.5, Ubuntu 19.10):

```
[2019-11-11 09:16:23 +0000] [11635] [ERROR] Error handling request
Traceback (most recent call last):
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 111, in handle_request
    resp.write_file(respiter)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 397, in write_file
    if not self.sendfile(respiter):
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/http/wsgi.py", line 387, in sendfile
    sent += os.sendfile(sockno, fileno, offset + sent, count)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  File "/data-hub-api/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 36, in _gevent_sendfile
    return os.sendfile(fdout, fdin, offset, nbytes)
  [Previous line repeated 936 more times]
RecursionError: maximum recursion depth exceeded
```

Is there any known solution?
Now that v20 has been released, this bug is causing more problems (e.g., #2164). 

gevent provides `socket.sendfile` on all versions of Python 3. One option would be to change `gunicorn.http.wsgi.Resoponse:sendfile` to use `socket.sendfile` instead of `os.sendfile` if `hasattr(sock, 'sendfile')`.  Another option would be to override `Response.sendfile` in the gevent worker. Of course the obvious option would be to go back to what gunicorn 19 did and capture the non-patched `os.sendfile` as a variable and call it instead of calling `os.sendfile` from the patch that replaces `os.sendfile`.
@jamadden what would you suggest? 
Python 3.4 has been end-of-life for sometime now. According to https://pypistats.org/packages/gunicorn, it makes up less than 1% of gunicorn downloads, less than 3.8 (those numbers are skewed, of course, but there may be value in them nonetheless). So rather than deal with the `hasattr` dance, I would just drop 3.4 and assume `socket.sendfile` exists 😄 

But that's not something that can happen in a 20.0.1 bugfix release, so the most minor change that solves the problem is probably the obvious one: capture `os.sendfile` before patching and use that capture in the patch.
That sounds good. Let's capture the old sendfile. Let's also bump the Python version for the next release.

Can we make a 20.x release branch, then, and start working on 21?
Thank you for the report! #1925 documents a very similar issue with gevent, but I'm not sure we realized it was also an issue with eventlet.

Resolving patch:
diff --git a/gunicorn/workers/geventlet.py b/gunicorn/workers/geventlet.py
--- a/gunicorn/workers/geventlet.py
+++ b/gunicorn/workers/geventlet.py
@@ -26,10 +26,10 @@
 from gunicorn.workers.base_async import AsyncWorker
 
 
-def _eventlet_sendfile(fdout, fdin, offset, nbytes):
+def _eventlet_sendfile(fdout, fdin, offset, nbytes, _os_sendfile=os.sendfile):
     while True:
         try:
-            return os.sendfile(fdout, fdin, offset, nbytes)
+            return _os_sendfile(fdout, fdin, offset, nbytes)
         except OSError as e:
             if e.args[0] == errno.EAGAIN:
                 trampoline(fdout, write=True)
diff --git a/gunicorn/workers/ggevent.py b/gunicorn/workers/ggevent.py
--- a/gunicorn/workers/ggevent.py
+++ b/gunicorn/workers/ggevent.py
@@ -30,10 +30,10 @@
 VERSION = "gevent/%s gunicorn/%s" % (gevent.__version__, gunicorn.__version__)
 
 
-def _gevent_sendfile(fdout, fdin, offset, nbytes):
+def _gevent_sendfile(fdout, fdin, offset, nbytes, _os_sendfile=os.sendfile):
     while True:
         try:
-            return os.sendfile(fdout, fdin, offset, nbytes)
+            return _os_sendfile(fdout, fdin, offset, nbytes)
         except OSError as e:
             if e.args[0] == errno.EAGAIN:
                 socket.wait_write(fdout)


Test patch: