Repository: benoitc/gunicorn

Issue:
OSError [Error 29] llegal seek
Hi

   I have a python application on hand ,based on falcon(https://github.com/falconry/falcon),  and actually , I used the application as a proxy to access another one.

  in the application,  it just using requests.request to access another url and feed back current response
  when I run the application with 

gunicorn "abc.wsgi:make_api()" --bind="127.0.0.1:8000" --timeout=600 -w 1

  and run the 'curl -i http://127.0.0.1:8000/', it will failed with 

  2015-05-24 03:45:10 [129721] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/sync.py", line 134, in handle_request
    resp.write_file(respiter)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/http/wsgi.py", line 341, in write_file
    fd_offset = os.lseek(fileno, 0, os.SEEK_CUR)
OSError: [Errno 29] Illegal seek

  any if I write a verify simple app and run with gunicorn , I can't reproduce it 

  so what do you think can cause such error?



Discussion:
Does your app send a file over the socket? If yes, is the user launching gunicorn is able to read it.

I don't think it send a file. and  with debugging , I saw that my app had already get the expected content as below.

actually, the following URL and content are my expected content through my app as a proxy, it return Json. 

norman@ubuntu:~$ curl -i http://192.168.94.129:5000/
HTTP/1.1 300 Multiple Choices
Date: Sun, 24 May 2015 12:42:43 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
Content-Length: 597
Content-Type: application/json

{"versions": {"values": [{"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://192.168.94.129:5000/v3/", "rel": "self"}]}, {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.94.129:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

and in my app, the main code as below:

```
class OpenstackStream(object):
    def __init__(self, stream, size=None):
        self.stream = stream
        self.size = size

    def __len__(self):
        return self.size

    def read(self, size=None):
        return self.stream.read(size=size)

    def __iter__(self):
        return self.stream.__iter__()

    def __next__(self):
        return self.stream.__next__()
    next = __next__


class OpenStackResponder(object):
.....................
    def _standard_responder(self, req, resp, **_):
        data = None
        if (req.method == 'POST' or req.method == 'PUT'):
            if req.content_length:
                data = OpenstackStream(req.stream, size=req.content_length)

        ...............
        os_resp = requests.request(req.method,
                                   endpoint,
                                   data=data,
                                   headers=req.headers,
                                   stream=True)

        resp.status = os_resp.status_code
        content_type = os_resp.headers.pop(
            'Content-Type', 'application/json').split(';', 1)[0]

        # Hack for some reason
        if content_type == 'text/html':
            content_type = 'text/plain; charset=UTF-8'
        resp.content_type = content_type
        resp.stream_len = os_resp.headers.pop('Content-Length', 0)
        resp.set_headers(os_resp.headers)
        resp.stream = os_resp.raw

    on_get = _standard_responder
```

The error happen because your application is passing the `wsgi.file_wrapper`setting to the environ. The value given should be a [file-like object](https://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling).

Normally Gunicorn should detect if the file is a real file  or not:
https://github.com/benoitc/gunicorn/blob/master/gunicorn/util.py#L511-L521

Do you have any example application I could check? Which library does it use to handle WSGI?

I did not have an example application for you yet.

The library is falcon (https://github.com/falconry/falcon) if I understand right.

after grep the code,  it seem to be related with the code I paste above :

```
  resp.stream=os_resp.raw
```

  since in the falcon response.py  

```
        stream: Either a file-like object with a `read()` method that takes
            an optional size argument and returns a block of bytes, or an
            iterable object, representing response content, and yielding
            blocks as byte strings. Falcon will use *wsgi.file_wrapper*, if
            provided by the WSGI server, in order to efficiently serve
            file-like objects.
```

and  I sat that it (falcon) also just refer to env[wsgi.file_wrapper]

so I am wondering that who set the env value?

Gunicorn provides the file wrapper but it expects a file-like object. It may be something missing in the object provided by falcon or a bug in Gunicorn. I am currently investing it.

thank a lot

Turn out that this issue is caused by request. The raw object provided [implements](https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/response.py#L68) the `tell`and `fileno` functions which let gunicorn to think this is a real file object.

I am not sure where we should fix it right now. Also I wonder why this object is providing the `tell` function since you can' go back in the past. cc @kennethreitz . One temporary solution is to disable the sendfile option. For the fix I am still thinking of it. Maybe we could try an except on the first seek? 

We seek on the `fileno()` response. Not sure how that could cause any problem since requests will throw an error when we try to call that if it can't return us an actual fd.

If that wasn't clear, I'm not sure how we could get to the line where this errors unless we have a valid `fd`.

@tilgovi Not sure I follow but the raw object is quite like a wrapper over the socket. You can't seek over a socket which create the error above imo.

and this error happen because the sendfile API is used if we expect a file there.

Right. A socket has a file descriptor but isn't seekable. Of course.

I was just trying to point out that the lines of code we have in util are:

``` python
def is_fileobject(obj):
    if not hasattr(obj, "tell") or not hasattr(obj, "fileno"):
        return False

    # check BytesIO case and maybe others
    try:
        obj.fileno()
    except (IOError, io.UnsupportedOperation):
        return False

    return True
```

Requests is fine. There's no reason why we can't `tell` where we are in a stream that isn't seekable. It throws an exception if there's no `fileno`. The probably is definitely that we assume a fileno means we can seek. Maybe instead of a try/catch around the `lseek` in `http` we should be doing it in `is_fileobject`.

Or, it also occurs to me to ask why we call `lseek` at all. The first one seems wholly superfluous because we can get the same information from `tell` (and we do, on the very next line). Then we `lseek` at the end. Is there some spec that says we have to leave the file as we found it? In all cases where we use this function we finish the request right afterward. Why rewind the file?

I need to check for linux but te thing is that `sendfile(1)` can only be used to stream a file to a socket. This why we should test if a file is given or not at first hand. 

Anyway I  agree there is nothing preventing a file-object to expose the `tell`and `fileno` methods/properties. So it let us to try/catch on the first `seek` I guess except if there is an api that would allows us to test if we're using a file. At least that way we make sure we have file behind so we can use the `sendfile`api. 

About removing the seeks, that sounds like a good idea, just at the moment I don't remember the reason for them, i need to look at my notes if I've still have them. I propose in between to do the following:
1. Patch the `is_fileobject` function to make the first seek
2. Open a ticket to remove the seeks that revert to the last position if they are useless

Thoughts?

Depends on the schedule. If we can remove the seeks and it won't take too much time to figure that out, that seems better to me. If we patch it with a try/catch and remove those later it's not particularly helpful for @wpf710 unless we also make a release. If we don't, @wpf710 can do that patch themselves while we figure out whether these seeks are necessary.

So, if it matters that much, mark it as a blocker and on a milestone for the next release. If we hit a deadline and want to release we can try/catch. If, before that, someone investigates and can explain why we do or don't need them, then we'll know.

oh i think the try catch is here to stay. Don't get me wrong :) this is
only about the second part. i would do that fix now.
On Thu 28 May 2015 at 22:11 Randall Leeds notifications@github.com wrote:

> So, if it matters that much, mark it as a blocker and on a milestone for
> the next release. If we hit a deadline and want to release we can
> try/catch. If, before that, someone investigates and can explain why we do
> or don't need them, then we'll know.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/benoitc/gunicorn/issues/1038#issuecomment-106569112.

Don't get me wrong. Defensive programming is great. But in this case we are talking about a try/catch around a line that might not need to exist. So if that line goes away, the try/catch certainly will ;)

So, yeah, throw in the try/catch. I'll take a look and see whether it's necessary at all.

Opened #1040 and assigned to me to look into it.

@tilgovi I saw your point :) Thanks for taking care about it. I will provide a patch later in the day that implement the try/catch solution

@benoitc  Hi what's the status for it? 
Thanks


Resolving patch:
diff --git a/gunicorn/http/wsgi.py b/gunicorn/http/wsgi.py
--- a/gunicorn/http/wsgi.py
+++ b/gunicorn/http/wsgi.py
@@ -12,7 +12,6 @@
 from gunicorn._compat import unquote_to_wsgi_str
 from gunicorn.six import string_types, binary_type, reraise
 from gunicorn import SERVER_SOFTWARE
-import gunicorn.six as six
 import gunicorn.util as util
 
 try:
@@ -24,6 +23,10 @@
     except ImportError:
         sendfile = None
 
+# Send files in at most 1GB blocks as some operating systems can have problems
+# with sending files in blocks over 2GB.
+BLKSIZE = 0x3FFFFFFF
+
 NORMALIZE_SPACE = re.compile(r'(?:\r\n)?[ \t]+')
 
 log = logging.getLogger(__name__)
@@ -344,77 +347,51 @@ def write(self, arg):
         util.write(self.sock, arg, self.chunked)
 
     def can_sendfile(self):
-        return (self.cfg.sendfile and (sendfile is not None))
-
-    def sendfile_all(self, fileno, sockno, offset, nbytes):
-        # Send file in at most 1GB blocks as some operating
-        # systems can have problems with sending files in blocks
-        # over 2GB.
-
-        BLKSIZE = 0x3FFFFFFF
-
-        if nbytes > BLKSIZE:
-            for m in range(0, nbytes, BLKSIZE):
-                self.sendfile_all(fileno, sockno, offset, min(nbytes, BLKSIZE))
-                offset += BLKSIZE
-                nbytes -= BLKSIZE
-        else:
-            sent = 0
-            sent += sendfile(sockno, fileno, offset + sent, nbytes - sent)
-            while sent != nbytes:
-                sent += sendfile(sockno, fileno, offset + sent, nbytes - sent)
+        return self.cfg.sendfile and sendfile is not None
 
-    def sendfile_use_send(self, fileno, fo_offset, nbytes):
-
-        # send file in blocks of 8182 bytes
-        BLKSIZE = 8192
-
-        sent = 0
-        while sent != nbytes:
-            data = os.read(fileno, BLKSIZE)
-            if not data:
-                break
+    def sendfile(self, respiter):
+        if self.cfg.is_ssl or not self.can_sendfile():
+            return False
 
-            sent += len(data)
-            if sent > nbytes:
-                data = data[:nbytes - sent]
+        try:
+            fileno = respiter.filelike.fileno()
+            offset = os.lseek(fileno, 0, os.SEEK_CUR)
+            if self.response_length is None:
+                filesize = os.fstat(fileno).st_size
 
-            util.write(self.sock, data, self.chunked)
+                # The file may be special and sendfile will fail.
+                # It may also be zero-length, but that is okay.
+                if filesize == 0:
+                    return False
 
-    def write_file(self, respiter):
-        if self.can_sendfile() and util.is_fileobject(respiter.filelike):
-            # sometimes the fileno isn't a callable
-            if six.callable(respiter.filelike.fileno):
-                fileno = respiter.filelike.fileno()
+                nbytes = filesize - offset
             else:
-                fileno = respiter.filelike.fileno
+                nbytes = self.response_length
+        except (OSError, io.UnsupportedOperation):
+            return False
 
-            fd_offset = os.lseek(fileno, 0, os.SEEK_CUR)
-            fo_offset = respiter.filelike.tell()
-            nbytes = max(os.fstat(fileno).st_size - fo_offset, 0)
+        self.send_headers()
 
-            if self.response_length:
-                nbytes = min(nbytes, self.response_length)
+        if self.is_chunked():
+            chunk_size = "%X\r\n" % nbytes
+            self.sock.sendall(chunk_size.encode('utf-8'))
 
-            if nbytes == 0:
-                return
+        sockno = self.sock.fileno()
+        sent = 0
 
-            self.send_headers()
+        for m in range(0, nbytes, BLKSIZE):
+            count = min(nbytes - sent, BLKSIZE)
+            sent += sendfile(sockno, fileno, offset + sent, count)
 
-            if self.cfg.is_ssl:
-                self.sendfile_use_send(fileno, fo_offset, nbytes)
-            else:
-                if self.is_chunked():
-                    chunk_size = "%X\r\n" % nbytes
-                    self.sock.sendall(chunk_size.encode('utf-8'))
+        if self.is_chunked():
+            self.sock.sendall(b"\r\n")
 
-                self.sendfile_all(fileno, self.sock.fileno(), fo_offset, nbytes)
+        os.lseek(fileno, offset, os.SEEK_SET)
 
-                if self.is_chunked():
-                    self.sock.sendall(b"\r\n")
+        return True
 
-            os.lseek(fileno, fd_offset, os.SEEK_SET)
-        else:
+    def write_file(self, respiter):
+        if not self.sendfile(respiter):
             for item in respiter:
                 self.write(item)
 
diff --git a/gunicorn/util.py b/gunicorn/util.py
--- a/gunicorn/util.py
+++ b/gunicorn/util.py
@@ -7,7 +7,6 @@
 
 import email.utils
 import fcntl
-import io
 import os
 import pkg_resources
 import random
@@ -517,19 +516,6 @@ def to_latin1(value):
     return value.encode("latin-1")
 
 
-def is_fileobject(obj):
-    if not hasattr(obj, "tell") or not hasattr(obj, "fileno"):
-        return False
-
-    # check BytesIO case and maybe others
-    try:
-        obj.fileno()
-    except (IOError, io.UnsupportedOperation):
-        return False
-
-    return True
-
-
 def warn(msg):
     print("!!!", file=sys.stderr)
 


Test patch: