Repository: Pylons/pyramid

Issue:
Pyramid makes it pretty hard to stream responses
We have a decently sized Pyramid application running. We have a certain view that basically proxies content from Amazon S3 to the requesting user after doing security checks and other stuff.  I noticed that the developer who originally wrote this code was pretty naive and simply downloaded the content and then pushed it out by basically doing `return HTTPOk(body=storedfile.content, ...)`. Since boto (the Python S3 client) allows for streaming content since you can iterate over over that `Key` object (which represents a single file stored in S3) I figured that it might be possible to stream this data thought Pyramid to the client. After some research I came to the conclusion that it indeed should be possible by simply taking this iterator and passing it though the WSGI chain as `app_iter`.

So I wrote simply code like `return HTTPOk(content_type=..., content_length=..., app_iter=iterableKeyInstance)`. To my surprise it didn't work. No matter what I did to create the response, the browser would never receive the response until `boto` finished downloading the whole file. In fact, that browser would not even receive the HTTP headers which also meant the browser had no way of telling how big the response is gonna be. It just sat their and said "waiting".
# Problem

After some digging I finally came to the root cause of this:
1. `httpexceptions.HTTPException` (the base class for all `Http...` response classes) has a [`prepare()`](https://github.com/Pylons/pyramid/blob/master/pyramid/httpexceptions.py#L246) method. As far as I understand the code it is basically there to generate a HTML in case there is no body. It does this by doing a very pythonic check `if not self.body...`.
2. `httpexceptions.HTTPException` derives from `webob.Response` and the `body` attribute is actually a `@property`. 
3. `webob.Response#body` tries to ensure the presence and validity of the `Content-Length` header. To do this it actually [evaluates the underlying `app_iter` by doing `b''.join(app_iter)`](https://github.com/Pylons/webob/blob/master/webob/response.py#L355) since it wants to know the actual length of the to-be body. There is no way to avoid it...as soon as you access `body` it will iterate the `app_iter`.

Combine all of this and you simply can't generate a streaming response from Pyramid.
# Workaround

I managed to somewhat work around this issue we returning a plain `webob.Response` from my view:

```
return webob.Response(app_iter=iterableKeyInstance, headerlist=[
        ('Content-Type', content_type),
        ('Content-Length', content.size)
    ]
})
```

However, this is not really nice because now you have to write a long comment on why you are using a plain `Response` here and you also have to build your own headerlist.
# Possible solutions
1. Rewrite the check in `httpexceptions.HTTPException` to not access `body`. 
2. Introduce a new class that is specifically designed for streamed response (in the style of `response. FileResponse` that doesn't derive from `httpexceptions.HTTPException`.
3. Don't change any code but document the problem in the docstring for `response.Response` and explain to use a plain `webob.Response`.
4. Decide that this is not a Pyramid issue since `webob` doing insane things and close the issue.

Solution 1 would probably the solution with the most work and it also might break certain code that specifically expects this behaviour. However, I would also say that this is the nicest solution from an engineering standpoint. Solution 2 would probably the solution I would vote for since it has no way of breaking any existing code and also is dead simple to understand.
Solution 3 is the way to go if you argue that there is really no real need for additional code since the use case is pretty rare. 
Solution 4 would make me slightly mad since. While I believe that the behaviour of `webob` is a bit strange I can't see _any_ way of fixing it there since I'm almost sure there is code somewhere that depends on `webob.Response#body` reading the underlying `app_iter` since `webob` is widely used.

**I'm willing to do the actual work** but since I never contributed to Pyramid before I would rather have a discussion and a decision from some core devls about the solution before diving into it.



Discussion:
My vote is solution 1. I think that check is incorrect and should guard against someone like you setting `app_iter` prior to checking the full `body` object as obviously checking the `body` unrolls the entire `app_iter`. I think adding that check in `prepare` should be enough. What do you think?

For the record, I think you're right to use webob.Response() instead of any HTTPException in this particular case (personally I would have never thought to use any httpexception class for this), but also we should fix httpexceptions as @mmerickel described too.

The check could be modified to be `if app_iter is None:`

If the response has a body, the `app_iter` will not be None. This should satisfy the check so that it doesn't attempt to read the full body, and will correctly allow it to return HTML as necessary.

@bertjwregeer Your last sentence was truncated.

@tseaver My last sentence was a misfire, edited to remove it.

Also, checking for just `app_iter` doesn't work, because WebOb will happily hand an empty `app_iter` back, which is truthy... I am not sure what the best way to go about this is :-/ I'd have to dig into it a little more. Maybe add a `has_body()` function or something that does the checks without reading the whole thing.

```
    def has_body(self):
        app_iter = self._app_iter

        if isinstance(app_iter, list) and len(app_iter) == 1:
            if app_iter[0] != b'':
                return True
            else:
                return False

        if app_iter is None:
            return False

        return True
```

Adding this to `webob.Response` and using it to with `resp.has_body()` in the Exception stuff allows tests to pass in WebOb.

The default is to return `True`. This basically takes everything from `body__get` except for the reading the entire `app_iter`...

@martinth or anyone else - if you are interested in working on this it should be possible to:
1) Add the method to webob that @bertjwregeer proposed
2) Fix the HTTPException class to use this method.
3) Either pin the next release of Pyramid to the next release of WebOb or (preferable) add a shim to `pyramid.request.Request` which can use the current private API if the new API is not found.

That sounds like a nice plan. I will be doing it this way.

@martinth Thanks for the PR on WebOb; I've tagged it to be added to 1.7.0 for now.

Once WebOb is released this will be an easy fix, added a "blocked-external-lib" label.

Webob 1.7rc1 is out. Who wants to make a PR for this?

Taking.


Resolving patch:
diff --git a/pyramid/httpexceptions.py b/pyramid/httpexceptions.py
--- a/pyramid/httpexceptions.py
+++ b/pyramid/httpexceptions.py
@@ -246,7 +246,7 @@ def _json_formatter(self, status, body, title, environ):
                 'title': self.title}
 
     def prepare(self, environ):
-        if not self.body and not self.empty_body:
+        if not self.has_body and not self.empty_body:
             html_comment = ''
             comment = self.comment or ''
             accept_value = environ.get('HTTP_ACCEPT', '')
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -41,7 +41,7 @@
 
 install_requires = [
     'setuptools',
-    'WebOb >= 1.3.1', # request.domain and CookieProfile
+    'WebOb >= 1.7.0rc2', # Response.has_body
     'repoze.lru >= 0.4', # py3 compat
     'zope.interface >= 3.8.0',  # has zope.interface.registry
     'zope.deprecation >= 3.5.0', # py3 compat


Test patch: