Repository: wagtail/wagtail

Issue:
Error serving documents with Django Cache Middleware enabled
I get an error each time I serve an uploaded document (pdf file) and my Django cache middleware is enabled. I'm using Redis as a cache backend, probably happens with any backend but I'm not sure. Some of the details are:

```
Django Version: 1.7.1
Exception Type: TypeError
Exception Value:    
can't pickle instancemethod objects
Exception Location: /var/www/bcito/venvs/test.com/lib/python2.7/copy_reg.py in _reduce_ex, line 70
Python Executable:  /usr/bin/uwsgi-core
Python Version: 2.7.6
```

I've found a workaround editing the way I include the wagtaildocs urls.py in my app. I make use of the _never_cache_ decorator. Instead of having:

```
urlpatterns = patterns(
    ...
    url(r'^documents/', include(wagtaildocs_urls)),
    ...
```

Have this:

```
...
from wagtail.wagtaildocs.views import serve
...
urlpatterns = patterns(
    ...
    url(r'^documents/(\d+)/(.*)$', never_cache(serve.serve), name='wagtaildocs_serve'),
    ...

```

Cheers,
Jordi



Discussion:
Thank you Jordi,

@kaedroho are you happy with the proposed solution? 
Does this relate to what you are thinking on #911? 

@jordij Can you paste the full stack trace, please?

TypeError at /documents/51/testar2013spreads.pdf
can't pickle instancemethod objects
Request Method: GET
Request URL:    http://test.co.nz/documents/xxx.pdf
Django Version: 1.7.1
Exception Type: TypeError
Exception Value:  
can't pickle instancemethod objects
Exception Location: /var/www/test/venvs/test.org.nz/lib/python2.7/copy_reg.py in _reduce_ex, line 70
Python Executable:  /usr/bin/uwsgi-core
Python Version: 2.7.6
Python Path:  
['.',
 '',
 '/var/www/test/venvs/test.org.nz/src/wagtailembedder',
 '/var/www/test/venvs/test.org.nz/src/wagtailgmaps',
 '/var/www/test/venvs/test.org.nz/lib/python2.7',
 '/var/www/test/venvs/test.org.nz/lib/python2.7/plat-x86_64-linux-gnu',
 '/var/www/test/venvs/test.org.nz/lib/python2.7/lib-tk',
 '/var/www/test/venvs/test.org.nz/lib/python2.7/lib-old',
 '/var/www/test/venvs/test.org.nz/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/var/www/test/venvs/test.org.nz/local/lib/python2.7/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/var/www/test/test.org.nz/test',
 '/var/www/test/test.org.nz/test']
Server time:    Thu, 12 Feb 2015 17:31:58 +1300

Traceback Switch back to interactive view

Environment:

Request Method: GET
Request URL: http://test.co.nz/documents/51/testar2013spreads.pdf

Django Version: 1.7.1
Python Version: 2.7.6
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'compressor',
 'import_export',
 'taggit',
 'modelcluster',
 'djcelery',
 'core',
 'lookingforwork',
 'floppyforms',
 'widget_tweaks',
 'wagtail.contrib.wagtailsitemaps',
 'wagtail.contrib.wagtailroutablepage',
 'wagtail.wagtailcore',
 'wagtail.wagtailadmin',
 'wagtail.wagtaildocs',
 'wagtail.wagtailsnippets',
 'wagtail.wagtailusers',
 'wagtail.wagtailsites',
 'wagtail.wagtailimages',
 'wagtail.wagtailembeds',
 'wagtail.wagtailsearch',
 'wagtail.wagtailredirects',
 'wagtail.wagtailforms',
 'wagtailembedder',
 'utils',
 'debug_toolbar')
Installed Middleware:
(u'debug_toolbar.middleware.DebugToolbarMiddleware',
 'django.middleware.common.BrokenLinkEmailsMiddleware',
 'django.middleware.cache.UpdateCacheMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.cache.FetchFromCacheMiddleware',
 'wagtail.wagtailcore.middleware.SiteMiddleware',
 'wagtail.wagtailredirects.middleware.RedirectMiddleware')

Traceback:
File "/var/www/test/venvs/test.org.nz/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
1.                 response = middleware_method(request, response)
   File "/var/www/test/venvs/test.org.nz/local/lib/python2.7/site-packages/django/middleware/cache.py" in process_response
2.                 self.cache.set(cache_key, response, timeout)
   File "/var/www/test/venvs/test.org.nz/local/lib/python2.7/site-packages/redis_cache/cache.py" in set
3.             result = self._set(key, pickle.dumps(value), timeout, client, _add_only)
   File "/var/www/test/venvs/test.org.nz/lib/python2.7/copy_reg.py" in _reduce_ex
4.             raise TypeError, "can't pickle %s objects" % base.**name**

Exception Type: TypeError at /documents/xxx.pdf
Exception Value: can't pickle instancemethod objects

OK - it looks like the problem is in the `wagtail.wagtaildocs.views.serve` view, which uses `FileWrapper` to generate a streamed response that avoids loading the file into memory. (It turns out that FileWrapper is not a documented Django feature - it's provided by the `wsgiref` package, and the relevant alias has been [quietly dropped from Django as of several days ago](https://github.com/django/django/commit/bbe28496d32f76ca161f5c33787d6ad62267fcc6), so it's a good thing we're spotting this now! I suspect I borrowed the code from [this StackOverflow question](http://stackoverflow.com/questions/8600843/serving-large-files-with-high-loads-in-django).)

The cache middleware can't cache streamed responses, and usually it will identify and ignore those because they're using `StreamingHttpResponse` instead of `HttpResponse`... however, our code is currently using a plain HttpResponse. (Sure enough, that StackOverflow question [was also using HttpResponse until someone fixed it a few days ago](http://stackoverflow.com/posts/8601118/revisions)...)

So, I think what we need to do here is:
- ~~check that we can rely on the `wsgiref` package always being available. (I think it's part of the Python standard library, but would be good to make sure)~~ ([it is](https://docs.python.org/2/library/wsgiref.html))
- change the import to `from wsgiref.utils import FileWrapper`
- update the view to return a StreamingHttpResponse


Resolving patch:
diff --git a/wagtail/wagtaildocs/views/serve.py b/wagtail/wagtaildocs/views/serve.py
--- a/wagtail/wagtaildocs/views/serve.py
+++ b/wagtail/wagtaildocs/views/serve.py
@@ -1,6 +1,6 @@
 from django.shortcuts import get_object_or_404
-from django.core.servers.basehttp import FileWrapper
-from django.http import HttpResponse
+from wsgiref.util import FileWrapper
+from django.http import StreamingHttpResponse
 
 from wagtail.wagtaildocs.models import Document, document_served
 
@@ -8,7 +8,7 @@
 def serve(request, document_id, document_filename):
     doc = get_object_or_404(Document, id=document_id)
     wrapper = FileWrapper(doc.file)
-    response = HttpResponse(wrapper, content_type='application/octet-stream')
+    response = StreamingHttpResponse(wrapper, content_type='application/octet-stream')
 
     # TODO: strip out weird characters like semicolons from the filename
     # (there doesn't seem to be an official way of escaping them)


Test patch: