Repository: encode/django-rest-framework

Issue:
Caching in DRF?
Awesome work on v2, I'm really liking what I've seen so far. Big thumbs up!

One problem arose when I was migrating from tastypie to DRF. I use Django's cache framework to cache everything on the site I'm developing (it's very read-heavy), but when I try to access the API I get `TypeError: can't pickle _Input objects`. It works just fine with caching disabled.

It should be noted that I use nginx + uwsgi instead of the internal dev server.

Here's the traceback:

```
Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
    response = self.get_response(request)
  File ".../lib/python2.7/site-packages/django/core/handlers/base.py", line 192, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File ".../lib/python2.7/site-packages/django/core/handlers/base.py", line 221, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File ".../lib/python2.7/site-packages/django/core/handlers/base.py", line 188, in get_response
    response = middleware_method(request, response)
  File ".../lib/python2.7/site-packages/django/middleware/cache.py", line 112, in process_response
    lambda r: self.cache.set(cache_key, r, timeout)
  File ".../lib/python2.7/site-packages/django/template/response.py", line 91, in add_post_render_callback
    callback(self)
  File ".../lib/python2.7/site-packages/django/middleware/cache.py", line 112, in <lambda>
    lambda r: self.cache.set(cache_key, r, timeout)
  File ".../lib/python2.7/site-packages/redis_cache/cache.py", line 218, in set
    result = self._set(key, pickle.dumps(value), int(timeout), client, _add_only)
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle _Input objects
```

`_Input` object is of type `uwsgi._Input`



Discussion:
Wow, that's an interesting one.
So Django's caching layer adds a post_render_callback to the response which is used to cache the response once it's been rendered.  So far so good.  The problem is that it uses pickle to do so, which is blowing up for Response objects (where it would pass for HttpResponse objects).

I've not looked at the pickle API enough to know exactly how this needs to work, but seems like we need to implement `__getstate__` on Response to do something other than the default behavior.

I reckon we can probably get this sorted, just needs a bit of digging to figure out how to make Response objects properly pickle-able.

First step would be to throw together some failing tests.

I believe the fix would be to ensure that Response.**getstate** calls the superclass, then pops off .accepted_renderer / .renderer_context / .data from the dict before returning, (since those bits of may not be pickable.)

That should work okay, because the restored, unpicked Response will include the required `.content`, it just won't include the intermediate state it used along the way to get.

There's probably a slightly nicer solution that _does_ correctly restore all the state, but I think this should get the job done and cover all sensible use-cases.

Sounds great. I was just about to write some tests for this, but maybe you're already on your way with a fix?

@jmagnusson No, not started on anything.  Be mighty pleased if you could tackle it, or provide tests.

Okay, I'll see what I can do. Hopefully I will get some time to do it later this week :-)


Resolving patch:
diff --git a/rest_framework/response.py b/rest_framework/response.py
--- a/rest_framework/response.py
+++ b/rest_framework/response.py
@@ -45,3 +45,13 @@ def status_text(self):
         # TODO: Deprecate and use a template tag instead
         # TODO: Status code text for RFC 6585 status codes
         return STATUS_CODE_TEXT.get(self.status_code, '')
+
+    def __getstate__(self):
+        """
+        Remove attributes from the response that shouldn't be cached
+        """
+        state = super(Response, self).__getstate__()
+        for key in ('accepted_renderer', 'renderer_context', 'data'):
+            if key in state:
+                del state[key]
+        return state
diff --git a/rest_framework/runtests/settings.py b/rest_framework/runtests/settings.py
--- a/rest_framework/runtests/settings.py
+++ b/rest_framework/runtests/settings.py
@@ -21,6 +21,12 @@
     }
 }
 
+CACHES = {
+    'default': {
+        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
+    }
+}
+
 # Local time zone for this installation. Choices can be found here:
 # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
 # although not all choices may be available on all operating systems.


Test patch:
diff --git a/rest_framework/tests/renderers.py b/rest_framework/tests/renderers.py
--- a/rest_framework/tests/renderers.py
+++ b/rest_framework/tests/renderers.py
@@ -1,6 +1,8 @@
+import pickle
 import re
 
 from django.conf.urls.defaults import patterns, url, include
+from django.core.cache import cache
 from django.test import TestCase
 from django.test.client import RequestFactory
 
@@ -83,6 +85,7 @@ def get(self, request, **kwargs):
 urlpatterns = patterns('',
     url(r'^.*\.(?P<format>.+)$', MockView.as_view(renderer_classes=[RendererA, RendererB])),
     url(r'^$', MockView.as_view(renderer_classes=[RendererA, RendererB])),
+    url(r'^cache$', MockGETView.as_view()),
     url(r'^jsonp/jsonrenderer$', MockGETView.as_view(renderer_classes=[JSONRenderer, JSONPRenderer])),
     url(r'^jsonp/nojsonrenderer$', MockGETView.as_view(renderer_classes=[JSONPRenderer])),
     url(r'^html$', HTMLView.as_view()),
@@ -416,3 +419,89 @@ def assertXMLContains(self, xml, string):
         self.assertTrue(xml.startswith('<?xml version="1.0" encoding="utf-8"?>\n<root>'))
         self.assertTrue(xml.endswith('</root>'))
         self.assertTrue(string in xml, '%r not in %r' % (string, xml))
+
+
+# Tests for caching issue, #346
+class CacheRenderTest(TestCase):
+    """
+    Tests specific to caching responses
+    """
+
+    urls = 'rest_framework.tests.renderers'
+
+    cache_key = 'just_a_cache_key'
+
+    @classmethod
+    def _get_pickling_errors(cls, obj, seen=None):
+        """ Return any errors that would be raised if `obj' is pickled
+        Courtesy of koffie @ http://stackoverflow.com/a/7218986/109897
+        """
+        if seen == None:
+            seen = []
+        try:
+            state = obj.__getstate__()
+        except AttributeError:
+            return
+        if state == None:
+            return
+        if isinstance(state,tuple):
+            if not isinstance(state[0],dict):
+                state=state[1]
+            else:
+                state=state[0].update(state[1])
+        result = {}
+        for i in state:
+            try:
+                pickle.dumps(state[i],protocol=2)
+            except pickle.PicklingError:
+                if not state[i] in seen:
+                    seen.append(state[i])
+                    result[i] = cls._get_pickling_errors(state[i],seen)
+        return result
+
+    def http_resp(self, http_method, url):
+        """
+        Simple wrapper for Client http requests
+        Removes the `client' and `request' attributes from as they are
+        added by django.test.client.Client and not part of caching
+        responses outside of tests.
+        """
+        method = getattr(self.client, http_method)
+        resp = method(url)
+        del resp.client, resp.request
+        return resp
+
+    def test_obj_pickling(self):
+        """
+        Test that responses are properly pickled
+        """
+        resp = self.http_resp('get', '/cache')
+
+        # Make sure that no pickling errors occurred
+        self.assertEqual(self._get_pickling_errors(resp), {})
+
+        # Unfortunately LocMem backend doesn't raise PickleErrors but returns
+        # None instead.
+        cache.set(self.cache_key, resp)
+        self.assertTrue(cache.get(self.cache_key) is not None)
+
+    def test_head_caching(self):
+        """
+        Test caching of HEAD requests
+        """
+        resp = self.http_resp('head', '/cache')
+        cache.set(self.cache_key, resp)
+
+        cached_resp = cache.get(self.cache_key)
+        self.assertIsInstance(cached_resp, Response)
+
+    def test_get_caching(self):
+        """
+        Test caching of GET requests
+        """
+        resp = self.http_resp('get', '/cache')
+        cache.set(self.cache_key, resp)
+
+        cached_resp = cache.get(self.cache_key)
+        self.assertIsInstance(cached_resp, Response)
+        self.assertEqual(cached_resp.content, resp.content)