Repository: ckan/ckan

Issue:
Large memory leak in datastore API
When querying the datastore API on a large dataset (eg. querying 10,000 rows at a time) the memory used by the process keeps growing. On our dataset this is about 20MB leaking per request - more than can be reasonably mitigated by restarting the worker after a number of requests.

This happens both when running the application using Paster and when running as a WSGI script under Apache2.

It seems the string containing the response is kept in memory. This is referenced by the response object, which is referenced by the pylon context and (a few steps up) by the WSGI environment. The environment itself is referenced numerous times, so it's hard to track this down.

As a proof of concept, the symptom can be removed by this "amusing" code:

``` python
diff --git a/ckan/controllers/api.py b/ckan/controllers/api.py
index 08ba3a8..4b09102 100644
--- a/ckan/controllers/api.py
+++ b/ckan/controllers/api.py
@@ -19,6 +19,12 @@ import ckan.lib.munge as munge

 from ckan.common import _, c, request, response

+from threading import Timer
+
+
+def _clean_links(env):
+    env['pylons.controller']._py_object.response._body = ''
+

 log = logging.getLogger(__name__)

@@ -67,7 +73,9 @@ class ApiController(base.BaseController):

         # avoid status_code_redirect intercepting error responses
         environ['pylons.status_code_redirect'] = True
-        return base.BaseController.__call__(self, environ, start_response)
+        result = base.BaseController.__call__(self, environ, start_response)
+        Timer(10, _clean_links, [environ]).start()
+        return result

     def _finish(self, status_int, response_data=None,
                 content_type='text'):
```

Obviously this is not a workable solution!



Discussion:
@aliceh75 , is it possible to get some more details on the size of the dataset and the types of queries you are performing on the datastore? I need to be able to reliably recreate this to get a deeper look at it.

Our dataset is about 3,000,000 records. The queries are just simple fetches using the datastore API. I still have the script I used to test this at the time. If I run:

``` python
import json
import urllib
import urllib2

while True:
    request_params = {
        'resource_id': '...',
        'offset': 0,
        'limit': 10000 
    }
    request = urllib2.Request('http://.../api/action/datastore_search')
    response = urllib2.urlopen(request, urllib.quote(json.dumps(request_params)))
    str = response.read(100)
    print len(str)
    print str
    response.close()
```

Then I can see the memory used by CKAN go up and up and up. This is true both when ckan is running as a WSGI script and when it is running using paster.

Ensuring the response string is cleared (as I do in the code provided in an earlier comment) removes the symptom - no more memory leak. But the code I wrote is obviously not a proper solution - it's just there to demonstrate where the problem comes from.

If you change the end of _finish() in the api controller to write the response data instead of returning it do you see any difference?  I realise this also seems a bit hacky ...

```
        response.write(response_msg)
        return ''
```

instead of 

```
return response_msg
```

just a guess, but could something be logging the SQL issued to the DB? I know another web framwork records all the SQL issued when run in debug mode.

I've been poking around using heapy and it looks like the previous responses are hanging around in memory, but it's nontrivial to trackdown why they aren't getting garbage collected.

```
ipdb> heap[0].byid
Set of 151666 <str> objects. Total size = 226687040 bytes.
 Index     Size   %   Cumulative  %   Representation (limited)
     0 21196016   9.4  21196016   9.4 '{"help": "ht...l": 2173762}}'
     1 21196016   9.4  42392032  18.7 '{"help": "ht...l": 2173762}}'
     2 21196016   9.4  63588048  28.1 '{"help": "ht...l": 2173762}}'
     3 21196016   9.4  84784064  37.4 '{"help": "ht...l": 2173762}}'
     4 21196016   9.4 105980080  46.8 '{"help": "ht...l": 2173762}}'
     5 21196016   9.4 127176096  56.1 '{"help": "ht...l": 2173762}}'
     6 21196016   9.4 148372112  65.5 '{"help": "ht...l": 2173762}}'
     7 21196016   9.4 169568128  74.8 '{"help": "ht...l": 2173762}}'
     8 21196016   9.4 190764144  84.2 '{"help": "ht...l": 2173762}}'
     9 21196016   9.4 211960160  93.5 '{"help": "ht...l": 2173762}}'

...
```

e: Below is the shortest paths of references that heapy can find for the above strings

```
ipdb> pprint([strings.byid[x].shpaths for x in range(0,10)])
[ 0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x4e56250>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x4e56e50>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x4e565f0>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x4ce2430>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x5433ab0>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.t139785342744320_c_traceobj.__self__.__dict__['curframe'].f_locals['result'][0]
 1: hpy().Root.t139785342744320_c_traceobj.__self__.__dict__['curframe_locals']['result'][0]
 2: hpy().Root.t139785342744320_c_traceobj.im_self.__dict__['curframe'].f_locals['result'][0]
 3: hpy().Root.t139785342744320_c_traceobj.im_self.__dict__['curframe_locals']['result'][0],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x55288f0>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x4ce28d0>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x4ce21f0>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body'],
  0: hpy().Root.i0_modules['routes'].__dict__['_RequestConfig'].__dict__['_RequestConfig__shared_state'].??[<weakref...x54339b0>]['environ']['pylons.controller'].__dict__['_py_object'].__dict__['response'].__dict__['_body']
```

e: .rp shows a tree of references, so in this case,the str is onctained in a dict of pylons.controllers.util.Response and that's contained in 3, dict of pylons.util.PylonsContext and so on and so forth

```
...
ipdb> strings.byid[0].rp
Reference Pattern by <[dict of] class>.
 0: _ --- [-] 1 <id 0x7f21a9792010>: 0x7f21a9792010
 1: a      [-] 1 dict of pylons.controllers.util.Response: 0x7f224a002550
 2: aa ---- [-] 1 pylons.controllers.util.Response: 0x7f224a002550
 3: a3       [-] 1 dict of pylons.util.PylonsContext: 0x7f224a002790
 4: a4 ------ [-] 1 pylons.util.PylonsContext: 0x7f224a002790
 5: a5         [-] 1 dict of pylons.util.AttribSafeContextObj: 0x7f224a002ad0
 6: a6 -------- [-] 1 pylons.util.AttribSafeContextObj: 0x7f224a002ad0
 7: a7           [^ 3] 1 dict of pylons.util.PylonsContext: 0x7f224a002790
 8: a4b ------ [-] 1 dict of ckan.controllers.api.ApiController: 0x7f224a002c10
 9: a4ba        [-] 1 ckan.controllers.api.ApiController: 0x7f224a002c10
<Type e.g. '_.more' for more.>
ipdb> strings.byid[0].rp.more
10: a4baa ------ [+] 1 dict (no owner): 0x7f222405ef30*50
11: a4bab        [+] 1 types.MethodType: <ckan.controllers.api.ApiController ...
```

@joetsoi When I looked at this the reference chain I found was something like:

response string <- response object <- pylon context <- ... <- WSGI environment <- ...

The WSGI environment itself was referenced numerous times, I couldn't really make anything out of it.

So this isn't a memory leak. I'm just going to document what I did here, just in case it's of interest to anyone else, or if I find it useful in the future when I've forgotten what happened.

So the `"_RequestConfig__shared_state'].??[ < weakref...x4e56250>"` in my last reply suggested that these objects should of been cleaned up once there was no longer a reference to them, a manual `gc.collect()` didn't seem to do anything and inspecting the`RequestConfig shared_state` showed it to be a thread.local object.

So at this point i'm thinking we must be holding a reference to it in the controller somewhere that we odn't clean up.

using heapy and inserting some ipdb breakpoints before the controller returns the result from the api, something like

```
from guppy import hpy
hp = hpy()
heap = hp.heap()
strings = heap[0]
import ipdb; ipdb.set_trace()
```

 it's clear that the response objects, specifically the response strings are the problem

```
ipdb> heap.byrcs
Partition of a set of 343689 objects. Total size = 266893568 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0     19   0 190780624  71 190780624  71 dict of pylons.controllers.util.Response
     1     11   0 21199320   8 211979944  79 dict of pylons.controllers.util.Response, list
     2 107194  31 10007816   4 221987760  83 types.CodeType
     3  41823  12  4018400   2 226006160  85 tuple
     4  24904   7  3071408   1 229077568  86 function
     5   9287   3  2350432   1 231428000  87 dict of module
     6   6440   2  2324560   1 233752560  88 function, tuple
     7   7144   2  2229040   1 235981600  88 type
     8  15536   5  1949840   1 237931440  89 dict of type
     9  19440   6  1858816   1 239790256  90 dict (no owner)
Partition of a set of 343689 objects. Total size = 266892800 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 137776  40 225205640  84 225205640  84 str
     1  83479  24  7054200   3 232259840  87 tuple
     2   8486   2  4350608   2 236610448  89 dict (no owner)
     3  18988   6  2430464   1 239040912  90 types.CodeType
     4  19980   6  2397600   1 241438512  90 function
     5    842   0  2387696   1 243826208  91 dict of module
     6   2190   1  1984280   1 245810488  92 type
     7   2190   1  1815888   1 247626376  93 dict of type
     8   9956   3  1585184   1 249211560  93 list
     9   2563   1  1509984   1 250721544  94 unicode
```

and by inspecting those response strings using byid as above in my previous comment we see they're the api responses. 

So I wanted to see if what we have in memory is exactly the responses that we requested (or was it copying one request over and over? etc)
I ended up tweaking your script a bit and removing the `total` from the result dict returned by `datastore_search`

```
import json
import urllib
import urllib2

i = 0
while True:

    request_params = {
        'resource_id': 'blah',
        'offset': 0,
        'limit': 100000 + i
    }
    request = urllib2.Request('http://localhost:5000/api/action/datastore_search')
    response = urllib2.urlopen(request, urllib.quote(json.dumps(request_params)))
    str = response.read()
    print len(str)
    print str[-100:]
    response.close()
    i += 1
```

This way, when I ran it the next time, the output from the script is something like

```
21195957
store_search", "next": "/api/action/datastore_search?offset=100000"}, "offset": 0, "limit": 100000}}
21196181
store_search", "next": "/api/action/datastore_search?offset=100001"}, "offset": 0, "limit": 100001}}
21196406
store_search", "next": "/api/action/datastore_search?offset=100002"}, "offset": 0, "limit": 100002}}
```

and when we insepct byid we get something like

```
ipdb> strings.byid
Set of 137776 <str> objects. Total size = 225205640 bytes.
 Index     Size   %   Cumulative  %   Representation (limited)
     0 21198840   9.4  21198840   9.4 '{"help": "ht...it": 100013}}'
     1 21198632   9.4  42397472  18.8 '{"help": "ht...it": 100012}}'
     2 21198424   9.4  63595896  28.2 '{"help": "ht...it": 100011}}'
     3 21198200   9.4  84794096  37.7 '{"help": "ht...it": 100010}}'
     4 21197976   9.4 105992072  47.1 '{"help": "ht...it": 100009}}'
     5 21197768   9.4 127189840  56.5 '{"help": "ht...it": 100008}}'
     6 21197552   9.4 148387392  65.9 '{"help": "ht...it": 100007}}'
     7 21197336   9.4 169584728  75.3 '{"help": "ht...it": 100006}}'
     8 21197120   9.4 190781848  84.7 '{"help": "ht...it": 100005}}'
     9 21196896   9.4 211978744  94.1 '{"help": "ht...it": 100004}}'
<137766 more rows. Type e.g. '_.more' to view.>
```

This way, we can directly see which of the requests we made are in memory. So we look at the next couple of strings

```
ipdb> strings.byid.more
 Index     Size   %   Cumulative  %   Representation (limited)
    10    77528   0.0 212056272  94.2 '============...elease.\n\n\n'
    11    30712   0.0 212086984  94.2 'Provide a re...n\n\n        '
    12    22352   0.0 212109336  94.2 '(?<=\\()(\\*...ng |zero\\? )'
    13    18016   0.0 212127352  94.2 'Return a new....\n\n        '
    14    13656   0.0 212141008  94.2 '\n.. dialect...stgreSQL.\n\n'
    15    12480   0.0 212153488  94.2 'subprocess -...cess.Popen.\n'
    16    12088   0.0 212165576  94.2 "Create a new...k'``.\n\n    "
    17    11088   0.0 212176664  94.2 'Create a SQL....\n\n        '
    18    10904   0.0 212187568  94.2 '\n        Co....\n\n        '
    19    10504   0.0 212198072  94.2 '\n    Format...output.\n    '
<137756 more rows. Type e.g. '_.more' to view.>
```

At this point, the first requests we made no longer appear in memory, the requests for 100000-100003 don't appear to be in memory. 

So by doing some stepping through each request and stopping before the controller returned each time, we can see that it maintains a maximum of 10 response strings in memory.

```
ipdb> c
2015-01-15 17:54:57,892 INFO  [ckan.lib.base]  /api/action/datastore_search render time 33.155 seconds
> /home/joe/projects/json_datastore/src/ckan/ckan/controllers/api.py(82)__call__()
     81             import ipdb; ipdb.set_trace()
---> 82         log.debug('mem {0}'.format(result[0][-20]))
     83 

ipdb> strings.byid
Set of 151739 <str> objects. Total size = 226701936 bytes.
 Index     Size   %   Cumulative  %   Representation (limited)
     0 21199056   9.4  21199056   9.4 '{"help": "ht...it": 100014}}'
     1 21198840   9.4  42397896  18.7 '{"help": "ht...it": 100013}}'
     2 21198632   9.4  63596528  28.1 '{"help": "ht...it": 100012}}'
     3 21198424   9.4  84794952  37.4 '{"help": "ht...it": 100011}}'
     4 21198200   9.4 105993152  46.8 '{"help": "ht...it": 100010}}'
     5 21197976   9.4 127191128  56.1 '{"help": "ht...it": 100009}}'
     6 21197768   9.4 148388896  65.5 '{"help": "ht...it": 100008}}'
     7 21197552   9.4 169586448  74.8 '{"help": "ht...it": 100007}}'
     8 21197336   9.4 190783784  84.2 '{"help": "ht...it": 100006}}'
     9 21197120   9.4 211980904  93.5 '{"help": "ht...it": 100005}}'
<151729 more rows. Type e.g. '_.more' to view.>
```

100004 has been wiped off. At this point after some discussion on irc, we thought it might be the fact that we're in debug mode, and it's just retaining the last 10 responses and their debugging information. but the problem remains if we turn on "debug = false"

At some point I pieced together the weakref, thread.local stuff with http://pythonpaste.org/paste-httpserver-threadpool.html and notice that [_the default number of threads for the paste http server is 10_](http://pythonpaste.org/modules/httpserver.html). If you test using "threadpool_workers=n" in your development.ini, it'll match the number of strings that will show up in heap[0].byid

The other thing to notice is that if you make 10 new  requests to api endpoints that return a much smaller response

```
Set of 151950 <str> objects. Total size = 78397616 bytes.
 Index     Size   %   Cumulative  %   Representation (limited)
     0 21199280  27.0  21199280  27.0 '{"help": "ht...it": 100015}}'
     1 21199056  27.0  42398336  54.1 '{"help": "ht...it": 100014}}'
     2 21198840  27.0  63597176  81.1 '{"help": "ht...it": 100013}}'
     3    77528   0.1  63674704  81.2 '============...elease.\n\n\n'
     4    30712   0.0  63705416  81.3 'Provide a re...n\n\n        '
     5    22352   0.0  63727768  81.3 '(?<=\\()(\\*...ng |zero\\? )'
     6    18016   0.0  63745784  81.3 'Return a new....\n\n        '
     7    13656   0.0  63759440  81.3 '\n.. dialect...stgreSQL.\n\n'
     8    12480   0.0  63771920  81.3 'subprocess -...cess.Popen.\n'
     9    12088   0.0  63784008  81.4 "Create a new...k'``.\n\n    "
<151940 more rows. Type e.g. '_.more' to view.>
ipdb> strings.byid.more
  Index     Size   %   Cumulative  %   Representation (limited)
    10    11088   0.0  63795096  81.4 'Create a SQL....\n\n        '
    11    10904   0.0  63806000  81.4 '\n        Co....\n\n        '
    12    10584   0.0  63816584  81.4 '{"help": "ht...facets": {}}}'    (the smaller api responses)
    13    10584   0.0  63827168  81.4 '{"help": "ht...facets": {}}}'
    14    10584   0.0  63837752  81.4 '{"help": "ht...facets": {}}}'
    15    10584   0.0  63848336  81.4 '{"help": "ht...facets": {}}}'
    16    10584   0.0  63858920  81.5 '{"help": "ht...facets": {}}}'
    17    10584   0.0  63869504  81.5 '{"help": "ht...facets": {}}}'
    18    10504   0.0  63880008  81.5 '\n    Format...output.\n    '
    19     9168   0.0  63889176  81.5 'Represent a ...ents.\n\n    '
<151930 more rows. Type e.g. '_.more' to view.>

```

you'll see the total size has dropped. from 226701936 bytes to 78397616 bytes. But the entire time I've been debugging, I've been running

```
$ watch "cat /proc/`pgrep paster`/status"
...
VmPeak:  3497708 kB
VmSize:  3497708 kB
...
```

Which still reports the memory usage the same as when the larger requests were still in memory. This is apparently just hte way python works (see https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw). If you've just made a response that took up 500mb of memory, it stands to reason that you might have to do the same again quite soon, so python will hang onto the (virtual) memory. But it's not 'leaked'

**TL;DR**, this isn't a memory leak, each thread will hold onto the memory that it has used and another thread can't garbage collect another thread's data, so, the issue at this point is whether we can clear out the response string once the response has been returned and whether it is safe to do so.

e:http://pylons-webframework.readthedocs.org/en/latest/controllers.html#special-methods, next step will probably to investigate whether it's ok to clear out the string in something like the `__after__()` method of the controller.

heapy docs
http://guppy-pe.sourceforge.net/heapy-thesis.pdf
http://smira.ru/wp-content/uploads/2011/08/heapy.html
http://haypo-notes.readthedocs.org/en/latest/heap_fragmentation.html
https://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html
http://forthescience.org/blog/2014/08/16/python-and-memory-fragmentation/

weak references
https://docs.python.org/2/library/weakref.html
http://pymotw.com/2/weakref/

pylons docs
http://pylons-webframework.readthedocs.org/en/latest/execution.html
http://pythonpaste.org/paste-httpserver-threadpool.html

awesome stuff @joetsoi, super useful!

I think it might be possible to have some cleanup middleware as suggested in these docs, https://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode. Pylons does a similar-ish thing at the end of requests https://github.com/Pylons/pylons/blob/master/pylons/wsgiapp.py#L118-L122.

This wouldn't stop say 15 simultaneous requests returning large results eating up a lot of memory at once.

So if I understand correctly, you are suggesting that the apparent memory usage should max out after 10 requests (with the occasional increase if a given request is larger than any that have been before).

I remember this growing steadily over a long period of time, not maxing out quickly - but it was a while ago so I will run tests again to see if what I see is consistent with that idea.


Resolving patch:
diff --git a/ckan/config/middleware.py b/ckan/config/middleware.py
--- a/ckan/config/middleware.py
+++ b/ckan/config/middleware.py
@@ -75,6 +75,8 @@ def make_app(conf, full_stack=True, static_files=True, **app_conf):
 
     # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
     #app = QueueLogMiddleware(app)
+    if asbool(config.get('ckan.use_pylons_response_cleanup_middleware', True)):
+        app = execute_on_completion(app, config, cleanup_pylons_response_string)
 
     # Fanstatic
     if asbool(config.get('debug', False)):
@@ -360,3 +362,41 @@ def __call__(self, environ, start_response):
             self.engine.execute(sql, key, data.get('url'), data.get('type'))
             return []
         return self.app(environ, start_response)
+
+
+def generate_close_and_callback(iterable, callback, environ):
+    """
+    return a generator that passes through items from iterable
+    then calls callback(environ).
+    """
+    try:
+        for item in iterable:
+            yield item
+    except GeneratorExit:
+        if hasattr(iterable, 'close'):
+            iterable.close()
+        raise
+    finally:
+        callback(environ)
+
+
+def execute_on_completion(application, config, callback):
+    """
+    Call callback(environ) once complete response is sent
+    """
+    def inner(environ, start_response):
+        try:
+            result = application(environ, start_response)
+        except:
+            callback(environ)
+            raise
+        return generate_close_and_callback(result, callback, environ)
+    return inner
+
+
+def cleanup_pylons_response_string(environ):
+    try:
+        msg = 'response cleared by pylons response cleanup middleware'
+        environ['pylons.controller']._py_object.response._body = msg
+    except (KeyError, AttributeError):
+        pass


Test patch:
diff --git a/ckan/tests/config/test_middleware.py b/ckan/tests/config/test_middleware.py
new file mode 100644
--- /dev/null
+++ b/ckan/tests/config/test_middleware.py
@@ -0,0 +1,25 @@
+import ckan.tests.helpers as helpers
+
+from nose.tools import assert_equals, assert_not_equals
+from routes import url_for
+
+
+class TestPylonsResponseCleanupMiddleware(helpers.FunctionalTestBase):
+    @classmethod
+    def _apply_config_changes(cls, config):
+        config['ckan.use_pylons_response_cleanup_middleware'] = True
+
+    def test_homepage_with_middleware_activated(self):
+        '''Test the home page renders with the middleware activated
+
+        We are just testing the home page renders without any troubles and that
+        the middleware has not done anything strange to the response string'''
+        app = self._get_test_app()
+        response = app.get(url=url_for(controller='home', action='index'))
+
+        assert_equals(200, response.status_int)
+        # make sure we haven't overwritten the response too early.
+        assert_not_equals(
+            'response cleared by pylons response cleanup middleware',
+            response.body
+        )