Repository: ckan/ckan

Issue:
ckan dies downloading 1GB file in debug mode (oom_killer)
### CKAN Version if known (or site URL)
2.6.0

### Please describe the expected behaviour
ckan "paster serve" web server process does not mysteriously exit when I download the 1GB file.

### Please describe the actual behaviour
ckan "paster serve" web server process mysteriously exits when I download the 1GB file. No error message is produced to stdout/stderr or apache logs.

The error shows up in /var/log/messages and dmesg: "python2 invoked oom-killer" and "Out of memory: Kill process 1699 (python2) score 418 or sacrifice child".

### What steps can be taken to reproduce the issue? 
Set debug to true in ckan.ini. (debug mode must be on to hit the problem)
Increase file-size limits to handle 1GB files: Increase max_resource_size (and nginx client_max_body_size if nginx is used).
Create 1GB file eg dd if=/dev/zero of=test1G bs=1GB count=1
Run "paster serve". (Running apache requires debug to be false so the problem does not occur)
Create a dataset and upload a 1GB file to the new initial resource
Click the download link on this 1GB file. (/dataset/.../resource/.../download/test1g)
The ckan "paster serve" silently exits with no error or stack trace. The download fails.

### Details

I found the problem line like this:

//pdb breakpoint right in the download code before it dies:
vi controllers/package.py +/"def resource_download"
before "return app_iter", insert the line:
import pdb; pdb.set_trace()

I just kept pressing "r" to return up the stack until the last func before ckan mysteriously exited with no error or information. Then I stepped forward a few lines to the exact line which died.

Problem line is in site-packages/weberror/evalexception.py:
return_iter = list(app_iter)

The surrounding code shows more about the problem:

```
B->             if isinstance(app_iter, fileapp._FileIter):
                     return app_iter

                  try:
                     return_iter = list(app_iter)
                     return return_iter
```

So if type is _FileIter, it works fine and does not die. But there is a feature in ckan called "ckan.use_pylons_response_cleanup_middleware" which is on by default. When it is off, the problem goes away. It wraps _FileIter inside generate_close_and_callback. It does that here:

/usr/lib/ckan/default/src/ckan/ckan/config/middleware/pylons_app.py
return generate_close_and_callback(result, callback, environ)

So type is no longer _FileIter so evalexception.py runs "list(app_iter)" and that triggers oom_killer.

When debug is false, pylons ErrorHandler middleware does not use EvalException so the problem is avoided. (see pylons/middleware.py function ErrorHandler)

### Possible ideas for a solution/improvement?

Perhaps option "ckan.use_pylons_response_cleanup_middleware" should default to false? Perhaps if the ckan wrapper was a subclass of _FileIter, that would fix the problem? Perhaps the oom_killer should only kill the 1 child process, not all the web server processes? Perhaps oom_killer should have larger memory limits in debug mode? Can EvalException (or the ErrorHandler middleware in general) somehow be skipped for downloads?

I notice that when debug is false, weberror does not call "list(app_iter)" but does wrap app_iter in wraps app_iter in CatchingIter (see weberror/errormiddleware.py:188). Perhaps this is a better way to wrap it? I am not sure if these 2 actions are comparable.


Discussion:
@amercader this is related to the flask changes, what do you think?

Resolving patch:
diff --git a/ckan/config/middleware/pylons_app.py b/ckan/config/middleware/pylons_app.py
--- a/ckan/config/middleware/pylons_app.py
+++ b/ckan/config/middleware/pylons_app.py
@@ -10,6 +10,7 @@
 from paste.registry import RegistryManager
 from paste.urlparser import StaticURLParser
 from paste.deploy.converters import asbool
+from paste.fileapp import _FileIter
 from pylons.middleware import ErrorHandler, StatusCodeRedirect
 from routes.middleware import RoutesMiddleware
 from repoze.who.config import WhoConfig
@@ -223,33 +224,58 @@ def can_handle_request(self, environ):
             return (False, self.app_name)
 
 
-def generate_close_and_callback(iterable, callback, environ):
-    """
-    return a generator that passes through items from iterable
-    then calls callback(environ).
+class CloseCallbackWrapper(object):
+    def __init__(self, iterable, callback, environ):
+        # pylons.fileapp expects app_iter to have `file` attribute.
+        self.file = iterable
+        self.callback = callback
+        self.environ = environ
+
+    def __iter__(self):
+        """
+        return a generator that passes through items from iterable
+        then calls callback(environ).
+        """
+        try:
+            for item in self.file:
+                yield item
+        except GeneratorExit:
+            if hasattr(self.file, 'close'):
+                self.file.close()
+            raise
+        finally:
+            self.callback(self.environ)
+
+
+class FileIterWrapper(CloseCallbackWrapper, _FileIter):
+    """Same CloseCallbackWrapper, just with _FileIter mixin.
+
+    That will prevent pylons from converting file responses into
+    in-memori lists.
     """
-    try:
-        for item in iterable:
-            yield item
-    except GeneratorExit:
-        if hasattr(iterable, 'close'):
-            iterable.close()
-        raise
-    finally:
-        callback(environ)
+    pass
 
 
 def execute_on_completion(application, config, callback):
     """
     Call callback(environ) once complete response is sent
     """
+
     def inner(environ, start_response):
         try:
             result = application(environ, start_response)
         except:
             callback(environ)
             raise
-        return generate_close_and_callback(result, callback, environ)
+        # paste.fileapp converts non-file responses into list
+        # In order to avoid interception of OOM Killer
+        # file responses wrapped into generator with
+        # _FileIter in parent tree.
+        klass = CloseCallbackWrapper
+        if isinstance(result, _FileIter):
+            klass = FileIterWrapper
+        return klass(result, callback, environ)
+
     return inner
 
 


Test patch: