Repository: ckan/ckan

Issue:
exception after upgrade to 2.8.1 - Popped wrong app context
After upgrade from 2.7 to 2.8.1 - I'm seeing some exceptions in a production server log, couldn't reproduce the error locally

### CKAN Version if known (or site URL)
2.8.1

### Please describe the expected behaviour
no exception (or less cryptic exception..)

### Please describe the actual behaviour
exception - 
```
-------------------------------------------------------------
Error - <type 'exceptions.AssertionError'>: Popped wrong app context.  (<flask.ctx.AppContext object at 0x7f3ac4b90b10> instead of <flask.ctx.AppContext object at 0x7f3ac4b90690>)
URL: http://www.odata.org.il/organization/hatzlacha?license_id=cc-by&tags=%D7%94%D7%AA%D7%A4%D7%A8%D7%A6%D7%95%D7%AA+%D7%9C%D7%9E%D7%A7%D7%95%D7%9D+%D7%9E%D7%92%D7%95%D7%A8%D7%99%D7%9D&organization=hatzlacha&res_format=DOC&tags=%D7%A2%D7%99%D7%9B%D7%95%D7%91+%D7%94%D7%9C%D7%99%D7%9B%D7%99%D7%9D
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
  app_iter = self.application(environ, sr_checker)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
  return request.get_response(self.app)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
  resp = self.call_func(req, *args, **self.kwargs)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
  return self.func(req, *args, **kwargs)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
  response = request.get_response(self.app)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
  application, catch_exc_info=False)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
  app_iter = application(self.environ, start_response)
File '/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/pylons_app.py', line 265 in inner
  result = application(environ, start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
  return self.app(environ, start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/beaker/middleware.py', line 156 in __call__
  return self.wrap_app(environ, session_start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/common_middleware.py', line 30 in __call__
  return self.app(environ, start_response)
File '/usr/lib/ckan/venv/src/ckan/ckan/config/middleware/common_middleware.py', line 56 in __call__
  return self.app(environ, start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
  return controller(environ, start_response)
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 240 in __call__
  res = WSGIController.__call__(self, environ, start_response)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
  response = self._dispatch_call()
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
  response = self._inspect_call(func)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
  return func(**args)
File '/usr/lib/ckan/venv/src/ckan/ckan/controllers/group.py', line 230 in read
  extra_vars={'group_type': group_type})
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 125 in render
  return cached_template(template_name, renderer)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
  return render_func()
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 162 in render_template
  return render_jinja2(template_name, globs)
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 94 in render_jinja2
  return template.render(**extra_vars)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
  return self.environment.handle_exception(exc_info, True)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
  reraise(exc_type, exc_value, tb)
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/organization/read.html', line 1 in top-level template code
  {% extends "organization/read_base.html" %}
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/organization/read_base.html', line 1 in top-level template code
  {% extends "page.html" %}
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/page.html', line 1 in top-level template code
  {% extends "base.html" %}
File '/ckanext-odata_org_il/ckanext/odata_org_il/templates/base.html', line 1 in top-level template code
  {% ckan_extends %}
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/base.html', line 101 in top-level template code
  {%- block page %}{% endblock -%}
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/page.html', line 125 in block "page"
  {%- block footer %}
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/page.html', line 126 in block "footer"
  {% include "footer.html" %}
File '/ckanext-odata_org_il/ckanext/odata_org_il/templates/footer.html', line 3 in top-level template code
  {% block footer_content %}
File '/ckanext-odata_org_il/ckanext/odata_org_il/templates/footer.html', line 32 in block "footer_content"
  {% block footer_lang %}
File '/ckanext-odata_org_il/ckanext/odata_org_il/templates/footer.html', line 33 in block "footer_lang"
  {% snippet "snippets/language_selector.html" %}
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/jinja_extensions.py', line 268 in _call
  return base.render_snippet(args[0], **kwargs)
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 84 in render_snippet
  output = render(template_name, extra_vars=kw)
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 125 in render
  return cached_template(template_name, renderer)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
  return render_func()
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 162 in render_template
  return render_jinja2(template_name, globs)
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/base.py', line 94 in render_jinja2
  return template.render(**extra_vars)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
  return self.environment.handle_exception(exc_info, True)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
  reraise(exc_type, exc_value, tb)
File '/usr/lib/ckan/venv/src/ckan/ckan/templates/snippets/language_selector.html', line 2 in top-level template code
  <form class="form-inline form-select lang-select" action="{% url_for controller='util', action='redirect' %}" data-module="select-switch" method="POST">
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/jinja_extensions.py', line 297 in _call
  return h.url_for(*args, **kwargs)
File '/usr/lib/ckan/venv/src/ckan/ckan/lib/helpers.py', line 326 in url_for
  _auto_flask_context.pop()
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/ctx.py', line 376 in pop
  app_ctx.pop(exc)
File '/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/ctx.py', line 193 in pop
  % (rv, self)
AssertionError: Popped wrong app context.  (<flask.ctx.AppContext object at 0x7f3ac4b90b10> instead of <flask.ctx.AppContext object at 0x7f3ac4b90690>)


CGI Variables
-------------
  CKAN_CURRENT_URL: '/organization/hatzlacha?license_id%3Dcc-by%26tags%3D%25D7%2594%25D7%25AA%25D7%25A4%25D7%25A8%25D7%25A6%25D7%2595%25D7%25AA%2B%25D7%259C%25D7%259E%25D7%25A7%25D7%2595%25D7%259D%2B%25D7%259E%25D7%2592%25D7%2595%25D7%25A8%25D7%2599%25D7%259D%26organization%3Dhatzlacha%26res_format%3DDOC%26tags%3D%25D7%25A2%25D7%2599%25D7%259B%25D7%2595%25D7%2591%2B%25D7%2594%25D7%259C%25D7%2599%25D7%259B%25D7%2599%25D7%259D'
  CKAN_LANG: 'he'
  CKAN_LANG_IS_DEFAULT: True
  CONTENT_LENGTH: '0'
  HTTP_ACCEPT: '*/*'
  HTTP_ACCEPT_ENCODING: 'gzip'
  HTTP_CF_CONNECTING_IP: ' --- '
  HTTP_CF_IPCOUNTRY: 'US'
  HTTP_CF_RAY: '450eef3ef9fa5afd-HEL'
  HTTP_CF_VISITOR: '{"scheme":"https"}'
  HTTP_HOST: 'www.odata.org.il'
  HTTP_USER_AGENT: ' --- '
  HTTP_X_FORWARDED_FOR: ' --- '
  HTTP_X_FORWARDED_HOST: 'www.odata.org.il'
  HTTP_X_FORWARDED_PORT: '443'
  HTTP_X_FORWARDED_PROTO: 'https'
  HTTP_X_FORWARDED_SERVER: 'traefik-77ccf967dc-4dfwt'
  HTTP_X_REAL_IP: '10.132.0.5'
  PATH_INFO: '/organization/hatzlacha'
  QUERY_STRING: 'license_id=cc-by&tags=%D7%94%D7%AA%D7%A4%D7%A8%D7%A6%D7%95%D7%AA+%D7%9C%D7%9E%D7%A7%D7%95%D7%9D+%D7%9E%D7%92%D7%95%D7%A8%D7%99%D7%9D&organization=hatzlacha&res_format=DOC&tags=%D7%A2%D7%99%D7%9B%D7%95%D7%91+%D7%94%D7%9C%D7%99%D7%9B%D7%99%D7%9D'
  REMOTE_ADDR: '10.20.1.228'
  REQUEST_METHOD: 'GET'
  SERVER_NAME: '0.0.0.0'
  SERVER_PORT: '5000'
  SERVER_PROTOCOL: 'HTTP/1.1'


WSGI Variables
--------------
  __no_cache__: True
  application: <fanstatic.publisher.Delegator object at 0x7f3ac38876d0>
  beaker.cache: <beaker.cache.CacheManager object at 0x7f3ac3887690>
  beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f3ac38875d0>>
  beaker.session: {'_accessed_time': 1535377065.228329, '_creation_time': 1535377065.228329}
  ckan.app: 'pylons_app'
  fanstatic.needed: <fanstatic.core.NeededResources object at 0x7f3aa9ff7450>
  paste.cookies: (<SimpleCookie: >, '')
  paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x7f3ac4b8e910>
  paste.parsed_dict_querystring: (MultiDict([('license_id', 'cc-by'), ('tags', '\xd7\x94\xd7\xaa\xd7\xa4\xd7\xa8\xd7\xa6\xd7\x95\xd7\xaa \xd7\x9c\xd7\x9e\xd7\xa7\xd7\x95\xd7\x9d \xd7\x9e\xd7\x92\xd7\x95\xd7\xa8\xd7\x99\xd7\x9d'), ('organization', 'hatzlacha'), ('res_format', 'DOC'), ('tags', '\xd7\xa2\xd7\x99\xd7\x9b\xd7\x95\xd7\x91 \xd7\x94\xd7\x9c\xd7\x99\xd7\x9b\xd7\x99\xd7\x9d')]), 'license_id=cc-by&tags=%D7%94%D7%AA%D7%A4%D7%A8%D7%A6%D7%95%D7%AA+%D7%9C%D7%9E%D7%A7%D7%95%D7%9D+%D7%9E%D7%92%D7%95%D7%A8%D7%99%D7%9D&organization=hatzlacha&res_format=DOC&tags=%D7%A2%D7%99%D7%9B%D7%95%D7%91+%D7%94%D7%9C%D7%99%D7%9B%D7%99%D7%9D')
  paste.parsed_querystring: ([('license_id', 'cc-by'), ('tags', '\xd7\x94\xd7\xaa\xd7\xa4\xd7\xa8\xd7\xa6\xd7\x95\xd7\xaa \xd7\x9c\xd7\x9e\xd7\xa7\xd7\x95\xd7\x9d \xd7\x9e\xd7\x92\xd7\x95\xd7\xa8\xd7\x99\xd7\x9d'), ('organization', 'hatzlacha'), ('res_format', 'DOC'), ('tags', '\xd7\xa2\xd7\x99\xd7\x9b\xd7\x95\xd7\x91 \xd7\x94\xd7\x9c\xd7\x99\xd7\x9b\xd7\x99\xd7\x9d')], 'license_id=cc-by&tags=%D7%94%D7%AA%D7%A4%D7%A8%D7%A6%D7%95%D7%AA+%D7%9C%D7%9E%D7%A7%D7%95%D7%9D+%D7%9E%D7%92%D7%95%D7%A8%D7%99%D7%9D&organization=hatzlacha&res_format=DOC&tags=%D7%A2%D7%99%D7%9B%D7%95%D7%91+%D7%94%D7%9C%D7%99%D7%9B%D7%99%D7%9D')
  paste.registry: <paste.registry.Registry object at 0x7f3ac3f14450>
  paste.throw_errors: True
  pylons.action_method: <bound method OrganizationController.read of <ckan.controllers.organization.OrganizationController object at 0x7f3aa8826b10>>
  pylons.controller: <ckan.controllers.organization.OrganizationController object at 0x7f3aa8826b10>
  pylons.environ_config: {'session': 'beaker.session', 'cache': 'beaker.cache'}
  pylons.pylons: <pylons.util.PylonsContext object at 0x7f3aa8826d50>
  pylons.routes_dict: {'action': u'read', 'controller': u'organization', 'id': u'hatzlacha'}
  repoze.who.api: <repoze.who.api.API object at 0x7f3ac37da0d0>
  repoze.who.logger: <logging.Logger object at 0x7f3ac3887890>
  repoze.who.plugins: {'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f3ac3acd090>, 'friendlyform': <FriendlyFormPlugin 139890367716112>, 'auth_tkt': <CkanAuthTktCookiePlugin 139890367715984>}
  routes.route: <routes.route.Route object at 0x7f3ac4b13f10>
  routes.url: <routes.util.URLGenerator object at 0x7f3ac40d9690>
  webob._parsed_query_vars: (GET([('license_id', 'cc-by'), ('tags', '\xd7\x94\xd7\xaa\xd7\xa4\xd7\xa8\xd7\xa6\xd7\x95\xd7\xaa \xd7\x9c\xd7\x9e\xd7\xa7\xd7\x95\xd7\x9d \xd7\x9e\xd7\x92\xd7\x95\xd7\xa8\xd7\x99\xd7\x9d'), ('organization', 'hatzlacha'), ('res_format', 'DOC'), ('tags', '\xd7\xa2\xd7\x99\xd7\x9b\xd7\x95\xd7\x91 \xd7\x94\xd7\x9c\xd7\x99\xd7\x9b\xd7\x99\xd7\x9d')]), 'license_id=cc-by&tags=%D7%94%D7%AA%D7%A4%D7%A8%D7%A6%D7%95%D7%AA+%D7%9C%D7%9E%D7%A7%D7%95%D7%9D+%D7%9E%D7%92%D7%95%D7%A8%D7%99%D7%9D&organization=hatzlacha&res_format=DOC&tags=%D7%A2%D7%99%D7%9B%D7%95%D7%91+%D7%94%D7%9C%D7%99%D7%9B%D7%99%D7%9D')
  webob.adhoc_attrs: {'response': <Response at 0x7f3aa9ff7990 200 OK>, 'language': 'en-us'}
  wsgi process: 'Multithreaded'
  wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x7f3ac40d9690>, {'action': u'read', 'controller': u'organization', 'id': u'hatzlacha'})
------------------------------------------------------------
```

### What steps can be taken to reproduce the issue? 
I'm just seeing those exception in the server logs, couldn't reproduce locally


Discussion:
seems to be solved by switching to gunicorn to serve wsgi (used paster serve before)

still interesting to understand why it happens
Same error here, but I'm using Apache instead of Nginx. Could you give me more information about the solution adopted?
Install Gunicorn

```
pip install gunicorn
```

Start the server using Gunicorn

```
gunicorn --paste /path/to/ckan.ini
```
When I launch the application, it asks me to install each CKAN module, including the plugins that I have enabled. I already have an instance of CKAN installed with these plugins working perfectly, running on Apache. Is it necessary to reinstall everything?

Thx
Probably you are running gunicorn from a different Python virtualenv then your CKAN instance - causing it to not recognize the installed plugins.
Thanks @OriHoch, I have managed to configure CKAN with gunicorn but due to its complexity (I have not configured it completely, there are things that do not work correctly), I think I'm not able to replace Apache or Nginx for this in my production environment.

@amercader, Do you plan to take a fix for this bug soon?

Thx.
This is also happening with an upgrade to 2.8.2.  I can provide a full error log with debug output, if needed.  To see how we're deploying, folks can refer to our Ansible playbook:

https://github.com/City-of-Bloomington/ckan-deploy


This seems to be related to problems with multithreading.  Either flask or the ckan code doesn't seem to be thread-safe.  As soon as more than one request is in flight at the same time, the popped wrong app context error occurs, and the site crashes.

In Apache, I've limited our daemon to only one process and one thread.  For now, this seems to have alleviated the problem.

```apache
WSGIScriptAlias / "/srv/sites/ckan/public/ckan.wsgi"
WSGIPassAuthorization On
WSGIDaemonProcess ckan processes=1 threads=1
WSGIProcessGroup ckan

<Directory "/srv/sites/ckan/public">
    Options FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

```
Looking into this. It's reproducible for example by generating a large number of organizations (I generated 1360), setting processes and threads to low numbers, then requesting the organization list page repeatedly.

I'm seeing `ckan.lib.helpers.url_for` calling `_get_auto_flask_context` and receiving `_internal_test_request_context` back. The `_request_ctx_stack.top` it's probably supposed to return for real calls is `None`.
Diving deeper, the expected flask app context is pushed in a different thread than where it is supposed to be popped. So thread T1 may push app context C1 to be used for request context R1, and T2 push C2 for R2, but R1 ends up being exectuted in T2 and get C2 and vice versa. Trying to parse my thread-identified logging it looks like this:

    T1, R3: push C3
    T1, R1: push C1
    T1, R1: pop -> C1
    T2, R2: push C2
    T1, R2: pop -> C3 (error, expected C2)
    T2, R3: pop -> C2 (error, expected C3)

I instrumented `/usr/lib/ckan/default/lib/python2.7/site-packages/flask/ctx.py` app and request context push/pop functions to print `thread.get_ident()` and `hex(id(self))`. I then search-and-replaced the memory addresses into the more readable format above.

    T1: Push app context C1
    T1: Push request context R1
    T1: Pop request context R1
    T1: Popped correct app context C1
    T1: Push app context C2
    T1: Push request context R1
    T1: Pop request context R1
    T1: Popped correct app context C2
      T2: Push app context C3
      T2: Push request context R1
      T2: Pop request context R1
      T2: Popped correct app context C3
      T2: Push app context C4
      T2: Push request context R1
      T2: Pop request context R1
      T2: Popped correct app context C4
      T2: Push app context C5
      T2: Push request context R1
    T1: Push app context C6
    T1: Push request context R1
      T2: Pop request context R1
      T2: Popped wrong app context.  (C5 instead of C6)
    T1: Pop request context R1
    T1: Popped wrong app context.  (C6 instead of C5)

As can be seen, as long as the same thread pushes and pops the app context before the other gets its turn, everything works out fine. But the moment there's a push from both threads consecutively, the error manifests. The app context stacks are thread-local, so the requests should be handled in the thread they were created at.

A curious addition is that all the request contexts are `R1`, meaning they all point to the same object regardless of thread. Not sure yet if this can cause issues, but it sure sounds suspicious.
As near as I can tell, the request context created in `ckan.config.middleware.AskAppDispatcherMiddleware.__call__` on the line `with flask_app.test_request_context():` in pylons core request handling doesn't live as long as the request handling takes. The request context is popped from the stack as soon as the `self.apps[app_name](environ, start_response)` returns. Then the thread moves on to process the next request.

Now, pylons seems to do multithread request handling, so the request context is actually needed longer than this. Because the `_request_ctx_stack` is now empty, `_get_auto_flask_context` returns the same (AFAICT) entirely thread-unsafe global `ckan.config.middleware._internal_test_request_context` for every pylons call, which even has the comment `"This is a test Flask request context to be used internally. Do not use it!"`.

This may be the main culprit. As the error message "Popped wrong app context" implies, the Flask `AppContext` already knows what should be on the top of the thread-local `_app_ctx_stack` when `pop()` is called. This means that the `AppContext.pop()` is being called in the wrong thread, which suggests the `AppContext` object has somehow traveled from one thread to another and I'm guessing it's through the shared `RequestContext` object.

Pylons needs to run single-threaded if it's embedded with flask contexts like this. The problem only shows when there are enough multiple concurrent requests, because otherwise the same thread seems to get to handle all parts of generating the response, meaning the correct `AppContext` will always be available.
Last clue for now: Calling the same organization list view just once results in

    // These are executed under /etc/ckan/default/apache.wsgi
    push C1
    push R1
    execute pylons middleware
    pop R1
    pop C1

    // Then, these are executed under /usr/lib/ckan/default/lib/python2.7/site-packages/paste/registry.py
    push C2
    push R2
    pop R2
    pop C2
    push C3
    push R2
    pop R2
    pop C3
    ...

The actual request context, `R1`, is only available for the first part of the full handling run by `apache.wsgi`. Each subsequent part run by paste with `RegistryManager` receives the same global test request context because the request context stack is empty. Based on where the app is running during the latter events, namely template rendering (hence `url_for` I reported earlier), I'm guessing the response isn't sent yet.
Waiting for an opinion on the findings above and potential solutions before digging deeper. Someone more familiar with the code can probably see the issue more clearly. Is the context pushed in `apache.wsgi` supposed to be available when the request is actually processed? What is the threading instance that runs the `RegistryManager`-rooted parts of the request processing and how are they synchronized with the wsgi handling?
@bzar a potential workaround for this could be to make the default request context objects thread-safe. We do that for some other wrappers in `ckan/common.py`. Can you try if applying this patch and see if it helps? (it's based on the `dev-v2.8` branch):

```diff
diff --git a/ckan/config/middleware/__init__.py b/ckan/config/middleware/__init__.py
index 082372d70..fd3eeccc5 100644
--- a/ckan/config/middleware/__init__.py
+++ b/ckan/config/middleware/__init__.py
@@ -5,6 +5,7 @@ import urllib
 import urlparse
 import urllib
 
+from werkzeug.local import Local
 import webob
 from routes import request_config as routes_request_config
 
@@ -64,7 +65,11 @@ def make_app(conf, full_stack=True, static_files=True, **app_conf):
     # Set this internal test request context with the configured environment so
     # it can be used when calling url_for from tests
     global _internal_test_request_context
-    _internal_test_request_context = flask_app._wsgi_app.test_request_context()
+
+    local = Local()
+    local(u'_internal_test_request_context')
+    local._internal_test_request_context = flask_app._wsgi_app.test_request_context()
+    _internal_test_request_context = local._internal_test_request_context
 
     return app

```
Okay, after extensive testing I can say this does not seem to fix the issue, but affects it somehow. It did seem like it worked at first, but ultimately it seems to merely have narrowed the cases it happens in.

With my current test case I can still reproduce the issue, but only with a newly started apache. I think this is because it takes longer to handle the first calls, so if I picked even longer page loads it would probably still pop up.

However, this means the application context thread-hopping probably happens somewhere else, or all the thread-local duplicated test contexts refer to some same global data, even if they themselves are thread-local.
nice work @bzar ! I tried to investigate myself initially but gave up much sooner then you :) very hard to debug it
Setting `streaming=False` here <https://github.com/ckan/ckan/blob/master/ckan/config/middleware/pylons_app.py#L144> seems to fix the issue completely as far as I can see. It makes the entire pylons call synchronous within the wsgi call handling. The `streaming=True` was introduced in CKAN 2.8.
Just to explain how I got there: After trying amercader's patch I realized the issue was going to be really hairy to fix by adding synchronization primitives. There were always going to be places where the entire thing could fall apart since the system wasn't designed to be used this way. So I went back to finding out the reason for the parallelization in the first place. WSGI should already parallelize handling the calls, so further parallelization in pylons shouldn't be necessary. After poring through Pylons' code and CKAN pylons app layer, I came across the streaming functionality right where I saw the break point between the WSGI handler and RegistryManager stack traces earlier. This led me to try disabling the streaming and without any additional changes things started working and stack traces began containing the entire execution tree from WSGI to our template code.

Resolving patch:
diff --git a/ckan/config/middleware/pylons_app.py b/ckan/config/middleware/pylons_app.py
--- a/ckan/config/middleware/pylons_app.py
+++ b/ckan/config/middleware/pylons_app.py
@@ -138,10 +138,7 @@ def make_pylons_stack(conf, full_stack=True, static_files=True,
     )
 
     # Establish the Registry for this application
-    # The RegistryManager includes code to pop
-    # registry values after the stream has completed,
-    # so we need to prevent this with `streaming` set to True.
-    app = RegistryManager(app, streaming=True)
+    app = RegistryManager(app, streaming=False)
 
     if asbool(static_files):
         # Serve static files


Test patch: