Repository: ckan/ckan

Issue:
Unsustainable Beaker cache growth on CKAN 2.9
**CKAN version**

2.9 (tested on 2.9.0 and 2.9.5)

Does not occur on 2.8.8.

**Describe the bug**

Beaker session cache files are created on every unauthenticated request to non-API URLs. This did not occur under CKAN 2.8.

**Steps to reproduce**

* Create a CKAN 2.9 instance on a Linux system.
* Check total inode usage with `df -i`
* Send a request to /dataset, eg `curl http://my-site/dataset/`
* Check total inode usage again and confirm that it has risen by 2.
* Check the session files in `/tmp/*/sessions` to confirm that two new files have been created, `find /tmp/*/sessions -type f -cmin -5`

**Expected behavior**

Unauthenticated requests should not create Beaker session files, as per CKAN 2.8 behaviour.

**Additional details**

This can easily lead to hundreds of thousands of files being created per day, rapidly reaching the inode limit of the hard disk. It could also be deliberately exploited as a Denial of Service attack.



Discussion:
Further testing shows that CKAN 2.8 does create session files for requests that go to Flask, such as `/user` or `/ckan-admin`
We're looking at resolving this on our own system by using Redis as our Beaker storage backend, with a suitable session expiry.

    beaker.session.timeout = 3600
    beaker.session.type = ext:redis
    beaker.session.url = <same as ckan.redis.url>

something from https://github.com/ckan/ckan/issues/3196
and looking at 
https://flask-session.readthedocs.io/en/latest/#quickstart

it seems a 'session' is created on each request and anon is for the single request and thrown. Since flask session has beaker backing it, it still creates a file, from what i can see, ckan could be tweaked to give anon a cookie/session that sticks around and reduces this.

I think for @ThrawnCA problem, with the 2.8.x to 2.9.x upgrade, search engines are now doing a massive re-crawl and generating heaps of hits and doing active probing compared to the usual passive (check if page changed etc)

Found the culprit!

Here's the step by step rundown:

* At the end of each request, Flask checks if there is something in the session, and if so, saves it (in our case this means that Beaker will store a session file):

    https://github.com/pallets/flask/blob/0305d31cb8c62b26c1e073995114963c88ca962e/src/flask/app.py#L1887-L1888

* We would assume that in an anonymous request where nothing has been stored in the session, calling `self.session_interface.is_null_session(ctx.session)` would return True, but that is not the case. If we inspect the session we see it contains data:
    ```
    {'_accessed_time': 1656666227.368082, '_creation_time': 1656666227.368082}
    ```
* These entries are _always_ added by Beaker when initializing the session class:

    https://github.com/bbangert/beaker/blob/d4ab7d37387feddd0d544792d9e32533e39ff868/beaker/session.py#L222-L224

* The cleanest approach I can think of is to override the `is_null_session()` method in our custom `SessionInterface` class to take this into account:

```diff
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
index eb9ff8155..ef6b8b63e 100644
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -174,6 +174,17 @@ def make_flask_stack(conf):
         def save_session(self, app, session, response):
             session.save()
 
+        def is_null_session(self, obj):
+
+            is_null = super(BeakerSessionInterface, self).is_null_session(obj)
+
+            if not is_null:
+                # Beaker always adds these keys on each request, so if these are
+                # the only keys present we assume it's an empty session
+                is_null = sorted(obj.keys()) == ["_accessed_time", "_creation_time"]
+
+            return is_null
+
     namespace = 'beaker.session.'
     session_opts = {k.replace('beaker.', ''): v
                     for k, v in six.iteritems(config)

```

This was already happening on CKAN 2.8 but because there weren't many Flask-based endpoints the issue was not made apparent. On CKAN 2.9 where everything goes trough Flask the sesssions were being stored in every single request. We definitely need to backport this. I'll send a PR


Good trouble shooting. Sounds like an easy fix 😊

Resolving patch:
diff --git a/ckan/config/middleware/flask_app.py b/ckan/config/middleware/flask_app.py
--- a/ckan/config/middleware/flask_app.py
+++ b/ckan/config/middleware/flask_app.py
@@ -10,7 +10,7 @@
 import logging
 
 from logging.handlers import SMTPHandler
-from typing import Any, Iterable, Optional, Union, cast
+from typing import Any, Iterable, Optional, Union, cast, Dict
 
 from flask import Blueprint, send_from_directory
 from flask.ctx import _AppCtxGlobals
@@ -126,6 +126,29 @@ def _ungettext_alias():
     return dict(ungettext=ungettext)
 
 
+class BeakerSessionInterface(SessionInterface):
+    def open_session(self, app: Any, request: Any):
+        if 'beaker.session' in request.environ:
+            return request.environ['beaker.session']
+
+    def save_session(self, app: Any, session: Any, response: Any):
+        session.save()
+
+    def is_null_session(self, obj: Dict[str, Any]) -> bool:
+
+        is_null = super(BeakerSessionInterface, self).is_null_session(obj)
+
+        if not is_null:
+            # Beaker always adds these keys on each request, so if these are
+            # the only keys present we assume it's an empty session
+            is_null = (
+                sorted(obj.keys()) == [
+                    "_accessed_time", "_creation_time", "_domain", "_path"]
+            )
+
+        return is_null
+
+
 def make_flask_stack(conf: Union[Config, CKANConfig]) -> CKANApp:
     """ This has to pass the flask app through all the same middleware that
     Pylons used """
@@ -200,15 +223,6 @@ def make_flask_stack(conf: Union[Config, CKANConfig]) -> CKANApp:
         from werkzeug.debug import DebuggedApplication
         app.wsgi_app = DebuggedApplication(app.wsgi_app, True)
 
-    # Use Beaker as the Flask session interface
-    class BeakerSessionInterface(SessionInterface):
-        def open_session(self, app: Any, request: Any):
-            if 'beaker.session' in request.environ:
-                return request.environ['beaker.session']
-
-        def save_session(self, app: Any, session: Any, response: Any):
-            session.save()
-
     namespace = 'beaker.session.'
     session_opts = {k.replace('beaker.', ''): v
                     for k, v in config.items()


Test patch:
diff --git a/ckan/tests/config/test_middleware.py b/ckan/tests/config/test_middleware.py
--- a/ckan/tests/config/test_middleware.py
+++ b/ckan/tests/config/test_middleware.py
@@ -1,8 +1,9 @@
 # encoding: utf-8
 
 import pytest
-import six
+from unittest import mock
 from flask import Blueprint
+from ckan.config.middleware.flask_app import BeakerSessionInterface
 
 import ckan.plugins as p
 from ckan.common import config, _
@@ -53,7 +54,7 @@ def test_flask_core_route_is_served(patched_app):
     assert res.status_code == 200
 
     res = patched_app.get(u"/flask_core")
-    assert six.ensure_text(res.data) == u"This was served from Flask"
+    assert res.get_data(as_text=True) == "This was served from Flask"
 
 
 @pytest.mark.ckan_config(u"SECRET_KEY", u"super_secret_stuff")
@@ -75,3 +76,20 @@ def test_no_beaker_secret_crashes(make_app):
     # RuntimeError instead (thrown on `make_flask_stack`)
     with pytest.raises(RuntimeError):
         make_app()
+
+
+def test_no_session_stored_by_default(app, monkeypatch, test_request_context):
+
+    save_session_mock = mock.Mock()
+
+    class CustomBeakerSessionInterface(BeakerSessionInterface):
+
+        save_session = save_session_mock
+
+    monkeypatch.setattr(
+        app.flask_app, 'session_interface', CustomBeakerSessionInterface())
+
+    with test_request_context():
+        app.get("/")
+
+    save_session_mock.assert_not_called()