Repository: mozilla/bugbug

Issue:
Make the models cache an actual cache
Right now, it's a simple dict. We preload all models and keep them in memory forever.
We should instead only preload the ones we use often, and unload models from the cache when they haven't been used for a while.


Discussion:
Also, not cache the seldomly requested ones.
So we could do the following:
- [x] cache a model only if it was loaded at least X times in two hours (maybe starting with a low value, like X = 2);
- [ ] set a TTL of two hours for cache entries;
- [x] preload only the component model, which is used hourly by relman-auto-nag (and in the future the test selection models).
I'm interested in working on this, and curious as to the motivation for this enhancement. Are there any specific operational issues the service is currently experiencing, or that you foresee it experiencing, if this enhancement is not implemented and the service keeps loading all models into a dict?
> I'm interested in working on this

@mvkski better to start with an issue with the good-first-bug label: https://github.com/mozilla/bugbug/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-bug.
You can still work on this if you want of course, but it might not be as easy as those.
NOTE: The only thing missing here is removing items from the cache after two hours.

> , and curious as to the motivation for this enhancement. Are there any specific operational issues the service is currently experiencing, or that you foresee it experiencing, if this enhancement is not implemented and the service keeps loading all models into a dict?

The reason behind this change is reducing memory usage of the HTTP service. It is currently high, and will be too high if we add more models.
@marco-c Thanks! I'd like to take a crack at this task for now, but will take a step back if it doesn't work out. I think I have a handle on what needs to be done here

 If I understand correctly, the dict on the following line:
https://github.com/mozilla/bugbug/blob/22b27b97b0a7ec4d84c8b374cb56c2c5f3f9d734/http_service/bugbug_http/models.py#L41
Needs to be switched out from a dict to a cache supporting a TTL. Currently I think https://cachetools.readthedocs.io/en/stable/#cachetools.TTLCache with a TTL parameter of two hours would do the trick, but am still researching (not sure if this TTL will refresh when a model is retrieved). 

I had more questions on what problems specifically you might expect to see due to high memory usage, but I think I'll refrain from taking up too much of your attention with them right now(I might feel emboldened to ask if a PR of mine for this issue is accepted 🙂)
>  Needs to be switched out from a dict to a cache supporting a TTL. Currently I think https://cachetools.readthedocs.io/en/stable/#cachetools.TTLCache with a TTL parameter of two hours would do the trick, but am still researching (not sure if this TTL will refresh when a model is retrieved).

I've looked at cachetools, but it looks like the cache is only cleaned up when you actually access it, so it doesn't help a lot by itself in reducing memory usage. We could set up a background thread which periodically performs the cleanup.

> I had more questions on what problems specifically you might expect to see due to high memory usage, but I think I'll refrain from taking up too much of your attention with them right now(I might feel emboldened to ask if a PR of mine for this issue is accepted slightly_smiling_face)

Simply: running out of available memory :)
On a Heroku `Standard 2x` worker we only have 1 GB available, so we have to use it carefully.

> I've looked at cachetools, but it looks like the cache is only cleaned up when you actually access it, so it doesn't help a lot by itself in reducing memory usage. We could set up a background thread which periodically performs the cleanup.
 
Thanks for catching that! If we have a cleanup thread run only once every two hours it seems to me like it would solve the problem.  If I understand correctly, the problematic nature of the [global interpreter lock](https://wiki.python.org/moin/GlobalInterpreterLock) applies to this situation, but since the cleanup thread would only need to run very infrequently negative performance effects on inference should be negligible. 
 
> Simply: running out of available memory :)
> On a Heroku `Standard 2x` worker we only have 1 GB available, so we have to use it carefully.

That makes a lot of sense! Thanks - its very satisfying to work on something when you have full visibility into why its needed 😃.
> Thanks for catching that! If we have a cleanup thread run only once every two hours it seems to me like it would solve the problem. If I understand correctly, the problematic nature of the global interpreter lock applies to this situation, but since the cleanup thread would only need to run very infrequently negative performance effects on inference should be negligible.

Indeed!
Also, If we're going to be writing our own cleanup thread, it seems like we may as well go the distance and implement our own TTL cache from scratch rather than take on an external dependancy with more features than we need.

It shouldn't be too much extra code to do so (I'm estimating 20-50 lines?). Having a small amount of extra code in the project seems to me like a fair price to pay for the cost of reducing the chance we end up with [dependancy hell](https://en.wikipedia.org/wiki/Dependency_hell) related problems.
It sounds good to me, we already partially have it now (see https://github.com/mozilla/bugbug/blob/b72e96b269478cdce2b6aceef4beb190238c0eea/http_service/bugbug_http/models.py#L40 and https://github.com/mozilla/bugbug/blob/b72e96b269478cdce2b6aceef4beb190238c0eea/http_service/bugbug_http/models.py#L67-L71), so we just need the cleanup thread to go through the entries and remove stale ones.
Created PR for the issue. Wrote unit tests and an integration test for the cache, but haven't verified end to end yet due to having issues launching http_service locally. Asking in matrix how to launch service.

Resolving patch:
diff --git a/http_service/bugbug_http/download_models.py b/http_service/bugbug_http/download_models.py
--- a/http_service/bugbug_http/download_models.py
+++ b/http_service/bugbug_http/download_models.py
@@ -3,15 +3,30 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
+import logging
+
 from bugbug import db, repository, test_scheduling, utils
-from bugbug_http.models import MODELS_NAMES, get_model
+from bugbug_http import ALLOW_MISSING_MODELS
+from bugbug_http.models import MODEL_CACHE, MODELS_NAMES
+
+LOGGER = logging.getLogger()
 
 
 def download_models():
     for model_name in MODELS_NAMES:
         utils.download_model(model_name)
         # Try loading the model
-        get_model(model_name)
+        try:
+            MODEL_CACHE.get(model_name)
+        except FileNotFoundError:
+            if ALLOW_MISSING_MODELS:
+                LOGGER.info(
+                    "Missing %r model, skipping because ALLOW_MISSING_MODELS is set"
+                    % model_name
+                )
+                return None
+            else:
+                raise
 
     db.download_support_file(
         test_scheduling.TEST_LABEL_SCHEDULING_DB,
diff --git a/http_service/bugbug_http/models.py b/http_service/bugbug_http/models.py
--- a/http_service/bugbug_http/models.py
+++ b/http_service/bugbug_http/models.py
@@ -6,20 +6,18 @@
 import logging
 import math
 import os
-from datetime import datetime
-from typing import Dict
+from datetime import timedelta
 
 import numpy as np
 import orjson
 import requests
-from dateutil.relativedelta import relativedelta
 from redis import Redis
 
 from bugbug import bugzilla, commit_features, repository, test_scheduling
 from bugbug.model import Model
 from bugbug.models import load_model
 from bugbug.utils import get_hgmo_stack
-from bugbug_http import ALLOW_MISSING_MODELS
+from bugbug_http.readthrough_cache import ReadthroughTTLCache
 
 logging.basicConfig(level=logging.INFO)
 LOGGER = logging.getLogger()
@@ -33,41 +31,14 @@
     "testlabelselect",
     "testgroupselect",
 ]
-DEFAULT_EXPIRATION_TTL = 7 * 24 * 3600  # A week
-
-
-MODEL_LAST_LOADED: Dict[str, datetime] = {}
-MODEL_CACHE: Dict[str, Model] = {}
-
 
+DEFAULT_EXPIRATION_TTL = 7 * 24 * 3600  # A week
 redis = Redis.from_url(os.environ.get("REDIS_URL", "redis://localhost/0"))
 
-
-def get_model(model_name):
-    if model_name not in MODEL_CACHE:
-        LOGGER.info("Recreating the %r model in cache" % model_name)
-        try:
-            model = load_model(model_name)
-        except FileNotFoundError:
-            if ALLOW_MISSING_MODELS:
-                LOGGER.info(
-                    "Missing %r model, skipping because ALLOW_MISSING_MODELS is set"
-                    % model_name
-                )
-                return None
-            else:
-                raise
-
-        # Cache the model only if it was last used less than one hour ago.
-        if model_name in MODEL_LAST_LOADED and MODEL_LAST_LOADED[
-            model_name
-        ] > datetime.now() - relativedelta(hours=1):
-            MODEL_CACHE[model_name] = model
-    else:
-        model = MODEL_CACHE[model_name]
-
-    MODEL_LAST_LOADED[model_name] = datetime.now()
-    return model
+MODEL_CACHE: ReadthroughTTLCache[str, Model] = ReadthroughTTLCache(
+    timedelta(hours=1), load_model
+)
+MODEL_CACHE.start_ttl_thread()
 
 
 def setkey(key, value, expiration=DEFAULT_EXPIRATION_TTL):
@@ -99,7 +70,7 @@ def classify_bug(model_name, bug_ids, bugzilla_token):
     if not bugs:
         return "NOK"
 
-    model = get_model(model_name)
+    model = MODEL_CACHE.get(model_name)
 
     if not model:
         LOGGER.info("Missing model %r, aborting" % model_name)
@@ -183,7 +154,7 @@ def get_runnables(granularity):
             commit_test["test_job"] = data
             commit_tests.append(commit_test)
 
-        probs = get_model(f"test{granularity}select").classify(
+        probs = MODEL_CACHE.get(f"test{granularity}select").classify(
             commit_tests, probabilities=True
         )
         selected_indexes = np.argwhere(probs[:, 1] > test_selection_threshold)[:, 0]
diff --git a/http_service/bugbug_http/readthrough_cache.py b/http_service/bugbug_http/readthrough_cache.py
new file mode 100644
--- /dev/null
+++ b/http_service/bugbug_http/readthrough_cache.py
@@ -0,0 +1,74 @@
+# -*- coding: utf-8 -*-
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this file,
+# You can obtain one at http://mozilla.org/MPL/2.0/.
+
+
+import datetime
+import logging
+import threading
+import time
+from datetime import timedelta
+from typing import Callable, Dict, Generic, TypeVar
+
+LOGGER = logging.getLogger()
+
+# A simple TTL cache to use with models. Because we expect the number of models
+# in the service to not be very large, simplicity of implementation is
+# preferred to algorithmic efficiency of operations.
+#
+# Called an 'Idle' TTL cache because TTL of items is reset after every get
+Key = TypeVar("Key")
+Value = TypeVar("Value")
+
+
+class ReadthroughTTLCache(Generic[Key, Value]):
+    def __init__(self, ttl: timedelta, load_item_function: Callable[[Key], Value]):
+        self.ttl = ttl
+        self.load_item_function = load_item_function
+        self.items_last_accessed: Dict[Key, datetime.datetime] = {}
+        self.items_storage: Dict[Key, Value] = {}
+
+    def __contains__(self, key):
+        return key in self.items_storage
+
+    def get(self, key, force_store=False):
+        store_item = force_store
+        if key in self.items_storage:
+            item = self.items_storage[key]
+        else:
+            item = self.load_item_function(key)
+            # Cache the item only if it was last accessed within the past TTL seconds
+            # Note that all entries in items_last_accessed are purged if item was not
+            # accessed in the last TTL seconds.
+            if key in self.items_last_accessed:
+                store_item = True
+
+        self.items_last_accessed[key] = datetime.datetime.now()
+        if store_item:
+            LOGGER.info(
+                f"Storing item with the following key in readthroughcache: {key}"
+            )
+            self.items_storage[key] = item
+
+        return item
+
+    def purge_expired_entries(self):
+        purge_entries_before = datetime.datetime.now() - self.ttl
+        for key, time_last_touched in list(self.items_last_accessed.items()):
+            if time_last_touched < purge_entries_before:
+                LOGGER.info(
+                    f"Evicting item with the following key from readthroughcache: {key}"
+                )
+                del self.items_last_accessed[key]
+                del self.items_storage[key]
+
+    def start_ttl_thread(self):
+        def purge_expired_entries_with_wait():
+            while True:
+                self.purge_expired_entries()
+                time.sleep(self.ttl.total_seconds())
+
+        thread = threading.Thread(target=purge_expired_entries_with_wait)
+        thread.setDaemon(True)
+        thread.start()


Test patch:
diff --git a/http_service/tests/conftest.py b/http_service/tests/conftest.py
--- a/http_service/tests/conftest.py
+++ b/http_service/tests/conftest.py
@@ -322,9 +322,10 @@ def classify(self, items, probabilities=False):
                             results.append([0.9, 0.1])
                 return np.array(results)
 
-        def mock_get_model(modelname):
-            return Model(modelname)
+        class MockModelCache:
+            def get(self, model_name):
+                return Model(model_name)
 
-        monkeypatch.setattr(bugbug_http.models, "get_model", mock_get_model)
+        monkeypatch.setattr(bugbug_http.models, "MODEL_CACHE", MockModelCache())
 
     return do_mock
diff --git a/http_service/tests/test_readthrough_cache.py b/http_service/tests/test_readthrough_cache.py
new file mode 100644
--- /dev/null
+++ b/http_service/tests/test_readthrough_cache.py
@@ -0,0 +1,175 @@
+# -*- coding: utf-8 -*-
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this file,
+# You can obtain one at http://mozilla.org/MPL/2.0/.
+
+
+import time
+from datetime import datetime, timedelta
+from unittest.mock import MagicMock, patch
+
+from bugbug_http.readthrough_cache import ReadthroughTTLCache
+
+
+class MockDatetime:
+    def __init__(self, mock_now):
+        self.mock_now = mock_now
+
+    def now(self):
+        return self.mock_now
+
+    def set_now(self, new_mock_now):
+        self.mock_now = new_mock_now
+
+
+class MockSleep:
+    def __init__(self, mock_datetime):
+        self.mock_datetime = mock_datetime
+        self.wakeups_count = 0
+
+        # count variable used to ensure that context switch into
+        # thread calling sleep occurred
+
+    def sleep(self, sleep_seconds):
+        self.wakeup_time = self.mock_datetime.now() + timedelta(seconds=sleep_seconds)
+        while self.mock_datetime.now() < self.wakeup_time:
+            pass
+        self.wakeups_count += 1
+
+
+class PurgeCountCache(ReadthroughTTLCache):
+    def __init__(self, ttl, load_item_function):
+        super().__init__(ttl, load_item_function)
+        self.purge_count = 0
+
+        def purge_then_increment(purge):
+            purge()
+            self.purge_count += 1
+
+        super().purge_expired_entries = purge_then_increment(
+            super().purge_expired_entries
+        )
+
+
+def test_doesnt_cache_unless_accessed_within_ttl():
+    mockdatetime = MockDatetime(datetime(2019, 4, 1, 10))
+    cache = ReadthroughTTLCache(timedelta(hours=4), lambda x: "payload")
+
+    with patch("datetime.datetime", mockdatetime):
+        cache.get("key_a")
+
+        # after one hour
+        mockdatetime.set_now(datetime(2019, 4, 1, 11))
+        assert "key_a" not in cache
+
+        # after two hours
+        mockdatetime.set_now(datetime(2019, 4, 1, 12))
+        cache.get("key_a")
+
+        # after three hours
+        mockdatetime.set_now(datetime(2019, 4, 1, 13))
+        assert "key_a" in cache
+
+
+def test_cache_purges_after_ttl():
+    mockdatetime = MockDatetime(datetime(2019, 4, 1, 10))
+    cache = ReadthroughTTLCache(timedelta(hours=2), lambda x: "payload")
+
+    with patch("datetime.datetime", mockdatetime):
+        cache.get("key_a", force_store=True)
+
+        # after one hour
+        mockdatetime.set_now(datetime(2019, 4, 1, 11))
+        cache.purge_expired_entries()
+        assert "key_a" in cache
+
+        # after two hours one minute
+        mockdatetime.set_now(datetime(2019, 4, 1, 12, 1))
+        cache.purge_expired_entries()
+        assert "key_a" not in cache
+
+
+def test_cache_ttl_refreshes_after_get():
+    mockdatetime = MockDatetime(datetime(2019, 4, 1, 10))
+    cache = ReadthroughTTLCache(timedelta(hours=2), lambda x: "payload")
+
+    with patch("datetime.datetime", mockdatetime):
+        cache.get("key_a", force_store=True)
+
+        # after one hour
+        mockdatetime.set_now(datetime(2019, 4, 1, 11))
+        cache.purge_expired_entries()
+        assert "key_a" in cache
+        assert cache.get("key_a") == "payload"
+
+        # after three hours
+        mockdatetime.set_now(datetime(2019, 4, 1, 13))
+        cache.purge_expired_entries()
+        assert "key_a" in cache
+
+        # after three hours one minute
+        mockdatetime.set_now(datetime(2019, 4, 1, 13, 1))
+        cache.purge_expired_entries()
+        assert "key_a" not in cache
+
+
+def test_force_store():
+    def with_spied_storage(cache):
+        cache.storage_access_count = 0
+        cache_getitem = cache.items_storage.__getitem__
+
+        def spy_getitem(key):
+            cache.storage_access_count += 1
+            return cache_getitem(key)
+
+        mock_items_storage = MagicMock()
+        mock_items_storage.__getitem__.side_effect = spy_getitem
+        mock_items_storage.__setitem__.side_effect = cache.items_storage.__setitem__
+        mock_items_storage.__contains__.side_effect = cache.items_storage.__contains__
+        cache.items_storage = mock_items_storage
+        return cache
+
+    cache = with_spied_storage(
+        ReadthroughTTLCache(timedelta(hours=2), lambda x: "payload")
+    )
+    cache.get("key_a", force_store=True)
+
+    assert "key_a" in cache
+    assert cache.get("key_a") == "payload"
+    assert cache.storage_access_count == 1
+
+
+def test_cache_thread():
+    def cache_with_purge_count(cache):
+        cache.purge_count = 0
+        purge = cache.purge_expired_entries
+
+        def purge_then_increment():
+            purge()
+            cache.purge_count += 1
+
+        cache.purge_expired_entries = purge_then_increment
+        return cache
+
+    mockdatetime = MockDatetime(datetime(2019, 4, 1, 10))
+    cache = cache_with_purge_count(
+        ReadthroughTTLCache(timedelta(hours=2), lambda x: "payload")
+    )
+    mocksleep = MockSleep(mockdatetime)
+    with patch("datetime.datetime", mockdatetime):
+        with patch("time.sleep", mocksleep.sleep):
+            cache.get("key_a", force_store=True)
+            cache.start_ttl_thread()
+
+            # after one hour
+            mockdatetime.set_now(datetime(2019, 4, 1, 11))
+            assert "key_a" in cache
+            assert mocksleep.wakeups_count == 0
+
+            # after two hours one minute
+            before_timechange_purge_count = cache.purge_count
+            mockdatetime.set_now(datetime(2019, 4, 1, 12, 1))
+
+            while cache.purge_count == before_timechange_purge_count:
+                time.sleep(0)
+            assert "key_a" not in cache