Repository: WordPress/openverse-api

Issue:
Count provider occurrences in executed query results
## Problem

<!-- Describe a problem solved by this feature; or delete the section entirely. -->
We do not currently have any visibility into the number of times a provider shows up in search queries or how many queries they show up in period.

When considering the concerns @AetherUnbound brought up in [this Make post about iNaturalist](https://make.wordpress.org/openverse/2023/01/14/preparing-for-inaturalist/), it became clear that we need to have visibility into these metrics so that we know how dramatic changes to existing providers and the addition of new providers is actually reflected in real user searches.

## Description

<!-- Describe the feature and how it solves the problem. -->
In the `search_controller.search` method, before returning the search results, create a tally of the providers for each result present in the page being returned:

```py
provider_occurrences = defaultdict(int)
for result in results:
    provider_occurences[result["provider"]] += 1

for provider, occurrences in provider_occurrences:
    redis.incr(f"provider_occurrences:{week_timestamp}:{provider}", occurrences)
    redis.incr(f"provider_appeared_in_searches:{week_timestamp}:{provider}", 1)
```

This will allow us to see (a) how many searches each provider actually shows up in over the course of a week and (b) how many results from each provider were shown in search. If we can collect this data before iNaturalist data is fully available in the API, then we'll be able to directly see the effect of adding iNaturalist, both in the count of how many iNaturalist results start to show up _and_ in and potential decreases in result counts for other providers.

## Alternatives

<!-- Describe any alternative solutions or features you have considered. How is this feature better? -->
Store the entire list of result ids returned for all queries in a time series. This would allow us to do more different kinds of analysis long term but would take way longer for us to get usable results from for the purposes of measuring the impact of iNaturalist on the actual user query results.

Set up a `tallies` cache alias for tracking longer-term values
## Problem

<!-- Describe a problem solved by this feature; or delete the section entirely. -->

We are presently using the default `django.core.cache` alias for tracking view caches & authentication information _in addition to_ general tallies and aggregates we wish to track. This means that we have both cache and (ostensibly) longer-term data living in the same Redis database, and (unless explicitly adjusted) existing under the same TTLs.

## Description

<!-- Describe the feature and how it solves the problem. -->

We would like to set up a separate cache alias, `tallies`, which could be used to store longer-living values. The two examples that come to mind would be the dead link tallies that are accumulated in `validate_images`, and the thumbnail domain timeouts that were added in #982.

This alias could have a longer (or nonexistent) TTL, with the option to define individual timeouts when writing to the cache.

We should also check through other uses of the `cache` and swap any uses that are not view caches & authentication info to use `tallies`.




Discussion:

> This alias could have a longer (or nonexistent) TTL, with the option to define individual timeouts when writing to the cache.

I think it should specifically have no TTL for this usecase and if entries need to be timestamped (or a record kept of "last updated") then secondary keys can be used to accomplish that.

The other use of cache that exists in the validate images function is the dead link mask. This needs to retain its existing behaviour and uses a hand-configured TTL. Just want to note that for whomever works on this issue: be sure to keep an eye out for not accidentally changing the behaviour of the dead link masks :slightly_smiling_face: 

Resolving patch:
diff --git a/api/catalog/api/controllers/search_controller.py b/api/catalog/api/controllers/search_controller.py
--- a/api/catalog/api/controllers/search_controller.py
+++ b/api/catalog/api/controllers/search_controller.py
@@ -17,6 +17,7 @@
 from elasticsearch_dsl.response import Hit, Response
 
 import catalog.api.models as models
+from catalog.api.utils import tallies
 from catalog.api.utils.dead_link_mask import get_query_hash, get_query_mask
 from catalog.api.utils.validate_images import validate_images
 
@@ -412,6 +413,12 @@ def search(
     result_count, page_count = _get_result_and_page_count(
         search_response, results, page_size, page
     )
+
+    if results and page <= 4:
+        # We ignore tallies for deep pages because they're not likely to
+        # be as important for search relevancy for most users at this point
+        tallies.count_provider_occurrences(results, index)
+
     return results or [], page_count, result_count
 
 
diff --git a/api/catalog/api/utils/tallies.py b/api/catalog/api/utils/tallies.py
new file mode 100644
--- /dev/null
+++ b/api/catalog/api/utils/tallies.py
@@ -0,0 +1,39 @@
+from collections import defaultdict
+from datetime import datetime
+
+import django_redis
+from django_redis.client.default import Redis
+
+
+def _get_weekly_timestamp() -> str:
+    """Get a timestamp for the Monday of any given week."""
+    now = datetime.now()
+    return datetime(
+        now.year,
+        now.month,
+        # Set the day to Monday of the current week
+        now.day - now.weekday(),
+    ).strftime("%Y-%m-%d")
+
+
+def count_provider_occurrences(results: list[dict], index: str) -> None:
+    # Use ``get_redis_connection`` rather than Django's caches
+    # so that we can open a pipeline rather than sending off ``n``
+    # writes and because the RedisPy client's ``incr`` method
+    # is safe by default rather than Django's handspun method which:
+    # 1. Takes two requests to execute; and
+    # 2. Raises a ``ValueError`` if the key doesn't exist rather than
+    # just initialising the key to the value like Redis's behaviour.
+    tallies: Redis = django_redis.get_redis_connection("tallies")
+
+    provider_occurrences = defaultdict(int)
+    for result in results:
+        provider_occurrences[result["provider"]] += 1
+
+    week = _get_weekly_timestamp()
+    with tallies.pipeline() as pipe:
+        for provider, occurrences in provider_occurrences.items():
+            pipe.incr(f"provider_occurrences:{index}:{week}:{provider}", occurrences)
+            pipe.incr(f"provider_appeared_in_searches:{index}:{week}:{provider}", 1)
+
+        pipe.execute()
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -178,32 +178,31 @@
 REDIS_HOST = config("REDIS_HOST", default="localhost")
 REDIS_PORT = config("REDIS_PORT", default=6379, cast=int)
 REDIS_PASSWORD = config("REDIS_PASSWORD", default="")
-CACHES = {
-    # Site cache writes to 'default'
-    "default": {
+
+
+def _make_cache_config(dbnum: int, **overrides) -> dict:
+    return {
         "BACKEND": "django_redis.cache.RedisCache",
-        "LOCATION": f"redis://{REDIS_HOST}:{REDIS_PORT}/0",
+        "LOCATION": f"redis://{REDIS_HOST}:{REDIS_PORT}/{dbnum}",
         "OPTIONS": {
             "CLIENT_CLASS": "django_redis.client.DefaultClient",
-        },
-    },
+        }
+        | overrides.pop("OPTIONS", {}),
+    } | overrides
+
+
+CACHES = {
+    # Site cache writes to 'default'
+    "default": _make_cache_config(0),
     # For rapidly changing stats that we don't want to hammer the database with
-    "traffic_stats": {
-        "BACKEND": "django_redis.cache.RedisCache",
-        "LOCATION": f"redis://{REDIS_HOST}:{REDIS_PORT}/1",
-        "OPTIONS": {
-            "CLIENT_CLASS": "django_redis.client.DefaultClient",
-        },
-    },
+    "traffic_stats": _make_cache_config(1),
     # For ensuring consistency among multiple Django workers and servers.
     # Used by Redlock.
-    "locks": {
-        "BACKEND": "django_redis.cache.RedisCache",
-        "LOCATION": f"redis://{REDIS_HOST}:{REDIS_PORT}/2",
-        "OPTIONS": {
-            "CLIENT_CLASS": "django_redis.client.DefaultClient",
-        },
-    },
+    "locks": _make_cache_config(2),
+    # Used for tracking tallied figures that shouldn't expire and are indexed
+    # with a timestamp range (for example, the key could a timestamp valid
+    # for a given week), allowing historical data analysis.
+    "tallies": _make_cache_config(3, TIMEOUT=None),
 }
 
 # If key is not present then the authentication header won't be sent


Test patch:
diff --git a/api/test/unit/conftest.py b/api/test/unit/conftest.py
--- a/api/test/unit/conftest.py
+++ b/api/test/unit/conftest.py
@@ -1,6 +1,6 @@
 from unittest.mock import MagicMock
 
-from rest_framework.test import APIClient
+from rest_framework.test import APIClient, APIRequestFactory
 
 import pytest
 
@@ -16,3 +16,10 @@ def capture_exception(monkeypatch):
     monkeypatch.setattr("sentry_sdk.capture_exception", mock)
 
     yield mock
+
+
+@pytest.fixture
+def request_factory() -> APIRequestFactory():
+    request_factory = APIRequestFactory(defaults={"REMOTE_ADDR": "192.0.2.1"})
+
+    return request_factory
diff --git a/api/test/unit/controllers/test_search_controller.py b/api/test/unit/controllers/test_search_controller.py
--- a/api/test/unit/controllers/test_search_controller.py
+++ b/api/test/unit/controllers/test_search_controller.py
@@ -9,6 +9,8 @@
 from elasticsearch_dsl import Search
 
 from catalog.api.controllers import search_controller
+from catalog.api.serializers.media_serializers import MediaSearchRequestSerializer
+from catalog.api.utils import tallies
 from catalog.api.utils.dead_link_mask import get_query_hash, save_query_mask
 
 
@@ -377,3 +379,61 @@ def test_paginate_with_dead_link_mask_query_mask_overlaps_query_window(
     assert (
         actual_range == expected_range
     ), f"expected {expected_range} but got {actual_range}"
+
+
+MOCK_RESULTS = [{"provider": "a provider", "identifier": i} for i in range(20)]
+
+
+@pytest.mark.parametrize(
+    "index",
+    (
+        "image",
+        "audio",
+    ),
+)
+@pytest.mark.parametrize(
+    ("page", "does_tally"),
+    (
+        (1, True),
+        (2, True),
+        (3, True),
+        (4, True),
+        (5, False),
+    ),
+)
+@mock.patch.object(
+    tallies, "count_provider_occurrences", wraps=tallies.count_provider_occurrences
+)
+@mock.patch(
+    "catalog.api.controllers.search_controller._post_process_results",
+    return_value=MOCK_RESULTS,
+)
+@pytest.mark.django_db
+def test_search_tallies_pages_less_than_5(
+    _,
+    count_provider_occurrences_mock: mock.MagicMock,
+    page,
+    does_tally,
+    index,
+    request_factory,
+):
+    serializer = MediaSearchRequestSerializer(data={"q": "dogs"})
+    serializer.is_valid()
+
+    search_controller.search(
+        search_params=serializer,
+        ip=0,
+        index=index,
+        page=page,
+        page_size=20,
+        request=request_factory.get("/"),
+        filter_dead=False,
+    )
+
+    if does_tally:
+        count_provider_occurrences_mock.assert_called_once_with(
+            mock.ANY,
+            index,
+        )
+    else:
+        count_provider_occurrences_mock.assert_not_called()
diff --git a/api/test/unit/serializers/media_serializers_test.py b/api/test/unit/serializers/media_serializers_test.py
--- a/api/test/unit/serializers/media_serializers_test.py
+++ b/api/test/unit/serializers/media_serializers_test.py
@@ -4,7 +4,7 @@
 
 from django.conf import settings
 from rest_framework.exceptions import NotAuthenticated, ValidationError
-from rest_framework.test import APIRequestFactory, force_authenticate
+from rest_framework.test import force_authenticate
 from rest_framework.views import APIView
 
 import pytest
@@ -14,15 +14,6 @@
 from catalog.api.serializers.media_serializers import MediaSearchRequestSerializer
 
 
-# TODO: @sarayourfriend consolidate these with the other
-# request factory fixtures into conftest.py
-@pytest.fixture
-def request_factory() -> APIRequestFactory():
-    request_factory = APIRequestFactory(defaults={"REMOTE_ADDR": "192.0.2.1"})
-
-    return request_factory
-
-
 @pytest.fixture
 def access_token():
     token = AccessTokenFactory.create()
diff --git a/api/test/unit/utils/tallies_test.py b/api/test/unit/utils/tallies_test.py
new file mode 100644
--- /dev/null
+++ b/api/test/unit/utils/tallies_test.py
@@ -0,0 +1,113 @@
+from datetime import datetime
+
+import pytest
+from freezegun import freeze_time
+
+from catalog.api.utils import tallies
+
+
+# Use a fake media type to avoid having to run each test case for each media type
+FAKE_MEDIA_TYPE = "this_is_not_a_media_type"
+
+
+@pytest.mark.parametrize(
+    ("now", "expected_timestamp"),
+    (
+        pytest.param(datetime(2023, 1, 19), "2023-01-16", id="midweek"),
+        pytest.param(datetime(2023, 1, 16), "2023-01-16", id="start_of_week"),
+        pytest.param(datetime(2023, 1, 15), "2023-01-09", id="end_of_week"),
+    ),
+)
+def test_count_provider_occurrences_uses_week_timestamp(now, expected_timestamp, redis):
+    results = [{"provider": "flickr"} for _ in range(4)] + [
+        {"provider": "stocksnap"} for _ in range(6)
+    ]
+    with freeze_time(now):
+        tallies.count_provider_occurrences(results, FAKE_MEDIA_TYPE)
+
+    assert (
+        redis.get(f"provider_occurrences:{FAKE_MEDIA_TYPE}:{expected_timestamp}:flickr")
+        == b"4"
+    )
+    assert (
+        redis.get(
+            f"provider_occurrences:{FAKE_MEDIA_TYPE}:{expected_timestamp}:stocksnap"
+        )
+        == b"6"
+    )
+    assert (
+        redis.get(
+            f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{expected_timestamp}:flickr"
+        )
+        == b"1"
+    )
+    assert (
+        redis.get(
+            f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{expected_timestamp}:stocksnap"
+        )
+        == b"1"
+    )
+
+
+def test_count_provider_occurrences_increments_existing_tallies(redis):
+    results_1 = [{"provider": "flickr"} for _ in range(4)] + [
+        {"provider": "stocksnap"} for _ in range(6)
+    ]
+
+    results_2 = [{"provider": "flickr"} for _ in range(3)] + [
+        {"provider": "inaturalist"} for _ in range(7)
+    ]
+
+    now = datetime(2023, 1, 19)  # 16th is start of week
+    timestamp = "2023-01-16"
+    with freeze_time(now):
+        tallies.count_provider_occurrences(results_1, FAKE_MEDIA_TYPE)
+
+    assert (
+        redis.get(f"provider_occurrences:{FAKE_MEDIA_TYPE}:{timestamp}:flickr") == b"4"
+    )
+    assert (
+        redis.get(f"provider_occurrences:{FAKE_MEDIA_TYPE}:{timestamp}:stocksnap")
+        == b"6"
+    )
+    assert (
+        redis.get(f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{timestamp}:flickr")
+        == b"1"
+    )
+    assert (
+        redis.get(
+            f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{timestamp}:stocksnap"
+        )
+        == b"1"
+    )
+
+    with freeze_time(now):
+        tallies.count_provider_occurrences(results_2, FAKE_MEDIA_TYPE)
+
+    assert (
+        redis.get(f"provider_occurrences:{FAKE_MEDIA_TYPE}:{timestamp}:flickr") == b"7"
+    )  # 4 + 7
+    assert (
+        redis.get(f"provider_occurrences:{FAKE_MEDIA_TYPE}:{timestamp}:stocksnap")
+        == b"6"
+    )  # no change
+    assert (
+        redis.get(f"provider_occurrences:{FAKE_MEDIA_TYPE}:{timestamp}:inaturalist")
+        == b"7"
+    )
+    assert (
+        redis.get(f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{timestamp}:flickr")
+        == b"2"
+    )
+    assert (
+        redis.get(
+            f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{timestamp}:stocksnap"
+        )
+        == b"1"
+    )
+    assert (
+        redis.get(
+            f"provider_appeared_in_searches:{FAKE_MEDIA_TYPE}:{timestamp}:inaturalist"
+        )
+        == b"1"
+    )
diff --git a/api/test/unit/utils/throttle_test.py b/api/test/unit/utils/throttle_test.py
--- a/api/test/unit/utils/throttle_test.py
+++ b/api/test/unit/utils/throttle_test.py
@@ -1,6 +1,6 @@
 from test.factory.models.oauth2 import AccessTokenFactory
 
-from rest_framework.test import APIRequestFactory, force_authenticate
+from rest_framework.test import force_authenticate
 from rest_framework.views import APIView
 
 import pytest
@@ -28,13 +28,6 @@ def get_redis_connection(*args, **kwargs):
     fake_redis.client().close()
 
 
-@pytest.fixture
-def request_factory() -> APIRequestFactory():
-    request_factory = APIRequestFactory(defaults={"REMOTE_ADDR": "192.0.2.1"})
-
-    return request_factory
-
-
 @pytest.fixture
 def access_token():
     token = AccessTokenFactory.create()