Repository: WordPress/openverse-api

Issue:
Dead link filtering reduces number of queryable pages 
## Description
<!-- Concisely describe the bug. -->
The dead link filtering algorithm significantly limits the pages that can be retrieved.

## Reproduction

0. Go to audio search endpoint: `http://localhost:8000/v1/audio/`. Note the number of pages.
1. Go to a page number that's quite high but less than 250, like 200: `http://localhost:8000/v1/audio/?page=200`
2. See the deep pagination error. This should not be happening.

## Expectation
<!-- Concisely describe what you expected to happen. -->
Page count calculation should be correct and should allow pagination upto the stated limit.

## Resolution
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in resolving this bug.


Discussion:
One way to fix this would be to check if `result_count` is divisible by `page_size`. If not, increment `page_count` by 1.
Updated this issue to actually reflect the second half of the bug report, the first being resolved in #172.
HI @dhruvkb 
Can you please provide more details about this bug?

A brief search shows that `api / controllers / search_controller` is file where the bug is. 

Methods
1. _paginate_with_dead_link_mask
2. _get_query_slice

Is this correct?
@ritesh-pandey I added the steps to reproduce. You're right about the bug being in the `search_controller.py` file.
The problem for audio is actually the fact that we only have one provider at the moment, and when we send 20 requests at the same time to check if the links are dead or not, a large portion of the responses return errors. 

One [temporary] solution could be to suspend checking for dead links for audio, since we have ingested the audio data only recently, and they are most likely all current.
Do we experience this bug in non-audio search as well?

I agree with @obulat. The system starts deleting images in cache when we go for high page numbers. This sometimes causes zero result set. 
This seems to be the same problem of #13 and the solution is not trivial. I'm going to remove the "good first issues" label.
One mitigation approach we could employ here ([based on this discussion from the Make WP Slack](https://wordpress.slack.com/archives/C02012JB00N/p1669749455402879?thread_ts=1669738284.048119&cid=C02012JB00N)):

For cases where the number of results on a page is less than the page size, and the page itself is less than the expected page count, adjust the `result_count` and `page_count` values to reflect the actual number of results returned. An example: 
> Let's say that we have 100 results in ES, but only 50 of those are live links. We could iterate through the first 3 pages, and then on page 3 (of a projected 5 total pages) we'd have 10 results instead of 20 which is the page size. It's at page 3 that we'd report the actual total result count and number of pages.
I know this is a bigger change than we can implement for the existing `v1` API, but this kind of problem makes me wonder whether we would be better off returning a pagination cursor that is returned as `null` when there are no more pages. The fact is that due to dead link filtering we have very limited ability to accurately report the number of pages and it's not unreasonable for someone to write code like this (which is not reliable with our current API):

```py
first_page = requests.get(".../images?q=birds")
page_count = first_page["page_count"]

i = 2
while i <= page_count:
  # request new page, do whatever you want
  i += 1
```

For any query that returns a page count I would personally fully expect it to be stable through the iterations of the pages. But we cannot guarantee that because the page count is based on the full number of not-yet-validated results. We currently calculate `page_count` based on the total number of hits that ES reports. It's not even possible for us to easily apply our dead link masks to this, at least not that I can tell: https://github.com/WordPress/openverse-api/blob/4edae23b8e12b8ceb19c75652b215621128fb4fa/api/catalog/api/controllers/search_controller.py#L512-L513

Maybe we could `sum` the dead link mask and reduce that from the total count, but unless a query has been paginated to the end of all the ES results, it won't matter. And we know that won't happen because some queries will return more results than we have ever made possible to retrieve via the API due to ES deep pagination limits.

I've assigned myself the issue to work on the solution that Madison shared above, but I think as we look forward to a potentially v2 of the API, it would be good to rethink how we present pagination entirely.
> adjust the `result_count`

@AetherUnbound Did you have a way of doing this in mind? I'm not sure how to do this on the last page of a query given that we've skipped over a certain number of results. Given we already limit pagination far lower than some queries would ever be able to be exhausted (i.e., we have "inaccessible results" even that aren't dead links) I propose we leave the result_count alone and only update the page_count to be set to the currently requested page (when we detect that we've reached the last possible page for a query based on the current number of results for the page).

Resolving patch:
diff --git a/api/catalog/api/controllers/search_controller.py b/api/catalog/api/controllers/search_controller.py
--- a/api/catalog/api/controllers/search_controller.py
+++ b/api/catalog/api/controllers/search_controller.py
@@ -410,7 +410,7 @@ def search(
     )
 
     result_count, page_count = _get_result_and_page_count(
-        search_response, results, page_size
+        search_response, results, page_size, page
     )
     return results or [], page_count, result_count
 
@@ -447,7 +447,7 @@ def related_media(uuid, index, request, filter_dead):
         s, start, end, page_size, response, request, filter_dead
     )
 
-    result_count, _ = _get_result_and_page_count(response, results, page_size)
+    result_count, _ = _get_result_and_page_count(response, results, page_size, page)
 
     return results or [], result_count
 
@@ -496,7 +496,7 @@ def get_sources(index):
 
 
 def _get_result_and_page_count(
-    response_obj: Response, results: list[Hit] | None, page_size: int
+    response_obj: Response, results: list[Hit] | None, page_size: int, page: int
 ) -> tuple[int, int]:
     """
     Elasticsearch does not allow deep pagination of ranked queries.
@@ -506,14 +506,22 @@ def _get_result_and_page_count(
     :param results: The list of filtered result Hits.
     :return: Result and page count.
     """
-    if results is None:
-        return 0, 1
+    if not results:
+        return 0, 0
 
     result_count = response_obj.hits.total.value
-    page_count = int(result_count / page_size)
-    if page_count % page_size != 0:
-        page_count += 1
-    if len(results) < page_size and page_count == 0:
-        result_count = len(results)
+    page_count = ceil(result_count / page_size)
+
+    if len(results) < page_size:
+        if page_count == 1:
+            result_count = len(results)
+
+        # If we have fewer results than the requested page size and are
+        # not on the first page that means that we've reached the end of
+        # the query and can set the page_count to the currently requested
+        # page. This means that the `page_count` can change during
+        # pagination for the same query, but it's the only way to
+        # communicate under the current v1 API that a query has been exhausted.
+        page_count = page
 
     return result_count, page_count
diff --git a/api/catalog/api/examples/audio_responses.py b/api/catalog/api/examples/audio_responses.py
--- a/api/catalog/api/examples/audio_responses.py
+++ b/api/catalog/api/examples/audio_responses.py
@@ -58,7 +58,7 @@
 audio_search_200_example = {
     "application/json": {
         "result_count": 1,
-        "page_count": 0,
+        "page_count": 1,
         "page_size": 20,
         "page": 1,
         "results": [
@@ -108,7 +108,7 @@
 audio_related_200_example = {
     "application/json": {
         "result_count": 10000,
-        "page_count": 0,
+        "page_count": 1,
         "results": [
             {
                 "title": "File:Mozart - Eine kleine Nachtmusik - 1. Allegro.ogg",
diff --git a/api/catalog/api/examples/image_responses.py b/api/catalog/api/examples/image_responses.py
--- a/api/catalog/api/examples/image_responses.py
+++ b/api/catalog/api/examples/image_responses.py
@@ -68,7 +68,7 @@
 image_search_200_example = {
     "application/json": {
         "result_count": 1,
-        "page_count": 0,
+        "page_count": 1,
         "page_size": 20,
         "page": 1,
         "results": [base_image | {"fields_matched": ["title"]}],
@@ -109,7 +109,7 @@
 image_related_200_example = {
     "application/json": {
         "result_count": 10000,
-        "page_count": 0,
+        "page_count": 1,
         "results": [
             {
                 "title": "exam tactics",


Test patch:
diff --git a/api/test/unit/controllers/test_search_controller.py b/api/test/unit/controllers/test_search_controller.py
--- a/api/test/unit/controllers/test_search_controller.py
+++ b/api/test/unit/controllers/test_search_controller.py
@@ -13,36 +13,49 @@
 
 
 @pytest.mark.parametrize(
-    "total_hits, real_result_count, page_size, expected",
+    "total_hits, real_result_count, page_size, page, expected",
     [
         # No results
-        (0, 0, 10, (0, 0)),
+        (0, 0, 10, 1, (0, 0)),
         # Setting page size to 0 raises an exception
+        # Not possible in the actual API
         pytest.param(
-            0, 0, 0, (0, 0), marks=pytest.mark.raises(exception=ZeroDivisionError)
+            5, 5, 0, 1, (0, 0), marks=pytest.mark.raises(exception=ZeroDivisionError)
         ),
         # Fewer results than page size leads to max of result total
-        (5, 5, 10, (5, 0)),
-        # Even if no real results exist, total result count and page count are returned
-        # (seems like an impossible case IRL)
-        (100, 0, 10, (100, 10)),
+        (5, 5, 10, 1, (5, 1)),
+        # If no real results exist, even if there are hits, fall back to 0, 0
+        # (This case represents where all the links for a result are dead, for example)
+        (100, 0, 10, 1, (0, 0)),
         # If there are real results and ES reports no hits, nothing is expected
         # (seems like an impossible case IRL)
-        (0, 100, 10, (0, 0)),
+        (0, 100, 10, 1, (0, 0)),
         # Evenly divisible number of pages
-        (25, 5, 5, (25, 5)),
+        (25, 5, 5, 1, (25, 5)),
+        # An edge case that previously did not behave as expected with evenly divisble numbers of pages
+        (20, 5, 5, 1, (20, 4)),
         # Unevenly divisible number of pages
-        (21, 5, 5, (21, 5)),
-        # My assumption would be that this yields (20, 4), but the code is such that
-        # when the "natural" page count can't be cleanly divisible by the page size,
-        # We increment it plus one. Why would that be the case? 20 results, with 5
-        # results per-page, would seem to result in 4 pages total not 5 🤷‍♀️
-        (20, 5, 5, (20, 5)),
+        (21, 5, 5, 1, (21, 5)),
         # Fewer hits than page size, but result list somehow differs, use that for count
-        (48, 20, 50, (20, 0)),
+        (48, 20, 50, 1, (20, 1)),
+        # Despite us applying a pagination limit, that is applied further up in the API, not at this low a level
+        (2000, 20, 20, 2, (2000, 100)),
+        # Page count is reduced to the current page number even though 2000 / 20 is much larger than 5
+        # because despite that we have result count < page size which indicates we've exhausted the query
+        (2000, 5, 20, 5, (2000, 5)),
+        # First page, we got all the results and there are no further possible pages with the current page count
+        (10, 10, 20, 1, (10, 1)),
+        # This is here to test a case that used to erroneously produce (10, 2) by adding 1
+        # to the page count when it wasn't necessary to do so.
+        (10, 10, 10, 1, (10, 1)),
+        # This is technically impossible because we truncate results to the page size before entering this method
+        # I think the handling of this case is a likely source for the bug in the previous case
+        (10, 10, 9, 1, (10, 2)),
     ],
 )
-def test_get_result_and_page_count(total_hits, real_result_count, page_size, expected):
+def test_get_result_and_page_count(
+    total_hits, real_result_count, page_size, page, expected
+):
     response_obj = mock.MagicMock()
     response_obj.hits.total.value = total_hits
     results = [mock.MagicMock() for _ in range(real_result_count)]
@@ -51,6 +64,7 @@ def test_get_result_and_page_count(total_hits, real_result_count, page_size, exp
         response_obj,
         results,
         page_size,
+        page,
     )
     assert actual == expected