Repository: WordPress/openverse-api

Issue:
Move off of grequests
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
The API currently uses [the `grequests` library](https://github.com/spyoungtech/grequests) for handling some of the `head` requests it completes when returning results (specifically in [`validate_images.py`](https://github.com/WordPress/openverse-api/blob/main/api/catalog/api/utils/validate_images.py#L37)).

## Description
<!-- Describe the feature and how it solves the problem. -->
The README for the project says:

> Note: You should probably use requests-threads or requests-futures instead.

The project has also not had a release since 2020. We might want to consider using one of those proposed alternatives.

## Alternatives
<!-- Describe any alternative solutions or features you have considered. How is this feature better? -->
Leave as is and hope for the best!

## Additional context
<!-- Add any other context about the feature here; or delete the section entirely. -->

Some more context in Sentry: https://sentry.io/share/issue/061ba99fc3df4c23bdb7643d337bbda0/

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:
This would be a great issue for a new contributor to pick up, if anyone is looking for something to get started with 😀 
Hi @sarayourfriend,

I came across your project and I am interested in contributing! I am new to opensource community contributions but I have 5 years of experience working as an engineer with python, C++ and django frameworks. Can I work on this issue?

From the description, what I concluded is that we want to migrate from grequests to some other multi-threaded handling of requests, possibly requests-threads? Please share any other information which you think might be needed to work on the issue.

Thanks!
@samajain Hi! Welcome to Openverse :grin: It'd be great to have you work on this issue. Your understanding of the issue and one of the potential solutions is correct.

`request-threads` could be a good issue but we don't currently use Twisted, which `requests-threads` depends on :thinking: I wonder if we could also look into using [Django's builtin asyncio support](https://docs.djangoproject.com/en/4.0/topics/async/) and using something like [httpx](https://github.com/encode/httpx) or one of the other popular async HTTP libraries. `requests-threads` hasn't been updated in a few years (including not having technical support [for the latest Python versions](https://github.com/requests/requests-threads/blob/master/setup.py#L98)).

If you feel comfortable doing that kind of research for the trade-offs between the different options then let us know :slightly_smiling_face: Unfortunately this issue might not be as clear cut as it seems now that I've looked into it a bit more :confused: If you'd prefer working one something more straight forward then also let me know and I can help you find something else.

Thanks for your interest in the project!
I can do the research if you want me to and share my findings.

If not this issue then I can take up something else too 🙂 Since you have been part of this project, I leave it to your call.

I'll dig up for something from the good first issues label. 
We'd be more than happy to let you take this one on! If you're comfortable gathering the research and posting it here, we can work together to decide which library/path forward might be best :slightly_smiling_face: 
Sure! I can do that. What kind of tradeoff aspects are you looking for? Shall I compare them based on their performance metrics or are you looking for a comparison on anything specific other than this?
I'll let @sarayourfriend chime in with any other thoughts, but I think my greatest concern would be 1) what is the maintenance burden we incur with whatever dependency we go with and 2) how much restructuring of the codebase do we need to do in order to properly integrate the library. For instance, if we do go with an async-based library, how much will we need to change to make the application support `async` calls. Performance metrics would be useful but secondary I think to these factors :smile: 
I'm less concerned with the maintenance burden initially than with the performance impact of using async in Django 4. The Django docs mention that using async views (which is fully supported and does not require any restructuring, apparently :thinking:) has some performance trade offs. It doesn't explain what those are, so it'd be nice to understand them. We could move the Django app to ASGI and run it with uvicorn instead of gunicorn, but that my gut tells me that's probably a bigger move than we need to solve this particular problem. It would also need to involve larger infrastructure changes that we don't necessarily have capacity for at the moment. To quote the Django docs:

> Async views will still work under WSGI, but with performance penalties, and without the ability to have efficient long-running requests. 

The only place we're using grequests right now is basically to fire off a bunch of HTTP requests in parallel, so the long-running requests issue isn't important. However, the usage is in the main search routine, so it's extremely critical path. It's perhaps the view where performance matters the most for us.

My preference would be to use the built in Django async support as it's in line with mainline Python support (and I think is generally where things are headed, mostly away from Twisted, etc). But if the performance impact is bad, then we'll necessarily need to suffer the maintenance burden of introducing a "bolted on" asynchronous library for now.

All of that to say, I'd recommend looking into what the trade offs are with using Django's built in asyncio view support under WSGI first, then moving onto seeing what other options exist if the penalties for partial async in Django WSGI app are bad.
> However, the usage is in the main search routine, so it's extremely critical path. It's perhaps the view where performance matters the most for us.

That's a great point, thanks for mentioning it! And I'm glad that it doesn't seem like async support in Django will be too difficult from a code perspective.
@AetherUnbound @sarayourfriend Thank you for your inputs. I will do my research and get back soon!
@samajain Did you get a chance to look into this?

Given Django's recent release of 4.2 adding async to the ORM, I think it is a good time to start planning a switch over to the async Django API. I don't think this will be a trivial change to do correctly and I don't know if it can be done in stages. Would it be detrimental, for example, to delay switching to async Redis and Elasticsearch clients and Django ORM to happen after switching to async Django generally to be able to use an async http library to replace our usage of grequests?
I was reading more about Django's async support. It looks like we might be able to relatively easily get off of grequests by using Django's in-depth `async_to_sync` function to run an `async` version of the `validate_images` function.

@WordPress/openverse-maintainers I think we could prioritize this issue higher if someone wanted to try out `async_to_sync` with a regular asyncio HTTP Python library like httpx or similar.
I am all for this!
Hii @AetherUnbound I also want to contribute to fix this issue
@Prakharkarsh1 please go ahead! Feel free to comment here if you need any pointers getting started.
I took a peek at this today and moving to ASGI for our application would be mostly trivial (I think). The bulk of the work would be to remove grequests and replacing it with something like httpx or aiohttp (plus maybe AnyIO to make "grouping" the requests easier, if the libraries themselves do not natively support it).

Updating unit tests would probably take up most of the time spent on this. Due to grequests nature, the tests do a fair bit of unusual mocking to try to stay out of the way as much as possible, but it's very specific to grequests. Perhaps adding an abstraction layer here would help to simplify the tests in general.
Here is a small diff that moves the app to ASGI:

```diff
diff --git a/api/catalog/asgi.py b/api/catalog/asgi.py
new file mode 100644
index 00000000..6b3169f4
--- /dev/null
+++ b/api/catalog/asgi.py
@@ -0,0 +1,7 @@
+import os
+
+from django.core.asgi import get_asgi_application
+
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "catalog.settings")
+
+application = get_asgi_application()
diff --git a/api/catalog/settings.py b/api/catalog/settings.py
index 268ebb62..192feca2 100644
--- a/api/catalog/settings.py
+++ b/api/catalog/settings.py
@@ -236,7 +236,7 @@ TEMPLATES = [
     },
 ]
 
-WSGI_APPLICATION = "catalog.wsgi.application"
+ASGI_APPLICATION = "catalog.aisgi.application"
 
 # Database
 # https://docs.djangoproject.com/en/2.0/ref/settings/#databases
diff --git a/api/catalog/wsgi.py b/api/catalog/wsgi.py
deleted file mode 100644
index f8458d06..00000000
--- a/api/catalog/wsgi.py
+++ /dev/null
@@ -1,23 +0,0 @@
-"""
-WSGI config for catalog project.
-
-It exposes the WSGI callable as a module-level variable named ``application``.
-
-For more information on this file, see
-https://docs.djangoproject.com/en/2.0/howto/deployment/wsgi/
-"""
-from gevent import monkey
-
-
-monkey.patch_all()
-import os
-
-from django.core.wsgi import get_wsgi_application
-
-from wsgi_basic_auth import BasicAuth
-
-
-os.environ.setdefault("DJANGO_SETTINGS_MODULE", "catalog.settings")
-
-application = get_wsgi_application()
-application = BasicAuth(application)
```

I've done no performance checking on this, but adding `async` to view routes and switching to the async base classes (if we need to do that?) seems like it would be the only bit left to make it work fully async. ORM methods might need to be changed as well...

It doesn't look like we're using any middleware that causes Django to have to adapt each request. I checked this by setting the log level to debug and checking for the `django.logger` logs about middleware adapting.
@sarayourfriend would switching to ASGI mean that we need to change the server from Gunicorn to [something else](https://asgi.readthedocs.io/en/latest/implementations.html#servers)?
Yes, we would switch the production server to hypercorn, uvcorn, etc. I've used hypercorn in the past and liked it, it has almost the same api as gunicorn.
It completely slipped by me that we would need DRF to also support async views for migrating to async Django to be possible. [DRF still does not support async](https://github.com/encode/django-rest-framework/pull/8617) and it seems like it might not natively support it for a while (though a package could do the job).

The two alternatives listed by grequests (and mentioned in the issue description) are actually less recently maintained than grequests itself.

We could try just using Django's `async_to_sync` but I think we would pay a performance penalty from spinning up an event loop.

The Sentry issue continues to record events, so this problem does still exist today.

As I see it, there are two options:

1. Try `async_to_sync` and measure whether there is a difference in performance in production. Revert if there is an unacceptable performance difference or a regression otherwise.
2. Pull the async DRF implementation out of the PR and write our own async shim as recommended in the PR itself and dive head first into full async Django.

My preference is the second as (a) we'll want to move to async Django in the future anyway and (b) if we do the shimming right, it should be possible to swap it out for whatever more developed async DRF package comes out of the PR _or_ we can just switch to async DRF when it is officially supported (if ever).

I'm going to tinker a bit with the second option as I am working on this mostly as a curiosity rather than it being a priority. I hope that is okay with folks. If anything I'm sure I'll learn something useful about our app and async generally in the process that will be applicable to a future async conversion (if one doesn't come out of this practice).
I do not think it is worth the fuss to do number 2 right now. I've tried for a bit to get the second option working and it is going to take quite a bit of work from us to convert things to async with DRF.

Trying `async_to_sync` is probably the best approach here. We will want to implement some timings for the API first, however. Adding timeit around the `validate_images` function and logging the timings would be a good first step here.

Resolving patch:
diff --git a/api/catalog/api/utils/validate_images.py b/api/catalog/api/utils/validate_images.py
--- a/api/catalog/api/utils/validate_images.py
+++ b/api/catalog/api/utils/validate_images.py
@@ -1,11 +1,14 @@
+import asyncio
 import logging
 import time
 
 from django.conf import settings
 
+import aiohttp
 import django_redis
-import grequests
+from asgiref.sync import async_to_sync
 from decouple import config
+from elasticsearch_dsl.response import Hit
 
 from catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask
 
@@ -28,7 +31,29 @@ def _get_expiry(status, default):
     return config(f"LINK_VALIDATION_CACHE_EXPIRY__{status}", default=default, cast=int)
 
 
-def validate_images(query_hash, start_slice, results, image_urls):
+async def _head(url: str, session: aiohttp.ClientSession) -> tuple[str, int]:
+    try:
+        async with session.head(url, timeout=2, allow_redirects=False) as response:
+            return url, response.status
+    except aiohttp.ClientError as exception:
+        _log_validation_failure(exception)
+        return url, -1
+
+
+# https://stackoverflow.com/q/55259755
+@async_to_sync
+async def _make_head_requests(urls: list[str]) -> list[tuple[str, int]]:
+    tasks = []
+    async with aiohttp.ClientSession(headers=HEADERS) as session:
+        tasks = [asyncio.ensure_future(_head(url, session)) for url in urls]
+        responses = asyncio.gather(*tasks)
+        await responses
+    return responses.result()
+
+
+def validate_images(
+    query_hash: str, start_slice: int, results: list[Hit], image_urls: list[str]
+) -> None:
     """
     Make sure images exist before we display them. Treat redirects as broken
     links since 99% of the time the redirect leads to a generic "not found"
@@ -57,25 +82,10 @@ def validate_images(query_hash, start_slice, results, image_urls):
             to_verify[url] = idx
     logger.debug(f"len(to_verify)={len(to_verify)}")
 
-    reqs = (
-        grequests.head(
-            url, headers=HEADERS, allow_redirects=False, timeout=2, verify=False
-        )
-        for url in to_verify.keys()
-    )
-    verified = grequests.map(reqs, exception_handler=_validation_failure)
+    verified = _make_head_requests(to_verify.keys())
+
     # Cache newly verified image statuses.
-    to_cache = {}
-    # relies on the consistenct of the order returned by `dict::keys()` which
-    # is safe as of Python 3 dot something
-    for idx, url in enumerate(to_verify.keys()):
-        cache_key = CACHE_PREFIX + url
-        if verified[idx]:
-            status = verified[idx].status_code
-        # Response didn't arrive in time. Try again later.
-        else:
-            status = -1
-        to_cache[cache_key] = status
+    to_cache = {CACHE_PREFIX + url: status for url, status in verified}
 
     pipe = redis.pipeline()
     if len(to_cache) > 0:
@@ -98,10 +108,7 @@ def validate_images(query_hash, start_slice, results, image_urls):
     # Merge newly verified results with cached statuses
     for idx, url in enumerate(to_verify):
         cache_idx = to_verify[url]
-        if verified[idx] is not None:
-            cached_statuses[cache_idx] = verified[idx].status_code
-        else:
-            cached_statuses[cache_idx] = -1
+        cached_statuses[cache_idx] = verified[idx][1]
 
     # Create a new dead link mask
     new_mask = [1] * len(results)
@@ -145,6 +152,6 @@ def validate_images(query_hash, start_slice, results, image_urls):
     )
 
 
-def _validation_failure(request, exception):
-    logger = parent_logger.getChild("_validation_failure")
+def _log_validation_failure(exception):
+    logger = parent_logger.getChild("_log_validation_failure")
     logger.warning(f"Failed to validate image! Reason: {exception}")


Test patch:
diff --git a/api/test/dead_link_filter_test.py b/api/test/dead_link_filter_test.py
--- a/api/test/dead_link_filter_test.py
+++ b/api/test/dead_link_filter_test.py
@@ -1,5 +1,5 @@
 from test.constants import API_URL
-from unittest.mock import MagicMock, patch
+from unittest.mock import patch
 from uuid import uuid4
 
 from django.conf import settings
@@ -48,44 +48,39 @@ def get_empty_cached_statuses(_, image_urls):
     )
 
 
-def _patch_grequests():
-    def grequests_map(reqs, *_, **__):
-        """
-        Patch for ``grequests.map`` used by ``validate_images`` to filter
-        and remove dead links
-        """
+_MAKE_HEAD_REQUESTS_MODULE_PATH = (
+    "catalog.api.utils.validate_images._make_head_requests"
+)
+
+
+def _patch_make_head_requests():
+    def _make_head_requests(urls):
         responses = []
-        for idx in range(len(list(reqs))):
-            mocked_res = MagicMock()
-            mocked_res.status_code = 200 if idx % 10 != 0 else 404
-            responses.append(mocked_res)
+        for idx, url in enumerate(urls):
+            status_code = 200 if idx % 10 != 0 else 404
+            responses.append((url, status_code))
         return responses
 
-    return patch("grequests.map", side_effect=grequests_map)
+    return patch(_MAKE_HEAD_REQUESTS_MODULE_PATH, side_effect=_make_head_requests)
 
 
 def patch_link_validation_dead_for_count(count):
     total_res_count = 0
 
-    def grequests_map(reqs, *_, **__):
+    def _make_head_requests(urls):
         nonlocal total_res_count
-        """
-        Patch for ``grequests.map`` used by ``validate_images`` to filter
-        and remove dead links
-        """
         responses = []
-        for idx in range(len(list(reqs))):
+        for idx, url in enumerate(urls):
             total_res_count += 1
-            mocked_res = MagicMock()
-            mocked_res.status_code = 404 if total_res_count <= count else 200
-            responses.append(mocked_res)
+            status_code = 404 if total_res_count <= count else 200
+            responses.append((url, status_code))
         return responses
 
-    return patch("grequests.map", side_effect=grequests_map)
+    return patch(_MAKE_HEAD_REQUESTS_MODULE_PATH, side_effect=_make_head_requests)
 
 
 @pytest.mark.django_db
-@_patch_grequests()
+@_patch_make_head_requests()
 def test_dead_link_filtering(mocked_map, client):
     path = "/v1/images/"
     query_params = {"q": "*", "page_size": 20}
diff --git a/api/test/unit/utils/validate_images_test.py b/api/test/unit/utils/validate_images_test.py
--- a/api/test/unit/utils/validate_images_test.py
+++ b/api/test/unit/utils/validate_images_test.py
@@ -1,10 +1,9 @@
-from collections.abc import Callable
-from dataclasses import dataclass
+from unittest import mock
 
+import aiohttp
+import pook
 import pytest
 from fakeredis import FakeRedis
-from grequests import AsyncRequest
-from requests import Response
 
 from catalog.api.utils.validate_images import HEADERS, validate_images
 
@@ -22,48 +21,26 @@ def get_redis_connection(*args, **kwargs):
     fake_redis.client().close()
 
 
-@dataclass
-class GRequestsFixture:
-    requests: list[AsyncRequest]
-    response_factory: Callable[
-        [AsyncRequest], Response
-    ] = lambda x: GRequestsFixture._default_response_factory(x)
-
-    @staticmethod
-    def _default_response_factory(req: AsyncRequest) -> Response:
-        res = Response()
-        res.url = req.url
-        res.status_code = 200
-        return res
-
-
-@pytest.fixture(autouse=True)
-def grequests(monkeypatch) -> GRequestsFixture:
-    fixture = GRequestsFixture([])
-
-    def map_reqs(reqs, **kwargs):
-        nonlocal fixture
-        fixture.requests += list(reqs)
-        responses = [fixture.response_factory(r) for r in fixture.requests]
-        return responses
-
-    monkeypatch.setattr("grequests.map", map_reqs)
-
-    return fixture
-
-
-def test_sends_user_agent(grequests):
+@mock.patch.object(aiohttp, "ClientSession", wraps=aiohttp.ClientSession)
+@pook.on
+def test_sends_user_agent(wrapped_client_session: mock.AsyncMock):
     query_hash = "test_sends_user_agent"
     results = [object() for _ in range(40)]
-    image_urls = [f"http://example.org/{i}" for i in range(len(results))]
+    image_urls = [f"https://example.org/{i}" for i in range(len(results))]
     start_slice = 0
 
+    head_mock = (
+        pook.head(pook.regex(r"https://example.org/\d"))
+        .times(len(results))
+        .reply(200)
+        .mock
+    )
+
     validate_images(query_hash, start_slice, results, image_urls)
 
-    requested_urls = [r.url for r in grequests.requests]
+    assert head_mock.calls == len(results)
+    requested_urls = [req.rawurl for req in head_mock.matches]
     for url in image_urls:
         assert url in requested_urls
 
-    assert len(grequests.requests) > 0
-    for r in grequests.requests:
-        assert r.kwargs["headers"] == HEADERS
+    wrapped_client_session.assert_called_once_with(headers=HEADERS)