Repository: WordPress/openverse-api

Issue:
Increase the dead links cache expiration timeout
## Problem
<!-- Describe a problem solved by this feature; or delete the section entirely. -->
Currently, every front-end search results in requesting the provider image URL (the direct URL) twice: once for the API dead links filtering and once for the thumbnail generation.
The thumbnails are cached for a period of around a year (to be confirmed), and the link validation cache expires after 24 hours for active links and 120 days for broken links. This means we're re-requesting popular valid images every 24 hours. We could increase the cache expiration timeout to reduce the number of network requests to the providers (which have often been 429-ed lately).

## Description
<!-- Describe the feature and how it solves the problem. -->
We could set the cache expiration timeout for active links based on an environment variable to make it easily tweakable.
I think increasing the active link cache TTL to **one week** could be a good balance between keeping the link status up-to-date and reducing the number of requests. What do you think about the timeout period, @openverse-team?

## Alternatives
<!-- Describe any alternative solutions or features you have considered. How is this feature better? -->
Another option for reducing the number of requests to providers could be to request the thumbnail endpoint first, and if the request headers show that the response header comes from the Cloudflare cache, consider the link not dead. If, however, the response is coming from the proxy and does not have a 200 status, then we can consider the link dead.

## Implementation
<!-- Replace the [ ] with [x] to check the box. -->
- [ ] 🙋 I would be interested in implementing this feature.



Discussion:
> Another option for reducing the number of requests to providers could be to request the thumbnail endpoint first, and if the request headers show that the response header comes from the Cloudflare cache, consider the link not dead. If, however, the response is coming from the proxy and does not have a 200 status, then we can consider the link dead.

If we did this then the API would need to create an access token for itself to use, just a heads up. Otherwise the API would be rate limiting itself :slightly_smiling_face: 
One year is correct for the thumbnails:

![CleanShot 2022-08-09 at 10 19 04@2x](https://user-images.githubusercontent.com/6351754/183673043-662d03f2-ecd6-48df-915e-6edd60fac2be.png)

The cache-control max age is 31536000 seconds, or 365 days (31536000s / 60s / 60m / 24hrs).

Been thinking about this a bit. Have some thoughts and complications:

1. Thumbnail proxy does not solve the problem for audio, but the same process is followed for audio (requesting upstream `url` and using response status code to determine liveness of the result). Audio does use the thumbnail proxy but for the artwork. Audio link liveness check uses the actual result url, not the upstream `thumbnail` location on the audio result.
2. If we're relying on Cloudflare's caching, are we accepting that a year is a reasonable amount of time to assume that a result is still valid after checking it? If so.... why not just increase the expiration in our existing validation cache to 1 year and skip the network request entirely.
3. If we use thumbnail proxy then we're adding some potentially non-trivial latency to the validation check:
    1. Send request to thumbnail proxy -> hits cloudflare -> hits the API -> redirect to the internal thumbnail service -> request goes upstream to the result url -> thumbnail proxy processes and returns the response -> now we have the status code we wanted.
    2. Currently the only process is: request goes to upstream `url` -> how we have the status code we wanted.


Resolving patch:
diff --git a/api/catalog/api/utils/validate_images.py b/api/catalog/api/utils/validate_images.py
--- a/api/catalog/api/utils/validate_images.py
+++ b/api/catalog/api/utils/validate_images.py
@@ -3,6 +3,7 @@
 
 import django_redis
 import grequests
+from decouple import config
 
 from catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask
 
@@ -18,6 +19,10 @@ def _get_cached_statuses(redis, image_urls):
     return [int(b.decode("utf-8")) if b is not None else None for b in cached_statuses]
 
 
+def _get_expiry(status, default):
+    return config(f"LINK_VALIDATION_CACHE_EXPIRY__{status}", default=default, cast=int)
+
+
 def validate_images(query_hash, start_slice, results, image_urls):
     """
     Make sure images exist before we display them. Treat redirects as broken
@@ -69,14 +74,17 @@ def validate_images(query_hash, start_slice, results, image_urls):
         # Cache successful links for a day, and broken links for 120 days.
         if status == 200:
             logger.debug("healthy link " f"key={key} ")
-            pipe.expire(key, twenty_four_hours_seconds)
+            expiry = _get_expiry(200, twenty_four_hours_seconds * 30)
         elif status == -1:
             logger.debug("no response from provider " f"key={key}")
             # Content provider failed to respond; try again in a short interval
-            pipe.expire(key, thirty_minutes)
+            expiry = _get_expiry("_1", thirty_minutes)
         else:
             logger.debug("broken link " f"key={key} ")
-            pipe.expire(key, twenty_four_hours_seconds * 120)
+            expiry = _get_expiry("DEFAULT", twenty_four_hours_seconds * 120)
+
+        pipe.expire(key, expiry)
+
     pipe.execute()
 
     # Merge newly verified results with cached statuses


Test patch: