Repository: pulp/pulpcore

Issue:
Intermittent 403s when using Redis + Azure Storage backend
**Version**
`pulpcore 3.40.4`

**The bug**
Cloud storage options often have a timeframe that a given signed URL is valid for. Let's suppose that I have set [AZURE_URL_EXPIRATION_SECS](https://django-storages.readthedocs.io/en/latest/backends/azure.html#settings) = 3600 (an hour), which I have.

Pulp's Redis cache has a [default TTL](https://docs.pulpproject.org/pulpcore/configuration/settings.html#cache-settings) of 600 seconds (ten minutes). Let's suppose that I have not changed that, which is true.

A naive developer (me) would assume that since ten minutes is much smaller than an hour there will be no issue with enabling the Redis cache and that users would never get redirected to an expired Azure Storage URL. This is false. The issue is with how pulp-content interacts with Redis.

Pulp-content caches responses (which in this case are always redirect urls to Azure Storage) in hashes with `HSET`. The hash's overall key will be related to the _whole repo_ (or more accurately the distribution, but whatever), something like `yumrepos/sherr-test`. Then individual url/response pairs will get set as key/values of that hash, where an example key might be `/yumrepos/sherr-test/repodata/repomd.xml:GET`. I assume this is so that it's easy to wipe out all data related to a repo when a new version gets published.

The Redis TTL (10 minutes) gets set on that distribution-level hash object. So one would assume that all cached responses related to that distribution will get wiped every 10 minutes, even if some of them are much younger than that. **However**, pulp-content [resets the TTL for the hash every time it caches a new response](https://github.com/pulp/pulpcore/blob/main/pulpcore/cache/cache.py#L88).

This means that as long as users are requesting _some new file_ from the repo before each reset 10 minute TTL is reached, this hash object will persist in Redis _indefinitely_. And after an hour those earlier-cached Azure Storage URLs are in fact expired, and users requesting those files will get 403s.

[You cannot set](https://stackoverflow.com/questions/16545321/how-to-expire-the-hset-child-key-in-redis) a TTL on individual child keys of hashes in Redis. So it seems to me that the options for fixing this are:
1. Don't use hashes, and `SET` a top-level key in Redis for every url/response. This makes invalidating the cache after a publication harder.
2. Attempt to only set the TTL on the hash one time, when it is first created. This would set the behavior back to "all info for the whole repo is wiped every 10 minutes, even if some responses are much younger", which was probably the original intention.
3. Store an additional field on each child key that pulp-content can evaluate to see if _this particular_ response is past the Redis TTL, and refresh it if so.

I don't have any strong opinions on which of those is the correct thing to do, but the current behavior of always resetting the TTL is definitely wrong.

**Additional context**
I am aware that this sounds very similar to the issue @vonsch reported in #3077. However his debugging sounds like it _might actually_ be a different issue, with `django_storages` returning about-to-expire S3 urls to a correctly-behaving `redis` cache, while in this case it's actually entirely Redis' fault, or rather pulp-content's usage of it. If they come back and say that it is in fact the same issue then I have no problem with merging this issue into that one.



Discussion:
Good find! I would go with the third option. Setting the additional field in the cache entry here (https://github.com/pulp/pulpcore/blob/main/pulpcore/cache/cache.py#L200) and adding an extra check here (https://github.com/pulp/pulpcore/blob/main/pulpcore/cache/cache.py#L187-L188) should handle this. The same change will have to be mirrored across the `SyncContentCache` and `AsyncContentCache`.
Reopening requested on matrix.
I have discovered  an issue with the current patch that we probably should fix before release:

The current patch considers the non-existence of the `expires` key to mean "does not expire". It should _instead_ mean "already expired". Otherwise cached responses from the previous version of pulp will persist indefinitely, causing this issue to remain.

Resolving patch:
diff --git a/pulpcore/cache/cache.py b/pulpcore/cache/cache.py
--- a/pulpcore/cache/cache.py
+++ b/pulpcore/cache/cache.py
@@ -186,7 +186,8 @@ def make_response(self, key, base_key):
             return None
         entry = json.loads(entry)
         response_type = entry.pop("type", None)
-        expires = entry.pop("expires", None)
+        # None means "doesn't expire", unset means "already expired".
+        expires = entry.pop("expires", -1)
         if (not response_type or response_type not in self.RESPONSE_TYPES) or (
             expires and expires < time.time()
         ):
@@ -202,9 +203,12 @@ def make_entry(self, key, base_key, handler, args, kwargs, expires=DEFAULT_EXPIR
         """Gets the response for the request and try to turn it into a cacheable entry"""
         response = handler(*args, **kwargs)
         entry = {"headers": dict(response.headers), "status": response.status_code}
-        if expires:
+        if expires is not None:
             # Redis TTL is not sufficient: https://github.com/pulp/pulpcore/issues/4845
             entry["expires"] = expires + time.time()
+        else:
+            # Settings allow you to set None to mean "does not expire". Persist.
+            entry["expires"] = None
         response.headers["X-PULP-CACHE"] = "MISS"
         if isinstance(response, HttpResponseRedirect):
             entry["redirect_to"] = str(response.headers["Location"])
@@ -373,7 +377,8 @@ async def make_response(self, key, base_key):
             entry["body"] = bytes.fromhex(binary)
 
         response_type = entry.pop("type", None)
-        expires = entry.pop("expires", None)
+        # None means "doesn't expire", unset means "already expired".
+        expires = entry.pop("expires", -1)
         if (not response_type or response_type not in self.RESPONSE_TYPES) or (
             expires and expires < time.time()
         ):
@@ -392,9 +397,12 @@ async def make_entry(self, key, base_key, handler, args, kwargs, expires=DEFAULT
             response = e
 
         entry = {"headers": dict(response.headers), "status": response.status}
-        if expires:
+        if expires is not None:
             # Redis TTL is not sufficient: https://github.com/pulp/pulpcore/issues/4845
             entry["expires"] = expires + time.time()
+        else:
+            # Settings allow you to set None to mean "does not expire". Persist.
+            entry["expires"] = None
         response.headers.update({"X-PULP-CACHE": "MISS"})
         if isinstance(response, FileResponse):
             entry["path"] = str(response._path)


Test patch: