Repository: pulp/pulpcore

Issue:
Gunicorn consuming excessive amounts of memory
**Version**
3.16.z

**Describe the bug**
Gunicorn consuming excessive amounts of memory, 3.5-4gb

**To Reproduce**
Unclear

**Expected behavior**
Probably not to have a single gunicorn process use 4gb of memory

**Additional context**

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2035873
Katello forum discussion: https://community.theforeman.org/t/katello-4-5-foreman-3-3-memory-leak-in-gunicorn/29658/22


Discussion:
One workaround is to enable process recycling for gunicorn workers. This doesn't get at the root cause though, which I'm curious about. Also how does it grow over time, does it grow slowly or are there specific requests that leave large amounts of memory pinned?
> One workaround is to enable process recycling for gunicorn workers. 

We have a user doing exactly this, with some success. See https://bugzilla.redhat.com/show_bug.cgi?id=1994397#c43


copy pasting for transparency so it's in one place
```
 To enable gunicorn worker recycling the procedure goes like this:

1.  Create a gunicorn.conf.py under /var/lib/pulp containing max_requests and max_requests_jitter values

# cat /var/lib/pulp/gunicorn.conf.py
max_requests = 1000
max_requests_jitter = 50

2.  restart the specific service systemctl restart pulpcore-api

3.  gunicorn workers will then be automatically recycled every 1000 requests (with a jitter to avoid all workers restarting at the same time) One can reduce the max_requests value to have more aggressive recycling.

```
The installer [passes the gunicorn options along via command line params](https://github.com/pulp/pulp_installer/blob/669dbdde710762c59a6840c1b97cca119d6f6c0a/roles/pulp_content/templates/pulpcore-content.service.j2#L27-L32), so we could use `--max-requests` and `--max-requests-jitter` options like we do today.

To implement this as a default I expected an installer + operator + single container to all make the changes, and pulpcore to document it's benefit.
This might not (fully) be a leak, look at what it's doing, it's fetching the full details of RPM packages from the /pulp/api/v3/rpm/packages/ endpoint.  So: serializing 100-1000 packages at a time.  That might still be a stretch but maybe if it's doing so really inefficiently?

@dralley I agree. My concern (besides the efficiency of that code) is that for subsequent requests the memory isn't released until the recycling occurs.
Added a link to a Katello forum discussion that is ongoing.  Some discoveries so far:  

* it's definitely the API server rather than anything else
* process recycling does help, but the memory is re-leaked extremely quickly
* it's worse with multiple API server processes running
Trimming the (RPM) changelogs appears to help.  So there is likely overlap with https://github.com/pulp/pulpcore/issues/2138

Sounds like it's responsible for not just memory usage issues but also potentially timeouts (there is a BZ for that as well)
Closing in favor of https://github.com/pulp/pulpcore/issues/2138

Katello / Satellite are the main impacted users right now, and they now have multiple workarounds.
Could this issue be reopened? This is not solved yet. We are running pulpcore version 3.21.4

We have some scripts in place for automating some pulp settings. The scripts that we are using use the python pulp client bindings, in these it is not possible to set the fields that the api returns in these bindings.

I can replicate the issue with the following api commands:

Ram usage (of the virtual machine) before the command: 3.7Gi of 24 Gi

```
# pulp rpm content list --name 'self-build-package' --limit 1000 | jq .[].location_href | wc -l
267
```

Ram usage (of the virtual machine) after execurting the command: 6.6Gi of 24Gi

Our pulp virtual machine ran out of memory once when the pulpcore-api service was using 12.7Gi of memory. This was cleared after restarting the service, but that should not be the case.
@maartenbeeckmans How old is this instance, and are you working with RHEL RPMs synced from at least 6 months ago?  If so, could you try out the changelog trimming feature and report back on whether it helps?

Also what is your pulp_rpm version?  You only mentioned pulpcore.
Versions can be seen below, we are using the rpm's provided by theforeman.

```sh
# pulp status | jq .versions
[
  {
    "component": "core",
    "version": "3.21.4",
    "package": "pulpcore"
  },
  {
    "component": "rpm",
    "version": "3.18.9",
    "package": "pulp-rpm"
  },
  {
    "component": "file",
    "version": "1.11.1",
    "package": "pulp-file"
  },
  {
    "component": "deb",
    "version": "2.20.0",
    "package": "pulp_deb"
  },
  {
    "component": "container",
    "version": "2.14.3",
    "package": "pulp-container"
  }
]
```
I've tried a number of different debugging approaches, I know that the issue is all of the RPM changelogs loaded while listing the packages, but I don't know why they're leaking.  My best guess is that maybe the django queryset cache is being used on the viewset, but I'm not confident of that.

https://github.com/pulp/pulpcore/issues/2138 would help to work around it as does process recycling
So finally we have an answer to this saga.  [This line](https://github.com/pulp/pulpcore/pull/2826/files/bc85beb34796682071e72d5ee44a9425c2b86b75#diff-f3d57ed180a61179704601b8a978567bf8e5c5c54e95458ed8b0b49bbc150f3cR15) was caching the view objects, rather than serving the intended goal of reducing the number of lookups.  

Thanks to Hao Yu for finding the root cause (I need find out how!) and also Gerrod for putting in some time investigating once that was identified.

Resolving patch:
diff --git a/pulpcore/app/access_policy.py b/pulpcore/app/access_policy.py
--- a/pulpcore/app/access_policy.py
+++ b/pulpcore/app/access_policy.py
@@ -1,4 +1,3 @@
-from functools import lru_cache
 from rest_access_policy import AccessPolicy
 from rest_framework.exceptions import APIException
 
@@ -12,7 +11,6 @@ class AccessPolicyFromDB(AccessPolicy):
     """
 
     @staticmethod
-    @lru_cache
     def get_access_policy(view):
         """
         Retrieves the AccessPolicy from the DB or None if it doesn't exist.


Test patch: