Repository: pulp/pulpcore

Issue:
on-demand content results in unclosed, idle, database connections
**Version**
* core/main, pulp_rpm/main
* core/3.16, pulp_rpm/3.17
* pulp3-source-centos7

**Describe the bug**
A client asking for content from an "on_demand" remote results in a database connection remaining open for each piece of RemoteArtifact content that was streamed back to the client. This can run a pulp instance out of database connections, if (for example) a downstream Pulp attempts to sync 'immediate' from an on_demand upstream.

The idle connections all show an identical last-query, where they asked for the remote of the not-yet-streamed content. Here's an example:
```
SELECT "core_remote"."pulp_id", "core_remote"."pulp_created", "core_remote"."pulp_last_updated", "core_remote"."pulp_type", "core_remote"."name", "core_remote"."url", "core
_remote"."ca_cert", "core_remote"."client_cert", "core_remote"."client_key", "core_remote"."tls_validation", "core_remote"."username", "core_remote"."password", "core_remote
"."proxy_url", "core_remote"."proxy_username", "core_remote"."proxy_password", "core_remote"."download_concurrency", "core_remote"."max_retries", "core_remote"."policy", "co
re_remote"."total_timeout", "core_remote"."connect_timeout", "core_remote"."sock_connect_timeout", "core_remote"."sock_read_timeout", "core_remote"."headers", "core_remote".
"rate_limit" FROM "core_remote" WHERE "core_remote"."pulp_id" = '949036ee-cb03-4bfd-aaf8-739e2af1a0fc'::uuid LIMIT 21
```

If multiple clients ask for the same content at the same time, you can end up with more idle connections than content-units, due to the occasional concurrency-collisions.

**To Reproduce**

The following bash script will result in one connection to the database for each piece of content in the /rpm-signed fixture:

```
#!/usr/bin/bash
pulp rpm remote create --name test --policy on_demand --url https://fixtures.pulpproject.org/rpm-signed/
pulp rpm repository create --name test --remote test --autopublish
pulp rpm repository sync --name test
pulp rpm distribution create --name test --base-path test --repository test

echo "Connections at start:"
psql -U pulp -d pulp --host 127.0.0.1 -c "SELECT count(pid) FROM pg_stat_activity WHERE datname = 'pulp' and query like '%remote%';"

FILES=(\
b/bear-4.1-1.noarch.rpm \
c/camel-0.1-1.noarch.rpm \
c/cat-1.0-1.noarch.rpm \
c/cheetah-1.25.3-5.noarch.rpm \
c/chimpanzee-0.21-1.noarch.rpm \
c/cockateel-3.1-1.noarch.rpm \
c/cow-2.2-3.noarch.rpm \
c/crow-0.8-1.noarch.rpm \
d/dog-4.23-1.noarch.rpm \
d/dolphin-3.10.232-1.noarch.rpm \
d/duck-0.6-1.noarch.rpm \
d/duck-0.7-1.noarch.rpm \
d/duck-0.8-1.noarch.rpm \
e/elephant-8.3-1.noarch.rpm \
f/fox-1.1-2.noarch.rpm \
f/frog-0.1-1.noarch.rpm \
g/giraffe-0.67-2.noarch.rpm \
g/gorilla-0.62-1.noarch.rpm \
h/horse-0.22-2.noarch.rpm \
k/kangaroo-0.2-1.noarch.rpm \
k/kangaroo-0.3-1.noarch.rpm \
l/lion-0.4-1.noarch.rpm \
m/mouse-0.1.12-1.noarch.rpm \
p/penguin-0.9.1-1.noarch.rpm \
p/pike-2.2-1.noarch.rpm \
s/shark-0.1-1.noarch.rpm \
s/squirrel-0.1-1.noarch.rpm \
s/stork-0.12-2.noarch.rpm \
t/tiger-1.0-4.noarch.rpm \
t/trout-0.12-1.noarch.rpm \
w/walrus-0.71-1.noarch.rpm \
w/walrus-5.21-1.noarch.rpm \
w/whale-0.2-1.noarch.rpm \
w/wolf-9.4-2.noarch.rpm \
z/zebra-0.1-2.noarch.rpm \
)

for f in ${FILES[@]}; do
  echo "FILE ${f}"
  FPATH="test/Packages/${f}"
  ( http GET :/pulp/content/${FPATH} ) &
done

sleep 10
echo "Connections at end:"
psql -U pulp -d pulp --host 127.0.0.1 -c "SELECT count(pid) FROM pg_stat_activity WHERE datname = 'pulp' and query like '%remote%';"
```

**Notes**
* Restarting pulpcore-content releases the idle connections.
* Issuing the GET calls in the script above sequentially results in fewer stale connections (e.g., 22)

**Expected behavior**
Database connection must be closed as soon as the streamed Artifact has been successfully saved.

**Additional context**

See the discussion starting at https://bugzilla.redhat.com/show_bug.cgi?id=2062526#c19 for the investigation that leads us here. The BZ says "deadlock", this problem interferes with verifying that fix, and will have its own BZ "soon".  Some of the discussion on '2526 contains machines and access-info, and is alas private. I have made as much of it public as I reasonably can.



Discussion:

Resolving patch:
diff --git a/pulpcore/content/__init__.py b/pulpcore/content/__init__.py
--- a/pulpcore/content/__init__.py
+++ b/pulpcore/content/__init__.py
@@ -12,10 +12,6 @@
 import django
 
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "pulpcore.app.settings")
-# Until Django supports async ORM natively this is the best we can do given these parts of Pulp
-# run in coroutines. We try to ensure it is safe by never passing ORM data between co-routines.
-os.environ.setdefault("DJANGO_ALLOW_ASYNC_UNSAFE", "true")
-
 django.setup()
 
 from django.conf import settings  # noqa: E402: module level not at top of file
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -648,7 +648,7 @@ async def _stream_content_artifact(self, request, response, content_artifact):
         """
 
         def get_remote_artifacts_blocking():
-            return list(content_artifact.remoteartifact_set.order_by_acs())
+            return list(content_artifact.remoteartifact_set.select_related("remote").order_by_acs())
 
         remote_artifacts = await sync_to_async(get_remote_artifacts_blocking)()
         for remote_artifact in remote_artifacts:


Test patch: