Repository: pulp/pulpcore

Issue:
--url file:///foo/bar/PULP_MANIFEST with policy of on_demand or streamed raises unhandled RuntimeError
**Version**

Using the all-in-one pulp container with pulpcore 3.28.1 and pulp-file 1.14.3

**Describe the bug**

With a remote with --url file:///foo/bar/ using either on_demand or streamed policy, when attempting to retrieve a file via wget, a 500 Internal Server Error is returned, and the logs show an unhandled RunTimeError

**To Reproduce**

```
pulp file remote create --name foobar --url file:///foo/bar/PULP_MANIFEST --policy on_demand
...
wget http://myserver/pulp/content/foobar/somefile
...
HTTP request sent, awaiting response... 500 Internal Server Error
2023-07-08 01:19:34 ERROR 500: Internal Server Error.
```

**Expected behavior**

The file is successfully downloaded

**Additional context**

I mentioned this issue in IRC and it was confirmed as reproducible 




Discussion:
Reproduced using pulp_rpm. The problem is definitely in pulpcore. I'm moving this issue to the correct component.

RPM Reproducer:
```
# get a small RPM repo to a local-dir on the Pulp instance
$ cd /tmp
$ wget --recursive --no-parent https://fixtures.pulpproject.org/rpm-signed/
# Sync and distribute it
$ pulp rpm remote create --name foo --url file:///tmp/fixtures.pulpproject.org/rpm-signed/ --policy on_demand
$ pulp rpm repository create --name foo --remote foo --autopublish
$ pulp rpm repository sync --name foo
$ pulp rpm distribution create --name foo --repository foo --base-path foo
$ wget http://localhost:5001/pulp/content/foo/Packages/b/bear-4.1-1.noarch.rpm
--2023-07-12 13:52:28--  http://localhost:5001/pulp/content/foo/Packages/b/bear-4.1-1.noarch.rpm
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:5001... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:5001... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-07-12 13:52:28 ERROR 500: Internal Server Error.
$ 
```

Same traceback in logs:
```
[pulp]  | 127.0.0.1 [12/Jul/2023:13:48:44 +0000] "GET /pulp/content/foo/Packages/b/bear-4.1-1.noarch.rpm HTTP/1.0" 500 315 "http://localhost:5001/pulp/content/foo/Packages/b/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
[2023-07-12 13:52:28 +0000] [1366] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/src/pulpcore/pulpcore/content/authentication.py", line 48, in authenticate
    return await handler(request)
  File "/src/pulpcore/pulpcore/content/handler.py", line 248, in stream_content
    return await self._match_and_stream(path, request)
  File "/src/pulpcore/pulpcore/content/handler.py", line 611, in _match_and_stream
    return await self._stream_content_artifact(
  File "/src/pulpcore/pulpcore/content/handler.py", line 764, in _stream_content_artifact
    response = await self._stream_remote_artifact(request, response, remote_artifact)
  File "/src/pulpcore/pulpcore/content/handler.py", line 1042, in _stream_remote_artifact
    download_result = await downloader.run()
  File "/src/pulpcore/pulpcore/download/base.py", line 257, in run
    return await self._run(extra_data=extra_data)
  File "/src/pulpcore/pulpcore/download/file.py", line 58, in _run
    await self.handle_data(chunk)
  File "/src/pulpcore/pulpcore/content/handler.py", line 1027, in handle_data
    await response.write(data)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_response.py", line 512, in write
    raise RuntimeError("Cannot call write() before prepare()")
RuntimeError: Cannot call write() before prepare()
```

Resolving patch:
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -1012,6 +1012,13 @@ async def handle_response_headers(headers):
 
         async def handle_data(data):
             nonlocal data_size_handled
+            # If we got here, and the response hasn't had "prepare()" called on it, it's due to
+            # some code-path (i.e., FileDownloader) that doesn't know/care about
+            # headers_ready_callback failing to invoke it.
+            # We're not going to do anything more with headers at this point, so it's safe to
+            # "backstop" the prepare() call here, so the write() will be allowed.
+            if not response.prepared:
+                await response.prepare(request)
             if range_start or range_stop:
                 start_byte_pos = 0
                 end_byte_pos = len(data)


Test patch:
diff --git a/pulpcore/tests/functional/api/using_plugin/test_content_access.py b/pulpcore/tests/functional/api/using_plugin/test_content_access.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/tests/functional/api/using_plugin/test_content_access.py
@@ -0,0 +1,40 @@
+"""Tests related to content delivery."""
+import pytest
+import uuid
+
+from pulpcore.client.pulp_file import (
+    RepositorySyncURL,
+)
+
+from pulp_file.tests.functional.utils import (
+    download_file,
+)
+
+
+@pytest.mark.parallel
+def test_file_remote_on_demand(
+    basic_manifest_path,
+    file_distribution_factory,
+    file_fixtures_root,
+    file_repo_with_auto_publish,
+    file_remote_api_client,
+    file_repository_api_client,
+    gen_object_with_cleanup,
+    monitor_task,
+):
+    # Start with the path to the basic file-fixture, build a file: remote pointing into it
+    file_path = str(file_fixtures_root) + basic_manifest_path
+    kwargs = {
+        "url": f"file://{file_path}",
+        "policy": "on_demand",
+        "name": str(uuid.uuid4()),
+    }
+    remote = gen_object_with_cleanup(file_remote_api_client, kwargs)
+    # Sync from the remote
+    body = RepositorySyncURL(remote=remote.pulp_href)
+    monitor_task(file_repository_api_client.sync(file_repo_with_auto_publish.pulp_href, body).task)
+    repo = file_repository_api_client.read(file_repo_with_auto_publish.pulp_href)
+    # Create a distribution from the publication
+    distribution = file_distribution_factory(repository=repo.pulp_href)
+    # attempt to download_file() a file
+    download_file(f"{distribution.base_url}/1.iso")