Repository: pulp/pulpcore

Issue:
DNF (still) gets wrong RPM download size for on-demand repo
Re-opening of https://pulp.plan.io/issues/9213, running the supposedly fixed pulp-rpm versions, dnf still gets the wrong sizes for RPM files downloaded from an on-demand repository. Fetching with curl or wget _sometimes_ causes dnf to then pull the correct content, but not other times. Pulp is in use here via Foreman/Katello nightly (updated regularly).

Issue reliably reproduceable with https://rpm.releases.hashicorp.com/. Every newly released RPM has the same issue.

Installed versions (fix for #9213 supposedly in 3.16.0):
- rubygem-pulp_rpm_client-3.16.1-1.el8.noarch
- python38-pulp-rpm-3.16.1-1.el8.noarch
```
[root@alma-canary ~]# dnf upgrade consul -y
Dependencies resolved.
=================================================================================================================================
 Package                 Architecture            Version                      Repository                                    Size
=================================================================================================================================
Upgrading:
 consul                  x86_64                  1.11.2-1                     sihnon_hashicorp_stable-el8                   37 M

Transaction Summary
=================================================================================================================================
Upgrade  1 Package

Total download size: 37 M
Downloading Packages:
[MIRROR] consul-1.11.2.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 38148438 but expected size is: 38490036
[MIRROR] consul-1.11.2.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 38148438 but expected size is: 38490036
[MIRROR] consul-1.11.2.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 38148438 but expected size is: 38490036
[MIRROR] consul-1.11.2.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 38148438 but expected size is: 38490036
[FAILED] consul-1.11.2.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success                      

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Cannot download Packages/c/consul-1.11.2.x86_64.rpm: All mirrors were tried
```

```
[root@alma-canary ~]# wget https://whitefall.jellybean.sihnon.net/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/c/consul-1.11.2.x86_64.rpm
--2022-01-19 20:41:21--  https://whitefall.jellybean.sihnon.net/pulp/content/sihnon/Library/custom/hashicorp/stable-el8/Packages/c/consul-1.11.2.x86_64.rpm
Connecting to whitefall.jellybean.sihnon.net (whitefall.jellybean.sihnon.net)|81.187.154.141|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 38148438 (36M) [binary/octet-stream]
Saving to: ‘consul-1.11.2.x86_64.rpm.5’

consul-1.11.2.x86_64.rpm.5       100%[=======================================================>]  36.38M  11.5MB/s    in 3.2s    

2022-01-19 20:41:24 (11.5 MB/s) - ‘consul-1.11.2.x86_64.rpm.5’ saved [38148438/38148438]

[root@alma-canary ~]# rpm -K --nosignature consul-1.11.2.x86_64.rpm.5 
consul-1.11.2.x86_64.rpm.5: DIGESTS NOT OK
```

Meanwhile downloading directly from upstream fetches the correct size:

```
[root@alma-canary ~]# dnf install --downloadonly consul-1.11.2 --disablerepo=* --enablerepo=hashicorp
Updating Subscription Management repositories.
Hashicorp Stable - x86_64                                                                        1.0 MB/s | 671 kB     00:00    
Last metadata expiration check: 0:00:01 ago on Wed 19 Jan 2022 21:31:04 UTC.
Dependencies resolved.
=================================================================================================================================
 Package                      Architecture                 Version                         Repository                       Size
=================================================================================================================================
Upgrading:
 consul                       x86_64                       1.11.2-1                        hashicorp                        37 M

Transaction Summary
=================================================================================================================================
Upgrade  1 Package

Total download size: 37 M
DNF will only download packages for the transaction.
Is this ok [y/N]: y
Downloading Packages:
consul-1.11.2.x86_64.rpm                                                                         8.9 MB/s |  37 MB     00:04    
---------------------------------------------------------------------------------------------------------------------------------
Total                                                                                            8.9 MB/s |  37 MB     00:04     
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

[root@alma-canary ~]# ls -l /var/cache/dnf/hashicorp-164999f2fbadbd87/packages/
total 105192
-rw-r--r-- 1 root root 38490036 Jan 19 21:31 consul-1.11.2.x86_64.rpm
```


Discussion:
~~Probably the same~~ A similar issue: https://github.com/pulp/pulpcore/issues/2147
@optiz0r Just to be clear on one detail, it's meant to be fixed in pulpcore 3.16, not pulp_rpm 3.16.  What's your pulpcore version?
Nonetheless, I've reproduced it.  It does still seem to just be the first download though (the streamed one) - further hits work as expected.  Which means that `immediate` mode syncing should be OK.


![Screenshot from 2022-01-25 11-02-59](https://user-images.githubusercontent.com/2751996/151014581-b9c822f4-8b12-41fc-987c-2f336d7ee1d0.png)
![Screenshot from 2022-01-25 11-10-31](https://user-images.githubusercontent.com/2751996/151014583-616d1b30-553b-4b17-ae66-16553ff9cad4.png)


Sorry for not specifying the pulp core versions:

```
pulpcore-selinux-1.2.6-2.el8.x86_64
python38-pulpcore-3.16.1-1.el8.noarch
rubygem-pulpcore_client-3.16.0-1.el8.noarch
```

I'm not currently able to reproduce the "first download fails, subsequent downloads succeed" reliably. In fact I have packages that no matter how many times I download them (via dnf, curl, wget), I consistently get the incorrect size.

![image](https://user-images.githubusercontent.com/1030831/151029321-181a5e82-6c2f-4924-8af6-2f27036cfa40.png)
![image](https://user-images.githubusercontent.com/1030831/151029549-aa862c36-b431-471c-b39d-12b620f4c19d.png)

I still haven't been able to reproduce that - yet.  Maybe there's a bug that was unintentionally fixed on the main branch (which is what I'm using) vs. 3.16.

But we can focus on this first issue and see where we end up after fixing that.
Brain dump so I don't lose anything:

The size of the "incorrectly" downloaded file is identical to the `Content-Length` of the response from original server / repo.  The difference is that the original response has `Content-Encoding: gzip` set which tells the client "I have applied this encoding, you should reverse it to get the correct data" and the streamed response from Pulp does not.  ~~Thus for the first (streaming) download, depending entirely on the configuration of the remote server / repo, clients of Pulp could end up getting with a gzip-compressed version of the file they were supposed to get because Pulp forgot to tell them to decode the file.~~

[edit] nope, Pulp clients always get the decompressed bytes. But the `Content-Length` header is forwarded from the original request, and it seems that when the payload's actual size differs from `Content-Length`, the header wins, and the payload gets truncated.  And that's why the files differ.

```
(pulp) [vagrant@pulp3-source-fedora35 ~]$ cmp -n 31024532 consul-1.11.0-1.x86_64.rpm consul-1.11.0-1.x86_64.rpm.1 
(pulp) [vagrant@pulp3-source-fedora35 ~]$ cmp -n 31024533 consul-1.11.0-1.x86_64.rpm consul-1.11.0-1.x86_64.rpm.1 
cmp: EOF on consul-1.11.0-1.x86_64.rpm after byte 31024532, in line 121994
```

(confirming that the file streamed from Pulp is a prefix of the file downloaded into Pulp)

The reason it isn't broken most of the time is that most of the time the remote server / repo doesn't try to gzip-compress RPM files.  It's kind of pointless since they're already internally compressed.

The artifacts that get saved into Pulp's datastore seem to be *correct*, i.e. they are the decompressed versions - on the latest release.  **TODO**: investigate if this was ever not the case in the past and whether that might cause "RPMs always have the wrong sizes"

If the actual cause is the `Content-Length` header being forwarded when the upstream server uses `Content-Encoding`, then the options are:

* Find a way to use a restrictive `Accept-Encoding` that prevents the server from setting `Content-Encoding` (disadvantage: for *some* files, but probably not RPMs, this might be less efficient)
* Disable `Content-Length` header for all streamed responses (disadvantages: TBD, but `Content-Length` is generally useful)
* Disable `Content-Length` header for all streamed responses IF the remote server used `Content-Encoding` (disadvantages: TBD)
* Some magic not known to me which that would allow us  to know the size of the decompressed file while still in the process of downloading the compressed bytes?
Having dipped a toe into the downloader an streaming logic, i really think forwarding the headers from the upstream server we are downloading in the very same moment is probably a convenient but bad idea. We should really provide the same headers to the client whether we serve streaming or form saved artifacts.

Resolving patch:
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -102,6 +102,7 @@ class Handler:
     hop_by_hop_headers = [
         "connection",
         "content-encoding",
+        "content-length",
         "keep-alive",
         "public",
         "proxy-authenticate",
@@ -833,10 +834,9 @@ def cast_remote_blocking():
         async def handle_response_headers(headers):
             for name, value in headers.items():
                 lower_name = name.lower()
-                if lower_name in self.hop_by_hop_headers:
-                    continue
-
-                if response.status == 206 and lower_name == "content-length":
+                if lower_name not in self.hop_by_hop_headers:
+                    response.headers[name] = value
+                elif response.status == 206 and lower_name == "content-length":
                     content_length = int(value)
                     start = 0 if range_start is None else range_start
                     stop = content_length if range_stop is None else range_stop
@@ -849,9 +849,6 @@ async def handle_response_headers(headers):
                     response.headers["Content-Range"] = "bytes {0}-{1}/{2}".format(
                         start, stop - 1, content_length
                     )
-                    continue
-
-                response.headers[name] = value
             await response.prepare(request)
 
         data_size_handled = 0


Test patch: