Repository: pulp/pulpcore

Issue:
HTTPs Proxies not working.
**Version**
tfm-pulpcore-python3-pulpcore-3.17.7-1.el7.noarch
tfm-pulpcore-python3-aiohttp-3.8.1-2.el7.x86_64

**Describe the bug**
There was HTTPS proxy tunneling support added to the aiohttp library for 3.8.1 => https://github.com/aio-libs/aiohttp/pull/5992
However pulpcore does not 

1. Have the necessary bindings to get this mode to run.
2. Expects the CA Cert of the proxy to be concatenated to the repo's remote CA cert instead of expecting it in the default trust store (as it used to do in pulp2). 

**To Reproduce**
Steps to reproduce the behavior:
- Setup a https proxy or ping me about one
- Add the proxy's cacert to the default trust store.
- Add a repo remote with the feed and proxy urls
- Sync

**Expected behavior**
Clean Sync

**Actual behavior**
SSL error while trying to connect to the proxy.




Discussion:
Bugzilla 
https://bugzilla.redhat.com/show_bug.cgi?id=1993917
This works with Python 3.11. Closing the issue since Pulp is not in the position to update Python for users. 
@dkliban Does this work *immediately* with Python 3.11 or are other patches required?  Because I see Partha's PR which monkeypatches the internal flag that enables this support, but that's not all it does, it does other things too.

Will we need to adopt some or all of those other changes in order for this to work?  If so then this issue ought to be reopened because we won't get this resolved for free by upgrading the Python runtime.
This should just work with Python 3.11. There are no code changes in Pulp required. 
Reopening - while python/3.11 and recent aiohttp will (finally) allow this, Pulp still needs to load the system-allowed certstore to be able to trust an HTTPS proxy's CA.  

Resolving patch:
diff --git a/pulpcore/download/factory.py b/pulpcore/download/factory.py
--- a/pulpcore/download/factory.py
+++ b/pulpcore/download/factory.py
@@ -120,6 +120,8 @@ def _make_aiohttp_session_from_remote(self):
             sslcontext.verify_mode = ssl.CERT_NONE
         if sslcontext:
             tcp_conn_opts["ssl_context"] = sslcontext
+            # Trust the system-known CA certs, not just the end-remote CA
+            sslcontext.load_default_certs()
 
         headers = MultiDict({"User-Agent": DownloaderFactory.user_agent()})
         if self._remote.headers is not None:


Test patch:
diff --git a/pulpcore/tests/functional/api/using_plugin/test_proxy.py b/pulpcore/tests/functional/api/using_plugin/test_proxy.py
--- a/pulpcore/tests/functional/api/using_plugin/test_proxy.py
+++ b/pulpcore/tests/functional/api/using_plugin/test_proxy.py
@@ -4,6 +4,7 @@
 from pulpcore.client.pulp_file import (
     RepositorySyncURL,
 )
+import sys
 
 
 def _run_basic_sync_and_assert(
@@ -163,3 +164,34 @@ def test_sync_http_through_https_proxy(
         file_content_api_client,
         monitor_task,
     )
+
+
+@pytest.mark.parallel
+def test_sync_https_through_https_proxy(
+    file_remote_ssl_factory,
+    file_repo,
+    file_bindings,
+    file_content_api_client,
+    https_proxy,
+    basic_manifest_path,
+    monitor_task,
+):
+    """
+    Test syncing http through an https proxy.
+    """
+    if not (sys.version_info.major >= 3 and sys.version_info.minor >= 11):
+        pytest.skip("HTTPS proxy only supported on python3.11+")
+    remote_on_demand = file_remote_ssl_factory(
+        manifest_path=basic_manifest_path,
+        policy="on_demand",
+        proxy_url=https_proxy.proxy_url,
+        tls_validation="false",
+    )
+
+    _run_basic_sync_and_assert(
+        remote_on_demand,
+        file_repo,
+        file_bindings,
+        file_content_api_client,
+        monitor_task,
+    )