Repository: pulp/pulpcore

Issue:
Pulpcore prevents the use of keep-alive and may exhaust port numbers in large syncs
**Version**
Probably all.

**Describe the bug**
Pulp set's force_close on all downloaders and all aiohttp sessions. This prevents any keep-alive functionality and can lead to an exhaustion of tcp ports for large syncs. Some suboptimal network configuration and you end up with servers closing connections, because they believe they are still in time_wait state.

**Steps to reproduce**
Unclear. But maybe a proxy is involved, multiplying the connection creation dilemma.

**Expected behavior**
Pulp should use HTTP/1.1 features when available.

**Additional context**
Creating millions of tcp connections may even be a performance bottleneck.



Discussion:

Resolving patch:
diff --git a/pulpcore/download/factory.py b/pulpcore/download/factory.py
--- a/pulpcore/download/factory.py
+++ b/pulpcore/download/factory.py
@@ -48,10 +48,6 @@ class DownloaderFactory:
     http://aiohttp.readthedocs.io/en/stable/client_quickstart.html#timeouts Behaviorally, it should
     allow for an active download to be arbitrarily long, while still detecting dead or closed
     sessions even when TCPKeepAlive is disabled.
-
-    Also for http and https urls, even though HTTP 1.1 is used, the TCP connection is setup and
-    closed with each request. This is done for compatibility reasons due to various issues related
-    to session continuation implementation in various servers.
     """
 
     def __init__(self, remote, downloader_overrides=None):
@@ -102,7 +98,7 @@ def _make_aiohttp_session_from_remote(self):
         Returns:
             :class:`aiohttp.ClientSession`
         """
-        tcp_conn_opts = {"force_close": True}
+        tcp_conn_opts = {}
 
         sslcontext = None
         if self._remote.ca_cert:
@@ -133,17 +129,17 @@ def _make_aiohttp_session_from_remote(self):
                     headers["User-Agent"] = f"{headers['User-Agent']}, {user_agent_header}"
                 headers.extend(header_dict)
 
-        conn = aiohttp.TCPConnector(**tcp_conn_opts)
-        total = self._remote.total_timeout
-        sock_connect = self._remote.sock_connect_timeout
-        sock_read = self._remote.sock_read_timeout
-        connect = self._remote.connect_timeout
-
         timeout = aiohttp.ClientTimeout(
-            total=total, sock_connect=sock_connect, sock_read=sock_read, connect=connect
+            total=self._remote.total_timeout,
+            sock_connect=self._remote.sock_connect_timeout,
+            sock_read=self._remote.sock_read_timeout,
+            connect=self._remote.connect_timeout,
         )
         return aiohttp.ClientSession(
-            connector=conn, timeout=timeout, headers=headers, requote_redirect_url=False
+            connector=aiohttp.TCPConnector(**tcp_conn_opts),
+            timeout=timeout,
+            headers=headers,
+            requote_redirect_url=False,
         )
 
     def build(self, url, **kwargs):
diff --git a/pulpcore/download/http.py b/pulpcore/download/http.py
--- a/pulpcore/download/http.py
+++ b/pulpcore/download/http.py
@@ -71,10 +71,6 @@ class HttpDownloader(BaseDownloader):
     allow for an active download to be arbitrarily long, while still detecting dead or closed
     sessions even when TCPKeepAlive is disabled.
 
-    If a session is not provided, the one created will force TCP connection closure after each
-    request. This is done for compatibility reasons due to various issues related to session
-    continuation implementation in various servers.
-
     `aiohttp.ClientSession` objects allows you to configure options that will apply to all
     downloaders using that session such as auth, timeouts, headers, etc. For more info on these
     options see the `aiohttp.ClientSession` docs for more information:
@@ -165,7 +161,7 @@ def __init__(
             self._close_session_on_finalize = False
         else:
             timeout = aiohttp.ClientTimeout(total=None, sock_connect=600, sock_read=600)
-            conn = aiohttp.TCPConnector({"force_close": True})
+            conn = aiohttp.TCPConnector()
             self.session = aiohttp.ClientSession(
                 connector=conn, timeout=timeout, headers=headers, requote_redirect_url=False
             )


Test patch: