Issue 1:
Sending an https request with timeout, does not make retries when it gets a timeout.

Code:

import requests
s = requests.session()
s.mount('https://localhost:10000', requests.adapters.HTTPAdapter(max_retries = 3))
s.post('https://localhost:10000', timeout = 3)

Raises this exception (with no retries):

File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 498, in post
return self.request('POST', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
SSLError: _ssl.c:492: The handshake operation timed out

A similar request (even using the same session object) with http instead of https raises the exception Timeout: HTTPConnectionPool(host='localhost', port=10000): Read timed out. (read timeout=1) after 3 retries as expected.

This does not seem to be a general issue with ssl + retries - I've successfully used both

The problem was found on version 2.3.0 both on linux and windows, and also on the latest version - 2.4.3 (tried on linux only).

Issue 2:
Alternative title: A bugfix in urlib3 raises a new bug in requests. (I tried to figure out what changed in urllib3 but was not able to do so).

I believe the issue has to do with these lines (

requests/requests/adapters.py

Lines 509 to 539
in
ac3be98

try:

skip_host = "Host" in request.headers

low_conn.putrequest(

request.method,

url,

skip_accept_encoding=True,

skip_host=skip_host,

)

for header, value in request.headers.items():

low_conn.putheader(header, value)

low_conn.endheaders()

for i in request.body:

low_conn.send(hex(len(i))[2:].encode("utf-8"))

low_conn.send(b"\r\n")

low_conn.send(i)

low_conn.send(b"\r\n")

low_conn.send(b"0\r\n\r\n")

# Receive the response from the server

r = low_conn.getresponse()

resp = HTTPResponse.from_httplib(

r,

pool=conn,

connection=low_conn,

preload_content=False,

decode_content=False,

)

). It seems like the library is uploading the body with a PUT while it should first follow the redirection before uploading the body.

We've traced this down, confusingly, to different behavior for requests depending on the urllib3 version. Essentially, on an older urllib3, python requests will send the entire file twice(!) which can be time-consuming for larger files. This is incorrect behavior. Upon a PUT to a node that responds with a 307/redirect, requests sends the whole body then handles redirects, and then PUT again.

Expected Result

I expected for a PUT request receiving a 307 to redirect and re-PUT (It does) but doesn't send data immediately until after redirects are resolved. This should probably behave more like what curl is doing (see output below).

Actual Result

Data is sent immediately instead of checking for redirects, causing the file to be sent at least twice.

Reproduction Steps

import requests
import subprocess
import logging

# These two lines enable debugging at httplib level (requests->urllib3->http.client)
# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# The only thing missing will be the response.body which is not logged.
try:
import http.client as http_client
except ImportError:
# Python 2
import httplib as http_client
http_client.HTTPConnection.debuglevel = 1

# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

print("working with big file")
fpath = "big_file.txt"
fp = open(fpath, 'rb')

sever = "https:// .... " # our internal server

try:
ret = requests.put(f"{server}", headers={"Content-Type":"application/octet-stream"}, data=fp, allow_redirects = True, stream = True)
print(ret.status_code)
except Exception as exception:
print(exception)

Meanwhile, curl seems to do the RightThing™ here:

$ time curl -L -v https://server/large_file.txt --upload-file ./big_file.txt
* About to connect() to server port 443 (#0)
* Trying ADDRESS...
* Connected to server(ADDRESS) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=CN_INFO
* start date: Jun 27 21:45:02 2022 GMT
* expire date: Aug 01 21:45:02 2023 GMT
* common name: SERVER
* issuer: CN=CN_AUTHORITY
> PUT /large_file.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: server:443
> Accept: */*
> Content-Length: 1048576
> Expect: 100-continue
>
< HTTP/1.1 307 TEMPORARY_REDIRECT
< Connection: Keep-Alive
< Server: XrootD/5.5.5
< Content-Length: 0
< Date: Tue, 20 Dec 2022 14:34:33 GMT
< ETag: "751376756371357696:00000000"
< Location: http://server_redirect/large_file.txt
<
* Closing connection 0
* Issue another request to this URL: 'http://server_redirect/large_file.txt'
* About to connect() to server_redirect port 443 (#1)
* Trying ADDRESS...
* Connected to server_redirect (ADDRESS) port 443 (#1)
> PUT /server_redirect/large_file.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: server_redirect:443
> Accept: */*
> Content-Length: 1048576
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Connection: Close
< Server: OURSERVER
< Date: Tue, 20 Dec 2022 14:34:33 GMT
* We are completely uploaded and fine
< HTTP/1.1 201 CREATED
< Connection: Keep-Alive
< Server: OURSERVER
< Content-Length: 0

System Information

$ python -m requests.help

This (older) version silently works:

{
"chardet": {
"version": "4.0.0"
},
"charset_normalizer": {
"version": "2.0.12"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.10"
},
"implementation": {
"name": "CPython",
"version": "3.7.9"
},
"platform": {
"release": "22.1.0",
"system": "Darwin"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.27.1"
},
"system_ssl": {
"version": "1010113f"
},
"urllib3": {
"version": "1.26.3"
},
"using_charset_normalizer": false,
"using_pyopenssl": false
}

This version breaks (correctly):

{
"chardet": {
"version": null
},
"charset_normalizer": {
"version": "2.1.1"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "3.4"
},
"implementation": {
"name": "CPython",
"version": "3.10.6"
},
"platform": {
"release": "22.1.0",
"system": "Darwin"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.28.1"
},
"system_ssl": {
"version": "1010113f"
},
"urllib3": {
"version": "1.26.13"
},
"using_charset_normalizer": true,
"using_pyopenssl": false
}