Raibek/requests-github-issues huggingface dataset:
Misusing of system-wide proxy on Windows / HTTPS connections over system-wide proxy on Windows do not work

<!-- Summary. -->

On Windows 10 and 11, if a system-wide proxy is configured in the OS settings, Requests won't load HTTPS websites.
Instead of the regular CONNECT header, Requests sends TLS request to the proxy server (ip and port). The TLS ClientHello does not have Server Name Indication field, that's why I assume that Requests misinterprets the proxy as a Secure Web Proxy (a regular proxy with TLS encapsulation on top), which Windows does not support.

## Expected Result

If system-wide proxy is configured in Windows, Requests sends CONNECT header to the proxy server when HTTPS website is requested.

## Actual Result

If system-wide proxy is configured in Windows, Requests sends TLS request to the proxy address when HTTPS website is requested.

## Reproduction Steps

1. Configure system-wide proxy on Windows
2. 
```python
import requests
requests.get("https://w3.org/")
```

Result:
My proxy does not respond to the packets it does not understand, and I receive the following error:
`HTTPSConnectionPool(host='w3.org, port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:1114: The handshake operation timed out')))`

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.4"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010ef"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


This is a Python's urllib bug, which is triggered by newer urllib3 version, thus affecting requests.

https://bugs.python.org/issue42627
https://github.com/python/cpython/issues/86793
https://github.com/python/cpython/commit/9743524fc14db39e2f309cf3d372c7e8e18fc93a

>This bug has existed for a very long time (since Python 2.0.1 if not earlier), but it was exposed recently when urllib3 added support for HTTPS-in-HTTPS proxies in version 1.26. Before that, an `https` prefix on the HTTPS proxy url was silently treated as `http`, accidentally resulting in the correct behavior.

Fixed in Python 3.10.5, all the previous versions are still affected.

https://github.com/urllib3/urllib3/issues/2164
Allow tests to run if HTTP proxy env variables are already present,

Hello,

The following tests fail if HTTP proxy environment variables are already set:

```
FAILED tests/test_requests.py::TestRequests::test_mixed_case_scheme_acceptable
FAILED tests/test_requests.py::TestRequests::test_HTTP_302_ALLOW_REDIRECT_GET
FAILED tests/test_requests.py::TestRequests::test_errors[http://doesnotexist.google.com-ConnectionError]
FAILED tests/test_requests.py::TestRequests::test_respect_proxy_env_on_send_self_prepared_request
FAILED tests/test_requests.py::TestRequests::test_respect_proxy_env_on_send_session_prepared_request
FAILED tests/test_requests.py::TestRequests::test_respect_proxy_env_on_send_with_redirects
FAILED tests/test_requests.py::TestRequests::test_respect_proxy_env_on_get - ...
FAILED tests/test_requests.py::TestRequests::test_respect_proxy_env_on_request
```

The variables affecting the tests are:

 * `http_proxy` (and `https_proxy` potentially)
 * `no_proxy`

Fixed by overriding their value if they are already present in the environment.

The `override_environ` function would fail if attempting to delete a variable that is not in the environment. Added a `suppress(KeyError)` construct around the `del` instruction to fix it.

Thanks,
Oilvier
Fix output of test_lowlevel tests in case of timeout

Hello,

The tests in `test_lowlevel.py` use the `testserver.server.Server` thread object.

When using the object as a context manager, we wait until the server thread is ready with a timeout.

Unfortunately, if the timeout is reached, we would not propagate the error. Instead, we would return from the `__enter__` method and access the `with` body with an uninitialized value for `port` (i.e., `port = 0`).

Therefore, the tests would fail with:

```
E           urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc3a3e30a90>: Failed to establish a new connection: [Errno 111] Connection refused

/usr/lib/python3/dist-packages/urllib3/connection.py:181: NewConnectionError
```
Reading this error does not make it obvious that this is the result of a timeout.

Fixed by raising an error in case of timeout, which makes it more obvious:

```
        if not self.ready_event.wait(self.WAIT_EVENT_TIMEOUT):
>           raise RuntimeError("Timeout waiting for server to be ready.")
E           RuntimeError: Timeout waiting for server to be ready.

tests/testserver/server.py:119: RuntimeError
```
_The reason why I had a timeout on my end was because my environment was not resolving "localhost" as it should have. Requests were going through the DNS and therefore ended-up hitting the timeout_

_The name resolution taking forever was happening during this instruction (where `self.host` is `localhost`):_
```
sock.bind((self.host, self.port))
```

Thanks,
Olivier
Make requests work for pyright in strict mode

Running pyright in strict mode errors currently for requests function calls:

```
  48:19 - error: Type of "post" is partially unknown
  Type of "post" is "(url: str | bytes, data: str | bytes | MutableMapping[str, Any] | Iterable[Tuple[Text, str | None]] | IO[AnyStr] | None = ..., json: Any | None = ..., **kwargs) -> Response" (reportUnknownMemberType)
  59:18 - error: Type of "json" is partially unknown
  Type of "json" is "(**kwargs) -> Any" (reportUnknownMemberType)
```

## Expected Result

Basic usage of `requests` should pass strict typechecking

## Actual Result

Failure

## Reproduction Steps

```python
# pyright: strict
import requests

r = requests.post("http://httpbin.org/post", json={"a": 1})
print(r.json())
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.12"
  },
  "platform": {
    "release": "19.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010ff"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


Hi @brandonchinn178, we don't provide any typing information in requests currently. This is either being derived from pyright or a type stubbing implementation. Neither of those are in our control, so you may need to direct this elsewhere.

ah yes, sorry, thanks!

Moved to https://github.com/python/typeshed/issues/7869
How to handle this issue ? - [Errno 110] Connection timed out') - Max retries exceeded with url - HTTPSConnectionPool(host=..., port=443

Good evening,

After having searched in stackoverflow for answers to my question, I was not able to find any solution. Currently I'm trying to use the request package to retrieve environmental data from a gobernment site by using url get requests. When I do the request through postman it works, but surprisingly with my python (jupyter notebook) scripts it does not.

It was working perfectly before but after starting using sessions ( s = requests.Session()) then it began to produce problems.

This is the code:
s = requests.Session()
my_uri = '.....'
r_metadata= s.get(my_uri, headers={'Authorization': 'ApiToken ..............'})
print(r_metadata.status_code)

---------------------------------------------------------------------------------------------------------
This is the error encounter:
HTTPSConnectionPool(host='.....', port=443): Max retries exceeded with url: ....... (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff8256a0310>: Failed to establish a new connection: [Errno 110] Connection timed out')


------------------------------------------------------------------------------------
What I think is that these persistent session from requests are still alive, and for that reason the third party server from which I'm trying to get the data, dont allow me to enter anymore and take the information(Max retries exceeded with url:).

Would you mind giving me a suggestion or solution?.

Cordially,

Juan David Briceno Guerrero

Hi @juandavidbriceno, what you're observing is likely intentional behavior from the website to prevent web scraping. Requests often ends up on fingerprinting lists which CDNs or websites can use to prevent programmatic access. The service is refusing to respond within the default connection timeout, which implies it's just rejecting the connection.

You can try extending the timeout which there is guidance for in the docs. Otherwise, I don't believe we'll be able to help because this is a server side issue.
Requests taking too long or stucking ( GNU-Linux ) 

requests are taking more than 1 minute to process if they succeed at all on 2 of my linux machines at home . 
On windows a simple : 
```
>>> import requests as r
>>> r.get("http://google.com")
```
takes miliseconds , 
on debian bullseye takes more than 1 minute for simple requests,
and sometimes they are "freezing"

Any hints on how to debug this welcome.
I just upgraded to latest and still the issue persist

This isn't debuggable without a packet capture or logs, can you provide those? You can use Wireshark to capture packets and look at the timings.

I added a small timeout and its fixed now. Thanks for the hint
Fix error caused by invalid charset in response headers

Ran into an error where we were having an exception raised from the requests library: `'bool' object has no attribute 'strip'`

Seems the server was returning an empty charset value in the Content-Type response header of `application/json;charset`. This was causing the line below to default the value to a boolean `True`:

https://github.com/psf/requests/blob/2a6f290bc09324406708a4d404a88a45d848ddf9/requests/utils.py#L524-L529

This meant that attempting format the charset on the line below, the value was a boolean rather than a string:

https://github.com/psf/requests/blob/2a6f290bc09324406708a4d404a88a45d848ddf9/requests/utils.py#L545-L548

This change adds a type check so that the charset is ignored if a value is not provided. The unit test has been modified for coverage, along with expanding the other test scenarios slightly.

Thanks for the PR, @deosrc! You've chosen an interesting time to raise this. We're currently working on options for typing in Requests and this was a case where there actually appears to be a bug in the code. Note that the response the server is giving you is not a valid response though and should be fixed on their end.

`True` was chosen for a default in #4442 but doesn't make sense according to the [RFC](https://datatracker.ietf.org/doc/html/rfc7231#section-3.1.1.1). A parameter is defined to be:
```
     media-type = type "/" subtype *( OWS ";" OWS parameter )
     parameter = token "=" ( token / quoted-string )
```

Which means the minimum viable `parameter` is `<token>=`. If we read farther in the RFC we find that quoting in parameters is interchangeable without affecting the semantics so `<token>=` is equivalent to `<token>=""`. Our default in this case should just be an empty string rather than `True`.

 We don't rely on value being boolean anywhere and there's no way to leave the function with the boolean and emit a successful request. The fix we'd proposed on the typing branch was to change the default. This will result in an empty `charset=` being treated as UTF-8 when decoding rather than an unhelpful ValueError.

> Note that the response the server is giving you is not a valid response though and should be fixed on their end.

Absolutley, this has been fixed.

Your response all makes sense to me and sounds like it's fixed in the typing branch. I'll leave this up to you if you want to merge it in as an fix while the typing is ready, or for those unable to use typing. Happy for it to just be closed if it's not worth it as it's a rather obscure bug.
How to implement oauth token on requests library?

Thanks
netanel.cyber@gmail.com

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

I'm sorry but the [stackoverflow's](http://Stackoverflow.com )account No longer accepting questions and So I write the question here and not in [STACKOVERFLOW](http://Stackoverflow.com)

thanks
netanel

Stackoverflow is definitely still accepting questions.
Git.io deprecation notice


Hello, I've found that some of your files in this repo is still using https://git.io/, and this service will be discontinued in the future. (ref: [Git.io deprecation](https://github.blog/changelog/2022-04-25-git-io-deprecation/))
Theses files are: .github/workflows/codeql-analysis.yml

> This is an automated message from NPMMirror(https://github.com/npmmirror), a bot developed by Nova Kwok(https://github.com/n0vad3v).
    

For reference the line being flagged is a comment from the original source of this action, it's not used in the code itself.

Given there's no impact on our infrastructure, I'm going to leave the comment in place as it appears the resource will remain available as read-only.
What is the correct way to pass the proxy username and proxy password ?

Hey,

After reading and trying out many things, I am confused how the proxy username and password is passed in the follows -

proxy = {'https' : 'http://{username}:{password}@host:port',
              'http': 'http://{username}:{password}@host:port' }
requests.get(url, proxies=proxy)

1. Do I have to use quote() from urllib.parse to encode the username and password before making the proxy dict ?
2. My username , password having "@" sign works fine without encoding, Still do I have to encode for any other chars ?

Version
requests = 2.20.0
urllib3 = 1.24.2




Hi @pghole,

It's not clear from the post if you're encountering an issue or asking for general information about URL syntax. You can find the relevant information on URL syntax in the [RFC](https://datatracker.ietf.org/doc/html/rfc3986#section-3.2.1). Requests will simply proxy along whatever characters you provide, so as long as they conform to the RFC 3986 specification, we'd expect them to work.

@nateprewitt, I am interested in knowing the behaviour of request library and how the username/password is passed to the proxies dictionary

For example -(having special chars)
username = "username@#$"
password = "password@#"

which of the following is correct ?

'http://{username}:{password}@host:port'. format(username, password)

OR
from urllib.parse import quote
'http://{username}:{password}@host:port'. format(quote(username), (password))

@nateprewitt has answered your question then
Rewrite version checks and split using iteration protocol

Make use of `map` and sequence unpacking in an iteration context to make the codes a little bit declarative.

Hi @oyeyipo, thanks for the PR. This is actually a duplicate of #5334 and #5281. Please make sure to check open and closed PRs before opening new ones.

Hey @nateprewitt, thanks for the suggestion. Did do a cursory scan of opened pr first page. Now, took a practical look at search techniques at [github pr search](https://docs.github.com/en/enterprise-cloud@latest/search-github/searching-on-github/searching-issues-and-pull-requests) and also search for filename and code, could not surface this and the other closed PRs. How would you go about this? thanks.

Searching either the function name "[check_compatibility](https://github.com/psf/requests/pulls?q=is%3Apr+check_compatibility+is%3Aclosed)" or phrase "[version check](https://github.com/psf/requests/pulls?q=is%3Apr+version+check+is%3Aclosed+)" like you've used will return the previous issues.

confirmed :thumbsup: thanks
Session.close() doesn't work within WSL2

<!-- Summary. -->

## Expected Result

<!-- What you expected. -->

## Actual Result

<!-- What happened instead. -->
I have a nodejs server uses session to store user login info and the auth will immediately invalidated after session closed. However, it still has access to the server after session closed. 
## Reproduction Steps

```python
import requests
s=requests.Session()
url_login = 'http://localhost:8080/login'
url_postStatus = 'http://localhost:8080/postStatus'
loginInfo={'UserEmail':test@test.test','UserPassword':'test'}
s.post(url_login,data=loginInfo)
s.post(url_postStatus)
s.close()
s.post(url_postStatus)
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.2"
  },
  "platform": {
    "release": "5.10.16.3-microsoft-standard-WSL2",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010bf"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


This isn't a bug. You haven't read the documentation to understand what session.close does

> This isn't a bug. You haven't read the documentation to understand what session.close does

Hi @sigmavirus24 , it's my bad that I didn't understand what session.close does. As I read from the documentation, quotes here: "Closes all adapters and as such the session". If that doesn't mean invalidate the current session, I guess I totally misunderstand the purpose of this method.
May I ask what is the appropriate way to close the current session?
Potential security vulnerability

A potential security vulnerability has been disclosed for this repo https://huntr.dev/bounties/2c5789ab-213c-4568-9380-683d0612f1aa/ 3 months ago .
But i see it still not addressed .
Kindly validate the bug

@ranjit-git we don't have an interest in third-party applications which require access to personal information to view the vulnerability.

Please follow our [security disclosure instructions](https://docs.python-requests.org/en/latest/community/vulnerabilities/).

@nateprewitt huntr.dev team already sent you the report details  to provided email in https://docs.python-requests.org/en/latest/community/vulnerabilities/ .
You should have received the report details in mail already . In that mail there is a magick unique link , by using this you can view the report details without signup . In mail just search for "https://huntr.dev" .
I will ask the admin  to send the report details again .

https://huntr.dev/ is opensource bugbounty platform running since last 2.5 year
Requests not returning full html

When using requests.post/get(url...) The html that is returned by using request.content/request.text is not the full html.

For example I am trying to code a bot that will get data from a website called investopedia. This html is cut short and does not return the same html that can be seen in inspect element.

## Reproduction Steps

```python
import requests

with open("test.html", "w") as file:
    file.write(requests.get("https://www.google.com/").text)

If you compare the html of test.html to the actual google it is very different
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "3.0.4"
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.6"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010bf"
  },
  "urllib3": {
    "version": "1.24.2"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}
```


Hi @YeetmanLord, please check open and closed issues before filing new ones. You'll find most servers either truncate or completely change content for scrapers to reduce bandwidth or access. This is the behavior you're encountering with Google. The content you receive is still a complete and valid html file.
Add support for non-latin1 basic auth encodings (such as utf-8)


Hi @timbedard, thanks for the PR. Unfortunately, I don't believe we'll be able to accept it. The high level Requests APIs have been frozen for the majority of 2.x and we don't intend to change those interfaces.

I believe the general consensus has also been to favor bytes in most cases going forward. It removes any guessing (which is impossible to do right consistently) and avoid us leaking an `encoding` argument on to every interface in Requests.

Oh hey, no worries. It was a draft for a reason. I came to the same conclusion shortly after this and just came back to close it. Thanks for the fast response and keep up the good work!
Add automatic code formatting to Requests

### Overview
This is a draft PR for adding standardized code formatting to Requests. Now that we intend to drop Python 2 support, we have the opportunity to clean up multiple conflicting code styles that have crept into Requests over the years.

This PR will introduce a few standard changes:

* Integration of `black` for code formatting, bringing Requests in-line with what has largely become the standard.
* Integration of `isort` to ensure our imports are automatically grouped and cleaned.
* Integration of `pyupgrade` which will automatically pull most of the code base to Python 3.7 syntax.
* Addition of a `.pre-commit-config.yaml` to orchestrate the above tools.

### TODO
- [x] Rebase on to #6091 before merge
- [x] Resolve formatting issues with server response line wrapping in `tests/test_lowlevel.py`
- [x] Add .git-blame-ignore-revs file (#6116)
- [x] Fix inconsistent f-string upgrades
- [x] Enable and fix flake8 issues

https://github.com/psf/requests/blob/main/docs/dev/contributing.rst#kenneth-reitzs-code-style is no longer necessary if you're formatting the code
HTTPBasicAuth does not handle non-latin-1 characters

Hi,
I have a password containing Norwegian characters, like "ø". I found that `HTTPBasicAuth` generates a wrong `Authentication` header, as the base64-encoded password differed from what I computed and successfully tested.

It looks like `latin-1` is the default encoding instead of  `utf-8`:
https://github.com/psf/requests/blob/79f60274f7e461b8fd2f579e741f748438d7eadb/requests/auth.py#L59-L63

My workaround is to encode strings to `utf-8` before passing them to `HTTPBasicAuth`, but it looks like this would break starting with `requests >= 3.0.0`:

https://github.com/psf/requests/blob/79f60274f7e461b8fd2f579e741f748438d7eadb/requests/auth.py#L36-L41

I do not know if different servers have different defaults, but if `requests` do not want to handle binary sequences, then it should probably allow specifying different encodings.

Hi @frafra, this warning is a remnant from when Python 2 was still the main Python and `bytes`/`str` were synonymous. You'll see if you look at the definition for [`basestring`](https://github.com/psf/requests/blob/main/requests/compat.py#L79) that it is that same type union in Python 3. We may be able to make "Non-string" a bit clearer for Python 3.

`bytes` will continue to be supported going forward and would in fact be our general recommendation for most Requests arguments. The library still assumes `latin-1`/`iso-8859-1` because that's what's formally documented for the web. You'll find in practice though that `utf-8` has superseded that in many cases.
request.get adds %20 to the end of the link and gives a 404 error

I am browsing the links from a list, the links are working, the headers are also defined, but when I send a request with requests.get, the response returns 404. When I look at it with debug, it adds %20 to the end of the main url, so the links are broken and the error returns. How do I solve this?
i am giving an example
mylink= https://example.net///2022/V/0/1/p/2710/583/742/2/w/2048/2710583742_6_2_1.jpg?ts=1645610622959
error= https://example.net///2022/V/0/1/p/2710/583/742/2/w/2048/2710583742_6_2_1.jpg?ts=1645610622959%20

Hi @Turkmen1Mehmet,

Please make sure to read the issue template before filing new issues. We won't be able to help with general usage questions on this issue tracker, those will need to be directed to [StackOverflow](https://stackoverflow.com/). Generally speaking, %20 denotes whitespace, so you'll likely want to take a close look at your url string.
Is there any disconnection detection exist on Stream feature?

I have a post request to a stream API server. I tried to simulate disconnection during streaming. However, in my case, if I disconnect my connection to the API server which streams data, "requests" continues without breaking the connection or raising any exceptions. Is there any help to overcome this issue? Or is there a feature needs to be added to this kind of scenario?

Hi @VahidTa,

If you're receiving incomplete data from the broken connection and no error, this is likely an issue in urllib3 that's been [fixed](https://github.com/urllib3/urllib3/pull/2514) in the upcoming urllib3 2.0. We don't have control over detecting this in Requests unfortunately though.

I think it is good idea to have "keep-alive" like mechanism for connection status in streaming.
Remove support for Python 2.7, 3.6 and Pypy3.6

This PR is a proposal to officially remove support for Python 2.7 as discussed in #6023. The next minor release (Requests 2.28.0) will no longer provide support for any of the end of life Python runtimes. Users are encouraged to upgrade to Python 3.8 or later. If you're unable to do so, we'd recommend pinning to `requests<2.28`.

This will be the first of a few PRs that will be doing some general code clean up now that we're moving to a Python 3-only code base. I'll post the others as drafts shortly.

Thanks @hugovk, those are all great call outs! `use_chardet_on_py3` is an unfortunate misnomer because it just forcing use of `chardet`. The only reason it references py3 is because chardet was the default for Python 2. I don't think we can change the extra name at this point.

I addressed the rest of @sethmlarson's comments too and added a hard failure in the setup.py for anyone installing on `pip<9`.
POST a Multipart-Encoded File freezes response code 500

<!-- Summary. -->

I am trying to send a single PDF/DOCX file using a single rest API and performing a single Post request.

<!-- What you expected. -->

When I run the code it hangs for almost 3 min, and then I got a response `{"error":true,"code":500,"message":"Error: Timed Out"}` from the server where I am trying to send the file to,

<!-- What happened instead. -->

## Reproduction Steps

```python

def send_document(ID, document):


    headers = {
    }

    params = {
        "key": userKey,
    }

    data = {
        "id": ID
    }

    files= {
    'file': ('application.pdf', open(document, 'rb'), 'application/pdf')
    }

    response = post(f"{baseUrl}message/doc", headers=headers, params=params, data=data, files=files).content
    
    print(response)

```

## System Information

    $ python -m requests.help

```json
{
 {
  "chardet": {
    "version": "3.0.4"
  },
  "charset_normalizer": {
    "version": "2.0.7"
  },
  "cryptography": {
    "version": "36.0.0"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.0"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "101010cf",
    "version": "21.0.0"
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010cf"
  },
  "urllib3": {
    "version": "1.26.7"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->
Requests does not pass 'params

<!-- Summary. -->

## Expected Result

<!-- What you expected. -->

## Actual Result

<!-- What happened instead. -->

## Reproduction Steps

```python
import requests

```

## System Information

    $ python -m requests.help

```json
{
  "paste": "here"
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


Sorry, I did not intend to open two issues...
Session preparation does not preserve the underlying cookie jar policy

Python (through `http.cookiejar`) allows different cookie policies to be set to customize the behavior of certain attributes (e.g secure); but on the `prepare_request` stage of sessions, `requests` looses the information regarding the custom policy.

https://github.com/psf/requests/blob/79f60274f7e461b8fd2f579e741f748438d7eadb/requests/sessions.py#L433-L435

More specifically, the `merge_cookies` method only carries out the policy of the newly created `RequestsCookieJar`, even though the cookie jar that is bound to that session might have different properties (including a policy)

## Reproduction Steps

```python
import json
import requests
from http import cookiejar
from requests.cookies import RequestsCookieJar


class AllowEverythingPolicy(cookiejar.DefaultCookiePolicy):
    def return_ok_secure(self, cookie, request):
        """Check whether the given cookie is sent to a secure host."""
        return True


jar = RequestsCookieJar(policy=AllowEverythingPolicy())
jar.set('secure_cookie', 'value', secure=True)
jar.set('insecure_cookie', 'value', secure=False)

session = requests.Session()
session.cookies = jar

request = requests.Request(
    'GET',
    'http://localhost',
)
prepared_request = session.prepare_request(request)
print(prepared_request.headers['Cookie'])
```

## Expected Result

```
insecure_cookie=value; secure_cookie=value
```

## Actual Result

```
insecure_cookie=value
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "4.0.0"
  },
  "charset_normalizer": {
    "version": "2.0.10"
  },
  "cryptography": {
    "version": "3.4.8"
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.5"
  },
  "platform": {
    "release": "5.11.0-49-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "101010cf",
    "version": "20.0.1"
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010af"
  },
  "urllib3": {
    "version": "1.26.8"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


Hi @isidentical, thanks for the ticket. In the future please check open and closed issues before posting new ones. This is a long standing issue in Requests that has had a number of attempts to address. The main problem is doing this in a non-breaking manner. I'm going to resolve this in favor of the existing [ticket](https://github.com/psf/requests/issues/5449).

Relevant:
#3416
#3463
#4000
#4042
#4513
response.json() raises inconsistent exception type

This is a follow up on https://github.com/psf/requests/issues/5794, the issue is marked as resolved, but it's not completely resolved.

## Expected Result

When no encoding is returned, you have simplejson installed and the content is not JSON, a requests.exceptions.JSONDecodeError should be raised.

## Actual Result

A simplejson.errors.JSONDecodeError is raised.

## Reproduction Steps

In the PR, related to the issue mentioned, the JSONDecodeError is only caught when there's an encoding (https://github.com/psf/requests/pull/5856/files#diff-633feaee006d302c2988a1f73e39e76a55ec077b595d8c0a093f72d75fb1799cR909-R911), this except clause should also be there a few lines above (when there's no encoding).


Hi @JeremyVriens, thanks for bringing this to our attention! This was definitely a miss in the original PR. I've pushed up a quick test and fix (#6097) to appropriately catch and wrap the JSONDecodeError. Please let us know if you have any comments or concerns on the PR, otherwise, this should be addressed in the next minor release.

We've gone ahead and merged #6097. I'll close this since it's now on the path for release, but please feel free to leave any further comments here or on the PR.
ValueError is raised instead of InvalidHeader

When using an invalid header name it is not checked in the `requests.utils.check_header_validity` function, only the value is checked.

This causes the underline HTTPConnection to fail when trying to write the header on `ValueError`, seems like that it is checked against:
`_is_legal_header_name = re.compile(rb'[^:\s][^:\r\n]*').fullmatch`

The expected exception, in this case, will be `InvalidHeader`

```python
import requests
requests.get("https://www.google.com", headers={":bad": "header"})
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.10"
  },
  "platform": {
    "release": "5.4.0-1068-aws",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "1010100f"
  },
  "urllib3": {
    "version": "1.26.8"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}
```


In `requests.utils.check_header_validity` there is a validation only for value, I think we have to validate both name and value of the header. 

As per the IMF (refer [https://www.rfc-editor.org/rfc/rfc2822#section-2.2](https://www.rfc-editor.org/rfc/rfc2822#section-2.2)) 

> A field name MUST be composed of printable US-ASCII characters (i.e., characters that have values between 33 and 126, inclusive), except colon.

I figured the fix for this one (the regex for validating header name goes like this `'^[!-9\[;-~]*$'`), I am a first timer, but seasoned developer. Please assign it to me, I would like to contribute.

Thanks for volunteering @Khasim481. The RFC linked is quite old and has been obsolete for almost a decade. I believe you're looking for the definition in the current RFC [7230](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2). You'll find the exact ABNF is 1 or more `tchar` as defined in [section 3.2.6](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.6).

We strictly adhere to the spec for field-values but it's correct we omitted the field-name when this was first done. The change was explicitly to solve security risks with header values. This issue is more cosmetic, but I do agree we should have consistency with the error type. We actually already have a [branch](https://github.com/psf/requests/tree/improved_header_validation) in place with most of the fix for this but it's still needs some final refactoring.

The changes in the [branch](https://github.com/psf/requests/tree/improved_header_validation) looks perfect, If you mention those cosmetic changes broadly here, I can be of help.

@Khasim481 Any estimation when this will be fixed? 

It's currently a candidate fix for the next release, 2.28.0.

This has been merged in #6154 and will be available in the next release. Thanks everyone!
Call request api very slowly 

<!-- Summary. -->
Macos: Monterey (12.0.1)
Python call request API very slowly (>8 minutes).
Please check the video recording screen bugs

## Expected Result

<!-- What you expected. -->

## Actual Result

<!-- What happened instead. -->

## Reproduction Steps

```python
import requests
# This is a sample Python script.im
import requests
import logging
import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

def print_hi(name):
    # Use a breakpoint in the code line below to debug your script.
    print(f'Hi, {name}')  # Press ⌘F8 to toggle the breakpoint.


# Press the green button in the gutter to run the script.
if __name__ == '__main__':
    print_hi('PyCharm')
    r = requests.get('https://www.google.com/')
    print(r.text)

```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.2"
  },
  "platform": {
    "release": "21.1.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010df"
  },
  "urllib3": {
    "version": "1.26.8"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->
Video Bugs:






Hi @workingxx92, this isn't something that is affected by Requests, either your IDE is halting at a break point or there's something in your local network delaying the response. We typically wouldn't expect the connection to the server to be kept open for that long to begin with.

For questions on debugging, please use [StackOverflow](https://stackoverflow.com/). We won't be able to provide help with that on this issue tracker.
How to uinstall requests library using setup.py?

We are in a factory environment where we cannot use pip. 
We installed request library using python install setup.py.
Is it possible to uninstall requests library using setup.py. Please share the command. 

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0335' in position 19: ordinal not in range(256)

https://github.com/psf/requests/blob/79f60274f7e461b8fd2f579e741f748438d7eadb/requests/auth.py#L60

I got the below error on authenticating a Gmail user in my app, P.S: I have client-id and client-secret stored in a file from GCP project.

`File "/venv/lib/python3.9/site-packages/google_auth_oauthlib/flow.py", line 240, in fetch_token
  return self.oauth2session.fetch_token(
File "/venv/lib/python3.9/site-packages/requests_oauthlib/oauth2_session.py", line 341, in fetch_token
  r = self.request(
File "/venv/lib/python3.9/site-packages/requests_oauthlib/oauth2_session.py", line 521, in request
  return super(OAuth2Session, self).request(
File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 515, in request
  prep = self.prepare_request(req)
File "/venv/lib/python3.9/site-packages/requests/sessions.py", line 443, in prepare_request
  p.prepare(
File "/venv/lib/python3.9/site-packages/requests/models.py", line 322, in prepare
  self.prepare_auth(auth, url)
File "/venv/lib/python3.9/site-packages/requests/models.py", line 558, in prepare_auth
  r = auth(self)
File "/venv/lib/python3.9/site-packages/requests/auth.py", line 96, in __call__
  r.headers['Authorization'] = _basic_auth_str(self.username, self.password)
File "/venv/lib/python3.9/site-packages/requests/auth.py", line 60, in _basic_auth_str
  username = username.encode('latin1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0335' in position 19: ordinal not in range(256)
`
Throwing a UnicodeEncodeError


Traceback (most recent call last):
  File "/usr/share/doc/hackingtool/hackingtool.py", line 109, in <module>
    all_tools.show_options()
  File "/usr/share/doc/hackingtool/core.py", line 161, in show_options
    self.show_info()
  File "/usr/share/doc/hackingtool/hackingtool.py", line 71, in show_info
    print(logo + '\033[0m \033[97m')
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 9-10: ordinal not in range(256)


Hi @killabit,

Nothing in this stacktrace is related to Requests. We won't be able to help here.
Drop EOL Python 3.6

See:

- https://endoflife.date/python
- https://www.python.org/dev/peps/pep-0494/#lifespan

We haven't even dropped 2.7. Please don't spam this project

@sigmavirus24 I know we haven't dropped Python 2, that won't happen until v3 AFAIU. Are you saying dropping EOL 3.x is blocked on dropping 2.7? I saw commits dropping [3.4](https://github.com/psf/requests/commit/68e581f1b5b4425a0ee950a2433fc3c92e671d0f) and [3.5](https://github.com/psf/requests/commit/f6c0619d892a41dcf84933810ffda89e9f6b10d4) which is why I thought it was alright to open this PR.

Also, I don't think it's quite right to characterize a contribution as spam. I came along and saw something that might need doing so I tried to help out (was going push a final commit to update docs files later today).

I understand there's a lot going on in the world right now though, so no worries.

Hi @ofek,

We are planning to drop [Python 2.7](https://github.com/psf/requests/issues/6023) and 3.6 in the next minor release. We're currently keeping `main` clean for any other hotfix issues that may come in before that release. Thanks for contribution and we'll look at trying to pull some of this in when it's time.

@nateprewitt Oh that's awesome, thank you!
Fix environment CA Bundle resolution

This is a proposal to address #6071. Rather than settling on the final value from `CURL_CA_BUNDLE` we'll end the conditional change with `verify`. This ensures we don't accidentally take a "falsey" value and disable verification for misconfigured environments.

So... we considered this a feature and not a bug. How can we control `verify=False` through environmental variables now?

Use case when call of `requests` api is buried deep in the dependency stack and you **need to force** `verify=False`
Problem with cert between python2 and python3

**My package i used**
+ requests: 2.27.0
+ openssl: 1.1.1m 2021
+ certifi: 2021.10.8

**Problem**
+ python2: can use param cert=**("cert.pem", "cert.key")** or add **br** to Accept-Encoding (python2 only has gzip and deflate), headers={"Accept-Encoding: "gzip, deflate, **br**"} to access cloudflare checking
+ python3: i used same method but seem like it is not working

i tried to fix the requests module but its still not working, if you know anything please tell me, your every word is a good suggestion :)

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Migrate test suite to support Flask>2.0

As shown in #5846 and #6069, our test suite is now thoroughly broken from Flask 2.0 onwards. We never did a deep dive to understand what caused the breakage and now changes in the underlying modules (markupsafe, werkzeug) are causing further problems. We need to sit down and work out the issue, unblocking upgrading Flask if possible.


Ping.

Resolved in #6496.
Pin markupsafe for older versions of Jinja2

Fix for build issue in #6068. The release of `markupsafe==2.1` broke older versions of Jinja2 (specifically <3.0). We're still stuck on Flask<2.0 because our test suite broke when it was released and never got fixed. This is a temporary patch to get the test suite working again and I'm going to open an issue to track resolving the Flask issue.

Upstream issue: pallets/markupsafe#283
Reword proxies section and add warning

This adds a warning around the Session proxies issues in Requests 2.x and cleans up a few erroneous sentences in the following section. Notably, rewording the environment intro to match the warning above, and removing `curl_ca_bundle` which was added in #5670. It's not particularly relevant to this section and is covered elsewhere.

## Preview Render
<img width="738" alt="Screen Shot 2022-02-18 at 6 20 06 PM" src="https://user-images.githubusercontent.com/5271761/154780254-5e7e16ac-b446-4fe8-93d4-917029f8f50e.png">
Pydantic SecretStr support

This post is about guidance on a specific feature request that I came accross when using `requests` and `pydantic`. 
I also raised the issue in `pydantic` forum:  [link to issue](https://github.com/samuelcolvin/pydantic/discussions/3808)

### Description*
I store credentials in a dictionary such as: 
```
some_headers = { 'user' : 'myself', 'password' : SecretStr('asdfghk') }
```
When passing this to `requests`, I get something of the type: 
```the_response = requests.get('http://some-url.com', headers=some_headers)
InvalidHeader: Value for header {user: **********} must be of type str or bytes, not <class 'pydantic.types.SecretStr'>```

I've read the documentation on handling the headers through Session and Request objects.  
And I did manage to solve it in my specific use case.   
Would this be somethiing that the package is interested in handling altogether?  

I would also want to try my skills on contributing, but I may need some assurence that the issue is handleable and interesting. 

Thank you for the consideration. 

 

Unless you're trying to send `*****` over the wire I don't believe this is going to work. The class also doesn't support base `str` operations like `decode`, which means this class can't be converted to `bytes` easily. I don't believe we have intention to support something like this at this time. You might consider some intermediary function to convert from this class into `bytes` for use with Requests.

I would go further and say that Requests handles native types only. We don't need to support every library's custom data types as those should be converted to a native type by that library's developers or by users of both who know what behavior they want. 

Thank you for your answers.   That makes sense.  
I'll use it in my own classes, and if I find a stronger case of use, I'll come back to the forum. 
Best. 

@Diego-MX to make sure I was clear - We will not accept any contribution that adds support for SecretStr.