Raibek/requests-github-issues huggingface dataset:
[Doc] ALL_PROXY is undocumented 

## Expected Result

The doc for [Proxies](https://2.python-requests.org/en/master/user/advanced/#proxies) should include `ALL_PROXY` introduced by https://github.com/psf/requests/pull/3185 and its precedence compared to `HTTP_PROXY` and `HTTPS_PROXY`.

## Actual Result

The document [Proxies](https://2.python-requests.org/en/master/user/advanced/#proxies) lacks the description for `ALL_PROXY`, and the only appearance of `ALL_PROXY` is in the [Release History](https://2.python-requests.org/en/master/community/updates/#id37).



If anyone would like to provide a PR for this, we'd be fine reviewing it.
about redirection

i know python requests is dynamic httplib, but some problem with requests. 

'requests.exceptions.TooManyRedirects: Exceeded 30 redirects.'
i am already increase max redirect , but same response.

i am try using other ISP 'requests' is perfect and nothing problem. and cross check , problem is ISP . there injection ads from my ISP

how i can baypass injection ads  ? 

> how i can baypass injection ads ?

That's not a feature of any HTTP client and does not and will not exist in Requests. You'll need to ask StackOverflow questions like this
Requests now forcing TLS 1.2 on Python 3.6.10

Due to the changes in [this PR](https://github.com/psf/requests/pull/5443) that made it into release 2.24.0 our stack went from using TLS 1.3 through `pyopenssl` to TLS 1.2 through the new requests/`ssl` path that is forced when `ssl` has SNI. This occurs when using Python 3.6.10.

## Expected Result

On Python 3.6.10 with `pyopenssl` installed requests should use the highest TLS available, 1.3 (from `pyopenssl`), not 1.2 (from `ssl`).

## Actual Result

On Python 3.6.10 TLS 1.2 is being used from `ssl`.

## Reproduction Steps

```python
import requests
tls_actual = requests.get('https://www.howsmyssl.com/a/check', verify=False).json()['tls_version']
print(tls_actual)
```

You can run the above easily from py36/py37 to observe the difference using conda:
- `conda create -n py36-test requests=2.24.0 python=3.6.10 pyopenssl=19.10.1`
  - **Output:** TLS 1.2
- `conda create -n py37-test requests=2.24.0 python=3.7 pyopenssl=19.10.1`
  - **Output:** TLS 1.3

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.8"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.10"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010105f",
    "version": "19.1.0"
  },
  "requests": {
    "version": "2.24.0"
  },
  "system_ssl": {
    "version": "1000214f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": true
}
```
**Note:** this really isn't a blocker for us at all since we are currently upgrading from Python 3.6.10 -> 3.7 which fixes the issue since the 3.7 ssl has TLS 1.3 support. But for any users using Python 3.6.10 there may be unexpected consequences from this update.

Hey @cleebp,

Thanks for bringing this up! I don't think we'd taken this into account when pushing pyopenssl to the background. You can still opt into using pyopenssl on 3.6 with:

```
from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
```

It may be worth adding some kind of messaging around a potential TLS downgrade but I don't know if a warning is too invasive.

Resolving since we haven't seen any further feedback on this and it's easily resolvable by the end user.
WIP: Make Brotli support enabled by default, if appropriate urllib3 version is available

This is still a work in progress - I have to make sure that the tests are actually running, not skipped every time 😅 but [I would like to get feedback if the direction is right](https://github.com/psf/requests/blob/master/docs/dev/contributing.rst#get-early-feedback).

I had to update some deps to make the tests pass after a lot of packages being updated after I added `brotli` to dev-packages. But I assume that this should be done anyway. Especially as there were discrepancies in the versions - f.e. `pytest` version in `setup.py` commit message declared v. 4 compatibility while the version in `Pipenv` was set to be at most v. 3.10.1.

Speaking of dependencies I have a doubt how to declare required urllib3 version. Technically the first with brotli support is 1.25.1, but this is a version not supported by Requests... Should I update all the references to require urllib3 >= 1.25.2 then instead of >= 1.25.1?

Ping! :)

FYI Requests already supports Brotli [via urllib3](https://urllib3.readthedocs.io/en/latest/advanced-usage.html#brotli-encoding):

```python
import requests

http = requests.Session()
http.headers = {"Accept-Encoding": "br"}

resp = http.request("GET", "https://cloudflare.com")
print(resp.text)
print("Content-Encoding:", resp.headers["Content-Encoding"])
```

@sethmlarson : yes, I know. Please see the updated PR title. I hope my intent is more clear now. :) 

At https://github.com/psf/requests/pull/5783 I took a different approach: rely on urllib3 to determine whether to include `,br` in the `Accept-Encoding` header.

Closing in favor of #5783.
Requests module detected by antibot protection, but curl not

I am tring to get pixabay.com content.
Requests module detected by antibot protection, but curl command working perfectly.
(Please don't tell anything about javascript, ...)
This is the second website I seen detected with python module requests.

## Expected Result
Response status code: 200
pixabay website content

## Actual Result
Response status code: 403
cloudflare page


## Reproduction Steps

python requests code to load page
```
import requests
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en-US;q=0.5,en;q=0.3',
    'Connection': 'keep-alive',
}

response = requests.get('https://pixabay.com/photos/search/?cat=animals&min_height=2400&min_width=3000&orientation=horizontal&pagi=2', headers=headers)
print(response.text)
print(response.status_code, len(response.text))
print(response.request.headers)
```
curl command (for windows)
```
curl "https://pixabay.com/photos/search/?cat=animals&min_height=2400&min_width=3000&orientation=horizontal&pagi=2" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: en-US;q=0.5,en;q=0.3" -H "Connection: keep-alive" --output f
```

## System Information
Python version: 3.6.10 :: Anaconda, Inc.
requests==2.24.0




Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Allow [] in params

Currently, it seems to not be possible to have [] in query parameters at all using requests since it url-encodes them even though these characters are essential parts of many API's (e.g. they require filters like `?filters[key1]=val1&filters[key2]=val2`). There seems to be no other method to obtain the filters as mentioned above either.

## Expected Result

The `[` and `]` characters should not be encoded when they are present in the key of a query parameter.

## Actual Result

They are url-encoded.

## Reproduction Steps

```python
import requests

request = requests.get('http://example.com', params={'filter[key1]': 'val1'})
print(request.url)

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "4.14.181-142.260.amzn2.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": false
}
```


Hi @Luttik,

It sounds like there may be some confusion about valid query params. Query parameters are defined in [RFC 3986](https://tools.ietf.org/html/rfc3986#section-3.4) to explicitly be a `pchar`*.

If your application isn't able to properly decode URIs, you may be able to look at using PreparedRequests and manually modifying the URL. That service is likely going to have issues with other clients going forward though. If you need further help getting the above setup, StackOverflow would be the best place to get assistance.

* `pchar` = ([a-zA-Z]/[0-9]/ "-" / "." / "_" / "~"/ "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" / ":" / "@"), all other chars **MUST** be percent encoded.
Is chardetect.exe required for get request and content?

Hi - is "chardetect.exe" executable required for a get request and content?  Just wanted to know.  If not needed, then is it okay to delete it so that I can store the chardet files in bitbucket/git?

```
req = requests.get(url)
with io.BytesIO() as buf:
    buf.write(req.content)
    buf.seek(0)

```
Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.


Requests currently requires the `chardet` package but doesn't rely on any of the CLI tools.
Since this isn't a defect with the library please ask questions like this on StackOverflow in the future.
Raising readtimeout instead of connecttimeout

My timeout is set to `(1,10)` AKA connect=1, read=10 and when it times out, I can see either:

> HTTPSConnectionPool(host='cloud-collector.newrelic.com', port=443): Read timed out. (read timeout=10)

or 

> HTTPSConnectionPool(host='cloud-collector.newrelic.com', port=443): Read timed out. (read timeout=1)

<img width="1320" alt="Screenshot 2020-07-30 at 16 36 08" src="https://user-images.githubusercontent.com/1932338/88945113-7d20b400-d285-11ea-8189-ea68dfc7e9ff.png">

The second one looks like definitely a connect timeout and should mention `connect` instead of `read`. Currently it's possibly making one believe that the timeouts are not being applied correctly.

I'm not sure if this is related, but it's a read timeout issue.  I have timeout=60 but this exception is thrown <= 154ms after the post?
```
2020-09-06 09:28:12,599 Controller polyinterface      DEBUG    pgSession: Ecobee Controller:post: Sending: url=https://api.ecobee.com/token payload={}
2020-09-06 09:29:12,735 Controller polyinterface      ERROR    pgSession: Ecobee Controller:post: Connection error for https://api.ecobee.com/token: HTTPSConnectionPool(host='api.ecobee.com', port=443): Read timed out. (read timeout=60)
```


I have the same problem. 

Please do not ping random people.

> Please do not ping random people.

I'm sorry, I didn't notice the problem.

I have the same issue too. And when I narrow down connect timeout to 0.01, It raise the correct ConnectTimeout Exception.

both are related...depending on your OS, if our OS have TCP keepalive enable you will get a ReadTimeout, if not, probably will get ConnectionTimeout or ConnectionReset.... please make sure you have tcp KA enabled


I've found this issue at the urllib3 github page - they seems to be related ;)
https://github.com/urllib3/urllib3/issues/1366
Making code examples in documentation more consistent with contributi…

…on guidelines

@AaronRobson hello, I recognised that you approved these changes a while ago. I tried pulling the changes into my forked repo, but now it has left this massive PR... Should I just close this? It seems like a pain to deal with.

> @AaronRobson hello, I recognised that you approved these changes a while ago. I tried pulling the changes into my forked repo, but now it has left this massive PR... Should I just close this? It seems like a pain to deal with.

I see, it's gone and done a huge merge commit with everything that's been done to upstream since you made your change.
There's a couple of ways to deal with this:
1. Rewind back you branch all the way back to your commit and then debase on the new master branch.
2. Note your commit hash locally, delete your branch, remake your branch from latest master and cherry pick your original commit.

Then check the changes make sense with a `git log` or `git log -p` and overwrite your branch on your fork with `git push -f`.

Depending on what's changed in the meantime and the complexity of the changes it may be easier to start again though.

Resolving as this branch doesn't appear to be in a stable state and hasn't seen updates in the last 6 months.
PyPi website description is showing HTML

When viewing the package on PyPi using Edge I am seeing HTML where it should just be a text description.

**Details:**

* Browser: Edge (Chromium) version 84.0.522.40 (Official build) (64-bit)
* OS: Windows 10

## Screenshot of the problem:

![image](https://user-images.githubusercontent.com/2363526/88360396-5cf37100-cd2a-11ea-9d09-404f25d4896a.png)


Hi @LyalinDotCom, this is a dupe of #5501. Please make sure to check existing issues before posting new ones. Thanks!

Will do next time sorry 🙏

Why is this closed? The problem is still prevalent on both Firefox and Chrome.

![image](https://user-images.githubusercontent.com/47085752/89333803-cb72f000-d695-11ea-8b88-1c42b82ecb09.png)
(August 4, 2020)
Deadlock / infinite stall when using HTTPAdapter and Retry

Summary.

## Expected Result

The code to finish or error out.

## Actual Result

Execution stalls / never finishes.

## Reproduction Steps

Run the code below. Tried multiple versions without status_forcelist, without Retry altogether, just with max_retries=2. I cannot get any version to finish.

```
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) # disabled SSL warnings
 
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'}
TIMEOUT = 5

s = requests.Session()
retries = Retry(total=3, backoff_factor=1, status_forcelist=[503])
s.mount("https://", HTTPAdapter(max_retries=retries))
response = s.get('https://employdentllc.com', headers=HEADERS, timeout=TIMEOUT, verify=False)
```

## System Information

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": true
}
```

Sorry about this, just found out about the `retry-after` header and `respect_retry_after_header` option.Closing issue...
https request via http proxy lead to hangs

```
http_proxy = "http://192.168.128.3:58591"

proxyDict = {
    "http": http_proxy,
    "https": http_proxy,
}

resp = requests.get("https://example.com/", proxies=proxyDict, timeout=3)
print(resp.text)
```

When I use this proxy to request, it hangs at ssl handshake.So I change to sock proxy, it works.

And then I use node to request with this http proxy, it works too.

Of course, I use this proxy url with Chrome, it works.

So my question is that `requests` only send https request via https proxy?
Update Python Version in Shell Example in README

Python 3.8.4 is now released, but the version in README is still 3.7.4. Let's update it.

Why? This isn't necessary to illustrate the code in the console

Hi @wyz23x2, thanks for contributing to Requests! Unfortunately, we don't usually accept cosmetic changes to documentation and I'm not sure this provides new info for the reader. I think we'll pass on this for now.
I get 403 from a get request

I don't know if it's ok to post this url? 
[url](https://gcpvuclip-a.akamaihd.net/exp=1594464888~acl=%2f11125cc917d31e89f39b04e48017551f%2f*~data=cid%3d1165763700%2ciid%3dc85a94cc-1674-4eda-b955-fb54269101a6%2csid%3dakamai-890d3df2-8143-41ae-a5ca-2d6e8a8d1be4%2csiteId%3dviu%2cslotId%3dwebapp2%2cclientId%3dnone%2cn%3d1594452428397%2ch%3d63150D199AB469D014D2F6C869D5CA1D%2cip%3d103.78.114.109%2ccdnid%3dakamai%2ccdnprefix%3dgcpvuclip-a.akamaihd.net%2f~hmac=88bef1822223760250961d1ff745a9e4995af5d804ca88155146267ca3289e65/11125cc917d31e89f39b04e48017551f/vp63207_V20200707045412/hlsc_e2931_7.m3u8)
When I make a get request on the above url I get 403 from requests but if I use curl, urllib or anything else I get 200

## Expected Result
A 200 response


## Actual Result

403 response

## Reproduction Steps

```python
import requests
r = requests.get(url)
print(r)
```

## System Information

    $ python -m requests.help

```
The problem seems to be on all systems with latest requests version
```


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

> 
> 
> Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

ok. I thought it was a bug when all other libs managed to make a get request to the url I entered with a response code 200 except just requests

Nope. 403 means the server thinks you're forbidden. That means Requests did everything correct but you're missing something the server wants. Hence user error and thus it being proper for this to be on StackOverflow

Thanks for the answer.. I hope someone will find what the issue is.

There is no bug here
Avoid reading unlimited body of the redirect

https://github.com/psf/requests/blob/1b417634721ec377abb7f17bc1f215e07202c2f7/requests/sessions.py#L160:L164
Here socket is read till the end, but in case of unlimited or extremely huge body the process is stuck.
Solution I can assume - drop connection in case of extremely huge body (unlimited by default), as redirect meaning is kept inside headers, unlimited body seems to be a vulnerability.

Not reading the entire body is also a vulnerability and was a deliberate choice. There's no way to close the socket from Python without draining it or otherwise causing a significant memory leak. So on one hand, you read a lot of data off the socket, on the other your program OOMs

Hi @Sindbag, sigmavirus24 is correct that we have to read everything on the socket. Choosing an arbitrary threshold for what constitutes "unlimited" becomes difficult because many users use Requests for long lived responses. There are a handful of libraries out there that will allow you to set a hard timeout around your request. That is probably the best option for defending against this kind of issue.
is there a way to use a requests session with tasks e.g. celery?

Hey,
first, I have already created a question on stack-overflow, but didn't recieve an answer. I want to write this down here, because I know about the discusion about threat safity of requests sessions and so on. 
You can find the Question [here](https://stackoverflow.com/questions/62648309/is-there-a-way-to-use-a-requests-session-with-tasks-e-g-celery)
I try only to use some task-manager to automate some requests on a session. For more details see please the question in stack-overflow. Sorry, for reposting here, but I need a solution or work-arround. :-)


I have read your problem but can you simplify what you need about this ?

It looks like the original stack overflow question was removed for moderation violations. If you search Stack Overflow for "Requests Celery" you'll find many answers to this question. The short answer is yes there are options to use them in conjunction.
Requests and chrome browser requests behave differently when params contains '|'

when request URL: `http://192.168.12.202/cgi-bin/rtpd.cgi?echo&AdminPasswd_ss|tdb&get&HTTPAccount`,It urlencode the '|' char to '%7C' when use requests.But when I use chrome request the url,It seems the chrome doesn't encoded the '|' char, I have try the flowing way to disable url encode,but It seem not work.Does there any way disable requests encode url?
```python
def raw_url_request(method,url, **kwargs):
    req = requests.Request(method, url, **kwargs)
    prepped = req.prepare()
    prepped.url = url
    with requests.Session() as s:
           return s.send(prepped, timeout=timeout, verify=verify)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.6.1"
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.8"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010102f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.24.0"
  },
  "system_ssl": {
    "version": "1000211f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": true
}
```

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
response.text is too slow

if response.content is too long(3MB) and there's no encoding, response.text will auto calculate the encoding, it takes me 15 seconds(33 for the debug mode)
so, what if we calculate the encoding only for content[0:1024]?
(if failed, we can add content[1024:2048] into the encoding-calculater and get the newest result)

`chardet` (the package which guesses the encoding) works better with the more data you give it. If you already need to _guess_ the encoding, you need to look at as much of the data as possible to guess more accurately. Guessing off the first 1kiB wouldn't be useful for a number of cases (e.g. if the response has a lot of ambiguous preamble), so it wouldn't be a good idea for the `requests` library to do this. It also doesn't "fail" per se - it may determine the wrong encoding due to insufficient data, but that won't necessarily throw an error.

You can set `reponse.encoding` yourself if you know it ahead of time. But if you truly need to make guesses and use the `chardet` functionality, then you do need to let it operate over the whole response for best results. In this case, you may want to monkey-patch `requests` with something like [cChardet](https://pypi.org/project/cchardet/) to speed it up.

> `chardet` (the package which guesses the encoding) works better with the more data you give it. If you already need to _guess_ the encoding, you need to look at as much of the data as possible to guess more accurately. Guessing off the first 1kiB wouldn't be useful for a number of cases (e.g. if the response has a lot of ambiguous preamble), so it wouldn't be a good idea for the `requests` library to do this. It also doesn't "fail" per se - it may determine the wrong encoding due to insufficient data, but that won't necessarily throw an error.
> 
> You can set `reponse.encoding` yourself if you know it ahead of time. But if you truly need to make guesses and use the `chardet` functionality, then you do need to let it operate over the whole response for best results. In this case, you may want to monkey-patch `requests` with something like [cChardet](https://pypi.org/project/cchardet/) to speed it up.

yeah, I did like this. It's easy

You can use `chardet` incrementally (https://chardet.readthedocs.io/en/latest/usage.html#advanced-usage). Maybe feeding the document in piece by piece (and stopping as soon as chardet is confident about the encoding) would solve the problem?

Another suggestion for replacing `chardet` is https://github.com/google/compact_enc_det . It's what Google Chrome uses to detect the encoding. It's written in C++ and I'm not aware of any Python bindings for it though, so you'd have to write them yourself.

Read the HTTP headers for content lenght , use chunks retrival iterations  with AJAX or DOM... as action

Well, you can use it just if you know how the data is being encasulated...



Oh, I forget to close this! Thanks for everyone.
Support for multipart/form-data boundary

If custom boundary is set in content-type header, use it.

Relates to issues:
#1997
#3744
#621
#4740
#4589
#4554

Shouldn't need adding another library for something which could easily be achieved in the base lib

The library is in feature-freeze and this is the worst way to implement this. So many users copy headers out of their Chrome inspector without knowing what they mean and this has wildly undefined behaviour with the mix of other parameters a user might actually pass.

You're also breaking the guarantee of ordering with this PR for something that a third-party library does _better_.

@sigmavirus24 
> this has wildly undefined behaviour with the mix of other parameters a user might actually pass.

- it is utilizing a function very similar to a function which already exists in the lib: get_encoding_from_headers
- it is looking for specifically only 'boundary' param passed in the content-type header, nothing else.

Can you please help me understand why would this have wildly undefined behavior?

I don't believe we intend to accept a PR for this at this time, resolving as it's currently unactionable.
Removing extra documentation parameter json when undefined

On the functions `put` and `patch` the parameter `json` is not defined as function parameter but it is on the docstring.
This commit is to remove the extra lines from the docstring. 
The json parameter can be passed on the kwargs but never as positional parameter as it would be following the current docstring.  

Hi @tuxskar,

Thanks for the contribution! This is a reversion of #3666 where these were specifically added. They're intended to make the docstrings consistent and clarify that all of the APIs support the json parameter. I think we'll leave it in place for now.

I see, thanks for the response. 

If the idea is to have consistency with the `post` method it could be added the `json` parameter on `patch` and `put` then, to avoid having calls like:
`requests.patch('<url>', None, 'jsondata')`
raising errors but in reality are documented that could be used like this.

@tuxskar, just to clarify, `requests.patch('<url>', json=jsondata)` works as intended and that's what's documented here. If there's some confusion about how keyword args work, there's a number of more general [resources](https://realpython.com/python-kwargs-and-args/) detailing them.

@nateprewitt Thanks for the answer.

The example `requests.patch('<url>', json=jsondata)`works as intended because the key `json` is added as a key of kwargs when making the call to the function but on the docstring says that there are 3 paramaters defined using `:param` which means that json can be used passed as key-value (and will be added to kwargs) and **as a positional argument** (added to the args if it is defined, or using the positional arguments).

Because it is defined as a positional argument on the docstring it would allow to do calls like:
`requests.patch('<url>', None, 'jsondata')`
But in reality the 3rd parameter is not defined on the function and this call will raise an error.

In order to have the json parameter properly defined and not been used as other kwarg argument it must be added as a parameter of the function like:
`def patch(url, data=None, json=None, **kwargs):`
And then the calls:
`requests.patch('<url>', None, 'jsondata')`
and
`requests.patch('<url>', json='jsondata')`
Will work properly with no errors and it will be aligned with the docstring.
Remove copyrighted media from documentation

The [advanced usage](https://requests.readthedocs.io/en/master/user/advanced/) page contains a derivative of a [copyrighted album cover](https://wilderun.bandcamp.com/album/veil-of-imagination) the copyright for which is not owned by the PSF.

This must be removed.

Hi @brandondube,

I believe the work is question is actually Veil of Summer by Adrian Cox from an art gallery in San Franciso. That and the photo in question both predate the album by several years. That said, I don't think it's necessary here and opens the psf to unnecessary liability. I've merged 1b41763 to resolve the issue.

@nateprewitt more generally speaking, I think all of the images in the docs are copyright Kenneth who hasn't provided a license to the PSF or project to use those. We should probably just wipe them all and maybe commission a new logo since that was also copyright Kenneth and not licensed.

@sigmavirus24, sure I think that's totally fair. Seth sent out an email to Ernest earlier this week about options for cleaning the docs up on a wider scale. We're waiting for some input and then I think we'll be going down that route. 

For what it's worth, the contributing page also contains Tool lyrics - https://requests.readthedocs.io/en/master/dev/contributing/
Cannot jump to Japanese documentation

The link to Japanese documentation is redirected to English one.

## Expected Result
As you know, there are translation links in the sidebar of [documentation page](https://requests.readthedocs.io/en/latest/), 
but I can't jump to the Japanese documentation page.
I expect to jump to [this page](https://requests-docs-ja.readthedocs.io/en/latest/)

## Actual Result
The link to Japanese translation page is redirected to the English page.

## Reproduction Steps
Just clicking the link of documentation


i tried a few other links including https://requests.readthedocs.io/jp/latest/ and https://requests.readthedocs.io/zh_JP/latest/,
seems like it's just not ready yet.

Hi @senbeiman, thank you for reaching out about this. The doc translations have all been owned by separate maintainers since their inception. The Japanese documentation is hosted [here](https://github.com/requests/requests-docs-jp) but appears to be almost 7 years out of date. I'm not sure if we have a current alternative unfortunately.
Read timeout errors are converted to connection errors by adapter

When adding an adapter with a retry object to a session, read timeout errors are converted to connection errors. 

```python 
>>> s = requests.Session()
>>> s.get("http://whatever.com", timeout=(3.05, 0.001))
Timeout: HTTPConnectionPool(host='whatever.com', port=80): Read timed out. (read timeout=0.001)
>>> a = HTTPAdapter(max_retries=Retry(total=3))
>>> s.mount("http://", a) 
>>> s.get("http://whatever.com", timeout=(3.05, 0.001))
ConnectionError: HTTPConnectionPool(host='whatever.com', port=80): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPConnectionPool(host='whatever.com', port=80): Read timed out. (read timeout=0.001)"))
```

Is this intentional? If so, what's the reasoning behind it? I'd like to catch these errors, but handle connection errors differently (e.g. warn the user to check his internet connection) from timeout errors (e.g. inform the user that the requested resource is possibly overloaded). Wouldn't it make more sense to raise the original error type once the max number of retries is exceeded?

Whoops... just saw this is a duplicate of #5430. From what I understood the consensus is that it would indeed be better to change this behavior, but for backwards compatibility it will not be done before a new major release of the package (i.e. 3.0).

Is there already a PR for this? I'd be happy to give it a go...

Nope...if the socket is dropped because  your OS have  lower layer tcp keep alive disabled, your will get a Connection Timeout...because you are asking the server something but you are not being heard...

This work inLinux clients...there are the same for all OS


 
    **import socket
    from urllib3.connection import HTTPConnection
    HTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
        ,(socket.SOL_TCP, socket.TCP_KEEPIDLE, 45)
        ,(socket.SOL_TCP, socket.TCP_KEEPINTVL, 10)
        ,(socket.SOL_TCP, socket.TCP_KEEPCNT, 6)
         ]**

    from requests import session
    self.session = session()

this way....when a HTTP Persistence Connecion is requested and no data is being transfered , the client will notify the server that stills up....they are Persistence , so the client needs to  keep online and waiting...if not, the server will drop the connection without advise.
[Question] Is there a way to get file size of downloaded(chunk) file while downloading. 

How can I get size of file while downloading it in Python so that I can show downloaded size and pending size of file to download? if file size is 2GB and 700mb is downloaded then how will we get that in python. Currently using Python requests module to download file.

import requests
r = requests.get(download_link)
with open('setup.exe','wb') as file:
    file.write(r.content)


Hi,

You can get the total size of the file from the `Content-Length` header. 
As for the the download progress, you need to  pass a` stream=True` parameter to the request and then use `iter_content` method on the response to read the output in chunks. As you read the chunks you can keep track of the size of the data received. It's a pretty basic functionality of requests described thoroughly in documentation.

https://requests.readthedocs.io/en/master/user/advanced/#body-content-workflow


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

Thanks for the reply. Same has been pending for answer at stackOverflow.
https://stackoverflow.com/questions/62508831/how-can-i-get-size-of-file-while-downloading-it-in-python
Case sensitive in the URL host

Hello,

when making a normal requests.get I have verified that a lowercase is made to the host that is sent. Is there a way to disable this action? I am dealing with URLs where the 'host' is case sensitive (Freenet URL) and this fact causes me an error.

Thank you.

This answer fixes it, sorry: https://stackoverflow.com/questions/55607769/uppercase-url-in-requests-returns-name-does-not-resolve
How can I send HTTP/1.0 request ?

Would like to send HTTP/1.0 request using python requests library. can anyone help me here?

UseCase:
Need to test, proxy (Device under Test) upgrades the protocol version automatically.
So let's say If I send HTTP/1.0 request, it automatically upgrades it to HTTP/1.1 and send that request to server.

https://stackoverflow.com/questions/37816596/restrict-request-to-only-ask-for-http-1-0-to-prevent-chunking-error
This is the only help till now I have found, apart from that anything else I can try to send HTTP/1.0 request?

#python-requests

Hi @contactparthshah,

As the issue originally stated, please avoid posting requests for assistance on this issue tracker. We try to keep it clear for defects in Requests.

We state both in the docs and in a handful of issues on StackOverflow that Requests is an http/1.1 client.

https://stackoverflow.com/questions/31728124/how-to-define-the-http-protocol-version-in-requests

@nateprewitt, makes sense. Thank you.
connection_from_url already parses and handles lower casing scheme, it also lower cases the host, for us so there is no need to perform mutations on a url when a proxy is not in use inside of HTTPAdapter.get_connection.


cc @sigmavirus24 

Re: [psf/requests] connection_from_url already parses and handles lower
casing scheme, it also lower cases the host, for us so there is no need to
perform mutations on a url when a proxy is not in use inside of
HTTPAdapter.get_connection. (#5511)

---------- Forwarded message ---------
From: David Black ***@***.***>
Date: Thu, Jun 17, 2021, 4:50 PM
Subject: Re: [psf/requests] connection_from_url already parses and handles
lower casing scheme, it also lower cases the host, for us so there is no
need to perform mutations on a url when a proxy is not in use inside of
HTTPAdapter.get_connection. (#5511)
To: psf/requests ***@***.***>
Cc: Subscribed ***@***.***>


cc @sigmavirus24 <https://github.com/sigmavirus24>

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/psf/requests/pull/5511#issuecomment-863631506>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AUCGM533F7LMQUKSGJMKO2TTTKC47ANCNFSM4OEOHJLQ>
.


cc @sigmavirus24  

This change has reasonable potential to cause issues for users subclassing this adapter. I'm hesitant to make a change like this because it doesn't provide clear value that outweighs the risk.
incremental rest api data 

Hello, I am trying to fetch incremental data from localhost:8080/test url. This url has json rest api data and it has incremental data inside of it. I fetched data but When I fetch it, it finishes. When I run the code again or run the function every 5 seconds, it gets all data again. How can I get the new data? 

```
import requests
import json
request = requests.get("http://localhost:8080/test").text
a = json.loads(request)
print(a['items'])

```

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
requests HTTP2


requests怎么发送http2版本的情求呢？
How can requests send http2 request?

It can't

@lei375809291, [`hyper`](https://github.com/python-hyper/hyper) is a possible Python alternative. I believe they still support using the [Requests interface over the top of their H2 stack](https://hyper.readthedocs.io/en/latest/api.html#requests-transport-adapter).

thanks
2.24.0 requests and boto3 appear to be incompatible

boto3 calls broken since upgrade to requests 2.24.0. Running `pip install 'requests<2.24.0' before installing boto3 fixes the issue. I'm not sure if this is a problem with the new version of requests or if boto3 is using requests in some unexpected way that is causing this issue. Since reverting to the old version seems to workaround the issue I thought I'd start here.

## Expected Result

Calls using boto3 to succeed as they always have.

## Actual Result

Get an error:
`An HTTP Client raised and unhandled exception: 'SSLSocket' object has no attribute 'connection'`

## Reproduction Steps

Just attempt to access S3 using boto3

## System Information

Using Docker image `python:3.7-slim`

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.9.2"
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.7"
  },
  "platform": {
    "release": "5.4.0-7634-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010107f",
    "version": "19.1.0"
  },
  "requests": {
    "version": "2.24.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": true
}

```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Hi @jw-maynard, Boto3 hasn't used Requests since 2018 and at that time was using a vendored copy. It should only be relying on urllib3 since version 1.8.0.

Could you open a ticket in the [Boto3](https://github.com/boto/boto3) repository with a minimal set of steps you used to recreate the issue, including the output of `python -m pip freeze`.

Thanks!

@nateprewitt Sorry about that. Sure will do that!
Fix misformatted project description in PyPI for version 2.24.0

The project description used to display fine at https://pypi.org/project/requests/2.23.0/. It doesn't at https://pypi.org/project/requests/2.24.0/. It is a reasonable expectation that it displays correctly.

Hi @impredicative,

Thanks for the ticket. We're aware there were some issues with the long description upload this morning. Some of the PyPI infrastructure changed since our last upload and we ended up with incorrectly transformed markdown.

We're working on some larger changes to the README file itself and will upload a new one in a 2.24.1 patch. Current release time is TBD but we'll post an update here when it's done. I'll leave this open to track the issue for now.

By the way, you could remove everything related to long_description from `setup.py` and instead put this in `setup.cfg`:

```
long_description = file: README.md
long_description_content_type = text/markdown
```

The Markdown of the README could also be simplified (essentially stripping the HTML). That part seems rather straight-forward. Should I make a PR?

I did a test upload of `master` to TestPyPI and the README is now correctly rendered by PyPI:

* https://test.pypi.org/project/requests-readme-md-test/2.24.0/

There are changes coming to the README to make sure it's pure Markdown as well.

Is this going to be ever fixed as it looks really awful. IMHO, it would be much better UX to stick to pure text without any images or big advertisements, especially at the top of the page.

This was resolved in v2.25.0.
v2.24.0

2.24.0 (2020-06-17)
-------------------

**Improvements**

- pyOpenSSL TLS implementation is now only used if Python either doesn't have an `ssl` module or doesn't support SNI. Previously pyOpenSSL was unconditionally used if available. This applies even if pyOpenSSL is installed via the `requests[security]` extra (#5443)

- Redirect resolution should now only occur when `allow_redirects` is True. (#5492)

- No longer perform unnecessary Content-Length calculation for requests that won't use it. (#5496)

For those of you following along at home, we're having some issues with the existing README file and new linting rules PyPI is enforcing. The release may take a bit longer than expected today.
Allow TLS version used, to be discovered

It's not possible to find which version of TLS was used, or indeed any of the SSL parameters, after a transfer. 
Could some of this metadata be made available, perhaps only if specified in the request.

Is there any indication that TLS was used at all ?

In the future, please search **closed and** open issues before creating new ones that are duplicates. This has been discussed ad nauseum and is blocked on urllib3 supporting it for which there's an open issue discussing the idea and implementation.

Apologies, thought I had searched in open issues, and could not find it in the manual, hence thought it was not a duplicate.

eg Curl does allow discovery of the SSL/TLS phases in a connection.

Resolving as duplicate of previous tickets. We'll need work in urllib3 before we'd be able to consider exposing this in Requests.
Avoid unnecessary data collection conversion by making use of urllib3's HTTPHeaderDict for request headers

Avoid unnecessary data collection conversion by making use of urllib3's [`HTTPHeaderDict`](https://github.com/urllib3/urllib3/blob/master/src/urllib3/_collections.py#L106) for request headers & response headers. Currently request http headers are placed in a [`CaseInsensitiveDict`](https://github.com/psf/requests/blob/master/requests/structures.py#L15) but before a request is made the headers are converted to urllib3's [`HTTPHeaderDict`](https://github.com/urllib3/urllib3/blob/master/src/urllib3/_collections.py#L106) & similar seems to occur for responses (in reverse). 

Related: 
* https://github.com/psf/requests/issues/4214

I'm 99% certain it's a backwards incompatible change at the moment. Also the 3.0 PR is not active and shouldn't be considered something under active development.

It might be possible to have a kind of container structure that wraps/resolves differences in behaviour.

It probably is, but will it be faster if we have to layer tonnes of extra logic on top of things to support backwards compatibility? Also our current CaseInsensitiveDict's interactions with the outside world probably aren't well tested enough to have any confidence in that migration

HTTP Handshake requires some information exchange, made by headers....
Call url.lower() once inside get_adapter


Hi @dbaxa,

edit: Redacting my first statement, I read over the PR too quickly. I think this is a semi-reasonable optimization, but it's probably not entirely necessary given the number of adapters typically loaded. Given this isn't a high touch point, I'm not sure it's necessary right now. Thanks for the suggestion though!

@nateprewitt can we apply this PR then ...?

Re: [psf/requests] Call url.lower() once inside get_adapter (#5497)



On Thu, Jun 17, 2021, 4:45 PM David Black ***@***.***> wrote:

> @nateprewitt <https://github.com/nateprewitt> can we apply this PR then
> ...?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/psf/requests/pull/5497#issuecomment-863629982>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AUCGM52V3IE5MAJWTE7LQNTTTKCKHANCNFSM4N7QHJTQ>
> .
>
Instacrack termux bruteforce 

Traceback (most recent call last):
  File "crack.py", line 108, in <module>                  sess = instabrute.Login(password)
  File "crack.py", line 83, in Login                      sess.headers.update({'X-CSRFToken' : r.cookies.get_dict()['csrftoken']})
KeyError: 'csrftoken'

Any solution to fix this error

Don't access a key that isn't in the dictionary? There's no bug in this library based off of what very little information you've provided

> Don't access a key that isn't in the dictionary? There's no bug in this library based off of what very little information you've provided

Any code for this error

Cupp.py


Hi @gehanshanaka, as Sigmavirus24 pointed out, the cookie your code is looking for doesn't exist in this case. This isn't a defect in Requests from what you've provided. You'll need to update the referenced code to be more fault tolerant.
Remove what appears to be an un-needed call to self.resolve_redirects when not following redirects as part of creating a request's history.


At one point, `resolve_redirects` used to bail early if we weren't following redirects but I guess that changed. Now if there's a redirect to follow, we make the request but we don't use that response which is pretty bad.

Good catch. @nateprewitt @sethmlarson this PR looks good

@sigmavirus24 let me know if you want me to fix up the source branch name and or  commit message :-) .

I'm not a maintainer so not my call. 🙂