Raibek/requests-github-issues huggingface dataset:
Why does the same request return different things on burp suite and python?

When I make a post with Burpsuite the post is executed correctly. But whe I try to do the same post using python requests I get the 401 error. I am using the same User agent and the same headers for everything. There is a way for the site to recognize that the request isn’t from a browser? Do you have any ideias for what could I do? 

I look for the answer everywhere, mainly on stack overflow, but I didn’t find the answer.
Thank for the help

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please ask for help on [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Cookies d'ont work

Well i wanted to Login to my textnow account  to send smsusing cookies
i copied from firefox to curl and the request done and worked very well
when i use python
requests.get("https://www.textnow.com/api/sessions",
    headers={
        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8"
    },
    cookies={},
)
i tried use Cookies in Cookies KeyWord i tried use requests.cookies.set ,set-cookie header, cookie header , tried injecting cookies to cookie jar and tried use alot of things and everything i found on stackoverflow i tried


Every New thing i try Cookies Fail
## Reproduction Steps

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

I TRIED ALl StackOVerflow posts no solution


but none of them did help sir 

that why i did come here to demand Solution in CURL , PyCurl Works Perfectly but i prefer Requests but is don't know why if you can help thanks you d'ont send a solution that you didn't try because as i did say no solution found till now
GET Method + HTTPS does not append request Body

I'm writing a Script to interact with an Elasticsearch Instance. The ES Server blocks POST Requests, therefore i have to use GET Methods with a request body.

## Expected Result

A HTTP GET Request with Content-type and Content-length set with the corrensponding body
## Actual Result

The Headers are always been set, but if the protocol is HTTPS the body will not be appended.

## Reproduction Steps

```python
import requests, json

proxies={'http':'http://localhost:8080',
         'https':'https://localhost:8080'
        }
test={'test':'test'}
payload=json.dumps(test)

#Does not append Body (GET,HTTPS)
r=requests.request(method='get',url='https://google.com',json=payload,proxies=proxies)
#Does Append Body (GET, HTTP)
r=requests.request(method='get',url='http://google.com',json=payload,proxies=proxies)
#Does append Body (POST HTTPS)
r=requests.request(method='post',url='https://google.com',json=payload,proxies=proxies)

```

Hey, I know about Elasticsearch because I'm the Python ES client maintainer. Elasticsearch accepts POST in place of GET for all requests that require a body. The client automatically uses POST for you for this reason.

Hi Sethmlarson, thanks for the quick answer.
There is a Proxy between the client and the ES instance.
The proxy is blocking any POST Method.

A few things:

1. Semantically according to the RFCs, GET bodies are meaningless and _any intermediary_ (i.e., a proxy) can decide to strip it
2. `json=json.dumps(...)` negates the purpose of the `json=` parameter. Stop that
3. The issue sounds completely isolated to your proxies and not to requests as I know for a fact that Requests adds the body regardless.
Support for logging is abysmal, and documentation is missing

Attempting to change logging for requests is very complicated, can only be done application-wide (I think), and the documentation is missing.

Google finds many incorrect suggestions (mainly pointing to readthedocs documentation of requests version 0.x), the only workable information I found was at this stack overflow thread: https://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application

As a first step it would be good if the information from this thread (insofar as it is still correct) was included in the documentation somewhere.

But it would be a lot nicer if there was easier support for logging, possibly through a toolbelt module or something. Controlling logging on a fine-grained scale is vital, especially when you are trying to debug things like SSL-related errors (which are rather unintelligible in their own right) in a large scale application.

I understand that part of the problem is with urllib3, but I guess I have to start somewhere.

And I'm willing to help, if personpower is an issue.

I guess nobody would mind if information from https://docs.python-requests.org/en/master/api/#api-changes will get its own Debugging chapter.

Merging the actual code into helper method might not be accepted as that will lead to many pull requests with different formats that people want. For example, sometimes I want HAR dump (#16) to compare with browser, sometimes to compare to `curl` output, and sometimes colorful thing like `httpie -v`.

I think it's important to support logging. There are many code like this where I use requests:
```
res = requets.post(url, data) 
if res.status_code >= 400:
    logger.error(res.text)
res.raise_for_status()
```

I think the `raise_for_status` should get the requests logger and log the error response.

Further more, the method should add parameter like `check=True` as [subprocess](https://docs.python.org/3/library/subprocess.html) did.

@ramwin as the OP of this issue I think your problem is completely different: you want help from requests to make it easier to do logging for your own application.

This report is about logging what requests itself is doing internally, to debug internal issues inside requests (or, more likely, in my code using requests or in the server that requests is communicating with).
To work just like normal browsers, default headers in  requests.api.get(), requests.api.request(), requests.models.Request.__init__() and requests.sessions.Session.request() are set

The headers argument in `requests.api.get()`, `requests.api.request()`, `requests.models.Request.__init__()` and `requests.sessions.Session.request()` are set to a default dictionary value rather than NoneType Data Type to handle errors like 503 server error though the URL is correct. Some websites block webscrapers. To avoid being blocked, this default headers is very useful. This works in Mac OS X, Linux and Windows as of January 2020.

For example, https://www.amazon.com blocks Web-scrapers nowadays. Before this change, the returned response object's status code was always 503 though the website existed. 

![before](https://user-images.githubusercontent.com/74560907/105088248-a2b6a980-5ac1-11eb-9bcd-8b50717e7a31.png)

After this change, the returned response object's status code is always 200.

![after](https://user-images.githubusercontent.com/74560907/105088322-bcf08780-5ac1-11eb-9216-f566fa137d46.png)


Hi @shreyanavigyan,

I don't think it makes sense to add these user agents to Requests. They aren't actually accurate and the goal of Requests isn't to impersonate a browser. Users are free to add headers as they see fit, but we're not trying to violate protective restrictions services have put in place by default.

Thanks for the suggestion though.
requests library code walkthrough

Wannabe opensource contributor and intermediate python developer here. I am wondering if any contributors of this  library has done or could do a code walkthrough video ?
- It is difficult for newbies to undestand the code base from a bigger perspective, what are the patterns used, design decisions made etc merely going throught the code
- Most of the time there is a lot of context that is required by the reader, around why code is written the way it is - which cannot be understood simply reading the code
- There are a lot of tutorials on how to use the repo functionality, but I could not find any on the implementation details and the design.
- This library has  been reccomended by many as  a good case study of readable python code. It would highly benifit the community if there is a free-form walkthrough of the code base - which gives a good intro to the internal of the library - and give a bigger picture on how a successful python opensource project is structured and run.

>     * This library has  been reccomended by many as  a good case study of readable python code.

As a maintainer for several years, I've always disagreed with this notion. That said, those folks would be the people who should be doing the walk-through. I would never feel honest giving such a walk through, nor does anyone directly involved in the project have the sufficient free time to write, record, edit, and post such a walk through (or series of such)

I am also a begineer and got overwhelmed by the code so, I would also request someone to give us a full walkthrough of the entire codebase.

I also don't recommend this project as somewhere that beginners should get their start. We're unlikely to create any resources for this either as our limited time is better spent maintaining the project itself. Good luck finding a project that speaks to you and finding time to invest in OSS if you'd really like to make an impact those two components are critical.
Requests is using dependencies with copyleft licenses

We are using request's latest version but it uses certifi as depedency which is under MPL 2.0 license. This makes certifi a plugin with a copyleft license. Our company does not want to use any copyleft license plugin. I know that requests added certifi as separate plugin dependency after v 2.16 and onwards but I am not sure if its okay to use v2.15 because a lot of other bugs would also be fixed in latest versions. Is there a way we can exclude certifi depedency to avoid using a copyleft license plugin?

The MPL-licensed code included in certifi is not modified so doesn't trigger the MPL license terms. Please search for existing closed issues before opening a new one.
Old Dependencies

Just wondering why the dependencies haven't been updated for a while now.   Seems to be requiring a pretty old version of IDNA

**requests 2.25.1 requires idna<3,>=2.5**

Ran it with the latest 3.1 and still seems to work fine.  Wonder if the older version is a real requirement?


Hi @gtoph,

idna 3.0 was only released 2 weeks ago. We've restricted version caps at major version bumps to avoid pulling in breaking API changes without testing. There's currently a PR (#5711) open discussing this, so we'll close this in favor of that. Thanks.

No worries, it's just my ocd to have the latest versions ;)
Make raise_for_status() chainable

Frequently, I find myself doing the following:
```python
r = requests.get(url)
r.raise_for_status()
json = r.json()
```

I thought it may be useful to be able to do the following instead, where `raise_for_status()` is chainable:
```python
r = requests.get(url)
json = r.raise_for_status().json()
```

Unfortunately, Requests is under a feature-freeze. As a result, we won't be expanding the behaviour of `raise_for_status()`
Add json arguments to .put and .patch

This PR adds the `json` argument to both the `put` and `patch` function definitions. This is to bring them into agreement with the docstring of each function and to mirror the definition of `post`.
This should not change any functionally as the `json` arg would be captured by the `**kwargs` expansion. 

Obviously, if this difference was intentional, feel free to close without merging. Alternatively, if you prefer I can quickly put together a pr to update the docstring. It was just something that briefly tripped me up and I figured it was quick and easy to change.  

Duplicate of https://github.com/psf/requests/pull/3664

Swear I searched before submitting but must have only looked in issues not closed PRs, sorry about that.

Ironically, it was the PR from that (#3666) that introduced the issue I was trying to solve. Currently, the documentation specifically calls out JSON as a parameter, but it is not actually a parameter in the function definition (obviously it is just grabbed by **kwargs). It creates seemingly broken documentation as pictured below.

<img width="745" alt="Screen Shot 2021-01-08 at 4 37 59 PM" src="https://user-images.githubusercontent.com/117248/104066809-055d9a80-51d0-11eb-9632-bf533ecdd432.png">


However, perhaps the current state is the lesser of all evils. Happy to drop it and not take up any more time.
PUT call to unify does not work, curl call does

Hi all!

Best wishes (first things first!)

I want to enable/disable a PoE port on my UniFi switch. For this I aim using Python 3.9.1 (first time) with the following code:

```
import requests
import json
import sys
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

gateway = {"ip": "MYSERVER.COM", "port": "8443"}
headers = {"Accept": "application/json", "Content-Type": "application/json"}

login_url = f"https://{gateway['ip']}:{gateway['port']}/api/login"
login_data = {
    "username": "MYUSERNAME",
    "password": "MYPASSWORD"
}

session = requests.Session()
login_response = session.post(login_url, headers=headers, data=json.dumps(login_data), verify=False)

if (login_response.status_code == 200):
    poe_url = f"https://{gateway['ip']}:{gateway['port']}/api/s/default/rest/device/MYDEVICEID"
    json_poe_state_off = '{"port_overrides": [{"port_idx": 6, "portconf_id": "MYPROFILE2"}]}'
    post_response = session.put(poe_url, headers=headers, data=json.dumps(json_poe_state_off))   
    print('Response HTTP Request {request}'.format(request=post_response.request ))
else:
    print("Login failed")
```


The login works (I get the 2 security cookies and tried them in Paw (a macOS REST API client) to see if these were ok)) but the second call, the. PUT, returns OK but noting happens.
Before I've done this in Python, I tried all my calls in Paw first and there it works. I tried everything in bash with curl and there it works too. So I am a bit at a loss here.

Anyone has an idea?

Thank you for your time!

Best regards!
Peter


It's likely your device is being super picky about how the HTTP request looks.

Try using Wireshark to capture the exact bytes / headers that are being sent by Requests and curl and compare them. If you aren't able to discern a difference attach the output/.pcap files here and we can maybe lend an eye. Without this information it'll be tough to figure out what's causing this.

Hi Seth!
Thanks for your hint! I tried with Wireshark and saw indeed that the payload was different. The culprit was the json.dumps function which encoded the string by putting a backslash in front of each double quote.
It works now!

Thanks!
Unable to to get request

I was following a tutorial on W3School on how to do get request. The link is here: https://www.w3schools.com/python/ref_requests_get.asp

I expected the status code of 200 when I run the code 

## Actual Result

I got a error saying:
  File "api2.py", line 3, in <module>
    x = requests.get('https://w3schools.com')
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 61, in request      
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 412, in send
    conn = self.get_connection(request.url, proxies)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\adapters.py", line 315, in get_connection
    conn = self.poolmanager.connection_from_url(url)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\urllib3\poolmanager.py", line 293, in connection_from_url
    return self.connection_from_host(
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\urllib3\poolmanager.py", line 240, in connection_from_host
    return self.connection_from_context(request_context)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\urllib3\poolmanager.py", line 253, in connection_from_context
    pool_key = pool_key_constructor(request_context)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\urllib3\poolmanager.py", line 119, in _default_key_normalizer
    return key_class(**context)
TypeError: <lambda>() got an unexpected keyword argument 'key_strict'
PS C:\Users\zekgo> & C:/Users/../AppData/Local/Programs/Python/Python39/python.exe 

## Reproduction Steps

```
import requests

x = requests.get('https://w3schools.com')
print(x.status_code)
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

My Python Version is 3.9, version of requests is 2.25.1 and my operating system is Windows. Any help will be appreciated

This exception is coming from a dependency of requests. Without the information you were asked to provide, we can't provide any guidance as to what might be causing it.

Ok understood

https://github.com/urllib3/urllib3/commit/b6061f09741082bed3d8b7db10c25bf1ae683a43
Can requests work with file:/// urls?

for example my application uses requests for downloading some special data. i want that to be compatible with `file:///` type urls while also supporting `http://`. does requests have that ability builtin or not?

Also if you know a work around please help me

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Avoid zip extract racing condition by using read+write instead extract

Extract also creates the folder hierarchy, however we do not need that,
the file itself being extracted to a temporary folder is good enough.
Instead we read the content of the zip and then write it. The write is
not locked but it's OK to update the same file multiple times given the
update operation will not alter the content of the file. By not creating
the folder hierarchy (default via extract) we no longer can run into the
problem of two parallel extracts both trying to create the folder
hierarchy without exists ok flag, and one must fail.

Resolves #5223.

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>


@sigmavirus24 @nateprewitt would you be able to accept this, so we can fix a long outstanding pip issue? Thanks!

@sigmavirus24 @nateprewitt do you have any plan to have a look at this?
Documentation bug - missing cookies documentation

On the current latest documentation, there is a `cookies` link under the `quickstart` option.

![image](https://user-images.githubusercontent.com/73482956/102895815-4d08c600-445d-11eb-8ab5-fa98620efc85.png)

We can see that this section should be displayed between `Response Headers` and `Redirection History`.

![image](https://user-images.githubusercontent.com/73482956/102895893-76295680-445d-11eb-996a-edf5dd6dcebb.png)

This section does not currently exist, and the link in the sidebar list is dead - clicking it does nothing.

Is this section missing, or should this link be removed?

This section is missing!


Do we have a template for what information we would want to put in here? I would happily put up a PR if so.

The `Cookies` section under `Developer Interface` is missing as well

What domain are you accessing the documentation at? Both sections appear at:

https://requests.readthedocs.io/en/master/
https://requests.readthedocs.io/en/master/user/quickstart/#cookies
https://requests.readthedocs.io/en/master/api/#cookies

They are also present in the source files under the docs in this repository. 

Of those affected, how many are using 3rd party browser extensions? Is it possible this content is being removed client side?

Looks like the problem I originally reported is fixed now, and I can see it at the links you gave! Perhaps GitHub was serving an outdated cache at the time?

I am happy to close this, but will leave it for you guys to decide since it sounds like some other people might be having issues too.

Thanks for getting back to me.

> Looks like the problem I originally reported is fixed now

Nothing changed
BUG REPORT: 2.25.1 breaks SSL for proxy connections. Reverting back to 2.24.0 fixes issue.

Using requests 2.25.1:

		session = requests.session()
		session.proxies.update(proxies_dict)

should setup connections over our proxy server. 

Unfortunately this is what we see:

    SSLError: HTTPSConnectionPool(host='api.xyz.com', port=443): Max retries exceeded with url: /my/url (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1123)')))

The error disappears when reverting back to 2.24.0

Please help!

thanks 👍 


I guess this urllib3 bug is related: https://github.com/urllib3/urllib3/issues/2075
Since requests 2.25.1 urllib 1.16 is used, which fixed a bug regarding proxies and https connections.

The 'solution' posted there fixed the issue for me.
Don't perform authentication if no credential provided

## The old behavior:

```python
>>> requests.get('http://127.0.0.1:8080', auth=(None, None))
```

```http
GET / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: python-requests/2.25.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Basic Tm9uZTpOb25l (None:None)
```

This is weird, right!

## Now:
```python
>>> requests.get('http://127.0.0.1:8080', auth=(None, None))
```

```http
GET / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: python-requests/2.25.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
```
(No authentication)

This will be easier for applications that allow auth username and password from arguments:

```python
response = self.session.request(
                    self.httpmethod,
                    url,
                    data=self.data,
                    proxies=proxy,
                    verify=False,
                    allow_redirects=self.redirect,
                    headers=dict(self.headers),
                    timeout=self.timeout,
                    auth=(self.user, self.pass)
                )
```

If the user didn't provide credentials, which means they don't need to authenticate, then `self.user` will be `None` and `self.pass` will be `None` too. And in this situation, `requests` with my update won't perform authentication!

But in the old version (without my update), we must do this:

```python
if self.user and self.pass:
    response = self.session.request(
                        self.httpmethod,
                        url,
                        data=self.data,
                        proxies=proxy,
                        verify=False,
                        allow_redirects=self.redirect,
                        headers=dict(self.headers),
                        timeout=self.timeout,
                        auth=(self.user, self.pass)
                    )
else:
    response = self.session.request(
                        self.httpmethod,
                        url,
                        data=self.data,
                        proxies=proxy,
                        verify=False,
                        allow_redirects=self.redirect,
                        headers=dict(self.headers),
                        timeout=self.timeout
                    )
```

Also, None username or password will become an empty username/password

Requests is in a feature-freeze and this is strictly backwards incompatible

Hi, what's the problem? I just want to help!!

Totally, change the `None` object to the username `None` (string) is really crazy
Don't perform authentication if no credential provided

## The old behavior:

```python
>>> requests.get('http://127.0.0.1:8080', auth=(None, None))
```

```http
GET / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: python-requests/2.25.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Basic Tm9uZTpOb25l (None:None)
```

The same thing will happen with:

```python
>>> requests.get('http://127.0.0.1:8080', auth=('None', 'None'))
```

This is weird, right!

## Now:
```python
>>> requests.get('http://127.0.0.1:8080', auth=(None, None))
```

```http
GET / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: python-requests/2.25.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
```
(No authentication)

This will be easier if the application allows auth username and password from arguments:

```python
response = self.session.request(
                    self.httpmethod,
                    url,
                    data=self.data,
                    proxies=proxy,
                    verify=False,
                    allow_redirects=self.redirect,
                    headers=dict(self.headers),
                    timeout=self.timeout,
                    auth=(self.user, self.pass)
                )
```

If the user didn't provide credentials, which means they don't need to authenticate, then `self.user == None` and `self.pass == None`. And in this situation, `requests` with my update won't perform authentication!

In the old version (without my update), it will be like:

```python
if self.user and self.pass:
    response = self.session.request(
                        self.httpmethod,
                        url,
                        data=self.data,
                        proxies=proxy,
                        verify=False,
                        allow_redirects=self.redirect,
                        headers=dict(self.headers),
                        timeout=self.timeout,
                        auth=(self.user, self.pass)
                    )
else:
    response = self.session.request(
                        self.httpmethod,
                        url,
                        data=self.data,
                        proxies=proxy,
                        verify=False,
                        allow_redirects=self.redirect,
                        headers=dict(self.headers),
                        timeout=self.timeout
                    )
```

Also, `None` username or password will become an empty username/password
Document that the timeout can be an int

Requests simply delegates to `urllib3.util.Timeout`, and urllib3 has of course supported ints for a very long time – at least Sept 2013 per the docstring changes, but likely much longer than that. The [timeout section in the advanced docs](https://github.com/psf/requests/blob/c2b307dbefe21177af03f9feb37181a89a799fcc/docs/user/advanced.rst#timeouts) actually has an example that uses an int, but the API docs (via function docstrings) don't mention this so far.

`int`s are just converted to `float`s under the covers and really floats are the better thing to be using. It'd probably be better to convert the timeout section in our docs to using floats.

I disagree. Despite the internal conversion, from the perspective of the user code, the best data type to represent an integer is obviously an int, and it has always been supported by the code. There's no reason not to document this and instead give the appearance of requiring a float for such timeouts (adding unnecessary `.0` clutter to the code), especially since integer timeouts are far from rare and likely already very widely used.

There is a point to be made that technically, any object with a `__float__` method (or an `__index__` method since Python 3.8) is also supported, but that would be a quite obscure use case and is not documented in urllib3's documentation. Another little detail is that due to the conversion to a float, very large ints would not be treated accurately, but that's irrelevant for real-world usage.

Going to agree with @sigmavirus24, int is a subclass of float so is covered. Closing this PR.

@sethmlarson Uhm, at least in Python 3, `int` is definitely not a subclass of `float`. They both inherit directly from `object`. I'm pretty sure this was true on Python 2 as well, but I've long purged that from my systems, so can't test.

```
>>> isinstance(42, float)
False
>>> int.mro()
[<class 'int'>, <class 'object'>]
>>> float.mro()
[<class 'float'>, <class 'object'>]
```
Boundary and content disposition appears in content during PUT request python

Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.
Hi ,

I am new to requests lib.

I am generating a presigned url for s3 and use that to upload csv file using below command
files = {'file':open(<my file name'>,'rb')}
response = requests.put(<signed_url>,files=files)

The file is getting uploaded successfully. But when i download the same file using **requests.get** am getting boundary and content before my actual data as shown below

--cd5a636f460e663ab07e090ab029aac3
Content-Disposition: form-data; name="file"; filename=<filename>

Actual data

Please help me to remove the boundary string and content-disposition. I need only actual data to be uploaded


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Support Chardet version 4

It was released yesterday: https://github.com/chardet/chardet/releases and I would like to use it together with requests.

Someone will need to investigate how to test compatibility of Requests with the new `chardet` version.

PR for this is already in progress 
https://github.com/psf/requests/pull/5688

#5688 is merged. It should be available in the next release.
Upgrade to chardet 4.x

I just released [chardet 4.0.0](https://github.com/chardet/chardet/releases/tag/4.0.0) today, and it's faster and fully backward compatible with chardet 3.x (as long as you aren't mucking around in the models it uses under-the-hood directly). The next major release will be Python 3.6+, but seeing as it took me three years to put out this one, that's unlikely to be soon.

`requests/__init__.py` lines 69 to 71 need to be updated otherwise one gets a warning:

```
/usr/lib/python3.8/site-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.2) or chardet (4.0.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
```
```
assert major == 3
assert minor < 1
assert patch >= 2
```

I got this patch

```diff
diff --git a/setup.py b/setup.py
index 4ff0cfe..eca8bf5 100755
--- a/setup.py
+++ b/setup.py
@@ -42,7 +42,7 @@ if sys.argv[-1] == 'publish':
 packages = ['requests']
 
 requires = [
-    'chardet>=3.0.2,<4',
+    'chardet>=4.0.0,<5',
     'idna>=2.5,<3',
     'urllib3>=1.21.1,<1.27',
 
-- 
2.29.2

diff --git a/requests/__init__.py b/requests/__init__.py
index c00f556..11f0e97 100644
--- a/requests/__init__.py
+++ b/requests/__init__.py
@@ -66,9 +66,9 @@ def check_compatibility(urllib3_version, chardet_version):
     major, minor, patch = chardet_version.split('.')[:3]
     major, minor, patch = int(major), int(minor), int(patch)
     # chardet >= 3.0.2, < 3.1.0
-    assert major == 3
-    assert minor < 1
-    assert patch >= 2
+    assert major == 4
+    assert minor >= 0
+    assert patch >= 0
 
 
 def _check_cryptography(cryptography_version):
```

This should be goodish enough (idk if you can test `>=` and `<` in the same assert though):

```diff
diff --git a/requests/__init__.py b/requests/__init__.py
index c00f556..9bfe511 100644
--- a/requests/__init__.py
+++ b/requests/__init__.py
@@ -65,10 +65,10 @@ def check_compatibility(urllib3_version, chardet_version):
     # Check chardet for compatibility.
     major, minor, patch = chardet_version.split('.')[:3]
     major, minor, patch = int(major), int(minor), int(patch)
-    # chardet >= 3.0.2, < 3.1.0
-    assert major == 3
-    assert minor < 1
-    assert patch >= 2
+    # chardet >= 3.0.2, < 5.0.0
+    assert major >= 3 < 5
+    assert minor >= 0
+    assert patch >= 0
 
 
 def _check_cryptography(cryptography_version):
diff --git a/setup.py b/setup.py
index e714bfa..7ba4b2a 100755
--- a/setup.py
+++ b/setup.py
@@ -42,7 +42,7 @@ if sys.argv[-1] == 'publish':
 packages = ['requests']
 
 requires = [
-    'chardet>=3.0.2,<4',
+    'chardet>=3.0.2,<5',
     'idna>=2.5,<3',
     'urllib3>=1.21.1,<1.27',
     'certifi>=2017.4.17'
```

I've updated the check in `requests/__init__.py`
Uri and request from other language

Hello.
I am currently tring to request a page using .get 
my problem is that when i use specific chars become %something but i what be completed like this
"https://test:search?keyphrase=ΤεστΕδωΕχειΕλληνικα"
but the requests changes those greek chars before sending it .
Is there any way to fix this.
Thanks in advance.

Requests is doing the right thing by percent-encoding that phrase.

Browsers have this exact behavior because it's complying with standards (see RFC 3986). This is what the request looks like from Firefox:

```
https://test:search?keyphrase=%CE%A4%CE%B5%CF%83%CF%84%CE%95%CE%B4%CF%89%CE%95%CF%87%CE%B5%CE%B9%CE%95%CE%BB%CE%BB%CE%B7%CE%BD%CE%B9%CE%BA%CE%B1
```

The server should be following this standard as well, otherwise many clients (Requests, Firefox, Chrome, etc) will not work.
Broken sponsor link

Summary.

## Expected Result

I and others can sponsor your team for this awesome project

## Actual Result

I can't because the sponsor link has been broken

## Reproduction Steps

N/A

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.6"
  },
  "platform": {
    "release": "4.4.0-18362-Microsoft",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.25.0"
  },
  "system_ssl": {
    "version": "1010108f"
  },
  "urllib3": {
    "version": "1.24.3"
  },
  "using_pyopenssl": false
}
```



Hi there,

Thanks for bringing this to our attention. Unfortunately, the folks who maintain this project don't control that form. Supporting the PSF (especially during the Q4 drive right now: https://www.python.org/psf/donations/2020-q42020-drive/ ) will undoubtedly benefit this project.

Cheers,
Ian

cc @ewdurbin in the event you know what's up with the form link

Thanks for this notification @shelld3v, I've opened a PR at #5684 to update with the new link after the PSF relaunched our sponsorship program.
When I upload empty file with streaming, `Transfer-Encoding` added to request header.

When I upload empty file with streaming, `Transfer-Encoding` added to request header.

## Expected Result

What you expected.

```
def upload_streaming(file):
    with open(file, mode="rb") as f:
        r = requests.put("https://httpbin.org/put", data=f)
        print(r.request.headers)

upload_streaming("empty.txt")
# {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
```


## Actual Result

What happened instead.

```
def upload_streaming(file):
    with open(file, mode="rb") as f:
        r = requests.put("https://httpbin.org/put", data=f)
        print(r.request.headers)

upload_streaming("empty.txt")
# {'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Transfer-Encoding': 'chunked'}
```


## Reproduction Steps

```python
import requests

def upload_streaming(file):
    with open(file, mode="rb") as f:
        r = requests.put("https://httpbin.org/put", data=f)
        print(r.request.headers)

upload_streaming("empty.txt")
```

## System Information


```
$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.6"
  },
  "platform": {
    "release": "4.18.0-16-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.24.0"
  },
  "system_ssl": {
    "version": "1010100f"
  },
  "urllib3": {
    "version": "1.25.11"
  },
  "using_pyopenssl": false
}

```



Hi there,

When we cannot determine a length for a file, we default to chunked transfer-encoding which is entirely valid approach. It's what many HTTP clients do in this case where the file size is unavailable to them. Your expectation is also entirely flawed. If you're uploading a file anywhere (regardless of length) you either need `Content-Length` to be present or a chunked transfer-encoding. Given we cannot provide the former with confidence, we provide the latter.

Cheers
Response object does not handle encoding properly

GET request to an API returned a response with broken encoding.
More specifically, the response body was encoded in ASCII even though the data sent via API is encoded in UTF-8.
The result is escaped non-ASCII characters in the response body.
This behaviour can be observed on r.content, r.text, and r.json().

Following the quickstart guide, I tried to "force encoding" with `r.encoding = 'UTF-8'` but it had no effect.
I made sure to include `charset=UTF-8` in request header, which also didn't help.

What ended up working was `json.loads(r.text)`.

Very simple I know, but locating the problem was difficult,
not to mention I had to be absolutely sure the API was sending a data that was encoded in UTF-8.
This problem is not mentioned or, at least, not easily found on Stackoverflow.
I thought I would mention it here, so that others can find it.

issue #5577 seems to be similar in nature, which only proves that Encoding is an issue that seems to pop up every now and then.
The answer given for that issue seems to be "there is an easy solution so there's no need to fix it on our ends".
But nonetheless, a bug is a bug.
Besides, it would be nice to mention how to troubleshoot encoding on the quickstart guide.

Sorry it's not so much of a "summary", but things had to be said.

## Expected Result

A response body properly encoded in UTF-8.
Like so, `{'body': '상품등록'}`

## Actual Result

A response body encoded in ASCII even though it's supposed to be UTF-8.
`{'body': '\uc0c1\ud488\ub4f1\ub85'}`

## Reproduction Steps

```python
import requests

url = 'https://www.sorryicannottellyoutheactualapi.com/api'
headers = {'Content-Type': 'application/json;charset=UTF-8'}
r = requests.get(url, headers=headers)
print(r.content)
print(r.text)
print(r.json())

```

## System Information

```
chardet 3.0.4
cryptography 2.8
idna 2.9
implementation CPython 3.6.7
platform Linux 4.15.0-101-generic
pyOpenSSL 19.1.0 (openssl 1010101f)
requests 2.23.0
system_ssl 1010104f
urllib3 1.25.7
using_pyopenssl True

python 3.6.7
```

You're right, this is a duplicate of old bug reports. The fact that you found those and opened a new one is confusing. We don't need 1 bug report for everyone who has found the same bug
[DOC] Misleading Statement in Chinese Translation of Documentation

In the first line of the [Chinese translation](https://requests.readthedocs.io/zh_CN/latest/#id3), the translator used a statement as:

> "Requests 唯一的一个非转基因的 Python HTTP 库，人类可以安全享用。"


which means:

> "Requests is the only non-genetically-modified HTTP library in Python, safe for human beings to use"


This statement is scientifically misleading with the improper metaphor and apparently diverged from the English version 

> "Requests is an elegant and simple HTTP library for Python, built for human beings."


I'd be happy to open a PR to make it match the English version but failed to find the file that contains the translation - can anyone give me a pointer?

https://github.com/requests/requests-docs-cn is the repository with that translation. It's from an older version of the english docs
request defaulting to GET if url missing ending '/' on rasberry pi

Summary.

If I send a POST request with http://...., the request is sent as GET and the body is stripped.  If I add the ending '/' http://.../, the request sends as expected.

## Expected Result

requests.request("POST", **kwargs) sends a POST request regardless of URL.   Same for requests.POST(url, **kwargs)

## Actual Result

```python
url = "http://192.168.1.37:8080/api
r = requests.request(url, data={'sid':'test', 'value':911})
# Yields GET request being sent to endpoint

url = "http://192.168.1.37:8080/api/
r = requests.request(url, data={'sid':'test', 'value':911})
# Yields POST request being sent to endpoint
```

## Reproduction Steps
On Rasberry Pi 4, rasbian, using pipenv (requests installed via: pipenv install requests)

```python
import requests
url = "http://192.168.1.37:8080/api
r = requests.request(url, data={'sid':'test', 'value':911})
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "5.4.72-v7l+",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.25.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.26.2"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Sounds like the API you're hitting is returning a redirect that for compatibility reasons transforms POST to GET. Try the request with redirects=False and inspect the status code to verify this.

If you control the API I recommend using 307/308 redirect status codes.

You nailed it.  If I use allow_redirects=False in the request, the endpoint doesn't process the request at all.  I'll look into adding redirect catches.  

Thanks for helping guide my search further.  Found someone with practically the exact problem: https://rtfm.co.ua/en/http-redirects-post-and-get-requests-and-lost-data/ (python app, golang webserver).
Waiting on 503

Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.

Hello!
I'm encountering a 503 error of DDOS protection by cloudfare when using the library.
My code is as follows:
s = requests.Session()
h = s.get('https://www.pcpartpicker.com')
print(h.status)code)
>>> 503

So what I want to happen is for there to be a minimum 5 second wait from doing the get request to actually getting the page and printing the page contents.

Is this doable? I find that timeout only works by waiting for a response, but the 503 response comes immediately so that's unhelpful.

Thanks for reading!

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
[Bug/Feature?] HTTP 206 Multipart Byterange (Multiple HTTP Range response)

Not sure if this is a bug or a feature? 

Requests happily will build and parse an HTTP GET with the ```Range:``` header to request specific bytes from a server. (Assuming the server supports it, of course.)

Specifically, setting the header to say:

```python 
headers = {'Range':  'bytes 0-100, 1000-2000'} 
```

See [the Mozilla HTTP reference on HTTP Range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range) for some more examples. 

However, supplying a list of ranges in the request causes response processing to break down - the response is there, but it's not fully parsed: per [RFC 7233 Section 4.1](https://tools.ietf.org/html/rfc7233#section-4.1) the server *MUST* respond with a ```Content-Type:``` of ```multipart/byteranges``` and the associated boundary parameter that separates the ranges. In the raw response, each range is separated by the boundary parameter and then has a few small headers for each of the ranges in the response. 

## Expected Result

Parsing of the returned individual ranges into something usable - perhaps one of the following:
- A collection of some sort that contains the individual parsed headers and range data (this might be a bigger way to do it - instead of a Response object, perhaps a MultipartResponse that then parses the individual headers and allows access to the separate ranges)
- At a minimum, transparent concatenation of the response data into a contiguous stream that can be accessed/iterated as a normal response can.   

## Actual Result

The response data contains the unparsed/unseparated ranges of data instead of header parsed data.

## Reproduction Steps

```python
import requests as req

headers={'Range':  'bytes 0-100, 1000-2000'} 

response = req.get('<Some URL from a server that supports range>', headers=headers)

response.content  

```

## System Information

Tested on Python 3.8/3.9 - Linux - Requests v2.25.0, urllib3 1.26.2


This would be a feature request and the project is presently under a feature freeze. To handle Multipart responses you can use the toolbelt's multipart decoder to accomplish much of this yourself.
Offline installation in python2.7

Thank you for taking the time to review this request,I want install this package , but I use python2.7, I can't install urllib3,So what can I do to finish this installation
requires = [
    'chardet>=3.0.2,<4',
    'idna>=2.5,<3',
    'urllib3>=1.21.1,<1.26,!=1.25.0,!=1.25.1',
    'certifi>=2017.4.17'

]


I need work in python2.7, I can't change the version.(cry)

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
updated `get_encoding_from_headers` to return utf-8 if the content type is set to application/json

This PR fixes #5667 by adding a small bit of code to `get_encoding_from_headers` that returns `utf-8` if the content type is set to `application/json` and `charset` is not set. This follows [RFC 4627](https://www.ietf.org/rfc/rfc4627.txt) (ctrl+f for `shall be`) which reads: "JSON text SHALL be encoded in Unicode.  The default encoding is UTF-8."

Awesome, thanks for the fast responses here and for merging!