Raibek/requests-github-issues huggingface dataset:
Unexpected Prepared Request behavior

`requests.Session.send` raises random error if you change `headers` in Prepared Request and send a `POST` request with `json` parameter.

## Reproduction Steps

```python
import requests as rq
s = rq.Session()
r = rq.Request('POST', 'https://google.com', json={'foo':'bar'})
p = r.prepare()
p.headers = {}
s.send(p)
```
**Error:**
```
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/hairygeek/.local/share/virtualenvs/slink-9m8Vdza8/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/hairygeek/.local/share/virtualenvs/slink-9m8Vdza8/lib/python3.8/site-packages/requests/adapters.py", line 470, in send
    low_conn.send(hex(len(i))[2:].encode('utf-8'))
TypeError: object of type 'int' has no len()
```
However this works:
```python
r = rq.Request('POST', 'https://google.com')
p = r.prepare()
p.headers = {}
s.send(p)
<Response [405]>
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.2"
  },
  "platform": {
    "release": "5.5.10-arch1-1",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010105f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": false
}
```


Also this works too:
```python 
r = rq.Request('POST', 'https://google.com', json={'foo':'bar'})
p = r.prepare()
p.headers.update({'bar':'baz'})
s.send(p)
<Response [405]>
```

Calling `prepare` generates all the headers that Requests has to pass to urllib3 including content-length and content-type. You're altering those and it seems you're expecting magic to happen. Removing essential headers will never Just Work.

You're right. But in this case it would be better to throw a specialized exception, something about there are missing essential headers. 

Why? The presumption is that if you're using the PreparedRequest flow you know enough to not need safety rails

I don’t think we can easily special case every possible action a user can take with a PreparedRequest. The PreparedRequest flow is for users making targeted changes that still operate within the realm of reasonable requests.

This specific case will actually go away if we move to using urllib3 for chunked encoding but that still needs through testing. I’m going to close this out since we don’t have any immediate actions take here.
docs: make danger of verify=False clearer

This PR makes the danger of setting `verify=False` clearer to users of requests. In many cases, users may set this to `False` when running into problems locally with self-signed certificates, but not realize the risk they are opening themselves up to.

As an example of how the dangers of not verifying TLS (SSL) certificates is communicated, see Go's `InsecureSkipVerify` docs: https://golang.org/pkg/crypto/tls/

```
    // InsecureSkipVerify controls whether a client verifies the
    // server's certificate chain and host name.
    // If InsecureSkipVerify is true, TLS accepts any certificate
    // presented by the server and any host name in that certificate.
    // In this mode, TLS is susceptible to man-in-the-middle attacks.
    // This should be used only for testing.
```

Happy to take feedback on the wording / doc-style - this is my first PR to requests but I care a lot about 'safe defaults' in OSS!

Fixed some RST formatting / consistency.

Gentle ping - does this need anything else to be merged?
Add response.future_requests_uri, deprecate response.is_permanent_redirect

`response.is_permanent_redirect` is unreliable/useless/dangerous for chains of redirects, just give the user the correct URI instead.

example use-case: `foo` redirects temporarily to `bar`, which in turn redirects permanently to `baz`. future requests should still be made to `foo`, but using `is_permanent_redirect` would lead to them being made to `baz` instead.

That method was not introduced for users to use and was part of internal machinery that was ripped out

https://requests.readthedocs.io/en/master/api/#requests.Response.is_permanent_redirect ?

I think an alternative check for the first history item being a permanent redirect would be a more accurate way to calculate this?  The actual permanent redirect url would be gotten from the last permanent redirect among the contiguous group of initial permanent redirects.  Maybe?

@SoniEx2 I believe `autoclass` will auto-document a public method. That doesn't mean it's encouraged for you to use

Could we get it removed from the docs?  It's hard to expect developers to consider a documented non-leading-`_` attribute private...  Or implement it more sensibly (based on my minute or two of thought, maybe I'm missing something).

it does add value fwiw - if you're doing things yourself.

but don't rely on it as representative of the whole response chain.
QUES: Default port for Dockerfile

What port do I need to map to in Dockerfile so requests can make a request out?


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

Thanks for your Shitty developer pedantry —  I tried Stack and I labeled
this ques as a question.

On Thu, Apr 23, 2020 at 1:15 PM Ian Stapleton Cordasco <
notifications@github.com> wrote:

> Hi there! Thanks for opening this issue. Unfortunately, it seems this is a
> request for help instead of a report of a defect in the project. Please use
> StackOverflow <https://stackoverflow.com> for general usage questions
> instead and only report defects here.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/psf/requests/issues/5435#issuecomment-618563690>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABHVQHBQ27H7ZIVOSYY6VXLROCASPANCNFSM4MPGOFCA>
> .
>


>  I labeled this ques as a question.

The project doesn't accept questions in issues. If you'd followed the issue template then you'd have seen that and not opened this in the first place.
Requests library can hang indefinitely on windows, even if a timeout is set

Summary.

Assume you have a website that listens for and accepts connections, but never responds.

`requests.get('https://sitethatneverresponds.com', timeout=1)`

I expect this should *always* raise a timeout error.

Instead, if you put Windows into "sleep" mode during an openssl handshake, then wake it up a few minute later, socket events are silently dropped and the socket hangs forever.

## Reproduction Steps

On a laptop run do this a bunch of times:

`requests.get('https://sitethatneverresponds.com', timeout=1)`

Then close the laptop lid, and lift it up 5 minutes later (or whatever the tcp discard window is).  The get request never times out, and never returns.    If someone knows a better way of making a socket "disappear" and not time out, let me know.

It's possible to get a call stack by turning on python's tracing system and killing the process:

## Call stack of hung thread:

```
Thread 0x000070000823a000 (most recent call first):
  File "site-packages/OpenSSL/SSL.py", line 1813 in recv_into
  File "site-packages/urllib3/contrib/pyopenssl.py", line 313 in recv_into
  File "socket.py", line 586 in readinto
  File "http/client.py", line 258 in _read_status
  File "http/client.py", line 297 in begin
  File "http/client.py", line 1331 in getresponse
  File "site-packages/urllib3/connectionpool.py", line 421 in _make_request
  File "site-packages/urllib3/connectionpool.py", line 677 in urlopen
  File "site-packages/requests/adapters.py", line 449 in send
  File "site-packages/requests/sessions.py", line 643 in send
```

## Related Issue with pyopenssl:

pyopnessl, by not doing it themselves, has essentially decided that the caller must handle this by putting the socket into non-blocking mode, and calling select in a loop:

See: https://github.com/pyca/pyopenssl/issues/168 for potential solutions.


## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.8"
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.8"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010104f",
    "version": "19.1.0"
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1000211f"
  },
  "urllib3": {
    "version": "1.25.8"
  },
  "using_pyopenssl": true
```


I'm going to write a program that makes this easy to reproduce, then reopen if I can figure it out.

@earonesty I belive you're confused about what functionality `timeout` provides in `requests`. Please search open and closed issues and read those, or read the docs
Timeout exception not being raised

I am testing a backend api and expect an exception to be raised when the request takes longer than specified timeout. I am running a flask app behind nginx behind an aws elb. The app is running in a private subnet and the elb is in the public subnet. Don't know if this is an issue related to the backend architecture or a requests bug. When I go to the url in a browser flask returns the json response immediately. Interestingly, using the requests library, the response is returned at a time about double the specified timeout (see screenshots). 

## Expected Result

Raised exception

## Actual Result

Returns response after specified (2 * timeout)

## Reproduction Steps
<img width="510" alt="image" src="https://user-images.githubusercontent.com/25521899/79810594-3fc49600-8328-11ea-96d4-c067679433a4.png">
<img width="497" alt="image" src="https://user-images.githubusercontent.com/25521899/79810605-45ba7700-8328-11ea-9fb5-fbcc32a12c52.png">


```python
import time
start = time.time()
print(requests.get("https://******.com/test", timeout=2).json())
print(time.time() - start)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "19.4.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000213f"
  },
  "urllib3": {
    "version": "1.25.6"
  },
  "using_pyopenssl": false
}
Python 3.7.4
requests==2.22.0
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Hi @alexbode,

If you look at the [docs](https://requests.readthedocs.io/en/master/user/quickstart/#timeouts) or source code on timeouts, we’ve got a couple call outs on this. The timeout you’re specifying here is a Read Timeout, not a Total Timeout.

> timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do not time out.

The extra time you’re seeing is either because we’ve received some data from the server, resetting the timeout, or because the connection step is taking up the rest of the time.

@nateprewitt 

Thank you for your quick response.
Solved: Misconfigured Aws Elastic Load Balancer.
Very interesting how the miscofigured load balancer affected the requests client in that way but the issue was on my end. I deployed the elb in a public and private availability zone, should have been only in a public availability zone. Thanks for the help again.
Add more details in error reponse of "Bad Request"

**The problem**: When there is an exception raised on a 400 "Bad Request" from the server. The information provided on the exception is not enough to debug problems. 

**Resolution**: Adding "content" attribute from the response to understand the error from the server's perspective.

This doesn't scale. `response.content` isn't guaranteed to be pre-populated and in the event of a malicious server, this causes us to hang before we even raise the exception because a server could send an endless stream of bytes and requests has no mechanism to stop it.

Also, there's already a response attribute. If you want more information  while handling this, log the `error.response.content` value yourself and at your own risk

Hi @bitsapien,

Sigmavirus24 already laid out the reasoning it looks like. We already have cases of the scenario he describes happening in the wild, so for that reason, I think we'll pass on this. The workaround provided should get you to where you want.
Not able to import requests 

If I am trying to install requests it is showing requirement already satisfied but If I am trying to import then it is throwing error. Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'.
 I am running the command pip3 list then it is showing requests   2.23.0. but not able to import.  

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Received response with content-encoding: gzip, but failed to decode it.

I get an error while trying to download from https://www.huffingtonpost.gr/feeds/index.xml.

## Expected Result

To download the data.

## Actual Result

I get an exception.
('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))

## Reproduction Steps

```python
import requests
requests.get('https://www.huffingtonpost.gr/feeds/index.xml', timeout=5.0, allow_redirects=True)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.6.1"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.6"
  },
  "platform": {
    "release": "5.5.15-200.fc31.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010104f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.25.7"
  },
  "using_pyopenssl": true
}

```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


This isn't a bug in Requests. The server your talking to (or an intermediary) is mucking with the compression of the response.

Going to resolve since there's nothing we can do here. This would need to be resolved with the server in the network chain.

On the other hand, other libraries (wget, firefox), seem to download the data.

Any fix for this would be in urllib3
Cannot download files larger than 124MB

I'm using requests to download files from my remote server. All files under 124MB are downloaded and saved properly. Files larger than that are only downloaded up to 124MB and the rest of the data is skipped. 

Originally I hadn't set **stream** to **True** and I was simply saving the data using **r.content**. I thought setting **stream** to **True** and iterating over chunks with **iter_content()** would fix my problem but it persists.

 Am I doing something wrong?

Here's my code
```python
for file in files_list:
    url = "http://server:8080/path/" + file
    r = requests.get(url, stream=True)
    with open(file, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024): 
            if chunk:
                f.write(chunk)
                f.flush()
```

**Edit:** I previously stated that it downloaded up to 130MB, but it's actually 124MB. 

**Edit 2:** Solution that worked for me: https://serverfault.com/questions/470403/apache-2-4-on-windows-2008-large-file-downloads-are-corrupt

What file type were you trying to download and what type of webserver are you running? (If you don't know what type you might be able to find it in the "X-Powered-By" or "Server" response headers) I think I have a solution but still trying to debug. 

Webserver is:  **Apache/2.4.37 (Win64) PHP/7.3.1**
The file type is a Unity Engine binary file with **.resS** extension

I still haven't found a solution and I was forced to split the file in smaller pieces and combine them as they're being downloaded.

what does 

```python
r.headers["Content-Length"]
```

Say?

> what does
> 
> ```python
> r.headers["Content-Length"]
> ```
> 
> Say?

It says ~219 million bytes. Which is correct since the file is 219MB.

Stupid question but what happens if you increase the chunk size to like **8192**?

> Stupid question but what happens if you increase the chunk size to like **8192**?

Same thing. Downloads up to 124MB and stops. Last chunk was 514 bytes in size. I don't know why though.

And it downloads ok via wget / curl? can we get the url to test?

Dunno if related.. https://serverfault.com/questions/470403/apache-2-4-on-windows-2008-large-file-downloads-are-corrupt

> Dunno if related.. https://serverfault.com/questions/470403/apache-2-4-on-windows-2008-large-file-downloads-are-corrupt

That seems to be the solution! File downloads completely now. Thank you so much for your help! :heart:
word correction


This isn't a correction. `sans` means `without`.

Mea culpa... I did not know that this word exists in english. Thought that
it's a leftover from french translation.




On Thu, Apr 16, 2020 at 5:34 PM Ian Stapleton Cordasco <
notifications@github.com> wrote:

> This isn't a correction. sans means without.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/psf/requests/pull/5426#issuecomment-614691816>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AF5R6QQLWAF24ZO7OYZUYI3RM4JPTANCNFSM4MJPVOSQ>
> .
>


It's one of those French words that actually gets used in (at least American) English with relative frequency. I'm not against replacing it with "without" but it's not a faux pas in this case
WIP: Allow use of system CA certs

This allows setting `REQUESTS_CA_BUNDLE=SYSTEM` to make Requests behaves like the base python standard library with respect to CA cert trust.

Based on a comment on the original issue (https://github.com/psf/requests/issues/2966#issuecomment-613577111) this is not trustworthy (i.e., we can't guarantee that the systems will remotely do the right thing) enough of a fix to accept.

This commit doesn't change default behavior.

I don't see what your comment has to do with my point that anyone attempting to use this value would have no visibility into whether it's remotely doing the right thing because this library and it's dependencies can't guarantee that.

Agree, it doesn't guarantee anything other than that it will cause Requests to use the underlying python ssl behavior. And users have to explicitly enable it, therefor assume responsibility for use.

Right. And users can already do that without overloading the usage of this environment bundle with a transport adapter. For example, the [toolbelt](/requests/toolbelt) has a SSL Adapter which allows for modification of the defaults and makes it easy to search for usage of non-default TLS settings. A similar approach could be taken by a user to force the default certificate locations to be used instead of certifi including doing something like:

```
REQUESTS_CA_BUNDLE="$(python -c 'import ssl; print(ssl.get_default_verify_paths().cafile)')"
```
Usage of SourceAddressAdapter to create multiple source ip

Hi,
Can you tell how can i test it with multiple source ip address . Something i tried like
```python
from requests_toolbelt.adapters import source
session = PACSession(pac)
list_source_ip_and_url = [("10.129.xx.yy", "https://google.com"), ("10.189.yy.xx", "https://random.com/Surveyor.Web/")]
responses = []
try:
    for source_ip, url in list_source_ip_and_url:
        new_source = source.SourceAddressAdapter(source_ip)
        session.mount('http://', new_source)
        session.mount('https://', new_source)
        responses.append(session.get(url))
    print(responses)
except Exception as e:
    print(e)
```
I am getting error like:
HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x033D5547>: Failed to establish a new connection: [WinError 10049] The requested address is not valid in its context'))
Looks like im hitting some requests module error.
Is it because i can't assign source ip to my interface?


This isn't appropriate for this project. `requests_toolbelt` is a separate third-party project.

ok, i will open it there, thanks
How can I use sendfile?

I tried

```
r = requests.get(SOME_URL, stream=True, verify=False)

with open('/path/to/dest', 'wb') as f:
    socket.fromfd(r.raw.fileno(), socket.AF_INET, socket.SOCK_STREAM).sendfile(f)
```


but it doesn't work.


You can't use sendfile.

Hi @ImSingee,

This isn't something Requests supports and I don't think we can provide help on it. You'd need to do some more digging with the socket library.
json parameter should transform booleans

Summary.
`json=` parameter should correctly transform booleans.

## Expected Result
Given this:

```
resp = req.post('https://example.com/endPoint',
  json={ "foo": false }
)
```
I'd expect this to work, and python to correctly transform `false` to `False`.

## Actual Result
Instead, I get `"foo": false, NameError: name 'false' is not defined`

This works:

```
resp = req.post('https://example.com/endPoint',
  json={ "foo": False }
)
```

And this works:

```
payload = json.loads("{ "foo": false }")
resp = req.post('https://example.com/endPoint',
  json=payload
)
```

Therefore this is an request for enhancement that the `requests` library handle this for us, so that it "just works".

The requests library can not change the actual syntax of Python

Yes, but you can include some rudimentary transformation inside the `post` method?

I don't think you understand what is happening here, @Dynatrace-Adam-Gardner .

Try just doing:

```py
json={ "foo": False }
```

You're binding a dictionary using Python's syntax to a name - `json`. With this you could then do

```py
requests.post(url, json=json)
```

Likewise you can pass that dictionary in as you are trying to but Python's lexer and syntax are inescapable inside the `(` and `)` because `requests` literally can neither change the lexer nor the syntax of the compiled Python binary.

There is absolutely nothing a library like Requests can do in this case.

If you want you can do:

```py
false = False
```

And thus you've done your own rudimentary transformation, but there is no pre-processing that Requests can do because it's executed after Python has lexed the code and is executing it. The `NameError` you're getting comes directly from Python.

You guessed correctly. I'm very very green with Python. Thanks for the info. I'll close this now :) ;)
"Authorization" header is rewritten for HTTP Basic Auth, even when it is explicitly set in the headers kwarg

I tried setting the `Authorization` header manually to make a request to the Github API, but authentication failed despite the same token working fine when making the request using cURL, `urllib.request.Request()`, and other methods. 

## Expected Result

Successful auth and a `200 OK` response.

## Actual Result

Authentication fails with a `401 Bad Credentials` response.

Upon inspecting the response's `request` attribute, I found that the `Authorization` header had been overwritten (looks like this happens [here](https://github.com/psf/requests/blob/9cdf294/requests/auth.py#L95-L97)).

I've since worked around this, but in my opinion, one should expect that if they are setting the `Authorization` header manually (rather than using the `auth` param), the header should not be overwritten.

I can draft a PR to address this if the project maintainers agree to such a change.

## Reproduction Steps

Create an [access token](https://github.com/settings/tokens) and then try to auth to the API like so:

```python
requests.get(
    'https://api.github.com/notifications',
    headers={'Authorization': 'token MY_TOKEN_HERE'}
)
```

## System Information

```python
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.2"
  },
  "platform": {
    "release": "5.6.3-arch1-1",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.25.8"
  },
  "using_pyopenssl": false
}
```

> Upon inspecting the response's `request` attribute, I found that the `Authorization` header had been overwritten (looks like this happens [here](https://github.com/psf/requests/blob/9cdf294/requests/auth.py#L95-L97)).

I believe your diagnosis is wrong. You likely have a `.netrc` file with basic credentials in it and need to set `trust_env=False`.

You're right. I knew about the .netrc parsing feature, but I just got a new laptop and imported my dotfiles using [chezmoi](https://github.com/twpayne/chezmoi) and didn't realize I had added an old .netrc file to chezmoi. After deleting it, auth works fine.

This then becomes a duplicate of an issue about `trust_env`
Remove extraneous paramater from doctring for put() method

There's an unused `json` parameter in the docstring from the put() method.

Hi @gregpoulos, thanks for the contribution! This was added in #3666 because there was some confusion on whether verbs other than POST supported the json argument. The documentation change is intended to make it clear despite it not being an explicit arg. I think we'll probably leave this in place for now.

@nateprewitt Understood. Thanks for the quick response!
Downloading corrupted files with Python

Hello,
I have a XAMPP Server on Windows, from which I aim to download remotely a pdf file located in C:/xampp/htdocs/ using requests library from python to generate it in the very same folder but with another name (later I'll change the direction to downloads folder once it works)
The server is running in IP address 192.168.1.134 port 8081. 
This is the part of the code:
  ```
  url="http://192.168.1.134:8081/c:/xampp/htdocs/informe.pdf"
  response=requests.get(url)
  tofile=open('informelocal.pdf','wb')
  tofile.write(response.content)
  tofile.close()

```
The thing is, once I try to open the local file, I can't open it, because it's corrupted.
I think something is matter with that URL, but I don't know how to put it right.
Thanks for your time

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Add a Pull request template

To make easier for the contributors and guide them in the right way to contribute with the deploy preview check.

Hi @to09,

Thanks for the interest! You can find the full contribution guide [here](https://requests.readthedocs.io/en/master/dev/contributing/) which would be the best place to get that information.

That said, Requests is both on a semi-permanent feature freeze and under limited support. We're only able to reliably review security issues or critical bugs right now. For that reason, I think adding a template may generate more churn for all parties.
Specifying number of retries in requests.get

Is it possible to specify number of retries in requests.get?

No

Is this a feature that would be considered for a feature addition? I'd be happy to write if so.

No. The project's been under a feature freeze for years - most especially when it comes to the API

Sigmavirus24 is right, we're currently under a feature freeze for Requests. There are a handful of guides available for using urllib3's `Retry` class with a Requests `Session`, but we don't support retries with the `requests.get` api.
Add text_with_headers to Response model

This property will return the entire HTTP response including the headers. There doesn't seem to be a `test_models.py` file in the tests folder so I'm not sure if I should make a test for this, but if so where would I make it?

Hi @JonPizza,

Requests is currently under a feature freeze. This will probably be useful to a subset of users but is pretty straight forward to implement if needed. I think we'll pass on this for now. Thanks!
Remove shebang from nonexecutable script

When packaging requests and pip (which bundles it) in Fedora, we have realised
that there is a nonexecutable file with a shebang line.

It seems that the primary purpose of this file is to be imported from Python
code and hence the shebang appears to be unnecessary.

Shebangs are hard to handle when doing downstream packaging, because it makes
sense for upstream to use `#!/usr/bin/env python` while in the RPM package, we
need to avoid that and use a more specific interpreter. Since the shebang was
unused, I propose to remove it to avoid the problems.
missing implementation of "json" argument in "put" method

It seems that the implementation of "json" argument was forgot, see [1].

[1] https://github.com/psf/requests/blob/bebf5250b0df1b922ff534e987a4481cc4dde99d/requests/api.py#L128

Nope. It's accepted via `**kwargs*`.

@sigmavirus24 thanks, you are right.

It would be great to keep it in similar format as `post` method as they both accept this argument. Is there any specific reason why they are written in different ways?

Maintenance overhead is much higher if you have to keep argument lists updated in two places

Okay, I see, thanks for your explanation.
Commercial Usage Rights

We are using this module in our application with Django.

Can you please confirm if its okay to be used commercially.

Thank you in advance

Hi @hi2rashid, this project is released under the Apache-2.0 license as noted [here](https://github.com/psf/requests/blob/master/LICENSE). You can use it freely within the constraints of that licensing agreement.

Here's a quick explanation of expectations: https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)
Send Request to HAProxyProtocol Enabled Endpoint

Trying to send request to below endpoint which implements haproxy protocol (v1 and v2) using requests module. I am not seeing any specific options to pass to `requests.request()` or `requests.get`

```
curl "https://localhost/health/local?skipCacheCheck=true" -ik --haproxy-protocol**
```

**Without Flag:**

```
[root@ip-10-89-83-80 ~]# curl "https://localhost/health/local?skipCacheCheck=true" -ik ;echo
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:443 
```

**With Flag:**

```
[root@ip-10-89-83-80 ~]# curl "https://localhost/health/local?skipCacheCheck=true" -ik --haproxy-protocol;echo
HTTP/2 200 
server: envoy
date: Fri, 27 Mar 2020 14:43:18 GMT
content-type: text/plain
x-envoy-upstream-service-time: 2

Health Check Ok
```

Please suggest?

I used `pycurl` for this.

```python
import pycurl


url = "https://localhost/health/local"


# LibCurl Options as per https://github.com/curl/curl/blob/master/include/curl/curl.h
CURLOPT_HAPROXYPROTOCOL = 274
CURLOPT_SSL_VERIFYPEER = 64
CURLOPT_SSL_VERIFYHOST = 81
ENABLE = 1
DISABLE = 0


buffer = BytesIO()
curl = pycurl.Curl()
curl.setopt(curl.URL, url)
curl.setopt(curl.WRITEDATA, buffer)

curl.setopt(CURLOPT_HAPROXYPROTOCOL, ENABLE)  # Equivalent to --haproxy-protocol in curl
curl.setopt(CURLOPT_SSL_VERIFYHOST, DISABLE)  # Equivalent to -k/--insecure in curl
curl.perform()

body = buffer.getvalue()
response_text = body.decode('iso-8859-1')
status_code = curl.getinfo(curl.RESPONSE_CODE)
curl.close()

if status_code == 200 and response_text == "Health Check Ok":
    return True
return False
```
Missing Occasional *space char/url %20* in HTML?

Requesting HTML to grab some links out of <script> at the end of the HTML, but I noticed that some strings being pulled from the requests_html.HTML object were missing space characters compared to strings in my Firefox developer source file. 

## Expected Result

Expected to find strings as they are in my Firefox Developer Debugger.
They have occasional spaces that I can't systematically understand, so I just went to get them from the script as they are.

Like this one with a weird space before the parentheses:
```'11 - Im Gonna Be Ft Jhene Aiko  (Prod by keY Wane).mp3'```

!Actually, it's not even showing the extra space here when I submit my post, but I can see it in the raw text?

## Actual Result

When parsing the requests_html.HTML object for the same string, I found that the weird space wasn't there.

Like this: 
```'11 - Im Gonna Be Ft Jhene Aiko (Prod by keY Wane).mp3'```

## Reproduction Steps

```
import requests
from requests_html import HTML

player_link = f"https://embeds.datpiff.com/mixtape/390127"
player_html = HTML(html=requests.get(player_link).text)

js = player_html.find("body>script")[-1].text
print(js)

"""
requests html: '11 - Im Gonna Be Ft Jhene Aiko (Prod by keY Wane).mp3'
js in Firefox: '11 - Im Gonna Be Ft Jhene Aiko  (Prod by keY Wane).mp3'
"""
```


> 
> 
> Requesting HTML to grab some links out of <script> at the end of the HTML, but I noticed that some strings being pulled from the requests_html.HTML object were missing space characters compared to strings in my Firefox developer source file.
> ## Expected Result
> 
> Expected to find strings as they are in my Firefox Developer Debugger.
> They have occasional spaces that I can't systematically understand, so I just went to get them from the script as they are.
> 
> Like this one with a weird space before the parentheses: '11 - Im Gonna Be Ft Jhene Aiko (Prod by keY Wane).mp3'
> 
> !Actually, it's not even showing the extra space here when I submit my post, but I can see it in the raw text?
> ## Actual Result
> 
> When parsing the requests_html.HTML object for the same string, I found that the weird space wasn't there.
> 
> Like this: '11 - Im Gonna Be Ft Jhene Aiko (Prod by keY Wane).mp3'
> ## Reproduction Steps
> 
> ```
> import requests
> from requests_html import HTML
> 
> player_link = f"https://embeds.datpiff.com/mixtape/390127"
> player_html = HTML(html=requests.get(player_link).text)
> 
> js = player_html.find("body>script")[-1].text
> print(js)
> 
> """
> requests html: '11 - Im Gonna Be Ft Jhene Aiko (Prod by keY Wane).mp3'
> js in Firefox: '11 - Im Gonna Be Ft Jhene Aiko  (Prod by keY Wane).mp3'
> """
> ```

Seems like extra spaces are just compacted in HTML due to the spec, so never mind I guess
SSLError: HTTPSConnectionPool(host='free-proxy-list.net', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_record', 'wrong version number')])")))

**I am getting this error in return when running Twitterscraper on Python. 

This is the the code that was used:** 

# -*- coding: utf-8 -*-
"""
Spyder Editor

This is a temporary script file.
"""

from twitterscraper import query_tweets
import datetime as dt
import pandas as pd

begin_date = dt.date(2020, 2, 17)
end_date = dt.date(2020, 3, 20)

limit = 2000
lang = 'english'

tweets = query_tweets('coronavirus deaths', begindate = begin_date, enddate = end_date, limit = limit, lang = lang)

df = pd.DataFrame(t.__dict__ for t in tweets)

filt = df['text'].str.contains('coronavirus deaths')

df = df.loc[filt]

export_excel = df.to_excel (r'C:\Users\victo\Desktop\coronavirus.xlsx', index = None, header=True)

**How can I fix this issue?**

Hi @victorb37,

This is due to either the web server you're hitting being unable to negotiate a proper TLS connection or potentially an issue with the version of OpenSSL you have installed on your machine.

Neither of these are issues directly with Requests though which is what we have this issue tracker reserved for. Your best bet will be to create a question on Stackoverflow.com with the `python-requests` tag to debug there. Thanks!
Request returns broken encoding (how to use with BeautifulSoup)?

Hello,

I am trying to web scrap a site that contains lots of ELA questions (permission obtained). I have been trying to implement the code below:
```python
    login_url = 'https://www.helpteaching.com/registration/login.htm?'
    data = {
        'name': '<myUserName>',
        'password': '<myPassword>',
        'remember_me': 1
        }

    with requests.Session() as s:
        response = requests.post(login_url , data)
        s.encoding='utf_8'
        index_page= s.get('https://www.helpteaching.com/questions/Reading_Strategies')
        soup = BeautifulSoup(index_page.text, 'html.parser')
```

Note that the encoding used by the site is utf-8, and I even manually adjusted the encoding to 'utf-8'.
However, when execute the code above, the texts are all broken.

Is there any way that I can use Requests with BeautifulSoup?

I am using Python 3.7.0.

Thank you,

Hi @h56cho,

`Session` objects don't have an `encoding` parameter. With the web today, it will rarely work to apply a single encoding to all response. You can try to set the encoding on the `Response` object, but that shouldn't be needed in this case.

You should validate the output from Requests before passing it to BeautifulSoup. If it's still coming back broken (with and without setting the encoding), the response from the webserver isn't in UTF-8. This unfortunately is outside of the scope for this issue tracker. You'll probably have the best luck asking a question on Stackoverflow.com with the `python-requests` tag.
ssl.SSLError: [SSL: KRB5_S_TKT_NYV]

Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.
I was logging in with facebook on this python script the day before 


 it was working good  but now it throw the following error

> Traceback (most recent call last):
File "fbwv2b.py", line 553, in
login()
File "fbwv2b.py", line 116, in login
br.select_form(nr=0)
File "/data/data/com.termux/files/usr/lib/python2.7/s
ite-packages/mechanize/_mechanize.py", line 633, in sel
ect_form
global_form = self._factory.global_form
File "/data/data/com.termux/files/usr/lib/python2.7/s
ite-packages/mechanize/_html.py", line 313, in global_f
orm
self.forms()
File "/data/data/com.termux/files/usr/lib/python2.7/s
ite-packages/mechanize/_html.py", line 320, in forms
self._current_forms, self._current_global_form = se
lf._get_forms()
File "/data/data/com.termux/files/usr/lib/python2.7/s
ite-packages/mechanize/_html.py", line 335, in _get_for
ms
if self.root is None:
File "/data/data/com.termux/files/usr/lib/python2.7/s
ite-packages/mechanize/_html.py", line 289, in root
raw = self._response.read() if self._response else
b''
File "/data/data/com.termux/files/usr/lib/python2.7/s
ite-packages/mechanize/_response.py", line 199, in read
self.__cache.write(self.wrapped.read())
File "/data/data/com.termux/files/usr/lib/python2.7/s
ocket.py", line 355, in read
data = self._sock.recv(rbufsize)
File "/data/data/com.termux/files/usr/lib/python2.7/h
ttplib.py", line 607, in read
s = self.fp.read(amt)
File "/data/data/com.termux/files/usr/lib/python2.7/s
ocket.py", line 384, in read
data = self._sock.recv(left)
File "/data/data/com.termux/files/usr/lib/python2.7/s
sl.py", line 754, in recv
return self.read(buflen)
File "/data/data/com.termux/files/usr/lib/python2.7/s
sl.py", line 641, in read
v = self._sslobj.read(len)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof whil
e reading (_ssl.c:1946)

What is your openssl version? Try to update the openssl to the most recently version.

@eliasfank i was problem only in  the termux new verision in old version it worked fine 

Hi @faizanwahla,

This is an SSL issue with how you're accessing Facebook. This isn't in the scope of what we support on this issue tracker though. Please follow the instructions at the top of your issue to get assistance with this. Thanks!
time limit / lifetime of request

I have a program and in order verify that the user doesnt download such big files using input i need a time limit on how long each request is allowed to take.

Does anyone know a good way to put a time limit(/lifetime) on each python requests get requests so if it takes 10 seconds an exception will be thrown.

Thanks


Please refer to the [Stack Overflow question here as well](https://stackoverflow.com/questions/60798728/put-a-time-limit-on-a-request).


Hi @flipchan,

Requests doesn't currently support duration timeouts, only read and connect. We've had various discussions about this over the past and have mostly arrived at using [timeout-decorator](https://github.com/pnpnpn/timeout-decorator) or something similar to wrap your requests. This should be pretty simple to implement and avoids a lot of the technical complexities of introducing this functionality in Requests.

**Edit:**

Previous tickets for reference:
* #2327
* #2685
* #1928
* requests/toolbelt#51

 okay
requests.get failed with AuthenticationRequired error but urllib.request.urlopen works

Summary.
requests.get failed with AuthenticationRequired error but urllib.request.urlopen works.
The computer access google url with shadowsocks client.

```
>>> requests.__version__
'2.23.0'
>>>
>>> url = 'http://storage.googleapis.com/tfds-data'
>>>
>>> response = requests.get(url)
>>> response
<Response [401]>
>>> response.content
b"<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>"
>>>
>>> urllib.request.urlopen(url).msg
'OK'
```

## Expected Result
requests.get(url) works.

## Actual Result
```
>>> response
<Response [401]>
>>> response.content
b"<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>"
```
## Reproduction Steps
The computer access google url with shadowsocks client.

```python
import requests
import urllib.request
url = 'http://storage.googleapis.com/tfds-data'
response = requests.get(url) # It doesn't work.
response
response.content
urllib.request.urlopen(url).msg # It works.
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "19.3.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1000213f"
  },
  "urllib3": {
    "version": "1.24.3"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Hi @jiongjiongli,

I'm unable to reproduce this issue with the code provided, I get identical responses across both Requests. Something you're sending in the first request is triggering an authentication challenge. I'd verify you're not providing credentials from a .netrc file or another mechanism. The easiest way to check is seeing if there's an `Authentication` header in your initial `request` object. You can access that off your response with `response.request`.

If that doesn't resolve the issue, I'd suggest opening a new question on Stackoverflow.com to continue debugging. This is a local configuration issue rather than a defect in Requests, so I don't believe this is the best place to track. I'm going to resolve for now.

@nateprewitt OK. Thanks a lot!
Send only one Host header in chunked request

Send only one Host header when specifying a custom Host header in a chunked request.

Before this fix, the default Host header was sent along with the custom Host header.

Closes #5274 

Hi @AaronRobson :)
Can you take another look at this?

@nateprewitt What's the state of this PR?

It's ready, it just needs to be rebased to rerun tests and we can merge. It's on my list for Monday now that 2.27.0 is unblocked.

Thanks again @hodbn!
[Question/Suggest] Session requests return

As most of the people who will make requests to web sites will need to have the source, why don't you implement an optional parameter so when creating the session object we can set it to true so we have an attribute of the object that store the return of the last requests, with that we wouldn't need to create a new variable each time, that will save lot of times for coders and is easy to implement (from my point of view), I can do a PR if you think this suggestion is Good. Of course, I could modify my local version to do that but that wouldn't make any sense when sharing to others

It sounds like you're suggesting that a session store the last response on the session itself. In a single-threaded context, that's all well and good. In a multi-threaded/coroutine context this just flat out doesn't work.

Yep, sigmavirus24 is correct. This brings a lot of issues in with shared state in a parallelized environment. I don't think we'll be able to implement this due to those constraints. Thanks for the suggestion though @Rasko1!
kali linux mimtf installation error

Received response with content-encoding: gzip, but failed to decode it  error -3 while installing mimtf 


Hi @dr55555, there isn't enough information here for us to work with. It's not clear what Requests has to do with the installation of mimtf. If you can provide a full stack trace or a small reproduction sample that shows a defect in Requests that would be helpful.

For the time being I'm going to resolve the ticket since there aren't any actions we can take.
Fixed a decode problem when missing `charset`

When the `charset` is not declared in `Content-type`, the `get_encoding_from_headers` function returns `ISO-8859-1` as the response encoding by default, but the encoding is a single-byte encoding system, so if the response contains other multi-byte encoding languages like Chinese, `Response.text` will return garbage characters.

So I think it's better to leave it to `Response.apparent_encoding` to determine the encoding.

for example, if the response is like this
```python
r.headers['Content-Type'] == 'text/html'
r.content == b'<title>\xe4\xb8\xad\xe6\x96\x87</title>'
r.encoding == 'ISO-8859-1'
r.text == '<title>ä¸\xadæ\x96\x87</title>' # garbage characters
# but if we leave it to `Response.apparent_encoding`
r.encoding == 'utf-8'
r.text == '<title>中文</title>'
```
Make the `data` vs `json` parameters more clear

As https://stackoverflow.com/q/26685248/2693875 question
and answer popularity shows that there is a lot of people who
do not understand the distinction.

Ping! Can I get a review, pretty please? :) 

Can we get this merged? I just lost half an hour on this

The current wording is counter intuitive, I recommend @gdubicki's PR.