Raibek/requests-github-issues huggingface dataset:
Update documentation for response.json()

The current documentation says that the function is encoding - actually, it is decoding!

I'm also wondering if it might be worth putting in some type annotations at some point

It doesn't return JSON body. It returns Python object.

Fixes #5245 

As long as this is open - I would like to add that the "if any" clause is very unclear, from the code I can't tell what is meant to say.

From what I can see the function tries to return the decoded representation *as if* the content were JSON - there are no checks if it actually is, is declared by the response to be, or even exists at all.

I think we'll keep the current docstring, thanks!
Incorrect json() docstring

What `json()` actually does - it deserializes JSON into a Python object.

So the docstring below must be wrong:
https://github.com/psf/requests/blob/fab1fd10d0b115e635b9ef1364f8444089725000/requests/models.py#L894-L918

Do you mind if I take this task as an intro issue to the project? I'm looking to be a new contributor

This has tripped me up as well. I'm new to Python. I expected response.json() to return JSON, but it doesn't. I don't know if I'm expected to do something further to turn it into actual JSON (and if so, what), or if this is an oversight in the requests code.

I also ran into issues with this - besides the fact that it indeed returns a decoded Python representation of the content, I'm confused by the "if any" clause - there is nothing that checks the existence of content, nor if the response actually has Content-Type application/json set, it simply dumps `self.text` into a json loader.

So a better way to phrase this might be along the lines of:
"Tries to return a decoded representation of the response's content as if it were JSON"

Plus one.
Why not fix it after passed so many years?
Change documentation website to requests.readthedocs.io

Fixes: #5212

Looks like there's one more reference remaining in `docs/_templates/sidebarlogo.html` on line 57:

```
$ grep -n -r "docs\.python" *
docs/_templates/sidebarlogo.html:57:    <li><a href="http://docs.python-requests.org/">English</a></li>
...
```
Python crashed while run requests.get() in a daemon process with pandas.

Run requests.get() in a daemon process with `import pandas` then python would be crashed.

* Comment `import pandas` it would be fine.
* Comment `.. os.fork() ...` it would be fine.

The problem was found on macOS only. (macOS Mojave 10.14.6)

## Expected Result

200

## Actual Result

```
objc[89863]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[89863]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
```

## Reproduction Steps

Would be crashed.

```python
import os

import pandas
import requests

def main():
    if os.fork() > 0:
        exit(0)

    resp = requests.get('https://duckduckgo.com')
    print(resp.status_code)

if __name__ == '__main__':
    main()
```

It's OK!

```python
import os

# import pandas
import requests

def main():
    if os.fork() > 0:
        exit(0)

    resp = requests.get('https://duckduckgo.com')
    print(resp.status_code)

if __name__ == '__main__':
    main()
```

It's OK!

```python
import os

import pandas
import requests

def main():
    #if os.fork() > 0:
    #    exit(0)

    resp = requests.get('https://duckduckgo.com')
    print(resp.status_code)

if __name__ == '__main__':
    main()
```

## System Information

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "18.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000213f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}
```

This is hard to say to be issue of the requests. It crashes on purpose in case of user does n't know what  he is doing. 

> objc[1936]: +[NSValue initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

If you do know what you are doing, the "crashing script"(test.py) can avoid to crash by :
`$export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ; python3 test.py`

This is definitely an issue outside of Requests. It looks like you're running into an issue with Pandas. I would suggest posting something on StackOverflow and reviewing the bug tracker for Pandas.

Sounds related to #4768
Caused by NewConnectionError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000001F06C779470>)

partial code as follows:
header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, 
                                          like Gecko) Chrome/74.0.3729.131 Safari/537.36",
                  "Content-Type": "text/html; charset=utf-8"
                } 
r = requests.get('https://nvd.nist.gov/vuln/detail/CVE-1999-0802',headers = header,verify = False)
    

When running this,I always meet such an error "HTTPSConnectionPool(host='nvd.nist.gov', port=443): Max retries exceeded with url: /vuln/detail/CVE-1999-0802 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000016033BC9828>: Failed to establish a new connection: [WinError 10060] 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败。'))
",which comes from <class 'requests.exceptions.ConnectionError'>. When i open this url in google,it is ok.But why i cant get right response when using requests in pycharm?I wish that someone could help me.Appreciate very much! 

Are you sure this is not a network issue on your end? This appears to be working for me. Have you tried URLs other than the one you have mentioned here?

Hi @Hacker1faker, this is a network issue likely with the interface you’re issuing the request on or perhaps the firewall. There isn’t anything we can do from Requests side in this case.
Fix broken URLS in readme

[This commit](https://github.com/psf/requests/commit/74b72ce4265583db0430080ab67d3d5e0c4b44b2) removed several references to the URL 2.python-requests.org but left several behind. This PR cleans those up and replaces with requests.readthedocs.io

@ssbarnea looks like you were much more thorough in replacing the old URLs...

Admins - approve #5236 instead!
Add timeout as a default attribute for Session objects

We encountered a bug today wherein we mistakenly assumed it was already supported as an attribute like all the other kwargs, https://github.com/DataDog/integrations-core/pull/4811

@Lukasa @kennethreitz @nateprewitt Hello there! After opening this I came across https://github.com/psf/requests/issues/3341

Does that decision (from way back then) still remain true today? As it is, the behavior is not very user-friendly and is entirely unexpected, hence the reason so many people keep opening the same PR 😄
Support for HTTP/3 (QUIC)

Is there currently a way to have requests to use HTTP/3 and thus QUIC as the transport protocol?  If not, has there been discussion about supporting this in future releases?

Thank you

Hi mikehwang@, we don't have a way to support HTTP/3 or QUIC directly with Requests right now. I'd suggest taking a look at either the Python-Hyper project or httpx to see if they have plans for integrating that support.
#3070 Defaults on session level

See #3070 and [this](https://stackoverflow.com/questions/41295142/is-there-a-way-to-globally-override-requests-timeout-setting). Also, compare #3341 

TODO:
- [ ] docs for the session instance attributes
- [ ] tests for this attributes' usage
- [ ] decision on default value for timeout

----------------------------

At first commits I've added session-level `timeout=None` and `allow_redirects=True` attributes that are now used in `x or self.x` in `request(...)` method. I've also removed `allow_redirects` defaults in `get(...)` and friends.

Default values are a subject for discussion, but let's start at least drafting this.

I don't believe we'll be able to accept this. `timeout` was intentionally omitted from the Session class in Requests 2.x and we don't intend to add it within this major version.
requests.utils.super_len() can't cope with tarfile members

## Expected Result

POSTing a large file that I have extracted from a tarfile.  I expect that `requests` should be able to determine the length of that file, set the `Content-Length` header, and upload it efficiently.

## Actual Result

`requests` fails to determine the file length, goes with `Transfer-Encoding: chunked`, and the upload is much slower than desired.

## Reproduction Steps

Start with a small tarfile, say

```bash
echo foo > file1
tar cf tarfile.tar file1
```

Then use `requests.utils.super_len` to try and get the length of `file1` in the tarfile:

```python
#!/usr/bin/env python
import tarfile
import requests

tar = tarfile.open("tarfile.tar")
member = tar.extractfile("file1")
print(f"super_length of member is {requests.utils.super_len(member)}")
```

result:

```
Traceback (most recent call last):
  File "./demo.py", line 7, in <module>
    print(f"super_length of member is {requests.utils.super_len(member)}")
  File "/data/developer/.virtualenvs/req-tar/lib/python3.6/site-packages/requests/utils.py", line 119, in super_len
    fileno = o.fileno()
AttributeError: '_FileInFile' object has no attribute 'fileno'
```

The behavior of the member is quite strange here: `hasattr(member, fileno)` is true, but trying to call `member.fileno()` results in an `AttributeError`.

I expect that a sensible fix might be for [this exception](https://github.com/psf/requests/blob/fab1fd10d0b115e635b9ef1364f8444089725000/requests/utils.py#L121) to catch a broader range of errors.  

One can get the length of the member by seeking, so I expect that `super_len()` should then succeed.

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.8"
  },
  "platform": {
    "release": "3.10.0-1062.1.1.el7.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.25.6"
  },
  "using_pyopenssl": false
}
```

I feel it should be raising unsupported exception when the underlying attribute doesn't have fileno. I have opened https://bugs.python.org/issue38572

Sounds reasonable, thanks - though obviously I hope that meanwhile we can get a workaround/fix here, per #5239.

Resolved with #5239
num_connections is bigger than pool_maxsize

Hi,
I use requests.Session() to send a request in many threads. When timeout error occurs, I see num_connections increased and it's bigger than pool_maxsize.

Can you tell me if it is correct behaviour?

## Expected Result

num_connections is always less than pool_maxsize.

## Actual Result

num_connections is bigger than pool_maxsize when timeout error occur.

## Reproduction Steps

```python
import requests
import logging
import json
from threading import Thread

logger = logging.getLogger(__name__)

POOL_MAXSIZE_CONST = 2


def start_new_thread(target_function):
	def decorator(*args, **kwargs):
		t = Thread(target=target_function, args=args, kwargs=kwargs)
		t.start()
	return decorator


@start_new_thread
def send_post_request_using_pool(method, url, data=None, headers=None, timeout=5):
	try:
		HTTPConnectionManager.get_instance().session.request("POST", url, data=data, headers=headers, timeout=timeout)
		return r
	except Exception as ex:
		conn = HTTPConnectionManager.get_instance().adapter.poolmanager.connection_from_url(url)
		print ('num_connections: ' + str(conn.num_connections) + '\n')


class HTTPConnectionManager:
	__instance = None
	session = None
	adapter = None

	@staticmethod
	def get_instance():
		if HTTPConnectionManager.__instance is None:
			HTTPConnectionManager()
		return HTTPConnectionManager.__instance

	def __init__(self):
		if HTTPConnectionManager.__instance is not None:
			raise Exception("This class is a singleton!")
		else:
			self.session = requests.Session()
			self.adapter = requests.adapters.HTTPAdapter(pool_maxsize=POOL_MAXSIZE_CONST, pool_block=True)
			self.session.mount('http://', self.adapter)
			self.session.mount('https://', self.adapter)
			HTTPConnectionManager.__instance = self

url = 'http://slowwly.robertomurray.co.uk/delay/2000/url/http://www.google.co.uk'
logging.basicConfig(level=logging.DEBUG)
for i in range(0,100):
	try:
		response = send_post_request_using_pool("GET", url, timeout=1)
	except Exception as ex:
		print("ex: " + str(ex))

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.13"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "100020af"
  },
  "urllib3": {
    "version": "1.25.6"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Hi @joripage,

This doesn't appear related to Requests or a defect in our software. I'd recommend seeking advice on a platform like StackOverflow. Thanks!
request timeouts connect vs read - clarification needed

documentation reads :

`requests.get('https://github.com/', timeout=0.001)`

> timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do not time out.

description seems to indicate the timeout specified is used as a "connect" timeout

however tracing the code i see : 
same timeout value is being used for `connect` and `read`
https://github.com/psf/requests/blob/master/requests/adapters.py#L435


looking for some clarity on the matter,
do i trust the documentation (am i missing something here?)
or does the documentation need an update ?

It expects a tuple in the format (connect timeout, read timeout) which you can see being used [here](https://github.com/psf/requests/blob/master/requests/adapters.py#L425)

Ln435 is what happens if only one value is passed (else).

hmm, then the documentation needs an update... 

`requests.get('https://github.com/', timeout=0.001)`
> timeout is not a time limit on the entire response download

this is clearly not the case, as the same timeout value is getting duplicated for `connect` and `read` in Ln435



Hi @adithyaonline,

This should all be covered in our [timeout docs](https://requests.readthedocs.io/en/master/user/advanced/#timeouts). Passing a single value will be used for both the connect and read timeout. Passing a tuple will allow you to modify the two independently. The section quoted in the first comment is referencing the read timeout and that we'll wait <timeout> seconds between response packets.

oh, I think I got mislead by the Note on the quickstart guide thats all 
https://2.python-requests.org/en/master/user/quickstart/#timeouts

thanks for clarifying...
Fix duplicated idna requirement

Hello everyone,

The `idna` is in the `install_requires` already, so it makes sense to remove it from extra requirements.

Also, `poetry` has been confused by this duplicated requirement.
https://github.com/sdispater/poetry/issues/1449

Best regards!

Hello @nateprewitt,
Could you please take a look at this PR?
Thank you!

Also looks to cause problems with recent setuptools/pip.

It looks like cryptography [removed its dependency of idna](https://github.com/pyca/cryptography/commit/7e422821b9f800f5345c37011c510dc9e76f552c), so we're free to remove this. @Jamim, can you rebase this onto the tip of master and we can merge. Thanks!

I've pulled this commit into https://github.com/psf/requests/pull/5342.

@sethmlarson - you've merged this commit so I suspect this PR can be closed :-)

@cjw296 Thanks for the ping, closed in favor of #5342.
Give the body of the response when raising an exception for a 400 error.

This makes debugging the reason for the error much easier, e.g. in keycloak / OpenID.

Before the patch:
400 Client Error: Bad Request for url: <url>

After the patch:
400 Client Error: Bad Request for url: <url>, response body is {"error":"unauthorized_client","error_description":"Invalid client secret"}

Hi @RubenGarcia,

We've reviewed this proposal in the past but there are a number of cases, both unintentional and intentional, where an unreasonable amount of data is sent back through a connection. In that case you may not be able to do anything useful with this information, or more severe negative implications may happen with your program. For that reason, I don't think we'll accept this at this time.

Would something more intelligent, like sending the first 128 bytes of the body, be acceptable? It would be enough for APIs, and common cases, without being overly verbose in misconfigured servers.

No.
utf-8 chars in name of file in post requests

I have some code like this:
``` python
files = {"file": ("some utf-8 chars like '№'", open("file.jpg", "rb"), "image/jpeg")}
r = requests.post("localhost", files=files)
```

and I receive broken jpg file (actually, 2x size of original)


## Expected Result

I expect to receive file.jpg

## Actual Result

But requests send to me broken file.

## System Information

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.9"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000211f"
  },
  "urllib3": {
    "version": "1.24.2"
  },
  "using_pyopenssl": true
}
```
Change raise_for_status message to hide password in URL

This PR fixes #5021 by replacing a password in the URL with asterisks.

Example:

```python
import requests

r = requests.get("http://foo:secret@httpbin.org/status/401")
r.raise_for_status()
```

Output:

```
Traceback (most recent call last):
  File "5021.py", line 4, in <module>
    r.raise_for_status()
  File "***/python-requests/requests/requests/models.py", line 968, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: UNAUTHORIZED for url: http://foo:***@httpbin.org/status/401
```

@jackyzha0 @barroca I'll have time to get back to this next weekend.

@Overv I opened a similar PR for this that might help you to write a test #5244 let me know if you need help.

I've added a unit test.

Running into this same issue for requests with `client_id` and `client_secret` parameters in the url.   Those values would get exposed despite the proposed fix here.  Seems like a more general solution is needed.
Fix connection pool managers not taking varying TLS parameters into account

This PR fixes #5140 and #4325 by returning different connection and proxy pools based on the connection parameters for the current request.

I tested it with the following code:

```python
import requests

session = requests.Session()

print(session.get('https://client.badssl.com/'))
print(session.get('https://client.badssl.com/', cert=('client.crt', 'client.key')))
print(session.get('https://client.badssl.com/', cert=('client.crt', 'client.key')))
print(session.get('https://client.badssl.com/'))
print(session.get('https://client.badssl.com/'))
```

Which correctly outputs:

```
<Response [400]>
<Response [200]>
<Response [200]>
<Response [400]>
<Response [400]>
```

Instead of the previous output:

```
<Response [400]>
<Response [400]>
<Response [400]>
<Response [400]>
<Response [400]>
```
Change requests.ConnectionError to inherit standard ConnectionError

This PR changes the requests version of `ConnectionError` to also inherit from the built-in `ConnectionError`, see #5162 

This PR doesn't solve the issue that `requests.exceptions.ConnectionError` redefines the builtin `ConnectionError`. What about renaming this to `RequestsConnectionError` or another name altogether?

Also - does this actually produce the result you're expecting? You expected:

(<class 'requests.exceptions.ConnectionError'>, <class 'requests.exceptions.RequestException'>, <class 'ConnectionError'>, <class 'OSError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>)

☝️ but the PR doesn't inherit from `ConnectionError` - it inherits from `OSError`. I suspect you'd find this PR creates a slightly different inheritance like this:

(<class 'requests.exceptions.ConnectionError'>, <class 'requests.exceptions.RequestException'>, <class 'OSError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>)

Note how this only shows the redefined builtin, not the actual builtin per the issue.

I don't believe this is an approach we can accept for #5162. We can't overwrite the standard lib ConnectionError.
REQUESTS_CA_BUNDLE overrides request.Session verify property

When `REQUESTS_CA_BUNDLE` is defined, the CA certificate passed to `requests.Session` constructor is overridden when `requests.Session.request` is called.

Guilty method/lines:
https://github.com/psf/requests/blob/master/requests/sessions.py#L694-L710

## Expected Result

I would expect for the priority to be:

1. `requests.Session.request` verify kwarg
2. `requests.Session` verify property
3. `REQUESTS_CA_BUNDLE` / `CURL_CA_BUNDLE`

## Actual Result

As `requests.Session.request` verify kwarg was not set, it took the value from `REQUESTS_CA_BUNDLE` and only then `requests.Session.verify` was checked, but ignored, resulting in an invalid CA used (for that Session).

Related issue (in docker-py project) https://github.com/docker/docker-py/issues/2433

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "18.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000213f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": true
}
```

Same issues I presume:
https://github.com/psf/requests/issues/4938
https://github.com/home-assistant/home-assistant-cli/issues/184

Had also opened a pull request, hopefully one of them gets merged in. This is a tiresome issue.
https://github.com/psf/requests/pull/5172

This looks like a duplicate of https://github.com/psf/requests/issues/3829
Part II: The Principle of Polarity

Love & Hated appear orthogonal, yet are extracted from the same substance & essences. The opposite of Love is not Hatred — but Apathy.

> Everything is dual; Everything has poles; Everything has its pair of opposites; Like and unlike are the same; Opposites are identical in nature, but different in degree; Extremes meet; All truths, are but half-truths; All paradoxes may be reconciled.

[As above, so below](https://www.kennethreitz.org/as-above-so-below).

☤

Part III is a :sparkles: :cake: :sparkles:
Tidy community files

These files can live under `.github/`, and there are already community files there.

Let's keep them together to keep the root directory a bit neater.

* https://help.github.com/en/articles/about-community-profiles-for-public-repositories

Also: 

* Fix the image for Markdown not RST
* Fix headers, compare https://github.com/not-kennethreitz/requests3/blob/9b8206fdb74e4b9649b2bf3367af7a506f9b30a6/docs/community/vulnerabilities.rst
Obtain all the content from https://free-ss.site/ by requests.

Hi,

I try to obtain the content from https://free-ss.site/ by requests.

But it seems that the above tools cannot get all of the content as the
stuff obtained by the firefox.

Is is possible to get all of the content just as the results via firefox
for this website by using requests?

Regards

Not necessarily. Reference https://github.com/psf/requests/issues/5189

@hongyi-zhao, Requests is not a browser, it's an HTTP client. It won't perform asynchronous calls or evaluate scripts in response bodies. I believe the issue you're hitting may be better solved with something like Selenium or Ghost.js.
SSL: CERTIFICATE_VERIFY_FAILED with urllib 1.25.4

Have a bunch of code using verify=False because it's a local server with no certificate.

Fails with something like:
```
requests.exceptions.SSLError: HTTPSConnectionPool(host='hostname', port=80): Max retries exceeded with url: /api (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
```

I was able to work around this by reverting back to 1.25.3 instead of 1.25.4

## Expected Result

Expected it to not check SSL Certificate.

## Actual Result

It checked SSL Certificate

## Reproduction Steps

```python
import requests

requests.get('https://nocertdomain.com/', verify=False)
# Certificate_verify_failed
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.7"
  },
  "platform": {
    "release": "4.15.0-60-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010100f"
  },
  "urllib3": {
    "version": "1.25.4"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Thanks for opening a ticket @xNinjaKittyx. The issue is being tracked in urllib3 at urllib3/urllib3#1682. We’ll leave this open for visibility until the issue is resolved.

This issue is resolved in urllib3 1.25.5.

Thanks Seth! I’ll close this out then.
Multipart post of image does not get the same result in requests as in Insomnia/Postman

I am trying to upload a .jpg file to a server using HTTP POST with the requests library in Python 3.7.

Target URL has some PHP code that handles the upload, taking 'fileToUpload' as the multipart variable.

I tried putting the request into a **with-statemen**t for opening the file, changing the **data=files to files=files** (as recommended by some example code), or setting the **headers** to multipart/form-data specifically. 

```python
import requests

url = 'http://someurl.com/upload/dir/post.php'
files = {'fileToUpload' : open('image.jpg', 'rb')}
r = requests.post(url, data=files)
```
If I run the script I raise every single error message in the post.php file, while if I take the thing to Insomnia or Postman the upload works just fine, so the server-side seems to be working.

After all it seems that the problem is with opening the file, but I tried to stick to the [documentation ](https://2.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file)and still can't figure out what is wrong here. 


Just a thought on how to debug the issue: create a preparedRequest() and then .send() that. If the same behavior arises, then just investigate what's in the prepared request and see if the binary data matches what you wish to send, etc.
This way you might be able to find a cut off point for where the code is either breaking or behaving in a manner you find unexpected.

Use `files`, not `data`:

```
r = requests.post(url, data=files)
```

vs

```
r = requests.post(url, files=files)
```

@samrussell is correct
Python 2.7 Build Still Broken - Incompatible Requirements

E.g. looking at https://travis-ci.com/psf/requests/jobs/237028717 or some of the more recent builds, we notice that there are issues in installing dependencies during the Python 2.7 build. In particular, the issue arises due to attempting the install `more-itertools==7.2.0`.
Looking at https://pypi.org/simple/more-itertools/ we find that it does exist and that the hashes do match those in the Pipfile.lock https://github.com/psf/requests/blob/146291795130cbe73e2a64f53c2a819244a4bd6b/Pipfile.lock#L435
However, inspect element on https://pypi.org/simple/more-itertools/ reveals that every version > 5.0 requires a Python version of the 3.X series (>= 3.4 in particular).

The solution is to either lower the used version of more-itertools to 5.0.0 or to drop Python 2.7 support.

Update:
We can see that this was the commit that resulted in this issue (load the second diff and CTRL+ f more-iter). So it seems it was originally kept at 5.0.0. Presumably reverting that is the desired approach?
https://github.com/psf/requests/commit/6c322457a219f9f18e3aa8ed83ad1326aa565210#diff-46bb8405eeb4d8a11deb635f562f7c48L500

Python 2.7 support is ending at the end of 2019. It might be better to just drop 2.7 support in requests too. https://pythonclock.org/

Hi @AndTheDaysGoBy, I've resolved the issue by pinning more-itertools at 5.x (a5f4f3c9), thanks for letting us know.

@abaham, Requests has a large user base with varying needs which means we'll need to maintain 2.7 support for sometime beyond EoL. Once pip is migrated to being Python 3 only (no ETA to my knowledge), we would be able to consider it, but there aren't any concrete plans at the moment.
Timeout parameter is ignored when using HTTP proxy

When I use a HTTP proxy, the `timeout` parameter is ignored.

## Expected Result

The request raising a TimeoutError after a given time.

## Actual Result

A blocking request that sometimes finishes with a 200 code, after much more than the given timeout.

## Reproduction Steps

```python
import requests
requests.get('http://example.com/', proxies={'http': '108.61.186.207:8080'}, timeout=5)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "5.0.0-29-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "1010102f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


As I read in the requests reference: ```timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds).```

So it seems that after starting the response there is no way to timeout requests.
Host header is propagated on allow_redirects

## Summary: 

We use a combination of explicit IP addresses and Host header to contact a website explicitly via IPv4 and IPv6. The Host header is propagated in every redirect, which causes problems, as the Host header and the address will not match the visited address anymore. This might cause an infinite redirect loop if a server tries to correct the host with a redirect.

## Example:
```python
requests.get('http://40.101.81.152:80', allow_redirects=True, headers={'Host': 'autodiscover.drimmelen.nl', 'random_header': "test"})
```
If you debug this request, you'll see that both the random_header and Host are sent with every request. See code below...

## Expected Result
From the support request i've seen that requests does not explicitly allows navigation between IPv6 or IPv4. This is too bad, but that is not the problem here.

In this case the problem is that the Host header is not overwritten after a redirect. We'd expect the IP(v4 or v6) address be used in the redirect, with the Host url received. If that's not possible, we'd expect the Host be overwritten with the location we're at now, completely ignoring the concept of IPv4 or IPv6 (and just going the default route of underlying network settings).

## Actual Result
After 30 redirect, the Host was still set to the value specified. This creates a mismatch between the requested address (translated from the url) and the actual url.

The following headers where observed:
...ccept: */*\r\nConnection: keep-alive\r\nHost: autodiscover.drimmelen.nl\r\nrandom_header: test\r\nCookie: ClientId=3127...

Which means the random_header was progragated correctly... and also the Host header. Propagating the Host header after a redirect is obviously wrong and can cause all kinds of misunderstanding.

## Reproduction Steps

```python
import requests
requests.get('http://40.101.81.152:80', allow_redirects=True, headers={'Host': 'autodiscover.drimmelen.nl', 'random_header': "test"})
```

The code responsible seems to be here:
https://github.com/psf/requests/blob/3e7d0a873f838e0001f7ac69b1987147128a7b5f/requests/sessions.py#L149

This code blindly copies all kwargs, except a few chosen ones such as content-length (of course). We think Host should be one of the things changed as well to prevent the address and url mismatch on redirects.

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.6.1"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.6"
  },
  "platform": {
    "release": "18.0.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010102f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "100020ff"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": true
}
```


There has been some discussion about this before in #4002.
json.dumps() fails for CookieJar deserialization after serialization

Summary.

## Expected Result

Seamless serialization/deserialization to and from dictionary consisting of name:value pairs.

## Actual Result

Serialization succeeds but the object it returns is not a serialized dictionary. If one then tries to deserialize this resulting object using `json.loads(serialized_obj)` it would return an exception.
Since, deserialization requires a proper serialized dictionary, it fails there and then.

## Reproduction Steps

```python
import requests
import json

serialized_cookie_dict = ""
response = requests.get('example') 
serialized_cookie_dict = json.dumps(response.cookies)
print(serialized_cookie_dict)     #--- Notice the resulting object is nothing like a dictionary
deserialized_cookie_dict = requests.cookies.cookiejar_from_dict(json.loads(serialized_cookie_dict))     #--- Error!

```

## System Information
This is irrelevant for the concerned bug.
Tested on Python version 3.6

## Possible Solution/Workaround

```python
import requests
import json

serialized_cookie_dict = ""
response = requests.get('example') 
serialized_cookie_dict = json.dumps(dict(response.cookies))
print(serialized_cookie_dict)     #--- Correctly serialized CookieJar
deserialized_cookie_dict = requests.cookies.cookiejar_from_dict(json.loads(serialized_cookie_dict))     #--- Successful!

```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


I cannot reproduce this. In your first example I get an error during serialization (`Object of type RequestsCookieJar is not JSON serializable`).

@Overv Forgive me for the late reply. Since I've moved on to other projects, I don't have time to look into this again.

If you do not see this being a problem then I welcome you to close this bug. In future projects if I face something like this again, I'll surely raise the issue in more detail and will try my best to follow up!

Thank you.
Requests module get method response says "JavaScript has been disabled on your browser"

Summary.
I have a url whose html response has to be assigned to a variable. In order to do that I am using the below code
resp = requests.get(url, timeout=05, verify=False)
html_output = resp.text
from the above html_output variable I am expecting the total html response of the page. However I am seeing a intermediate html output which says "**If you're seeing this message, that means <strong>JavaScript has been disabled on your browser**"
## Expected Result
The response should contain actual html source code of the url which i have given.
What you expected.
The response should contain actual html source code of the url which i have given.
## Actual Result
The below is the output of resp.text
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="manifest" href="/manifest.json"><meta name="mobile-web-app-capable" content="yes"><meta name="apple-mobile-web-app-title" content="react boilerplate"><link rel="apple-touch-icon" sizes="120x120" href="/icon-120x120.png"><link rel="apple-touch-icon" sizes="152x152" href="/icon-152x152.png"><link rel="apple-touch-icon" sizes="167x167" href="/icon-167x167.png"><link rel="apple-touch-icon" sizes="180x180" href="/icon-180x180.png"><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500"><link rel="icon" href="/favicon.ico"/><title>F1</title></head><body><noscript>If you're seeing this message, that means <strong>JavaScript has been disabled on your browser</strong>, please <strong>enable JS</strong> to make this app work.</noscript><div id="app" style="height:100%"></div><link href="https://fonts.googleapis.com/css?family=Open+Sans:400,700" rel="stylesheet"><script type="text/javascript" src="/main.7be1cefcdb84757d693b.js"></script></body></html>
What happened instead.

## Reproduction Steps

```python
import requests
url = "http://adwords-iq-staging-84824396.us-east-1.elb.amazonaws.com"
resp = requests.get(url, timeout=05, verify=False)
print resp.text
```

## System Information

    $ python -m requests.help

```
python 2.7 and OS is both Amazon linux and Mac OS
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Could be my ignorance, but to me that has nothing to do with requests. I.e. this has to with the page being able to render before responding to your GET. I.e. sort of like how you need selenium for certain tasks because it allows certain necessary browser functionalities to exist.

Hi @vempasiddhartha, 

@AndTheDaysGoBy is correct. The server is rendering a page that requires JavaScript to load content. Requests is an http client rather than a browser, so we won’t have output that requires script execution. I’d recommend looking at something like selenium for this use case.
Correct Travis 2.7 build

Signed-off-by: Gábor Lipták <gliptak@gmail.com>

failing build https://travis-ci.com/psf/requests/jobs/228220372

https://github.com/psf/requests/pull/5174

@kennethreitz please review

Hi @gliptak, this is a duplicate of #5174. We’re going to resolve this in favor of the outcome of that ticket. Thanks!
Fix encode data which has a dict value

If data contains a dict in dict value like {'a': 'b', 'c': {'d': 'e'}}, it will be encoded as a=b&c=d which is obviously wrong.

requests.post('https://httpbin.org/post', data={'a': 'b', 'c': {'d': 'e'}}).text

```
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "a": "b",
    "c": "d"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "7",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.21.0"
  },
  "json": null,
  "origin": "x.x.x.x",
  "url": "https://httpbin.org/post"
}
```

urlopen(Request('https://httpbin.org/post', data=urlencode({'a': 'b', 'c': {'d': 'e'}}).encode())).read().decode()

```
{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "a": "b",
    "c": "{'d': 'e'}"
  },
  "headers": {
    "Accept-Encoding": "identity",
    "Content-Length": "30",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "Python-urllib/3.7"
  },
  "json": null,
  "origin": "x.x.x.x",
  "url": "https://httpbin.org/post"
}
```


This PR is connected to https://github.com/psf/requests/pull/5253


Hi @sunshineplan,

This functionality already exists today in the Request-Toolbelt library where some of the more non-standard or involved changes have been kept. You can find full documentation on how to use it here: https://toolbelt.readthedocs.io/en/latest/formdata.html.
different header handling

There is a difference between the `requests.Request.prepare()` call and the `requests.Session().send()` call. I would like to explain a little:
When a `requests.Session()` is initialized, different `default_headers` are set ('Accept-Encoding', 'Accept' and 'Connection'). Such a session is also initialized by `requests.get()`. Then the `custom_headers` are merged with the `default_headers`. For the query of my URL it is important that 'Accept' and 'Connection' are set. However, calling `Request.prepare()` does not set these `default_headers` and calling `requests.Session().send()` overwrites any values already set (including the `default_headers`).

## Expected Result

`requests.Session()` and `requests.get()` have the same headers

## Actual Result

The default_headers are overwritten by `requests.Session().send()`

## Reproduction Steps

```python
import requests

resource = 'https://www.accuweather.com/en/de/hamburg/20095/biking-weather/178556'
method = 'GET'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36',
    }
verify_ssl = True
timeout = 10

# this works fine
response = requests.get(resource, headers=headers, timeout=timeout)

# that causes an requests.exceptions.ReadTimeout
request = requests.Request(method, resource, headers=headers, auth=None, data=None).prepare()
with requests.Session() as sess:
    response = sess.send(request, timeout=timeout, verify=verify_ssl)

# now this works fine again
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36',
    'Accept': '*/*',
    'Connection': 'keep-alive',
    }

request = requests.Request(method, resource, headers=headers, auth=None, data=None).prepare()
with requests.Session() as sess:
    response = sess.send(request, timeout=timeout, verify=verify_ssl)

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "5.0.0-27-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "1010102f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}

```
Re-add newline to docstring for proper rendering

#4987 had [removed the existence of a blank line before a doctest block](https://github.com/psf/requests/pull/4987/files#diff-35c843f55d3e3abf0fca8d08b73bc26cL7), which results in [the documentation for `requests.codes`](https://2.python-requests.org/en/master/api/#status-code-lookup) being improperly rendered:
![Screen Shot 2019-08-30 at 1 04 03 PM](https://user-images.githubusercontent.com/7754936/64048738-70e91f80-cb27-11e9-9d66-251ad9cce653.png)

This PR re-introduces the blank line, which appears to be? suggested by [reST spec](http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#doctest-blocks) and worked for NumPy as well [at some point](https://github.com/numpy/numpy/pull/14085).

Hi @convoliution, would you mind rebasing this onto the current master branch?

Hi @nateprewitt!

It looks like the problem addressed by this PR has effectively been resolved by someone else, because [the status codes section of the docs](https://requests.kennethreitz.org/en/master/api/#status-code-lookup) currently render nicely.

I'll go ahead and close this PR.
Documentation says there should be 4 hooks

Documentation says there should be 4 event hooks but there is only 1 in the code.

## Expected Result

4 available hooks per documentation: args, pre_request, post_request, response
https://2.python-requests.org//en/v0.10.6/user/advanced/#event-hooks

## Actual Result

<img width="720" alt="requests_hook_error" src="https://user-images.githubusercontent.com/8445610/64031455-71c58580-cb16-11e9-8108-5125ea54546d.png">

Only 1 valid hook: response

```python

"""
requests.hooks
~~~~~~~~~~~~~~

This module provides the capabilities for the Requests hooks system.

Available hooks:

``response``:
    The response generated from a Request.
"""
HOOKS = ['response']


def default_hooks():
    return {event: [] for event in HOOKS}

# TODO: response is the only one
```

## Reproduction Steps

```python
import requests

requests.get('<any_url>', hooks=dict(post_request=print_req))

def print_req(request):
    print(request)

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "18.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000213f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": true
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Hi valmack@, this is a link to the docs from 7 years ago. You’ll find current documentation by looking at version 2.22.0.