Raibek/requests-github-issues huggingface dataset:
Pass Multiple Auth Handlers

Summary.

I would like to pass multiple authentication handlers to the auth variable on requests. 

In the urllib.request package when you build an opener, you can chain hanlders.  

Example:

```
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password.
# If we knew the realm, we could use it instead of None.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)

handler1 = urllib.request.HTTPBasicAuthHandler(password_mgr)
handler2 = urllib.request.HTTPDigestAuthHandler(password_mgr)
# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener([handler1, handler2])

# use the opener to fetch a URL
opener.open(a_url)

# Install the opener.
# Now all calls to urllib.request.urlopen use our opener.
urllib.request.install_opener(opener)

```

It takes each handler in order and tried one from the list to see what will take.  This will help for sites that use multiple forms of authentication or with site where you don't know the auth type.

Is there a way to do this with requests?  If I pass in a list of authhandlers to the auth variable, I see the ```__str__``` representation of the class being given in the auth part of the request.

Thank you


There is no way with the core Requests package, but your use-case can be solved with [the Requests toolbelt](http://toolbelt.readthedocs.io/en/latest/authentication.html#guessauth) package's `GuessAuth` class. For more general use, [`AuthHandler`](http://toolbelt.readthedocs.io/en/latest/authentication.html#authhandler) might suit your needs.

@Lukasa Thanks.  Do you think this could be an added feature for the next release?

I think your solution will work.  The ```GuessAuth``` class seems to extend the ```BaseAuth``` class and does the ```40x``` checks.



Nope. =) Generally speaking we only add features when they're needed by an enormous number of people and when it's more convenient for us to do it than for a third-party. In this case the third-party package works great, so there's no real reason to bring it into Requests.
Pep8ify

PEP8 forbids inventing __double_leading_and_trailing_underscore__
name, so don't do it.

The first rule of PEP8 is that a foolish consistency is the hobgoblin of little minds. In this case, the double-underscore is safe and reasonable. :smile:

@Lukasa Why is it safe since when we don't know when Python core add such thing?

It also makes someone new to Python think that `__attrs__` is a magic method when it doesn't. `requests` code base is great and recommended to read in many sources, IMHO, it's a poor choice for chosing such name.

Because there's no need for us to be predictive. Requests is one of the most widely-used Python packages: we'll know when such a thing is added, if for no other reason than because most of the Python core team uses Requests and will spot the problem if and when it's introduced. So for now we're in a safe place.

e.g. people probably rely on this — we don't want to change anything, especially for pedantic reasons, unless we have a good reason to.
When running lib under PyPy -X track-resources WARNING about unclosed socket object raised

Closing this socket object further passed to pypy garbage collector.
What I found from PyPy documentation is 

To the best of our knowledge this problem has no better solution than fixing the programs. If it occurs in 3rd-party code, this means going to the authors and explaining the problem to them: they need to close their open files in order to run on any non-CPython-based implementation of Python.
Here is the [link](http://doc.pypy.org/en/latest/cpython_differences.html)

tried with requests=2.18.4, pypy=5.6

Here is the trace
```
ERROR 2017-09-09 17:29:30,345 log:29 STDERR     import requests
ERROR 2017-09-09 17:29:30,345 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/requests/__init__.py", line 43, in <module>
ERROR 2017-09-09 17:29:30,345 log:29 STDERR     import urllib3
ERROR 2017-09-09 17:29:30,345 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/urllib3/__init__.py", line 8, in <module>
ERROR 2017-09-09 17:29:30,346 log:29 STDERR     from .connectionpool import (
ERROR 2017-09-09 17:29:30,346 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/urllib3/connectionpool.py", line 29, in <module>
ERROR 2017-09-09 17:29:30,346 log:29 STDERR     from .connection import (
ERROR 2017-09-09 17:29:30,346 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/urllib3/connection.py", line 39, in <module>
ERROR 2017-09-09 17:29:30,346 log:29 STDERR     from .util.ssl_ import (
ERROR 2017-09-09 17:29:30,346 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/urllib3/util/__init__.py", line 3, in <module>
ERROR 2017-09-09 17:29:30,346 log:29 STDERR     from .connection import is_connection_dropped
ERROR 2017-09-09 17:29:30,346 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/urllib3/util/connection.py", line 130, in <module>
ERROR 2017-09-09 17:29:30,346 log:29 STDERR     HAS_IPV6 = _has_ipv6('::1')
ERROR 2017-09-09 17:29:30,346 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/urllib3/util/connection.py", line 119, in _has_ipv6
ERROR 2017-09-09 17:29:30,346 log:29 STDERR     sock = socket.socket(socket.AF_INET6)
ERROR 2017-09-09 17:29:30,347 log:29 STDERR   File "/home/davit/venvs/pypy-5.6/site-packages/gevent/socket.py", line 227, in __init__
ERROR 2017-09-09 17:29:30,347 log:29 STDERR     self._sock = _realsocket(family, type, proto)
```

The traceback you have there is from urllib3: please open the issue there.
FR: Allow SSL verification to be disabled with an env var

Summary.

Lots of libraries use `requests`. If there's an error verifying SSL in a third-party library it'd be useful to be able to just set an env var to disable SSL verification while the issue is fixed in the failing library. 

This issue is currently affecting me and [others](https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/192) after upgrading `google-cloud`. Even modifying that library to pass `verify=False` would be no mean feat.

I guess it'd make sense if it used the `PYTHONHTTPSVERIFY` env var that [urllib](https://www.python.org/dev/peps/pep-0493/) uses.

@boosh, I believe this has been discussed and soundly rejected before. Please search open **and** closed issues prior to opening new ones. 

Further, the PEP you're referencing is not intended to be used like this. For one thing, it was added because the standard library was *insecure by default* and the transition to making it secure by default was deemed problematic by some. This was an escape hatch to avoid having to update a formerly insecure block of code to properly handle being secure. I don't fully understand what is happening with `google-cloud` but I suspect that the solution to their problem doesn't actually belong here.

This has been requested before and I think our position is the same: global switches to disable certificate validation are an attractive nuisance that tend to be used to mask actual problems. 

I don't think the brokenness of other libraries requires us to provide this functionality.
Update allow_redirects docstrings defaults


# [Codecov](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=h1) Report
> Merging [#4277](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/fdb9b753b47e7abead1ec75d860c9c9b929ca5f5?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4277/graphs/tree.svg?token=mZGswIiXLa&src=pr&height=150&width=650)](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4277   +/-   ##
=======================================
  Coverage   88.66%   88.66%           
=======================================
  Files          18       18           
  Lines        2021     2021           
=======================================
  Hits         1792     1792           
  Misses        229      229
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/sessions.py](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=tree#diff-cmVxdWVzdHMvc2Vzc2lvbnMucHk=) | `95.55% <ø> (ø)` | :arrow_up: |
| [requests/api.py](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=tree#diff-cmVxdWVzdHMvYXBpLnB5) | `81.81% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=footer). Last update [fdb9b75...853a4c4](https://codecov.io/gh/requests/requests/pull/4277?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is still relevant imho, do you think we can reopen this PR, @kenneth-reitz @Lukasa ?

Well, that's a blast from the past. I'm happy to fix it- does it just require the removal of that second, incorrectly updated, docstring?

@geokala I suppose it is. :) If it's of no relevance for you at this point, I will be equally happy to take this over.

Just ran into this myself, would be great if this could be merged so as to not cause confusion.

@Unit03 Happy for you to take it over if you're happy to.

@geokala I will do just that, then, thank you for your input. :)

@Lukasa I've addressed your remarks and opened a new PR (to not take more of @geokala's time): #5127 - but I'd be ok to contribute this change in some other way if you think it's appropriate.
Something more robust to deal with the following

Summary.

Seems a little overkill to have to do https://stackoverflow.com/questions/22346158/python-requests-how-to-limit-received-size-transfer-rate-and-or-total-time to deal with sites like the below. Would be nice to have that built in. But maybe thats asking to much.

Question on Stack: https://stackoverflow.com/questions/46061786/python-requests-timeout-not-working/46061958#46061958

## Expected Result
Error with running:

requests.get('http://arboleascity.com',timeout=(5,5),verify=False)

What you expected.

Would timeout in 5 seconds. But it does not. It works 99.9% of the time but for this site it does not. 

Probably a bug. Any help?

## Reproduction Steps

```python
import requests
requests.get('http://arboleascity.com',timeout=(5,5),verify=False)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "4.4.0-1032-aws",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

The answer is in your StackOverflow question. Your connection is likely taking less than 5 seconds and you're probably receiving some data at least once every 5 seconds. What you seem to be expecting is some kind of total timeout on the time it takes to complete downloading the response and that's not something that we can provide at this point in time.

You can search closed issues for other discussions of this and look specifically for "total timeout" or "wall clock timeout". There are other 3rd party solutions that can help you achieve your goal.
Requests tries to resolve `user:pass@host` on OS X

If you have proxy bypass settings, requests tries to resolve the hostname before stripping credentials from the URL.

It looks like this happens because the [proxy bypass](https://github.com/requests/requests/blob/4bda7b66e7ece5be51b459edd046a70915b4792c/requests/utils.py#L78) logic is called before [urldefragauth](https://github.com/requests/requests/blob/b9638ec7be9134786e28684ca17f3d161297ac5e/requests/utils.py#L879) strips the credentials from the URL.

## Expected Result

Should not try to resolve an invalid hostname.

## Actual Result

DNS resolution for `user:pass@host` is made:

![screenshot 2017-09-05 11 04 11](https://user-images.githubusercontent.com/107611/30056277-0279af7a-922a-11e7-9223-ef66b5bd2af6.png)

## Reproduction Steps

On OS X with proxy bypass hosts configured:

![screenshot 2017-09-04 18 08 46](https://user-images.githubusercontent.com/107611/30056416-864198b8-922a-11e7-9448-f2e702c95927.png)


```python
import requests
requests.get('http://user:pass@localhost')
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.10"
  },
  "platform": {
    "release": "15.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

So this seems to be an issue with macOS specifically, and in particular how it handles proxy bypass. We should probably strip the auth from the host before we bypass.
allow_redirects default for 'head' not shown in help text

When checking help(requests.request) to get the default parameters, the text is misleading if a head request is going to be used.

## Expected Result

The help text should indicate where the default differs OR the help text for .head should indicate that it overrides the default.

e.g. 
    :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``False`` for head requests, ``True`` for all others.

## Actual Result

The 'head' help text displays only that it takes the args of a 'request' object, and request displays:
    :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.

## Reproduction Steps

```python
import requests
help(requests.head)
help(requests.request)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": ""
  }, 
  "idna": {
    "version": "2.6"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.6"
  }, 
  "platform": {
    "release": "3.16.0-30-generic", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "", 
    "version": null
  }, 
  "requests": {
    "version": "2.18.4"
  }, 
  "system_ssl": {
    "version": "1000106f"
  }, 
  "urllib3": {
    "version": "1.22"
  }, 
  "using_pyopenssl": false
}
```

HEAD overrides the default and should be updated. 

I've created a PR for this here: https://github.com/requests/requests/pull/4277

I tried updating just in requests.head, but then the requests.request still has an incorrect docstring if it is only updated there so for a first pass I've updated in just the one place. I'm not sure if this is 'right' for requests, or if there are any similar examples (I had a quick skim through api.py but am possibly being oblivious this evening).

I also applied the same (almost) change to session.py since the same different default is present there.

It would be nice to document somewhere the motivation for setting `allow_redirects` to False for `HEAD` requests.
Handling Multiple Sessions with requests.Session()


Hello, Im trying to make multiple Sessions at once, I need to login multiple users into my webpage, but when instantiating requests.Session() it seems like the session is shared, and if I call requests.Session() again there I have the same cookies and the same headers that I stored in the previus object.

Sorry for my english, Im Spanish

Thank you!

The `Session` object is not shared: it's a class object, it cannot be shared. Can you provide some sample code that reproduces the issue?

Wow so fast Lukasa! Thank you for answering, I solved my problen not using Session, instead getting the cookiejar and store it into my class variable.

Anyway My code was like the next one:

```python
class User:
    def __init__(self,username, password,session=None):
        self.username = username
        self.password = password
        self.session= session or requests.Session()
    def login(self):
        result = self.session.post( "url", data={"username":self.username, "password": self.password} )
        print( "%s: %s" % (self.username,result.text) )

if __name__ == "__main__":
    u = User( "pepe", "asdf" )
    u.login()
    u2 = User("juan", "asdf" )
    u2.login()
```

And it the first time return the data concerning the user "pepe", but the second time, it returns two times the data concerning "juan".

I dont know if I explain myself :/ my english are very bad :P 

Anyway I solve my problem using cookiejar :) 

Very Very thank you anyway :D


Happy to be of some service! I'm glad you solved your problem. :smile:

I found the error, I was instantiatting the logger twice xD :P haha! :)
Recieving 400 Bad Request on PUT against Ranger API 

Summary.

Submitting a PUT request against the Ranger API: https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management#RESTAPIsforPolicyManagement-UpdatePolicy. I can create the policy fine, but seems with PUT, I get bad request and cannot determine why.

Headers on GET
```
Retrieving and exporting policy 16: <POLICY_NAME>
{'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 14:57:25 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'}
```

Headers on successful POST
```

{'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 15:34:04 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'}
Policy applied <POLICY_NUM>
```

Headers on PUT (400 respose)
```
Submitting PUT API request: http://HOST:PORT/service/public/api/policy/<POLICY_NUM>
{'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DIOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Tue, 29 Aug 2017 15:19:58 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'}
```

The object exists at the endpoint, so a PUT needs made.

## Expected Result

Response code 200

What you expected.

## Actual Result

```
Traceback (most recent call last):
  File "ranger-policy-manager.py", line 183, in <module>
    result.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request
```

## Reproduction Steps

Hard to reproduce without a Hadoop Ranger environement.

400 bad request generally suggests that this is a problem with your data not matching what Hadoop wants. Without knowing what that looks like, I'm afraid we simply cannot help you.
Document Community Extensions

Exceptional:

- https://github.com/requests/toolbelt
- https://github.com/ionrock/cachecontrol

Note-worthy:

- https://github.com/kislyuk/requests-http-signature
- https://github.com/ross/requests-futures
- https://pypi.python.org/pypi/requests-mock
- https://vcrpy.readthedocs.io/en/latest/
- https://pypi.python.org/pypi/requests-cache

Participation award:

- https://github.com/twisted/treq (maybe https://pypi.python.org/pypi/requests-twisted/0.1.1 instead?)
- https://github.com/getsentry/responses (maybe not mention)
- https://httpie.org

Let's list others. 

I would say `cachecontrol` falls under `Exceptional`. `responses` isn't an extension and if it is then there are far better ones out there.

@sigmavirus24 thanks for the feedback :)

updated

@kennethreitz  i would like to work on this. Should they be documented in the ReadMe.md file?

I think these are already done, actually, @vhawk19 http://docs.python-requests.org/en/latest/community/recommended/
A problem with proxies parameters

I have a problem about proxies, when I was using ip agent
`requests.get('http://1212.ip138.com/ic.asp', headers=headers,proxies=pro, timeout=9)`
How can i know if I succeed to use the ip agent, sometimes the result from the url is my own ip instead of the ip in proxies.Does that means the ip in proxies is useless?
Thx for your reply.


Without seeing how your proxies are configured I have no idea if that will work or not.

Given that we haven't seen in additional information to help debug this, I'm going to close this out. Feel free to ping if you have more information @SongJee.
Add environment info to prepared requests docs

When using prepared requests, the environment is not taken into account. This should be reflected in the documentation.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4264?src=pr&el=h1) Report
> Merging [#4264](https://codecov.io/gh/requests/requests/pull/4264?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/369be8afd07e7a0f38bc4f78d6f3764a800711aa?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4264/graphs/tree.svg?token=mZGswIiXLa&src=pr&height=150&width=650)](https://codecov.io/gh/requests/requests/pull/4264?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4264   +/-   ##
=======================================
  Coverage   88.66%   88.66%           
=======================================
  Files          18       18           
  Lines        2021     2021           
=======================================
  Hits         1792     1792           
  Misses        229      229
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4264?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4264?src=pr&el=footer). Last update [369be8a...785c733](https://codecov.io/gh/requests/requests/pull/4264?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Pin to httpbin 0.5.0.

The release of httpbin 0.6.0 seems to have broken pytest-httpbin in a surprising way. We want to pin that version out, and probably also alert @kevin1024 about the problem.

Resolves #4259.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4261?src=pr&el=h1) Report
> Merging [#4261](https://codecov.io/gh/requests/requests/pull/4261?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/5dcbdb50fc9652cbeffeec22a21d1196e938bfca?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4261/graphs/tree.svg?token=mZGswIiXLa&src=pr&width=650&height=150)](https://codecov.io/gh/requests/requests/pull/4261?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4261   +/-   ##
=======================================
  Coverage   88.66%   88.66%           
=======================================
  Files          18       18           
  Lines        2021     2021           
=======================================
  Hits         1792     1792           
  Misses        229      229
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4261?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4261?src=pr&el=footer). Last update [5dcbdb5...3a2ea9d](https://codecov.io/gh/requests/requests/pull/4261?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


OK, this is a really crazy rabbit hole - I think @Lukasa is actually more up what's going on than I.

I think the problem is caused by this commit to httpbin by @kennethreitz:

https://github.com/kennethreitz/httpbin/commit/2e94212886e4c9531b8ba7f55f4fb0b8134e10ac

which is an attempt to fix kennethreitz/httpbin#340

BUT I think it only fixes it on Gunicorn?  And pytest-httpbin uses werkzeug which has chunked encoding issues so it ends up causing problems: https://github.com/pallets/werkzeug/issues/1149

@kevin1024 we can remove that — I don't think it fixed anything. 

@kennethreitz sweet, I think this problem should magically disappear in that case 🌟
tests are broken

Specifically, this hangs https://github.com/requests/requests/blob/5dcbdb50fc9652cbeffeec22a21d1196e938bfca/tests/test_requests.py#L151

It can be seen in #4258 https://travis-ci.org/requests/requests/jobs/268764155

This occurs locally for me as well.

I'm loving these new GitHub line previews. 

So fancy!

See #4261.
info for #4259 DO NOT MERGE


@kennethreitz Hey there! Could you please re-open this? There is a hanging test I want to verify in CI.

sorry!

@kennethreitz No problem! I submitted #4259 

Ok, so the first note is that I don't reproduce this failure locally. That suggests that the failure is environmental: either OS-based (I'm on macOS) or something more nuanced. My first guess is that it's related to proxies, so I'm going to chase that rabbit hole.

Ok if we fail out of the `mixed_case_scheme_acceptable` test the next test hangs instead. Let me see if I can escape that one too.

Ok so what seems to be happening is that the HTTP server is blocking forever. Let's try to find out why.

Hrm, here's a weirdness: this doesn't appear to happen when I run the tests under tox.

Ohhhhh I know what this will be. Ok, one sec.

Yup, got it. Ok, will open a PR that fixes this problem.

Between this and that macOS memory allocation thing, you're the King of Bug Hunting!

Heh, thanks!
Requesting the ability to globally override the CA bundle path

Currently, requests.certs.where() provides the path to the CA bundle by means of the certifi library.  While this is fine it presents issues for developers who are using third party libraries requiring the requests library in situations where they need to use a internal or self signed CA.

A method prescribing a global override for this CA bundle path as a first pass would be extremely helpful.  This method could be as easy as setting an environment variable, ie. CA_PATH=/path/to/ca/bundle.crt

Such an environment variable already exists: `REQUESTS_CA_BUNDLE`. I hope that helps!
Git Clone Fails

The repository does not clone, due to a bad commit. Let me know if I can do anything more to help!

## Expected Result

The repository should be cloned successfully with the command:

```
git clone git@github.com:requests/requests.git
```

## Actual Result

I get this message:

```
Cloning into 'requests'...
remote: Counting objects: 20967, done.
remote: Compressing objects: 100% (25/25), done.
error: object 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8: badTimezone: invalid author/committer line - bad time zone
fatal: Error in object
fatal: index-pack failed
```

## Reproduction Steps

run:

```
git clone git@github.com:requests/requests.git
```

Thanks for the report. This is a known problem: see #3088, #3008, and #2690 for previous discussions. The problem can be easily resolved on your end either by performing a shallow clone (if you don't need the history) or turning off the fsck timestamp check. This cannot be fixed without rewriting our history and causing *way* more problems that way than by just letting it sit.

Please, search open **and** closed issues before opening new ones in the future, @evanshauser
If there is a value in the body, setting the content-length in the header does not transfer it.

If there is a value in the body, setting the content-length in the header does not transfer it.
I wonder if you are willing to give me the option to ignore the body value and pass in the value set by content-length
If you are thinking about providing options, I am willing to send a pull-request

## Expected Result
I set the header to a content-length of 100, and the size of the body is 10.
I expected the content-length to be 100, but actually it was 10.

## Actual Result
Looking at the code, there was logic to update the content-length by calculating the value of the body if it had a value for the body.
(requests/models.py - line 515, in prepare_content_legnth method)

## Reproduction Steps
This happens when you set the content-length and enter a different size than the value set in the body.


## System Information
It is not system dependent.

Hey @kssim, you should be able to accomplish what you're looking for by creating a PreparedRequest and then modifying the value of the header before sending.

You really shouldn't be sending incorrect content lengths though as this can cause issues for some servers and will often leave sockets hanging open waiting for more data.

Let us know if you're still experiencing issues with the above approach.

I know that sending invalid values can affect the server.
I'm trying to validate my server using requests, but I need to intentionally send the wrong value.

So, I asked if you could provide that functionality as a send option, not a formal function.
For example, through a boolean value like allow_redirects

Sorry if I didn't provide enough information in the last response. Requests is fully capable of handling your use case, you'll just need to use what we call the "PreparedRequest workflow".

The Content-Length calculation should only be performed when prepare() is called. This will produce a PreparedRequest object which will have the newly computed Content-Length based on the body size. You can override the Content-Length header on the PreparedRequest object and then send it using the send method on a Session.

As for a specific option, the Requests API is fairly frozen at this point. I think it's unlikely we'd provide an option or any further modifications to address this since it's already achievable though other means.

Thanks for the kind answer.
I think it's better to override and use it as needed.
Thank you :)
get an error OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF'),

get an error OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF'),
## Expected Result
I just make a simple request, and i just  want to see what is the response 

## Actual Result

OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

## Reproduction Steps
here is my code

```python
import requests
r = requests.post("https://gateway.gopay.com.cn/Trans/WebClientAction.do", data=param_dict)
```

## System Information

    $ python -m requests.help

```
<paste here>
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Hi @liying1993,

Please provide more information about your system. Without that we cannot provide much help.

In the absence of more detail I'm assuming this is a dupe of #4244.

@Lukasa sorry,I have not check my email boxes in time,
I have asked our cooperative corporation which is the server I requested.They told me they need   to bound our company's.something about DNS.So,thank you
Configuration to follow redirect method after http redirect

Summary.
Basically HTTP 301 redirection replaces POST method with a GET.
More Info: https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect
Need an config option in python requests to override this behavior and follow redirect method after http redirection.

Thanks
Raja


For overriding our redirect logic we have a coarse grained switch: `allow_redirecte`, which can be set to `False`. Failing that, more modern versions of Requests have a `Session.rebuild_method` method that can be overridden to change the behaviour to get what you're looking for. 

There is no plan to add an additional config option for this behaviour.
Error with relative url

I have a problem with a specific relative URL, but I suppose it's not the only one.
When i do:
`print(requests.get('http://www.resultados.gob.ar/escrutinio/dat99/../../javascript/mun.js')`

## Expected Result

The content of the js

## Actual Result

> b'File not found."'

## Reproduction Steps

```python
import requests
print(requests.get('http://www.resultados.gob.ar/escrutinio/dat99/../../javascript/mun.js').content)
```

## Other info
With other links in the same web, it not happened, for example:
`print(requests.get('http://www.resultados.gob.ar/escrutinio/dat99/../dat01/DDN01001A.htm').content)`

If i do it with no relative path:
`print(requests.get('http://www.resultados.gob.ar/javascript/mun.js').content`
is ok either

And in firefox or chrome the same relative url is ok too

So i think it can be the '.js' at the end in combination with relative urls?

## System Information
```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "4.10.0-32-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

Yeah, it's currently the case that Requests does not clean up relative URLs. This is not planned to change in the short term: eventually we'll want to add a dependency on a proper URL handling module in urllib3, but until that work gets done we can't resolve this. 

If you're interested in doing that work please open an issue on urllib3 and we can discuss steps forward.
requests-http-signature

I have published a new PyPI module, https://github.com/kislyuk/requests-http-signature, to implement the IETF draft for HTTP signatures (https://tools.ietf.org/html/draft-cavage-http-signatures-07). This draft RFC formalizes the use of symmetric (HMAC-SHA1, HMAC-SHA256) and asymmetric (RSA-SHA256, ECDSA-SHA256) message signing in HTTP.

Is there interest in bringing this module under the requests organization?

Very nice work! For the moment I think it would be best to keep it separate; this ensures we don't bottleneck on the limited resources of the Requests team. However, if there is significant uptake and you no longer have the resources to do solo maintainership please let us know and we'll re-evaluate. 

For reference, here is an issue tracking HTTP Signatures standard finalization: https://github.com/w3c-dvcg/http-signatures/issues/1

We should have a document in our docs for modules like this

https://github.com/requests/requests/issues/4267
Fix parse_header_links on empty header

When receiving an empty `Link:` header, one bogus entry was returned anyway. This breaks some applications.

I agree that web servers shouldn't be sending those but they do so here we are. Also fixes docstring with proper return type.

Fixes https://github.com/halcy/Mastodon.py/issues/74

# [Codecov](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=h1) Report
> Merging [#4245](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/af9dd61fcba5a1d0b3f75abf5499f6e1f845349c?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4245/graphs/tree.svg?width=650&token=mZGswIiXLa&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4245      +/-   ##
==========================================
+ Coverage   88.65%   88.66%   +0.01%     
==========================================
  Files          18       18              
  Lines        2018     2021       +3     
==========================================
+ Hits         1789     1792       +3     
  Misses        229      229
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/utils.py](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=tree#diff-cmVxdWVzdHMvdXRpbHMucHk=) | `82.87% <100%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=footer). Last update [af9dd61...8e411d0](https://codecov.io/gh/requests/requests/pull/4245?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Sure thing!

Congratulations on your first Requests contribution! ✨🍰✨

You guys are very friendly. Thanks for the kind words. I'll try to patch the bogus server as well.
Slow charset detection on Json file with no explicit encoding

Hello!

In Reproduction Steps a large JSON file is downloaded - about 180MB

When I called rs.text (step 6), I did not wait for the result - I waited about 25 minutes

Console:

```
None
[1] total time: 4.0396250532594795 sec
[2] total time: 3.9574704779202534 sec
[3] total time: 0.271754903759728 sec
[4] total time: 4.078747523628547 sec
[5] total time: 0.27294884912872064 sec
[6] total time: 1515.3671759879826 sec
```

## Reproduction Steps

```python
class TimeThis(object):
    def __init__(self, title="TimeThis"):
        self.title = title
        self.start_time = None

    def __enter__(self):
        import time
        self.start_time = time.clock()

        return self

    def __exit__(self, exc_type, exc_value, exc_traceback):
        import time
        print('[{}] total time: {} sec'.format(self.title, time.clock() - self.start_time))


url = 'http://opendata.mkrf.ru/opendata/7705851331-register_movies/data-16-structure-3.json'

import requests
rs = requests.get(url)
print(rs.encoding)

with TimeThis("1"):
    rs.json()

with TimeThis("2"):
    import json
    json.loads(rs.content, encoding='utf-8')

with TimeThis("3"):
    rs.content.decode('utf-8')

with TimeThis("4"):
    import json
    json_data = rs.content.decode('utf-8')
    json.loads(json_data)

with TimeThis("5"):
    rs.encoding = 'utf-8'
    rs.text
    rs.encoding = None

with TimeThis("6"):
    rs.text

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.1"
  },
  "platform": {
    "release": "7",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.3"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

Windows7, Python 3.6.1 | Anaconda 4.4.0 (64-bit)

Thanks for this! This is a dupe of #4235: closing to centralise.
.get params is ignored if there is a double slash in the url

Summary.

The behaviour doesn't seem to be consistent everywhere, but when I run the following with Python 3 / requests 2.14-2, params is ignored in the double-slash case:

```python
import requests

response = requests.get("https://erikberg.com/nba/results/atlanta-hawks.json", params={"season": "2014"}, verify=False)
print("No double slashes", response.url)

response2 = requests.get("https://erikberg.com//nba/results/atlanta-hawks.json", params={"season": "2014"}, verify=False)
print("With double slashes", response2.url)
```


## Expected Result

`response2.url` should be the same as `response.url`:

No double slashes https://erikberg.com/nba/results/atlanta-hawks.json?season=2014
With double slashes https://erikberg.com/nba/results/atlanta-hawks.json?season=2014

## Actual Result

`response2.url` was the raw url, with no params attached to it:

No double slashes https://erikberg.com/nba/results/atlanta-hawks.json?season=2014
With double slashes https://erikberg.com/nba/results/atlanta-hawks.json

## Reproduction Steps

```python
import requests

response = requests.get("https://erikberg.com/nba/results/atlanta-hawks.json", params={"season": "2014"}, verify=False)
print("No double slashes", response.url)

response2 = requests.get("https://erikberg.com//nba/results/atlanta-hawks.json", params={"season": "2014"}, verify=False)
print("With double slashes", response2.url)

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "1.8.1"
  },
  "idna": {
    "version": ""
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.1"
  },
  "platform": {
    "release": "3.18.0-14875-g438cb8ab27c6",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "100020bf",
    "version": "17.0.0"
  },
  "requests": {
    "version": "2.18.3"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}
```

Did you say this happens with Requests 2.14? Why does your help output identify the Requests version as 2.18.3?

The case above seems to be very specific, testing with any other url that has a double slash in it seems to work. 

    >>> response = requests.get("https://www.sinax.be/blog", params={"season": "2014"})
    >>> response.url
    'https://www.sinax.be/blog/?season=2014'
    >>> response = requests.get("https://www.sinax.be//blog", params={"season": "2014"})
    >>> response.url
    'https://www.sinax.be/blog/?season=2014'


Note: reproduced with requests version 2.18.4


This is not an issue with requests, but with the website. 
What happens is that the site redirects the url `https://erikberg.com//nba/results/atlanta-hawks.json?season=2014` to `https://erikberg.com/nba/results/atlanta-hawks.json`. 

Indeed, check `response.history` for more information.

Thanks for investigating and sorry for the hassle!
attempted fix of travis


# [Codecov](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=h1) Report
> Merging [#4241](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/1c2022cf868cb503815f34901ad8e85cf524d01a?src=pr&el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4241/graphs/tree.svg?width=650&height=150&src=pr&token=mZGswIiXLa)](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4241      +/-   ##
==========================================
+ Coverage   88.65%   88.85%   +0.19%     
==========================================
  Files          18       18              
  Lines        2018     2054      +36     
==========================================
+ Hits         1789     1825      +36     
  Misses        229      229
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/utils.py](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=tree#diff-cmVxdWVzdHMvdXRpbHMucHk=) | `84.17% <0%> (+1.42%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=footer). Last update [1c2022c...ade8379](https://codecov.io/gh/requests/requests/pull/4241?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


not working

do not merge
Adding header name to exception

Adds the name of the header to the invalid header exception raised on TypeError.

Fixes #4239 

I like this. 

# [Codecov](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=h1) Report
> Merging [#4240](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/1c2022cf868cb503815f34901ad8e85cf524d01a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4240/graphs/tree.svg?token=mZGswIiXLa&width=650&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4240   +/-   ##
=======================================
  Coverage   88.65%   88.65%           
=======================================
  Files          18       18           
  Lines        2018     2018           
=======================================
  Hits         1789     1789           
  Misses        229      229
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/utils.py](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=tree#diff-cmVxdWVzdHMvdXRpbHMucHk=) | `82.75% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=footer). Last update [1c2022c...d528106](https://codecov.io/gh/requests/requests/pull/4240?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Can we add a test for this change?

@Lukasa I modified the existing tests to include a basic assertion that the header name is included in the exception message - If you would prefer that assertion be more specific to match the expected exception statement exactly, I can definitely add that in there as well.

History updated, thanks again for your feedback!

Thanks for this! :sparkles: