Raibek/requests-github-issues huggingface dataset:
2.11.1: Using OrderedDict to order headers on session doesn't allow new headers to be added during later request:s

For kicks I decided to order my headers.  According to the directions http://docs.python-requests.org/en/master/user/advanced/ here I just constructed a OrderedDict and set the session.headers value to it.  Now when I use the session for requests e.g. session.get("site", headers={"Referer": "test"}) and try to add a new header is doesn't take.  Only if the header value already exists can it be overriden.  According to the documentation it should be merged.


I can't reproduce this. Can you please provide some sample code? Here is mine:

``` python
>>> import requests
>>> requests.__version__
'2.11.0'
>>> import collections
>>> s = requests.Session()
>>> s.headers = collections.OrderedDict()
>>> s.headers['X-Custom-Whatever'] = 'val'
>>> r = s.get('http://http2bin.org/get', headers={'Custom-Key': 'Custom-Value'})
>>> print r.text
{
  "args": {}, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Connection": "keep-alive", 
    "Custom-Key": "Custom-Value", 
    "Host": "http2bin.org", 
    "Via": "1.1 http2bin.org", 
    "X-Custom-Whatever": "val"
  }, 
  "origin": "24.185.108.14", 
  "url": "http://http2bin.org/get"
}
>>> r.request.headers
{'Custom-Key': 'Custom-Value', 'X-Custom-Whatever': 'val'}
```


Seems to work for me as well for what it is worth. Python 3.5.2, requests 2.11.1, OSX El Captian 10.11.6

``` python
Python 3.5.2 (default, Jul 28 2016, 21:28:00)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.11.1'
>>> import collections
>>> s = requests.Session()
>>> s.headers = collections.OrderedDict()
>>> s.headers['X-Custom-Whatever'] = 'val'
>>> r = s.get('http://http2bin.org/get', headers= {'Custom-Key':'Custom-Val'})
>>> print(r.text)
{
  "args": {},
  "headers": {
    "Accept-Encoding": "identity",
    "Connection": "keep-alive",
    "Custom-Key": "Custom-Val",
    "Host": "http2bin.org",
    "Via": "1.1 http2bin.org",
    "X-Custom-Whatever": "val"
  },
  "origin": "24.107.123.117",
  "url": "http://http2bin.org/get"
}

>>> r.request.headers
{'X-Custom-Whatever': 'val', 'Custom-Key': 'Custom-Val'}
```


Since @phrozetime hasn't responded and we have multiple people unable to reproduce it. I'm closing this until we have more details.
connection: keep-alive not working with requests used in pyArango

Hi, 
requests is used in the connection object of pyArango:
https://github.com/tariqdaouda/pyArango/blob/master/pyArango/connection.py
Though I only see one instance of the connection object being created and used through out the life of https://github.com/tariqdaouda/pyArango/blob/master/examples/debiangraph.py (I've added printfs in these cases) the http connection isn't re-used.

I've followed the connection between pyArango and ArangoDB with Wireshark, to find both of them sending `connection: Keep-Alive` thus the client instantly disconnecting the tcp connection after the HTTP-Requests is done. 

I'm a bit out of Ideas howto better debug this, is there a smart way? 


You're saying that wire shark shows the requests-based client sending `Connection: keep-alive`, is that right? If so, what is the server sending?


yes, its sending that. The server does send that too. Then the client closes the connection, just to open a new one in subsequent requests.


@dothebart It's hard to know exactly what's going on here, but the connection will be left open unless the `Session` is leaked, closed, or otherwise lost. The only other way the connection gets lost is if `Connection: close` is set.


Ok, we found the issue. The bearer object keeping the session object went out of scope and thus closed the connection.
Setting session cookies from cookiejar

I'm trying to set the session cookies from a cookiejar, but having no joy. Here is my current code, cookies, and the error.

Surely there has to be a way to set the session cookies from a cookiejar?

```
def cookieLogin(cookies):

    with open(cookies) as f:
        cookies = requests.utils.cookiejar_from_dict(json.load(f)[0] )

    session = requests.session()
    session.cookies = cookies

    r = session.get('https://example.com')
```

Cookie example:

```
[
{
    "domain": ".example.com",
    "hostOnly": false,
    "httpOnly": false,
    "name": "act",
    "path": "/",
    "sameSite": "no_restriction",
    "secure": false,
    "session": true,
    "storeId": "0",
    "value": "875",
    "id": 1
},
{
    "domain": ".example.com",
    "expirationDate": 1479135421.720188,
    "hostOnly": false,
    "httpOnly": false,
    "name": "c_user",
    "path": "/",
    "sameSite": "no_restriction",
    "secure": true,
    "session": false,
    "storeId": "0",
    "value": "109",
    "id": 2
},
...
```

Error:

```
Traceback (most recent call last):
  File "./cookieInviter.py", line 37, in <module>
    session = cookieLogin(args.cookies)
  File "/Users/jj/Projects/Jim/login.py", line 78, in cookieLogin
    r = session.get('https://example.com')

  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 487, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 461, in request
    prep = self.prepare_request(req)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 394, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/Library/Python/2.7/site-packages/requests/models.py", line 297, in prepare
    self.prepare_cookies(cookies)
  File "/Library/Python/2.7/site-packages/requests/models.py", line 518, in prepare_cookies
    cookie_header = get_cookie_header(self._cookies, self)
  File "/Library/Python/2.7/site-packages/requests/cookies.py", line 136, in get_cookie_header
    jar.add_cookie_header(r)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cookielib.py", line 1342, in add_cookie
_header
    attrs = self._cookie_attrs(cookies)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cookielib.py", line 1301, in _cookie_at
trs
    self.non_word_re.search(cookie.value) and version > 0):
TypeError: expected string or buffer
```

CookieJar:

```
<RequestsCookieJar[<Cookie domain=.example.com for />, <Cookie hostOnly=False for />, <Cookie httpOnly=False for />,
<Cookie id=1 for />, <Cookie name=act for />, <Cookie path=/ for />, <Cookie sameSite=no_restriction for />, <Cookie s
ecure=False for />, <Cookie session=True for />, <Cookie storeId=0 for />, <Cookie value=... for />]>
```


No worries, solved it:

```
def cookieLogin(cookies):

    with open(cookies) as f:
        j = json.load(f)

    session = requests.Session()
    for cookie in j: session.cookies.set(cookie['name'], cookie['value'])

    r = session.get('https://example.com')
```
ALL_PROXY is incorrectly preferred to HTTP(S)_PROXY

Related to #3516.

We should aim to use HTTP(S)_PROXY in priority to ALL_PROXY. Generally speaking, more specific proxy settings should be preferred to less-specific ones.


BTW, to any experienced contributor reading this: please can you leave this on the pile? This is a fairly easy change, so it'd be great to let someone who hasn't had the opportunity to get started in OSS take a swing at this if we possibly can.


Resolved.
It seems that I cannot hide my IP using the proxy?

I want to hide my own IP address but it seems unstable as follows. 

```
import requests
url = "http://httpbin.org/get"
proxy0 = {"http":"http://118.179.220.186:8080"} # just an example, it may have expired
proxy1 = {"http":"http://41.79.60.202:8080"}
r0 = requests.get(url, proxies=proxy0, timeout=5)
r1 = requests.get(url, proxies=proxy1, timeout=5)
r0.json()["origin"] 
r1.json()["origin"] 
```

When I tested it, the origin in r0 contained only one IP while that of r1 contained two IPs including the source IP. 

The same problem was posted on StackOverflow, please refer to [here](http://stackoverflow.com/q/27824055/3552975).


@eduOS In this instance, it's possible that the proxy is setting X-Forwarded-For header, which will contain the IP address of your source. Alternatively, it's possible that you're getting forwarded to a HTTPS address. Either way, it's likely that the connection is still using the proxy: so longa s the scheme of the two URLs match, we _will_ use it.
Not raise ProxyError when I use proxy `https://...:8080` 

Code:

```
proxies = {'https': u'https://...:8080'} # or {'HTTP':'http://:80' }
r = requests.head(u'http://example.com', proxies=proxies, timeout=10)
r.status_code
>>> 200
```

The proxy is broken, seems requests do not use this proxy and make request directly,
it should raise `ProxyError('Cannot connect to proxy.'` here.


That proxy is not broken. We are using the configuration you set. If you set the scheme `{'https': u'https://...:8080'}`, and make a request to a URL beginning with `http://`, you have not set any proxy for us to use: as a result, we don't use it.

The mere presence of a proxy dictionary does not indicate that the user wants _every_ request they issue to be proxied.
Prepare for 2.11.1 bugfix release.

@kennethreitz, this is all we need from a boring administrative perspective. If you merge this PR, you only need to do three things to release:
1. Update the version number to 2.11.1.
2. Put the date you do this in the changelog instead of the XX.
3. Push the release.

It'd be good to get this release out this week: it should put the nastiest bugs of 2.11 behind us and get people back to working again. =)


@kennethreitz let's :shipit:


Awesome! My main machine is down at the moment. I should be able to do this today, though — once I find my laptop (which I was using just a few days ago).


@kennethreitz I think @Lukasa and I can still do it for you if you want us to. Just let us know. We'd both be happy to help out.


I like making the releases. 


Released.
doc updates

Various doc updates for clarity. Some of these may not match @kennethreitz's aesthetic sensibilities, which may require an inversion of the proposed change. I was mainly trying to make things consistent, input on preferences would be greatly appreciated.

Some notes:
- I think at least the faq.rst changes _need_ to be merged because there is still live documentation stating that Requests supports 3.1 and 3.2. I can open a separate PR if we want those changes quicker.
- What was ScraperWiki, is now found at quickcode.io and appears to either be a paid or account-walled service. Wayback Machine is showing the site as a free service when the documentation was created, so I'm not sure if Kenneth still wants this included. At the very least, the URL should be updated.
- ~~Does Requests support pypy 2.2 officially? faq.rst says yes, but it's other versions were wrong.~~
- While I realize it's common to use `class.method` as a pattern when describing code, I felt like `r.json` is ambiguous when referenced next to `r.text`. Calling exactly `r.json` won't function as described in the provided `ValueError` example. Explicitly defining the call as `r.json()` reduces the chance of misinterpretation.
- api.rst has a pared down set of Exceptions and classes. I assume this is to keep extraneous information from bloating the page to a point of being unhelpful. I do think it may be worth including `BaseAdapter` for easy reference since anyone looking into a custom _non-HTTP_ Transport Adapter will likely need to start there.


@nateprewitt is this related to #2228?


@sigmavirus24 It wasn't originally, I found a few minor hiccups making the `.json` change and then did a pass through the docs from there. I realize this is kind of a mess of unrelated doc changes so I can break it up into different commits or PRs if needed.

It looks like most of the discussion in #2228 is about timeouts documentation. Was there something low hanging you wanted to see integrated?


> Was there something low hanging you wanted to see integrated?

No because the last person to comment on that thread wants to contribute and I wanted to make sure we gave them the first chance. If this is unrelated that's fine.


Sorry, I missed that. No, none of these changes touch the timeouts docs, so I think all is well.


These are all fanatastic changes, and ones I would have made myself, given an infinite amount of time :)

Thank you very much!
Requests 2.11 regression - Site ID needs to be String

Using https://github.com/timotheus/ebaysdk-python  v 2.1.3 (latest)

Which transitively uses requests...

Requests 2.10 was good, with Requests 2.11 I get:

```

<Truncated>
  File "/webapps/listing_mirror/lib/python3.5/site-packages/ebaysdk/connection.py", line 119, in execute
    self.build_request(verb, data, verb_attrs, files)
  File "/webapps/listing_mirror/lib/python3.5/site-packages/ebaysdk/connection.py", line 159, in build_request
    self.request = request.prepare()
  File "/webapps/listing_mirror/lib/python3.5/site-packages/requests/models.py", line 251, in prepare
    hooks=self.hooks,
  File "/webapps/listing_mirror/lib/python3.5/site-packages/requests/models.py", line 295, in prepare
    self.prepare_headers(headers)
  File "/webapps/listing_mirror/lib/python3.5/site-packages/requests/models.py", line 409, in prepare_headers
    check_header_validity(header)
  File "/webapps/listing_mirror/lib/python3.5/site-packages/requests/utils.py", line 756, in check_header_validity
    "not %s" % (value, type(value)))

requests.exceptions.InvalidHeader: Header value 961 must be of type str or bytes, not <class 'int'>
```

Can't say for sure yet where the bug is (requests vs ebaysdk)... but at least something changed in a breaking manner.

Trying to investigate more...


Hey @listingmirror, you can find more information on this in #3477. 2.11.0 brought in more rigid header validation, requiring values to be passed as either strings or bytes. I'm betting ebaysdk is passing the Content-Length header as an int. This will need to be converted to a string for use with 2.11.0+.


@nateprewitt  Perfect I will chase it down there ;)
adding passthrough in close() for non-urllib3-like Responses

This is minor, but when testing `resolve_redirects` I encountered an `AttributeError` when passing a non-urllib3 Response. This is because the method unconditionally calls [`close`](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L864) on the Response which in turn calls `release_conn` on the `Response.raw` attribute.

If the desire for requests is to maintain support for non-urllib3 objects in custom adapters, I think this would be a simple and useful addition. If urllib3 is going to be a de facto pattern for Responses though, this is probably just unnecessary code bloat.


Ok, I have put some small notes in. I don't want to merge this until after 2.11.1 gets out the door though.


Test updated and lambda logic changed to a conditional. No rush on this, just wanted to get it on the record before I forgot.


Just reminding ourselves not to merge this until after 2.11.1 is released


Now that 2.11.1 is out, I don't think I have anything else on my checklist. Let me know if this needs any other final tweaks.


Thanks @nateprewitt!
session() got an unexpected keyword argument 'headers'

When trying the second example of [http://docs.python-requests.org/en/v0.10.6/user/advanced/#session-objects](http://docs.python-requests.org/en/v0.10.6/user/advanced/#session-objects) i get the following error:

`TypeError: session() got an unexpected keyword argument 'auth'`

Did something change there and the documentation didn't get updated?


@breize You're linking to an ancient version of the documentation. Try this: http://docs.python-requests.org/en/latest/user/advanced/


@Lukasa this is still in the latest docs version: `http://docs.python-requests.org/en/latest/api/#requests.Session.auth`. I have the same with `headers=`. Are the docs out of date?

Any reason not to accept this as a init kwarg? Seems a bit silly that you have to do:
```
s = requests.Session()
s.headers.update('foo', 'bar')
```
Grammar error in document: Quickstart

In section [Binary Response Content](http://docs.python-requests.org/en/master/user/quickstart/#binary-response-content), `r.content` returns an instance of `bytes` while `StringIO` is an in-memory stream for text only. It should use `BytesIO` instead.

``` python
from PIL import Image
from io import BytesIO

i = Image.open(BytesIO(r.content))
```

Use original code will raise `TypeError: initial_value must be str or None, not bytes.`


@forrestchang Thanks, would you like to open a PR to change it?


@Lukasa Yes, I would like to. I have modified the mistake in `.rst` file. Should I generate it to the HTML file? 


Hey @forrestchang, there's no need to run the docs build. If you open a PR with the rst changes, read the docs will generate changes on merge.


@nateprewitt Thanks.


Resolved by #3511.
Prevent environment variables from overridding Session settings

In some cases, `Session` settings are overridden by environment settings. Session should inherit from its own settings before trying to pull in environment settings. 

For example, say environment variable `REQUESTS_CA_BUNDLE` is set to a certificate bundle, a session object is created and `session.verify = False`, and then that session is mounted to an adapter. If subsequent requests made do not explicitly pass in `verify=False`, certificate verification will be enabled despite having set `session.verify = False`. This is due to `merge_setting` defaulting to use the `request_setting` when merging verify, which is currently incorrectly set to the environment variables certificate file before checking `session.verify`.

This will ensure that `Request` object settings take precedence, then `Session` settings, then global environment settings.


Thanks! This is a known issue (see #2018), but fixing it is a breaking change that needs to go into 3.0.0. Can you please open a new pull request targeting the proposed/3.0.0 branch?


It looks like this has already been fixed in proposed/3.0.0!


It may be worth noting that it's PR #2839 that fixes this.
Any way to set verify=False on session object?

Is there any way to set `verify=False` on a session object rather than having to use it on every HTTP request made by the session?


No. =) If you're going to make yourself unsafe, we'd like you to do it every time. Think of us as like your better angel, sitting on your shoulder saying "are you _sure_ you want to turn off your security?".


Ah ok. I was debugging HTTP requests with Burp Suite. It's the only time I will be setting `verify=False`. I wanted to do it once for development purposes only.


Oh sure, there are lots of totally legitimate reasons to turn it off, that's why we have the switch at all. =D But we just never want to make it _too_ comfortable to turn off.
2.11.0 causing issues with missing attribute 'reason' in raise_for_status()

Apologies for the vague header, I'm still narrowing this down. It appears that this commit (https://github.com/kennethreitz/requests/commit/7700ecae14930fd078e28e35425661d46778bfa9) is using self.reason, and in our implementation, its throwing errors for a missing attribute 'reason'.

The lib in question is django_openstack_auth; all our tests began failing when 2.11.0 released. 2.10.0 still works fine. I'm unsure whether this is an issue with a clumsy test implementation on our side or a false assumption on the requests side. Will keep digging.


I suspect the issue is a bit of both. The higher level requests code expects a reason phrase, but they're not always common, especially in test implementations. That said, I recommend all test implementations _set_ a reason: the code here always expected one, it's just that now it's always checked for, rather than being checked for only in error cases.

If you have no reason phrase to set, just set `''`: that'll work fine.


Makes sense. This can be closed then, thanks for the quick response!


My pleasure!
#3490 removing Content-Type and Transfer-Encoding headers on redirect

This will remove additional headers (`Content-Type` and `Transfer-Encoding`) from non-temporary/non-permanent redirects in `resolve_redirects` to address #3490.

This functionality isn't currently being tested but I'm happy to add a test or two for it if desired.


I would like to see tests for this please! :sparkles:


I was on it too. Did not find a way to properly add tests though since httpbin does not seem to be able to post to an endpoint and respond with a 301/302. Am I wrong on this one?

Anyway since @nateprewitt is on it I will not open my PR ;)


On my version of the fix (https://github.com/kennethreitz/requests/compare/master...saveman71:master) I moved [this line](https://github.com/kennethreitz/requests/pull/3493/files#diff-28e67177469c0d36b068d68d9f6043bfR151) a little higher and removed the try/catch below as it seems the general way (at least in this file) is to check the existence of the key prior to the deletion.


@saveman71, sorry, I didn't mean to jump on your PR. I wasn't sure if anyone had started working on it.

@Lukasa I added a test to check the header removal and empty body.


@Lukasa re: [this line](https://github.com/kennethreitz/requests/pull/3493/files#diff-28e67177469c0d36b068d68d9f6043bfR151)

Do you have an opinion on moving the declaration farther up to avoid the multiple calls to the headers attribute on the prepare_request? Also as for the conversion from try/except, if that was done would you want it in a separate PR to avoid semi-unrelated changes?


@nateprewitt no worries at all, I should have stated I started working on it.
As for the line thing, I do not think it is a performance issue (I believe, but I might be mistaken) because well requests is a network library and "premature optimization is the root of all evil". (https://www.xkcd.com/1691/)

I was suggesting that merely because it's already been like that [here](https://github.com/nateprewitt/requests/blob/20a2b5ac71f1d6ce674ca2633172ef5964ec9c38/requests/sessions.py#L192) and [here](https://github.com/nateprewitt/requests/blob/20a2b5ac71f1d6ce674ca2633172ef5964ec9c38/requests/sessions.py#L223) for example.

On a side note, :+1: for a separate PR.


@saveman71, I only bring up the optimization because @sigmavirus24 has [brought it up before](https://github.com/kennethreitz/requests/pull/3362#discussion_r68398047). If I'm misunderstanding his original intent though, I vote we completely remove the declaration, in favor of `prepared_request.headers`. `headers` currently doesn't provide a lot of use in `resolve_redirects`.

Edit: Clarity and Link


Quick semi-unrelated question, what would be the timeline of a release on PyPI after this gets merged?

cc @Lukasa 


We'd be aiming to release this early next week if we can merge it this week. 


To be clear, this is on my queue, but I'll review tomorrow. 


Alright, fab, I'm happy with this. @sigmavirus24 you ok with us merging this for 2.11.1?


Alright, all is well. @sigmavirus24, you want this in 2.11.1?


👍
Requests is no longer converting Boolean values in headers to strings

This results in an InvalidHeader error. 

For requests 2.10.0:
In [1]: import requests
In [2]: r = requests.get('http://www.google.com', headers={'fake-header':True})
In [3]: r.request.headers
Out[3]: {'fake-header': True, 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '_/_', 'User-Agent': 'python-requests/2.10.0'}

For requests 2.11.0:
In [1]: import requests
In [2]: r = requests.get('http://www.google.com', headers={'fake-header':True})
<Traceback snip>
InvalidHeader: Header value True must be of type str or bytes, not <type 'bool'>


Please see #3477.


See also #3477, #865. Requests has always documented that headers _must_ be strings. We just didn't enforce it very well. It's also entirely unclear how to appropriately represent True on Python.


@Lukasa @sigmavirus24 It looks like the [HISTORY.rst change](https://github.com/kennethreitz/requests/blame/master/HISTORY.rst#L33) specifies only integers. Perhaps that should be changed to 'non-string'?


I found all the history after I posted the bug.  It would have been nice to have at least included this change in the changelog (the entry lists only integers as being nixed).  It would also have been nice to continue supporting python's bools type for True and False at least.


@nateprewitt Yes, it should.

@jidar How do we represent True/False as strings? "True"/"False"? Why that over "true"/"false", "1"/"0", or "Yes"/"No"? Requests is simply refusing to guess in this case, and asking that you unambiguously decide what you want to send.


I should note that, as far as I know, there are _no_ headers defined that take "True"/"False" or any variation thereof as header field values. There are one or two that can conceptually accept "1"/"0". That makes booleans bad candidates for special casing: we'd be much more likely to special-case datetimes and integers than bools.


It's, you know, none at all! :)


@Lukasa str(value) seems acceptable.  No need for special casing anything really.  It's what the user is going to be forced to do on their end now anyway.


@jidar str(value) is not acceptable, nor is it what anyone will be forced to do. That may work in your case with your data type, but with many data types it will not work and will lead to us emitting bizarre headers that make no sense. For example, consider datetime objects, which print internal representations when str() is called on them. 


@jidar if that's acceptable for you, I'm very happy. You should only ever be treating your headers as strings though. I would not expect any movement on this.
adding in pickling tests for PreparedRequest

This is a continuation of @keyan's work in #2757 and addresses issue #1558 regarding pickling Request objects (specifically `PreparedRequest` objects in this PR).

These tests ensure that a standard `PreparedRequest`, a `PreparedRequest` with a file object as the body, and a `PreparedRequest` with hooks defined outside of the `locals` scope will all pickle properly.

This also works for data passed to the `json` parameter and I can add my test for that as well if it's deemed helpful.

Hooks defined inside of a method (such as in the [original test](https://github.com/kennethreitz/requests/pull/2757/files#diff-56c2d754173a4a158ce8f445834c8fe8R843) by @keyan) will fail because you can't pickle local objects. Is this a use case that needs to be handled?

@Lukasa any thoughts here? We still have edge cases where if the `data` param is a generator or file-like object, it can't be pickled. Is there any utility in adding this subset of tests, or should #1558 be closed as can't fix?


I have no particular problems with merging this PR, though I'd like to confirm we're on the same page regarding file objects.


Yeah, I think we're on the same page. I'm also not terribly tied to the PR, I started it trying to wrap up #1558 and Keyan's work, since Kenneth felt there was some value in having these tests.
requests 2.11.0 raises when header contains leading space

With the release of requests 2.11.0 we're seeing errors where things were working with requests 2.10.0. The exception that's raised is "Invalid return character or leading space in header".

We can work around the problem and make sure we set headers without leading spaces, just making sure that this was the intention to have things that were working with 2.10 stop working with 2.11.


Thanks for this!

This is deliberate. Headers with leading spaces in them are frequently mis-handled by intermediaries and servers. This can lead to attacks or misbehaviour, where any or all of a server, client, and all involved proxies disagree on what headers are actually present. This is called a smuggling attack, and it's not a good thing.

Given that this misbehaviour exists, and given that it is not really _sensible_ to add extra whitespace to the header field value, Requests has simply decided to forbid it entirely. This was entirely the intention of the change: we're trying to help Requests emit better-formed HTTP data.


ok, thanks for the explanation.


> This is deliberate

heh, awesome

https://bugs.launchpad.net/python-swiftclient/+bug/1614280
https://review.openstack.org/#/c/356802/1
reverting 3362

This addresses #3481 and will revert #3362 back to its prior state. We'll likely want to reopen #3359 since this will reintroduce the bytes/text issue.


I'm good with this. How about you @Lukasa? (To be honest, I would have rathered let GitHub generate the revert than have it done artisanally.)


I'm fine with this too.


@sigmavirus24 sorry, I tried the revert button and got an error so did it by hand.
Commit 386382b18caab6a3a8a111fa8e24875dca27b0aa is kicking my ass on Python 3

Hi,

Today our automated builds started failing because 2.11.0 was pulled and 386382b18caab6a3a8a111fa8e24875dca27b0aa causes a request's body to be a bytes object in Python 3 but a str in Python 2.7, which is breaking both my code and https://github.com/seanbrant/requests-wsgi-adapter:

https://gist.github.com/alexntt/b181b562852c6850a83269b7401d3bef

Can this be made a bit more "backwards-compatible"?

Cheers,

Álex


Bytes and str are the same type. In fact, bytes aliases to str on Python 2.6. For that reason, it would seem like the code to work around this problem is simple: `if not isinstance(bytes, request.body): request.body.encode('utf-8')`

Indeed, I'd like to point out that this has worked around a bug in requests-wsgi-adapter: specifically, if provided a unicode string containing code points outside U+0000 to U+007F then the content-length would have been incorrect. Because we're now providing bytes to you, this error cannot occur. This change was introduced for exactly this reason.


I've patched the library I depend on and our project so this is no longer an issue, thank you
Can't connect to HTTPS URL because the SSL module is not available.

If I try to fetch any https url, It throws an error on the server, but when I try it on the local machine, It works.

Code:

```
def test():
  data = requests.get('https://newsroom.co',verify=False)
  return data;
```


@rohitkhatri This is almost certainly a problem with the OpenSSL shipped on your server. You'll likely need to elaborate for us to help you: for example, _what_ error do you see?


@Lukasa You can check this out here http://52.40.67.18/testing


Oh that's easy: the Python on your server was compiled without the SSL module. It will be impossible to make HTTPS requests without the SSL module. You'll need to take this up with whomever built your application environment.


So what do I need to do in order to resolve this issue, any suggestion?


My suggestion is the same as what I said in the last comment: you'll need to contact whomever built your application environment. If you're deploying on something like Google App Engine, you'll need to consult their docs. If you're using something like Docker, you'll need to consider changing your base image.


I'm using amazon ec2 instace with ubuntu 14.04 installed on it.


Where is your Python coming from? Because the Python shipped with Ubuntu 14.04 definitely has the SSL module installed.


@rohitkhatri How did you resolve this? I'm getting the same error?

and me

All: you need to contact whomever built your Python install. If it was you, then you need to check the Python documentation because you didn't link against OpenSSL.
Clarify Python versions supported by requests

Remove some confusion in the Python versions supported by Requests.


endashes (–) are appropriate for ranges, not hyphens. 

Let's also add "Requests officially supports".


en dash, not em dash :)

It's cool, I'll fix it. 


✨🍰✨


@kennethreitz thanks, tried copy pasting your en dash from above. Apparently GitHub's UI wasn't a fan of my methods.
requests v2.11 has a syntax error in python3.2

First of all thanks for the awesome library. I am really enjoying using requests. This is my favorite library.

In the last version of requests, I started to get an error on my own library and it turned out it was because of my requests dependency. 

In python3.2, requests v2.11 gives following error:

```
  File "/home/travis/build/huseyinyilmaz/placebo/requests/__init__.py", line 64, in <module>
    from .models import Request, Response, PreparedRequest
  File "/home/travis/build/huseyinyilmaz/placebo/requests/models.py", line 856
    http_error_msg = u'%s Client Error: %s for url: %s' % (self.status_code, reason, self.url)
                                                      ^
SyntaxError: invalid syntax
```

Here is the fix
https://github.com/kennethreitz/requests/pull/3478

Again sorry about the crappy way of testing. I could not install python3.2 on my local machine so I could not test it with requests tests.


Hey @huseyinyilmaz, I think this may have been discussed in #3385 (particularly [here](https://github.com/kennethreitz/requests/pull/3385#discussion_r69560890)). Removing the `u` designation on the string would revert the unicode support added in #3385.


Ah, I see @nateprewitt already chimed in over here.


I see. So, python3.2 is not supported by requests library. Than I will just close the issue and pr.

Thanks for the link @nateprewitt .


Hey,
I just wanted to give some extra feedback. I got this error because when I installed requests with following command:

```
pip install requests
```

It installed an unstable version in python 3.2.
Is it possible to not to release unsupported versions in pypi. So python3.2 would not get any updates but it would always load supported version but old version.


@huseyinyilmaz That is not possible, I'm afraid: you'd need to ask @dstufft to provide tools to do that. It's a nice idea, certainly, although it requires much more formalisation of the trove classifiers system that currently exists.


@huseyinyilmaz I think you're going to have a bad time as more and more projects find it unreasonable to support Python 3.2. There are _very_ few users on that version and it does not even receive security updates. 


There are updates being made to PEP 503 that will make it possible. It'll take some time for setuptools/pip/etc to filter out that respects that.
Fix python3.2 unicode literral error.

This is a specific fix for python3.2.
When I install requests version v2.11.0, I get an invalid syntax error.
It turns out the reason for that error is because "u" prefix for unicode strings are not allowed in python3.2. So I removed the "u" prefix to fix the issue.

To test it ,I moved the request library into my codebase and get syntax error:

https://travis-ci.org/huseyinyilmaz/placebo/builds/150770392

Than I fixed the issue and ran my tests again:

https://travis-ci.org/huseyinyilmaz/placebo/builds/150770803

I realized that this is not a right way to test this. But I could not install python3.2 on my mac. So I decided to use travis-ci's python version.


We have not supported Python 3.2 for ... over a year? I don't think we'll accept this. @kennethreitz, thoughts?


@sigmavirus24 . I did not realize that pyhon3.2 is not supported. Thanks for the explanation. I will close the pr.


I'm running into the same issue, where this totally worked before, and stopped working with today's release. Looked through the changelog but found nothing, should this have been publicized?


```
  File "/home/travis/virtualenv/python3.2.6/lib/python3.2/site-packages/requests/models.py", line 856
    http_error_msg = u'%s Client Error: %s for url: %s' % (self.status_code, reason, self.url)
```


Requests no longer supports Python 3.2. As referenced by @sigmavirus24 this has been the case for ever a year.

If you view the setup.py for requests it specifies 3.3 and higher, or 2.6 and higher.


The setup.py specifies 3.3 or higher, however the documentation homepage and the PyPi page both say: 

> Requests supports Python 2.6 — 3.5, and runs great on PyPy.

These should be amended for clarity, if you're not looking closely this is a pretty innocent mistake.


I've submitted a PR for this in #3480
Prepare for 2.11 release.

@kennethreitz, this is all we need from a boring administrative perspective. If you merge this PR, you only need to do three things to release:
1. Update the version number to 2.11.
2. Put the date you do this in the changelog instead of the XX.
3. Push the release.

@sigmavirus24 I'm inclined to want to let all currently open PRs remain open rather than try to rush merge any: they can always wait for a 2.11.1 or 2.12 without any risk.


> I'm inclined to want to let all currently open PRs remain open rather than try to rush merge any: they can always wait for a 2.11.1 or 2.12 without any risk.

My opinion has been (for a while now) that release numbers are cheap. I'm on board with this. 👍 :shipit: 


Released! ✨🍰✨
process is terminated while starting new https protocol

I use requests pacakge to call nexmo api with post method: requests.post(url)

It has worked well for a month, but after my service is terminated and I check logging throw lastest row: 
 requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): rest.nexmo.com

I catched exception but no errors are written to logging.


That information is nowhere near detailed enough to tell us what happened. You'll need to provide a traceback or something similar.


We can reopen this if and when we get more information.
post with unicode data and the post body will be truncated!!!

when i use requests post data with unicode, and the content-length will be wrong, and the post data will be truncated, test with requests 2.10.0

test code

---

```
# -*- coding: utf-8 -*
import json
import requests
# use utf-8
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

data = json.dumps({'test': u"测试"}, ensure_ascii=False)
print data
resp = requests.post('https://github.com/kennethreitz/requests/issues', data=data, verify=False)
```

fiddler snapshot

---

![u](https://cloud.githubusercontent.com/assets/1853121/17411410/052ec118-5aab-11e6-9486-ab29fd6a7a9e.png)

reason

---

```
data = json.dumps({'test': u"测试"}, ensure_ascii=False)
the unicode length: len(data) = 14
the utf-8 chars length len(data.encode('utf-8')) = 18
encode with utf-8 but the content-length not change, so, the post body is truncated data
data.encode('utf-8')[:len(data)] = {"test": "测
```

is it a bug? need help


Why are you passing `ensure_ascii=False`?


Anyway, that's kind of neither here nor there. There are several solutions to your problem:
1. Don't use `ensure_ascii=False`. There's no requirement to do it: JSON allows unicode escape characters.
2. Alternatively, don't pass Requests a unicode string in `data`. This kind of works in Python 2, but I wish it didn't: it makes no sense. Encode it yourself using `utf-8`.
3. Just defer all of this decision making and use `requests.post(url, json={'test': u"测试"})`.
OverflowError: requested number of bytes is more than a Python string can hold

I'm facing some weird problem.

I'm not able to post ~3GB (more than 2GB) file using requests > 0.7.4 module because of getting following err.

File "/build/toolchain/noarch/requests-2.5.3/lib/python2.6/site-packages/requests/api.py", line 83, in post
  return request('post', url, data=data, *_kwargs)
File "/build/toolchain/noarch/requests-2.5.3/lib/python2.6/site-packages/requests/api.py", line 38, in request
  return s.request(method=method, url=url, *_kwargs)
File "/build/toolchain/noarch/requests-2.5.3/lib/python2.6/site-packages/requests/sessions.py", line 200, in request
  r.send(prefetch=prefetch)
File "/build/toolchain/noarch/requests-2.5.3/lib/python2.6/site-packages/requests/models.py", line 387, in send
  fields.update({k: (fn, fp.read())})
OverflowError: requested number of bytes is more than a Python string can hold

But, with requests<=0.7.4 its working fine. No problem of OverflowError.

So, I was wondering why it started throwing OverflowError in latest version of requests module (or from requests > 0.7.4)?

I tested this with python2.6 and python2.7.

PS: I know that instead of posting huge file at once, we can post same file via file chunks. But, I do not want chucks way.


The problem here is that you're sending the file using a multipart file upload. For requests, this causes us to read in the entire file in order to build the body.

You can resolve this by either not using multipart file uploads (if that's possible), or else by using the [streaming multipart data encoder](https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-multipart-data-encoder) from the Requests toolbelt.


Are you saying requests<=0.7.4 handles multipart file uploads well by default?
But, I dont think this below code uses multipart format

1501         res = requests.post(
1502             server_url,
1503             files={
1504                 'package': open(package, 'rb')},
1505             data={
1506                 'md5sum': md5sum,
1507                 'length_in_bytes': packagelen
1508             })

May I know is it save using requests<=0.7.4 for file uploads?


> Are you saying requests<=0.7.4 handles multipart file uploads well by default?

No. That is not what @Lukasa is saying.

>  But, I dont think this below code uses multipart format

You're using the files parameter, so yes, it is sending a multipart message.

If you describe what you want the body to look like, we _might_ be able to help you, but you should be using [StackOverflow](stackoverflow.com) for help with this module, not the defect tracker.


> May I know is it save using requests<=0.7.4 for file uploads?

It is extremely _unsafe_ to use such an old version of Requests, for any purpose.
Add option to specify custom DNS resolver

My services are running on AWS with some of the applications being external. We would run Nagios check to make sure web applications are still accessible (internally and externally). One challenge I have is to run the check as an external user because our Nagios server is in the VPC so the check is performed over the internal VPC network.

There are many ways to solve my problem, for example recompile libcurl to support resolver option, or run my nagios in another VPC without any knowledge of my external applications, or adopt a proxy server.

However, I found http://stackoverflow.com/questions/22609385/python-requests-library-define-specific-dns. As a Python programmer I like to stick with Python if requests could offer this out of the box.

Is anyone interested and willing to accept an option in requests to specify custom DNS resolver?


Thanks for this request!

Requests is certainly not going to take a patch use use a custom DNS resolver, partly because we don't want to add flags and options for such low-level functionality, and partly because that functionality is owned entirely by the urllib3 library, which we don't carry patches for.

However, this is the kind of thing that urllib3 would be willing to accept in some form, I suspect, so I recommend you open an issue on that repository.

Thanks again for reaching out!


@Lukasa  are you saying if urllib3 is updated, then requests could be in the position of having this option?


No, I'm saying that if urllib3 is updated you can write a [transport adapter](http://docs.python-requests.org/en/master/user/advanced/#transport-adapters) that uses it.
cannot pass realm parameter to HTTPDigestAuth

I need to specify a custom realm parameter for HTTPDigestAuth.
It does not seems to be supported.


When you say you "need to specify a custom realm parameter", what do you believe that parameter will do? The realm is provided by the remote host in the auth challenge.


I get auth failed if I do not specify it. Besides urllib2 allows to pass this parameter.


urllib2 allows you to pass it because it associates a specific realm with a given username/password tuple, such that it will only respond for challenges using that specific realm. Requests has a simpler Digest Auth model which does not have realm-specific passwords.

When you say you get auth failed, I'm going to need some more information. What does your code look like for Requests, and what does it look like for urllib2?


The management interface of Wildfly Jboss is exposed via http/JSON. Basically  it requires to pass realm = 'ManagementRealm' to Digest Auth  otherwise I cannot login as a superuser and get 401. 

The code looks like this:

url = 'http://{host}:{port}/management/host/{host}/server/{server}/core-service/platform-mbean/type/garbage-collector/name/G1_Young_Generation?include-runtime=true'
username = 'admin'
password = 'password'
realm = 'ManagementRealm'

passman = urllib2.HTTPPasswordMgr()  # this creates a password manager
passman.add_password( realm, url, username, password)
auth = urllib2.HTTPDigestAuthHandler( passman)
urlopener = urllib2.build_opener( auth)
request = urllib2.Request(url)
response = urlopener.open(request).read()
d = json.loads( response)
pprint.pprint( d)


What's your equivalent requests code, please?


import requests
from requests.auth import HTTPDigestAuth
r = requests.get(url, auth = HTTPDigestAuth('admin', 'password'))
r.request.headers
r.status_code
r.headers['content-type']
pprint.pprint(r.json())


I assume your passwords here are replacements, and not the actual password? Because they aren't the same.

What does r.history contain?


r.history contains: <html><head><title>Error</title></head><body>401 - Unauthorized</body></html>
user/password are real but I don't know if they're the defaults after installation


Ok, can you please print: `r.history[0].headers` please? I'm interested to see what the challenge is.


BTW, if the username and password are real, why are they different in each example?


ok fixed now


Date: Wed, 03 Aug 2016 11:27:31 GMT
Content-Type: text/html
Connection: keep-alive
Content-Length: 77
WWW-Authenticate: Digest realm="ManagementRealm",domain="/management",nonce="v4CCqEhQR2wNMTQ3MDIyMzY1MTEwMoryoyih0IMIqIg2nq4ZP2M=",opaque="00000000000000000000000000000000",algorithm=MD5

So as you can see, the realm is ManagementRealm in the challenge. So at best the realm portion of this is a red herring. We're using the same realm as urllib2 is. The problem has to be elsewhere.


I'm closing this due to about a month of inactivity. If you can provide more information @alextorex0 such that we can continue debugging this, we will happily reopen this.
Replacing HTTP Content-Length header

Good day,

We use your lib for testing S3 protocol, and found minor issue with HTTP headers:
some of header parameters, such as Content-Length, were override by sessions.prepare_request()
is it possible to add logic not to do this (smth like if request.headers['Content-Length']:         p.headers['Content-Length'] = request.headers['Content-Length']) if header value is provided.

Thank You


You can change these values by using the [`PreparedRequest` flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests), which allow you to either skip the Session preparation step or to change the request after it has been prepared by the Session.

_However_, I strongly recommend that you do not change the Content-Length: Requests has almost certainly got it right, and if you change it to a different value you are likely to break the connection.