Raibek/requests-github-issues huggingface dataset:
Requests causing NoneType error

I've pip installed requests (requests-2.9.1-py2.py3-none-any).  I'm working on a script to download an excel file from my company's SharePoint onto my local drive using Python 2.7.  Even if I comment out every last bit of code other than "import requests" at the beginning, I get this error message:

_"AttributeError: 'NoneType' object has no attribute 'a2b_hex' "_

So something about importing the requests module is causing this problem. Here's the full traceback:

_Traceback (most recent call last): File "H:\PythonSandbox\GetSharePoint.py", line 1, in import requests

File "C:\Python27\ArcGIS10.1\lib\site-packages\requests__init__.py", line 58, in from . import utils

File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\utils.py", line 26, in

from .compat import parse_http_list as _parse_list_header

File "C:\Python27\ArcGIS10.1\lib\site-packages\requests\compat.py", line 29, in

import json

File "C:\Python27\ArcGIS10.1\Lib\json__init__.py", line 108, in from .decoder import JSONDecoder

File "C:\Python27\ArcGIS10.1\Lib\json\decoder.py", line 24, in NaN, PosInf, NegInf = _floatconstants()

File "C:\Python27\ArcGIS10.1\Lib\json\decoder.py", line 18, in _floatconstants _BYTES = '7FF80000000000007FF0000000000000'.decode('hex')

File "C:\Python27\ArcGIS10.1\Lib\encodings\hex_codec.py", line 42, in hex_decode

output = binascii.a2b_hex(input)

AttributeError: 'NoneType' object has no attribute 'a2b_hex'

Failed to execute (GetSharePoint)._

Apologies if this isn't worded well, I'm a relative novice and new to github.  Thanks!


This looks like an error in your system. For some reason, the hex_codec module's import of binascii is broken: something is patching it away or otherwise messing with it such that it doesn't not function correctly.

Taking you through debugging that is not really something that we can do here (this is a requests bug tracker, and the issue here is not with requests), but nonetheless, you should be investigating why binascii is not functioning well here. Given that this is in the standard library and that your code apparently _only_ imports requests, either ArcGIS have busted something or your system is pretty remarkably borked.

FWIW, you should be able to reproduce this error by using just the following script:

``` python
import json
```

That involves no third-party code, which should demonstrate that the problem is not in requests. =)
Consolidate logic for changing method during redirects

I only moved the code into a function, there was no actual change to the code. I added a few tests to ensure we're doing things correctly.

The real point of me doing this is to make it easier to bring back `strict_mode` functionality. For you requests youngsters in the crowd, `strict_mode` followed the spec for redirects meaning the method wouldn't change to a GET. The current code follows the browser convention of changing the method to a GET when doing a 302 redirect. However, lots of servers want you to follow the standards (the nerve!) so I'd like to override the logic. Now that the method changing logic is in `rebuild_method`, I can simply override that function instead of overriding the entire `resolve_redirects` function as suggested by kennethreitz/requests#1325


So, I'm totally happy with this code change: I think it's a reasonable refactoring that helps improve some clarity, _as well_ as allow @benweatherman to perform the override you want.

I'm also delighted to see some extra tests added for this. I think this change is wonderful! Pinging @kennethreitz or @sigmavirus24 for an extra round of review before merging.


:sparkles: :cake: :sparkles:


Yay! Thanks y'all for getting this merged quickly. Now for the release... :joy_cat:
Raise a ProxyError for proxy related connection issues

This one aims to address https://github.com/kennethreitz/requests/issues/3050


Cool, I'm happy with this. @sigmavirus24 `ProxyError` is a `ConnectionError` subclass, so I think this can go in the next minor release. Agreed?


I agree. Do we have things we've merged that are not suitable for a minor release though?


I don't think so.
Clarify that SSL verification is on by default

Generally if a kwarg is present it indicates that an option other than the
default is being specified. Putting `verify=True` in the first code sample
for SSL confused me, because it seemed to indicate that you had to specify
`verify=True` to get SSL verification. The opposite is true; SSL verification
is turned on by default and you have to specify `verify=False` to opt out of
SSL verification.

Updates the docs to make this more clear. Furthermore, connections to
https://kennethreitz.com currently time out instead of presenting an invalid
certificate, so I replaced this domain with https://requestb.in, which presents
the same error message as is currently there.


Thanks @kevinburke! :sparkles: :cake: :sparkles:


What about https://self-signed.badssl.com/? (I am not affiliated, just an occasional user)
The problem is, that is validates without SNI...
Import native Python JSON only

Pull Request derived from the discussion on issue #3052


LGTM.


:sparkles: :cake: :sparkles:


Really, no longer a need for `json` to be in `compat` at all anymore, let alone be called `complexjson`. 

But, that would require touching a lot more code. 


@kennethreitz I actually was going to rename complexjson to json but there are json variable references all over the models.py file and decided it was probably best not to refactor it. If you still think it's a good idea I'm game to change it.


Perhaps there a different name we could use for the import so it would require less touching of code. `jsonp`?


Ah, prob best to `import json as complexjson` in `models.py` then (instead of `compat.py`).

It's a clever name, and I enjoy cleverness. 


Ha! Agreed, complexjson is a good alt name. Best to either change it to json or leave it as is.


PR looks perfect. 


The reason we need `complexjson` in `models.py` is because we accept `json` as a method/function parameter and for those cases we can't also reference `json` the module without a name collision. Leaving it as such makes life a bit simpler.

That said, I'm open to changing it to one of the following
- `loljson`
- `whyjson`
- `hailjson`
- `zomgjson`
- `whatevenisjson`

And probably quite a few other names. ;)


- `mwson` (micro web services object notation)


- `perhapsjson`
- `itcouldbejson`
- `pleasedontsayjason`


No, it's fine as is. 


I know. I'm just kidding about alternate names. Alternatively
- `mycathatesjson`
- `oneweirdtrickforserializinganddeserializingjson`


- `magicjson`
- `sacredjson`
- `chopjsoncarrywater`
- `cosmicjson`


I got it ...
- `trumpjson`

You guys are gonna delete the PR now aren't you :(


TrumpJSON™: It's unbelievable, really, the best JSON you'll ever experience. Trust me, I know JSON. 


It overwrites all your code with this way better code, trust me it's better....


It's a huuuuuuuuuuuuuuuuuuge download. It makes serialization great again


anyway this can get in a 2.x release? It's been two years now and `simplejson` is even less relevant than when this was authored
Set a default timeout on a Session instance

It would be useful to be able to set a default timeout on a Session instance, which would get used for any requests initiated from that Session in lieu of an explicitly supplied timeout.


Please search the issue repository before you open new issues. If you had done so you'd have seen #2856 and #2011 which both cover this issue.


In the future please search the closed issues for similar ideas. This is one we have rejected numerous times. Further it is trivial to subclass the session class to provide this.
remove simplejson import, native python json import only

Pull Request derived from the discussion on issue https://github.com/kennethreitz/requests/issues/3052


This seems reasonable to me, and it's certainly the smallest diff that could achieve the job. I do wonder if we shouldn't just remove this from `compat` now though, and instead change the imports in the modules that use it.


@Lukasa That's a good point. There is actually only file that is importing json, [models.py](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L34) So the diff would be about the same. Updated the PR


This should target proposed/3.0.0 not master.
Fix 'Transfer-Encoding: chunked' bug when upload an empty file

Though we have the `stdout` (which is also a `fileno` type with length=0) to care about, when we post or put a normal empty file, we still can not force to set the `Transfer-Encoding` to `chunked`.

I think it is clearly meaning that "I don't want to transfer the file in chunked type", when someone set the `Content-length=0` in headers. `requests` does too much in this case, and it will bring out some problem when requesting some servers which not support chunked type. The `Transfer-Encoding` parameter should be user's response to add into the headers.


I am :-1: on this change.

The reality is that I don't care about servers that don't support chunked transfer encoding. [RFC 7230 Section 4.1](https://tools.ietf.org/html/rfc7230#section-4.1) says:

> A recipient MUST be able to parse and decode the chunked transfer coding.

And this is hardly a new requirement. RFC 2616, the original HTTP/1.1 specification that was published _nearly 15 years ago_ (in 1999), says in [Section 3.6.1](https://tools.ietf.org/html/rfc7230#section-4.1) that:

> All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding

This means that any server that cannot handle chunked transfer encoding is either a) a HTTP/1.0-only server, which is incompatible with requests (a HTTP/1.1 library); or b) deliberately non-compliant with the now _fifteen year old_ HTTP/1.1 specification. Requests has _no duty_ to bow to the whims of server developers that chose only to support more than 15 year old protocols or to ignore specifications when they do so, or to operators of websites that are stuck in the 1990s.

And even if that were not true, your reasoning about chunked transfer encoding:

> The Transfer-Encoding parameter should be user's response to add into the headers.

is not now and has never been how requests operates. The requests team has quite consistently told users not to add their own content-length or transfer-encoding headers to the headers dictionary, because requests will ignore them.

The upshot is that sending an empty chunked body is far and away the nicest way to deal with this situation.
Added unit tests for hooks module


This also seems reasonable to me, but @Stranger6667 is sat right next to me. ;) @sigmavirus24/@kennethreitz, mind doing an extra review for me? This is +1 from me.


Hello folks seems somehow I got placed on this email alias By mistake.
Anyway I can be removed.

Thank you
-Ryan
On Sun, Mar 13, 2016 at 7:24 AM Cory Benfield notifications@github.com
wrote:

> This also seems reasonable to me, but @Stranger6667
> https://github.com/Stranger6667 is sat right next to me. ;)
> @sigmavirus24/@kennethreitz https://github.com/kennethreitz, mind doing
> an extra review for me? This is +1 from me.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/3049#issuecomment-195938314
> .


@ryandebruyn This is managed by GitHub, it's not an email alias. You'll need to take some action to remove yourself via GitHub.


@ryandebruyn you may be "Watching" this repository in which case only you can fix that by visiting the project page while logged in and "un-watching" it.


Hello, @sigmavirus24 @Lukasa !
I've updated this PR :)


I have one small note. =)


@Stranger6667 I don't like having tons of pull requests open, so I'm merging this. If you wouldn't mind, it would be appreciated if you opened another PR that addresses the notes @Lukasa left on your test.
Added unit tests for structures module


@sigmavirus24 @kennethreitz Can one of you two do a separate code review of this? @Stranger6667 is here with me at PyCon SK writing some of these tests. I'm happy with these, but it'd be good if one of you two gave the ok/not-ok.


Hello @Lukasa @sigmavirus24 !
I've updated this PR :)


Awesome, we're very close now @Stranger6667. One small note. =)


Looks pretty solid!
python 3.5 requests 2.9.1 can not upload Chinese filename

source code

`import requests

headers = {
    'Origin': 'http://home.ctfile.com',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36',
    'Accept': '_/_',
    'Referer': 'http://home.ctfile.com/',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'zh-CN,zh;q=0.8',
}

files = {'file': open('中文.txt', 'rb')}
values = {'name': r'中文.txt', 'filesize': '101'}

url = 'xxxx'
r = requests.post(url, files=files, data=values, headers=headers)

print(r.content)`

I use fiddler to capture browser data.

The right data is:
Content-Disposition: form-data; name="file"; filename="中文.txt"

The python script data is 
Content-Disposition: form-data; name="file"; filename*=中文.txt

so I change 

Python35\Lib\site-packages\requests\packages\urllib3\fields.py

line 46-47 to

`# value = email.utils.encode_rfc2231(value, 'utf-8')
    value = '%s="%s"' % (name, value)`

It is OK now.


> The right data is:
> Content-Disposition: form-data; name="file"; filename="中文.txt"

That is not the right data. This is:

```
Content-Disposition: form-data; name="file"; filename*=utf-8''%E4%B8%AD%E6%96%87.txt
```

This uses RFC 2231 already. To achieve it, change your code to this:

``` python
files = {'file': open(u'中文.txt', 'rb')}
values = {'name': r'中文.txt', 'filesize': '101'}

url = 'xxxx'
r = requests.post(url, files=files, data=values, headers=headers)
```


The fiddler right data screenshot

![img1](https://cloud.githubusercontent.com/assets/8475089/13702662/36592330-e7cb-11e5-8d99-7ddb6d0739e8.jpg)

![img2](https://cloud.githubusercontent.com/assets/8475089/13702668/3ed008f8-e7cb-11e5-8fad-80bd2bbf3ad6.jpg)

not like
 filename*=utf-8''%E4%B8%AD%E6%96%87.txt


@pc10201 Did you change your code to match mine, _including_ the unicode string for opening the file?


I will try tomorrow.I will tell you result.


I tried your code.
It do not work
![img3](https://cloud.githubusercontent.com/assets/8475089/13703163/d4c33026-e7ce-11e5-9db1-efa3ec9ece0c.jpg)
![img4](https://cloud.githubusercontent.com/assets/8475089/13703164/d51a0ebe-e7ce-11e5-8c39-74adbd66d6b7.jpg)

If you use windows,you can use fiddler to capture http data.
if you use linux or mac os,you can use wireshark.


@pc10201 In what sense is that not working? I see the filename field formatted correctly.


The server response is not right.
The right data

![img1](https://cloud.githubusercontent.com/assets/8475089/13703995/74192cf2-e7d4-11e5-88a0-e9bebf7755b4.jpg)

The bad data
![img3](https://cloud.githubusercontent.com/assets/8475089/13704002/7b5e663a-e7d4-11e5-9d25-a4fb06d627e2.jpg)

Please check the third line with each image.


Assuming for a moment that the different filenames are unrelated to this problem, the issue here is that the "right" case is not using RFC 2231 encoding. This is common, but _wrong_: the server needs to be able to handle the RFC 2231-encoded filename, which is what we're providing.


You are right.The server may cause this problem.Please close this issue.
Suggest to user to install 'certifi' Python module if CA bundle is not found and 'certifi' is not loaded

Requests grabs its CA bundle by either using the 'certifi' library, or if it's not installed, looking at the hard-coded filename 'cacert.pem'. The problem is that if you're on a Linux distro (ex: OpenSUSE) that doesn't have 'certifi' installed by default and doesn't have the CA bundle named 'cacert.pem', all you get is an unhelpful error telling you it could not find a suitable SSL CA certificate bundle.

What I'm suggesting is that, if the filename look fails and 'certifi' is not loaded, Requests adds a message in the "could not find a suitable SSL CA certificate bundle" error message that suggests that the user install 'certifi' to resolve the problem. It would save a lot of headache for people who aren't savvy enough to search the source code and figure out what they need to install in the first place.


@aahancoc Requests ships the `cacert.pem` bundle by default, which means that if you're on a Linux distribution and it cannot be found your distribution removed that file themselves. This means your distribution broke your install.

I recommend you take it up with the people breaking our software downstream. =)


We may not want to encourage the distros to modify thecodebase even further.

Ugh.
Refactoring and Improving Readability

Since `and` condition is there, `verify is not True` will be always false.


Hey @tusharmakkar08,

Thanks for sending this pull request. Could you explain what bug you were seeing that made you change this? What instigated this change? It's not clear that this exhibits the same behaviour it did before, especially since tests have failed pretty consistently across all versions of Python.


@sigmavirus24 : In line 174 in `adapters.py` `if url.lower().startswith('https') and verify`  is there and then `if not verify` is there. Since `and` condition is there, `if not verify` won't be `True`. The other thing is more pythonic way of if/else i.e in line 200 


@tusharmakkar08 your commit no longer affects that region and simply turns an easy to read if/else statement into a ternary which is not "pythonic" nor is it easier to read.

I'll ask again: What bug are you seeing that prompted this (if there was any bug)?


There was no bug per se. I was trying to remove lines for increase in readability.


Fewer lines does not lead to an increase in readability. In this instance, the ternary conditional is particularly unclear when combined with tuple unpacking assignment as well as creating an intermediate tuple in many cases. 

In this instance, I suggest that the only clear readability improvement would come from using tuple unpacking in the first branch of the conditional. I don't think any change beyond that is warranted. 


We're not interested in random refactorings that change logic for no apparent reason. Sorry @tusharmakkar08 


@Lukasa : I have done couple of more changes and few of them are [anti-patterns](https://www.quantifiedcode.com/app/issue_class/62b0f5b7b69e4a2498568b32bfa30991) as suggested by http://docs.quantifiedcode.com/python-code-patterns/ . This isn't random refactoring, we are removing anti-patterns from code @sigmavirus24. 

Thanks. 


@tusharmakkar08 while your contributions are appreciated, I want to give you feedback — all of the changes present in this pull request actually _reduce_ readability, significantly.


@sigmavirus24 @Lukasa @kennethreitz: Since requests is being mentioned over [here](http://docs.python-guide.org/en/latest/writing/reading/) among the best python codes, I believe these kind of antipatterns shouldn't exist over here. 


@tusharmakkar08 please keep in mind that your continuous arguments are emailing roughly 841 people. Since it doesn't seem as if you will respectfully let this conversation end and it serves to benefit none of those 841 people (because it has little value to the project as a whole and is likely _not_ why they are subscribed) I am ending the conversation forcefully by locking it.


Antipatterns are an antipattern. :)
got response with 'Connection: close', instead of closing connection but still send message body

When I post a big file to a server with invalid authorization, the server response with a `400` error response and with `"Connection: close"` header before read all the request body  and then close the connection, requests still send message body and get `connection aborted` exception, but according to [RFC7230](https://tools.ietf.org/html/rfc7230#section-6.5), the client `SHOULD` monitor the network connection for an error response and immediately cease transmitting the body and close its side of the connection.


@zjulmh This is a known problem. Unfortunately, the underlying HTTP library we're using makes it essentially impossible to actually do this: we only find out about the error response when the connection gets closed.

We'd need to replace httplib to avoid this problem, which is a substantial amount of work.
Proxy settings not working

HI, using urllib2, I can connect through a proxy without problems. But, using requests it's not working:

http://stackoverflow.com/questions/35851227/python-requests-behind-proxy

Can anyone help?


Don't use `https` schemes. Your proxy dict should be:

``` python
proxy_dict = {
    'http': 'http://10.20.23.5:8080',
    'https': 'http://10.20.23.5:8080',
}
```


It is not working...

Why urllib2 is doing it correctly and requests is unable to handle this?


I notice in your urllib2 code you aren't actually providing a list of proxies. That means it is reading the proxy information from the environment. Requests does that automatically, with no need to provide the proxies dictionary. Try removing it entirely and see if requests sorts the problem out for you.


Same result with status code 407.

page = requests.get('http://www.google.com')

This returns:
'407 Proxy Authentication Required' 


@janscas What operating system are you using?


It's windows 8.1.
I'm behind a corporate proxy (Isa Server Forefront TMG)

It seams the urllib2 can take the proxy settings from the envirenment.

After doing this: 

proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

urllib2.getproxies() returns a dictionary with the correct proxies.


What's the result of `urllib.getproxies()`, please?


it s python dict containing the proxies used by the system:

proxy_dict = urllib2.getproxies()
print proxy_dict
{'ftp': 'ftp://10.20.23.5:8080', 'http': 'http://10.20.23.5:8080', 'https': 'https://10.20.23.5:8080'}


Ok, can you print the headers from the 407 please?


Content-Length: 4128
Via: 1.1 SRVMADISA08
Proxy-Connection: Keep-Alive
Proxy-Authenticate: Negotiate, Kerberos, NTLM
Connection: Keep-Alive
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html

Ok, so the problem here is that Requests does not out of the box support any of those authorization schemes for proxies. [Requests-NTLM](https://github.com/requests/requests-ntlm) does provide support for authenticating to a proxy, but it will require that you pass your domain credentials to requests NTLM.


But... why urllib2 does this out of the box?


@janscas I have no idea. If you want to try to dive deeper into what urllib2 is doing that we're not, you could use Wireshark to check the differences between the request that requests sends and the one that urllib2 sends.


ok, I'll check this and report back.

Thanks a lot
using ipv6 with requests 

Hi I read previous issue on using requests on ipv6, but I don't quite understand how to update the HOST header, could someone point out? (suppose I already have ipv6 address of a website, how do request it?)
Or is there any better solution available now? 

Thank you! 


It's not clear what you mean when you say "the previous issue", but I assume it is #3002, which is a bug. You can work around it by doing `requests.get('https://[fe80::00]', headers={'Host': 'fe80::00'})`. Alternatively, wait for the next release, which should contain a fix.
Sending piped stdin results in OSError: [Errno 29] Illegal seek

I'm working on adding support for streamed uploads via `stdin` to HTTPie.  It works fine when `stdin` redirected from a file, but fails with `OSError: [Errno 29] Illegal seek` when piping programs together.

**`test.py`:**

``` python
#!/usr/bin/env python
import sys
import requests

r = requests.post(
    'http://httpbin.org/post', 
    data=getattr(sys.stdin, 'buffer', sys.stdin)
)
print(r.text)
```

:smile: **works:** 

``` bash
$ python test.py < test.py
```

:disappointed: **fails:**

```
$ cat test.py | python test.py
```

``` python
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    r = requests.post('http://httpbin.org/post', data=sys.stdin.buffer)
  File "python3.5/site-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "python3.5/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "python3.5/site-packages/requests/sessions.py", line 454, in request
    prep = self.prepare_request(req)
  File "python3.5/site-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "python3.5/site-packages/requests/models.py", line 296, in prepare
    self.prepare_body(data, files, json)
  File "python3.5/site-packages/requests/models.py", line 430, in prepare_body
    length = super_len(data)
  File "python3.5/site-packages/requests/utils.py", line 86, in super_len
    current_position = o.tell()
OSError: [Errno 29] Illegal seek
```

---

Tested on OS X 10.11.3 + Python 3.5.1 & 2.7.11 with `requests==2.9.1`.

---

I imagine checking for that error in the appropriate [`except` clause in model.py:431 ](https://github.com/kennethreitz/requests/blob/4f378b0e1a2f5f60e2e57daaec80081483f7150e/requests/models.py#L431) would do the trick. Or a more robust `super_len()`. I'll be happy to submit a pull request, if interested.


@jkbrzt I think you're probably right. I'll have a little play today and see if I can convince myself you're definitely right, and then submit a patch.
Support non-ASCII Reason Phrase as per RFC2616

RFC2616 specifies that header values and reason phrases are encoded in in ISO-8859-1 or RFC2047 (MIME for text). RFC7230, which obsoletes RC2616, recommends treating non-ASCII header values as 'opaque data'.

In the case of the Reason Phrase, however, we should try to decode it as described due to its possible/encouraged localization and due to its purpose for display to the user; treating it as 'opaque data' is not useful and prevents its intended use.

Note that several production servers, including Apache Tomcat, send responses conforming to RFC2616. Servers that respond only in US-ASCII are unaffected by this change. This _only_ regresses (by introducing possible encoding errors) for servers that send responses conforming to neither RFC, which I believe are either uncommon or non-existent.

Fixes kennethreitz/requests#3009


I'm sorry, but this reasoning is still not accurate, and so this code is still not right.

Firstly, RFC 2616 is not relevant anymore except inasmuch as it provides guidance to what older implementations may do.

Secondly, this argument is wrong:

> In the case of the Reason Phrase, however, we should try to decode it as described due to its possible localization and its purpose for display to the user; treating it as 'opaque data' is not useful and prevents its intended use.

RFC 7230 is very clear on this:

> The reason-phrase element exists for the sole purpose of providing a textual description associated with the numeric status code, mostly out of deference to earlier Internet application protocols that were more frequently used with interactive text clients.  A client SHOULD ignore the reason-phrase content.

That is to say, we are strongly advised by RFC 7230 to ignore the reason phrase _entirely_ rather than to do any special handling of it.

Now, I do believe that we should aim to fix the original problem, but the original problem is _entirely_ that the default encode/decode cycle of Python was busted. For that reason, we should restrict the scope of our fix to the `raise_for_status` method (any anywhere else where we try to do computation on the reason phrase), at which point we should use `requests.utils.to_native_string` with the `ISO-8859-1` encoding. That does the minimal translation to avoid anything exploding when doing the string interpolation, but otherwise leaves the reason phrase unmolested.


@erydo expressed on the original issue that they were going hands off. That implies to me that they're done working on this if we're not going to accept their changes. If I'm wrong, I'll be happy to reopen this if they wish to follow our recommendations for the appropriate fix. Otherwise, this will sit open for a while and I'd rather that not happen.
Changes for #3028

I've overwritten the `__contains__` method of `RequestsCookieJar` as discussed in the issue. It looks to be working as expected and tests pass, but this is my first contribution to this project so I could be missing something.


Thanks for this @davidsoncasey, this pretty much looks right! I left a few small notes inline for you to take a look at, but one those are addressed we can merge this!


@Lukasa great, thanks for taking a look! I'll have a chance later today to make those edits.


@Lukasa I added a couple fixes, take a look and let me know if there's anything else!


\o/ Thanks so much @davidsoncasey! :sparkles: :cake: :sparkles:
Strange behaviour in class dict

Apparently the class does not like hyphens or percent sign.

cookies = dict(hottest-count='working')
SyntaxError: keyword can't be an expression

Same problem with cookies like %FOO%='bar'.


@coveritytest That has nothing to do with requests, and everything to do with Python: as the error reports to you, your _syntax_ is wrong.

Try: `cookies = {"hottest-count": "working"}`.
"in" operation on resp.cookies should not raise CookieConflictError

Hosts which have the same cookie define for multiple domains should not raise 
"CookieConflictError: There are multiple cookies with name,"  when performing "in" 
 It should short cut and return True immediately when found.


So the problem here is that the default implementation of `collections.Mapping.__contains__` calls in to `__getitem__`, which for the `RequestsCookieJar` requires that there be no duplicate. In this instance, we probably get the most bang for our buck by just overriding `__contains__` to call the parent implementation, and then catch `CookieConflictError` and return `True` in that instance.

I'm marking this as contributor friendly because the fix is fairly easy, and it'd be a good one for someone to pick up as a first open source change.


I've been using requests for a while and thinking I'd contribute, so I might take a shot at this.


@davidsoncasey Awesome!
requests can't make a request to the server it's running on

Here's a test case that never returns a response. Shouldn't this work?

``` python
import requests
from flask import Flask
app = Flask(__name__)

@app.route('/')
def Welcome():
    return "Testing requests module's ability to call the service in which it's running"

@app.route('/test')
def TestRequest(): 
    resp = requests.get('http://0.0.0.0:5004')
    print('Response from TestRequest: '+resp.text)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5004, debug=True)
```


Flask's debug server cannot handle multiple requests at once. Run Flask behind a proper WSGI server and you'll be fine. =)


Thanks for the tip @Lukasa. I'm a newbie when it comes to Python web serving. Been struggling getting Apache WSGI module + Flask on OS X el capitan all day. First it was SIP (system integrity protection) and now it's getting Apache/OS X python to recognize my anaconda python modules. This whole process is extremely unpleasant. Do you happen to know what "proper WSGI server" folks tend to use?


@rajrsingh there are too many to list, but here are a few:
- Apache+mod_wsgi
- gunicorn
- uwsgi
- twisted


Use gunicorn. :)


Thanks for the info. I'll put some of my research here for future Googlers. 
1. Getting mod_wsgi installed on OS X El Capitan with SIP is annoying. This is the best resource on that: https://github.com/GrahamDumpleton/mod_wsgi/issues/98. I ended up using the technique of disabling SIP, then installing mod_wsgi. 
2. Then my problem was either getting Python modules (like pandas and Flask) installed in the system Python, which Apache insists on using, or figuring out how to get Apache to recognize my Anaconda Python install. I gave up at this stage because...
3. I found a way to get Flask to be multi-threaded here: http://stackoverflow.com/questions/14672753/handling-multiple-requests-in-flask so I can continue my dev work on the laptop with no "real" WSGI server.
4. Next step is to try out gunicorn and get it all running on my eventual production environment on http://bluemix.net. 


So the strict answer to the question is nothing is wrong with requests, I just needed to add the `Threaded=true` parameter to the test server.

``` python
import requests
from flask import Flask
app = Flask(__name__)

@app.route('/')
def Welcome():
    return "Testing requests module's ability to call the service in which it's running"

@app.route('/test')
def TestRequest(): 
    resp = requests.get('http://0.0.0.0:5004')
    print('Response from TestRequest: '+resp.text)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5004, Threaded=True, debug=True)
```
option to enable TCP level keepalive

When there is firewall between client and server, http session will be killed even with keepalive enabled because no TCP level keepalive probe is sent. 

according to [here](http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html)

> Remember that keepalive support, even if configured in the kernel, is not the default behavior in Linux. Programs must request keepalive control for their sockets using the setsockopt interface. There are relatively few programs implementing keepalive

that said, is it possible to enable this behavior as an option in the library? 


It is already possible: the requests tool belt provides a [Socket Options transport adapter](http://toolbelt.readthedocs.org/en/latest/adapters.html#socketoptionsadapter) that allows you to turn on TCP_KEEPALIVE.


There's also a separate keep alive adapter which subclasses the socket options adapter. That will ideally make things simpler for your usage


If we're going to be responding to tickets like this, then we need to make Requests-Toolbelt more prominent in the documentation.
Proxies not working

My script shows that proxies aren't working. Am I doing something wrong? 
Here's the code if you need:
[Script.zip](https://github.com/kennethreitz/requests/files/149201/Script.zip)


Code here:

``` python
# Important stuff here
from json import load
import requests
from random import choice, randint, shuffle
from urllib2 import urlopen

url = 'http://epiccodes.com' 

# Random proxies from - http://proxygaz.com
proxies = {'https': 'https:131.96.228.236:9064',
           'https': 'https:54.85.61.208:80',
           'https': 'https:54.209.36.19:80',}

# Retrieves IP Address
def IPAddress():
    IP = load(urlopen('http://httpbin.org/ip'))['origin']
    return(IP)

# Retrieves Cookie
def CookieIP():
    r = requests.get(url,
                     # Using proxy
                     proxies=proxies)
    try:
        Cookie = r.headers['Set-Cookie']
        return(Cookie)
    except KeyError as e:
        return("Website might be down")

# This prints your IP
print("IP Address: " + IPAddress())

# The website's cookie has your IP in it. If proxy works, then the IP on the
# cookie should be different from the other IP.
print("\nProxy's IP: " + CookieIP())

```


A few things:
1. If you look at [the proxy documentation](http://docs.python-requests.org/en/master/user/advanced/?highlight=proxies#proxies) you'll see that your proxy dictionary should look like this:
   
   ``` py
   proxies = {'https': 'https://131.96.228.236:9064'}
   ```
   
   Note the `//`.
2. You haven't described what kind of proxies these are. (Hint: If they're SOCKS proxies, those will not work with requests... yet.)
3. You haven't described what you're seeing versus what you're expecting.
4. Questions belong on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests). Please ask future questions there.


1. If you look at my script (the one @kennethreitz  posted), that's what I did. 
   proxies = {'https': 'https:131.96.228.236:9064',
          'https': 'https:54.85.61.208:80',
          'https': 'https:54.209.36.19:80',}
2. The proxies I tried to use are HTTPS
3. If you look at the cookies for http://epiccodes.com, you'll see your IP in the cookies, I was expecting to see the proxy's IP in the cookie when I use proxies=proxies in the get request, instead of my actual IP.
4. My bad, I didn't know. Will do in the future if I have any questions.


> If you look at my script (the one @kennethreitz posted), that's what I did. 

It isn't. Please look closely at what you wrote and what I wrote. Further, which proxy do you expect to be used? Creating a dictionary with a repeated key is undefined behaviour (as to which value is kept) in Python.


@Pr0xy671 you are specifying 3 proxies. Try only doing one. Also, add a `//` to your proxy URL.

p.s. you can use `http://httpbin.org/ip` to get ip address easily. 


Ok, I tried doing this way and still nothing.

r = requests.get('http://httpbin.org/ip', proxies={'https': 'https://89.163.131.168:3128'})
print(r.json()['origin'])

It then prints out my real IP and not 89.163.131.168


@Pr0xy671 it's because you specified a proxy only for HTTPS (`{'https': …}`) but requested a HTTP page.
Added unit tests for utils module

More unit tests for `utils` module. Also there is small cleanup for this module - removed unused imports & variable.
Lines coverage increased from 46% to 75%, but I'd like to ask about some functions, which are not used in `requests` at all (e.g `dict_to_sequence`, `from_key_val_list`, etc). Is there some plan for them? I wanted to write unit tests for it, but I'm not sure, may be they are staged to be deleted and it is not reasonable to cover them with unit tests.

BR


Some of the utils module functions are left in place to avoid breaking users that may be using them directly, even though they're not used in requests any more. There's certainly no plan to remove them prior to 3.0.0.


I'm down for removing ones that _don't belong there_ for 3.0.
Add exceptions into __init__.py

Addresses Issue #3022.

Also added the JetBrains IDE settings folder, `.idea/`, to `.gitignore`.


I'm going to leave this up to @kennethreitz. As you stated in #3022, this is totally unnecessary, but neither is it a maintenance burden, so I have no objection to it. Thanks for your contribution! :cake:


We already have all the exceptions available in the public API:

``` python
from .exceptions import (
    RequestException, Timeout, URLRequired,
    TooManyRedirects, HTTPError, ConnectionError,
    FileModeWarning,
)
```


I'm with @kennethreitz this isn't necessary. Because of the line @kennethreitz is citing, users can already just do

``` py
import requests

try:
    r = requests.get(url)
except requests.ConnectionError:
    pass

# or

try:
   requests.get(url)
except requests.exceptions.ConnectionError:
   pass
```

Both work perfectly fine. Beyond that, we don't need a gitignore line for PyCharm/idea projects because that should already be in your [global gitignore](https://help.github.com/articles/ignoring-files/#create-a-global-gitignore).


@kennethreitz Thanks, I was wondering about that line. Two questions:
1. That import makes them available as `requests.RequestException`. It's a subtle difference, but [this doc page](http://docs.python-requests.org/en/master/api/#exceptions) instructs users to include them as `requests.exceptions.RequestException`. Which one is the preferred or supported method?
   - If we want to include both, we could specify both in `__init__.py`.
   - If `requests.RequestException` is preferred (which I like), perhaps we should update the docs.
2. It looks like `ConnectTimeout` is missing from the list. Should it be added?


@sigmavirus24 Thanks for the global gitignore tip! I hadn't heard about that.


@jtpereyda it's a subtle but unimportant difference. Try this in an interactive session:

``` py
>>> import requests
>>> try:
...     requests.get('http://localhost:8080')
... except requests.ConnectionError:
...     pass
...
>>> try:
...     requests.get('http://localhost:8080')
... except requests.exceptions.ConnectionError:
...     pass
...
>>>
```

Both already work perfectly fine. You can use either one or both interchangeably. If `ConnectTimeout` is missing, that should be added to the list of exceptions imported. But like I said, as it is now, we do not need to change anything for `requests.exceptions.RequestException` to work. It will simply work.


@jtpereyda the "recommended" way is, in fact, `requests.RequestException`, as `requests.exceptions` is really just an internal namespacing.

That section of the docs are auto-generated from the codebase. I'd love to fix that section if we can. 

However, at this time, there's nothing broken about the docs or the code. We're just discussing room for possible usability improvement (which is the main goal of this library). 


@jtpereyda fixed the docs! http://docs.python-requests.org/en/master/api/#exceptions


@kennethreitz Thanks!
exceptions missing from __init__.py

It is traditional to include the public API in `__init__.py`.

Consider this test code:

``` python
import requests
import requests.exceptions

a = requests.exceptions.ConnectTimeout
```

This will run properly, but PyCharm raises a warning, saying, "Cannot find reference 'exceptions' in `__init__.py`".

This could be fixed by changing an `__init__.py` line from:

```
from . import utils
```

to

```
from . import requests, utils
```

While this is technically a [bug ](https://youtrack.jetbrains.com/issue/PY-18550) in PyCharm, requests' **init**.py already imports utils, apparently for this very purpose. So it would be consistent to import exceptions as well.


This isn't _technically_ a bug in PyCharm. It is a bug that they should be fixing. If we did something like this for each editor that does nonsensical things like this, I'm not sure we'd have a very coherent project.


@kennethreitz Resolved this by changing the docs (see PR #3023).

Looks like http://docs.python-requests.org/en/master/api/#exceptions is still missing ConnectTimeout and ReadTimeout. Should they be added to `__init__.py`?


@jtpereyda oddly, those two exceptions are being _raised_ during the sphinx build. I haven't dug into it yet. 

But.... anyone can investigate! ;)


Is this still a reproducible bug? Both (python2 and python3) sphinx builds are clean:

```
(p2_venv) requests $ make docs
cd docs && make html
make[1]: Entering directory '/home/kmaduri/Documents/programming/python/requests/docs'
sphinx-build -b html -d _build/doctrees   . _build/html
Running Sphinx v1.3.5
loading pickled environment... failed: unsupported pickle protocol: 4
loading intersphinx inventory from http://urllib3.readthedocs.org/en/latest/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 18 source files that are out of date
updating environment: 18 added, 0 changed, 0 removed
reading sources... [100%] user/quickstart                                     
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] user/quickstart                                      
generating indices... genindex py-modindex
highlighting module code... [100%] requests.auth                              
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in _build/html.
make[1]: Leaving directory '/home/kmaduri/Documents/programming/python/requests/docs'
\033[95m\n\nBuild successful! View the docs homepage at docs/_build/html/index.html.\n\033[0m
```


This may have been fixed now; there was a change made around this recently.
Fix #3017: Whitespace characters surrounding a URL should be ignored

My attempt to fix [the issue I reported](https://github.com/kennethreitz/requests/issues/3017). Feel free to comment my changes, especially if you don't agree with them.


@geckon thank you!

For the record, this is a potentially breaking change for _someone_, hence why it has to wait for inclusion in the 3.x series. 


:sparkles: :cake: :sparkles:


Thanks for this @geckon
Fix #3017: Whitepace characters surrounding a URL should be ignored

My attempt to fix [the issue I reported](https://github.com/kennethreitz/requests/issues/3017). Feel free to comment my changes, especially if you don't agree with them.


@geckon please rebase this against that branch instead. If you need help, let me know and I can give you some instructions that should work in this case. Thanks for handling this! :cake: 


Hopefully I did it right the third time: https://github.com/kennethreitz/requests/pull/3021
I believe this PR can be closed, right?
Fix #3017: Whitepace characters surrounding a URL should be ignored

My attempt to fix [the issue I reported](https://github.com/kennethreitz/requests/issues/3017). Feel free to comment my changes, especially if you don't agree with them.


Looks perfect to me!


@geckon although, we're going to be merging this into the `proposed/3.0.0` branch. would you like to make your PR against that branch instead?


No problem: https://github.com/kennethreitz/requests/pull/3020
Is that alright?


Hopefully I did it right the third time: https://github.com/kennethreitz/requests/pull/3021
I believe this PR can be closed, right?
"Request.hooks" and "Session.hooks" merged inappropriately.

Both `Session` and `Request` have attribute `hooks`. The hook function list execute sequentially. In this situation, `Session.hooks` executed for every request and `Request.hooks` executed for a specifical request.

But in fact, `Session.hooks` were replaced with `Request.hooks` when code ran into
`merge_setting`:

``` Python
    merged_setting = dict_class(to_key_val_list(session_setting))
    merged_setting.update(to_key_val_list(request_setting))
```


@evilerliang Do the requests docs guarantee that `Session.hooks` will not be replaced by `Request.hooks`? In general, things declared for specific requests _override_ the things declared on the `Session`, rather than _supplementing_ them.


I reviewed the doc, thk.

> Any dictionaries that you pass to a request method will be merged with the session-level values that are set. The method-level parameters override session parameters.
> 
> Note, however, that method-level parameters will not be persisted across requests, even if using a session. 


Hmmm, we should consider appending any additional hooks provided by a method-level request, instead of overwriting. 

_e.g._

```
# session has a 'response' hook defined; additionally provided hooks get *merged*.
session.get(..., hooks={'respose': ...}

# session has a 'response' hook defined; all 'response' hooks removed.
session.get(..., hooks={'respose': None}
```

This is tricky, because in this scenario there is no way to specify nullifying the session hooks while simultaneously providing your own for the request. I believe this must be why I implemented it the way it is today, and justifies the current behavior as _correct_. 

---

**Random hooks thoughts:** Not sure if it's worth changing any hooks behavior at this point, since hooks were at one point in time much more powerful (since there were more of them). Now they're really just used for internal stuff, e.g. custom authentication providers. 

I don't actually perceive hooks, in their current state, being used much by users, and as such could potentially be removed from the documentation, to further their internal-ness. Or, possibly, they could be moved to a new section of the documentation for people extending requests.

Just random thoughts. 


That's fine. I changed the hook behavior in my package. Thanks again, you help me a lot.