Raibek/requests-github-issues huggingface dataset:
utils.select_proxy raises an exception when URL has no hostname

URLs sometimes have no hostname (e.g. file URLs, or unix socket URLs). The parsed URL `hostname` attribute returns `None` in this case, making `utils.select_proxy` complain about concat'ing a `str` with `None`.


@chipaca keep in mind that whatever adapter you're using will have to deal with older versions of requests and if it's not meant to touch the network should not be inheriting from the HTTPAdapter.


@sigmavirus24 I'm using https://github.com/msabramo/requests-unixsocket/ which does inherit from `HTTPAdapter`. I'm not sure what you mean, though; where does "touching the network" become a factor?


If you're using a `file://` URI, you're never going to talk to something over the network using HTTP (for example). I'll talk to @msabramo about fixing unixsocket as this problem will persist otherwise for several people who will not get this fix.


I'm curious how you saw this exception while using that library. Looking at unixsocket's code, it overrides the method which is the one place where [`select_proxy` is used in requests](https://github.com/kennethreitz/requests/blob/60aee145b96a4cf13a7daefd8578d538fe8459a8/requests/adapters.py#L250).


it's used in one more place: `HTTPAdapter.request_url`.

On Tue, Apr 26, 2016 at 3:56 PM, Ian Cordasco notifications@github.com
wrote:

> I'm curious how you saw this exception there. Looking at unixsocket's
> code, it overrides the method which is the one place where select_proxy
> is used in requests
> https://github.com/kennethreitz/requests/blob/60aee145b96a4cf13a7daefd8578d538fe8459a8/requests/adapters.py#L250
> .
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/3131#issuecomment-214772376

## 

John Lenton (jlenton@gmail.com) ::: http://chipaca.com


As to where I'm using it, I'm using it to connect to a unix socket :-)
I packaged httpie as a snap (originally for snappy ubuntu core systems, now on ubuntu classic itself), and added support for unix sockets so that it could talk to the snappy daemon. That's on https://github.com/chipaca/httpie-snap fwiw.


Ah so @msabramo should be overriding that too if he continues to subclass the adapter. (I missed the usage in the `request_url` method because GitHub search hasn't indexed that yet apparently.)


I didn't know about requests-unixsocket! We must catalogue these extensions somewhere. 


Hmmm, but don't unixsocket URLs always have a "hostname" Because the "hostname" is a URL-encoded path to the socket on the filesystem. E.g.:

```
[38] > /Users/marca/dev/git-repos/requests/requests/utils.py(592)select_proxy()
-> proxy = proxies.get(urlparts.scheme+'://'+urlparts.hostname)
(Pdb++) urlparts
ParseResult(scheme='http+unix', netloc='%2Ftmp%2Ftest_requests.132820227_21127_b4f4da9f', path='/path/to/page', params='', query='', fragment='')
(Pdb++) urlparts.scheme
'http+unix'
(Pdb++) urlparts.hostname
'%2ftmp%2ftest_requests.132820227_21127_b4f4da9f'
```
HTTPAdapter now updates its PoolManager connection_pool_kw

This is not particularly polished yet, and depends on the as-yet unreleased
urllib3 version 1.16. Still, I'd greatly appreciate some early feedback on how
(in)sane this approach is. 

With the addition of https://github.com/shazow/urllib3/pull/830 requests
should update the connection_pool_kw on the PoolManager so that new
ConnectionPools get created when TLS/SSL settings change. This ensures
that users can update the CA certificates used to verify servers as well
as the client certificate and key it uses to authenticate with servers.

This fixes issue #2863

---

Closes #2863 


Thanks for this @jeremycline!

In principle this is fine. However, it changes the interface of the HTTPAdapter which is IMO part of our public interface, so I'd want to sit on this until 3.0.0. Happily, 3.0.0 shouldn't be too far away now so we can progress with this work. =)


I'm confused why we would remove `cert_verify` as part of this.


@sigmavirus24 Because it sets things on the connection pool directly, which is Not Right. We want to get a different pool if those are different.


I thought about leaving the method name as `cert_verify`, but did it this way because the behaviour changed a great deal. I am, of course, not apposed to alternate approaches. I just wanted to make sure it was clear the way certificates are paired up with connections changed.


Since urllib-1.16 is out now, I suppose I should get this ready to go :smile:

Other than rebasing once 1.16 is pulled in, is there anything that jumps out as lacking?


Nothing leaps out, no. =)


I've rebased onto 2.11 and all the tests are passing now.


@jeremycline can you rebase once more? I promise we won't lose track of this again. =(


@sigmavirus24 no problem, rebased!


wait a second, @Lukasa can this not go into a pre-3.0 release?


@sigmavirus24 Nope, we removed cert_verify which is public on the HTTPAdapter.


@sigmavirus24 Should I add a 3.0 changelog entry or is that something you'd prefer to handle?


Looks good to me then!
Flip conditional in session.send()

Previously we checked that the `request` being sent was an instance of a
PreparedRequest. If a user somehow created a PreparedRequest using a different
Requests library instance, this check makes the request un-sendable.

(This happened recently - unbeknownst to me, my server was running an outdated
version of pip, vulnerable to this issue - pypa/pip#1489, which creates
multiple subdirectories (src/requests, src/requests/requests) when you rerun
pip install --target. So the PreparedRequest was being created in one version
of the library and compared against the other version of the library, and
throwing this exception, even though they were both PreparedRequest instances!)

It would probably be preferable to check the object's behavior (instead of
its type), but a PreparedRequest has a lot of behavior, and it wouldn't be
really feasible or allow us to provide a helpful error message to check all
of it here. Instead flip the conditional to guard against the user sending an
unprepared Request, which should still give us most of the benefits of the
better error message.

Fixes #3102


I'm happy with this! Go for it @kennethreitz, merge if you'd like to. =D


Not if I merge it first ;)
Redirected request.post turns into a GET

```
SITE = 'HTTP@{}'.format(API_ENDPOINT)
w_, krb_context = kerberos.authGSSClientInit(SITE)
kerberos.authGSSClientStep(krb_context, '')
AUTH = HTTPKerberosAuth(mutual_authentication=requests_kerberos.DISABLED)

r = requests.post(url, json = {'key':'value'}, auth=AUTH)
print 'MY REQUEST'
pprint.pprint(vars(r.request))
r = requests.post('http://httpbin.org/post', json = {'key':'value'})
print 'THEIR REQUEST'
pprint.pprint(vars(r.request))
```

```
MY REQUEST
{'_cookies': <RequestsCookieJar[Cookie(version=0, name='xxx', value='xx', port=None, port_specified=False, domain='xxx', d\
omain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=1461145561, discard=False, comment=None, comment_url=None, rest={'HttpOnly\
': None}, rfc2109=False), Cookie(version=0, name='xxx', value='xxxx', port=None,\
 port_specified=False, domain='xxx', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=True, expires=146114\
5561, discard=False, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)]>,
 'body': None,
 'headers': {'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.9.1', 'Connection': 'keep-alive', 'Cookie': 'xxx', 'Content-Type': 'application/json'},
 'hooks': {'response': [<bound method HTTPKerberosAuth.handle_response of <requests_kerberos.kerberos_.HTTPKerberosAuth object at 0x10c0a6310>>]},
 'method': 'GET',
 'url': 'xxx'}
THEIR REQUEST
{'_cookies': <RequestsCookieJar[]>,
 'body': '{"key": "value"}',
 'headers': {'Content-Length': '16', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.9.1', 'Connection': 'keep-alive', 'Content-Type': 'a\
pplication/json'},
 'hooks': {'response': []},
 'method': 'POST',
 'url': 'http://httpbin.org/post'}
```

As you can see, body becomes empty and the method is POST.

Looking at the response history shows that it gets redirected from the authenticator, then the request becomes a GET.

Shouldn't the POST request propagate all the way to the end?


Nope. =) Not all redirects preserve the verb: if the website in question wants to preserve the verb, they should use 307 or 308 instead.

Requests behaves like a browser when redirected. That means that it does the following:
- on 303, changes everything except HEAD/GET to GET, as specified in RFC 7231 Section 6.4.4
- on 302, changes everything except HEAD/GET to GET, as allowed by the note in RFC 7231 Section 6.4.3 and as done by browsers
- on 301, changes POST to GET, as allowed by the note in RFC 7231 Section 6.4.2 and as done by browsers
Congrats on worst naming standards for Python modules 2015+

`
pkg_resources.VersionConflict: (requests 2.2.1 (/usr/lib/python2.7/dist-packages), Requirement.parse('requests>=2.3.0'))
`

> Reasons for using a naming convention (as opposed to allowing programmers to choose any character sequence) include the following:
> - to reduce the effort needed to read and understand source code;
> - to enable code reviews to focus on more important issues than arguing over syntax and naming standards.
> - to enable code quality review tools to focus their reporting mainly on significant issues other than syntax and style preferences.
> - to enhance source code appearance (for example, by disallowing overlong names or unclear abbreviations).
>   [https://en.wikipedia.org/wiki/Naming_convention_%28programming%29#cite_ref-1](https://en.wikipedia.org/wiki/Naming_convention_%28programming%29)

Without prior knowledge into Python syntax, Bash syntax and the name of the module, error fixing took several extra hours to solve. Another nail in the coffin.

Thank you for a great module regardless of name. It provides one of the strongest features I need in a bot that checks URLs. Too bad I have to be a wizard to swish and flick my wand to fix the errors.
Sticker Design [Kenneth]

Using this as a place to throw things i find that I like. 

Going to use StickerMule, not StickerRobot. Undecided on size or design, yet. 

I like the idea of using 99designs to come up with the sticker. Would be far more expensive than doing it myself, but I'd rather be handing out _amazing_ stickers at a conference than mediocre ones (for an amazing project).


http://99designs.com


Turtles (evolutions of Rezzy the sea turtle, possibly):
- http://graphicriver.net/item/turtle/10194964?s_rank=6 (cute, but not a sea turtle)
- http://graphicriver.net/item/cartoon-turtle/12865148?s_rank=38 (running, meh)
- http://graphicriver.net/item/cartoon-turtle/9991241?s_rank=49 (i really like this one, very similar to current Rezzy)
- http://graphicriver.net/item/vector-turtle-tattoo/8725608?s_rank=157 (super dope, very different)

Undecided if "sea turtle" is important. I feel like it is. 


Experimented with using the current Rezzy, of course. This could work, but I think the stickers are an opportunity to play with Rezzy as a concept, not as that specific image. 


> Would be far more expensive than doing it myself, but I'd rather be handing out amazing stickers at a conference than mediocre ones (for an amazing project).

I'll be handing out mediocre ones. =D


99design's prices have gone up, won't be using them. 


$ pip install requests


Shipped!
Implement #2149

Though #2149 is marked as closed, I had to do this because 1) I can't just give up requests and 2) I can't change the non-standard URLs I'm using. So I did a little hack. Now it goes like this:

```
>>> from urllib.parse import quote
>>> requests.models.DEFAULT_QUOTE_VIA = quote
>>> requests.models.DEFAULT_SAFE = '/?$=%'
>>> r = requests.get('http://reddit.com/', params={'$foo': 'bar baz'})
>>> r.url
'https://www.reddit.com/?$foo=bar%20baz'
```


I'm -1 on this: we have an appropriate hook to do this already (the PreparedRequest), and I'm disinclined to want to add configurability here, _particularly_ in the form of global singleton state: this is a recipe for really tricky to debug issues down the road. 


@Lukasa Thanks for pointing to the right direction.
How to find what proxies Requests is using?

I set no proxies in environment variables, but I set proxies in system configuration on my MacOS. And I find that Requests use it automatically (by checking what responses I get), but it seems this feature is not pointed out in your documentation. So is there any way to see what proxies are Requests is using? Will it automatically use what urllib.request.getproxies() return?


You can call `requests.utils.get_environ_proxies(url)`, where the URL is the URL you're accessing. That will show you what proxies we found.
Be able to set _MAXHEADERS in httplib.py

I am making a GET request which returns a response with 100+ headers because I am printing debug statements in it, so when I use requests to get the URL, I am a similar error like below:

Traceback (most recent call last):
  File "......", line ....., in get_response
    response = requests.get(url)
  File "/Users/loh/myvirtualenvs/auto_ssai2.7/lib/python2.7/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, *_kwargs)
  File "/Users/loh/myvirtualenvs/auto_ssai2.7/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, *_send_kwargs)
  File "/Users/loh/myvirtualenvs/auto_ssai2.7/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/Users/loh/myvirtualenvs/auto_ssai2.7/lib/python2.7/site-packages/requests/adapters.py", line 426, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', HTTPException('got more than 100 headers',))

Need a way to change _MAXHEADERS in httplib.py as suggested here:
http://stackoverflow.com/questions/23055378/http-client-httpexception-got-more-than-100-headers


I'm sorry. I don't understand. Does something like

``` py
import httplib  # or http.client if you're on Python 3

httplib._MAXHEADERS = 1000
```

Not work for you? Given that this seems to be exceptionally rare, I'm not sure we need a way to do this (or override this).
Token string add error

``` python
url = "https://graph.facebook.com/v2.6/me/subscribed_apps?access_token="+self.token
        data = {}
        if url:
            data['url'] = url
        result = requests.post(url, data)
```

Above code generating following error, even though schema is defined as in url. how to fix?

```
requests.exceptions.MissingSchema: Invalid URL '%sreally long string , facebook token':
No schema supplied. Perhaps you meant http://%sreally long string , facebook token?
```


EAAHEisMO9u8BABL41ZCoUFeXRWzhhjxDb0jDNzyH7YdJeFVpUoYpRrqhp96YKkDrfzqFJfclbqhOZB4MMvihAOS2XBJLYpLDGLIyZAuXBuo8gpddNICEgc7Xo8mRGBH6F9q38ZALItZCaGfytrs59dPLRhi9ibEYeC2Pi0AbXXXXXXXX my token is as following


@Nuruddinjr Can you print the actual error message? It seems like the URL you've built is invalid, which suggests that the code you're using right now isn't correct in some way.


@Nuruddinjr please answer @Lukasa soon. Otherwise, I'm going to close this issue as it does not appear to be a bug.
Problem in json.dump with headers dictionary

I really don't know what is happing. I make a `requests.get` and save the returned headers in a dictionary:

```
r = requests.get("http://myurl.com")
my_dic = {'my_header' :  r.headers}
```

When I try to parse my_dic to JSON, I get an error. 

```
with open('arq.txt', 'w') as fp:
    json.dump(my_dic, fp)
```

For no special reason, it fail! I thought there was an illegal character or duplicate key, but no. It only work when I created this function:

```
def fix(old):
    new = {}
    for key, value in old.items():
        new[key] = value
    return new
```

So, I need to do this:

```
r = requests.get("http://myurl.com")
my_dic = {'my_header' : fix(r.headers)}
```

Why? Is this a bug? I am using the python 3.5 and believe all most update libraries. 


So you didn't actually print your error, but the reason it's a problem is that the json module really only accepts dictionaries. `r.headers` is not actually a dictionary:

```
>>> type(r.headers)
<class 'requests.structures.CaseInsensitiveDict'>
```

You can resolve the problem by simply calling `dict()` on it:

``` python
r = requests.get("http://myurl.com")
my_dic = {'my_header' :  dict(r.headers)}
```


Yeah, now I understand it. But... Why not make it a dictionary? I am new in python and it take hours to understand WHY the JSON was having problem. My first try was some illegal character. Because the error print it like a dictionary. I have the same problem with cookies. 

So I suggest you to make every "like a dictionary" to be a "real dictionary" I can imagine so many problems with it in others frameworks. 


The reason it's not a dictionary is because generally speaking, types defined in Python should not subclass the built-in types. This is because the built-in Python types have various shortcuts in their C implementation that means that they don't call back into Python code, which means they don't see method overrides defined in Python.

For that reason, it is intended that modules use the [collections module's Abstract Base Classes](https://docs.python.org/2/library/collections.html#collections-abstract-base-classes): in our case, [Mutable Mapping](https://docs.python.org/2/library/collections.html#collections.MutableMapping). It is unfortunate that the JSON module doesn't like them, but I believe that's a flaw in the JSON module rather than the data type we've constructed.
Clear any pooled proxy connections

Based on the explanation of the bug in #3090 this PR closes any pooled connections for any active ProxyManager object associated with the session.


This looks great to me. @bodgit could you add some tests to ensure this doesn't regress?


I'm wondering what/how I can test this. I had a look through `tests/test_requests.py` but I could only see one proxy-related test. Do you have any suggestions?


Create a dictionary with values which are mocks and then assert that `clear()` was called once on each one?


To be clear, my code review is +1 on this too. @sigmavirus24 has the best testing approach here, I think we just need to mock it out.


The suggestion makes sense, however I don't know how to do this with pytest. I'm trying to find an existing example I can use as a starting point...


I've managed to come up with the following:

``` python
    def test_session_close_proxy_clear(self, mocker):
        proxies = {
          'one': mocker.Mock(autospec=True),
          'two': mocker.Mock(autospec=True),
        }
        session = requests.Session()
        mocker.patch.dict(session.adapters['http://'].proxy_manager, proxies)
        session.close()
        proxies['one'].clear.assert_called_once_with()
        proxies['two'].clear.assert_called_once_with()
```

I had to pull in `pytest-mock`. It passes with my change and fails when I revert it so it looks like it does the job.


I think that looks fine to me. If you want to go ahead and update this pull request (you can just push your new commits to the branch), we'll review more formally. =)


Fixed.


Thanks @bodgit! :tada:
add requests.util.parse funtions

help things easy


Thanks for this!

However, I don't believe we need either of these.

`parse_rawpostdata_to_dict` is a less general version of `urlparse.parse_qs` from the standard library, so users should simply use that. `parse_rawcookies_to_dict` is a less general buggy version of `cookielib.CookieJar.add_cookie_header`. Both of these have better versions available in the standard library, so I'm afraid that we're not going to merge this: they simply add no useful function that is not available elsewhere.

Thanks for the work though!


@Lukasa 
But i can't use `cookielib.CookieJar.add_cookie_header` to add a raw cookie to requests
can you give me a example?
thanks!


@sleshep What specifically are you trying to achieve? You appear to have a cookie header on hand and want to attach it to a request. Is that right?


@Lukasa 
Yes, I copy a cookie from browser want to attach it to a request.
How to do this with `cookielib.CookieJar.add_cookie_header`?


Oh I'm sorry, I didn't fully understand your use case. You want to do this:

``` python
import Cookie
import requests

c = Cookie.SimpleCookie()
c.load('user=a;a=b;c=1')

s = requests.Session()
for v in c.values():
    s.cookies.set_cookie(requests.cookies.morsel_to_cookie(v))
```

Then, running requests through that session will use the cookies appropriately.

Note that this won't tightly scope them to domains unless the domain is set in the cookie directly, but you can use the standard functionality in the Cookie module to manipulate the morsels appropriately.


@Lukasa 
Thanks.
 but I think use `Cookie.SimpleCookie` is too complex,use this pull request to very easy to achive.


@sleshep As a project, we don't agree. This pull request does not implement the same functionality that the standard library does. It also implements only a portion of what can be found without this pull request regardless of one's perception of ease of use.

If these functions are working for you, you should keep using them, but we are not going to accept them into requests or maintain them since there are significantly more mature and robust solutions available to us that we do not have to maintain.


@sigmavirus24 
Ok.thank you


if found a simple way to  achieve my target.

``` python
from urlparse import parse_qsl
from requests import Session
cookies='user=a;a=b;c=1'
params='user=a&a=b&c=1'
s=requests.Session()
s.cookies.update(dict(parse_qsl(cookies)))
s.post('http://someurl',data=dict(parse_qsl(params)))
```

it work well.
Thanks all of you.
@kennethreitz 
@Lukasa 
@sigmavirus24
badTimezone: cannot clone with git 2.8.1

I can't clone this repo. It complains:

```
$ git clone https://github.com/kennethreitz/requests
Cloning into 'requests'...
remote: Counting objects: 17048, done.
remote: Compressing objects: 100% (39/39), done.
error: object 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8: badTimezone: invalid author/committer line - bad time zone
fatal: Error in object
fatal: index-pack failed
```


This is a known problem: see #3008 and #2690 for previous discussions. The problem can be easily resolved on your end either by performing a shallow clone (if you don't need the history) or turning off the fsck timestamp check.


Thanks for the quick response.


@rillian did you enable verification yourself or did another tool do that?


You're right, I have those options set in my ~/.gitconfig; it's nothing to do with the git version. I don't know if I did it manually or if one of our audit tools added it, because how is that not the default? :/

``` ini
[transfer]
          fsckobjects = true
[fetch]
          fsckobjects = true
[receive]
          fsckObjects = true 
```

I've been trying to figure out how to get YCM to work around this, but it's tricky with submodules because an initial shallow clone generally fails. There doesn't seem to be a `fetch.fsckBadTimezone`. I couldn't get your `$ git config fsck.badTimezone ignore` suggestion to work, just disabling `fetch.fsckObjects` entirely. 


```git config --global fetch.fsck.badTimezone ignore```

Works for me when cloning submodules, this just disables it for fetch.
Is there a mistake on requests/docs/user/quickstart.rst

I am reading the official document of requests. There is one code in quickstart.rst

``` python
>>> payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
>>> r = requests.get('http://httpbin.org/get', params=payload)
>>> print(r.url)
```

However, the output is

``` python
http://httpbin.org/get?key2=value2&key2=value3&key1=value1
```

instead of what is in rst?

Is this a small mistake?


No. =)

Python dictionaries are _unordered_: that is, when iterated over the order in which elements are returned is not well defined. That means that iterating over the dictionary above can give you two possible options: first `'key1'` then `'key2'`, or first `'key2'` then `'key1'`. The first case gives you the code from the documentation, the second one gives you the code you saw.

If you repeatedly re-run the same code, you should see both URLs. This is not a bug in requests or in Python, it's just an expected side effect of the way Python works.
Is there a way to specify filename in the post data?

Hi, 
For example, I want to upload a video called "测试中文视频.mp4", which will be converted to`filename*=utf-8''%E6%B5%8B%E......`, the problem is that my working server does not recognize this format, so, is there a method to tell requests to convert the request message to like this: `filename=测试中文视频.mp4` or like this: `filename=u'\u6d4b\u8bd5\u4e2d\u6587\u89c6\u9891.mp4'`.

Thanks very much!


I haven't worked on this, but a PreparedRequest I believe is what you want.

On Mon, 11 Apr 2016 15:04 imnisen notifications@github.com wrote:

> Hi,
> For example, I want to upload a image called "测试中文视频.jpg", which will be
> converted tofilename*=utf-8''%E6%B5%8B%E......, the problem is that my
> working server does not recognize this format, so, is there a method to
> tell requests to convert the request message to like this:
> filename=测试中文视频.jpg or like this:
> filename=u'\u6d4b\u8bd5\u4e2d\u6587\u89c6\u9891.mp4'.
> 
> Thanks very much!
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/3085


There is no builtin method to make requests do this, nor will there be: servers that don't understand the format requests uses are old an non-standards-compliant. However, as @TetraEtc points out, you can use the [PreparedRequest flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests) to mutate the body as you wish to, which would allow you to change the multipart-encoded body to whatever form you like.

You can also manually set the filename yourself by using longer tuples in the [files parameter](http://docs.python-requests.org/en/master/api/#requests.request): in particular, if you use a _bytestring_ in the filename portion then requests will leave it alone.


@TetraEtc thanks, I have succeed using  urllib2 to send the data which encoded manually, although it is a boring work. I will also try the PreparedRequest method.

@Lukasa  do you mean use  something like `"测试中文视频.mp4".encode('utf-8')` in the filename portion?
 when I try this, because of `result.encode('ascii')` (at requests/packages/urllib3/fields line38) fail, the post content is also  changed to something like this `filename*='%E6%B5%8B%E8%AF%95%E4%B8%AD%E6%96%87%E...`, which is not I want

Thank you.


Ah, yup, that'll do it. So yes, in that case I'm afraid you will have to use the prepared request flow to tweak the body.
requests attempts to use IPv6 even when IPv6 is disabled

This is an issue when running on a server without IPv6 (must be disabled because the network does not support it). Example when connecting to https://graph.facebook.com and IPv4 happens to fail:

Python 3.4.3, requests 2.9.1, `ipv6.disabled=1` in kernel, `gai.conf` set to prefer IPv4

``` python
HTTPSConnectionPool(host='graph.facebook.com', port=443): Max retries exceeded with url: /v2.5/me/feed (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f4dbd158518>: Failed to establish a new connection: [Errno 97] Address family not supported by protocol',))
Traceback (most recent call last):
  File "/home/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/lib/python3.4/site-packages/requests/packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/home/lib/python3.4/site-packages/requests/packages/urllib3/util/connection.py", line 71, in create_connection
    sock = socket.socket(af, socktype, proto)
  File "/usr/lib/python3.4/socket.py", line 126, in __init__
    _socket.socket.__init__(self, family, type, proto, fileno)
OSError: [Errno 97] Address family not supported by protocol
```


Requests doesn't do much special here. Requests calls `getaddrinfo` and then iterates through the entries in the order returned from `getaddrinfo` attempting to connect to each one in turn.

That exception only gets raised if we've attempted all our connections and they all fail. We then raise the final exception we saw. The bigger problem here seems to be that we're swallowing perfectly reasonable exceptions earlier in the chain that reveal the underlying problem in situations when IPv6 or IPv4 are enabled.

That code all lives in [urllib3](https://github.com/shazow/urllib3), would you like to open this issue over there so we can track it appropriately?


That's fair. I'll open a ticket on urllib3.
Improvement to hook response.

Hello,

It would be nice if the _response_ hook was called even if there is an error during the send() process flow.

Example, if a timeout occurs, the cb is not used and a requests constructed url and query data that can normally be saved via the cb is not available - that is, URL that caused a timeout.

Thanks for your consideration.


@JackDandy If you pass `stream=True`, then the callback should fire once the response has been received, so that resolves the problem where timeouts are encountered downloading the response data.

If the timeout is encountered while actually downloading the response, we can't call the response hook because we don't have a response to call it with. If you need to intercept that data the best place to do it is likely to be in the HTTPAdapter itself (by overriding the `send` method, which gets a `PreparedRequest` object), or by using the [prepared request flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests) which will give you access to those things, also on a `PreparedRequest` object.

Do either of those work for you?


If there is no response to provide to the hook, I'm uncertain what we would call the hook with. I do not think we could safely construct a fake response object just so we can call the response hook (which wouldn't confuse existing response hooks).

You can introspect what requests constructs for the url, query data, etc. by using the prepared request flow.


The hook is a nice and cheap way to get the constructed URL that is used in the request for logs. To add the prepared request flow seems overkill for URL reporting. A server response has content or none so there is value to add by calling the hook in any outcome.

[edit] If too much hassle to create response objects, then save the built URL/post data in the requests object like _headers_ in order to give a full picture of what requests sent after its pre-processing.

So, yeah, just suggestions.


Sounds like `PreparedRequest` is what you want — it was built for exactly this purpose, if I understand you correctly.
Added a test to show a faulty behaviour when posting binary data for an object with no __iter__

An object with read / seek / tell but without __iter__ triggers a code path in requests which causes the stream to seek to the start instead of the position it was when it was passed into the data of the request. The faulty code is in models.py in the function prepare_content_length where it seeks to the start of the stream.

The tests fail currently because this is a test to expose the bug.


Ok awesome, this is a great first step! Are you interested in providing a fix as well?


I have made a dummy test so it works both on python 2 and 3. I hope the added fix is ok with all possible interactions (first time looking at requests code), but it might still not work with the redirect bug.


I think this works well, at least in the basic case. As you correctly pointed out, there's a question about how this will work with redirects, but that's harder for us to test at this moment (though we'll get there shortly I hope).

@kennethreitz, want a final review here?


This looks like a sane approach to me. 


Ok @kennethreitz, you can hit the big green merge button whenever you're ready.


Gah, @sigmavirus24 beat me to it
Missing double quotation marks at "requests/packages/urllib3/fields.py" Line 46

value = '%s_=%s' % (name, value)
should not be
value = '%s_="%s"' % (name, value)
:)


Thanks for this report!

No double quotation mark is required here. All the characters that would require a quotation mark (e.g. spaces) are percent-encoded by `encode_rfc2231`. This means that, per that RFC, no quotation mark is required. =)
Add 421 Misdirected Request.

The 421 Misdirect Request status code was originally added in [RFC 7540](https://tools.ietf.org/html/rfc7540#section-11.7), and is going to be actively used in [RFC 7838](https://www.rfc-editor.org/rfc/rfc7838.txt). This adds it to our status code registry.


:sparkles: :cake: :sparkles:
The request module redirects to a wrong URL if the 'Location' field of the response header contains utf-8 characters.

Hello!

The requests module redirects to a wrong URL if the 'Location' field of the response header contains UTF-8 characters (for example,  https://picsrch.me/wCtd3uQF). The server typically does not return a response containing UTF-8 characters in the 'Location' field, but many HTTP clients (Firefox, Chrome, Wget, etc.) react correctly. As I can see, this module tries to solve this problem as well.

The main problem is that http.client module (which is used by requests module) decodes all fields in latin1, but requests module uses utf-8 encoding to prepare URL from 'Location' field (we can see this behavior in urllib.parse.quote function).

P.S.:
Python    - 3.4.0
requests - 2.8.1


Hi there!

This is a known issue that has a fix prepared for the 3.0.0 release: see #2754. Unfortunately, this change will subtly change the logic of the code for a lot of people, which is why it's scheduled for 3.0.0. However, we're aiming to have the 3.0.0 release out in the first half of this year.


Ok. Thanks! :)


when will the 3.0 release be out? from the situation I have this issue persists and looks like its not just for utf-8 Location header.

@1600 we don't have a hard date for 3.0 at this time as we're waiting for some work on urllib3 to be completed before we can look at creating a release.
proxy-scheme is case sensitivity or not?

This question came out when I tried to validate some proxies.
If  proxy-scheme is uppercase as follows, it will return 200. (The host of proxy is not a true one.)
import requests
proxy = {
    'HTTP': 'HTTP://111.1.11.1:8000'
}
r = requests.get('http://www.sina.com',proxies = proxy)
print (r.status_code)

However, if changing 'HTTP' to 'http', it will raise an error about httpconnectionerror.
So, it is confused me that proxy-scheme is case sensitivity or not? Or something went wrong about my code ?

Hoping to receive your instruction. 


Thanks for this!

This is a real bug, but it's a duplicate of #3075. Please see shazow/urllib3#833 to track work on this problem.
Redacted

Redacted


did you try `OrderedDict`? Python dict has unordered keys by design


@arcan1s you don't need an OrderedDict, you only need to provide a list of tuples. And questions should be asked on [StackOverflow](https://stackoverflow.com).


@MrAureliusR The terseness comes from the fact that we get several new issues a day that require that the core maintainers interrupt their days to address. Each bug report costs at least one maintainer a few minutes and a context switch, and may cost more than that if multiple maintainers stop to address it. That's unfortunate when you consider that the majority of the maintainers are unpaid, and all of us are overstretched.

This is doubly problematic for questions. When you opened this issue you were shown this screen:

![screen shot 2016-04-01 at 13 56 23](https://cloud.githubusercontent.com/assets/1382556/14207380/951c6ec6-f811-11e5-9d51-b7842fd8023a.png)

Notice the yellow bar at the top. Had you clicked through and read that link you'd have seen [this document](https://github.com/kennethreitz/requests/blob/master/CONTRIBUTING.md), which contains in its very first section this text:

> The GitHub issue tracker is for _bug reports_ and _feature requests_. Please do not use it to ask questions about how to use Requests. These questions should instead be directed to Stack Overflow. Make sure that your question is tagged with the `python-requests` tag when asking it on Stack Overflow, to ensure that it is answered promptly and accurately.

This is why the interaction was strained: you demonstrated that you had not read that note. Those notes are important to ensure that everyone interacts as efficiently as possible and to ensure that the limited resources of the project are allocated in the most useful way available.

Each issue opened on this bug tracker costs the core team at least 5 minutes of time. We're at issue 3073 right now, meaning that if these had all been _just questions_ we'd have spent nearly _11 days_ just swapping to handle those questions.

I say all of this not to chastise you, but to remind you that open source is a predominantly volunteer operation where the primary resource is time. Asking questions on bug trackers is a common source of wasted time on the part of maintainers, which is why maintainers can be terse when questions are asked in inappropriate forums.


It's triply problematic here because our documentation describes how to handle this exact scenario.


Read through the original thread, and I want it to be known that I think @MrAureliusR's submission was perfectly fine. Not every interaction here has to be handled solely by the maintainers (in this instance, a community member was already stepping in to help).

The "be cordial or be on your way" rule applies to maintainers too :) These interactions were fairly cordial, but far from friendly.


No further comment needed. Sorry for your bad experience, @MrAureliusR
find_no_duplicates raise KeyError

Hi there!

I am currently on `requests version: 2.9.1` I have getting the following error in my code:

```
Traceback (most recent call last):
  File "./add.py", line 98, in <module>
    'SESSID':term.cookies['SESSID'],
  File "/usr/local/lib/python2.7/site-packages/requests/cookies.py", line 276, in __getitem__
    return self._find_no_duplicates(name)
  File "/usr/local/lib/python2.7/site-packages/requests/cookies.py", line 331, in _find_no_duplicates
    raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='SESSID', domain=None, path=None"
```

Line 98 in `add.py` refers to the following:

```
# Grab the Add/Drop Class page cookies                                
ad_cookie = {                                                         
    'SESSID':term.cookies['SESSID'],                                  
    'IDMSESSID':username                                              
} 
```

Just to make sure that the particular cookie in that page was not removed or missing, I checked the cURL for when I hit that page, and surely enough, both `SESSID` and `IDMSESSID` were in that cURL request.

Wondering what the issue is?


Don't check curl, check requests. =) Can you print out:

```
for resp in response.history:
    print resp.headers
print response.headers
```

That should provide information about what cookies got set.


@Lukasa I feel dumb! Turns out my `username` and `password` were incorrect, and thus I was not authenticated, and thus no cookie. My bad. Thanks for the super quick reply though, Lukasa!


No problem, glad you got it sorted out!
allow certs.where to return a file like object

Currently `certs.where` expects a file name that is then passed along and eventually opened.  This means it is up to the caller to deal with eggs, wheels, install locations, zip_safe vs not etc, to eventually figure out where something is going to end up on the file system..  None of these issues are particularly fun.  Instead of certs.where could return a file like object and a custom implantation could be as simple as `StringIO.StringIO(pkgutil.get_data(__name__, 'cacert.pem'))`


Unfortunately, `certs.where` cannot return a file-like object: due to several limitations in the way OpenSSL's APIs behave across versions of Python the lowest common denominator is that we need to provide a _path on the filesystem_ to OpenSSL.

This makes it impossible for us to do that. Sorry, it's otherwise a good idea!
Update docstring and API doc to document ability to add per-file headers in multipart POST

I recently needed this function and could not find it in the docs - luckily SO pointed me at PR #1640.

However, I feel like this really should be documented, so here is my attempt.

Note:
- I have not run any tests whatsoever over this as this is "just a docs change" - I don't know if that's ok. 
- The API docs seem a bit inaccurate in that they do not mention that `files` can also be a list. Similarly the docsstring in `_encode_files` does not discuss the case where instead of a tuple a file-obj is provided.


\o/ This looks great, thanks @achermes! :sparkles: :cake: :sparkles:
Response read hangs

python 2.7.11
requests installed from pip and up-to-date

URL -> 'http://archive.mid.ru/bdomp/zu_r.nsf/strawebeng!OpenView'

GET request hangs at read time.
I tried to use `stream=True` and then read from `response.raw` and it works. But doesn't finish on normal usage.

I looked at the headers and the `Content-Length` is duplicated.
When I try to read it from `response.headers['Content-Length']` I find a single string: `'133172, 133172'`.
The length is correct otherwise.

I am not entirely sure of what's causing this but my guess is that requests is getting confused by the duplicate header.

_thanks for everything_


> I tried to use `stream=True` and then read from `response.raw` and it works.

When you did this read, did you just do `read()`, or did you provide a length to the `read()` call?

So, the server is getting this wrong. It is strictly forbidden to send multiple content-length headers, and generally speaking I oppose having code to defend against idiot servers doing things wrong. In this specific instance, however, we have another very similar problem with servers sending duplicate `Location` headers, so I wouldn't be opposed to us providing fixes for both at once, as the fix is basically the same.


Yes, I provided the value I get from the duplicate `Conten-Length` header.


@medecau Yeah, so you'll almost certainly find that if you call `read()` with no arguments (which is what requests normally does), that hangs the same way that requests does. This is because httplib controls the HTTP framing and is presumably confused about what the actual content-length is.

Strictly speaking the best way to fix this would be to raise a bug against httplib, though the _actual_ best fix is to tell the server to sort itself out.


Since this is strictly not something we can fix in requests (or urllib3) I think we should close this. Thoughts @Lukasa ?


I think we technically _can_ fix it in urllib3/requests, but doing it is pretty invasive. I'd like to approach getting it fixed in httplib first so at least python-dev can say we're lunatics.
No way to make requests redirect POST/PUT/DELETE requests

In the documentation (see http://docs.python-requests.org/en/latest/api/#requests.request) it says that specifying _allow_redirects_ to `True` means that redirecting POST/PUT/DELETE is allowed.

Quick attempts show that the reality is that those requests _ALWAYS_ get redirected as `GET`s, even when this parameter is True (which it is by default). It seems like the tests [here](https://github.com/kennethreitz/requests/blame/3acf3a723892550719ca23bfe1be07549a8ad201/tests/test_requests.py#L165) support this claim.

The one commented out test [here](https://github.com/kennethreitz/requests/blame/3acf3a723892550719ca23bfe1be07549a8ad201/tests/test_requests.py#L208) supports the statement from the documentation, but is commented out (the reason seems to be a bug in httpbin, but no further explanation is supplied)


Requests will change the verb used for a redirect in the following cases:
- On a 303 See Other response, HEAD and GET are left alone: all other verbs get changed to GET, as per [RFC 7231 Section 6.4.4](https://tools.ietf.org/html/rfc7231#section-6.4.4).
- On a 302 Found response, HEAD and GET are left alone: all other verbs get changed to GET. This is allowed by the note in [RFC 7231 Section 6.4.3](https://tools.ietf.org/html/rfc7231#section-6.4.3), and is extremely common behaviour with all browsers.
- On a 301 Moved Permanently response, POST is changed to GET. All other verbs are left alone. Again, this is allowed by the note in [RFC 7231 Section 6.4.2](https://tools.ietf.org/html/rfc7231#section-6.4.2) and is extremely common behaviour with browsers.

While requests allows redirects, it uses the semantic meaning in the status code to potentially adjust the redirect verb. If a server explicitly wants the redirected user agent to keep the verb the same in all circumstances, servers should use 307 Temporary Redirect and 308 Permanent Redirect.


:+1: thanks. You learn something new every day...


@lukasa we should add your snippet of text there to the docs. Doubt it'll reduce incoming issues (questions) around this, but it'd be great for the docs.
Honor HTTP strict transport security headers

[HTTP strict transport security header](https://scotthelme.co.uk/hsts-the-missing-link-in-tls/) helps protect web applications against communications interception and MITM attacks. I would recommend that requests look for and honor the header by default.


See shazow/urllib3#608, and the comment I just made on #3063: the same limitations that apply there apply here.


Perfect, thank you! :bow:
Honor HTTP public key pinning headers for server certificate validation

An interesting protection against MITM attacks could be provided by default if requests looked for and honored [Public-Key-Pins](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning) headers in the server response for additional validation of the certificate presented.


@asieira This is an idea we've been interested in for a long time: see shazow/urllib3#607.

Unfortunately, any support we're likely to have will be of minimal use in practice unless deliberate effort is taken to harden your application. We'd need a cache in non-ephemeral storage (or the exit of your app cleans the in-memory cache), and PKP headers are extremely uncommon.

Regardless, the issue to track is the one linked above.


Perfect, thank you! :bow: