Raibek/requests-github-issues huggingface dataset:
fix: check 'cookie.value' before actions

Just a small fix :)


Can you add a test to make sure this doesn't regress?


Yes of course, how i can do that ?


Can I get clarification here: is cookie.value `None`, or just the empty string?


In some situation cookie.value is `None` (i can't explain ...)

```
Traceback (most recent call last):
  File "/home/checkmyws/pylib/jobs/httpping.py", line 60, in httpping
    r = requests.get(configs['url'], verify=verify_ssl, timeout=timeout)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 364, in send
    r = self.build_response(request, resp)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 178, in build_response
    extract_cookies_to_jar(response.cookies, req, resp)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/cookies.py", line 117, in extract_cookies_to_jar
    jar.extract_cookies(res, req)
  File "/usr/lib/python2.7/cookielib.py", line 1645, in extract_cookies
    self.set_cookie(cookie)
  File "/home/checkmyws/opt/venv/local/lib/python2.7/site-packages/requests/cookies.py", line 270, in set_cookie
    if cookie.value.startswith('"') and cookie.value.endswith('"'):
AttributeError: 'NoneType' object has no attribute 'startswith'
```


Fair enough. When checking for `None` you should really use `is not None`.


Thanks Lukasa, i have fixed my PR


Beautiful, I'm happy with Kenneth merging this. =D Thanks so much for your work!


I want tests before we merge it but I can added them for @william-p afterwards.


:cake:
Requests doesn't clean up on failed streaming uploads.

From [Reddit](http://www.reddit.com/r/Python/comments/1njuk5/python_for_network_engineers/ccjh56a):

> For instance, if you use it in a generator to stream content from a request and the connection dies, it doesn't clean up properly. You have to call close on the underlying socket but that's not done and it's not exposed to you.


Yikes, this seems rather serious.


Fixed in 2.1.0 (hooray!).
Cannot send Single Key, Multiple value data in post because of the use of pythons dict.

Some Horrible APIs like Rightscales expect you to send the same key with multiple values. For example to set 2 security groups with rightscale you need to do the equivalent of:

curl ... 
-d server[instance][security_group_hrefs][]=group1 
-d server[instance][security_group_hrefs][]=group2 

This isn't possible using the python dictionary approach.

http://support.rightscale.com/12-Guides/RightScale_API_1.5/1_Overview/Parameters#Array


1. You can always provide your own post data.
2. Support for this is built in :)  

```
r = requests.post('http://httpbin.org/post', data={'a': [1,2,3]})
```


Oh awesome, thanks for the quick response too! :+1:  Thats worked for me.
Accept per-file headers in multipart file POST messages.

- Each file in the `files` argument can now be a 4-tuple, instead of
  just a file, 2-tuple, or 3-tuple. If it is, the last value in the
  tuple is a dictionary of extra headers.
- See [#1639](https://github.com/kennethreitz/requests/issues/1639) and [StackOverflow](http://stackoverflow.com/questions/19104924/how-to-add-per-file-custom-header-to-multipart-upload-with-requests/19105672#19105672) for more details.


This would have to wait for 3.0 since it's a significant API change unless @kennethreitz or @Lukasa disagree.


This in principle looks great @abarnert, thanks so much!

We wouldn't need to wait until 3.0, because this change isn't backwards incompatible: adding the ability to pass a 4-tuple can be done in a minor release (e.g. 2.1.0) according to [semver](http://semver.org/).

The real question is whether we think this API extension is the right way to handle it. As you pointed out in #1640, we already poorly document this corner of the API so that definitely has to be fixed.

@kennethreitz: Are you happy with this extension to the API, or would you like to reconsider the multipart file API entirely?


Also, this is probably necessary for the [amount of control](http://stackoverflow.com/questions/19124965/http-put-with-base64-encoded-attachment-using-requests/19128178) people are expecting


I have a specific use case where this would come in very handy. The `data` dict I pass in contains plaintext XML data, but by default there is no `Content-Type: text/xml` header set. I also have no way of setting that header through `requests`' API.


I'm confused: if you pass a dictionary to `data` you're requesting an `application/x-www-form-urlencoded` Content-Type. If you want to post the plaintext data, pass a string to `data` then set the Content Type in the headers dictionary.


@Lukasa Here's an example of what I'm trying to do: http://paste2.org/kgNztbBv

Basically, each part of the multipart has a different `Content-Type`, separate and distinct from the one in the message header.


Oh, sure. OK, yes, this is exactly what this PR is intended to allow. =)


This code using the `poster` library does exactly what I needed. http://paste2.org/HUYM3fDp


Very related: shazow/urllib3/issues/215


Is shazow/urllib3#215 (or, rather, the patch in shazow/urllib3#220) the functionality I used in this patch? (It's hard to check on the phone, it sounds like it, and it looks like since that commit, at least one new urllib3 has been released and requests has updated its urllib3 twice…)


@abarnert Yes it is. =)


I'm +1, specifically because this changes makes the code much easier to understand.

Thanks, @abarnert.
Support custom per-part headers in multipart upload

As discussed on stackoverflow:
http://stackoverflow.com/questions/19104924/how-to-add-per-file-custom-header-to-multipart-upload-with-requests

A solution is proposed, something along these lines could be implemented in requests.


My solution on SO was a hack, to allow monkeypatching as little as possble. #1640 is a better solution to actually go upstream, and has a nicer API as well (the headers are just a fourth member of the file tuple, instead of turning the third member into a tuple of content-type and headers). Although I think it might be nicer to revamp the API in some way; passing a dict or a-list of `name: (filename, fp-or-contents, content-type, headers)` values isn't exactly clear (especially since the content-type, and now the headers, aren't even documented)…


Let's concentrate the discussion of this over on #1640. =)
Don't send the full URL on CONNECT messages.

This should resolve the problem from #1631.

@atomontage, can you try testing with this version of Requests to confirm that it resolves your problem?


I had 'Response [599]' errors, have been breaking my head the last 2 days over this. Finally I was sure it was a bug in requests, then I just woke up and found this patch. It solves my https proxy issues, you've made my day, thanks!!


Glad to hear this was helpful! This should hopefully be in the next release but that might not be for a couple of weeks, so feel free to use the version from my branch until we cut the next release. =)


Fixes the issues I had.


Most excellent.
Fixed #1630 and added test case.

I've added logic for adding cookies to the Session cookie jar in `Session.request`.


This is an awesome start! The test failures aren't relevant to your work, so that's fine.

I've made a note in the diff: if you go ahead and implement that I'll get Kenneth to take a look. =)


@Lukasa I pushed updated version, Thanks!


Awesome, that looks much better. I'll grab Kenneth and get him to take a look. =)


LGTM to me too. :+1: @daftshady 

Please add yourself to the [AUTHORS.rst](https://github.com/kennethreitz/requests/blob/master/AUTHORS.rst#patches-and-suggestions) file?
Mimic CURLOPT_HTTPAUTH from curl 

I believe it'd be nice if we could have the option to define more than one authorisation methods in a request. It is useful in cases where one is not aware of the auth methods the servers supports.
The way it could work is by polling the server, finding out which methods it supports and picking the best one to proceed.

i.e  auth=[HTTPBasicAuth(user,pass), HTTPDigestAuth(user,pass)]

I'm OK to submit a patch.


I'm -1 on this personally but @Lukasa and @kennethreitz might have different opinions.


Yeah, I'm -1 on this as well. If you genuinely have to do this, you can just check the `Authorization` header on the original 401 and then follow up with the correct kind of Auth. =)


@Lukasa `WWW-Authenticate` you meant...


I apologise, yes, that's what I meant. =)
2.0.0 fails to handle some 307 redirects

Requests fails to handle the final 307 redirect at this URL. Instead of using the Location: supplied in the 307, it continually requests the previous location (until exceeding 30 redirects):

```
import requests
req = requests.get('http://ensembl.org/saccharomyces_cerevisiae/Location/View?r=II:5175-7175;contigviewbottom=das:http://foo.bar.com:4895/123abc/das/yeast_variants=normal')
print req.status_code
```

Uninstalling 2.0.0 and installing 1.2.3 fixes the problem. 

It seems to be some kind of parsing issue. Removing the :// in the query string of the original URL (or URL encoding the : to %3A) also eliminates the redirect loop.

Note that 2.0.0 does properly handle the 301 and 302 replies also returned in that request; it's just the 307 that fails for some reason.


Thanks for raising this issue!

As you suggest above but don't really commit to above, this has actually got nothing to do with the status code and everything to do with the URL. The redirects I get when I do this are as follows:

```
301 -> 'http://www.ensembl.org/saccharomyces_cerevisiae/Location/View?r=II:5175-7175;contigviewbottom=das:http://foo.bar.com:4895/123abc/das/yeast_variants=normal'
307 -> '/Saccharomyces_cerevisiae/Location/View?r=II:5175-7175;contigviewbottom=das:http://foo.bar.com:4895/123abc/das/yeast_variants=normal'
```

The key bit about the 307 is that the redirect is relative. Our flawed bit of redirect logic is [here](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L95-L98) (reproduced for posterity):

``` python
# The scheme should be lower case...
if '://' in url:
    scheme, uri = url.split('://', 1)
    url = '%s://%s' % (scheme.lower(), uri)
```

Of course, the second URL does contain a `://`, but not in the place we thought it would be. The fix here should be reasonably simple: get urllib to parse the URL and work out if it has a scheme.


Actually, the trigger condition is even smaller: the redirected-to URI must also be not all lower case _and_ case sensitive. That makes this a very small window condition indeed.


Awesome. Thanks for the quick response! ^_^

Cheers,

--Rob


This was fixed by #1637. =)
Add helper method analogous to urllib.urlretrieve

The method [`urllib.urlretrieve`](http://docs.python.org/2/library/urllib.html#urllib.urlretrieve) downloads a file from a URL and saves it to disk. 

> Copy a network object denoted by a URL to a local file, if necessary. If the URL points to a local file, or a valid cached copy of the object exists, the object is not copied. 
> 
>  urlretrieve() will raise ContentTooShortError when it detects that the amount of data available was less than the expected amount (which is the size reported by a Content-Length header). This can occur, for example, when the download is interrupted.

That's a useful helper method. Is there anything like this in Requests?


There is no such analogous method, and the core library shouldn't have one. =) Instead, it should make writing one easy, which it does:

``` python
def urlretrieve(path, url):
    with open(path, 'wb') as f:
        r = requests.get(url, stream=True)
        r.raise_for_status()  # Replace this with better error handling.

        for chunk in r.iter_content(1024):
            f.write(chunk)
```

Thanks for suggesting this though! I might create a repository with useful snippets for Requests, in case people find the need for things like this with some frequency.


Further more @matt-hickford in order to use requests for local files you must define your own file adapter. We have been asked for it several times and I believe we all three maintain the opinion that trying to use requests for local files makes absolutely no sense.


Makes sense. Thanks for the snippet.
Requests 2.0 breaks POST requests made through HTTPS proxy

System configured HTTPS proxy, reported by urllib2.getproxies() as:

```
{'https' : 'https://10.1.1.1:3128'}
```

The target webserver is 2.2.2.2.
Requests 2.0 doing a POST through the system configured proxy:

```
requests.post('https://2.2.2.2:6666/test', verify=False)
```

The web server receives the POST as:

```
[2013-09-27 10:01:36] "POST https://2.2.2.2:6666/test HTTP/1.1" 404 918 0.000754
```

which is obviously wrong.

To verify with curl:

```
curl -x https://10.1.1.1:3128 -i -k -X POST https://2.2.2.2:6666/test 
```

The web server receives:

```
[2013-09-27 10:04:11] "POST /test HTTP/1.1" 404 861 0.000694
```

which is the expected behavior.


Thanks for raising this issue!

I believe this bug is a hangover from 1.X, when we didn't do HTTP CONNECT processing. Because we didn't do CONNECT, the proxy always needed the full url in the status line. Now that we do CONNECT, we should stop doing that for HTTPS requests. I'll prepare a pull request.


Patch by Lukasa fixes the issue for me.
Fixed #1623. Added 'MD5-sess' algorithm to HTTPDigestAuth

See https://github.com/kennethreitz/requests/issues/1623
I've added md5-sess algorithm referencing 'http://en.wikipedia.org/wiki/Digest_access_authentication'
Any criticism is welcome :)


Thanks for this!

I'm not in the best place to review this, having not contributed to this section of our codebase, but I can point out a few things. @sigmavirus24 would do a better job of reviewing this change.

Firstly, according to RFC 2617, ['If the "algorithm" directive's value is "MD5-sess", then A1 is calculated only once - on the first request by the client following receipt of a WWW-Authenticate challenge from the server.  It uses the server nonce from that challenge...'](http://tools.ietf.org/html/rfc2617#page-13). This is something that is very difficult for our current authentication handler system to do, but appears to mean that we are severely limited in our ability to properly execute 'MD5-sess'.

Secondly, the `build_digest_header` method is now _massive_ and quite difficult to follow. I think we should be aiming to refactor this method, either as part of this pull-request or more generally.


I'll take a look at this later today
Cann't run request in crontab

I write a python script for sending post request, and add the script to crontab.
The code like this:

for i in range(10):
      requests.post(url, data=data, files=files)

When I run the script on shell, it working well. 
But run it in crontab, the loop just execute once, not 10 times.


Thanks for raising this issue!

Uh..that doesn't seem at all possible. Can you redirect stdout and stderr to a file and see if anything is being outputted?


@massifor do you have any more details for us?


Closing due to inactivity.
kbutton.org button added to README file to clone requests in Koding cloud.

It adds the kbutton to clone requests repository into Koding easily. You can find the details in http://kbutton.org

Thank you so much! :)
github: @f
twitter: @fkadev
Fatih Kadir Akin


I'm totally +0 on this, it's up to @kennethreitz. =)


@Lukasa Sure. It's just a request. :)


Is this necessary for users to use that site or is this just a way of promoting that site? If the former then I'm +0.5 otherwise -1


@fka This is really cool! I think I'll hold off on it for now, though. :)


Thanks @kennethreitz :) It's up to you.
bad authentication with Twitter and update_with_media

Hi guys, thank you for this great library!

I'm trying to publish a tweet with a picture in it, by calling update_with_media, without success. This is my code so far:

``` python
import requests_oauthlib
import requests
auth = requests_oauthlib.OAuth1(YOUR_APP_KEY, YOUR_APP_SECRET, USER_OAUTH_TOKEN, USER_OAUTH_TOKEN_SECRET)
req = requests.request('POST', 'https://api.twitter.com/1.1/statuses/update_with_media.json', data={'status':'test'}, files={'media[]' : open('/tmp/tmp.jpg', 'rb').read()})
print req.json() # error!
```

and the result is:

```
{u'errors': [{u'message': u'Bad Authentication data', u'code': 215}]}
```

That's not true, because I can call any other API's end point, both via POST and GET. For example, I can tweet a text-only message:

``` python
import requests_oauthlib
import requests
auth = requests_oauthlib.OAuth1(YOUR_APP_KEY, YOUR_APP_SECRET, USER_OAUTH_TOKEN, USER_OAUTH_TOKEN_SECRET)
req = requests.request('POST', 'https://api.twitter.com/1.1/statuses/update.json', data={'status':'test'})
print req.json() # ok!
```

This is what I found reading some Twitter docs and support pages:

```
Parameters being sent to the API that are not the oauth_* named parameters do not become part of the OAuth signature base string when the HTTP method being used is POST but the post body is not of the "Content-Type: x-www-form-urlencoded" variety.

Since you're using multipart POST, you're not sending a POST body with the Content-Type set as x-www-form-urlencoded and it's therefore erroneous to include parameters like "status" or "media[]" in the OAuth signature base string.

Some implementations of OAuth do not account for this gotcha.
```

I sense there's something wrong with the headers or the body: maybe a missing parameter in my request() call? 

Requests version: 1.2.3
Python version: 2.7.3


One thing I noticed is that you're never using the auth instance you create. Is this just a mistake in your example code or are you actually never using it? On top of that, OAuth is only supposed to sign certain kinds of data if I remember correctly so and multipart form-data is not one of those kinds.

Can I ask why you aren't using something like Twython for this? It should handle properly setting up the authentication for you and posting the data correctly.


Twython do exactly this and use Requests + requests-oauthlib to do it. I imagine the problem is not using the auth object. =)


OK this is an absolute facepalm: I left out the auth object since the beginning... Twython: I will definitely check it out. I'm very sorry, please forgive the noise :-)


No worries @monocasual 

Also, @Lukasa sometimes you have to ask the obvioius question
Use MD5-sess authentication got  "UnboundLocalError: local variable 'hash_utf8' referenced before assignment"

Error Message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, *_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 360, in request
    resp = self.send(prep, *_send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 468, in send
    r = dispatch_hook('response', hooks, r, *_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/hooks.py", line 41, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/auth.py", line 163, in handle_401

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/auth.py", line 96, in build_digest_header
    KD = lambda s, d: hash_utf8("%s:%s" % (s, d))
UnboundLocalError: local variable 'hash_utf8' referenced before assignment


I just saw this is not implemented yet, so please ignore.


I don't think we should ignore this, I think we should fix it. =) I'll try to take a run at it sometime soon.


I think it's working correctly when i tested with md5.
In my opinion, that UnboundLocalError only happens when 'www-authenticate' header has another hashing algorithm (not md5, sha1)
@linuxhngg Are you trying to use 'MD5' digest authentication?


Here is the header I got as below:
(Pdb) res.headers
CaseInsensitiveDict({'transfer-encoding': 'chunked', 'set-cookie': 'MIEP_digest_session=186eec93ce4429853ca5e68e4a546b63; Path=/', 'vary': 'Accept-Encoding', 'www-authenticate': 'Digest realm="iptv-portal", qop="auth", nonce="888c6a6441d8791165887559aaf96b5f", opaque="186eec93ce4429853ca5e68e4a546b63", algorithm=MD5-sess, domain="private domains now showing here"'})

I removed the domain information.  

What I did in curl as below and it is working:

curl -vvv -c 1tmpcookies.txt --digest -u xxxxx:yyyyy http://172.16.73.49(private_network)/iap-dataapi/private/bc/channels?X-Claimed-UserId=xxxxx -o channels.json


@linuxhngg 
oh.. there is no way to parse 'MD5-sess' in current DigestAuthenticator.
According to http standard, 'MD5-sess' is valid algorithm name.
I will attach quick fix on it.


wow ~ sweet! must try it now.
thanks a lot for your help on this.


it is working great! thanks a lot for this quick fix.
Added migrating to 2.x docs

In a similar vein to the migrating to 1.x docs, this section details changes that can break existing code in order to ease the migration to 2.x. I didn't talk about any of the added APIs since those don't really break existing code. Compared with the last major release, there's not much to say.

This is largely based on @Lukasa's [blog post](http://lukasa.co.uk/2013/09/Requests_20/) and the [changelog](https://github.com/kennethreitz/requests/blob/master/HISTORY.rst). I did not know how folks feel about linking to a blog post from the docs so I didn't add the link.


:cake:


@davidfischer I think the link would be fantastic, actually. Want to add it?


@kennethreitz linked!


:100: 


This is awesome @davidfischer, thanks! =D :cake:
requests doesn't fill CookieJar

First of all, i cannot use session to manage cookies across many requests. 

Based on this answer http://stackoverflow.com/questions/6878418/putting-a-cookie-in-a-cookiejar, requests should fill CookieJar object

jar = cookielib.CookieJar()
r = requests.get(URL, cookies=jar)

But it doesn't work for me.

Is this a bug or not?

> jar = cookielib.CookieJar()
> r = requests.get('http://192.168.226.141/test.php', cookies=jar)
> r.cookies
> <<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='test1', value='1', port=None, port_specified=False, domain='192.168.226.141', domain_specified=False, domain_initial_dot=False, path='/', path_specified=False, secure=False, expires=1380060661, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>
> jar
> <cookielib.CookieJar[]>


Why can't you use a session?


To answer the question:

That answer is two years old. Today marked the release of the second major API breaking version release. Answers from two years prior are not even remotely applicable to today's versions of requests. In other words, no that is not still possible and it is an intentional design (if I remember correctly), i.e., not a bug.

I'd also like to know why you can't use a session.

Third, when asking questions, why not use [StackOverflow](http://stackoverflow.com/questsions/tagged/python-requests)?

Finally why can you not use the cookies returned on the response?


Sorry. Thanks!
I will use cookies returned on the response
Encountering off-host redirects using allow_redirects=True doesn't handle authentication sanely

Problem:

A host which requires authentication redirects off-hostname. Using `allow_redirects=True` will then send the target the current authentication headers. This is almost certainly wrong behaviour.

Probably a solution:
- Simple: Just reset the authentication when encountering an off-hostname redirect.
- Complex: Introduce a dict of authentication objects keyed on hostname. May break or introduce new API functionality though.


Thanks for raising this issue!

I seem to recall we discussed this in the past, and decided that what we do here is a user-level issue. Some users may absolutely want their authentication headers to redirect off-host. Does that sound right to you @kennethreitz?


Ah, here we go. Original discussion on #975, relevant bits start [here](https://github.com/kennethreitz/requests/issues/975#issuecomment-12616989). Never mind Kenneth, you can ignore this. =)


Yeah I was of a similar opinion @smokey42 but I agree with the decision in the issue linkes by @Lukasa. It allows for the most degrees of freedom.


Fair enough, yet to quote PEP 20: "Explicit is better than implicit."

It was undocumented, unintuitive behavior and also took me a while to figure out what was going wrong. I had to enable urllib3 debugging to see the request headers.

I don't have any objections this being the default behavior, but it should be either documented or settable. Both would make it explicit.

Thanks.


Adding docs is easy and if you'd like you can send a PR with that update to make sure it gets done.
Incorrect key types in request headers (python3)

Request header keys are being encoded and then checked for later. In python 3 this is causing issues. 

In an example use case, if `Content-Type` is explicitly set, when making a post request, you end up getting both the user-set type and the default type, eg: `application/x-www-form-urlencoded`.

https://github.com/kennethreitz/requests/blob/master/requests/models.py#L364

``` py
headers = dict((name.encode('ascii'), value) for name, value in headers.items())
```

https://github.com/kennethreitz/requests/blob/master/requests/models.py#L417

``` py
# Add content-type if it wasn't explicitly provided.
if (content_type) and (not 'content-type' in self.headers):
            self.headers['Content-Type'] = content_type
```


Thanks for raising this issue!

This is a well known bug. We've had a fix for this for a very long time (more than 7 months), which has been scheduled for our next major release (2.0). This is planned to be our next release. In the meantime, the current `2.0` branch includes that change and many others. =)

Additionally, in future can you please do a quick GitHub issue search for your issue? If you had, you'd have found the 5 or 6 duplicates of this issue that have been raised in the last two months as well as the 7-month-old pull request that fixed it. =)


@Lukasa my searches were bad! Thanks for the quick response.
Update urllib3 to b4391c1

urllib3 recently added support for https proxies (https://github.com/shazow/urllib3/pull/170). This has been a blocking issue for our us of the requests library.

All tests pass, and no backwards incompatible changes are noted in the urllib3 docs.


Thanks for raising this!

These changes are actually already available in the `2.0` branch. We're just as excited to have them merged as you are! The 2.0 release is intended to be our next release, but if you can't wait until then you should feel free to use Requests as it stands in the `2.0` branch. We've got a set of fairly up-to-date release notes for `2.0` in #1541.
Multi-part file encoding requires types that allow buffer interface

"TypeError: 'long' does not have the buffer interface" fixed for multi-part file encoding.


Your tests are failing on Python 3. Could you fix them please?


@sanmayaj do I understand correctly that you expect requests to string-ify everything you send in? Why can you not do this yourself on your inputs?

> Be liberal in what you receive and conservative in what you send.

You should be assuming that everything is a string (or byte) that leaves your computer via requests. This includes headers too. Personally I'm -1 on this. Up to @kennethreitz and @Lukasa though


Sorry, I should have spotted this, but we fixed it already! Merged into the 2.0 branch as PR #1537. @sanmayaj Can you try the 2.0 branch and confirm that it works fine for your use case?
Specifying the source_address parameter for HTTP and HTTPS connections.

Add a possibility specify the source_address parameter.
HTTPConnection and HTTPSConnection support this parameter since Python 2.7 and Python 3.2.
This is necessary when one server has multiple ip.


Thanks for raising this issue!

There's no particular interest in providing this parameter on the functional API at this time. By and large we expect that this behaviour will be controlled via [Transport Adapters](http://docs.python-requests.org/en/latest/user/advanced/#transport-adapters). For an example of how Transport Adapters are implemented, you can see [this example](https://lukasa.co.uk/2012/12/Writing_A_Transport_Adapter/).


Thanks for answer.

Add this functional is not difficult, small code writing and not break current API.  Instead, writing own Transport Adapter is huge effort for minimal changes in current mechanism. I think this is same case when it should be realize inside the request.
After all, it's only require add one parameter which pass through inside to [HTTPConnection](https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L206) and [HTTPSConnection](https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L557) and this is just enough.


Oh, I guess I understood all complexity. Yes, this is not that trivial than I suggested before.


The question isn't one of triviality. Instead, it's about API cleanliness and separation of concerns. Our primary functional API confines itself to talking about HTTP, and HTTP alone. The notion of 'source address' is a connection-level issue, and so really belongs at the transport-adapter layer.


Thanks again, it issue has become clear for me now.
Add copyright header to each source files to pass Debian's licensecheck.pl script

It is a side effect of using requests in the Chromium project.

We have over 100 third party libraries in the codebase. While we audit all of their licenses, it's also helpful for Open Source Distributions like Debian and others to at least semi-automatically see what licenses apply to our 100+ libraries. For a team of volunteers this is a pretty big task, and having license headers in each file is helpful. Note that it doesn't need to be a full text of the license. For example in Chromium project we use this:

// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

Still, it's preferred to use a full header if possible to help avoid any ambiguity. When people copy your library around, and bundle it as part of other third party project (the nesting can be 1 or even 2 levels deep), finding the right scope of a LICENSE file can be tricky, especially if it's missing (fortunately not the case here, but in general nothing is guaranteed). And then is it LICENSE, LICENSE.txt, COPYING, COPYING.txt and so on - it's trivial for people, but becomes increasingly non-trivial to automate.

Also see the following for recommendations (not all directly related to BSD, but still):

http://producingoss.com/en/license-quickstart.html

"The standard way to do this is to put the full license text in a file called COPYING (or LICENSE) included with the source code, and then put a short notice in a comment at the top of each source file, naming the copyright date, holder, and license, and saying where to find the full text of the license."

http://www.gnu.org/licenses/gpl-howto.html

"This statement should go near the beginning of every source file, close to the copyright notices."

http://www.mozilla.org/MPL/2.0/FAQ.html

"To apply the Mozilla Public License to software that you have written, add the header from Exhibit A of the license to each source code file in your project."


Thanks for raising this!

Yeah, this is something I'm aware of. I recall briefly discussing with Kenneth when he switched licenses to Apache, which includes a similar recommendation. The way I remember it, I'm pretty strongly opposed to putting the full license in each file, and the only person more opposed to it than me is Kenneth. =D

With that said, I'm definitely more open to using a shorter license-type string as you've suggested.

This is fundamentally Kenneth's call though. I consider this low priority for his attention, so I'm marking it as Needs BDFL Attention and leaving it open. =)


Frankly I've always found these annoying. They only really make sense when you have one source file and there's no real need to have a LICENSE file.


@maruel if you send a pull request, i'll merge it :)
status_code wrong

```
In [1]: import requests

In [2]: requests.__version__
Out[2]: '1.2.3'

In [3]: requests.get('http://mock.issues.com/404').status_code
Out[3]: 200

In [4]: requests.get('http://mock.issues.com/500').status_code
Out[4]: 200
```

mock.issues.com seems to be sending the correct HTTP status codes using other tools to check.

Am I doing something wrong, or is this broken?


@chadgh `mock.issues.com` is down, whereas `issues.com` isn't. Can you compare the results with `httpie` and `curl`. `http -h http://mock.issues.com/500`, `curl -i http://mock.issues.com/500`


Is `mock.issues.com` down, or nonexistent? I can't resolve the domain at all.


In fact @chadgh, if your ISP is anything like mine they'll find they can't resolve the DNS and then secretly point you to their own servers for advertising purposes. That would cause the behaviour you're seeing.


Also, `issues.com` appears to be being domain squatted. I'm reasonably convinced `mock.issues.com` does not exist as a real domain.


Fellows,

I'm sorry to have wasted your time. The url I was trying to use is mock.isssues.com looks like I messed the extra 's'. For future reference, it is a great site to use to test out making web service calls. I guess just don't forget the extra 's'.

Sorry again.


BTW, requests works great with the actual url. http://mock.isssues.com/500 gives a 500 error as expected.

thanks 


Not a problem, I'm glad you got it sorted out!
CaseInsensitiveDict problem with python3 result in duplicated Content-Type/Length headers

Headers are stored in a CaseInsensitiveDict but this class accept both bytes and str as key in python3. This result in duplicated headers when you want to provide Content-Type/Length by yourself.

When using python3 requests converts headers keys to bytes then set some headers with str as keys here https://github.com/kennethreitz/requests/blob/master/requests/models.py#L416 and here https://github.com/kennethreitz/requests/blob/master/requests/models.py#L422 

This result in the fact that you can't provide those headers by yourself. Requests should always check if the header exist first (both with bytes and str key)
Or at least replace the existing key (bytes not str)

Example:

```
>>> from requests.structures import CaseInsensitiveDict
>>> d = CaseInsensitiveDict()
>>> d[b'Content-Type'] = 'text/html'
>>> 'Content-Type' in d
False
>>> d['Content-Type'] = 'application/json'
>>> d
CaseInsensitiveDict({b'Content-Type': 'text/html', 'Content-Type': 'application/json'})
>>> 
```


Thanks for raising this issue!

We've known about this bug for some time, and had a fix ready for a long time (see #1338). This fix is a breaking API change, so we wanted to hold off on making it. However, it has been merged into the upcoming 2.0 release. 2.0 is intended to be the next release, so this should be fixed in the wild fairly soon. =)
expose urllib3's assert_hostname argument (v1 for 2.0)

This allows the user to control SSL CommonName verification.
For example,

``` python
requests.get('https://github.com/', assert_hostname='foo')
```

Will fail with an requests.exceptions.SSLError


Hi @borfig! Thanks for doing this work!

I have no idea if this'll get accepted or not. We're very reluctant to add new things to the Requests functional API at this stage, so this boils down to whether or not @kennethreitz believes this is a feature worth exposing at that level. I simply don't know what he'll decide. =)

Regardless of what Kenneth decides, the patch itself appears to be in good shape, so if he wants the feature this patch is a good solid implementation of it.


Hi, you are welcome :)

I have submitted the relevant urllib3 patch (shazow/urllib3@dfd9e0dc7ecbe4bb7abe723b5fd0e01688096d02) 3 months ago, but I still prefer to use requests.

The use-case of this feature is for testing purposes:
this allows us (@ravello) to test exact replicas of our production HTTPS service, without creating and signing certificates for each replica.


If this gets accepted, we could also add support for `assert_fingerprint`


And @t-8ch's suggestion is exactly why I'm kind of -1 on this. I would love extra conveniences for accessing some urllib3's extra features but I don't want us caught in parameter hell.

Would it make sense to consolidate a bunch of these options into a dictionary? 


I'm equally reluctant to having a dictionary of 'super magic urllib3 parameters'. If we're going to support something let's support it. Otherwise, it's still a Transport Adapter problem.


Once upon a time a `Verification`-object for urllib3 has been proposed, encapsulating all those options.
Unfortunately I don't have time for the next two weeks, else I'd love to implement this.
If this doesn't have to be sooner and nobody beats me to it I'll take a shot then.


this should only exist in the transport adapter, if at all
expose urllib3's assert_hostname argument

urllib3 has the assert_hostname argument to control verification of the Common Name of the server's SSL certificate
(shazow/urllib3@2493c9b3c0ce90b82209b93cad84296244e2763a), and this was also merged to requests (@2ed976ea7147a9d0c18998e02b16d691b6798a3e).
Therefore I would like to let the user use it.


1. We have been asking that all Pull Requests be pointed at the requests 2.0 branch
2. This is a major API change and as such should only be considered for 2.0. I'm not adverse to exposing this API, but I'm also not convinced this is necessary. I don't think we expose everything `urllib3` can do and so I don't see us as being obligated to expose this.
3. Regardless, @borfig if I were you I would close this, and open a new PR that instead chooses kennethreitz:2.0 as the base instead of kennethreitz:master.


You'll also want to rebase your commit onto the 2.0 branch.


ok. no problem.
(when will 2.0 be released?)


By the way, in your README.rst you still ask to PR to master...
urllib3 was update to1.7， and support connection pooling.

urllib3 was update to1.7， and support connection pooling.

you should update the request inline package->urllib3 (version was dev) to out inline.
just import from python-site-packages,  not import from request.packages.

​did you change the inline packge urllib3?​


Intentional design decision. Thanks, though.


could you explain why used this design ？ I want to fork and change it， but worry bugs.


Dependency management is still far from perfect in Python. If we vendor `urllib3` we can control and understand exactly which version is used and we can more easily debug user issues. Relying on pip would result in quite a few headaches for us. Also you should take note that `urllib3` itself vendors all of its dependencies.
Handle multiple qop types

In Digest Access Authentication there are two possible values (four if you count the not-present and both cases) for authentication. We were narrowly handling one of the four cases. Now we handle two.

Fixes #1601.

As for logic: I reordered the branches because if `qop is None` then we can not call `split` on it. So if we handle that case first, we shouldn't hit any exceptions. :)

I think I'll also be sending a PR to handle the `auth-int` `qop` value for DigestAuth. Is there a reason we never did do it?


As a side note: the Travis builds will continue to fail on this branch until someone (:eyes: @kennethreitz) deploys the latest version of httpbin. :wink:


Presumably because it's not commonly used: note that #1601 is the first issue we've had that mentions it.

This PR is awesome, as always I'm :+1:. =D


@Lukasa yeah that makes perfect sense actually (about `auth-int`). It [`auth-int`] looks (at a glance) like it is a variation on the `auth` algorithm. I would be perfectly willing to punt on this but at some point it looks like it was planned to be added: https://github.com/sigmavirus24/requests/commit/22e31b4b737c2a3b61b3ab4fccd534b2eee65a87#L0R126

Regardless #1601 will be fixed by this PR and since we haven't seen many complaints I would guess that `auth-int` isn't as widely used.
qop problem on HTTPDigestAuth

When Server responses 401 with qop in WWW-Authenticate header is : "auth,auth-int". Requests package send "None" as Authorization header. It's wrong.

```
www-authenticate: ...qop=auth,auth-int...
```

Requests package does correctly for:

```
www-authenticate: ...qop=auth...
```

from debuging, file <code>requests/auth.py</code>, line 109:

``` python
    if qop == 'auth':
        if nonce == self.last_nonce:
            self.nonce_count += 1
        else:
            self.nonce_count = 1

        ncvalue = '%08x' % self.nonce_count
        s = str(self.nonce_count).encode('utf-8')
        s += nonce.encode('utf-8')
        s += time.ctime().encode('utf-8')
        s += os.urandom(8)

        cnonce = (hashlib.sha1(s).hexdigest()[:16])
        noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, hash_utf8(A2))
        respdig = KD(hash_utf8(A1), noncebit)
    elif qop is None:
        respdig = KD(hash_utf8(A1), "%s:%s" % (nonce, hash_utf8(A2)))
    else:
        # XXX handle auth-int.
        return None
```

I saw that when you parse headers for qop argument, you did not split it into some option.

Did I misunderstand or something else?


Nice catch! Frankly I'm not certain how we should be handling the case with `qop=auth,auth-int` but I know where to look. :)

Thanks for reporting this! :+1: If it turns out not to be a bug on our side, I'll let you know.


I really made me in some troubles when I connected with others, but fortunately I can force the serve side to response qop=auth instead of qop=auth,auth-int. However, after that I did a check into document, qop=auth,auth-int is legal.

Thank you for very quick response.


Right, i was fairly certain it would be permitted. That is not what I'm checking. I want to see what the RFC says we should do and it doesn't seem to specify. It seems that this only means that we can respond with either the `auth` algorithm or the `auth-int` algorithm (which we don't currently handle). The fix for this is actually fairly trivial. I'll send a pull request for it in a couple minutes.


Yeah, when I wrote my RESTful client by PHP (we use both PHP & Python for our services) I ignored auth-int too. But I think you should give a warning in log when Server side specify 'auth-int' only (as you do not handle it now), it will be easier for developers to debug.


I may take a crack at implementing `auth-init` but not tonight. Tonight I catch up on sleep.

Thanks again for reporting this! :cake:


Thank you very much.

I currently install requests through pip, could you push your new update to PyPI?


This will be in 2.0. That doesn't have a set release date though it should be out "soon".
Request hook

Hey,

How do you feel about adding a `request` hook?

My use case would be to print the requested URL with its parameters and headers. It's not possible with the `response` hook.

Let me know what you think. If that sounds like a good idea, I'll implement it and send you a PR.

Best,

Chx


We had that and removed it intentionally. If you want to print what is being sent you can print them from the `request` attribute on the `Response` object, e.g.,

``` python
r = requests.get('https://httpbin.org/get')
print r.request.url
print r.request.headers
```


Nice! Should this be added to the doc?


Pretty sure it is in the docs that the request object has those properties and that the Response has the request property. Whether it is in (or belongs in) the quick start docs is probably what you mean. I'm pretty sure this is in the advanced section but can't easily context switch on my phone. I can check tomorrow or Sunday (when I get a chance).


Found it: http://docs.python-requests.org/en/latest/user/advanced/?highlight=request%20object#request-and-response-objects.
Use of mutable defaults in Request class

Incorrect use of mutable default dictionary in the Request class can lead to incorrect behaviour in advanced usage.

The mutable defaults are specified at:

https://github.com/kennethreitz/requests/blob/master/requests/models.py#L189

and should be None, not dict(). In order to maintain interface consistency it may be necessary to change the following lines to ensure data is set to a dict() if data is None, otherwise it will change to an empty list in the default case.

A demonstration of the flaw is:

```
>>> import requests
>>> s = requests.Session()
>>> r = requests.Request('GET','http://google.com/')
>>> r.data
{}
>>> r.data['foo'] = 'cat'
>>> r2 = requests.Request('GET','http://google.com/')
>>> r2.data
{'foo': 'cat'}
```


Heh, I briefly got totally misled by the appearance of `Session()` in your example. =D

Yeah, that absolutely looks wrong. We notably do this for everything else except those two parameters. I'll fix this up today. Thanks for raising this issue! :cake:


Resolved by #1598. Thanks for pointing this out! =D :cookie:
Automatically detect authentication

Would it be feasible for requests to check the `WWW-Authenticate` response header and send the appropriate authentication?

I'm trying to build a REST API client for a web service, but found that some of the servers where the service runs require basic auth, while others require digest.  I want to, if possible, make it so the user doesn't need to care which one it is.


We do not perform authentication for anyone automatically and there's a difference in how servers and APIs respond. The GitHub API does not normally return the header so it would be useless there. Several other prominent APIs do the same and for good reason. Beyond that you can do this for your user without having requests do it. In fact you have better knowledge of their needs than we do and we already meet our users needs. Is it something useful that might be appreciated by users? Yes. Are those users your typical requests users? Not entirely. Maybe some of them are but certainly not enough for me to like or approve of this idea. I'm :-1: on this personally. 


OK, fair enough.  I'll close this and handle it myself.