Raibek/requests-github-issues huggingface dataset:
HTTPDigest working with CURL but not Requests?

I am making a request using:

``` python
r = requests.get(url, auth=HTTPDigestAuth(username, password))
```

but this shoots back a 401. However, when I use the cURL command:

```
curl --digest -u username:password url
```

it works and I receive the correct response.

Is there a difference between cURL's "--digest" tag and the HTTPDigestAuth class?


Could you provide the version of requests you're using?


Additionally, can you provide `r.history` for your failed request?


Using pip freeze | grep requests shows that I am on 1.2.0.

My r.history returns "[<Response [401]>]".


Ok, can you get cURL to print out the headers it has sent, and also print out `r.request.headers`?


cUrl:

```
HTTP/1.1 401 Unauthorized
Path=/; HttpOnly
WWW-Authenticate: Digest realm="Contacts Realm via Digest Authentication"
qop="auth"
Content-Type= text/html; charset=utf-8

HTTP/1.1 200 OK
Path=/; HttpOnly
Vary: Accept-Encoding
```

r.request.headers:

```
Authorization :
    realm="Contacts Realm via Authentication"
    uri="/"
    qop=auth
    nc=00000001
Accept-Encoding : gzip, deflate, compress
Accept: "*/*"
User-Age : python-requests/1.2.0 CPython/2.7.3 CYGWIN_NT-6.1-WOW64/1.7.17(0.262/5/3)
```

Since the URL is third party I tried to remove anything incriminating in the headers. Let me know if you need something I may have omitted. I'm new to this but it appears that I get two responses using cURL?


You get two responses using Requests too: the difference is that they're a 401 followed by a 401. This is why I'm wondering if you're using an incorrect username/password combination: you seem not to be being logged in.


I apologize for not mentioning this sooner but part of the response HTML states:

```
<h1>HTTP Status 401 - Response realm name {0} does not match system realm name of {1}</h1>
```

Maybe that is telling of something?

I've triple checked the username and password with the one I use for the cURL request and it's the same one.


/me suspects it's this one: https://github.com/kennethreitz/requests/pull/1242/files (`Digest` is missing in request.headers).
Maybe the call to `pat.sub()` should specify `count=1` (Only guessing, not familiar with the code)


You're absolutely correct, thanks! It seems that one of my headers in the first response is:

```
'Digest realm="Contacts Realm via Digest Authentication"
```

and that pattern actually eats both Digests, when it should really look like:

```
'realm="Contacts Realm via Digest Authentication"
```

When I manually replace the 'realm' header and send the second response I receive a 200 reply.


Just confirmed it locally using apache. PR is submitted, should I also write a test? FYI httpbin doesn't verify the realm (its realm is hardcoded in fact)


I will close this tonight if everything checks out on the original example. Thanks for all of your help!
Fixing the adapter matching to be most-specific first.

While trying to create a custom adapter I noticed that it wasn't actually using my custom adapter, instead it was using the default adapter.

When the `get_adapter` is called there is no order for the matching so it was matching the default `https://` adapter even though I called `mount` with a `https://www.example.com`.

This change should make it always match the most specific path that matches a URL.

For example, the following mounted adapters should always match in this order:

https://www.example.com/sub/dir
https://www.example.com
https://


Looks like this might be a duplicate of #1327. Not sure method is better, I'll leave it up to you to decide :)


I'm personally in favor of #1327 because it doesn't require re-sorting and reversing the list of adapter prefixes every-time we make a request (we use `get_adapter` every time we call `request` which is every time we make a request). Sorry @Zoramite 


@sigmavirus24 no worries :) That is a better approach, I should have done a search for it before just blindly fixing it :)


Thanks for the request anyway @Zoramite ! Your other one looks good though. :cake:
Prevent a KeyError when there's no link headers

This nipped me in the bud with github3.py+github-cli. Previously, the `CaseInsensitiveDict` behaved like a `defaultdict` that returned None when the key wasn't there. Now the CID behaves like a regular dict and raises a `KeyError`. To dodge this on our attribute, we should use `.get` to avoid a `KeyError`.


This is an obvious :+1:. =D


:trollface:
Reverse HTTP?

I'm working with an HTTP-like server application that is able to upgrade a TCP connection (after client request) to send back event updates to the client according to: http://tools.ietf.org/id/draft-lentczner-rhttp-00.txt.

Is there a way to do this with Requests?


The short answer is no.

The long answer is still basically no. The core limitation is that Requests' default transport adapter is based on urllib3, and urllib3 doesn't support reverse HTTP.

Unfortunately, simply adding that support would not help. Requests is very closely tied to the client-server architecture, and serious refactoring would be needed to make this use-case behave sensibly. If you decided you wanted to implement this yourself in a Requests-y manner, a fork of Requests would be the way to go.

Thanks for opening the issue!
threaded_requests.GetThread, threaded_requests.DownloadThread

I've added classes that I've made to have requests on their own thread with variables and methods that allow the main thread to monitor their status.

This is my first attempt at contributing to anything like this.
I tried to keep it simple. I hope you like it.


This would definitely do much better as it's own separate package that requires requests. I can almost assure you that this will not be included in requests itself. Thanks for the idea though!


Thanks Sigmavirus24.  I'll figure out how to do that on git hub. I'll make a threaded_requests package with a sub-package wx_requests for wxPython ProgressDialog classes that monitor the request threads.

Cheers.
I use requests so often, I forget all of the hard work that went into it.

A reminder...


:+1:


@Lukasa concerns?


Heh, this is awesome. Not sure I appreciate printing this unconditionally, but otherwise nice.

Oh, and btw, `zlib.decompress` returns bytes on Python 3, so this throws an exception on Python 3. You might want to add a call to `.decode('ascii')`.


:+1: 


@jcarbaugh :cake:


I was just notified that this pull request violates a number of patents held by a very litigious company. Closing now. Thank you for your consideration.


Well done pull request
params and data are easily confused

The `params` argument refers to GET query string parameters and the `data` argument refers to POST parameters. However it's my experience that these two are easily mixed up.

See for example:

http://forum.twilio.com/twilio/topics/python_requests_lib_from_phone_number_is_required - The user believes he's sending the data in the post body, but instead he's attaching it to the URL as query parameters.

Or here:

https://github.com/kevinburke/twilio-jsonapi/commit/db30eea4513be30e40c9187a9d3d73b8df165e2d

where I assumed incorrectly that `data=data` would add query params to a GET but form values to a POST.

It's certainly legitimate to make a POST request with query parameters. It's unlikely though.

The reverse case seems like it's always a problem - attaching form values to a GET request, and it may be nice to warn in this case.


Thanks for raising this issue!

Respectfully, I disagree to both of your later points. Firstly, I use query parameters along with POST data in my Mozilla Persona implementation. Second, RFC 2616 does not prevent sending body data along with GET requests: it is perfectly valid, if highly unusual.

Requests absolutely should not prevent you doing either. =) Furthermore, Requests _definitely_ should not change the meaning of a function parameter based on the value of a different one.

I'm pretty sure that we're happy with the way the API is now. Thanks for raising the issue though! =)
Issue #749: Add optional SNI support for python2.

SNI support will be enabled for python2 if ndg-httpsclient and pyopenssl are available.


Is that seriously all that's needed to set up SNI support in it's totality?


@sigmavirus24: If it is that's awesome.


Yes. It's merely disabled in urllib3 by default due to the additional requirements and due to the fact that ssl is way more tested than pyopenssl.  That's why they've kept it in the `contrib` package, and needs to be manually enabled, but all the hard work is already done.
See [urllib#153](https://github.com/shazow/urllib3/pull/156) for more information.


Then I'm :+1: on this.


:cake:
Could not find a suitable SSL CA certificate bundle.

I get this error when I try to use requests

 File "C:\Program Files (x86)\Python\lib\site-packages\requests\models.py", line 562, in send
    raise Exception("Could not find a suitable SSL CA certificate bundle.")
Exception: Could not find a suitable SSL CA certificate bundle.

I'm running windows 7 and Python 2.7.3


What version of requests are you using? Because that stack seems to be weird.


I'm using version 0.13.0


Right, that's several versions older than we're currently developing on. The last stable version with the same API was v0.14.2, so if you don't want to change API I recommend upgrading to that. If you are able to work around some changes, I recommend upgrading to the latest version, v1.2.0. 


To be clear, I closed this issue because this is not a bug in a current version of Requests, so it doesn't belong on the GitHub issue tracker. However, if you want support, feel free to continue commenting here (I'll get emails) or to join us in the IRC channel and ask for support there.


Thanks for the help,  I'll grab 1.2.0 then.
Add support for HTTP TRACE method

Adding support for this underappreciated method.

Reference: http://tools.ietf.org/html/rfc2616#section-9.8


95% sure this will be rejected. We're currently in a feature freeze (#1165) so no changes to the API are allowed to be accepted unless they were previously planned. Sorry @maurycyp 


I see.  I wasn't aware of the feature freeze, I was just missing TRACE support when using requests in my projects.  Maybe this can be added to a future release?


I'm -1 on this as well, but for different reasons.

Basically, I've considered adding TRACE on multiple occassions before. The interface always felt weird, basically.
Refactor merge_kwargs

This refactors the `merge_kwargs` function for sessions to (hopefully) be clearer, and fix some bugs with it. It fixes #1321, as well as fixing an issue where arguments other than headers were being improperly merged case insensitively.


:+1: this makes the function easier to understand for future contributors and adds the flexibility we need for headers.


Can you do a rebase?


Sure thing, done.


:cake:
Remove 3.1 and 3.2 from setup.py, add tox.ini to simplify testing on multiple versions

This PR removes Python 3.1 and 3.2 from the trove classifiers in `setup.py` since @kennethreitz decided to drop support for those versions in the rejected PR #1326. 

It also introduces a rudimentary "tox.ini" which makes local tests with multiple Python versions trivial:

```
$ pip install tox
$ cd requests-sources/
$ tox
GLOB sdist-make: setup.py
py26 inst-nodeps: .tox/dist/requests-1.2.0.zip
py26 runtests: commands[0]
.........................................
Ran 41 tests in 24.071s
OK
py27 inst-nodeps: .tox/dist/requests-1.2.0.zip
py27 runtests: commands[0]
.........................................
Ran 41 tests in 23.269s
OK
py33 inst-nodeps: .tox/dist/requests-1.2.0.zip
py33 runtests: commands[0]
.........................................
Ran 41 tests in 22.674s
OK
________________ summary ________________
  py26: commands succeeded
  py27: commands succeeded
  py33: commands succeeded
  congratulations :)
```


I'll merge this once you remove tox :)

Tox used to be included in the repository a few years ago. It was removed for various reasons.


No problem, tox.ini removed. Can you point me at the reasons? It's worth knowing why it might be unwanted.


I love tox, I just found it to always need configuration modifications. 

The root directory of the project also already has too many files — tying to minimize that :)
Rewrite CaseInsensitiveDict to work correctly/sanely

Fixes #649 and #1329 by making Session.headers a CaseInsensitiveDict,
and fixing the implementation of CID. Credit for the brilliant idea
to map `lowercased_key -> (cased_key, mapped_value)` goes to
@gazpachoking, thanks a bunch.

Changes from original implementation of CaseInsensitiveDict:
1.  CID is rewritten as a subclass of `collections.MutableMapping`.
2.  CID remembers the case of the last-set key, but `__setitem__`
   and `__delitem__` will handle keys without respect to case.
3.  CID returns the key case as remembered for the `keys`, `items`,
   and `__iter__` methods.
4.  Query operations (`__getitem__` and `__contains__`) are done in
   a case-insensitive manner: `cid['foo']` and `cid['FOO']` will
   return the same value.
5.  The constructor as well as `update` and `__eq__` have undefined
   behavior when given multiple keys that have the same `lower()`.
6.  The new method `lower_items` is like `iteritems`, but keys are
   all lowercased.
7.  CID raises `KeyError` for `__getitem__` as normal dicts do. The
   old implementation returned `None`
8.  The `__repr__` now makes it obvious that it's not a normal dict.

See PR #1333 for the discussions that lead up to this implementation


I am a big, big fan of this Pull Request. :smile: Absolutely +1.


All I ask is that a doc-string be added to the new test in `TestRequests`, but otherwise :+1: x10. This is great work.

(Also brought this up on [IRC](https://botbot.me/freenode/python-requests/msg/2961988/))


:sparkles: :cake: :sparkles:
Replacing existing session headers is not done case-insensitively

When updating a requests.sessions.Session object's headers with a differently-cased existing header name, subsequent requests made with the session object include both the default and the wrongly-cased version. This is surprising considering the docs mention the session's header dict is case insensitive.

While I would normally just think "ok, i'll just use the same casing as the default key", the examples in the docs use lowercase header names. The session header behavior also differs from the behavior of headers when supplied in the request itself with the `headers` kwarg:

``` python
import requests
s = requests.Session()
# Update dict as suggested by the session docs
s.headers.update({'accept': 'application/json'})
s.get('http://localhost:8080')
```

The request as received by the server (note "accept" and "Accept"):

```
GET / HTTP/1.1
Host: localhost:8080
accept: application/json
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.2.0 CPython/2.7.3rc2 Linux/3.2.0-4-amd64
```

Versus:

``` python
headers = {'accept': 'application/json'}
requests.get('http://localhost:8080', headers=headers)
```

Received request:

```
GET / HTTP/1.1
Host: localhost:8080
Accept-Encoding: gzip, deflate, compress
Accept: application/json
User-Agent: python-requests/1.2.0 CPython/2.7.3rc2 Linux/3.2.0-4-amd64
```

Obviously the docs are misleading, but I happen to really like the behavior demonstrated in the second request. I'm not sure how to try to fix the behavior. Perhaps a add_header method? That seems ugly though.


Ok, so based on a suggestion from @Lukasa in IRC, I'm considering switching `.sessions.Session.headers` over to `.structures.CaseInsensitiveDict`, but it looks like its `.update` method doesn't do what I'd like. However, I don't really know what the sane behavior would be for edge cases where the input mapping has two keys that have the same `.lower()`...

This needs some extra thinking, I'll ponder it over the weekend.


I think the correct behaviour there is to throw a `ValueError`. We can't meaningfully merge the values: the user might want any of about three different things to happen and we don't know which, so I think we want to die.


I've made significant progress implementing CID as a wrapper around (not a subclass of) dict. All that remains to be done is flesh out the tests completely.

I'll take the suggestion to throw a ValueError under advisement. I noticed that the fix for #649 wasn't merged. My fix for this issue should fix that one as well. I also believe it will be a better solution for #1321.


So far: https://github.com/cdunklau/requests/commit/6e476452f11c18a1be61905d1f12ae2679aee124


Fixed in e6e9b55.


But the 'latest' [document](http://docs.python-requests.org/en/latest/user/quickstart/#response-headers) still says, 

```
If a header doesn’t exist in the Response, its value defaults to None:
```

I hope someone can update it...


Mm. Do we want that behaviour? I feel like we should bring the docs in line with the current dict-like behaviour.


:+1: for updating the docs to reflect the change. Sorry for not catching this.
Fixed tests failing on Python 3.2 

A test introduced in #1279 used the `u''` literal which is not present on Python 3.2 which made unit tests fail.

This PR fixes the bug and introduces a rudimentary "tox.ini" which makes local tests with multiple Python versions trivial:

```
$ pip install tox
$ cd requests-sources/
$ tox
GLOB sdist-make: setup.py
py26 inst-nodeps: .tox/dist/requests-1.2.0.zip
py26 runtests: commands[0]
.........................................
Ran 41 tests in 24.071s
OK
py27 inst-nodeps: .tox/dist/requests-1.2.0.zip
py27 runtests: commands[0]
.........................................
Ran 41 tests in 23.269s
OK
py32 inst-nodeps: .tox/dist/requests-1.2.0.zip
py32 runtests: commands[0]
.........................................
Ran 41 tests in 21.836s
OK
py33 inst-nodeps: .tox/dist/requests-1.2.0.zip
py33 runtests: commands[0]
.........................................
Ran 41 tests in 22.674s
OK
________________ summary ________________
  py26: commands succeeded
  py27: commands succeeded
  py32: commands succeeded
  py33: commands succeeded
  congratulations :)
```


Nice. =) :+1:


In related news, is the omission of Python 3.2 in `.travis.yml` deliberate? This is the reason the tests show as green while they actually fail under 3.2.

`setup.py` lists this Python version as compatible in the trove classifiers.


Not merging for one reason only: `u''` was added to Python 3.3 because of a tweet I made because of this exact issue. I intend to take advantage of this now that it's there :)


You should then remove 3.1 and 3.2 from trove classifiers for future requests releases.
Session with CookieJar/LWPCookieJar cause AttributeError

In 1.2.0 , 

```
>>> import cookielib
>>> import requests
>>> jar = cookielib.LWPCookieJar()
>>> s = requests.session()
>>> s.cookies = jar
>>> s.get('http://httpbin.org/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 369, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 297, in request
    merged_cookies = self.cookies.copy()
AttributeError: LWPCookieJar instance has no attribute 'copy'
```

Works in older version. (0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.1.0)


So the code which causes this break was introduced partially because of Transport Adapters and partially because the re-factor broke the old code. This is a far more elegant way of managing cookies (in my opinion) than we used to use and I'm really not 100% comfortable changing it back. However, this is a major break in the API (although entirely unintended and not exactly forseen) so I am guessing that @kennethreitz will be entirely in favor of finding a solution to return to the old behaviour.

This especially is a pain because @gazpachoking and I put a decent amount of effort into restoring cookie persistance and making it elegant. Perhaps he would be interested in tackling this? _Nudge, nudge_


Hmm, yeah, I wasn't planning for the case that `cookies` attribute of a session would not be a `RequestsCookieJar` or a dict. I'll check it out.


@gazpachoking seems to have fixed this.
Actually choose the best adapter match

So it turns out the documentation here has been wrong the whole time! Who knew?

This PR follows #1320, and selects the best matching prefix from the dict in a _really stupid_ way. The _right_ way to do this would be to use a trie, but I thought we probably didn't want the cost of a new structure, especially as it's unlikely a user will mount a large number of adapters to any one session.

Let me know if you think that's a bad idea, and I'll whip up a quick trie and add it to this PR.


@sigmavirus24: Changes made. =)


Closed based on the discussion in #1320.
Clear preset request headers case-insensitively

While request headers can be removed by setting them to `None`:

```
>>> r = requests.get('http://github.com', headers={'Accept-Encoding': None})
>>> 'Accept-Encoding' in r.request.headers
False
```

the following code gives an `KeyError` instead of doing the same job:

```
>>> # Remove preset header
>>> r = requests.get('http://github.com', headers={'accept-encoding': None})
Traceback (most recent call last):
  (...)
  File "..\requests\sessions.py", line 313, in request
    headers = merge_kwargs(headers, self.headers)
  File "..\requests\sessions.py", line 76, in merge_kwargs
    del kwargs[k]
  File "..\requests\packages\urllib3\packages\ordered_dict.py", line 59, in __delitem__
    dict_delitem(self, key)
KeyError: 'accept-encoding'
```

With this patch the key to remove can be specified case-insensitively.

```
>>> r = requests.get('http://github.com', headers={'accept-encoding': None})
>>> 'Accept-Encoding' in r.request.headers
False
```


Can you fix the test for python 3.3?


@sigmavirus24 Thank you. I will fix it ASAP.


@shaung when you fix it, ping me. GitHub won't tell me when you've added commits so don't be afraid to mention me in the commit message or just add a comment here. :)


@sigmavirus24 Fixed :-)


Well this looks good to me. :+1:


:+1: 


This looks great! Can you do a rebase so I can merge it in? 

Many thanks :)


@cdunklau is this fixed by your pull request?


@sigmavirus24 Just tested against the new master after my PR was merged, this issue is not fixed.


@kennethreitz Thanks! I'm going to do it tonight.


@kennethreitz  Ready for merging (=´ー｀)ノ


I think #1343 + #1339 will make this pull obsolete to a degree. The other issue is that previously we were merging the arguments in a case insensitive way which was only correct for headers. This is better handled by @gazpachoking's PR.
Issue of sending a HTTPS request use http proxy

I try to sending a HTTPS requests by http proxy.

the code is like

``` python
proxy = 'host:port'
proxies = {'https' : proxy}
r = requests.post('https://url', proxies=proxies, timeout=10)

But it just raises the error below, while I tested it OK when using pycurl.

Traceback (most recent call last):
  File "c:\python32\lib\site-packages\requests-1.2.0-py3.2.egg\requests\packages\urllib3\connectionpool.py", line 421, in urlopen
    body=body, headers=headers)
  File "c:\python32\lib\site-packages\requests-1.2.0-py3.2.egg\requests\packages\urllib3\connectionpool.py", line 273, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "c:\python32\lib\http\client.py", line 964, in request
    self._send_request(method, url, body, headers)
  File "c:\python32\lib\http\client.py", line 1002, in _send_request
    self.endheaders(body)
  File "c:\python32\lib\http\client.py", line 960, in endheaders
    self._send_output(message_body)
  File "c:\python32\lib\http\client.py", line 805, in _send_output
    self.send(msg)
  File "c:\python32\lib\http\client.py", line 743, in send
    self.connect()
  File "c:\python32\lib\site-packages\requests-1.2.0-py3.2.egg\requests\packages\urllib3\connectionpool.py", line 104, in connect
    ssl_version=resolved_ssl_version)
  File "c:\python32\lib\site-packages\requests-1.2.0-py3.2.egg\requests\packages\urllib3\util.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\python32\lib\ssl.py", line 181, in wrap_socket
    _context=self)
  File "c:\python32\lib\ssl.py", line 268, in __init__
    raise x
  File "c:\python32\lib\ssl.py", line 264, in __init__
    self.do_handshake()
  File "c:\python32\lib\ssl.py", line 443, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:376: The handshake operation timed out
```


Unfortunately, Requests is in an awkward place with proxies and HTTPS at the moment. You cannot use the HTTP CONNECT verb, which means the only kind of proxy you can use with HTTPS traffic is an SSL-terminating proxy like mitmproxy.

The specific bug here is that we're attempting to connect to your proxy and perform the SSL handshake, but the proxy is either not responding or taking too long to respond.

Until the CONNECT verb is implemented in urllib3 (see shazow/urllib3#170), you will not be able to do what you're attempting to do using Requests.

Sorry we can't do more to help you, and thanks for reporting the bug!
Don't reuse the same prepared request for all redirects

Updated #1275

Example:

``` python
import requests

resp = requests.get("http://httpbin.org/redirect/4")
for r in resp.history:
    print("url: {} request.url: {}".format(r.url, r.request.url))
```

Output:

```
url: http://httpbin.org/redirect/4 request.url: http://httpbin.org/redirect/4
url: http://httpbin.org/redirect/3 request.url: http://httpbin.org/get
url: http://httpbin.org/redirect/2 request.url: http://httpbin.org/get
url: http://httpbin.org/redirect/1 request.url: http://httpbin.org/get
```

Desired:

```
url: http://httpbin.org/redirect/4 request.url: http://httpbin.org/redirect/4
url: http://httpbin.org/redirect/3 request.url: http://httpbin.org/redirect/3
...
```


:+1:


Nice! Can you do a rebase so I can merge? 

Many thanks :)


@kennethreitz done. 

Let me know if something is wrong, I'm doing rebase first time.


:+1: on merging this before a new rebase is necessary


i missed it :(


Did a rebase again.


thanks!
AttributeError: 'HTTPResponse' object has no attribute 'fileno'

If I pass requests.get(url).raw into something that uses select, like subprocess, then I get AttributeError: 'HTTPResponse' object has no attribute 'fileno'

It looks like requests suffers from the problem fixed in urllib2 here:

   http://bugs.python.org/issue1327971

Is there a simple work around?

Thanks!


`HTTPResponse` is actualy from urllib3 in this case, not urllib2. The simple work around would be to check urllib3 for similar issues and if none exist (open or closed) open a new one to see if @shazow likes the idea of making the Responses usable via select. You might be better off (in the interim) looking into the Respones attributes for something that might allow you to use select. It won't be as elegant as doing `Response.raw` but it shouldn't be worse than `Response.raw.attribute`.
Change the method when it isn't already GET/HEAD

For some reason it was only change the method when a POST was being made. This
is almost certainly my fault.

Fixes #1303


cc @Lukasa for code-review


I pronounce this one line change: REVIEWED. =D


I was going to reflect on this for a while, but someone can always simply do `allow_redirects=False`, so....


:hatched_chick:
this can solve [Errno 185090050] when built to exe

[Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load
_cert_crl_file:system lib


I'm pretty sure this will break currently existing code, by requiring that cacert.pem be in the working directory. A more complete solution is needed.


just a temporarily solution

2013/4/13 Cory Benfield notifications@github.com

> I'm pretty sure this will break currently existing code, by requiring that
> cacert.pem be in the working directory. A more complete solution is needed.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1310#issuecomment-16328720
> .


No this will almost certainly break requests. `os.path.join` will use the current path of execution of the python binary to form the path so if you `pip install requests`, and then from some project directory try to use it, it will search in that directory for `cacert.pem` not in the directory requests was installed into.


Whoops, closed this by accident.


I would set the CURL_CA_BUNDLE environment variable when building an exe, myself. I think it should work with relative paths.


how to do in windows?

2013/4/14 Kenneth Reitz notifications@github.com

> I would set the CURL_CA_BUNDLE environment variable when building an exe,
> myself. I think it should work with relative paths.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1310#issuecomment-16338812
> .
Changed HTTPAdapter to allow max retries to be specified when initializing.

Currently, to bump the number of `max_retries`, you need to reach into the `HTTPAdapter` instance & manipulate `HTTPAdapter.max_retries`. The other pool options are passed, with sane defaults, to the constructor so it makes sense this should be configurable in the same breath.

Passes all tests & updated documentation. Should be 100% backward-compatible, since it's a new parameter that's not being passed anywhere else & is being default to the same constant as before.


I'm +0 on this. It adds no extra complexity.


I'm also +0 on this.


Well I'm :+1: to this.


There's no toast emoji!


This will have to do...

:bread: :fire:


It's roughly the same. :sake:
Fix Advanced docs Twitter Streaming example

I don't know how you feel about adding the dependency of `requests_oauthlib` in the documentation, but Twitter API v1.1 Streaming now requires OAuth Authentication rather than XAuth. I believe v1 is being blacked out sometime at the beginning of May.

(And yeah, I edited this fron the GitHub website... **judge** me :smile:)


I think if we're doing this the preamble needs to be updated to explain how to get `requests-oauthlib`.


+1 to this and @Lukasa 's suggestion


@Lukasa Good idea. I'm not too good at copy, I'd probably just confuse the hell out of somebody; so if anybody has suggestions as to what to update the preamble to, I'm open to suggestions :D


+1, i'll merge then


@kennethreitz Do _yoooou_ have suggestions on how to introduce `requests_oauthlib` into the docs :smirk: 


A new section here :) http://docs.python-requests.org/en/latest/user/authentication/#other-authentication


Alright! I'll _try_ and think of some copy that sounds like it was written while I was sipping wine and eating cheese squares that I've never heard of before ;D


@kennethreitz I've made changes to the Other Authentication section, let me know what you think!


Thanks so much! Merging :)
Make it possible to attach additional data to request

It would be useful to be able to attach additional data to request by passing these data to `send()` and other api functions like `get()` etc. We can think of these data as a context in which given request was sent so `context` seems like appropriate name for such a parameter. This is especially useful in async scenario but not only. See kennethreitz/grequests/issues/33 The idea is to have request store only these data for later retrieval by user - it wouldn't be used by Requests anywhere in the process of sending request.


This isn't necessary, especially given the fact that there's a perfectly good hook in that issue. The parameters to any of the methods (or to `request`) itself should deal solely with the request data sent or the conditions under which it must be sent (`verify`, `allow_redirects`, etc.). With that being the API, consistency is key and adding this would be wholly inconsistent.

The hook might be better written as

``` python
def set_context(context):
    def hook(resp, **data):
        resp.request.context = context
        return resp
    return hook
```


I agree.
Documents the actual logging methods #1297

- Documented the logging, requested in #1297
- Added build directory and *.egg to .gitignore
- Added sphinx as setup requirement in order to be able to build documentation with `pyhton setup.py build_sphinx`

modified:   .gitignore
modified:   docs/api.rst
modified:   setup.py


**Once fixed**, I'm :+1: on this. I was going to work on the docs (including this), this weekend anyway, but this is one less thing I have to do. :)


Thanks!
Set max_redirects per request

I expected this code works:

``` python
>>> import requests
>>> requests.__version__
'1.2.0'
>>> requests.get('http://www.google.com', max_redirects=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
TypeError: request() got an unexpected keyword argument 'max_redirects'
```

To use `max_redirects`, I have to do it:

``` python
>>> session = requests.Session()
>>> session.max_redirects = 1
>>> session.get('http://www.google.com')
<Response [200]>
```

It is not the obvious way.


Closing in deference to #1301 


@sigmavirus24 this issue is different than #1301. Here, I just expected the `max_redirects` work per request, because it is the obvious way. I don't want to create a `requests.Session` to be able to set `max_redirects`.


I think it's important to note that 'obvious' varies depending on where you stand. In this case, Requests seems to be aiming for API consistency. Anything that is a form of 'configuration' belongs on the `Session`. Anything that is intimately related to the sending of data belongs on the `HTTPAdapter`. Anything that is related intimately to the body of a HTTP request belongs in the request. Maximum number of redirects is configuration, and so belongs on the session. =)


To continue what @Lukasa explained, `max_retries` was an option pre 1.x but was explicitly removed with malice of forethought during the refactor. Really, when you make a specific request there are things that pertain to that specific request, e.g., certificate verification, whether that request should follow redirects, etc. but not how many times that request should be sent. A request is a single object. A browser may retry a request a maximum number of times, but a request is not a browser, a session is the closest thing to a browser you will ever find in requests (but it is not a browser).

That said, this is a feature request and combined with the fact this feature was already removed and that we're in a feature freeze this will remain closed.


Understood. The difference between Session and Request was not clear to me. Thanks for the explanation!


You're very welcome!


And thank _you_ for raising the issue. =) More people who are prepared to raise issues is always good.
Transport Adapter Docs

A documentation Pull Request, from @Lukasa? How unexpected! :grin: 

There comes a point where I should stop writing blog articles about Requests-y things and start actually adding some documentation to Requests. This will add the Transport Adapter to the API documentation and a description of them to the Advanced docs.


:space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: :space_invader: 

:video_game:
Add helpful error message to r.json() method

It took me a while to figure out that this error was coming from the newest version of Requests rather than somewhere in Django. This updates the documentation to be more control-F friendly. 


+1. I'm always in favour of better documentation.


I'm not sure this is the only error the json module raises but I'm generally +1 for this.


Good point @sigmavirus24. Fixed my wording.


Thanks very much. :+1: I was going to look to see if `json.decode` raised more than a ValueError but I'm too busy with school work so this is a good starting point. I also had plans of digging deep into the docs and checking the validity of them. In particular the hooks documentation, so if you'd like you're more than welcome to attack that too. :)


:sparkles: :cake: :sparkles:
MissingSchema error raised when schema present

After [this](https://github.com/kennethreitz/requests/commit/ca85b4ec6ce2428891d2e1ae72dad2b873c87bbe) commit, a `MissingSchema` error can be raised when schema is present, but network location isn't. Seems this should be handled another way.


Yeah, I guess `InvalidURL` would be better in this case. Sorry about that.
Don't set all session cookies on response.cookies

This prevents all the session cookies from being set in response.cookies. I believe the old behavior was a workaround due to how session cookies were being updated before #1290
After this, response.cookies will match up with the set-cookies header for the response.

As discussed here: https://botbot.me/freenode/python-requests/msg/2568299/


:sparkles: :cookie: :sparkles:


Mmm, cookies. :tongue: 


:cake:
fix #1287: Make sure expired cookies get removed from session.cookies

This changes sessions to update their cookiejar directly from the headers rather than the cookiejar attached to responses. This way cookies that the server marks as expired are removed from the session cookiejar.


This is absolutely fantastic and I can confirm it works with [this](https://gist.github.com/sigmavirus24/5314553).


And in case it wasn't obvious, +1000


:cake:
Add missing return_response parameter support according to the docstring...

.... This time with proper whitespaces.


Sorry for improper indent last time.


@KamilSzot why would you want the unprepared request in the first place? You can do this very easily on your own and that aside, you can not send an unprepared request to `Session#send` or the underlying adapter. The changes don't make any sense from that point of view. Beyond that, the ability to return the unsent request was **deliberately** removed during the refactor for the 1.x release


@KamilSzot would you mind closing this?


I don't mind you closing this if you don't want this feature.
Fix for the issue https://github.com/kennethreitz/requests/issues/1280

https://github.com/kennethreitz/requests/issues/1280


:metal:


@kennethreitz why reintroduce this behaviour? It was a documentation mistake on my part.


Woah why did you open this?


Oh, you guys have the same avatar. :(


Reverted.


So you won't reintroduce this behavior? Could you suggest what should I do instead to achieve same thing?
Unicode strings in multipart post requests

When attempting to post a file along with some other data including a unicode string (demonstrated in the added test case) I was running into the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in position 0: ordinal not in range(128)

I traced the issue back to `_encode_files` where `compat.builtin_str`, in my case `str`, was being applied to the unicode string causing the above error. I changed it to use `compat.str` which seems to have resolved the issue.

``` bash
> python --version
Python 2.7.2
```


Hi, and thanks for the bug report!

Unfortunately, you've run straight into Python 2's implicit string encoding/decoding, and your code changes don't resolve the problem: they simply move it somewhere else. In Requests, `str` is aliased to `unicode` in Requests on Python 2. This means your change will call `unicode` on all data passed in to the `data` parameter. If the user has done this:

``` python
import requests
r = requests.post('http://httpbin.org/post', data={'key': u'ëlïxr'.encode('utf-8')})
```

they will now get `UnicodeEncodeError`s. I think in this case the correct fix is for Requests to explicitly encode any unicode data as UTF-8 on Python 2.7, so that we only send bytes to urllib3.


Thanks Cory for your feedback and sorry for my lack of understanding around some of the issues with Python 2's string encoding/decoding...

What about modeling `_encode_files` handling of the data after `_encode_params` as in e7247ce since this issue does not seem to happen when just posting data instead of files and data or is this coming back to your point about moving the issue somewhere else?


No worries. =) Python 2's string encoding issues is one of the reasons Python 3 is backward incompatible: to make these issues more apparent. This has been tripping up developers for a very long time, so I'm not surprised you're not familiar with it.

This fix looks a lot better! This should resolve the original problem you had nicely. I think there are still a few ways to hit some Unicode errors, but we really need a proper issue that tackles them, because they're sitting in quite a few places in the library. I'm happy with this PR as-is.

+1 :+1: :cake:


:cake:


Excellent work folks, found this after isolating a problem we had with this today. Looking forward to a pypi release :)
Requests in history are overridden on redirect

Example:

``` python
import requests

resp = requests.get("http://httpbin.org/redirect/4")
for r in resp.history:
    print("url: {} request.url: {}".format(r.url, r.request.url))
```

Output:

```
url: http://httpbin.org/redirect/4 request.url: http://httpbin.org/redirect/4
url: http://httpbin.org/redirect/3 request.url: http://httpbin.org/get
url: http://httpbin.org/redirect/2 request.url: http://httpbin.org/get
url: http://httpbin.org/redirect/1 request.url: http://httpbin.org/get
```

Simplest solution is to copy `prepared_request` for each redirect. Also `prepared_request` can be created inside the while loop (maybe it's better?).


My sincere concern with this pull request is that you could have someone making a lot of requests to resources that have at least one redirect each. At that point their memory consumption will grow a lot faster than now.

I would be 100% ok going back to the original 1.x behaviour of only attaching the original prepared request, but I understand that's not what you're asking. This is, however, how requests worked even before 1.x. The final decision is, however, entirely Kenneth's call.


v1.1.0 doesn't have this issue. It uses `Session.request()` instead `Session.send()`, so new request is created for each redirect anyway.

With this new behaviour I have problems in `requests-cache` library, because I'm using `response.request` from `response.history` for cache key.


Ah, I see what the problem is. I think creating the prepared request would be fine inside the while loop. It reduces the cost of the import (yes importing is costly in terms of time).


It can't be merged now. I'll reopen.
API ratelimit timer

Can you please add a pre-request timer to the session class to comply with certain api ratelimit rules?

Requestobj.ratelimit = 1

Requestobj.get(url) # will wait one second before making the request


We're under a feature freeze (#1165) which means this won't be accepted and on top of that you can do the following:

``` python
import time


class MySession(requests.Session):
     def __init__(self):
         super(MySession, self).__init__()
         self.ratelimit = 0

     def request(*args):
         if self.ratelimit > 0:
            time.sleep(self.ratelimit)
         return super(MySession, self).request(*args)
```
Multipart files are only multipart/form-data

Unable to specify multipart/related or application/xop+xml when trying to send files with the file option.

If headers are passed in, no additional information is added to them based on the other options used.

Request:
If header options are passed in, use those for multipart files, and add the boundary information.  Currently I see no way to use SOAP with attachments using the multipart 


Requests only does the following encodings:
- multipart/form-data
- application/x-www-urlencoded

The rest is done by the user. The former is specified by using the files parameter as you tried here. We also do chunked encodings but for that we set the `Transfer-Encoding` header and not the `Content-Type` header. We've also debated this before and consistently come to the same conclusion.

Also, requests is in a Feature Freeze (#1165) so this won't be accepted for that reason as well.

Sorry
Response manipulation hook no longer in Session object after request?

Commit a721d590b4e13b641b3171a081e7d6a7b3d8b6f2 as applied to request/session.py removes the following (lines 297-299, after request call succeeds):

``` python
# Response manipulation hook.
self.response = dispatch_hook('response', hooks, resp)
```

At the moment, my code which relies on this response property in the Session object works fine with the current 1.1.0 release from PyPI but breaks with this commit. Is there a reason why this hook was removed? It presents a significant compatibility issue.


I've made pull request #1273 if you want to restore this response property to an instance of Session.


We removed it because when we asked if anyone was using it, no one replied in the affirmative.


Closing to centralize discussion on #1273
add 'max_redirects' to Session's __attrs__

This patch is to ensure the **proper serialization** with, like, the standard module _pickle_.

See  [Issue #1269](https://github.com/kennethreitz/requests/issues/1269) for specific potential error it may produce.

Thanks for reviewing ;)


Hey, this looks likely! Nice detective work. :+1:


@Lukasa 

Thanks ;)
Glad to be helpful.


:+1:


:cake:
One last pull request before 1.2 ideally

This fixes a few issues once and for all and I updated the HISTORY for the 1.2 release to include all the big changes.


@Lukasa if you have a chance, some code review would be great. Thanks


Some of your additions split comments from the relevant code. 


Comments from the diff, otherwise I'm +1 on this PR. Gotta make sure the tests pass too. =)


repoening for travis test


:metal:


I don't know how I missed that paren. -_- At least it was caught before 1.2