Raibek/requests-github-issues huggingface dataset:
Fix order of data+params

There's an important fix in the code that I found when Travis ran the tests after the first commit in this PR. It was a mistake I made in not keeping my naming consistent.

Related to #795


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2171155) (merged 6d418d91 into b130b980).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2171159) (merged 6d418d91 into b130b980).


Thanks!


You're welcome. Thanks for requests.
Fix ordering of data + params

Also there's an important fix in the code that I found when Travis ran the tests after the first commit. It was a mistake I made in not keeping my naming consistent.


Actually, I have some bizarre commits here, let me fix that and submit a new pr.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2171113) (merged 4fdcfe1b into b130b980).
Order of operations

So I realized in changing github3.py's download creation function that when @jkbr changed the encoding of files (`_encode_files`) he inserted the file tuples first. So making a call like:

``` python
from requests import post


r = post('http://httpbin.org/post', data=[('some', 'data'), ('some', 'more data')], files=[('file', open(__file__, 'rb'))])
```

Will actually have the structure (sent to urllib3) of

``` python
[('file', '<data from __file__>'), ('some', 'data'), ('some', 'more data')]
```

Whereas before that it would be:

``` python
[('some', 'data'), ('some', 'more data'), ('file', '<data from __file__>')]
```

And this changes the previously expected/exhibited behavior. I kept that construction assuming it had been approved, but I can't find that and I'm not sure if anyone was aware of this. I can change the ordering very easily, but I was wondering if I should or if this is ok.


The ordering change I made wasn't really intentional. I think it's a good idea to switch back to the original ordering for the sake of consistency.


Ah ok, I'll just wait for word from @kennethreitz before issuing the pull request.
Comma encoding in request params

I'm making requests against an API which takes comma separated parameter values.

A working request looks like this:

```
curl http://localhost:4444/endpoint?val=555,666
```

When I pass these parameters in with `requests.get()` however, the resulting URL looks like this: 

```
>>> r = requests.get('http://localhost:4444/endpoint/', params={'val': '555,666'})
>>> r.url
u'http://localhost:4444/endpoint/?val=555%2C666'
```

I can build the URL myself, but I'm wondering if this is the intended behavior?


Ehh...standards are hard.

The comma (along with some other characters) is defined in [RFC 3986](http://pretty-rfc.herokuapp.com/RFC3986#reserved) as a reserved character. This means the comma has defined meaning at various parts in a URL, and if it is not being used in that context it needs to be percent-encoded.

That said, the [query parameter](http://pretty-rfc.herokuapp.com/RFC3986#query) doesn't give the comma any special syntax, so in query parameters we probably shouldn't be encoding it. That said, it's not entirely Requests' fault: the parameters are encoded using [`urllib.urlencode()`](http://docs.python.org/library/urllib#urllib.urlencode), which is what is percent-encoding the query parameters.

This isn't easy to fix though, because some web services use `,` and some use `%2C`, and neither is wrong. You might just have to handle this encoding yourself.

@kennethreitz, can you think of anyone who would have a better insight into this problem?


If you are passing it in via `params`, we should be doing a full encoding. If you're adding it to the URL yourself, it should not be escaped.


Thanks for the quick response and clarification!


thanks.it cleared the mist in my brain

It seems a little inconsistent to encode the comma despite the standard allowing commas in that context, while a param set to a list is encoded like `?myparam=foo&myparam=bar`, which has different behavior in different languages/frameworks (some parse this as an array, some take the last value, etc) due to the standard not defining correct behavior there.

To prevent comma encoding, it's as simple as calling `urlencode` with kwarg `safe` set to "," if using python3. I'm sure a suitable polyfill to support python2 would be trivial to implement.

Obviously the default behavior of passing a list as a params dict entry value shouldn't change, but perhaps a configuration parameter could be passed to override this behavior?

If either/both of those endeavors get a maintainer's blessing, I'd be happy to whip up a pull request or two.
Typo fix in tests.

Addresses #792.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2163928) (merged a04775f8 into cb0fdd80).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2163932) (merged a04775f8 into cb0fdd80).


thanks!
Typo in tests/test_requests_ext.py

tests/test_requests_ext.py seems to contain a typo in RequestsTestSuite.test_cookies_on_redirects().
The last lines in this function are:

```
        # get cookies on another domain
        r2 = s.get(url='http://httpbin.org/cookies')
        # the cookie is not there
        self.assertTrue('preview' not in json.loads(r2.text)['cookies'])

        # this redirects to another domain, httpbin.org
        # cookies of the first domain should NOT be sent to the next one
        r3 = s.get(url='http://tinyurl.com/7zp3jnr')
        assert r3.url == 'http://httpbin.org/cookies'
        self.assertTrue('preview' not in json.loads(r2.text)['cookies'])
```

r2.text should be r3.text in last line.


Fix is in Pull Request #793.


Nice catch
Key val lists

Accept lists of `(key, val)` tuples everywhere.

This should close/solve issue #179. Unfortunately @jkbr had duplicated some of the effort so I had to remove his contributions there, but all the tests pass.

Also, I apologize in advance for the likely very ugly history. I wanted to keep up-to-date while working on this so there are a few (possibly unnecessary) merge commits.


Scratch that, python 3 is failing. 


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2163101) (merged f01694e2 into cb0fdd80).


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2163113) (merged f01694e2 into cb0fdd80).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2163403) (merged ab56e4a9 into cb0fdd80).


I need to rework how headers are dealt with in the Request object, but until then most of the rest of the parameters that only accepted dictionaries before now accept lists of `(key, value)` tuples. When I am certain that I have the headers logic fixed, I'll submit a separate pull request for that.


This is serious work, I'm very impressed. I'm sure Kenneth will be too. =)

Quick note: to avoid merge commits, use `git pull --rebase` to get the new commits without adding merge commits.


@kennethreitz @Lukasa should I start a new branch off of develop and cherry pick my commits on top of it to avoid the merges? Besides, one of the merges would not have been able to be automatic, so it would have required a merge commit anyway.


:sparkles: :cake: :sparkles:

Dude, this is excellent. Thank you.

---

I'll merge this if you change one thing: removing those newlines you added in the middle of a function call to follow PEP8. I hate those :)


@kennethreitz done.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2164620) (merged a5d462e1 into cb0fdd80).


<3
HTTPDigestAuth fixes

Fixes issues described in #788.

Also extended the digest test case to do two requests with digest auth and check the history length to confirm the second one is zero (ie no second 401).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2158977) (merged 6d020360 into cb0fdd80).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2158981) (merged 6d020360 into cb0fdd80).


This is beautiful. Thanks!
HTTPDigestAuth doesn't retain credentials

HTTPDigestAuth always tries to make a request first and only responds with an Authorization header when it receives a 401 (and then promptly forgets). This doesn't match my understanding of how it's supposed to work and doesn't match the behaviour of browsers eg Chrome.

The correct behaviour (I believe) is to store the credentials information the server gives you, and send Authorization on all subsequent requests. The old server nonce, etc is reused but the client creates a new cnonce and increments the nonce counter. If the server decides to change its nonce, it will send a new 401.

Draft fix is here: 5017aeb06c3893a1e500ae73d7d343b480a9c8c2

I'm storing the state in the HTTPDigestAuth object itself, which made some sort of sense to me. This means you have to reuse the same HTTPDigestAuth in subsequent requests to reuse the credentials.


Sorry, should mention that there are two commits here. The previous one is 53d454c8137e3c4863d7c1ecef8e3664f399f883 on which i based the one above.

The purpose of that was to get the digest handler to release the connection with the 401 before it reconnects. I was having deadlocks when forcing requests to only open a single outbound connection using the code below because the original connection was still open. The same technique is used by requests when handling redirects.

``` python
class MySession(requests.Session):
    def init_poolmanager(self):
            self.poolmanager = PoolManager(num_pools=2,maxsize=1,block=True)
```
"There many ..." to "There are many ..." in quickstart


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2150797) (merged 76f90054 into 4130cf9b).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2150801) (merged 76f90054 into 4130cf9b).


Thanks!
Lint fixes following PEP8.

These are generally only whitespace fixes. I excluded the `packages` directory for now. Let me know if you want that code cleaned up also. 

I can do other PEP8 fixes if interested; unused imports, one import per line for different modules, etc.

=)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2146435) (merged 42d0a216 into 4130cf9b).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2146439) (merged 42d0a216 into 4130cf9b).


Excluding `packages` is correct, those are brought in from their respective projects.


Thanks for the attention detail. This is fantastic :)
Allow lists in the dict values of the hooks argument

Currently the Request class has a .register_hook() method but it parses the dictionary it expects from it's hooks argument weirdly: the argument can only specify one hook function per hook.  If you pass in a list of hook functions per hook the code in Request.**init**() will wrap the list in a list which then fails when the hooks are consumed (since a list is not callable).  This is especially annoying since you can not use multiple hooks from a session.  The only way to get multiple hooks now is to create the request object without sending it, then call .register_hook() multiple times and then finally call .send().

This would all be much easier if Request.**init**() parsed the hooks parameter in a way that it accepts lists as it's values.


If anyone OKs this feature request, I'd be happy to dig into it.


@sigmavirus24 :+1:


Just need to make sure that the current workflow also continues to work with this change.


Once @kennethreitz has time to review #833, I'll start working on this. I have a feeling opening a branch for this would cause a merge conflict if I were to have two Pull Requests that are ignorant of each other for the same file. Could be wrong though. Also, I'm in no rush since I'm fairly busy and I know @kennethreitz is more busy than I am with conferences and whatnot. Just wanted to keep @flub updated.


I'm going to start work on this Friday at the earliest.
Change variable '_r' to 'r' for consistency.

I don't think there is a reason for the leading underscore in this section of documentation, but if there is, this pull request can be ignored, or used to clarify the documentation. 


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2145448) (merged ff55d737 into 4130cf9b).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2145452) (merged ff55d737 into 4130cf9b).


Ooh, this one I can explain!

The intent is that you can walk through the quickstart documentation and get the same results. The purpose of the underscore was so that the user wouldn't clobber the previous `r` variable when they reached this part of the documentation. You'll notice that after the block with `_r`, we immediately call `r.raise_for_status()`. If we'd used `r` in the section we used `_r`, the second `raise_for_status()` call would also throw an exception, which would be no good.

You can certainly use a different variable name than `_r`, but it shouldn't be `r`.


That's reasonable. My brain immediately thought "are they implying something about this instance being private?", which did not make sense. Want me to rename to something else, `bad_r`, or leave it be? 


I'm happy for it to be named something else, `bad_r` would do great. =) You're quite right, I did violate a Python convention with that name, so it _should_ be changed.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2145708) (merged b9d3ce2e into 4130cf9b).


Done! Let me know if that works for you.


I like it. =) I don't accept Pull Requests, that's Kenneth's job, but this should be good to go.

Thanks!


Your very welcome. On a different note, my sublimetext setup strips trailing whitespace, would you like me to do that for all the docs (in this same pull request, or a separate)?


There's no reason to keep it. By all means remove it as part of this pull request. =)


What's funny is, I started using SublimeText2 with the linter plugin due to Kenneth's blog post about it. =)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2145886) (merged b31a4399 into 4130cf9b).


Fantastic, thanks! :cake:
add .show() to Response

i find this very handy while figuring out how to call into an API (especially the ones made for humans). Thoughts on adding something like this into requests?


Obviously, whether this gets pulled is up to @kennethreitz, but I would suspect that answer will be that it won't be.

Requests is not intended for formatting or displaying data, but for obtaining it. Even the `Response.json` method only really exists as a courtesy, with any fault-tolerant JSON handling being done by the end user (see #691). Requests generally does not deal with the data beyond decoding the response and handing it back to the programmer.

As @kennethreitz [has said before](https://github.com/kennethreitz/requests/issues/769#issuecomment-7622419), Requests is an HTTP library, not anything else. This is **not** my decision, but I think it's likely that the answer will be that this pull request is not accepted. Don't take my word for it though, wait for Kenneth to look at it. I might be talking total rubbish. =)


haha i might as well close this then, thanks


It's totally up to you, but if I it were me I'd wait for Kenneth's word on it. This is his baby, not mine. =)


@doda @Lukasa I agree :)
dispatch_hook eats exceptions

I tried to signal failures in a hook via exceptions, but all I got was two tracebacks printed. This happens because `dispatch_hook` catches `Exception`.

https://github.com/kennethreitz/requests/blob/develop/requests/hooks.py#L50

It seems to me like that's a bad idea.


This is intentional, but is certainly work reconsidering now :)


Out of interest, what was the reasoning behind the decision?


I'm trying to recall :)

I believe I was under the impression that people would be writing all kinds of hooks and that if one failed, your request should still go through. For example, our cookie parsing used to be quite problematic, so instead of just halting when there was an issue with it, it would raise an error and carry on.

---

I see little value/purpose in this today, so I'm +1 on removing the suppression, unless anyone objects.


Fair enough.

Should be an easy fix. =)


@Lukasa want to send a pull request? :)


Happy to. Will save me from writing Objective-C as a bonus. =)


I have one here https://github.com/kennethreitz/requests/pull/811. Not sure how I could get github to link it here.


Just got to mention it in a comment, like this: #811. Looks good!


Thanks!
permissive implementation of iter_content

This allows iter_content and iter_lines to succeed without
crashing even after the response content has been fetched
(iter_content gives you a one-item iterator containing the
content).

It should mitigate the API break I introduced in #760 (as discussed on #775 and #597).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2139018) (merged 4095a9f3 into 4130cf9b).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2139022) (merged 4095a9f3 into 4130cf9b).


Hmm, this is an interesting fix. 

Why would we return everything at once instead of spitting up the response as if it wasn't downloaded already?


Hmm, as in, it would be better to split it into strings of length `chunk_size` (currently 1 byte by default)?

Basically, I like this for simplicity, and because generally when you specify a read size it is understood to be only a hint. (Although typically when your hint is not respected, it's because you're getting less data than you asked for, not more as in this patch.) Anyone who specifies a chunk size of, say, 10, and then expects to get exactly 10 bytes back each time is in trouble already, because the final segment of the page may have fewer than 10 bytes in it.

I'm not totally sure.


Thoughts? If you think `iter_content` should always return strings of length <= `chunk_size`, I can definitely do it that way.


@slingamn if you do that, I'll accept it :)
minor cleanup

Very minor change, but thought I'd submit anyway.  Figured 3 less LOC might be considered a contribution. :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2136498) (merged 025d0a73 into 4130cf9b).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2136502) (merged 025d0a73 into 4130cf9b).


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2180815) (merged cc5a6305 into 4889adce).


(closed/opened to retrigger build)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2256427) (merged c72071c1 into b9ecf456).
Make sure proxy exclusions work properly.

This should fix the issue in #776.

Any unit test for this problem requires either the existence of a proxy to test against or the using of some pretty gnarly test hooks. You want it anyway?


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2132558) (merged 00b4ca3b into 4130cf9b).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2132562) (merged 00b4ca3b into 4130cf9b).


Thanks!
POST file tests fail when run from byte code (.pyc)

Four test methods in `test_requests.pyc` fail:

```
test_POSTBIN_GET_POST_FILES
test_POSTBIN_GET_POST_FILES_STRINGS
test_POSTBIN_GET_POST_FILES_WITH_HEADERS
test_POSTBIN_GET_POST_FILES_WITH_PARAMS
```

due to http://httpbin.org/post returning 500. All 4 methods, at some point, send the running script (file) as the POST data. Tests pass when `test_requests.py` is run for the first time (and ascii sent), but fail the next and every other time, when python byte code (`test_requests.pyc`) exists and is run (binary file sent).

As it seems, the culprit is `httpbin` itself, which  fails when raw binary data is sent to the `/post` endpoint. I submitted the [patch](https://github.com/kennethreitz/httpbin/pull/60) that should fix this issue, but I'm also documenting it here, just in case someone else runs into the same problem.


Yeah I'd been seeing this occasionally so I got into the habit of just running `make clean && make travis`


HTTPbin is fixed for this now. 

In genreal though, I advocate the use of `PYTHONDONTWRITEBYTECODE`
no_proxy handling doesn't account for port in URL

On https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L556

``` python
if proxy and not any(map(_p.netloc.endswith, no_proxy)):
```

I don't believe `_p.netloc.endswith` is the correct way to check if the hostname is in the `$no_proxy` ENV variable (or `proxies` dict param). If the URL passed to requests contains a port, then `netloc` will look like `127.0.0.1:8000`, which will incorrectly fail to exempt `127.0.0.1` from the proxy.

I'd submit a pull request, but the test suite doesn't play nicely with the proxy I'm behind to begin with. It should be as simple as replacing:

`_p.netloc.endswith`

with:

`_p.netloc.split(':')[0].endswith`

But I don't know if you want to account for more complex scenarios.


Splitting on the colon is an accident waiting to happen: the [relevant RFC](http://pretty-rfc.herokuapp.com/RFC3986) seems to indicate that you are allowed additional colons in the `netloc` section returned by urlparse. It might be better to use the `urlparse(url).hostname` method on the original URL, e.g.:

``` python
>>> from urlparse import urlparse
>>> urlparse('//127.0.0.1:8000').hostname
'127.0.0.1'
```

There's probably a bit of a perf hit for hitting urlparse twice. Not too bad though.


@Lukasa It appears that `hostname` is already available on `_p`, from the previous `urlparse()`.

So `_p.hostname.endswith` should be the new callable to pass to `map`, in my opinion.


Good spot, you're quite right.

If you want to file a pull request, go ahead, TravisCI will run tests against it for you. If you don't want to, I'm sure someone will be happy to write it on your behalf.


Resolved by #779.
Content marked as consumed in 0.13.6

Content is immediately marked as consumed in 0.13.6, causing calls to e.g. response.iter_content() to throw an error.

Test code (tested with python 2.6):

```
import requests

r = requests.get('http://docs.python-requests.org/')
if r._content_consumed:
    print 'consumed'
else:
    print 'not consumed'
```

In 0.13.5 this prints:
not consumed

In 0.13.6 this prints:
consumed


This is a known bug in 0.13.6, and was resolved by #764. If you need immediately to fix the problem you can do one of the following: clone the development branch in its current state, use 0.13.6 with the patch from #764 applied, or wait until the next minor version is released, which will include the fix.


I think to some extent, this is an API change rather than a bug. In 0.13.6, the default value of prefetch changed from False to True. Thus, unless you explicitly pass `prefetch=False`, calls to `iter_content` will throw exceptions even in develop:

```
>>> import requests
>>> r = requests.get('http://docs.python-requests.org/')
>>> list(r.iter_content())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/models.py", line 752, in iter_content
    'The content for this response was already consumed'
RuntimeError: The content for this response was already consumed
```

The rationale for this change is at #520. Basically, without making this change, it was difficult to guarantee that Requests would not leak sockets.


Thanks, that helps a lot, downgrading to 0.13.5 and explicitly setting prefetch=False for the next release should work for me.


Thanks for understanding. I didn't anticipate this problem when I changed the default, unfortunately.

@kennethreitz maybe `iter_content` and `iter_lines` should still work after the content is fetched, for example, `iter_content` could return an iterator containing a single item, the entire response body?
Throw clear exceptions

I'm not super invested in this: I think the UnicodeError is reasonably clear, but it probably helps to throw a Requests based exception when there is an error in the URL. The InvalidURL exception subclasses ValueError, which means anyone catching this by looking for ValueErrors will still catch this exception.

That said, you might just want to keep throwing the Unicode Error.

(Inspired by me really wanting to close #697.)


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2087975) (merged 5b5cff77 into 64646182).


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2087975) (merged 5b5cff77 into 64646182).


@travisbot, you want to try testing this again? /poke


Oh it'll only test if there's no merge conflicts. Need to rebase :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2111792) (merged 79d53d3b into 27b55a74).


Yeah, damn straight it does @travisbot. Would have passed the first time too, if your VM hadn't broken.


ouch :)


=P Been a long day, taking my stress out on an inanimate bot seemed like the way to go.


:cake:
Tests: python2.6 compat (`assertIn` new in python2.7)

"test_requests_ext.py" should now be python 2.6 compatible (where `unittest.TestCase` does not have `assertIn` and `assertNotIn` methods).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2087478) (merged 4da47928 into 40ba6b4e).


thanks!


Heh, I thought Travis was covering Python 2.6 --- I wonder how these were passing?


Dunno, they were passing for me when I ran them with a 2.6 virtualenv too.


I'm not sure about Travis, but `unittest` in Python 2.6 _does not_ support `assertIn` (and many assert methods, see [docs](http://docs.python.org/library/unittest.html#unittest.TestCase.assertIn)). There is, of course the `unittest2` (a backport of new unittest features from Python 2.7 to older Pythons), but I don't think some assert-sugar (in this case) justifies requiring an additional package.


I think neither I nor @slingamn are saying this pull request is anything other than brilliant. We just don't understand how the tests were passing to begin with. I always test locally before I submit a pull request, including in 2.6, and never had the tests fail.

For me, this is just a curiosity, nothing more. =)


Just to satisfy the curiosity, here's why Travis didn't complain.. :-)
`Makefile:16`:

```
# test_requests_ext.py depends on external services, and async doesn't work under Python 3
# Travis/Jenkins should be ensuring that all other tests pass on all supported versions
CI_TESTS=$(shell find tests/ -name "*.py" ! -name "test_requests_ext.py" ! -name "test_requests_async.py")
```

Note: the `test_requests_ext.py` is excluded.


@radomir: @Lukasa is right, this change was great. I was just worried that the testing infrastructure for Requests was broken somehow. But `test_requests_ext.py` is intentionally excluded, so that explains it. Thanks for your contribution!
First pass at documenting encodings and RFC compliance.

My first run at dealing with #524 and #527. Let me know what you think of the changes and if you want me to take a different approach with some or all of it.

I've set the Compliance section up so that it is easily expanded if/when you want to document other compliance cases.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2086391) (merged 7a9419ce into 40ba6b4e).


Beautiful! I couldn't have written it better myself.

I'm merging now, I like it so much. Keep sending pull requests if you have additional changes ;)
add request.json

I use Requests to test a JSON over HTTP API - which I suppose is a pretty common usecase. Given that, I find the response.json property to be pretty convenient and would find a request.json property to be similarly useful (for put and post requests). A patch to do such a thing would be trivial, which I could provide. Prior to doing that though, I was wondering if you had any objection to such a patch?

So it'd go like:

``` Python
r = requests.post(...., body=json.dumps(....))
=> <response 400>
hmmmm, I wonder what I did wrong
r.request.json
=> {'nmae': 'tim', ....}
```

Actually, it'd be nice to be able to instantiate a request directly with json content i.e.

``` python
req = requests.post(...., json={'name': 'tim'})
as opposed to:
req = requests.post(....., body=json.dumps({'name': 'tim'}))
```


I've considered this heavily in the past, but decided to draw the line and not support this. Requests is a HTTP library, not anything else :)
Wrap socket.error in ConnectionError (+ unit tests)

On connection refused, requests recently started throwing lower-level socket.error exceptions instead of ConnectionError, which does not match the API docs and breaks existing code. I reported this here: https://github.com/kennethreitz/requests/issues/748

I believe the following commit introduced this bug: https://github.com/kennethreitz/requests/commit/e02fb2eb6c2acfa67793a2c77a3c7bd47998a024

This pull request simply catches socket.error in Request.send, and re-raises ConnectionError

It also adds two (now passing) unit tests, based on existing ones that are commented out (I left them untouched).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2069544) (merged 875c9e18 into bc7b2f9a).


A recent comment on #748 suggests that the underlying problem actually belongs to `urllib3`.


Thanks!
Accept objects with string representations as URLs.

This should provide the functionality requested in #423. This does _not_ implement schema-less URLs, and if we want that it should be filed as a separate issue.

I've tested with `furl` on Python 2.7, and this seems to work fine.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2066367) (merged 6166ba7e into bc7b2f9a).


Beautiful.
test and fix regressions from #760

This adds unit tests corresponding to the issues @dhagrow identified on #760, then fixes them.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2052399) (merged 000c1053 into a94fcb87).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2052525) (merged 291859c1 into a94fcb87).


Thanks :)
prefetch=False isn't respected for redirects

```
import requests
r = requests.get('http://httpbin.org/status/301', prefetch=False, allow_redirects=True)
next(r.iter_lines())

Traceback (most recent call last):
  File "<ipython-input-8-a2f7272b22ca>", line 1, in <module>
    next(r.iter_lines())
  File "python2.7/site-packages/requests/models.py", line 740, in iter_lines
    decode_unicode=decode_unicode):
  File "python2.7/site-packages/requests/models.py", line 712, in iter_content
    'The content for this response was already consumed'
RuntimeError: The content for this response was already consumed
```


Fixed by #764 !


Awesome :)
address connection leak issue from #520

Piggybacking on the recent upgrade of urllib3 inside Requests, here's my much-delayed shot at a real fix for #520. From the commit message:
- prefetch now defaults to True, ensuring that by default, sockets
  are returned to the urllib3 connection pool on request end
- sessions now have a close() method, notifying urllib3 to close pooled
  connections
- the module-level API, e.g., `requests.get('http://www.google.com')`,
  explicitly closes its session when finished

When prefetch is False, the open socket becomes part of the state of the
Response object, and it's the client's responsibility to read the whole
body, at which point the socket will be returned to the pool.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2044485) (merged 0caaf682 into 1898077f).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2044576) (merged 3c0b9404 into 1898077f).


The first Travis failure was due to a Python 3 issue, which I subsequently fixed.

Unrelated: `test_POSTBIN_GET_POST_FILES_WITH_PARAMS` is flaking locally for me, has anyone else seen this?


Beautiful work. :)

:sparkles: :cake: :sparkles:


RELEASED! :shipit:


This breaks prefetch=False in certain cases.

Redirect:

```
>>> res = requests.get('http://google.com', prefetch=False)
... next(res.iter_content(10))
Traceback (most recent call last):
RuntimeError: The content for this response was already consumed
```

return_response=False (unless prefetch=False is also passed to .get):

```
>>> req = requests.get('http://www.google.com', return_response=False)
... req.send(prefetch=False)
... next(req.response.iter_content(10))
Traceback (most recent call last):
RuntimeError: The content for this response was already consumed
```


Thanks for the report! I'll send a pull request including regression tests and fixes.
long type not accepted as a parameter when files provided

When we provide a file and a long-type parameter through to the request functions, we get an exception thrown.
urllib3 seems to turn integer types into strings, but doesn't convert python long types.
The exception is raised by urllib3's encode_multipart_formdata function.

Not sure if this is a urllib3 issue as in their function,  "# Backwards compatibility" is written next to their int conversion code. 
As that might be removed at some point, maybe integer/long conversions should not be relied upon within that library.

``` python
import requests
import tempfile

filename = tempfile.mkstemp()[1]
file1 = open(filename, 'w')
file1.write('test')
file1.close()
file1 = open(filename, 'rb')

response = requests.post(
        url='http://www.google.com',
        data={'aparam1': 12L},
        files={testfile': file1,}
        )
```


This _is_ a `urllib3` issue. It's also a stupid Python version problem.

In Python 2, the correct code would read:

``` python
if isinstance(data, (int, long)):
    # do type coercion
```

In Python 3, that's a NameError because the `long` type was removed. @shazow, does this fix look useful?

Whack this function definition into `util.py`:

``` python
import platform

def integer_types():
    if (platform.python_version_tuple[0] == '3'):
        return int
    else:
        return (int, long)
```

Then just replace the test with:

``` python
if isinstance(data, integer_types()):
    # do type coercion
```


Nope, @deadbeef404 is correct. You shouldn't depend on urllib3 to coerce your types for you. If we want ints, longs, and whatever else to be auto-coerced into strings, then please do that in Requests before passing it to urllib3.

I do plan to remove the backwards compatibility block, so please make sure everything will continue working in Requests. :)


Ok cool. =)


Resolved in #762.


Cheers for the quick turnaround. Also, I commend you all on the library's API. Very nice!
Develop

Fixes kennethreitz/requests#747.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2033776) (merged 2d321b22 into b5ca4b44).


hmm, looks like this fails on Python 3 because of your str import.


Sorry about that I thought I tested it on Python 3, but I obviously didn't. Let me try something else.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2035479) (merged f60579dd into b5ca4b44).


Excellent, thanks!
Allow a quote_plus option to the API, some sites give a 401 with default urlencode

Some sites give a 401 error when encoding url parameters and body data with the default urlencode function, as they send `+` instead of `%20` for encoding spaces.

An issue for this has been posted to [urllib](http://bugs.python.org/issue13866) but as it doesn't seem to be nowhere near to be accepted. I propose that an additional parameter can be added to requests API to replicate that feature.


Hi,

So how to use it ?

```
import requests

requests.post("https://www.google.fr", data={'q': 'test 1'}, quote_plus=True)
```

`
TypeError: request() got an unexpected keyword argument 'quote_plus'
`
  INSTALLED: 2.13.0 (latest)



The option does not exist: this was never added. We recommend mutating a preparedrequest object instead. 

@zen0x90 this was related to the pull request #757 which was no accepted. This feature wasn't accepted either. If you want to have your data `quote_plus`'d, you should do it yourself with [`quote_plus`](https://docs.python.org/3.5/library/urllib.parse.html#urllib.parse.quote_plus)

ok I see thank you.
Multipart filenames only sent if provided.

Before this change, `filename` parameter (in `Content-Disposition`) was sent
with every subpart in a multipart even if one was not provided.  This changes
request semantics, indicating to the server that the part is indeed a file.

This commit fixes #732.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2029683) (merged c50239b2 into 664880db).


@enginous if you pull develop and merge, your PR should pass unless there's a way to kick @travisbot to merge it against the current head.


There seem to be errors related to my change as well. I'll make sure to pull and merge after I've fixed those.


Ah, okay. I hadn't tried them, I just know that @travisbot doesn't 
always keep up.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2029758) (merged c50239b2 into 664880db).


It's breaking tests and it's dawned on me reading through the tests that this isn't actually solving anything. See #732.
Fix raise_for_status()

Found by @enginous https://github.com/kennethreitz/requests/commit/204649521bf8d165690b78a349d0f7c10d03bb99#requests/models.py-P27


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2029564) (merged b4bc76f8 into 664880db).


Thank you sir!


Not a problem.
file posting example looked wrong

From what I can tell from the source code (down thru urllib3) the key in the `files` dict is always the 'fieldname', while specifying a custom 'filename' is optional.

So in this example it was confusing to have a `fieldname` of 'report.xls' (the same as the filename) when all the other examples use a fieldname of 'file'.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2018277) (merged fdec88de into b4741853).


Great insight. Thanks. :)


thank _you_ for the awesome module!
requests.{post,patch} set Content-Type to application/x-www-form-urlencoded even when ``data`` arg is a string

the Requests documentation [clearly states](http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests),

> There are many times that you want to send data that is not form-encoded. If you pass in a `string` instead of a `dict`, that data will be posted directly.

i presume that it's safe to interpret "`string`" here as meaning "`str` or `unicode`" (minor documentation error).  i'm not sure what the string being "posted directly" is supposed to mean.  i suppose it should mean that the request's `Content-Type` is `text/plain`, since that's a reasonable interpretation of a generic string.  in any case, i'm quite sure that it's _not_ supposed to mean that the string is passed in as form-encoded data.  that would defeat the stated purpose of passing in a string instead of a `dict`.

unfortunately, passing a `str` as the `data` does in fact cause the request to have `Content-Type` `application/x-www-form-urlencoded`, just as in the `dict` case.

either this behavior is incorrect or the documentation is incorrect.  i'd prefer it to be the former.  the automatic conversion of a `dict` of parameters as form-encoded data seems to be specific to the case where `data` is a mapping (in fact, i'd prefer this special case to likewise interpret any `collections.Mapping` objects rather than just `dict`s).  when `data` is a string, on the other hand, it seems natural to interpret it as a plain-text string (`text/plain`) unless otherwise specified.  in other words, the particular media type of the request entity is application-dependent; automatically applying `application/x-www-form-urlencoded` is an act of incorrectly imposing one application's assumptions on another.  the current behavior leads to confusing bugs when a user forgets to specify the proper media type, Requests (incorrectly) applies `application/x-www-form-urlencoded`, and the receiving server subsequently parses the entity (incorrectly) as such.  `text/plain` has no inherent syntax, so an unsuspecting web server will not mangle it if the user forgets to specify the correct media type.


It's intentionally labelled as "string" because you can pass either byte strings or unicode. Depends on the version of Python and your use case.

We should not be setting that content-type header though, nice catch :)


I started working on this and wrote a regression test that exposes the problem, but I wanted to get your opinion on how you want this fixed @kennethreitz . My first question is do you want the content-type header to be text/plain or do you just want it unset? The second question is in how to fix it. The issue is that the compat.py rebinds str to be the unicode class so in models.py line 496 where there is a check for `instanceof(self.data, str)` where str is actually unicode and not the __builtin__.str. Should I just add in another isinstance check for the __builtin__ str type or do something else? Let me know how you would like this fixed and I can put in a pull request.


It should be unset.
Fixed encoding of fields with the same name, multipart/form-data fixes.

- Properly handle repeated data fields for multipart/form-data requests (#737)
- Allow a list of 2-tuples as the `files` agument.
- Consistently serialize lists a of parameters (#729).
- Use BytesIO for bytes.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1970172) (merged 1f7aeacb into 02dcc40a).


Looks good to me. I'm uncomfortable merging pull requests on @kennethreitz 's behalf, so I'll let him do it.

That TravisCI failure is worrying me, because it seems to come and go, and not be related to the contents of a particular build. /pensiveface


Yeah, It looks like Travis or httpbin.org is having hiccups:

```
HTTPError: 503 Server Error: Service Unavailable
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): httpbin.org
requests.packages.urllib3.connectionpool: DEBUG: "GET /redirect/4 HTTP/1.1" 503 601
49--------------------- >> end captured logging << ------------------
```

I was running the tests locally and got some 5xx's as well but they seemed unrelated to my changes and also happened with the stable requests release, but still it would be good to have a successful run of the tests.


What bothers me more is not that these issues are happening, but that on Travis running Python 3.2 the actual cause of the error is totally obscured by some exceptions thrown in `nose` or `traceback`. If I could at least see for sure that they're 5xx's I'd be happier ignoring the failure.


@Lukasa if you edit the verbosity of the tests in a local branch of yours and set up Travis with it, you should be able to get more information on your own from Travis.


Ah, good idea. =)

If they're 5xx's though, it's probably on httpbin.org's side (aka. it's @kennethreitz 's problem, not mine. =D ).


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1988839) (merged 181d8967 into 02dcc40a).


Any feedback on the changes?


Hmm, looks like your branch is a bit stale — would you mind updating it?


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/2088411) (merged dee3693e into 64646182).


Sure, rebased against upstream. All tests pass when run locally.


Not on Python 3 :) 


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/2092783) (merged 88c762e2 into 64646182).


Woo!


Thanks for the awesome work :sparkles: :cake: :sparkles:


Ups, fixed now. Got confused by the strange Travis log.

BTW, I have good experience with [Tox](http://tox.testrun.org/latest/) ([example config](https://github.com/jkbr/httpie/blob/master/tox.ini)). It makes it very easy to run tests on multiple Pythons also locally.
Set content type before authentication

AWS S3 authentication adds content type header (when it exist) to
canonical string that is signed. Since it is set after authentication
is done authentication on S3 fails


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1968492) (merged 5e175ef2 into 02dcc40a).


A little more context: https://github.com/tax/python-requests-aws this is the authentication backend. 


Awesome, thanks!
Case for OAuth with multipart/form-data signing parameters when files are present

Not 100% sure if there is a wider use case for this, but working with the tumblr API (and I believe others, will investigate further soon) submitting a post request with an attached file, as well as other data, requires that the data dict be included in the signature.

The following code does not sign the data dict at all as it removes it from the request.

``` python
if r.files:
    # Omit body data in the signing and since it will always
    # be empty (cant add paras to body if multipart) and we wish
    # to preserve body.
    r.headers['Content-Type'] = 'multipart/form-encoded'
    r.url, r.headers, _ = self.client.sign(
        unicode(r.full_url), unicode(r.method), None, r.headers)
```

I can confirm that if I bypass the check for files, the signature will include the data dict (but not the attached files), and produce the needed signed request.

Is Tumblr out of spec for this, or could it be that the check for files is not needed?


@idan @dgouldin @gulopine?


Interesting. Dropbox puts their parameters in the QueryString, even for a file PUT. Of course, they also use a plaintext signature, which doesn't take the parameters into account anyway.
SSL certs aren't working with https://graph.facebook.com in Mac OS

I hope it's not something I'm doing wrong but...

> import requests
> requests.get('https://graph.facebook.com')

and I get this:

> requests.exceptions.SSLError: [Errno 1] _ssl.c:480: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

in line 604 of requests/models.py

My cert_loc seems to be going with the DEFAULT_CA_BUNDLE_PATH, which is certifi/cacert.pem.

I made sure that I have the latest certifi, too.  Anyone have any ideas?


Usually an OpenSSL version difference.


Looks like I was on 0.9.8r. I just upgraded to 1.0.1c and I'm still getting the same error.


Please make sure you're using the latest version of Requests as well.


Closing until further reports emerge.


I'm having exactly this issue on MacOSX 10.8.3 with OpenSSL 0.9.8r Feb 2011. My Windows 7 machine will connect and verify just fine, but the Mac throws the SSL3_GET_SERVER_CERTIFICATE:certificate verify failed error as listed above.
This is using 1.2.0 of requests, Python 2.7.2 (On Mac)/Python 2.7 (on Windows).


What URL is causing it? I have the same setup, so I'm interested to see if I can reproduce it.


connection = requests.get(u'https://forums.eveonline.com', verify=True, stream=True)


Hmm, works fine for me. What version of Requests are you using?


For me:

``` python
>>> import requests 
>>> r = requests.get('https://forums.eveonline.com', verify=True, stream=True)
>>> r
<Response [200]>
>>> q = requests.get('https://graph.facebook.com')
>>> q
<Response [400]>
```

On linux with python 2.6 and python 3.2. I think python 2.7.4 (which was recently released) contained some corrections to the ssl module, could you try installing that?


For me the same as for sigmavirus24 with py26,27,31 and 32.
What response does the following give you?

``` sh
openssl s_client -connect forums.eveonline.com:443  -CAfile `python -c 'import requests; print(requests.certs.where())'`
```

(Note the backticks)

The commandline version has much better error reporting than pythons' version.