Raibek/requests-github-issues huggingface dataset:
Documentation is wrong: Response has no attribute 'text'

The very first example on the Website and in the README.md produces an error because Response has no attribute 'text'. Maybe you meant 'content'?

But there is another mistake in that example – the github API call returns nothing. See http://developer.github.com/v3/#authentication.

What could be a better example? If someone makes up a good example I'll fix the documentation. 

Great work so far, I really love requests!


Are you using an old release?


Oh, sorry. I had 0.7 and now upgraded to 0.12.1 and it works. Thanks.


Yes, I had same error. Using "pip install requests --upgrade" worked for me.
Better content-type detection and unicode fix

Better content-type detection for request.auth.OAuth1, fixes [OAuthLib #38](https://github.com/idangazit/oauthlib/issues/38). All requests with files are now treated as multipart and r.data is not included in the signature creation.

Small fix to a UnicodeDecodeError during httplibs concatenation of the unicode Authorization header and file data.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1336709) (merged c24665b5 into 763393f7).


Looks like this breaks Python 3.1.


Yea sorry about that. Only use 2.7 myself and forgot that 3 does not allow u''. Nice work @travisbot =) 

brb updating...


@idangazit, any comments on this?


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1336823) (merged 6a9d59ea into 763393f7).


Thank you sir :)


Yes! Thanks :heart:


@ib-lundgren 

3.3 will [support](http://docs.python.org/dev/whatsnew/3.3.html#pep-414-explicit-unicode-literals) explicit unicode literals.
exception in requests.models.Request.cookies.items() 

It seems like the keys in the cookie jar are not names, but the actual cookie objects, so callings req.cookies.items() fails with an exception:

``` ipython
In [19]: requests.__version__
Out[19]: '0.12.1'

In [20]: r.request.cookies.keys()
Out[20]: [Cookie(version=0, name='csrftoken', value='65cab054cf10a7c14785ad7b254e44c4', port=None, port_specified=False, domain='192.168.230.1', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=1368513530, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]

In [21]: r.request.cookies.items()
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/home/arteme/http-tester/<ipython-input-21-93b115547bb8> in <module>()
----> 1 r.request.cookies.items()

/usr/lib/python2.6/_abcoll.pyc in items(self)
    348 
    349     def items(self):
--> 350         return [(key, self[key]) for key in self]
    351 
    352     def values(self):

/home/arteme/http-tester/requests/cookies.pyc in __getitem__(self, name)
    161 
    162     def __getitem__(self, name):
--> 163         return self._find(name)
    164 
    165     def __setitem__(self, name, value):

/home/arteme/http-tester/requests/cookies.pyc in _find(self, name, domain, path)
    176                         return cookie.value
    177 
--> 178         raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
    179 
    180     def __getstate__(self):

KeyError: "name=Cookie(version=0, name='csrftoken', value='65cab054cf10a7c14785ad7b254e44c4', port=None, port_specified=False, domain='192.168.230.1', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=1368513530, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), domain=None, path=None"
```


@arteme Check out #578. There are plans to implements keys(), values(), and items(), so that a user of Requests can call dict(resp.cookies) and get a name value dict.
Improve developer documentation.

I cloned, followed the todo and had failures as no certifi or chardet.

Fixed up so easy to get started.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1322966) (merged 2c114738 into 40d32536).


May be related to Issue #601 


I was going to argue against teaching virtualenv in here, but I can definitely see it being useful to new developers.

Thanks!


Is the documentaion incomplete ?


@toutouastro always a work in progress :)


in and after july (summer break) I'am planing to work on some open source projects and I liked requests and I think I'll participate in this project !
async.py doesn't have code

Seems async.py file doesn't have any code. So async will not be supported from requests? 


It's now a separate project: https://github.com/kennethreitz/grequests


yep, I am aware of it, if upgrade requests and code which consumes async will break, is there any plan to support async.py by internally importing GRequests ?


@kracekumar no, this is a backwards incompatible change.


 :( 
`from request.__past__ import async` isn't possible ?


It's possible, but it's not a good idea. I'm removing it for a reason.


For reference: http://semver.org/ :)
async broken on Windows

Running below code

``` python
from requests import async

if __name__ == '__main__':

    req = async.get('http://httpbin.org/get')
    res = async.map([req])[0]
    print res.content
```

I get

```
Traceback (most recent call last):
  File "C:\python\virtualenv\requests\lib\site-packages\gevent-0.13.7-py2.7-win-amd64.egg\gevent\greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "C:\Users\Piotr\projects\requests\requests\models.py", line 606, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /get
<Greenlet at 0x35e59c8L: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError

None
```

in about 90% cases.

Requests 0.12.1
gevent 0.13.7-py2.7-win-amd64
greenlet 0.3.4-py2.7-win-amd64
Python 2.7.3 64bit
Windows Vista 64bit


See also #500.


Closing in favor of kennethreitz/grequests#2
New implementation of safe_mode. 

We throw exceptions in models.py even if in safe_mode. We catch those exceptions at the API level and return a blank Response with the error field filled with the exception thrown. See safe_mode.py for details.

Discussed here: #583


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1295925) (merged cff70e75 into 2d328d8a).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1296854) (merged db43eb4c into 2d328d8a).


Fantastic work! This is much cleaner. 

:sparkles: :cake: :sparkles:
document testing practices

Currently, as @djco and @arfrever note, the Makefile doesn't make it into the Pypi tarball, and a plain invocation of `nosetests` doesn't run the right tests. So if you've downloaded Requests and you want to run the tests, it's not clear how. This is a reasonable thing to want to do.

Also, we should document which tests contributors should run before they submit their pull requests. (Ideally we would encourage contributors to run `make pyflakes`.)


In general, we need to document development workflow.


Does the entire repo need to be included in the PyPI tarball? Is it unreasonable to ask that people who plan to make pull requests on the repo actually get the repo from the origin? (As mentioned here https://github.com/kennethreitz/requests#contribute)

I feel like the PyPI tarball should only include what end-users need to use the library, not necessarily what's needed to contribute back to it.


I think the idea isn't so much that they want to hack on the tarball or patch it, but that they want to be sure that everything is working as expected on their system.


I not a fan of including extra things in the distribution. As a user, downloading extra stuff when installing a module often bothers me.


Some package managers, like Gentoo's, can run tests to check the package is working before installing it in the system. Though in requests' case it wouldn't be useful, as they forbid making network requests.


Portage (official package manager in Gentoo) does not forbid network requests, so tests of dev-python/requests package pass when internet connection is available.


@slingamn do #611 and #962 satisfy your desire for better documented testing practices?


Yeah, legit.

As a sidenote: the tests should not have to be run (or pass) as part of any automated package build.
Stream POST requests

I want to make POST requests with a big payload, and right now I have to be able to hold the full payload in memory to do the POST.

It be cool to accept a generator as the "data" argument of requests.post.


This is very much planned :)


I've actually got a need for this, too. Could I take a stab at implementing it, or would you prefer I hold off?


There's a major refactor in progress ([outlined here](http://kennethreitz.com/the-future-of-python-http.html)) that will facilitate this


[Armin Ronacher's Thoughts and Writings - I am Doing HTTP Wrong](http://lucumr.pocoo.org/2012/4/14/im-doing-http-wrong/)


@piotr-dobrogost indeed, the plan is to disable streaming by default but give it a simple flag


I'd like this for POST and PUT requests.


It will be for all requests.
```chunk_size``` for ```iter_lines()``` versus ```iter_content()```

Seriously, I think you should merge this one. The latest release and the `develop` branch still have the default `chunk_size` "reversed" for `iter_content()` and `iter_lines()` contrarilly to what you mentioned here: https://github.com/kennethreitz/requests/pull/589#issuecomment-5567553

This pull request may be canned by Travis CI again, but as I mentioned in the above pull request, the issue is located either in the test logic or the way lines ending in `\r\n` in `iter_lines()` depending how you interpret the issue... (I think the later is the real problem as even without changing `chunk_size`, `iter_lines()` could return different results for the same input depending on where the line-ending ends inside the buffer read by `iter_content()`). Not knowing how you perceive the issue, I did not attempt to fix it.

----- Original idea:

Following discussion about issue #539 Kenneth agreed that a default
`chunk_size` of 1 for `iter_lines()` would make more sense, but
apparently changed `chunk_size` for `iter_content()` by mistake
instead, which did solve the `iter_lines()` issue but also changed the
default `iter_content()` chunk to 1 bytes which is not desirable.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1276587) (merged 87f0a92d into 0308a280).


As expected. See comments above and for my previous pull request here: https://github.com/kennethreitz/requests/pull/589#issuecomment-5538019


I agree with @mponton — the current default chunk_size value of 1 for iter_content() seems broken. We were bitten by this performance regression when upgrading to 0.13.2.


Did this get merged, or ignored?  Would have been useful to have had a comment made when the pull request was closed.

Looks like it was ignored, which ... means I had a massive performance regression just now.


Duly noted but 'ignored' for now. Definitely needs to change, but we need to find a good answer to the issue noted above first.


I don't entirely know what I'm talking about, but here are some guesses about how `iter_content` should maybe work:
1. The core use case for this feature seems to be streaming APIs like Twitter's "firehose", which use HTTP 1.1 and chunked encoding
2. As per Twitter's guidelines [https://dev.twitter.com/docs/streaming-apis/processing], the right way to consume an API like this is to use chunked encoding to tell how much data to read at a time. They suggest deferring the details of this to the underlying HTTP library.
3. Unfortunately, it looks like we can't reuse the chunked encoding implementation from `httplib`, because it's hard-wired to consume the entire document at once and return it. We might need to make our own --- possibly by copying and pasting code from httplib?
4. For HTTP 1.0 and non-chunked pages, we should probably just try to read the entire document and yield it as a single string?

As mentioned at #775, I am also interested in changing the `iter_content` behavior so that if the body has already been consumed, it yields a one-item iterator containing the whole body, e.g., `iter([self.body])`.
Session cookies not saved when Session.request is called with return_response=False

For example, I want to use `request.async` and save cookies to session:

```
import requests
from requests import async

s = requests.Session()
async.map([async.get('http://httpbin.org/cookies/set/async/yes', session=s)])
print 'cookies:', s.cookies
```

cookies are empty. 
It's because when `Session.request()` is called with `return_response=False`, it returns request without updating session cookies [sessions.py#L234](https://github.com/kennethreitz/requests/blob/9bf53676b87fa4951e23f67d2251f5dfae4e5206/requests/sessions.py#L234)

So, I think we can solve it by moving cookies saving logic to `Request._build_response()`. Or maybe there is more suitable place?


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1274659) (merged 33f10875 into 0308a280).


Nice catch!

LGTM, except I don't think your test case is actually hitting the async API. Should it be calling `async.map` to make the request?

Add yourself to `AUTHORS.rst` too? :-)


> Should it be calling async.map to make the request?

Yes, but `get` is patched in this testcase, see [test_requests_async.py#L39](https://github.com/kennethreitz/requests/blob/317f64a11f56d89119baf5db3af65c0343464bc1/tests/test_requests_async.py#L39) so it calls `async.map`.

> Add yourself to AUTHORS.rst too? :-)

Ok :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1275024) (merged c010f589 into 0308a280).


Oh, yeah. Awesome.


:sparkles: :cake: :sparkles:


@reclosedev 

Nice work.

> Yes, but `get` is patched in this testcase

... and all sync tests are being run using async api as well. As your two requests in the async test are sequential you might want to replace sync test with the async one (which tests more) and remove the latter. You might also want to reference request from reply instead the other way around in case the link in this direction is removed in the future :)

Btw, please remember not to link to moving target (develop) as this doesn't make sense.


@piotr-dobrogost 

Maybe you are right about  sync test, but I think it's not superfluous, because, for example, async api may use another way to create requests in the future, but `return_response=False` should be tested.

> You might also want to reference request from reply instead the other way around in case the link in this direction is removed in the future :)

Can you elaborate?

> Btw, please remember not to link to moving target (develop) as this doesn't make sense.

Are you talking about links in comments? Fixed :)


> Can you elaborate?

See issue #520.


Oh, I don't think it matters; if something like that gets implemented, we'll just change the test.
unbreak the Jenkins build again/hopefully :-/

This makes it so the Python 3 and PyPy Jenkins environments don't try to run Pyflakes, since they can't.

If you're fed up with all my attempts to get Jenkins to check pyflakes, that's cool; we can just remove the target.

However, there's another issue. We should take `oauthlib` out of `requirements.txt`, as per this failure: http://ci.kennethreitz.com/job/requests/244/PYTHON=3.2/console

`oauthlib` will get pulled in for Python 2 only when `setup.py` runs, right?


I'd rather not have it honestly :)


But if it works, it works :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1273543) (merged 175945b0 into 6fd03d5f).


Yay! http://ci.kennethreitz.com/job/requests/245/
XML parse issue with requests, not with urllib2

This is the XML response i get https://gist.github.com/2624993

When i try to parse it with cElementTree, first i get ascii codec error, then if i encode `response.text.encode(response.encoding)` i get tag mismatched error (62 col 2).

With urllib2 it seem to work ok.

I'm lost as to what can cause this issue, i would be glad to provide more info


Are you using `response.content` or `response.text`?


text


Text is unicode.


Yeah i know that, what's weird is that if get that XML from github, like:

```
ipdb> url = 'https://raw.github.com/gist/2624993/03b7d5e7c9041138f9da132def3e8e1e0c30fa85/gistfile1.xml'
ipdb> ET.fromstring(requests.get(url).text)
```

it works, but if i get the same XML from goodreads.com, it does not

```
ipdb> url = 'http://www.goodreads.com/book/show/43615?key=REDACTED'
ipdb> ET.fromstring(requests.get(url).text)
*** UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 67922: ordinal not in range(128)
```


XML is typically not a unicode-friendly format.
Fixed chunk_size values in iter_content() and iter_lines(). False default instead of None.

Following discussion about issue #539 Kenneth agreed that a default
`chunk_size` of 1 for `iter_lines()` would make more sense, but
apparently changed `chunk_size` for `iter_content()` by mistake
instead, which did solve the `iter_lines()` issue but also changed the
default `iter_content()` chunk to 1 bytes which is not desirable.

Also changed default for `decode_unicode` to `False` instead of `None` in `iter_lines()`.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1259283) (merged 85e83c40 into c84012ed).


Hmmm, I think the logic in

``` python
if pending is not None:
    chunk = pending + chunk
lines = chunk.splitlines()

if lines and lines[-1] and chunk and lines[-1][-1] == chunk[-1]:
    pending = lines.pop()
else:
    pending = None
```

Does not handle "\r\n" as `splitlines()` does when using a `chunk_size` of 1. The quote:

```
quote = (
            '''Agamemnon  \n'''
            '''\tWhy will he not upon our fair request\r\n'''
            '''\tUntent his person and share the air with us?'''
        )
```

used in `test_requests.py` contains `\r\n` which ends up becoming a line in `iter_lines()` when the iterator hits the `\r` character, next the `\n` character is read and an empty line is spit by `chunk.splitlines()`. Thus the test fails as 4 lines are returned instead of 3. If the `chunk_size` is larger and `\r\n` are read in a single chunk, then it works as expected.


decode_unicode is `None` for a reason :)


`iter_content` also has a proper size now, so closing. Thanks though!


OK, not sure why `decode_unicode` should be `None` but you may then want to change it to `None` in `iter_content()` too...? :-)
Add response.reason

At work we have an in-house REST API that puts useful information in the text portion of the HTTP status line.  I was surprised that this wasn't exposed on the Response object, and I had to go to response.raw.reason to get it.

My fork has a fix  (https://github.com/btubbs/requests/commit/742c1681d12f7a4b079c4c3ac49c4991ce898fd1).  I think it's worth adding, but am interested in others' thoughts before making a pull request.


Thanks for the suggestion!

Honestly though, I don't really see how useful the response reason is. The number is all that matters to me. 


In the API I'm working with, the main culprit is status 409 responses, which use the reason field to say things like "Bad Username/Password" or "Already Exists", typically in response to POSTs.

Having that information in an intuitive place on the response object could save future users the 20 minutes or so of head-scratching that I had, trying to figure out where the reason is.


Hmm, interesting. I didn't even realize that was customizable. 


I had to double-check, but they are. http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1.1


tl;dr: +1: 

Yeah, use of the reason phrase for explanatory detail is part of the spec. As another data point, I, myself, use requests with a small wrapper, and one of the things it does is tack the reason phrase onto the response object for convenient access. In the app I'm currently working on, several of the APIs I pull from do indeed have cases where the reason phrase is the only way to know which of several possible causes actually triggered a particular error status code.


Let's do this ;)
unbreak the Jenkins build by excluding ./env/*

Doh! http://ci.kennethreitz.com/job/requests/236/PYTHON=2.7/console

This might be a reason to take pyflakes out of the Jenkins target (`ci`).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1243968) (merged d4227051 into 3d373e2a).


But personally, I'm a really huge fan of pyflakes as a build step. Catches a ton of issues.


thank you sir :P
Problem installing pycrypto on Windows

oauth library depends on pycrypto library which is not pure Python library (has some c extensions). For this reason pycrypto can't be installed from source on Windows without Visual Studio 2008 installed (needed to compile these c extensions of pycrypto). The simplest solution is to download binary installers and install them using easy_install (pip doesn't support installing of binaries). 32bit binary installers for Python 2.2-2.7 and 64bit binary installer for Python 2.6 are at [The Voidspace Python Modules](http://www.voidspace.org.uk/python/modules.shtml#pycrypto) and 64bit binary installer for Python 2.7 is at [Compiling pycrypto on Win7-64](http://yorickdowne.wordpress.com/2010/12/22/compiling-pycrypto-on-win7-64/). Binary installers can be installed in virtualenv - see http://stackoverflow.com/q/3271590/95735


I think we should just focus on fixing #584; this really seems like a problem with `oauthlib`, not with Requests.

You could file a bug at idangazit/oauthlib ?


Sure we should fix #584. I opened this issue just in case someone had problems installing pycrypto which is needed for oauthlib right now. As to filing bug at oauthlib I don't see what bug I could file :)


er, s/bug/issue/ --- and file this same one?


> You could file a bug at idangazit/oauthlib ?

Issue https://github.com/idangazit/oauthlib/issues/34


Closing for idangazit/oauthlib#34
Fix out-of-range issue with iter_lines (#515).

This will fix the iter_lines IndexError, which may be caused either by `chunk == ''` (empty string), or `chunk.endswith('\n\n')`.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1243718) (merged c333bd14 into 3d373e2a).


Excellent, thanks!
Make oauth an optional dependency

It would be nice not to force oauth on all users especially as it drags in pycrypto which has some binary parts making it more difficult to install.


+1.

Basically, if you don't have oauth, all of Requests should work fine except `requests.auth.OAuth1`. I would make it so that `OAuth1` throws an appropriate exception (probably `NotImplementedError`) in `__init__()` if oauth is unavailable.

If that sounds good, I'll send a pull request.


How about `ImportError` instead of `NotImplementedError`? The latter one suggests there's no implementation whereas there is one but it needs external module to work.


It is an optional dependency, like chardet. 

Better yet, it would better if oauthlib had a soft dependency on pycrypto,


It's getting pulled in by setuptools, though. For example, Travis is having to compile it:

http://travis-ci.org/#!/kennethreitz/requests/jobs/1243719

It might be more appropriate to put it in `extras_require`, like `gevent`.


Ideally oathlib won't require pycrypto soon. We'll see.


I guess the real reason is that `chardet` is more important to Requests than `oauthlib` is, e.g., a bunch of the unit tests will `ValueError` without it.


@piotr-dobrogost can you clarify what happens now if you `pip install requests` on Windows? Does it crash?


Windows can build pycrypto just fine if you have the proper compilers available. 


From the sound of it, though, the typical Windows user trying to install Requests doesn't.


Again, ideally oathlib won't require pycrypto soon. We'll see.


> can you clarify what happens now if you pip install requests on Windows? Does it crash?

Of course it crashes :) which is expected having pycrypto as a strictly required dependency.

> From the sound of it, though, the typical Windows user trying to install Requests doesn't.

That's right. Why would someone need C compiler to program in Python? :)

> I guess the real reason is that chardet is more important to Requests than oauthlib is, e.g., a bunch of the unit tests will ValueError without it.

Ideally chardet should be optional as well.


We've dropped pycrypto from oauthlib, it now uses the pure-python [python-rsa](http://stuvel.eu/rsa) instead.


:sparkles::cake::sparkles:
new implementation for safe_mode?

The current implementation of `safe_mode` seems to be a source of bugs. In safe mode, many members of the `Request` and `Response` objects can be `None`, so they have to be checked manually. Also, 

Would it be possible to reimplement safe_mode by catching all known exception types (`RequestException`, `ConnectionError`, `HTTPError`) at the outermost API level? Then we would respond to exceptions by falling immediately back down to the API level.

The implementation of this catch-and-save behavior could probably be a decorator on each API method.


Basically, if I understand the problem correctly, here's how to implement this change.
1. `git grep -i safe`
2. Remove all checks for safe mode
3. Remove `if baz is None` tests that are only relevant in safe mode
4. Decorate all the functions in `requests/api.py` with something that catches the relevant exception types, constructs an empty `Response` object, populates its `error` field with the exception, and returns it

The implementation of `danger_mode` can stay as-is.


+1


Sounds reasonable.


#TODO as per #517, also catch `socket.timeout`.


I would like to begin contributing to Requests. This seems like something I could handle. If this seems like an okay problem to start with, I'll give it a try.


@joshimhoff go for it :)


I have finished my implementation, but I am unsure where in the file-system the safe_mode decorator should live. Anyone have an opinion? Currently, I created a new module, safe_mode.py, as the decorator did not seem to fit into any existing modules.


@joshimhoff fantastic work.
A utility for using environment proxies

This provides a utility function that reads the environment proxies. It returns proxies in a format compatible with request's proxy parameter.

Moreover, it can be used in the request models for proxy defaults.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1242492) (merged abc38cf1 into 1adaba38).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1242554) (merged 7cf6ca80 into 1adaba38).


This is fantastic, thanks a ton!


:sparkles: :cake: :sparkles:
Fix and test for #458

Fix and test for #458.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1242233) (merged 35ec6bb6 into 317f64a1).


Heh, if the user sets `headers={'connection': 'keep-alive'}` and `config={'keep_alive': False}`, which should win? Right now it's the config.


You're amazing.
Changes for #574

Mostly what was discussed for #574.

This patch will run pyflakes on a whitelisted set of files during the Jenkins build (but not the Travis); I can remove that if you'd prefer to keep it separate.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1241977) (merged bcc4125a into 317f64a1).


BTW, Jenkins used to run the `init` target, but doesn't anymore after 57ccd7e65c30 (I foolishly assumed no one was using the original `ci` target). Does Jenkins need that to work?


Thanks!
local testing mock server

Right now, it's not really possible to test things like secure cookies that aren't supported by httpbin.org. urllib3 has a "dummy server":

https://github.com/shazow/urllib3/blob/master/dummyserver/server.py

that can be used to serve one-off canned responses from `localhost` --- providing full control over headers, body, etc. I wouldn't want to depend on `tornado` for this, so I'm not suggesting copying and pasting `dummyserver`, but do you think something similar would be useful?


I run httpbin locally :)


Hmm, yeah, the `response_headers` method of httpbin should provide most of this functionality.

But would it be OK to start an httpbin (possibly in-process) from inside the unit tests? And to have them depend on the httpbin package?


We used to do that, but had to stop for Python 3, unfortunately. Werkzeug and Flask will support Python 3 within the next few months, if things go according to plan. So, I'm holding out until then ;)


Wait, never mind, that's not necessary either to test secure cookies. The remote httpbin should work just fine.

When I get some time, I'll poke around in `test_requests_ext.py` and see if any important tests in there can't be converted to use remote httpbin.


That would be fantastic.


I'll close this in favor of incorporating any necessary functionality into a localized httpbin.
Redirect of multipart-encoded post

I'm scripting a site that returns a redirect in response to the post of a multipart-encoded form.  When I do so, I get the following exception:

```
File "C:\Python27\lib\site-packages\requests\sessions.py", line 259, in post
    return self.request('post', url, data=data, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 209, in request
    r.send(prefetch=prefetch)
File "C:\Python27\lib\site-packages\requests\models.py", line 624, in send
    self._build_response(r)
File "C:\Python27\lib\site-packages\requests\models.py", line 297, in _build_response
    cert=self.cert
File "C:\Python27\lib\site-packages\requests\models.py", line 119, in __init__
    self.files, self._enc_files = self._encode_files(files)
File "C:\Python27\lib\site-packages\requests\models.py", line 346, in _encode_files
    fields = dict(self.data)
TypeError: 'NoneType' object is not iterable
```

Browsing through the code, I found the following snippet in models.py -  _build_response:

``` python
                # Do what the browsers do if strict_mode is off...
                if (not self.config.get('strict_mode')):

                    if r.status_code in (codes.moved, codes.found) and self.method == 'POST':
                        method = 'GET'
                        data = None

                    if (r.status_code == 303) and self.method != 'HEAD':
                        method = 'GET'
                        data = None
```

I'm guessing that `self.files` should also get set to `None` here, so that it doesn't get re-used in the subsequent request (causing the exception).


Interesting catch, thanks!
run pyflakes during CI builds?

Sorry about that missing import :-/

Would a Travis build step that runs pyflakes on all relevant files be helpful? I'd whitelist the following:
- `docs/`
- `requests/packages/urllib3/`
- `tests/`
- anything named `__init__.py` or `compat.py`.


Oh yeah, `requests/packages/oreos/` too.


No need for travis, but I wouldn't mind adding it to http://ci.kennethreitz.com tests.


Ah. What make target does that service use?


`make ci`. We can add another if needed.


Cool, I added a separate target for travis; I'll send a pull request for this after #572 gets merged.


I've been wanting to find a purpose for a `make lemonade`. Let's have that be the metrics (pyflakes, coverage, etc)


:-)

BTW, re. #572, you may want to fastforward master into develop before doing any other merging.


(It's one commit ahead.)


Actually, if Travis doesn't run pyflakes and Jenkins does, isn't there a risk of pull requests passing Travis and then breaking the Jenkins build?

I guess the question is whether it's worth enforcing pyflakes-compliance for every pull request or not.


Compliance? No. It'll be a nice metric though. I'll make it a separate job.


:-)
Lazily encode data, params, files

Previously, data, params, and files were encoded and stored in
Request.**init**, and subsequently put into service during
Request.send. The problem with this approach is that hooks and auth
callables need to be aware of the eager encoding, and if they touch the
originals, make sure to update the encoded versions.

A better approach is to only encode late in the sending process. This
way, hooks and auth callables can safely make changes without fear of
the old, encoded variant overriding it.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1230802) (merged cf6b2042 into 53d60cac).


:sparkles: :cake: :sparkles:
cleanup, redux

- Raise a ValueError on .text when chardet is unavailable
- `make ci` runs everything except test_requests_async and test_requests_ext


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1224921) (merged d4f48c0d into 53d60cac).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1224957) (merged 628ffd65 into 53d60cac).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1225144) (merged 37361dd8 into 53d60cac).


It would probably be good to cherry-pick cf34ccecec81 at least; otherwise Travis won't run `cookie_tests.py` against new pull requests.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1233185) (merged 853a0711 into 373a34a1).


(Sorry, I rebased; cf34ccecec81 is now 57ccd7e65c3.)
Various forms of "cleanup"

Here are several atomic, largely-unrelated changes, ordered roughly from least to most controversial. Feel free to take whichever ones you like :-)
- Restore something I accidentally removed from the .gitignore
- Small documentation fix for the new cookie API
- Don't crash if we can't import chardet --- assume utf8 instead
- Cache response.text, since computing it can require a call into chardet
- Change setup.py to provide guidance for packagers (I think this is part of what @jnrowe was asking for)
- Have Travis run all the tests, not just `test_requests.py`


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1224813) (merged a453a78c into 45be14ed).


merged, thanks!

:cake:
Unnecessary use of list() and items() in "for" loops

In many `for` loops there is unnecessary use of list() and items() instead of iteritems(). One example
[`for (k, v) in list(defaults.items()):`](https://github.com/kennethreitz/requests/blob/775b6f6f0098ffa9edd3874b1af4b98378377211/requests/sessions.py#L83). Adding list() and changing iteritems() to items() was done in changeset b0e2e4f284f619536b7f405122bb1a93900cd743 with commit message "blah"...
What is the point?


Based on the commit message, I'm guessing it's something to do with Py32. :P


I'm sure there was a reason at the time :)


How about reverting to the proper state until you find out what was the reason? :)


:bike: :house:
Proper test for str vs unicode when preparing data

In python 2.x, requests sets str = unicode in requests/compat.py. This causes
isinstance(foo, str) to return True even if foo is a unicode string.

Supersedes #556, which I totally broke.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1213530) (merged 0ec0e9f3 into ef0bf79f).


Le big sigh, this breaks something. Working it out, fix coming shortly.


Let's all agree to forget I ever made this PR.
Import error using requests.async with 10000 URLs

I'm using requests to download asynchronously and parse the content of a bunch of URLs from different hosts. When the number of URLs is small (up to 1000 URLs) all works fine. When I run the same script with 10000 I get the following error **only when there are about 10 URLs left to download**:

``` python
Traceback (most recent call last):
  File "many_async_req.py", line 35, in <module>
  File "/home/alex/code/.virtualenvs/testreq/local/lib/python2.7/site-packages/requests-0.11.2-py2.7.egg/requests/async.py", line 83, in map
  File "/home/alex/code/.virtualenvs/testreq/local/lib/python2.7/site-packages/gevent/greenlet.py", line 487, in joinall
ImportError: No module named queue
```

Sometimes the script just hangs with about 10 URLs left. I observed this behavior with requests installed from pipy and with the code downloaded from the development branch. 

This happened to me using the version of gevent from pypi **and** using the [latest version](http://code.google.com/p/gevent/downloads/list) downloaded from gevent's Google Code repository.  
I'm using a virtualenv with only the relevant modules installed:

``` bash
(testreq)alex@haiku:~/code$ pip freeze
certifi==0.0.8
chardet==1.0.1
distribute==0.6.19
gevent==0.13.7
greenlet==0.3.4
requests==0.11.2
wsgiref==0.1.2
```

To reproduce the error I'm using the following script (that is a stripped-down version of the original) that basically just print the status code of each URL.

``` python
import codecs
import random
from requests import async
from requests import defaults
import sys

defaults.defaults['base_headers']['User-Agent'] = "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101 Firefox/11.0"
defaults.defaults['max_retries'] = 5
defaults.defaults['verbose'] = sys.stderr


def check(response):
    global counter
    print "\t %s HTTP status code: %s" % (counter, response.status_code)
    counter += 1


def load_urls(fname):
    """ Read URLs from file and remove duplicates """
    with codecs.open(fname, 'r', 'utf-8') as fh:
        urls = list(set([url.strip() for url in fh.readlines()]))
        random.shuffle(urls)
        return urls

if __name__ == '__main__':

    urls = load_urls('urls.txt')
    limit = int(sys.argv[1])
    urls = urls[:limit]
    rs = []
    counter = 0
    todo = len(urls)
    for url in urls:
            rs.append(async.get(url, hooks=dict(response=check), timeout=5.0))
    async.map(rs, size=10)
    print "To check: %s" % todo
    print "To checked: %s" % counter
```


It is curious to me why you aren't using a generator for this, but it looks like it wouldn't work anyway due to this sorcery
https://github.com/kennethreitz/requests/blob/develop/requests/async.py#L79. 

Well there is the imap function that looks to be generator compatible, but its not mentioned in the docs anywhere. http://docs.python-requests.org/en/latest/search/?q=imap&check_keywords=yes&area=default


@pythonben It's just because I'm a n00b :-)


Async is removed from requests now. A new module, [grequests](https://github.com/kennethreitz/grequests) will replace it.
Update AUTHORS.rst

lulz


:trollface:


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1182970) (merged 72a75838 into ef0bf79f).


![noooooooo](http://www.soshichan.org/g/src/132358305526.jpg)


FUCK THIS SHIT


:-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1::-1:

You can't change the authors THE TESTS FAIL


this made my day.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1183066) (merged 72a75838 into ef0bf79f).


Two failures?! this is like failing calculus, retaking it in a summer course and _failing that too_. :cry:
replace utils.randombytes with os.urandom

My understanding is that Python's `random` module is not recommended for cryptographic use. `os.urandom` should provide random bytes on all major platforms: Linux, OS X, *nix, and Windows:

http://docs.python.org/library/os.html#os.urandom
http://docs.python.org/release/3.0.1/library/os.html#os.urandom

Since the function provides a `str` under 2.x and a `bytes` under 3.x, and the result is fed directly into SHA, there shouldn't be an issue with the encoding.

Thoughts? Thanks for your time.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1217425) (merged f1b054e4 into 2159c802).


I'll stick with it the way it is _for now_ because that's the same behavior the stdlib uses


Thanks though :)


I see this in my version of urllib2:

``` python
def randombytes(n):
    """Return n random bytes."""
    # Use /dev/urandom if it is available.  Fall back to random module
    # if not.  It might be worthwhile to extend this function to use
    # other platform-specific mechanisms for getting random bytes.
    if os.path.exists("/dev/urandom"):
        f = open("/dev/urandom")
        s = f.read(n)
        f.close()
        return s
    else:
        L = [chr(random.randrange(0, 256)) for i in range(n)]
        return "".join(L)
```

so it's preferring the secure randomness functionality to the `random` library --- I guess they thought it preferable to fall back to `random` rather than choke.


I guess this does leave one question --- why doesn't Windows use `os.urandom` (backed by their own `/dev/urandom` analogue) instead of `random`? But still, `os.urandom` seems closer to the stdlib behavior.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1342925) (merged f25827c5 into 6a9d59ea).


Awesome, thanks so much ;)
Redefine order of pre_request hook and auth execution

## Background

Auth callables, in their current incarnation, run much in the same fashion as a hook—they get the request at some point after it is asked to execute itself, and may mutate it. Currently, auth callables run before the pre_request hook.

The problem with this arises with auth schemes that expect the request's information to be immutable, like OAuth, because they are signing data in the request. If the pre_request hook comes along and changes a header, the signature would no longer be valid.

I'm suggesting that we swap the order of auth and pre_request: auth would run later in the cycle, directly after pre_request.
## Impact

Existing pre-request hooks which rely on the request content not changing after they run would suffer the same problems as the OAuth callable scenario described above. We'd need to document the fact that pre-request hooks do not have a guarantee of immutability if used together with some kinds of auth callables.

I don't think this is a big deal to change, but wanted to hear if anybody had some case I hadn't thought of.
### Edit

Worth noting that a potential solution here is simply to document that Auth callables are the ones with no guarantees. This would allow us to leave auth callables in their current location.
Remove chardet from dependencies

Chardet is LGPL, which is unfortunate to have installed when using an ISC licensed library.


It is indeed a shame, but it's the only available module that works. 

Luckily, Requests works fine without it. You just don't get character ecoding guessing.


Yes, it is conditionally imported, but it gets installed when installing requests via pip. Is it possible to remove it from the setup.py?


Unfortunately, no. Requests is not bound by the terms of the LGPL.


I understand that Requests is not, but by requiring chardet on installation, it effectively is. If its not required and conditionally imported, why can chardet not be removed from the setup.py?

Alternatively, is there a way to black list modules in requirements.txt files?


It effectively is not.


Ok, thank you.


Not trying to be short, sorry :) 

It's a pain point for me at the moment. I've definitely put a lot of thought into it though.


No worries. I understand the issue.


I don't even see how LGPL could be an issue for a library. It does not restrict software linking to it.


@laurentb exactly.


While I personally with this interpretation of the LGPL agree with you, others do not.

As always, thanks for your time.
Proper test for str vs unicode when preparing data

In python 2.x, requests sets str = unicode in `requests/compat.py`. This causes
`isinstance(foo, str)` to return True even if foo is a unicode string.

Would appreciate review of this change because my understanding of how this impacts py3x compatibility is tenuous.


I just pushed a change for issue 526. because of the compat (byte=str; str=unicode) i did a isinstance(x, byte)  when i was checking for str. 

Is this wrong? Should i update to be consistent with your change?

https://github.com/kennethreitz/requests/pull/559


@barberj I don't know—that's the point. I'm not sure what the correct approach is for py2 AND py3.


D'oh, accidentally included more commits in this than intended. Please don't merge the request, but discussion is desired.


@idangazit can you make this PR again?


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1213297) (merged 3857de64 into ef0bf79f).


Superseded by #569
IronPython: LookupError: unknown encoding: idna

C:\MyProjects\IronPython>ipy -X:FullFrames
IronPython 2.7.2.1 (2.7.0.40) on .NET 4.0.30319.239 (32-bit)
Type "help", "copyright", "credits" or "license" for more information.

> > > import requests
> > > r = requests.get('https://github.com/timeline.json')
> > > Traceback (most recent call last):
> > >   File "C:MyProjects\IronPython\requests\api.py", line 52, in get
> > >   File "C:MyProjects\IronPython\requests\sessions.py", line 208, in request
> > >   File "C:MyProjects\IronPython\requests\models.py", line 349, in full_url
> > >   File "<stdin>", line 1, in <module>
> > >   File "C:\MyProjects\IronPython\requests\api.py", line 40, in request
> > >   File "C:\MyProjects\IronPython\requests\models.py", line 424, in send
> > > LookupError: unknown encoding: idna

As expected, the same works fine with CPython 2.7.3. I am using requests-0.11.1.


IronPython should really fix that ;)


@kennethreitz: Can you suggest a workaround?


No suggestions at this time. That encoding is pretty key to requests's functionality. I'd follow up here: 

http://ironpython.codeplex.com/workitem/4565
Breaks Py3k

The u before the string in the tests breaks py3k


I'll likely be moving to support only 3.3 soon, because of this.


This will leave next Ubuntu LTS users without python 3 support by default. Isn't it better to drop python 2.5 support (PSF already drops python 2.5 support) and use `from __future__ import unicode_literals` across the codebase?


I mean only official 2.6, 2.7, and 3.3 support. 

Python 2.5 support was dropped when support for 3.1 and 3.2 was added several months ago.


Did my understanding correct and you're going to leave u in codebase as-is and drop python 3.2 support because of this? What's wrong with official python 3.2 support using `from __future__ import unicode_literals` approach?


This is just a mean of adding a import, removing u's and adding b's when appropriate - not a big price for the most used python 3 version support :) I really don't care about 3.0 or 3.1 but 3.2 is mainstream and it is default python3 in a lot of distros. 


It has nothing to do with the codebase, only the tests. 

I needed explicit unicode literals for my test suite. I tweeted my frustration. Armin made it happen :)


Still don't get what are the practical reasons tests can't be run under 3.2 but ok, no more drama as you say code works :)


@kmike essentially i need "native" strings (unicode in 3, bytes in 2) with the same literal, since these are integration tests, not unit tests. That's why I can't use just do `from __future_ import unicode_literals`.

But I also need explicit unicode strings and explicit bytecode strings.
IronPython: import issue

It will be great if requests is made compatible with IronPython. This is a pure python library. So it should be feasible.


+1 for this, also for PyPy support.

Does requests not work out of the box in these environments right now? Are there any specific bugs?


requests does not work out of box in IronPython. "import requests" fails.


Do you have a traceback?


I actively test Requests against PyPy.

There's no reason IronPython should have any problems.


File requests\packages\urllib3\packages\six.py", line 307, in exec_
AttributeError: 'module' object has no attribute '_getframe'

sys._getframe(1) is not supported in IronPython.2.7.2.1.


This is a bug with urllib3, not Requests.

You can file a bug [here](https://github.com/shazow/urllib3)


 sys._getframe(1) is supported with the option -X:FullFrames. So import requests is now working with ipy -X:FullFrames.


_sigh_ :)


Closing for out of scope.