Raibek/requests-github-issues huggingface dataset:
Ability to interrupt / stop iter_content

When using iter_content, in some instances it would be beneficial to have the ability to interrupt the request. If you take the Twitter streaming API as an example, there is no current way to exit cleanly as iter_content will continuously block. For example, in the following code:

```
for line in self.conn.req_content(self.ITER_BYTES):
                self.status += line
                if line.endswith(self.DIVIDER) and self.status.strip():
                    print self.status
                    self.status = ""
```

The only way to I can stop the stream connection is by either running it in a separate thread and joining or by manually setting the raw property of the request to None (and catch the AttributeError that subsequently gets thrown): 

```
self.req.raw = None
```

I understand that currently only the Twitter streaming API makes this issue relevant but going forward I'm sure more streaming APIs will emerge and it'd be useful if there was a nicer way to unblock.


Can't you just break out of the generator when you so desire?


Twitter will endeavour to always keep the connection open for as long as possible meaning if you are tracking a particularly low frequency predicate, it could potentially be hours before the request receives a response and you are able to break.
Session.prefetch not pickled and causing AttributeError on next request after unpickle

Code that reproduces the problem:

``` python
import pickle
import requests

url = 'http://httpbin.org/cookies'
session = requests.session()
session.get('%s/set/key1/value1' % url)
before_pickle = session.get(url).text

pickled_session = pickle.dumps(session)
session = pickle.loads(pickled_session)

# AttributeError: 'Session' object has no attribute 'prefetch'
after_pickle = session.get(url).text
assert before_pickle == after_pickle
```

Ways to solve:
1. Dirty one: dont' save `prefetch`, just replace [sessions.py#L158](https://github.com/kennethreitz/requests/blob/develop/requests/sessions.py#L158)
   `prefetch = self.prefetch or prefetch`
   to
   `prefetch = getattr(self, 'prefetch', False) or prefetch`
2. Add prefetch to `Session.__attrs__`, [`args` at sessions.py#L169](https://github.com/kennethreitz/requests/blob/develop/requests/sessions.py#L169) and to `Request.__init__()`
3. Another one?
A get with an invalid port should wrap urllib3's LocationParseError exception with our own.

Fixes this issue:

``` python
>>> import requests
>>> requests.get('http://google.com:banana')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
[...snip...]
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py", line 579, in get_host
    raise LocationParseError("Failed to parse: %s")
requests.packages.urllib3.exceptions.LocationParseError: (LocationParseError(...), 'Failed to parse: Failed to parse: %s')
```

To now return a wrapped exception, which hides the use of urllib3 and is more intuitive to catch:

``` python
>>> import requests
re>>> requests.get('http://google.com:banana')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 208, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 506, in send
    raise InvalidURL(e)
requests.exceptions.InvalidURL: (LocationParseError(...), 'Failed to parse: Failed to parse: google.com')
```


You're amazing: 

:cake:
Fix infinite loop on wrong Digest Authentication

The most straightforward way to fix #541 seems to be to deregister the hook on the first 401. This is what I have implemented here.


Hmm, the second commit is something different (I should have created a branch).
It fixes the conflict between Digest Auth and Multipart Post (#298).

I am less convinced that my implementation is the right way to do it there, so take it as a suggestion (and motivation to fix a 4 months old bug hit by one of our clients :p).


This is fantastic. Thanks!

:cake:
I think cookie_header should not use DOUBLE SPACE after ";"

At line 550 in file models.py, I found the code like this:

... sep='; ' ...

There is one space character after semicolon character. so that the result of cookie string in the header like this:

airkey=;  clientkey=;  clientuin=;  confirmuin=0;

There are two space characters after ";".
I checked IE and Firefox that are just generate one space character after ";".
And, some website can not recognize the formation of cookie which have two space characters as separator.


This is fixed in master:

```

>>> import requests
>>> c = requests.get('http://httpbin.org/cookies/set/key/value').cookies
>>> c = requests.get('http://httpbin.org/cookies/set/key2/value2', cookies=c).cookies
>>> print requests.get('http://httpbin.org/headers', cookies=c).text
{
  "headers": {
    "Content-Length": "", 
    "Accept-Encoding": "identity, deflate, compress, gzip", 
    "Connection": "keep-alive", 
    "Accept": "*/*", 
    "User-Agent": "python-requests/0.12.0", 
    "Host": "httpbin.org", 
    "Cookie": "key2=value2; key=value", 
    "Content-Type": ""
  }
}
```


<3
get() partial download

The command
resp = get(url, verify=False, prefetch=True, timeout=20)

does not read the whole content, i.e., len(resp.text) differs from the expected value.
The expected page size is in the range 50-60 KB, while len(resp.text) is about 8 KB.
I have tried get(url) as well, but I have not noticed any difference.
I have made some tests with other libraries (i.e., https://github.com/Lispython/human_curl or a Java client), and I get the expected behavior.


`resp.text` is unicode, so measuring its length has little to do with its size :)
Inclusion in the standard library.

Has any thought been put into proposing (via the guidelines of PEP2) `requests` as a standard module (please excuse me if I missed this discussion elsewhere. Hard to google for "python" and "requests" :)).

With full support for older versions of Python and Py3k, it seems like a good candidate. Interacting with HTTP is one of the biggest uses of Python and will likely to continue to become more important. `requests` is battle-tested, portable, extremely popular and actively developed. In this manner, it would be very similar to the inclusion of `simplejson` (as the `json` module) since 2.6.


I've chatted with Guido about this, his advice was that we want to stay away from the standard library. Being included in the standard library requires a great deal of stability, and the process to introduce changes is slow. Requests is still rapidly improving and being included in the standard library will likely stunt its growth more than help.

Regarding Googling for "requests", have a look at this thread: https://github.com/kennethreitz/requests/issues/538


Excellent point. I would think however that a stable release would be suitable for inclusion, with development continuing rapidly in parallel, much the same as pysqlite (the sqlite3 module) and simplejson (the json module). If only included in the standard library for 3.*, then there will be more opportunities to include an updated stable, and users will expect potentially breaking changes.

Just my two cents. I trust Guido's input above my own :) Glad to see I'm not the only one who was having google-fu issues.


Now, when there's [PEP 411 - Provisional packages in the Python standard library](http://www.python.org/dev/peps/pep-0411/) we can eat this cake and have it too.

> Other possible future use cases include:
> - Improved HTTP modules (e.g. requests)


This literally appears to have been made with requests in mind (and is mentioned in the examples as an acceptable project). Input @shazow, if you don't mind?


I started a big thread in python-dev about making the Python standard docs more inclusive of third-party solutions: http://mail.python.org/pipermail/python-dev/2012-March/thread.html#117518

Looks like this PEP was updated the same week, perhaps it took some of the feedback into account. Who knows, we'll see what happens. I personally don't have high hopes or desire to deal with the standard library at the moment, given the feedback from that thread. :)


Sorry for the lack of response here. I've given this extensive thought for this and I'm currently :-1: on it. 

---

I briefly talk about it at the end of [this talk](http://www.leancrew.com/all-this/2012/04/where-modules-go-to-die/).
Attempting to address issue #377 by making content on the response model a little smarter

Calling the content attribute on a response that has no content causes an error after the second time. On async requests, prefetch is set to true by default so all calls to content from an async response cause an error.

My change does change the default value for _content. All tests pass and I added a new one to replicate the issue and ensure that it works with my patch.


This is a fantastic contribution. Thanks for the catch/fix!

Bonus cake for tests: :cake:
Name:"Requests" is too general

The name of the module "requests" is really general, resulting in "python requests"-related question hardly searchable with search engine.


A search for ["python-requests"](https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=python-requests) is all good results.


I'm sorry that I'm not specific enough, now for an example:

I want to search for "python requests post xml", since "requests" isn't really specific to the name of the module, most of the result returned is not "requests" module-related.


Search instead for this:

`python-requests post xml`


We could rename it to requests3. :P


urlrequests5


coreurlrequestslib5


python-coreurlrequestslib5


@EwyynTomato 

It's true that the library's name which consists of two general words doesn't help when searching the web. Searching would be easier if the lib was named for instance pyrequests following the common pattern among Python libs. Well, I guess you have to deal with it.

Searching for word `python-requests` (http://stackoverflow.com/search?q=python-requests&submit=search) gives 45 results and searching for tag `python-requests` (http://stackoverflow.com/questions/tagged/python-requests) gives 10 results. If you don't find the question you're looking for please ask one.


Well, I guess as long as I limit my search my question in stack overflow, this wouldn't be a problem.

And I really should read the docs.
requests.get throws TooManyRedirects in safe_mode

``` python
>>> import requests
>>> requests.__version__
'0.11.1'
>>> res = requests.get("http://httpbin.org/redirect/3", config=dict(safe_mode=True, max_redirects=2))
Starting new HTTP connection (1): httpbin.org
"GET /redirect/3 HTTP/1.1" 302 229
"GET /redirect/2 HTTP/1.1" 302 229
"GET /redirect/1 HTTP/1.1" 302 215
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/spulec/Development/yipit-env/lib/python2.7/site-packages/requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "/Users/spulec/Development/yipit-env/lib/python2.7/site-packages/requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Users/spulec/Development/yipit-env/lib/python2.7/site-packages/requests/sessions.py", line 208, in request
    r.send(prefetch=prefetch)
  File "/Users/spulec/Development/yipit-env/lib/python2.7/site-packages/requests/models.py", line 597, in send
    self._build_response(r)
  File "/Users/spulec/Development/yipit-env/lib/python2.7/site-packages/requests/models.py", line 226, in _build_response
    raise TooManyRedirects()
TooManyRedirects
```


Thanks for the report! This is definitely a bug.


Different traceback now :-/

```
>>> import requests
>>> res = requests.get("http://httpbin.org/redirect/3", config=dict(safe_mode=True, max_redirects=2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 52, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 232, in request
    for cookie in r.response.cookies:
TypeError: 'NoneType' object is not iterable
>>> requests.__version__
'0.12.0'
```


Fixed!
Fix decoding of invalid %encodings in URLs

Currently a URL with invalid %encodings will cause an exception to be thrown in unquote_unreserved.

This patch reimplements unquote_unreserved to be almost identical to urllib.unquote, thus fixing the problem and simplifying the code.


Awesome, thanks! 

Looks like there's a merge conflict now. Would you mind bringing it back up to date?


Rebased and fixed, should merge cleanly now.


You're the best. Thanks!


Reverting for now.
Broken HTTPResponse.iter_lines() in 0.11.1

Code like `for line in http_response.iter_lines():` works fine with 0.10.8 but when used with 0.11.1, lines are not yielded as expected and when the server closes the connection, the following exception is raised:

`httplib.IncompleteRead: IncompleteRead(354 bytes read)` (where that 354 bytes is about the size of the lines I should I received ;-))

I did not have time to further investigate but opened this ticket in case you have an idea what recent change may have introduced this issue.

Will try to dig in further when I have some spare time.


"Issue" is really #539. Not really broken. Just working differently :-)

Closed and commented on #539.
fixed bug in setup.py where running from outside project dir would fail

If you run setup from outside the project directory it will fail on attempting to open the README, etc. files.

for eg.

python requests/setup.py install

now works because the full path name is used in reading the file. implement as a separate func since it is used three times. 

this is a pretty common bug, but I see it a lot because of my deploy scripts


Unnecessary, but good changes. :) 

Thanks!
Multipart (files=) Strings


do you mean like: 
r = requests.post(url, files={'file.txt': 'some_data', 'rb')})


exactly


https://github.com/kennethreitz/requests/pull/559


just occured to me you probably want unicode string support. I'll update tomorrow


Hows the Pull request look? Consider yourself bugged.


I thought you were going to update? :P


i did. commit 3d097b2cfea47917d03486951b54f937868bbd15 covers unicode. wasn't aware, but since on same branch it included with original pull request.


i'll go about updating the docs as well if its approved.


docs added


:cake:
Distibute Workload

This is not scalable.

---
### First steps:
- Think through contribution workflow
- Heavily document said workflow.
### Next steps:
- Move personal emails to mailing list
- Heavily encourage someone that "gets it" to become heavily involved and get collab'd


Development is quite fine, actually. Support / Questions are the issue. 


A guide through the modules and bug triaging would be great too.


+100


From my point of view, the PyDev collaboration workflow is the best: http://pydev.org/about.html
- Users support => stackoverflow
- Discussion between contributors => mailing-list

Detailed explanations from Fabio (PyDev project leader): http://pydev.blogspot.com/2012/02/pydev-forums-stackoverflow.html


Documentation on triaging would be great. I've gone through a large number of Issues that have been merged and could be closed but don't know how best to convey this knowledge.


http://geemus.com/blog/2011/11/27/less-is-more


:thumbsup:


If you have some time, you should watch this video about the opensource community construction: http://video.fosdem.org/2012/maintracks/k.1.105/You_re_doing_it_wrong_.webm
I've seen his presentation in live at FOSDEM, it was very interesting.
The abstract: https://archive.fosdem.org/2012/schedule/event/really_grow_community.html
Accept  "dictionary update sequences" everywhere.

Blocking OAuthLib, right @idangazit ?


Not sure.

Updating data dicts is problematic because order might be sensitive, however we can get around that by just returning the form-urlencoded string.


While I'm looking at #785, I was wondering what exactly you meant by this. Do you mean something like:

``` python
from requests import session

s = session(params={'page': '2'}, headers={'Content-Type': 'application/json'})
# use s for some requests
s.params.update(page='3', user='sigmavirus24')
```


@idan @kennethreitz you never answered my question, and beyond that accepting update sequences everywhere seems a bit unreasonable


This was a terrible idea.
Treating resource as file-like object seems to not work

For instance, in previous versions of requests it was possible to pass a repines to `open()` which would read the resource, such as an image, from the response object.

It seems as though the `raw` attribute should handle this however when trying to pass the raw attribute of a response to `open()` an error is raised:

```
>>> open(requests.get('http://www.iana.org/_img/iana-logo-pageheader.png').raw)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: coercing to Unicode: need string or buffer, HTTPResponse found
```

What is the proper way of handling this kind of functionality? Is there a preferred method for pulling down raw data and parsing it as a file-like object?


Note that while you can use StringIO to convert `response.content` it doesn't seem to play nice with `open()`. However for my particular use case (pulling down an image) StringIO did the trick.


You should be using `response.raw` directly instead of running `open` on it. It should work with most things.


Cool. Thanks!
Updates Tests to use new Exceptions

Tests now use MissingSchema and InvalidSchema added in b973d08b0e46d25c4e5b9b7e8e5e6a4ca440594a.


This is great, but it looks like it was done off a really old copy of the codebase. Can chance you could update this with the latest version of `develop`?


Hrm, must have gone off master instead of develop. I'll update it.


Thanks a ton.


Going to start over, I merged my master with your develop.
Response.content raises timeout in safe_mode

This seems to happen very infrequently and I'm not able to recreate it, but I figured that I'd throw the traceback up here and see if someone can figure it out.

``` python
response = requests.get(url, timeout=10, config={'safe_mode': True})
if response.ok:
  content = response.content
```

File "/requests/models.py", line 757, in content
    self._content = None
File "/requests/models.py", line 668, in generate
    chunk = self.raw.read(chunk_size)
File "/requests/packages/urllib3/response.py", line 168, in read
    self.release_conn()
File "/python2.7/httplib.py", line 561, in read
    s = self.fp.read(amt)
File "/python2.7/socket.py", line 384, in read
    raise


This should be better now.


Great, thanks.
Move pre_request to a more usable place, followup to #382

Having the hook called so late makes it impossible to overwrite the HTTP request
calling behavior and set self.sent to True in case of possible cache matches
for example.

This way we can interfere before we waste any CPU time to calculate
a request.

As a side note I am not sure if this whole process should be splitted into two parts: pre_prepare_request and pre_request so that one hook really can interfere before we calculate anything and the other can use the certificate and gathered authentication information to make it's decisions... not sure 


Hmm, interesting. Maybe a whole new hook should be added instead.


Yea, as I said, but I am pretty unsure about where would be the best place, suggestions welcome, I can adapt the patch.

The question is how many hooks are user friendly in the end and if it makes even sense to create a whole bunch of new hooks like for:
- prepare
- pre_request
- post_request


I've encountered the same problem with `pre_request` at beginning of writing [requests-cache](https://github.com/reclosedev/requests-cache) (solved with monkey-patching `send`). 

I'm agree that new hook like `pre_send` can be better. Another option, is to check for `if not self.sent`  right after `dispatch_hook`.


+1 for `pre_send`


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1217420) (merged f2590187 into 2159c802).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1231720) (merged f2590187 into 2159c802).


Merged, thanks guys!
Suggestion to change iter_lines() logic

`def iter_lines(self, chunk_size=10 * 1024, decode_unicode=None):`

`iter_lines()` uses chunk_size, rather than that it would be better if `\r\n` is used, `iter_lines()` is used when connection is open for long timing, I recently tried the example(twitter streaming api), code - http://kracekumar.com/post/19556427690/n00bs-epic-research-on-twitters-streaming-api-on snippet found exception like `String Index Out of Range`. 


Here is the example error, 
IndexError: string index out of range
[2012-03-29 15:12] ERROR: Generic: Uncaught exception occurred
Traceback (most recent call last):
  File "collector.py", line 63, in <module>
    store_live_tweets()
  File "collector.py", line 54, in store_live_tweets
    for line in r.iter_lines():
  File "/home/kracekumar/lib/python2.7/requests/models.py", line 729, in iter_lines
    if lines[-1][-1] == chunk[-1]:
IndexError: string index out of range


I changed chunk_size to 2*1024, seems to work. If actual size is less than chunk_size wondering what will happen?(last message sent)


This shouldn't happen anymore.


So what is the take away, user should specify according to the content, or requests will figure out ?


Today again I faced the same issue. 


Same thing here (Twitter streaming API). Please reopen.

```
Traceback (most recent call last):
  File "twtrack.py", line 26, in <module>
    for line in r.iter_lines():
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 729, in iter_lines
    if lines[-1][-1] == chunk[-1]:
IndexError: string index out of range
```


I'm sorry, I installed requests without using pip's `-U` switch and didn't realize that an older version of requests was already installed... After the update it seems to work :)


FYI, I use version `0.10.8` stil same error. 


Sorry for all that buzz. A friend of mine uses requests v0.11.1 and the original bug (string index out of range) appeared again.


I have tracked down the iter_lines IndexError to this: If the chunk is `''` (empty string), `chunk[-1]` will fail. If the chunk ends with `'\n\n'`, `lines[-1][-1]` will fail, as `lines[-1] = ''`.

So to check `lines[-1][-1] == chunk[-1]`, you must make sure that `lines and lines[-1] and chunk`.
latest source doesn't supports Python 2.5.4 on OS X

```
$ sudo python2.5 setup.py develop
Traceback (most recent call last):
  File "setup.py", line 6, in <module>
    import requests
  File "/Users/lee/Downloads/py-pkgs/requests/requests/__init__.py", line 25, in <module>
    from . import utils
  File "/Users/lee/Downloads/py-pkgs/requests/requests/utils.py", line 232
    return b"".join(L)
             ^
SyntaxError: invalid syntax

$ python2.5 --version
Python 2.5.4
```


Python 2.5 support was intentionally dropped.


It's not a good idea, lots of modules still support Python 2.5。

On Mac OS X 10.6, I have to use Python 2.5(because of it works with Tkinter, but Py2.6 or Py2.7 doesn't), 
requests is a great project if it could supports Python 2.5 and help us get out of the hell of httplib/urllib/urllib2.


I can assure you, it's a great idea :)

---

You're always welcome to use an older release.
Added url to LocationParseError message

requests/packages/urllib3/connectionpool.py
- Added the url that couldn't be parsed to LocationParseError message.


Sorry, I didn't realize the 2 commits were going to get tracked into the same pull request.

The first commit simply interpolates the URL into error message passed into the exception.

The other commit copies form data into the request that gets made after a redirect.  I couldn't create a unit test for this, but I have solved a problem I'm having at work POSTing data on the intranet after receiving a 301.


Hi @kvonhorn, thanks for noticing that bug in urllib3. Could you make this change to urllib3 and pull request there?


Closing for urllib3.


Is there any chance of getting commit 8c5d06148bfa94ba87f369202b909d8c79897568 pulled, @kennethreitz?
Suggesting changing "async.map(rs)" to "responses = async.map(rs)" just ...

...to show that it returns a list rather than mutating the original? Maybe this is apparent for most, but I was a bit confused =/ http://goo.gl/gE4up


I like it, but that isn't how the interactive interpreter actually works. 


My apologies, technically you would want to

```
>>> responses = async.map(rs)
```

then 

```
>>> print responses 
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>]
```

to show the list of request objects as you do in the original documentation.


all better :) 

thanks!


(Seems I was trying to reply to the OP and somehow only saw his pull and not this thread. Ignore the above post. Need a bit more practice with the GitHub interface...)
[docs] Confusing sidebar title "Support requests"

Every time I read the sidebar in the docs, I stumble upon "Support Requests", automatically assuming I'll find info on issue trackers or other means to request support. While I get: "If you love Requests, consider making a small donation on Flattr: "

But of course "requests" here is the name of the project and not the verb.

Could you maybe change the title to "Donate"?


Ah, nice observation. "Donate" sounds good :)


Honestly, I might remove it soon. It doesn't really do anything :)


Wow, it's up to ~$56.09 now. I guess I will keep it :)
Allow disabling of cookie persistance/sending

It should be possible to disable the cookie persistence functionality of a `Session`, possibly by passing the `config` parameter `False`.

Assuming it is not already possible, it would also be good to be able to stop cookies from being sent when generating a request.


+1


On second thoughts, using `cookie=False` to signify not wanting to store or send cookies would be ambiguous. That wouldn't allow for someone to have default cookies for their session, but not store cookies sent by servers.

So maybe a `store_cookies` option would be a better idea.


@johtso are you working on this?


@barberj I'm afraid I haven't had a chance to work on this, no.


I'd like to work on this issue.

Which one of below are best for API design?

``` python
import requests

s = requests.session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies", store_cookies=False)
```

``` python
import requests

s = requests.session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
config = dict(store_cookies=False)
r = s.get("http://httpbin.org/cookies", config=config)
```


@berkerpeksag awesome! The second one (with `config`) is definitely best


@kennethreitz thanks!


Relevant: http://tools.ietf.org/html/rfc6265#section-7.2
They call it a **private browsing** mode


My understanding is that:
1. Private browsing means "save all cookies during the session, but don't persist them to disk when the session is closed" --- that is to say, the current behavior of the `session` object
2. This ticket is to implement the option, "discard all cookies during the session", which is somewhat different

@berkerpeksag there's an unmerged branch out for review (#565) that will modify cookie handling. Wild guess: if it gets merged, you could implement this functionality using a `MockCookieJar` that discards all cookies.


@berkerpeksag actually, `MockCookieJar` is a terrible suggestion, please ignore me :-)


@berkerpeksag Sorry I fixed the one which you had picked originally ( I followed the suggestions you made here though :D ). I am new to python and I found this one to be one of the easier ones to fix.


@amalakar no problem, have fun with Python :)

@slingamn thanks for the suggestions :)


Done :)
Some notes for the quickstart tutorial

Decided to do a smaller script today that uses requests, so I opened up the tutorial and spent too much time trying to figure out following two things:
- What should I import to get started?
  
     IMHO, first code example in the tutorial should be: `import requests`
- How can I nicely build urls for simple get request that contain variables/parameters...
  
     The `params` argument to `Request` class should be mentioned to show how to build urls with variables. URLs with various variables are quite common and finding the `params` is not trivial unless you know where to look (and I eventually had to ask).


+1 especially for the missing import requests. People ALWAYS copy and paste the example code into a console. If it doesn't work you'll loose half of the audience. Please fix it


Half of python developers don't know they have to import modules?


They know, but they don't know what module should they import. Consider e.g. "from datetime import datetime" or "pip install scikit-learn" but 'import sklearn'. 


No, they do. But most of the people just copy and paste. So please do them a favor and do not make them search for what to import. It's faster and easier to use.


Have the criteria for this issue been met? This issue seemed like a good place to start contributing, but it looks like information for `import` and `params` has been included in the pull request Lukasa linked. Thanks.


Pull #619 has addressed these issues. Closing \o/


@plaes Awesome, thanks for the quick response.


Cool, it's much better now. It is not perfectly copy'n pastable because of the >>> prefix in each line.
@kylerob I would remove that ">>>" because for every pythonista it is clear that you can type into the terminal. A side effect would be that one can copy and paste.


@jups23 that's a good idea. @kennethreitz, what do you think about me removing all of the >>> prefixes from all code examples to allow easy copy/paste?


`>>>` should only be present in a code block when it's necessary: when it's showing interactive examples.


While we're here, the first code block [here](http://docs.python-requests.org/en/latest/user/quickstart/#response-status-codes) isn't rendering properly in my browser (Chrome 20.0.1132.34 beta and Safari 5.1.7 (7534.57.2) on OS X 10.7). Might be worth tackling as part of this issue as well. (Not sure why it's not displaying properly, because [GitHub shows it fine](https://github.com/kennethreitz/requests/blob/develop/docs/user/quickstart.rst#response-status-codes).)

I'm all for @kylerob removing the >>> prefixes where appropriate. Might even be worth adjusting the way some of the text flows to remove as many as possible, allowing for more copy-pasting.
AuthBase has unwieldy access to r.params

Because `_enc_params` seems to already be built by the time auth is applied, and r.params is no longer a dictionary, authentication schemes that require signatures to be passed as a param (like LastFM's, possibly others) end up being kind of unwieldy.  Here's some working LastFM auth code, for example:

``` python

from hashlib import md5
from requests.auth import AuthBase
from urllib import urlencode

key = "xxx"
secret = "xxx-secret"

class LastFMAuth(AuthBase):
    """Implement LastFM's annoying authentication."""
    def __init__(self, session_key):
        self.session_key = session_key

    def __call__(self, r):
        r.params.append(('sk', self.session_key))
        r.params.append(('api_key', key))
        encode = lambda x: unicode(x).encode("utf-8")
        signature = list(sorted(r.params))
        signature = "".join([encode(k) + encode(v) for k,v in signature]) + secret
        r.params.append(("api_sig", md5(signature).hexdigest()))
        r._enc_params = urlencode(r.params)
        return r
```

I think it would suffice to delay setting `_enc_params` until after auth is applied;  even if such a thing is documented it still feels kind of strange.


I believe this has changed recently.


request.params is indeed a dictionary in requests-0.13.1, which means the code in this ticket no longer works and the `_enc_params = urlencode(...)` step is no longer necessary for auth using params.


@jmoiron @kennethreitz I'm sorry, but how can it be currently done?
Building API client for another service that uses signature-based authentication and when I'm trying to add parameters by editing r.params, I receive:
AttributeError: 'PreparedRequest' object has no attribute 'params'


@pitsevich the API has changed in the most recent releases of requests. As of 1.x you have no access to the params attribute. If you want to modify the params you'll have to parse the url and unquote the param string, add your params, re-quote it and then unparse the url.


@sigmavirus24 thanks, this worked!


@pitsevich glad to help.
Response.ok raises Timeout Exception in safe_mode

``` python
>>> import requests
>>> requests.__version__
'0.10.8'
>>> res = requests.get("http://localhost:8000/", timeout=1, config={'safe_mode': True})
>>> res.ok
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "../requests/models.py", line 650, in ok
    self.raise_for_status()
  File "../requests/models.py", line 800, in raise_for_status
    raise self.error
Timeout: Request timed out.
>>> 
```

This certainly feels unintuitive as a user. It should probably always return a boolean.

A simple solution is to have raise_for_status catch RequestException instead of just HTTPError, but I'm not sure if that would have other implications.


This is definitely a bug.


Merged!
requests fails on request.get call in Google App Engine python environment

Duplicate: http://stackoverflow.com/questions/9762685/using-the-requests-python-library-in-google-app-engine

Google App Engine is used by a lot of developers. I use the python environment myself. It's sandboxed and can't rely on any non-Python code. It also has other restrictions (for example, you need to use this patched version of urllib3  https://github.com/shazow/urllib3/issues/61.)

I can successfully

> > > import requests

but then

> > > response = requests.get('someurl')

fails with the following traceback. Does anyone know what's going on?

Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/admin/init.py", line 317, in post exec(compiled_code, globals()) File "", line 6, in File "/Users/Rohan/Dropbox/MuktiTechnologiesINC/MuktiTechnologies/GAE/humanmictweet/GAE/libraries/requests/api.py", line 52, in get return request('get', url, *_kwargs) File "/Users/Rohan/Dropbox/MuktiTechnologiesINC/MuktiTechnologies/GAE/humanmictweet/GAE/libraries/requests/api.py", line 40, in request return s.request(method=method, url=url, *_kwargs) File "/Users/Rohan/Dropbox/MuktiTechnologiesINC/MuktiTechnologies/GAE/humanmictweet/GAE/libraries/requests/sessions.py", line 208, in request r.send(prefetch=prefetch) File "/Users/Rohan/Dropbox/MuktiTechnologiesINC/MuktiTechnologies/GAE/humanmictweet/GAE/libraries/requests/models.py", line 458, in send self.auth = get_netrc_auth(url) File "/Users/Rohan/Dropbox/MuktiTechnologiesINC/MuktiTechnologies/GAE/humanmictweet/GAE/libraries/requests/utils.py", line 43, in get_netrc_auth for loc in locations: File "/Users/Rohan/Dropbox/MuktiTechnologiesINC/MuktiTechnologies/GAE/humanmictweet/GAE/libraries/requests/utils.py", line 40, in locations = (os.path.expanduser('~/{0}'.format(f)) for f in NETRC_FILES) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 260, in expanduser userhome = pwd.getpwuid(os.getuid()).pw_dir AttributeError: 'module' object has no attribute 'getuid'
Adds support for Google App Engine

Issue #194


I'd be happy to accept a pull request that works with  2.6–3.2 (I know App Engine supports 2.7), but the urllib3 changes need to occur in the urllib3 repository.


There are several problems with this pull request:
- Attempts to reintroduce Python 2.5 support while breaking Python 3 support.
  - Removes usage of bytes
  - Removes new exception syntax
  - Backports LifoQueue which is unnecessary since urllib3 works fine with standard Queue when Lifo is not available.
- Removes is_connection_dropped which will regress several important bugs in non-AppEngine environments. A better solution is to check if select is available and pass if it's not.
- Some weird getattr stuff in the set_cert method, not sure I understand what it's doing and why.

Overall it makes urllib3 and Requests very specific to Python 2.5 and AppEngine. I'm not a fan of this. I'd say Python 2.6+ and Python 3+ support is more important than AppEngine, but I'm confident we can do a better job to support AppEngine without breaking everything in the process.


Interestingly 2.5 support isn't even necessary for App Engine, which has had a good Python 2.7 runtime for a while now.  (The 2.5 runtime is single-threaded CGI, so it requires many more instances to handle any significant request volume; in other words, it is legacy).


Urllib3 supports app engine now.


:cake:


Btw, @kennethreitz: Requests still does not support AppEngine, since you're depending on various filesystem things for configuration magic that AppEngine doesn't have. Not sure if you want to reopen this bug.


@shazow those should all be supplementary. I guess I should fire up an app engine app and find out :)


@kennethreitz Here's a failure for example: http://stackoverflow.com/questions/9762685/using-the-requests-python-library-in-google-app-engine/9763217


I wasn't expecting this pull request to be accepted: I simply submitted it for other people to take a look at. I need `requests` to work with Python 2.5 on App Engine, which is why I made some of the changes I did. I'm happy to create a separate pull request to make `requests` run on Python 2.7 on App Engine without breaking forwards-compatibility, once the updated `urllib3` code is pulled in: I'm not sure what the protocol is for that.


If anyone cares, this would still be awesome.


yeah, it would still be appreciated! @singingwolfboy, is requests using the 2.5 environment functional on App Engine as is..?


@rdixit It was working when last I touched my fork. I haven't touched it in a _long_ time, though.
Adding headers to a request causes .get() to not follow redirects

The GET request is not following redirects when setting a header even with allow_redirects set to True. In this example setting a header manually for the user-agent causes the request to not follow redirects. I would expect that even with setting headers it would give the same output as not setting a header.

```
url = "http://t.co/ms7J7HJ0"
r = requests.get(url)
r.url 
>>> u'http://www.inc.com/magazine/20070501/column-guest.html'

headers = { 'User-Agent' : 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21' }
r = requests.get(url, headers=headers, allow_redirects=True)
r.url
>>> u'http://t.co/ms7J7HJ0'

```


What you're actually seeing here is that `t.co` is actually doing an HTTP redirect for scripts and a JavaScript Location redirect (in HTML) for browsers :)


Ok so by setting user-agent header I'm getting the js redirect, but it does nothing since it's not a real browser? Just trying to understand the behavior. Is that how bit.ly works as well?


Exactly. I didn't know this until just now ;)

If you look at the response.content of the second request there you'll see :)


ok cool, makes sense. Thanks!
Remove redundant generator for chunked transfer encoding

Generator for non-chunked encoding uses method read of
httplib.HTTPResponse which also checks and correctly handles
chunked encoding.


 @ronnypfannsch ?


Actually this also fix at least one bug resulting from interfering HTTPResponse internals: I cannot tell you the principle exactly, but for some requests with chunked encoding and empty body like method HEAD, it keeps hanging when you try to access property content:

import requests
r = requests.head('http://www.httpwatch.com/httpgallery/chunked/'
r.content


Thanks!
Access to response.content fails with response hook

Trying to access the response.contents  in an async setting and, f.e. pass them to a variable, fails somehow: 

Traceback (most recent call last):
  File "build/bdist.macosx-10.6-x86_64/egg/gevent/greenlet.py", line 390, in run
    result = self._run(_self.args, *_self.kwargs)
  File "build/bdist.macosx-10.6-x86_64/egg/requests/models.py", line 569, in send
    self.response.content
AttributeError: 'int' object has no attribute 'content'
<Greenlet at 0x102584eb0: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with AttributeError

However, printing the response via hook methods works correct...


I ran into this today where if the function called by hook returns something it causes this attribute error.

i.e. 

```
def good(x):
    print(x.text)

def bad(x)
    return x.text
```

This is more just confirmation.  I don't know if this is expected behaviour...


I have this problem, too. 

I am using grequests to send POST requests to an external API, which gives me a nice speedup. However, when I use a hook method and access response.text, this happens:

Traceback (most recent call last):
  File "/python2.7/site-packages/gevent/greenlet.py", line 328, in run
    result = self._run(_self.args, *_self.kwargs)
  File "python2.7/site-packages/requests/models.py", line 630, in send
    self.response.content
AttributeError: 'str' object has no attribute 'content'
<Greenlet at 0x10143e5f0: <bound method Request.send of <Request [POST]>>(prefetch=True)> failed with AttributeError

Despite this message, the code still works. 


I have experimented with different ways of setting prefetch to False. This makes no difference, the same error still appears.

Simply commenting out the offending line solves my particular problem. My hook method does everything I want with the response so I don't need this. 

```
        # If prefetch is True, mark content as consumed.
        if prefetch or self.prefetch:
            # Save the response.
            #self.response.content
            pass
```

Someone with a better grasp of the overall design should probably take a look at this.
Use of requests.async removes select.poll

You can reproduce this:

```
In [1]: import select; select.poll
Out[1]: <function select.poll>

In [2]: from requests import async

In [3]: import select; select.poll
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/opt/VirtualEnvs/weather-daemon/src/wxcdc/wxcdc/<ipython-input-3-a25f519ae2a6> in <module>()
----> 1 import select; select.poll

AttributeError: 'module' object has no attribute 'poll'
```

The line that seems to be causing this is the monkey patching done in async:
https://github.com/kennethreitz/requests/blob/develop/requests/async.py#L20

```
curious_george.patch_all(thread=False)
```

We can remove the patching of select from this and things seem to work. Do we need to patch select? Should we look into what else we don't need to patch in this documentation: http://www.gevent.org/gevent.monkey.html ?

```
curious_george.patch_all(thread=False, select=False)
```


Wow, that's bizarre. Thanks.
Client SSL Cert Support

Exposed key_file and cert_file in requests, to support https client certificates.


Ideal:

``` python
# Default, CA Bundle, like browser. Today.
requests.get(..., verify=True)

# No SSL check. Today.
requests.get(..., verify=False)

# Private PEM
requests.get(..., cert='/path/to.pem')

# Private Cert / Key
requests.get(..., cert=('cert', 'key'))

# No SSL check.
requests.get(..., cert='/path/to.pem', verify=False)
```


My apologies, but I'm not currently in a position to be able to supply a good, or repeatable, test-case for this new functionality, as it requires a correctly configured server.

I've tested this in my own environment, and it suits my needs well enough; if there are any issues, please let me know.


Same. I trust that it works properly though. If it doesn't, it's urllib3's fault :)


:cake:


Confirming - works perfectly :) Thank you very much.
_LegalChars should include more characters.

I found _LegalChars in monkeys.py file is not include "@" and it will output as Cookie: key="@abc" instead of Cookie: key=@abc

so that the server can not recognize the key of cookies. I think maybe more characters should be including in _LegalChars.


The whole reason that file exists is to add more and more _LegalChars, one at a time. :) 

Send a pull request!


This code looks isolated now:

```
[shivaram@good-fortune ~/workspace/requests]$ git grep monkeys
requests/packages/oreos/core.py:from .monkeys import SimpleCookie
requests/packages/oreos/monkeys.py:oreos.monkeys
```


Oreos has been removed.
Feature any http verb in docs


As @kennethreitz knows, I'm preparing to use requests in a different project, and was testing out the GitHub API. The following is something I did from the python command line interpreter:

``` python
import requests
a = ('sigmavirus24', 'not_my_real_password')
url = 'https://api.github.com/repos/sigmavirus24/issues.py/issues/comments/4524692'  # This no longer exists, you'll see why
data = u'{"body": "Amend comment."}'
r = requests.patch(url, data, auth=a)
# Checked what GitHub returned (response code 200)
r = requests.request('DELETE', url, auth=a)
# GitHub responded with code 204 -- successful deletion of a comment on an issue.
```


@kennethreitz: Whereabouts in the docs did you want these? The only place that currently exists that looks suitable is docs/quickstart.rst, but that document is getting increasingly large. Should we consider busting the verbs out into a new file? Or are you happy to extend the quickstart? Either way, I'm happy to write these and add them into #619.


Probably in the "advanced" section


I love requests so much, thanks for your excellent work!


Sounds great! I'll get right on it!


@Fighter42 I don't know why you commented 11 times but this is already taken care of [here](http://docs.python-requests.org/en/latest/user/advanced/#http-verbs)


I feed my cat!


@RAINCEN Really? _Really?_


@Lukasa the emails I received from @RAINCEN had the same message that @Fighter42 posted about 20 times (of which I deleted all but one). I have to wonder if these are just spam accounts or if perhaps this is related to [recent attempts to brute force passwords on accounts](https://github.com/blog/1698-weak-passwords-brute-forced). Regardless, it's probably advisable to just ignore everyone who makes similar comments on this issue. It's been closed for so long.


github great
LookupError when you read the text of pages with an unknown encoding

On one of the pages that I'm parsing I got:

  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 777, in text
    content = str(self.content, encoding, errors='replace')
  LookupError: unknown encoding: ISO-LATIN-1

I think it would be more sensible to try a default encoding (utf-8 and windows-1252 as a fallback) when Python doesn't support the encoding defined in the page.

Also, using encoding.lower().replace('-latin-', '-8859-') will go a long way fixing the common case of pages using iso-latin-N instead of iso-8859-N.


duplicate of #338


We currently have a pretty robust encoding detection system, but it's only a fallback for when the server doesn't specify an encoding.


We're currently blindly trusting that the encoding that the server is responding with will exist. 

Perhaps a map of encodings (e.g. ISO-LATIN-1 => latin1) is best?


Closing for #338