Raibek/requests-github-issues huggingface dataset:
Strange encoding(?) problem

When I try to GET a twitter API search url, with `requests` i get the following output:

```
>>> requests.get('http://search.twitter.com/search.json?q=pythonrequests').read()
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x02\xffl\x8f\xc1\x0e\x820\x10D\xffe\x0f\x9e\x1aS\xc0"4!~\x881\r\x81U\x9a\x14(\xdb\x92H\x8c\xff\xee\x1e@\x0fx\xdb\xc9\xbc\x99\xdd}A3\xf6\xdea\xc4\xd6\xd8\x01\xb4<\xcaT@_?\x8dmA\'IY\x9c\xb2$S*?\xa7e\xa1r\xb9y&D\x02\r{\x00\x04\xf8\xfa\x81\x9c\x150\xcdH\x0bS~\x89\xdd8\x10\xb2\x0e10Ax\'\x0c\x9d\x99\xc9\xb1}\tvh\x90K\xab}\xdda\xaa\xfe\xa4\xc3\xecx\xd2\xd7\xdbW\x18\x8fd\xd6\xc5J\xc0\xd6\xc8\x0f\xfd\xc4z\xb3\x84\xf7\x07\x00\x00\xff\xff\x03\x00g3\x0ex\xf6\x00\x00\x00'
```

These are the response headers:

```
>>> requests.get('http://search.twitter.com/search.json?q=pythonrequests').headers
{'content-length': '173', 'x-varnish': '1594769474', 'content-encoding': 'gzip', 'age': '0', 'expires': 'Fri, 30 Sep 2011 18:43:40 GMT', 'vary': 'Accept-Encoding', 'server': 'tfe', 'connection': 'close', 'via': '1.1 varnish', 'cache-control': 'max-age=15, must-revalidate, max-age=300', 'date': 'Fri, 30 Sep 2011 18:38:40 GMT', 'content-type': 'application/json;charset=utf-8'}
```

I can't utf8-decode the returned string.

```
>>> response.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/danilo/ve/studentenportal/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte
```

Using urllib2, the response is returned properly decoded.

```
>>> urllib2.urlopen('http://search.twitter.com/search.json?q=pythonrequests').read()
'{"completed_in":0.0090,"max_id":119843090793111552,"max_id_str":"119843090793111552","page":1,"query":"pythonrequests","refresh_url":"?since_id=119843090793111552&q=pythonrequests","results":[],"results_per_page":15,"since_id":0,"since_id_str":"0"}'
```

Is this a bug?


This is not a bug (yay!).

Use `response.content` instead of `response.read()`. Read is giving you the raw response bytes from the connection. It's for advanced usage.


Oh, sorry for not reading the docs carefully enough, and thanks a lot for the quick comment :) I'm urllib2-spoiled ;)
Fixed #174 and refactored urls quoting/concatenation in one function in utils.py.

We were only quoting path but params and query also need to be encoded. Everything is now in utils.py / get_clean_url (feel free to rename the function if you wish).


Nice one! :sparkles: ⍥ :cake: om nom nom


Beautiful! I can merge this in once it's brought up to date with the current `develop` branch. :)


Done, but btw, why don't we have a response.url anymore?


We will, as soon as I figure out where to get it from w/ httplib :)
PEP8ified all the things

I ran the pep8 checker (version 0.6.1) on the code base and tried to clean up as many pep8 errors as I could.  Everything except line-width errors are gone (I was worried about messing up the documentor since I.haven't fiddled with it).  Code base didn't have very many pep8 errors (awesome!).


Thank you, sir! 

I'm quite religious about these sort of things myself. But contributors, not so much. :) 

:sparkles: :cake: :sparkles:


I love it when contributors PEP8'ify all the things. Yay!
redirect fails

When response contains redirection, it fails to reencode correctly. For example, headers contain:

``` python
{'location': '/relative/uri/init.do;jsessionid=SOMEDATA!234567889?param=value'}
```

When redirecting, it becomes urlencoded, which is not correct, since results in 404:

```
https://example.com/relative/uri/init.do%3Bjsessionid%3DSOMEDATA%211234567889%3Fparam%value
```


This is not an issue with master branch. Pip version only. 


master branch or develop branch?


Right, develop:

```
(new) skapis requests # git branch
* develop
```


Excellent, just making sure. 

I'm glad you raised this issue, actually.


Related #174
Feature/improve test structure

Successively adding unit tests for request here. Reopened from [here](https://github.com/kennethreitz/requests/pull/166).


![i'm lovin it](http://www.freelancewritinggigs.com/wp-content/uploads/2010/09/Lovin-ItMcDs_Print.jpg)


Excellent work. Thanks :)


So pull requests get closed on merge automatically? Looks like we need to decide on leaving it open and you merge at the end or I'll submit a pull request every time I've added some commits. Or maybe see if `git cherry-pick` is not detected by Github. But I think we should prefer one of the first two solutions.
Expose HTTP status message resp.status_msg

This change adds a new attribute to the Response objects,
status_msg.  This typically returns something like "OK"
for 200 responses or "Not Found" for 404 responses.

The status message can provide some additional semantic information
about the response outside of the response body and response headers.


Is this good?  Or should I squash these revisions together?


Sorry for the delay!

I'm torn about adding this in right now. I'm currently working at porting the entire backend of Requests to urllib3. part of that will be to give the response a simple reference to the original httplib response object w/ `r.raw`. With that, the `status_msg` will be available.


Yeah, I saw poster removal was targeting 0.8 and figured this might be good to have for another 0.6 or an 0.7 release in the meantime.

If urllib3 is around the corner and/or the api's (r.raw.status_msg vs. r.status_msg) won't be compatible then you probably shouldn't take this.


Closing, but I'll add you to the contributors list. If the urllib3 integration falls through (doubtful), I'll def add this.
Improve test structure (proposal)

This might be somewhat related to issue #153. I think it would make sense to split up testing into a unit testing and an integration testing part (loosely defined). In the unit tests the actual methods are tested with everything mocked out. This way it can be assured that the respective external methods are always called with the right arguments. In the integration testing suite it is allowed to call external entities (like `httpbin.org`) for testing the overall function. So I just moved the existing tests there.

I have restructured the tests organization a bit and added some basic unit tests for the `api` module. However I would like to get your feedback on this, before going on. As this will definitely be some work and take a while to reach a somewhat good level of coverage.


Excellent work, thanks! 

I have a good feeling about this. I'm going to think about it for a few days, but I know this is absolutely the direction I'd like to go.


I'm 100% in. I'll merge this in, but keep the pull request open so you can add your further changes.

:cake:


Cool. I'll update and push the unit tests subsequently. What are you using as a test runner? I saw that you removed the tox.ini some time ago.


I typically use `py.test`, with or without tox.


I'd love to open this, if I could figure out how :)


@mrtazz, can you send another one?


Done. Forgot to add the original commits, though. But I thought it's ok since they are merged anyway.


#169
models.Response can't easily be subclassed

I'm making the assumption that there isn't a way that I don't know about :)

I'd like to be able to pass a subclass of `models.Response` to `get`, `post` etc. I know there is a `response` hook however I don't want to do something like `response.__class__ = MyResponse` or add custom methods to the `Response` instance one by one. Being able to subclass would be much cleaner, I have a patch (hopefully I can work out how to attach it).


see #165, it has code
Tests throw error

Running test_requests.py -v with develop version (commit a8e77688f0) I get the following exception:

```
======================================================================
ERROR: test_settings (__main__.RequestsTestSuite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_requests.py", line 283, in test_settings
    with requests.settings(timeout=0.0000001):
TypeError: 'Settings' object is not callable

----------------------------------------------------------------------
Ran 47 tests in 29.054s

FAILED (errors=1)

```


In development :)


Uppps didn't think about that, sorry. :-[
data arg submitted to 'patch' method is ignored

The data parameter passed to the 'patch' method in api.py is not being included in the parameter list passed to the 'request' method. The following change shown below fixes the issue.

In api.py:

```
def patch(url, data='', **kwargs):
    """Sends a PATCH request. Returns :class:`Response` object.

    :param url: URL for the new :class:`Request` object.
    :param data: (optional) Dictionary or bytes to send in the body of the :class:`Request`.
    :param **kwargs: Optional arguments that ``request`` takes.
    """

-   return request('patch', url, **kwargs)
+   return request('patch', url, data=data, **kwargs)
```


Yikes, thanks for the fix.

Much appreciated :)


I think this bug has regressed into the codebase again. I have a failing unit test for this in https://github.com/kennethreitz/requests/pull/215 and was wondering how it got there.


You are right. Looking through the commit history, I'm not seeing the 
commit for that issue. Seems it might have gotten inadvertently wiped.

On 10/20/2011 03:57 PM, Daniel Schauenberg wrote:

> I think this bug has regressed into the codebase again. I have a failing unit test for this in https://github.com/kennethreitz/requests/pull/215 and was wondering how it got there.


Sorry about the bug, guys. The rewrite to urllib3 was previously taking place in the develop branch, but was taking longer than expected (was blocking releases), so I had to back it out into a feature branch. The codebase is in a good state again, and pull requests that were merged into that branch are being backported into new releases.
Possibly fix GET with a 302 redirect response causing params to be glued on incorrectly

A case can arise where a GET request, with params,  receives a 302
redirect and the subsequent request has param garbase appended to the
url.

Example:

```
client> GET /some/url?id=1 HTTP/1.1
client> Host: example.com

server< HTTP/1.1 302 Moved Temporarily
server< Location: http://example.com/some/url?id=1&extra_data=2

## this is the bad request right here. Note the [('id', 1)] stuck on
# the end
client> GET /some/url?id=1&extra_data=2&[('id', 1)] HTTP/1.1.
client> Host: example.com

# the server of course doesn't like it
server< HTTP/1.0 400 Bad request.
```

This issue is caused by the conditional branch in _encode_params for the
second request not urlencoding the list of tuples for the
Request._enc_params attribute.

github note: possible fix for kennethreitz/requests#157
GET with a 302 redirect response causing params to be glued on incorrectly

I am calling a service like this:

```
import requests
resp = requests.get('http://example.com/some/url', params={'id': 1})
```

The service responds with a series of 302 redirects.
I am seeing this exchange.

```
client> GET /some/url?id=1 HTTP/1.1
client> Host: example.com

server< HTTP/1.1 302 Moved Temporarily
server< Location: http://example.com/some/url?id=1&extra_data=2

## this is the bad request right here. Note the [('id', 1)] stuck on the end
client> GET /some/url?id=1&extra_data=2&[('id', 1)] HTTP/1.1.
client> Host: example.com

# the server of course doesn't like it
server< HTTP/1.0 400 Bad request.
```

Any ideas why this may be happening?


Adding a print statement on [line 79](https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L79) of models.py like this:

```
print 'params: %r, %r, %r' % (params, self.params, self._enc_params)
```

Shows me the following output:

```
params: {'id': 1}, [('id', 1)], 'id=1'
params: [('id', 1)], [('id', 1)], [('id', 1)]
```

So it looks like the branch in _encode_params is not doing the right thing for the second request (after redirect).


This seems to fix it, but I don't know if it has other unintended side effects.

```
diff --git a/requests/models.py b/requests/models.py
index 6180e96..b3e32d2 100644
--- a/requests/models.py
+++ b/requests/models.py
@@ -278,7 +278,9 @@ class Request(object):
                                    v.encode('utf-8') if isinstance(v, unicode) 
             return result, urllib.urlencode(result, doseq=True)
         else:
-            return data, data
+            if data == None:
+                return data, data
+            return data, urllib.urlencode(data, doseq=True)


     def _build_url(self):
```


previous print output now looks like this:

```
params: {'id': 1}, [('id', 1)], 'id=1'
params: [('id', 1)], [('id', 1)], 'id=1'
```


Excellent! Thanks for both the bug and the apparent solution.

Can you send a pull request?


Hmm. Note that the patch provide does not guard against the addition of duplicate results in the request.

I am seeing this now (note the additional copy of id=1 in the qs):

```
client> GET /some/url?id=1&extra_data=2&id=1 HTTP/1.1.
client> Host: example.com
```

Is there a mechanism to _not_ supply params again for redirect requests, but to simply follow the redirect?


Also, by simply appending (even if the location redirect provides it) identical elements, could there arise a situation where a form array post type query string could accidentally get more elements added to it?

Such as: `http://example.com/some/url?arr[]=val1&arr[]=val2`


In response to your question, I sure can send a pull request. I just wanted to make sure the solution was tenable (and correct) before doing so.


pull request kennethreitz/requests#158


Fixed!


great! thanks. :)
get_unicode_from_response does not check charsets from meta tags

Also affects Response.content which calls this utility method.

According to documentation, utils.get_unicode_from_response should check meta tags for additional charsets to try (presumably with get_encodings_from_content) before falling back to using unicode with errors="replace". However, the code seems to miss this step, rather the encoding is extracted from the headers, tried and then directly falls back to replacement.


Sorry about that — the documentation is in development and needs to be update. 

The decision was that Requests is an HTTP module, not an HTML module. There's a helper function available from within `requests.utils` that will attempt to decode the html content for you based on meta tags though.


I see. Thanks for the quick reply. Let me know if you prefer I close this or leave it open.


I'll leave it open to remind me to update the docs :)

:cake:


Hi, I want to reopen issue, because property .text came in game.

You actually use content to detect encoding with _detected_encoding() but forgot get_encodings_from_content() branch.
chardet wont use html meta too and I can't see how to hint .text to use meta. 
Should I write "response.encoding = request.utils.get_encodings_from_content(response.content)" before using response.text? I think I need config option to do it automaticly or just try get_encodings_from_content() before chardet in _detected_encoding()

Also, fallback to 'ISO-8859-1' in get_encoding_from_headers() when no suitable headers found looks wrong for me because it makes false sense of determined encoding. Why "content-type: text/html" means ISO-8859-1? It means utf-8 more often in my non-english world.


This is intentional, actually.

The initial implementation of response encoding did take meta tags into consideration, but I decided to remove it to keep things within a small scope. In short, meta tags are HTML, not HTTP :)

You can see #480 for a bit more detail.


Actually, you are usng that html in _detected_encoding() with unreliable result (via chardet).

If I understand what regular expression based parsing is error prone, maybe I can get option like 
s = requests.session(config = {'look_for_meta':True})

And what about 'ISO-8859-1' in get_encoding_from_headers()?


@Savier: `_detected_encoding()` is checking the bytes of the response for an encoding. Chardet does a damn good job to :)
Malformed headers cause later headers to be ignored

If an HTTP header is malformed then the headers after it will be ignored. 

E.g. the line after the "Server" header is missing a key and value. The "Date" and "Server" headers are stored in r.headers but the "Vary", "Connection" and "Content-Type" headers are not:

``` shell
wget -S http://www.example.com/
--2011-09-08 08:56:59-- http://www.example.com/
Resolving www.example.com... XX
Connecting to www.example.com|XX|:80... connected.
HTTP request sent, awaiting response... 
HTTP/1.0 200 OK
Date: Thu, 08 Sep 2011 08:57:00 GMT
Server: Apache
: 
Vary: Accept-Encoding,User-Agent
Connection: close
Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]
Saving to: `index.html'
```

Request:

``` python
>> import requests
>>> r = requests.get("http://www.example.com/")
>>> r.headers
{'date': 'Thu, 08 Sep 2011 08:57:00 GMT', 'server': 'Apache'}
```


Looks likes this comes from urllib2

``` python
>>> import urllib2
>>> request = urllib2.Request('http://www.example.com/')
>>> request.get_method = lambda : 'HEAD'
>>> response = urllib2.urlopen(request)
>>> print response.info()
Date: Thu, 08 Sep 2011 13:38:22 GMT
Server: Apache
```


Actually, it comes from httplib.py L#317 where it breaks out of the header parsing. Unfortunately this status message isn't passed on from urllib2.py L#1177. Not sure if the behavior is specified in a RFC.


This is a bit out of scope — at least for now.

If someone wants to take the time to overload httplib's internal methods, I wouldn't mind adding the patch at all, but that will likely get complicated while supporting Python 2.5–3.3.


[RFC2616 section 4](http://tools.ietf.org/html/rfc2616#section-4) says field-name must be a token, so the header is malformed. 
The [section on tolerant applications](http://tools.ietf.org/html/rfc2616#section-19.3) only mentions line terminators, whitespace and date handling (the RFC was written in the pre-apocalyptic year of 1999)

The protocol gives the server all kinds of 4xx messages to point out epic client failz, but what is sane to do as a client?

"Be liberal in what you accept, and conservative in what you send"

But a malformed header could be a sign of more things mal. Maybe a "quirksmode" that silently tries to fix the servers intent (IE6 anyone?) and a strict-mode that raises warnings.


I'm currently porting the codebase to be powered by the wonderful `urllib3` project, which sits directly on top of httplib. This will make handing a case like this much easier.


I'm closing this ticket for being out of scope.

If the headers are being malformed, I think there are bigger issues at hand :)
Content-Type overwritten by urllib2 during POST

Got a bug, where urllib2 set a "Content-type" header, even if the "Content-Type" was set.

``` python
    r = requests.post('http://example.com', headers={'Content-Type':'application/json'})
```

vs

``` python
    r = requests.post('http://example.com', headers={'Content-type':'application/json'})
```

Content-Type failed, because urllib2 has written it's own header ("Content-Type: application/x-www-form-urlencoded"). I think it's a case (in)sensitive problem. Didn't have the time to look exactly into it, sorry.

The error occured with Python 2.7.2.


Thanks for the report. I know exactly what's causing this — should have a fix pushed up soon.


Awesome, thanks! :)


urllib2 is being removed, so this shouldn't be a problem in the next release.
Feature content streaming

Make it possible to not load the whole response into memory at once.  This is especially helpful if you download huge responses from the interwebs.


Yes, there are no tests now, but I don't know how you want to have this tested. Open for suggestions.


Beautiful — Thanks Armin.

---

Testing shouldn't need to be too thorough for this. For now, we can send a request to Httpbin's `/gzip` endoint and join the iterator to confirm that it's the same as a non-iterator version.
HTTPS client-side certificate

About certificates and validation: it would be really great to have client-side certificate authentication. Submit a PKCS#12 identity and its associated password (if any) and let requests use it for authentication to a remote HTTPS site. This is the kind of thing you need more and more as some public APIs require strong authentication. pyCurl has it but it is a real pain to use.


Thanks for the thoughts! 

Check out #30 for the current plans.
bytecode string returned when page has charset=UTF-8

I had a situation (with both requests and urllib2) where a page that had `<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />` was being returned as a bytestring `<type 'str'>` but did contain unicode characters (due to a server misconfiguration I assume). So when I tried to use it I got the classic `UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 419: ordinal not in range(128)`

Is this something that requests could fix? Is this something that requests would want to fix?


Requests only attempts to decode charsets specified in HTTP Headers (in the upcoming release).

However, there is a utility function that will attempt to decode based on the HTML tags. If the content isn't actually in the specified encoding, however, there's nothing that can be done (aside from ignoring the invalid charecters).
Move tests into submodule

- Moved tests into .test submodule. As you can see instead of `import requests` i used `import __init__ as requests` — this will work in the same way. Some relative import magic might have been used but AFAIK it won't work on python 2.5
- Changed `open('test_requests.py')` to `StringIO('Test.')` — this will always work and behave properly. Using plain file open might not work if user is testing requests from some different location (e.g. test module after install via pip.) Although it is possible to make usual file open work by some clever path calculations but i'm not sure if this is really needed.
- Added additional try/except during json import to ensure proper import.


Thanks for the contribution — I'm still a little hesitant about doing this though. 

I'll update this pull request once I give it some more thought.
readonly input fields

How does requests handle readonly (hidden) input fields like [nonce](http://en.wikipedia.org/wiki/Cryptographic_nonce) ?

In [mechanize](https://github.com/jjlee/mechanize) , the readonly values are automatically posted.
Are there any plans of implementing this , since this is a Huge Time Saver.


Can you give me more context?

Requests is used for crafting HTTP requests, not necessarily filling out HTML forms.
Fixed logic for auto redirect in GET and HEAD methods.

This patch allow you turn off allow_redirects for GET or HEAD methods. 


Please merge this into next release, this will make testing redirect code much easier


Can you explain in more detail?


The `allow_redirects` parameter is only for PUSH/POST/_&c_ requests, to allow the behavior described in [RFC 2616](http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html):

> If the 301 status code is received in response to a request other than GET or HEAD, the user agent MUST NOT automatically redirect the request unless it can be confirmed by the user, since this might change the conditions under which the request was issued.


I can see how the name of the argument could be confusing. I need to make an explicit note of it in the docs. 

Can you explain the use case for wanting to disable redirect-following for a GET/HEAD? I'm not against adding it (at all), if it's needed.


> > Can you explain the use case for wanting to disable redirect-following for a GET/HEAD? I'm not against adding it (at all), if it's needed.

For debug my http application. (i want to see full sequence of requests/responses)


You should use `Response.history`, which contains a list of the redirect response objects.


On second thought, I think I will add this in. I'll have `follow_redirects` default to True for GET/Heads.


> I'll have follow_redirects default to True for GET/Heads

This is exists in my patch :)

https://github.com/kennethreitz/requests/pull/142/files#diff-0


:thumbsup:


Good though, kennethreitz! Please, apply this pull request, it will make world better ;-) 


I merged it in yesterday :)
Session local argument merging


Beautiful. This will kill two birds with one stone.

My plan of late has been: if you pass in a dictionary update sequence, instead of a dictionary, it will be converted to a dictionary internally. 

I can also use this to power local argument overrides for sessions. When a dict is passed in locally, it'll be merged. When a sequence is, it will be overridden.

``` python
import requests

headers = {'X-Life': '42'}

with requests.session(headers=headers) as s:

    _headers_merged = {'X-Question': 'What is six times nine?'}

    r = s.get('http://..', headers=_headers_merged)
    # request has both X-Question and X-Life headers

    _headers_override = [('X-Question', 'We\'ll never know.')]

    r2 = s.get('http://..', headers=_headers_override)
   # request has only X-Question header
```

I can pretty much just check to see if the passed in item has `__getitem__`, and if does not, assume it's a dict sequence. If it doesn't work as one, just let the `dict()` error raise.


New proposal, as discussed with @robmadole:

All locally passed dictionaries will merge with the session-level dictionaries. If a value is set to None, that key will be removed before being passed to the request method.
HTTP Headers w/ Multiple Values


http://www.w3.org/Protocols/9707-link-header.html


https://gist.github.com/210535


Document this.


Does it mean something like this for example?

``` python
from unittest import TestCase
import requests


class TestHeaderLink(TestCase):

    def test_header(self):
        response = requests.get('http://response_header_link_url')
        self.assertEquals(response.headers['link'], {
            'url_link?query': {'rel': 'some', 'author': 'test'},
            'url_link1?query': {'rel': 'some', 'author': 'test'},
            'url_link2?query': {'rel': 'some', 'author': 'test'}})
```
Issue #97 - Turn a request into a CURL command string

For Issue #97. Once you create a request, just run r.request.curl for the curl command. For example: 

``` python
>>> import requests
>>> r = requests.get("http://httpbin.org");
>>> r.request.curl
'curl -L -X GET -H "Accept-Encoding:gzip" -H "User-Agent:python-requests.org" "http://httpbin.org"'
```

Still have the following to do:
- OAuth


This is so awesome! I'm going to toy with the API a bit, if you don't mind.

I think the way to go is have a `curl_from_request` function in the `requests.utils` module.


Sounds good. 

Yeah, requests.utils sounds like a better idea. I'll dig through that code and maybe send an update. 


Just made a few modifications. You can pull from my `feature/curl` branch to get them.

Thanks again for this! I'm really excited to see this going in.


So, new syntax is:

``` python
>>> import requests
>>> from requests.utils import curl_from_request

>>> r = requests.get('http://httpbin.org/get')
>>> curl_from_request(r.request)
'curl -L -X GET -H "Accept-Encoding:gzip" -H "User-Agent:python-requests.org" "http://httpbin.org/get"'

```


Looks good!


Sorry about the double commits, created another branch by mistake. Still learning my way around github. 

I merged in your change from feature/curl and everything should be in mawaldne:curl_feature

Still a few things left to do. 


No worries ;)


Apart from Oauth, this is finished. I know there's some work going on with Oauth in #65. Once that's complete I'll add update the curl util method.  


Excellent! Thanks for the great work :) 

This will make an excellent addition to the next release.
Progressive downloading

Is there any hooks in requests to enable progressive loading of data? Please don't tell me I have to go back to urllib2 to create a simple progressbar. :(


There are no hooks, but the content is not downloaded until you access the `Response.content` attribute.

Instead of using the `content` attribute, you can read bytes from the socket directly with `Response.read`. You can easily set up a loop to pull down 1/100th of your content at a time while updating your progress bar.

---

**Note:** In the next release of Requests, this will be available as `Response.fo.read()`.


Is there an example in the documentation on how to do that? Also note, that the headers need to be available so that you can check the total size of the download. And preferably, you would have hooks that trigger after dns lookups and after connection established. I guess that may be hard to accomplish using urllib2 though.


No documentation on that, but I'll add it in soon.

The request is made as soon as you call the request functions, so the headers are all there and everything, the content just isn't downloaded yet until you access `r.content`, which is usually immediately after.


@bjourne: Check out #90 for an example. Your use case is a bit different, but this is the general idea.
Added multiple hooks support.

As per feature request #134


I apologize for not getting back to you about this sooner. 

To be honest, I'm a bit torn about including a library of hooks with requests. I'm thinking about it. 


Hi Kenneth,
Since the multiple hooks feature is more interesting than a library of hooks, I just reverted from sub-package to module. Hooks library was just an idea.. and honestly, I couldn't think to something more than `post_request` HTTP status code handler or `response` content parsing.


I'm considering utilizing some hooks internally now, actually. It will likely be a more elegant solution going forward as more decodings are added.


Let's do this.
Add test_requests.py to manifest

People that download the tarball should also be fortunate enough to test that Requests works as intended :-)


Thanks!


IMO it might be good idea to move test_requests.py into requests.test submodule, what do you think?


@densh, I've considered this before, but I like being able to 'import requests' directly from `test_requests.py` without having have requests installed into a site-packages w/ `setup.py develop`.

If there's an elegant way to accomplish both at the same time, I'm game.


@kennethreitz: You shouldn't really assume PYTHONPATH=`pwd` anyway (since different flavors of python won't like running the same bytecode) - so if you want to create the subdir, either inject path to `ls -d build/lib-*/` depending on python version or more commonly let the user decide by setting PYTHONPATH. I guess one could say that the latter is already in play (disregarding moving tests) :-)


I noticed e8dbecb4dbed3e835021977cf43f225a76bc5ec3 adding this to MANIFEST.in, but AFAICT test_requests.py still isn't included in the 0.10.6 or 0.10.8 tarballs, so I think this might need to be reopened?


Sorry, I thought this was an issue... should I open an issue to get this revisited?
More content encodings.

Hi Kenneth,
I don't see a reason why not to support more content encodings.

I' would contribute to this by writing a module with a set of decoding functions, a way to get the list of supported encodings, and a main decoder which parse the response header and apply the decoding.


I mean http://en.wikipedia.org/wiki/HTTP_compression


Absolutely. I'd love to add both deflate and sdch soon.


Added from urllib3.
More hooks, in pipe

Hi Kenneth,
I'd like to see this feature: allow developers to specify more hooks per category, and let them be applied in pipe.
I'll be very glad to produce a patch.


I would love that as well. What did you have in mind?


Sorry, I messed up with issues and pull requests :P See the issue #137
But.. I played a little more with the package and I've come up with more changes.
You can take a look at https://github.com/monkeython/requests/tree/develop/requests
Basically:
- `hooks` is a package.
- `hooks.response` is a module that now contains 2 hooks: one for unicode and another to decode the `response`.
- `hooks.setup_hooks` is a function that builds a dictionary of sets of hooks (easily handle duplicates). Based on `config.settings` it may add the unicode and uncompress response hooks.
- `hooks.dispatch_hooks` now take just an iterable of hooks and an argument, and apply each hook on the argument.

No unit testing yet, but there will be soon. 


Closing.
Optional base url attribute on Session object

Is it possible to add an optional base url attribute to the Session object ? 

For example;

```
 auth = ('user', 'pass')
 with requests.session(url='http://api.twitter.com', auth=auth) as s:
     resp = s.get('/1/statuses/home_timeline.json')
```


Or I can just use a hook;

```
auth = ('user', 'pass')
def twitter(args):
    args['url'] = 'http://api.twitter.com%s' %args['url']

with requests.session(auth=auth, hooks=dict(args=twitter)) as s:
    resp = s.get('/1/statuses/home_timeline.json')
```


I've considered this myself, but opted against it. I think this is a good use-case for the hooks.


I don't think this can be done with hooks anymore.

I think @tback , it's not possible to do with hooks anymore. The only hook available according to the docs is the `response` hook. Any update on this issue?

https://toolbelt.readthedocs.io/en/latest/sessions.html#baseurlsession
Refactor Response.content into a property

Originally my main goal was to just make `Response.content` attribute visible in `dir(response)`. After writing a custom `__dir__` it struck me that a simple refactoring of the content attribute into a property would make a much more cleaner solution.


Much cleaner, thanks!


Sphinx autodoc understands property docstrings fine, no need to set it to `None`.

I'm using `Request._content` to cache the value of `Request.content`, for now.
Added readline method to response object.

Reading single lines is really useful when dealing with APIs , for example consuming streaming data from twitter. 


Thanks!

---

Hmm, I'm hesitant to add this. Adding `r.read()` has already caused some confusion. I'm leaning more towards having a `r.fo` w/ these methods.


Thanks, yes. I think the Response abstraction is excellent for most uses. Sometimes, you need to access the file handle methods. `r.fo` sounds like the best way of exposing these methods. Thanks for the great work with requests.
Issue 128

Hello! I've just fixed issue #128: 
https://github.com/kennethreitz/requests/issues/128

Actual problem was quite subtle:
`Response.__getattr__` didn't trow AttributeError
on attributes other then `content` so you would
get `None` for any non-existent attribute.

lxml tried to perform `.getvalue()` call and it resulted into:
- `response.getvalue -> None`
- `response.getvalue() -> TypeError: ...`


Nice catch, thanks!
Response not working with lxml

This looks like it should work but doesn't, but it does with the urllib2 line that's commented out. I'm using requests 0.6 and lxml 2.3

```
import urllib2
import requests
import lxml.etree as etree
#page = urllib2.urlopen(url)
r = requests.get('http://www.google.com')
parser = etree.HTMLParser()
tree = etree.parse(r, parser) # first parameter can be a file like object http://lxml.de/tutorial.html#parsing-from-strings-and-files
```

I get `TypeError: 'NoneType' object is not callable`


Sorry about the bug. This will be fixed in the next release. In the meantime, i recommend treating parsing from the `r.content` string.
gzip decompression not working

I tested `requests` on [This Link](http://encodable.com/uploaddemo/)
and the `requests.read()` function doesn't seem to work. It returns strange text because the response in Uncompressed. I noticed that requests uses `zlib` for Decompression.

Although using `r.content` worked perfectly.


`response.read()` is for grabbing raw chunks of bytes from a response's content, just like a file.

You should always be using `response.content`, unless you have a special reason not to. 

Sorry for the confusion, it'll be in the docs soon.
Error fetching non-existant pages using authentication

Trying to fetch a non-existant page when authenticated behaves strangely. Steps to reproduce:
- Configure your web server to require HTTP basic authentication
- Execute code such as the following:

``` python
import requests
resp = requests.get("http://my.web.server/some_non_existant_path", auth=("username", "password"))
print resp.status
print resp.content
```
- Check output of your web server logs and output of the code snippet

Expected output:
- The web server receives two requests: the first returns status 401 and the second returns status 404.
- The code snippet prints 404, then the contents of your 'Not Found' page

Actual output:
- The web server receives seven requests in immediate succession: the first returns status 401 and all the others return status 404.
- The code snippet prints 401, then traces back with the following:
  
  Traceback (most recent call last):
    File "my_example_script.py", line 2, in **main**
      print resp.content
    File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 422, in **getattr**
      self._content = self.read()
  TypeError: 'NoneType' object is not callable

Using requests 0.6.0 and Python 2.7


Urllib2 gets stuck in an infinite redirect loop when invalid credentials are passed. Requests stops this at 7 requests.

I'm not sure about the TypeError.

---

Is there a public URL you can reproduce this with?


This shows the problem using httpbin.

``` python
>>> response = requests.get("http://httpbin.org/status/404", auth=("user", "pass"))
>>> response.status_code
401
>>> response.content
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests-0.6.0-py2.7.egg/requests/models.py", line 422, in __getattr__
    self._content = self.read()
TypeError: 'NoneType' object is not callable
```


Very odd. 

Thanks, I'll look into it.


Update: I'm still working on this. It's actually a very complicated issue.

My (tentative) plan going forward is to implement the ForcedBasicAuth handler by hand, instead of using a hacked up urllib2 auth handler. 


This will be fixed in the next release. 


I can't tell by your last response if you have fixed the issue upstream or not, so I thought you might be interested in the following:

```
In [3]: response = requests.get("http://httpbin.org/status/404", auth=("user", "pass"))
{}
FUCK
Max retries exceeded for url: http://httpbin.org/status/404
```

It looks as if the issue is still not fixed (like I said, I don't know if it is supposed to be fixed yet or not)


Works fine for me.

``` python
>>> import requests
>>> response = requests.get("http://httpbin.org/status/404", auth=("user", "pass"))
>>> response.status_code
404
>>> response.content
u''
```


@kazbak, this will not be an issue in the next release.


@kazbak for further context, the entire core is being rewritten on top of `httplib`. This will make issues like this one go away. All of the weird bugs (like this one) that were caused by `urllib2` should be gone once it is ready :)
Fix a potential memory leak, raise a TooManyRedirects exception with more than 30 redirections, and fix unicode urls.

This pull request contains 3 fixes:
- Fix a potential memory leak. See http://bugs.python.org/issue1208304 for more details.
- Raise a TooManyRedirects exception with more than 30 redirects in order to prevent infinite redirections.
- Correctly handle any kind of unicode url. Works for urls with the path not encoded (ex: http://fr.news.yahoo.com/mariage-gay-débat-relancé-à-lassemblée-065517796.html) and so on...

```
>>> r = requests.get(u'http://fr.news.yahoo.com/mariage-gay-débat-relancé-à-lassemblée-065517796.html')
>>> r.status_code
200
>>> r = requests.get('http://fr.news.yahoo.com/mariage-gay-débat-relancé-à-lassemblée-065517796.html')
>>> r.status_code
200
```


Added support for redirection without scheme (Location starting with //, for example http://goo.gl/UYpIU).


Thanks for the excellent contributions! 

I've made the max_redirects configurable.
Not compatible with python 2.5

```
  File "/home/xxx/env/tran++/lib/python2.5/site-packages/requests/async.py", line 41
    from .core import *
SyntaxError: 'import *' not allowed with 'from .'
```

The kind of import isn't support in python 2.5


Seems like a _feature_ to me :)


Because it is :)


Very funny ;)

Sorry for the bug. `requests.async` is relatively useless as-is, and is fully undocumented. A major rework of the module will be developed soon :)
Feature/session

One feature I consistently use, especially for writing quick one-off scripts, is automatic cookie handling.  I've implemented this via a Session object whose instance properties can be set to be automatically passed whenever an http method call is performed.  A CookieJar is instantiated by default with each Session instance.  If you're interested in merging this feature but feel the docs/tests I added are lacking let me know and I can go over them more thoroughly.


Thanks a ton for the (excellent) contribution!

This is _very_ interesting. I'm going to have to give it some thought. This will affect the API quite a bit moving forward.

My initial reaction is to make it a context manager:

``` python
with requests.session as s:
    s.get(...

```


So, my pattern with wrapping requests so far is a pre and post callback that modifies the args going in, and the response going out.

I've been tossing the idea around of just supporting optional callbacks right into the request functions.

These two ideas might play nicely together. 


This is merged into develop now. Going to try to release this tonight.

Usage:

``` python
with requests.session(auth=('user', 'pass')) as c:
    c.get('http://httpbin.org/basic-auth/usr/pass')
```
Get request headers...

There is no easy way to get the used **request headers** from urllib2, see: http://stackoverflow.com/questions/603856/how-do-you-get-default-headers-in-a-urllib2-request

I made https://github.com/jedie/django-tools/blob/master/django_tools/utils/http.py for this. e.g.:

``` python
from django_tools.utils.http import HttpRequest
r = HttpRequest("http://www.python.org/index.html")
response = r.get_response()
```

You can get the raw sended request headers:

```
print response.request_header
```

Output (striped \r\n):

```
GET /index.html HTTP/1.1
Accept-Encoding: identity
Host: www.python.org
Connection: close
User-Agent: Python-urllib/2.7
```

Get the request headers as a list:

```
print response.request_headers
```

Output:

```
[('Accept-Encoding', 'identity'), ('Host', 'www.python.org'), ('Connection', 'close'), ('User-Agent', 'Python-urllib/2.7')]
```

Is there any interests to merge this? If so, I would try to contribute it ;)


Interesting.

So, I added in `Response.request`, which is the Requests's Request object that was used to create the Response.

So, you can:

``` pycon

>>> r = requests.get('http://httpbin.org/get')
>>> r.request.headers
{'Accept-Encoding': 'gzip', 'User-Agent': 'python-requests.org'}
```

This doesn't include all of the urllib2-supplied headers. I'm thinking, if those are needed, it would be best to update this dictionary after a request is sent. 

Thoughts?


updating this dict is ok, but IMHO it's nice to get the raw sended headers as a string, too. e.g.: the dictionary is not sorted.


@jedie: I can make it an OrderedDict, so the order won't be lost. 

Making the raw request headers available is interesting. Same with the response. 

**Random ideas:** 

Raw property on the headers would be useful: `r.headers.raw` and `r.request.headers.raw` perhaps.


OK, but IMHO where the info are stored is not really important ;)

The Important questions is, how can we get theses information. In my solution i capture the header in `httplib.HTTPConnection.putheader()` and `httplib.HTTPConnection.send()`

And i don't know if these exist a easier way access theses captured headers.


Where the info is stored and accessed is _extremely_ important. That's what Requests is all about :)

I'll poke around and see what works consistently. Shouldn't be too difficult.


I've decided to leave this out for now. I'll be reconsidering it soon.
Use socket.setdefaulttimeout(timeout) only if timeout argument doesn't ex

Use socket.setdefaulttimeout(timeout) only if timeout argument doesn't exist (Python <2.6) - https://github.com/kennethreitz/requests/issues/112


This is not thread-safe. It may be OK but this should be at least mentioned somewhere. 

The thread-safe alternative is to reimplement request sending using raw sockets (see https://bitbucket.org/kmike/vkontakte/src/e89d4cb94902/vkontakte/http.py for an example) but this can be an overkill.


Excellent. Thanks a ton!
socket.setdefaulttimeout(timeout)

I read your source a little bit and found socket.setdefaulttimeout(timeout) in models.Request().

IMHO it's not a good idea to do this, because this is not thread safe. See also: http://kurtmckee.livejournal.com/32616.html

There exist timeout on several methods since Python v2.6

In my code i try to use the "new" timeout arguments and fall-back to socket.setdefaulttimeout() if it's intended:
https://github.com/jedie/django-tools/blob/master/django_tools/utils/http.py


Have made changes here: https://github.com/kennethreitz/requests/pull/113
content-encoding zlib.decompression issue

requests.get does not decompress data.

Try:
requests.get('http://crunchbase.com').read() 
vs
urllib2.urlopen('http://crunchbase.com').read()


Oops. Someone posted this before. Problem solved. User r.content instead


I assure you that it does :)

Compression occurs when accessing via `r.content` instead of `r.read()`.

Only gzip compression is supported at this time.
Content-Encoding issue **edit: issue due to ambiguous documentation**

I'm not sure what's going on here, but I'm assuming it's an issue regarding Content-Encoding. Have a look.

With Requests:

```
>>> import requests
>>> r = requests.get('http://www.elacarte.com/challenge/city_grid.txt')

>>> r.headers['content-encoding']
'gzip'

>>>print r.read()[:100]
��[��J����Q����?�s�"��S���e���$貥��?������AS��c�ܤ������D���;�    t+��~��&�N��$��

>>> import pprint
>>> pprint.pprint(r.headers)
{'accept-ranges': 'bytes',
 'connection': 'close',
 'content-encoding': 'gzip',
 'content-type': 'text/plain',
 'date': 'Sun, 14 Aug 2011 09:34:29 GMT',
 'etag': '"8a24017-1d1672-4aa6f3fd72a00"',
 'last-modified': 'Sun, 14 Aug 2011 04:04:56 GMT',
 'server': 'Apache/2.2.14 (Ubuntu)',
 'transfer-encoding': 'chunked',
 'vary': 'Accept-Encoding'}
```

With `urllib2`:

```
>>> import urllib2
>>> u = urllib2.urlopen('http://www.elacarte.com/challenge/city_grid.txt')

>>> print u.read()[:100]
>>> O   O   P   P   P   P   P   P   P   P   P   P   P   P   P   P   P   O   P   P   P   P   P   O   O
```


It turns out it's a feature—to get the response, I should use `r.content`, not `r.read()`. **That isn't in the documentation** (or at least, not in the front-page example or in the API documentation for Response)—perhaps it should be changed to include illustrating how to read the actual response, as that's the most common use-case?


Absolutely. I've received quite a few questions about this. :)


I encountered the same "problem". Please disregard my email :))


All better :)