Raibek/requests-github-issues huggingface dataset:
Content-type problem depending on python version 

Hi, I've changed python version from 2.6 to 2.7 and I've found this weird error at sending headers.

If I put 'Content-Type' then no content-type is sent :

```
>>> requests.post('http://httpbin.org/post', '',{'Content-Type' : 'application/json'}).read()
'{"origin": "::ffff:184.73.223.212", "files": {}, "form": {}, "url": "http://httpbin.org/post", "args": {}, "headers": {"Content-Length": "0", "Accept-Encoding": "gzip", "X-Forwarded-For": "::ffff:184.73.223.212", "Connection": "close", "User-Agent": "python-requests.org", "Host": "httpbin.org", "X-Forwarded-Protocol": "", "Content-Type": "application/x-www-form-urlencoded"}, "data": ""}'
```

But if I put 'Content-type' then it works perfectly:

```
>>> requests.post('http://httpbin.org/post', '',{'Content-type' : 'application/json'}).read()
'{"origin": "::ffff:184.73.223.212", "files": {}, "form": {}, "url": "http://httpbin.org/post", "args": {}, "headers": {"Content-Length": "0", "Accept-Encoding": "gzip", "X-Forwarded-For": "::ffff:184.73.223.212", "Connection": "close", "User-Agent": "python-requests.org", "Host": "httpbin.org", "X-Forwarded-Protocol": "", "Content-Type": "application/json"}, "data": ""}'
```

In python 2.6\* both examples work well.

For more information, the same example but using urllib2 is working ok.

```
req = urllib2.Request('http://httpbin.org/post', '',{'Content-Type' : 'application/json'})
>>> urllib2.urlopen(req).read()
'{"origin": "::ffff:184.73.223.212", "files": {}, "form": {}, "url": "http://httpbin.org/post", "args": {}, "headers": {"Content-Length": "0", "Accept-Encoding": "identity", "X-Forwarded-For": "::ffff:184.73.223.212", "Connection": "close", "User-Agent": "Python-urllib/2.7", "Host": "httpbin.org", "X-Forwarded-Protocol": "", "Content-Type": "application/json"}, "data": ""}'
```


This workaround works, but I am having the same problem on 2.7
Revisit Async Module

Useless as is.


### Possible API:

``` pycon
>>> from requests import async
>>> request_list = [async.get('http://github.com'), async.get('http://kennethreitz.com')]

>>> async.map(request_list, size=5)
[<Response [200]>, <Response [200]>]
```

**Changes Required:**
- `Response.request` for context.


+1


Unless I'm missing something, this wouldn't allow adding new requests while other requests are in progress.  I think that's important ... but I may well be missing something!

Some documentation of async might the most important bit :)


rboulton, good point. There will have to be a callback-style interface too.


fwiw, I would vote for callbacks over a twisted style yield based api that seems to be en vogue these days (just in case a yield style async api was in the running)


Planned: gevent


3092483433ebf0f5802677aa9b8a74ee4dc1e64b


``` python
# -*- coding: utf-8 -*-

import requests
from requests import async

urls = [
    'http://www.readability.com',
    'http://tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

urls = urls + urls

```

**Standard (14.125s)**

``` python
for url in urls:
     r = requests.get(url)
     r.content
```

**Standard + Keep-Alive (6.390s)**

``` python
with requests.session() as s:
    for url in urls:
         r = requests.get(url)
         r.content
```

**Standard + Gevent (1.589s)**

``` pycon
>>> rs = [async.get(u) for u in urls]
>>> async.map(rs)
[<Response [200]>, <Response [200]>, ...]
```

**Gevent + Keep-Alive (1.132s)**

``` pycon
>>> rs = [async.get(u) for u in urls]
>>> async.map(rs, keep_alive=True)
[<Response [200]>, <Response [200]>, ...]
```


`async.map(map(async.get, urls))`?  :)


Pure awesomesauce !!!!


But quite unstable of course. Looking forward to release ;-)


I :heart: gevent.

Looking forward to this!


#201
Fixes correct "relative" redirections.

Fixes correct "relative" redirections. Example: http://digg.com/d1qIwX, which redirects to "/news/story/A_Manifesto_How_to_Save_Media", which requests transforms to: http://digg.com//news/story/A_Manifesto_How_to_Save_Media" which results in an endless redirect loop.


Just ran into this bug today, thanks for the fix @jeremys


Thanks a ton for the contribution! Sorry the delay on merging this
requests.models.Request has allow_redirects and redirects

The api uses allow_redirects to control redirect following but the in the Request:185 it uses _redirects_ 

requests.post is following a 302 redirect even though allow_redirects is explicitly set to False. 

I think Request:185 should be; 

```
        if r.status_code in REDIRECT_STATI and self.allow_redirects:
```


`allow_redirects` is used to allow 301-like behavior for non-GET-like requests (like a POST). 

`_redirects` is used internally to flag if a request is part of a redirect chain.


But the following code follow redirects for 302. 

```
resp = requests.post(uri, data=urlencode(datadict), allow_redirects=False, headers=headersdict)
```

Server first responds with;

```
Status Code:302 Moved Temporarily

Cache-Control:no-store, no-cache, must-revalidate
post-check=0, pre-check=0
Content-Length:0
Date:Mon, 01 Aug 2011 04:04:21 GMT
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Location:http://server/downloadTable?fileTemplate=templates%2FTemplate.csv
Pragma:no-cache
Server:Apache-Coyote/1.1
```

then a GET request is sent to the Location


Yes, that is correct behavior. The `allow_redirects` parameter is only an opt-in to allow redirects on POSTs. Not disallow regular redirects.


Well the name is ambiguous `allow_redirects` suggests that it will disallow any redirects. What are _regular redirects_ ?
PATCH error

Hi!

I'm trying to make a request using PATCH, but it show this error (the same when I tried to make this request using urllib2 only):

'PATCH' is an unrecognized HTTP method


What version of Python?


2.6.1, using Mac OS X 10.6.8

One more detail, maybe this is the problem: I'm using Google AppEngine, but just the launcher on my machine by now.


That shoudn't be happening :)
.history doesn't work

If I do:

```
>>> import requests
>>> r = requests.get('http://bit.ly/kCCnc0')
>>> r.history
[<Response [301]>]
```

I just get the first request in the history, instead, I should get:

```
>>> import requests
>>> r = requests.get('http://bit.ly/kCCnc0')
>>> r.history
[<Response [301]>, <Response [301]>, <Response [302]>]
```


Yikes, I added some changes to the redirection handling last release.

I added a customizable redirection testing system to httpbin to test this (http://httpbin.org/redirect/6) , but haven't written the unit tests yet. 

Thanks.


history works but error is defined to the redirect error. It was not the case before 0.5.1.

```
>>> import requests
>>> r = requests.get('http://httpbin.org/redirect/6')
>>> r.history
[<Response [302]>, <Response [302]>, <Response [302]>, <Response [302]>, <Response [302]>, <Response [302]>]
>>> r.error
HTTPError()
>>> str(r.error)
'HTTP Error 302: FOUND'
>>> 
```


@jerem: This is fixed now!
keep-alive

Does requests have support for keep-alive type requests? I am using an API that streams data through an open POST connection. If the connection is still active, the request function does not return. Maybe it could be done by adding file handling to  the Response.content property.


It does not, currently. Pull requests welcome though :)


I'm using Twitter streaming api with requests, perhaps this code is useful for you

``` python
import requests

def on_receive(data):
    print data

username = 'XXXX'
password = 'XXXX'
url = 'http://stream.twitter.com/1/statuses/filter.json'
# Sadly 'bieber' keyword is a good test :S
response = requests.post(url, data={'track' : 'bieber'}, headers={'Connection' : 'Keep-Alive'}, auth=(username, password))
while True:
    on_receive(response.read(100))
```


Thanks @moliware, this is what I did.


@moliware: Excellent! I'm going to add this to the upcoming cookbook :)


nice :)
We can represent multivalued params with arrays

If we need to use a multivalued param like in http://localhost/service?q=*&facet.field=fieldA&facet.field=fieldB, now we can use arrays in the params dictionary:

``` python
r = requests.get('http://localhost/service',
                 params={'q':'*', 'facet.field':['fieldA','fieldB']})
```


Very interesting. Sorry I haven't merged this yes. I've been giving it a lot of thought.

I'm wondering if a MultiDict might be more appropriate? 


It should be easy to use either MultiDict(http://www.dalkescientific.com/Python/MultiDict.py) or a custom class to replace params dictionary.

Let me have a look.


On second thought, I'm going to merge this in. This is a good solution :)

Thanks! :cake:


Ok, it was an easier approach ;)
basic auth not working with github api (v3)

when i try to get info of authenticated user it fails with 404:

```
>> import requests
>> r = requests.get(' https://api.github.com/user', auth=('garbas', 'PASS')
>> r.ok
False
>> r.headers
{'status': '404 Not Found', 'x-ratelimit-remaining': '4997', 'content-length': '23', 'server': 'nginx/1.0.4', 'connection': 'close', 'x-ratelimit-limit': '5000', 'date': 'Thu, 30 Jun 2011 22:51:40 GMT', 'content-type': 'application/json'}
```

if i do it with curl it works:

```
% curl -u "garbas:PASS' https://api.github.com/user
{
  "public_gists": 29,
  "type": "User",
   ...
} 
```

am i doing something wrong?


You aren't doing anything wrong. The way Basic Auth works is the client requests the url w/o auth, the server challenges it (`403`), and the client sends the auth. GitHub is sending a not found `404` for security reasons, which isn't standard protocol.

So, requests isn't sending the auth at all, since it isn't being challenged, which _is_ proper behavior. I'm working on an elegant api to force-send the auth, which will cover this use case. Hopefully I'll have something great within a few days.

In the meantime, you can build the Auth header yourself:

``` python
import requests
from base64 import encodestring as base64

username = 'kennethreitz'
password = 'xxxxxxxx'

repo_url = 'https://api.github.com/repos/kennethreitz/reflog'   # private repo

encoded_auth = base64('%s:%s' % (username, password)).replace('\n', '')
headers = {'Authorization': 'Basic %s' % encoded_auth}
```

Yay!

``` pycon
>>> requests.get(repo_url, headers=headers)
<Response [200]>
```


Alright, a new ForcedHTTPBasicAuthenticationHandler was added, and it's default.

This should work fine now::

requests.get('https://api.github.com', auth=('user', 'pass'))
:)
Be able to get the response content as a file like object

In some particular cases, it is necessary to open a request without automatically downloading the body.

Example:

With the actual behaviour:

```
>>> import requests
>>> r = requests.get('http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2')
-> Here it blocks until the body it downloaded (72.6 MB in this case!).
>>>
```

But with a file like object, here what we can do:

```
>>> import requests
>>> r = requests.get('http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.2.tar.bz2')
>>> r.status_code
200
>>> r.read(1)
'B'
```

That means we can access the headers without waiting for the body...

Of course, we can keep the compatibility with response.content by adding a **getattr** method to the response that read and return the body when the content is requested.


Interesting thoughts. 

Can't you just use a `HEAD` request though?


Eventually, but it's not the best way to do it.

For example, if you want to crawl web pages, you do a `GET`, check if the content-type == 'text/html', if yes you read and parse the body, otherwise you close the connection.

With a `HEAD` you have to do 2 requests, `HEAD` + `GET`. Plus the fact that some servers block or don't respond correctly to `HEAD`
add doseq to urlencode in model._encode_params to enable list as dict value

parse_qs and then urlencode, things will changed.

[test code and result](http://min.us/ldLJUS)

Thank you for this great lib,

the result of parse_qs is something like {'oauth': ['sig'], 'status': ['hanzi ok']}
urlencode can't handle list as dictionary value without doseq
doseq does three things,
1. if value is not list or unicode, behave like doseq=False
1. if value is unicode, decode unicode
2. if value is list, handle list as dict value

it will fall to decode if the unicode value is not encoded in ascii, Chinese for example.
but _encode_params already made sure that no unicode will pass to urlencode,
so I think it's safe to add doseq.

snippet from Python2.7/Lib/urllib.py

``` python
   if not doseq:
        # preserve old behavior
        for k, v in query:
            k = quote_plus(str(k))
            v = quote_plus(str(v))
            l.append(k + '=' + v)
    else:
        for k, v in query:
            k = quote_plus(str(k))
            if isinstance(v, str):
                v = quote_plus(v)
                l.append(k + '=' + v)
            elif _is_unicode(v):
                # is there a reasonable way to convert to ASCII?
                # encode generates a string, but "replace" or "ignore"
                # lose information and "strict" can raise UnicodeError
                v = quote_plus(v.encode("ASCII","replace"))
                l.append(k + '=' + v)
            else:
                try:
                    # is this a sufficient test for sequence-ness?
                    len(v)
                except TypeError:
                    # not a sequence
                    v = quote_plus(str(v))
                    l.append(k + '=' + v)
                else:
                    # loop over the sequence
                    for elt in v:
                        l.append(k + '=' + quote_plus(str(elt)))
```

it's the first time I use github, I'm sorry for the duplicated pull request
reallly appreciate for this great lib, it helps me a lot.
and I'm looking forward to oauth feature so I don't need to roll my own:)

``` diff
@@ -230,7 +230,7 @@ class Request(object):
             for (k, v) in data.items():
                 result[k.encode('utf-8') if isinstance(k, unicode) else k] \
                      = v.encode('utf-8') if isinstance(v, unicode) else v  
-            return result, urllib.urlencode(result)
+            return result, urllib.urlencode(result, doseq=True)
         else:
             return data, data
```


Merged! Thank you :)
post file to flask with requests-lib result is different than from a sever

``` python

from flask import Flask, request

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def upload_file():
    if request.method == "POST":
        print request.form
        print request.files
        return 'fin'
    else:
        return '''
        <!doctype html>
        <title>Upload new File</title>
        <h1>Upload new File</h1>
        <form action="" method=post enctype=multipart/form-data>
          <p><input type=file name=file>
             <input type=submit value=Upload>
        </form>
        '''

if __name__ == '__main__':
    app.run(host="localhost", port=7998, debug=True)
```

``` python
import requests
import unittest

class TE(unittest.TestCase):
    def test_url3(self):
        response = requests.post('http://localhost:7998/', data={}, files={"file":"/home/px/wbbase/douban.png"}).read()
        print response

if __name__ == "__main__":
    unittest.main()
```

this request is send from unittest

``` bash
--> p2 up.py
 * Running on http://localhost:7998/
 * Restarting with reloader...
ImmutableMultiDict([('file', u'/home/px/wbbase/douban.png')])
ImmutableMultiDict([])
127.0.0.1 - - [26/Jun/2011 15:34:10] "POST / HTTP/1.1" 200 -
```

this request is send from chromium

``` bash
--> p2 up.py
 * Running on http://localhost:7998/
 * Restarting with reloader...
127.0.0.1 - - [26/Jun/2011 15:35:00] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [26/Jun/2011 15:35:01] "GET /favicon.ico HTTP/1.1" 404 -
ImmutableMultiDict([])
ImmutableMultiDict([('file', <FileStorage: a.torrent' ('application/x-bittorrent')>)])
127.0.0.1 - - [26/Jun/2011 15:35:07] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [26/Jun/2011 15:35:07] "GET /favicon.ico HTTP/1.1" 404 -
```


You have to add the file object to the dictionary, not just the path to the file.
[feature request] support multiple values with same name in get/post data

``` python
# *-* coding: utf-8 *-*

from flask import Flask
from flask import request
import requests

app = Flask(__name__)

@app.route('/', methods=["POST", "GET"])
def index():
    if request.method == "GET":
        return str(len( request.args.getlist('ck') ))
    else:
        return str(len( request.form.getlist('ck') ))

if __name__ == '__main__':
    app.run(host="localhost", port=7999, debug=True)
```

``` python
import requests
import unittest

class TE(unittest.TestCase):
    def test_url1(self):
        self.assertEqual('1', requests.get('http://localhost:7999?ck=1').read())

    def test_url3(self):
        self.assertEqual('3', requests.get('http://localhost:7999?ck=1&ck=2&ck=3').read())

    def test_get1(self):
        self.assertEqual('1', requests.get('http://localhost:7999', data={
            "ck":"1"
        }).read())

    def test_get3(self):
        self.assertEqual('3', requests.get('http://localhost:7999', data={
            "ck":"1",
            "ck":"2",
            "ck":"3",
        }).read())

    def test_post1(self):
        self.assertEqual('1', requests.post('http://localhost:7999', data={
            "ck":"1"
        }).read())

    def test_post3(self):
        self.assertEqual('3', requests.post('http://localhost:7999', data={
            "ck":"1",
            "ck":"2",
            "ck":"3",
        }).read())

if __name__ == "__main__":
    unittest.main()
```

``` bash
.-(~/multipart)----------------------------------------------------(px@Rainbow)-
`--> nosetests -s
.F.F..
======================================================================
FAIL: test_get3 (test.TE)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/px/multipart/test.py", line 20, in test_get3
    "ck":"3",
AssertionError: '3' != '1'

======================================================================
FAIL: test_post3 (test.TE)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/px/multipart/test.py", line 32, in test_post3
    "ck":"3",
AssertionError: '3' != '1'

----------------------------------------------------------------------
Ran 6 tests in 0.427s

FAILED (failures=2)
```

It's  related to my previous issues post..
And the origin of all my questions is 

``` python
response = requests.post(url, data=oauth_request.to_postdata(), files=files).read()
```

oauth_request.to_postdata() will return a string of encoded url, 
without files, it works great,
but if files present, requests.post will try to update a dict with a string

``` python
if self.data:
    self.files.update(self.data)
```
add doseq to urlencode in model._encode_params to enable list as dict value

parse_qs and then urlencode, things will changed.

[test code and result](http://min.us/ldLJUS)

Thank you for this great lib,

the result of parse_qs is something like {'oauth': ['sig'], 'status': ['hanzi ok']}
urlencode can't handle list as dictionary value without doseq
doseq does three things,
1. if value is not list or unicode, behave like doseq=False
1. if value is unicode, decode unicode
2. if value is list, handle list as dict value

it will fall to decode if the unicode value is not encoded in ascii, Chinese for example.
but _encode_params already made sure that no unicode will pass to urlencode,
so I think it's safe to add doseq.

snippet from Python2.7/Lib/urllib.py

``` python
   if not doseq:
        # preserve old behavior
        for k, v in query:
            k = quote_plus(str(k))
            v = quote_plus(str(v))
            l.append(k + '=' + v)
    else:
        for k, v in query:
            k = quote_plus(str(k))
            if isinstance(v, str):
                v = quote_plus(v)
                l.append(k + '=' + v)
            elif _is_unicode(v):
                # is there a reasonable way to convert to ASCII?
                # encode generates a string, but "replace" or "ignore"
                # lose information and "strict" can raise UnicodeError
                v = quote_plus(v.encode("ASCII","replace"))
                l.append(k + '=' + v)
            else:
                try:
                    # is this a sufficient test for sequence-ness?
                    len(v)
                except TypeError:
                    # not a sequence
                    v = quote_plus(str(v))
                    l.append(k + '=' + v)
                else:
                    # loop over the sequence
                    for elt in v:
                        l.append(k + '=' + quote_plus(str(elt)))
```

it's the first time I use github pull request, I'm sorry for the duplicated pull request
reallly appreciate for this great lib, it helps me a lot.
and I'm looking forward to oauth feature so I don't need to roll my own:)
add doseq to urlencode in model._encode_params to enable list as dict value

parse_qs and then urlencode, things will changed.

[test code and result](http://min.us/ldLJUS)

Thank you for this great lib,

the result of parse_qs is something like {'oauth': ['sig'], 'status': ['hanzi ok']}
urlencode can't handle list as dictionary value without doseq
doseq does three things,
1. if value is not list or unicode, behave like doseq=False
1. if value is unicode, decode unicode
2. if value is list, handle list as dict value

it will fall to decode if the unicode value is not encoded in ascii, Chinese for example.
but _encode_params already made sure that no unicode will pass to urlencode,
so I think it's safe to add doseq.

snippet from Python2.7/Lib/urllib.py

``` python
   if not doseq:
        # preserve old behavior
        for k, v in query:
            k = quote_plus(str(k))
            v = quote_plus(str(v))
            l.append(k + '=' + v)
    else:
        for k, v in query:
            k = quote_plus(str(k))
            if isinstance(v, str):
                v = quote_plus(v)
                l.append(k + '=' + v)
            elif _is_unicode(v):
                # is there a reasonable way to convert to ASCII?
                # encode generates a string, but "replace" or "ignore"
                # lose information and "strict" can raise UnicodeError
                v = quote_plus(v.encode("ASCII","replace"))
                l.append(k + '=' + v)
            else:
                try:
                    # is this a sufficient test for sequence-ness?
                    len(v)
                except TypeError:
                    # not a sequence
                    v = quote_plus(str(v))
                    l.append(k + '=' + v)
                else:
                    # loop over the sequence
                    for elt in v:
                        l.append(k + '=' + quote_plus(str(elt)))
```

reallly appreciate for this great lib, it helps me a lot.
and I'm looking forward to oauth feature so I don't need to roll my own:)
Please add doseq to urlencode in model._encode_params

Thank you for this great lib

parse_qs and then urlencode, things will changed.
the result of parse_qs if something like {'oauth': ['sig'], 'status': ['hanzi ok']}
urlencode can't handle list as dictionary value without doseq

My English is poor so I'll show you my code directly.

[code and result](http://min.us/ldLJUS)

reallly appreciate for this great lib, it helps me a lot.
and I'm looking forward to oauth feature so I don't need to roll my own:)
Doesn't return complete URL in response.url when url has a hash (#) character

This may or may not be correct HTTP behavior, but when using a URL with a hash (#) character, response.url chops off the URL at the hash character. I would have expected the full URL.  (For redirects, I can retrieve it from the location header in the history.)

``` pycon
>>> import requests
>>> url = 'http://twitter.com/#!/charliesheen/status/42731720402931712'
>>> shortened = 'http://bit.ly/fxDgIl'
>>> response = requests.get(url)
>>> response.url
'http://twitter.com/'
>>> response = requests.get(shortened)
>>> response.url
'http://twitter.com/'
>>> response.history[0].headers['location']
'http://twitter.com/#!/charliesheen/status/42731720402931712'
```


This is correct behavior. 

---

Traditionally, The `#` is used as a fragment identifier on the client side, to specify an ancor tag (e.g. http://docs.python-requests.org/en/latest/api/#requests.get)

More info:  http://en.wikipedia.org/wiki/Fragment_identifier

However, hacking the fragment identifiers into "hashbang urls" (containing `/#!/`) has been a bit of a (controversial) trend lately. They were used for a while for javascript client applications to be relatively semantic. Some websites (gawker up until recent, twitter) are using them as their main URL system, which is confusing to users and developers alike.

If you look closely at what your web browser does when you request a URL like `http://twitter.com/#!/charliesheen/status/42731720402931712`, the `GET` request is actually only to `http://twitter.com/`. The browser receives a generic page that contains javascript that carries out `window.location` search, and loads the tweet in the hashbang url.

You can learn more about hashbangs here: 
- http://danwebb.net/2011/5/28/it-is-about-the-hashbangs

---

**Update:** Tweets have actual semantic urls too:
    https://twitter.com/charliesheen/status/42731720402931712


I believe this is correct behaviour, and a good example of why using fragments in this way is a really bad idea.

> Fragment identifiers have a special role in information retrieval systems as the primary form of client-side indirect referencing, allowing an author to specifically identify aspects of an existing resource that are only indirectly provided by the resource owner.  As such, the fragment identifier is not used in the scheme-specific processing of a URI; instead, the fragment identifier is separated from the rest of the URI prior to a dereference, and thus the identifying information within the fragment itself is dereferenced solely by the user agent, regardless of the URI scheme. - [RFC 3986](http://tools.ietf.org/html/rfc3986)

**Edit**: Damn, @kennethreitz, you fast!


:cake:
Request_gzip = True

In order to receive gzipped content, requests should send the header:

```
'Accept-Encoding': 'gzip'
```


This is intentional and correct behavior. 

If the server responds with gzip-encoded content, requests will handle it. 

I think the user should be explicit with the headers they send up if they'd like to change the default behavior of the server. 


Perhaps a `requests.settings.request_gzip = True` is in order?


Yes, but what about deflate compression?
Maybe simply something like `settings.compression = True` that enable both gzip and deflate compression?
Requests does not support Internationalized domain name (idna)

The following url http://➡.ws/gggo should redirect to google.com but it doesn't work with requests.

See the following code:

```
>>> import requests
>>> a = requests.get(u'http://➡.ws/gggo')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/jerem/dev/resolver/lib/python2.7/site-packages/requests/api.py", line 75, in get
    timeout=timeout, proxies=proxies)
  File "/Users/jerem/dev/resolver/lib/python2.7/site-packages/requests/api.py", line 53, in request
    r.send()
  File "/Users/jerem/dev/resolver/lib/python2.7/site-packages/requests/models.py", line 294, in send
    resp = opener(req)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/urllib2.py", line 392, in open
    response = self._open(req, data)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/urllib2.py", line 410, in _open
    '_open', req)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/urllib2.py", line 370, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/urllib2.py", line 1186, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/urllib2.py", line 1155, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/httplib.py", line 941, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/httplib.py", line 974, in _send_request
    self.putheader(hdr, value)
  File "/usr/local/Cellar/python/2.7.1/lib/python2.7/httplib.py", line 921, in putheader
    hdr = '%s: %s' % (header, '\r\n\t'.join([str(v) for v in values]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u27a1' in position 0: ordinal not in range(128)
>>> a = requests.get('http://➡.ws/gggo')
>>> a.content
'<html>\n<head>\n\t<title>.WS Internationalized Domain Names</title>\n</head>\n<frameset rows="100%,*" border="0" frameborder="0">\n\t<frame src="http://zoomdns.net" scrol
ling="auto">\n\t<noframes>\n\t\t<p> Your browser does not support frames. Continue to <a href="http://zoomdns.net">http://zoomdns.net</a>.</p>\n\t</noframes>\n</frameset>\
n</html>'
>>> a.url
'http://\xe2\x9e\xa1.ws/gggo'
```

As you can see, it doesn't redirect to google.com but to a compability page for client that doesn't upport idna.
workaround for urllib2 md5-sess bug

Feature Wishlist :-)

The bug with patch is here:

http://bugs.python.org/issue2202

I suspect this is pretty trivial to set up on an IIS server, sadly, I don't have access to one :-S


Actually, slight lie, I do have access to an IIS server, but it's a customer's. I obviously can't give out the password, and I doubt they'd like me advertising the url ;-)

However, if you can get things integrated on a branch, lemme know and I'll check it out and test!


I'm going to close this issue for now — we've moved off of urllib2, so a new approach can be taken if we decide to tackle this problem.
please support https client certificate authentication

The following Apache config section enforces the kind of auth I'm talking about:

SSLCACertificateFile /root/CA/cacert.pem
SSLVerifyClient require
SSLVerifyDepth 1

Using httplib2 (http://code.google.com/p/httplib2/) this is done as follows:

``` python
from httplib2 import Http
http = Http()
http.add_certificate(keyfile,certfile,urlparse(url).netloc)
```

...which is great, except for the suck of _having_ to parse the netloc.

My wishlist for requests would be:

``` python
requests.get(full_url,auth=AuthObject(method='cert',keyfile=keyfile,certfile=certfile))
```

However, on the subject, it's important for auth types of be used in parallel.
The specific case I have here has both client certificate _and_ basic auth at the same time.
I have no idea how I'd choose to spell that nicely, although httplib2 does it quite nicely:

``` python
http = Http()
http.add_credentials(username,password)
http.add_certificate(keyfile,certfile,urlparse(url).netloc)
```

thanks!

Chris
v0.5 is backwards incompatible

You may already know and be totally cool with this, but requests v0.5 is backwards incompatible due to [this commit](https://github.com/kennethreitz/requests/commit/80d860d2d409b6d2a6b4c133ef0eef4de6b32ef2).

```
>>> import requests
>>> requests.__version__
'0.4.0'
>>> # Sends a HTTP GET request to http://example.org with data in the query string
>>> requests.request('GET', 'http://example.org', data={'foo': 'bar'})

>>> import requests
>>> requests.__version__
'0.5.0'
>>> # Sends a HTTP GET request to http://example.org with data in the request body
>>> requests.request('GET', 'http://example.org', data={'foo': 'bar'})
```


IMO data should map to the query string for a GET.

Some more discussion here: http://stackoverflow.com/questions/978061/http-get-with-request-body


This change was introduced because parameters are now mapped on all query types, instead of only GET/HEADs. 

This leads to a much more consistent internal API, with less request-type branching.

`params` = querystring dict
`data` = body dict or bytes


It should also be noted that `data` is not an available keyword on the `get` function, which should be the primary method of making GET requests. 

@jgorset, are you using `request.request` directly? I view that as an internal-only function.


@kennethreitz,

if `request` is an internal function, I don't think it should be defined in the `api` module. I'd argue that it's [very useful](https://github.com/jgorset/facepy/blob/master/facepy/graph_api.py#L106), though.


@jgorset, can you hop on freenode #python-requests? 


Re: params = querystring, data=request body for all methods, I like that even better. :+1:
Python 3.x Support

It's closer than you think.


Are you going to use 2to3 at build-time, or manually maintain two branches, or support both lines from a single codebase?


@merwok: support both from the same codebase, ideally (this is what I do for [tablib](https://github.com/kennethreitz/tablib)). 


Okay.  I guess you’ll have a small compat module, like six (look for it on PyPI if you don’t know what it is), and sprinkle the ugly b function everywhere to have sane bytes/string separation.


https://github.com/kennethreitz/tablib/blob/develop/tablib/compat.py :D


Hey, if you don't mind my asking - is there any progress on it?


Yes, great progress! The `feature/rewrite` branch has urllib2 and poster removed, and has a nice new codebasethat will be wonderfully simple to port to 3.x :)


Requests v0.8.x has been released for several weeks now, which uses urllib3. Getting closer!


Hey, is there any progress on py3 compatibility?


Well, kind of... Team behind pip installer may pretty soon decide to use requests instead of what they currently have and when this happens the pressure for porting will be enormous :)


Actually, no pressure from us, really, take your time, get this right the first time.


@jezdez nothing wrong with some motivation :)


:sparkles: :cake: :sparkles:


**IT IS FINISHED**


Awesome, thank you!
Parameters are getting out of hand. 

Use context manager as an alternative way to make requests?

Currently, settings context manager is a singleton. I suspect this will cause multiprocessing issues.


Interesting idea!

Are you thinking something along the lines of...

``` python
with requests.settings(path='http://example.org', timeout=10) as r:
    response = r.get('/path/to/file.txt')
```

...?


Heh, neat path idea. 

I was thinking:

``` python
with requests.settings(timeout=10, auth=('a', 'b'), proxies={'http': 'localhost:8080'}):
    requests.get('http://blah.com/', params={'x': 'y'})
```


either way I think that's awesome.


Let's do it :)


Maybe this would be better:

``` python
with requests.settings(timeout=10, auth=('a', 'b'), proxies={'http': 'localhost:8080'}) as r:
    r.get('http://blah.com/')
```

This would escape the threading/processes issues.


If there's a way to detect if someone's using `as r`, then I'm all for doing it now. Otherwise, I'm going to think about it for a while.


My thought is always that explicit is better than implicit. And intuitiveness goes along with it. So then are context processors more explicit and are they intuitive?

This does look like a divergence from the intuitive aspect of Requests that I like so much. I can point other newbies at it and they **just get it**. I'm open to ideas, but keep in mind your audience. Use of context processors could be for advanced users, and be given appropriate documentation.


Great points.

If I do decide to implement this, it would be supplementary. All of the functions will accept the same paramaters they do now as well.


Maybe I'll hold off on this for now. 

Currently, you can do the following

``` pycon
>>> with requests.settings(verbose=sys.stderr):
...    requests.get('http://httpbin.org/ip')
2011-06-23T21:23:18.422726   GET   http://httpbin.org/ip
<Response [200]>

>>> requests.get('http://httpbin.org/ip')
<Response [200]>
```

As an alternative to:

``` pycon
>>> requests.settings.verbose=sys.stderr
>>> requests.get('http://httpbin.org/ip')
2011-06-23T21:23:18.422726   GET   http://httpbin.org/ip
<Response [200]>

>>> requests.settings.verbose=None
>>> requests.get('http://httpbin.org/ip')
<Response [200]>
```

I love this. 

Maybe, it would just be best to move proxies into the settings module instead of being _required_ to be passed into each request function for now.


**Decided**: current `settings` singleton use is a non-issue because `request.settings` should only manage (supplementarily) all-encompassing settings (timeouts, proxies, etc). 

Auth Management still needs to be rethought.


See #75 for Re: Proxy 


I'd also like to move all non-essential keyword arguments (timeout, proxies, cookies) into a well-documented and constient `**kwargs` again. This will solve the 'paramaters getting out of hand' issue.

**Closed.**
Remove Poster

Great project, but an unnecessary dependency. It is also the main barrier to 3.x compatiblity.


Useful links: 
- http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
- http://www.doughellmann.com/PyMOTW/urllib2/
- http://stackoverflow.com/questions/1270518/python-standard-library-to-post-multipart-form-data-encoded-data


Done.
Support for proxies

Implements the feature explained in issue #54

Now you can do a GET/POST/.... through a proxy.

E.g: 

``` pycon
>>> requests.get('http://docs.python.org/', proxies={'http' : 'localhost:3128'})
< Response [200]>
```


This is awesome, thanks! I'm going to toy around with it first before I merge. I'll get back shortly.


:cake:


Thanks! I was using requests ( nice job, it's really useful ) for a couple of weeks and I need this feature on a project at work.
Allow querystring parameters with POST, PUT

It would be nice to allow querystring parameters to be supplied for a POST or PUT request.

currently, I have to do something like:

```
request.post("http://localhost/test?" + urllib.urlencode(dict(foo=bar)), data="body")
```

I'd like, instead, to be able to do

```
request.post("http://localhost/test", params=dict(foo=bar), data="body")
```


I've pushed some changes to my fork which implement this; hopefully in a sane way.  The public API change is simply to add params parameters to the request.post and request.put functions (adding params at the end to avoid breakage if people are already calling the functions using positional arguments).


Thanks for the feedback.

I was torn between putting querystring params in the non-get methods, and was waiting to see if anyone needed it.


Merged. 


Released!!!
settings.verbose stream/logger support

Print all requests to `stdout`:

``` python
>>> requests.settings.verbose = sys.stderr
```


This sounds very useful; but I hope it won't be a global configuration as the above implies - I'd like to be able to log some requests, but not all.


`requests.settings` is also a context manager :)

``` pycon
>>> with requests.settings(verbose=sys.stderr):
...    r = requests.get('http://httpbin.org')
...
GET http://httpbin.org
```

I really need to update the docs...


In all honesty, @kennethreitz, it is I who should've updated the docs when I committed that. ;-)


@jgorset: feel free to :)


Done.
Header capitalization

 Fixed bug where not capitalizing a header properly caused that header to not be set.

I noticed when doing a post to a service that expected a json request body that my typo "content-type" rather than "Content-type" in my headers dictionary caused the Content-type to default to "application/x-www-urlencoded" rather than "application/json".  

I added a pair of tests that illustrate the problem with posts to postbin.


HTTP header names are case-insensitive, per spec: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html

Perhaps the website on the other side is not following the spec on this?


There was an issue in an early release where the wrong casing for `content-type` would fail to override `urllib2`s default Request `Content-Type` header, but it was fixed.

It's possible when I implemented `CaseInsensitiveDict` for `headers`, that something changed there. I'll take a look this evening.


Thanks for taking a look.


This was an error with your server.
Auth fail for private github repo

I've got a private repo here on github.  I want to suck down all the issues.  This works in curl (password removed):

```
$ curl -u "mw44118:XXXXXXXX" https://api.github.com/repos/spothero/SpotHero-Splash-Page/issues
{ 
... real data here
}
```

And this is what happens when I don't include the -u authentication stuff:

```
$ curl https://api.github.com/repos/spothero/SpotHero-Splash-Page/issues
{ 
  "message": "Not Found"
}
```

Then I tried the same thing with requests (again XXXXXX ain't  really my password):

```
>>> requests.get('https://api.github.com/repos/spothero/SpotHero-Splash-Page/issues', auth=('mw44118', 'XXXXXXXX'))
<Response [404]>
```

I wondered if maybe you don't actually send the auth stuff unless the server replies with a 403, and since github is replying with a 404, the requests code never knows to send the credentials.

I tried reading through the requests code, but didn't see anything that supported that guess.


I'm going to have to play with this. #74 may fix the problem, once it is implemented.


Same problem here. From Wireshark,

``` python
requests.get('https://api.github.com', auth=('user', 'pass'))
```

seems to not send the HTTP authentication headers.


I'm going to get on this tonight. 

It's because GitHub's api isn't asking for authentication. 


Until I figure out a good method for overriding default HTTP behavior, you can do this:

http://stackoverflow.com/questions/2407126/python-urllib2-basic-auth-problem/2955687#2955687


Alright, a new ForcedHTTPBasicAuthenticationHandler was added, and it's default.

This should work fine now::

```
requests.get('https://api.github.com', auth=('user', 'pass'))
```

:)


Thanks for writing this!  

I just ran easy_install -U and now it seems to work, because I get a <Response 200> instead of the <Response 404>.

Of course when I tried to read from the response, I got a huge stream of gibberish, but I'll try to figure that out myself.
Add support for proxies

It would be nice to be able to provide a dict of proxies to use eg

```
proxies = {'http': 'http://someproxy.com:8080', 'https': 'http://anotherproxy.com:8443'}
requests.get(url, params={}, headers={}, cookies=None, auth=None, proxies={}, **kwargs)
```


It's better do like here, with HTTP and SOCKS5 support:
http://code.google.com/p/httplib2/wiki/Examples#Proxies


Done.
Timeouts throw exception from urllib2

Slight grump. When I set a timeout (say `requests.get('some url', timeout=1.0)`), if the server on the other end doesn't reply in time, I get a `urllib2.URLError` thrown. Yay that it timed out right, but in order to catch the exception, I have to import `urllib2` as well. Any chance there could be a `requests` exception thrown there instead?


Luckily, the latest release raises a `requests.exceptions.Timeout` exception :)
Add verbose logging support for debugging

``` pycon
>>> requests.settings.log = True
# Logs all HTTP Traffic to stderr

>>> requests.settings.log = steam_like_object
# Logs all HTTP Traffic to given stream

>>> requests.settings.log = False
# [Default] No HTTP Logging
```


Done.


I can't find any documentation on this and it doesn't work for me, even after pip installing right from GitHub:

```
In [9]: import requests

In [10]: requests.settings.log = True
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/Users/fletcher/Work/Current/twrss/<ipython-input-10-e3570368ba7d> in <module>()
----> 1 requests.settings.log = True

AttributeError: 'module' object has no attribute 'settings'

In [11]: requests.__version__
Out[11]: '0.10.2'
```

What am I doing wrong?


Same here with current HEAD:

``` python
>>> import requests
>>> requests.settings.log = True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'settings'
>>> requests.__version__
'0.12.01'

```

Also, I can only find a couple of mentions to settings:

``` bash
$ grep -rnsI settings lib/python2.6/site-packages/requests/*
lib/python2.6/site-packages/requests/models.py:186:            # Pass settings over.
lib/python2.6/site-packages/requests/sessions.py:7:This module provides a Session object to manage and persist settings across
```

Has this been removed after closing this issue or something?
requests.post should accept strings in addition to dicts

The specification for HTTP POST is that it has a "message body". HTML forms sticks encoded key/values representing form inputs into that message body when submitted. However, this means that currently requests.post can't accept JSON or XML message bodies, which could make it hard to use with some APIs. Therefore, I propose the following:

``` python

"""inside of requests.post method"""

if isinstance(data, dict):
    # process as existing code now handles POST
elif isinstance(data, str) or isinstance(data, unicode):
    # Stick into message body
    # Do encoding if that is what the spec demands (I'm not sure)
else:
    # raise InvalidRequestDataFormat

"""Example of implementation"""

import json

import requests

post_dict = {"hello":"world"}
r = request.post(form_url, post_dict)

post_json = json.dumps(post_dict)
r = request.post(json_url, post_json)
```


`Request.post` accepts both dicts and bytestrings for the data parameter already :)


Thanks for submitting this issue though. Clearly, the docs need to reflect that capability. :)