Raibek/requests-github-issues huggingface dataset:
Facilitate for multiple hooks

I implemented support for multiple hooks by passing a list of callables:

```
requests.get(
    url = 'http://example.org',
    hooks = {
        'args': [do_something, do_something_else]
    }
)
```

In the interest of backwards compatibility (but mostly just because I think it's practical),
it remains possible to pass a single hook without encapsulating it in a list:

```
requests.get(
    url = 'http://example.org',
    hooks = {
        'args': do_something
    }
)
```


You're the man! 


This probably a wise approach. I feel like there should ben an append-mechanism though. LIke a hook register.


I'll add an `add_hook` method to `Request`. We can then have the top-levels just utilize that internally.


`register_hook`.
Default the encoding of "text" media subtypes to "ISO-8859-1"

Ref. HyperText Transfer Protocol, [section 3.7.1](http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1) on Canonicalization and Text Defaults.


I'm going to merge this, but into the `redsea` branch.


Merged.


What is the `redsea` branch, exactly?


It'll be 0.10.0 ;)

It makes `Response.content` bytes-only, and a new `Response.text` unicode-only.


oh, that's awesome!


so, you're +1 then?


Yes; I already decode `response.content` all over the place. I would consider renaming `response.content` to `response.bytes` in an attempt make its differences from `response.text` more immediately obvious, but I can see why you might not want to do that for compatibility reasons.


well `response.content` now is unicode when it can be, bytes when it can't.


Hmm. What are your thoughts on keeping the current implementation of `response.content` in the interest of backwards compatibility and introducing `response.bytes` and `response.text` for its typed representations... too much?


Too much. The whole point is because in python3, you can't mix the two ;(


Yes; `response.content` would need to be deprecated and removed in due time, leaving `response.bytes` and `response.text` for the encoded and decoded representations of the response body, respectively.

Then again, it's probably not worth breaking backwards compatibility for a fundamental part of the API over a naming preference that may even only pertain to myself. ;-)
Impossible to submit multiple query parameter values on a single key cleanly

Related is #179, which theoretically fixes this.

If there are a bunch of query parameters all with the same key and differing values, it is impossible to cleanly pass them in using the `params` kwarg, because whatever object is passed in on that route is `dict`ified.

Real use case: Ganeti's API has a few calls which expect URLs to be in the following format:

```
https://ganeti.example.org:5800/2/some/tag/modifying/api/modify?tag=spam&tag=eggs&tag=spam&tag=spam&tag=bacon
```

This is circumventable by prepacking the troublesome parameters onto the URL instead of as kwargs, but it's a bit cumbersome and a better remedy would be preferable.


Actually, this is super simple :)

``` python
params={'tag': ('spam', 'eggs', 'spam', 'spam', 'bacon')}
```

This should be documented.


Sorry for not confirming your fix yet; it was a long weekend. I'll close upon confirmation. Should I also submit a docs patch for this?


No worries :)

A pull request would be quite fantastic.


I feel like it should go in the advanced section.


Confirmed working. Thanks a bunch. Pull request for docs will follow in a bit.


I don't see an obvious way to do this with files though. I assume the following, but it doesn't work. Looking through the code I don't see an obvious way to do it. 

```
files={'attachment': [('name1', file1), ('name2', file2)]}
```

I now see #285 covers files.


This should be solved soon. Requests will accept " dictionary update sequences" everywhere.
Install Failed on CentOS 5.6 + Python 2.4

[root@raullen ~]# easy_install requests
Searching for requests
Reading http://cheeseshop.python.org/pypi/requests/
Reading http://python-requests.org
Reading http://cheeseshop.python.org/pypi/requests/0.9.3
Best match: requests 0.9.3
Downloading http://pypi.python.org/packages/source/r/requests/requests-0.9.3.tar.gz#md5=b13b6fbfa8fc3fc3c25bae300748053f
Processing requests-0.9.3.tar.gz
Running requests-0.9.3/setup.py -q bdist_egg --dist-dir /tmp/easy_install-jyVdnT/requests-0.9.3/egg-dist-tmp-8CYvTK
Traceback (most recent call last):
  File "/usr/bin/easy_install", line 7, in ?
    sys.exit(
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 1670, in main
    with_ei_usage(lambda:
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 1659, in with_ei_usage
    return f()
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 1674, in <lambda>
    distclass=DistributionWithoutHelpCommands, **kw
  File "/usr/lib/python2.4/distutils/core.py", line 149, in setup
    dist.run_commands()
  File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 211, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 446, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 471, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 930, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
    DirectorySandbox(setup_dir).run(
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in <lambda>
    {'**file**':setup_script, '**name**':'**main**'}
  File "setup.py", line 6, in ?
  File "/tmp/easy_install-jyVdnT/requests-0.9.3/requests/**init**.py", line 26
    from . import utils
         ^
SyntaxError: invalid syntax

I will probably take a look on that!


Python 2.4 is not supported.


If this is the case, it would be better to mention it explicitly in the README or other instructing sheets to avoid confusion.
Added requests.Response.response_time

Added requests.Response.response_time to keep track of how long requests take. The user could keep track of this himself except for when he is using requests.async. 


Two notes:
1. `float(datetime.now().strftime("%s.%f"))` is exceedingly inefficient compared to `time.time()`, which returns a float directly.
2. I would measure the end time immediately before the response hook call so you don't measure other things than the actual request processing.


updated


ya know, it might be a good idea for me to run my code before pushing it :) . fixed and tested


> ya know, it might be a good idea for me to run my code before pushing it :) 

No, it's not enough to merely run code. You should run tests before submitting code.


I am often times too much in the hacker state of mind and not enough in the developer state of mind. I wrote a test for the response_time and made sure that I didn't break anything else.


Interesting.

Holding off on this for now. Going to think about it ;)


@kennethreitz Note that this would be a trivial user addition if multiple hooks were supported, which would be a nice way to add this for people who care without adding two syscalls (albeit an extremely cheap call on modern Unix variants) per request for everyone.


@acdha multiple hooks would be huge. I want. Badly.


@acdha multiple hooks are in!


(Closing for hooks)

Maybe it would be worthwhile having this hook be built in.
Make requests.async.send asynchronous 

The async.send method wasn't really doing anything useful. It was just calling the r.send() method of the Request object 'r'. It took an optional kwarg pools, and assigned it to r._pools which isn't used by anything. This situation was eaither broken or not fully implemented yet. I modified async.map to call r.send rather than send(r) and then modified async.send to actually make use of the pool that it is passed.


Love it. Thanks!


Btw, any idea how http pools (provided by urllib3) mix with gevent pools?


each request is handled separately, and an http pool is contained within each request.


Hmm, actually, if the `session` argument was passed implicitly to each request as it was created, they'd all be using the same pool. We should do some tests.
chunked streaming+line reading

this branch adds custom reading for chunked responses
that way the iter_content function has chunk boundaries as additional boundaries,

additionally iter_lines will no longer keep a completed line in pending

that way using requests for chunk streamed update notifications from couchdb gets actually usefull

there are probably some rough edges i need to check, but the basic stuff works,
please review and yell at me


Looks good. I'm about to release this. Objections?


@RonnyPfannschmidt I'm looking into debugging issue #436 with generate_chunked and noticed you merged this. I was wondering if you had any insight on what the trouble is there?
Session + params giving a 411 error

One test in requests-oauth is failing and I think is due to requests itself. It's this one:
https://github.com/maraujop/requests-oauth/blob/dev/tests/tests.py#L77

When passing parameters to a session,  I'm getting a 411 Length Required error. I've sniffed the traffic and `Content-Length` is not in headers.

Cheers,
Miguel


Same problem here:

``` python
import requests
from lxml.html import fromstring


s = requests.session()

login_req = s.get('http://music.google.com/')
login_tree = fromstring(login_req.content)
login_form = login_tree.forms[0]
login_values = dict(login_form.form_values())
login_values.update({'Email': 'whatever@gmail.com', 'Passwd': 'ilovethepony'})

g_music_req = s.post(login_form.action, data=login_values, allow_redirects=True)
print g_music_req.status_code
# 411
```


Closing for #223
Fix bug in HTTP-digest auth w/ URI having query string

When there's a query string in the request URL, we need to separate it from the parsed path with a question mark when constructing the digest input.

(Sorry, I didn't add a test case for the problem.)


I can confirm that this patch fixes a bug I ran into where all of my HTTP Digest Auth request calls worked except ones with query strings (like /whatever?foo=bar). I could observe this in wireshark where the query strings were being truncated by requests.


Thanks a ton!
Wrong handling of params given as bytes object

``` python
from collections import OrderedDict
from urllib import urlencode
import requests

params = OrderedDict([('first', 1), ('second', 2), ('third', 3)])
requests.get('https://example.org/', params=urlencode(params))
```

ends with

`ValueError: dictionary update sequence element #0 has length 1; 2 is required` at `models.py", line 82` (`self.params = dict(params or []`)
Check cookie's expiry time when using sessions

Hi,
Just stumbled across requests, makes things a lot easier for me.

One thing i came across, was one site im trying to request, sets some expired cookies (along with genuine cookies). The set cookie would look something like this

```
'_xxxx=deleteMe; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/, yyyy="delete me"; Version=1; Max-Age=0; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/, zzzzz="000000000000000000000"; Version=1; Max-Age=1799; Expires=Tue, 10-Jan-2012 08:36:05 GMT; Path=/, yyyy="delete me"; Version=1; Max-Age=0; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/
```

and so on...

In the case of request , it resends the Expired cookies, but real browsers do not! My guess is that this site uses such cookies to possibly detect non-browser users. In the above example only zzzz should be sent on subsequent request, cause other cookies expired 32 years ago.

I built some very very ugly code to fetch the page using httplib2 and parse cookies myself, the code would only work on this particular site.. i thought requests may be a relief from my crappy code.


Cookies are key/value only for now. I hope to change this soon.
When using session objects, deleting a key from headers fails

Using version 0.91, downloaded 1/09/12, this code produces the specified exception:

> > > import requests
> > > s = requests.session(headers={"foo": "bar"})
> > > s.get('http://httpbin.org/headers', headers={"foo": None})
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "C:\Python27\lib\site-packages\requests\sessions.py", line 217, in get
> > >     return self.request('get', url, **kwargs)
> > >   File "C:\Python27\lib\site-packages\requests\sessions.py", line 159, in request
> > >     headers[k] = header_expand(v)
> > >   File "C:\Python27\lib\site-packages\requests\utils.py", line 140, in header_expand
> > >     for i, (value, params) in enumerate(headers):
> > > TypeError: 'NoneType' object is not iterable

The problem is that expand_header, in requests/utils.py:140, because "None" is not enumerable. 

The fix I've patched in is to add 'elif headers is None: return headers' before enumerating the headers variable (part of the prior if/elif block).

Great library, thanks!


This is definitely a problem. Sorry for the bug!


From my messing about, this also happens if you aren't using sessions:

```
import requests
r = requests.get('http://httpbin.org/headers, headers={"foo":None})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/sessions.py", line 217, in get
    return self.request('get', url, **kwargs)
  File "requests/sessions.py", line 159, in request
    headers[k] = header_expand(v)
  File "requests/utils.py", line 157, in header_expand
    for i, (value, params) in enumerate(headers):
TypeError: 'NoneType' object is not iterable
```
request.request doesn't accept bytes for params argument since 0.8 version

In the latest versions of `requests`, passing bytes to `requests.request` and all helper functions (for example `requests.get('http://google.com', params='q=test%20me')`) will trigger the following error:

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/requests/api.py", line 50, in get
    return request('get', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 38, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 190, in request
    r = Request(**args)
  File "/Library/Python/2.7/site-packages/requests/models.py", line 86, in __init__
    self.params = dict(params or [])
ValueError: dictionary update sequence element #0 has length 1; 2 is required
```

The last known version of requests where code like this works fine is 0.7.6.


Correct. It makes no sense for us to support bytes in query strings. You're more than welcome to add it to your url yourself, though.


I'm not going to argue against closing, but http://docs.python-requests.org/en/latest/api/ needs a small update for description of params argument. `params – (optional) Dictionary or bytes to be sent in the query string for the Request.` should become `params – (optional) Dictionary to be sent in the query string for the Request.`
ConnectionError in requests, not urllib2

```
>>> url = 'http://www.rpgwatch.com?rwsiteid=1'
>>> len(urllib2.urlopen(url).read())
112749
>>> len(requests.get(url).content)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 50, in get
  File "build/bdist.macosx-10.7-intel/egg/requests/api.py", line 38, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/sessions.py", line 200, in request
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 498, in send
requests.exceptions.ConnectionError: Max retries exceeded for url: /?rwsiteid=1
```


Hmm, not sure about the specifics of URL rules, but 'http://www.rpgwatch.com/?rwsiteid=1' works fine. Maybe this is the same as #323


Between then and now, this has been resolved. Unintentionally or not, I'm going to go ahead and close this as fixed. 

:cake:
LookupError in requests, not in urllib2

```
>>> url = 'http://dilbert.com'
>>> len(urllib2.urlopen(url).read())
30194
>>> len(requests.get(url).content)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 713, in content
LookupError: unknown encoding: utf-8lias
>>> 
```


Heh, some heavy testing here... :)


What can I say, I run an RSS feed reader. I fetch millions of URLs a day. About half are ported over to requests, and I'd like to move the other half.


It looks like they are sending this trash for at least 3 years now... :)

> Posted by: Ryan | April 21, 2008 at 01:04 PM
> Yeah, WTH is up with the new Dilbert.com page? Tell whoever it is that runs the server that "text/html; charset=utf-8lias" > is not a valid Content-Type response header.

Taken from http://dilbertblog.typepad.com/the_dilbert_blog/2008/04/apr-for-houses/comments/page/2/

Funny, someone already had a problem with parsing this in Python; see http://objectmix.com/python/784974-where-locate-existing-standard-encodings-python-2.html


We should definitely be catching this.


Looks like this is fixed?


Not really. `res.content` is no more unicode that's why there's no encoding done when accessing it

``` py
>>> type(requests.get('http://dilbert.com').content)
<type 'str'>
```

However the problem is still present with `res.text` which is unicode

``` py
>>> len(requests.get('http://dilbert.com').text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\requests\requests\models.py", line 774, in text
    content = str(self.content, encoding, errors='replace')
LookupError: unknown encoding: utf-8lias
```

The error is present regardless of `chardet` module being installed or not.


We currently have a pretty robust encoding detection system, but it's only a fallback for when the server doesn't specify an encoding.


We're currently blindly trusting that the encoding that the server is responding with will exist.

Perhaps a map of encodings (e.g. ISO-LATIN-1 => latin1) is best?


> Perhaps a map of encodings (e.g. ISO-LATIN-1 => latin1) is best?

We can't keep adding misspelled and imaginary (ISO-LATIN-1 is not registered alias per http://www.iana.org/assignments/character-sets)  encodings' names found by people on the Internet :)
Providing a hook seems a better approach.


I agree.

You're welcome to do `response.encoding = None`, and it'll use the built-in detection.


so we should make sure that the encoding actually exists in the header detection step
zlib error in Requests and not urllib2

```
>>> url = 'http://www.federaltimes.com/rss/'
>>> len(urllib2.urlopen(url).read())
70636
>>> requests.get(url)
<Response [200]>
>>> requests.get(url).content
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.macosx-10.7-intel/egg/requests/models.py", line 693, in content
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/urllib3/response.py", line 142, in read
  File "build/bdist.macosx-10.7-intel/egg/requests/packages/urllib3/response.py", line 33, in decode_deflate
zlib.error: Error -3 while decompressing data: invalid block type
>>> 
```


Fixed!
Add eager mode for raising errors immediately.

I'm not 100% on:
1. The name of the setting. Settled on `eager_mode`.
2. The placing of the `raise_for_status()` method call. Settled on the very end of `Request.send()`.

Let me know if this is something you guys are interested in adding or not. It's pretty lightweight and I've included a test case.

_PS: the test `test_invalid_content` was failing before and after I added this feature._


Hmm, interesting. I have some reservation about adding this, but it would save a lot of `raise_for_status` raising. Hmm..


Yeah, I did a quick count and found 91 matches for `raise_for_status` in my code. This should cut that down substantially. :-) 

If you have a different idea on implementing this, I'm happy to whip up another commit. Such a feature would definitely help me (and even more so when debugging while in a shell).


Yeah, I vote for an option like this too. We currently use a subclass of Session, where we add a `raise_status` option that works similarly.

Ours is on by default, and I must admit, I turn it off more often than I thought would be necessary, but the number of calls that raise still outnumber the number I don't want to. Not sure about the name `eager_mode` though..


@gazpachoking Yeah, a bit torn on the name. `raise_status` or `always_raise` may be better.


I like `eager_mode`. It complements `safe_mode` well.


@bryanhelmig Heh, both of those name have actually been in my code at some point.
@kennethreitz Yeah, now that you mention it as a complement to `safe_mode` I like it a bit better. I just feel like the name doesn't imply that it is going to raise errors. Feels more like it would do something like prefetch mode to me.


Perhaps: `danger_mode`™


+1 `danger_mode` ;-)


`danger_mode` it is.


:)


@kennethreitz Let me know if this is enough accept the pull, I'm happy to do any additional legwork.


@bryanhelmig I'd be happy to pull now, but I'll give you a sparkly piece of cake if you add it to the docs :)


@kennethreitz I do love cake. :-) Where in the docs is this appropriate to mention? I assume the makefile will grab the appropriate docstring and add to http://docs.python-requests.org/en/latest/api/#configurations automatically?


Possible name: `fail_fast`.


I'd love to see a quick mention of it in the "advanced" section. 


@kennethreitz Would http://docs.python-requests.org/en/latest/user/quickstart/#errors-and-exceptions be appropriate?


@bryanhelmig even better!


@kennethreitz Added something rather minor.

On another note, it might be a good idea to flesh out the configuration section of the advanced page at some point in the future, as it just points to the API docs right now.


`fail_fast` is nice and much better than `danger_mode`


I haven't tried yet, but what happens if `safe_mode` and `danger_mode` are both on?


Actually, that won't be a problem at all — they cover different error scenarios.

Beautiful. Enjoy your cake!

:sparkles: :cake: :sparkles:


Perfect.
Cookies are sent regardless to domain when redirecting

Assume an HTTP server which sets the following headers when a GET request is made to "http://example1.foo/bar.cgi":

Location: http://example2.foo/bar.cgi
Set-Cookie: mycookie=something; path=/

In the case of running:

```
requests.get("http://example1.foo/bar.cgi")
```

results sending the mycookie to example2.foo/bar.cgi.

According to RFC 6265, I think the http client must not send the cookie in this case to example2.foo as example2.foo doesn't match the domain example1.foo (which is set implicitly by omitting the domain attribute in the Set-Cookie header).


Correct. Cookies are currently not tied to a domain. Having Cookiejar-backed cookies once again will solve this, but this isn't exactly trivial to do with httplib (As opposed to urllib2).
Returning request.content in preferred format [feature request]

`r = requests.get("some api which results contents in json")
type(r.content) => unicode.`

`r = requests.get("some api which results contents in json", return_format = 'json")
type(r.content) => json`

In current scenario request will return r,content in unicode format even though api results json or xml contents.
It will be easy for developer to mention return_type = 'json', so the request will convert unicode to json.

*_Shortcoming: *_
 If the api returns xml and user specifies return_type: json it is tedious to convert. If like to like conversion is possible it will be cool feature.


This would be a cool feature, but it's way out of scope for requests. I've considered it many a time, but it comes down to this: requests is an http library, not a json, html, xml, yaml, &c library. 


Yes, you are absolute correct. Converting unicode to json hardly takes few lines, I just thought it will be awesome out of box feature. 


We're on the same page. I think it would be neat too, but I think it would be the start of a bad trend.


Fine.
Clean way to silence warnings in connectionpool?

Logging messages in connectionpool.py are set to level warning, and there appears to be no clean (eg. in config/defaults) way to silence them (please correct me if I am wrong about that!).

Currently, after importing requests I am doing this:

```
logging.getLogger('requests.packages.urllib3').setLevel(logging.CRITICAL)
```

It seems a bit inelegant though. Is there a better way?


This is not inelegant in any way. That's how logging module is supposed to work.


I realize that is how it works (or I wouldn't have known to do it!). :P

As to 'inelegant', it just seemed that way to me because urllib3 is kind of a submodule of requests, and is otherwise relatively hidden underneath requests' api model layer and required special knowledge of non interface code to silence unexpected logging.

I have no problem doing it the way I did though. 

This issue was created because I was wondering if the way I did it was the intended/recommended way (e.g. avoid api changes breaking the work around) or if I missed a config element or other suggested mechanism somewhere.


Should we agree that this is a documentation problem?


(I never use logging myself)


The only stuff you should do in requests is add a NullHandler so your lib doesn't create warnings about missing handlers.

See here : http://docs.python.org/howto/logging.html#configuring-logging-for-a-library

Every thing else is done by the user


> The only stuff you should do in requests is add a NullHandler so your lib doesn't create warnings about missing handlers.
> 
> See here : http://docs.python.org/howto/logging.html#configuring-logging-for-a-library

I believe that is already being done in the **init**.py. 

> Every thing else is done by the user

I wonder if 'warning' is the right error level for library logging though. As an exception is reraised, and is part of the parent api expectation, warning level will simply "leak" logs if the user does not manually override it with a default log level. Maybe a debug log level makes more sense there?
SOCKS proxy support

Maybe this exists and I just can't find it, but it'd be awfully nice if requests had support for SOCKS proxies. If this exists already, could someone give me a simple example? Thanks.


I patched in socks5 support in my fork (https://github.com/kbatten/requests/commit/fb4676bcabcb53471272b6bfbecada7ce6161be6). I'm not sure if this specific implementation is appropriate for Requests though because it bypasses the proxies={'scheme':'host:port'} method and instead globally sets up the proxy to handle any socket connections with the specified remote port (instead of scheme.)


Try [PySOCKS5](https://bitbucket.org/shugelee/pysocks5)

Simple demo
https://bitbucket.org/shugelee/pysocks5/src/23625b3ad726/demos/using_socks5_with_auth_and_requests.py


See issue #478. It seems to solve this.
Add ending slash to raw domains before query insertion

Method 'get' with  not empty params ends with error:

File "/usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py", line 524, in get_host
    port = int(port)
ValueError: invalid literal for int() with base 10: '7766?test=s'

It is probably caused by invalid parsing of port in method get_host.


Can you attach some code to reproduce?


import requests
requests.get("http://localhost:80", params=dict(test="s")) 


Ah, we should fix that. Nice catch.

Technically, that's an invalid URL. Try this:

``` python
requests.get("http://localhost:80/", params=dict(test="s"))
```


Ok, thanks.


Keeping this open, because requests should compensate for this.


This issue was fixed by Pull Request #401.
Test failures with requests 0.9.0

I get lots of these errors (most likely related to gunicorn starting correctly):

<pre>
# PYTHONPATH=build/lib python test_requests.py 
EEEEEEEEEEEEEEEEEE.E..EEEEEEEEEEEEEEEEEEEEEE
======================================================================
ERROR: test_BASICAUTH_HTTP_200_OK_GET (__main__.RequestsTestSuite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_requests.py", line 176, in test_BASICAUTH_HTTP_200_OK_GET
    r = requests.get(url, auth=auth)
  File "/home/jbergstroem/requests/requests/api.py", line 50, in get
    return request('get', url, **kwargs)
  File "/home/jbergstroem/requests/requests/api.py", line 38, in request
    return s.request(method=method, url=url, **kwargs)
  File "/home/jbergstroem/requests/requests/sessions.py", line 200, in request
    r.send(prefetch=prefetch)
  File "/home/jbergstroem/requests/requests/models.py", line 498, in send
    raise ConnectionError(e)
ConnectionError: Max retries exceeded for url: /basic-auth/user/pass
</pre>


requests 0.9.0, envoy 0.0.2, gunicorn 0.13.4, certifi 0.0.6. tested on python2.5-2.7.


looks like you aren't running httpbin correctly.
Make safe_mode handle exceptions more consistently

I'm not sure how I feel about this patch, but I thought I'd submit a pull request regardless because if nothing else it raises a couple of issues that need some attention, IMHO.

Instead of passing None to `_build_response` when a request failed in safe mode, we now pass an empty `urllib3.HTTPResponse` object instead. One advantage is that you no longer need to check for None in the `iter_*` methods, because reading the bare response works as expected (i.e., you end up with an empty iterator). Having a regular object is also nice because you can save suppressed exceptions in an attribute before passing it to `_build_response`, which allows us to avoid breaking the current pattern of having the nested build function there build a `requests.Response` by inspecting attributes of the raw response.

I took out the optional `is_error` arg in `_build_response` because it was not getting used anywhere, nor was it part of the public API.

But: is it a good idea to store suppressed exceptions in safe mode on `Response.error`? This makes `Response.ok` work as expected (or at least as _I_ expect :P), but maybe this encourages bad habits. If you need to know what exceptions (if any) were raised by your HTTP request, you probably shouldn't be suppressing them.

On the other hand, the current implementation seems a little inconsistent:

``` python
>>> import requests
>>> r = requests.get('http://__bad_url__/', config=dict(safe_mode=True))
>>> r.ok
True
>>> r = requests.get('http://httpbin.org/status/501', config=dict(safe_mode=True))
>>> r.ok
False
```

Thoughts?


Thanks for the thoughts — I'm glad someone noticed. I lot of these oddities that you noticed are remnants from old iterations of requests that need to be cleaned up. 


I'm going to accept this. I think it's a good step going forward.


Thanks. I like this a lot.

Last thing to do is have safe_mode be default for async requests.
fix iteration on null responses in safe_mode

Previously, calling `iter_lines` or `iter_content` on a blank response caused an `AttributeError` to be thrown, because it called `self.raw.read()` where `raw` was `None`. In my opinion, the correct behavior should be to return an empty iterator.


Excellent, thanks!

For the record, I commend you for adding tests to all of your pull requests. I wish everyone was like you.

:sparkles: :cake: :sparkles:
Fix for #30: options for certificate validation and client certs.

Quick fix for issue #30. Note that this doesn't do hostname matching when verifying, although this is an issue in urllib3 - maybe it should use [`backports.ssl_match_hostname`](http://pypi.python.org/pypi/backports.ssl_match_hostname/) if available.

For example, you can set the global `ca_certs` and enable cert verification with

```
import requests

requests.defaults.defaults['verify_cert'] = True
requests.defaults.defaults['ca_certs'] = '/etc/ssl/certs/ca-certificates.crt'
```


Awesome!

Those default dicts shouldn't be touched by any users. There needs to be a better API for this.


> Those default dicts shouldn't be touched by any users.

This would be not quite pythonic I guess :)


shazow/urllib3#25


Requests v0.8.8 was just released that includes ssl verification!

http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification
Exceptions inside greenlets

Hi,

When using the async stuff, how do I correctly handle an exception that occurs inside the greenlet?

```
Traceback (most recent call last):
  File "/Users/jamie/.virtualenvs/pulse/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/jamie/.virtualenvs/pulse/lib/python2.7/site-packages/requests/async.py", line 50, in send
    r.send()
  File "/Users/jamie/.virtualenvs/pulse/lib/python2.7/site-packages/requests/models.py", line 454, in send
    raise ConnectionError(e)
ConnectionError: Max retries exceeded for url: /followers/ids.json?user_id=21762493
<Greenlet at 0x111fe05f0: send(<Request [GET]>)> failed with ConnectionError
```


aync requests now have `safe_mode` enabled by default.


This doesn´t solve the problem. Same issue here. Adding "ConnectionError" on line 650 in models.py solves the problem, buts it´s just a workaround
Unhandled exception on timeout in safe_mode

If safe_mode is specified, and a timeout occurs, the following error is raised:

``` python
>>>requests.get('http://timeout.com', timeout=0.01, config=dict(safe_mode=True))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build\bdist.win32\egg\requests\api.py", line 79, in get
  File "build\bdist.win32\egg\requests\api.py", line 66, in request
  File "build\bdist.win32\egg\requests\sessions.py", line 191, in request
  File "build\bdist.win32\egg\requests\models.py", line 468, in send
UnboundLocalError: local variable 'r' referenced before assignment
```
Add Proxy Authorization Support

A new class is auth.py adds the `Proxy-Authorization` header to a Request object.

If a proxy is specified in the request, the `send()` method is going to look for authentication info in the proxy URL string and set up the proxy auth from there.

None of this works with out the changes to urllib3 in #293, however.


Thanks a ton for this!


(Great pull request btw — honored to be your first :)
Client Certificates in Requests

In requests there doesn't seem to be a way to specify an SSL key and certificate for client certificate auth.
urllib and httplib both support the client cert approach. This is a commonly requested implementation for business to business communications.

Is this something that we would use the custom auth approach for, using the interface to bind the values to the internal urllib object... or, what?

Just want to get a handle on whether or not this is achievable using requests.

John


Hey there. I'm looking forward to adding support for this soon.

See #30 for discussion and details.
Digest Authentication in 0.7.3 versus 0.8.4

After upgrading from 0.7.3 to 0.8.4:

A digest request such as:

``` python
r = requests.get('http://app.com/sensors.json',  auth=('digest', 'username', 'password'))
```

Yields:

```
matt@Jenkins:~/Development/sensor-tools$ python pollsensors.py
Traceback (most recent call last):
  File "pollsensors.py", line 84, in <module>
    s = PollSensors()
  File "pollsensors.py", line 12, in __init__
    r = requests.get('http://app.com/sensors.json', auth=('digest', 'username', 'password'))
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 80, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in request
    prefetch=prefetch
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 191, in request
    r.send(prefetch=prefetch)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 400, in send
    r = self.auth(self)
TypeError: 'tuple' object is not callable
```


There was a backwards incompatible change. 

Here's the new usage:

http://docs.python-requests.org/en/latest/user/quickstart/#digest-authentication


ah, shit. I didn't see that

works now.


Sorry for the backwards-incompatible changes. They're for the best :)


no worries, I like this new syntax better!
prefetch=True and RuntimeError: The content for this response was already consumed

Hello!

requests 0.8.3 (was installed with pip), python 2.7.1

> > > import requests
> > > r = requests.get("http://www.httpbin.org/headers", prefetch=True)
> > > print r.content
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/home/hurr/venvs/durr/local/lib/python2.7/site-packages/requests/models.py", line 584, in content
> > >     'The content for this response was already consumed')
> > > RuntimeError: The content for this response was already consumed


Yikes, thanks for finding this. I'll push a fix up soon.
File upload not working with HTTP digest authentication

This is due to a couple of issues involved with the fact that it is necessary to create two requests and the file already having been read and a boundary set in the content-type header by the first request, resulting in the second (authenticated) request lacking file data and having an incorrect boundary even if you reset the file input stream.

However, more than that it seems like a waste of bytes to send POST data up with the initial request which you know will come back as 401. As a quick fix in my own project I have added an if statement around lines 360-388 in models.py with the conditional 

```
if (not isinstance(self.auth, HTTPDigestAuth)) or 'Authorization' in self.headers:
```

but I expect there is a more elegant way. If there already exists a way to do an empty first request when using HTTP Digest then I was unable to find it.


Thanks for the report. I'm really unhappy with the post-hook thing that's being used in the Digest Auth handler. It feels really messy.

I'd love to take a more elegant approach, but nothing seems immediately obvious.
File download: get the filename without the data

Is it possible to just retrieve the filename when doing a get request without the file itself?

For example, any browsers prompts for a filename to save before actually downloading the file if the save button is hit.


This might help: http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow


Wow it seems the doc has been updated since last time I checked. Sorry for that useless issue, I may be able to read the filename in one of the header.

Thanks!
Documentation fix: clarification of keep-alive behavior.

The existing behavior was a little opaque for me--I was making a PUT request which returned a HTTP 204 with a 0-byte body. I was never reading the body and was wondering why my connections were not being reused.

Thanks so much for the requests library--we use it at App.net and absolutely love it. I threw away a big spaghetti'd-out mess of a homegrown urllib2 wrapper and haven't looked back.


That's for the kind words!

This is a great addition. Thanks!
Proxy authentication broken with 0.76 -> 0.83 upgrade.

Proxy authentication was possible in 0.76 using a string like "user:password@ip:port".  In 0.83 I am getting the following error:

Traceback (most recent call last):
  File "req.py", line 10, in <module>
    print get('http://ipcheckit.com', proxies=p).content
  File "build\bdist.win32\egg\requests\api.py", line 80, in get
  File "build\bdist.win32\egg\requests\api.py", line 67, in request
  File "build\bdist.win32\egg\requests\sessions.py", line 191, in request
  File "build\bdist.win32\egg\requests\models.py", line 409, in send
  File "build\bdist.win32\egg\requests\packages\urllib3\poolmanager.py", line 127, in proxy_from_url
  File "build\bdist.win32\egg\requests\packages\urllib3\connectionpool.py", line 521, in connection_from_url
  File "build\bdist.win32\egg\requests\packages\urllib3\connectionpool.py", line 497, in get_host
ValueError: invalid literal for int() with base 10: '(censored)@184.154.139.226:60099'


This may be an unexpected side effect of @shazow's hated for the urlparse module :)


You are correct. :(

I actually had a thought that @ would break my parsing last week but didn't get around to fixing it yet.

Sorry about this, I'll fix it this week. It's being tracked here: https://github.com/shazow/urllib3/issues/12

If everyone would prefer I switch to using urlparse proper, I'm willing to do that begrudgingly. :P


This specific case is fixed in https://github.com/shazow/urllib3/commit/e5c2c193a0a93f97202ad76749231723f8d60156


Fixed.
Accept file-like objects as request data

I'd like it to be possible to pass in a file-like object as the source of request data, provided `Content-Length` and `Content-Type` headers have been specified already. Here's an example of what I'd like to do:

```
response = requests.put(
    'http://example.com/endpoint',
    data=open('filename', 'rb'),
    headers={
        "Content-Type": "text/plain",
        "Content-Length": os.stat('filename').st_size})
```

Alternatively, requests could even recognise that I've passed in a `file` object and perform an `os.fstat(file_like_obj.fileno())` automatically.

Here's the bit of code that seems to be breaking:

https://github.com/kennethreitz/requests/blob/c7b99f78192bdff62fad4f92039396c0dfbe4fb8/requests/models.py#L267

When I pass in an object that is neither a string nor a dictionary, the call to `dict()` fails.

I'm able and willing to work on this in a fork, I just figured it would be better to submit it as an issue first.


Interesting. I see no reason why we couldn't support that.

Supporting this everywhere would also force us to revisit the `files` parameter. Currently, it expects file-like objects. It should also accept bytes/unicode strings.


In this case, I'd imagine that the `files` parameter wouldn't even be used in `Request.send()`; by that point, the data could be put into one large file-like object with a computed `Content-Length`.
Add use_session argument to request method.

This allows usage of an existing session to make a request with the request and the get, post etc. function.

I made this so that it is easier to create async requests with pre-existing sessions. Currently, we have to do

```
async.map([my_session.get(url, return_response=False, prefetch=True) for url in urls])
```

I got the `return_response` and `prefetch` arguments from inspecting the source of async module. This isn't very pleasant.

With the proposed simple change in this pull request, one can do

```
async.map([async.get(url, use_session=my_session) for url in urls])
```

which feels more natural.


Thanks! I've been thinking of this lately myself. 

Can you change from `user_session` to `session` ?


That's what I initially did, but that would mean changing the session's import. Ok with that?


That's fine ;)

## 

Kenneth Reitz

On Nov 25, 2011, at 12:04 AM, Shrikant Sharat Kandulareply@reply.github.com wrote:

> That's what I initially did, but that would mean changing the session's import. Ok with that?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/288#issuecomment-2871395


Done. Thanks a lot Kenneth :)


Thanks a ton!
Allow generators or any iterators for async.map

Hey Kenneth,

If I call `async.map` this way

```
async.map(async.get(u) for u in urls)
```

I get an empty list as the list of responses. But it works if I do

```
async.map([async.get(u) for u in urls])
```

The proposed change fixes this.

If in case, this kind of usage is not intended, i.e., if only a list has to be passed to the `async.map` function, please make it clear in the documentation. Currently it says "A _collection_ of Requests objects" which does not necessarily imply a _list_.

Thanks.

Note: I just hacked up a simple solution in the github editor, you might want to ignore it and fix it up your style :)


:cake: 

Thanks a ton!
Requests don't allow multiple files for a key

We are a startup developing a programmable email platform and we love using Requests.
However we have encountered a problem recently.

Our API tends to send data in key value pairs where a key could have multiple values.
In order to do so we used multi dictionary and it worked pretty well with Requests 0.7.5.
But with new versions of Requests e.g. 0.8.2 it looks like we won't be able to do that.
The problem is that data structures to store 'params', 'data' and 'files' are explicitly set to be dictionaries.
For 'params' and 'data' we could use a list of values for a key but it won't work for 'files'.

I don't know what the mentioned changes in Requests are based on but
is there a chance that the situation would change regarding multi valued keys?
Any advise regarding the issue?


This applies to any HTTP request in general: multiple values for a single key must be allowed, not just file uploads.


@mailgun Multiple values for singe key **are** allowed. See [`test_urlencoded_post_query_multivalued_and_data`](https://github.com/kennethreitz/requests/blob/develop/test_requests.py#L390)


So the problem exists only for uploaded files? 


This appears to be a bug in the file upload logic. I'll investigate.


Thanks!

On Sat, 26 Nov 2011 07:33:24 -0800
Kenneth Reitz
reply@reply.github.com
wrote:

> This appears to be a bug in the file upload logic. I'll investigate.
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/issues/285#issuecomment-2882085


My pull request was rejected. https://github.com/kennethreitz/requests/pull/422#issuecomment-4058039
I'm going to submit a new issue for urllib3.


Now the urllib3's encode_multipart_formdata function supports the list of tuples. Could you please take in the latest urllib3 and then modify like this patch?

```
diff --git a/requests/models.py b/requests/models.py
index 8368c9f..ca7d0b2 100644
--- a/requests/models.py
+++ b/requests/models.py
@@ -404,11 +404,6 @@ class Request(object):
         if self.files:
             if not isinstance(self.data, str):

-                try:
-                    fields = self.data.copy()
-                except AttributeError:
-                    fields = dict(self.data)   
-
                 for (k, v) in list(self.files.items()):
                     # support for explicit filename
                     if isinstance(v, (tuple, list)):
@@ -416,9 +411,9 @@ class Request(object):
                     else:
                         fn = guess_filename(v) or k
                         fp = v
-                    fields.update({k: (fn, fp.read())})
+                    self.data.append((k: (fn, fp.read())))

-                (body, content_type) = encode_multipart_formdata(fields)
+                (body, content_type) = encode_multipart_formdata(self.data)
             else:
                 pass
                 # TODO: Conflict?
```


Is this fixed now that #179 has been resolved?
Support passing data and params in ordered dict

Order of multiple params with the same name is important. To support sending such requests there should be an option to pass data and params in ordered dict (and then keep the ordering when building request of course).


Item sequences will be supported everywhere soon (e.g. `[('name', 'kennethreitz'), ('key', 'value')]`


It has to be supported in urllib3 then, right?


urllib3 is, as always, a non-issue.


> Item sequences will be supported everywhere soon (e.g. `[('name', 'kennethreitz'), ('key', 'value')]`

Well this would allow for ordering of arbitrary items, yes. What I need is multivalued param in body and as far as I can see `Request._encode_params` supports this and is used not only for query params but for data as well (`self.data, self._enc_data = self._encode_params(data)`). The name is a bit misleading as it suggests it's being used only for query parameters.


Closing for #179