Raibek/requests-github-issues huggingface dataset:
Trivial typo in README

Utterly trivial :-)


Thanks for the pull request @poirier! However, that's not a typo: it means 'highlight this like it's python in an interactive console'. =D


D'oh!  Thinking too much about PyCon lately and assumed that's what happened to someone else.  Sorry for the bother.


Pull requests are never a bother, we love having them, even if we don't accept them. Thanks so much for contributing! =)


> Pull requests are never a bother, we love having them, even if we don't accept them. Thanks so much for contributing! =)

This x2. Please don't be afraid to open other PRs. And if you'd like to ask someone in private before doing so, you can email me and I'll happily answer.
Changed the "im_used" informational status code for the value given by IANA (226)

Changed the "im_used" informational status code for the value given by IANA (226)

See RFC 3229 at http://tools.ietf.org/html/rfc3229#section-10.4.1 and HTTP status codes at http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml


@phndiaye Awesome, thanks for this! While we're here, do you want to fix up 208? It should be "already_reported".


Done. By the way, i've seen some status codes that did not match with the ones in the IANA list, but i don't if @kennethreitz did that intentionnaly or if it's just some errors. 

Eg:

`uri_too_long` is set to 122 in requests but the IANA set it to 414 (103 to 199 are unassigned)


Some are intentional, some are errors. `uri_too_long` looks like it's wrong to me. =)


That's what i thought too :). I will be searching for some status like the `uri_too_long` and do a "global" commit of the corrections.
requests force override query string encoding

I am dealing with a legacy platform that does not understand URL encoded parameters.

I have the following parameters

```
params = {'fiz': 'foo:bar', 'biz': 'barfoo'}
```

url generated by requests

```
http://localhost/?fiz=foo%3Abar&biz=barfoo
```

required url for the legacy platform

```
http://localhost/?fiz=foo:bar&biz=barfoo
```

I tried creating a custom query string and adding that to the url, however requests will decode this query, and re-encode it.

```
improt urllib
qry = urllib.urlencode(params).replace('%3A', ':')
requests.get(url+'?'+qry) #: requests encodes the colon back to %3A
```

Is there any quick work around that doesn't look as verbose as using straight urllib2? 

is it possible to get a "safe" or "replace" list hooked into the API, this would need to be done at the requests layer as python 2.7 is required.


This is a bit awkward to do in Requests. As I see it, you have two options. The first is to prepare the `Request` objects yourself:

``` python
import requests
import urllib

url = 'http://test.com/'
qry = urllib.urlencode(params).replace('%3A', ':')

s = requests.Session()
req = requests.Request(method='GET', url=url)
prep = req.prepare()
prep.url = url + qry
r = s.send(prep)
```

Your other option would be to build a Transport Adapter that mutates the URLs before sending them on, like so:

``` python
import requests
import urllib

class MutatingAdapter(requests.adapters.HTTPAdapter):
    def send(self, request, **kwargs):
        main, querystr = request.url.split('?') # Nasty hack, not always valid.
        querystr = urllib.unquote(querystr)
        querystr = urllib.quote(querystr, '/&:=')
        request.url = '?'.join([main, querystr])

        super(MutatingAdapter, self).send(request, **kwargs)
```


I recommend the first option.


you could try the json parameter of requests.post() and requests.get() http://stackoverflow.com/a/26344315/1056345
Provide way to cache DNS lookups when making repeated requests to the same host

A nice feature would be the ability to cache DNS requests when making multiple requests to the same host. This would include:
- an optional timeout for invalidating a cached DNS lookup
- possibly persist the cache across multiple python processes


Hi @alanhamlett, thanks for raising this issue!

Unfortunately, this would not be simple to do. Requests doesn't handle any of the low-level HTTP connection processing, we pass that off to [urllib3](https://github.com/shazow/urllib3). urllib3 also appears not to have any explicit hostname resolution calls in it, which means that it passes that job off to httplib in the standard library.

If httplib provided that feature this would be easy enough to plumb through, but it doesn't. This means any change made in urllib3 would have to circumvent the standard httplib hostname resolution behaviour, which would be pretty awkward. Altogether I doubt there's going to be much appetite for this feature.


My use case:
For some reason, requests to my local server via its vhost name are ultra slow, and I suspect DNS to be the culprit. Everyone suggests to use 127.0.0.1 rather than `localhost` or whatnot. Unfortunately, I can't do that - the server runs on HTTPS which needs the host name to work. So I'm stuck with DNS query every cron job.


@rr- You're taking a long time to resolve `localhost`? That name should be served from your hosts file, and shouldn't take long _at all_. If you're only using `localhost` as an example, you should consider adding a hosts file entry for the server in question.


I'm sorry but I jumped to conclusions. The bottleneck was somewhere else altogether and the logs I had fooled me into thinking it was the connection that was slow. You're right, the requests are fast. Yep, I used `localhost` only as example. Yep, using `/etc/hosts` improves situation (and seems to be downright necessary for certain scenarios involving NAT).


@rr- You should not need `/etc/hosts` for anything else except `localhost`.
If you need a DNS cache check out `nscd` or `dnsmasq`
The users' doc is mixing the python2, python3 syntax and error.

Should we fix this? 
if I file a patch, which python version syntax is perfered?

for example:

python2:
    >>> print r.url

python3:
    >>> r.content
    b'[{"repository"


Hi @s7v7nislands, thanks for opening this issue!

We're always open to having documentation updates! If you feel the need to make this change, we would expect Python 2 to be the version in the documentation.


@s7v7nislands basically I think it somewhat safe to assume that people using python 3000 realize that print is now a function that requires parentheses. As for displaying data, you can use `b'[{"repository": "kennethreitz/requests"}]'` because on python 2 `b'string' == 'string'` unlike python 3000 so that distinction is actually good to point out.

I guess, I'm saying that the better way is to put as much subtext in as possible (if that makes sense). I've had "improve requests documentation" on my todo list for a while now but I keep finding it hard to find the time. I know I won't have time this weekend but ideally next weekend I might be able to get to some of this. We really need to explicitly define the behaviour differences on python 2 and python 3. For simple examples, you can stick to whichever you feel is best. Python 2 (or python 3.3) might give you the most flexibility so you can clearly illustrate  "HEY THIS IS UNICODE!" with a `u'string'`.

And either way, it's good practice whether writing python 2 or 3 to use parentheses when writing print statements (especially if you want your code to work on both without much effort) so tacitly enforcing that with users is a good thing. :) In all the examples for github3.py (for [example](http://github3py.readthedocs.org/en/latest/#example)) I use syntax which I'm certain will work on python 2.6+ since the library works on python 2.6+.
View a Request, PreparedRequest as a curl command

In #139 and #97 there was work to view a Request as a curl command. It looks like it's gone. I can't seem to navigate the revision history to find when it was removed and see if there's any explanation why.

Would you be interested in bringing it or something like it back?


I would be -1 on this. It's far beyond the scope of the library. We don't log anything (if I remember correctly) currently and adding a method like that isn't something we have had for quite some time (since at least 0.11 since that's around when I started using requests, over a year ago).


Thanks for the pull request @matthewlmcclure! Unfortunately, I think Ian is right on this one. We haven't had this feature for a long time and it's not really one that's missed.


No problem. I'll explore doing it outside of Requests proper. Thanks for the quick replies. 


Very nice idea. I agree there's no place for this in the lib but still a very nice feature.


I man, I totally forgot that we used to do this. Good times :)


Here's [the same implementation in a separately importable module](https://gist.github.com/matthewlmcclure/5980698). Improvements welcome.
The problem with stream=True

``` python
import requests

r = requests.get('http://servag.rupsy.ru/media/2/Amanojaku_and_ZY_-_Womtek_Julla-2013/Mensaje.jpg', stream=True)
cc = 1
while True:
    print('read ' + str(cc*1024*100))
    r.iter_content(1024*100).__next__()
    cc += 1
```

Output

```
read 102400
read 204800
read 307200
read 409600
read 512000
read 614400
```

Thereafter, the script is not responding.

If you press Ctrl-C:

```
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    r.iter_content(1024*100).__next__()
  File "/opt/python33/lib/python3.3/site-packages/requests-1.2.0-py3.3.egg/requests/models.py", line 528, in generate
    chunk = self.raw.read(chunk_size)
  File "/opt/python33/lib/python3.3/site-packages/requests-1.2.0-py3.3.egg/requests/packages/urllib3/response.py", line 148, in read
    data = self._fp.read(amt)
  File "/opt/python33/lib/python3.3/http/client.py", line 503, in read
    return super(HTTPResponse, self).read(amt)
  File "/opt/python33/lib/python3.3/http/client.py", line 542, in readinto
    n = self.fp.readinto(b)
  File "/opt/python33/lib/python3.3/socket.py", line 297, in readinto
    return self._sock.recv_into(b)
```

Debian 6.0, Python 3.2 or Python 3.3, requests 1.2.3 or last or 1.2.0
Under Python 3.3 and Windows 7 is no such problem


You should not be using `iter_content` like that. You should do `for i in r.iter_content():`


In particular, what you're doing here is creating a new generator each time you run around the loop. `r.iter_content()` returns a _new_ generator, on which you call `__next__`. In some cases this _might_ work, but that's simply good luck and not something on which you want to rely.


Ok

``` python
import requests

r = requests.get('http://servag.rupsy.ru/media/2/Amanojaku_and_ZY_-_Womtek_Julla-2013/Mensaje.jpg', stream=True)
cc = 1
for i in r.iter_content(1024*100):
    print('read ' + str(cc*1024*100))
    cc += 1
```

But this not resolve problem!


I'm installing a Debian VM right now, I'll see what I can do.


I'll test as well in 20


Ok, with a freshly installed Debian 7.1, Python 3.3 and Requests 1.2.3 I can't reproduce this.


Please, try with another url:

``` python
import requests

r = requests.get('http://servag.rupsy.ru/media/2/Manifest_-_Expedientes_X-2013/00-Manifest-X_Files-A-Front_Cover.jpg', stream=True)
cc = 1
for i in r.iter_content(1024*100):
    print('read ' + str(cc*1024*100))
    cc += 1
```


Again, the request completes successfully for me under Python 3.3 on Debian 7.1.


Thank you for all - problem not in requests!
Also, I found this problem when i get these files with wget.
Response hook's callback function returns a value to cause exception

The documentation for response hooks says "If the callback function returns a value, it is assumed that it is to replace the data that was passed in. If the function doesn’t return anything, nothing else is effected"

Now i am trying to return a value(int in my case) from my hook function and it throws an exception. This will be valid in all the cases when the return value is an object that DOESNOT have the raw() method defined for it.

Here is some code

```
def hook(resp,**kwargs):
    print resp.url
    return 1

def main()
    s = requests.Session()
    s.hooks = {"response":hook}
    r = s.get("http://localhost/index.html")
```

And here is the exception:

```
http://localhost/index.html
Traceback (most recent call last):
 File "/home/talha/ws/test.py", line 85, in <module>
   main()
 File "/home/talha/ws/test.py", line 72, in main
   r = s.get("http://localhost/index.html")
 File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 347, in get
   return self.request('GET', url, **kwargs)
 File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
   resp = self.send(prep, **send_kwargs)
 File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 446, in send
   extract_cookies_to_jar(self.cookies, request, r.raw)
 AttributeError: 'int' object has no attribute 'raw'
```

The code in sessions.py @line 446 is trying to extract cookies after the dispatch_hook..From source

```
    # Response manipulation hooks
    r = dispatch_hook('response', hooks, r, **kwargs)

    # Persist cookies
    extract_cookies_to_jar(self.cookies, request, r.raw)
```

Either the documentation needs to change or the handling needs to be re-worked. What is the best way to handle this ?


Sorry, the documentation could be more clear here. The value you return from the hook _must_ be a `Requests.Response` object or something that duck-types to it (e.g. a subclass).


You traced it back to the wrong line. The problematic line is probably inside `dispatch_hooks`. I'll look into this in an hour or so


And sorry for closing, hate this app


Nah, Ian, you just can't return an integer from a response hook. That would be crazy. =) An integer is not a response.


Agreed. The correct approach to be taken should be a subclass of `response`

```
class MyIntResponse(Response):
def __init__(self, integer):
    super(MyIntResponse, self).__init__()
    self._content_consumed = True
    self._content = integer

def hook(resp,**kwargs):
    print resp.url
    newresp = MyIntResponse(1)
    newresp.raw = resp.raw  # copy across original HTTP response object
```


I solved it after digging more into it..Will close. Thanks


Yeah the docs are bad and I feel bad
Cookies not passed on through redirect

I've been using requests 1.2.0 to log into a server, but when I upgraded to 1.2.3 today I found that a change in 1.2.1 breaks my code.

I log in with the line

```
response = requests.post(<endpoint>, data={'username':<username>, 'password':<password>})
```

The server responds with a 302 that sets the authentication cookies and redirects me to the welcome page. In 1.2.0, the authentication cookies can be found in the `response` variable's cookies, but not in 1.2.1.

I haven't found what change in the code is causing this different behavior, but I did notice that in the `response` variable I can access the history and get the cookies from the 302 response, so I'm guessing the issue is with the redirect.


Hi @yunruiwatts! Thanks for raising this issue!

The breaking change was 49a3664222d9e17ab83d089e4d1b1e30c998f59c. The reason for this change is that the old behaviour was incorrect. `Response.cookies` [is defined as](http://docs.python-requests.org/en/latest/api.html#requests.Response):

> A CookieJar of Cookies the server sent back.

The intention behind this documentation is that it's a CookieJar of Cookies the server sent back _on that response_. Those aren't the cookies you want, you want the ones the server sent on the _previous_ response.

There are two ways for you to get the cookies you want: either extract them from a `Session` or extract them from `r.history[-1]`. Either should work. If your use case is only that you're later going to make requests that you want to authenticate with that cookie, you should just be using a `Session` to send all your requests: that will handle cookie persistence for you.


Thanks for the informative response! I had not seen the `Session` objects, but that seems to make more sense.

I saw in the history that version 1.2.0 has "Fixed cookies on sessions and on requests", but 1.2.1 does not have anything related to cookies in its section of the history. It might be a good idea to put something in the documentation so it's clear that this is an intentional change of behavior.
tests fail on Python 2.7 OSX

The following tests fail:

```
#### ERROR: test_custom_content_type (__main__.RequestsTestCase)

    File "test_requests.py", line 397, in test_custom_content_type
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11167: ordinal not in range(128)

Fails due to UTF-8/Unicode characters in the test_requests.py document; if you replace `__file__` with `__file__.txt` that only contains ASCII data, it passes.


#### ERROR: test_different_encodings_dont_break_post (__main__.RequestsTestCase)

    File "test_requests.py", line 357, in test_different_encodings_dont_break_post
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11162: ordinal not in range(128)

Fails due to UTF-8/Unicode characters in the test_requests.py document; if you replace `__file__` with `__file__.txt` that only contains ASCII data, it passes.


#### ERROR: test_unicode_header_name (__main__.RequestsTestCase)

    File "test_requests.py", line 345, in test_unicode_header_name
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

No idea how to fix.

#### ERROR: test_unicode_multipart_post (__main__.RequestsTestCase)

    File "test_requests.py", line 363, in test_unicode_multipart_post
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 84: ordinal not in range(128)

No idea how to fix.

#### ERROR: test_mixed_case_scheme_acceptable (__main__.RequestsTestCase)

    Traceback (most recent call last):
    SSLError: [Errno 1] _ssl.c:499: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
```

I'd suggest a two part fix:

1- rewrite test to not verify ssl certificates:

```
>   def test_mixed_case_scheme_acceptable(self):
>+        verify = False
...
>-        r = s.send(r.prepare())
>+        r = s.send(r.prepare(), verify=verify)
```

2- write another test suite to ensure `verify` can be turned on and off, and that PEM can be sent [ ie, in line with http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification and http://stackoverflow.com/questions/10667960/python-requests-throwing-up-sslerror ]


Unfortunately, I can't reproduce this against HEAD:

```
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- pytest-2.3.4
collected 71 items 

test_requests.py .......................................................................

========================== 71 passed in 21.62 seconds ==========================
```


I'll try upgrading my Python ( it's 2.7 ) but also try a fresh install to see if this happens again.

This seems to work fine on a debian 2.6 and ubuntu 2.7.3 ;

My distro could have been built wrong.  


Confirmed that it passes on 2.7.5; sorry for sounding the alarm.  The 2.7 Python.org distro must have been built wrong.  


No worries. Thanks for being patient enough to file and then double-check.


Yeah, I want to echo that. =) If you think something has gone wrong, please tell us. It's so valuable. :cake:
feature request - cookiejar to accept and return actual cookies

I'm setting up regression testing on an application , which has me doing a lot cookie manipulation : pulling cookies out of a session, retrying a request, putting them back in, etc.

the current cookiejar has great behaviors which I wouldn't change – i'd just want to extend them a bit...

right now the cookiejar builds a new cookie via kwargs, and returns the values.  that loses a lot data that is really useful on testing.  i can still do what i need to do, it's just more code.

i'd like to propose these 2 additions :
- `cookiejar.cookie_add` - adds a Cookie object
- `cookiejar.cookie_get` - gets a Cookie object


Hi @jvanasco, thanks for opening this issue!

Have you tried `RequestsCookieJar.set_cookie()`? This should do your cookie adding. As for retrieving cookies, you should be able to iterate over the `RequestsCookieJar` to get the cookies out. Does that seem like what you need?


I missed `RequestsCookieJar.set_cookie` --  thanks!  ( i can't believe i missed it! I was even digging in the source files  )

I am iterating through `RequestsCookieJar` to find the cookies and grab them.  I was just thinking that this seemed like a common enough practice, and the addition of a `kwarg` to the `get` and `find` functions could easily return a cookie object (instead of the value).

Just to be clear - I have everything working for me right now.  I just think the extra code needed to grab the cookie out of a jar is a common enough situation that it would be better placed in `requests`.


I'm +0 on this. I can see the utility of having such a function, but it's not as though the building blocks for it aren't exposed. If it was implemented, I'd recommend using new functions (not adding a keyword argument). This is just because a function should really only return objects of one type (plus None).


Well if you change your mind, I tossed what I wrote into a changeset :  https://github.com/jvanasco/requests/commit/ef59f15dd4a0017f872536744ed1733e18da11ca

I also added two new test suites ( btw, the existing tests failed in 4 places )


If the existing tests are failing, please do file a bug report with details.


@sigmavirus24  done - https://github.com/kennethreitz/requests/issues/1443


At this stage, I'm fairly convinced we won't be implementing this. Thanks so much for the suggestion!
Remove urllib3-specific kwargs from non-urllib3 branch.

As pointed out by @lukesneeringer in #1395, the kwargs in the non-urllib3 branch of the `iter_content` code are urllib3 specific. This PR will finish the split I originally made in #1425.

/cc @sigmavirus24 and @lukesneeringer who have both done work that has been rendered unnecessary (though not useless) by this change.


My work can be considered unnecessary for newer versions of requests, but anyone intent on using an older version will not find it unnecessary ;)


Also :+1:
unquote double-quotes cookie values

Reading the spec, seems to me that quote-enclosed values should be returned unclosed. My interpretation is that the quotes are not part of the value, but they are considered 'punctuation'.

I'm not confident I'm reading the spec right, and I don't have a big knowledge of the many actual server implementations out there (for example: does anyone actually send cookies with `Version="1"`?), so feel free to prove me wrong.


to expand on this, a request to a url with `Set-Cookie: foo="bar:baz";` currently result in a cookie with value `"\"bar:baz\""`

with this patch, the value returned will be `bar:baz`.

Alternatively, we might want to have `"bar:baz"` as the returned value instead.


Hi @fcurella! Thanks so much for the pull request! This is good work. :cake:

The relevant section of RFC 6265 is this one:

```
 cookie-pair       = cookie-name "=" cookie-value
 cookie-name       = token
 cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
```

This suggests to me that the cookie value includes the double-quotes. Otherwise, the spec would read:

```
 cookie-pair       = cookie-name "=" cookie-value / ( DQUOTE cookie-value DQUOTE )
 cookie-name       = token
 cookie-value      = *cookie-octet
```

Of course, reading RFCs is a dark art, and so I could be wrong here. Nevertheless, I think we're safest if we assume that the quotes should be included. That way, if the remote end considers them semantically meaningful, it'll get them back; and if it doesn't, it should be stripping them (since it added them in the first place).

I do, however, agree that we shouldn't be _escaping_ them in the cookie values. =)


Ready to merge?


I think we're holding out for tests.


_tear_ look at us, being all responsible.


It's almost like this is a real open-source project! :car:


@Lukasa: I've added https://github.com/kennethreitz/requests/pull/1440/files#L2R173 already, but I can write more. What kind of tests would like fro me to add?


Ah, sorry, I totally missed that.

@kennethreitz this is good-to-go. =)


:sparkles: :cake: :sparkles:
Make sure netrc doesn't override any authentication settings explicitly set by the client

This fixes issue #1438.

.netrc is still going to be used by default if it's available (that's the intended behavior based on #446), but now any explicit settings (whether they're set on the session or passed to request()) will take precedence.


This looks excellent @voberoi, thank you! :cake:


No, thank _you_ for all the time and thoughtful effort going into requests.
netrc auth settings override explicit session settings

Netrc authentication will blow away any explicit authentication settings you've set on a session, and it causes unexpected behavior that is annoying to debug:
1. "Why are these credentials not working?" 
2. Print out what requests is sending over the wire.
3. Decode the basic authentication header, realize it's wrong
4. Poke through library to figure out where these credentials that aren't anywhere in your code are coming from.
5. Realize it's the way netrc is handled, modify the code to pass in authentication info in request(...), because that's the only way it'll work.

``` python
session = requests.session()
session.auth = ("username", "password")

# This will not use "username", "password", but whatever's in .netrc
r = session.get("http://www.hostinnetrc.com")

# This works
r = session.get("http://www.hostinnetrc.com", auth=("username", "password"))
```

Seems like both of those examples above should use "username", "password".


I'm happy to patch this and send you a pull request if you agree, just drop a note here.


Yeah, I agree that this is wrong. Feel free to submit a Pull Request. =)


This is absolutely a bug. Sorry about that :(


No problem!
stream_decode_response_unicode fails in Python 3 at rv = decoder.decode('', final=True)

```
 rv = decoder.decode('', final=True) should be
 rv = decoder.decode(b'', final=True)
```

otherwise, this fails:

```
import codecs
decoder = codecs.getincrementaldecoder("utf-8")(errors='replace')
rv = decoder.decode('', final=True)
```

Traceback:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-86c335a3edfc> in <module>()
----> 1 rv = decoder.decode('', final=True)

/Users/chen/Virtualenvs/python3Env/lib/python3.3/codecs.py in decode(self, input, final)
    297     def decode(self, input, final=False):
    298         # decode input (taking the buffer into account)
    --> 299         data = self.buffer + input
    300         (result, consumed) = self._buffer_decode(data, self.errors, final)
    301         # keep undecoded input until the next call

TypeError: can't concat bytes to str
```

Original Code attached:

```
def stream_decode_response_unicode(iterator, r):
    """Stream decodes a iterator."""

    if r.encoding is None:
        for item in iterator:
            yield item
        return

    decoder = codecs.getincrementaldecoder(r.encoding)(errors='replace')
    for chunk in iterator:
        rv = decoder.decode(chunk)
        if rv:
            yield rv
    rv = decoder.decode('', final=True)
    if rv:
        yield rv
```


Hi @chinux23, thanks for raising this issue! I agree, that looks fairly clearly wrong to me. I don't have time to fix it right now, so you should feel free to open a Pull Request if you want to fix it yourself. Otherwise, I'm marking this as Contributor Friendly until I can get around to it. =)


I can make a fix for this. I'll submit a pull request once I am done. Thanks.


Fixed by #1435. Thanks so much!
Issue trying to specify http_pool_maxsize

When trying to specifiy other values for http_pool_maxsize

self.session.mount('http://', requests.adapters.HTTPAdapter(pool_connections=self.pool_connections,
                                                                    pool_maxsize=self.pool_maxsize,
                                                                    max_retries=self.max_retries,
                                                                    pool_block=self.is_pool_block))
Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
..........
    return self.session.put(self.swiftAuthPath + SWAUTH_API_VERSION_2 + "/" + account,  headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 387, in put
    return self.request('PUT', url, data=data, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 272, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 199, in get_connection
    conn = self.poolmanager.connection_from_url(url)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/poolmanager.py", line 123, in connection_from_url
    return self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/poolmanager.py", line 109, in connection_from_host
    pool = self._new_pool(scheme, host, port)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/poolmanager.py", line 79, in _new_pool
    return pool_cls(host, port, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 189, in **init**
    for _ in xrange(maxsize):
TypeError: an integer is required

Problem is that it tries to create a xrange when doing **init** in connectionpool.py and supplies a string and I get this above exception

Am I doing something wrong? This only seems to work with the default

Thanks

Dave


Hi there, thanks for raising this issue!

This works fine for me if I use an integer as the `pool_maxsize` argument. Out of interest, what is `self.pool_maxsize` in your example?


Hey thanks Lukasa,

Sorry - I have just converted the string to an int and its working. 

Dave
Https and proxies problem (Fiddler2)

I am trying to redirect my python requests through Fiddler2 so I can monitor what is going on.  As far as I know I have set up Fiddler correctly because I can see my http requests going through.  However, my https requests are getting this error:

An exception of type SSLError occured, these were the arguments:
(SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:547)'),),)

My code looks like this:

```
proxies = {
    "http":"http://127.0.0.1:8888",
    "https":"https://127.0.0.1:8888",
}
url1='https://membership.usairways.com/Login.aspxReturnUrl=http:%2f%2fwww.usairways.com%2fdefault.aspx'
s = requests.Session()
s.mount('https://', ssladapter.SSLAdapter(ssl_version = PROTOCOL_SSLv3))
r1 = s.get(url1, proxies = proxies)                                      #USair Home Page
```

I have tried other https web addresses and I get the same error.
The code works fine if I remove the proxies argument(but is fine for http).   Any ideas?


Are you trying to do transparent HTTPS proxying (e.g. just forward the encrypted traffic through) or are you trying to use Fiddler2 as a MITM proxy?


I am trying to debug some website scraping code and wanted to see what was going on with the headers, post data and cookies that I am sending using Requests.  I wanted to make sure that I was using Requests properly.  I use Fiddler2 and Tamper Data to watch the data being sent from the browser to the target server.  I matched the data being sent using Requests but I am still getting an 400 error.  

Now, here is the kicker.  When I don't include all the header data (took some out. found by trial and error) I get a correct 200 response and the data I need.  I just wanted to know why and wanted to investigate what Requests was actually sending (outside of IDE).

Since I'm new to all of this and judging from your comment I'm guessing this can't be done because of the ssl encryption hence not a problem with Requests at all.  Correct?  :)  If there is a way I surely would like to know ;)


You have two options here:
1. Requests keeps all that data lying around. You can see what was sent on a request by examining `Response.request`, e.g. `Response.request.headers`. This is a `PreparedRequest` object, so has all the data of that object. The history is also available, if you saw any redirects: `Response.history`.
2. Use a Man-in-the-Middle proxy. You _can't_ use a transparent proxy here, partly because Requests doesn't support them (yet) and partly because all the traffic would be encrypted and you wouldn't see a thing. I don't know much about Fiddler2, but [mitmproxy](http://mitmproxy.org/) is totally awesome and does MITM stuff. =)


Thanks for all your help.  I will check out mitmproxy. :)
url encoding issue

The Hacker News API requires a '+' in the URL for certain kinds of searches.

If I put one in a parameter, it is escaped by requests and doesn't work.
A workaround is to put the parameter in the URL string.


Hi @Chris2048! Does the Hacker News API not treat the encoded version of the `+` character the same as an unencoded version?


No, right now (for me) "http://api.thriftdb.com/api.hnsearch.com/items/_search?filter[fields][create_ts]=[2013-06-23T00:00:00Z%20+%20TO%20+%20*]" returns 3451 hits, but "http://api.thriftdb.com/api.hnsearch.com/items/_search?filter[fields][create_ts]=[2013-06-23T00:00:00Z%20%2B%20TO%20%2B%20*]" returns 'null' hits...


I'm pretty sure that the API is doing the wrong thing here.

The '+' character has a syntactic meaning in the query string. Spaces may be encoded either to the literal `+` or to `%20`. For this reason, a non-space `+` character _must_ be encoded to `%2B`. Given that the HN API is clearly using the `%20` representation of a space, I think someone needs to tell them that they can't use the `+` like that.

Unfortunately, this makes it a lot harder for us to help you on this problem. You do have some options. One would be to break open the flow, building `PreparedRequests` yourself and adjusting their URLs. You could potentially use @sigmavirus24's [uritemplate](https://github.com/sigmavirus24/uritemplate) library to help with that.


Just tried "http://api.thriftdb.com/api.hnsearch.com/items/_search?filter[fields][create_ts]=[2013-06-23T00:00:00Z%20TO%20*]" and it works. Wonder why the tutorials indicate the use of a '+'...


Hmm, that is odd. Well, I'm glad you got your code working! Let me know if you have any further problems. =)
Note regarding embedding requests and cacerts.pem

Adding a note about modifying setup.py for a package that embeds requests so that SSL will still work.  Hopefully this avoid several hours of pain for someone else in the future.


HI @andrewgross, thanks for this Pull Request! I like the idea, I'm just wondering if there's a better place in the docs for this. I'm going to have a think about it, but I didn't want you to think no-one had seen your PR, so I thought I'd let you know. =)


Fair enough, just hoping to get the information visible.  Not particular about the wording or the placement.
Adding Cookie to Cookiejar problems.

I am trying to add a cookie to an existing cookiejar using the python requests 1.2.3 library.  Every time I add the new cookie, the data in the jar is munged for the new cookie.  Keys missing, Values missing or matched to incorrect Keys.  I'm not sure if it's a Request library bug or I'm not sending the cookie correctly.  I'm using the following code that is resulting in a bad cookie in cookiejar.  Am I formatting the cookie correctly? Any ideas?

```
    my_cookie = {
           'domain':'www.mydomain.com',
           'expires':None,
           'name':'COOKIE_NAME',
           'path':'/',
           'value':'the cookie works',
           'version':0
}

s = requests.Session()
requests.utils.add_dict_to_cookiejar(s.cookies, my_cookie)
```


Sorry @mrfatboy for not spotting this last night, but `add_dict_to_cookiejar()` doesn't allow for cookies specified in that manner. `add_dict_to_cookiejar()` creates a set of cookies whose names are equal to the keys of your dict, and whose values are their values. You can't set their properties using that method.

Instead, try the following:

``` python
my_cookie = {
    'domain':'www.mydomain.com',
    'expires':None,
    'name':'COOKIE_NAME',
    'path':'/',
    'value':'the cookie works',
    'version':0
}
s = requests.Session()
s.cookies.set(**my_cookie)
```


Thanks @Lukasa.  I found out a way to do by importing CookieJar, Cookie, and cookies.  I like your way better :)  However, with your way I was not able to specify the "port_specified", "domain_specified", "domain_initial_dot" or "path_specified" attributes.  The "set" method does it automatically with default values.   I'm trying to scrape a website and their cookie has different values in those attributes. As I am new to all of this I'm not sure if that really matters yet.   I was able to make this work with your help. :)

   my_cookie = {
        "version":0,
        "name":'COOKIE_NAME',
        "value":'true',
        "port":None,
//       "port_specified":False,
        "domain":'www.mydomain.com',
//       "domain_specified":False,
//        "domain_initial_dot":False,
        "path":'/',
//        "path_specified":True,
        "secure":False,
        "expires":None,
        "discard":True,
        "comment":None,
        "comment_url":None,
        "rest":{},  
        "rfc2109":False
    }

s = requests.Session()
s.cookies.set(**my_cookie)

note: the '//' were replacements for the '#'  (comment out) because they were causing funky formatting in this post.


Awesome. Glad to know you got to somewhere useful. =) Let me know if you have any further problems!
latest requests version breaks with ipv6 URL

```
>>>  requests.get('http://[2002:1234::213:1002:37:ed36]/xxxy')
[...]
LocationParseError: Failed to parse: Failed to parse: 2001:1234::213:5002:37:ed36
```

This works fine with requests 0.14.2, looks like a urllib3 regression (I can't find their closed bug report about it anymore..) and I don't know much about that library to open a bug report there.


Problem is with `requests.models.PreparedRequest.prepare_url()` where the url is reconstructed and the braces are lost.
While doing the request `requests.packages.urllib3.util.parse_url` is called _two_ times. The first time (called from `prepare_url` itself) everything is fine. The second time it blows up. The question here is, if a) `prepare_url` has to add back the braces which it doesn't get from `parse_url` b) `parse_url` should return the braces as part of the host.


`urlparse` from the stdlib keeps the braces.

cc @shazow 


I'm +1 for keeping the braces. Anyone want to do a urllib3 PR with a test + fix?


This should be fixed with the next Requests release.
Use new urllib3 'stream' parameter.

In shazow/urllib3#186 I proposed to add a `.stream()` generator to the urllib3 response object. This avoids some of the nasty problems with the `.read()` method on that object (apparent zero-sized reads when there's still data, all sorts of stuff). It's also a much more natural way to stream data.

This PR would update urllib3 to include that PR, and updates our streaming methods to use it where possible.

@michaelhelmick: Do you want to try this branch with your twitter issue? Use one-byte chunk sizes.


I'll give it a shot in a half hour! Thanks!

Sent from my iPhone

> On Jun 18, 2013, at 1:26 PM, Cory Benfield notifications@github.com wrote:
> 
> In shazow/urllib3#186 I proposed to add a .stream() generator to the urllib3 response object. This avoids some of the nasty problems with the .read() method on that object (apparent zero-sized reads when there's still data, all sorts of stuff). It's also a much more natural way to stream data.
> 
> This PR would update urllib3 to include that PR, and updates our streaming methods to use it where possible.
> 
> @michaelhelmick: Do you want to try this branch with your twitter issue? Use one-byte chunk sizes.
> 
> You can merge this Pull Request by running
> 
>   git pull https://github.com/Lukasa/requests stream
> Or view, comment on, or merge it at:
> 
>   https://github.com/kennethreitz/requests/pull/1425
> 
> Commit Summary
> 
> Update urllib3 to cffbd6b317
> Use the new urllib3 stream generator.
> File Changes
> 
> M requests/models.py (17)
> M requests/packages/urllib3/response.py (24)
> M requests/packages/urllib3/util.py (15)
> Patch Links:
> 
> https://github.com/kennethreitz/requests/pull/1425.patch
> https://github.com/kennethreitz/requests/pull/1425.diff


:+1: 


:cake: :cake: :cake: :cake: :cake: :cake: :cake: :cake:
event hooks require kwargs now?

The balanced-python library uses a response event hook. [The hook](https://github.com/balanced/balanced-python/blob/82670e5c5f6052c19cd062df11095c576dfc9b14/balanced/http_client.py#L63) does not take kwargs, and under requests 1.1.0 this works fine, but under 1.2.3 this results in TypeError, `wrapper() got an unexpected keyword argument 'timeout'`.

This change in behavior doesn't appear to be reflected in the [event hook documentation](http://docs.python-requests.org/en/latest/user/advanced/#event-hooks).

Should hooks not require kwargs, or should the docs be updated?


Yeah, I need to update the docs. Was hoping to do it last weekend but I have been far too busy. I thought I had ticketed it, but perhaps not.


Cool, no worries. :)

!m @sigmavirus24 


I just ran into this problem where the hook needs to take kwargs.  Thanks.
need to send request in multipart/form-data without files and specify mine boundary

In developing a program using Phase II CORE protocol.  I need to send HTML request in 
multipart/form-data but without file object. e.g. the file data should be sent without 
a filename. The boundary must be 'XbCY':
   --XbCY 
   Content-Disposition: form-data; name=“CORERuleVersion" 2.2.0 
   --XbCY 
   Content-Disposition: form-data; name=“Payload" 
   <contents of file go here -- 1674 bytes long as specified above> 
   --XbCY--

I patched models.py for my project like this(help it can fixed in Requests)

```
87c87
<     def _encode_files(files, data, boundary=None):

---
>     def _encode_files(files, data):
95c95
<         if (not files and boundary is None) or isinstance(data, str):

---
>         if (not files) or isinstance(data, str):
131,133c131
<         if not boundary:
<             boundary = None
<         body, content_type = encode_multipart_formdata(new_fields, boundary)

---
>         body, content_type = encode_multipart_formdata(new_fields)
174,177d171
<     # patched by IIC 
<     :param boundary:
<                      used for mime type encoding. it sever a flag to forced mime encoding even without files.  
<                      set to '' to force mime encoding and use auto-generated boundary 
195,198c189
<         hooks=None,
<         
<         # patched by IIC
<         boundary=None):

---
>         hooks=None):
221,223d211
<         
<         # patched by IIC
<         self.boundary = boundary
236c224
<         p.prepare_body(self.data, self.files, self.boundary)

---
>         p.prepare_body(self.data, self.files)
342c330
<     def prepare_body(self, data, files, boundary=None):

---
>     def prepare_body(self, data, files):
380,381c368,369
<             if files or boundary is not None:
<                 (body, content_type) = self._encode_files(files, data, boundary)

---
>             if files:
>                 (body, content_type) = self._encode_files(files, data)
```


Hi @gangli, thanks for opening this issue and thanks for providing a patch!

This is a fairly specialised use-case. Requests primarily attempts to solve the 90% use-case, which this does not come under. Additionally, we are currently in feature-freeze, which means we aren't accepting new features at this time.

Unfortunately, this means it's unlikely we are going to implement these changes in the main library itself.

I'm sorry that I can't be of more help. Please keep contributing to the library, and let me know if you need anything else! Thanks again. =) :cake:
Connection timeout doesn't work for chunked-encoded request 

Sample code:

```
def gen():
    for i in xrange(1, 1000):
        yield 'x'
import requests, datetime

print datetime.datetime.now()

try: requests.post(url='http://1.1.1.1', timeout=5, data=gen())
except Exception as e: print e

print datetime.datetime.now()

try: requests.post(url='http://1.1.1.1', timeout=5, data='a')
except Exception as e: print e

print datetime.datetime.now()
```

gives the following output

```
2013-06-14 00:33:03.079389
[Errno 110] Connection timed out
2013-06-14 00:34:06.236742
HTTPConnectionPool(host='1.1.1.1', port=80): Request timed out. (timeout=5)
2013-06-14 00:34:11.242869
```

The first request blocked for more than a minute, before a timeout has been thrown, while the second one is ok.

Requests 1.2.3
Python 2.7.3 


Thanks for opening this issue!

Looking at the Requests code, I suspect you might be right. I'm at work at the moment and won't have internet at home until the 18th, but when I can get time I'll investigate this.

To begin with, I think the problem is around [here](https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L300). For chunked uploads, we do this:

``` python
low_conn = conn._get_conn(timeout=timeout)
low_conn.putrequest(request.method, url, skip_accept_encoding=True)

for header, value in request.headers.items():
    low_conn.putheader(header, value)

low_conn.endheaders()

for i in request.body:
    low_conn.send(hex(len(i))[2:].encode('utf-8'))
    low_conn.send(b'\r\n')
    low_conn.send(i)
    low_conn.send(b'\r\n')

low_conn.send(b'0\r\n\r\n')

r = low_conn.getresponse()
resp = HTTPResponse.from_httplib(r,
    pool=conn,
    connection=low_conn,
    preload_content=False,
    decode_content=False
 )
```

The only place we set the timeout is in the call to `urllib3.HTTPConnectionPool._get_conn()`, which passes down into its `LifoQueue`. This timeout affects how long we wait to obtain a connection object, but does not set the timeout on that connection.

I need to think about what's best to do here, but I think the answer is to set `low_conn.timeout = timeout` before the initial call to `putrequest()`. Feel free to try that out if I don't get around to testing it first.


I wonder if our new Timeout system will fix this.


Seems this is still happening with 2.18.4. I was able to fix it using `low_conn.timeout = timeout.connect_timeout`, like @Lukasa mentioned. Should I write a test and submit a pull request or just create a new issue?

Raise a new issue, link this one, and include your test case that can reproduce the problem consistently. Then you can submit a PR and link to your new issue if you have time :) Thanks @isobit!