Raibek/requests-github-issues huggingface dataset:
do not update the request.url from the OAuth signing call as it results ...

...in the request URL params all being doubled (and the URL doesn't need to be updated based on the signing anyway)

This was noticed when attempting a file upload OAuth1a request to PyPI - the params were all doubled-up when performing POST with URL params (the only way to do OAuth1a file upload AFAIK.)

I don't believe the OAuth signing should ever modify they URL so I believe this modification is safe.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1957207) (merged 6f0a2242 into 52b55ccf).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1965210) (merged 0b8b859f into 52b55ccf).


... 80 changed files?


Closing this, @r1chardj0n3s thanks for the patch! Can you send again with only 6f0a224? 


I have absolutely no idea what the hell happened there. Will do.


I believe this was fixed in #722 actually.


Yes, that appears to fix the problem.


Awesome, will be released shortly.
Make OAuth path hack platform independent.

This _should_ resolve issue #735. I'd love it if someone could run a test on a Windows box, as I don't have easy access to one at the minute.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1922177) (merged 79aa9edd into 876e1744).


Thought I'd fix up some typos while I had a pull request open. Can't hurt.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1950590) (merged 55237ad6 into 876e1744).


Thank you sir :)
NameError: global name 'Client' is not defined

I ran into trouble running [Twython](https://github.com/ryanmcgrath/twython). It seems to be a problem with `requests`, since my authentication keys are correct (and the script runs on another computer).

```
Traceback (most recent call last):
  File "C:\Users\Yi-Yi\Documents\Yi-Yi\Projects\Programs\Twitter Art Bot\foo.py", line 11, in <module>
    oauth_token_secret=ACCESS_TOKEN_SECRET)
  File "C:\Python27\lib\site-packages\twython-2.3.2-py2.7.egg\twython\twython.py", line 131, in __init__
    signature_type='auth_header')
  File "C:\Python27\lib\site-packages\requests-0.13.3-py2.7.egg\requests\auth.py", line 57, in __init__
    self.client = Client(client_key, client_secret, resource_owner_key,
NameError: global name 'Client' is not defined
```

The code the error is from:

```
from twython import Twython

CONSUMER_KEY = '...'
CONSUMER_SECRET = '...'
ACCESS_TOKEN = '...'
ACCESS_TOKEN_SECRET = '...'

t = Twython(app_key=CONSUMER_KEY,
            app_secret=CONSUMER_SECRET,
            oauth_token=ACCESS_TOKEN,
            oauth_token_secret=ACCESS_TOKEN_SECRET)

print t.getHomeTimeline()
```

I'm running `requests.__version__` 0.13.3. I'm running this on Windows Vista 64-bit, Python 2.7.3.


This is the result of a bug in the OAuth path hack, I think. Specifically, [this path hack](https://github.com/kennethreitz/requests/blob/develop/requests/_oauth.py#L19) works great on *nix where the path separator is /, but does not work on Windows where it is .

If you want a quick fix, you can install OAuthLib from PyPI. I recommend using `pip install oauthlib` but [here is a direct link if you prefer](http://pypi.python.org/pypi/oauthlib/).

EDIT: I have opened pull request #736 to deal with this issue.


Installing `oauthlib` fixed it for me. Thank you for addressing the issue!
Include response.content in raised HTTPErrors


I don't think this is a good idea. I wouldn't be against adding `response.reason` though.


So, I had a niggling feeling that it might not be, but no idea why; I'd be curious to hear your reasoning...


Requests doesn't read the response bodies until the user access them.

Also, many responses are binary and therefore can't be printed to the screen without encoding errors.


Keep the pull requests coming though ;)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1915888) (merged 6eb1ac44 into 876e1744).


Ah, hadn't realised `reason` had been added. 

I'd erroneously thought [RFC2616](http://www.ietf.org/rfc/rfc2616.txt) had some plain-text-y limitations on `message-body` format in the event of an error, but I can't seem to find anything saying that, so presumably imagined it, apologies.

Thanks for the feedback


No need to apologize, it was a great idea ;)


#734 adds `response.reason` as you mention above then :)
breaks under python 2.7, without the from __future__ import 

If python support < 2.6 is required, will need to make this import conditional


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1893840) (merged 2c241d28 into 73815e2e).


No need to make the import conditional: [Requests only supports 2.6 and higher](http://docs.python-requests.org/en/latest/community/faq/#python-3-support).


Ah, thanks. Do I need to do anything else? Or will this minor change be incorporated into trunk at some point? (sorry, just starting out on github).


I checked it out on my machine and it looks good, to me, so you don't need to do anything else. @kennethreitz will pop by at some point when he has time and will handle merging into the development branch, but it'll merge nicely so it shouldn't be a big deal. If you want to, you could add your name to the Authors file, but that's really up to you. As for trunk, it should make it into the next minor version of requests.


Sounds good - I will now remove my attention from this. Thanks for info.


Thanks guys!

And thanks for the rundown @Lukasa :)
HTTPS failing with protocol error / timeout

Connecting to some HTTP sites is failing. It seems to be sites that use TLS and OpenSSL isn't selecting the right SSL protocol to communicate with the server. This seems to be an issue on Ubuntu 12.04 so far..

There is more discussion here: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/965371

Example to test is below:

```
>>> import requests
>>> requests.get("https://www.acgov.org/alco_ssl_app/rov/voter_info/voter_profile.jsp?formLanguage=E")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/api.py", line 54, in get
    return request('get', url, **kwargs)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/safe_mode.py", line 37, in wrapped
    return function(method, url, **kwargs)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/api.py", line 42, in request
    return s.request(method=method, url=url, **kwargs)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/sessions.py", line 230, in request
    r.send(prefetch=prefetch)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/models.py", line 613, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol
>>> requests.get("https://www.acgov.org/alco_ssl_app/rov/voter_info/voter_profile.jsp?formLanguage=E", verify=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/api.py", line 54, in get
    return request('get', url, **kwargs)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/safe_mode.py", line 37, in wrapped
    return function(method, url, **kwargs)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/api.py", line 42, in request
    return s.request(method=method, url=url, **kwargs)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/sessions.py", line 230, in request
    r.send(prefetch=prefetch)
  File "/home/dpn/.virtualenvs/scrapers/local/lib/python2.7/site-packages/requests/models.py", line 615, in send
    raise Timeout('Request timed out.')
requests.exceptions.Timeout: Request timed out.
>>> 
```

It seems that manually specifying the SSL version to use may be the best option until the problem is fixed.

It seems the issue of selecting SSL version has come up here before: https://github.com/kennethreitz/requests/issues/606
Added needed cast for when POSTing data and files (Issue #661)

Co-Authored By: Timnit Gebru tgebru@gmail.com @tgebru
Co-Authored By: Sarah Gonzalez smar.gonz@gmail.com @smargonz
Co-Authored By: Leila Muhtasib muhtasib@gmail.com @muhtasib

This is a fix for Issue #661.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1867038) (merged f9c0ddf4 into a97a5133).


Thanks!
Document Logging, issue #695

This is a fix for issue #695. 

Additional co-authors:
@smargonz
@vickimo
@tgebru


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1866717) (merged 9135530d into a97a5133).


@smargonz and I noticed this hasn't been implemented yet. Would you like us to implement it?  Let us know if you have any design considerations you'd like us to follow.


Are you thinking of the urllib3 logging?


Closing until further update :)
Fix infinite loop on wrong Digest Authentication (Issue #541 and #547)

Modified code to use the current fix versus the old fix, which was broken.

Co-Authored By: Timnit Gebru tgebru@gmail.com @tgebru
Co-Authored By: Sarah Gonzalez smar.gonz@gmail.com @smargonz
Co-Authored By: Leila Muhtasib muhtasib@gmail.com @muhtasib

This is a fix for issue #541 and #547.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1866392) (merged 5296b8be into a97a5133).


looks good to me, thanks!


Great! Thanks for checking :).

Vicki

On Sat, Jul 14, 2012 at 6:48 PM, Kenneth Reitz <
reply@reply.github.com

> wrote:
> 
> looks good to me, thanks!
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/727#issuecomment-6987314
how to set max_retries

I don't know how to set max_retries.

In a low speed, how to optimize the configuration to Improve the connection success rate?

thank you.

---------I LOVE REQUESTS--------


Check out this gist. Those are 2 ways that you can achieve it.

https://gist.github.com/3116429

I also believe that you can import the requests defaults and set those as well, haven't tried that before though


This gist does not work for me on requests v2.21:

```
a = requests.get('http://gidjisao.com',config={"max_retries":10},timeout=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pbaxi/.virtualenvs/qualysapi/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Users/pbaxi/.virtualenvs/qualysapi/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
TypeError: request() got an unexpected keyword argument 'config'
```


@paragbaxi this issue was last commented on over 2 years ago. The proper place for questions about requests is [StackOverflow](http://stackoverflow.com/questions/tagged/python-requests) which you've already done. I will answer your question there.
Differentiate between two types of timeout errors from urllib3

This is a fix for issue #699

A simple solution was to bubble up the error from urllib3. TimeoutError inherits from _HTTPError, so we could just add a line and raise a Timeout.

The error message will indicate whether the error is of type socket timeout or of type no connections available in the connection pool.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1859013) (merged 318f2460 into c3ab38ed).


\o/


:sparkles: :cake: :sparkles:
Differentiate between two types of timeout errors from urllib3.

This is a fix for issue #699

A simple solution was to bubble up the error from urllib3. TimeoutError inherits from _HTTPError, so we could just add a line and raise a Timeout.

The error message will indicate whether the error is of type socket timeout or of type no connections available in the connection pool.


Hmm, would you mind merging with the latest 'release' branch? This is a bit out of date, so has some merge conflicts :)

It looks great though! Excellent contribution. :)


Yes, I'll send a more updated pull request in a few.


thanks for the fix for this issue @muhtasib and @kennethreitz  .
Plesse Support Socks

I am very love requests. I hope it can support socks(socks4/socks5).I know can use socksipy,But inefficient.I hope you can improve socksipy and added to the requests.thank you.
sorry, I am not good at english.


No worries :) 

Work is being done on socks, actually! You can track the development here: https://github.com/shazow/urllib3/pull/68
adding clarification to the history API.

The history list after a few redirects is sorted with the oldest first and the most recent one being the last one of the list. Added a simple line to explain it. It was not obvious at first when you do something like. 

```
for resp in history:
    print resp 
```


Beautiful.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1849932) (merged 9fe4a993 into 884fc333).
Workaround for an ImportError on AppEngine

On Google App Engine (runtime 2.7 thread safe) there is a ImportError
when ever calling get_netrc_auth (stacktrace at the end of the comment)
I'm adding a except pass on the Import error to workaround that error. I
will open an issues on the App Engine tracker so that will fix that on
thier end also.

File "/base/data/home/apps/.../requests/api.py", line 87, in post
return request('post', url, data=data, *_kwargs)
File "/base/data/home/apps/.../requests/safe_mode.py", line 37, in
wrapped
return function(method, url, *_kwargs)
File "/base/data/home/apps/.../requests/api.py", line 42, in request
return s.request(method=method, url=url, **kwargs)
File "/base/data/home/apps/.../requests/sessions.py", line 230, in
request
r.send(prefetch=prefetch)
File "/base/data/home/apps/.../requests/models.py", line 483, in send
self.auth = get_netrc_auth(url)
File "/base/data/home/apps/.../requests/utils.py", line 79, in
get_netrc_auth
for loc in locations:
File "/base/data/home/apps/.../requests/utils.py", line 76, in <genexpr>
locations = (os.path.expanduser('~/{0}'.format(f)) for f in NETRC_FILES)
File "/base/python27_runtime/python27_dist/lib/python2.7/posixpath.py",
line 259, in expanduser
import pwd
ImportError: No module named pwd


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1849548) (merged f823a2b2 into b222ff94).
no option to turn off retry

it is good that requests could handle retrying upon general connection errors. but in a web scraper project, it'll be better for the outside scheduler to handle these kind of exceptions and make corresponding logging procedures, don't you think? 


Requests doesn't do retries :)


@kennethreitz Then what does the max_retries do in requests.defaults?

http://docs.python-requests.org/en/latest/api/#configurations


It's set to 0 :)


Ahh, it was just the way you stated that **requests doesn't do retries**, I thought you meant it collectively, rather than just as a default.


Sorry for the confusion :)


Documentation should be improved here.


the confusion starts with a testdrive. I inserted a invalid Host string in the header, and launch a request, it got immediately bounced back with the exception:

requests.exceptions.ConnectionError: MaxRetryError("HTTPConnectionPool(host='202.104.xx.xxx', port=80): Max retries exceeded with url:
401 digest fail causes infinite loop

I'm using 0.13.2
In models.py, line 491
# Allow auth to make its changes.

r = self.auth(self)

In https://github.com/kennethreitz/requests/pull/547
it will deregister the hook, after the first request failed, but in the second request it will register a new hook 'response', which causes infinite loop.

Is this the right way to fix it?
if not anyway:
                r = self.auth(self)


Fixed!


Do yo mean it's already a fixed issue when I open this one?
Timeouts not working when importing gevent

I'm going to open this here, as I believe the issue/bug is in the requests library, rather than the gevent library, as other http libs work with gevent. If you don't agree with this, feel free to close.

After importing gevent and monkeypatching the socket class, Requests simply ignores the timeout parameter. This wouldn't be a huge problem on its own, the problem is that it then defaults to **never** timing out, bringing the whole script to a halt if you try requesting a page that never responds, or takes a extremely long time. I don't have time right now to dig through the requests code (or likely enough Python knowledge), but the below example demonstrates the error:

Example 1:

``` python
>>> import requests
>>> from gevent import monkey
>>> monkey.patch_all(thread=False, select=False)

# This site takes a ridiculously long time to respond, so brings the whole script to a stop.
>>> requests.get('http://www.whatzonip.com/wiki9/index.php', timeout=5)
```

Example 2:

``` python
>>> import requests
>>> from gevent import monkey
>>> monkey.patch_all(thread=False, select=False)

# This shouldn't return a response, as should raise a TimeoutError first.
# But ignores the timeout, and returns the Response object after 10 seconds.
>>> requests.get('http://httpbin.org/delay/10', timeout=5)
<Response [200]>
```


Hi,

I'm looking into this for you. I'll let you know what I find by this evening. 


It might be gevent related after all. See this: https://github.com/shazow/urllib3/issues/69


Seems it was gevent related. The fix isn't in any releases yet though, so have to install from the git repo.

Thanks for your help.
Do not hide SSLErrors behind Timeouts.

Following discussion on Issue #701, it seems like we should rethrow SSLErrors regardless of whether `verify` is true or not. SSLErrors are not raised by invalid certificates with `verify = False` anyway, so anything caused by anything else should be rethrown.

I can't think of a good test for this, so if you have an idea I'd love to hear it.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1837636) (merged 523906c5 into c19ff038).


I agree, thanks for the contribution!
fixed param handling in Request.full_url

I noticed that the params were not correctly handled in the presence of a fragment.

``` python
requests.get('http://example.com/path?key=value#fragment',
    params={'a': 'b'}, return_response=False).full_url
```

Is my fix ok?


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1821406) (merged 08cebb3c into b7fd0e81).


this looks great, thanks!
Add try/catch for simplejson vs json

Just a minor change that opts to use the `simplejson` library if it's available rather than the standard library's `json` package. Nothing too fancy — could provide some minor speed-ups from the `simplejson` C extension.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1806954) (merged 25299400 into 808b0ad8).


thanks!
Have User Agent include more details

With this change the default user agent will include more details by default.

Python:

```
python -c "import requests; print requests.get('http://httpbin.org/user-agent').json"
{u'user-agent': u'python-requests/0.13.2 CPython/2.7.3 Darwin/11.4.0'}
```

PyPy:

```
pypy -c "import requests; print requests.get('http://httpbin.org/user-agent').json"  
{u'user-agent': u'python-requests/0.13.2 PyPy/1.9.0 Darwin/11.4.0'}
```


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1805051) (merged 11433d94 into 808b0ad8).


Awesome! i might swap this out to use its.py


its.py?


actually, most of it exists in requests.compat

## 

Kenneth Reitz

On Sunday, July 8, 2012 at 12:18 AM, Donald Stufft wrote:

> its.py?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/709#issuecomment-6828782


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1807040) (merged 523433f2 into 808b0ad8).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1807082) (merged 21f60fe8 into 808b0ad8).


\o/


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1807579) (merged 21f60fe8 into 808b0ad8).


Finally tried this. Absolutely incredible :)
Adding nice rounded corners for the pre element of Sphinx theme

Just adding some nice rounded corners for the **pre** element of Sphinx theme, I don't know if you want to commit it, it looks like in this project: http://odbrasil.readthedocs.org


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1753881) (merged c19a83d1 into d983cfab).


Thanks for the contribution. I've actually played around with adding rounded corners to the theme a few times, and decided against it. 

Thanks though :)
Duplicate cookie in session

I am using `requests` 0.13.1, this code produces the specified exception:

```
  >>> import requests
  >>> s = requests.session()
  >>> s.get("http:///www.google.com")
  >>> s.cookies.items()
  [('NID', '61=WDudE-2jKmX8WQjZbiH0QUi__G_kGOVDbs5XCTQO634x51d99mHYi-VptWUURDSJco_IwTTJLAM-Dp_LurBU-WWyz_3xy0BTsMHBJ5Sf6wZrgGwvyDDupLLcA86ypDUt'), ('PREF', 'ID=a7055cc9168cd7d3:FF=0:TM=1341167514:LM=1341167514:S=vbL-Ih__itkxmGxQ'), ('NID', '61=gLhy_qaWV5hZ9VcKxzUXfeJt1yRYmkVzPDVmtbYJshr01gWXNtDSwmHDaRByaNTYWJDzP_z_FjzPhy5EsZ6k74uaZDjX0sUNzgecF7si6Z-50GaemVbN3eNa22Z86T7s'), ('PREF', 'ID=b06021225aa97775:FF=0:TM=1341167515:LM=1341167515:S=wdaMCYeYlNZ-3tnS')]         
```

There are two cookies with name `NID`? It seems like when server update old cookies, session did not update it but insert new one. 


Could you show more details of the cookie objects? What are the domains, etc.?


@kiennt Google is redirecting to a localized domain, so there is a cookie for `google.com` and `google.xyz`.

You can verify this with:

```
>>> s.get("http://www.google.com", allow_redirects=False)
<Response [302]>
>>> s.cookies.items()
[('NID', '61=pNDDIx0Bj7mHfVejY_ZUY1h-8VYOo-W6GuYkX-lyeAPHHrFxLDb-I7wR8vspIl2Yf0x
F1lpK8ReuFa6ZTkxVUcHIX1_m7uy5KjPK9KUNwdy7k7PrOsdMOoMEnteMdE1w'), ('PREF', 'ID=6d
e3b5315cd0659c:FF=0:TM=1341305909:LM=1341305909:S=R3lLroX9nHdc_Ixa')]
```


Those look like two cookies to me?


The confusion here is probably related to the fact that a `RequestsCookieJar` isn't a dictionary, but can be manipulated like a dictionary. There is no guarantee that a cookie has a unique name, only that its combination of name and domain (and maybe path?) are unique. Each regional Google homepage sets two cookies, one called NID and one called PREF.

This behaviour is totally correct.
SSL errors shouldn't be converted to Timeouts when verify=False

Spent a while figuring out why I was getting timeouts that return almost instantly.  It's possible to use SSL and not verify certificates (yes, this is actually what I want), so SSLError exceptions shouldn't be converted to Timeout exceptions unless they really are timeouts.


They aren't. I'm assuming you're referring to [this section](https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L603) of `models.py`, which seems to throw `Timeouts` if it gets urllib3 `SSLErrors`. However, if you examine the code that runs before that try;except block, you'll note that [if verify is set to `False`, we tell urllib3 not to check the certificate](https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L536). The relevant section is reproduced here:

``` python
if url.startswith('https') and self.verify:
    [...snip...]
else:
    conn.cert_reqs = 'CERT_NONE'
    conn.ca_certs = None
```

This change prevents the python `ssl` module from verifying the certificate in any way, which means it won't throw `SSLErrors`, which means you won't hit that try;except block.

Would it be possible for you to provide a code snippet that can reproduce the problem, so I can take a look at it?

**EDIT**: I find it helpful to format things correctly.


Yes, but SSL errors are not limited to problems with the certificate. I'm seeing this same problem with a server that requires sslv3 while requests or urllib3 or whatever assumes something different. I think it has something to do with the ssl libraries that ship with ubuntu precise, because curl suffers from similar issues.


If curl is suffering it is probably the fault of your distro, because the Python `ssl` module uses system calls to the OS socket APIs.

As for your response, you're correct that `ssl.SSLError` is not only raised when the certificate fails to match, but when any problem occurs in the SSL layer. Within the scope of the OPs question, however, my answer was correct: his `Timeout`s were symptomatic of some other problem, not of `requests` or `urllib3` checking the certificate and throwing a `Timeout`.

With that said, I think the idea that legitimate SSLErrors shouldn't be hidden is correct. It's unclear to me why that check of `self.verify` is present at all. I've opened a Pull Request removing that check as Issue #715.


I can confirm this issue too. The bug causing this is much lower level, but it's still weird to see a Timeout error:  https://github.com/kennethreitz/requests/issues/730


This _should_ (no guarantees) have been fixed in Pull Request #715. This pull request was included as part of [Requests v0.13.3](https://github.com/kennethreitz/requests/zipball/v0.13.3), so if you run your code in Issue #730 using that version of Requests you should get an SSLError.

Doesn't fix your main problem of course. =P


Ah missed the pull request. Looks suitable indeed. Thanks :)
more information about cause when a Timeout exception occurs

We were experiencing a lot of time out exceptions in production which were triggered on a get using requests. 

```
response = requests.get(url,verify=False, prefetch=True)
```

was occasionally raising a Timeout exception ('Request timed out.')

We weren't setting timeout explicitly in the request so it was confusing that our request was timing out at all.

I dug around in urllib3 and realised there are two types of timeout error that can be raised, a socket timeout and a timeout caused by no connections available in the connection pool:

line 406 in requests/packages/urllib3/connectionpool.py:

```
    except Empty as e:
        # Timed out by queue
        raise TimeoutError(self, "Request timed out. (pool_timeout=%s)" %
                           pool_timeout)

    except SocketTimeout as e:
        # Timed out by socket
        raise TimeoutError(self, "Request timed out. (timeout=%s)" %
                           timeout)
```

currently both of thse get raised as a generic Timeout errors by requests:

in requests/models.py line 618 

```
except (_SSLError, _HTTPError) as e:
       if self.verify and isinstance(e, _SSLError):
           raise SSLError(e)

        raise Timeout('Request timed out.')
```

I think its usefull for us to know the difference between a socket time out error or a timeout due to no more connections available in the connection pool .

Let me know if you are interested in accepting a patch for this issue.

Thanks
Anthony


Hi @amenasse, I'm +1 on differentiating between the two timeouts. If you're going to do this on the urllib3 side, please make the pull request on that repo.

I'd suggest keeping TimeoutError as the base exception class for the two new ones you want to create. Something like SocketTimeout(TimeoutError) and PoolTimeout(TimeoutError).


Done!
Catch UnicodeError coming from encodings/idna.py

Here's a Python 2.6 Requests 0.13.1 traceback.  Looks like the call to `netloc.encode('idna').decode('utf-8')` needs a try/except.

```
File "/srv/import-service/lib/python2.6/site-packages/requests/api.py", line 76, in head
    return request('head', url, **kwargs)
  File "/srv/import-service/lib/python2.6/site-packages/requests/safe_mode.py", line 37, in wrapped
    return function(method, url, **kwargs)
  File "/srv/import-service/lib/python2.6/site-packages/requests/api.py", line 42, in request
    return s.request(method=method, url=url, **kwargs)
  File "/srv/import-service/lib/python2.6/site-packages/requests/sessions.py", line 230, in request
    r.send(prefetch=prefetch)
  File "/srv/import-service/lib/python2.6/site-packages/requests/models.py", line 618, in send
    self._build_response(r)
  File "/srv/import-service/lib/python2.6/site-packages/requests/models.py", line 305, in _build_response
    request.send()
  File "/srv/import-service/lib/python2.6/site-packages/requests/models.py", line 474, in send
    url = self.full_url
  File "/srv/import-service/lib/python2.6/site-packages/requests/models.py", line 388, in full_url
    netloc = netloc.encode('idna').decode('utf-8')
  File "/srv/import-service/lib/python2.6/encodings/idna.py", line 164, in encode
    result.append(ToASCII(label))
  File "/srv/import-service/lib/python2.6/encodings/idna.py", line 73, in ToASCII
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long
```


This incredibly unhelpful exception is sometimes thrown because the URL is invalid. For example:

``` python
>>> u'google.com'.encode('idna')
'google.com'
>>> u'.google.com'.encode('idna')
UnicodeError: label empty or too long
```

Would it be possible for you to check the URL you're using, or alternatively to post it here so I can take a look?


@terrycojones Is this bug still affecting you, or has it been resolved?


Fixed by #774.


Hi, sorry, I missed the comment from 4 days ago about whether this was still affecting me. Is that still relevant?


No worries.

I decided to be conservative and filed a Pull Request that replaces the `UnicodeError` with a `requests.exceptions.InvalidURL`, which is the actual problem. If that solves your problem, all is well in the world. If not, I've misunderstood what your problem is. =)


Replacing with requests.exceptions.InvalidURL sounds good to me. Thanks!

Terry
client certificates are ignored when verify=False

Our dev machines don't have valid server certificates for a web service we run. So we use `verify=False` for dev machines.

However our web service requires client certificates, so we use `cert=(cert, key)`. This fails because requests only sets the client certificate if `verify=True`.

Tested with 0.13.1 and latest develop branch. This pull request fixes the issue.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1706719) (merged 74a4d06d into 8f0ac668).


So you want to provide client certificates but not verify them?


From what I understand `verify=True` verifies server certs, it's not related to client certs. The server can verify the client certs, that's not `requests`'s job.


Ah, understood. Thanks for the patch!


Can we make verify=False for all the requests?? Kind of default option for all the requests...


Yes you can:

``` python
import requests
s = requests.Session()
s.verify = False

r = s.get(url)
```

Use a `Session` object to make all your requests. The [relevant part of the documentation is here](http://docs.python-requests.org/en/latest/user/advanced/#session-objects).

In the future, requests for help shouldn't be made on this issue tracker, they should be made on [StackOverflow](http://stackoverflow.com/). =)


@Lukasa Thanks :) Sorry for posting here, will take care from next time.
Ipv6 support : integration of urllib3's patch?

urllib3 has very recently been patched to support ipv6 (https://github.com/shazow/urllib3/issues/72). I reckon that's the only thing that prevents requests from supporting it.

How long would it take to integrate this bug into the packaged version that requests ship with?

Many thanks!


Yes this would be very good, I'm currently bumping up against this, I'm wondering why git sub-modules are not used here

Note it looks like this was fixed in 8f0ac668da796b04df9901db58f5d4215be308dd 

But it's yet to be pushed to PyPi. @kennethreitz can you do this?
Temporary fix for unicode literal breaking test.

As discussed in #687.

I'm not really wild about this commit. The awkward `if` block mentioned in #687 is present here, which isn't lovely, as is the awkward dependence on `sys.version_info`. This also relies on the fact that the parser will only look for a method when it's called, not when it's parsed, which means that `str.decode()` won't throw `AttributeErrors` in Python 3.

This is butt-ugly, and if it had to live in the codebase for all time I would ask you not to actually pull it. If you're planning to only test against 3.3 when it's released, however, then it's probably ok.

If you (or anyone else) can think of a nicer way to use this split, by all means use it and don't accept this pull request. =D


Thanks :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1657575) (merged f8076fee into bcf04d5e).
urllib3 overwrites 'Accept' header

Custom 'Accept' headers are overwritten by the ProxyManager._set_proxy_headers function in urllib3's poolmanager.py (line 124)

I did a bit of googling but couldn't come up with a reason why this should be done; if this modification of the 'Accept' header is not needed, would it be possible to get it removed somehow?


You should open this on urllib3's repo :)


Ah good point, will do :)
requests.get() doesn't use given proxy

I'm trying to use a proxy in a GET request

proxy = {'http:': '127.0.0.1:8080'}
...
...
res = requests.get(url, headers=headers, prefetch=True, proxies = proxy)
...

the request is made directly without using the http proxy I have running (BURP).
is there a way to force the request?

Thanks!


Try this: 

```
proxy = {'http:': 'http://127.0.0.1:8080'}
```


will be updated.


I can't understand why the request object isn't using the proxy all the time, I'm trying to create requests in different variations (I'm using PyScripter).

> > > re = requests.get("http://www.google.com", proxies = {'http:': 'http://127.0.0.1:8080'})
> > > I'm using python's interpreter, this line of code used to use the proxy I have installed, I then changed to fetch other URLs and passed the dict proxy as a variable (proxies = proxy, prefetch = True).

The code doesn't use the proxy here -

> > > re = requests.get("http://www.google.com", proxies = proxy, prefetch = True)

WireShark shows regular direct HTTP requests.

How can I debug it?


Works fine for me. Check your `proxy` variable is actually the dictionary you want to pass, e.g.:

``` python
proxy = {'http': 'http://127.0.0.1:8080'}
```

Make sure that you have the full schema present.

EDIT: Just did a copy-paste of @kennethreitz's code, and noticed that it just made standard requests. The issue seems to be that there is a colon in the dict key that shouldn't be there. Try removing that colon.


I've double checked it, still happens. 
what may cause it?


That's very odd. Can you copy and paste the exact code you're using so I can take a look? (Removing personally identifying information, obviously.)
Unicode Strings in Tests

Pull request #686 is failing Travis-CI testing because of [lines 354 and 355](https://github.com/kennethreitz/requests/blob/develop/tests/test_requests.py#L354) in [`test_requests.py`](https://github.com/kennethreitz/requests/blob/develop/tests/test_requests.py). This failure occurs only in Python 2.*, and is because the string needs to be a unicode literal.

An simple and butt-ugly fix would be to change the section of the test from:

``` python
post6 = post(url, files={'fname.txt': '\xe9'})
self.assertEqual(post6.status_code, 200)
```

to:

``` python
if (sys.version_info[0] == 2):
    fnamedata = u'\xe9'
else:
    fnamedata = '\xe9'
post6 = post(url, files={'fname.txt': fnamedata})
self.assertEqual(post6.status_code, 200)
```

I'm not putting this into a pull request because I hate this code so much it makes me want to break things. I'm happy to look into a better way, but I'm putting this issue up so that people are aware of it. Also so that someone smarter than me can fix it. ;)


That actually won't work because unicode literals aren't valid in 3 until 3.3 :)

Once 3.3 is out though, I'll drop official support for 3.1 and 3.2 and only test 3.3, which will make things a LOT easier.


Good point, forgot the parser would still encounter the literal and choke. 

As for 3.3, according to [PEP398](http://www.python.org/dev/peps/pep-0398/) the final release date is August 25. Do we want a (potentially very hacky) fix for this test before then, or are we happy to just let it fail/comment it out?


Yeah, either that or we can comment it out :)
Fix failing setup.py

Commit d3acb783b95623d0378db02bd9832fb11850204b updated `urllib3`, which involved the removal of the `mimetools_choose_boundary` package. This package is still being referred to in `setup.py`, which is causing the install to fail. This pull request resolves this problem, and therefore also resolves Issue #685.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1654489) (merged 80ceaa47 into 38db8c21).


Thanks!
Sign querystring parameters in OAuth 1.0

Existing usage doesn't pass GET querystring parameters along to oauthlib,
so it wasn't signing those properly, which causes problems with APIs that
rely heavily on GET parameters. By passing in r.full_url instead of r.url,
oauthlib can parse out the correct parameters and sign them properly.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1652457) (merged 4e568c18 into 38db8c21).


Yes! Thank you.
I was actually just going to make an issue. 

I was updating a lib and was wondering why my signatures were invalid when passing the `params` kwarg and were valid when I just did `http://api.example.com/method/?myquerystring=parameters` and thought of this issue to be the only explanation 

@gulopine Think you can get @travisbot to successfully pass?

 :+1:


I'm pretty sure Travis was breaking on some things unrelated to my code. It's been failing for several days now, but it looks like some other work was just done (ba7215f8f53e7cd820b8f79d0a87f20573a167d2 and 28dc45a30cba480177f7bfad5ee13fd87014bf4e) to resolve those issues. I can't find a way to re-run the Travis job without committing more code, though, so I think somebody will have to just merge it anyway. :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1669371) (merged 4e568c18 into 38db8c21).


Thanks, Marty! 


@kennethreitz @gulopine I feel like this didn't fix the issue. I went to update `Twython` to use

``` python
requests.get(url, params=params)
```

instead of 

``` python
url = url + '?' + urllib.urlencode(params)
requests.get(url)
```

and the request failed, response was None :(


The response was None? I can't think of why that would happen. If it was just a signing issue, you should get a response from Twitter saying so. I'll do some more testing tonight and see if I can reproduce it. To make sure I'm doing the same you are, can you provide a the `url` and `params` you're using?


url = 'https://api.twitter.com/1/statuses/home_timeline.json' 
params = {'count': 2}

You'll have to grab your own app key/secret, token key/secret if that's okay? haha 

## 

Mike Helmick
@mikehelmick

On Friday, June 29, 2012 at 12:49 PM, Marty Alchin wrote:

> The response was None? I can't think of why that would happen. If it was just a signing issue, you should get a response from Twitter saying so. I'll do some more testing tonight and see if I can reproduce it. To make sure I'm doing the same you are, can you provide a the `url` and `params` you're using?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/684#issuecomment-6667600
Cast the URL as string before using it

This patch solves the issue #423 https://github.com/kennethreitz/requests/issues/423

We added a small test file showing that now a request like
<code>requests.get(furl('http://example.com/ ').set({'query': 'string'}))</code>
is working properly.

Authors:
Iulian Stana
Vlad Traistă-Popescu
Constantin Șerban-Rădoi


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1635261) (merged 48c94214 into f178219d).
File upload with digest auth doesn't work

I think this issue still exists

https://github.com/kennethreitz/requests/issues/298

I am using latest version of Requests as of 15/06/2012

It still unnecessary uploads the file even though it knows it is going to receive a 401 but then fails to upload the file for the second request

My code...

import requests
from requests.auth import HTTPDigestAuth

url = 'http://localhost:8080/'
user = 'demo'
password = 'test'
filename = 'C:\Users\Public\Pictures\Sample Pictures\koala.jpg'

file = {'file': (filename, open(filename, 'rb'))}

r = requests.post(url, auth = HTTPDigestAuth(user, password), files = file)

print r.status_code


Thanks for the report! Investigation will continue.
Fixes #649, take 2.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1630497) (merged ede7b7eb into 8de9b2e1).


Fails because my tests are broken. Will look again as soon as they're fixed.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1630512) (merged 50988dc4 into 8de9b2e1).


Can you merge upstream? Tests are fixed now :)


Closing until further update.
Additional Documentation for Requests and Responses

I've edited the advanced docs with a simple tutorial of how to access the request object from a response, and I also provided a listing of methods and attributes accessible using a Request object and a Response object. This section is entitled "Request and Response Objects". I would have found this simple listing tremendously useful had it existed in the original documentation, and I hope others find it useful as well.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1620348) (merged 87637c9b into 1d193133).


This is awesome — thanks!


I'll be removing the big list of attributes though, we should just be linking to the [api docs](http://docs.python-requests.org/en/latest/api/)


Sounds good. I do agree about the list now that i think about it. Glad I could help!


Keep the pull requests coming :)
Make sure OAuth doesn't fall off

In particular, this happens when encountering redirects. The first request doesn't have a Content-Type, but the second request does (it was added by the OAuth1 object), so when it tries to apply the authorization details again, it falls off and returns None instead of the original response. This change simply moves the return up one level, so it always returns the request regardless of whether it was modified along the way.


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1616331) (merged 9a30e029 into 1d193133).


Thanks!
Response Hook not updating when non-2XX

When using response hooks, if I transform the passed-in response and the it becomes a non-2XX response, requests does not honor the new value.

This is important when dealing with successive failing requests. Test case:

```
import requests

def try_again(response):
   if response.status_code == 401:
      return requests.get('http://httpbin.org/status/400', hooks={'response': try_again})

response = requests.get('http://httpbin.org/status/401', hooks={'response': try_again})

print response.status_code
```

Expected output: `400`
Actual output: `401`


Added a test suite to the latest master branch, and turns out this is fixed in 0.13.1 (is broken in 0.12.1)
support for chunked Transfer-Encoding

I'm tried to use requests to PUT a big file to a WebDev server

that server that support only support chunked Transfer-Encoding
(see http://en.wikipedia.org/wiki/Chunked_transfer_encoding)

I know it's kind of a broken server, that it can't stream those files, with out this mechnizem.

it would be nice if it can be implemented on requests


This is currently planned, but not supported at the moment.
Programmatically Finding requests Object's HTTP Verb

I would like to do something equivalent to the following. I have looked through the documentation and the source code and it does not currently look like this is possible, but feel free to tell me if I'm wrong. I realize this is a redundant example, but it's the clearest way to illustrate my problem. My real use for knowing the HTTP verb from a requests object is for writing my own AuthBase, since our auth needs to know what type of method is being used.

``` python
r = requests.get('http://some.url.com/endpoint')
print r.verb
>>> 'GET'
```


``` pycon
>>> r.method
'GET'
```


Thanks! Is there a way that can be added to the documentation? I did a Ctrl+f on all the docs and didn't find it (on both python-requests.org and github) and couldn't find anything. I'm assuming this kind of thing is also possible for any of the params `r.data`, for example. If you don't have time I'd be happy to make an attempt at adding it myself sometime in the next day or two.


Working from the repository I don't see this attribute on the Response object.

```
>>> import requests                                                                       
>>> requests                                                                              
<module 'requests' from 'requests/__init__.py'>                                           
>>> requests.__build__, requests.__version__
(4865, '0.13.1')
>>> r = requests.get('http://httpbin.org/')                                               
>>> r                                                                                     
<Response [200]>                                                                          
>>> r.method                                                                              
Traceback (most recent call last):                                                        
  File "<stdin>", line 1, in <module>                                                     
AttributeError: 'Response' object has no attribute 'method'
```


Ah, sorry, it's on the Request object.

`r.request.method`


Ah yeah. Knew I'd done something like that before. Just wanted to make sure the functionality had not been removed.


+1 to adding it to the docs


Is there any point in adding a method attribute on the Response object to make access to the request attribute easier? If not, I'll see where this can be added to the docs.


No, not when the full request object is available.


Didn't think there would be. Thanks for the confirmation.


Thanks all for the responses! I've edited the docs and submitted a pull request, so if y'all want to discuss this further we could talk there. But as this issue is resolved I am closing it.
Fixes safe mode when not sending requests immediately (issue #639)


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1597016) (merged 52f3344b into 1d193133).


@kennethreitz Any comments why it was closed? I believe that failing tests have nothing to do with this pull request.


Seems this is related to #888?


@sigmavirus24
Yes, I believe it fixes #888 too.


@piotr-dobrogost can you not re-open this? If not, rebase and push to the same branch and re-open. It'll trigger Travis and your fix should work.


Well I don't see any option to reopen this or any of issues I created for a matter of fact... Should I?


Either way it has been done.
test_file_post_data error on Windows

```
(requests) C:\requests\tests>python test_requests.py RequestsTestSuite.test_file_post_data
E
======================================================================
ERROR: test_file_post_data (__main__.RequestsTestSuite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_requests.py", line 476, in test_file_post_data
    r = post(service('post'), data=open(testfile.name, "rb"),
IOError: [Errno 13] Permission denied: 'c:\\users\\piotr\\appdata\\local\\temp\\tmpfpitom'

----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (errors=1)
```

see [How do you fix the following Django Error: “Type: IOError” “Value: [Errno 13] Permission denied”](http://stackoverflow.com/questions/4125859/)