Raibek/requests-github-issues huggingface dataset:
Session dropping proxy for https redirects

I noticed it when trying to add proxy support to python-foreman, but have reduced it down to just being an issue with requests.  The cert on my destination is self-signed, but verify is set to false.

The following code reproduces the problem.  When I observe the traffic I see it make a request out to the proxy server, then it follows up by trying to make a request directly to the server without using the proxy and times out because it can't get there.

I'm not seeing other people complain about this issue, and wonder if I'm just missing something.

```
#!/usr/bin/env python
import requests
s = requests.Session()
s.proxies = {
        'http': 'proxy.example.com:3128',
        'https': 'proxy.example.com:3128'
    }
s.get('https://uri.that.redirects.example.com', verify=False)
```

I've tried setting the environment variables HTTP_PROXY and HTTPS_PROXY, and setting explicitly on my session object, or passing it to each Session.get() request as part of the **kwargs (like verify).  I've tried with http:// URI's and with just host:port for the proxy values in both the internal dict and the env variables, as the documentation seems to suggest both in different places.

I can provide more specific details or a packet dump if necessary.


Ah, yes, this is a well known issue. It's the same as pypa/pip#1805. This should be fixed in the next release of requests. =)
Multiple consequtive requests on Session throws TunnelError on https proxies.

If you make consecutive requests on Session with set https proxy.
    import requests

```
s = requests.session()
s.proxies = {'https': 'https://xxx.xxx.xxx.xxx:8888'}
s.get("https://www.tumblr.com/")
s.post("https://www.tumblr.com/")
```

You get this error
`ProxyError: Cannot connect to proxy. Socket error: Tunnel connection failed: 400 Bad Request.`


@m-novikov we need more information.
1. What kind of proxy are you using?
2. What is the full traceback?
3. What is the exact version of python you're using (e.g., 3.4.1)
4. What is the version of requests you're using?
5. Can you still reproduce this with the version of requests present in `master`?


We should also note that this may well be related to pypa/pip#1805.


@Lukasa that was why I asked question 5. I just didn't want to jump the gun on the association.


@sigmavirus24 
Sorry for late answer.
- Random https proxies from public lists
- Traceback

``` python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 425, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 381, in send
    raise ProxyError(e)
requests.exceptions.ProxyError: Cannot connect to proxy. Socket error: Tunnel connection failed: 400 Bad Request.
```
- Python 2.7.6
- tested on requests 2.2.1, requests 2.3.0
- Looks like it's fixed in master.


Suspect then that it is identical to pypa/pip#1805. =)

Thanks for the report!
Cookies lost on redirect

Hi, i'm using requsts 2.3.0 from PyPi and get blank cookies after 302 redirect

``` python
Python 2.7.5+ (default, Sep 19 2013, 13:48:49) 
[GCC 4.8.1] on linux2
>>> import requests
>>> requests.__version__
'2.3.0'
>>> r = requests.get('http://mlgame.us/?state=register_simple_new')
>>> print r.cookies
<<class 'requests.cookies.RequestsCookieJar'>[]>
>>> print r.history[0].cookies
<<class 'requests.cookies.RequestsCookieJar'>[<Cookie PORTAL_SESSIONID=vhjyfqxwh3oh1mtvxly7lqwf1 for mlgame.us/>]>
```

In version 1.2.0 all worked fine:

``` python
Python 2.7.3 (default, Sep 26 2012, 21:53:58) 
[GCC 4.7.2] on linux2
>>> import requests
>>> requests.__version__
'1.2.0'
>>> r = requests.get('http://mlgame.us/?state=register_simple_new')
>>> print r.cookies
<<class 'requests.cookies.RequestsCookieJar'>[<Cookie PORTAL_SESSIONID=8r6iqae7yirj1gxx7x9biwiov for mlgame.us/>]>
```


@fenume they're not "lost". See @lukasa's [comment](https://github.com/kennethreitz/requests/issues/1446#issuecomment-20658864) on #1446. This was intentional because a single response's CookieJar should represent the cookies sent by the server in _that_ response. If you want what used to work in version 1.2.0, I would advise you start using a session.

``` pycon
>>> s = requests.Session()
>>> r = s.get('http://mlgame.us/?state=register_simple_new')
>>> s.cookies
<<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='PORTAL_SESSIONID', value='15nalzaawi5yf9njecyoec73f', port=None, port_specified=False, domain='mlgame.us', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False)]>
```
add auto detect charset from http body when http headers not seted

requests detect page's charset/encoding from responsed http headers, but many servers won't set it in headers.

In this case, requests will set the charset/encoding to the default 'ISO-8859-1', and result in decoded error, many Chinese developers get lose and complain.

For example:

```
print requests.get('http://www.126.com/').text
```

This commit will detect charset/encoding from http headers, if no, from http body, for most situations, everything goes well now.


Thanks for this!

This is a change that has been discussed many, many times. A sample of issues that touch it are: #1604, #1774, #1604, #1589, #1588, #1546, and #2042. We also mention it in the documentation [here](http://docs.python-requests.org/en/latest/user/advanced/#encodings). Though it's an easy mistake to make, in future you really should search both open and closed issues before making a change like this because you'll save yourself time and effort.

We aren't going to merge this. Previously we wouldn't have merged this because of the explanation given in the issues mentioned above (namely, that the behaviour is deliberate, and that we aren't a HTML library so we won't introspect content). Currently, this behaviour is tracked under the _open_ issue #2086, which will change this behaviour in a different way. If you'd found that issue you'd have saved yourself some time.

I'm sorry that we can't take this change, please don't let it dissuade you from continuing to contribute!
requests.codes.continue raises SyntaxError

``` python
>>> import requests
>>> requests.codes.continue
  File "<stdin>", line 1
    requests.codes.continue
                          ^
SyntaxError: invalid syntax
```

We expect to see the Informational HTTP status code, 100, but instead an error is raised. `requests.codes['continue']` works as a workaround. Presumably this happens because `continue` is a keyword in Python and therefore cannot be used as an attribute name. Python 2.7.6, OS X 10.10.


It's not really clear what we can do about this. Continue is and always will be a keyword.


The _best_ we could do is support case-insensitive attribute access but that would certainly be useful to an extremely tiny subset of users. (In other words, you would be able to get around this with requests.codes.Continue.)


The fact that Python does not have contextual kewords is PITA. Compare with C# - http://msdn.microsoft.com/en-us/library/x53a06bb.aspx

Wait, there [was](http://fiber-space.de/wordpress/2009/03/22/contextual-keywords/) one contextual keyword - 

> The word `as` was such a contextual keyword in Python until it became a proper keyword in version 2.5.

Relevant SO question - [Why can't attribute names be Python keywords?](http://stackoverflow.com/q/9746838/95735)
Changed url in the sidebar for github stars iframe

This is the one which shows up first in Google search results and looks odd with broken sidebar below the logo.
![requests-docs-broken](https://cloud.githubusercontent.com/assets/769447/3776643/f362af58-1957-11e4-94ce-93806eca40b0.png)


As I see, this issue was fixed on the master branch on the latest version. 
Is there a way to edit the documentation here : http://docs.python-requests.org/en/v0.10.7/user/quickstart/ ?


@atmb4u Sadly there isn't. That is built directly off a git tag, which is a static reference to a single commit, we can't update it.
Not supported proxy scheme socks5

I see https://github.com/kennethreitz/requests/pull/478 is closed, but why I can not use socks5 proxy?

define as:
`
PROXIES = 
    {
      "http": "socks5://127.0.0.1:7070",
      "https": "socks5://127.0.0.1:7070",
    }
`

I use above proxy , and requests raise error `Not supported proxy scheme socks5`


The error is perfectly clear. We do not support SOCKS5 proxies at this time. If you'd read #478 more closely you'd have seen [this comment](https://github.com/kennethreitz/requests/pull/478#issuecomment-24914571) I made:

> We currently do not have SOCKS proxy support, we're waiting for that code to be included in urllib3. There are plans afoot for this, but no definitive schedule at this time.

This remains true.


[`urllib3` is an implementation detail](https://github.com/kennethreitz/requests/pull/1812#issuecomment-30854316). 

If the issue is not fixed; it should not be closed unless the answer is that `requests` will never support SOCKS5 proxies.


The fact that urllib3 is an implementation detail is relevant for our features. For features we don't have, implementation is critical. For example, `httplib` is an implementation detail but it's also _why_ we don't have Twisted support. 

I was not blaming urllib3, I was providing a path to get this feature.
HTTP Status Code always 404

http status code always 404 even though link actually exists and can be accessed directly.


This bug report is seriously incomplete.

Can you suggest what the link is? Can you provide some sample code that reproduces the bug?


When I try to open that URL from my browser I also get a 404. Are you sure this URL is correct?


sample code:

import requests

if **name** == "**main**":
    r = requests.get("http://movie.douban.com/subject/1459804/")
    print(r.status_code)
    input()

output:
![default](https://cloud.githubusercontent.com/assets/237062/3711232/1bfc271a-14ca-11e4-86d1-2c5a4122e6d2.png)

the page:
![2](https://cloud.githubusercontent.com/assets/237062/3711238/5e542c66-14ca-11e4-8c4c-ffef3e6a7c54.png)


The fact that we 404 in this case suggests a number of possibilities. One option is that the website is only available from a certain set of IP addresses, e.g. via a proxy. Another option is that you need to have the appropriate User Agent set. You can do both of those things in requests, but you'll have to investigate: without knowing the configuration of your machine I cannot help debug it.

Regardless, it is the _website_ that is breaking your code, not requests. =)
requests loosing cookies

Because of dict-like headers requests can loose one ore more cookies 
(i.e. http://www.lineage2dex.com/):

Set-Cookie: __HFCTL=0; path=/
Set-Cookie: PHPSESSID=7c3129a98cc6a75e0cbf71b7a3513364; path=/

But in requests i have only one (the first).
I think the solution is to take othe data structure (like http://werkzeug.pocoo.org/docs/datastructures/#werkzeug.datastructures.MultiDict from werkzeug) that will not break api, but can hadle this situation.


@s0rg you didn't tell us which version of requests you're using so we can't even test this to ensure that it's a problem.


Besides that, this appears to be an issue with `httplib`:

``` pycon
>>> conn = httplib.HTTPConnection('www.lineage2dex.com')
>>> conn.request('GET', 'http://www.lineage2dex.com/')
>>> r = conn.getresponse()
>>> r
<httplib.HTTPResponse instance at 0x10d4b8488>
>>> r.getheaders()
[('content-length', '633'), ('set-cookie', '__HFCTL=1; path=/'), ('expires', 'Fri, 25 Jul 2014 14:24:57 GMT'), ('ddos', 'Protected by HyperFilter : http://www.hyperfilter.com'), ('server', 'HyperFilter FastCache v1.0'), ('hf-wall', 'HIT-3'), ('connection', 'close'), ('cache-control', 'no-cache'), ('date', 'Fri, 25 Jul 2014 14:24:58 GMT'), ('p3p', 'CP="IDC DSP COR CURa ADMa OUR IND PHY ONL COM STA"')]
```

Therefore it is something requests can not fix for you. Using `urllib2` would cause the same problem:

``` pycon
>>> r = urllib2.urlopen('http://www.lineage2dex.com/')
>>> r.headers
<httplib.HTTPMessage instance at 0x10e00ad40>
>>> r.headers.getheaders('set-cookie')
['__HFCTL=1; path=/']
```


Ok, thanks.


@s0rg if you upgrade to Python 3 this should be fixed http://bugs.python.org/issue1660009
How to make spaces to encode as %20 instead of +?

`print requests.get("http://url.com", params={"key" : "asd asd").url`

I want this output: `http://url.com?key=asd%20asd`, but I get this: `http://url.com?key=asd+asd`


Requests provides no easy way to do this, because it follows the W3C standard for urlencoding that specifies that urlencoded spaces map to '+' chars, not %20.

You'll notice we map to %20 in other parts of the URL.
Handle socket errors in iter_content

My first PR for Requests, so I probably did something terribly wrong.

Closes #2144.


![:+1:](https://d16jfyletng9p5.cloudfront.net/media/bOUbNyILuU56U/giphy.gif)


This looks good to me. =)


@sigmavirus24 @alex Fixed. Not sure what I was thinking, there.


:sparkles: :cake: :sparkles:
Invocation of `SSLContext.load_verify_locations` causes intermittent hanging

I have a background worker that makes many http requests to various APIs. I've found that every time my job runs, a few requests appear to just "hang" for quite a long time.
I've investigated further and it appears the requests are not actually hanging, but my python code "hangs" when it invokes `OpenSSL's` `SSL_CTX_load_verify_locations` function.

I forked `requests` and added some more logging. The last entry to be logged in my code is `begin: ssl_wrap_socket: load verify locations` and that corresponds to [this line](https://github.com/WiFL-co/requests/blob/master/requests/packages/urllib3/util/ssl_.py#L118).

One possible conclusion is that SSL_CTX_load_verify_locations is hanging because of parallel file access or the version of OpenSSL on my Heroku dyno has a bug. FYI the `requests` library is calling python's `load_verify_locations` which is what then calls `OpenSSL`. I am not invoking these functions manually.

[I've created a question on StackOverflow](http://stackoverflow.com/questions/24628866/why-does-my-program-hang-after-urllib3-logs-starting-new-https-connection) that provides extensive information. I am using the following dependencies:

```
python 3.4
requests==2.3.0.  
Rabbit 3.1.3 celery:3.1.11 (Cipater w/ Prefork) kombu:3.0.16
billiard:3.3.0.17 py-amqp:1.4.5
```

@Lukasa has been really helpful thus far, and I've been able to narrow it down. I figured i'd throw this issue up and see if anyone else has some insight.


I'm really interested to see what people can come up with here. I cannot see any good reason for `SSL_CTX_load_verify_locations` to hang.


Closed for inactivity.
Update faq.rst

Removed an unnecessary "to" in the "What are "hostname doesn't match" errors?" section.


Why don't we just rewrite that entire sentence? Something like:

"This is important when servers are using Virtual Hosting (include link). When these servers are hosting several SSL sites they need to be able to return the appropriate certificate based on the hostname the client is connecting to."


@sigmavirus24 I've updated the wording - comments?


It may be beneficial to include a link to confirm if SNI support is available. For this, https://sni.velox.ch seems reliable (has been around more than a year and is referenced in the Mozzila bug tracker). It's also the URL used by PHP's tests (http://gcov.php.net/viewer.php?version=PHP_5_4&func=tests&file=ext%2Fopenssl%2Ftests%2Fsni_001.phpt)


@TkTech I completely agree! A pull requests that added that would be accepted :)
socket.error exception not caught/wrapped in a requests exception (ConnectionError perhaps?)

I just noticed a case where I had a socket reset on me, and was raised to me as a raw socket error as opposed to something like a requests.exceptions.ConnectionError:

```
  File "/home/rtdean/***/***/***/***/***/***.py", line 67, in dir_parse
    root = ElementTree.fromstring(response.text)
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/models.py", line 721, in text
    if not self.content:
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/models.py", line 694, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/models.py", line 627, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/response.py", line 240, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/response.py", line 187, in read
    data = self._fp.read(amt)
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/httplib.py", line 543, in read
    return self._read_chunked(amt)
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/httplib.py", line 612, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/httplib.py", line 658, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/home/rtdean/.pyenv/versions/2.7.6/lib/python2.7/site-packages/gevent-1.0.1-py2.7-linux-x86_64.egg/gevent/socket.py", line 385, in recv
    return sock.recv(*args)
socket.error: [Errno 104] Connection reset by peer
```

Not sure if this is by accident or design... in general, I guess I'd expect a requests exception when using requests, but I can start looking for socket errors and the like as well.


No, this looks like an error.

`iter_content` doesn't seem to expect any socket errors, but it should. We need to fix this.
Upgrade urllib3 to 1.9.x

It seems urllib3 changed the timeout logic such that, we now have a failing test (`test_stream_timeout`). We now receive a `MaxRetryError` instead of a `TimeoutError` from urllib3. Either we have to adjust or this is a bug that needs to be fixed in urllib3.

Thoughts @shazow @lukasa?


Don't think it's strictly a urllib3 bug: I seem to recall @shazow had a comment in his urllib3 code wondering why `Timeout`s weren't wrapped in `MaxRetryError`s.

As for what to do, I'm not sure. Check what the wrapped exception is, maybe?


I'm not convinced that `ReadTimeoutError`s are the same as a `MaxRetryError`. I certainly agree that a `ConnectTimeoutError` is, I'm just not convinced that a `ReadTimeoutError` is. Perhaps `urllib3` should be checking more carefully which `TimeoutError` it is receiving and rethrowing as a `MaxRetryError`?


This is intentional. Our default retry logic handles a lot more cases now,
including read errors in some scenarios.

You can disable urllib3 retries altogether by passing False and the
exceptions won't get wrapped.
On Jul 21, 2014 12:05 PM, "Ian Cordasco" notifications@github.com wrote:

> I'm not convinced that ReadTimeoutErrors are the same as a MaxRetryError.
> I certainly agree that a ConnectTimeoutError is, I'm just not convinced
> that a ReadTimeoutError is. Perhaps urllib3 should be checking more
> carefully which TimeoutError it is receiving and rethrowing as a
> MaxRetryError?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2142#issuecomment-49650248
> .


I'm going to investigate the urllib3 API some more to investigate the ability to pass a list of exceptions to not retry. If that's not possible, I might be able to override one or more of the helper methods around this.

I should also mention that I'm 100% against inspecting the error messages in this case because it will inevitably lead to someone else sending a PR to unveil a wrapped exception that they want to deal with (in requests) in some specialized way and they'll point to this as a reason why their PR should be accepted. I'm basically 110% in favor of anything that keeps us from having to inspect the exceptions urllib3 is wrapping.


If you pass `Retry(read=False)` then it won't retry anything related to read errors [as defined here](https://github.com/shazow/urllib3/blob/master/urllib3/util/retry.py#L186). I believe it won't wrap it either (I could be mistaken).


Assuming we're good now.
proxy not used for second connect

Hi,

we encountered a strange problem where requests stops using the proxy. 

if a http(s) connection to a server is not using keep-alive a new connection to that server is made per request. while the first connection uses the proxy well the second connection times out because it does not use the proxy anymore.

i figured out a fast-fix by re-evaluating the proxy object always. i just removed the line 209 in requests/adapters.py to always call proxy_from_url. this works for us.

if needed i could dive deeper into it to help to get more information if wanted.

regards,
michael


This seems like it is related to https://github.com/pypa/pip/issues/1805. 


Agreed.


That means this will be fixed by the next release of requests, when we pull in a new urllib3.
Quickstart doc updates

- Add the Timeout exception in API doc so it gets hyperlinked properly from quickstart
- Restructure the redirection and history section so it's a bit easier to read
- Couple of other minor tidy-ups

Note: the documentation build does not work on my machine, so I have not tested this fully.


LGTM.

While we're here, can we change the `history` section to stop referring to `Response.history` as a method? It's a property. =) Might be ok to leave the `:meth:` reStructuredText directive for it though.


I actually changed it to `:meth:` from `:class:`.

Is there a directive specially for properties?


Not that I'm aware of. What I was getting at is the text below (emphasis mine):

> We can use the `history` **method** of the Response object to track redirection. The `Response.history` list contains the `Request` objects that were created in order to complete the request. The list is sorted from the oldest to the most recent request.

That text should say 'property', not 'method'.


OK, I see what you mean. I amended the commit to make that change.


I see CI builds are now being done with Jenkins.  Nice.

Are the jobs configured in git somewhere?  I'd like to have a look and see if I can add a documentation build, which would help to verify changes like this.


No, they're configured on the system. I can add one though, we should add it to the list of issues. Otherwise this is good, I'm happy with it. :cake:


Builds fine, looks like the `:meth:` directive doesn't link though.


OK, I'll look at it again tomorrow and try to get the doc build working on my machine.

Would be good if you could set up a doc build on Jenkins, thanks.  Would be even better if the jobs were configured with JJB and kept in git, then I'd just be able to add it myself and submit another PR :)


The `:meth:` and `:class:` directive links are fixed in #2135
verify=False,but still requests.exceptions.SSLError

I have set the ssl verify=False, and I used a proxy(charles) with a Fake SSL cert,works fine with some website  ,but some still raise SSLError,such as 
https://pcs.baidu.com/rest/2.0/pcs/file?method=rapidupload&content-length=1643802624&content-md5=a655883b853e766e0cdf961bda894df9&slice-md5=fc7812dbdce256902d9277cef2c9b0da&path=/tmp2/a/ak.iso&app_id=250528&ondup=newcopy&filename=asas.iso

---

Traceback (most recent call last):
  File "/home/khalid/py/py-mime/py20140710/baidupan.py", line 260, in <module>
    main(sys.argv[1:])  
  File "/home/khalid/py/py-mime/py20140710/baidupan.py", line 238, in main
    upload_rapid('/media/khalid/数据盘/迅雷下载/VS2012_ULT_chs.iso')
  File "/home/khalid/py/py-mime/py20140710/baidupan.py", line 136, in upload_rapid
    ls.get(url, proxies=proxies, verify=False)
  File "/home/khalid/py/py-mime/py20140710/lsession3.py", line 73, in get
    self._r = self._rqs.get(url, *_kw)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error


Setting `verify=False` does not mean you will never get an `SSLError`, it just means we won't validate the certificate. In this case we tried to perform the TLS handshake and encountered an internal error. We can't ignore this kind of error, we have to raise it.


I am having a similar error. I am trying to request a url that has an expired certificate and I set verify to false I get:

Traceback (most recent call last):
  File "/media/tj/Windows 8.1/Users/tj/Desktop/sockets.py", line 25, in <module>
    soc1.connect(ADDR)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 113] No route to host

This is weird because if I use urllib2 for the same exact url it works fine.


@Fire30 that traceback doesn't appear to have anything to do with requests. Can you explain the relationship?


Closing due to lack of activity. If there are any updates let us know and we'll reopen this.
SOCK4/5 Proxies

Hello, is there any built-in support for socks4 and socks5 proxies?
i'am running a tor process at port 7575 and i would like to use it in fetching data.


Currently no. We need to work on getting shazow/urllib3#284 up to scratch and merged to get that support. 


Also if you have questions, the proper place to ask them is on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests). Also, had you performed a search you would have found your answer.
Depend on certifi

Resolves #2123.

This represents one possible approach to certifi: always take it if it's present. This is probably not the most secure approach, however. If we wanted to be more intelligent we could embed the version number of certifi that requests contains and use the most up-to-date version. If you're interested in doing that let me know and I'll add it as an enhancement.

Note also that this adds certifi as a hard dependency. I'd be equally happy to not install it in the install process but take it if it's there.


Lets do it!

:sparkles:


I don't think making certifi a hard-dependency is a good idea. There are reasons to prefer the distribution-wide ca-bundle.


Sure there are, but that's orthogonal to certifi being a dependency. All we've done is change our logic from preferring our own bundled certs to certifi's. If you override our logic (as all distros do) then you will continue not to use certifi.


I am the maintainer of the packages for archlinux and I override the logic there. The problem is that I also have to remove the requires in setup.py because, for example, scripts using load_entry_point will fail to import the required certifi module.

Essentially I think the certs.py logics are fine but the requirement in setup.py should be lifted.


That's a risk. If we take the policy that our new dependence on certifi means we no longer need to update the build-in bundle we've exposed our users to risk.

I have no particular objection, so I'll let @kennethreitz make the call: should the hard dependency on certifi be removed?


Certifi should be a soft-requirement, always used if present. 

Ideally, people should always have an up-to-date version of certifi installed. We can document this, and recommend this. 


Ok, we'll remove certifi as a hard dependency then. =)


Done, see #2203. 


<3
Response encoding detect

Take one link for example: http://baike.baidu.com/view/115789.htm

This HTTP response has header `Content-Type:text/html`, no encoding, however, the
returned html content has `<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />`, actually using requests, I got encoding `'ISO-8859-1'`.


Thanks for this issue!

This is a well known requests behaviour that is very clearly documented. You can see the documentation section [here](http://docs.python-requests.org/en/latest/user/advanced/#encodings). It has been raised in prior issues as well: #1604 and #2042 for some of the most recent discussions of it.

However, we're looking to remove this fallback: see #2086. Note, however, that we're not sure we'll examine the HTML for the meta tag. Our position on this has been that we're not a HTML library, we're a HTTP library, and therefore examining the body of the request is outside our remit.

In the meantime, if you want to do this you can take advantage of one of our utility functions: `requests.utils.get_encodings_from_content`. You use it like this:

``` python
import requests
from requests.utils import get_encodings_from_content

r = requests.get('http://baike.baidu.com/view/115789.htm')
codings = get_encodings_from_content(r.content)
if codings:
    r.encoding = codings[0]

# Do stuff with r.text
```


Thanks a lot, :)
python-requests does not support rfc6874 URLs with link-local addresses and zone identifiers 

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:
always

Steps to Reproduce:
1.ip addr add fe80::1 dev p14p14
2. nc -6 -l 1234
3. python -c "import requests; requests.get('http://[fe80::1%25p14p1]:1234')"
3. nc -v6 'fe80::1%p14p1' 1234  
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to fe80::1:1234.

Actual results:
The python script shows a traceback:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, *_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='fe80::1%25p14p1', port=1234): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

Expected results:
python requests should connect to netcat.

Additional info:
https://tools.ietf.org/html/rfc6874


Please see https://bugzilla.redhat.com/show_bug.cgi?id=1076822 for more information


This is a duplicate of #1985. I'm going to close this to centralise discussion there.
Update how we check verify when inspecting env variables

Closes #2092


Assigned to @Lukasa for review


Notes inline. =)


LGTM. :star: :cake:


We should do `if verify`, not `if verify is True`.


@kennethreitz Disagree: verify may be set to a string (path to the cert bundle) and we shouldn't overwrite it.


Ah, gotcha. Good catch :)
Update sessions.py

allow Session to set a timeout for requests


Thanks for this!

Unfortunately, this is a feature request that comes up often but that we don't have any real interest in. See #2011 for a longer form justification. For this reason, I'm afraid that we can't accept this pull request. Thanks so much for the work though, and please do keep contributing! :cake:


Thanks. The documentation should probably updated. http://docs.python-requests.org/en/v0.10.6/api/#requests.Session


You just linked to the v0.10.6 documentation. Try looking at the latest version: http://docs.python-requests.org/en/latest/api/#requests.Session


@manizzle you're linking to an ancient version of the documentation (you can see it in the URL `v0.10.6`). The current documentation is fine: http://docs.python-requests.org/en/latest/user/advanced/#session-objects


Okay thanks
[idea] Change how we merge request and session settings

# Problem

Currently, tools (e.g., [openstack/python-swiftclient](https://github.com/openstack/python-swiftclient/blob/3d0de79e26e2aa6285742c60aca3c164e9c2fbb9/swiftclient/client.py#L942..L945)) fight how requests sets the `Content-Type` header for a request. Notice that if the user of swiftclient doesn't provide their own `content_type` then the library sets the header's value to `''`. Ideally, setting `None` in a situation like this (where the Session has no default `Content-Type`) would prevent the request preparation from setting its own `Content-Type` header. This doesn't work because the per-session and per-request settings are [merged at the session level](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L361).
# Potential solutions
## Split the responsibility of merging settings

The merge at the session level could just take care of ensuring that the per-request settings have priority while waiting for the request preparation to remove `None`s. The downside of this behaviour is that it breaks backwards compatibility. Why? Consider the following:

``` python
import requests

s = requests.Session()
s.headers['Content-Type'] = 'application/json'
s.post(url, data={'some': 'formdata'}, headers={'Content-Type': None})
```

This allows for requests to detect that it is in fact `application/x-www-form-urlencoded` by removing the default set by the session. Keeping backwards compatibility would mean that we would have to do extra work while merging settings on the session-level. We would have to check if the header was set on the session level, and then we would delete headers with value `None`, otherwise, persist the `None` value. This is clearly a lot of extra logic.
## Create a constant/singleton to mean "Do not autogenerate any of this for me"

We could potentially expose a new constant or singleton in requests to allow users to specify when they want to prevent requests from handling it for them. I dislike this idea a lot, but it's potentially easier to implement and a bit more explicit than overloading the already overloaded meaning of `None` in this context.
gevent ssl and WantWrite error

Hi!

Using `gevent` with  `pyOpenSSL 0.14` I got `OpenSSL.SSL.WantWriteError` error.  Here is simple script to reproduce the issue:

```
from gevent import monkey
monkey.patch_all()

import requests

payload = 'x' * 1024 * 1000
requests.get('https://example.org/', data=payload, verify=False)
```

As a workaround you can use `pyOpenSSL 0.13`.  BTW it seems to me that SSL's `WrappedSocket.sendall()` method should do the same thing it does for SSL handshake. But it seems to be a problem since pyOpenSSL doesn't report number of bytes already sent. I got `OpenSSL.SSL.Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]` on retry.


Here is possible fix:

```
class WrappedSocket(object):
...
...
    def send_until_done(self, data):
        while True:
            try:
                return self.connection.send(data)
            except OpenSSL.SSL.WantWriteError:
                # TODO: implement timeouts
                select.select([], [self.socket], [])
                continue

    def sendall(self, data):
        while len(data):
            sent = self.send_until_done(data)
            data = data[sent:]
```


Hang on: if this works for PyOpenSSL 0.13 and breaks for PyOpenSSL 0.14, in what way is this our bug? (Just want to clarify my understanding, I'm not saying it's not our bug. Yet.)


I'm not sure about pyOpenSSL but it seems to me that in 0.14 they switched to ffi instead of C extension module. Also I'm not sure that it works correct in 0.13 as it handles WantWriteError somewhere inside pyOpenSSL C extension module bypassing `gevent's` select.


Sorry, I'm wrong 0.13 is also affected.


@Lukasa: anyway it's urllib3 related issue


Aha, if it's a urllib3 related issue can we open it over there?


Started new issue https://github.com/shazow/urllib3/issues/412
Obsolete link address in docs/user/install.rst

Hi, the link which describes pip and easy_install's difference in `docs/user/install.rst` are obsolete.

```
But, you really `shouldn't do that <http://www.pip-installer.org/en/latest/other-tools.html#pip-compared-to-easy-install>`_.
```

But I found that this page is not available in pip's document any more, maybe we should point this link to some place else instead?


Good spot, we should fix that up. Maybe [this StackOverflow answer](http://stackoverflow.com/questions/3220404/why-use-pip-over-easy-install)?


I think that StackOverflow answer is good. :smile:
Update test_expires_valid_str in test_request.py

"assert cookie.expires == 1" is changed to "assert cookie.expires is not None" at line 1166.


What's the purpose of this change?


When running on my system, the previous test failed, I think it should be
tested against None, not 1
2014年6月23日 下午1:02于 "Cory Benfield" notifications@github.com写道：

> What's the purpose of this change?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2106#issuecomment-46806552
> .


So this change isn't going to be right. We tracked this issue previously in #1859 and #1860. The specific problem is that our cookie parsing seems to be weirdly timezone dependent. Testing against `None` isn't particularly helpful: we want to make sure the response is right, not that it's present.


It might be enough simply to reopen #1860 and do the investigation we never got around to doing.


@xiongxoy can you apply the commit in the PR that @Lukasa referenced (#1860) and see if that fixes the test for you?
setup.py says requests does not support python3.4

Since Ubuntu 14.04 LTS comes with python 3.4, this is quite a big deal.

I haven't tested it yet with python 3.4 myself


@robvdl it does not say that it "does not support Python 3.4" it is merely an omission that we do in fact support 3.4. I'd like to wait for @kennethreitz to add a Python 3.4 builder to the Jenkins server before adding it however.


Requests works fine with Python 3.4, I get some test coverage of it from hyper. =)


@sigmavirus24 you're too abrasive :)


@robvdl many thanks! fixed by d22b8d8e7e8fcb8d16efba6841977dc81ac2c935
make requests.request('HEAD',...) behave the same as requests.head(...)

Currently requests.head() behaves differently (doesn't follow redirects) than requests.request('HEAD', ...)

This is, I believe, undesirable behaviour, and also disagrees with the documentation (http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history).

This also addresses the problem described in kennethreitz/grequests#45


I disagree. If you're using `requests.request` with the verb explicitly you should know that you should be setting `allow_redirects=False`. Note the bug you reference in grequests was never actually acknowledged by Kenneth as a bug.

There is a bug in the documentation, but the bug is that it doesn't say "when using `requests.head`". I see no reason to special case this at this level. There's also nothing in [RFC 7231](http://tools.ietf.org/html/rfc7231#section-4.3.2) to justify not following redirects.

In short I'm :-1: on this fix.


I'm generally in agreement with @sigmavirus24. More generally, if you want redirects on (or off), you should explicitly turn them on (or off). That's the only way to write code that is never going to change under your feet.


Also, regardless of whether this is a good idea or not, this solution only works for the functional API. It is in the wrong place to be even remotely acceptable.


Many thanks for the contribution! Unfortunately, due to some design considerations, we can't accept this at this time. 

Basically, this is intended behavior. Thanks again though :) :cake:
How to set requests logging level?

I just want to set requests logging level up to warning cause I'm not interested in that information. But I don't wish to impact my logging leveling with root logger I set, please help me?


Setting logging levels for a module is very easy:

``` python
import logging
log = logging.getLogger('requests')
log.setLevel(logging.WARNING)
```


In the future, please ask your question on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests)


Ok, thank you so much.—
Sent from my mobile device. Please excuse typos.

On Fri, Jun 20, 2014 at 8:32 PM, Ian Cordasco notifications@github.com
wrote:

> ## In the future, please ask your question on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests)
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/issues/2103#issuecomment-46673063
Is there any methods to get the source TCP port of the requests?

I want to get the source TCP port of the socket in requests I send. 
Any answers will be appreciated.


As the documentation indicates, there is no builtin method for that. If you can grab the connection from urllib3 there might be a way to do it there. In the future, please ask your _questions_ on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests)
can you guys help me out with json, et al?  It's for a good cause.

I need a super easy way to send and receive json.
Nothing fancy.

We want to send/receive json very similar to google geocode api.

I will put kudos on wikispeedia.org

thanks for consideration.


The proper place to ask questions is [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).


no thanks

On Wed, Jun 18, 2014 at 9:23 PM, Ian Cordasco notifications@github.com
wrote:

> The proper place to ask questions is StackOverflow
> https://stackoverflow.com/questions/tagged/python-requests.
> 
> ## 
> 
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2101#issuecomment-46517125
> .
Fix Event Hooks Documentation or the Hook Handling Code

After looking at [this StackOverflow question](http://stackoverflow.com/a/24214101/1953283), I realized that our documentation about [Event Hooks](http://docs.python-requests.org/en/latest/user/advanced/?highlight=hook#event-hooks) is wrong. Specifically,

```
If the function doesn’t return anything, nothing else is effected.
```

It makes sense to me that a callback would not be required to return anything for it to be a successful hook. There's two avenues to fix this: 
1. Make the behaviour match the documentation
2. Make the documentation match the behaviour

Both are easy, but it depends on what we want the behaviour to actually be.


Wait, is this wrong? If a function doesn't return anything it returns `None`, and our code seems to handle that appropriately. What have I missed?


This is my brain on very little (if any) sleep. Sorry all
Default timeout for requests.Session object

The idea is to have a default timeout for each request you make with a session object. 

Example:

``` python
>>> import requests
>>> s = requests.Session()
>>> s.timeout = 1
>>> s.get('https://httpbin.org/delay/10')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/sessions.py", line 473, in get
    return self.request('GET', url, **kwargs)
  File "requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "requests/sessions.py", line 564, in send
    r = adapter.send(request, **kwargs)
  File "requests/adapters.py", line 384, in send
    raise Timeout(e, request=request)
requests.exceptions.Timeout: HTTPSConnectionPool(host='httpbin.org', port=443): Read timed out.
```

I came up with this PR to meet a need of a project based on requests => https://github.com/RedTuna/mysolr/issues/40

Thanks!


Thanks for this!

Unfortunately, this is a feature request that comes up often but that we don't have any real interest in. See #2011 for a longer form justification. For this reason, I'm afraid that we can't accept this pull request. Thanks so much for the work though, and please do keep contributing! :cake:


Thanks for the answer!


Thanks for all the effort you put in @moliware ! Even though we didn't accept the work, we appreciate your effort to make requests better :)
REQUESTS_CA_BUNDLE is not correctly check in ‘requests/sessions.py’

When ‘verify’ is explicitly set to ‘True‘, CA_BUNDLE ENV variables are not checked:
'REQUESTS_CA_BUNDLE' and 'CURL_CA_BUNDLE'

**requests.get('https://my.trusted.server:8080', verify=True)**

requests/sessions.py line 434

# Look for configuration.
**if not verify** and verify is not False:
    verify = os.environ.get('REQUESTS_CA_BUNDLE')

# Curl compatibility.
**if not verify** and verify is not False:
    verify = os.environ.get('CURL_CA_BUNDLE')

You should probably use: 
**if verify is True or (not verify and verify is not False):**


Thanks for this, it's a good spot!

I feel like the correct logic is probably: `if verify is True or verify is None`. I don't believe the other falsy values (empty string, empty list etc.) should cause us to accidentally turn verification on.


I second @Lukasa's far more explicit check for `None`.


I ran into this issue also. It isn't completely clear from [the documentation](http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification) whether this is the intended behavior.

Currently, `verify=True` means "verify with a default set of CAs," correct? Hopefully nobody is relying on that.

For me this issue would be resolved with either the proposed code change or a change to the documentation to make it clear that `REQUESTS_CA_BUNDLE` is not checked when `verify=True` is passed.
handle 308 redirection the same as 301 and 302

308 is also a redirect.  It is the permanent version of 307


Thanks for this!

We should go the whole hog though. HTTP 308 is not 'Resume' anymore, it's 'Permanent Redirect', as defined in [RFC 7238](http://tools.ietf.org/html/rfc7238). Mind making the relevant updates in `status_codes.py`?


Alright, went ahead and made changes to status_codes and everywhere else resume was used.


LGTM. :+1: 


LGTM as well, thanks for this! :+1:

What I've just noticed, however, is that this would be a breaking change. =P Sorry I didn't notice this earlier, but can we re-add the `resume_incomplete` and `resume` options? We'll remove them at 3.0 when we can break other people's code. =)


Put resume_incomplete and resume back in there.  Commented them to be removed for 3.0


Brilliant, thank you so much. I'm happy. =D :+1: :cake:


Thanks @ericfrederich !


:sparkles: :cake: :sparkles:
Update README.rst

Translated some important message to Chinese.


You likely meant to submit this [here](https://github.com/requests/requests-docs-cn).


Thanks a lot

On Wed, Jun 11, 2014 at 12:25 AM, Ian Cordasco notifications@github.com
wrote:

> You likely meant to submit this here
> https://github.com/requests/requests-docs-cn.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2089#issuecomment-45637055
> .
various pep-8 cleanups; remove unsused imports; remove unused variables

A number of PEP-8 related cleanups. Also removing unused imports and unused variables.


@jeffknupp In general this is great and I'm sure we'll be happy to have it. However, please note that we view PEP8 more as guidelines than actual rules (see #1899), and there's some things that Kenneth actively dislikes, so don't get too invested by changing other parts of the code. =)


:sparkles: :cake: :sparkles:

Many thanks.