Timeout for connecting to proxy

I provided a bogus proxy, set a very short timeout and expected requests to fail very fast. Unfortunately the opposite happened. Is this behavior intentional? As far as I can tell there is no way to set a timeout for the proxy connection.

``` Python
time python -c "import requests; print requests.__version__; requests.get('https://google.com', timeout=0.01, proxies={'https':'https://1.1.1.1'})"
2.4.3
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 413, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(110, 'Connection timed out')))

real    2m7.350s
user    0m0.090s
sys 0m0.030s
```


That's an interesting question. @kevinburke, got any idea?


A guess without looking at anything is that the ProxyManager doesn't apply the timeout to connections to the proxy.


Will take a look when I get an computer

On Thursday, November 13, 2014, Ian Cordasco notifications@github.com
wrote:

> A guess without looking at anything is that the ProxyManager doesn't apply
> the timeout to connections to the proxy.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2336#issuecomment-62920205
> .

## 

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com


Hmmm seems to time out right away if you make the request over HTTP. here is my investigation so far.

https://gist.github.com/kevinburke/e862b5d6a56a6d99de9d

About to start reading code.


So... this is an error. 

The gist of it is, with HTTPS connections connect() gets called earlier in the urlopen() process than with a HTTP connection. Specifically, for HTTPS _new_conn will call _prepare_conn, which calls connect() on the socket, where for HTTP, the connect() call is delayed all the way until httplib.request.

urllib3 sets the connection timeout until after calling _get_conn, but before httplib.request, which explains why the timeout isn't set.

The obvious question is why isn't this an issue when you're not using a proxy? Well, prepare_conn only attempts the connect() if you're using a proxy, with the following note:

```
        # Establish tunnel connection early, because otherwise httplib
        # would improperly set Host: header to proxy's IP:port.
        conn.connect()
```

I'll discuss with @shazow and see if we can figure out a good course of action.


Added https://github.com/shazow/urllib3/issues/504. Maybe just leave this open until we come to a resolution over there


I also encoutered a deadlock problem caused by this similar timeout issue.

What I did is sending a GET request to some site (like http://jsonip.com) using an HTTP proxy to get proxy's public ip address, the code is like:

```
s = requests.Session()
s.proxies = {'http': 'http://xx.xx.xx.xx:xxx', 'https': 'http://xx.xx.xx.xx:xxx'}
r = s.get('http://jsonip.com', timeout=5)
r.raise_for_status()
...
```

After deadlock, the stack trace of the problmatic thread is (other threads are waiting for its completion):

```
File: "C:\Python27\lib\threading.py", line 783, in __bootstrap
  self.__bootstrap_inner()
File: "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
  self.run()
File: "C:\Python27\lib\threading.py", line 763, in run
  self.__target(*self.__args, **self.__kwargs)
File: "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 73, in _worker
  work_item.run()
File: "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 61, in run
  result = self.fn(*self.args, **self.kwargs)
File: "E:\zbb\projects\crawler\new\zbb-crawler\zbb_proxymanager\app\proxy_filter.py", line 45, in test
  ip = self.public_ip.get(host)
File: "E:\zbb\projects\crawler\new\zbb-crawler\zbb_proxymanager\utils.py", line 91, in get
  ip = func(session, timeout)
File: "E:\zbb\projects\crawler\new\zbb-crawler\zbb_proxymanager\get_ip.py", line 43, in get_ip_4
  r = session.get('http://jsonip.com', timeout=timeout)
File: "C:\Python27\lib\site-packages\requests\sessions.py", line 473, in get
  return self.request('GET', url, **kwargs)
File: "C:\Python27\lib\site-packages\requests\sessions.py", line 461, in request
  resp = self.send(prep, **send_kwargs)
File: "C:\Python27\lib\site-packages\requests\sessions.py", line 599, in send
  history = [resp for resp in gen] if allow_redirects else []
File: "C:\Python27\lib\site-packages\requests\sessions.py", line 192, in resolve_redirects
  allow_redirects=False,
File: "C:\Python27\lib\site-packages\requests\sessions.py", line 573, in send
  r = adapter.send(request, **kwargs)
File: "C:\Python27\lib\site-packages\requests\adapters.py", line 370, in send
  timeout=timeout
File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 513, in urlopen
  conn = self._get_conn(timeout=pool_timeout)
File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 231, in _get_conn
  return conn or self._new_conn()
File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 717, in _new_conn
  return self._prepare_conn(conn)
File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 690, in _prepare_conn
  conn.connect()
File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connection.py", line 217, in connect
  self._tunnel()
File: "C:\Python27\lib\httplib.py", line 752, in _tunnel
  (version, code, message) = response._read_status()
File: "C:\Python27\lib\httplib.py", line 365, in _read_status
  line = self.fp.readline(_MAXLINE + 1)
File: "C:\Python27\lib\socket.py", line 476, in readline
  data = self._sock.recv(self._rbufsize)
```

I think it's the same bug discussed in this issue, right? The version of Requests is 2.5.0.


We haven't shipped the fix for this yet. Ping @shazow and @sigmavirus24

On Wednesday, December 3, 2014, Zhang Yibin notifications@github.com
wrote:

> I also encoutered a deadlock problem caused by this similar timeout issue.
> 
> What I did is sending a GET request to some site (like http://jsonip.com)
> using an HTTP proxy to get proxy's public ip address, the code is like:
> 
> s = requests.Session()
> s.proxies = {'http': 'http://xx.xx.xx.xx:xxx', 'https': 'http://xx.xx.xx.xx:xxx'}
> r = s.get('http://jsonip.com', timeout=5)
> r.raise_for_status()
> ...
> 
> After deadlock, the stack trace of the problmatic thread is (other threads
> are waiting for its completion):
> 
> File: "C:\Python27\lib\threading.py", line 783, in __bootstrap
>   self.__bootstrap_inner()
> File: "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
>   self.run()
> File: "C:\Python27\lib\threading.py", line 763, in run
>   self.__target(_self.__args, *_self.__kwargs)
> File: "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 73, in _worker
>   work_item.run()
> File: "C:\Python27\lib\site-packages\concurrent\futures\thread.py", line 61, in run
>   result = self.fn(_self.args, *_self.kwargs)
> File: "E:\zbb\projects\crawler\new\zbb-crawler\zbb_proxymanager\app\proxy_filter.py", line 45, in test
>   ip = self.public_ip.get(host)
> File: "E:\zbb\projects\crawler\new\zbb-crawler\zbb_proxymanager\utils.py", line 91, in get
>   ip = func(session, timeout)
> File: "E:\zbb\projects\crawler\new\zbb-crawler\zbb_proxymanager\get_ip.py", line 43, in get_ip_4
>   r = session.get('http://jsonip.com', timeout=timeout)
> File: "C:\Python27\lib\site-packages\requests\sessions.py", line 473, in get
>   return self.request('GET', url, *_kwargs)
> File: "C:\Python27\lib\site-packages\requests\sessions.py", line 461, in request
>   resp = self.send(prep, *_send_kwargs)
> File: "C:\Python27\lib\site-packages\requests\sessions.py", line 599, in send
>   history = [resp for resp in gen] if allow_redirects else []
> File: "C:\Python27\lib\site-packages\requests\sessions.py", line 192, in resolve_redirects
>   allow_redirects=False,
> File: "C:\Python27\lib\site-packages\requests\sessions.py", line 573, in send
>   r = adapter.send(request, **kwargs)
> File: "C:\Python27\lib\site-packages\requests\adapters.py", line 370, in send
>   timeout=timeout
> File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 513, in urlopen
>   conn = self._get_conn(timeout=pool_timeout)
> File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 231, in _get_conn
>   return conn or self._new_conn()
> File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 717, in _new_conn
>   return self._prepare_conn(conn)
> File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 690, in _prepare_conn
>   conn.connect()
> File: "C:\Python27\lib\site-packages\requests\packages\urllib3\connection.py", line 217, in connect
>   self._tunnel()
> File: "C:\Python27\lib\httplib.py", line 752, in _tunnel
>   (version, code, message) = response._read_status()
> File: "C:\Python27\lib\httplib.py", line 365, in _read_status
>   line = self.fp.readline(_MAXLINE + 1)
> File: "C:\Python27\lib\socket.py", line 476, in readline
>   data = self._sock.recv(self._rbufsize)
> 
> I think it's the same bug discussed in this issue, right? The version of
> Requests is 2.5.0.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2336#issuecomment-65412519
> .

## 

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com


In my case, I didn't use proxy to visit any HTTPS site. Does it fix my problem?

UPDATE: Now I have to use "socket.setdefaulttimeout(30.0)" everytime I use requests to access proxy, sigh...


The urllib3 fix was merged before the v2.5.1 release.

With HTTPS proxy:

```
time ./bin/python -c "import requests; print requests.__version__; requests.get('https://google.com', timeout=0.01, proxies={'https':'https://1.1.1.1'})"
2.5.1
```

```
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "app_main.py", line 562, in run_it
  File "<string>", line 1, in <module>
  File "/tmp/req/site-packages/requests/api.py", line 65, in get
    return request('get', url, **kwargs)
  File "/tmp/req/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/tmp/req/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/req/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/req/site-packages/requests/adapters.py", line 424, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(110, 'Connection timed out')))
```

```
real    2m7.502s
user    0m0.192s
sys 0m0.025s
```

And with the HTTP proxy:

```
time ./bin/python -c "import requests; print requests.__version__; requests.get('https://google.com', timeout=0.01, proxies={'http':'http://1.1.1.1'})"
2.5.1
```

```
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "app_main.py", line 562, in run_it
  File "<string>", line 1, in <module>
  File "/tmp/req/site-packages/requests/api.py", line 65, in get
    return request('get', url, **kwargs)
  File "/tmp/req/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/tmp/req/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/req/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/req/site-packages/requests/adapters.py", line 433, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='google.com', port=443): Read timed out. (read timeout=0.01)
```

```
real    0m0.281s
user    0m0.201s
sys 0m0.024s
```


Thanks for your work all!


Erm, IMHO the problem is not fixed :)


In what sense?


The request with HTTPS proxy is still not timing out as set by the parameter.


Apologies, I don't know how I missed that. You're quite right.


I think we need to pull urllib3 in before the next release to fix this. 2.5.1 was released mostly with the intention of fixing a bug with HTTPDigestAuth. @glaslos if we pull in a more modern copy of urllib3, can you test master for us to see if it fixes the problem? If it does I'm comfortable cutting 2.5.2


Sounds good to me.


Master has an up-to-date copy of urllib3 @glaslos. Test away


@kevinburke could you confirm that your changes to urllib3 are in the requests master branch?


urllib3 is still ignoring the timeout parameter if you use an HTTPS proxy:

```
(req)lrist@laptop-lrist /tmp/req $ cd requests/ && git rev-parse --short HEAD && cd ..
673bd6a
```

```
(req)lrist@laptop-lrist /tmp/req $ time ./bin/python -c "import requests; print requests.__version__; print requests.__file__; requests.get('https://google.com', timeout=0.01, proxies={'https':'https://1.1.1.1'})"
2.5.1
/tmp/req/site-packages/requests-2.5.1-py2.7.egg/requests/__init__.py
```

It takes more than 2 seconds to timeout with `timeout=0.01`

``` python
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "app_main.py", line 562, in run_it
  File "<string>", line 1, in <module>
  File "/tmp/req/site-packages/requests-2.5.1-py2.7.egg/requests/api.py", line 65, in get
    return request('get', url, **kwargs)
  File "/tmp/req/site-packages/requests-2.5.1-py2.7.egg/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/tmp/req/site-packages/requests-2.5.1-py2.7.egg/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/req/site-packages/requests-2.5.1-py2.7.egg/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/req/site-packages/requests-2.5.1-py2.7.egg/requests/adapters.py", line 424, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(110, 'Connection timed out')))
```

Timings:

```
real    2m7.452s
user    0m0.194s
sys 0m0.024s
```


Any progress on this issue?


FYI, requests doesn't support connections to HTTPS proxies. See https://github.com/kennethreitz/requests/issues/1622 and https://github.com/kennethreitz/requests/issues/2386


Ah I see. Still, urllib3 is not handling the timeout parameter.


> Ah I see. Still, urllib3 is not handling the timeout parameter.

This works for me on master, should be fixed with https://github.com/kennethreitz/requests/commit/d2d576b6b1101e2871c82f63adf2c2b534c2dabc


I confirmed this is fixed in master. Thanks!


not work with requests 2.10.0,

a fake proxies, this work fine.after 0.1s timeout.

```
requests.get("http://baidu.com",timeout=0.1,proxies={'http':'http://120.0.0.1:80'})
```

this code with https proxies , get no response, just hang ther.

```
requests.get("http://baidu.com",timeout=0.1,proxies={'http':'http://120.0.0.1:80'})
```


@Petelin The code in both cases is identical: there is no difference. Please note, the timeout is _per socket event_: it is possible that you are hanging elsewhere in the stack.


@Lukasa my bad 

```
requests.get("https://baidu.com",timeout=0.1,proxies={'https':'http://120.0.0.1:80'})
```

use a fake https proxie to access real https://baidu.com,it will just hang there.

i do this in ipython with no context.


I don't see the same behaviour: I see an instant timeout. It'd be good to know where you're blocking: what system are you running on?


it's Ubuntu 14.04.2 LTS   , 3.13.0-32-generic , all two version python  not work.

```
╰─$ ipython
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
Type "copyright", "credits" or "license" for more information.

IPython 4.1.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import requests;requests.get("https://baidu.com",timeout=0.1,proxies={'https':'http://120.0.0.1:80'})


```

```
╰─$ python2.7                                                                                                                 1 ↵
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests;requests.get("https://baidu.com",timeout=0.1,proxies={'https':'http://120.0.0.1:80'})


```

but on my mac ipython5 python3 it works too.