Inconsistent ConnectionError Behavior

Looking for guidance to solve the following problem:

On dev env / MacOSX, the following:

``` python
url = "http://insurance-guide.netfirms.com"
response = requests.get(url, timeout=90, verify=False, stream=True)
```

results in:

``` python
ConnectionError: HTTPConnectionPool(host='insurance-guide.netfirms.com', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno 8] nodename nor servname provided, or not known)
```

on prod env / Centos 6, the exact same code never timesout and just stays stuck forever...

both env are python 2.7.5 and requests 1.2.3

what could be the cause of the diff behavior on prod vs dev?


It should be noted that the `ConnectionError` is raised without the parameters you specify:

``` python
response = requests.get('http://insurance-guide.netfirms.com')
```

produces this exception as reliably as your example does. Oddly enough, I get a different reason for the exception though:

```
requests.exceptions.ConnectionError: HTTPConnectionPool(host='insurance-guide.netfirms.com', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno -3] Temporary failure in name resolution)
[Errno -3] Temporary failure in name resolution)
```

I pulled the relevant bit into the second line. 

Why this doesn't happen on Centos is an excellent question. We don't currently test on any Centos (or similar) systems so we can't really say. Were I you, I would set the timeout considerably lower. Try less than 1 second (say 0.1 or 0.01).

All of that advice aside, this seems more like you're looking for the answer to a question that I doubt Kenneth, Cory or I could answer without getting our hands on a Centos system. You might have better luck on [StackOverflow](http://stackoverflow.com/questions/tagged/python-requests). I would use both the Centos and Requests tags. I can also promote it from there if it doesn't receive enough attention.


One thought that just came to me, however, is that I'm having name resolution issues with that URL. On the Centos system (presuming you have shell access to it), what is the result of

```
dig netfirms.com
```


dig result is consistent across both systems:
on centos:
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.17.rc1.el6_4.4 <<>> netfirms.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46225
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;netfirms.com. IN A

;; ANSWER SECTION:
netfirms.com. 2756 IN A 65.254.227.16

;; Query time: 10 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Sep 24 02:26:25 2013
;; MSG SIZE  rcvd: 46

on macosx:
; <<>> DiG 9.6-ESV-R4-P3 <<>> netfirms.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2977
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;netfirms.com. IN A

;; ANSWER SECTION:
netfirms.com. 3600 IN A 65.254.227.16

;; Query time: 142 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Mon Sep 23 19:26:13 2013
;; MSG SIZE  rcvd: 46

Fred~

On Mon, Sep 23, 2013 at 7:14 PM, Ian Cordasco notifications@github.comwrote:

> One thought that just came to me, however, is that I'm having name
> resolution issues with that URL. On the Centos system (presuming you have
> shell access to it), what is the result of
> 
> dig netfirms.com
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1617#issuecomment-24970370
> .


tried with lower timeout:
on centos:
url = "http://insurance-guide.netfirms.com"
response = requests.get(url, timeout=0.1, verify=False, stream=True)

> > requests.exceptions.Timeout: HTTPConnectionPool(host='
> > insurance-guide.netfirms.com', port=80): Request timed out. (timeout=0.1)

on macosx:
same code

> > requests.exceptions.ConnectionError: HTTPConnectionPool(host='
> > insurance-guide.netfirms.com', port=80): Max retries exceeded with url: /
> > (Caused by <class 'socket.gaierror'>: [Errno 8] nodename nor servname
> > provided, or not known)

The bottomline tho, is that I need this call to timeout properly on centos,
why there is a diff is not that interesting to me... ;-)
and I need the timeout to stay high, ie 90, coz I have sites that are
sometimes backed up and I want to wait that long...

is this related to?
https://github.com/kennethreitz/requests/issues/1577

Fred~

On Mon, Sep 23, 2013 at 7:12 PM, Ian Cordasco notifications@github.comwrote:

> It should be noted that the ConnectionError is raised without the
> parameters you specify:
> 
> response = requests.get('http://insurance-guide.netfirms.com')
> 
> produces this exception as reliably as your example does. Oddly enough, I
> get a different reason for the exception though:
> 
> requests.exceptions.ConnectionError: HTTPConnectionPool(host='insurance-guide.netfirms.com', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno -3] Temporary failure in name resolution)
> [Errno -3] Temporary failure in name resolution)
> 
> I pulled the relevant bit into the second line.
> 
> Why this doesn't happen on Centos is an excellent question. We don't
> currently test on any Centos (or similar) systems so we can't really say.
> Were I you, I would set the timeout considerably lower. Try less than 1
> second (say 0.1 or 0.01).
> 
> All of that advice aside, this seems more like you're looking for the
> answer to a question that I doubt Kenneth, Cory or I could answer without
> getting our hands on a Centos system. You might have better luck on
> StackOverflow http://stackoverflow.com/questions/tagged/python-requests.
> I would use both the Centos and Requests tags. I can also promote it from
> there if it doesn't receive enough attention.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1617#issuecomment-24970344
> .


@laruellef yeah I was just about to go looking for that issue. It almost certainly seems related to #1577. We're still waiting on the urllib3 PR that @Lukasa alluded to in that issue so I'm going to close this while we wait for that to work itself out so we can provide that interface to you.

That said, if you run into timeout issues again, and setting it lower doesn't resolve your issue, then open an issue. Otherwise it might be safe to assume it's related to #1577.


oh, but wait,
the proposed work around is not working here... :-(
ie
https://github.com/kennethreitz/requests/issues/1577#issuecomment-23794576

Fred~

On Mon, Sep 23, 2013 at 7:46 PM, Ian Cordasco notifications@github.comwrote:

> @laruellef https://github.com/laruellef yeah I was just about to go
> looking for that issue. It almost certainly seems related to #1577https://github.com/kennethreitz/requests/issues/1577.
> We're still waiting on the urllib3 PR that @Lukasahttps://github.com/Lukasaalluded to in that issue so I'm going to close this while we wait for that
> to work itself out so we can provide that interface to you.
> 
> That said, if you run into timeout issues again, and setting it lower
> doesn't resolve your issue, then open an issue. Otherwise it might be safe
> to assume it's related to #1577https://github.com/kennethreitz/requests/issues/1577
> .
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1617#issuecomment-24971338
> .


I never said it would. One solution does not fit all related problems.

In this case, I expect that the connection is being made but the server is not sending any data. Until it starts, I think even with `stream=True` requests will not return. You're expecting `stream=True` to make requests behave asynchronously when in order to do so you need erequests or grequests or requests-futures.


Ok, so what is the work around for this then?
The timeout param should really be universal and more robust, no?
Or add a param as suggested in other case,
But please capture all cases...

Sent from my iPad

On Sep 23, 2013, at 8:13 PM, Ian Cordasco notifications@github.com wrote:

> I never said it would. One solution does not fit all related problems.
> 
> In this case, I expect that the connection is being made but the server is not sending any data. Until it starts, I think even with stream=True requests will not return. You're expecting stream=True to make requests behave asynchronously when in order to do so you need erequests or grequests or requests-futures.
> 
> —
> Reply to this email directly or view it on GitHub.


Hum, we have prod processes that are currently on hold because of his problem,
Is there any someone who could provide a work around?
Are there other good choices to replace requests?, coz this timeout issue is becoming a real problem for us... :-(

Per http://docs.python-requests.org/en/latest/
requests "makes your integration with web services seamless", 
the number of work arounds we have to put in place to handle the lack of a robust timeout is not so seamless ... :-(

Sent from my iPad

On Sep 23, 2013, at 8:13 PM, Ian Cordasco notifications@github.com wrote:

> I never said it would. One solution does not fit all related problems.
> 
> In this case, I expect that the connection is being made but the server is not sending any data. Until it starts, I think even with stream=True requests will not return. You're expecting stream=True to make requests behave asynchronously when in order to do so you need erequests or grequests or requests-futures.
> 
> —
> Reply to this email directly or view it on GitHub.


The urllib3 timeout parameter has been implemented. I'm waiting on Kenneth to provide an idea of what kind of API he wants before I go ahead and plumb some part of it through.