Support DNS roundrobin with failure detection

As we move into the cloud world with multiple subnets failing over transparently is becoming more and more of a pain.

Many database systems for instance support the DNS entry for the server having multiple IP addresses in a set. The client is smart enough to know which IP is up and listening and failover to the other quickly. 

HTTP doesn't come with such facilities, but Requests is a perfect place to add it.

Imagine website.com having a record set of 10.10.10.5/24 and 10.10.11.5/25. Imagine these are load balancers such as nginx or haproxy sitting in two distinct subnets perhaps even in two data centers.

The client is going to make web service requests of website.com. If 10.10.11.5/24 goes down the clients should make requests to 10.10.10.5/24 only.

This is more a request for peoples opinions on whether this should be added to the framework and where. Typically this kind of functionality is accomplished transparently to the web clients using a "smart" dns server with health checks among other things. IMHO I feel like that just creates complexities that could be done better and more importantly faster at the client level.

Thanks,
Eric


If you specify a connection timeout and Requests can't connect to the first host specified via a DNS lookup, it will try the next host. 

This request will take 18 seconds for example: 

``` python
start = time.time()
try:
  requests.get('http://google.com:81', 
               timeout=3)
except ConnectTimeout: pass
print time.time() - start 
```

(via [my slides](https://kev.inburke.com/slides/reliable-http/#server-unreachable))

This is because google.com has 6 IP addresses listed. Note, this functionality is super below Requests, it's at the OS level I am pretty sure.

Caching the good DNS host is another matter; as you mentioned you might want to try something like requests talking to haproxy locally (this is what we used at Twilio for example). HAProxy will do health checks and stop sending traffic to a down host.


This functionality is not that far below requests, it's in urllib3, [here](https://github.com/shazow/urllib3/blob/master/urllib3/util/connection.py#L46-L90).

We effectively get this logic because we attempt to connect to each result returned by the DNS server, and once we do we do our best to continue to use that connection. If we can't, we'll attempt to reconnect, including trying other IP addresses. This is important because setting multiple A records for your domain name is commonly used to attempt to balance load.

Stepping further ahead, I don't know what this means:

> Imagine website.com having a record set of 10.10.10.5/24 and 10.10.11.5/25.

That's just not how DNS works, unless the domain name actually has 384 A records (256 for the /24 and 128 for the /25). Additionally, your addresses are weird, given that they'd ordinarily be represented 10.10.10.0/24 and 10.10.11.0/25 (neither of the 5s have any high bits that would be frozen by their mask).

Regardless, the summary is that we currently have this functionality. I can see some advantage in trying to short-circuit some of the DNS logic in userspace, but it's a fraught approach that can be easy to get wrong.


/25 should have been /24. Just denoting that I have boxes in 2 subnets. The point there being I'm trying to make my front end boxes use the "closest(s)" servers first. 

Maybe change this to a documentation request(or do it myself). I don't see anywhere in the docs that talk about A record sets with multiple IPs.

Just to clarify is Requests looping through the list of IP addresses coming from DNS or is Urllib3 doing it, that urllib3 snippet above doesn't look like it is looping.

Thanks!!!!


@kevinburke thanks for your response too. I'm actually already using timeout in my code, but I think that also controls how long the request can take. We have some requests that take quite a while like 25 seconds (not great form) based on some app requirements we have.

I'll see how hard it might be to add a connection_timeout option and maybe put in a request or pull request for that. 

:update looking at the code I immediately see something about connection timeout in a tuple of some sort.) So hopefully that works.

Thanks!!!!


urllib3 does it: the loop is `for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)`