avoid reverse DNS lookup?

I have a url that I can load instantly in my browser.  However, when I load it from Python (I tried urllib2 also), I get about a 5 second delay.  The same delay happens if I use curl.

I did a little bit of profiling, and it looks like it's getting hung up in a getaddrinfo() call, which I'm not sure why it needs to make, but I would like to bypass.  I can bypass it if I access the url with an IP instead of hostname, but for maintainability, I would rather not do this in the long term.

Wondering if you know why some of these libraries are doing the reverse DNS lookup, and if you would consider providing an option to bypass it?


This doesn't sound like a reverse DNS issue. `getaddrinfo()` is a standard DNS resolution function, so if you're using a hostname (i.e. not a raw IP address), you need to make this call to resolve the URL.

If I had to guess, I'd say your browser implements a DNS cache, so it can avoid the need to resolve the hostname most of the time.

If this bothers you, you could attempt to implement a DNS cache in your Python code. Probably wouldn't be too hard. =)


Hmm.  Ok.  Now I'm confused, because the forward lookup should be very fast, since it comes from my /etc/hosts file, and should not even need to be cached.  Anyway, here is the stack trace, in case it's useful

  File "./find_references.py", line 68, in findExistingRefs
    r = requests.get(url)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/api.py", line 54, in get
    return request('get', url, *_kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/safe_mode.py", line 37, in wrapped
    return function(method, url, *_kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/api.py", line 42, in request
    return s.request(method=method, url=url, *_kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/sessions.py", line 230, in request
    r.send(prefetch=prefetch)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/models.py", line 604, in send
    timeout=self.timeout,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 384, in urlopen
    body=body, headers=headers)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.13.1-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 254, in _make_request
    conn.request(method, url, *_httplib_request_kw)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 958, in request
    self._send_request(method, url, body, headers)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 992, in _send_request
    self.endheaders(body)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 954, in endheaders
    self._send_output(message_body)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 814, in _send_output
    self.send(msg)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 776, in send
    self.connect()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 757, in connect
    self.timeout, self.source_address)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
KeyboardInterrupt


I had a similar issue a while back with grequests (previously requests.async) as a result of http://code.google.com/p/gevent/issues/detail?id=21 . Aside from that though, I can't see why this should be happening. requests relies on urrlib3->httplib->socket for getaddrinfo. Socket definitely respects /etc/hosts.


Can you try making the raw socket call yourself? For instance, if I try to lookup the nonexistent domain name `test.lukasa.co.uk`, by calling `socket.getaddrinfo('test.lukasa.co.uk', 'http')`, I timeout unless I put that domain in my hosts file. Can you check whether you can reproduce this behaviour?


@jric: Is this still a problem?


Closed for inactivity.


I just ran into this problem on a new machine. `socket.getaddrinfo` takes several seconds to resolve a name from `/etc/hosts`. I then changed the `/etc/hosts` entries from `127.0.0.1` to `::1` and resolution was instant. It appears `getaddrinfo` is attempting IPv6 lookup _before_ consulting the hosts file for IPv4. FWIW, this isn't a requests bug.