Repository: scrapy/scrapy

Issue:
TypeError: 'float' object is not iterable (on Twisted dev + Scrapy dev)
This happens on Twisted trunk and with latest Scrapy master.

```
$ scrapy shell http://localhost:8081/
2016-12-22 12:52:01 [scrapy.utils.log] INFO: Scrapy 1.2.2 started (bot: scrapybot)
2016-12-22 12:52:01 [scrapy.utils.log] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-12-22 12:52:01 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2016-12-22 12:52:01 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-12-22 12:52:01 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-12-22 12:52:01 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2016-12-22 12:52:01 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-22 12:52:01 [scrapy.core.engine] INFO: Spider opened
Traceback (most recent call last):
  File "/Users/rolando/miniconda3/envs/dev/bin/scrapy", line 11, in <module>
    load_entry_point('Scrapy', 'console_scripts', 'scrapy')()
  File "/Users/rolando/Projects/sh/scrapy/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/rolando/Projects/sh/scrapy/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/Users/rolando/Projects/sh/scrapy/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/Users/rolando/Projects/sh/scrapy/scrapy/commands/shell.py", line 71, in run
    shell.start(url=url)
  File "/Users/rolando/Projects/sh/scrapy/scrapy/shell.py", line 47, in start
    self.fetch(url, spider)
  File "/Users/rolando/Projects/sh/scrapy/scrapy/shell.py", line 112, in fetch
    reactor, self._schedule, request, spider)
  File "/Users/rolando/Projects/gh/twisted/src/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "/Users/rolando/Projects/gh/twisted/src/twisted/python/failure.py", line 372, in raiseException
    raise self.value.with_traceback(self.tb)
TypeError: 'float' object is not iterable
```

```
(Pdb) w
  /Users/rolando/Projects/sh/scrapy/scrapy/utils/defer.py(45)mustbe_deferred()
-> result = f(*args, **kw)
  /Users/rolando/Projects/sh/scrapy/scrapy/core/downloader/handlers/__init__.py(65)download_request()
-> return handler.download_request(request, spider)
  /Users/rolando/Projects/sh/scrapy/scrapy/core/downloader/handlers/http11.py(61)download_request()
-> return agent.download_request(request)
  /Users/rolando/Projects/sh/scrapy/scrapy/core/downloader/handlers/http11.py(286)download_request()
-> method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  /Users/rolando/Projects/gh/twisted/src/twisted/web/client.py(1601)request()
-> parsedURI.originForm)
  /Users/rolando/Projects/gh/twisted/src/twisted/web/client.py(1378)_requestWithEndpoint()
-> d = self._pool.getConnection(key, endpoint)
  /Users/rolando/Projects/gh/twisted/src/twisted/web/client.py(1264)getConnection()
-> return self._newConnection(key, endpoint)
  /Users/rolando/Projects/gh/twisted/src/twisted/web/client.py(1276)_newConnection()
-> return endpoint.connect(factory)
  /Users/rolando/Projects/gh/twisted/src/twisted/internet/endpoints.py(779)connect()
-> EndpointReceiver, self._hostText, portNumber=self._port
  /Users/rolando/Projects/gh/twisted/src/twisted/internet/_resolver.py(174)resolveHostName()
-> onAddress = self._simpleResolver.getHostByName(hostName)
  /Users/rolando/Projects/sh/scrapy/scrapy/resolver.py(21)getHostByName()
-> d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
> /Users/rolando/Projects/gh/twisted/src/twisted/internet/base.py(276)getHostByName()
-> timeoutDelay = sum(timeout)
```

After digging, I found out that the addition of `DNS_TIMEOUT` was not effective at all: https://github.com/scrapy/scrapy/commit/85aa3c7596c6e9c66daaa5503faadd03a16e1d59#diff-92d881d6568986904888f43c885240e2L13

Previously, on Twisted <=16.6.0, the method `getHostByName` was always called with a default timeout: https://github.com/twisted/twisted/blob/twisted-16.6.0/src/twisted/internet/base.py#L565-L573

But now, on Twisted trunk, the method is called without a timeout parameter: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/_resolver.py#L174

This makes the caching resolver to use the default value `timeout=60.0` which causes the error: https://github.com/twisted/twisted/blob/twisted-16.6.0/src/twisted/internet/base.py#L259-L268


Discussion:
Given `DNS_TIMEOUT` was not used at all, perhaps a straightforward fix is to make it a tuple and compatible with what twisted expects:

> timeout: Default number of seconds after which to reissue the query. When the last timeout expires, the query is considered failed. (type: Sequence of int )
Related: https://github.com/scrapy/scrapy/pull/2462
Have you found a way to work around this? I'm having the exact same error as you have. I've tried downgrading scrapy to version 1.2, but this doens't help.
@tituskex , are you using Twisted>=16.7 (I see there's a RC1 on PyPI)?
It should work if you downgrade Twisted to 16.6, until we find a fix.
So, the traceback here is because technically Scrapy is violating the formal interface in Twisted (it says you have to provide your own default value for `timeout` and scrapy's custom resolver didn't).

We _could_ go ahead and backport a fix into the release that passes this value so that scrapy continues to work - would that be valuable to you, or would you be OK fixing the interface implementation to be correct?

(BTW, for future testing, Zope Interface has [some tools](http://zopeinterface.readthedocs.io/en/latest/verify.html) for verifying that interfaces match their implementations at least a little automatically.)
(Oops, on second thought, `zope.interface.verify` wouldn't have caught this particular issue, since a default was provided but it was just the wrong _type_.)
Thanks @glyph. I would say that Scrapy needs to fix it and provide a default timeout value.

Resolving patch:
diff --git a/scrapy/resolver.py b/scrapy/resolver.py
--- a/scrapy/resolver.py
+++ b/scrapy/resolver.py
@@ -16,8 +16,11 @@ def __init__(self, reactor, cache_size, timeout):
     def getHostByName(self, name, timeout=None):
         if name in dnscache:
             return defer.succeed(dnscache[name])
-        if not timeout:
-            timeout = self.timeout
+        # in Twisted<=16.6, getHostByName() is always called with
+        # a default timeout of 60s (actually passed as (1, 3, 11, 45) tuple),
+        # so the input argument above is simply overridden
+        # to enforce Scrapy's DNS_TIMEOUT setting's value
+        timeout = (self.timeout,)
         d = super(CachingThreadedResolver, self).getHostByName(name, timeout)
         d.addCallback(self._cache_result, name)
         return d


Test patch: