Repository: scrapy/scrapy

Issue:
empty WARNING message in scrapy.core.downloader.tls (1.1.0rc4/master)
Sometimes I'm getting empty warnings now, on 1.1.0rc4 and master branch.
(at least on rc3 as well)

```
2016-05-07 00:33:46 [scrapy.core.downloader.tls] WARNING: 
2016-05-07 00:33:47 [scrapy.core.downloader.tls] WARNING: 
2016-05-07 00:33:48 [scrapy.core.downloader.tls] WARNING: 
```

It happens in a broad linkcheck crawl; so I couldn't pinpoint what URLs might be responsible for that, at this time. The only other observation so far is, that it doesn't happen on a cache-replayed run (which might be obvious, as there is no TLS there).



Discussion:
@nyov , I'm not sure I understand _"(at least on rc3 as well)"_
you're seeing this with 1.1 RC3 too? or only from 1.1 RC4?
A [`logger.warning()` call was indeed added in RC4](https://github.com/scrapy/scrapy/blob/1.1.0rc4/scrapy/core/downloader/tls.py#L47) for hostname verification failures, but it was not in RC3

> A logger.warning() call was indeed added in RC4

Actually that was exactly why I tested RC3 as an afterthought, and why this puzzled me at the time. But it seems I got the wrong branch or something, it was probably a bit late; I can't reproduce it now.

After another testing round, RC3 is throwing the `ResponseNeverReceived: [<twisted.python.failure.Failure service_identity.exceptions.VerificationError: >]` errors, but not the TLS warning. It's safe to say the warning is actually the one added in RC4.

It's not supposed to be empty, is it? (This is on Python 2.7, btw.)
Maybe it could at least add a small text with the `self._hostnameASCII` info to it, if it is empty.

`OpenSSL.__version__` is `'0.15.1'`.
Oh I see `VerificationError` is from twisted.
`twisted.__version__` is `'15.5.0'`.

When I tested the patch, the warning was not empty, but it may depend on Twisted versions indeed.
I'll have to fool around this a bit...

I am able to reproduce the empty warning with Twisted/PyOpenSSL/service-identity v14:

```
(twisted14) $ scrapy version -v
Scrapy    : 1.1.0rc4
lxml      : 3.6.0.0
libxml2   : 2.9.2
Twisted   : 14.0.0
Python    : 2.7.9 (default, Apr  2 2015, 15:33:21) - [GCC 4.9.2]
pyOpenSSL : 0.14 (OpenSSL 1.0.2d 9 Jul 2015)
Platform  : Linux-4.2.0-36-generic-x86_64-with-Ubuntu-15.10-wily
(twisted14) $ pip freeze
PyDispatcher==2.0.5
Scrapy==1.1.0rc4
Twisted==14.0.0
argparse==1.2.1
attrs==15.2.0
cffi==1.6.0
characteristic==14.3.0
cryptography==1.3.2
cssselect==0.9.1
enum34==1.1.5
idna==2.1
ipaddress==1.0.16
lxml==3.6.0
parsel==1.0.2
pyOpenSSL==0.14
pyasn1==0.1.9
pyasn1-modules==0.0.8
pycparser==2.14
queuelib==1.4.2
service-identity==14.0.0
six==1.10.0
w3lib==1.14.2
wsgiref==0.1.2
zope.interface==4.1.3
(twisted14) $ scrapy shell https://www.knorr-bremsecvs.com
2016-05-11 12:35:31 [scrapy] INFO: Scrapy 1.1.0rc4 started (bot: scrapybot)
2016-05-11 12:35:31 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-05-11 12:35:31 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2016-05-11 12:35:31 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-05-11 12:35:31 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-05-11 12:35:31 [scrapy] INFO: Enabled item pipelines:
[]
2016-05-11 12:35:31 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-05-11 12:35:31 [scrapy] INFO: Spider opened
2016-05-11 12:35:31 [scrapy] WARNING: 
2016-05-11 12:35:31 [scrapy] DEBUG: Retrying <GET https://www.knorr-bremsecvs.com> (failed 1 times): [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionLost'>>]
2016-05-11 12:35:31 [scrapy] WARNING: 
2016-05-11 12:35:31 [scrapy] DEBUG: Retrying <GET https://www.knorr-bremsecvs.com> (failed 2 times): [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionLost'>>]
2016-05-11 12:35:32 [scrapy] WARNING: 
2016-05-11 12:35:32 [scrapy] DEBUG: Gave up retrying <GET https://www.knorr-bremsecvs.com> (failed 3 times): [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionLost'>>]
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/twisted14/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/scrapy/commands/shell.py", line 71, in run
    shell.start(url=url)
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/scrapy/shell.py", line 47, in start
    self.fetch(url, spider)
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/scrapy/shell.py", line 112, in fetch
    reactor, self._schedule, request, spider)
  File "/home/paul/.virtualenvs/twisted14/local/lib/python2.7/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "<string>", line 2, in raiseException
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionLost'>>]
(twisted14) $ 
```

Probably related: [changed in service-identity 16](https://service-identity.readthedocs.io/en/stable/changelog.html#changes):

> Add a **str** method to VerificationError


Resolving patch:
diff --git a/scrapy/core/downloader/tls.py b/scrapy/core/downloader/tls.py
--- a/scrapy/core/downloader/tls.py
+++ b/scrapy/core/downloader/tls.py
@@ -44,7 +44,9 @@ def _identityVerifyingInfoCallback(self, connection, where, ret):
                 try:
                     verifyHostname(connection, self._hostnameASCII)
                 except VerificationError as e:
-                    logger.warning(e)
+                    logger.warning(
+                        'Remote certificate is not valid for hostname "{}"; {}'.format(
+                            self._hostnameASCII, e))
 
 except ImportError:
     # ImportError should not matter for older Twisted versions


Test patch: