Repository: scrapy/scrapy

Issue:
fetch errors on some https sites
$ scrapy fetch 'https://flixbus.de'

```
2014-12-12 10:01:50+0100 [scrapy] INFO: Scrapy 0.24.4 started (bot: scrapybot)
2014-12-12 10:01:50+0100 [scrapy] INFO: Optional features available: ssl, http11
2014-12-12 10:01:50+0100 [scrapy] INFO: Overridden settings: {}
2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled extensions: LogStats, TelnetConsole,  CloseSpider, WebService, CoreStats, SpiderState
2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-12-12 10:01:50+0100 [scrapy] INFO: Enabled item pipelines: 
2014-12-12 10:01:50+0100 [default] INFO: Spider opened
2014-12-12 10:01:50+0100 [default] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-12-12 10:01:50+0100 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2014-12-12 10:01:50+0100 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2014-12-12 10:01:50+0100 [default] DEBUG: Retrying <GET https://flixbus.de> (failed 1 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
2014-12-12 10:01:50+0100 [default] DEBUG: Retrying <GET https://flixbus.de> (failed 2 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
2014-12-12 10:01:50+0100 [default] DEBUG: Gave up retrying <GET https://flixbus.de> (failed 3 times): [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
2014-12-12 10:01:50+0100 [default] ERROR: Error downloading <GET https://flixbus.de>: [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]
2014-12-12 10:01:50+0100 [default] INFO: Closing spider (finished)
2014-12-12 10:01:50+0100 [default] INFO: Dumping Scrapy stats:
    {'downloader/exception_count': 3,
     'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 3,
     'downloader/request_bytes': 627,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 12, 12, 9, 1, 50, 463722),
     'log_count/DEBUG': 5,
     'log_count/ERROR': 1,
     'log_count/INFO': 7,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2014, 12, 12, 9, 1, 50, 288873)}
2014-12-12 10:01:50+0100 [default] INFO: Spider closed (finished)
```

$ pip freeze

```
Scrapy==0.24.4
Twisted==14.0.2
argparse==1.2.1
cffi==0.8.6
characteristic==14.2.0
cryptography==0.6.1
cssselect==0.9.1
distribute==0.7.3
lxml==3.4.1
pyOpenSSL==0.14
pyasn1==0.1.7
pyasn1-modules==0.0.5
pycparser==2.10
queuelib==1.2.2
service-identity==14.0.0
six==1.8.0
w3lib==1.10.0
wsgiref==0.1.2
zope.interface==4.1.1
```

Same error on scrapinghub ;)



Discussion:
Quick shot: Something to do with SSL implementation changes due to the SSL bug Google published? Several (most?) providers stopped supporting SSL 3.0. Maybe this is related, but don't know and haven't digged in deeper.

@aufziehvogel yes that was my first thought.
flixbus.de doesn't support SSLv2 nor SSLv3, only TLSv1+.
Additional a question, does scrapy/twisted-webclient support SNI?
The Flixbus site is on amazon cloudfront and expected SNI-support to present the correct certificate.

@toolking By default we use TLSv1, but you can change it by extending `ContextFactory`.
[`ContextFactory`](https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/contextfactory.py#L13)
[`ContextFactorySetting`](https://github.com/scrapy/scrapy/blob/master/scrapy/settings/default_settings.py#L75)

See also #82 

@AlanChien @toolking 
You can fix by extending ContextFactory:

```
from OpenSSL import SSL
from twisted.internet.ssl import ClientContextFactory
from twisted.internet._sslverify import ClientTLSOptions
from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
class CustomClientContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = ClientContextFactory.getContext(self)
        # Enable all workarounds to SSL bugs as documented by
        # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html
        ctx.set_options(SSL.OP_ALL)
        if hostname:
            ClientTLSOptions(hostname, ctx)
        return ctx
```

(Thanks to @darkrho for the fix)

As an additional note, Twisted [handles this issue](https://github.com/twisted/twisted/blob/trunk/twisted/internet/_sslverify.py#L38) for `twisted.web.client.Agent` by using this class [BrowserLikePolicyForHTTPS](https://github.com/twisted/twisted/blob/trunk/twisted/web/client.py#L846).

@AlanChien Did you set up the change correctly?

Because it works for me:

```
$ scrapy shell https://app.gomlab.com/jpn/gom/GOMPLAYERJPSETUP.EXE
2015-02-26 12:46:09-0400 [scrapy] INFO: Scrapy 0.25.1 started (bot: scrapybot)
2015-02-26 12:46:09-0400 [scrapy] INFO: Optional features available: ssl, http11, boto, django
2015-02-26 12:46:09-0400 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0}
2015-02-26 12:46:09-0400 [scrapy] INFO: Enabled extensions: TelnetConsole, CloseSpider, CoreStats, SpiderState
2015-02-26 12:46:10-0400 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-02-26 12:46:10-0400 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-02-26 12:46:10-0400 [scrapy] INFO: Enabled item pipelines: 
2015-02-26 12:46:10-0400 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-02-26 12:46:10-0400 [default] INFO: Spider opened
2015-02-26 12:47:53-0400 [default] DEBUG: Crawled (200) <GET https://app.gomlab.com/jpn/gom/GOMPLAYERJPSETUP.EXE> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f47577f14d0>
[s]   item       {}
[s]   request    <GET https://app.gomlab.com/jpn/gom/GOMPLAYERJPSETUP.EXE>
[s]   response   <200 https://app.gomlab.com/jpn/gom/GOMPLAYERJPSETUP.EXE>
[s]   settings   <scrapy.settings.Settings object at 0x7f47577ebe50>
[s]   spider     <DefaultSpider 'default' at 0x7f475678dc50>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

In [1]: len(response.body)
Out[1]: 22235496
```


Resolving patch:
diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py
--- a/scrapy/core/downloader/contextfactory.py
+++ b/scrapy/core/downloader/contextfactory.py
@@ -1,11 +1,17 @@
 from OpenSSL import SSL
 from twisted.internet.ssl import ClientContextFactory
+try:
+    # available since twisted 14.0
+    from twisted.internet._sslverify import ClientTLSOptions
+except ImportError:
+    ClientTLSOptions = None
 
 
 class ScrapyClientContextFactory(ClientContextFactory):
     "A SSL context factory which is more permissive against SSL bugs."
     # see https://github.com/scrapy/scrapy/issues/82
     # and https://github.com/scrapy/scrapy/issues/26
+    # and https://github.com/scrapy/scrapy/issues/981
 
     def __init__(self):
         # see this issue on why we use TLSv1_METHOD by default
@@ -17,4 +23,6 @@ def getContext(self, hostname=None, port=None):
         # Enable all workarounds to SSL bugs as documented by
         # http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html
         ctx.set_options(SSL.OP_ALL)
+        if hostname and ClientTLSOptions is not None: # workaround for TLS SNI
+            ClientTLSOptions(hostname, ctx)
         return ctx


Test patch: