Repository: scrapy/scrapy

Issue:
SSL error
I have lot of errors with SSL websites.
For exemple, when I call : `scrapy shell https://subscribe.wsj.com/printpack/`

I have this error : 

```
2016-10-06 22:15:40 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6043
2016-10-06 22:15:40 [scrapy] INFO: Spider opened
2016-10-06 22:15:40 [scrapy] DEBUG: Retrying <GET https://subscribe.wsj.com/printpack/> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]>]
2016-10-06 22:15:40 [scrapy] DEBUG: Retrying <GET https://subscribe.wsj.com/printpack/> (failed 2 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]>]
2016-10-06 22:15:40 [scrapy] DEBUG: Gave up retrying <GET https://subscribe.wsj.com/printpack/> (failed 3 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]>]
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/shell.py", line 71, in run
    shell.start(url=url)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/shell.py", line 47, in start
    self.fetch(url, spider)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/shell.py", line 112, in fetch
    reactor, self._schedule, request, spider)
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "<string>", line 2, in raiseException
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]>]
```

How fix that ?



Discussion:
Hi @ricoxor ,

It seems that Scrapy (via Twisted) is not sending a CipherSuite that suits the remote `subscribe.wsj.com` server. It's the server closing the TLS connection with a handshake failure message.

`curl -v https://subscribe.wsj.com/printpack/` and `openssl s_client -connect subscribe.wsj.com:443` manage to connect.

So Scrapy should be able to connect though, because `openssl s_client` (on the same machine) can.
It should be possible to configure the TLS ClientHello initiated by Scrapy with these cipher suites.
Although I'm not sure how to do that yet.

I've put all the Wiresharp/tcpdump and console logs at https://github.com/redapple/scrapy-issues/tree/master/2311

Note that https://www.ssllabs.com/ssltest/analyze.html?d=subscribe.wsj.com&hideResults=on states that the server is using insecure cipher suites and grades it "C":

```
(...)
TLS_RSA_WITH_RC4_128_MD5 (0x4)   INSECURE   128
TLS_RSA_WITH_RC4_128_SHA (0x5)   INSECURE   128
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) 112
```

Some logs from curl and openssl s_client:

```
$ curl -v https://subscribe.wsj.com/printpack/
* Connected to subscribe.wsj.com (205.203.148.137) port 443 (#0)
* ALPN, offering http/1.1
* SSL connection using TLS1.0 / RSA_3DES_EDE_CBC_SHA1
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: subscribe.wsj.com (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: C=US,ST=New Jersey,L=Princeton,O=Dow Jones and Company,OU=EIS SUB PRINT WEB,CN=subscribe.wsj.com
*    start date: Wed, 21 Sep 2016 00:00:00 GMT
*    expire date: Fri, 22 Sep 2017 23:59:59 GMT
*    issuer: C=US,O=Symantec Corporation,OU=Symantec Trust Network,CN=Symantec Class 3 Secure Server CA - G4
*    compression: NULL
```

```
$ openssl s_client -connect subscribe.wsj.com:443
(...)
New, TLSv1/SSLv3, Cipher is RC4-MD5
(...)
```

Oh, and Scrapy 1.0.6 is able to connect, so it's a regression.

```
$ scrapy shell https://subscribe.wsj.com/printpack/ 
2016-10-07 10:58:21 [scrapy] INFO: Scrapy 1.0.6 started (bot: scrapybot)
2016-10-07 10:58:21 [scrapy] INFO: Optional features available: ssl, http11
2016-10-07 10:58:21 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-10-07 10:58:22 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-10-07 10:58:22 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-10-07 10:58:22 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-10-07 10:58:22 [scrapy] INFO: Enabled item pipelines: 
2016-10-07 10:58:22 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2016-10-07 10:58:22 [scrapy] INFO: Spider opened
2016-10-07 10:58:23 [scrapy] DEBUG: Redirecting (meta refresh) to <GET https://buy.wsj.com/wsjcol16v2?inttrackingCode=aaqnz43b&icid=WSJ_ON_PHP_ACQ_NA> from <GET https://subscribe.wsj.com/printpack/>
2016-10-07 10:58:24 [scrapy] DEBUG: Redirecting (301) to <GET https://buy.wsj.com/wsjcol16v2/?inttrackingCode=aaqnz43b&icid=WSJ_ON_PHP_ACQ_NA> from <GET https://buy.wsj.com/wsjcol16v2?inttrackingCode=aaqnz43b&icid=WSJ_ON_PHP_ACQ_NA>
2016-10-07 10:58:25 [scrapy] DEBUG: Crawled (200) <GET https://buy.wsj.com/wsjcol16v2/?inttrackingCode=aaqnz43b&icid=WSJ_ON_PHP_ACQ_NA> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f6f3d36e090>
[s]   item       {}
[s]   request    <GET https://subscribe.wsj.com/printpack/>
[s]   response   <200 https://buy.wsj.com/wsjcol16v2/?inttrackingCode=aaqnz43b&icid=WSJ_ON_PHP_ACQ_NA>
[s]   settings   <scrapy.settings.Settings object at 0x7f6f35461b50>
[s]   spider     <DefaultSpider 'default' at 0x7f6f33d89910>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser
2016-10-07 10:58:26 [root] DEBUG: Using default logger
2016-10-07 10:58:26 [root] DEBUG: Using default logger
```


Resolving patch:
diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py
--- a/scrapy/core/downloader/contextfactory.py
+++ b/scrapy/core/downloader/contextfactory.py
@@ -13,7 +13,7 @@
     from twisted.web.client import BrowserLikePolicyForHTTPS
     from twisted.web.iweb import IPolicyForHTTPS
 
-    from scrapy.core.downloader.tls import ScrapyClientTLSOptions
+    from scrapy.core.downloader.tls import ScrapyClientTLSOptions, DEFAULT_CIPHERS
 
 
     @implementer(IPolicyForHTTPS)
@@ -46,7 +46,9 @@ def getCertificateOptions(self):
             #   not calling super(..., self).__init__
             return CertificateOptions(verify=False,
                         method=getattr(self, 'method',
-                                       getattr(self, '_ssl_method', None)))
+                                       getattr(self, '_ssl_method', None)),
+                        fixBrokenPeers=True,
+                        acceptableCiphers=DEFAULT_CIPHERS)
 
         # kept for old-style HTTP/1.0 downloader context twisted calls,
         # e.g. connectSSL()
diff --git a/scrapy/core/downloader/tls.py b/scrapy/core/downloader/tls.py
--- a/scrapy/core/downloader/tls.py
+++ b/scrapy/core/downloader/tls.py
@@ -28,6 +28,7 @@
         SSL_CB_HANDSHAKE_START = 0x10
         SSL_CB_HANDSHAKE_DONE = 0x20
 
+    from twisted.internet.ssl import AcceptableCiphers
     from twisted.internet._sslverify import (ClientTLSOptions,
                                              _maybeSetHostNameIndication,
                                              verifyHostname,
@@ -60,6 +61,8 @@ def _identityVerifyingInfoCallback(self, connection, where, ret):
                         'from host "{}" (exception: {})'.format(
                             self._hostnameASCII, repr(e)))
 
+    DEFAULT_CIPHERS = AcceptableCiphers.fromOpenSSLCipherString('DEFAULT')
+
 except ImportError:
     # ImportError should not matter for older Twisted versions
     # as the above is not used in the fallback ScrapyClientContextFactory


Test patch: