Issue 1:
Scrapy : 1.5.0

lxml : 4.1.1.0

libxml2 : 2.9.5

cssselect : 1.0.3

parsel : 1.4.0

w3lib : 1.19.0

Twisted : 17.9.0

Python : 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)]

pyOpenSSL : 17.6.0.dev0 (OpenSSL 1.1.0g 2 Nov 2017)

cryptography : 2.1.4

Platform : Windows-7-6.1.7601-SP1

2018-03-12 21:20:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.farmerscompress.com/> (referer: None)

2018-03-12 21:20:46 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <POST https://www.farmerscompress.com/ProcessUser.aspx> (failed 1 times): []

2018-03-12 21:20:50 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <POST https://www.farmerscompress.com/ProcessUser.aspx> (failed 2 times): []

2018-03-12 21:20:50 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <POST https://www.farmerscompress.com/ProcessUser.aspx> (failed 3 tim

es): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection

lost.>]

2018-03-12 21:20:50 [scrapy.core.scraper] ERROR: Error downloading <POST https://www.farmerscompress.com/ProcessUser.aspx>: [<twisted.python.failure.F

ailure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]

2018-03-12 21:20:51 [scrapy.core.engine] INFO: Closing spider (finished)

2018-03-12 21:20:51 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

Issue 2:
Hi,

I'm attempting to run scrapy on https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences.

Despite upgrading all my brew packages, scrapy installation via pip install -U scrapy, the website disconnects uncleanly. Specifying -s DOWNLOADER_CLIENT_TLS_METHOD=TLSv1.0 makes no difference to the following run. Neither does a user-agent clear things up with -s USER_AGENT="Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"

I am uncertain as how to debug this further.

scrapy shell https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences

2018-01-09 18:03:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
2018-01-09 18:03:33 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.3.1, w3lib 1.18.0, Twisted 17.9.0, Python 3.6.2 (default, Jul 17 2017, 16:44:45) - [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g 2 Nov 2017), cryptography 2.1.4, Platform Darwin-16.7.0-x86_64-i386-64bit
2018-01-09 18:03:33 [scrapy.crawler] INFO: Overridden settings: {'DOWNLOADER_CLIENT_TLS_METHOD': 'TLSv1.0', 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'LOGSTATS_INTERVAL': 0}
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-01-09 18:03:33 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-01-09 18:03:33 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-01-09 18:03:33 [scrapy.core.engine] INFO: Spider opened
2018-01-09 18:03:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
2018-01-09 18:03:34 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences> (failed 2 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
2018-01-09 18:03:35 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences> (failed 3 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
Traceback (most recent call last):
File "/Users/BenJolitz/.virtualenvs/cpython36/bin/scrapy", line 11, in <module>
sys.exit(execute())
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py", line 150, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
func(*a, **kw)
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/cmdline.py", line 157, in _run_command
cmd.run(args, opts)
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/commands/shell.py", line 73, in run
shell.start(url=url, redirect=not opts.no_redirect)
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/shell.py", line 48, in start
self.fetch(url, spider, redirect=redirect)
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/scrapy/shell.py", line 115, in fetch
reactor, self._schedule, request, spider)
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
result.raiseException()
File "/Users/BenJolitz/.virtualenvs/cpython36/lib/python3.6/site-packages/twisted/python/failure.py", line 385, in raiseException
raise self.value.with_traceback(self.tb)
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]

scrapy version -v:

Scrapy : 1.5.0
lxml : 4.1.1.0
libxml2 : 2.9.7
cssselect : 1.0.3
parsel : 1.3.1
w3lib : 1.18.0
Twisted : 17.9.0
Python : 3.6.2 (default, Jul 17 2017, 16:44:45) - [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
pyOpenSSL : 17.5.0 (OpenSSL 1.1.0g 2 Nov 2017)
cryptography : 2.1.4
Platform : Darwin-16.7.0-x86_64-i386-64bit

curl -sLv https://agences.creditfoncier.fr/credit-immobilier/toutes-nos-agences 2>&1 | head -20:

* Trying 160.92.134.3...
* TCP_NODELAY set
* Connected to agences.creditfoncier.fr (160.92.134.3) port 443 (#0)
* TLS 1.0 connection using TLS_RSA_WITH_3DES_EDE_CBC_SHA
* Server certificate: agences.creditfoncier.fr
* Server certificate: RapidSSL RSA CA 2018
* Server certificate: DigiCert Global Root CA
> GET /credit-immobilier/toutes-nos-agences HTTP/1.1
> Host: agences.creditfoncier.fr
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 10 Jan 2018 02:06:54 GMT
< Server: Microsoft-IIS/6.0
< X-Powered-By: ASP.NET
< Content-Length: 35884
< Content-Type: text/html
< Set-Cookie: ASPSESSIONIDQQBACDTS=JHOJNNCCJNIFABOPLNCEEAFH; path=/
< Cache-control: private