Issue 1:
Hi

I am having problem using proxy with Scrapy, I have private proxy with HTTPS supported.

I tried with simplest spider without proxy with the following extensions:

EXTENSIONS = {

'scrapy.extensions.telnet.TelnetConsole': None,

}

SPIDER_MIDDLEWARES = {

'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,

}

DOWNLOADER_MIDDLEWARES = {

'scrapy_splash.SplashCookiesMiddleware': 723,

'scrapy_splash.SplashMiddleware': 725,

'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,

}

It worked fine 200 ok response

Then I configured the proxy as suggested in the documenation:

https://docs.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=proxy#module-scrapy.downloadermiddlewares.httpproxy

added this to the middle ware:

DOWNLOADER_MIDDLEWARES = {

...

'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 1

...

}

I added the meta key proxy per each request as follows:

request = SplashRequest(url='https://www.example.com', callback=self.parse, endpoint='execute',

args={'wait': 1,

'lua_source': login_script,

'timeout':90,

}

)

request.meta['proxy'] = proxy

yield request

where proxy = 'https://username:password@proxyip:proxyport'

I am getting the error below when running scrapy:

2019-07-19 16:11:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://127.0.0.1:8050/execute> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL rout

ines', 'ssl3_get_record', 'wrong version number')]>]

I tried to fix by using the values of DOWNLOADER_CLIENT_TLS_METHOD such as:

'TLSv1.0' 'TLSv1.1' 'TLSv1.2' 'SSLv3'

but it didn't change anything same error.

Thanks

Issue 2:
I'm unable to scrape https sites through https supported proxies. I've tried with proxymesh as well as other proxy services. I can scrape most of this sites without proxies or using Tor.

Curl seems to work fine too:

curl -x https://xx.xx.xx.xx:xx --proxy-user user:pass -L https://www.base.net:443

Retrieves the site's html.

Setup:

OS:

OS X El Capitan v10.11.3

Scrapy:

scrapy version -v
Scrapy : 1.0.5
lxml : 3.5.0.0
libxml2 : 2.9.2
Twisted : 15.5.0
Python : 2.7.11 (default, Dec 7 2015, 23:36:10) - [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)]
pyOpenSSL : 0.15.1 (OpenSSL 1.0.2g 1 Mar 2016)
Platform : Darwin-15.3.0-x86_64-i386-64bit

Solutions tried:

1 - Installing Scrapy-1.1.0rc3

2016-03-09 12:44:59 [scrapy] ERROR: Error downloading <GET https://www.base.net/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]>]

Other website:

2016-03-09 12:56:45 [scrapy] DEBUG: Retrying <GET https://es.alojadogatopreto.com/es-es/> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl23_read', 'ssl handshake failure')]>]

2 - #1764 (comment)

Using SSLv23_METHOD

2016-03-09 12:22:40 [scrapy] ERROR: Error downloading <GET https://www.base.net/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]>]

Using other SSL methods

2016-03-09 12:24:11 [scrapy] ERROR: Error downloading <GET https://www.base.net/>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_RECORD', 'wrong version number')]>]

3 - #1227 (comment) | Get same errors as in 1 & 2.

4 - #1429 (comment) | Get same errors as in 1 & 2.