Repository: scrapy/scrapy

Issue:
OpenSSL unsafe legacy renegotiation disabled error
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description

I get an SSL issue on a working [site ](https://dorotheum.com) 
`twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unsafe legacy renegotiation disabled')]>]`

### Steps to Reproduce

1. `scrapy shell https://dorotheum.com`

**Expected behavior:** HTML page

**Actual behavior:** the error above

**Reproduces how often:** 100%

### Versions

```
Scrapy       : 2.6.1
lxml         : 4.8.0.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.4.0
Python       : 3.9.12 (main, Mar 26 2022, 15:44:31) - [Clang 13.1.6 (clang-1316.0.21.2)]
pyOpenSSL    : 22.0.0 (OpenSSL 3.0.3 3 May 2022)
cryptography : 37.0.2
Platform     : macOS-12.2.1-arm64-arm-64bit
```



Discussion:
This is explicitly disabled in OpenSSL 3: https://www.openssl.org/docs/manmaster/man7/migration_guide.html

We may need to add some new OP_* flags to support connecting to these outdated servers.
@wRAR Can the downgrade of OpenSSL, pyopenssl ang cryprography help in this situation?
You should downgrade pyOpenSSL to a version that ships 1.1, assuming it uses a bundled one in your environment (which I suspect it does).
@wRAR looks like twisted does not support OP flasgs... 
You can usually access the underlying OpenSSL.SSL.Context object and call `set_options()` on it.
@wRAR fixed by downgrading cryptography to  36.0.2
Here's my current `scrapy version --verbose ` result
```
Scrapy       : 2.6.1
lxml         : 4.8.0.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 22.4.0
Python       : 3.9.12 (main, Mar 26 2022, 15:44:31) - [Clang 13.1.6 (clang-1316.0.21.2)]
pyOpenSSL    : 22.0.0 (OpenSSL 1.1.1n  15 Mar 2022)
cryptography : 36.0.2
Platform     : macOS-12.3.1-arm64-arm-64bit
```
Now pyOpenSSL uses OpenSSL 1.1.1n
Even though the problem is fixed, the flags enhancement would be great.
Thank you so much @wRAR !

Right, it's `cryptography` that bundles libssl.
I'm having the same problem when trying to scrape the Cisco website for security advisories. How can I downgrade cryptography to a specific version inside of the scrapy package ?
It's not "inside of the scrapy package", it's in your Python environment.
I've having the same problem with Government of Ireland websites. E.g., 

```
scrapy fetch https://www.courts.ie/
```

Comes back with the error, 

```
2022-07-05 17:33:05 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.courts.ie/robots.txt> (failed 1 times): [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', '', 'unsafe legacy renegotiation disabled')]>]
```

Curl and various web browsers do fine with it. Downgrading `cryptography` works. Thank you! 

Does anyone know how to determine exactly what these websites need to upgrade? I.e., if I were going to get in touch with the Government of Ireland webmaster, what would I say? It's not immediately obvious that the bug is not with `cryptography`.
If you are going to contact the website owner just point them to https://www.ssllabs.com/ssltest/ results for their website.

I mean, if the task is "make it work with OpenSSL 3.0 clients" then the fix is to implement  RFC 5746, but I'm not sure the website owner would really want to fix it, considering that the website works in browsers, and it may not be easy or even possible with the stack they are now using, looking at its other problems. And if the task is "make it more secure" then the more important problem is SSL3 not being disabled. 

> It's not immediately obvious that the bug is not with cryptography.

I agree it's not immediately obvious and some research would be needed. I did it earlier and linked the OpenSSL 3.0 changelog which explicitly lists things that are not supported anymore.
I want to drop this relevant link here:

https://stackoverflow.com/questions/71603314/ssl-error-unsafe-legacy-renegotiation-disabled/72374542#72374542

Here is a very useful piece of code to circumvent the issue without downgrading:

````
import requests
import urllib3
import SSL


class CustomHttpAdapter (requests.adapters.HTTPAdapter):
    # "Transport adapter" that allows us to use custom ssl_context.

    def __init__(self, ssl_context=None, **kwargs):
        self.ssl_context = ssl_context
        super().__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = urllib3.poolmanager.PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_context=self.ssl_context)


def get_legacy_session():
    ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
    ctx.options |= 0x4  # OP_LEGACY_SERVER_CONNECT
    session = requests.session()
    session.mount('https://', CustomHttpAdapter(ctx))
    return session
````

````
with (
    get_legacy_session() as s,
    s.get("some-url") as response
):
    print(response.json())
````
(note that Scrapy doesn't use `requests`, so accessing the OpenSSL context from a spider is left as an exercise to the reader)
I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:

contextfactory.py (within MyProject folder)

```
from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = self.getCertificateOptions().getContext()
        ctx.set_options(0x4)
        return ctx

```

within my_spider.py

```
class MySpider(Spider):
    name = "my_spider"

    custom_settings = {
        'DOWNLOADER_CLIENTCONTEXTFACTORY': 'MyProject.contextfactory.LegacyConnectContextFactory',
    }
```



Thanks @aysegulc, your suggestion works.
> I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:
> 
> contextfactory.py (within MyProject folder)
> 
> ```
> from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
> 
> 
> class LegacyConnectContextFactory(ScrapyClientContextFactory):
> 
>     def getContext(self, hostname=None, port=None):
>         ctx = self.getCertificateOptions().getContext()
>         ctx.set_options(0x4)
>         return ctx


this code works too.

contextfactory.py
```python
from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
from cryptography.hazmat.bindings.openssl.binding import Binding


class LegacyConnectContextFactory(ScrapyClientContextFactory):

    def getContext(self, hostname=None, port=None):
        ctx = super(LegacyConnectContextFactory, self).getContext()
        binding = Binding()
        ctx.set_options(binding.lib.SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION)
        return ctx

```
> I ran into the same problem, and built a custom context factory to solve it. I hope it will be helpful for others:
> 
> contextfactory.py (within MyProject folder)
> 
> ```
> from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
> 
> 
> class LegacyConnectContextFactory(ScrapyClientContextFactory):
> 
>     def getContext(self, hostname=None, port=None):
>         ctx = self.getCertificateOptions().getContext()
>         ctx.set_options(0x4)
>         return ctx
> ```
> 
> within my_spider.py
> 
> ```
> class MySpider(Spider):
>     name = "my_spider"
> 
>     custom_settings = {
>         'DOWNLOADER_CLIENTCONTEXTFACTORY': 'MyProject.contextfactory.LegacyConnectContextFactory',
>     }
> ```

thanks.

Resolving patch:
diff --git a/scrapy/core/downloader/contextfactory.py b/scrapy/core/downloader/contextfactory.py
--- a/scrapy/core/downloader/contextfactory.py
+++ b/scrapy/core/downloader/contextfactory.py
@@ -61,7 +61,9 @@ def getCertificateOptions(self):
     # kept for old-style HTTP/1.0 downloader context twisted calls,
     # e.g. connectSSL()
     def getContext(self, hostname=None, port=None):
-        return self.getCertificateOptions().getContext()
+        ctx = self.getCertificateOptions().getContext()
+        ctx.set_options(0x4)  # OP_LEGACY_SERVER_CONNECT
+        return ctx
 
     def creatorForNetloc(self, hostname, port):
         return ScrapyClientTLSOptions(hostname.decode("ascii"), self.getContext(),
diff --git a/scrapy/core/downloader/tls.py b/scrapy/core/downloader/tls.py
--- a/scrapy/core/downloader/tls.py
+++ b/scrapy/core/downloader/tls.py
@@ -17,10 +17,10 @@
 
 
 openssl_methods = {
-    METHOD_TLS: SSL.SSLv23_METHOD,                      # protocol negotiation (recommended)
-    METHOD_TLSv10: SSL.TLSv1_METHOD,                    # TLS 1.0 only
-    METHOD_TLSv11: getattr(SSL, 'TLSv1_1_METHOD', 5),   # TLS 1.1 only
-    METHOD_TLSv12: getattr(SSL, 'TLSv1_2_METHOD', 6),   # TLS 1.2 only
+    METHOD_TLS: SSL.SSLv23_METHOD,      # protocol negotiation (recommended)
+    METHOD_TLSv10: SSL.TLSv1_METHOD,    # TLS 1.0 only
+    METHOD_TLSv11: SSL.TLSv1_1_METHOD,  # TLS 1.1 only
+    METHOD_TLSv12: SSL.TLSv1_2_METHOD,  # TLS 1.2 only
 }
 
 
diff --git a/scrapy/utils/ssl.py b/scrapy/utils/ssl.py
--- a/scrapy/utils/ssl.py
+++ b/scrapy/utils/ssl.py
@@ -1,14 +1,9 @@
-import OpenSSL
+import OpenSSL.SSL
 import OpenSSL._util as pyOpenSSLutil
 
 from scrapy.utils.python import to_unicode
 
 
-# The OpenSSL symbol is present since 1.1.1 but it's not currently supported in any version of pyOpenSSL.
-# Using the binding directly, as this code does, requires cryptography 2.4.
-SSL_OP_NO_TLSv1_3 = getattr(pyOpenSSLutil.lib, 'SSL_OP_NO_TLSv1_3', 0)
-
-
 def ffi_buf_to_string(buf):
     return to_unicode(pyOpenSSLutil.ffi.string(buf))
 
@@ -22,9 +17,6 @@ def x509name_to_string(x509name):
 
 
 def get_temp_key_info(ssl_object):
-    if not hasattr(pyOpenSSLutil.lib, 'SSL_get_server_tmp_key'):  # requires OpenSSL 1.0.2
-        return None
-
     # adapted from OpenSSL apps/s_cb.c::ssl_print_tmp_key()
     temp_key_p = pyOpenSSLutil.ffi.new("EVP_PKEY **")
     if not pyOpenSSLutil.lib.SSL_get_server_tmp_key(ssl_object, temp_key_p):


Test patch:
diff --git a/tests/mockserver.py b/tests/mockserver.py
--- a/tests/mockserver.py
+++ b/tests/mockserver.py
@@ -19,7 +19,6 @@
 from twisted.web.util import redirectTo
 
 from scrapy.utils.python import to_bytes, to_unicode
-from scrapy.utils.ssl import SSL_OP_NO_TLSv1_3
 from scrapy.utils.test import get_testenv
 
 
@@ -350,7 +349,7 @@ def ssl_context_factory(keyfile='keys/localhost.key', certfile='keys/localhost.c
     if cipher_string:
         ctx = factory.getContext()
         # disabling TLS1.3 because it unconditionally enables some strong ciphers
-        ctx.set_options(SSL.OP_CIPHER_SERVER_PREFERENCE | SSL_OP_NO_TLSv1_3)
+        ctx.set_options(SSL.OP_CIPHER_SERVER_PREFERENCE | SSL.OP_NO_TLSv1_3)
         ctx.set_cipher_list(to_bytes(cipher_string))
     return factory