Repository: urllib3/urllib3

Issue:
Deprecate [secure] extra
## Are you seeing a DeprecationWarning?

Hello! If you're seeing one of the below messages in `DeprecationWarning` you're in the right place:

> 'urllib3[secure]' extra is deprecated and will be removed in a future release of urllib3 2.x.

> 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in a future release of urllib3 2.x.

These warnings mean you're using deprecated code. But no worries! You can stop the deprecation warnings by following these steps:

- **If you're using Python 2.7.10 or later and OpenSSL 1.1.1+** (this is almost everyone) then you won't need the `urllib3[secure]` extra or `urllib3.contrib.pyopenssl` module. Both were necessary a long time ago, but aren't necessary any more! You can read more context below if you're interested.

- **If you're using an old version of Requests** (ie <2.26.0) then you should upgrade to a newer version. [Requests v2.26.0](https://github.com/psf/requests/blob/main/HISTORY.md#2260-2021-07-13) stopped using `urllib3[secure]` and `urllib3.contrib.pyopenssl` and didn't encounter any issues for users upgrading.

- **If you're using an old version of Selenium** (ie <4.4.3) then you should upgrade to a newer version. [Selenium v4.4.1](https://github.com/SeleniumHQ/selenium/blob/trunk/py/CHANGES) stopped using urllib3[secure], but only in v4.4.3 was an appropriate dependency on certifi added.

- Ensure that you're not installing `urllib3[secure]` anywhere, either in a `requirements.txt` or directly. You should remove all instances of `[secure]` and instead only install `urllib3`.

- ~~Ensure that your code isn't using the `urllib3.contrib.pyopenssl` module.~~ We've decided to undeprecate this module in https://github.com/urllib3/urllib3/issues/3126.

- Ensure that none of your dependencies are installing the `urllib3[secure]` extra. We have [created a list](https://github.com/urllib3/urllib3/issues/2700) of projects using the `urllib3[secure]` extra which we have notified and created a PR for each but not all have responded. If you find additional packages then notifying the package about this issue is appreciated.

- ~~Ensure that none of your dependencies are using the `urllib3.contrib.pyopenssl` module. If you find a package that is using this module then I suggest notifying the package by pointing to this issue.~~ We've decided to undeprecate this module in https://github.com/urllib3/urllib3/issues/3126.

- **Finally, to silence the DeprecationWarning**, uninstall the `urllib3-secure-extra` package with:
  ```bash
  $ python -m pip uninstall urllib3-secure-extra
  ```

## Context

There are not nearly as many reasons to use the pyOpenSSL TLS implementation as there previously were. The primary motivations behind creating the implementation were to support SNI before it was available in Python's `ssl` module everywhere. Requests started using this implementation unconditionally by default but two years ago this was changed to [only on Python versions without SNI support](https://github.com/psf/requests/pull/5443). I suspect this number is shrinking.

The `[secure]` extra began as a way to install all dependencies for our pyOpenSSL TLS implementation but now is only an unfortunate misnomer as it's no more secure than our default implementation on all supported Python versions.

My proposal for now is:

- Create documentation for pyOpenSSL's deprecation in the docs. We'll be targetting a "future 2.x release" with the actual removal.
- Begin unconditionally emitting a `DeprecationWarning` when `urllib3.contrib.pyopenssl.inject_into_urllib3` is called and point to the documentation.
- Remove the `[secure]` extra on as many packages as possible. If any actually require one of the dependencies (pyOpenSSL, certifi, idna) then add that dependency explicitly to their packages' `install_requires`. Remove use of `inject_into_urllib3` if used anywhere in the package.
- Search for `inject_into_urllib3` on GitHub and see what we find? If there are big projects using it we should alert them.

## Timeline for deprecation and removal

In a urllib3 v2.1.0:

- Make the `[secure]` extra empty.
- Make the `inject_into_urllib3` function raise an exception with a URL to the documentation.

In a version after that:

- Remove the `[secure]` extra
- Remove the `inject_into_urllib3` function and `pyopenssl` contrib module.


Discussion:
You can make `urllib3[secure]` depend on something like `urllib3-deprecated-secure` and then raise a DeprecationWarning if you see that module installed

You'll also be able to use pypi stats to see how many people install it

You can then yank `urllib3-deprecated-secure` with a message to show when someone tries to install it
Some numbers!

A large proportion, the majority, of urllib3-secure-extra's installs (August 2022) are for EOL Pythons:

| category | percent | downloads |
|:---------|--------:|----------:|
| 3.6      |  50.95% |   337,687 |
| 3.8      |  17.18% |   113,866 |
| 3.10     |  11.13% |    73,766 |
| 3.9      |  10.39% |    68,839 |
| 3.7      |   6.50% |    43,062 |
| 2.7      |   2.58% |    17,109 |
| null     |   1.23% |     8,162 |
| 3.11     |   0.05% |       326 |
| 3.5      |   0.00% |         2 |
| Total    |         |   662,819 |

Source: `pip install -U pypistats && pypistats python_minor urllib3-secure-extra --last-month`

Compared with a better distribution for urllib3 itself:

| category | percent |   downloads |
|:---------|--------:|------------:|
| 3.7      |  31.40% |  85,496,159 |
| 3.8      |  19.33% |  52,646,133 |
| 3.6      |  13.93% |  37,930,892 |
| null     |  12.55% |  34,165,499 |
| 3.9      |  10.06% |  27,383,154 |
| 2.7      |   6.03% |  16,422,259 |
| 3.10     |   5.83% |  15,870,891 |
| 3.5      |   0.50% |   1,356,362 |
| 3.4      |   0.19% |     512,132 |
| 3.11     |   0.19% |     508,843 |
| 3.12     |   0.00% |       2,073 |
| 3.3      |   0.00% |         390 |
| 3.2      |   0.00% |          18 |
| 2.6      |   0.00% |           9 |
| 3.1      |   0.00% |           1 |
| Total    |         | 272,294,815 |

Source: `pip install -U pypistats && pypistats python_minor urllib3 --last-month`

@sethmlarson FTR it seems like `requests-toolbelt` also triggers this code path: https://github.com/requests/toolbelt/blob/69e2487494b7f8a5951ed92ed014137b8381814c/requests_toolbelt/_compat.py#L56.

Oh, it's already reported at https://github.com/requests/toolbelt/issues/331.
@webknjaz Thank you for the heads up! Wasn't aware of the requests-toolbelt issue. I answered there but don't think there's a bug. Any help reproducing this would be appreciated.

Nevermind: I thought this was about the secure extra, but it's in fact about the pyOpenSSL deprecation, sorry.
This shouldn't be removed (and in fact the deprecation notice should even be removed in the meantime) until Python core `ssl` module allows to load certificates and keys directly from strings instead of just files. PyOpenSSL allows to do that without fuss (cf  https://www.pyopenssl.org/en/latest/api/ssl.html#OpenSSL.SSL.Context.use_certificate and https://www.pyopenssl.org/en/latest/api/ssl.html#OpenSSL.SSL.Context.use_privatekey) and it is even silly not to have it in core as C `_ssl.c` function `load_certificate_chain` is given file paths to call OpenSSL `SSL_CTX_use_certificate_chain_file` where that library has exactly as well `SSL_CTX_use_certificate_chain` to pass content directly, without files.

Using strings instead of files is a VERY common use case (among multiple reasons, when handling certificates and keys in external vaults/secret stores, which you don't want to convert to on-disk files), you can find loads of workarounds everywhere, all equally bad and often up to even monkeypatching core modules. Hence Python core `ssl` module should have that feature, but this is irrelevant here, so until then the current `pyopenssl.py` contrib should be left as is and not generate warnings, as it is the only proper solution to do the above in `requests` (using an adapter).
@pmevzek-godaddy Unfortunately a single feature is unlikely to make us change course here. There is no doubt that the pyOpenSSL library provides more functionality than the builtin ssl module, however we are unable to commit to maintaining the module long-term and will likely need to spend even more effort trying to maintain compatibility with the pyOpenSSL contrib library as we add more features to urllib3 in the future.

In the short term if this code is critical to your application then you can vendor the code and maintain it yourself. Long term there's value in contributing to Python's stdlib to fix the gaps between the two libraries.
> @pmevzek-godaddy Unfortunately a single feature is unlikely to make us change course here. There is no doubt that the pyOpenSSL library provides more functionality than the builtin ssl module, however we are unable to commit to maintaining the module long-term and will likely need to spend even more effort trying to maintain compatibility with the pyOpenSSL contrib library as we add more features to urllib3 in the future.
> 
> In the short term if this code is critical to your application then you can vendor the code and maintain it yourself. Long term there's value in contributing to Python's stdlib to fix the gaps between the two libraries.

I can't disagree with what you say at the end, except that it is far over both my pay range and decision reach. Yet, while the core issue is most probably Python core `ssl` module not having the appropriate feature, I find it a bit simplified to say it is a single feature as if it doesn't count by arguing on compatibility issues and new features you don't disclose, so it remains for me to be seen what this would break really.

I know that I won't make you change your mind and decision on this, yet I wanted to record I believe this change to be the wrong move, making life of users more complicated because something working right now will stop working completely, without (simple) alternatives. If you really have to add in the future new things that would somehow conflict with this you could let your users chose either new features, or this. Which is impossible if "this" is just removed from code source.

Thanks for your time.
I'd just like to add my voice in too that losing the ability to load TLS certificate and key data directly from bytes is really unfortunate.
> If you're using an old version of Selenium (ie <4.4.3) then you should upgrade to a newer version. [Selenium v4.4.1](https://github.com/SeleniumHQ/selenium/blob/trunk/py/CHANGES) stopped using urllib3[secure].

This seems inconsistent: it was either 4.4.3 or 4.4.1 that changed this!
@gsnedders You're right, it is inconsistent but also correct! There was an issue where `urllib3[secure]` was removed as a dependency but that also removed the implicit dependency on `certifi` through `urllib3[secure]` so it had to be added back in v4.4.2, but was done so with an upper bound. v4.4.3 had the proper dependency configuration with `certifi` explicitly depended on but with no upper bound on the version.
> @gsnedders You're right, it is inconsistent but also correct! There was an issue where `urllib3[secure]` was removed as a dependency but that also removed the implicit dependency on `certifi` through `urllib3[secure]` so it had to be added back in v4.4.2, but was done so with an upper bound. v4.4.3 had the proper dependency configuration with `certifi` explicitly depended on but with no upper bound on the version.

While this is true, it does make it confusing for people just reading the quick summary of what to change; I'd suggest something like:

> If you're using an old version of Selenium (ie <4.4.3) then you should upgrade to a newer version. [Selenium v4.4.1](https://github.com/SeleniumHQ/selenium/blob/trunk/py/CHANGES) stopped using `urllib3[secure]`, but only in v4.4.3 was an appropriate dependency on `certifi` added.
I agree with the maintainers here.

Most likely, what will happen is that someone else will take this code and turn it into a new package, and we'll just add that new package as a dependency on top of urllib3 to be able to use in-memory certificates and keys.

Eventually the team behind the new package will hit the same problem though - it will be tough to maintain parity.

Alas, I do not hold high hopes that this will be fixed in core Python however - after over a decade, https://github.com/python/cpython/issues/60691 was recently closed. (And the reason behind that is troubling - IIUC it's quite hard to do it all in C, as would be required to be part of the core ssl library, and the team members who maintain that part of Python don't think that it can be pulled off.)
On a more hopeful note, monkey-patching Python's ssl library itself isn't out of the realm of possibility.

I came up with a working POC, if a very fragile one (based on the longstanding Stack Overflow answer).

The problem is that this requires us to manually compute an offset into a C struct used by the C implementation of Python (so it would only work on CPython). This is very fragile - when I use my system-installed Python of 3.10.11 on GNU/Linux (x86-64), it works. When I use the same version of Python, but in a virtual env, the offset has changed somehow and it segfaults.

There's actually a feature request to deal with this as well, https://github.com/python/cpython/issues/88068 - if we had this then the below monkey-patch could be made to work a lot more reliably. I don't like the odds on this one either - but at least it is still open.


```import requests
import ssl
from OpenSSL.crypto import PKCS12, X509, PKey
from cryptography.hazmat.bindings.openssl.binding import Binding


def _is_key_file_encrypted(keyfile):
    # In memory key is not encrypted
    if isinstance(keyfile, PKey):
        return False
    return _is_key_file_encrypted.original(keyfile)


class PyOpenSSLContext(ssl.SSLContext):
    # Support loading certs from memory
    def load_cert_chain(self, certfile, keyfile=None, password=None):
        if isinstance(certfile, X509) and isinstance(keyfile, PKey):
            pkey = keyfile
            cert = certfile

            binding = Binding()
            lib = binding.lib
            ffi = binding.ffi

            self_addr = id(self)
            self_ctx_field = ffi.cast("long *", self_addr+16)
            ctx_list = ffi.unpack(self_ctx_field, 1)
            ssl_ctx_addr = ctx_list[0]
            context = ffi.cast("SSL_CTX *", ssl_ctx_addr)

            lib.SSL_CTX_use_PrivateKey(context, pkey._pkey)
            lib.SSL_CTX_use_certificate(context, cert._x509)
        else:
            super().load_cert_chain(certfile, keyfile=keyfile,
                                    password=password)


class HTTPAdapter(requests.adapters.HTTPAdapter):
    # Handle a variety of cert types
    def cert_verify(self, conn, url, verify, cert):
        if cert:
            # PKCS12
            if isinstance(cert, PKCS12):
                conn.cert_file = cert.get_certificate()
                conn.key_file = cert.get_privatekey()
                cert = None
            elif isinstance(cert, tuple) and len(cert) == 2:
                # X509 and PKey
                if isinstance(cert[0], X509) and hasattr(cert[1], PKey):
                    conn.cert_file = cert[0]
                    conn.key_file = cert[1]
                    cert = None
                # cryptography objects
                elif hasattr(cert[0], 'public_bytes') and hasattr(
                      cert[1], 'private_bytes'):
                    conn.cert_file = X509.from_cryptography(cert[0])
                    conn.key_file = PKey.from_cryptography_key(cert[1])
                    cert = None
        super().cert_verify(conn, url, verify, cert)


def patch_requests(adapter=True):
    # You can perform a full patch and use requests as usual:

    # >>> patch_requests()
    # >>> requests.get('https://httpbin.org/get')

    # or use the adapter explicitly:

    # >>> patch_requests(adapter=False)
    # >>> session = requests.Session()
    # >>> session.mount('https', HTTPAdapter())
    # >>> session.get('https://httpbin.org/get')
    
    if hasattr(requests.packages.urllib3.util.ssl_, '_is_key_file_encrypted'):
        _is_key_file_encrypted.original = \
            requests.packages.urllib3.util.ssl_._is_key_file_encrypted
        requests.packages.urllib3.util.ssl_._is_key_file_encrypted = \
            _is_key_file_encrypted
    requests.packages.urllib3.util.ssl_.SSLContext = PyOpenSSLContext
    if adapter:
        requests.sessions.HTTPAdapter = HTTPAdapter
 ```
Thanks @gsnedders I updated the post above to use your wording.
There doesn't seem to be an `inject_into_pyopenssl`. 😉

![image](https://github.com/urllib3/urllib3/assets/4003950/cc854a99-c8af-4fa9-b718-6124afb1ab03)

Thanks @jiasli, fixed.

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -9,7 +9,7 @@
 
 def tests_impl(
     session: nox.Session,
-    extras: str = "socks,secure,brotli,zstd",
+    extras: str = "socks,brotli,zstd",
     byte_string_comparisons: bool = True,
 ) -> None:
     # Install deps and the package itself.
@@ -65,7 +65,7 @@ def test_brotlipy(session: nox.Session) -> None:
     'brotlicffi' that we still don't blow up.
     """
     session.install("brotlipy")
-    tests_impl(session, extras="socks,secure", byte_string_comparisons=False)
+    tests_impl(session, extras="socks", byte_string_comparisons=False)
 
 
 def git_clone(session: nox.Session, git_url: str) -> None:
@@ -156,7 +156,7 @@ def mypy(session: nox.Session) -> None:
 @nox.session
 def docs(session: nox.Session) -> None:
     session.install("-r", "docs/requirements.txt")
-    session.install(".[socks,secure,brotli,zstd]")
+    session.install(".[socks,brotli,zstd]")
 
     session.chdir("docs")
     if os.path.exists("_build"):
diff --git a/src/urllib3/__init__.py b/src/urllib3/__init__.py
--- a/src/urllib3/__init__.py
+++ b/src/urllib3/__init__.py
@@ -44,23 +44,6 @@
             "See: https://github.com/urllib3/urllib3/issues/2168"
         )
 
-# === NOTE TO REPACKAGERS AND VENDORS ===
-# Please delete this block, this logic is only
-# for urllib3 being distributed via PyPI.
-# See: https://github.com/urllib3/urllib3/issues/2680
-try:
-    import urllib3_secure_extra  # type: ignore # noqa: F401
-except ModuleNotFoundError:
-    pass
-else:
-    warnings.warn(
-        "'urllib3[secure]' extra is deprecated and will be removed "
-        "in urllib3 v2.1.0. Read more in this issue: "
-        "https://github.com/urllib3/urllib3/issues/2680",
-        category=DeprecationWarning,
-        stacklevel=2,
-    )
-
 __author__ = "Andrey Petrov (andrey.petrov@shazow.net)"
 __license__ = "MIT"
 __version__ = __version__
diff --git a/src/urllib3/contrib/pyopenssl.py b/src/urllib3/contrib/pyopenssl.py
--- a/src/urllib3/contrib/pyopenssl.py
+++ b/src/urllib3/contrib/pyopenssl.py
@@ -8,10 +8,10 @@
 
 * `pyOpenSSL`_ (tested with 16.0.0)
 * `cryptography`_ (minimum 1.3.4, from pyopenssl)
-* `idna`_ (minimum 2.0, from cryptography)
+* `idna`_ (minimum 2.0)
 
-However, pyOpenSSL depends on cryptography, which depends on idna, so while we
-use all three directly here we end up having relatively few packages required.
+However, pyOpenSSL depends on cryptography, so while we use all three directly here we
+end up having relatively few packages required.
 
 You can install them with the following command:
 


Test patch: