Repository: pyca/cryptography

Issue:
Decide when to deprecate 1.0.1
At some point we should start warning about users with OpenSSL 1.0.1, as we did with 0.9.8 and 1.0.0, now that upstream is no longer supporting it. We don't have to make a decision about when to actually drop it, but at some point we should be warning on it.

However, I don't think we should do it right now, as it'd be too easy to just contribute to warning fatigue, 71% of our downloads are currently using 1.0.1. Here's the last 11 months of progress: 

![screen shot 2017-03-18 at 1 30 00 pm](https://cloud.githubusercontent.com/assets/772/24074446/07214410-0bdf-11e7-90be-695b14698b25.png)

And the SQL I used to generate that:

```sql
SELECT
  STRFTIME_UTC_USEC(timestamp, "%Y-%m") AS yyyymm,
  ROUND(100 * SUM(CASE WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "0.9.8" THEN 1 ELSE 0 END) / COUNT(*), 1) AS percent_098,
  ROUND(100 * SUM(CASE WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.0.0" THEN 1 ELSE 0 END) / COUNT(*), 1) AS percent_100,
  ROUND(100 * SUM(CASE WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.0.1" THEN 1 ELSE 0 END) / COUNT(*), 1) AS percent_101,
  ROUND(100 * SUM(CASE WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.0.2" THEN 1 ELSE 0 END) / COUNT(*), 1) AS percent_102,
  ROUND(100 * SUM(CASE WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.1.0" THEN 1 ELSE 0 END) / COUNT(*), 1) AS percent_110,
  COUNT(*) as download_count
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -1, "year"),
    CURRENT_TIMESTAMP()
  )
WHERE
  details.openssl_version IS NOT NULL AND file.project = 'cryptography' and details.system.name NOT IN ("Windows", "Darwin")
GROUP BY
  yyyymm
ORDER BY
  yyyymm DESC
LIMIT 100
```


Discussion:
I agree -- I think we're at least a year away from starting to raise warnings, but we can keep an eye on the decay. As a note: that's 71% of our downloads that report OpenSSL version and are not on macOS or windows, not 71% of our total downloads. Doesn't change the conclusion in any way though :)
I think we should try to pick a number at which we start warning. To throw
out an opening bid: 35%.

On Sat, Mar 18, 2017 at 5:37 PM, Paul Kehrer <notifications@github.com>
wrote:

> I agree -- I think we're at least a year away from starting to raise
> warnings, but we can keep an eye on the decay. As a note: that's 71% of our
> downloads that report OpenSSL version and are not on macOS or windows, not
> 71% of our total downloads. Doesn't change the conclusion in any way though
> :)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3460#issuecomment-287576682>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAADBMgUAX34JnWy6xi1z19Q2GP9Y7TSks5rnE6JgaJpZM4Mhe4b>
> .
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

35% sounds like a good place to revisit and discuss. Right now Ubuntu 12.04
(supported until April 2017), Ubuntu 14.04, RHEL/CentOS 7 (for now?), and
Debian Jessie all ship 1.0.1. By the time we hit 35% hopefully there'll be
a new Debian and RHEL will have rebased on 1.0.2

On March 18, 2017 at 5:37:59 PM, Alex Gaynor (notifications@github.com)
wrote:

I think we should try to pick a number at which we start warning. To throw
out an opening bid: 35%.

On Sat, Mar 18, 2017 at 5:37 PM, Paul Kehrer <notifications@github.com>
wrote:

> I agree -- I think we're at least a year away from starting to raise
> warnings, but we can keep an eye on the decay. As a note: that's 71% of
our
> downloads that report OpenSSL version and are not on macOS or windows, not
> 71% of our total downloads. Doesn't change the conclusion in any way
though
> :)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3460#issuecomment-287576682>,
> or mute the thread
> <
https://github.com/notifications/unsubscribe-auth/AAADBMgUAX34JnWy6xi1z19Q2GP9Y7TSks5rnE6JgaJpZM4Mhe4b
>
> .
>



--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/pyca/cryptography/issues/3460#issuecomment-287576727>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAJ216jfoXHCnpzaUR0-BYT_CkU57TlSks5rnE62gaJpZM4Mhe4b>
.

That all sounds right. 14.04 is in an awkward spot, I hope Ubuntu does
something proactive there, rather than wait another 3 years before it limps
off stage.
On Sat, Mar 18, 2017 at 11:52 PM, Paul Kehrer <notifications@github.com>
wrote:

> 35% sounds like a good place to revisit and discuss. Right now Ubuntu 12.04
> (supported until April 2017), Ubuntu 14.04, RHEL/CentOS 7 (for now?), and
> Debian Jessie all ship 1.0.1. By the time we hit 35% hopefully there'll be
> a new Debian and RHEL will have rebased on 1.0.2
>
> On March 18, 2017 at 5:37:59 PM, Alex Gaynor (notifications@github.com)
> wrote:
>
> I think we should try to pick a number at which we start warning. To throw
> out an opening bid: 35%.
>
> On Sat, Mar 18, 2017 at 5:37 PM, Paul Kehrer <notifications@github.com>
> wrote:
>
> > I agree -- I think we're at least a year away from starting to raise
> > warnings, but we can keep an eye on the decay. As a note: that's 71% of
> our
> > downloads that report OpenSSL version and are not on macOS or windows,
> not
> > 71% of our total downloads. Doesn't change the conclusion in any way
> though
> > :)
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <https://github.com/pyca/cryptography/issues/3460#issuecomment-287576682
> >,
> > or mute the thread
> > <
> https://github.com/notifications/unsubscribe-auth/
> AAADBMgUAX34JnWy6xi1z19Q2GP9Y7TSks5rnE6JgaJpZM4Mhe4b
> >
> > .
> >
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: D1B3 ADC0 E023 8CA6
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3460#issuecomment-287576727>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-
> auth/AAJ216jfoXHCnpzaUR0-BYT_CkU57TlSks5rnE62gaJpZM4Mhe4b>
> .
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3460#issuecomment-287592434>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAADBO06QReLsxVvXPo1zCkfMb9FDXyyks5rnKaNgaJpZM4Mhe4b>
> .
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

Without actually doing any math, it looks like 1.0.1 drops by about 2% a month. So to get from 71% to 35% will be 18 months. See everybody in 2018!
This appears to be approximately on track, with July 2017 being at 65.6%
I'm going to be very interested to see what the reported OpenSSL version is for the users who don't get wheels once 2.0 comes out 😁 
Slightly updated query, instead of hardcoding exclusion of macOS and Windows, just only count sdist installs:
```sql
SELECT
  STRFTIME_UTC_USEC(timestamp, "%Y-%m") AS yyyymm,
  ROUND(100 * SUM(CASE
        WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "0.9.8" THEN 1
        ELSE 0 END) / COUNT(*), 1) AS percent_098,
  ROUND(100 * SUM(CASE
        WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.0.0" THEN 1
        ELSE 0 END) / COUNT(*), 1) AS percent_100,
  ROUND(100 * SUM(CASE
        WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.0.1" THEN 1
        ELSE 0 END) / COUNT(*), 1) AS percent_101,
  ROUND(100 * SUM(CASE
        WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.0.2" THEN 1
        ELSE 0 END) / COUNT(*), 1) AS percent_102,
  ROUND(100 * SUM(CASE
        WHEN SUBSTR(REGEXP_EXTRACT(details.openssl_version, r"^OpenSSL ([^ ]+) "), 0, 5) = "1.1.0" THEN 1
        ELSE 0 END) / COUNT(*), 1) AS percent_110,
  COUNT(*) AS download_count
FROM
  TABLE_DATE_RANGE( [the-psf:pypi.downloads], DATE_ADD(CURRENT_TIMESTAMP(), -1, "year"), CURRENT_TIMESTAMP() )
WHERE
  details.openssl_version IS NOT NULL
  AND file.project = 'cryptography'
  AND file.type = 'sdist'
GROUP BY
  yyyymm
ORDER BY
  yyyymm DESC
LIMIT
  100
```

Bad news:
![screen shot 2017-10-22 at 5 21 49 pm](https://user-images.githubusercontent.com/772/31866447-82327b50-b74d-11e7-9744-dc1ceecba027.png)

Seeing some good movement here, maybe this'll be the year!

![screen shot 2018-01-06 at 8 19 24 pm](https://user-images.githubusercontent.com/772/34645557-e9dd13f2-f31e-11e7-9948-0a99e514a4a2.png)

More good progress here! I dug in and the remaining 1.0.1 users are primarily Debian Jessie and Ubuntu 14.04:

![screen shot 2018-02-11 at 11 51 45 am](https://user-images.githubusercontent.com/772/36075822-f7687b8c-0f21-11e8-9f59-a4ecc881b811.png)

Interesting seeing the continually dropping download count there. Since our overall download count is actually rising I guess that's due to increased use of the binary wheels?
<img width="674" alt="screen shot 2018-03-18 at 9 26 09 am" src="https://user-images.githubusercontent.com/161495/37566365-6df07750-2a8e-11e8-81e8-9376f21519c7.png">

Getting tantalizingly close to the 35% number we talked about!
To add to the OpenSSL version deprecation, I'd suggest adding OpenSSL API deprecation alongside it.

The current latest stable (ie. 2.2.2) suffers from function signatures so legacy that they predate 0.9.8. (I'm looking at you, useless BN_zero). While this specific function has been removed in 528eb1b, building against an OpenSSL with a restricted API compatibility would actually allow to get proper warnings to detect APIs of OpenSSL version about to be dropped and detect that sooner.
We looked at that a bit over a year ago and at the time concluded it was too much effort given the large amount of deprecated APIs we needed to allow for older OpenSSL support. That may no longer be true (we've dropped 0.9.8 and 1.0.0, although I can't remember if that was before or after we checked) so it's worth revisiting at some point.
As it currently stands, OpenSSL supports 4 API levels :
  - None (which is basically 0.0.0.0)
  - 0.9.8
  - 1.0.0
  - 1.1.0

Except for the aforementioned BN_zero, cryptography supports 1.0.0 which is the maximum API usable if it aims to support 1.0.2.

So this API depreciation could be applied in the upcoming cryptography to ensure no outdated functions are introduced afterwards.
To do that we'd add a `#define OPENSSL_API_COMPAT=0x10000000L` before we pull in any OpenSSL imports, correct?

Current 1.0.1 numbers:

<img width="678" alt="image" src="https://user-images.githubusercontent.com/161495/42354678-11be689e-80e6-11e8-8b77-8bb783f43223.png">

Yep that's it, defining OPENSSL_API_COMPAT to 0x10000000L will disable all pre-1.0.0 wrappers and signatures, allowing for checks as to what's in use despite being deprecated at this stage. 
This is pretty encouraging. Time to add the warning?

![image](https://user-images.githubusercontent.com/772/44013362-0dd0f3f8-9e93-11e8-9f09-2fd2bee20322.png)


Resolving patch:
diff --git a/src/cryptography/hazmat/bindings/openssl/binding.py b/src/cryptography/hazmat/bindings/openssl/binding.py
--- a/src/cryptography/hazmat/bindings/openssl/binding.py
+++ b/src/cryptography/hazmat/bindings/openssl/binding.py
@@ -7,6 +7,7 @@
 import collections
 import threading
 import types
+import warnings
 
 from cryptography import utils
 from cryptography.exceptions import InternalError
@@ -150,9 +151,24 @@ def init_static_locks(cls):
             _openssl_assert(cls.lib, res == 1)
 
 
+def _verify_openssl_version(lib):
+    if (
+        lib.CRYPTOGRAPHY_OPENSSL_LESS_THAN_102 and
+        not lib.CRYPTOGRAPHY_IS_LIBRESSL
+    ):
+        warnings.warn(
+            "OpenSSL version 1.0.1 is no longer supported by the OpenSSL "
+            "project, please upgrade. A future version of cryptography will "
+            "drop support for it.",
+            DeprecationWarning
+        )
+
+
 # OpenSSL is not thread safe until the locks are initialized. We call this
 # method in module scope so that it executes with the import lock. On
 # Pythons < 3.4 this import lock is a global lock, which can prevent a race
 # condition registering the OpenSSL locks. On Python 3.4+ the import lock
 # is per module so this approach will not work.
 Binding.init_static_locks()
+
+_verify_openssl_version(Binding.lib)


Test patch: