Repository: pyca/cryptography

Issue:
scrypt failures on 32-bit arches
The Debian builds are failing on a whole bunch of arches (I think it's all of the 32-bit ones) in the scrypt  tests with `[_OpenSSLErrorWithText(code=101404844L, lib=6, func=181, reason=172, reason_text='error:060B50AC:digital envelope routines:EVP_PBE_scrypt:memory limit exceeded')]`.

You can see a full build log here: https://buildd.debian.org/status/fetch.php?pkg=python-cryptography&arch=i386&ver=1.7.1-1&stamp=1481779760

And the failing arches here: https://buildd.debian.org/status/package.php?p=python-cryptography



Discussion:
https://github.com/pyca/cryptography/pull/3117 contains a discussion of max mem. Is Debian compiling with a memory limit?
Maybe I'm misunderstanding the code, but Cryptography seems to be passing `sys.maxsize // 2` for maxmem; what kind of memory limit could affect `sys.maxsize`? Anyhow, I can reproduce the problem in a local i386 chroot that I haven't done anything special to wrt memory; I can provide more specific diagnostics from there if you know what I should check :imp: 
OpenSSL also has a global memory limit.

On Thu, Dec 15, 2016 at 8:31 AM, Tristan Seligmann <notifications@github.com
> wrote:

> Maybe I'm misunderstanding the code, but Cryptography seems to be passing sys.maxsize
> // 2 for maxmem; what kind of memory limit could affect sys.maxsize?
> Anyhow, I can reproduce the problem in a local i386 chroot that I haven't
> done anything special to wrt memory; I can provide more specific
> diagnostics from there if you know what I should check 👿
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3323#issuecomment-267327644>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAADBAV9cm79eXUP0VP1lJ4jAxaXxlQTks5rIUEUgaJpZM4LNw0T>
> .
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

The only place the OpenSSL source seems to reference `EVP_R_MEMORY_LIMIT_EXCEEDED` is http://sources.debian.net/src/openssl/1.1.0c-2/crypto/evp/scrypt.c/#L218-L221 which doesn't seem to check any global limit.
https://sources.debian.net/src/openssl/1.1.0c-2/crypto/evp/scrypt.c/#L215-L216
was what I was thinking of, I guess the global limit isn't used if you
explicitly pass one.

On Thu, Dec 15, 2016 at 8:33 AM, Tristan Seligmann <notifications@github.com
> wrote:

> The only place the OpenSSL source seems to reference
> EVP_R_MEMORY_LIMIT_EXCEEDED is http://sources.debian.net/src/
> openssl/1.1.0c-2/crypto/evp/scrypt.c/#L218-L221 which doesn't seem to
> check any global limit.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3323#issuecomment-267328231>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAADBMOpuz4ZuoaJCa755iPq3LZse_ATks5rIUHGgaJpZM4LNw0T>
> .
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

Hmm, I think the problem is simply that this vector exceeds the size limit on 32-bit platforms:
```
>>> sys.maxsize // 2
1073741823
>>> Blen = p * 128 * r
>>> Vlen = 32 * r * (N + 2) * 4
>>> Blen + Vlen
1073744896
>>> 1073744896 > 1073741823
True
```
Makes sense. We should a) Write a test that captures this error condition,
so we don't raise an InternalError, b) skip those vectors I guess on 32-bit

On Thu, Dec 15, 2016 at 8:36 AM, Tristan Seligmann <notifications@github.com
> wrote:

> Hmm, I think the problem is simply that this vector exceeds the size limit
> on i386:
>
> >>> sys.maxsize // 2
> 1073741823
> >>> Blen = p * 128 * r
> >>> Vlen = 32 * r * (N + 2) * 4
> >>> Blen + Vlen
> 1073744896
> >>> 1073744896 > 1073741823
> True
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/3323#issuecomment-267328800>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAADBGh_HRgedBTZfPCI23RFnWUBfi_1ks5rIUJ4gaJpZM4LNw0T>
> .
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6


Resolving patch:
diff --git a/src/cryptography/hazmat/backends/openssl/backend.py b/src/cryptography/hazmat/backends/openssl/backend.py
--- a/src/cryptography/hazmat/backends/openssl/backend.py
+++ b/src/cryptography/hazmat/backends/openssl/backend.py
@@ -9,7 +9,6 @@
 import collections
 import contextlib
 import itertools
-import sys
 from contextlib import contextmanager
 
 import six
@@ -59,6 +58,7 @@
 from cryptography.hazmat.primitives.ciphers.modes import (
     CBC, CFB, CFB8, CTR, ECB, GCM, OFB
 )
+from cryptography.hazmat.primitives.kdf import scrypt
 
 
 _MemoryBIO = collections.namedtuple("_MemoryBIO", ["bio", "char_ptr"])
@@ -1833,9 +1833,10 @@ def x509_name_bytes(self, name):
 
     def derive_scrypt(self, key_material, salt, length, n, r, p):
         buf = self._ffi.new("unsigned char[]", length)
-        res = self._lib.EVP_PBE_scrypt(key_material, len(key_material), salt,
-                                       len(salt), n, r, p, sys.maxsize // 2,
-                                       buf, length)
+        res = self._lib.EVP_PBE_scrypt(
+            key_material, len(key_material), salt, len(salt), n, r, p,
+            scrypt._MEM_LIMIT, buf, length
+        )
         self.openssl_assert(res == 1)
         return self._ffi.buffer(buf)[:]
 
diff --git a/src/cryptography/hazmat/primitives/kdf/scrypt.py b/src/cryptography/hazmat/primitives/kdf/scrypt.py
--- a/src/cryptography/hazmat/primitives/kdf/scrypt.py
+++ b/src/cryptography/hazmat/primitives/kdf/scrypt.py
@@ -4,6 +4,8 @@
 
 from __future__ import absolute_import, division, print_function
 
+import sys
+
 from cryptography import utils
 from cryptography.exceptions import (
     AlreadyFinalized, InvalidKey, UnsupportedAlgorithm, _Reasons
@@ -13,6 +15,11 @@
 from cryptography.hazmat.primitives.kdf import KeyDerivationFunction
 
 
+# This is used by the scrypt tests to skip tests that require more memory
+# than the MEM_LIMIT
+_MEM_LIMIT = sys.maxsize // 2
+
+
 @utils.register_interface(KeyDerivationFunction)
 class Scrypt(object):
     def __init__(self, salt, length, n, r, p, backend):


Test patch:
diff --git a/tests/hazmat/primitives/test_scrypt.py b/tests/hazmat/primitives/test_scrypt.py
--- a/tests/hazmat/primitives/test_scrypt.py
+++ b/tests/hazmat/primitives/test_scrypt.py
@@ -14,7 +14,7 @@
     AlreadyFinalized, InvalidKey, UnsupportedAlgorithm
 )
 from cryptography.hazmat.backends.interfaces import ScryptBackend
-from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
+from cryptography.hazmat.primitives.kdf.scrypt import Scrypt, _MEM_LIMIT
 
 from tests.utils import load_nist_vectors, load_vectors_from_file
 
@@ -22,10 +22,30 @@
     os.path.join("KDF", "scrypt.txt"), load_nist_vectors)
 
 
+def _skip_if_memory_limited(memory_limit, params):
+    # Memory calc adapted from OpenSSL (URL split over 2 lines, thanks PEP8)
+    # https://github.com/openssl/openssl/blob/6286757141a8c6e14d647ec733634a
+    # e0c83d9887/crypto/evp/scrypt.c#L189-L221
+    blen = int(params["p"]) * 128 * int(params["r"])
+    vlen = 32 * int(params["r"]) * (int(params["n"]) + 2) * 4
+    memory_required = blen + vlen
+    if memory_limit < memory_required:
+        pytest.skip("Test exceeds Scrypt memory limit. "
+                    "This is likely a 32-bit platform.")
+
+
+def test_memory_limit_skip():
+    with pytest.raises(pytest.skip.Exception):
+        _skip_if_memory_limited(1000, {"p": 16, "r": 64, "n": 1024})
+
+    _skip_if_memory_limited(2 ** 31, {"p": 16, "r": 64, "n": 1024})
+
+
 @pytest.mark.requires_backend_interface(interface=ScryptBackend)
 class TestScrypt(object):
     @pytest.mark.parametrize("params", vectors)
     def test_derive(self, backend, params):
+        _skip_if_memory_limited(_MEM_LIMIT, params)
         password = params["password"]
         work_factor = int(params["n"])
         block_size = int(params["r"])
@@ -77,6 +97,7 @@ def test_password_not_bytes(self, backend):
 
     @pytest.mark.parametrize("params", vectors)
     def test_verify(self, backend, params):
+        _skip_if_memory_limited(_MEM_LIMIT, params)
         password = params["password"]
         work_factor = int(params["n"])
         block_size = int(params["r"])