Repository: pyca/cryptography

Issue:
KDF Interfaces
Who has opinions? This is relevant for HKDF in #490 and upcoming PBKDF2.

Here's a straw man to knock down:

``` python
# This API is a bit challenging since each KDF has different features 
# (salt/no salt, maximum time, output length, iteration count, etc)

kdf = PBKDF2(iterations, digest, backend)
key = kdf.derive("key/password", "salt")
kdf.verify("key/password", salt="salt", key)

kdf = HKDF(algorithm, key_length, info, backend)
key = kdf.derive("key/password", "salt")
kdf.verify("key/password", salt="salt", key)

kdf = BCrypt(work_factor=100, backend)
key = kdf.derive("key/password")
kdf.verify("key/password", key)

kdf = SCrypt(key_length, salt_size=16, max_time=0.1, max_mem=4, backend)
key = kdf.derive("key/password") # really need to get key/salt out since auto-generated
kdf.verify("key/password", key)
```

There are also calibration routines available for some of these KDFs. (scrypt has one as well as CommonCrypto's `CCCalibratePBKDF` for PBKDF2).



Discussion:
It seems like the output for all of these should be actual key material suitable for passing directly to an algorithm constructor. However that makes `verify` hard to use because it can't recover the original parameters. It also means the API can't be used for transparently upgrading old keys on `verify`.

I don't think those are use cases this level of the API should actually support but it probably means e.g. `max_time` and `salt_size` shouldn't be arguments and that we should provide calibration externally. I'm not quite sure how `max_mem` actually behaves under memory pressure but it might also be problematic. The params to scrypt defined in the draft RFC (http://tools.ietf.org/id/draft-josefsson-scrypt-kdf-01.txt) are...

```
  Input:
            P       Passphrase, an octet string.
            S       Salt, an octet string.
            N       CPU/Memory cost parameter, must be larger than 1,
                    a power of 2 and less than 2^(128 * r / 8).
            r       Block size parameter.
            p       Parallelization parameter, a positive integer
                    less than or equal to ((2^32-1) * hLen) / MFLen
                    where hLen is 32 and MFlen is 128 * r.
            dkLen   Intended output length in octets of the derived
                    key; a positive integer less than or equal to
                    (2^32 - 1) * hLen where hLen is 32.

   Output:
            DK      Derived key, of length dkLen octets.

```

Do we want to provide the same interface for low-entropy KDFs (so, like scrypt, suitable for deriving a key from a password) as well as high-entropy ones (like HKDF)? I think the answer is yes, but I'm not sure. They do have some differences, e.g. HKDF can optionally skip the first step where a bunch of input is compressed into the input for the generation step. Also, maxtime, memory factors, iterations etc don't make sense for HKDF.

Salts and info are pretty important; I think they should be added to the API. If the underlying algorithm doesn't support them, we can raise exceptions. Plus, we can create wrappers for KDFs that do not support them to support them anyway: use dumb KDF to produce a key; use key as HMAC key, HMAC the info.

Also, all of these algorithms take salts, including bcrypt.

@lvh: I think @reaperhulk was proposing we have individual (but mostly common where possible) interfaces to each KDF? so e.g. max_time wouldn't be present for HKDF anyway.

I'm asking if we could :) I'm not even sure if there's any real benefit to doing so. I could see how e.g. a password library wants the agility to pick different KDFs, but that's something that can be accomplished at a higher level without being a huge footgun...

Oh right sorry :) I think that some sort of password storage/management/verification thing is probably a useful API for us to provide, as you observe it has different requirements to pure key derivation though.

@lvh, so are you proposing that we don't try to unify them at the hazmat layer and instead allow each to have their own API? So HKDF could have extract and expand separately, PBKDF2 would probably be a one-shot function that takes all params and outputs key material, scrypt would take the raw params (cost factor, block size, parallelization, etc). Then we could build a password verification API in recipes that allows for the behaviors @public mentioned.

Yes, unfortunately. I would like to have consistent APIs, but hazmat is the lowest layer, and the different algorithms just take completely different arguments, so I don't see how we can create a consistent API here. We can do so at a higher layer, so that's good I guess?

I'm not entirely sure if I'm happy with grouping HKDF in with all the others. They're pretty different beasts. But yes, they could all be supported by the high level passwordy API (although maybe we could just leave that up to passlib?)

Oops. I did not intend to close the issue. My bad.

Passlib does a better job than we ever will so that makes sense to me.

I think what @reaperhulk proposed makes sense, they all have constructors
that match their natural parameters, but the common interface is derive()
and verify().

On Mon, Jan 27, 2014 at 9:34 AM, Laurens Van Houtven <
notifications@github.com> wrote:

> Oops. I did not intend to close the issue. My bad.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pyca/cryptography/issues/511#issuecomment-33377525
> .

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

I just realized I was probably unclear: the derive/verify API can be the same, but obviously the "make an object that has that API" is going to have differing APIs :-)

I guess that's the only option we have for a consistent API, although I think it's a little strange that e.g. HKDF's salt and context are next to each other, when the context is obviously specific to derivation.

Maybe context should be an optional "derive" param?


Resolving patch:
diff --git a/cryptography/exceptions.py b/cryptography/exceptions.py
--- a/cryptography/exceptions.py
+++ b/cryptography/exceptions.py
@@ -38,3 +38,7 @@ class InvalidSignature(Exception):
 
 class InternalError(Exception):
     pass
+
+
+class InvalidKey(Exception):
+    pass
diff --git a/cryptography/hazmat/primitives/interfaces.py b/cryptography/hazmat/primitives/interfaces.py
--- a/cryptography/hazmat/primitives/interfaces.py
+++ b/cryptography/hazmat/primitives/interfaces.py
@@ -257,3 +257,19 @@ def e(self):
         """
         The public exponent of the RSA key. Alias for public_exponent.
         """
+
+
+class KeyDerivationFunction(six.with_metaclass(abc.ABCMeta)):
+    @abc.abstractmethod
+    def derive(self, key_material):
+        """
+        Deterministically generates and returns a new key based on the existing
+        key material.
+        """
+
+    @abc.abstractmethod
+    def verify(self, key_material, expected_key):
+        """
+        Checks whether the key generated by the key material matches the
+        expected derived key. Raises an exception if they do not match.
+        """


Test patch: