I meant to use this :
https://github.com/pyca/cryptography/pull/731/files#diff-9c5dedc98e79a81f0606e9dfc3d7fc2aR111
That's how it's done for RSA too.