erlang/otp — pull request #377 by dszoboszlay (2014-05-22 11:36:37+00:00)
Title: Allow using OpenSSL in FIPS mode

This pull request is about a new feature.

# EEP light

## Why do we need this new feature?

[FIPS 140-2](http://en.wikipedia.org/wiki/FIPS_140-2) is a U.S. government computer security standard used to accredit cryptographic modules. The U.S. government and other regulated industries may require their software vendors to use FIPS 140-2 validated cryptographic modules in their products.

Erlang/OTP already uses OpenSSL as a cryptographic module, and OpenSSL can [provide](https://www.openssl.org/docs/fips/fipsvalidation.html) FIPS 140-2 validated cryptography. By enabling the `crypto` application to use OpenSSL in FIPS mode Erlang applications could claim to be using a FIPS validated cryptographic module. This is necessary to enter said regulated markets.

## Risks or uncertain artifacts?

* First of all, FIPS validated cryptography is mostly a legal, not a technical challenge. This patch solves only the technical issues (that is, using the OpenSSL API properly). Turning `crypto` into FIPS mode only provides validated cryptographic services to any application if a **large number of administrative requirements are met** (most notably: the OpenSSL FIPS Module's source code was retrieved from a vendor's CD instead of being downloaded from the internet).

* In FIPS mode non-validated algorithms are disabled (and using them raises error `notsup`). This can break many applications (e.g. half of the SNMP v3 security protocols are not available). Properly configuring applications to use only validated algorithms is mostly left to the user.

* Most importantly `ssl` supports only TLS 1.2 in FIPS mode. Earlier versions of the protocol must use md5 (a non-validated algorithm). It is not illegal to use TLS 1.0 or 1.1 in FIPS mode (OpenSSL itself can do so), but there are a [lot of confusing restrictions](http://wiki.openssl.org/index.php/FIPS_mode_and_TLS) in various NIST standards on what is exactly allowed or denied. Not supporting TLS 1.0 and 1.1 in FIPS mode is a legally safe way and avoids adding exceptions to algorithm restrictions from the technical side too.

* A node can only enter FIPS mode when the `crypto` module (and its nif code) is loaded. The load will fail in case of any problems (e.g. the integrity check of the validated Module fails). It is not supported to enter or leave FIPS mode once `crypto` is loaded.

* FIPS support requires at least OpenSSL 1.0.1. To retain compatibility some code pathes are different on older OpenSSL versions or when FIPS support is disabled. This may make testing more difficult (more combinations to validate). Also, to test behaviour in FIPS mode one needs an OpenSSL library built with FIPS support. (It is fortunately not necessary to follow the Security Policy when building it - testing is a technical issue, it doesn't matter whether the libraries qualify as validated or not as long as they are binary compatible.)

* Because of the different C API used in FIPS mode, the (opaque) context values used by the streaming crypto functions are different in FIPS enabled builds and not compatible with contexts in 17.0.

## How did you solve it?

* A new configuration option (`--enable-fips`) is added to enable building OTP with FIPS support. This requires a FIPS-capable OpenSSL library, which is tested by the config script.

Erlang/OTP is by default compiled without FIPS support, because it requires OpenSSL 1.0.1 and would break compatibility with 17.0 (the streaming API's opaque context type changed).

* The crypto application got a new env variable, `fips_mode` that can be set in e.g. the `sys.config` file to `true` if the release needs to operate in FIPS mode.

* When the `crypto` module is loaded, the nif turns OpenSSL to FIPS mode based on the value of `fips_mode`. If FIPS mode is requested but cannot be turned on (because either OTP was compiled without FIPS mode support or OpenSSL reports a failure) the module fails to load.

This is to prevent the application accidentally running in non-FIPS mode and using non-validated cryptography.

* Runtime FIPS mode status can be queried with `crypto:info_fips()`.

* There is an undocumented function, `crypto:enable_fips_mode/1` that can turn FIPS mode on and off on a running system. However, it should not be used in production, and only provided for the test scripts (so it's not necessary to start new nodes for testing FIPS mode and non-FIPS mode respectively).

The problem with entering/leaving FIPS mode on a running system is that OpenSSL tries to prevent using non-validated cryptography so much that it would crash the emulator if any function is called after an attempt to enter/leave FIPS mode fails. This could be prevented only by an extreme amount of locking and global state checking in a concurrent environment like the beam VM.

* Internally the crypto nif code was changed to use the high-level EVP API, because the low-level API is disabled in FIPS mode. An additional benefit of the EVP API is that it makes possible to unify algorithm-specific nif functions to a single generic function. Effectively the same change that took place on the Erlang API in R16B01 is now applied to the C code.

This refactoring is in the last commit, so if it is not desired for some reason, it can be left out.

Switching to the high-level API in theory may come at some performance cost. I made some measurements using the benchmarks created from the `crypto_SUITE` CT tests, and the differences seem to be negligible. The benchmark code can be found at https://github.com/dszoboszlay/otp/tree/crypto_bench (it's not part of this pull request, because it has only a very little added value).

* The CT suites of the `crypto` application are extended to run the tests in both FIPS and non-FIPS modes. The FIPS mode verifies that non-validated algorithms are properly disabled.

* The `public_key`, `ssl`, `ssh`, `inets` and `snmp` applications were tested in FIPS mode too, but their test scripts are not modified to include a FIPS-run as well. Some changes were made to the tests to avoid using settings that would be problematic in FIPS mode (e.g. the test certificates are now created with sha-signatures, not md5 ones), but further modifications in the test structure would be necessary to execute them in FIPS mode (like skipping all TLS 1.1, 1.0 and SSL 3.0 tests in case of the `ssl` application). This was considered an unnecessarily large modification.