HADOOP-9333. Hadoop crypto codec framework based on compression codec

For use cases that deal with sensitive data, we often need to encrypt data to be
stored safely at rest. Hadoop common provides a codec framework for compression
algorithms. We start here. However because encryption algorithms require additional
configuration and methods for key management, we introduce a crypto codec framework
that builds on the compression codec framework. It cleanly distinguishes crypto
algorithms from compression algorithms, but shares common interfaces between them
where possible, and also carries extended interfaces where necessary to satisfy
those needs. We also introduce a generic Key type, and supporting utility methods
and classes, as a necessary abstraction for dealing with both Java crypto keys and
PGP keys.

Haifeng Chen <haifeng.chen@intel.com>
Tianyou Li <tianyou.li@intel.com>
Yi A. Liu <yi.a.liu@intel.com>
HADOOP-9332. Crypto codec implementations for AES

This provides two crypto codec implementations based on the Hadoop crypto codec
framework.
  - SimpleAES: AES codec implementation based on AES-NI. (Not splittable)
  - AES: AES codec implementation based on AES-NI in splittable format.

Haifeng Chen <haifeng.chen@intel.com>
Tianyou Li <tianyou.li@intel.com>
Yi A. Liu <yi.a.liu@intel.com>
Andrew Purtell <andrew.k.purtell@intel.com>
MAPREDUCE-5025. Key distribution and management for crypto codecs

This enables MapReduce to utilize the crypto codec framework for encryption and
decryption of data during MapReduce jobs. We address the following requirements:

  1. Different stages (input, output, intermediate output) should have the
     flexibility to choose whether encrypt or not, as well as which crypto codec
     to use.

  2. Different stages may have different scheme of providing the keys.

  3. Different files (for example, different input files) may have or use
     different keys.

  4. Support a flexible way of retrieving keys for encryption or decryption.

We provide the framework for supporting these requirements as well as implementation
for common use and key retrieving scenarios.

Haifeng Chen <haifeng.chen@intel.com>
Tianyou Li <tianyou.li@intel.com>
Yi A. Liu <yi.a.liu@intel.com>
Fix CMake hadoop-pipes link error with recent OpenSSL