md: Fix hashing for data >= 256 GB

* cipher/hash-common.h (gcry_md_block_ctx): Add "nblocks_high".
* cipher/hash-common.c (_gcry_md_block_write): Bump NBLOCKS_HIGH.
* cipher/md4.c (md4_init, md4_final): Take care of NBLOCKS_HIGH.
* cipher/md5.c (md5_init, md5_final): Ditto.
* cipher/rmd160.c (_gcry_rmd160_init, rmd160_final): Ditto.
* cipher/sha1.c (sha1_init, sha1_final): Ditto.
* cipher/sha256.c (sha256_init, sha224_init, sha256_final): Ditto.
* cipher/sha512.c (sha512_init, sha384_init, sha512_final): Ditto.
* cipher/tiger.c (do_init, tiger_final): Ditto.
* cipher/whirlpool.c (whirlpool_final): Ditto.

* cipher/md.c (gcry_md_algo_info): Add GCRYCTL_SELFTEST.
(_gcry_md_selftest): Return "not implemented" as required.
* tests/hashtest.c: New.
* tests/genhashdata.c: New.
* tests/Makefile.am (TESTS): Add hashtest.
(noinst_PROGRAMS): Add genhashdata
--

Problem found by Denis Corbin and analyzed by Yuriy Kaminskiy.

sha512 and whirlpool should not have this problem because they use 64
bit types for counting the blocks. However, a similar fix has been
employed to allow for really huge sizes - despite that it will be very
hard to test them.

The test vectors have been produced by sha{1,224,256}sum and the
genhashdata tool.  A sequence of 'a' is used for them because a test
using one million 'a' is commonly used for test vectors.  More test
vectors are required.  Running the large tests needs to be done
manual for now:

  ./hashtest --gigs 256

tests all algorithms,

  ./hashtest --gigs 256 sha1 sha224 sha256

only the given ones.  A configure option to include these test in the
standard regression suite will be useful.  The tests will take looong.

Signed-off-by: Werner Koch <wk@gnupg.org>
Avoid unneeded stack burning with AES-NI and reduce number of 'decryption_prepared' checks

* cipher/rijndael.c (RIJNDAEL_context): Make 'decryption_prepared',
'use_padlock' and 'use_aesni' 1-bit members in bitfield.
(do_setkey): Move 'hwfeatures' inside [USE_AESNI || USE_PADLOCK].
(do_aesni_enc_aligned): Rename to...
(do_aesni_enc): ...this, as function does not require aligned input.
(do_aesni_dec_aligned): Rename to...
(do_aesni_dec): ...this, as function does not require aligned input.
(do_aesni): Remove.
(rijndael_encrypt): Call 'do_aesni_enc' instead of 'do_aesni'.
(rijndael_decrypt): Call 'do_aesni_dec' instead of 'do_aesni'.
(check_decryption_preparation): New.
(do_decrypt): Remove 'decryption_prepared' check.
(rijndael_decrypt): Ditto and call 'check_decryption_preparation'.
(_gcry_aes_cbc_dec): Ditto.
(_gcry_aes_cfb_enc): Add 'burn_depth' and burn stack only when needed.
(_gcry_aes_cbc_enc): Ditto.
(_gcry_aes_ctr_enc): Ditto.
(_gcry_aes_cfb_dec): Ditto.
(_gcry_aes_cbc_dec): Ditto and correct clearing of 'savebuf'.
--

Patch is mostly about reducing overhead for short buffers.

Results on Intel i5-4570:

After:
 $ tests/benchmark --cipher-repetitions 1000 --cipher-with-keysetup cipher aes
 Running each test 1000 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 AES            480ms   540ms  1750ms   300ms  1630ms   300ms  1640ms  1640ms   350ms   350ms  2130ms  2140ms

Before:
 $ tests/benchmark --cipher-repetitions 1000 --cipher-with-keysetup cipher aes
 Running each test 1000 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 AES            520ms   590ms  1760ms   310ms  1640ms   310ms  1610ms  1600ms   360ms   360ms  2150ms  2160ms

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Camellia: Add AVX/AES-NI key setup

* cipher/camellia-aesni-avx-amd64.S (key_bitlength, key_table): New
order of fields in ctx.
(camellia_f, vec_rol128, vec_ror128): New macros.
(__camellia_avx_setup128, __camellia_avx_setup256)
(_gcry_camellia_aesni_avx_keygen): New functions.
* cipher/camellia-aesni-avx2-amd64.S (key_bitlength, key_table): New
order of fields in ctx.
* cipher/camellia-arm.S (CAMELLIA_TABLE_BYTE_LEN, key_length): Remove
unused macros.
* cipher/camellia-glue.c (CAMELLIA_context): Move keytable to head for
better alignment; Make 'use_aesni_avx' and 'use_aesni_avx2' bitfield
members.
[USE_AESNI_AVX] (_gcry_camellia_aesni_avx_keygen): New prototype.
(camellia_setkey) [USE_AESNI_AVX || USE_AESNI_AVX2]: Read hw features
to variable 'hwf' and match features from it.
(camellia_setkey) [USE_AESNI_AVX]: Use AES-NI/AVX key setup if
available.
--

Use AVX/AES-NI for key-setup for small speed-up.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
cipher: use size_t for internal buffer lengths

* cipher/arcfour.c (do_encrypt_stream, encrypt_stream): Use 'size_t'
for buffer lengths.
* cipher/blowfish.c (_gcry_blowfish_ctr_enc, _gcry_blowfish_cbc_dec)
(_gcry_blowfish_cfb_dec): Ditto.
* cipher/camellia-glue.c (_gcry_camellia_ctr_enc)
(_gcry_camellia_cbc_dec, _gcry_blowfish_cfb_dec): Ditto.
* cipher/cast5.c (_gcry_cast5_ctr_enc, _gcry_cast5_cbc_dec)
(_gcry_cast5_cfb_dec): Ditto.
* cipher/cipher-aeswrap.c (_gcry_cipher_aeswrap_encrypt)
(_gcry_cipher_aeswrap_decrypt): Ditto.
* cipher/cipher-cbc.c (_gcry_cipher_cbc_encrypt)
(_gcry_cipher_cbc_decrypt): Ditto.
* cipher/cipher-ccm.c (_gcry_cipher_ccm_encrypt)
(_gcry_cipher_ccm_decrypt): Ditto.
* cipher/cipher-cfb.c (_gcry_cipher_cfb_encrypt)
(_gcry_cipher_cfb_decrypt): Ditto.
* cipher/cipher-ctr.c (_gcry_cipher_ctr_encrypt): Ditto.
* cipher/cipher-internal.h (gcry_cipher_handle->bulk)
(_gcry_cipher_cbc_encrypt, _gcry_cipher_cbc_decrypt)
(_gcry_cipher_cfb_encrypt, _gcry_cipher_cfb_decrypt)
(_gcry_cipher_ofb_encrypt, _gcry_cipher_ctr_encrypt)
(_gcry_cipher_aeswrap_encrypt, _gcry_cipher_aeswrap_decrypt)
(_gcry_cipher_ccm_encrypt, _gcry_cipher_ccm_decrypt): Ditto.
* cipher/cipher-ofb.c (_gcry_cipher_cbc_encrypt): Ditto.
* cipher/cipher-selftest.h (gcry_cipher_bulk_cbc_dec_t)
(gcry_cipher_bulk_cfb_dec_t, gcry_cipher_bulk_ctr_enc_t): Ditto.
* cipher/cipher.c (cipher_setkey, cipher_setiv, do_ecb_crypt)
(do_ecb_encrypt, do_ecb_decrypt, cipher_encrypt)
(cipher_decrypt): Ditto.
* cipher/rijndael.c (_gcry_aes_ctr_enc, _gcry_aes_cbc_dec)
(_gcry_aes_cfb_dec, _gcry_aes_cbc_enc, _gcry_aes_cfb_enc): Ditto.
* cipher/salsa20.c (salsa20_setiv, salsa20_do_encrypt_stream)
(salsa20_encrypt_stream, salsa20r12_encrypt_stream): Ditto.
* cipher/serpent.c (_gcry_serpent_ctr_enc, _gcry_serpent_cbc_dec)
(_gcry_serpent_cfb_dec): Ditto.
* cipher/twofish.c (_gcry_twofish_ctr_enc, _gcry_twofish_cbc_dec)
(_gcry_twofish_cfb_dec): Ditto.
* src/cipher-proto.h (gcry_cipher_stencrypt_t)
(gcry_cipher_stdecrypt_t, cipher_setiv_fuct_t): Ditto.
* src/cipher.h (_gcry_aes_cfb_enc, _gcry_aes_cfb_dec)
(_gcry_aes_cbc_enc, _gcry_aes_cbc_dec, _gcry_aes_ctr_enc)
(_gcry_blowfish_cfb_dec, _gcry_blowfish_cbc_dec)
(_gcry_blowfish_ctr_enc, _gcry_cast5_cfb_dec, _gcry_cast5_cbc_dec)
(_gcry_cast5_ctr_enc, _gcry_camellia_cfb_dec, _gcry_camellia_cbc_dec)
(_gcry_camellia_ctr_enc, _gcry_serpent_cfb_dec, _gcry_serpent_cbc_dec)
(_gcry_serpent_ctr_enc, _gcry_twofish_cfb_dec, _gcry_twofish_cbc_dec)
(_gcry_twofish_ctr_enc): Ditto.
--

On 64-bit platforms, cipher module internally converts 64-bit size_t values
to 32-bit unsigned integers.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Use correct blocksize of 32 bytes for GOSTR3411-94 HMAC

* cipher/md.c (md_open): Set macpads_Bsize to 32 for
GCRY_MD_GOST24311_94.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add new MAC API, initially with HMAC

* cipher/Makefile.am: Add 'mac.c', 'mac-internal.h' and 'mac-hmac.c'.
* cipher/bufhelp.h (buf_eq_const): New.
* cipher/cipher-ccm.c (_gcry_cipher_ccm_tag): Use 'buf_eq_const' for
constant-time compare.
* cipher/mac-hmac.c: New.
* cipher/mac-internal.h: New.
* cipher/mac.c: New.
* doc/gcrypt.texi: Add documentation for MAC API.
* src/gcrypt-int.h [GPG_ERROR_VERSION_NUMBER < 1.13]
(GPG_ERR_MAC_ALGO): New.
* src/gcrypt.h.in (gcry_mac_handle, gcry_mac_hd_t, gcry_mac_algos)
(gcry_mac_flags, gcry_mac_open, gcry_mac_close, gcry_mac_ctl)
(gcry_mac_algo_info, gcry_mac_setkey, gcry_mac_setiv, gcry_mac_write)
(gcry_mac_read, gcry_mac_verify, gcry_mac_get_algo_maclen)
(gcry_mac_get_algo_keylen, gcry_mac_algo_name, gcry_mac_map_name)
(gcry_mac_reset, gcry_mac_test_algo): New.
* src/libgcrypt.def (gcry_mac_open, gcry_mac_close, gcry_mac_ctl)
(gcry_mac_algo_info, gcry_mac_setkey, gcry_mac_setiv, gcry_mac_write)
(gcry_mac_read, gcry_mac_verify, gcry_mac_get_algo_maclen)
(gcry_mac_get_algo_keylen, gcry_mac_algo_name, gcry_mac_map_name): New.
* src/libgcrypt.vers (gcry_mac_open, gcry_mac_close, gcry_mac_ctl)
(gcry_mac_algo_info, gcry_mac_setkey, gcry_mac_setiv, gcry_mac_write)
(gcry_mac_read, gcry_mac_verify, gcry_mac_get_algo_maclen)
(gcry_mac_get_algo_keylen, gcry_mac_algo_name, gcry_mac_map_name): New.
* src/visibility.c (gcry_mac_open, gcry_mac_close, gcry_mac_ctl)
(gcry_mac_algo_info, gcry_mac_setkey, gcry_mac_setiv, gcry_mac_write)
(gcry_mac_read, gcry_mac_verify, gcry_mac_get_algo_maclen)
(gcry_mac_get_algo_keylen, gcry_mac_algo_name, gcry_mac_map_name): New.
* src/visibility.h (gcry_mac_open, gcry_mac_close, gcry_mac_ctl)
(gcry_mac_algo_info, gcry_mac_setkey, gcry_mac_setiv, gcry_mac_write)
(gcry_mac_read, gcry_mac_verify, gcry_mac_get_algo_maclen)
(gcry_mac_get_algo_keylen, gcry_mac_algo_name, gcry_mac_map_name): New.
* tests/basic.c (check_one_mac, check_mac): New.
(main): Call 'check_mac'.
* tests/bench-slope.c (bench_print_header, bench_print_footer): Allow
variable algorithm name width.
(_cipher_bench, hash_bench): Update to above change.
(bench_hash_do_bench): Add 'gcry_md_reset'.
(bench_mac_mode, bench_mac_init, bench_mac_free, bench_mac_do_bench)
(mac_ops, mac_modes, mac_bench_one, _mac_bench, mac_bench): New.
(main): Add 'mac' benchmark options.
* tests/benchmark.c (mac_repetitions, mac_bench): New.
(main): Add 'mac' benchmark options.
--

Add MAC API, with HMAC algorithms. Internally uses HMAC functionality of the
MD module.

[v2]:
 - Add documentation for MAC API.
 - Change length argument for gcry_mac_read from size_t to size_t* for
   returning number of written bytes.
[v3]:
 - HMAC algorithm ids start from 101.
 - Fix coding style for new files.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add CMAC (Cipher-based MAC) to MAC API

* cipher/Makefile.am: Add 'cipher-cmac.c' and 'mac-cmac.c'.
* cipher/cipher-cmac.c: New.
* cipher/cipher-internal.h (gcry_cipher_handle.u_mode): Add 'cmac'.
* cipher/cipher.c (gcry_cipher_open): Rename to...
(_gcry_cipher_open_internal): ...this and add CMAC.
(gcry_cipher_open): New wrapper that disallows use of internal
modes (CMAC) from outside.
(cipher_setkey, cipher_encrypt, cipher_decrypt)
(_gcry_cipher_authenticate, _gcry_cipher_gettag)
(_gcry_cipher_checktag): Add handling for CMAC mode.
(cipher_reset): Do not reset 'marks.key' and do not clear subkeys in
'u_mode' in CMAC mode.
* cipher/mac-cmac.c: New.
* cipher/mac-internal.h: Add CMAC support and algorithms.
* cipher/mac.c: Add CMAC algorithms.
* doc/gcrypt.texi: Add documentation for CMAC.
* src/cipher.h (gcry_cipher_internal_modes): New.
(_gcry_cipher_open_internal, _gcry_cipher_cmac_authenticate)
(_gcry_cipher_cmac_get_tag, _gcry_cipher_cmac_check_tag)
(_gcry_cipher_cmac_set_subkeys): New prototypes.
* src/gcrypt.h.in (gcry_mac_algos): Add CMAC algorithms.
* tests/basic.c (check_mac): Add CMAC test vectors.
--

Patch adds CMAC (Cipher-based MAC) as defined in RFC 4493 and NIST
Special Publication 800-38B.

Internally CMAC is added to cipher module, but is available to outside
only through MAC API.

[v2]:
 - Add documentation.
[v3]:
 - CMAC algorithm ids start from 201.
 - Coding style fixes.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Tweak Camellia-AVX key-setup for small speed-up

* cipher/camellia-aesni-avx-amd64.S (camellia_f): Merge S-function output
rotation with P-function.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Camellia: fix compiler warning

* cipher/camellia-glue.c (camellia_setkey): Use braces around empty if
statement.
--

Patch silences following warning:

 camellia-glue.c: In function 'camellia_setkey':
 camellia-glue.c:183:5: warning: suggest braces around empty body in an 'if' statement [-Wempty-body]

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki-fi>
Initial implementation of GCM

* cipher/Makefile.am: Add 'cipher-gcm.c'.
* cipher/cipher-ccm.c (_gcry_ciphert_ccm_set_lengths)
(_gcry_cipher_ccm_authenticate, _gcry_cipher_ccm_tag)
(_gcry_cipher_ccm_encrypt, _gcry_cipher_ccm_decrypt): Change
'c->u_mode.ccm.tag' to 'c->marks.tag'.
* cipher/cipher-gcm.c: New.
* cipher/cipher-internal.h (GCM_USE_TABLES): New.
(gcry_cipher_handle): Add 'marks.tag', 'u_tag', 'length' and
'gcm_table'; Remove 'u_mode.ccm.tag'.
(_gcry_cipher_gcm_encrypt, _gcry_cipher_gcm_decrypt)
(_gcry_cipher_gcm_setiv, _gcry_cipher_gcm_authenticate)
(_gcry_cipher_gcm_get_tag, _gcry_cipher_gcm_check_tag): New.
* cipher/cipher.c (_gcry_cipher_open_internal, cipher_setkey)
(cipher_encrypt, cipher_decrypt, _gcry_cipher_authenticate)
(_gcry_cipher_gettag, _gcry_cipher_checktag): Add GCM mode handling.
* src/gcrypt.h.in (gcry_cipher_modes): Add GCRY_CIPHER_MODE_GCM.
(GCRY_GCM_BLOCK_LEN): New.
* tests/basic.c (check_gcm_cipher): New.
(check_ciphers): Add GCM check.
(check_cipher_modes): Call 'check_gcm_cipher'.
* tests/bench-slope.c (bench_gcm_encrypt_do_bench)
(bench_gcm_decrypt_do_bench, bench_gcm_authenticate_do_bench)
(gcm_encrypt_ops, gcm_decrypt_ops, gcm_authenticate_ops): New.
(cipher_modes): Add GCM enc/dec/auth.
(cipher_bench_one): Limit GCM to block ciphers with 16 byte block-size.
* tests/benchmark.c (cipher_bench): Add GCM.
--

Currently it is still quite slow.

Still no support for generate_iv(). Is it really necessary?

TODO: Merge/reuse cipher-internal state used by CCM.

Changelog entry will be present in final patch submission.

Changes since v1:
- 6x-7x speedup.
- added bench-slope support

Signed-off-by: Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
[jk: mangle new file throught 'indent -nut']
[jk: few fixes]
[jk: changelog]
Add some documentation for GCM mode

* doc/gcrypt.texi: Add mention of GCM mode.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: GHASH optimizations

* cipher/cipher-gcm.c [GCM_USE_TABLES] (gcmR, ghash): Replace with new.
[GCM_USE_TABLES] [GCM_TABLES_USE_U64] (bshift, fillM, do_ghash): New.
[GCM_USE_TABLES] [!GCM_TABLES_USE_U64] (bshift, fillM): Replace with
new.
[GCM_USE_TABLES] [!GCM_TABLES_USE_U64] (do_ghash): New.
(_gcry_cipher_gcm_tag): Remove extra memcpy to outbuf and use
buf_eq_const for comparing authentication tag.
* cipher/cipher-internal.h (gcry_cipher_handle): Different 'gcm_table'
for 32-bit and 64-bit platforms.
--

Patch improves GHASH speed.

Intel Haswell (x86-64):
Old:
       GCM auth |     26.22 ns/B     36.38 MiB/s     83.89 c/B
New:
       GCM auth |      3.18 ns/B     300.0 MiB/s     10.17 c/B

Intel Haswell (mingw32):
Old:
       GCM auth |     27.27 ns/B     34.97 MiB/s     87.27 c/B
New:
       GCM auth |      7.58 ns/B     125.7 MiB/s     24.27 c/B

Cortex-A8:
Old:
       GCM auth |     231.4 ns/B      4.12 MiB/s     233.3 c/B
New:
       GCM auth |     30.82 ns/B     30.94 MiB/s     31.07 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add Intel PCLMUL acceleration for GCM

* cipher/cipher-gcm.c (fillM): Rename...
(do_fillM): ...to this.
(ghash): Remove.
(fillM): New macro.
(GHASH): Use 'do_ghash' instead of 'ghash'.
[GCM_USE_INTEL_PCLMUL] (do_ghash_pclmul): New.
(ghash): New.
(setupM): New.
(_gcry_cipher_gcm_encrypt, _gcry_cipher_gcm_decrypt)
(_gcry_cipher_gcm_authenticate, _gcry_cipher_gcm_setiv)
(_gcry_cipher_gcm_tag): Use 'ghash' instead of 'GHASH' and
'c->u_mode.gcm.u_tag.tag' instead of 'c->u_tag.tag'.
* cipher/cipher-internal.h (GCM_USE_INTEL_PCLMUL): New.
(gcry_cipher_handle): Move 'u_tag' and 'gcm_table' under
'u_mode.gcm'.
* configure.ac (pclmulsupport, gcry_cv_gcc_inline_asm_pclmul): New.
* src/g10lib.h (HWF_INTEL_PCLMUL): New.
* src/global.c: Add "intel-pclmul".
* src/hwf-x86.c (detect_x86_gnuc): Add check for Intel PCLMUL.
--

Speed-up GCM for Intel CPUs.

Intel Haswell (x86-64):
Old:
AES     GCM enc |      5.17 ns/B     184.4 MiB/s     16.55 c/B
        GCM dec |      4.38 ns/B     218.0 MiB/s     14.00 c/B
       GCM auth |      3.17 ns/B     300.4 MiB/s     10.16 c/B
New:
AES     GCM enc |      3.01 ns/B     317.2 MiB/s      9.62 c/B
        GCM dec |      1.96 ns/B     486.9 MiB/s      6.27 c/B
       GCM auth |     0.848 ns/B    1124.8 MiB/s      2.71 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: Use counter mode code for speed-up

* cipher/cipher-gcm.c (ghash): Add process for multiple blocks.
(gcm_bytecounter_add, gcm_add32_be128, gcm_check_datalen)
(gcm_check_aadlen_or_ivlen, do_ghash_buf): New functions.
(_gcry_cipher_gcm_encrypt, _gcry_cipher_gcm_decrypt)
(_gcry_cipher_gcm_authenticate, _gcry_cipher_gcm_set_iv)
(_gcry_cipher_gcm_tag): Adjust to use above new functions and
counter mode functions for encryption/decryption.
* cipher/cipher-internal.h (gcry_cipher_handle): Remove 'length'; Add
'u_mode.gcm.(addlen|datalen|tagiv|datalen_over_limits)'.
(_gcry_cipher_gcm_setiv): Return gcry_err_code_t.
* cipher/cipher.c (cipher_setiv): Return error code.
(_gcry_cipher_setiv): Handle error code from 'cipher_setiv'.
--

Patch changes GCM to use counter mode code for bulk speed up and also adds data
length checks as given in NIST SP-800-38D section 5.2.1.1.

Bit length requirements from section 5.2.1.1:

 len(plaintext) <= 2^39-256 bits == 2^36-32 bytes == 2^32-2 blocks
 len(aad) <= 2^64-1 bits ~= 2^61-1 bytes
 len(iv) <= 2^64-1 bit ~= 2^61-1 bytes

Intel Haswell:
Old:
AES     GCM enc |      3.00 ns/B     317.4 MiB/s      9.61 c/B
        GCM dec |      1.96 ns/B     486.9 MiB/s      6.27 c/B
       GCM auth |     0.848 ns/B    1124.7 MiB/s      2.71 c/B
New:
AES     GCM enc |      1.12 ns/B     851.8 MiB/s      3.58 c/B
        GCM dec |      1.12 ns/B     853.7 MiB/s      3.57 c/B
       GCM auth |     0.843 ns/B    1131.4 MiB/s      2.70 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: Tweak Intel PCLMUL ghash loop for small speed-up

* cipher/cipher-gcm.c (do_ghash): Mark 'inline'.
[GCM_USE_INTEL_PCLMUL] (do_ghash_pclmul): Rename to...
[GCM_USE_INTEL_PCLMUL] (gfmul_pclmul): ..this and make inline function.
(ghash) [GCM_USE_INTEL_PCLMUL]: Preload data before ghash-pclmul loop.
--

Intel Haswell:
Old:
AES     GCM enc |      1.12 ns/B     853.5 MiB/s      3.58 c/B
        GCM dec |      1.12 ns/B     853.4 MiB/s      3.58 c/B
       GCM auth |     0.843 ns/B    1131.5 MiB/s      2.70 c/B
New:
AES     GCM enc |     0.990 ns/B     963.3 MiB/s      3.17 c/B
        GCM dec |     0.982 ns/B     970.9 MiB/s      3.14 c/B
       GCM auth |     0.711 ns/B    1340.8 MiB/s      2.28 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add aggregated bulk processing for GCM on x86-64

* cipher/cipher-gcm.c [__x86_64__] (gfmul_pclmul_aggr4): New.
(ghash) [GCM_USE_INTEL_PCLMUL]: Add aggregated bulk processing
for __x86_64__.
(setupM) [__x86_64__]: Add initialization for aggregated bulk
processing.
--

Intel Haswell (x86-64):
Old:
AES     GCM enc |     0.990 ns/B     963.3 MiB/s      3.17 c/B
        GCM dec |     0.982 ns/B     970.9 MiB/s      3.14 c/B
       GCM auth |     0.711 ns/B    1340.8 MiB/s      2.28 c/B
New:
AES     GCM enc |     0.535 ns/B    1783.8 MiB/s      1.71 c/B
        GCM dec |     0.531 ns/B    1796.2 MiB/s      1.70 c/B
       GCM auth |     0.255 ns/B    3736.4 MiB/s     0.817 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: Add stack burning

* cipher/cipher-gcm.c (do_ghash, ghash): Return stack burn depth.
(setupM): Wipe 'tmp' buffer.
(do_ghash_buf): Wipe 'tmp' buffer and add stack burning.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: Add clearing and checking of marks.tag

* cipher/cipher-gcm.c (_gcry_cipher_gcm_encrypt)
(_gcry_cipher_gcm_decrypt, _gcry_cipher_gcm_authenticate): Make sure
that tag has not been finalized yet.
(_gcry_cipher_gcm_setiv): Clear 'marks.tag'.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: add FIPS mode restrictions

* cipher/cipher-gcm.c (_gcry_cipher_gcm_encrypt)
(_gcry_cipher_gcm_get_tag): Do not allow using in FIPS mode is setiv
was invocated directly.
(_gcry_cipher_gcm_setiv): Rename to...
(_gcry_cipher_gcm_initiv): ...this.
(_gcry_cipher_gcm_setiv): New setiv function with check for FIPS mode.
[TODO] (_gcry_cipher_gcm_getiv): New.
* cipher/cipher-internal.h (gcry_cipher_handle): Add
'u_mode.gcm.disallow_encryption_because_of_setiv_in_fips_mode'.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
GCM: Use size_t for buffer sizes

* cipher/cipher-gcm.c (ghash, gcm_bytecounter_add, do_ghash_buf)
(_gcry_cipher_gcm_encrypt, _gcry_cipher_gcm_decrypt)
(_gcry_cipher_gcm_authenticate, _gcry_cipher_gcm_geniv)
(_gcry_cipher_gcm_tag): Use size_t for buffer lengths.
* cipher/cipher-internal.h (_gcry_cipher_gcm_encrypt)
(_gcry_cipher_gcm_decrypt, _gcry_cipher_gcm_authenticate): Use size_t
for buffer lengths.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>