blowfish: add amd64 assembly implementation

* cipher/Makefile.am: Add 'blowfish-amd64.S'.
* cipher/blowfish-amd64.S: New file.
* cipher/blowfish.c (USE_AMD64_ASM): New macro.
[USE_AMD64_ASM] (_gcry_blowfish_amd64_do_encrypt)
(_gcry_blowfish_amd64_encrypt_block)
(_gcry_blowfish_amd64_decrypt_block, _gcry_blowfish_amd64_ctr_enc)
(_gcry_blowfish_amd64_cbc_dec, _gcry_blowfish_amd64_cfb_dec): New
prototypes.
[USE_AMD64_ASM] (do_encrypt, do_encrypt_block, do_decrypt_block)
(encrypt_block, decrypt_block): New functions.
(_gcry_blowfish_ctr_enc, _gcry_blowfish_cbc_dec)
(_gcry_blowfish_cfb_dec, selftest_ctr, selftest_cbc, selftest_cfb): New
functions.
(selftest): Call new bulk selftests.
* cipher/cipher.c (gcry_cipher_open) [USE_BLOWFISH]: Register Blowfish
bulk functions for ctr-enc, cbc-dec and cfb-dec.
* configure.ac (blowfish) [x86_64]: Add 'blowfish-amd64.lo'.
* src/cipher.h (_gcry_blowfish_ctr_enc, _gcry_blowfish_cbc_dec)
(gcry_blowfish_cfb_dec): New prototypes.
--

Add non-parallel functions for small speed-up and 4-way parallel functions for
modes of operation that support parallel processing.

Speed old vs. new on AMD Phenom II X6 1055T:
                ECB/Stream         CBC             CFB             OFB             CTR
             --------------- --------------- --------------- --------------- ---------------
BLOWFISH      1.21x   1.12x   1.17x   3.52x   1.18x   3.34x   1.16x   1.15x   3.38x   3.47x

Speed old vs. new on Intel Core i5-2450M (Sandy-Bridge):
                ECB/Stream         CBC             CFB             OFB             CTR
             --------------- --------------- --------------- --------------- ---------------
BLOWFISH      1.16x   1.10x   1.17x   2.98x   1.18x   2.88x   1.16x   1.15x   3.00x   3.02x

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
rinjdael: add amd64 assembly implementation

* cipher/Makefile.am: Add 'rijndael-amd64.S'.
* cipher/rijndael-amd64.S: New file.
* cipher/rijndael.c (USE_AMD64_ASM): New macro.
[USE_AMD64_ASM] (_gcry_aes_amd64_encrypt_block)
(_gcry_aes_amd64_decrypt_block): New prototypes.
(do_encrypt_aligned) [USE_AMD64_ASM]: Use amd64 assembly function.
(do_encrypt): Disable input/output alignment when USE_AMD64_ASM is set.
(do_decrypt_aligned) [USE_AMD64_ASM]: Use amd64 assembly function.
(do_decrypt): Disable input/output alignment when USE_AMD64_AES is set.
* configure.ac (aes) [x86-64]: Add 'rijndael-amd64.lo'.
--

Add optimized amd64 assembly implementation for AES.

Old vs new, on AMD Phenom II:
          ECB/Stream         CBC             CFB             OFB             CTR
       --------------- --------------- --------------- --------------- ---------------
AES     1.74x   1.72x   1.81x   1.85x   1.82x   1.76x   1.67x   1.64x   1.79x   1.81x
AES192  1.77x   1.77x   1.79x   1.88x   1.90x   1.80x   1.69x   1.69x   1.85x   1.81x
AES256  1.79x   1.81x   1.83x   1.89x   1.88x   1.82x   1.72x   1.70x   1.87x   1.89x

Old vs new, on Intel Core2:
          ECB/Stream         CBC             CFB             OFB             CTR
       --------------- --------------- --------------- --------------- ---------------
AES     1.77x   1.75x   1.78x   1.76x   1.76x   1.77x   1.75x   1.76x   1.76x   1.82x
AES192  1.80x   1.73x   1.81x   1.76x   1.79x   1.85x   1.77x   1.76x   1.80x   1.85x
AES256  1.81x   1.77x   1.81x   1.77x   1.80x   1.79x   1.78x   1.77x   1.81x   1.85x

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
twofish: add amd64 assembly implementation

* cipher/Makefile.am: Add 'twofish-amd64.S'.
* cipher/twofish-amd64.S: New file.
* cipher/twofish.c (USE_AMD64_ASM): New macro.
[USE_AMD64_ASM] (_gcry_twofish_amd64_encrypt_block)
(_gcry_twofish_amd64_decrypt_block, _gcry_twofish_amd64_ctr_enc)
(_gcry_twofish_amd64_cbc_dec, _gcry_twofish_amd64_cfb_dec): New
prototypes.
[USE_AMD64_ASM] (do_twofish_encrypt, do_twofish_decrypt)
(twofish_encrypt, twofish_decrypt): New functions.
(_gcry_twofish_ctr_enc, _gcry_twofish_cbc_dec, _gcry_twofish_cfb_dec)
(selftest_ctr, selftest_cbc, selftest_cfb): New functions.
(selftest): Call new bulk selftests.
* cipher/cipher.c (gcry_cipher_open) [USE_TWOFISH]: Register Twofish
bulk functions for ctr-enc, cbc-dec and cfb-dec.
* configure.ac (twofish) [x86_64]: Add 'twofish-amd64.lo'.
* src/cipher.h (_gcry_twofish_ctr_enc, _gcry_twofish_cbc_dec)
(gcry_twofish_cfb_dec): New prototypes.
--

Provides non-parallel implementations for small speed-up and 3-way parallel
implementations that gets accelerated on `out-of-order' CPUs.

Speed old vs. new on Intel Core i5-4570:
                ECB/Stream         CBC             CFB             OFB             CTR
             --------------- --------------- --------------- --------------- ---------------
TWOFISH128     1.08x  1.07x    1.10x  1.80x    1.09x  1.70x    1.08x  1.08x    1.70x  1.69x

Speed old vs. new on Intel Core2 T8100:
                ECB/Stream         CBC             CFB             OFB             CTR
             --------------- --------------- --------------- --------------- ---------------
TWOFISH128     1.11x  1.10x    1.13x  1.65x    1.13x  1.62x    1.12x  1.11x    1.63x  1.59x

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add detection for Intel AVX2 instruction set

* configure.ac: Add option --disable-avx2-support.
(HAVE_GCC_INLINE_ASM_AVX2): New.
(ENABLE_AVX2_SUPPORT): New.
* src/g10lib.h (HWF_INTEL_AVX2): New.
* src/global.c (hwflist): Add HWF_INTEL_AVX2.
* src/hwf-x86.c [__i386__] (get_cpuid): Initialize registers to zero
before cpuid.
[__x86_64__] (get_cpuid): Initialize registers to zero before cpuid.
(detect_x86_gnuc): Store maximum cpuid level.
(detect_x86_gnuc) [ENABLE_AVX2_SUPPORT]: Add detection for AVX2.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add Serpent AVX2 implementation

* cipher/Makefile.am: Add 'serpent-avx2-amd64.S'.
* cipher/serpent-avx2-amd64.S: New file.
* cipher/serpent.c (USE_AVX2): New macro.
(serpent_context_t) [USE_AVX2]: Add 'use_avx2'.
[USE_AVX2] (_gcry_serpent_avx2_ctr_enc, _gcry_serpent_avx2_cbc_dec)
(_gcry_serpent_avx2_cfb_dec): New prototypes.
(serpent_setkey_internal) [USE_AVX2]: Check for AVX2 capable hardware
and set 'use_avx2'.
(_gcry_serpent_ctr_enc) [USE_AVX2]: Use AVX2 accelerated functions.
(_gcry_serpent_cbc_dec) [USE_AVX2]: Use AVX2 accelerated functions.
(_gcry_serpent_cfb_dec) [USE_AVX2]: Use AVX2 accelerated functions.
(selftest_ctr_128, selftest_cbc_128, selftest_cfb_128): Grow 'nblocks'
so that AVX2 codepaths are tested.
* configure.ac (serpent) [avx2support]: Add 'serpent-avx2-amd64.lo'.
--

Add new AVX2 implementation of Serpent that processes 16 blocks in parallel.

Speed old (SSE2) vs. new (AVX2) on Intel Core i5-4570:
                ECB/Stream         CBC             CFB             OFB             CTR
             --------------- --------------- --------------- --------------- ---------------
SERPENT128    1.00x   1.00x   1.00x   2.10x   1.00x   2.16x   1.01x   1.00x   2.16x   2.18x

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Add Camellia AES-NI/AVX2 implementation

* cipher/Makefile.am: Add 'camellia-aesni-avx2-amd64.S'.
* cipher/camellia-aesni-avx2-amd64.S: New file.
* cipher/camellia-glue.c (USE_AESNI_AVX2): New macro.
(CAMELLIA_context) [USE_AESNI_AVX2]: Add 'use_aesni_avx2'.
[USE_AESNI_AVX2] (_gcry_camellia_aesni_avx2_ctr_enc)
(_gcry_camellia_aesni_avx2_cbc_dec)
(_gcry_camellia_aesni_avx2_cfb_dec): New prototypes.
(camellia_setkey) [USE_AESNI_AVX2]: Check AVX2+AES-NI capable hardware
and set 'ctx->use_aesni_avx2'.
(_gcry_camellia_ctr_enc) [USE_AESNI_AVX2]: Add AVX2 accelerated code.
(_gcry_camellia_cbc_dec) [USE_AESNI_AVX2]: Add AVX2 accelerated code.
(_gcry_camellia_cfb_dec) [USE_AESNI_AVX2]: Add AVX2 accelerated code.
(selftest_ctr_128, selftest_cbc_128, selftest_cfb_128): Grow 'nblocks'
so that AVX2 codepaths get tested.
* configure.ac (camellia) [avx2support, aesnisupport]: Add
'camellia-aesni-avx2-amd64.lo'.
--

Add new AVX2/AES-NI implementation of Camellia that processes 32 blocks in
parallel.

Speed old (AVX/AES-NI) vs. new (AVX2/AES-NI) on Intel Core i5-4570:
                 ECB/Stream         CBC             CFB             OFB             CTR
              --------------- --------------- --------------- --------------- ---------------
CAMELLIA128    1.00x   0.99x   1.00x   1.53x   1.00x   1.49x   1.00x   1.00x   1.54x   1.54x
CAMELLIA256    0.99x   1.00x   1.00x   1.50x   1.00x   1.50x   1.00x   1.00x   1.54x   1.52x

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Optimize _gcry_burn_stack for 32-bit and 64-bit architectures

* src/misc.c (_gcry_burn_stack): Add optimization for 32-bit and 64-bit
architectures.
--

Busy looping 'tests/benchmark --cipher-repetitions 10 cipher blowfish' on ARM
Cortex-A8 shows that _gcry_burn_stack takes 21% of CPU time. With this patch,
that number drops to 3.4%.

On AMD64 (Intel i5-4570) CPU usage for _gcry_burn_stack in the same test drops
from 3.5% to 1.1%.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Check if assembler is compatible with AMD64 assembly implementations

* cipher/blowfish-amd64.S: Enable only if
HAVE_COMPATIBLE_GCC_AMD64_PLATFORM_AS is defined.
* cipher/camellia-aesni-avx-amd64.S: Ditto.
* cipher/camellia-aesni-avx2-amd64.S: Ditto.
* cipher/cast5-amd64.S: Ditto.
* cipher/rinjdael-amd64.S: Ditto.
* cipher/serpent-avx2-amd64.S: Ditto.
* cipher/serpent-sse2-amd64.S: Ditto.
* cipher/twofish-amd64.S: Ditto.
* cipher/blowfish.c: Use AMD64 assembly implementation only if
HAVE_COMPATIBLE_GCC_AMD64_PLATFORM_AS is defined
* cipher/camellia-glue.c: Ditto.
* cipher/cast5.c: Ditto.
* cipher/rijndael.c: Ditto.
* cipher/serpent.c: Ditto.
* cipher/twofish.c: Ditto.
* configure.ac: Check gcc/as compatibility with AMD64 assembly
implementations.
--

Later these checks can be split and assembly implementations adapted to handle
different platforms, but for now disable AMD64 assembly implementations if
assembler does not look to be able to handle them.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Make gpg-error replacement defines more robust.

* configure.ac (AH_BOTTOM): Move GPG_ERR_ replacement defines to ...
* src/gcrypt-int.h: new file.
* src/visibility.h, src/cipher.h: Replace gcrypt.h by gcrypt-int.h.
* tests/: Ditto for all test files.
--

Defining newer gpg-error codes in config.h was not a good idea,
because config.h is usually included before gpg-error.h and thus
gpg-error.h would be double defines to lead to faulty code there like

  typedef enum
    {
      [...]
      191 = 191,
      [...]
    };
Tweak ARM inline assembly for mpi

mpi/longlong.h [__arm__]: Enable inline assembly if __thumb2__ is
defined.
[__arm__]: Use __ARCH_ARM when defined.
[__arm__] [__ARM_ARCH >= 5] (count_leading_zeros): New.
--

Current ARM Linux distributions use EABI that enables thumb2, and therefore
inline assembly is disable (because !defined(__thumb__) selector). However
thumb2 allows the use of assembly instructions that longlong.h contains for
ARM. So this patch enables inline assembly for ARM when __thumb2__ is defined
in addition to __thumb__.

Patch also adds optimization for count_leading_zeros() macro for ARM.

Results on Cortex-A8, 1Ghz:
===

Before:

Algorithm         generate  100*sign  100*verify
------------------------------------------------
RSA 1024 bit         750ms    2780ms       110ms
RSA 2048 bit       14280ms   17250ms       300ms
RSA 3072 bit       38630ms   51300ms       650ms
RSA 4096 bit       60940ms   111430ms      1000ms
jussi@cubie:~/libgcrypt$ tests/benchmark dsa
Algorithm         generate  100*sign  100*verify
------------------------------------------------
DSA 1024/160             -    1410ms      1680ms
DSA 2048/224             -    6100ms      7390ms
DSA 3072/256             -   14350ms     17120ms
jussi@cubie:~/libgcrypt$ tests/benchmark ecc
Algorithm         generate  100*sign  100*verify
------------------------------------------------
ECDSA 192 bit         90ms    2160ms      3940ms
ECDSA 224 bit        110ms    2810ms      5400ms
ECDSA 256 bit        150ms    3570ms      6970ms
ECDSA 384 bit        340ms    8320ms     16420ms
ECDSA 521 bit        850ms   19760ms     38480ms

After:

jussi@cubie:~/libgcrypt$ tests/benchmark rsa
Algorithm         generate  100*sign  100*verify
------------------------------------------------
RSA 1024 bit         590ms    2230ms        80ms
RSA 2048 bit        2320ms   13090ms       240ms
RSA 3072 bit       60580ms   38420ms       460ms
RSA 4096 bit       115130ms   82250ms       750ms
jussi@cubie:~/libgcrypt$ tests/benchmark dsa
Algorithm         generate  100*sign  100*verify
------------------------------------------------
DSA 1024/160             -    1070ms      1290ms
DSA 2048/224             -    4500ms      5550ms
DSA 3072/256             -   10280ms     12200ms
jussi@cubie:~/libgcrypt$ tests/benchmark ecc
Algorithm         generate  100*sign  100*verify
------------------------------------------------
ECDSA 192 bit         70ms    1900ms      3560ms
ECDSA 224 bit        100ms    2490ms      4750ms
ECDSA 256 bit        120ms    3140ms      5920ms
ECDSA 384 bit        270ms    6990ms     13790ms
ECDSA 521 bit        680ms   17080ms     33490ms

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
mpi: add missing "cc" clobber for ARM assembly

* mpi/longlong.h [__arm__] (add_ssaaaa, sub_ddmmss): Add __CLOBBER_CC.
[__arm__][__ARM_ARCH <= 3] (umul_ppmm): Ditto.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
mpi: Add __ARM_ARCH for older GCC

* mpi/longlong.h [__arm__]: Construct __ARM_ARCH if not provided by
compiler.
--

GCC 4.8 defines __ARM_ARCH which provides forward compatible way to detect
ARM architecture. Use this when available and construct otherwise.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
bufhelp: Suppress 'cast increases required alignment' warning

* cipher/bufhelp.h (buf_xor, buf_xor_2dst, buf_xor_n_copy): Cast
to larger element pointer through (void *) to suppress -Wcast-error.
--

Patch disables bogus warnings caused by -Wcast-error. We know that byte
pointers are properly aligned at these phases, or that hardware can handle
unaligned accesses.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Fix i386/amd64 inline assembly "cc" clobbers

* cipher/bithelp.h [__GNUC__, __i386__] (rol, ror): add "cc" globber
for inline assembly.
* cipher/cast5.c [__GNUC__, __i386__] (rol): Ditto.
* random/rndhw.c [USE_DRNG] (rdrand_long): Ditto.
* src/hmac256.c [__GNUC__, __i386__] (ror): Ditto.
* mpi/longlong.c [__i386__] (add_ssaaaa, sub_ddmmss, umul_ppmm)
(udiv_qrnnd, count_leading_zeros, count_trailing_zeros): Ditto.
--

These assembly snippets modify cflags but do not mark "cc" clobber.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Remove duplicate header from mpi/amd64/mpih-mul2.S

* mpi/amd64/mpih-mul2.S: remove duplicated header.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Fix 'Please include winsock2.h before windows.h' warnings with mingw32

* random/rndw32.c: include winsock2.h before windows.h.
* src/ath.h [_WIN32]: Ditto.
* tests/benchmark.c [_WIN32]: Ditto.
--

Patch silences warnings of following type:
/usr/lib/gcc/i686-w64-mingw32/4.6/../../../../i686-w64-mingw32/include/winsock2.h:15:2: warning: #warning Please include winsock2.h before windows.h [-Wcpp]

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Fix memory leak in t-mpi-point test

* tests/t-mpi-point.c (basic_ec_math, basic_ec_math_simplified): add
calls to gcry_ctx_release() to free contexts after they become unused.

Signed-off-by: Dmitry Eremin-Solenikov <dbaryshkov@gmail.com>
Register DCO for Dmitry Kasatkin.

--
Fix a special case bug in mpi_powm for e==0.

* mpi/mpi-pow.c (gcry_mpi_powm): For a zero exponent, make sure that
the result has been allocated.
--

This code triggered the problem:

    modulus = gcry_mpi_set_ui(NULL, 100);
    generator = gcry_mpi_set_ui(NULL, 3);
    exponent = gcry_mpi_set_ui(NULL, 0);
    result = gcry_mpi_new(0);
    gcry_mpi_powm(result, generator, exponent, modulus);

gcry_mpi_new(0) does not allocate the limb space thus it is not
possible to write even into the first limb.  Workaround was to use
gcry_mpi_new (1) but a real fix is better.

Reported-by: Ian Goldberg
Signed-off-by: Werner Koch <wk@gnupg.org>
cipher: Prepare to pass extra info to the sign functions.

* src/gcrypt-module.h (gcry_pk_sign_t): Add parms flags and hashalgo.
* cipher/rsa.c (rsa_sign): Add parms and mark them as unused.
* cipher/dsa.c (dsa_sign): Ditto.
* cipher/elgamal.c (elg_sign): Ditto.
* cipher/pubkey.c (dummy_sign): Ditto.
(pubkey_sign): Pass 0 for the new args.

Signed-off-by: Werner Koch <wk@gnupg.org>