Optimize Blowfish weak key check

* cipher/blowfish.c (hashset_elem, val_to_hidx, add_val): New.
(do_bf_setkey): Use faster algorithm for detecting weak keys.
(bf_setkey): Move stack burning to do_bf_setkey.
--

Patch optimizes the weak key check for Blowfish. Instead of iterating through
sbox-tables for duplicates, insert values to hash-set and detect collisions.

Old check code was taking slightly longer time than the actual key setup of
Blowfish, which by itself is already quite slow.


After:

 $ tests/benchmark --cipher-with-keysetup --cipher-repetitions 10 cipher blowfish
 Running each test 10 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 BLOWFISH       410ms   440ms   430ms   370ms   440ms   370ms   430ms   440ms   370ms   370ms       -       -


Before:

 $ tests/benchmark --cipher-with-keysetup --cipher-repetitions 10 cipher blowfish
 Running each test 10 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 BLOWFISH       780ms   770ms   780ms   730ms   780ms   730ms   780ms   790ms   720ms   730ms       -       -


Without key-setup:

 $ tests/benchmark --cipher-repetitions 10 cipher blowfish
 Running each test 10 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 BLOWFISH        70ms    70ms    80ms    30ms    80ms    30ms    80ms    90ms    20ms    30ms       -       -


Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Tweak bench-slope parameters

* tests/bench-slope.c (BUF_STEP_SIZE): Half step size to 64.
(NUM_MEASUREMENT_REPETITIONS): Double repetitions to 64.
--

Tweak parameters for better repeatability of results with fast ciphers
(AES-NI).

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Tweak AES-NI bulk CTR mode slightly

* cipher/rijndael.c [USE_AESNI] (aesni_cleanup_2_5): Rename to...
(aesni_cleanup_2_6): ...this and clear also 'xmm6'.
[USE_AESNI && __i386__] (do_aesni_ctr, do_aesni_ctr_4): Prevent
inlining only on i386, allow on AMD64.
[USE_AESNI] (do_aesni_ctr, do_aesni_ctr_4): Use counter block from
'xmm5' and byte-swap mask from 'xmm6'.
(_gcry_aes_ctr_enc) [USE_AESNI]: Preload counter block to 'xmm5' and
byte-swap mask to 'xmm6'.
(_gcry_aes_ctr_enc, _gcry_aes_cfb_dec, _gcry_aes_cbc_dec): Use
'aesni_cleanup_2_6'.
--

Small tweak that yeilds ~5% more speed on Intel Core i5-4570.

After:
 AES            |  nanosecs/byte   mebibytes/sec   cycles/byte
        CTR enc |     0.274 ns/B    3482.5 MiB/s     0.877 c/B
        CTR dec |     0.274 ns/B    3486.8 MiB/s     0.876 c/B

Before:
 AES            |  nanosecs/byte   mebibytes/sec   cycles/byte
        CTR enc |     0.288 ns/B    3312.5 MiB/s     0.922 c/B
        CTR dec |     0.288 ns/B    3312.6 MiB/s     0.922 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Speed up Stribog

* cipher/stribog.c (STRIBOG_TABLES): Remove.
(Pi): Remove.
[!STRIBOG_TABLES] (A, strido): Remove.
(stribog_table): New table pre-reordered with Pi values.
(strido): Rewrite for new table.
(LPSX): Rewrite for new table.
(xor): Remove.
(g): Small tweaks.
--

Patch optimizes the table-lookup implementation a bit. Patch also removes
the unused non-table implementation from source.

On Intel Core i5-4570 (amd64, 3.2Ghz):

After:
                |  nanosecs/byte   mebibytes/sec   cycles/byte
 STRIBOG256     |      9.22 ns/B     103.4 MiB/s     29.53 c/B
 STRIBOG512     |      9.23 ns/B     103.4 MiB/s     29.53 c/B

Before:
                |  nanosecs/byte   mebibytes/sec   cycles/byte
 STRIBOG256     |     30.17 ns/B     31.61 MiB/s     96.56 c/B
 STRIBOG512     |     30.20 ns/B     31.57 MiB/s     96.68 c/B

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Modify encrypt/decrypt arguments for in-place

* cipher/cipher.c (gcry_cipher_encrypt, gcry_cipher_decrypt): Modify
local arguments if in-place operation.
--

Modify encrypt/decrypt argument variables instead of calling subfunction with
different arguments. This allows compiler to inline the subfunction for small
speedup.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Improve Serpent key setup speed

* cipher/serpent.c (SBOX, SBOX_INVERSE): Remove index argument.
(serpent_subkeys_generate): Use smaller temporary arrays for subkey
generation and perform stack clearing locally.
(serpent_setkey_internal): Use wipememory to clear stack and remove
_gcry_burn_stack.
(serpent_setkey): Remove unneeded _gcry_burn_stack.
--

Avoid using large arrays and large stack burning to gain extra speed for
key setup.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Avoid burn_stack in CAST5 setkey

* cipher/cast5.c (do_cast_setkey): Use wipememory instead of memset.
(cast_setkey): Remove stack burning.
--

Burning stack does not work properly when compiler inlines static functions,
therefore use wipememory to clear stack after use instead of relying on
_gcry_burn_stack.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Avoid burn stack in Arcfour setkey

* cipher/arcfour.c (arcfour_setkey): Remove stack burning.
--

Stack is already cleared in do_arcfour_setkey and GCC is inlining
do_arcfour_setkey to arcfour_setkey which renders this _gcry_burn_stack
broken anyways.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Speed-up AES-NI key setup

* cipher/rijndael.c [USE_AESNI] (m128i_t): Remove.
[USE_AESNI] (u128_t): New.
[USE_AESNI] (aesni_do_setkey): New.
(do_setkey) [USE_AESNI]: Move AES-NI accelerated key setup to
'aesni_do_setkey'.
(do_setkey): Call _gcry_get_hw_features only once. Clear stack after
use in generic key setup part.
(rijndael_setkey): Remove stack burning.
(prepare_decryption) [USE_AESNI]: Use 'u128_t' instead of 'm128i_t' to
avoid compiler generated SSE2 instructions and XMM register usage,
unroll 'aesimc' setup loop
(prepare_decryption): Clear stack after use.
[USE_AESNI] (do_aesni_enc_aligned): Update comment about alignment.
(do_decrypt): Do not burning stack after prepare_decryption.
--

Patch improves the speed of AES key setup with AES-NI instructions. Patch also
removes problematic the use of vector typedef, which might cause interference
with XMM register usage in AES-NI accelerated code.

New:
 $ tests/benchmark --cipher-with-keysetup --cipher-repetitions 1000 cipher aes aes192 aes256
 Running each test 1000 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 AES            520ms   590ms  1760ms   310ms  1640ms   300ms  1620ms  1610ms   350ms   360ms  2160ms  2140ms
 AES192         640ms   680ms  2030ms   370ms  1920ms   350ms  1890ms  1880ms   400ms   410ms  2490ms  2490ms
 AES256         730ms   780ms  2330ms   430ms  2210ms   420ms  2170ms  2180ms   470ms   480ms  2830ms  2840ms

Old:
 $ tests/benchmark --cipher-with-keysetup --cipher-repetitions 1000 cipher aes aes192 aes256
 Running each test 1000 times.
                 ECB/Stream         CBC             CFB             OFB             CTR             CCM
              --------------- --------------- --------------- --------------- --------------- ---------------
 AES            670ms   740ms  1910ms   470ms  1790ms   470ms  1770ms  1760ms   520ms   510ms  2310ms  2310ms
 AES192         820ms   860ms  2220ms   550ms  2110ms   540ms  2070ms  2070ms   600ms   590ms  2670ms  2680ms
 AES256         920ms   970ms  2510ms   620ms  2390ms   600ms  2360ms  2370ms   650ms   660ms  3020ms  3020ms

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Change 64-bit shift to 32-bit in AES AMD64 assembly

* cipher/rijndael-amd64.S (do16bit_shr): Change 'shrq' to 'shrl'.
--

64-bit shift is not needed here as registers are used for 32-bit values.

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Fix decryption function size in AES AMD64 assembly

* cipher/rijndael-amd64.S (_gcry_aes_amd64_decrypt_block): Set '.size'
for '_gcry_aes_amd64_decrypt_block', not '..._encrypt_block'.
--

Signed-off-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>