mask test: more thorough testing

use a random mask to flip bits instead of flipping all bits. This will
help to find problems where only a subset of the mask entries is
compared.

Signed-off-by: Matthias Kretz <kretz@kde.org>