Always compile xxhash with optimizations.

Makes the debug build faster.
Merge pull request #4331 from unknownbrackets/perf

Always compile xxhash with optimizations (debug only)