Enable gcc specific flags in tests correctly

... and fix issues found.
Don't use -Werror by default

It may be problematic when compilers add new warnings over time.
./configure && make should always just work.