Revert "Really only set CMAKE_CXX_FLAGS if empty on GCC"

This reverts commit 20c95ddcad13bce5b72ca6bc84368b3c6473a8e4.