CMake: Remove library-specific compiler flag logic