Move flags setup to the top of CMakeLists.txt to make sure it affects everything.