Fix thinko -- only add flags if debugging.

These CFLAGS/CXXFLAGS should only be added when building via --with debug, but in the current spec they're added regardless.  My bad. :-(