Avoid touching CFLAGS unless explicitly requested (#352)