Only pass cflags during configure and make

Fixes #24172.