Only use LDFLAGS and CFLAGS when needed