Honor LDFLAGS and explicitly link against GMP