Don't directly set global cflags for fpu

Change-Id: I3f502900b994ee3038ed44ef19f376ec70d266e0