Revert "vfp: since we use hardfloat, change the modifer accordingly"

This reverts commit 5e056532bf5b1119e5edbd09347ea1ab4a5fcc56.
Revert "enable various optimizations"

This reverts commit 50c1b1b24656e1e29526044cf5d8d4f929d88109.