Revert "vfp: since we use hardfloat, change the modifer accordingly"

This reverts commit b5f591ab882710bbd6ccd76f5a1bc58f57a7cbdb.