Revert "Enable hard vfp here as well"

This reverts commit cad1015c6d3debcf88b2ed9a2f93a716b731b0f0.