Revert "TST: Precision issues with infer on OSX"

This reverts commit 0a5026f2ce1c61a7aceb5f22addbac4b9f8fbab1.
BUG: downcasting via com._possibly_downcast_to_dtype now rounds before casting
     to avoid precision issues on platform; dtype conversion to int is not more robust
     related GH5174
Merge pull request #5368 from jreback/dtype_conv

BUG: downcasting is now more robust (related GH5174)