BUG/TST: fix downcasting of float-like object array
Merge pull request #5397 from jreback/infer

BUG/TST: fix downcasting of float-like object array