PERF: fix infer_dtype to properly infer a Series (GH5801)

closes #5801

```
------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timestamp_ops_diff2                          |  21.7124 | 2472.3583 |   0.0088 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234
```

really though this was in their before....oh well