PERF: fix infer_dtype to properly infer a Series (GH5801)