ENH: parse categoricals in read_csv
State: closed

Closes #10153 (at least partly)

Adds the ability to directly parse a `Categorical` through the `dtype` parameter to `read_csv`.  Currently just uses whatever is there as the categories, a possible enhancement would be to allow and enforce user-specified categories, through not quite sure what the api would be.

This only parses string categories - originally I had an impl that did type inference on the categories, but it added a lot of complication without much benefit, now the recommendation in the docs is to convert after parsing.

Here's an example timing.  For reasonably sparse data, a slightly worse than 2x speedup is what I'm typically seeing, along with much better memory usage.

``` python
group1 = ['aaaaa', 'bbbbb', 'cccccc', 'ddddddd', 'eeeeeeee']

df = pd.DataFrame({'a': np.random.choice(group1, 10000000).astype('object'),
                   'b': np.random.choice(group1, 10000000).astype('object'),
                   'c': np.random.choice(group1, 10000000).astype('object')})
df.to_csv('strings.csv', index=False)


In [14]: %timeit pd.read_csv('strings.csv').apply(pd.Categorical)
1 loops, best of 3: 6.66 s per loop

In [13]: %timeit pd.read_csv('strings.csv', dtype='category')
1 loop, best of 3: 3.68 s per loop
```


Labels:
name: IO CSV
description: read_csv, to_csv
name: Categorical
description: Categorical Data Type

Comments:
@jreback - see if you like this any better.  I am still using a fused type with inference functions, that's really the best I can think of to make them generic, still very open to suggestions though. 

I did add Cython's type specialization syntax, e.g. `_try_double[use_parser_data](self.parser, ...)` which at least make the intent pretty explicit.  It's not that far off something like [tag based dispatch](http://www.boost.org/community/generic_programming.html#tag_dispatching) in C++, though not sure if it's really idiomatic Cython.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13406?src=pr) is 85.30% (diff: 100%)

> Merging [#13406](https://codecov.io/gh/pydata/pandas/pull/13406?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13406   diff @@
==========================================
  Files           139        139          
  Lines         50108      50138    +30   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42744      42770    +26   
- Misses         7364       7368     +4   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [2beab41...c78f39f](https://codecov.io/gh/pydata/pandas/compare/2beab41aa288a98bb2f81e5aca0efa67792d3505...c78f39f982d1a3d788ff57a6a9707979a813e3a7?src=pr)

@chris-b1 like to update this. but also remove the fused type hack. see what you can do.

@jreback - just getting back to this.  How would you feel about only parsing string/object `Categoricals`?  That would significantly clean up the code but still cover the biggest use-case, and I could just show in the docs how to convert to numeric after the fact if needed.

@chris-b1 yes that sounds reasonable

@jreback - I've updated this to remove the type inference, if you want to have another look.

looks pretty - will have to have a more detailed look

can u post perf numbers? (or update top)
with new impl 

Updated performance at the top, although it's basically the same as it was before.

can you test when chunking with dtype='category', for with and w/o different categories in each chunk (it looks to work), but pls test.

@jreback - updated for your comments

@jreback, @jorisvandenbossche  - this is updated and rebased if you'd like to take another look - I think I have all the previous comments addressed.  In particular the `categories` of the parsed `Categorical` are now sorted.

thanks @chris-b1 

as usual, check out the docs when built and issue a followup if needed!