pydata/pandas — pull request #8823 by artemyk (2014-11-15 03:13:47+00:00)
Title: Sparse get dummies

For dataframes with a large number of unique values, `get_dummies` can use enormous amounts of memory. This provides a `sparse` flag to `get_dummies` which returns a much more memory-efficient structure.

For example:
```
import pandas as pd
df1 = pd.get_dummies(pd.DataFrame({'a':range(1000)}, dtype='category').a, sparse=False)
print 'dense:', df1.memory_usage().sum()
df2 = pd.get_dummies(pd.DataFrame({'a':range(1000)}, dtype='category').a, sparse=True)
print 'sparse:', df2.memory_usage().sum()
pd.util.testing.assert_frame_equal(df1, df2)
```
returns
```
dense: 8000000
sparse: 8000
```

Performance could probably be improved a lot.

Fails `pandas/tests/test_reshape.py:TestGetDummiesSparse.test_include_na` due to #8822 .