Get dummies
State: closed

fixes #4446, #4444

Added new functionality dummy_na (thoughts?). it's slightly different to a possible dropna argument, which I haven't included (which can be achieved using `pd.get_dummies(s.dropna())`.

Example:

```
In [3]: s = ['a', 'b', np.nan]

In [4]: pd.get_dummies(s)
Out[4]:
   a  b
0  1  0
1  0  1
2  0  0

In [5]: pd.get_dummies(s, dummy_na=True)
Out[5]:
   a  b  NaN
0  1  0    0
1  0  1    0
2  0  0    1

In [6]: pd.get_dummies(pd.Series(s).dropna())  # different
Out[6]:
   a  b
0  1  0
1  0  1
```

Note: atm there is a (strange) test Failure with above example, not quite sure what's going on:

```
res_na = get_dummies(s, dummy_na=True)
exp_na = DataFrame({nan: {0: 0.0, 1: 0.0, 2: 1.0},
                            'a': {0: 1.0, 1: 0.0, 2: 0.0},
                            'b': {0: 0.0, 1: 1.0, 2: 0.0}}).iloc[:, [1, 2, 0]]  # need to reorder cols
assert_frame_equal(res_na, exp_na)

-> assert(left.columns.equals(right.columns))
(Pdb) left
   a  b  NaN
0  1  0    0
1  0  1    0
2  0  0    1
(Pdb) right
   a  b  NaN
0  1  0    0
1  0  1    0
2  0  0    1
```


Labels:

Comments:
I _think_ this comes from:

```
(Pdb) np.testing.assert_array_equal(left.columns.values, right.columns.values)
*** AssertionError:
Arrays are not equal

(mismatch 33.3333333333%)
 x: array(['a', 'b', nan], dtype=object)
 y: array(['a', 'b', nan], dtype=object)
```

Which is confusing, as [`assert_array_equal`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.testing.assert_array_equal.html) is supposed to treat NaNs are equal...

> NaNs are compared like numbers, no assertion is raised if both objects have NaNs in the same positions.

```
(Pdb) np.isnan(left.columns.values[2]), np.isnan(right.columns.values[2])
(True, True)

(Pdb) left.columns[:2].equals(right.columns[:2])
True
```

the failing build is here: https://travis-ci.org/hayd/pandas/builds/9834728

Workaround by just setting the columns manually (which is kinda annoying, should create separate issue).

The other option is to have a more consistent `dropna` argument which defaults to True. However, it's not _really_ dropping the NaN row, it's just zeroing it. That's my reasoning behind using `dummy_na`...

@jreback speaking of assert_frame_equal did you see this weird bug?

Also, can we merge this?

@hayd what the `assert_frame_equal` bug?

I would add an example of `get_dummies` in v0.13.0txt

otherwise looks ok

Do you mean in reshaping.rst (it's not a new function btw just untested (!) and undocumented... is in wes' book :) )

but didn't you add the NA handling? (if yes, then need an example in whatsnew)

 (yes maybe an example/better example in reshape too!)

ah, I put it in the release.rst (should it also be somewhere else too?)

@hayd ahh...I just thought a mention of this new 'feature' should be in whatsnew (with an example)....seems like a nice feature

added to what's new.. think i will leave doc writing for another day/pr.

wow that build took a while: https://travis-ci.org/hayd/pandas/builds/10644820

whoop, get_dummies my favourite pandas function (now in the docs!)

@hayd awesom!

@jreback crap, this upset travis somehow (which is weird cos the last few commits have been green, and this was before I merged... in fact I linked to it above!) https://travis-ci.org/pydata/pandas/jobs/10646008