Repository: pandas-dev/pandas

Issue:
MultiIndex.get_level_values() replaces NA by another value
Test case:

```
In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: index = pd.MultiIndex.from_arrays([
    ['a', 'b', 'b'],
    [1, np.nan, 2]
])

In [4]: index.get_level_values(1)
Out[4]: Float64Index([1.0, 2.0, 2.0], dtype=object)
```

The expected output is
`Float64Index([1.0, nan, 2.0], dtype=object)`

This happens because NA values are not stored in the MultiIndex levels and the corresponding label is set to -1. Then when labels are used as indexes to values in `get_level_values()` that -1 points to the last (not null) value.

I tried to fix this by appending a NA to the values if -1 is in levels.
https://github.com/goyodiaz/pandas/commit/f028513ad96a
It needs to be improved in order to return the proper NA value (NaN, None, maybe NaT?) depending on the index type. Does this approach makes sense?



Discussion:
not a bad idea (this actually makes take deal with this correctly).

you can add a method `_fill_value` and have Index return `np.nan` and override in  DatetimeIndex return `NaT`

need some more tests, edge cases, e.g. empty index, multiple nan (datetime w/NaT), -1 at the end, beginning (you have in the middle case)

Better fix is to create a mask based on value == -1 and then fill with nan
afterwards. Otherwise you get weird behavior with setting levels and this
fix would complicate any future function that would remove extraneous
labels. Plus you'd always have to check for and remove null when outputting
levels. Might create a flag that checks for nan values so you use dtype
float instead. (or just check mask.any())

@goyodiaz something like (using @jtratner method)

```
mask = labels == -1
values = unique_values.take(labels)
values[mask] = np.nan
```

@goyodiaz do you have Travis set up? Interested if your solution passes. I didn't realize you were saying to just do it temporarily. I still think it'll create issues if you wanted to shorten levels later on, but I'm less convinced than in my previous comment :)

@jreback does append create a copy of the underlying memory? I guess it's happening in either case, but bool certainly smaller

slight tweak:

``` python
mask = labels == -1
values = unique_values.take(labels)
if mask.any():
    values = values.astype(float)
    values[mask] = np.nan
```

yes append copies

@jtratner yes, travis builds passed.
https://travis-ci.org/goyodiaz/pandas

Can you think of any possible side effect which should be tested? I did not understand well your concerns.

go ahead an open a pull-request, this will submit as a patch to the devs

Will do it in a while.

BTW there is nothing to fix with DatetimeIndex:

```
In [1]: index = pd.MultiIndex.from_arrays([
    pd.DatetimeIndex([pd.NaT, 0, 1]),
    ['a', 'b', 'b']
])
In [2]: index.get_level_values(0)[0]
Out[2]: NaT
```

But I guess this could change when numpy get proper integer nan support, if ever.

@goyodiaz - it's totally fine, I appreciate you figuring out what was going
on! I was thinking about this from an internals perspective (i.e., space
and time complexity for masking vs. appending a value to the end of a list)
and also what I'm looking to do going forward.

That said, figuring out that the issue was that it was taking the wrong
value was very helpful - thanks for that!

Behavior with NaT actually could be a bug, not sure.

@jratner I think you are right, it's a bug in `factorize()` if it is supposed to return unique values without missing values and NaT is to be treated as a missing value.

I guess I should link the PR: #5090


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -393,6 +393,9 @@ def values(self):
     def get_values(self):
         return self.values
 
+    _na_value = np.nan
+    """The expected NA value to use with this index."""
+
     @property
     def is_monotonic(self):
         return self._engine.is_monotonic
@@ -2256,7 +2259,8 @@ def get_level_values(self, level):
         num = self._get_level_number(level)
         unique_vals = self.levels[num]  # .values
         labels = self.labels[num]
-        values = unique_vals.take(labels)
+        values = Index(com.take_1d(unique_vals.values, labels,
+                                   fill_value=unique_vals._na_value))
         values.name = self.names[num]
         return values
 
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -495,6 +495,9 @@ def _mpl_repr(self):
         # how to represent ourselves to matplotlib
         return tslib.ints_to_pydatetime(self.asi8, self.tz)
 
+    _na_value = tslib.NaT
+    """The expected NA value to use with this index."""
+
     def __unicode__(self):
         from pandas.core.format import _format_datetime64
         values = self.values


Test patch:
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -1445,6 +1445,39 @@ def test_get_level_values(self):
         expected = self.index.get_level_values(0)
         self.assert_(np.array_equal(result, expected))
 
+    def test_get_level_values_na(self):
+        arrays = [['a', 'b', 'b'], [1, np.nan, 2]]
+        index = pd.MultiIndex.from_arrays(arrays)
+        values = index.get_level_values(1)
+        expected = [1, np.nan, 2]
+        assert_array_equal(values.values.astype(float), expected)
+
+        arrays = [['a', 'b', 'b'], [np.nan, np.nan, 2]]
+        index = pd.MultiIndex.from_arrays(arrays)
+        values = index.get_level_values(1)
+        expected = [np.nan, np.nan, 2]
+        assert_array_equal(values.values.astype(float), expected)
+
+        arrays = [[np.nan, np.nan, np.nan], ['a', np.nan, 1]]
+        index = pd.MultiIndex.from_arrays(arrays)
+        values = index.get_level_values(0)
+        expected = [np.nan, np.nan, np.nan]
+        assert_array_equal(values.values.astype(float), expected)
+        values = index.get_level_values(1)
+        expected = ['a', np.nan, 1]
+        assert_array_equal(values.values, expected)
+
+        arrays = [['a', 'b', 'b'], pd.DatetimeIndex([0, 1, pd.NaT])]
+        index = pd.MultiIndex.from_arrays(arrays)
+        values = index.get_level_values(1)
+        expected = pd.DatetimeIndex([0, 1, pd.NaT])
+        assert_array_equal(values.values, expected.values)
+
+        arrays = [[], []]
+        index = pd.MultiIndex.from_arrays(arrays)
+        values = index.get_level_values(0)
+        self.assertEqual(values.shape, (0,))
+
     def test_reorder_levels(self):
         # this blows up
         assertRaisesRegexp(IndexError, '^Too many levels',