ENH/BUG: Fix names, levels and labels handling in MultiIndex
State: closed

This PR covers:

Fixes: #4202, #3714, #3742 (there are might be some others, but I've blanked on them...)

Bug fixes:
- `MultiIndex` preserves names as much as possible and it's now harder to overwrite index metadata by making changes down the line.
- `set_values` no longer messes up names.

External API Changes:
- Names, levels and labels are now validated each time and 'mostly' immutable.
- names, levels and labels produce containers that are immutable (using new containers `FrozenList` and `FrozenNDArray`)
- `MultiIndex` now shallow copies levels and labels before storing them.
- Adds `astype` method to `MultiIndex` to resolve issue with `set_values` in `NDFrame`
- Direct setting of levels and labels is "deprecated" with a setter that raises a DeprecationWarning (but still functions)
- New `set_names`, `set_labels`, and `set_levels` methods allow setting of these attributes and take an `inplace=True` keyword argument to mutate in place.
- `Index` has a `rename` method that works similarly to the `set_*` methods.
- Improved exceptions on `Index` methods to be more descriptive / more specific (e.g., replacing `Exception` with `ValueError`, etc.)
- `Index.copy()` now accepts keyword arguments (`name=`,`names=`, `levels=`, `labels=`,) which return a new copy with those attributes set. It also accepts `deep`, which is there for compatibility with other `copy()` methods, but doesn't actually change what copy does (though, for MultiIndex, it makes the copy operation slower)

Internal changes:
- `MultiIndex` now uses `_set_levels`, `_get_levels`, `_set_labels`, `_get_labels` internally to handle labels and levels (and uses that directly in `__array_finalize__` and `__setstate__`, etc.)
- `MultiIndex.copy(deep=True)` will deepcopy levels, labels, and names.
- `Index` objects handle names with `_set_names` and `_get_names`.
- `Index` now inherits from `FrozenNDArray` which (mostly) blocks mutable methods (except for `view()` and `reshape()`)
- `Index` now actually copies ndarrays when copy=True is passed to constructor and dtype=None


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
as an aside, this leads to weird things like

``` python
idx.names = list("abcdef")
idx.droplevel("d") # Gives an index out of range-esque error
```

yep i used to get that and i hacked around by recreating frames/series and some other trickery that will hopefully never see the light of day :)

@cpcloud okay, I think I'm understanding more of the problem here: when you slice an index, the levels remain the same...e.g.:

``` python
chunklet = idx[-3:]
assert chunklet.levels[0] is idx.levels[0] # True
```

So, when you assign names, it mutates the underlying levels of _both_. This seems to follow the convention in `__new__` to assign names to the levels directly...but if you do this after the fact, you end up mutating earlier copies.

Moreover, if you pass levels to the `MultiIndex` constructor, it doesn't copy them, e.g.:

``` python
new_idx = MultiIndex(idx.levels, idx.labels)
assert new_idx.levels[0] is idx.levels[0] # True
```

so what ought to be happening here? Should names be assigned to underlying levels or just left alone?

@cpcloud made those two changes and rebased.

@wesm any reason why `names` should be settable to a sequence larger than `nlevels` after the fact?

nope. i just never added any validation and left names as a simple attribute. 

+1 for a validator here...@jtratner what breaks?

this is basically same issue as #3742

indeed...close this one, that one?

def should be some kind of validator on setting of `names` attrib in index; weird things can happen if e.g. levels are changed

can close that one (maybe move example to here though) as another test case

Example from #3742 cc @thriveth 

I have raised the issue in [this question](http://stackoverflow.com/questions/15621143/pandas-df-set-value-method-erases-resets-column-names-of-multiindex/15624426#15624426) on Stack Overflow, but I'm not sure it ever made it to the Pandas issue tracker.

I have a MultiIndex'ed DataFrame which I want to expand by using `set_value()`, but doing this destroys the `names` attribute of the index. This does not happen when setting the value of an already existing entry in the DataFrame. An easily reproducible example is to create the dataframe by:

``` python
lev1 = ['hans', 'hans', 'hans', 'grethe', 'grethe', 'grethe']
lev2 = ['1', '2', '3'] * 2
idx = pd.MultiIndex.from_arrays(
    [lev1, lev2], 
    names=['Name', 'Number'])
df = pd.DataFrame(
    np.random.randn(6, 4),
    columns=['one', 'two', 'three', 'four'],
    index=idx)
df = df.sortlevel()
df 
```

This shows a neat and nice object, just as I expected, with proper naming of the index columns. If I now run:

``` python
df.set_value(('grethe', '3'), 'one', 99.34)
```

the result is also as expected. But if I run:

``` python
df.set_value(('grethe', '4'), 'one', 99.34)
```

The column names of the index are gone, and the `names` attribute has been set to `[None, None]`.

also  #3714 same issue too, except assigning to levels, needs validation as well

This is what I was trying to get across earlier :) If you pass levels through the MultiIndex constructor, they have their names set to the `names` keyword argument or to `None` (and are not copied).

``` python
        if names is None:
            # !!!This is why names get reset to None
            subarr.names = [None] * subarr.nlevels
        else:
            if len(names) != subarr.nlevels:
                raise AssertionError(('Length of names (%d) must be same as level '
                                      '(%d)') % (len(names),subarr.nlevels))

            subarr.names = list(names)

        # THIS IS WHERE NAMES GET OVERWRITTEN WITHOUT BEING COPIED
        # set the name
        for i, name in enumerate(subarr.names):
            subarr.levels[i].name = name
```

An easy solution would be for the MultiIndex to copy the levels it receives first and then rename them. Then, any time you set the names attribute, it would set the name on every level and the `names` attribute on the copied levels, e.g. force `_ensure_index` to make a copy if it's already an index, so that at this point you'd be all good:

``` python
levels = [_ensure_index(lev) for lev in levels]
```

Because later on it just assigns it to the object:

``` python
subarr.levels = levels
```

I think levels should be a cached_readonly property, so that you don't end up creating indices twice (once in the `levels =` part and another on the setattr levels). If not,you could turn levels into a property that makes a copy before setting on the object.

If this all makes sense to you, I can write it up into a PR soon.

@jreback @cpcloud Are `Index` and `MultiIndex` supposed to be immutable? If so, then neither should be able to set names directly (like MultiIndex does in its `__new__` method). I think it makes more sense to allow them to mutate on names...much easier internally.

well they are immutable on the values
but not names / levels (though they really should be)
because state sharing is a problem then

@cpcloud I learned something today: Python cheats and compares lists first by object equality, _then_ checks individual items...I'm wondering if this error is lurking other places in the code:

``` python
>>> arr1, arr2 = np.array(range(10)), np.array(range(10))
>>> assert [arr1, arr2] == [arr1, arr2] # succeeds
>>> assert [arr1, arr2] == [arr1, arr2.copy()]
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
>>> assert_almost_equal([arr1, arr2], [arr1.copy(), arr2])
True
```

I found this in `test_copy`, where a test case was not actually testing what it claimed to be testing :P .

``` python

    def test_copy(self):
        i_copy = self.index.copy()

        # Equal...but not the same object
        self.assert_(i_copy.levels == self.index.levels)
        self.assert_(i_copy.levels is not self.index.levels)
```

Actually that whole test is not great, because it's actually (was) testing that two different lists were created.

@jtratner that is good to know.

i always assumed that sequence equality was done recursively. [docs](http://docs.python.org/2.7/reference/expressions.html#not-in) seem to imply that that is the case..strange

@cpcloud yep, that's exactly right. (I'm sure that's implementation dependent, but it is important to know if using numpy arrays.

maybe some sort of lightweight `Levels` class is in order? overloading `__eq__` to compare levels or something more general for use with `labels` and `levels`?

Maybe you could use `Categorical`?

@cpcloud maybe. Right now I just changed levels, names and labels to return tuples and used `assert_almost_equals`. Very simple.

On that note - is it okay to make that change? Much easier to prevent erroneous assignment (like `index.levels[i] = some_new_index`) by producing an immutable object  than to try to use something else. Have to change a ton of test cases that assumed `names` produces a list, but hopefully that's not a big deal...

`Categorical` might be jumping the gun a bit. I would rather have it immutable, but that prevents sharing. a  perf comparison might be useful here

@cpcloud well, I'm using shallow copies/views, which I think means that only metadata is copied. This is necessary anyways, because you want to be able to set names on the underlying levels without worrying about messing up other indexes.

(I just pushed what I have so far - it's failing because a _ton_ of tests assume that index names, levels and labels will be lists...)

@cpcloud also, categoricals aren't immutable, right?

Just the `levels` attribute because it's an `Index`

You'd still have to have a `tuple` of `Categorical` for multiindex which is not that different from what it sounds like you're doing