ENH: allow index of col names in set_index GH10797
State: closed

See #10797 

The desired result is to allow slices of the column index to be used in set_index. Like this:

```
>>> import pandas as pd
>>> data1 = 'x'*5 + 'y'*5
>>> data2 = ['a','b','c','d','e']*2
>>> data3 = range(10)
>>> data_dict = {'Cat': list(data1), 'SubCat': list(data2), 'Vals':data3}
>>> df = pd.DataFrame(data_dict)
>>> ordered_df = df[['Cat', 'SubCat', 'Vals']]
>>> ordered_df.set_index(ordered_df.columns[:2])
            Vals
Cat SubCat      
x   a          0
    b          1
    c          2
    d          3
    e          4
y   a          5
    b          6
    c          7
    d          8
    e          9
```

However, passing an Index of values to set_index currently has similar behavior to passing an array of values; the index will be used as the values of the new index. As to not break this behavior, this PR only treats the index like a list of column labels when the length of the index is not the same as the length of the DataFrame.


Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: MultiIndex

Comments:
you don't need to open a new PR when changing things , just force push to the old one.

@jreback - not sure if this approach is better or worse, but I gave it a shot. Instead of checking for specific values in the index, I've written it to check that the array underlying the `Index` passed as `keys` is the same array underlying the frame's `columns`. My thinking is that if the user is passing a slice of the columns index, they almost surely intend to use the Index as a list of existing columns to move to the row index. However, if the `Index` passed to `set_index` is constructed independently of `columns`, the user's intent is almost surely to set that as the index.

I feel like this will give the most intuitive behavior, but we do have the case where two `Index` objects that are identical (in terms of their values) yield different behavior.

What do you think?

I _think_ that we could allow a passed actual `Index` to simply set. That is unambiguous and clear. While other list-likes will work as they do now.

Further I think it makes sense to accept (only an `Index`) on `Series`. So this will be de-facto:

`df.set_index(i)` <-> `df.index = i`

@StephenKappel can you update

Yes; I will update. Just been tied up for the last week with some other stuff...

@StephenKappel np thanks

I don't feel like any solution here is a net improvement. Currently, when passed an `Index`, `set_index()` uses the passed index as the new index (ie. `df.set_index(i) <-> df.index = i`). Changing this behavior seems like a bad idea. The current behavior is used in the Pandas code base and tests (ex. https://github.com/pydata/pandas/blob/master/pandas/io/sas.py#L468 and https://github.com/pydata/pandas/blob/master/pandas/tests/test_multilevel.py#L2242-L2255) and presumably other projects. The only place that this current behavior is undesirable is when a slice of the columns index is being used with the intent of using the `Index` like a `list` of column labels.

`ordered_df.set_index(ordered_df.columns[:2])`

My initial attempts were kludgy hacks that attempted to guess the user's intention based on the characteristics of the passed `Index`. To avoid any ambiguity, I don't see any good solution other than a keyword argument. But, this doesn't seem like any great improvement over the current solution of having the user coerce the `Index` to a `list` before passing it to `set_index()`.

`ordered_df.set_index(list(ordered_df.columns[:2]))`

vs. 

`ordered_df.set_index(ordered_df.columns[:2], keys_as_col_labels=True)`

Thoughts?

@StephenKappel 

re-reading, I think this _would_ be extra confusing w/o a different keyword.

Ok, let's instead repurpose this to a doc-example in the set_index section [here](http://pandas.pydata.org/pandas-docs/version/0.17.1/indexing.html#set-reset-index)

and put an example for `.set_index(an_index)` (you can in fact use the example in the issue).

If its more clear then use a separate sub-section (or maybe put it all in a note).

Further can you enhance the doc-strings with examples like this?

thanks

pls rebase/update. 

Sorry for the delay; I'll update the documentation this week.

@StephenKappel I am a bit concerned that this is going to be confusing. `.set_index` by definition has to take names from `self` and use them to set, rather than taking in a completly new object.

yes it would be convenient to do:

`df.set_index(index)` rather than `df.index = index`. But can this be done completely unambiguously?

Yes - my understanding was that the current code shouldn't change, but I was going to update the docs to include examples of how the desired column subsetting could be done in the current code (i.e. converting index to a list). I'll remove my code changes and add the doc changes.

@StephenKappel the problem with docing this is that we have very different behavior for a list and an Index. 

I think this needs a re-think on the API.