ENH: Add set_index to Series
State: closed

- [x] closes #21684
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] examples for series.set_index
- [x] whatsnew entry

Following https://github.com/pandas-dev/pandas/issues/21684#issuecomment-401584844, I unified the method to `core/generic.py`. If this is approved in principle, I'll write some tests (both for DF/Series) and the whatsnew. On that note, it seems to me that `df.set_index` - particularly its kwargs - are basically untested. The only tests I found are:
- `pandas/tests/frame/test_indexing.TestDataFrameIndexingDatetimeWithTZ.test_set_reset`
- `pandas/tests/frame/test_indexing.TestDataFrameIndexingUInt64.test_set_reset`

both of which don't test any of the kwargs.


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
look in alter_axes

there are a ton of tests 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=h1) Report
> Merging [#22225](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/cb31b2b0912c99e40b22a26e598e640548ea8f14?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `97.14%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22225/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22225      +/-   ##
==========================================
- Coverage   92.38%   92.37%   -0.01%     
==========================================
  Files         166      166              
  Lines       52395    52398       +3     
==========================================
+ Hits        48403    48404       +1     
- Misses       3992     3994       +2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.8% <97.14%> (-0.01%)` | :arrow_down: |
| #single | `43.03% <60%> (+0.02%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.85% <100%> (-0.08%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.69% <100%> (+0.07%)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.73% <90%> (+0.17%)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.93% <0%> (-0.75%)` | :arrow_down: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `97.68% <0%> (-0.34%)` | :arrow_down: |
| [pandas/core/arrays/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3RpbWVkZWx0YXMucHk=) | `88.09% <0%> (-0.1%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88% <0%> (-0.1%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `99.34% <0%> (-0.03%)` | :arrow_down: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `98.52% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/arrays/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3BlcmlvZC5weQ==) | `98.52% <0%> (-0.01%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/pandas-dev/pandas/pull/22225/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=footer). Last update [cb31b2b...7f35d2c](https://codecov.io/gh/pandas-dev/pandas/pull/22225?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback I cleaned up the tests for `df.set_index` before I moved to the ones for `series.set_index`. Since I'm guessing that this would be preferred as separate PRs, here's just the cleanup for `df`, without any changes to functionality (except some better warnings): #22236
Hello @h-vetinari! Thanks for updating the PR.









Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on January 06, 2019 at 14:40 Hours UTC
Rebased this on #22236 #22526 and split of the changes from #22486 into a separate commit.

If you want to review this without #22486, just follow this link: ```https://github.com/pandas-dev/pandas/pull/22225/commits/3edbaeae7731885bf418c5ac50d0a9bcf82c0ed6```
@gfyoung @jreback @WillAyd @TomAugspurger 
Could someone please retrigger the failed travis job? Crazy waiting times on Circle/Appveyor currently.
Now that all the preps (#22236 #22526 #22486) are out of the way, can finally tackle this seemingly simple PR - adding `set_index` to `Series`. I moved the code from `frame.py` to `generic.py`, as mentioned in the issue by @gfyoung.

It's also green (Circle is behind but already passed before the last commit for linting)
@jreback @gfyoung 
ping :)
actually i just re-read this. This API is completely different than that of DataFrame, yet uses the same name.

This takes an array of values, while the DataFrame does a list of keys. Adding this is going to be very confusing. This is de-facto ``series.index = array``. Which is not the same API.
we don't really have something like this. There were several attempts to have DataFrame.set_index take an array as well, but these never got off the ground.

cc @TomAugspurger @jorisvandenbossche 
@jreback 
The only difference is that `DataFrame.set_index` *also* takes column keys. But it's completely usable (on master) to pass arrays, list of arrays, or even a mix of arrays and column keys into `keys` (which is confusing too...).

So effectively, `DataFrame.set_index` has `arrays_or_keys` and `Series.set_index` only has `arrays`. The other kwargs (`append`, `inplace`, `verify_integrity`) make sense for both as well.
@jreback 
> There were several attempts to have DataFrame.set_index take an array as well, but these never got off the ground.

I'm not sure *when*, but they certainly did get off the ground (this is v.0.23, before you asked me to expand the signature in #22486; also check the tests added/expanded in #22236):

```
>>> import pandas as pd
>>> import numpy as np
>>> pd.__version__
'0.23.4'
>>>
>>> df = pd.DataFrame(np.random.randint(0, 10, (4, 4)), columns=list('abcd'))
>>> df.set_index(['a',          # label
...               df.index,     # Index
...               df.b ** 2,    # Series
...               df.d.values,  # ndarray
...               list('ABCD'), # list
...               'c'])         # label again
              b  d
a   b      c
3 0 4  9 A 3  2  9
4 1 1  2 B 3  1  2
8 2 9  5 C 9  3  5
5 3 25 1 D 7  5  1
```
@TomAugspurger 
switched to `textwrap.dedent`. Since my answer to your question above has been eaten by github almost completely, here it is again:

> Function signatures of subclasses should match the parent. We should also accept drop in the Series version, even if it's not used..

That to me would then be an argument of *not* unifying the methods in `generic.py`. A parameter `keys` makes zero sense for Series (similarly for `drop`, which would be very confusing as well).

@jreback @TomAugspurger 
So, do you want me to implement `Series.set_index` separately (which would be duplicating code somewhat), or stay with a changed signature compared to `NDFrame`?

Personally, I don't see why the signatures can't be different - Series and DataFrame *are* different beasts on many levels, so why shouldn't their methods reflect that?
@h-vetinari this is simply a different function. we resisted implementing this a long time ago because it is simply confusing. It needs a new name.
@jreback 
Have you looked at https://github.com/pandas-dev/pandas/pull/22225#issuecomment-434719598 ?

`DataFrame.set_index` is fully compatible with using arrays. How is setting the index to an array not in line with the name `set_index`? This is exactly what this method does.

IMO the fact that `Series.set_index` does not support labels is a *much* smaller difference compared to the core functionality of setting the index to an array (I recall you being strongly against directly setting attributes like `.index` in general, but there's currently no other option for `Series`).

I don't care if the implementation is shared or not, but `Series` IMO needs a `set_index` with -- at least -- the ability to set the index to a given list-like / arrary.
@h-vetinari you are confusing the purpose of ``.set_axis``. It may be that the added tests previously are wrong. The problem with ``.set_index`` on a DataFrame with an array is that it technically can work with an array and not keys. (meaning its not unambiguous)
@jreback 
> @h-vetinari you are confusing the purpose of .set_axis.

I think this would not be a very well-named/designed API - it should be easy to avoiding direct setting of attributes. So instead of **set**ting the **`.index`** attribute directly (which works, but should be discouraged)
```
df.index = ['a', 'b', 'c']
```
it should be `df.set_index`.

>  It may be that the added tests previously are wrong.

The above behaviour (`df.set_index(['a', 'b', 'c'])`) works since long ago; the added tests just increased the number of tested combinations.

> The problem with .set_index on a DataFrame with an array is that it technically can work with an array and not keys. (meaning its not unambiguous)

I get (now) that you don't like the mixing of the functionality between `set_index` and `set_axis` but that is a very deeply ingrained reality already (and IMO makes sense from the point of usability/simplicity).
> I get (now) that you don't like the mixing of the functionality between set_index and set_axis but that is a very deeply ingrained reality already (and IMO makes sense from the point of usability/simplicity).

I am not sure why you think this is true. exactly how would this be done today?
@jreback 
>> I get (now) that you don't like the mixing of the functionality between set_index and set_axis but that is a very deeply ingrained reality already (and IMO makes sense from the point of usability/simplicity).

> I am not sure why you think this is true. exactly how would this be done today?

Let's start as follows:
```
>>> import pandas
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.23.4'
>>> df = pd.DataFrame(np.random.randint(0, 10, (3, 3)), columns=list('abc'))
>>> df
   a  b  c
0  3  7  5
1  0  9  4
2  2  8  6
```
IIUC, you're saying that:
* for keys, one should use `df.set_index('a')`
* for arrays, one should use `df.set_axis(df.a.values, inplace=False)`

However, the functionality of `df.set_index` is a proper superset of `df.set_axis(axis=0)`**, in that it's possible to fill `keys` with *only* `Series`/`Index`/`ndarray`/`list` etc.:
```
>>> df.set_index(pd.Index(df.a))  # same result as Series directly below
>>> df.set_index(df.a) 
   a  b  c
a
3  3  7  5
0  0  9  4
2  2  8  6
>>> df.set_index(df.a.values)  # same result as list directly below
>>> df.set_index([[3, 0, 2]])
   a  b  c
3  3  7  5
0  0  9  4
2  2  8  6
```

In fact, it's even possible to mix all of those input types with column keys in `keys`, as I pointed out in in https://github.com/pandas-dev/pandas/pull/22225#issuecomment-434719598.

IMO:
* `set_axis` should be able to do everything that `set_index` does, and just switch between operating on index/columns based on the `axis`-kwarg (after all, `index` and `columns` are the two axes of a DF).
* `set_index` *should* be able to process list-likes (as it currently does)
* it could be considered to add a method `set_columns` on a `DataFrame`
* The `axis`-kwarg of `set_axis` should just switch between the behaviour of `set_index` (i.e. dealing with keys *and* array-likes) and `set_columns`.
* For Series, the `set_index` and `set_axis` methods would/should be exactly the same.

** there is one caveat, in that lists (and only lists; out of all containers) need to be wrapped in another list, i.e. `df.set_index([[3, 0, 2]])` instead of `df.set_index([3, 0, 2])`
@jreback 
Can you have a look at my last response please and take another look at the PR. `set_index` *can* (and should) be able to deal with arrays.
@h-vetinari I have already commented. what you are doing is completely ambiguous and broken, and was never the intention of ``.set_index()`` to act as ``.set_axis``. It *happens* to work in the particular case you gave. That is exactly the problem. Until / unless you address this, we are not moving forward. You are trying to expand functionaility that is not allowed.

by your definition above this should produce a new Index, but instead this actually does what the doc-string says.
```
In [17]: df.set_index(['a', 'b', 'c'])
Out[17]: 
Empty DataFrame
Columns: []
Index: [(0, 6, 5), (5, 6, 6), (9, 2, 7)]

```

My point is using ``.set_index()`` for actual  list values and NOT column labels is, while not prohitibed not unambiguous.
@jreback 

The list-of column-labels is the *only* case that is ambiguous and needs some care.

And using arrays is not ambiguous or broken, even the (v0.23.4)-docstring mentions:
`keys : column label or list of column labels / **arrays**`

So, coming back to your example:
```
>>> df.set_index([['a', 'b', 'c']])
   a  b  c
a  3  7  5
b  0  9  4
c  2  8  6
>>> df.set_index(['a', 'b', 'c'])
Empty DataFrame
Columns: []
Index: [(3, 7, 5), (0, 9, 4), (2, 8, 6)]
```
I mentioned as much above:
> ** there is one caveat, in that lists (and only lists; out of all containers) need to be wrapped in another list, i.e. df.set_index([[3, 0, 2]]) instead of df.set_index([3, 0, 2])

The ambiguity for DataFrame *already* exists, but for `Series` there is *no* ambiguity (since the column keys aspect is disabled). In effect you're asking me to change something you don't like that:
* is already working for DataFrames
* is unchanged by this PR
* is (IMO) neither inconsistent/ambiguous nor broken

I went into all the points you addressed in detail in https://github.com/pandas-dev/pandas/pull/22225#issuecomment-436199850 and previous comments. 
@h-vetinari we had quite some discussion about this: https://github.com/pandas-dev/pandas/issues/14829

and never reached resolution. This is an API question.
Both this PR and #23192 have had persistent a `ResourceWarning` the last few runs. I first thought it was a flaky thing like those Warnings used to be, but this time, it stayed, and I can reproduce it locally (not with `pytest pandas/tests/io/test_parquet.py`, but at least with `pytest pandas/tests/io`).

For example in https://travis-ci.org/pandas-dev/pandas/jobs/453820311 or https://travis-ci.org/pandas-dev/pandas/jobs/453822449:
```
sys:1: ResourceWarning: unclosed <socket.socket fd=16, family=AddressFamily.AF_INET, type=2050, proto=0, laddr=('0.0.0.0', 0)>
sys:1: ResourceWarning: unclosed <socket.socket fd=15, family=AddressFamily.AF_INET, type=2050, proto=0, laddr=('0.0.0.0', 0)>
```
and 
```
=============================== warnings summary ===============================
pandas/core/frame.py::pandas.core.frame.DataFrame.to_parquet
  /home/travis/build/pandas-dev/pandas/pandas/io/parquet.py:129: ResourceWarning: unclosed file <_io.BufferedReader name='df.parquet.gzip'>
    **kwargs).to_pandas()
```

There's also a `stderr` (or `stdout`) warning from the parser-tests surfacing somewhere:
```
..............................................................x...........................................................s....
........................................Skipping line 3: Expected 3 fields in line 3, saw 4
.......................................s.......................................................................................
```

I've narrowed the ResourceWarning down to the parquet-s3 tests, but couldn't figure out a way to fix it despite *many* tries, so I'm skipping them for now.
@jreback 
> @h-vetinari we had quite some discussion about this: #14829 
and never reached resolution. This is an API question.

The discussion you linked is only tangentially related, plus the functionality I'm talking about is the currenct state of the library. Do you want me to open an issue to discuss what (if anything) to do with `df.update`?
@jreback
> > @h-vetinari we had quite some discussion about this: #14829 and never reached resolution. This is an API question.

> The discussion you linked is only tangentially related, plus the functionality I'm talking about is the currenct state of the library. Do you want me to open an issue to discuss what (if anything) to do with `df.update`?

Could you please tell me where you want to have this discussion? In a separate issue or in this PR?

BTW, failure is just a flaky hypothesis test.
Since I haven't been able to get an answer on how to proceed, I opened #24046 to discuss the capabilities of `DataFrame.set_index`. 
@TomAugspurger @jorisvandenbossche 
This would be ready (modulo a new docstring warning). Any comment before the cut-off?
Apologies for the delay. I think I'm +1 to the idea of this. Taking a look at the diff now.
I can't tell if the doc build error from https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=6464 is real, or whether it's a bug in the linter. I can take a look later if you want.