ENH: add StringMethods (.str accessor) to Index, fixes #9068
State: closed

fixes #9068


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: Strings
description: String extension data type and string data

Comments:
@jreback please let me know what you think of this, thanks 

this looks good
pls add a release note in v0.16.1

may also want to update the docs to mention u can do this on Index (I think it's in text.rst iirc)

@jreback both release note and `text.rst` have been updated 

Nice, but 2 points:
- There are some function returns `bool`, and `Index(dtype=bool)` is not usable very much (related to #8875). It may be better to return `np.array` in these cases. 
- Should prohibit to use `return_type="frame"` in `split` ?

@sinhrks can you elaborate on the first point (about `bool`)? an example?

@mortada 

I think add a tests for all of the `.str` methods (e.g. to test the return types issue), I think you will have to raise if `return_type='frame'` is defined

@jreback just added a check and unit test for the case of `Index` and `return_type='frame'`

@sinhrks I'm looking into the `bool` issue you referenced

@jreback @mortada some str methods return booleans, e.g., `str.isalpha`. For series, it makes sense to return a  As @sinhrks points out, we probably don't want to return a new Index object for these values, but rather a plain numpy array.

@shoyer, @sinhrks I see, indeed returning `np.array` in the `boolean` case seems a lot more useful. 

@jreback an example would be:

```
idx = Index(['a1', 'a2', 'b1', 'b2'])
s = Series(range(4), index=idx)
```

If we return an `np.array` then an expression like

```
s[s.index.str.startswith('a')]
```

would work naturally, whereas if we return a boolean `Index` that's not very useful

Just added a check for the `boolean Index` case, it will now return `np.array` instead. Please take a look, thanks! 

@mortada yeh I think we need to do something for

`Index(['foo_bar','bah_baz']).str.split('_',return_type='frame')`

@mortada Thanks for quick action:)

Didn't look in detail into this, but a quick question: what is the return type? I suppose in general a new index? But what if the method returns multiple elements for each entry? (eg returning a frame for a series)?

@jorisvandenbossche the return type is `Index` unless the `dtype` is `bool`, in which case the return type will be `np.array`.

if the method returns multiple elements it will be an `Index` of `list`s, and won't be a `DataFrame`. If the user specifies `return_type='frame'` it will raise. Here's an example:

```
In [1]: from pandas import Index
In [2]: idx = Index(['a b c', 'd e', 'f'])

In [3]: idx.str.split()
Out[3]: Index([['a', 'b', 'c'], ['d', 'e'], ['f']], dtype='object')

In [4]: idx.str.split(return_type='frame')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-d22b631e0f0f> in <module>()
----> 1 idx.str.split(return_type='frame')

/Users/mortada_mehyar/code/github/pandas/pandas/core/strings.py in split(self, pat, n, return_type)
    987     @copy(str_split)
    988     def split(self, pat=None, n=-1, return_type='series'):
--> 989         result = str_split(self.series, pat, n=n, return_type=return_type)
    990         return self._wrap_result(result)
    991

/Users/mortada_mehyar/code/github/pandas/pandas/core/strings.py in str_split(arr, pat, n, return_type)
    652         raise ValueError("return_type must be {'series', 'frame'}")
    653     if return_type == 'frame' and isinstance(arr, Index):
--> 654         raise ValueError("return_type='frame' is not supported for string "
    655                          "methods on Index")
    656     if pat is None:

ValueError: return_type='frame' is not supported for string methods on Index
```

@shoyer added examples to doc
@jreback now using `is_bool_dtype` 

please take a look, thanks. 

Only problem with raising on `return_type='frame'` is that this will be the default in some case (eg the method partition that is being added now), and not so nice to raise with the defaults

By the way, I think this is a very nice feature, especially useful for cleaning up your column names (stripping whitespace at beginning or end -> typical problem you see on SO, replacing whitespace with `_`, convert to all lower case, ..) For the docs, maybe it would be useful to also focus on this and give an example where the column names are used (but maybe not needed for this PR)

@jorisvandenbossche I don't think its a problem if certain methods which have a `return_type='frame'` raise if you try to call them on an `Index`. The only other option is to actually return a frame as I think the purpose is what you mentioned above, to allow cleaning type ops on an `Index`, which you expect to get an `Index`. 

Another option is to exclude the methods entirely from the `Index` (meaning ones that have a default `return_type='frame'`)

@jreback I added `reture_type='index'` as a alias for `'series'` as you suggested, please take a look 

lgtm. @jorisvandenbossche 

@mortada Are you sure you committed it? I still only see the frame/series options (or I am overlooking it completely ..)

@jorisvandenbossche oops I had to do another rebase and missed the last commit, thanks for catching! 

@jorisvandenbossche it's fixed, please take a look thanks 

The thing I am thinking now: won't it be confusing for users to get a Index back even if he/she supplies `return_type='series'` on a Index.str.split, or to get a series back with `Series.str.split(.., return_type='index')`? 

I was also thinking of another option: an `expand` keyword (or another name), that indicates for `False`: give same dimension back (so for series/index keep it a series/index), and for `True`: expand series to dataframe. 
This would then be a duplicate for `return_type` of course. But the `return_type` was only introduced in 0.15.1 (and for `partition` it is still in a PR), so if we want to change this: better now than later. Or has this ship sailed?

But maybe we shouldn't block this PR with this discussion and open a new issue for this. So I am OK with merging.

@jorisvandenbossche 

how about we just do

`return_type='same'|'expand'` to satisfy this need? (and can be easily back-compat)

but let's do that in another issue / PR

@jreback @jorisvandenbossche I'm happy to do another PR for the `return_type` 

ok, merging this, but @mortada I would appreciate another issue (and PR!) for the `return_type` issue. 

@mortada really a nice improvement!

I will open an new issue to discuss the `return_type` thing

@jorisvandenbossche thanks! 

Another thing you had mentioned was expanding the docs to cover some questions on SO about cleaning up column names. Could you please point me to some of those SO questions? I can add more examples to the docs. 

@mortada I don't have a specific SO question on my mind, but you often see questions like: "why does `df['my_col']` give a KeyError as it is in the columns?" with response -> "check `df.columns`, maybe you have starting or trailing spaces in the names" (something you don't easily see in the console output, you can only detect it through the alignment)

So for me, this are typical ways to clean up column names:
- remove starting/trailing whitespace (eg leftover of reading it in)
- convert all columns names to lower case (eg something typical you would want for writing it to certain databases/on certain platforms to not have to deal with case sensitivity issues)
- remove alle whitespace in the column names, or convert all whitespaces to underscores ("my col" -> "my_col")