API: allow scalar setting/getting via float indexer on integer indexes
State: closed

closes #12333 


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
not a lot of code changes, mostly making the tests robust & w/o repeating things.

haven't updated docs yet

ok, this is updated
- reorged indexing tests (mainly moved out float indexers from the monolith `test_indexing.py` and dropped some duplicate tests)
- bulit comprehensive float scalar/slice testing, separated out by index
- removed some net code! yeh
- cleaned up the some duplicated / awkward indexing code (_some_, but it does flow a bit better now). main issue was some duplicated code w.r.t. some of the slice indexing and checking. hopefully this will allow future changes to be in single places rather than all over the place.

net-net-net. I don't think actually much has changed from 0.17.1. (with the exception that I note when setting with `.ix` in the docs, IOW, using a float indexer on an object index now sets the label rather than positional sets).

 We have dropped the `FutureWarning` entirely on float indexers, but now only raise for `.iloc` (previously we were also raising on non-float indexes for `[]`, `.ix`, `.loc`).

So give it a whirl. If you find missing / unclear tests, let's fix em. The more comprehensive the better.

cc @shoyer @jorisvandenbossche @TomAugspurger @wesm 

Great!

Quick note: I think we should also raise on `[]` when this does positional indexing (eg in slicing) and using floats (just as `iloc`). Eg `s[1:2.5]` raised in 0.17.1, but now works with this PR.

I will try to take a more detailed look later this week.

```
In [1]: s = Series([1,2,3])

In [2]: s[1:2.5]
TypeError: cannot do slice stop value indexing on <class 'pandas.core.index.Int64Index'> with these indexers [2.5] of <type 'float'>

In [3]: s.loc[1:2.5]
Out[3]: 
1    2
2    3
dtype: int64
```

why are you calling out this a special case? This is the same path as `.loc`.

I get 

```
In [48]: s = Series([1,2,3])

In [49]: s[1:2.5]
Out[49]: array([2], dtype=int64)
```

using your branch (so something else as you show in your last comment)

Which now I note gives an array instead of series, which is also odd (wait a moment, maybe I have done something wrong in checkout out your branch, as this was a series when I first tested it)

@jorisvandenbossche yeh something odd going on. will update soon. In any event I don't see a reason to make this any different it is very consistent with the other indexers.

Very strange, it seems to depend on the values in the index:

```
In [14]: s = Series([1,2,3])

In [15]: s[1:2.5]
Out[15]: array([2], dtype=int64)

In [16]: s = Series([1,2,3], index=[1,2,3])

In [17]: s[1:2.5]
Out[17]:
2    2
dtype: int64
```

@jorisvandenbossche ok, adding tests for these off integers. 

But to come back to what I originally wanted to say: raising in []. 
You gave above the example (https://github.com/pydata/pandas/pull/12370#issuecomment-187725450):

```
In [1]: s = Series([1,2,3])

In [2]: s[1:2.5]
TypeError: cannot do slice stop value indexing on <class 'pandas.core.index.Int64Index'> with these indexers [2.5] of <type 'float'>

In [3]: s.loc[1:2.5]
Out[3]: 
1    2
2    3
dtype: int64
```

This _raises_ with floats when slicing in `[]`. And this is what I meant that I think should happen, but it is not what currently happens in the PR. 
What was your purpose in showing that example? That you think that should be the behaviour? If so, we are in full agreement :-) If not, it's not clear from that comment what you mean.

oh, I just generalized things. This is particular was not exactly tested :< oh
ok in agreement that this is NOT a special case then? gr8 (that example was from 0.17.1 btw)

@jreback yep, ok, in agreement it is not a special case and should stay as it was in 0.17.1 (your example). The confusion was that is actually behaved differently in the PR, which I tried to point out :-)

@jorisvandenbossche no, my example is from 0.17.1. But I am NOT treating it as a special case, which means this WILL work the same as `.loc` (aside fro the `np.array(..)` issue which I have to track).

why should this raise?

> why should this raise?

Because `[]` is not like `loc` in this case, but like `iloc` (which raises)

@jorisvandenbossche no i disagree, this is label based. Again this is introducing special cases where none should exist.

I am afraid that is not label based, but positional, and this is a special case that is already present ...

```
In [12]: pd.__version__
Out[12]: u'0.17.1'

In [13]: s = pd.Series(range(2,6), index=range(2,6))

In [14]: s
Out[14]:
2    2
3    3
4    4
5    5
dtype: int64

In [15]: s[2:4]
Out[15]:
4    4
5    5
dtype: int64

In [16]: s.loc[2:4]
Out[16]:
2    2
3    3
4    4
dtype: int64
```

this is exactly the `.ix` path

I am not saying this is ideal :-) But it is in any case positional and not label based, and that is not something we can change lightly

> this is exactly the .ix path

that does not seems to be the case in practice, as `ix` _is_ indeed label based in this case

```
In [17]: s.ix[2:4]
Out[17]:
2    2
3    3
4    4
dtype: int64
```

my point is that this is EXACTLY THE SAME PATH. It IS not a special case in this branch, nor should it. 

IIRC that was accidental before.

ok, you are right it IS positional. So this seems entirely reasonable then. Yes would be a slight API change, becuase now we accept something we don't before, but it IS consistent. Yes it IS like `.iloc`, but so is `[]` in this case. But again removing this is a special case.

```
In [1]: s = pd.Series([1,2,3],index=[4,5,6])

In [2]: s[1:3]
Out[2]: 
5    2
6    3
dtype: int64

In [3]: s[1.5:3]
Out[3]: 
5    2
6    3
dtype: int64

In [4]: s.ix[1.5:3]
Out[4]: Series([], dtype: int64)

In [5]: s.loc[1.5:3]
Out[5]: Series([], dtype: int64)
```

> ok, you are right it IS positional. So this seems entirely reasonable then. 

I am not sure how to interpret this (is it my or your point that is reasonable :-) ?) Can you show what you think the behaviour should be? (or is it like your last code example?)

My point of vue: independent of it is in `[]/ix/iloc`, we have the concept of 'positional' indexing. For `[]/ix` it depends on some factors if it will be positional or not. But once we have positional indexing, I would like to see the behaviour consistent for all indexer types within the concept of positional indexing. I think it would be really unfortunate that we introduce inconsistencies within that concept.

@jorisvandenbossche ok finally had time to update this.

lmk what you think

@jreback I still think positional indexing should always raise when using floats, also in slicing in `[]` (as it already raised in 0.17). See my last post above. 
What is your rationale to make an exception and now allow it?

Will try to take a more detailed look tomorrow

can u five an explicit example where u think I am doing positional indexing with floats? for scalars and slicing

It is for slicing (not scalars):

```
In [13]: pd.__version__
Out[13]: '0.18.0rc1+79.gff90e0a'

In [14]: s = pd.Series(range(2,6), index=range(2,6))

In [17]: s.loc[2:4]
Out[17]:
2    2
3    3
4    4
dtype: int64

In [18]: s[2:4]
Out[18]:
4    4
5    5
dtype: int64

In [19]: s[2:4.0]   <---------- this is positional indexing with floats -> so should raise?
Out[19]:
4    4
5    5
dtype: int64

In [20]: s.iloc[2:4.0]
TypeError: cannot do slice indexing on <class 'pandas.indexes.numeric.Int64Index
'> with these indexers [4.0] of <type 'float'>
```

vs

```
In [6]: pd.__version__
Out[6]: u'0.17.1'

In [7]:  s = pd.Series(range(2,6), index=range(2,6))

In [8]: s[2:4.0]          <------------ does raise in 0.17.1
TypeError: cannot do slice stop value indexing on <class 'pandas.core.index.Int6
4Index'> with these indexers [4.0] of <type 'float'>
```

ok let me adjust that

@jorisvandenbossche ok updated. I _think_ that we are now consistent with 0.17.1 (and now fully tested).

This looks good! Thanks a lot

An edge case: if you have mixed dtype index, you get some inconsistencies:

```
In [48]: s2 = pd.Series([1,2,3], index=['a', 'b', 'c'])

In [49]: s3 = pd.Series([1,2,3], index=['a', 'b', 1.5])

In [50]: s2[1.0]
TypeError: cannot do label indexing on <class 'pandas.indexes.base.Index'> with
these indexers [1.0] of <type 'float'>

In [51]: s3[1.0]
Out[51]: 2
```

So the second case (`s3[1.0]`) appears to do positional indexing with a float (which should raise?)