Repository: pandas-dev/pandas

Issue:
iloc with boolean mask
Currently masking by boolean vectors it doesn't matter which syntax you use:

```
df[mask]
df.iloc[mask]
df.loc[mask]
```

are all equivalent. Should mask `df.iloc[mask]` mask by position? (this makes sense if mask is integer index).

This [SO question](http://stackoverflow.com/questions/16603765/what-is-the-most-idiomatic-way-to-index-an-object-with-a-boolean-array-in-pandas).



Discussion:
normally the generated masks have the same index as to what you are doing, e.g in your example in the SO question.

I think `.iloc` could/should do this, does make sense. There is an alignment step for the mask in `.where`

Here's an example to summarise the current behaviours:

``` python
locs = np.arange(4)
nums = 2**locs
reps = map(bin, nums)
df = pd.DataFrame({'locs':locs, 'nums':nums}, reps)
print df
for idx in [None, 'index', 'locs']:
    mask = (df.nums>2).values
    if idx:
        mask = pd.Series(mask, list(reversed(getattr(df, idx))))
    for method in ['', '.loc', '.iloc']:
        try:
            if method:
                accessor = getattr(df, method[1:])
            else:
                accessor = df
            ans = bin(accessor[mask]['nums'].sum())
        except Exception, e:
            ans = str(e)
        print "{:>5s}: df{}[mask].sum()=={}".format(idx, method, ans)
```

with output

``` python
        locs  nums
0b1        0     1
0b10       1     2
0b100      2     4
0b1000     3     8

 None: df[mask].sum()==0b1100
 None: df.loc[mask].sum()==0b1100
 None: df.iloc[mask].sum()==0b1100
index: df[mask].sum()==0b11
index: df.loc[mask].sum()==0b11
index: df.iloc[mask].sum()==0b11
 locs: df[mask].sum()==Unalignable boolean Series key provided
 locs: df.loc[mask].sum()==Unalignable boolean Series key provided
 locs: df.iloc[mask].sum()==Unalignable boolean Series key provided
```

If I'm understanding your discussion correctly then in the last line the output should also be 0b11.

yes...that looks right, if you happen to think of a simple example where you would actually _use_ this pls post that

keep in mind that masks are the same length as what you are indexing, so I don't think there is ever ambiguity, but I could be wrong

Fair enough, I can't actually think of anything. It just seems that by symmetry the behaviour should be there.

Similarly, it doesn't seem quite right that in line 6 of my example above, `df.iloc[mask]` is actually realigning the mask based on mask.index rather than throwing an error.

Given these observations, I would probably vote for the following behaviour:
- df[mask] should only work with .values in the order they're given and do no realignment. Perhaps for performance reasons.
- df.loc[mask] should work as it currently does and realign based on mask.index if present. If mask.index is of the wrong type it should throw an Error.
- df.iloc[mask] should realign based on mask.index if present and of integer type. Otherwise throw an error.

It's true that I haven't thought about this for more than 10 minutes though so if there are many use cases where this causes a problem then nevermind.

I think it's possible there could be an ambiguity, if the index is in a different order (e.g. was taken from somewhere else where it may well mean the location rather than the label).

Also, I'm given a `/core/frame.py:1943: UserWarning: Boolean Series key will be reindexed to match DataFrame index.` somewhere along the way messing with integer index booleans.

Correct me if I wrong, but isn't "realign based on mask.index" different from location?

Still thinking about this, but I think a nice _feature_ of boolean masking is that since we don't ever have an ambiguity whether its label or position based (as the mask must be the same length as what you are indexing), then it can be used in either `.loc` or `.iloc`. I don't think restricting this is a good idea. 

The basic question is do we drop the index effectively (and make it not matter) when its the right length?

e.g. should

`df[mask_with_index]` and `df[mask_ndarray]` be the same?

or the first align (currently I retract my earlier statement, I don't think there is an alignment on the index itself)

but this may depend, e.g. `iloc/loc` don't align, but I think `[]` might....

@hayd  you make a good point, but to my knowledge that is the issue with an unlabeled index, the user has to make sure its in the _right_ order, pandas cannot help (but the case we are talking about it _could_ help, by making sure the index is aligned)

@jreback With regards to the alignment, see my example above. I threw in the reversed(...) to see whether it realigns based on the labels or not. The results above show that [], .loc and .iloc all perform an alignment step based on the **labels** when these are in a different order. Therefore I think the fact that the length is the same is lulling you into a false sense of security.

Apologies if my example isn't clear. I thought the binary representation was a nice way of concisely summarising which items were selected or not. The bottom line is that the results differ between ndarray and pd.Series because in the pd.Series case the .index is used to do an alignment step first.

Also there's mistake in my output because it should read:

```
df[mask]['nums'].sum()==...
```

@snth you are right....I was _eye-balling_ the code....as its a failry tricky path

ok....so bottom line is wether to make `iloc` align on the labels or position (for a boolean mask), but that itself is somewhat of an issue, the is certainly probably and likely that the index is not integer-based, so then `iloc` should throw an error

So from the original SO question, this should then raise?

```
In [1]: df = pd.DataFrame(range(5), list('ABCDE'), columns=['a'])

In [2]: mask = (df.a%2 == 0)

In [3]: mask
Out[3]: 
A     True
B    False
C     True
D    False
E     True
Name: a, dtype: bool

In [4]: df.iloc[mask]
Out[4]: 
   a
A  0
C  2
E  4
```

What about this?

```
In [5]: mask.nonzero()
Out[5]: (array([0, 2, 4]),)

In [6]: mask.nonzero()[0]
Out[6]: array([0, 2, 4])

In [7]: df.iloc[mask.nonzero()[0]]
Out[7]: 
   a
A  0
C  2
E  4

In [8]: df.iloc[Series(mask.nonzero()[0])]
Out[8]: 
   a
A  0
C  2
E  4

In [9]: Series(mask.nonzero()[0])
Out[9]: 
0    0
1    2
2    4
dtype: int64
```

I there is NO alignment happening, instead its using the **values** to actually index

I think iloc should throw an error if it's not integer based, it should _definitely_ use position.

These integers needn't be in order, just like the labels needn't be in order.

Putting an array to `iloc` should do what it does now, but a _boolean_ Series is a different beast:

```
msk = pd.Series([True, True, True, False, False])
df.iloc[msk] == df.iloc[0:3]
```

?

@jreback I agree that in your first example by my reasoning In[4] should raise an Exception rather than realign based on labels within in .iloc.

I don't understand your second example. There's no boolean indexing involved there and they just seem to be examples of .iloc. Is that what's happening in the source code?

@snth I was trying to have it index with a Series that had a different index, (that is essentially what boolean masking does)

Ok, if you had the one above, there is an easy workaround (with arrays):

```
msk = pd.Series([True, True, True, False, False])
df.iloc[msk.values]  # instead of df.iloc[msk]
```

But if your index was not in order:

```
msk = pd.Series([True, True, True, False, False], index=[1, 2, 3, 4, 0])
df.iloc[msk] == df[1:4]
```

?

@hayd In terms of workarounds you could probably do

```
df.iloc[msk.index][msk.values]
```

Nothing else broke...I'll put up the PR in a minute....pls try out

```
In [3]: df = DataFrame(range(5), list('ABCDE'), columns=['a'])

In [4]: mask = (df.a%2 == 0)

In [5]: df.iloc[mask]
ValueError: iLocation based boolean indexing can only have an integer index [string] was inferred
```


Resolving patch:
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -775,7 +775,14 @@ class _iLocIndexer(_LocationIndexer):
     _exception   = IndexError
 
     def _has_valid_type(self, key, axis):
-        return isinstance(key, slice) or com.is_integer(key) or com._is_bool_indexer(key) or _is_list_like(key)
+        if com._is_bool_indexer(key):
+            if hasattr(key,'index') and isinstance(key.index,Index):
+                if key.index.inferred_type == 'integer':
+                    raise NotImplementedError("iLocation based boolean indexing on an integer type is not available")
+                raise ValueError("iLocation based boolean indexing cannot use an indexable as a mask")
+            return True
+
+        return isinstance(key, slice) or com.is_integer(key) or _is_list_like(key)
 
     def _getitem_tuple(self, tup):
 
@@ -811,9 +818,11 @@ def _get_slice_axis(self, slice_obj, axis=0):
     def _getitem_axis(self, key, axis=0):
 
         if isinstance(key, slice):
+            self._has_valid_type(key,axis)
             return self._get_slice_axis(key, axis=axis)
 
         elif com._is_bool_indexer(key):
+            self._has_valid_type(key,axis)
             return self._getbool_axis(key, axis=axis)
 
         # a single integer or a list of integers


Test patch:
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -888,6 +888,60 @@ def test_multi_assign(self):
         df2.ix[mask, cols]= dft.ix[mask, cols].values
         assert_frame_equal(df2,expected)
 
+    def test_iloc_mask(self):
+
+        # GH 3631, iloc with a mask (of a series) should raise
+        df = DataFrame(range(5), list('ABCDE'), columns=['a'])
+        mask = (df.a%2 == 0)
+        self.assertRaises(ValueError, df.iloc.__getitem__, tuple([mask]))
+        mask.index = range(len(mask))
+        self.assertRaises(NotImplementedError, df.iloc.__getitem__, tuple([mask]))
+
+        # ndarray ok
+        result = df.iloc[np.array([True] * len(mask),dtype=bool)]
+        assert_frame_equal(result,df)
+
+        # the possibilities
+        locs = np.arange(4)
+        nums = 2**locs
+        reps = map(bin, nums)
+        df = DataFrame({'locs':locs, 'nums':nums}, reps)
+
+        expected = {
+            (None,'')     : '0b1100',
+            (None,'.loc')  : '0b1100',
+            (None,'.iloc') : '0b1100',
+            ('index','')  : '0b11',
+            ('index','.loc')  : '0b11',
+            ('index','.iloc') : 'iLocation based boolean indexing cannot use an indexable as a mask',
+            ('locs','')      : 'Unalignable boolean Series key provided',
+            ('locs','.loc')   : 'Unalignable boolean Series key provided',
+            ('locs','.iloc')  : 'iLocation based boolean indexing on an integer type is not available',
+            }
+
+        import warnings
+        warnings.filterwarnings(action='ignore', category=UserWarning)
+        result = dict()
+        for idx in [None, 'index', 'locs']:
+            mask = (df.nums>2).values
+            if idx:
+                mask = Series(mask, list(reversed(getattr(df, idx))))
+            for method in ['', '.loc', '.iloc']:
+                try:
+                    if method:
+                        accessor = getattr(df, method[1:])
+                    else:
+                        accessor = df
+                    ans = str(bin(accessor[mask]['nums'].sum()))
+                except Exception, e:
+                    ans = str(e)
+
+                key = tuple([idx,method])
+                r = expected.get(key)
+                if r != ans:
+                    raise AssertionError("[%s] does not match [%s], received [%s]" %
+                                         (key,ans,r))
+        warnings.filterwarnings(action='always', category=UserWarning)
 
 if __name__ == '__main__':
     import nose