ENH: add per axis, per level indexing with tuples using loc
State: closed

Selection only. Did not check (nor care) about performance at this point.
Full Tests, vbenches and docs to follow in due course.

Please hold off on PEP8 comments or suggestions on how this could be made more complex.
I'd like to get something functional, solid and understandable in master fairly early in 0.14. 

@jreback, I don't often wade this deep into the gore. Would welcome your review and suggestions.

Example of setting usage:
http://stackoverflow.com/questions/21539260/assign-to-multiindex-slice/21539479#21539479

xref: #5641, #3057, #4036, and others

``` python
In [1]: 
   ...: def mklbl(prefix,n):
   ...:     return ["%s%s" % (prefix,i)  for i in range(n)]
   ...: 
   ...: ix =pd.MultiIndex.from_product([mklbl('A',5),mklbl('B',7),mklbl('C',4),mklbl('D',2)])
   ...: df=pd.DataFrame(range(len(ix.get_values())),index=ix)

In [5]: df
Out[5]: 
              0
A0 B0 C0 D0   0
         D1   1
      C1 D0   2
         D1   3
      C2 D0   4
         D1   5
      C3 D0   6
         D1   7
   B1 C0 D0   8
         D1   9
      C1 D0  10
         D1  11

```

``` python
In [2]: df.loc[(slice('A1','A3'),slice(None), ['C1','C3']),:]
Out[2]: 
               0
A1 B0 C1 D0   58
         D1   59
      C3 D0   62
         D1   63
   B1 C1 D0   66
         D1   67
      C3 D0   70
         D1   71
   B2 C1 D0   74
         D1   75
      C3 D0   78
         D1   79
   B3 C1 D0   82
         D1   83
      C3 D0   86
         D1   87
   B4 C1 D0   90
         D1   91
      C3 D0   94
         D1   95
   B5 C1 D0   98
         D1   99
      C3 D0  102
         D1  103
   B6 C1 D0  106
         D1  107
      C3 D0  110
         D1  111
A2 B0 C1 D0  114
         D1  115
      C3 D0  118
         D1  119
   B1 C1 D0  122
         D1  123
      C3 D0  126
         D1  127
   B2 C1 D0  130
         D1  131
      C3 D0  134
         D1  135
             ...

[56 rows x 1 columns]
```


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
`df.loc[(slice('A1','A3'),slice(None), ['C1','C3']),:]`

I think accept `None` as well as `slice(None)` would be nice (and prob not a big deal)

as a side issue, we can support/eliminate xs by doing this:

`df.loc(level=4)[['C1:'C3']],:]` e.g. this would pass additional arguments to the `.loc` constructors (namely level)

I know the None convention is used elsewhere, but it seems awful strange to specify None, 
when you're actually requesting _everything_.

actually, maybe you can just to and empty slice, e.g. `df.loc[(:,:,:,['C1':'C3'])]` but I suspect that doesn't work (maybe elipsis?)

You know it doesn't. `Ellipsis` could work (as the singleton, `...` doesn't work inside a tuple either),
but it's not consistent with the rest of the indexing methods. If we did it uniformly as a refactor in the future?

Alternativelty. define something short in the pd namespace to be `slice(None)` and accept that everywhere.
Similar to `np.nan`. Not sure.

ok...I would be ok with

maybe `pd.slice`, `pd.allslice`, `pd.islice`

need to think of something good

But a big :+1:  on this....I think sorely needed. also FYI, setting shouldn't be too difficult once getting is done.

cc @dragoljub  as I know you are wanting this

Glad to hear it. I know there's the copy problem, but I'm not sure how you actually get past that.

Actually, once the indexer is found setting is a quite striaghtforward assignment of locations to array

kind of similar to 

`df.iloc[[0,1,3,4,10,12],column_indexer] = 2d numpy arrays here`

which is already handled

I agree the copy problem is tricky.

YES! I do want this. This will make my multi-index DFs much more user friendly, I may be able to avoid having duplicate index and column data moving forward. :+1:  

Not to get sidetracked but why does the empty slice not work? :confused: 

the ':' syntax only works in the [ ] 

which is a problem of we need to have multi axis and multi levels

the slice(None) syntax works fine to indicate give me a full slice for that level (iow don't exclude anything)

I think that is kind of cumbersome and I stead maybe something pd.allslice to represent

more of a syntax thing

If you're asking why you can't do `df.loc[(foo,:, bar),'baz']`, it's because the `:` token is not
a first class thing, but part of python's [grammer](http://docs.python.org/2/reference/grammar.html) (See the `subscript` rule),
and only recognised inside immediately `[]` when subscripting, not elsewhere (such as as an element in a tuple).

allslice? too spicy.

```
In [6]: def mklbl(prefix,n):
   ...:     return ["%s%s" % (prefix,i)  for i in range(n)]
   ...: 
   ...: ix =pd.MultiIndex.from_product([mklbl('A',25),mklbl('B',27),mklbl('C',14),mklbl('D',12)])
   ...: df=pd.DataFrame(range(len(ix.get_values())),index=ix)
   ...: df=df.sort_index()
   ...: r=pd.core.indexing._tuple_to_mi_locs(df.index,((slice('A1','A3'),slice(None), ['C1','C3'])))
   ...: ixs=pd.core.indexing._spec_to_array_indices(df.index,r)
   ...: 
   ...: print "MultiIndex size: %d" % len(ix)
   ...: %timeit -n10 -r 10 df.loc[(slice('A1','A3'),slice(None), ['C1','C3']),:]
   ...: %timeit -n10 -r 10 df.iloc[ixs]

ix =pd.MultiIndex.from_product([mklbl('A',25),mklbl('B',27),mklbl('C',14),mklbl('D',12)])
MultiIndex size: 113400
10 loops, best of 10: 10.6 ms per loop
10 loops, best of 10: 9.76 ms per loop

ix =pd.MultiIndex.from_product([mklbl('A',45),mklbl('B',27),mklbl('C',14),mklbl('D',12)])
MultiIndex size: 204120
10 loops, best of 10: 13.4 ms per loop
10 loops, best of 10: 12.6 ms per loop

 ix =pd.MultiIndex.from_product([mklbl('A',85),mklbl('B',27),mklbl('C',14),mklbl('D',12)])
MultiIndex size: 385560
10 loops, best of 10: 16.6 ms per loop
10 loops, best of 10: 13.5 ms per loop
```

After vectorizing it's pretty evenly matched with passing `iloc` the indices directly.

`pd.alls`?

could work

Just having the option to do `slice(None)` or `None` is a great start. 

Will this support indexing with a shorter tuple than the number of levels? That would be a nice feature to have.

`df.loc[(level0,level1)]` for a 5-level index will fill the remaining levels with `slice(None)`.

maybe: `pd.uncut`, `pd.whole`

You can omit trailing levels right now, it'll work. The example shows this, look at the last, 'D', level.

This is really good, MI slicing atm is fiddly...

FYI The only toplevel constant atm is pd.NaT. I think pd.Alls would work be better, and less confusing with all.

I was going to (jokingly) suggest passing the function all (or maybe not) :s, but interestingly... 

```
In [20]: df = pd.DataFrame([[1, 2], [3, 4]])

In [21]: df.loc[(1, all, 2),1]
Out[21]: 
1                           4
<built-in function all>   NaN
2                         NaN
Name: 1, dtype: float64
```

Users would have to be careful not to accidentally type in wrong thing here if we go with pd.something (I guess that's always the case).

What about leveraging the fact that Python is case sensitive and using `pd.ALL` It could be confusing but all caps will make it stand out `df.loc[(1, ALL, 2),1]`

Maybe accepting a string argument(s) like 'all' in place of a direct alias of `slice(None)`?

Let's leave that for now and pick it up in a subsequent PR. no clear winner.
afaic, @jreback can have the last word but it needs to be made consistent
across pandas.

I think let's get into 0.14 and try out....can use slice(None) until a decision is made

@y-p I think this should be moved to an object based protocol (as I outlined above); This will need some mods for setting - will be cleaner in that framework

What does "object based protocol" mean? I tend to disagree. What's cleaner then 
a pure function of it's arguments? there's no state here, why do we need an object?

on the allslice,alls, ALL, I suggest we punt and use a non-official `_things_to_treat_as_slice_none = (,)` with the  only  guarantee being that that it contains `slice(None)` by default.

@jtratner has used this trick in several places. I like it.

it moves a collection of functions into an object where they have a logical functionality, grouping them for ease of understanding. One of the functions is purely internal (even to the indexing routines) and thus just makes the indexing module even more complicated to understand. 

This breaks with the pattern all of the functionality grouped according to what it does in an object. Furthermore you may need to delegate functionality for different index types and/or setting which you might need to keep some state.

I get your approach and what I am suggesting is almost a stylistic change, except to someoneone looking at the module will see a bunch of objects and then a bunch of not-short functions.

how would you actually write the `all_slice_none`??

are we against just using `None`?

That makes sense, basically we already agreed on this [here](https://github.com/pydata/pandas/pull/6134#discussion_r9213636). Grouping by object is fine (and true, consistent with the rest of pandas) and I like the idea of keeping the dispatch logic inside such a thing, that
only hold a clear dispatch logic (with comments). It then only invokes self-contained functions
that do exactly one thing  in a readable manner. I like that a lot in fact.

re how to write `all_slice_none`:

```
if tuple: foo # means slice
elif list: foo # means discrete list
else isinstance(theThing, all_slice_none_is_any_of_these) # or is in.. etc'
```

I dislike None, yeah. And I don't think we need to make a call on the matter this way.

We don't really have a `WE` you know, we just try our best to...

The only issue with the slice object thing is that I don't want to hold this up for
a larger supposed refactor with an unknown arrival date. The change I'm fine with,
holding things for it, less so. If you put it in place, I'll play along, or we can refactor later,
which do you prefer?

u can refactor would be great

I think this warrants merging soon

as their are a number of things to test/fix after (setting, doc changes, tests with all index types ) - maybe open a new issue with a checklist for this as a master issue

let's open a new issue to defer the ALL change (or not change)