ENH: RangeIndex redux
State: closed

closes #939 
replaces #9977 

ToDo:
- [x] test for packers.py
- [x] more code review

Much commentary on the original issue #9977 

but in essence `RangeIndex` is a complete replacement for `Int64Index`, which all indexing semantics and interop. This is now the default indexer upon construction. It should be completely transparent to the end user.

It provides a constant memory footprint for any size of index. Their is a tiny penalty for < about 10 elements (which is actually trivial to fix, e.g. we could simply instantiate an `Int64Index` for these cases). But I think it is more natural to _always_ get a `RangeIndex`.

One other change here is to `assert_index_equal` the `exact` kw now takes `equiv` as the default (in addition to a boolean) to allow for exact comparisions except for `Int64Index`/`RangeIndex` are considered equivalent (as are string/unicode as inferred types, this was pre-existing).

```
In [1]: s = Series(range(5))

In [2]: s.index
Out[2]: RangeIndex(start=0, stop=5, step=1)

In [3]: s.nbytes
Out[3]: 40

In [4]: s.index.nbytes
Out[4]: 72

In [5]: s.index.astype(int).nbytes
Out[5]: 40

In [6]: s = Series(range(100))

In [7]: s.index.astype(int).nbytes
Out[7]: 800

In [8]: s.index.nbytes
Out[8]: 72
```


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
cc @jorisvandenbossche @shoyer @sinhrks 

@shoyer if you can review when you have a chance. as you had a number of comments on the original.

@shoyer all fixed up

@shoyer ok updated.

There are two aspects of the current design that I (still) don't like:
- The `RangeIndex` constructor is flexible enough that it accepts `range` and `RangeIndex` arguments as the `start` argument. This seems too flexible to me. I would prefer a stricter approach that requires using an explicit constructor rather than overloading the meaning of `start`.
- Sometimes, the `RangeIndex` constructor can create different types of indexes (e.g., if `dtype=object`). I don't see any good reason to allow this.

It can't actually create an `object` dtype `RangeIndex`, merely returning a regular `Index` that HAS `object` dtype. This really was for compat. Though its prob useless. ok I'll take though out.

Your first point I think makes usage just that much more confusing. As a user I would expect:

`Index(xrange(5))` to give me a `RangeIndex`, seems very natural and not a stretch to do this.

> Index(xrange(5)) to give me a RangeIndex, seems very natural and not a stretch to do this.

Yes, this should work. But `RangeIndex(xrange(5))` should not.

hmm, I also don't see why that should be special cased NOT to work. Its a valid constructor.

@shoyer https://github.com/jreback/pandas/commit/3a53067b2a8c1cb943915010a28a5c619d515b82

but this makes this a bit specially cased, though:

This just silenty ignores.....hmm

```
In [2]: pd.Int64Index([1,2,3],dtype=object)
Out[2]: Int64Index([1, 2, 3], dtype='int64')
```

@shoyer @jorisvandenbossche 

final comments?

Will try to look at this in the coming days as well. In any case, some more documentation will be needed I think (now it is not mentioned in the main docs, only in whatsnew)

@jorisvandenbossche that is on purpose. This is a complete replacement for `Int64Index` (which we don't document either, not that that is an excuse!).

I think adding to API.rst is ok (for all index-types). We only touch briefly on `Float64Index`, `CategoricalIndex` in main docs as these have special indexing behavior.

Again I think that all of these _should_ be more fully documented. I'll add an issue.

But then maybe adding a short section in advanced indexing (http://pandas-docs.github.io/pandas-docs-travis/advanced.html) like FloatIndex has there a section as well. Can largely be the same text as what is now in the whatsnew, but then instead of showing the difference between old and new behaviour, show the difference between explicitely making int64index vs default rangeindex. 
I would also add a small explanation on when this gets converted to a Int64Index

yep, just created #11962  for all of this 

any final comments?

@jreback I'm still :-1: on this stuff in the `RangeIndex` constructor!

``` python
        # RangeIndex
        if isinstance(start, RangeIndex):

            if not copy:
                return start
            if name is None:
                name = getattr(start, 'name', None)
            start, stop, step = start.start, start.stop, start.step

        # range
        elif isinstance(start, range):

            # seems we only have indexing ops to infer
            # rather than direct accessors
            if len(start) > 1:
                step = start[1] - start[0]
                stop = start[-1] + step
                start = start[0]
            elif len(start):
                start = start[0]
                stop = start + 1
                step = 1
            else:
                start = stop = 0
                step = 1

        # pass thru to main constructor
        elif isinstance(start, (Index, np.ndarray, ABCSeries)):
            return Index(start, dtype=stop, copy=step, name=name, **kwargs)
```

See my bulleted concerns about too much flexibility in the constructor above. Any other opinions here?

I’m only about halfway through reviewing. I’ll have some more time to play with it tomorrow.

-Tom

> On Jan 8, 2016, at 12:44 PM, Stephan Hoyer notifications@github.com wrote:
> 
> @jreback https://github.com/jreback I'm still  on this stuff in the RangeIndex constructor!
> 
> ```
>     # RangeIndex
>     if isinstance(start, RangeIndex):
> 
>         if not copy:
>             return start
>         if name is None:
>             name = getattr(start, 'name', None)
>         start, stop, step = start.start, start.stop, start.step
> 
>     # range
>     elif isinstance(start, range):
> 
>         # seems we only have indexing ops to infer
>         # rather than direct accessors
>         if len(start) > 1:
>             step = start[1] - start[0]
>             stop = start[-1] + step
>             start = start[0]
>         elif len(start):
>             start = start[0]
>             stop = start + 1
>             step = 1
>         else:
>             start = stop = 0
>             step = 1
> 
>     # pass thru to main constructor
>     elif isinstance(start, (Index, np.ndarray, ABCSeries)):
>         return Index(start, dtype=stop, copy=step, name=name, **kwargs)
> ```
> 
> See my bulleted concerns above. Any other opinions here?
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/pydata/pandas/pull/11892#issuecomment-170087206.

@shoyer I get your comments. But this fits the existing API. I _could_ make a `RangeIndex.from_range(...)`, and simply call it here. this is very natural to do.

are you arguing again accepting a `RangeIndex` in the `RangeIndex` constructor itself????

I think there may some benefit in making `list` works where `xrange` is accepted.  I'd expect, in P2, `RangeIndex(xrange(10))` and `RangeIndex(range(10))` to be the same, although the lazy version is  probably preferred.

Instead of the current main constructor (start, stop, & step), it could be a `Range` object. Then accepting a `RangeIndex` or list becomes less awkward.
And you could have `RangeIndex.from_attributes(start, stop, step)` for the current construction

A little bit usage feedback:
- I find it a bit strange that list and array are handled differently:
  
  ```
  In [17]: pd.RangeIndex([1,2,3])
  TypeError: RangeIndex(...) must be called with integers, [1, 2, 3] was passed fo
  r start
  
  In [18]: pd.RangeIndex(np.array([1,2,3]))
  Out[18]: Int64Index([1, 2, 3], dtype='int64')
  ```

This touches the point of @shoyer I think (the last part of the constructor code he mentioned) 
@jreback What is the reason that `Index, np.ndarray, ABCSeries` are passed through to the main constructor? 
An example that not all other index objects are that flexible:

```
In [58]: pd.Float64Index(np.array(['a', 'b']))
TypeError: String dtype not supported, you may need to explicitly cast to a numeric type

In [59]: pd.RangeIndex(np.array(['a', 'b']))
Out[59]: Index([u'a', u'b'], dtype='object')
```

One more comment on the `start`, `stop`, `step` attributes: I was wondering if it is a good idea to make these public. 
Because, if somebody starts using these attributes, depending on the operations you do they will suddenly be not available anymore on the index and raise a ValueError, as you will not always notice when the RangeIndex gets converted to a Int64Index.

I _could_ hide the `start,stop,step` (as they are on `xrange`), though its kind of annoying (as a user of the object :)

so @MaximilianR brings up a good point.

I _could_ take a list-like of integers and if its monotonic & equal-stepped (easy test, but not completely for free). make it into a `RangeIndex`. I think this would clear up some of the API.

Then `TypeError` or `ValueError` on non-supported (rather than deferring to `Index`)

> I could take a list-like of integers and if its monotonic & equal-stepped (easy test, but not completely for free). make it into a RangeIndex. I think this would clear up some of the API.
> Then TypeError or ValueError on non-supported (rather than deferring to Index)

But it would also make the constructor even more complex. What is the reason for not raising an Error in all these cases? Convenience for creating it?

I would like a stricter API -- one constructor for converting a `range` object and another for handling `start`/`stop`/`step`. It is not especially important to me which is which (so we could use @MaximilianR's alternative), but I think there is a good case to make the `RangeIndex` constructor handle `start`/`stop`/`step` because the generic `Index` constructor can already convert `range` objects.

I am would lean against converting lists, even if we have a dedicated constructor for this path. It's not hard to type `xrange` via `range` on Python 2. It just makes things more complex than necessary.

I'm in agreement with @shoyer. And one more point in favor of `start/stop/step` as the main constructor: it would seem weird for a `range*` function/constructor to have a different signature than `(x)range / arange`.

so what is the argument against allowing `start` be range (as we are currently doing). This is just an overloaded constructor, seems quite natural to me. Having to use a different factory doesn't gain you anything.

> so what is the argument against allowing start be range (as we are currently doing). This is just an overloaded constructor, seems quite natural to me. Having to use a different factory doesn't gain you anything.

It forces you to be explicit and code in a way that is more amenable to static analysis. These are both good things.

> It forces you to be explicit and code in a way that is more amenable to static analysis. These are both good things.

not true of pandas in general. This should be a conscious decision to break from the existing API flexibility. IMHO this quite in-line with the current functionaility.

I don't see any reason NOT to allow this.