RangeIndex as default index
State: closed

WIP on making RangeIndex (#9977) the default index


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance
name: API Design

Comments:
@jreback @shoyer I tried out what happend making RangeIndex the default index: nothing good. - A whopping 60 errors and 13 test failures. - Most of which meant absolutely nothing to me.

To make the problem more manageable, I started this tree with a minimal `RangeIndex` implementation: no optimization at all, pure fall-back to `Int64Index`. In principle it should behave pretty much like the usual default index - except with a different constructor.

Still I have about 16 errors and 2 failures - most of which I have not idea how they happen. I managed to diagnose and fix one and I am working on another.

That said, I will really need help debugging this if you really want to make `RangeIndex` the default index: I am fairly limited because I have already exceeded the time I was granted to work on `RangeIndex` by far. I will of course still work on it when I can, but I can no longer do it full-time - which would be required to understand all the pandas internals to fix these widely distributed errors.

Maybe you know another pandas developer would would be willing to help out with this - or take ownership?

Haha, that's not too surprising :). If you keep your branch up to date, I can test it and probably identify at least a few fixes.

On Mon, Apr 27, 2015 at 4:00 AM, ARF1 notifications@github.com wrote:

> @jreback @shoyer I tried out what happend making RangeIndex the default index: nothing good. - A whopping 60 errors and 13 test failures. - Most of which meant absolutely nothing to me.
> To make the problem more manageable, I started this tree with a minimal `RangeIndex` implementation: no optimization at all, pure fall-back to `Int64Index`. In principle it should behave pretty much like the usual default index - except with a different constructor.
> Still I have about 16 errors and 2 failures - most of which I have not idea how they happen. I managed to diagnose and fix one and working on another.
> That said, I will really need help debugging this if you really want to make `RangeIndex` the default index: I am fairly limited because I have already exceeded the time I was granted to work on `RangeIndex` by far. I will of course still work on it when I can, but I can no longer do it full-time - which would be required to understand all the pandas internals to fix these widely distributed errors.
> 
> ## Maybe you know another pandas developer would would be willing to help out with this - or take ownership?
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/9999#issuecomment-96610070

@shoyer That would be great. I managed to whittle the issues down to only a few with this minimal `RangeIndex` implementation. Now however I am a bit stumped by the error in `test_set_index2 (pandas.tests.test_frame.TestDataFrame)`. See: https://travis-ci.org/pydata/pandas/jobs/60231079#L1173

Also, I am not quite sure what to do with the failure in `test_constructor_mrecarray (pandas.tests.test_frame.TestDataFrame)`. (See: https://travis-ci.org/pydata/pandas/jobs/60231079#L1209) The issue seems to be that a dataframe that originally had a `RangeIndex` returns a dataframe with a `Int64Index` after some operation. That should in principle be accepted as ok, I think, but how would one adjust the test?

@jreback @shoyer Ok, it was truely painfuly but I managed to reduce the issues with the minimal `RangeIndex` implementation as default index to a single test failure. This last one though is stubbornly escaping my analysis.

See (`pandas.tests.test_graphics.TestDataFramePlots.test_legend_name`): https://travis-ci.org/pydata/pandas/jobs/60815604#L1173

I have narrowed it down to this line: https://github.com/ARF1/pandas/blob/rangeindex_as_default_index/pandas/tools/plotting.py#L1007

Before this line, `data.columns.name == 'new'`, after this line, `numeric_data.columns.name == None`. This only happens if `RangeIndex` is the default index. For the life of me I cannot figure out why the name is lost in translation. - Even with side-by-side step-by-step debugging of both versions.

Any chance you could help with this?

@ARF1  Tracking down bugs like this can be unforgiving, especially when it takes you on a deep dive through pandas internals. Thanks for the hard work!

Closing in favour of and rebasing changesets on #9977.