CLN/INT: remove Index as a sub-class of NDArray
State: closed

make `Index` now subclass `PandasObject/IndexOpsMixin` rather than `ndarray`
should allow much easier new Index classes (e.g. #7640)

This doesn't change the public API at all, and provides compat

closes #5080
back compat for pickles is now way simpler

ToDo:
- docs
  - [x] release note
  - [x] warnings in io.rst about this change
  - [x] fixes `.repeat` on MultiIndex (broken in master)
  - [x] minor API compat issue with comparisons of `DatetimeIndex` with `NaT` vs ndarrays
- [x] merge with searchsorted issues/PR ( #6712, #7447, #6469)
- [x] tests fixed (FIXMES), just a few left
- [x] perf
- [x] json completely broken ATM
- [x] #7796 FIXME (PeriodIndex not supported in HDF), not really a big deal
- [x] #7439 since `Index` now doesn't have implicit ops (aside from `__sub__/__add__`, these need to be added in (e.g. `__mul__,__div__,__truediv__`).
- [x] bool(Index/MultiIndex) : https://github.com/pydata/pandas/issues/7897 (will address this later)

closes #5155 (perf fix for Period creation), slight increase on the plotting
because of the the plottling routines holding array of Periods (rather than PeriodIndex).

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
period_setitem                               |  16.2210 | 122.9340 |   0.1319 |
timeseries_iter_periodindex                  | 1197.7839 | 6906.1464 |   0.1734 |
timeseries_iter_periodindex_preexit          |  12.4850 |  69.8563 |   0.1787 |
timeseries_period_downsample_mean            |  11.2850 |  11.1457 |   1.0125 |
plot_timeseries_period                       | 107.6056 |  86.5277 |   1.2436 |
```


Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: Internals
description: Related to non-user accessible pandas implementation
name: MultiIndex

Comments:
We will call you Mr Anti-NDArray :)

cc @Komnomnomnom if you could have a look at the json issues would be great!

`ndarray` sub-classing is SO old world. Its really quite annoying.

amen to that composition ftw

No problem guys, I'll take a look this weekend.

@sinhrks

would you mind taking a look at this PR and see if you can figure this out?

seem the PeriodIndex is getting converted to underlying (and not preserverd) like in master.....

thxs

```
======================================================================
ERROR: test_business_freq (pandas.tseries.tests.test_plotting.TestTSPlot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/tseries/tests/test_plotting.py", line 273, in test_business_freq
    self.assertEqual(PeriodIndex(data=idx).freqstr, 'B')
  File "/mnt/home/jreback/pandas/pandas/tseries/period.py", line 598, in __new__
    ordinal, freq = cls._from_arraylike(data, freq, tz)
  File "/mnt/home/jreback/pandas/pandas/tseries/period.py", line 667, in _from_arraylike
    raise ValueError('freq not specified and cannot be '
ValueError: freq not specified and cannot be inferred from first element

======================================================================
ERROR: test_mixed_freq_hf_first (pandas.tseries.tests.test_plotting.TestTSPlot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/pandas/pandas/tseries/tests/test_plotting.py", line 639, in test_mixed_freq_hf_first
    self.assertEqual(PeriodIndex(data=l.get_xdata()).freq, 'D')
  File "/mnt/home/jreback/pandas/pandas/tseries/period.py", line 598, in __new__
    ordinal, freq = cls._from_arraylike(data, freq, tz)
  File "/mnt/home/jreback/pandas/pandas/tseries/period.py", line 667, in _from_arraylike
    raise ValueError('freq not specified and cannot be '
ValueError: freq not specified and cannot be inferred from first element
```

@jreback I think following lines should be changed not to pass `PeriodIndex`.  Both tests can be passed by using `_mpl_repr` to pass `ndarray` of `Period` objects explicitly. It looks no affect to chart-looking, but will take a further look (especially resampling/replot case).

One difference is matplotlib axis no longer can hold `PeriodIndex` wigh `freq`, thus `freq` will be re-inferred during `PeriodIndex` reconstruction (and affects to other tests which checks `freq`)

```
-    lines = plotf(ax, series.index, series.values, **kwargs)
+    lines = plotf(ax, series.index._mpl_repr(), series.values, **kwargs)
```

https://github.com/pydata/pandas/blob/master/pandas/tseries/plotting.py#L64
https://github.com/pydata/pandas/blob/master/pandas/tseries/plotting.py#L155

@sinhrks ahh..ok, lmk try to replace that and see.

Will this make RangeIndexs a little easier to implement? xlink #3268 #2420

@hayd I certainly hope so!

I am super stoked about this change.

@hayd their is already a lot of work on RangeIndex here: https://github.com/jtratner/pandas/tree/add-range-index

I think it should be a bit easier. However, I have to think about how to fix the real problem, which is that `Index.__new__` unfortunatelly is still used (and each sub-class has a `__new__` as well). I think actually need a `IndexFactory` (which is then simply called by `__new__`), with the ability to register / override the creation mechanism.

That will have to wait though. @shoyer when you are ready to integrate `IntervalIndex` I think we can address that.

@sinhrks I have made the changes for plotting here. https://github.com/jreback/pandas/commit/1fc0a1fe53926ac32583da675be502667fec2d92

can you confirm that the graphs act/look the same? (esp when resampled/zoomed and such)

as an aside this now makes all lines whether `DatetimeIndex` or `PeriodIndex` be arrays of the underlying object (datetimes/Period).

@shoyer that was a good idea to use `_array_priority__`, see here: https://github.com/jreback/pandas/commit/29d2f44a245187b6d3e483635dcb331abd977849

you have to catch `Notimplemented` (type) but that 's ok 

@sinhrks I updated this commit a couple of times to put in some period construction optimizations....

pls lmk about the plottnig (if you think its ok). (all tests now pass). 

https://github.com/jreback/pandas/commit/2f32f462c04d7f0e10fdfc13348065187e1672c5

@jreback I've checked some plots, and these worked as the same as before. If anything, I'll confirm again. 

hey @jreback just taking a look at the json stuff now

@shoyer https://github.com/jreback/pandas/commit/7d66157004bd459070a0f82e27a62a99b6faf2e4

done.

also I have put in (well a little), structure on the Index testing for more generic testing, e.g. https://github.com/jreback/pandas/commit/d1c4fbb0d170cfaf920a27907c014e8cc45752d1

so prob worhwhile after this PR is merged to 'fix' the index tests to make it more class based (how Float64/Int64 are done). to make it a big more generic. E.g. 

ok monster is ready to merge. any further comments.
- has back compat pickle compat (I dropped < 0.8.0). I suppose could resurrect, any reason to though?
- perf is good
- testing / infrastructure is better for sub-classing
- couple of todos #7904

@jorisvandenbossche @cpcloud @shoyer @sinhrks @hayd 
cc @immerrr

I think I understand pickle and all of its evils now :) (not sure if that is a net benefit to society though)

There are some math related regressions:

```
idx = pd.Index(range(5))
print 1 * idx
print idx * idx
print np.sin(pd.Index(np.arange(5.0)))
print np.sin(idx)
print pd.Index(range(5)) * 1.0
```

On 0.14.1 I get:

```
Int64Index([0, 1, 2, 3, 4], dtype='int64')
Int64Index([0, 1, 4, 9, 16], dtype='int64')
Float64Index([0.0, 0.841470984808, 0.909297426826, 0.14112000806, -0.756802495308], dtype='float64')
Int64Index([0.0, 0.841470984808, 0.909297426826, 0.14112000806, -0.756802495308], dtype='float64')
Int64Index([0.0, 1.0, 2.0, 3.0, 4.0], dtype='float64')
```

(yes, those last two cases are already bugs! they shouldn't be Int64Index)

With this PR, I get:

```
TypeError: unsupported operand type(s) for *: 'int' and 'Int64Index'
TypeError: can only perform ops with scalar values
[ 0.          0.84147098  0.90929743  0.14112001 -0.7568025 ]
[ 0.          0.84147098  0.90929743  0.14112001 -0.7568025 ]
Int64Index([0.0, 1.0, 2.0, 3.0, 4.0], dtype='float64')
```

(e.g., ufuncs give us bare ndarrays, not indexes)

I think it is fine to only allow scalar operations on indexes and arguably more consistent (to avoid surprising conflicts with set-like `+` and `-`), but this should be documented and possibly go through a deprecation cycle.

Also, it is probably OK to return `ndarray` instead of `Index` objects, but the result from ufuncs should probably be consistent with the type returned by mathematical operations. Since bare `np.datetime64` is broken, everything should probably come back as an Index. In that case, to keep indexes as indexes after a ufunc, you need to add `__array_wrap__`:

``` python
def __array_wrap__(self, obj, context=None):
   return pd.Index(obj, name=self.name)
```

@shoyer good point, I forgot to put in `__array_wrap__`. going to put in tests to replicate current behavior (though I agree should be changed), see #7439 (`*,/,//`) work now

@shoyer all fixed by https://github.com/jreback/pandas/commit/d714d2b6457820488d16c870701b3b48e1c99948#diff-d41d8cd98f00b204e9800998ecf8427e

and results now as above.

further I think will now catch numeric ops on non-Float/Int Index types (and only *,/,//) allowed on those (for the time being)

interesting. (as Series doesn't have a priorty atm, so prob 1). make them both 1000?

@jreback you need checks for datetime/offset math, too, e.g.,
`np.timedelta64(1, 'D') + pd.date_range('2000-01-01', periods=3)`

unless that is covered somewhere else?

`np.timedelta64(1, 'D') * pd.date_range('2000-01-01', periods=3)`

what does that yield? (`TypeError`?)

Yes, actually I think datetime math is OK (I just didn't notice any reference to it in your code).

this already yields a `TypeError` (actually ALL ops for mul/div for non Float/Int Index do) (though I'll make the error a bit more intuite)

`__array_priority__` will not help for (index, series) operations unless you interpret it yourself, as neither object is an ndarray. But I think a fixed number for both (like 1000) is fine.

whosh, rewrote the whole ops handling section: https://github.com/jreback/pandas/commit/19bde6752e6c7fc36513b1930ae3f01c1cc68b1a

good new is its very easy to now add/change behavior on a per class level.

@jreback Added a bunch of comments (mainly on docs and public API, not familiar enough to comment on technical details)

Further, I wondered, are there things we have learned from the "Series -> NDFrame subclass and no longer ndarray subclass" move that can be relevant here? Issues that came up afterwards (where we had to say "series is no longer ndarray subclass, so this will not work anymore) that we can now warn for beforehand?
The whatsnew section then has some more warnings (http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#internal-refactoring).

@jorisvandenbossche 
https://github.com/jreback/pandas/commit/72f491a3ec37ca52a513b9f0b8bd4d68b3dedb70#diff-d41d8cd98f00b204e9800998ecf8427e

I just update the properties directly, better I think because then don't have to clutter with the full ndarray definitions.