DEPR: Remove int64 uint64 float64 index part 1
State: closed

This PR makes progress towards removing Int64Index, UInt64Index & FloatIndex from the code base, see #42717.

In this PR I make instantiation of `Index` with numpy numeric dtypes return a `NumericIndex` with the given dtype rather than converting to the old 64 bit only numeric index types, making all numeric dtypes available for indexes.

This is just the first part of the changes planned. In follow-ups PRs I will:
* remove Int64Index, Uint64Index & Float64Index from the top namespace (done, #49670)
* Remove int64Index, Uint64Index & Float64Index from the tess (partially done in #49682 & #49678, still some work needed)
* remove Int64Index, Uint64Index & Float64Index from the code base entirely (also internally)
* Move the functionality of `NumericIndex`into the base `Index` and remove `NumericIndex`
* Update docs to reflect the above changes

## related PRs
A part of the problems I had in #49494 was a few corner case bugs. They're pulled out into seperate PRs (#49536 & #49540). They're present in this PR at present, but will be removed after those seperate PRs have been merged.

## Notable changes from this PR:
As mentioned in #49494, one notable change from the PR is that because we now have all the numeric dtypes available for indexes, indexes created from DateTimeIndex (day, year etc.) will now be in int32, where previously they were forced to int64 (because that was the only index integer dtype available).

Using int32 is the more correct way, because the underlying `DateTimeArray` returns 32bit arrays. An example of this:

```ipython
>>> import pandas as pd
>>> 
>>> x = pd.date_range(start='1/1/2018', end='1/08/2018')
>>> x.array.day
array([1, 2, 3, 4, 5, 6, 7, 8], dtype=int32)  # 32bit, current and will not be changed in this series of PRs
>>> x.day
Int64Index([1, 2, 3, 4, 5, 6, 7, 8], dtype='int64')  # before this PR
NumericIndex([1, 2, 3, 4, 5, 6, 7, 8], dtype='int32')  # after this PR
Index([1, 2, 3, 4, 5, 6, 7, 8], dtype='int32')  # after follow-up PRs
```

The above will be included in the doc changes, when I do the docs

Progress towards #42717. See also #41272

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
Out of curiosity: the approach I would have taken would have been to start with changing `Index.__new__` to return NumericIndex instead of subclasses.  Would that be counter-productive?
> Out of curiosity: the approach I would have taken would have been to start with changing `Index.__new__` to return NumericIndex instead of subclasses. Would that be counter-productive?

Yes, possibly. My main problem is that on my computer intp means int64, while on some machines intp means int32. Also there have been some smaller bugs that I've found working on this that slowed me down.

I try again tonight or tomorrow and I will try the `Index.__new__` approach if I'm still having problems. Thanks for the suggestion.
Alright, it's all green finally! The failure in `Python Dev / actions-311-dev (macOS-latest) (pull_request)` says `ERROR: Could not find a version that satisfies the requirement setuptools==59.2.0`, so I assume unrelated to the PR).

This is quite a big PR, with lot of dtypes changes stemming from the fact that in this PR `Index` is able to return indexes with dtypes int8/int16/int32/uint8/uint16/uint32/float16/float32. There are some follow-ups after this as explained in the OP, but those should be much easier than this PR, because all the tricky dtype stuff is handled in this PR.

is there no reasonable way to split this into smaller chunks?
> is there no reasonable way to split this into smaller chunks?

I’ll look into it tonight.
I've handled the issues raised by @jbrockmendel, but I had to merge the all to many commits as part of a rebase at a point. Sorry anout that.

Some issues have been separated out in #50195 & #49536, (which this PR builds upon until they're merged), so  apart from those, this PR only deals with dtype changing part of allowing more flexible numeric indexes.
Updated & rebased to take into account that #49536 has been merged into main. This Pr still builds on #50195, so if someone could take a look at that also, then this PR can stand on its own. 
could split off a bunch of diff by temporarily patching `(Int64|UInt64|Float64Index).__repr__` to say "NumericIndex"?
Hi @jbrockmendel ,

I've looked into the failures after changing `np.int_` to `np.intp`, and the reason for the failures is that the use of `np.intp` is not equivalent to `np.int_` on Windows **and** `np.int_` is the default numpy integer dtype, according to [this page](https://docs.scipy.org/doc/numpy-1.10.1/user/basics.types.html).

So on all platforms  `np.array([1, 2, 3])` is equivalent to `np.array([1, 2, 3], dtype=np.int_)`

To the first point, on my Mac I have:

```python
>>> import numpy as np
>>> np.intp is np.int_
True
>>> np.int_ is np.int64
True
```
So on Mac (and Linux) it doesn't matter it we use `intp` or `int_`, they're exactly identical.

OTOH, on Windows (32-bit python installation) I have:

```python
>>> np.intp is np.int_  # different from Mac and Linux!
False
>>> np.int_ is np.int32
True
>>> np.intp  is np.intc
True
>>> type(np.int_(3))
int32
>>> type(np.intp(3))
intc # int32 or int64? This is weird
```

That `intp` is `intc` on Windows is very weird and caused many of the problems I had with this originally, because using `intp` passes all tests locally but fails on the CI Windows tests. My guess is that `intp` (and `intc`) is 32bit even on 64bit python installations on Windows, while `int_` always gives the same bit size as the numpy default (which is the correct bit size fort these tests).

So, in short & IMO, using `np.int_` is actually correct and using `np.intp` is wrong/will cause problems on Windows.

Can you comment, before I revert to `np.int_`?
For example, if you look at the failures in [this windows test](https://github.com/pandas-dev/pandas/actions/runs/3837454817/jobs/6532782315), you see they're all 32bit failures after applying `np.intp`, where they previously passed when using `np.int_`.
OK, thanks for taking a look
Ok thanks, I'll change it back then.
> could split off a bunch of diff by temporarily patching `(Int64|UInt64|Float64Index).__repr__` to say "NumericIndex"?

In a follow-up, if necessary? Getting this PR into main will make getting rid of those indexes a lot easier.
>> could split off a bunch of diff by temporarily patching (Int64|UInt64|Float64Index).__repr__ to say "NumericIndex"?

> In a follow-up, if necessary? Getting this PR into main will make getting rid of those indexes a lot easier.

This going in would make that idea unnecessary.  The idea of the suggestion was to make for a smaller diff here.
Ok, I understand.

I've made #50550, which is this PR, except not including the commits from #50195. If that passes the CI in the newest iteration, I plan to rebase this PR to exclude the #50195 commits, which will help as #50195 is not simple.

Also there is #50571 which should be excluded from this PR after being merged.

IMO the repr issue is relatively uncomplicated, and I'm a bit worried that changing the repr could introduce other problems (e.g. misunderstanding what we're looking at).
I'd like to see everything pass the CI again, before making new updates. I'll work on this again tomorrow.
the failing tests look like the same sqlalchemy stuff that is messing up the CI elsewhere, i.e. unrelated.  but being extra-careful is alright by me
Ping.

This PR builds on #50195, as mentioned, and that has to be pulled in before this PR. #50571 has been merged in.

There is currently a rebase issue, I'll handle that after eventual comments and hopefully merge of #50195.
looks like interval tests are failing
Updated after #50195 was merged + rebased. The remaining failure looks unrelated.
I'll wait for your response to the question about the `pd.Series.sparse.from_coo` question before I'll push an update for the PR.
Updated. All comments have been addressed AFAIKT.
Awesome, thanks @topper-123 
this is huge, thanks @topper-123