ENH: Create and propagate UInt64Index
State: closed

1) Introduces and propagates `UInt64Index`, an index specifically for `uint`.  xref #14935

2) <strike> Patches bug from #14916 that makes `maybe_convert_objects` robust against the known `numpy` bug that `uint64` cannot be compared to `int64`.  This bug was caught during testing of `UInt64Index`. </strike>

**UPDATE**: Patched in #14951


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can you separate out

> Patches bug from #14916 that makes maybe_convert_objects robust against the known numpy bug that uint64 cannot be compared to int64. This bug was caught during testing of UInt64Index.
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14937?src=pr) is 85.54% (diff: 95.00%)
> Merging [#14937](https://codecov.io/gh/pandas-dev/pandas/pull/14937?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14937   diff @@
==========================================
  Files           145        145          
  Lines         51288      51350    +62   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43872      43928    +56   
- Misses         7416       7422     +6   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [0e219d7...8ab6fbd](https://codecov.io/gh/pandas-dev/pandas/compare/0e219d79dcc6815ed651c2fc6d1b85e1e0d71bd6...8ab6fbd544c48f0b4a4b891016313abc7e070e91?src=pr)
@jreback : See #14951 for patching `maybe_convert_objects`
so need testing / fixes for 

- assignment to DataFrame
- set/reset index
- indexing (getting and setting), add a new test class

Assignment to DataFrame: `frame/test_alter_axes.py` ?
Add a new test class : `indexes/test_numeric.py` ?

Set / reset index ? (not sure what you mean)
Getting / setting ? (not sure what you mean)
> Set / reset index ? (not sure what you mean)
``df.set_index('uint64_col').reset_index()``

> Getting / setting ? (not sure what you mean)
df2 = df.set_index('uint64_col')
df2.loc[df.index[0]] (and =)

basically need a pandas/tests/indexing/test_uint64.py
Would appending `uint` to the battery of tests in `tests/indexing/test_indexing.py` suffice then?
@gfyoung yes adding uint to lots of tests would definitly help.
@jreback : Added `UInt64Index` to huge number of tests, and everything is green.
@gfyoung so some additional cases to check, e.g https://github.com/pandas-dev/pandas/issues/15027 (this may be quite involved actually. a uint group-and should be turned into a uint64 grouper).
@jreback : Yeah...looking at the code, it seems that this will be somewhat involved.  Can we patch that in a separate PR (and maybe just reopen it for starters)?
@jreback : Perhaps we can just focus on making sure in this PR that the changes I have proposed are good to merge.  How does the PR look at this point?
@jreback : Do you think you could do a re-look over this PR and tell me what things should be changed before this can be merged?
@jreback : Tons of new tests were added with the sub-class, all of the refactorings requested, and everything is still green!

My only concern is this doc-building instance <a href="https://travis-ci.org/pandas-dev/pandas/jobs/189708087">here</a>, which has consistently failed because for some reason it cannot pick up my changes to the `pxi.in` files.  I built the docs locally, and everything is fine.  Any ideas on how to remedy this?
@gfyoung looks pretty good. just a few more minor comments. I still have to play with this a bit more though.

@jorisvandenbossche 
I got the same import issues when I pulled this down. *maybe* some dep is not recompiling? (I didn't try, but I bet it works with a full clean first)

```
/Users/jreback/pandas/pandas/indexes/api.py in <module>()
      4 from pandas.indexes.category import CategoricalIndex  # noqa
      5 from pandas.indexes.multi import MultiIndex  # noqa
----> 6 from pandas.indexes.numeric import (NumericIndex, Float64Index,  # noqa
      7                                     Int64Index, UInt64Index)
      8 from pandas.indexes.range import RangeIndex  # noqa

/Users/jreback/pandas/pandas/indexes/numeric.py in <module>()
    171 
    172 
--> 173 class UInt64Index(NumericIndex):
    174     __doc__ = _num_index_shared_docs['class_descr'] % _uint64_descr_args
    175 

/Users/jreback/pandas/pandas/indexes/numeric.py in UInt64Index()
    184     _na_value = 0
    185 
--> 186     _engine_type = _index.UInt64Engine
    187 
    188     _default_dtype = np.uint64

AttributeError: module 'pandas.index' has no attribute 'UInt64Engine'

```
I also had to do a clean + full build to get the branch working (not convenient in this specific case, but is that actually indicating a problem?)
@jorisvandenbossche , @jreback : See my comment <a href="https://github.com/pandas-dev/pandas/pull/14937#issuecomment-271062525">above</a>.  This might be indicative of a `setup.py` issue since we have to do a `git clean` before installing.  It isn't breaking Travis, but this is a little disconcerting admittedly.
need 

``'depends': _pxi_dep['index']``

here `https://github.com/pandas-dev/pandas/blob/master/setup.py#L493
@jreback , @jorisvandenbossche : Added the dependency in `setup.py` as requested.  The installation should work.  If it does, then this should be ready to merge then.
Was playing a bit with it, and got into this:

```
In [15]: idx = pd.UInt64Index([1,2,3,4])

In [16]: s = pd.Series(range(4), index=idx)

In [17]: s.append(pd.Series([1,2], index=[-1,8])).index
Out[17]: Float64Index([1.0, 2.0, 3.0, 4.0, -1.0, 8.0], dtype='float64')
```
> Was playing a bit with it, and got into this:
> ```
> In [17]: s.append(pd.Series([1,2], index=[-1,8])).index
> Out[17]: Float64Index([1.0, 2.0, 3.0, 4.0, -1.0, 8.0], dtype='float64')
> ```

This looks like NumPy's default type promotion rules for combining int64 and uint64:
```
In [33]: np.concatenate([np.array([1, 2], dtype=np.int64), np.array([3], dtype=np.uint64)])
Out[33]: array([ 1.,  2.,  3.])
```
@shoyer , @jorisvandenbossche : At first glance, I would agree with that diagnosis.  Combining `int64` and `uint64` is "dangerous" from that perspective unfortunately (and out of our control).  For now, I would ensure that `uint64`-only interactions result in `uint64`.
Ah yes, of course, overlooked that. Naively thought for a moment this fitted in int64 and so should be that. But fully agree that we should not check for this and just follow the rules.
you whatsnew example is a little odd FYI, maybe expand a touch, and here's a bug

```
In [42]:    idx = pd.UInt64Index([1, 2, 3], name='foo')
    ...:    df = pd.DataFrame({'A' : ['a', 'b', 'c'], 'B' : range(3), 'C' : np.array([3,4, 5],dtype='uint64')}, index=idx)
    ...:    
    ...: 

In [43]: df2 = df.reset_index()

In [44]: df2
Out[44]: 
   foo  A  B  C
0    1  a  0  3
1    2  b  1  4
2    3  c  2  5

In [45]: df2.dtypes
Out[45]: 
foo    uint64
A      object
B       int64
C      uint64
dtype: object

In [46]: df2.iloc[1, 0] = np.nan

In [47]: df2
Out[47]: 
   foo  A  B  C
0  1.0  a  0  3
1  NaN  b  1  4
2  3.0  c  2  5

In [48]: df2.dtypes
Out[48]: 
foo    float64
A       object
B        int64
C       uint64
dtype: object

In [49]: df2.foo.fillna(-1)
Out[49]: 
0    1.0
1   -1.0
2    3.0
Name: foo, dtype: float64

In [50]: df2.foo.fillna(-1).astype('uint64')
Out[50]: 
0                       1
1    18446744073709551615
2                       3
Name: foo, dtype: uint64
```
[50] is wrong
@jreback :

1) What is odd about my example, and how should I expand it if it is lacking?

2) What is the expected behavior for [50] in your opinion, as I have a similar question in this example:
~~~python
>>> i = Index([1.1, 2, 3])
>>> i.astype('int64')  # Just plain INT64
0    1
1    2
2    3
dtype: int64
~~~
> What is odd about my example, and how should I expand it if it is lacking?

I find not naming columns as odd (in examples).

```
In [37]: pd.DataFrame({'A' : ['a', 'b', 'c']}, index=range(3))
Out[37]: 
   A
0  a
1  b
2  c
```
> What is the expected behavior for [50] in your opinion, as I have a similar question in this example:

I would raise, that is an invalid conversion if negative values or nan's are present

```
In [39]: Series([np.nan, 2, 3]).astype('int64')
ValueError: Cannot convert non-finite values (NA or inf) to integer
```

In fact I make sure to test lots of astyping (similar to what we do with ints)
@jreback : The point of my example in my response was to illustrate that `astype` is bugged far beyond anything that I could have added with `UInt64Index`.  I think it is better to save patching that for another time, as the bug extends beyond `uint64`.  I will update the `whatsnew` though.
@jreback : Updated `whatsnew`, and everything is still green.