ENH: Integer NA Extension Array 
State: closed

closes #20700 
closes #20747

```
In [1]: df = pd.DataFrame({
'A': pd.Series([1, 2, np.nan], dtype='Int64'), 
'B': pd.Series([1, np.nan, 3], dtype='UInt8'), 
'C': [1, 2, 3]})

In [2]: df
Out[2]: 
     A    B  C
0    1    1  1
1    2  NaN  2
2  NaN    3  3

In [3]: df.dtypes
Out[3]: 
A    Int64
B    UInt8
C    int64
dtype: object

In [4]: df.A + df.B
Out[4]: 
0      2
1    NaN
2    NaN
dtype: Int64

In [5]: df.A + df.C
Out[5]: 
0      2
1      4
2    NaN
dtype: Int64

In [6]: (df.A + df.C) * 3
Out[6]: 
0      6
1     12
2    NaN
dtype: Int64
In [7]: (df.A + df.C) * 3 == 1
Out[7]: 
0    False
1    False
2    False
dtype: bool

In [8]: (df.A + df.C) * 3 == 12
Out[8]: 
0    False
1     True
2    False
dtype: bool
```

Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
cc @TomAugspurger @jorisvandenbossche @cpcloud @jbrockmendel @Dr-Irv 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=h1) Report
> Merging [#21160](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/537b65cb0fd2aa318e089c5e38f09e81d1a3fe35?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `95.48%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21160/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21160      +/-   ##
==========================================
+ Coverage   91.96%   91.98%   +0.01%     
==========================================
  Files         166      167       +1     
  Lines       50329    50606     +277     
==========================================
+ Hits        46287    46551     +264     
- Misses       4042     4055      +13
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.39% <95.48%> (+0.02%)` | :arrow_up: |
| #single | `42.18% <34.51%> (-0.05%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.18% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.95% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jhc2UucHk=) | `87.85% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.37% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/missing.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbWlzc2luZy5weQ==) | `91.66% <100%> (+0.02%)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `88.52% <100%> (+0.16%)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.1% <100%> (-0.02%)` | :arrow_down: |
| [pandas/core/arrays/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL19faW5pdF9fLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `95.49% <100%> (+0.06%)` | :arrow_up: |
| [pandas/core/dtypes/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbW1vbi5weQ==) | `95.2% <100%> (+0.12%)` | :arrow_up: |
| ... and [6 more](https://codecov.io/gh/pandas-dev/pandas/pull/21160/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=footer). Last update [537b65c...4f04f90](https://codecov.io/gh/pandas-dev/pandas/pull/21160?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

There's a lot here, so preliminary thoughts based on a quick pass:

- Getting integer NA implemented will be a big win!
- If there are bite-sized pieces that can be broken off, that will make life easier
- For a bitmask is there a good heuristic for when to use sparse vs dense?  "Too early to worry about that" is a reasonable answer.

- For ops, this skips several of the previously-discussed steps in transitioning to One True Implementation.
    - IIRC there are inconsistencies in Series vs Index division by zero behavior for numeric types.  I'd prefer to get these aligned before transitioning the dispatch logic.
    - Should IntegerArray also back `Int64Index` and `UInt64Index`?

@jreback Here is my take on the operators implementation, based on what I did in #20889.  Looking at your implementation in `pandas/core/arrays/integer.py`, you've defined the operators on the arrays themselves.  I did set things up in #20889 so that if someone did that with `ExtensionArray`, there is a dispatch to those methods.  The important thing in  `pandas/core/ops.py`  is that you can't assume anything about the underlying implementation of `ExtensionArray`, and it's possible that your changes to `ops.py` might be assuming that `ExtensionArray` is using a `numpy` implementation under the hood. I think (but I'm not 100% sure) that if you take what I did in `pandas/core/ops.py` with what you've done here, things would work correctly. 
I split out #21185 with the changes to EA. To answer some of @jbrockmendel questions

> Getting integer NA implemented will be a big win!
yes

> If there are bite-sized pieces that can be broken off, that will make life easier
see above

> For a bitmask is there a good heuristic for when to use sparse vs dense? "Too early to worry about that" is a reasonable answer.

I think this would overcomplicate things.

> For ops, this skips several of the previously-discussed steps in transitioning to One True Implementation.

how so, the implementation is barely changed here

> IIRC there are inconsistencies in Series vs Index division by zero behavior for numeric types. I'd prefer to get these aligned before transitioning the dispatch logic.

yes this is tricky, we are matching on the current impl. if we were to change this we should adjust.

> Should IntegerArray also back Int64Index and UInt64Index?

sure, but we don't really support the concept of an ExtensionIndex yet.
latest push remove the type specific arrays, IOW everything is now just a ``IntegerArray`` with a dtype; and added a repr that shares code with Index. (so it looks similar).
thanks @Dr-Irv added to #21191 
ok finally rebased on top of current ops impl. @jbrockmendel a couple of comments I have added to make vectorized ops work correctly.
@Dr-Irv moved ``make_data()``
@jreback looked at the `make_data()` move and it looks good

going to add reduce & groupby operations in next PR. so pls have a look @TomAugspurger @jorisvandenbossche @jbrockmendel @WillAyd 
```
In [54]: s = pd.Series([1, 2, 3], dtype='Int64')

In [55]: s + 1
Out[55]: 
0    2
1    3
2    4
dtype: Int64

In [56]: s + 1.
Out[56]: 
0    2
1    3
2    4
dtype: Int64

In [57]: s + 1.1
Out[57]: 
0    2.1
1    3.1
2    4.1
dtype: float64
```

second case should also be float
fixed up if you want to have a relook @jorisvandenbossche @TomAugspurger @jbrockmendel 

ps. numpy scalars should be banned :<
Still on the branch I fetched this afternoon (so might already be fixed), but astype was not yet working:

```
In [123]: s = pd.Series([1, 2, 3], dtype='Int64')

In [124]: s.astype('Int8')
Out[124]: 
0    1
1    2
2    3
dtype: int8    <--- note lower case here
```

```
In [125]: s = pd.Series([1, 2, 3, None], dtype='Int64')

In [127]: s.astype('Int8')
...
ValueError: cannot convert float NaN to integer
```
https://github.com/pandas-dev/pandas/pull/21160/commits/1ff394a04b805358feb9a2167f24dc09deaf6c76 fixes ``.astype()`` generally for ExtensionArrays when the string is specified for a dtyep.
cc @jorisvandenbossche 
@jorisvandenbossche @TomAugspurger should be good
finally passing again
if any final comments. @pandas-dev/pandas-core 
> if any final comments. @pandas-dev/pandas-core

Please give some time if possible, we're at Scipy so might take a few days. Or is that very annoying for a next PR?
well this has been out for quite a while
it’s not that annoying but anything else should be in a follow up as this is already very large 
@Dr-Irv so that is a good point

numpy silently converts floats to ints. We *could* warn or raise.
```
In [2]: np.array([1.2, 2.3, 3.7]).astype('int64')
Out[2]: array([1, 2, 3])
```

We normally convert things to new dtypes that can be directly converted w.o loss of precision. So I am inclined to raise here (note that numpy *does* raise on the above if a NaN appears, but we don't have that issue :)
@jreback could you summarize the memory format for this extension dtype in the PR description? 

For example, Arrow implements nullable integer as follows:

* 1 contiguous memory buffer containing null bits. A set bit (1) indicates non-null where a cleared bit indicates null

* 1 contiguous memory buffer containing the integer values

If you wanted, you could clone this using the bitarray (https://pypi.org/project/bitarray/) package if you are not already
latest push raises on casting non-equivalent floats -> ints (numpy allows this, but raising is better I think).
addressing memory layout (@wesm); these are currently hidden behind private attributes.

- uses a int8 mask where set is null and clear is non-null (could prob flip this)
- uses a contiguous ndarray of the same dtype to hold the data

so we have a bit of overhead compared to bitmasks (we use a byte per). I will look at ``bitarray``
should be simple to use, though it would add a dep (possibly could do this an an optional depenedecy - if easy may just do that).

but would do this in a followup.
Ideally the user would not be exposed to these details at least unless they are constructing from large-ish datasets. Using bits would help with zero copy from Arrow-land
> latest push raises on casting non-equivalent floats -> ints (numpy allows this, but raising is better I think).


\+ 1
@jorisvandenbossche 

> I think we should consider not using np.nan for the missing value indicator (as the value what you get back on scalar access, on conversion to numpy array

what is the reasoning here? users are accustomed to using ``np.nan`` and ``pd.NaT`` exclusively. (and not really ``None``), and an already constructed float array that is using integers + nan is a natural conversion.


latest push fixes dtype comparisons with strings & xfails DataFrame tests (they will be implemented in a followup, #21789) 

I haven't moved any tests yet. have to think about how to do this for all extension array types. We for sure want to test the interface, but don't want to split arbitrarily.
anyhow any final comments @pandas-dev/pandas-core 

certainly will be follows in any event.
@jorisvandenbossche