API: added array
State: closed

Adds

- a new top-level `pd.array` method for creating arrays
- all our extension dtypes to the top-level API
- adds pd.arrays for all our EAs

TODO

- [x] Add the actual array classes somewhere to the public API (pandas.arrays?)
- [x] API docs for the rest of the arrays and dtypes.

Closes #22860 

supersedes #23532.

Labels:
name: API Design
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @TomAugspurger! Thanks for updating the PR.



























Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on December 28, 2018 at 22:13 Hours UTC
I'd like to restructure the documentation a bit. I want to collect similar things together, but we have two kinds of similarity:

First, we could group by "topic" (the type of data).

- Categorical
  - dtype
  - Categorical
  - CategoricalIndex?
- Period
  - dtype
  - scalar
  - array
  - PeriodIndex(?) or leave that under indexes probably
- Integer
  - ...
- Interval
  - ...
- Sparse
  - ...

and then eventually DatetimeArray & TimedeltaArray. 

Alternatively, we could group by "kind" first. So we'd have

- Dtypes
  - CategoricalDtype
  - PeriodDtype
  - ...
- Scalars
  - Period
  - ...
- Arrays
   - Categorical
   - PeriodArray
   - ...

Do people have a preference between "by topic" and "by kind" (cc. @jorisvandenbossche @jreback @jbrockmendel @datapythonista)?
> I'd like to restructure the documentation a bit.

I suppose you are only talking about the api.rst page?
Yes, sorry.
Not a strong opinion, but for adding new docs I would go by kind. 
But that is in the assumption that the above is only for listing the arrays / dtypes, and that there are still separate sections about each topic anyway? (eg with all Interval-specific or datetime-like methods/attributes)
The nice thing about "by topic" is that we can give a high-level summary of what, e.g. "Period" is for.  If we have things grouped "by kind" (dtypes, scalars, arrays), then we'd need to repeat that description, or just have it once.
> The nice thing about "by topic" is that we can give a high-level summary of what, e.g. "Period" is for

But in those places you would also put all the custom attributes and methods? 

I actually think we can be duplicated somewhat. Even if there are topical sections, I think it is still nice to have a short table of all dtypes and one for all array types.
Did a bit of inference in fe06de4. There are a few TODOs.

I think we don't handle intervals yet. https://github.com/pandas-dev/pandas/issues/23553
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=h1) Report
> Merging [#23581](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c1af4f5ede3f6dfb1d1652c62afbcdc88b5eef91?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23581/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23581      +/-   ##
==========================================
+ Coverage   92.31%   92.32%   +0.01%     
==========================================
  Files         165      166       +1     
  Lines       52194    52240      +46     
==========================================
+ Hits        48182    48231      +49     
+ Misses       4012     4009       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.74% <100%> (+0.01%)` | :arrow_up: |
| #single | `43.07% <28.94%> (+0.09%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3BlcmlvZC5weQ==) | `98.42% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVydmFsLnB5) | `93.04% <ø> (ø)` | :arrow_up: |
| [pandas/core/dtypes/dtypes.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2R0eXBlcy5weQ==) | `95.33% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL19faW5pdF9fLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/array\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2FycmF5Xy5weQ==) | `100% <100%> (ø)` | |
| [pandas/core/arrays/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jhc2UucHk=) | `98.23% <0%> (+0.03%)` | :arrow_up: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `92.17% <0%> (+0.06%)` | :arrow_up: |
| [pandas/core/arrays/numpy\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL251bXB5Xy5weQ==) | `93.51% <0%> (+0.46%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/pandas-dev/pandas/pull/23581/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=footer). Last update [c1af4f5...1b9e251](https://codecov.io/gh/pandas-dev/pandas/pull/23581?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

looks good @TomAugspurger mostly some docs & clarification comments, generally looks good though.
From https://github.com/pandas-dev/pandas/pull/23581#discussion_r232490269

> https://github.com/pandas-dev/pandas/pull/23581#discussion_r232490269

No, we don't because I hadn't really considered what to do about that. Do people have thoughts on how to handle 2-D (or n-d) input inside `pd.array`? In general, it shouldn't be up to `pd.array` what is and isn't valid input. That's up to the individual array constructors. But is 2-D special since NumPy handles it but EAs don't?
Another question: when given a series, index, or EA for `data`, we ignore `data.dtype` even if the user-specified dtype is `None`. So we get this (maybe strange?) result:

```python
In [2]: arr = pd.Categorical(['a', 'a', 'b'])

In [3]: pd.array(arr)
Out[3]: array(['a', 'a', 'b'], dtype=object)
```

Note, however, that we *cannot* just take a `dtype`  from an ndarray.  In the following, we ignore `data.dtype` (object) so that we can infer period.

```python
In [6]: arr = np.array([pd.Period('2000', 'D'), pd.Period('2000', 'D')])

In [7]: pd.array(arr)
Out[7]:
<PeriodArray>
['2000-01-01', '2000-01-01']
Length: 2, dtype: period[D]
```

So my proposal: when the user-specified `dtype` is None, pandas will

1. use the dtype from a Series / Index / EA, as if it was user-specfied. else
2. infer the the dtype from the data (including for ndarray)
sounds reasonable for https://github.com/pandas-dev/pandas/pull/23581#issuecomment-437869870
> So my proposal: when the user-specified dtype is None, pandas will
>
>    use the dtype from a Series / Index / EA, as if it was user-specfied. else
>    infer the the dtype from the data (including for ndarray)


I think the `pd.array(arr)` where `arr` is a Categorical returning not a categorical is something we should fix. 
But as I understand, that should be changed with above logic?


Yes,
838ce5e makes pd.array idempotent for ExtensionArrays. We extract dtype info from Series, Index or EAs when `dtype=None`
Revisiting this today. I think the biggest outstanding question is what to do about 2-D (or larger) input arrays.

I think attempting to detect 2-D arrays *before* they're created (from the raw `data`) is impossible. Especially if pandas or some 3rd party array wants to support nested data.

If so, our only option is to raise when we detect a 2D array output, which at this point just means an ndarray. Do we want that?
> If so, our only option is to raise when we detect a 2D array output, which at this point just means an ndarray. Do we want that?

I think we have checks in the pandas EA's for non-1D input. So if someone passes something non-1D and an EA dtype it won't return an ndarray (it will just raise), right?

can you show a case where there is a problem?
> I think we have checks in the pandas EA's for non-1D input. So if someone passes something non-1D and an EA dtype it won't return an ndarray (it will just raise), right?

Right.

There's no real problem with pd.array returning 2-D arrays

```python
In [2]: pd.array([[1, 2], [3, 4]])
Out[2]:
array([[1, 2],
       [3, 4]])
```

it's just a matter of setting expectations. I'm fine with letting that pass through.
@TomAugspurger yeah i agree, its either an EA, which may (or actually may not restrict) 2-Dness (e.g. imagine a Image EA array), or it passes to np.array
Pushed some tests with support for 2D.

Test failure is the rank segfault.

This should be good to go.
will look soon
Will try to do a review on Monday evening
lgtm.
> I think an arrow-backed Decimal array would make sense (this one is backed by an object-dtype ndarray).

I would leave that for a dedicated project for now (like fletcher), we already have a lot on our plate.

But that said: assume we add a DecimalArray (backed by arrow or not) in one of the next releases. How do we then see the "backwards compatibility guarantees" of this function? Because we would want to start returning a DecimalArray instead of object np.ndarray when decimals are passed to `pd.array`. But, this is of course not strictly backwards compatible. 
Assuming that situation, what should we do?

> Assuming that situation, what should we do?

I think it comes down to the dtypes registry. All we can say is that for a given registry, `pd.array(..., dtype=dtype)` will be backwards compatible, right?

We could perhaps give users an API for manipulating the dtypes registry.
> All we can say is that for a given registry, pd.array(..., dtype=dtype) will be backwards compatible, right?

That makes sense to me, but then we should make that clearer in the docstring? Because it basically means we guarantee no exact backwards compatibility if `dtype` is not specified.
Added a note to that @jorisvandenbossche 

```
    Omitting the `dtype` argument means pandas will attempt to infer the
    best array type from the values in the data. As new array types are
    added by pandas and 3rd party libraries, the best array type may
    change. We recommend specifying `dtype` to ensure that the correct
    array type is constructed.
```
lgtm. ping on green.
> lgtm. ping on green.

To be explicit about it this time, don't merge it yet on green, I am still reviewing it.
@jorisvandenbossche ok sure. in the future, easiest if you simply put a review block then.