Add default repr for EAs
State: closed

Closes https://github.com/pandas-dev/pandas/issues/22846
Closes https://github.com/pandas-dev/pandas/issues/23590

```python
In [4]: pd.core.arrays.period_array(['2000', '2001', None], freq='D')
Out[4]:
<PeriodArray>
['2000-01-01', '2001-01-01', 'NaT']
Length: 3, dtype: period[D]

In [5]: pd.core.arrays.period_array(['2000', '2001', None] * 100, freq='D')
Out[5]:
<PeriodArray>
['2000-01-01', '2001-01-01',        'NaT', '2000-01-01', '2001-01-01',
        'NaT', '2000-01-01', '2001-01-01',        'NaT', '2000-01-01',
 ...
        'NaT', '2000-01-01', '2001-01-01',        'NaT', '2000-01-01',
 '2001-01-01',        'NaT', '2000-01-01', '2001-01-01',        'NaT']
Length: 300, dtype: period[D]

In [6]: pd.core.arrays.integer_array([1, 2, None])
Out[6]:
<IntegerArray>
[1, 2, NaN]
Length: 3, dtype: Int64

In [7]: pd.core.arrays.integer_array([1, 2, None] * 1000)
Out[7]:
<IntegerArray>
[  1,   2, NaN,   1,   2, NaN,   1,   2, NaN,   1,
 ...
 NaN,   1,   2, NaN,   1,   2, NaN,   1,   2, NaN]
Length: 3000, dtype: Int64

```

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @TomAugspurger! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/core/arrays/base.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/core/arrays/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/categorical.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/core/arrays/categorical.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/integer.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/core/arrays/integer.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/interval.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/core/arrays/interval.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/period.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/core/arrays/period.py) !

 - There are no PEP8 issues in the file [`pandas/io/formats/printing.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/io/formats/printing.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/interval/test_interval.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/arrays/interval/test_interval.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/test_integer.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/arrays/test_integer.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/test_period.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/arrays/test_period.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/base/__init__.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/base/__init__.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/base/interface.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/base/interface.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/base/printing.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/base/printing.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/decimal/array.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/decimal/array.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/decimal/test_decimal.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/decimal/test_decimal.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/json/array.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/json/array.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/json/test_json.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/json/test_json.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/test_integer.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/test_integer.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/test_interval.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/test_interval.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/test_period.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/test_period.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/test_sparse.py`](https://github.com/TomAugspurger/pandas/blob/0fdbfd34add4fe9940cd1991122e851d62824724/pandas/tests/extension/test_sparse.py) !


```python
In [3]: integer_array([1, 2, 3])
Out[3]:
<IntegerArray>
[1, 2, 3]
Length: 3, dtype: Int64

In [4]: period_array(['2000', '2001'], freq='D')
Out[4]:
<PeriodArray>
[2000-01-01, 2001-01-01]
Length: 2, dtype: period[D]

In [5]: IntervalArray.from_breaks([1, 2, 3])
Out[5]:
<IntervalArray>
[(1, 2], (2, 3]]
Length: 2, dtype: interval[int64]

In [6]: integer_array([1, 2, 3] * 1000)
Out[6]:
<IntegerArray>
[1, 2, 3, 1, 2, 3, 1, 2, 3, 1,
 ...
 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
Length: 3000, dtype: Int64

```
For DatetimeArray & Timedelta I was waiting to see what happens on https://github.com/pandas-dev/pandas/pull/23587. If @jbrockmendel reverts the repr changes before merging then I'll add it here. Otherwise I'll just delete them after that's merged.

>  (in case we want to keep printing the freq for DatetimeArray/Timedelta)Array

> Do we need a mechanism to indicate which attributes to print in addition to length and dtype? (in case we want to keep printing the freq for DatetimeArray/Timedelta)Array

Sure. I suppose that info doesn't belong in the dtype (right?) so we can add hooks for extra attrs.

> Is there any control over the number of elements shown?

At the array level, no. But I think following the option at `pd.options.display.max_seq_items` is the right thing to do. I'll add a note to the docs.
The main difference for PeriodArray seems to be that the values are no longer quoted? (not a strong opinion here though, for Index reprs we use quotes and also numpy quotes dates)
```diff
diff --git a/pandas/core/arrays/period.py b/pandas/core/arrays/period.py
index f6996f8e6..4bc7841fe 100644
--- a/pandas/core/arrays/period.py
+++ b/pandas/core/arrays/period.py
@@ -330,6 +330,10 @@ class PeriodArray(dtl.DatetimeLikeArrayMixin, ExtensionArray):
     def end_time(self):
         return self.to_timestamp(how='end')
 
+    @property
+    def _formatter(self):
+        return "'{}'".format
+
     def __setitem__(
             self,
             key,   # type: Union[int, Sequence[int], Sequence[bool]]
```

will quote. Gonna mixup the py2 issues.
Update:

1. deprecated `ExtensionArray._formatting_values`
2. Change `_formatter` to a function that takes a flag for whether or we're printing inside a Series / DataFrame


```python
In [5]: integer_array([1, 2, None] * 1000)
Out[5]:
<IntegerArray>
[  1,   2, nan,   1,   2, nan,   1,   2, nan,   1,
 ...
 nan,   1,   2, nan,   1,   2, nan,   1,   2, nan]
Length: 3000, dtype: Int64

In [6]: IntervalArray.from_breaks(list(range(1000)))
Out[6]:
<IntervalArray>
[    (0, 1],     (1, 2],     (2, 3],     (3, 4],     (4, 5],     (5, 6],
     (6, 7],     (7, 8],     (8, 9],    (9, 10],
 ...
 (989, 990], (990, 991], (991, 992], (992, 993], (993, 994], (994, 995],
 (995, 996], (996, 997], (997, 998], (998, 999]]
Length: 999, dtype: interval[int64]

In [7]: period_array(['2000', '2001'] * 1000, freq='D')
Out[7]:
<PeriodArray>
['2000-01-01', '2001-01-01', '2000-01-01', '2001-01-01', '2000-01-01',
 '2001-01-01', '2000-01-01', '2001-01-01', '2000-01-01', '2001-01-01',
 ...
 '2000-01-01', '2001-01-01', '2000-01-01', '2001-01-01', '2000-01-01',
 '2001-01-01', '2000-01-01', '2001-01-01', '2000-01-01', '2001-01-01']
Length: 2000, dtype: period[D]
```
> deprecated: ExtensionArray._formatting_values

has this been around for a while? its a private attribute, why deprecate?
It's part of the interface and was around since 0.23.
This are into a bit larger of a refactor... I removed {Categorica,Period,Interval}ArrayFormatter in favor of a generic ExtensionArrayFormatter.

EAs will get control over formatting of individual values by overriding `ExtensionArray._formatter`.
FYI, I reverted the changes to the IntervalArray repr. Some doctests shared with IntervalIndex were failing.
This is also a base repr for 3rd parties. Having reprs that look like code but isn't is a bad practice. We shouldn't put that on other libraries by default.

We are of course able to override the reps of our first party EAs if we really want to, but a foolish consistency and all that.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=h1) Report
> Merging [#23601](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/08395af43b813b69b1cf038d260104cfeeb3e787?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `35.71%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23601/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23601      +/-   ##
==========================================
- Coverage   42.38%   42.37%   -0.02%     
==========================================
  Files         161      161              
  Lines       51701    51706       +5     
==========================================
- Hits        21914    21909       -5     
- Misses      29787    29797      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `42.37% <35.71%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVydmFsLnB5) | `35.08% <ø> (-0.09%)` | :arrow_down: |
| [pandas/core/indexes/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9wZXJpb2QucHk=) | `40.84% <0%> (ø)` | :arrow_up: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `52.19% <14.28%> (-0.18%)` | :arrow_down: |
| [pandas/core/arrays/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jhc2UucHk=) | `49.67% <25%> (-0.66%)` | :arrow_down: |
| [pandas/core/arrays/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3BlcmlvZC5weQ==) | `36.78% <25%> (-0.11%)` | :arrow_down: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `37.32% <28.57%> (-0.74%)` | :arrow_down: |
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `50.29% <31.25%> (-0.06%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `41.89% <50%> (ø)` | :arrow_up: |
| [pandas/core/arrays/sparse.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3NwYXJzZS5weQ==) | `40.45% <50%> (+0.02%)` | :arrow_up: |
| [pandas/io/formats/printing.py](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvcHJpbnRpbmcucHk=) | `67.18% <61.53%> (-0.02%)` | :arrow_down: |
| ... and [1 more](https://codecov.io/gh/pandas-dev/pandas/pull/23601/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=footer). Last update [08395af...6ad113b](https://codecov.io/gh/pandas-dev/pandas/pull/23601?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

General other question: should the default `_formatter` be `str` or `repr` ?

Eg, numpy uses `repr`:

```
In [31]: np.array([datetime.datetime(2012,1,1)])
Out[31]: array([datetime.datetime(2012, 1, 1, 0, 0)], dtype=object)
```

(which in the specific above case is useful to be able to see it are datetime.datetime objects)

I think for our own arrays, we typically use `str` (eg for Timestamp / Period, we don't include the class name in each scalar). 

Of course this is always easy to override by an EA author if the default is not what they want.
I suppose it make sense for `__repr__` to use `repr` of the scalar.
Changes from earlier:

1. Changed FutureWarning -> DeprecationWarning
2. Changed the default `ExtensionArray._formatter` to return `repr`, not `str`.
3. Changed the implementation to define `__repr__` instead of `__unicde__`. (I didn't realize that `__unicode__` wasn't part of the Python data model).
btw do you need the ``__bytes__`` ?
Hmm I was thinking people may have expected it to follow the behavior of Categorical, but now I'm thinking that's not a good idea in general (for the default). I think that EAs, being array-like, should follow NumPy and offer something that can be dumped in to a memory view:

```python
In [12]: np.frombuffer(memoryview(bytes(np.array([1, 2]))), dtype=np.int64)
Out[12]: array([1, 2])
```

But the default can't know what actual data needs to be dumped, so we should (I think) just not implement it.
All green here. I think this is a good default repr for 3rd party EAs.

We can talk further about our own EAs, but

1. I value consistency with Categorical over consistency with Index reprs, since Categorical is an EA.
2. I'm not especially interested in changing the Categorical repr to match the index ones.

Where do we stand here? Are we OK going forward with this as a base repr?
I am fine with merging this.

@jreback can you respond to the comments in https://github.com/pandas-dev/pandas/issues/22846
I responded. -1  on the specific changes as outlined. The general approach is nice to actually re-use existing code.
How should we proceed then?

On Thu, Nov 15, 2018 at 7:16 AM Jeff Reback <notifications@github.com>
wrote:

> I responded. -1 on the specific changes as outlined. The general approach
> is nice to actually re-use existing code.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/23601#issuecomment-439036371>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIrhq6BDlXwCgtRnDKMM5Ok8Pv5oXks5uvWksgaJpZM4YWyCT>
> .
>

Latest push simplifies `ExtensionArray._formatter` for the EA author. Now they just receive a keyword `boxed=False` for whether pandas is calling the `repr` to be printed inside a Series, DataFrame (or Index eventually).

We use that in PeriodArray to quote, or not, depending on the context.
Responding to a couple places (https://travis-ci.org/pandas-dev/pandas/jobs/457327081#L2551, 
https://github.com/pandas-dev/pandas/pull/23601#discussion_r234605325)

In terms of the repr I think ExtensionArray is closer to Series than Index. Since neither is (known) to be valid code, the values don't need to be quoted. And I would argue we don't want to quote here, since then you’d have ambiguity with an array of strings.
I think the outstanding discussions are

1. capital vs. lowercase "L" in "Length": 
https://github.com/pandas-dev/pandas/pull/23601#discussion_r234464865
2. The `boxed` argument passed to the `_formatter`: 
https://github.com/pandas-dev/pandas/pull/23601#discussion_r234464938 (reply at 
https://github.com/pandas-dev/pandas/pull/23601#issuecomment-439882626)
Question related to the boxed argument / quotes or not: assume we have a StringArray, how would we want it to look like? Quoted or not quoted? 
In a Series, we don't quote strings, but in such an array repr, I would actually be fine with having it quoted? So in a similar sense, it is maybe fine to have the Periods quoted in the Array repr?
> Question related to the boxed argument / quotes or not: assume we have a StringArray, how would we want it to look like? Quoted or not quoted?

I would say quoted. I don't have a strong preference for PeriodArray. In either case, it shouldn't be ambiguous what the contents are, since they're type-specific containers. I opted for no quotes since they seemed unnecessary.
Do people have preferences on quoting or not in Period/Datetime/TimedeltaArray?
I think we *should* quote in the arrays (e.g. like we do in Index), but not in Series.
Period quoting is fixed now (locally).

I'm playing around with short 1-line reprs. What do people think?

```python
In [3]: to_decimal([1, 2, 3])
Out[3]: <DecimalArray>[Decimal('1'), Decimal('2'), Decimal('3')], dtype: decimal

In [4]: pd.core.arrays.integer_array([1, None])
Out[4]: <IntegerArray>[1, NaN], dtype: Int64

In [6]: pd.core.arrays.period_array(['2000', '2001'], freq='D')
Out[6]: <PeriodArray>['2000-01-01', '2001-01-01'], dtype: period[D]
```

I'm not sure what I think.