ENH: add and register Arrow extension types for Period and Interval
State: closed

Related to https://github.com/pandas-dev/pandas/pull/28368, but now for Period and Interval for which we define extension types to store them with metadata in arrow.

Still needs some more tests and fixing corner cases. 
We probably also want to consolidate the pyarrow import checking somewhat.

I think a main question is how to deal with the import of pyarrow. The way I did it now makes that it is no longer a lazy import (as it was currently for the parquet functionality).

Labels:
name: Period
description: Period data type
name: Interval
description: Interval data type
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@jorisvandenbossche is this actionable?
I have been updating this PR locally last week, but still thinking through how we can deal with a delayed pyarrow import. 

I think a clear disadvantage of always trying to import pyarrow is import time (something we have been trying to reduce). 
But the main problem of only importing pyarrow when needed here, is that those extension types need to be registered when arrow is creating the data (eg from reading a parquet file, or receiving IPC messages). So it is not necessarily pandas that knows when it is needed to have those types registered (eg when using pyarrow to read a parquet file and convert to pandas instead of pandas' `read_parquet` function). 

So we could have a publicly exposed `register_arrow_types` function that does this, but that seems rather inconvenient if users have to call this manually in certain cases ..
Updated this, still todo:

- [x] Fix missing value handling for interval types
- [x] Clean-up conversion of numpy/pandas dtype to arrow dtype
- [x] Move some arrow conversion code to common utilities (will also be useful for integer/string array)
- [ ] Further look into if a lazy import is possible
Any more feedback on the import issue? (see https://github.com/pandas-dev/pandas/pull/28371#issuecomment-549177206)


@jorisvandenbossche the way you did the imports is fine; we must lazily import pyarrow or make it a hard dep

i think that would be better to make some functions to reduce the copy paste of what you did here in this PR. basically you isolate the arrow code (meaning you directly import it at the top of the module) in a separate module (file). then import that module conditionally.
> the way you did the imports is fine; we must lazily import pyarrow or make it a hard dep

Yes, but the lazily import can be done in two ways: always try on pandas import, or only try when someone uses the parquet functionality.

It's similar as with plotting: matplotlib is not a hard dependency, but before, we always tried to import it (and register some converters), while now we moved to only importing it when someone tries to plot.
pyarrow import takes ~160 ms for me, but 135 of that is numpy, so would only add about 25 ms to our import time.  If its easy to avoid that'd be nice, but not worth significant gymnastics
Ah, that's interesting (I tried to time it a while ago, but seem to remember is was rather fluctuating). 
In such a case, I would personally maybe prefer to always try to import (although matplotlib is also not that much slower to import ..)
@TomAugspurger @jreback more comments on the import?

For code organisation: I now put most of it in the array files (arrays/period.py and arrays/interval.py) and part in the dtypes/dtypes.py file (each time where it is used). Do people have preference to move the arrow related code to separate files? And eg make `arrays/period/` a directory that has a separate file for arrow stuff. Or a single arrays/arrow_utils.py file that combines the code of the different data types?
For now, importing arrow just to register the dtypes seems fine.

Long-term, I think that both we and pyarrow should be using entrypoints for this kind of thing. This kind of stuff is fixed at install time, so we should be using the tools for that (with fallback to the dynamic registration we've done so far). I can open an issue for pandas.

No preference on organization, the way you have it now seems fine.
For the entrypoints, Tom opened an issue at https://github.com/pandas-dev/pandas/issues/29774
@jorisvandenbossche does this ImportError make any sense to you? https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=21469

```
E   ImportError: cannot import name 'chunked_array'
```

that's internal to pyarrow, right?
@jorisvandenbossche pls rebase
@jorisvandenbossche pls rebase
I will try to rebase shortly. Any final concerns on the lazy vs non-lazy import? 

Right now, I would propose to keep this PR as is (i.e. non-lazy import -> always try to import pyarrow on pandas import if it is installed; Brock showed above that the additional import time (on top of importing numpy) is rather limited), but as I mentioned above, I can also refactor it to have the lazy import
I’m ok with non-lazy. 
There is a CI failure for the Linux py36_locale build: apparently the pyarrow install failed there (and for the other tests, this gets ignored / skipped, so that's the reason we didn't see it yet). So that raises two issues: 1) we should fix that CI env and 2) we should probably catch a more general error to avoid that a wrong pyarrow installation let the pandas import fail.
> There is a CI failure for the Linux py36_locale build: apparently the pyarrow install failed there (and for the other tests, this gets ignored / skipped, so that's the reason we didn't see it yet). So that raises two issues: 1) we should fix that CI env and 2) we should probably catch a more general error to avoid that a wrong pyarrow installation let the pandas import fail.

2) absolutely not

we need to see failures that are not the result of expected things 
OK, so the reason is not a failed installation, but actually that pyarrow up to version 0.12 also tries to import pandas, so you get a circular dependency which outs itself in the AttributeError.

So, one option is to increase the minimum pyarrow version to 0.13. But, the problem with that is that this will still give a confusing error message when people have pyarrow 0.12 installed.


> OK, so the reason is not a failed installation, but actually that pyarrow up to version 0.12 also tries to import pandas, so you get a circular dependency which outs itself in the AttributeError.
> 
> So, one option is to increase the minimum pyarrow version to 0.13. But, the problem with that is that this will still give a confusing error message when people have pyarrow 0.12 installed.

we already bumped to 0.12 for various things, you could push this to 0.13 (no objection). these conversions require a higher version anyhow? (e.g. interval/period)?
OK, for now (to get this PR in a mergeable state), I moved away from always trying to import. This will give some realistic corner cases (like reading a parquet file with pyarrow instead of with pandas will not give the extension type), but we can see this new feature as experimental anyway, and to be improved later.

By not importing it by default, we avoid the circular import problem with old pyarrow versions. Bumping the required pyarrow version might solve this, but that still causes pyarrow to become un-importable, if you install pyarrow 0.12 and new pandas side by side with a very cryptic error message. 
I might still try to add something more clever (based on `pkg_resources` to check the installed pyarrow version, to avoid needing to import pyarrow to know if it is recent enough to import, but that can go in a separate PR).
The remaining "failure" is codecov because the functionality I added is not run in the coverage build (only with arrow master)
> What's the behavior for __arrow_array__ pyarrow 0.15 and earlier? Is it just not called, so we don't need to worry about checking the pyarrow version within there?

Indeed, it should never be called with versions older than 0.15 (unless you would manually call the method, but the method is only called by pyarrow starting from 0.15)
Great, +1 then.