BUG: avoid attribute error with pyarrow >=0.16.0 and <1.0.0
State: closed

Problem: The minimum pyarrow [listed](https://pandas.pydata.org/docs/dev/whatsnew/v1.2.0.html#increased-minimum-versions-for-dependencies) as an optional dependency is 0.15.1.  Pyarrow added the compute module that is imported in the change here with pyarrow [0.16.0](https://github.com/apache/arrow/commit/27dded680e84f1a628de1bddff1f4eb62fbc5887#diff-3d08757408024228c4443730cc3536ab39c9436cd2e4cb63e5da34c69c18962f) but attributes imported by pandas are not available in that module until [1.0.0](https://github.com/apache/arrow/commit/dcd17bf36e0f7b18e8b8f466ed2cb3eb396955d8#diff-3d08757408024228c4443730cc3536ab39c9436cd2e4cb63e5da34c69c18962f) thus anyone using pyarrow [0.16.0, 1.0.0) will get an Attribute error.

This PR adds as try/expect around accessing the attributes such that they are only accessed if available (i.e. pyarrow >=1.0.0)

- [X] closes #38801
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Strings
description: String extension data type and string data
name: Dependencies
description: Required and optional dependencies
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @ADraginda for the report and the PR.

to having testing for this, and prevent future issues, we should pin one of the ci environments to the >=0.16.0 and <1.0.0 range.

indeed, creation of the ARROW_CMP_FUNCS was moved to the module level, https://github.com/pandas-dev/pandas/pull/35259#discussion_r522015282, but was initially guarded with a version check (in object creation).

the try/except is OK, or we could duplicate the version check or refactor the version check out to avoid duplication.  
@ADraginda Thanks for the PR! Your analysis sounds correct, and this seems a good fix for it. 

Now, I am also curious *how* you ran into it, because in principle we should only import this file (at least in the current version of pandas) if you are explicitly importing it, since it's not yet being used internally or publicly exposed. 

(but your fix is useful anyway, because once we will start exposing it publicly, it shouldn't error like that when having pyarrow 0.16 installed)
> Now, I am also curious _how_ you ran into it, because in principle we should only import this file (at least in the current version of pandas) if you are explicitly importing it

we have special casing in pandas/core/arrays/base.py that imports ArrowStringDtype
Ah, indeed, I forgot about that one (so runtime, I was only thinking about on import)
> to having testing for this, and prevent future issues, we should pin one of the ci environments to the >=0.16.0 and <1.0.0 range.

The `actions-37-locale.yaml` now has `>=0.17` (which results in 1.0.1 currently, but that version is pinned elsewhere), so maybe can change this to a pin to 0.17 (`=0.17`):

https://github.com/pandas-dev/pandas/blob/94810d121d6b547262a7659939aeff6b21016df7/ci/deps/actions-37-locale.yaml#L33
@jorisvandenbossche Thanks for the review! I added the pin from `>=` to `==`

@simonjayhawkins I'd prefer to punt on a refactor in favor of passing off to someone who knows this code better if that's ok.

@jreback  I'm hoping to avoid a 1.0.0 minimum version if at all possible. 
You can reproduce with pyarrow 0.17.1 (that's what I'm on, but breaks for other versions too) with the following example. Seems to be related to merging tow tables, one on a column of type `string` and the other of type `object`.

```python
import pandas as pd

example_dict = {'i': {0: 'foo'}, 'j': {0: 1}}

foo = pd.DataFrame(example_dict)
foo['i'] = foo['i'].astype(pd.StringDtype.name)
foo['j'] = foo['j'].astype(pd.Int64Dtype.name)

bar = pd.DataFrame(example_dict)
bar['j'] = bar['j'].astype(pd.Int64Dtype.name)

foo.merge(bar)
```
Hmm, apparently my suggestion of pinning to 0.17 in that environment was not as simple as that .. as it gives an unsolvable environment. Will do a check locally of what might work.
@jorisvandenbossche

```
=========================== short test summary info ============================
FAILED pandas/tests/io/test_user_agent.py::test_server_and_default_headers[ParquetPyArrowUserAgentResponder-read_parquet-34268-pyarrow]
FAILED pandas/tests/io/test_user_agent.py::test_server_and_custom_headers[JSONUserAgentResponder-read_json-34264-None]
FAILED pandas/tests/io/test_user_agent.py::test_server_and_custom_headers[ParquetPyArrowUserAgentResponder-read_parquet-34270-pyarrow]
FAILED pandas/tests/io/test_user_agent.py::test_server_and_custom_headers[GzippedJSONUserAgentResponder-read_json-34266-None]
= 4 failed, 136679 passed, 14311 skipped, 1054 xfailed, 63 xpassed in 1148.83s (0:19:08) =
```
Hmm the build we changed gives a different failure

```
FAILED pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_filter_row_groups
```
I think that test can be skipped for pyarrow 0.17, it's complaining about file-like object not yet being implemented.
I think the other failures on Azure are unrelated (it's not impacted by this PR), probably a flaky test that can be addressed elsewhere. Let's restart the build
> @jorisvandenbossche Thanks for the review! I added the pin from `>=` to `==`
> 
> @simonjayhawkins I'd prefer to punt on a refactor in favor of passing off to someone who knows this code better if that's ok.
> 
> @jreback I'm hoping to avoid a 1.0.0 minimum version if at all possible.
> You can reproduce with pyarrow 0.17.1 (that's what I'm on, but breaks for other versions too) with the following example. Seems to be related to merging tow tables, one on a column of type `string` and the other of type `object`.
> 
> ```python
> import pandas as pd
> 
> example_dict = {'i': {0: 'foo'}, 'j': {0: 1}}
> 
> foo = pd.DataFrame(example_dict)
> foo['i'] = foo['i'].astype(pd.StringDtype.name)
> foo['j'] = foo['j'].astype(pd.Int64Dtype.name)
> 
> bar = pd.DataFrame(example_dict)
> bar['j'] = bar['j'].astype(pd.Int64Dtype.name)
> 
> foo.merge(bar)
> ```

ok pls add this as a test. 

I don't know why anyone is trying to use StringArray before 1.0 but we need a test for this in any event.

put in tests/reshape/merge somewhere
@jreback good point. The root of it all is actually just this call:

```python
from pandas.core.arrays.string_arrow import ArrowStringDtype
```

So a test for the merge fn feels a bit incorrect for a unit test to me. Moreover, this import is [already in the tests](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/arrays/string_/test_string_arrow.py#L6) so is covered I think? 

Tests: I fixed a failing test that had the wrong minimum version of pyarrow for what the test wanted to accomplish (test_filter_row_groups in test_parquet)
> So a test for the merge fn feels a bit incorrect for a unit test to me. Moreover, this import is already in the tests so is covered I think?

Yes, this is already covered by the StringArray astype tests. We only didn't catch it, because we didn't have any CI build with this specific version of pyarrow. So no need to add an additional test.
@meeseeksdev backport 1.2.x
> can you put in bug fixes I/O section. ping on green.

for recent patch releases we have tended not to split into sub-sections, just regressions, bug-fixes and maybe other.

(yep, since there were already others, I thought we have to clean it up anyway, so not worth doing another round of changes here)