REGR: preserve reindexed array object (instead of creating new array) for concat with all-NA array
State: closed

Originally reported in GeoPandas: https://github.com/geopandas/geopandas/issues/2493

This worked in pandas 1.3.5, was then originally "broken" by https://github.com/pandas-dev/pandas/pull/43043 (as an unintended side effect of some of the (proper) changes in that PR) somewhere between 1.3 and 1.4.0 but got fixed again before the final 1.4.0 because of subsequent refactoring in the internal concat code. Some of those refactors were then reverted in 1.4.3 for being able to revert the all-NA change (https://github.com/pandas-dev/pandas/pull/47372), surfacing the "bug" again. 

This is not necessarily strictly a "bug" in pandas (I also don't know if we make any guarantees about preserving array objects, let alone its attributes), but I think this is a decent change nevertheless. 
Currently, when concatting/merging a column with all-NA data, we create the "empty" (all-NA) array for the result from scratch. While if there is no dtype change, and so the original column already has the correct dtype, we can actually just reindex that array instead of creating a new all-NA array. For builtin dtypes that might not matter much, but for custom ExtensionArrays that avoids going through some complex code to create this new all-NA array:

https://github.com/pandas-dev/pandas/blob/fc68a9a290fc314c090e037597138c74fa23ee6d/pandas/core/internals/concat.py#L479-L488

In GeoPandas, our ExtensionArray holds some optional attribute information (eg the coordinate reference system, a spatial index, ..) that is not part of the dtype. And so in that case, reindexing the original array lets geopandas preserve that information (if appropriate), while if creating an empty array from scratch with the code above, this information is definitely lost.

Apart from the two-line actual code change, this PR adds a new, minimal test ExtensionArray that exhibits this behaviour of having an attribute, to be able to test this without dependency on GeoPandas.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@jorisvandenbossche the added test passes without the code change?
Ah, I get that indeed as well. It was certainly failing, so it seems that in the meantime, there was another change in main that affected this. It seems that because of removing the usage of internal `concatenate_managers` in the merge code (https://github.com/pandas-dev/pandas/pull/48082), the code path changed a bit, and the `merge` test case actually doesn't get to this point in the code anymore (just reverting that patch, the test fails again without the changes in this PR). 
But so when directly calling `concat` in a certain way still triggers it. Updated the test to include that, and now the test fails on main as well (and I suspect that the merge example would also still fail on 1.4.x).
> This is not necessarily strictly a "bug" in pandas (I also don't know if we make any guarantees about preserving array objects, let alone its attributes), but I think this is a decent change nevertheless.

makes sense.



> Ah, I get that indeed as well. It was certainly failing, so it seems that in the meantime, there was another change in main that affected this. It seems that because of removing the usage of internal `concatenate_managers` in the merge code (#48082), the code path changed a bit, and the `merge` test case actually doesn't get to this point in the code anymore (just reverting that patch, the test fails again without the changes in this PR).
> But so when directly calling `concat` in a certain way still triggers it. Updated the test to include that, and now the test fails on main as well (and I suspect that the merge example would also still fail on 1.4.x).

indeed.

the two merge tests pass in 1.3.5, fail in 1.4.3 and then pass again on main.

the added concat test passes in 1.3.5 and fails in 1.4.3 and on main.
@simonjayhawkins thanks for the review!
@meeseeksdev backport 1.5.x
Thanks @jorisvandenbossche