BUG: Fix copy semantics in ``__array__``
State: closed

This should fix the semantics of ``__array__``.  While rejecting ``copy=False`` is OK even if unnecessary, ``copy=True`` should never have been ignored and is dangerous.

Closes gh-57739, closes gh-59932, closes https://github.com/pandas-dev/pandas/issues/59614

- [x] Still needs new tests.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

----

I have to figure out the tests.  I think a test in form of:
```
arr1 = np.asarray(obj, copy=True)
arr2 = np.asarray(obj, copy=True)
assert not np.may_share_memory(arr1, arr2)
# Check that without copy always works:
assert_array_equal(arr1, np.asarray(obj))

if np_ver < 2:
    return  # copy=False semantics not supported

try:
   arr1 = np.asarray(obj, copy=False)
except ValueError:
   return  # An error is acceptable for `copy=False`

# If no error is given, multiple returns must be views:
arr2 = np.asarray(obj, copy=False)
assert np.may_share_memory(arr1, arr2)
```
will work nicely.  But, right now I am not sure if there is a convenient pattern/parametrization to steal to cover all of the `__array__` implementations here.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Thanks for addressing this! Could this be released as a bug fix for v2.x?
FWIW, if anyone knows how to add decent tests for this, pushing to the PR is appreciated.  Otherwise, I may try to figure out where to add at least tests for a few of these paths.
Hi, Glad to see a PR addressing #59932 ! I was trying to reproduce that issue but couldn't install NumPy >= 2.0 with the main branch, which seems to be because the `environment.yml` on the current main branch requires  `numpy<2`. How did you resolve it?

And by the way, I asked about the bug fix release on Slack:

> Could this be released as a bug fix for v2.x?

Quoting the reply from @rhshadrach:

> I think there won't be another 2.2.x release; the next release will be 2.3.0.
You can just `pip install -U numpy` afterwards. Or edit the file.
Found a test that seemed reasonably to expand.  I doubt it covers _all_ the branches changed, though (and no, I don't love expanding tests, but I was lazy to split out the parametrization).
Ping @mroeschke just because it's been a while that it was idle without any tests.  Not sure if this needs a milestoned or a  release note.
Sorry for the delay. Overall looks good! Could you add a note to the `v2.3.0.rst` release notes?
Pushed fixes based on the review, thanks; it's quite tricky to know whether or not a copy may have been made.

Which also means that expandign the tests would be great (with the new changes also with `dtype=` which I am not sure how well it works in the current test).

@jorisvandenbossche would be great if you can look into tests (I can review), since I have to hunt/think a bit more about how to build examples.  But if I don't see movement in the next days, I'll hopefully dive in.
@seberg thanks for the update! I added a bunch of additional tests, that I think should now cover the corner cases for which you updated the code (i.e. PeriodArray and SparseArray corner cases that allow copy=False, MultiIndex never allowing copy=False). 
Also expanded the existing test a bit and added a basic base extension array test.
Thanks Joris.  With the last two commits, I think this should be good to go in.
The whatsnew note can probably be improved, but going to merge this already so I can start on backporting the code to 2.3.x.

Thanks @seberg!
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 eacf0326efb709169ebc49f040834670dfe4beb3
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #60046: BUG: Fix copy semantics in ``__array__``'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-60046-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #60046 on branch 2.3.x (BUG: Fix copy semantics in ``__array__``)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/60189
@seberg I am seeing some failures in the geopandas tests using pandas dev because of this change (https://github.com/geopandas/geopandas/actions/runs/11688928152/job/32550520328?pr=3455)

Essentially geopandas is using `np.array(pandas_object, copy=False)` in a few places. And in the past this has always worked, just silently returning a copy (I assume we just passed `copy=False` in the idea of avoiding a copy if not needed, it's not that the code relied on it returning a view). 
I think for numpy itself you made this behaviour change of `np.array` (to start raising an error) on 2.0? Did you warn about it in advance? (I think I remember updating such a few cases as part of "make the package compatible with numpy 2.0")

I am wondering if we should leave the `copy=False`-raising-error part of the changes for pandas 3.0, instead of including that in pandas 2.3.

The `copy=True` part of the change (actually copying when asked) is I think the main bug fix we definitely should include in 2.3, as far as I understand (I think the issues people have been reporting in https://github.com/pandas-dev/pandas/issues/57739 would be resolved by this part)


No, there was no warning about it, it was a hard change in 2.0...

You'll be the better judge for pandas users here.  It is wrong to not raise, but it does seem worse to backport `copy=False` for sure, since `copy=False` will be exceedingly rare anyway in code.
If this was right from the start geopandas would have had to fix this as part of NumPy 2 transition, but that ship has sailed.

So I guess, you may only want to backport the `copy=True` fix, since it seems important, but not touch `copy=False` even if it is a bug.
If the next release is 3.0, I guess that is good enough to just do the change without deprecation then.
> I am seeing some failures in the geopandas tests using pandas dev because of this change

@jorisvandenbossche now that this is merged and backported should we have a new issue for this and linked from the 2.3 release discussion for greater visibility?

> I am wondering if we should leave the copy=False-raising-error part of the changes for pandas 3.0, instead of including that in pandas 2.3.

Seems reasonable to me that we probably don't want to introduce breaking changes in 2.3
Yes, opened https://github.com/pandas-dev/pandas/issues/60340 and labeled it for 2.3.0