API: New copy / view semantics using Copy-on-Write
State: closed

This is a port of the proof of concept using the ArrayManager in https://github.com/pandas-dev/pandas/pull/41878 to the default BlockManager.

This PR is a start to implement the proposal described in more detail in https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2thyTb0/edit / discussed in https://github.com/pandas-dev/pandas/issues/36195 
A very brief summary of the behaviour you get:

- *Any* subset (so also a slice, single column access, etc) _behaves_ as a copy (using CoW, or is already a copy)
- DataFrame methods that return a new DataFrame return shallow copies (using CoW) if applicable (for now, this is only implemented / tested for `reset_index` and `rename`, needs to be expanded to other methods)

**Implementation approach**
This PR adds Copy-on-Write (CoW) functionality to the DataFrame/Series at the BlockManager level. It does this by adding a new `.refs` attribute to the `BlockManager` that, if populated, keeps a list of `weakref` references to the blocks it shares data with (so for the BlockManager, this reference tracking is done per block, so `len(mgr.blocks) == len(mgr.refs)`). 
This ensures that if we are modifying a block of a child manager, we can check if it is referencing (viewing) another block, and if needed do a copy on write. And also if we are modifying a block of a parent manager, we can check if that block is being referenced by another manager and if needed do a copy on write in this parent frame. (of course, a manager can both be parent and child at the same time, so those two checks always happen both)

---

**How to enable this new behaviour?** 
Currently this PR simply enabled the new behaviour with CoW, but of course that will need to be turned off before merging (which also means that some of the changes will need to put behind a feature flag. I only did that now in some places). 

I think that ideally, (on the short term) users have a way to enable the future behaviour (eg using an option), but _also_ have a way to enable additional warnings. 
I already started adding an option, currently the boolean flag `options.mode.copy_on_write=True|False`:

* Do we have a better name? I personally don't like that it uses "copy_on_write", because this is the internal implementation detail, and not what most end users really have to care about. But something like "new_copy_view_behaviour" is also not super ..
* In addition to True/False, we can probably add "warn" as a third option, which gives warnings in cases where behaviour would change.

---

Some notes:

- Not everything is already implemented (there are a couple of `TODO(CoW)` in the code), although the majority for indexing / setitem is done.
- This PR does not yet try to tackle copy/view behaviour for the constructors, or for numpy array access (`.values`). Given the size of this PR already, those can probably be done in separate PRs?
- Most tests are already passing (with changes), but still need to fix a few tests outside of /indexing
- We will also need to think about a way to test this (in a similar way as the ArrayManager with an environment variable?)


I will also pull out some of the changes in separate PRs (eg the new test file could already be discussed/reviewed separately (-> https://github.com/pandas-dev/pandas/pull/46979), and the `column_setitem` is maybe also something that could be done as pre-cursor(-> https://github.com/pandas-dev/pandas/pull/47074))

Labels:
name: Copy / view semantics

Comments:
@jreback I deleted your comment (included your phone number) - looks like you might have replied to the wrong email
thanks!
@jorisvandenbossche im questioning the accuracy of the "nothing is 'just a view'" heuristic.  Do any of the following cases break that:

- df.T  # single-block case
- df.astype(df.dtypes, copy=False)
- df.iloc[:]
- df.stack()/unstack()
- df.copy(deep=False)
> im questioning the accuracy of the "nothing is 'just a view'" heuristic

Do you mean in the current state of the PR? Or that those cases in general should/would break that in your opinion? 

The goal is that none of those examples would break that general rule. But at the moment, this PR only added the CoW mechanism to indexing and explicitly to a few methods (`copy`, `reset_index`, `rename`; and in addition methods that are based on (re)indexing operations under the hood will also already follow it), so others won't yet follow the intended semantics. 
But for example `copy` is already implemented, so `df.copy(deep=False)` will no longer return a "real" view (it returns a new dataframe with the same data in practice, but will stop being a view upon mutation through CoW).

`stack`/`unstack` already seem to return a copy on master / 1.4. 

`df.iloc[:]` is already implemented and thus follows the outlined rules (returning a view in practice, but guarded with CoW).

The other ones (astype, transpose) are good cases I still need to test and/or implement.
> Do you mean in the current state of the PR? Or that those cases in general should/would break that in your opinion?

Just in general, wanted to make sure that the question was written down in one of the relevant issues/PRs.  Thanks for taking a look.

Note for reviewers: there are currently a few tests failing because of the recent changes in main related to the in place setting with iloc deprecation. I need to decide whether to follow those deprecations, or already implement the future behavior in case of CoW, and update the tests accordingly. 

But apart from that (and that I need to merge master), the diff itself is certainly in a reviewable state.
@mroeschke would you have some time to further review this?
Thanks for the review!

> Main question is whether we can just have `self.refs: list[weakref | None]` instead of `self.refs: None | list[weakref | None`

See https://github.com/pandas-dev/pandas/pull/46958#discussion_r923811480 for my inline answer

> IMO I'm fine with `copy_on_write` as the configuration name with `True/False` While warn would be nice to implement, I am a little weary of maybe having another `SettingWithCopyWarning` situation.

The possible "warn" option (to implement in a separate PR if we do this) is something that I think we will want at some point, because it are basically the deprecation warnings we need at some point before we can switch behaviour. 
The reason for making it also part of the options, is that we might already implement those warnings at some point but not yet show them by default. 
One remaining question is how to test this (as I need to change the option default to False) for the time being. I can add one build that tests this? (similarly as there is one ArrayManager build)
> One remaining question is how to test this (as I need to change the option default to False) for the time being. I can add one build that tests this? (similarly as there is one ArrayManager build)

Sure a build can be added to the `include` matrix in the `.github/ubuntu.yml` GHA. Probably better than an autouse `pytest.fixture` with `copy_on_write=True/False`
There is one new test from a few days ago that still needs to be updated for CoW (the one failing build), will do that tomorrow.

> Were you planning on some docs + whatsnew in a separate PR?

Yes, would prefer to do that in a separate PR.
> Would you still like to include this for 1.5?

Yes, I would prefer that
Thanks @jorisvandenbossche! Maybe could reference or document the benchmarks in the follow-up documentation PR
way to go!