PDEP-7: Consistent copy/view semantics in pandas with Copy-on-Write
State: closed

This PR adds the [google doc](https://docs.google.com/document/d/1ZCQ9mx3LBMy-nhwRl33_jgcvWo9IWdEfxDNQ2thyTb0/edit#) about the Copy-on-Write proposal that came out of the discussion in https://github.com/pandas-dev/pandas/issues/36195 as a PDEP. 
It predates the PDEPs, but bascially back then I wrote what would now be a PDEP, and since this is still actively being developed, I think it would be useful to have an "official" reference to the proposal, instead of pointing to my google doc. 

The first commit is just a basic conversion of the google doc to the markdown. I will do a first follow-up commit updating some outdated parts (eg referencing the POC implementation). 

I am fine with still getting review of the text / details here, or we can also take it as it is. 

And then the question is also how to mark the status of the PDEP. Can we assume it is already accepted / under implementation? (it went through the somewhat vague pre-PDEP decision process of some kind of consensus / nobody loudly objecting anymore, and it's now available and documented as opt-in). But am also happy to let it pass through the PDEP process of approving this officially.

cc @pandas-dev/pandas-core @pandas-dev/pandas-triage 


Labels:
name: Copy / view semantics
name: PDEP
description: pandas enhancement proposal

Comments:
> but perhaps the `SettingWithCopy` warning could be updated to also ask users to consider enabling copy-on-write?

That's a good idea to do at the point when we are more confident to point general users towards it. In general we will have to add warnings for behaviours that will change (eg for chained assignment), and so that might already replace some of the current SettingWithCopyWarnings as well. 
In the current text, I mention adding warnings in general in the "Backward compatibility" section. I think that should be sufficient, or I can add this specific addition as well. In any case I tracked the idea in the CoW overview tracker: https://github.com/pandas-dev/pandas/issues/48998
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Updated a couple of sections to be more aligned with the current implementation, also mentioning things that we intend to do (warnings mode, upgrade, ...)
I changed the status to Under discussion to see if the docs will render properly.
@pandas-dev/pandas-core are there any more comments on the text of the proposal? 
This is meant as a final request for comments, before calling for a vote on this PDEP.
You won’t need a defensive copy anymore, ever. That’s one of the advantages
> You won’t need a defensive copy anymore, ever. That’s one of the advantages

Yes, but there is a lot of code out there that does have defensive copying going on (including things my staff has written), so the question is whether we help people identify it.  For example, we could raise a warning whenever `.copy()` is called.  Maybe we add a parameter to `DataFrame.copy()` such as `DataFrame.copy(intended: bool = False)` where we raise a warning if `intended==False`, which would help people pick up all the defensive copying in their code.  If they really want a copy, they then have to use `intended=True`

> so the question is whether we help people identify it

If the alternative is adding a fluffy keyword, then id go for not helping people identify it, just document it.

@jorisvandenbossche can you confirm that the TL;DR here is pretty much "make official that we will enable CoW for all users"?
> can you confirm that the TL;DR here is pretty much "make official that we will enable CoW for all users"?

That's correct, essentially what we already implemented under the optional `pd.options.mode.copy_on_write = True` becomes the default behaviour (and only option) in 3.0
Thanks Irv for the textual comments, will update for those. 

> > You won’t need a defensive copy anymore, ever. That’s one of the advantages
> 
> Yes, but there is a lot of code out there that does have defensive copying going on (including things my staff has written), so the question is whether we help people identify it.

I am also not fond of adding another keyword for this, but in theory we _could_ add some option to trigger a warning for this. For example, it should be quite easy to check within the `copy` method if the calling object fully owns its data and is called from a method chain (both kinds of checks we already do elsewhere), and in that case raise an (optional) warning. 
That should catch a typical example of `df_filtered = df[df["A"] > 1].copy()`. 

I don't know if it is worth adding a warning for this, but at least technically it shouldn't be hard. The main question would be how to let users enable it (I wouldn't do with a keyword in `copy()`, because then you need to modify your `copy()` calls, and in that case you could just as well read the code and remove the copy() call in case like the above?)
Could we move this discussion to a separate issue? I don't think that this should be part of the PDEP since it's not really relevant for CoW generally speaking.
/preview
No preview found for PR #51463. Did the docs build complete?
Just FYI: The preview thing is broken
Yeah, I emailed Marc about this and he said he refreshed the token that expired.

Looks like something else is still going wrong :(
I had a look on the server, and it seems like the docs artifact from PRs aren't getting stored on the preview server.

cc @datapythonista for more info.
Yes, sorry, I started to refresh the expired token but there were some problems and I couldn't finish until now. I see an unrelated error when uncompressing the artifact file in the server, but not sure if it's the specific file I was testing. I re-run the docs build here, when it finishes you can retry the preview again and hopefully it works, otherwise I'll have a look tomorrow. For other PRs, note that the preview is generated in the docs build, the `/preview` comment just posts the link, so previews won't be available for PRs when the docs build run while the server was broken (you can re-run the docs build to regenerate it as I did here).
/preview
No preview found for PR #51463. Did the docs build complete?
Started the vote at https://github.com/pandas-dev/pandas/issues/55511!
Looks like the vote passed in https://github.com/pandas-dev/pandas/issues/55511, I think we're good to set this as approved?