BENCH: Add decorator to parameterize benchmarks for copy on write
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is a POC for benchmarking copy on write. I wanted to get more feedback before moving forward

The idea here was to make a decorator to detect methods starting with ``time`` or ``peakmem`` and modify those methods to run with copy on write enabled or disabled.

I chose this method since it should be minimally invasive in terms of code changes - otherwise we would need to wrap every single method in a context manager by hand, and add the copy on write parameters to every single class by hand.

This would also cause problems when Copy on Write becomes the default, as we would have to do a bunch of changes to remove the copy on write option from the benchmarks at that time.

With this approach (despite its hackiness), all you have to do is put on or remove the decorator, which is much less error-prone.

Initial testing seems to show this working correctly, 
``rename`` is much faster under copy on write, which makes sense as it one of the methods where copy on write would eliminate a copy.

Labels:
name: Benchmark
description: Performance (ASV) benchmarks
name: Copy / view semantics

Comments:
> Why do we need this? Is there a reason that we can not just enable CoW globally if we care about CoW timings?

The goal is to compare copy on write perf to non copy on write.
You can simply do a commit on top of current main that enables CoW and then test against main.

You can also work with environment variables.

I am not generally opposed to adding a decorator, but I wouldn't spend more than 30 minutes on this and definitely wouldn't add complex code for it. It doesn't seem like a good time investment when the alternative is so easy and only relevant for about 2 months.

> You can simply do a commit on top of current main that enables CoW and then test against main.
> 
> You can also work with environment variables.
> 
> I am not generally opposed to adding a decorator, but I wouldn't spend more than 30 minutes on this and definitely wouldn't add complex code for it. It doesn't seem like a good time investment when the alternative is so easy and only relevant for about 2 months.

How would you tell ASV to differentiate between the copy on write results vs the non copy on write results?

The commit that enables Copy on Write works, but you also wouldn't be able to keep track of performance over time.

At any rate, if people don't care about this, happen to close.
Ah, you would want to enable this over time?

I'd rather avoid the additional strain on our benchmarks, but I am more open here
@rhshadrach

Do you know if your machine can handle running the full ASVs (or just a subset, e.g. frame_methods) for Copy on Write too?
Agree with @phofl - my recommendation would be to ensure with vs without looks good in one commit, and then switch everything over to using CoW. This keeps things quite simple.

I'm not -1 on adding additional benchmarks - but would like to know the impact on time here. The benchmark machine can "handle it", but longer run times means we will benchmark fewer commits, and add complexity to analyzing the benchmark data (at least, for a short window).