CoW: Push reference tracking down to the block level
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This pr removes the reference tracking from the manager level and adds it to the block level. Tests are passing locally.
Right now we are tracking the references also in non-CoW mode, but we are only accessing the object in CoW mode, so should not be harmful. Can also disable this, but wanted to avoid accessing ``using_copy_on_write()`` constantly.

Two things that the previous mechanism couldn't handle:

```
df = DataFrame({"a": [1, 2, 3]})
df = df.reset_index()
df.iloc[0, 1]
```

This triggered a copy, because the reference to the original df was kept alive. Same example: We were not able to free the memory of old object, if the result of an operation was assigned to the same variable.


Labels:
name: Copy / view semantics

Comments:
Some additional test ideas:

```
def test_out_of_scope():
    def func():
        df = DataFrame(..)
        # create some subset
        result = df[[cols]]
        return result
    df = func()
    # assert that df has no refs
```

```
def test_delete():
    df = DataFrame(np.random.randn(4, 3), columns=['a', 'b', 'c'])
    del df['b']
    # at this point, 'a' and 'c' each should only have 1 reference (in addition to itself)
    df = df[['a']]
    # 'a' should should be owned, and 'c' should have gone out of references (not sure if there is an easy way to test that)
```
Agree with parametrising the tests, this was one my todo list as well. Would prefer to do in a separate pr though.
Added the tests of your two cases.

The del right now sets up references so that we think that there is another reference alive, because we are creating shallow copies under the hood. 

Is there a way that delete would actually require us to keep references alive? Otherwise we could fix this in idelete on the manager level
Quick comments to your top-comment:

- I ensured that every function that was called behind a ref-update on the Manager level updates the refs correctly now. I went through all possible code paths down to the block level. The rest should be hidden behind general copies for CoW which we optimise later.
- Same for the 2nd point, I think we make general copies there without checking anything
- will do
- yes definitely. Most of the checks are more specific on the block level and easier as well I guess.


Yeah I mentioned the tracking In my top post as well. Would make things unnecessarily complex (and slower) imo if we only do this for CoW. Accessing it is what could cause a slowdown, which is not done without CoW enabled
Merging now to unblock other stuff, can still optimise later if necessary