BUG: Metadata propagation for groupby iterator
State: closed

Addresses part of #28283

This PR ensures that `__finalize__` is called for objects returned by iterator over `.groupby`-results.
Based on the discussion and benchmarks in PR #35688 (see https://github.com/pandas-dev/pandas/pull/35688#issuecomment-675367517), `__finalize__` is _not_ called for intermediate objects used by `.apply` (and maybe other code paths).

The implemented solution is the safe choice with regard to performance: An alternative solution would be to call `__finalize__` immediately after initialization (in `DataSplitter._chop()`).
The downside to doing so would be the performance hit due to the overhead of finalizing intermediate objects in `.apply` (and maybe other code paths).  In the benchmarks referenced above, this amounted to around 4%.
The upside of finalizing immediately after initialization would be reduced complexity and that it would allow `.apply` (and any other code paths using `DataSplitter._chop` to access metadata)..

- [x] closes #37343 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: metadata
description: _metadata, .attrs

Comments:
@jorisvandenbossche You asked about this PR. Do you have any position on the performance question? Would you be able to review?

I will have a look at the failing tests.
Thanks @Japanuspus for the PR.

> Addresses part of #28283

This PR may also fix #37343, see https://github.com/pandas-dev/pandas/issues/37343#issuecomment-718864907

If you could add a test for that issue, that'll be great. Also if you could keep the changes in this PR minimal as we would likely want to backport this.
Thanks @simonjayhawkins! 

> This PR may also fix #37343, see [#37343 (comment)](https://github.com/pandas-dev/pandas/issues/37343#issuecomment-718864907)
> 
> If you could add a test for that issue, that'll be great. Also if you could keep the changes in this PR minimal as we would likely want to backport this.

Yes, this was exactly the issue that I saw in my own code as well after the 1.1 update. I will add a test. It appears I need to merge 2414c75bfe06064a95d8da4db5c8248e5aa0cdbb to get passing tests: will rebase and force push to keep things clean for backport.

Still seeing unrelated test failures. Will try waiting for things to settle down once 1.1.4 is released.
Does something similar need to be done for the cython path?

> Does something similar need to be done for the cython path?

I believe not: The functional change is in `BaseGrouper.get_iterator` which does not appear to have a cython alternative path.

@jreback are you OK with backporting this?
> @jreback are you OK with backporting this?

actually this is fine to backport. 

@Japanuspus can you update the whatsnew note to put in 1.1.5
> lgtm if you can move the note and merge master. ping on green-ish.

32 bit failure unrelated. (caused by recent merge to master of #36842)
thanks @Japanuspus 
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 cc9c646463d4a93abdc7c61bbb47e7d2ccf2fc4b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #37461: BUG: Metadata propagation for groupby iterator'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-37461-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #37461 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).