BUG: DataFrame.groupby(., dropna=True, axis=0) incorrectly throws ShapeError
State: closed

- [x] closes #35612
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This PR makes a few more changes to propagate `dropna` correctly for sliced groupby objects. 

It depends on #35444 for the changes to `pandas/core/groupby/generic.py`. I put them in by hand for now so that the tests pass but there should be no diff once #35444 is merged.

Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@arw2019 #35444 has been merged.
> @arw2019 #35444 has been merged.

thanks @rhshadrach! 

@jreback this is ready for review
@jreback are you ok with this approach?

I'm down to 6 failing tests (haven't figured these out but I expect they're due to `dropna` propagation/default issues)
```
FAILED pandas/tests/reshape/test_crosstab.py::TestCrosstab::test_crosstab_normalize
FAILED pandas/tests/reshape/test_crosstab.py::TestCrosstab::test_crosstab_with_numpy_size
FAILED pandas/tests/reshape/test_crosstab.py::TestCrosstab::test_crosstab_normalize_multiple_columns
FAILED pandas/tests/reshape/merge/test_join.py::TestJoin::test_right_outer_join
FAILED pandas/tests/reshape/merge/test_join.py::TestJoin::test_full_outer_join
FAILED pandas/tests/reshape/test_pivot.py::TestPivotTable::test_margins - Ass...
```
This is ready for re-review. Green modulo unrelated CI failures (Parquet/S3 roundtrips)
let's not bundle extra things in this PR

pls do a single thing in each PR

so do these separately 
> let's not bundle extra things in this PR
> 
> pls do a single thing in each PR
> 
> so do these separately

Ok, will do
Thanks for looking into it @arw2019. When using `axis=1`, you want to specify which columns to group, rather than using something like `df.groupby('B', dropna=True, axis=1)`. Here is an example:

```
df = pd.DataFrame({"A": [0, 0, 7, 1], "B": [1, 2, 4, 3]}, index=list('abcd'))
gb = df.groupby([1, np.nan], dropna=True, axis=1)
result = gb.transform('sum')
```

This errors out in the same way master does, in `_set_result_index_ordered`.  I think all that needs to be done is to do the same operation you're adding for `axis=0`, but on the columns instead:

    result.columns = self._selected_obj.columns[result.columns]

I haven't tested this thoroughly though, and am okay with doing in a followup.
/azp run
<samp>
Commenter does not have sufficient privileges for PR 35751 in repo pandas-dev/pandas<br>

</samp>
Ready for re-review.
To make this easier to review I've moved the changes to `BaseGrouper` into a separate PR (#36604)
ping @jreback 
@arw2019: The failure looks potentially related:

> FAILED pandas/tests/groupby/test_grouping.py::TestGrouping::test_evaluate_with_empty_groups[transform-expected0]
> @arw2019: The failure looks potentially related:
> 
> > FAILED pandas/tests/groupby/test_grouping.py::TestGrouping::test_evaluate_with_empty_groups[transform-expected0]

Yes it does. Looking...
> @arw2019: The failure looks potentially related:
> 
> > FAILED pandas/tests/groupby/test_grouping.py::TestGrouping::test_evaluate_with_empty_groups[transform-expected0]

This was a genuine bug - fixed now
Addressed comments + CI green modulo a persistent docs failure (finding that hard to diagnose atm, build log isn't helpful...)
@arw2019: See [here](https://pandas-docs.github.io/pandas-docs-travis/development/contributing.html#id49) for building the docs if you haven't done so locally yet.

I get the following when doing so:

Edit: forgot to compile; only warning I see is:

    pandas/doc/source/whatsnew/v1.2.0.rst:487: WARNING: Inline literal start-string without end-string.

which looks unrelated.
> @arw2019: See [here](https://pandas-docs.github.io/pandas-docs-travis/development/contributing.html#id49) for building the docs if you haven't done so locally yet.
> 
> I get the following when doing so:
> 
> Edit: forgot to compile; only warning I see is:
> 
> ```
> pandas/doc/source/whatsnew/v1.2.0.rst:487: WARNING: Inline literal start-string without end-string.
> ```
> 
> which looks unrelated.

Right that's also what I get and what comes up in the CI. I'll see if merging again will get to green
I "fixed" the docs build by removing the whatsnew note

@rhshadrach Would it be ok to merge the code here (modulo further review comments) and I can add the whatsnew note in a separate PR once it's in? No idea what this failure is about but it's easier to debug that separately
I was able to replicate: the issue with the note you had was that there was no space between the \`\` and (:issue:...). I'm pretty confident adding that space in will resolve the warning.
CI green, responded to comments above

Thanks @rhshadrach for the docs fix!!!
if you can merge master will look again
> if you can merge master will look again

Merged, will ping once CI is green
Merged master, CI green. Addressed comments from previous reviews
cc @jreback in case you have time to look
ok this looks good, can you merge master and ping on green.
Ping (merged master + CI green)
thanks @arw2019