BUG: Fix fillna on multi indexed Dataframe doesn't work
State: closed

- [ ] closes #47649 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex
name: DataFrame
description: DataFrame data structure

Comments:
I also looked into this one and found that the issue is that somewhere downstream we explicitly check if the column names match exactly, but since we just do:
https://github.com/pandas-dev/pandas/blob/main/pandas/core/generic.py#L6677
we only hand the columns `['a', 'b']` to `loc` and not the correct multiindex.

So if we change the line refered to above to:
```python
                    if result[k].ndim > 1:
                        res = result[[k]].fillna(
                            v, limit=limit, downcast=downcast_k
                        )
                    else:
                        res = result[k].fillna(
                            v, limit=limit, downcast=downcast_k
                        )
                    result.loc[:, k] = res
```
the tests work. Feel free to continue from here if you think that's a suitable way.
The list indexer makes a copy and hence would avoid operating inplace I think, also we have this issue in multiple places, we have to fix this in loc itself, but not sure how to achieve this right now 
> The list indexer makes a copy and hence would avoid operating inplace I think, also we have this issue in multiple places, we have to fix this in loc itself, but not sure how to achieve this right now

Just checked, this solution works inplace. 

Let me break down as I understand the problem. Suppose we do:
```python
pdf = pd.DataFrame(
    {
        ("x", "a"): [np.nan, 2, 3, 4, np.nan, 6],
        ("x", "b"): [1, 2, np.nan, 4, np.nan, np.nan],
        ("y", "c"): [1, 2, 3, 4, np.nan, np.nan],
    },
    index=np.random.rand(6),
)

>>> pdf.fillna({"x": -1})
```

then at https://github.com/pandas-dev/pandas/blob/a7c5773d19819b750c719344143aa5b96102e124/pandas/core/generic.py#L6677
we will hand over 
```python
[In]: df
[Out]:
            a    b
0.666291 -1.0  1.0
0.874578  2.0  2.0
0.845121  3.0 -1.0
0.224814  4.0  4.0
0.442914 -1.0 -1.0
0.224514  6.0 -1.0
```
to `loc[:, k]` which then gets further downstreamed up until
https://github.com/pandas-dev/pandas/blob/a7c5773d19819b750c719344143aa5b96102e124/pandas/core/indexing.py#L1938-L1946
And here in line 1940 things get off the rails. Because now we explicitly check if `('x', 'a')` is in `df` which of course is not the case since we eliminated the multiindex by the we indexed. Fixing the indexing in case of dataframes fixes this issue.



The main issue is the following:

df.loc[a, :] = df.loc[a, :] * 2

does not work anymore if the MultiIndex is reduced. Switching to dataframe indexers is not a general solution, hence not desirable imo.

Did you check if a view on the original dataframe gets updated too? Like

view = df[:]
df…


This should update the view too
> The main issue is the following:
> 
> df.loc[a, :] = df.loc[a, :] * 2
> 
> does not work anymore if the MultiIndex is reduced. Switching to dataframe indexers is not a general solution, hence not desirable imo.
> 
> Did you check if a view on the original dataframe gets updated too? Like
> 
> view = df[:] df…
> 
> This should update the view too

Yes, it changes a view, too. But I see your point. Just as @xr-chen already tried, every change in `_setitem_with_indexer_frame_value` would lead to other undesired changes in behaviour. So, that's a tough one
https://github.com/pandas-dev/pandas/blob/a7c5773d19819b750c719344143aa5b96102e124/pandas/core/indexing.py#L1825-L1829
I found that if we call `_setitem_with_indexer_2d_value` instead of `_setitem_with_indexer_frame_value` when setting values of a multi-indexed Dataframe, `df.loc[a, :] = df.loc[a, :] * 2` will work, #47649 will be solved, and all tests in `\pandas\tests\indexing` will pass. Is that a suitable solution?
I think this would also change the behavior in general, similar to what you’ve tried initially
What about changing `df.loc` in such a fashion that it doesn't reduce the multiindex? That's also a change in behaviour but is it one we can't do?
https://github.com/pandas-dev/pandas/blob/3a39d2592d5f9090b3424f5d2f8f7ab453482e6c/pandas/core/generic.py#L6677-L6679
Could we do 
``result.loc[:, k] = result[k].fillna(
                        v, limit=limit, downcast=downcast_k
                    ).values``
in `fillna`? Since in this specific senario, we don't care `result`'s index. And we add a conveat in documentation, to change value of a multi-indexed dataframe under level `a`, we should use `df.loc[a, :] = df.loc[[a], :] * 2`  rather than `df.loc[a, :] = df.loc[a, :] * 2` which would bring unexpected `NAN` to original DataFrame.
Any suggestions on this regression fix? @phofl @simonjayhawkins 
sorry @xr-chen for the slow response. milestoning 1.4.4 to keep it on my radar and will look soon
I would be ok with the values changes for now, if you can add a TODO and link to the issue that was opened about the multiplication case. We should fix this in loc in the future.
Like TODO: we should fix GH47649 by fixing the issue in loc mentioned in #45751 in the future?
Something like:

Revert when issue number… is fixed
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 ae291e9b1543779ac8d4416e8160bef260d0dce5
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am "Backport PR #47774: BUG: Fix fillna on multi indexed Dataframe doesn't work"
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47774-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47774 on branch 1.4.x (BUG: Fix fillna on multi indexed Dataframe doesn't work)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).