BUG: fix Panel.fillna() ignoring axis parameter
State: closed

This fixes #8251. It may need some more comprehensive tests but before I go too much farther I want to make sure the axis numbering scheme makes sense . The default behavior of `fillna` is to fill along `axis=0`. In the case of a DataFrame this fills along the columns. However, in the case of a Panel, filling along `axis=0` means you're filling along items. I'm not sure if that's the behavior that most users would think is the default. You might note that I also removed the references to what the behavior of each axis value is in the docstring since it gets a bit arbitrary when you consider both DataFrames and Panels.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@0zeroth please take a look if you can

default filling should be along rows for frames (axis=0), this is the stat_axis
I think could be the same for Panel which has a stat_axis of 1

it just needs to be documented 
look through other functions and see if this is consistent

I've been trying to implement this with `Panel.apply` but there's an issue that perhaps I'm just not savvy enough to work around. It's simple enough to write the algorithm using `Panel.apply`  passing the chosen axis. However for a mixed-type Panel each constituent series get passed with a `dtype=object` even if the series it's passing are not of `dtype=object` to begin with. Hence, the `dtype` for all items in the panel becomes `object`.

```
In [50]:
from pandas.util.testing import makePanel
p = makePanel()
p.dtypes

Out[50]:
ItemA    float64
ItemB    float64
ItemC    float64
dtype: object


In [51]:
p['str'] = 'foo'
p.dtypes

Out[51]:
ItemA    float64
ItemB    float64
ItemC    float64
str       object
dtype: object


In [52]:
p.apply(lambda s: s).dtypes

Out[52]:
ItemA    object
ItemB    object
ItemC    object
str      object
dtype: object
```

Using `Panel.apply`, the resulting panels look ostensibly like they should, but their `dtypes` are wrong. Any ideas on how to work around this?

```
In [15]: p = tm.makePanel()

In [16]: p['str'] = 'foo'

In [17]: p.apply(lambda df: df.dtypes,axis=['items','major_axis'])
Out[17]: 
             A        B        C        D
ItemA  float64  float64  float64  float64
ItemB  float64  float64  float64  float64
ItemC  float64  float64  float64  float64
str     object   object   object   object
```

you need to specifiy multiple axes in order to get a frame.

Oh, I didn't mention my attempts with passing dataframes, which was actually my original idea. I played around with passing dual axes as you suggested, but the axes get transposed. That leaves you having to manually transpose the axes again, which is what we were trying to avoid.

```
In [35]: p
Out[35]:
<class 'pandas.core.panel.Panel'>
Dimensions: 3 (items) x 30 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemC
Major_axis axis: 2000-01-03 00:00:00 to 2000-02-11 00:00:00
Minor_axis axis: A to D

In [36]: p.apply(lambda df: df, axis=(0,1))
Out[36]:
<class 'pandas.core.panel.Panel'>
Dimensions: 4 (items) x 30 (major_axis) x 3 (minor_axis)
Items axis: A to D
Major_axis axis: 2000-01-03 00:00:00 to 2000-02-11 00:00:00
Minor_axis axis: ItemA to ItemC
```

However, now that I reconsider this, it guess it might be preferable to use `apply` and `transpose` rather than collect everything into a `dict` and rebuild the panel from that. 

I implemented the `Panel.apply` approach. Please take a look. If it looks OK I'll squash the commits.

Realized that this method was almost trivial to extend to Panel4D, so I implemented that (along with tests).

Unless I'm dopey, which is totally possible, I don't think your suggestion of using `range` and `remove` for determining the `apply_axes` is workable. In any event, as it turns out, for both Panel and Panel4D you want the `apply_axes=(0, 1)` for axis 0 and `apply_axes=(1, 2)` for any other chosen axis.

Also added some documentation

This works... mostly. Unfortunately it fails for categoricals since they don't have `apply` support yet. One option would be to hold off on this PR until they do support `apply` operations. The other options I suppose would be invoking one of the less elegant methods that I've proposed in earlier commits as they don't seem to suffer from the same problem.

I would also point out that I needed to invoke a `convert_objects` operation since `Panel.apply` converts everything to a dtype of `object` if anything in the frame is an `object` dtype. At first I thought this was a code smell, but I discovered `DataFrame.apply` uses the same technique. Perhaps an issue should be opened to address this point with `Panel.apply`.

I implemented a method for restoring categorical data. Not sure if it's robust enough to be widely applicable, but it does work for this use case and the categorical tests pass.

will have to look after 0.15.0 is released (end of the week).

If you want to take a look, I have time to work on this PR this week.

@stahlous can you rebase / revisit?

pls rebase/update. closing, but reopen if you want to work on.

Revisiting this after an extended hiatus. Do I need to submit another pull request, or can this one be re-opened?

no, you can just push to this one

I pushed to the `fillna_bug` branch of my fork, but it doesn't seem to have any effect here. I still only see the old commits in this PR.

Also, I'm having an odd problem with Travis. Anywhere that `LooseVersion()` is used I get an error. For example:

```
======================================================================
ERROR: test_deprecated_levels (pandas.tests.test_categorical.TestCategorical)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/stahlous/pandas/pandas/tests/test_categorical.py", line 1261, in test_deprecated_levels
    self.assertFalse(LooseVersion(pd.__version__) >= '0.18')
  File "/home/travis/miniconda/envs/pandas/lib/python3.5/distutils/version.py", line 70, in __ge__
    c = self._cmp(other)
  File "/home/travis/miniconda/envs/pandas/lib/python3.5/distutils/version.py", line 337, in _cmp
    if self.version < other.version:
TypeError: unorderable types: str() < int()
```

It's like `LooseVersion()` has lost its mind on the Travis machine. I can't reproduce the problem on my machine and the tests pass without a problem. Have you seen anything like this?

you need to rebase on master