API: clarify DataFrame.apply reduction on empty frames
State: closed

Add `is_reduction` argument to `DataFrame.apply` to avoid undefined behavior when `.apply` is called on an empty DataFrame when function being applied is only defined for valid inputs.

Currently, if the DataFrame is empty, a guess is made at the correct return value (either a `DataFrame` or a `Series`) by calling the function being applied with an empty Series as an argument:

``` python
if not all(self.shape):
    # How to determine this better?
    is_reduction = False
    try:
        is_reduction = not isinstance(f(_EMPTY_SERIES), Series)
    except Exception:
        pass

    if is_reduction:
        return Series(NA, index=self._get_agg_axis(axis))
    else:
        return self.copy()
```

For reduction functions which produce undefined results on unexpected input (ex, a function which doesn't expect an empty argument), this means that the the result of `apply` is also undefined.

This pull request adds an explicit `is_reduction` argument so that it's possible to explicitly control this otherwise undefined behavior.

**Update**: there has been the suggestion that the existing `reduce` argument should be used. Is this reasonable? The PR would be updated as follows:
- Remove the `is_reduction` argument
- Change the default value of `reduce` from `True` to `None` (to preserve the current behavior of checking the return value of the function being applied)
- In the case of an empty DataFrame, treat `reduce` in the same way that I'm currently treating `is_reduction`
- Otherwise treat `reduce` as normal

Ref:
- http://stackoverflow.com/q/21225608/71522
- http://stackoverflow.com/q/21225552/71522


Labels:

Comments:
their is already a reduce argument

The `reduce` argument has no effect if the DataFrame is empty.

don't u think it would be better to fix that than add another keyword?

Would it make sense to change the default value of `reduce` to `None` (so as not to break existing calls to `.apply()` which depend on the current empty-data-frame-reduction-guessing), then replace `is_reduction` with `reduce`? (I don't grok the semantics of `reduce` well enough to say for sure)

that's possible I think

can u give a use case for this?

apply on an empty frame should just return the frame - I don't think a reduction makes sense

My use case is that I've got a reduction function which behaves poorly (raises an exception) when the current implementation calls the reduction function with an empty Series to try and guess the return value (https://github.com/wolever/pandas/blob/5ca822b35d196928fc9ef1b14d457553ea7f3e68/pandas/core/frame.py#L3437)

This causes a problem because it's surprising that `.apply(…)` returns a `Series` _except_ when it's empty (which essentially means I've got to litter otherwise correct algorithms with checks to see if the input DataFrame is empty).

can u put a complete example up
as well as pandas version?

Specifically, the code I'm working with right now looks like this:

``` python
a = df.apply(row_reduction_func_a, axis=1)
b = df.apply(row_reduction_func_b, axis=1)
result = DataFrame({ "a": a, "b": b })
```

Which works fine, except when the `df` is empty (because then `a` will be a `DataFrame` and the DataFrame constructor raises an exception when it tries to create a column from a `DataFrame`)

Pandas version 0.12.0 but this issue is present in trunk (see the associated test case).

can u put up the function itself along with a sample frame

Is there something wrong with the test cases I supplied?

no I am trying to see if their is a more general issue

your solution is treating the symptom not the problem

Here is an example of a reduction function which will behave poorly (forgive me for being unable to share my specific function):

``` python
>>> def row_reduction_function(row):
...    return (row == correct).sum()
>>> correct = pd.Series(["a", "b"], index=["a", "b"])
```

And when applied to a full DataFrame, it produces the expected result (a Series with one element):

``` python
>>> nonempty = pd.DataFrame({ "first": ["a"], "second": ["x"] })
>>> nonempty.apply(row_reduction_function, axis=1)
0    1
dtype: int64
```

But when applied to an empty DataFrame, it doesn't produce an empty Series; it produces a DataFrame:

``` python
>>> empty = pd.DataFrame({ "first": [], "second": [] })
>>> empty.apply(row_reduction_function, axis=1)
Empty DataFrame
Columns: [first second]
Index: []
```

The problem is that, with an empty DataFrame, `.apply` tries to guess what the result should be, and in sitautions where the reduction function is only defined for a "real" row (as opposed to the empty row `.apply` uses to try and guess the result), the result is also undefined. There needs to be some way to explicitly define whether the result should be a Series of a DataFrame. If using the `reduce` argument for this purpose is appropriate, I'd be all for that!

I think it would make more sense to have a empty frame in this example.  I would rather not get an exception as missing data may throw exceptions in many functions.  

What would be the expected behavior expected for this:
http://stackoverflow.com/questions/21225608/pandas-dataframe-applyf-axis-1-on-an-empty-dataframe-ignores-exceptions

@MichaelWS the issue is not one of "when should an empty DataFrame be returned versus an empty Series".

The problem is that, when the input DataFrame is empty, _the result of `.apply` is undefined_ if the function being applied has undefined behavior for unexpected input.

@MichaelWS as for the question of expected behavior: my problem isn't with the expected behavior. My problem is that there's no way to override the current behavior and explicitly control the currently undefined behavior when the DataFrame is empty.

I get your point of view, but I agree with jreback and think the average user would be a bit confused with the reduce and is_reduction combination. It might be easier to use the reduce keyword to handle the empty frame.

Absolutely agree.

In that case, do you agree that it makes sense to update the PR as follows:
- Remove the `is_reduction` argument
- Change the default value of `reduce` from `True` to `None` (to preserve the current behavior of checking the return value of the function being applied)
- In the case of an empty DataFrame, treat `reduce` in the same way that I'm currently treating `is_reduction`
- Otherwise treat `reduce` as normal

That would be my preference, but I would wait for some others for a consensus.

Okay, cool. I'll update the description and see if we can get some more consensus. I don't grok `reduce` to know for sure that it will be okay.

why don't u give a try by changing reduce
I don't remember exactly why I added it anyhow 

Okay, cool — there we go. It does feel a _bit_ like we're overloading the `reduce` argument… but, like I said, I don't totally grok the regular functioning of `reduce`, so I'll trust you if you say this is the way to do it.

In fact, I noticed another issue today too: if the DataFrame is completely empty (no rows _or_ columns), `.apply(…)` _always_ returns a DataFrame, when in fact it should use the same logic to determine what type should be returned. PR has been updated.

I'll update the unit tests to make sure they are all passing after this gets approval.

I have 2 look at this some more
but can u add a release note in the API section (reference this pr number)

also pls put a short example in v0.13.1.txt (illustrating both cases)

@wolever ok...this looks fine...pls add the docs as I mentioned above and good 2 go

Okay! Docs and tests updated too. Unfortunately I wasn't able to test the docs since my laptop ran out of RAM while trying to build them… but hopefully they should look decent.

also, pls rebase and squash down to a single commit, see here: https://github.com/pydata/pandas/wiki/Using-Git

@wolever pls rebase this and squash

closed via https://github.com/pydata/pandas/commit/d74bd31a940cc64a7733798f09cdea06065402f5

thanks for the nice fix!

Urg, sorry for not doing this myself - I've been out of commission the last couple of days.

Thanks for the merge!

> On Jan 26, 2014, at 10:31, jreback notifications@github.com wrote:
> 
> closed via d74bd31
> 
> thanks for the nice fix!
> 
> —
> Reply to this email directly or view it on GitHub.