API: allow single element boolean Series to improve on numpy behavior( related GH4657)
State: closed

related #4657


Labels:

Comments:
@jseabold @jtratner @hayd 

Looks good to me... 

Can we ping out google groups to see what people think?

I still don't see the logic to this (other than numpy doing it), it seems to me there is always a non-ambiguous way to write things...

Practicality beats purity for me here. It saves typing and seems quite logical to me. At least, I don't see any ambiguity, especially if the empty Series/DataFrame case is explicitly handled to raise. Incidentally, the behavior for an empty objects differs from numpy (and Python) but I agree with the change.

To me this seems impractical and impure! :(

I'm yet to see an example where it saves typing (and which can't be done without item), and I don't see why the one item case should be special (if we're breaking numpy behaviour _anyway_....).

Every example I showed involved more typing?

I thought there was always a way it could be written to not be, e.g. the example with pd.isnull(df.ix[]) could be tweaked so the ix returned a value (not a Series)...

ix will only return a value if I don't use a boolean index, but this involves more code to get the single index value -- either a (horribly ugly) call to np.where, some pandas-fu that I didn't try yet, or setting an index on the frame (which isn't always possible).

Please do bring it up on the ML though. Input from others would be helpful.

I didn't realize how this handles empty Series. Should definitely be the same as Python/numpy:

``` python
>>> bool(Series([]))
False
>>> bool(np.array([])
False
>>> bool(np.array([[]])
False
>>> bool(DataFrame([[]]))
False
```

Ha, now there's some ambiguity if it's true there wasn't before. I thought this was an improvement over numpy and the warning is clear that maybe you should use 'empty'.

Anway, my vote is
1. Empty series raises. Maybe you screwed up your index? What is the 'correct' output of this?
   
   ```
   if pd.isnull(pd.DataFrame([])):
       print 'this dataframe has no missing values?'
   ```
   
   Ambiguous?
2. 1 element is fine. You know what you're doing, carry on. Also .all() == .any() in this case, so it's not ambiguous.
3. Length > 1 raises . This is ambiguous. Ask for all, any, or empty. Maybe you screwed up your index?

You can read old numpy threads about this if no one wants to discuss on the ML, but I think these issues may be of general interest.

current behavior (with this PR) is to raise on empty Series/DataFrame (note that this was not true in 0.12,
but actually WAS true IIRC before 0.12)

I agree that we should absolutely raise on an empty object

```
if not df:
   print 'ok'
```

should NEVER work (and if we allowed it to work for an empty frame it would)

correct is

```
if not df.empty:
     print 'ok'
```

@jseabold proposal is what this PR does

Yeah. I'm +1.

hmm...I'm mixed on only following some but not all of the numpy behavior
here...seems slightly confusing to me, but this is not an area I use much
with pandas.

Obviously I'm still massive -1, I think we should be raising on all `__nonzero__` (i.e. not this pr, but current master).

> 1 element is fine. You know what you're doing

Totally disagree. I would also argue that it _is_ ambiguous.

```
In [1]: bool([False])
Out[1]: True

In [2]: bool(np.array([np.nan]))
Out[2]: True

In [3]: bool(np.array([False]))
Out[3]: False
```

`pd.isnull(pd.DataFrame([]))` is an empty DataFrame (not ambiguous)...

On Tue, Sep 3, 2013 at 9:13 PM, Andy Hayden notifications@github.comwrote:

> Obviously I'm still massive -1, I think we should be raising on all
> **nonzero** (i.e. not this pr, but current master).
> 
> 1 element is fine. You know what you're doing
> 
> Totally disagree. I would also argue that it _is_ ambiguous.
> 
> In [1]: bool([False])
> Out[1]: True

Hmm, ok, this is a bit confusing, but I'd argue not a show stopper. I
can't speak for everyone, but I don't find this difficult to differentiate.
Most of Numpy/pandas operates element-wise. So element-wise it's not
ambiguous. If there's one element, we can return an answer.

> In [2]: bool(np.array([np.nan]))
> Out[2]: True

NumPy's nan handling is inconsistent and definitely one of the areas where
pandas should/exists to improve. It and operations on empty arrays are two
issues that are pointed to as warts (by some) in the numpy threads about
similar. That said, I'm not sure what bool(pd.Series([np.nan])) should do.
I'd probably say that it should raise an error.

> In [3]: bool(np.array([False]))
> Out[3]: False
> 
> pd.isnull(pd.DataFrame([])) is an empty DataFrame (not ambiguous)...

If the null DataFrame is the result of an indexing operation, then it is
ambiguous I'd say. If your question you want to answer is, given this
index, is this value True? The answer is, we can't evaluate this question
because your indexing operation didn't return any information.

we would have to special case Series([np.nan]) as I think this would evaluate to bool(np.nan) which is True

I think this special casing is going to confuse users (one rule for python, one (especially confusing) for numpy, one for pandas)... I think it's consistency to say no to `__nonzero__`. I think this is an improvement.

re the empty DataFrame it's just like doing a function over emptyset is empty:

```
map(pd.isnull, []) == []
```

the current behaviour is well-defined (and imo correct). However bool of it is ambiguous (if that's what you were saying then we agree :) ).

Some more random stuff we wanted to stop (but work in numpy):

```
In [4]: not np.array([False])
Out[4]: True

In [5]: np.array([False]) or np.array([np.nan])
Out[5]: array([ nan])
```

Special casing this will be a rabbit hole.

On Tue, Sep 3, 2013 at 9:41 PM, Andy Hayden notifications@github.comwrote:

> I think this special casing is going to confuse users (one rule for
> python, one (especially confusing) for numpy, one for pandas)... I think
> it's consistency to say no to **nonzero**. I think this is an improvement.
> 
> re the empty DataFrame it's just like doing a function over emptyset is
> empty:
> 
> map(pd.isnull, []) == []
> 
> the current behaviour is well-defined (and imo correct). However bool of
> it is ambiguous (if that's what you were saying then we agree :) ).

Yes, we agree.

> Some more random stuff we wanted to stop (but work in numpy):
> 
> In [4]: not np.array([False])
> Out[4]: True

 No problem here in current usage. It maps.

> In [5]: np.array([False]) or np.array([np.nan])
> Out[5]: array([ nan])
> 
> Either the RHS raises or it short circuits if the first element is True.
>  Special casing this will be a rabbit hole.

There are 5 (?) eventualities I can think of -- empty (raise), 1 element
castable to float (return a boolean), more than one element (raise), 1 nan
only (raise).

The other one I can think of is a single-element object array. I'd argue we
should raise in this case too. What else?

actually most of these are solved by limiting the single element resolving to only a boolean array
by definition then a single nan will raise, object array will raise, cast able to bool will raise (I have tests for these)

ONLY a single element series of True/False will pass thru the bool of the element 

the only remaining point is do we allow in `bool(Series([True])) == True` and `bool(Series([False])) == False`

numpy does this, but it is inconsistent with all other bool operations on a PandasObject which will raise

@hayd -1
@jseabold +1

any more strong opionions? @wesm @y-p ?

FWIW, I'm +1 so we're consistent.

so @jtratner vote I think is -1 (so with @hayd for consistency) (the +- are for this PR which implements the above)

@jreback no sorry I wasn't clear, I vote for being consistent with numpy.

Just to have a last comment on the subject (I think arguments are clear in this and the previous PR):

You're **not** voting for consistency with numpy, since we'd still raise on empty Series, but some kind of weird middle ground.

Weird...or sane... depending on how you come down I guess.

(slighter saner... ;) )

Should I put this on mailing list? I've _tried_ to sound impartial.

## changing bool behaviour of pandas objects

There's discussion on github to change the behaviour of `__nonzero__` for pandas objects.

Bool behaviour in pandas (and numpy) often trips up and surprises new (and experienced) users, for one thing because it differs from many python objects.
- For empty arrays it's Falsey
- For length one arrays it's bool of item (Note: bool(nan) is True)
- Otherwise it raises a ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

One option (currently implemented in master) is to turn off bool **always**:
- raise ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().

An alternative proposal being discussed is (4738):
- For length one arrays it's bool of item (Perhaps raising on bool(Series([nan])).)
- Otherwise raise a ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Note: bool of empty objets would be disallowed.

In both cases:
- not/and/or would be specifically disallowed.

sure...let's put it out there (numpy list too?)