API: Series.sum() will now return 0.0 for all-NaN series
State: closed

compat with `numpy` >= 1.8.2 and `bottleneck` >= 1.0, #9422
note that passing skipna=False will still return a NaN


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@shoyer 

So you think these are 'bugs' in numpy or just conventions?
and then should we follow these. ?

```
In [6]: np.nansum(np.array([np.nan],dtype='float64'))     
Out[6]: 0.0

In [7]: np.nansum(np.array([np.nan],dtype='object'))
Out[7]: nan

In [8]: np.nansum(np.array([np.timedelta64('nat')]))     
Out[8]: numpy.timedelta64('NaT')
```

Those look like numpy bugs to me

On Fri, Aug 14, 2015 at 11:56 AM, Jeff Reback notifications@github.com
wrote:

> @shoyer 
> So you think these are 'bugs' in numpy or just conventions?
> and then should we follow these. ?
> 
> ```
> In [6]: np.nansum(np.array([np.nan],dtype='float64'))     
> Out[6]: 0.0
> In [7]: np.nansum(np.array([np.nan],dtype='object'))
> Out[7]: nan
> In [8]: np.nansum(np.array([np.timedelta64('nat')]))     
> Out[8]: numpy.timedelta64('NaT')
> ```
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/10815#issuecomment-131209203

I am thinking this is actually a bad idea to make this change. This loses information in the `.sum()`. Maybe numpy doesn't care as it doesn't propogate nans. But I think users do.

@shoyer 
@jorisvandenbossche 
@sinhrks 

```
In [1]: df = DataFrame({'A' : [1,2,3], 'B' : np.nan})

In [2]: df
Out[2]: 
   A   B
0  1 NaN
1  2 NaN
2  3 NaN

In [3]: df.sum()
Out[3]: 
A    6
B    0
dtype: float64

In [4]: df.sum(1)
Out[4]: 
0    1
1    2
2    3
dtype: float64

In [5]: df2 = DataFrame({'A' : [1,2,3], 'B' : [np.nan,0,0]})

In [6]: df2.sum()
Out[6]: 
A    6
B    0
dtype: float64

In [7]: df2.sum(1)
Out[7]: 
0    1
1    2
2    3
dtype: float64
```

So your result now depends on whether you have 1 non-NaN in a reduction. And if it happens to be 0, they are indistinguishable.

@jorisvandenbossche @shoyer opinions on this?

Hmm, tough one, and I don't really have a strong opinion on this.

However, there is also some logic/consistency on returning 0, as an empty series gives you also 0:

```
In [30]: pd.Series([]).sum()
Out[30]: 0
```

And if you skip the NA's, you have in fact an empty series

I also don't have a strong opinion here. In practice, sum gets used much
less than mean, and even when it does I'm often dividing eventually by
counts, anyways, which means that this distinction doesn't matter. My
inclination is to copy NumPy, though this would also have implications for
other pandas functions (e.g., grouped and moving sum).

On Tue, Aug 18, 2015 at 4:47 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> Hmm, tough one, and I don't really have a strong opinion on this.
> 
> However, there is also some logic/consistency on returning 0, as an empty
> series gives you also 0:
> 
> In [30]: pd.Series([]).sum()
> Out[30]: 0
> 
> And if you skip the NA's, you have in fact an empty series
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10815#issuecomment-132392471.

actually my point wasn't the actual behavior, it was losing the propogation of all-NaN columns when you do a reduction. IOW, you essentially lose the notion that you aggregated on an all-NaN column.

The only reason I am pressing on this is I just installed bottleneck 1.0 and have a couple of tests I either need to mark as knownfails or fix this :)

I came across this issue after spending way too much time trying to debug my code after a sloppy package upgrade session that included bottleneck 1.0. For what it's worth, I very much prefer these behaviors:

``` python
In [1]: pd.Series([]).sum() # less important
Out[1]: NaN

In [2]: pd.Series([np.nan]).sum()
Out[2]: NaN
```

``` python
In [1]: df = DataFrame({'A' : [1,2,3], 'B' : np.nan})

In [2]: df
Out[2]: 
   A   B
0  1 NaN
1  2 NaN
2  3 NaN

In [3]: df.sum()
Out[3]: 
A    6
B    NaN
dtype: float64

In [4]: df.sum(1)
Out[4]: 
0    1
1    2
2    3
dtype: float64
```

@wholmgren FWIW, I agree with you. but this is getting pushed off anyhow, so current behavior will remain.

going to close for now