Repository: pandas-dev/pandas

Issue:
DOC: fill_value argument in Series.combine()
Series.combine() method is not handling NaN values correctly.
On running this code, NaN is still returned and isnt replaced by 5.

```
#importing pandas module
import pandas as pd

#importing numpy module
import numpy as np

#creating first series
first=[1,2,np.nan,5,6,3,np.nan,7,11,0,4,8]

#creating second series
second=[5,3,2,np.nan,1,3,9,21,3,np.nan,1,np.nan]

#making series
first=pd.Series(first)

#making seriesa
second=pd.Series(second)

#calling .combine() method
result=first.combine(second, (lambda x1, x2: x1 if x1 > x2 else x2) , fill_value=5)

#display
result
```

The NULL values in Caller series are replaced but the Null values in the passed series remain there. Unlike other similar methods like .add(), .sub() etc. this one isn't handling Null values of passed series correctly.

If this issue is approved, i would like to contribute and fix it


Discussion:
I would like to work this bug !
`>>> result`
0      5.0
1      3.0
2      2.0
3      NaN
4      6.0
5      3.0
6      9.0
7     21.0
8     11.0
9      NaN
10     4.0
11     NaN
dtype: float64
`>>> type(first[0])`
<class 'numpy.float64'>
`>>> first[0]`
1.0
`>>> first[0] > np.nan`
False

honestly, @kbhutan0001 , I don't know if we should even compare a `Not A Number` and a `float`
See this [answer from StackOverflow](https://stackoverflow.com/questions/1565164/what-is-the-rationale-for-all-comparisons-returning-false-for-ieee754-nan-values)
@brute4s99 We are not comparing, we are combining them. This bug is because the function is probably executed before na replacement and hence null is compared with numeric values

okay, @kbhutani0001 . I'll look more into this([.](https://github.com/pandas-dev/pandas/blob/v0.23.4/pandas/core/series.py#L2201-L2246)) tomorrow !
This is the correct behavior as the document states:

> optional fill value when an **index** is missing from one Series or the other

so `fill_value` represents the placeholder of the value corresponding to the index that doesn't exist not the `fill_value` of NaN values.

```
In [13]: second.loc[12] = 1

In [14]: result=first.combine(second, (lambda x1, x2: x1 if x1 > x2 else
    ...: x2) , fill_value=5)

In [15]: result
Out[15]:
0      5.0
1      3.0
2      2.0
3      NaN
4      6.0
5      3.0
6      9.0
7     21.0
8     11.0
9      NaN
10     4.0
11     NaN
12     5.0 # 5 > 1 so we get 5 here
dtype: float64
```

Would welcome an example in the docstring showing this behavior.
In other mathematical operation, fill_na fills NULL values in both series
so i don't think this one should be like this. Moreover what's the point of
null handling when you get NaN in output series anyway. Either don't do it
or do it for both😅

On Wed, 3 Oct 2018, 1:58 am Matthew Roeschke, <notifications@github.com>
wrote:

> This is the correct behavior as the document states:
>
> optional fill value when an *index* is missing from one Series or the
> other
>
> so fill_value represents the placeholder of the value that doesn't exist
> in the index not the fill_value of NaN values.
>
> In [13]: second.loc[12] = 1
>
> In [14]: result=first.combine(second, (lambda x1, x2: x1 if x1 > x2 else
>     ...: x2) , fill_value=5)
>
> In [15]: result
> Out[15]:
> 0      5.0
> 1      3.0
> 2      2.0
> 3      NaN
> 4      6.0
> 5      3.0
> 6      9.0
> 7     21.0
> 8     11.0
> 9      NaN
> 10     4.0
> 11     NaN
> 12     5.0 # 5 > 1 so we get 5 here
> dtype: float64
>
> Would welcome an example in the docstring showing this behavior.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/22953#issuecomment-426418015>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/Ae0Do9ZcAVoiT6bSv4QcXvhOBsYOYFbuks5ug8xvgaJpZM4XEwvC>
> .
>

The names of the arguments are slightly different `fill_na` vs `fill_value` (though `fill_value` isn't a great name for what it does).

Plus all comparisons with `np.nan` are False, so the given function `lambda x1, x2: x1 if x1 > x2 else x2`, `np.nan` comparisons will always fill with the 2nd value. 
> Would welcome an example in the docstring showing this behavior.

on it !
> The names of the arguments are slightly different `fill_na` vs `fill_value` (though `fill_value` isn't a great name for what it does).

how about `fallback_value` , @mroeschke ?
That'd be a better name, but I don't think changing this keyword is worth the API churn.

If it's properly documented with an example, the meaning would be entirely clear.
>If it's properly documented with an example, the meaning would be entirely clear.

I'll push a PR shortly, @mroeschke , reviewing for typos.

Resolving patch:
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -2281,37 +2281,71 @@ def _binop(self, other, func, level=None, fill_value=None):
 
     def combine(self, other, func, fill_value=None):
         """
-        Perform elementwise binary operation on two Series using given function
-        with optional fill value when an index is missing from one Series or
-        the other
+        Combine the Series with a Series or scalar according to `func`.
+
+        Combine the Series and `other` using `func` to perform elementwise
+        selection for combined Series.
+        `fill_value` is assumed when value is missing at some index
+        from one of the two objects being combined.
 
         Parameters
         ----------
-        other : Series or scalar value
+        other : Series or scalar
+            The value(s) to be combined with the `Series`.
         func : function
-            Function that takes two scalars as inputs and return a scalar
-        fill_value : scalar value
-            The default specifies to use the appropriate NaN value for
-            the underlying dtype of the Series
+            Function that takes two scalars as inputs and returns an element.
+        fill_value : scalar, optional
+            The value to assume when an index is missing from
+            one Series or the other. The default specifies to use the
+            appropriate NaN value for the underlying dtype of the Series.
 
         Returns
         -------
-        result : Series
-
-        Examples
-        --------
-        >>> s1 = pd.Series([1, 2])
-        >>> s2 = pd.Series([0, 3])
-        >>> s1.combine(s2, lambda x1, x2: x1 if x1 < x2 else x2)
-        0    0
-        1    2
-        dtype: int64
+        Series
+            The result of combining the Series with the other object.
 
         See Also
         --------
         Series.combine_first : Combine Series values, choosing the calling
-            Series's values first.
-        """
+            Series' values first.
+
+        Examples
+        --------
+        Consider 2 Datasets ``s1`` and ``s2`` containing
+        highest clocked speeds of different birds.
+
+        >>> s1 = pd.Series({'falcon': 330.0, 'eagle': 160.0})
+        >>> s1
+        falcon    330.0
+        eagle     160.0
+        dtype: float64
+        >>> s2 = pd.Series({'falcon': 345.0, 'eagle': 200.0, 'duck': 30.0})
+        >>> s2
+        falcon    345.0
+        eagle     200.0
+        duck       30.0
+        dtype: float64
+
+        Now, to combine the two datasets and view the highest speeds
+        of the birds across the two datasets
+
+        >>> s1.combine(s2, max)
+        duck        NaN
+        eagle     200.0
+        falcon    345.0
+        dtype: float64
+
+        In the previous example, the resulting value for duck is missing,
+        because the maximum of a NaN and a float is a NaN.
+        So, in the example, we set ``fill_value=0``,
+        so the maximum value returned will be the value from some dataset.
+
+        >>> s1.combine(s2, max, fill_value=0)
+        duck       30.0
+        eagle     200.0
+        falcon    345.0
+        dtype: float64
+"""
         if fill_value is None:
             fill_value = na_value_for_dtype(self.dtype, compat=False)
 
@@ -2352,16 +2386,26 @@ def combine(self, other, func, fill_value=None):
 
     def combine_first(self, other):
         """
-        Combine Series values, choosing the calling Series's values
-        first. Result index will be the union of the two indexes
+        Combine Series values, choosing the calling Series's values first.
 
         Parameters
         ----------
         other : Series
+            The value(s) to be combined with the `Series`.
 
         Returns
         -------
-        combined : Series
+        Series
+            The result of combining the Series with the other object.
+
+        See Also
+        --------
+        Series.combine : Perform elementwise operation on two Series
+            using a given function.
+
+        Notes
+        -----
+        Result index will be the union of the two indexes.
 
         Examples
         --------
@@ -2371,11 +2415,6 @@ def combine_first(self, other):
         0    1.0
         1    4.0
         dtype: float64
-
-        See Also
-        --------
-        Series.combine : Perform elementwise operation on two Series
-            using a given function.
         """
         new_index = self.index.union(other.index)
         this = self.reindex(new_index, copy=False)


Test patch: