Repository: pandas-dev/pandas
Pull request: Fix 40420:  Interpret NaN in clip() as no bound.
- [x] closes #40420
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Problem statement:
BUG: clip where the bound is a series with NA values returns NA  
```
In [1]: import pandas as pd

In [2]: pd.__version__
Out[2]: '1.2.3'

In [3]: tmp = pd.DataFrame({'f': [1,2]})

In [4]: import numpy as np

In [5]: tmp['h'] = np.nan
#This is the expected behaviour
In [6]: tmp['f'].clip(0, np.inf)
Out[6]: 
0    1
1    2
Name: f, dtype: int64
# This seems wrong:
In [7]: tmp['f'].clip(0, tmp['h'])
Out[7]: 
0   NaN
1   NaN
Name: f, dtype: float64
```

Hints from issue discussion:
This output seems correct to me. Why would an upper bound of `np.nan` behave the same way as `np.inf`?
Hi @dsaxton, please allow me to break my comments in two layers:
1. There is an inconsistency in pandas clip: as pointed out in the thread above, if you do df[col].clip(0, np.nan), it ignore the NA. Or, as you said, it gives you identical result as df[col].clip(0, np.inf). No clipping happens. However, if you do df[col].clip(0, df[col2]) where df[col2] is nothing but NA, then you get NA in return. The NA propagate.

2. Personally, I would prefer that NA values get ignored in clipping, regardless if it is a float or as part of a series/array. In pandas, by default, max, min, sum etc functions all ignore NA, so I have built up such an expectation. Personally I would feel it counter-intuitive and maybe even confusing if clip breaks this rule. 
Ah, I didn't realize `Series.clip(0, np.nan)` does indeed ignore the `np.nan` (in your scalar example you're using `np.inf` as the upper bound not `np.nan`). That _is_ a bug I would say:
```python
[ins] In [7]: s
Out[7]: 
0    1
1    2
2    3
dtype: int64

[ins] In [8]: s.clip(0, np.nan)
Out[8]: 
0    1
1    2
2    3
dtype: int64

[ins] In [9]: s.clip(0, [np.nan, np.nan, np.nan])
Out[9]: 
0   NaN
1   NaN
2   NaN
dtype: float64
```
However, I think the correct behavior would be the last case where the result is `NaN`. If we're clipping with an unknown or invalid upper bound, the only correct answer would be unknown / invalid. I don't think this case is really analogous to taking the min or max of a `Series` containing `NaN`, since then it is the input where `NaN` is ignored, rather than an argument that defines what operation we want to perform.
Actually, according to an old issue it looks like your preference was actually the original intention: https://github.com/pandas-dev/pandas/issues/17276. So _if_ we were to stay true to that then the "correct" behavior would be to not return `NaN` for an array-like upper bound.
Thanks @dsaxton for the information!
1. You are exactly right. Sorry there was a typo in my original post. I meant to use np.nan, not np.inf as a scalar inside .clip().
2. I think we are in consensus that no matter which choice we make, having consistent behavior between NA scalar or as part of a series is desirable. 
3. In the case of having NA in the input, should we return NA, or not do any clipping? In addition to staying compatible to original intention in #17276, I would make an additional argument here for "no clipping": consider the following example:
df = pd.DataFrame({'f': [1, 2], 'g': [np.nan, np.nan]})
I would think that we want df['f'].clip(upper=df['g']) to give the same result as df[['f', 'g']].min(axis=1), since they are mathematically equivalent operations. 
My guess would be that always treating `np.nan` as "no upper bound" (regardless of whether it's an array or scalar) would be the right way to fix this for now. If you wanted to work on a pull request, the part where it seems we're handling `NaN` is around here: https://github.com/pandas-dev/pandas/blob/7af47c999dc3d2782caf63fe8c42f0858937867b/pandas/core/generic.py#L7534
take

Commit message: Fix doctest dtype

Patch to review:
diff --git a/doc/source/whatsnew/v1.3.0.rst b/doc/source/whatsnew/v1.3.0.rst
index 1c7942dfedafa..5fcacc9b049fc 100644
--- a/doc/source/whatsnew/v1.3.0.rst
+++ b/doc/source/whatsnew/v1.3.0.rst
@@ -855,6 +855,7 @@ Other
 - Bug in :meth:`DataFrame.equals`, :meth:`Series.equals`, :meth:`Index.equals` with object-dtype containing ``np.datetime64("NaT")`` or ``np.timedelta64("NaT")`` (:issue:`39650`)
 - Bug in :func:`pandas.util.show_versions` where console JSON output was not proper JSON (:issue:`39701`)
 - Bug in :meth:`DataFrame.convert_dtypes` incorrectly raised ValueError when called on an empty DataFrame (:issue:`40393`)
+- Bug in :meth:`DataFrame.clip` not interpreting missing values as no threshold (:issue:`40420`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index bb8de35d22462..be74807d0ea55 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -7352,6 +7352,14 @@ def _clip_with_one_bound(self, threshold, method, axis, inplace):
                 threshold = self._constructor(threshold, index=self.index)
             else:
                 threshold = align_method_FRAME(self, threshold, axis, flex=None)[1]
+
+        # GH 40420
+        # In order to ignore nan values in the threshold, set the values in
+        # subset that correspond to these na values to True. This indicates to the
+        # final where() to not clip.
+        if is_list_like(threshold) and threshold.isna().any(axis=None):
+            subset_kwargs = {"axis": axis} if threshold.ndim != subset.ndim else {}
+            subset = subset.where(threshold.notna(), True, **subset_kwargs)
         return self.where(subset, threshold, axis=axis, inplace=inplace)
 
     @overload
@@ -7483,10 +7491,12 @@ def clip(
         ----------
         lower : float or array_like, default None
             Minimum threshold value. All values below this
-            threshold will be set to it.
+            threshold will be set to it. A missing
+            threshold (e.g `NA`) will not clip the value.
         upper : float or array_like, default None
             Maximum threshold value. All values above this
-            threshold will be set to it.
+            threshold will be set to it. A missing
+            threshold (e.g `NA`) will not clip the value.
         axis : int or str axis name, optional
             Align object with lower and upper along the given axis.
         inplace : bool, default False
@@ -7547,6 +7557,25 @@ def clip(
         2      0      3
         3      6      8
         4      5      3
+
+        Clips using specific lower threshold per column element, with missing values:
+
+        >>> t = pd.Series([2, -4, np.NaN, 6, 3])
+        >>> t
+        0    2.0
+        1   -4.0
+        2    NaN
+        3    6.0
+        4    3.0
+        dtype: float64
+
+        >>> df.clip(t, axis=0)
+        col_0  col_1
+        0      9      2
+        1     -3     -4
+        2      0      6
+        3      6      8
+        4      5      3
         """
         inplace = validate_bool_kwarg(inplace, "inplace")
 
@@ -7559,9 +7588,17 @@ def clip(
         # so ignore
         # GH 19992
         # numpy doesn't drop a list-like bound containing NaN
-        if not is_list_like(lower) and np.any(isna(lower)):
+        isna_lower = isna(lower)
+        if not is_list_like(lower):
+            if np.any(isna_lower):
+                lower = None
+        elif np.all(isna_lower):
             lower = None
-        if not is_list_like(upper) and np.any(isna(upper)):
+        isna_upper = isna(upper)
+        if not is_list_like(upper):
+            if np.any(isna_upper):
+                upper = None
+        elif np.all(isna_upper):
             upper = None
 
         # GH 2747 (arguments were reversed)
diff --git a/pandas/tests/frame/methods/test_clip.py b/pandas/tests/frame/methods/test_clip.py
index 8a2374a414482..6525109da4394 100644
--- a/pandas/tests/frame/methods/test_clip.py
+++ b/pandas/tests/frame/methods/test_clip.py
@@ -144,17 +144,25 @@ def test_clip_with_na_args(self, float_frame):
         tm.assert_frame_equal(float_frame.clip(np.nan), float_frame)
         tm.assert_frame_equal(float_frame.clip(upper=np.nan, lower=np.nan), float_frame)
 
-        # GH#19992
+        # GH#19992 and adjusted in GH#40420
         df = DataFrame({"col_0": [1, 2, 3], "col_1": [4, 5, 6], "col_2": [7, 8, 9]})
 
         result = df.clip(lower=[4, 5, np.nan], axis=0)
         expected = DataFrame(
-            {"col_0": [4, 5, np.nan], "col_1": [4, 5, np.nan], "col_2": [7, 8, np.nan]}
+            {"col_0": [4, 5, 3], "col_1": [4, 5, 6], "col_2": [7, 8, 9]}
         )
         tm.assert_frame_equal(result, expected)
 
         result = df.clip(lower=[4, 5, np.nan], axis=1)
         expected = DataFrame(
-            {"col_0": [4, 4, 4], "col_1": [5, 5, 6], "col_2": [np.nan, np.nan, np.nan]}
+            {"col_0": [4, 4, 4], "col_1": [5, 5, 6], "col_2": [7, 8, 9]}
         )
         tm.assert_frame_equal(result, expected)
+
+        # GH#40420
+        data = {"col_0": [9, -3, 0, -1, 5], "col_1": [-2, -7, 6, 8, -5]}
+        df = DataFrame(data)
+        t = Series([2, -4, np.NaN, 6, 3])
+        result = df.clip(lower=t, axis=0)
+        expected = DataFrame({"col_0": [9, -3, 0, 6, 5], "col_1": [2, -4, 6, 8, 3]})
+        tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/series/methods/test_clip.py b/pandas/tests/series/methods/test_clip.py
index 528e95f65c8f4..442718d677101 100644
--- a/pandas/tests/series/methods/test_clip.py
+++ b/pandas/tests/series/methods/test_clip.py
@@ -49,8 +49,13 @@ def test_clip_with_na_args(self):
         tm.assert_series_equal(s.clip(upper=np.nan, lower=np.nan), Series([1, 2, 3]))
 
         # GH#19992
-        tm.assert_series_equal(s.clip(lower=[0, 4, np.nan]), Series([1, 4, np.nan]))
-        tm.assert_series_equal(s.clip(upper=[1, np.nan, 1]), Series([1, np.nan, 1]))
+        tm.assert_series_equal(s.clip(lower=[0, 4, np.nan]), Series([1, 4, 3]))
+        tm.assert_series_equal(s.clip(upper=[1, np.nan, 1]), Series([1, 2, 1]))
+
+        # GH#40420
+        s = Series([1, 2, 3])
+        result = s.clip(0, [np.nan, np.nan, np.nan])
+        tm.assert_series_equal(s, result)
 
     def test_clip_against_series(self):
         # GH#6966


Code review comments:
diff_hunk: @@ -7352,6 +7352,14 @@ def _clip_with_one_bound(self, threshold, method, axis, inplace):
                 threshold = self._constructor(threshold, index=self.index)
             else:
                 threshold = align_method_FRAME(self, threshold, axis, flex=None)[1]
+
+        # GH 40420
path: pandas/core/generic.py
text: @user1:
um, can you not `subset = subset.fillna(np.inf)` (or -np.inf if the other way)?

@author:
Yep, not for subset, but for the thresholds.