Repository: pandas-dev/pandas

Issue:
handle nan values in DataFrame.update when overwrite=False #15593
closes #15593


Discussion:
can you add some tests from the issue in pandas/tests/frame//test_combine_concat.py
and a whatsnew note (0.20.0) in the bug fixes section
I think it should be 
```
if isnull(that).all():
    continue
```
The line of `self[col] = expressions.where(mask, this, that, raise_on_error=True)` should be skipped on the absence of values in `that` i.e. column of the other dataframe whereas `mask` is computed on `this` or the column values of base dataframe for `overwrite=False`.

The committed solution works but is not correct I feel.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15617?src=pr&el=h1) Report
> Merging [#15617](https://codecov.io/gh/pandas-dev/pandas/pull/15617?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3d699884e26120618bf0bb8869bc07f1e51a2935?src=pr&el=desc) will **decrease** coverage by `-0.03%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master   #15617      +/-   ##
==========================================
- Coverage   91.02%   90.99%   -0.03%     
==========================================
  Files         143      143              
  Lines       49301    49301              
==========================================
- Hits        44875    44863      -12     
- Misses       4426     4438      +12
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15617?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d699884e26120618bf0bb8869bc07f1e51a2935...02ebf34184dfe7ec3a46c52d059682bb446ffae7?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.87% <100%> (-0.1%)` | :x: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d699884e26120618bf0bb8869bc07f1e51a2935...02ebf34184dfe7ec3a46c52d059682bb446ffae7?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :x: |
| [pandas/tools/merge.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d699884e26120618bf0bb8869bc07f1e51a2935...02ebf34184dfe7ec3a46c52d059682bb446ffae7?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL21lcmdlLnB5) | `91.76% <0%> (-0.35%)` | :x: |
| [pandas/tseries/index.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d699884e26120618bf0bb8869bc07f1e51a2935...02ebf34184dfe7ec3a46c52d059682bb446ffae7?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvaW5kZXgucHk=) | `95.29% <0%> (-0.1%)` | :x: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/compare/3d699884e26120618bf0bb8869bc07f1e51a2935...02ebf34184dfe7ec3a46c52d059682bb446ffae7?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.25% <0%> (ø)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15617?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15617?src=pr&el=footer). Last update [3d69988...02ebf34](https://codecov.io/gh/pandas-dev/pandas/compare/3d699884e26120618bf0bb8869bc07f1e51a2935...02ebf34184dfe7ec3a46c52d059682bb446ffae7?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
closing as stale, if you'd like to update and add some tests pls do.
I still am receiving the invalid type promotion error in pandas 0.19.2. when using overwrite = False and both old df and new df contain NaN values for same indices. :-(

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3921,13 +3921,13 @@ def update(self, other, join='left', overwrite=True, filter_func=None,
 
                 if overwrite:
                     mask = isnull(that)
-
-                    # don't overwrite columns unecessarily
-                    if mask.all():
-                        continue
                 else:
                     mask = notnull(this)
 
+            # don't overwrite columns unecessarily
+            if mask.all():
+                continue
+
             self[col] = expressions.where(mask, this, that,
                                           raise_on_error=True)
 


Test patch:
diff --git a/pandas/tests/frame/test_combine_concat.py b/pandas/tests/frame/test_combine_concat.py
--- a/pandas/tests/frame/test_combine_concat.py
+++ b/pandas/tests/frame/test_combine_concat.py
@@ -763,3 +763,25 @@ def test_concat_datetime_datetime64_frame(self):
 
         # it works!
         pd.concat([df1, df2_obj])
+
+
+class TestDataFrameUpdate(TestData):
+
+    def test_update_nan(self):
+        # #15593 #15617
+        # test 1
+        df1 = DataFrame({'A': [1.0, 2, 3], 'B': date_range('2000', periods=3)})
+        df2 = DataFrame({'A': [None, 2, 3]})
+        expected = df1.copy()
+        df1.update(df2, overwrite=False)
+
+        tm.assert_frame_equal(df1, expected)
+
+        # test 2
+        df1 = DataFrame({'A': [1.0, None, 3], 'B': date_range('2000', periods=3)})
+        df2 = DataFrame({'A': [None, 2, 3]})
+        expected = DataFrame({'A': [1.0, 2, 3], 'B': date_range('2000', periods=3)})
+        df1.update(df2, overwrite=False)
+
+        tm.assert_frame_equal(df1, expected)
+