BUG:  DataFrame.update bool dtype being converted to object #55509
State: closed

- [x] Fixes #55509
- [x] Tests added and passed.
- [x] All code checks passed.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Blocker
description: Blocking issue or pull request for an upcoming release
name: Stale

Comments:
This is my first PR. For some reason that I still can't understand the commit that was immediately before the one I submitted got resubmitted. I imagine it can be something usual and easy to fix, but let me know if it is necessary that I  fix something (either on GitHub or on my local repository).
@rhshadrach

In short. To capture this bug I analyzed the "Unit Tests / Copy-on-Write 3.9 (pull_request)" and captured the single test giving the warning. After studying it it was possible to reproduce this test on my machine using the same pytest called by you on the GitHub CI with a proper environment setup for testing "copy-on-write". For the latter I tested using both the environment variable and `pandas.set_option()`. 

In the pull request above the code is failing on `FutureWarning`, and I believe that my code is giving the proper results (which were check through local builds), but that the test is outdated (it refer to #3217, which is from before 2013). 

Below I put the actual test that I think is wrong,  and the new proposed test (for which my new code on update passes):

File: `pandas/tests/frame/methods/test_update.py`
(Possibly) wrong test:
```python 
    def test_update_with_different_dtype(self, using_copy_on_write):
        # GH#3217
        df = DataFrame({"a": [1, 3], "b": [np.nan, 2]})
        df["c"] = np.nan
        if using_copy_on_write:
            df.update({"c": Series(["foo"], index=[0])})
        else:
            with tm.assert_produces_warning(FutureWarning, match="incompatible dtype"):
                df["c"].update(Series(["foo"], index=[0]))

        expected = DataFrame({"a": [1, 3], "b": [np.nan, 2], "c": ["foo", np.nan]})
        tm.assert_frame_equal(df, expected)
```

Curren proposed test:
```python 
    def test_update_with_different_dtype(self, using_copy_on_write):
        # GH#3217 #55509
        df = DataFrame({"a": [1, 3], "b": [np.nan, 2]})
        df["c"] = np.nan
        with tm.assert_produces_warning(FutureWarning, match="incompatible dtype"):
            if using_copy_on_write:
                df.update({"c": Series(["foo"], index=[0])})
            else:
                df["c"].update(Series(["foo"], index=[0]))

        expected = DataFrame({"a": [1, 3], "b": [np.nan, 2], "c": ["foo", np.nan]})
        tm.assert_frame_equal(df, expected)
```


Given this information I am not sure how to proceed:
1) Open an issue to correct the test. Wait for review.
2) Make a new pull request fixing test and the new code (separating the commits for history checks).
3) Close this pull request and wait for further discussion among core developers. 

Regards






> Looking good! I think we should add the test from the issue ([#55509 (comment)](https://github.com/pandas-dev/pandas/issues/55509#issuecomment-1767324695)).

Thanks for the return. I will add a test similar to the one presented in the discussion.
Organizing ideas:

Tests: 
-    [x] Add tests for when the index has duplicate values.
-    [x] Add the longer test checking various types, from issue #55509.

Investigate: 
-    [x] implementation of `DataFrame.clip()`, for ideas.
-    [x] performance regressions nonmonotonic case.
-    [x] performance regressions small dataframe with larger dataframe on argument.

Error messages:
-  [x] duplicate indexes.
-  [x] no overlap.

Docs:
-  [ ] Change whatsnew.rst, version 2.2

Questions:

- Was copy-on-write used on the tests above?
- Should strings be included in the longer test?
- What should I do about `test_update_with_different_dtype()`, discussed above?
The source of merging conflict is #56228. The merging conflict can be easily solved since it doesn't affect this PR.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
@aureliobarbosa - okay if I finish this up for 3.0?
@rhshadrach Thanks for bringing it back. Would you mind if I try to solve it until Feb 20th?
Sure!
@rhshadrach Tests are passing on my setup but the commit history of this PR got very noised. I would like to move it to a new PR, making reference to this. Is it OK?
@aureliobarbosa - I think in this case that's fine. For longer PRs (e.g. something like 300+ line diffs) it can create a lot of rework for reviewers, but that's not the case here.
@rhshadrach Done here. 

The intention of the  commit below was just a merge but I incorporate the main code fix on it (single line), by mistake.
[Merge remote-tracking branch 'upstream/main' into fix_gh55509](https://github.com/pandas-dev/pandas/pull/55634/commits/ce1b7a764d00ba6b88c84e9ed86e1278e64dbf33)

Also, I don't understand what is wrong with the doctest. 

I am available for any question or fix. 
@aureliobarbosa - thanks for the update. There are 2 questions/requests above that should be addressed.

> Also, I don't understand what is wrong with the doctest.

The doctest is failing on `DataFrame.to_xarray` and this method does use `DataFrame.update`. It looks like this needs to be addressed.
@rhshadrach Thanks for the feedback. I will try to address this issue this weekend.
@rhshadrach I decide to move proposed fix to PR #57637 because there were a few problems on the commit history of this PR that I couldn't fix myself, and also because it was possible to take a new approach on the code there, which I think is simpler and more maintainable. 
Closing in favor of https://github.com/pandas-dev/pandas/pull/57637