Repository: pandas-dev/pandas
Pull request: BUG: Copy attrs on pd.merge()
This uses the same logic as `pd.concat()`: Copy `attrs` only if all input `attrs` are identical.

I've refactored the handling in __finalize__ from special-casing based on th the method name (previously only "concat") to handling "other" parameters that have an `input_objs` attribute. This is a more scalable architecture compared to hard-coding method names in __finalize__.

Tests added for `concat()` and `merge()`.

Closes #60351.


Problem statement:
ENH: Copy attrs on join (possibly depending on left, right, etc.)
### Feature Type

- [ ] Adding new functionality to pandas

- [X] Changing existing functionality in pandas

- [ ] Removing existing functionality in pandas


### Problem Description

df.join() does not retain the attrs of the dataset. Given that the attrs manual states that "many operations that create new datasets will retain attrs", this seems like an omission.

### Feature Description

Join is different from concat because there is a clear dataframe that the operation is on. Therefore, it would seem natural if df.join() would retain the attrs of the initial dataframe.

### Alternative Solutions

It would also be possible to make the attrs dependent on "how" but this would only be natural for "left" and "right".

### Additional Context

_No response_

Hints from issue discussion:
This should be handled the same way as `concat`: Propagate only if all inputs have the same attrs.

`concat` is currently a hard-coded special case, https://github.com/pandas-dev/pandas/blob/7fe270c8e7656c0c187260677b3b16a17a1281dc/pandas/core/generic.py#L6056

but we may want to delegate the attrs combination back to the operation instead, i.e. 

```
if hasattr(other, "__combined_attrs__"):
      self.attrs = other.__combined_attrs__()
```
Note that `other` is the "combination object" for there calls, i.e. `_MergeOperation`, `_Concatenator` etc, which will have to grow the logic for combining attrs.

Alternatively, one could leave the combination logic in `__finalize__` but provide a uniform interface on all "combination objects" to give their inputs. Currently, thats non-uniform `_Concatenator.objs`, but `_MergeOperation.left`/`_MergeOperation.right`.
Thanks for the report! @timhoffm - can you post a reproducible example. 
 #60357 should fix this. I've choosen the somewhat smaller refactoring and not pushed the combination logic back into the "combination objects". In fact #59141 removed `_Concatenator` in favor of simple functions. Therefore, I've now choosen the common API to be "provides the inputs via `input_objs` parameter".

Note that `join()` is implemented via `concat()` or `merge()` depending on the case. I've only added explicit tests for these fundamental operations, not for `join()`, but could add that if desired.
> This should be handled the same way as `concat`: Propagate only if all inputs have the same attrs.

I disagree because concat is a symmetric operation operating on a list of dfs but join is an operation that joins other data to a specific dataframe. The properties of the initial dataframe should then be maintained unless pandas is instructed otherwise. It's as if you start dropping rows from your dataframe after a join just because the added data has some NaNs.
I recommend to be defensive on all operations that have multiple DataFrames as input. The inputs may have contradicting Information in their attrs and joining them can lead to misinterpretation. Say, `df1.attrs['date'] = "1.1.2024"; df2.attrs['date'] = "1.1.2025"` then, having one of the dates put on the join is not fair.

Instead, I propose to implement a `set_attrs()` helper as proposed in https://github.com/pandas-dev/pandas/issues/52166#issuecomment-2483031457 to enable easy explicit handling of attrs by the user.


Something like set_attrs() saves half a line of boilerplate code. But the main cost is cognitive: users constantly need to chase after the attrs to make sure they are copied. If the manual says that most operations copy them, I would expect any operation to copy those attrs that look like a reasonable default. Otherwise I expect the manual to say that attrs are usually dropped unless pandas is absolutely sure that the user wants to maintain the attrs.

I would even argue that on a concat the most sensible thing to do would be to merge the attrs by field and throw a warning in case of conflicting fields.
We can possibly improve the docs.

Concerning the behavior, there are two aspects stemming from the experimental state of the feature
- The intended behavior is not fully defined.
- The implementation is incomplete.

This has lead to the suggestion to deprecate `attrs` (#52166). I've [argued against the deprecation](https://github.com/pandas-dev/pandas/issues/52166#issuecomment-1661675919) because I believe attrs are valuable, but the topic is still an experimental and incomplete feature. That's also the reason for the vague "most operations copy" wording.

My personal take (as a non-pandas developer but advocate for attrs) on the intended behavior is to be very defensive and minimal:
- Operations `DataFrame -> DataFrame` should copy the attrs
- Operations `[DataFrame, DataFrame, ...] -> DataFrame` should copy the attrs only if all inputs are identical. - In the face of ambiguity, refuse the temptation to guess.
- Copies are always deep to prevent accidental modification of other DataFrames 



I'm supportive of @timhoffm's proposal. Users can also override this behavior via subclassing.

```python
import functools as ft

import pandas as pd

class Foo(pd.DataFrame):
    @property
    def _constructor(self):
        return Foo

    @ft.wraps(DataFrame.join)
    def join(self, other, *args, **kwargs):
        self_attrs = self.attrs
        other_attrs = other.attrs
        result = super().join(other, *args, **kwargs)
        result.attrs = self_attrs | other_attrs
        return result

df1 = Foo({"a": [1, 2, 3], "b": [3, 4, 5]}).set_index("a")
df1.attrs = {"key1": "value1"}
df2 = Foo({"a": [2, 3, 4], "c": [6, 7, 8]}).set_index("a")
df2.attrs = {"key2": "value2"}
result = df1.join(df2, how="inner")
print(result)
#    b  c
# a      
# 2  4  6
# 3  5  7
print(result.attrs)
# {'key1': 'value1', 'key2': 'value2'}
```

Perhaps that's not helpful in all cases, but hopefully is in many.

Commit message: Add release note

Patch to review:
diff --git a/doc/source/whatsnew/v2.3.0.rst b/doc/source/whatsnew/v2.3.0.rst
index a7f4c53b42f58..1073c72e5a17a 100644
--- a/doc/source/whatsnew/v2.3.0.rst
+++ b/doc/source/whatsnew/v2.3.0.rst
@@ -32,6 +32,9 @@ enhancement1
 Other enhancements
 ^^^^^^^^^^^^^^^^^^
 
+- :func:`pandas.merge` propagates the ``attrs`` attribute to the result if all
+  inputs have identical ``attrs``, as has so far already been the case for
+  :func:`pandas.copy`.
 - :meth:`pandas.api.interchange.from_dataframe` now uses the `PyCapsule Interface <https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html>`_ if available, only falling back to the Dataframe Interchange Protocol if that fails (:issue:`60739`)
 - The semantics for the ``copy`` keyword in ``__array__`` methods (i.e. called
   when using ``np.array()`` or ``np.asarray()`` on pandas objects) has been
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 8aae4609b1833..66188d9e91232 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -330,8 +330,8 @@ def attrs(self) -> dict[Hashable, Any]:
         -----
         Many operations that create new datasets will copy ``attrs``. Copies
         are always deep so that changing ``attrs`` will only affect the
-        present dataset. ``pandas.concat`` copies ``attrs`` only if all input
-        datasets have the same ``attrs``.
+        present dataset. :func:`pandas.concat` and :func:`pandas.merge` will
+        only copy ``attrs`` if all input datasets have the same ``attrs``.
 
         Examples
         --------
@@ -6090,11 +6090,11 @@ def __finalize__(self, other, method: str | None = None, **kwargs) -> Self:
                 assert isinstance(name, str)
                 object.__setattr__(self, name, getattr(other, name, None))
 
-        if method == "concat":
-            objs = other.objs
-            # propagate attrs only if all concat arguments have the same attrs
+        elif hasattr(other, "input_objs"):
+            objs = other.input_objs
+            # propagate attrs only if all inputs have the same attrs
             if all(bool(obj.attrs) for obj in objs):
-                # all concatenate arguments have non-empty attrs
+                # all inputs have non-empty attrs
                 attrs = objs[0].attrs
                 have_same_attrs = all(obj.attrs == attrs for obj in objs[1:])
                 if have_same_attrs:
diff --git a/pandas/core/reshape/concat.py b/pandas/core/reshape/concat.py
index 5efaf0dc051bd..cd7cc33e9ae7f 100644
--- a/pandas/core/reshape/concat.py
+++ b/pandas/core/reshape/concat.py
@@ -550,7 +550,7 @@ def _get_result(
             result = sample._constructor_from_mgr(mgr, axes=mgr.axes)
             result._name = name
             return result.__finalize__(
-                types.SimpleNamespace(objs=objs), method="concat"
+                types.SimpleNamespace(input_objs=objs), method="concat"
             )
 
         # combine as columns in a frame
@@ -571,7 +571,9 @@ def _get_result(
             )
             df = cons(data, index=index, copy=False)
             df.columns = columns
-            return df.__finalize__(types.SimpleNamespace(objs=objs), method="concat")
+            return df.__finalize__(
+                types.SimpleNamespace(input_objs=objs), method="concat"
+            )
 
     # combine block managers
     else:
@@ -610,7 +612,7 @@ def _get_result(
         )
 
         out = sample._constructor_from_mgr(new_data, axes=new_data.axes)
-        return out.__finalize__(types.SimpleNamespace(objs=objs), method="concat")
+        return out.__finalize__(types.SimpleNamespace(input_objs=objs), method="concat")
 
 
 def new_axes(
diff --git a/pandas/core/reshape/merge.py b/pandas/core/reshape/merge.py
index 6de3d5c3f9c09..72f25ffee9652 100644
--- a/pandas/core/reshape/merge.py
+++ b/pandas/core/reshape/merge.py
@@ -10,6 +10,7 @@
 )
 import datetime
 from functools import partial
+import types
 from typing import (
     TYPE_CHECKING,
     Literal,
@@ -1134,7 +1135,10 @@ def get_result(self) -> DataFrame:
         join_index, left_indexer, right_indexer = self._get_join_info()
 
         result = self._reindex_and_concat(join_index, left_indexer, right_indexer)
-        result = result.__finalize__(self, method=self._merge_type)
+        result = result.__finalize__(
+            types.SimpleNamespace(input_objs=[self.left, self.right]),
+            method=self._merge_type,
+        )
 
         if self.indicator:
             result = self._indicator_post_merge(result)
@@ -1143,7 +1147,9 @@ def get_result(self) -> DataFrame:
 
         self._maybe_restore_index_levels(result)
 
-        return result.__finalize__(self, method="merge")
+        return result.__finalize__(
+            types.SimpleNamespace(input_objs=[self.left, self.right]), method="merge"
+        )
 
     @final
     @cache_readonly
diff --git a/pandas/tests/frame/test_api.py b/pandas/tests/frame/test_api.py
index 2b0bf1b0576f9..af61cd75c4540 100644
--- a/pandas/tests/frame/test_api.py
+++ b/pandas/tests/frame/test_api.py
@@ -315,7 +315,7 @@ def test_attrs(self):
         result = df.rename(columns=str)
         assert result.attrs == {"version": 1}
 
-    def test_attrs_deepcopy(self):
+    def test_attrs_is_deepcopy(self):
         df = DataFrame({"A": [2, 3]})
         assert df.attrs == {}
         df.attrs["tags"] = {"spam", "ham"}
@@ -324,6 +324,30 @@ def test_attrs_deepcopy(self):
         assert result.attrs == df.attrs
         assert result.attrs["tags"] is not df.attrs["tags"]
 
+    def test_attrs_concat(self):
+        # concat propagates attrs if all input attrs are equal
+        df1 = DataFrame({"A": [2, 3]})
+        df1.attrs = {"a": 1, "b": 2}
+        df2 = DataFrame({"A": [4, 5]})
+        df2.attrs = df1.attrs.copy()
+        df3 = DataFrame({"A": [6, 7]})
+        df3.attrs = df1.attrs.copy()
+        assert pd.concat([df1, df2, df3]).attrs == df1.attrs
+        # concat does not propagate attrs if input attrs are different
+        df2.attrs = {"c": 3}
+        assert pd.concat([df1, df2, df3]).attrs == {}
+
+    def test_attrs_merge(self):
+        # merge propagates attrs if all input attrs are equal
+        df1 = DataFrame({"key": ["a", "b"], "val1": [1, 2]})
+        df1.attrs = {"a": 1, "b": 2}
+        df2 = DataFrame({"key": ["a", "b"], "val2": [3, 4]})
+        df2.attrs = df1.attrs.copy()
+        assert pd.merge(df1, df2).attrs == df1.attrs
+        # merge does not propagate attrs if input attrs are different
+        df2.attrs = {"c": 3}
+        assert pd.merge(df1, df2).attrs == {}
+
     @pytest.mark.parametrize("allows_duplicate_labels", [True, False, None])
     def test_set_flags(
         self,
diff --git a/pandas/tests/generic/test_duplicate_labels.py b/pandas/tests/generic/test_duplicate_labels.py
index cfa3cabbc1747..d0d1087dd8a08 100644
--- a/pandas/tests/generic/test_duplicate_labels.py
+++ b/pandas/tests/generic/test_duplicate_labels.py
@@ -164,7 +164,6 @@ def test_concat(self, objs, kwargs):
                     allows_duplicate_labels=False
                 ),
                 False,
-                marks=not_implemented,
             ),
             # false true false
             pytest.param(
@@ -173,7 +172,6 @@ def test_concat(self, objs, kwargs):
                 ),
                 pd.DataFrame({"B": [0, 1]}, index=["a", "d"]),
                 False,
-                marks=not_implemented,
             ),
             # true true true
             (
@@ -296,7 +294,6 @@ def test_concat_raises(self):
         with pytest.raises(pd.errors.DuplicateLabelError, match=msg):
             pd.concat(objs, axis=1)
 
-    @not_implemented
     def test_merge_raises(self):
         a = pd.DataFrame({"A": [0, 1, 2]}, index=["a", "b", "c"]).set_flags(
             allows_duplicate_labels=False
diff --git a/pandas/tests/generic/test_frame.py b/pandas/tests/generic/test_frame.py
index 1d0f491529b56..e927c17eceb76 100644
--- a/pandas/tests/generic/test_frame.py
+++ b/pandas/tests/generic/test_frame.py
@@ -80,12 +80,16 @@ def test_metadata_propagation_indiv(self, monkeypatch):
         def finalize(self, other, method=None, **kwargs):
             for name in self._metadata:
                 if method == "merge":
-                    left, right = other.left, other.right
+                    left, right = other.input_objs
                     value = getattr(left, name, "") + "|" + getattr(right, name, "")
                     object.__setattr__(self, name, value)
                 elif method == "concat":
                     value = "+".join(
-                        [getattr(o, name) for o in other.objs if getattr(o, name, None)]
+                        [
+                            getattr(o, name)
+                            for o in other.input_objs
+                            if getattr(o, name, None)
+                        ]
                     )
                     object.__setattr__(self, name, value)
                 else:
diff --git a/pandas/tests/generic/test_series.py b/pandas/tests/generic/test_series.py
index 7dcdcd96cce51..2ea4b8be5cf91 100644
--- a/pandas/tests/generic/test_series.py
+++ b/pandas/tests/generic/test_series.py
@@ -97,7 +97,7 @@ def finalize(self, other, method=None, **kwargs):
                     value = "+".join(
                         [
                             getattr(obj, name)
-                            for obj in other.objs
+                            for obj in other.input_objs
                             if getattr(obj, name, None)
                         ]
                     )


Code review comments:
diff_hunk: @@ -32,6 +32,9 @@ enhancement1
 Other enhancements
 ^^^^^^^^^^^^^^^^^^
 
+- :func:`pandas.merge` propagates the ``attrs`` attribute to the result if all
+  inputs have identical ``attrs``, as has so far already been the case for
+  :func:`pandas.copy`.
path: doc/source/whatsnew/v2.3.0.rst
text: @user1:
```suggestion
  :func:`pandas.concat`.
```

Also could you move this to `v3.0.0.rst`?