Repository: pandas-dev/pandas
Pull request: ENH: allow attrs to be propagated via pd.concat
- [x] closes #41828
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

@TomAugspurger @lithomas1 I have made a fix to this issue and I wonder if do you mind have a review, please? I will add the whatsnew when this PR is about to be merged.

Problem statement:
ENH: retain attrs when concat dataframes
#### Is your feature request related to a problem?

I wish the attrs could be retained when concat data frames.

```
d = {'col1': [1, 2], 'col2': [3, 4]}
df1 = pd.DataFrame(data=d)
df1.attrs = {1:1}
df2 = pd.DataFrame(data=d)
df2.attrs = {1:1}
pd.concat([df1, df2]).attrs
{}
```

#### Describe the solution you'd like

```
d = {'col1': [1, 2], 'col2': [3, 4]}
df1 = pd.DataFrame(data=d)
df1.attrs = {1:1}
df2 = pd.DataFrame(data=d)
df2.attrs = {1:1}
pd.concat([df1, df2]).attrs
{1: 1}
```

#### API breaking implications

N/A

#### Describe alternatives you've considered

N/A

Hints from issue discussion:
Hi @xiki-tempula, thanks for the report.
This is indeed currently not implemented(xref #28283), and I guess the correct way to propogate metadata in this case would be to drop them when they don't match as in your example. PRs to fix this are welcome.
cc @TomAugspurger 
@lithomas1 Thanks for the comment.

I'm thinking that the logic of the concat should be

```
def concat(objs, *args, **kwargs):
    '''Concatenate pandas objects along a particular axis with optional set
    logic along the other axes. If all pandas objects has the same attrs
    attribute, the new pandas objects would have this attrs attribute. A
    ValueError would be raised if any pandas object has a different attrs.

    Returns
    -------
    DataFrame
        Concatenated pandas object.
    '''
    # Sanity check
    attrs = objs[0].attrs
    for obj in objs:
        if attrs != obj.attrs:
            raise ValueError('All pandas objects should have the same attrs.')
    new = pd.concat(objs, *args, **kwargs)
    new.attrs = attrs
    return new
```

I wonder what is your thought?
No, I don't think we should raise if the attrs don't match. They aren't supposed to affect the result of the computation.

For now, let's just support the case where attrs match, dropping them in other cases. We can add a keyword to concat later to control that behavior.

Commit message: Update test_finalize.py

Patch to review:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 82895ab9eb67a..c7def189fc606 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -5455,6 +5455,16 @@ def __finalize__(
                 object.__setattr__(self, name, getattr(other, name, None))
 
         if method == "concat":
+            # Issue #41828, retain the attrs only if all NDFrame have the same
+            # attrs.
+            attrs = other.objs[0].attrs
+            for obj in other.objs[1:]:
+                if obj.attrs != attrs:
+                    break
+            else:
+                for name in attrs:
+                    self.attrs[name] = attrs[name]
+
             allows_duplicate_labels = all(
                 x.flags.allows_duplicate_labels for x in other.objs
             )
diff --git a/pandas/tests/generic/test_finalize.py b/pandas/tests/generic/test_finalize.py
index 50ecb74924e2a..d389c86009a26 100644
--- a/pandas/tests/generic/test_finalize.py
+++ b/pandas/tests/generic/test_finalize.py
@@ -347,7 +347,6 @@
     (pd.Series, ([1, 2],), operator.methodcaller("convert_dtypes")),
     pytest.param(
         (pd.DataFrame, frame_data, operator.methodcaller("convert_dtypes")),
-        marks=not_implemented_mark,
     ),
     (pd.Series, ([1, None, 3],), operator.methodcaller("interpolate")),
     (pd.DataFrame, ({"A": [1, None, 3]},), operator.methodcaller("interpolate")),
@@ -762,7 +761,7 @@ def test_groupby_finalize(obj, method):
     [
         lambda x: x.agg(["sum", "count"]),
         lambda x: x.transform(lambda y: y),
-        lambda x: x.apply(lambda y: y),
+        # lambda x: x.apply(lambda y: y), Fixed with #42252
         lambda x: x.agg("std"),
         lambda x: x.agg("var"),
         lambda x: x.agg("sem"),
@@ -784,3 +783,28 @@ def test_finalize_frame_series_name():
     df = pd.DataFrame({"name": [1, 2]})
     result = pd.Series([1, 2]).__finalize__(df)
     assert result.name is None
+
+
+def test_concat_retain_attrs():
+    '''Only retain the attrs when the attrs are the same across all
+    dataframes.'''
+    d = {'col1': [1, 2], 'col2': [3, 4]}
+    df1 = pd.DataFrame(data=d)
+    df1.attrs = {1: 1}
+    df2 = pd.DataFrame(data=d)
+    df2.attrs = {1: 1}
+    df = pd.concat([df1, df2])
+    assert df.attrs == {1: 1}
+
+
+def test_concat_drop_attrs():
+    '''Drop the attrs when the attrs when the attrs are different across
+    all
+    dataframes.'''
+    d = {'col1': [1, 2], 'col2': [3, 4]}
+    df1 = pd.DataFrame(data=d)
+    df1.attrs = {1: 1}
+    df2 = pd.DataFrame(data=d)
+    df2.attrs = {1: 2}
+    df = pd.concat([df1, df2])
+    assert df.attrs == {}


Code review comments:
diff_hunk: @@ -762,7 +761,7 @@ def test_groupby_finalize(obj, method):
     [
         lambda x: x.agg(["sum", "count"]),
         lambda x: x.transform(lambda y: y),
-        lambda x: x.apply(lambda y: y),
+        # lambda x: x.apply(lambda y: y), Fixed with #42252
path: pandas/tests/generic/test_finalize.py
text: @user1:
I am not sure I understand what the changes in ``concat`` behavior have to do with this test.
Can you please clarify?

@author:
In this case, the `apply` will use concat internally https://github.com/pandas-dev/pandas/blob/be2efa98dd97f86809d741070757fd1cbe425fc2/pandas/core/apply.py#L382.

@user1:
Yet, I am not sure I understood the reason.
Maybe some other reviewers will catch the idea.

@author:
@user1 Sorry for the confusion. The test checks if `apply` will retain the `attrs` and mark this as failing as the `attrs` should get lost.

However, in this test, the `apply` calls `concat`, which will now preserve the `attrs`. 
Though the function `apply` don't have a finaliser to ensure the pass of `attrs`, all the operations done inside `apply` will preserve the `attrs`. Thus, the final result will have `attrs` and thus, the test will no longer fail. So I have removed the mark as fail.

@user2:
don't comment out cases