REF: Store metadata in an attrs dict
State: closed

This aids in the implementation of
https://github.com/pandas-dev/pandas/pull/28394. Over there, I'm having
issues with using `NDFrame.__finalize__` to copy attributes, in part
because getattribute on NDFrame is so complicated.

This simplifies that PR because we only need to look in NDFrame.attrs (name borrowed from xarray), 
which is just a plain dictionary.

Aside from the addition of a public NDFrame.attrs dictionary, there
aren't any user-facing API changes.

Labels:
name: Refactor
description: Internal refactoring of code
name: metadata
description: _metadata, .attrs

Comments:
do you want to add docs anywhere / whatsnew? or just keep private mention for now?
> do you want to add docs anywhere / whatsnew? or just keep private mention for now?

I pushed some basics API docs. Will want a larger section once things settle I think.
thanks @TomAugspurger very nice!
In xarray, we currently use duck-typing to check for `.attrs` attributes in the `xarray.DataArray` constructor. So this PR broke our tests (see https://github.com/pydata/xarray/issues/3440), but that's probably our own fault.

My bigger concern is that I noticed that `attrs` is used in different way in pandas than xarray: `name` is a member of `attrs` vs. in xarray where `DataArray.name` is a separate attribute. This is not an unreasonable thing to do, but it does introduce some minor incompatible: in xarray (and h5py) `attrs` really is an arbitrary mapping, but that isn't quite true in pandas (because  `name` needs to be hashable). Also, what happens if you try to delete `name` from the `attrs` dict?
> Also, what happens if you try to delete name from the attrs dict?

From a quick test, it seems this deletes the name (`s.name` becomes `None`); which I *suppose* is the expected behaviour?

But you can indeed set it to a non-hashable value that way, which doesn't directly raise an error, but can confusingly give errors in later operations:

```
In [9]: s = pd.Series([1, 2, 3], name='a')  

In [10]: s.attrs  
Out[10]: {'name': 'a'}

In [12]: s.attrs['name'] = [1, 2] 

In [13]: s 
Out[13]: 
0    1
1    2
2    3
Name: [1, 2], dtype: int64

In [14]: s.to_frame()  
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-78ea8628cc3c> in <module>
----> 1 s.to_frame()

~/scipy/pandas/pandas/core/series.py in to_frame(self, name)
   1533         """
   1534         if name is None:
-> 1535             df = self._constructor_expanddim(self)
   1536         else:
   1537             df = self._constructor_expanddim({name: self})

~/scipy/pandas/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    456                 data_columns = list(data.dtype.names)
    457                 data = {k: data[k] for k in data_columns}
--> 458                 if columns is None:
    459                     columns = data_columns
    460                 mgr = init_dict(data, index, columns, dtype=dtype)

TypeError: unhashable type: 'list'
```
Do we need to further discuss @shoyer's concern?

I was planning to try this out in GeoPandas (to see if it can replace our usage of `_metadata`), to be able to give feedback on this feature that way, but didn't get to it yet.
Sorry I missed @shoyer's comment. I'll take a look.
Hi, not sure this is the best issue for my comments below, or too early, but I've been wanting for a while to store metadata along with Series or Dataframes. I've tried the subclassing approach, with `_constructor_expanddim`, `_constructor` etc., but eventually there was always some method that lead to the loss of my metadata. It seems that this new approach doesn't really solve the problem either (not sure if it's supposed to work like this):

```python
In [17]: s = pd.Series([0,1,2], name="x")                                                                                                                                                                                                                                       
In [18]: s.attrs["mydata"] = "test"                                                                                                                                                                                                                                             
In [19]: s.attrs                                                                                                                                                                                                                                                                
Out[19]: {'mydata': 'test'}

In [20]: df = s.to_frame()                                                                                                                                                                                                                                                      
In [21]: df.x.attrs                                                                                                                                                                                                                                                             
Out[21]: {}
```

Is this supposed to work? Or do I misinterpret the purpose of `attrs`?

Edit: 
For reference, it seems it may only be the "expanddim" direction from Series to DF doesn't seem to work (using s.to_frame() or the Dataframe constructor directly), while setting a column's attrs that already exists in a DF, and indexing or manipulating etc. seems to maintain the metadata correctly (though I haven't tested more complicated cases like aggregations etc.).
Pandas will need to call __finalize__ in more places to ensure the metadata is propagated.
@buhrmann can you open a new issue with this specific use case?
Is it appearing likely that `DataFrame.attrs` is here to stay? I'm considering using this for metadata used to differentiate DataFrames of two forms in [alchemlyb](https://alchemlyb.readthedocs.io/en/latest/parsing.html#standard-forms-of-raw-data), but wanted to check the temperature of the pandas devs on this question before relying on it. So far it works as expected (e.g. it survives such things as `DataFrame.copy()`)!