Repository: dask/dask

Issue:
Dropped Support for Custom Metadata Types
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:

A recent pull request titled "Fail if meta is not a pandas object": https://github.com/dask/dask/pull/8563 breaks support for custom metadata types in dask. This pull request was motivated by the issue: https://github.com/dask/dask/issues/8537.

In our codebase, we use a custom Python class which is iterable as the dask metadata value. I would like to kindly ask if this dropped support is intentional. According to the [documentation](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.map_partitions.html) of `DataFrame.map_partitions`, the `meta` parameter may be one of "pd.DataFrame, pd.Series, dict, iterable, tuple".

More specifically, this relates to the [Kartothek library](https://github.com/JDASoftwareGroup/kartothek) with this affected [custom metadata class](https://github.com/JDASoftwareGroup/kartothek/blob/1821ea5df60d4079d3911b3c2f17be11d8780e22/kartothek/io_components/metapartition.py#L220).

**What you expected to happen**:

The expected behavior is to maintain support for iterable custom metadata classes, as dask did before the aforementioned pull request. This change breaks our code.

**Minimal Complete Verifiable Example**:

Below is a simplified code snippet reproducing this new behavior.

```python
from typing import Iterable

import pandas as pd
import dask.dataframe as dd

class CustomMetadata(Iterable):
    """Custom class iterator returning pandas types."""

    def __init__(self, max=0):
        self.types = [(None, "f8")]

    def __iter__(self):
        self.n = 0
        return self

    def __next__(self):
        if self.n < len(self.types):
            ret = self.types[self.n]
            self.n += 1
            return ret
        else:
            raise StopIteration

df = pd.DataFrame({'x': [1, 2, 3, 4, 5], 'y': [1., 2., 3., 4., 5.]})
ddf = dd.from_pandas(df, npartitions=2)

def myadd(df, a, b=1):
    return df.x + df.y + a + b

# Works just fine
mapped_ddf = ddf.map_partitions(myadd, 1, b=2, meta=[(None, "f8")])
print(mapped_ddf)
# Dask DataFrame Structure:
#                      0
# npartitions=2
# 0              float64
# 3                  ...
# 4                  ...
# Dask Name: myadd, 4 tasks

# Fails
mapped_ddf_custom_meta = ddf.map_partitions(myadd, 1, b=2, meta=CustomMetadata)
print(mapped_ddf_custom_meta)
# Traceback (most recent call last):
#   File "/home/X931300/debugging/playground_dask/dask_metadata.py", line 35, in <module>
#     mapped_ddf_custom_meta = ddf.map_partitions(myadd, 1, b=2, meta=CustomMetadata)
#   File "/home/X931300/debugging/dask/dask/dataframe/core.py", line 772, in map_partitions
#     return map_partitions(func, self, *args, **kwargs)
#   File "/home/X931300/debugging/dask/dask/dataframe/core.py", line 6060, in map_partitions
#     raise ValueError(
# ValueError: Meta is not valid, `map_partitions` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns.
#
#
# Before the pull request #8563, the output was like above:
# Dask Series Structure:
# npartitions=2
# 0    object
# 3       ...
# 4       ...
# dtype: object
# Dask Name: myadd, 4 tasks
```

**Environment**:

- Dask version: 2022.1.0+27.g95e1cf31 (I pulled the most recent code on main and built/installed it locally)
- Python version: 3.10.2
- Operating System: Ubuntu 20.04.3 LTS
- Install method (conda, pip, source): source


Discussion:
Thanks for opening this @DamianBarabonkovQC this was definitely not an intentional effect. I will change it to accept an iterable an ping you on the PR.

Resolving patch:
diff --git a/dask/dataframe/backends.py b/dask/dataframe/backends.py
--- a/dask/dataframe/backends.py
+++ b/dask/dataframe/backends.py
@@ -1,4 +1,5 @@
 import warnings
+from typing import Iterable
 
 import numpy as np
 import pandas as pd
@@ -126,7 +127,7 @@ def make_meta_object(x, index=None):
         )
     if isinstance(x, tuple) and len(x) == 2:
         return _empty_series(x[0], x[1], index=index)
-    elif isinstance(x, (list, tuple)):
+    elif isinstance(x, Iterable) and not isinstance(x, str):
         if not all(isinstance(i, tuple) and len(i) == 2 for i in x):
             raise ValueError(f"Expected iterable of tuples of (name, dtype), got {x}")
         return pd.DataFrame(


Test patch:
diff --git a/dask/dataframe/tests/test_utils_dataframe.py b/dask/dataframe/tests/test_utils_dataframe.py
--- a/dask/dataframe/tests/test_utils_dataframe.py
+++ b/dask/dataframe/tests/test_utils_dataframe.py
@@ -1,4 +1,5 @@
 import re
+from typing import Iterable
 
 import numpy as np
 import pandas as pd
@@ -69,7 +70,7 @@ def test_make_meta():
     assert (meta.dtypes == df.dtypes).all()
     assert isinstance(meta.index, pd.RangeIndex)
 
-    # Iterable
+    # List
     meta = make_meta([("a", "i8"), ("c", "f8"), ("b", "O")])
     assert (meta.columns == ["a", "c", "b"]).all()
     assert len(meta) == 0
@@ -83,6 +84,31 @@ def test_make_meta():
     assert meta.dtype == "i8"
     assert meta.name == "a"
 
+    # Iterable
+    class CustomMetadata(Iterable):
+        """Custom class iterator returning pandas types."""
+
+        def __init__(self, max=0):
+            self.types = [("a", "i8"), ("c", "f8"), ("b", "O")]
+
+        def __iter__(self):
+            self.n = 0
+            return self
+
+        def __next__(self):
+            if self.n < len(self.types):
+                ret = self.types[self.n]
+                self.n += 1
+                return ret
+            else:
+                raise StopIteration
+
+    meta = make_meta(CustomMetadata())
+    assert (meta.columns == ["a", "c", "b"]).all()
+    assert len(meta) == 0
+    assert (meta.dtypes == df.dtypes[meta.dtypes.index]).all()
+    assert isinstance(meta.index, pd.RangeIndex)
+
     # With index
     idx = pd.Index([1, 2], name="foo")
     meta = make_meta(