Repository: scrapy/itemadapter

Issue:
Recursive conversion to a dictionary
It would be cool to be able to convert an item to a nested dictionary. Both with dataclasses and with attr.s you can define nested data:


```py
import attr

@attr.s(auto_attribs=True)
class Price:
    value: float   # fixme: Decimal?
    currency: str

@attr.s(auto_attribs=True)
class Product:
    name: str
    price: Price 
```

Such structures are common in real-world usage of these packages. For example, attr.s provides an utility to convert a nested structure to a dict: https://www.attrs.org/en/stable/api.html#attr.asdict. 

That's fine to add this feature later, but my worry is that the currently suggested way to convert ItemAdapter to a dict (`dict(adapter)`) is not future-proof; I'm not sure how to add nesting support to it in future, if we decide to. Likely it would require updating of all examples.

So, what do you think about adding a method (something like `ItemAdapter.asdict()`), and recommending it? It may support recursive export from the day 1, or just call `dict(self)` - in this case `recursive=True` flag can be added later.


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=h1) Report
> Merging [#29](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/itemadapter/commit/ec413c5da78e9aa9293cee5f3c71bcec8b1bc46c&el=desc) will **not change** coverage.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/scrapy/itemadapter/pull/29/graphs/tree.svg?width=650&height=150&src=pr&token=MDGbx42SRs)](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##            master       #29   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            2         2           
  Lines           98       108   +10     
=========================================
+ Hits            98       108   +10     
```


| [Impacted Files](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [itemadapter/adapter.py](https://codecov.io/gh/scrapy/itemadapter/pull/29/diff?src=pr&el=tree#diff-aXRlbWFkYXB0ZXIvYWRhcHRlci5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=footer). Last update [ec413c5...3493f75](https://codecov.io/gh/scrapy/itemadapter/pull/29?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/README.md b/README.md
index 01d95de..ecac8be 100644
--- a/README.md
+++ b/README.md
@@ -82,16 +82,40 @@ InventoryItem(name='bar', price=12.7, stock=9)
 True
 ```
 
-### Converting to dictionary
+### Converting to dict
 
-Passing an `ItemAdapter` to the `dict` built-in will create a new dictionary:
+The `ItemAdapter` class provides the `asdict` method, which converts
+nested items recursively. Consider the following example:
 
 ```python
->>> dict(adapter)
-{'name': 'bar', 'price': 12.7, 'stock': 9}
+from dataclasses import dataclass
+from itemadapter import ItemAdapter
+
+@dataclass
+class Price:
+    value: int
+    currency: str
+
+@dataclass
+class Product:
+    name: str
+    price: Price
 ```
 
-For more examples using different types, refer to the [examples section](#more-examples) below.
+```python
+>>> item = Product("Stuff", Price(42, "UYU"))
+>>> adapter = ItemAdapter(item)
+>>> adapter.asdict()
+{'name': 'Stuff', 'price': {'currency': 'UYU', 'value': 42}}
+```
+
+Note that just passing an adapter object to the `dict` built-in also works,
+but it doesn't traverse the object recursively converting nested items:
+
+```python
+>>> dict(adapter)
+{'name': 'Stuff', 'price': Price(value=42, currency='UYU')}
+```
 
 
 ## Public API
@@ -105,7 +129,7 @@ _class `itemadapter.adapter.ItemAdapter(item: Any)`_
 providing a `dict`-like API to manipulate data for the object it wraps
 (which is modified in-place).
 
-Two additional methods are available:
+Some additional methods are available:
 
 `get_field_meta(field_name: str) -> MappingProxyType`
 
@@ -127,6 +151,11 @@ for `scrapy.item.Item`s
 Return a [keys view](https://docs.python.org/3/library/collections.abc.html#collections.abc.KeysView)
 with the names of all the defined fields for the item.
 
+`asdict() -> dict`
+
+Return a `dict` object with the contents of the adapter. This works slightly different than
+calling `dict(adapter)`, because it's applied recursively to nested items (if there are any).
+
 ### `is_item` function
 
 _`itemadapter.utils.is_item(obj: Any) -> bool`_
diff --git a/itemadapter/adapter.py b/itemadapter/adapter.py
index 06653bd..38d06c5 100644
--- a/itemadapter/adapter.py
+++ b/itemadapter/adapter.py
@@ -120,3 +120,26 @@ class ItemAdapter(MutableMapping):
             return KeysView(self._fields_dict)
         else:
             return KeysView(self.item)
+
+    def asdict(self) -> dict:
+        """
+        Return a dict object with the contents of the adapter. This works slightly different than
+        calling `dict(adapter)`: it's applied recursively to nested items (if there are any).
+        """
+        return {key: _asdict(value) for key, value in self.items()}
+
+
+def _asdict(obj: Any) -> Any:
+    """
+    Helper for ItemAdapter.asdict
+    """
+    if isinstance(obj, dict):
+        return {key: _asdict(value) for key, value in obj.items()}
+    elif isinstance(obj, (list, set, tuple)):
+        return obj.__class__(_asdict(x) for x in obj)
+    elif isinstance(obj, ItemAdapter):
+        return obj.asdict()
+    elif is_item(obj):
+        return ItemAdapter(obj).asdict()
+    else:
+        return obj


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
index 988abdd..2adf70d 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -1,7 +1,11 @@
+from itemadapter.adapter import ItemAdapter
+
+
 try:
     import attr
 except ImportError:
     AttrsItem = None
+    AttrsItemNested = None
 else:
 
     @attr.s
@@ -9,11 +13,22 @@ else:
         name = attr.ib(default=None, metadata={"serializer": str})
         value = attr.ib(default=None, metadata={"serializer": int})
 
+    @attr.s
+    class AttrsItemNested:
+        nested = attr.ib(type=AttrsItem)
+        adapter = attr.ib(type=ItemAdapter)
+        dict_ = attr.ib(type=dict)
+        list_ = attr.ib(type=list)
+        set_ = attr.ib(type=set)
+        tuple_ = attr.ib(type=tuple)
+        int_ = attr.ib(type=int)
+
 
 try:
     from dataclasses import make_dataclass, field
 except ImportError:
     DataClassItem = None
+    DataClassItemNested = None
 else:
     DataClassItem = make_dataclass(
         "DataClassItem",
@@ -23,14 +38,37 @@ else:
         ],
     )
 
+    DataClassItemNested = make_dataclass(
+        "DataClassItem",
+        [
+            ("nested", DataClassItem),
+            ("adapter", ItemAdapter),
+            ("dict_", dict),
+            ("list_", list),
+            ("set_", set),
+            ("tuple_", tuple),
+            ("int_", int),
+        ],
+    )
+
 
 try:
     from scrapy.item import Item as ScrapyItem, Field
 except ImportError:
     ScrapyItem = None
     ScrapySubclassedItem = None
+    ScrapySubclassedItemNested = None
 else:
 
     class ScrapySubclassedItem(ScrapyItem):
         name = Field(serializer=str)
         value = Field(serializer=int)
+
+    class ScrapySubclassedItemNested(ScrapyItem):
+        nested = Field()
+        adapter = Field()
+        dict_ = Field()
+        list_ = Field()
+        set_ = Field()
+        tuple_ = Field()
+        int_ = Field()
diff --git a/tests/test_adapter.py b/tests/test_adapter.py
index 1bc7ac0..f7e2186 100644
--- a/tests/test_adapter.py
+++ b/tests/test_adapter.py
@@ -4,7 +4,14 @@ from typing import KeysView
 
 from itemadapter.adapter import ItemAdapter
 
-from tests import AttrsItem, DataClassItem, ScrapySubclassedItem
+from tests import (
+    AttrsItem,
+    AttrsItemNested,
+    DataClassItem,
+    DataClassItemNested,
+    ScrapySubclassedItem,
+    ScrapySubclassedItemNested,
+)
 
 
 class ItemAdapterReprTestCase(unittest.TestCase):
@@ -52,6 +59,7 @@ class ItemAdapterInitError(unittest.TestCase):
 class BaseTestMixin:
 
     item_class = None
+    item_class_nested = None
 
     def setUp(self):
         if self.item_class is None:
@@ -87,6 +95,30 @@ class BaseTestMixin:
         adapter = ItemAdapter(item)
         self.assertEqual(dict(name="asdf", value=1234), dict(adapter))
 
+    def test_as_dict_nested(self):
+        item = self.item_class_nested(
+            nested=self.item_class(name="asdf", value=1234),
+            adapter=ItemAdapter(dict(foo="bar", nested_list=[1, 2, 3, 4, 5])),
+            dict_={"foo": "bar", "answer": 42, "nested_dict": {"a": "b"}},
+            list_=[1, 2, 3],
+            set_={1, 2, 3},
+            tuple_=(1, 2, 3),
+            int_=123,
+        )
+        adapter = ItemAdapter(item)
+        self.assertEqual(
+            adapter.asdict(),
+            dict(
+                nested=dict(name="asdf", value=1234),
+                adapter=dict(foo="bar", nested_list=[1, 2, 3, 4, 5]),
+                dict_={"foo": "bar", "answer": 42, "nested_dict": {"a": "b"}},
+                list_=[1, 2, 3],
+                set_={1, 2, 3},
+                tuple_=(1, 2, 3),
+                int_=123,
+            ),
+        )
+
     def test_field_names(self):
         item = self.item_class(name="asdf", value=1234)
         adapter = ItemAdapter(item)
@@ -138,6 +170,7 @@ class NonDictTestMixin(BaseTestMixin):
 class DictTestCase(unittest.TestCase, BaseTestMixin):
 
     item_class = dict
+    item_class_nested = dict
 
     def test_get_value_keyerror_item_dict(self):
         """Instantiate without default values"""
@@ -161,6 +194,7 @@ class DictTestCase(unittest.TestCase, BaseTestMixin):
 class ScrapySubclassedItemTestCase(NonDictTestMixin, unittest.TestCase):
 
     item_class = ScrapySubclassedItem
+    item_class_nested = ScrapySubclassedItemNested
 
     def test_get_value_keyerror_item_dict(self):
         """Instantiate without default values"""
@@ -172,8 +206,10 @@ class ScrapySubclassedItemTestCase(NonDictTestMixin, unittest.TestCase):
 class DataClassItemTestCase(NonDictTestMixin, unittest.TestCase):
 
     item_class = DataClassItem
+    item_class_nested = DataClassItemNested
 
 
 class AttrsItemTestCase(NonDictTestMixin, unittest.TestCase):
 
     item_class = AttrsItem
+    item_class_nested = AttrsItemNested