Package: itemadapter
itemadapterTheItemAdapterclass is a wrapper for data container objects, providing a
common interface to handle objects of different types in an uniform manner,
regardless of their underlying implementation.Currently supported types are:scrapy.item.Itemdictdataclass-based classesattrs-based classespydantic-based classesAdditionally, interaction with arbitrary types is supported, by implementing
a pre-defined interface (seeextendingitemadapter).RequirementsPython 3.7+scrapy: optional, needed to interact withscrapyitemsattrs: optional, needed to interact withattrs-based itemspydantic: optional, needed to interact withpydantic-based itemsInstallationitemadapteris available onPyPI, it can be installed withpip:pip install itemadapterLicenseitemadapteris distributed under aBSD-3license.Basic usageThe following is a simple example using adataclassobject.
Consider the following type definition:>>>fromdataclassesimportdataclass>>>fromitemadapterimportItemAdapter>>>@dataclass...classInventoryItem:...name:str...price:float...stock:int>>>AnItemAdapterobject can be treated much like a dictionary:>>>obj=InventoryItem(name='foo',price=20.5,stock=10)>>>ItemAdapter.is_item(obj)True>>>adapter=ItemAdapter(obj)>>>len(adapter)3>>>adapter["name"]'foo'>>>adapter.get("price")20.5>>>The wrapped object is modified in-place:>>>adapter["name"]="bar">>>adapter.update({"price":12.7,"stock":9})>>>adapter.itemInventoryItem(name='bar',price=12.7,stock=9)>>>adapter.itemisobjTrue>>>Converting to dictTheItemAdapterclass provides theasdictmethod, which converts
nested items recursively. Consider the following example:>>>fromdataclassesimportdataclass>>>fromitemadapterimportItemAdapter>>>@dataclass...classPrice:...value:int...currency:str>>>@dataclass...classProduct:...name:str...price:Price>>>>>>item=Product("Stuff",Price(42,"UYU"))>>>adapter=ItemAdapter(item)>>>adapter.asdict(){'name':'Stuff','price':{'value':42,'currency':'UYU'}}>>>Note that just passing an adapter object to thedictbuilt-in also works,
but it doesn't traverse the object recursively converting nested items:>>>dict(adapter){'name':'Stuff','price':Price(value=42,currency='UYU')}>>>API referenceBuilt-in adaptersThe following adapters are included by default:itemadapter.adapter.ScrapyItemAdapter: handlesScrapyitemsitemadapter.adapter.DictAdapter: handlesPythondictionariesitemadapter.adapter.DataclassAdapter: handlesdataclassobjectsitemadapter.adapter.AttrsAdapter: handlesattrsobjectsitemadapter.adapter.PydanticAdapter: handlespydanticobjectsclassitemadapter.adapter.ItemAdapter(item: Any)This is the main entrypoint for the package. Tipically, user code
wraps an item using this class, and proceeds to handle it with the provided interface.ItemAdapterimplements theMutableMappinginterface, providing adict-like API to manipulate data for the object it wraps
(which is modified in-place).Attributesclass attributeADAPTER_CLASSES: collections.dequeStores the currently registered adapter classes. Being acollections.deque,
it supports efficient addition/deletion of adapters classes to both ends.The order in which the adapters are registered is important. When anItemAdapterobject is
created for a specific item, the registered adapters are traversed in order and the first
adapter class to returnTruefor theis_itemclass method is used for all subsequent
operations. The default order is the one defined in thebuilt-in adapterssection.See the section onextending itemadapterfor additional information.Methodsclass methodis_item(item: Any) -> boolReturnTrueif any of the registed adapters can handle the item
(i.e. if any of them returnsTruefor itsis_itemmethod withitemas argument),Falseotherwise.class methodis_item_class(item_class: type) -> boolReturnTrueif any of the registered adapters can handle the item class
(i.e. if any of them returnsTruefor itsis_item_classmethod withitem_classas argument),Falseotherwise.class methodget_field_meta_from_class(item_class: type, field_name: str) -> MappingProxyTypeReturn atypes.MappingProxyTypeobject, which is a read-only mapping with metadata about the given field. If the item class does not
support field metadata, or there is no metadata for the given field, an empty object is returned.The returned value is taken from the following sources, depending on the item type:scrapy.item.Fieldforscrapy.item.Itemsdataclasses.field.metadatafordataclass-based itemsattr.Attribute.metadataforattrs-based itemspydantic.fields.FieldInfoforpydantic-based itemsclass methodget_field_names_from_class(item_class: type) -> Optional[list[str]]Return a list with the names of all the fields defined for the item class.
If an item class doesn't support defining fields upfront, None is returned.get_field_meta(field_name: str) -> MappingProxyTypeReturn metadata for the given field, if available. Unless overriden in a custom adapter class, by default
this method calls the adapter'sget_field_meta_from_classmethod, passing the wrapped item's class.field_names() -> collections.abc.KeysViewReturn akeys viewwith the names of all the defined fields for the item.asdict() -> dictReturn adictobject with the contents of the adapter. This works slightly different than
callingdict(adapter), because it's applied recursively to nested items (if there are any).functionitemadapter.utils.is_item(obj: Any) -> boolReturnTrueif the given object belongs to (at least) one of the supported types,Falseotherwise. This is an alias, using theitemadapter.adapter.ItemAdapter.is_itemclass method is encouraged for better performance.functionitemadapter.utils.get_field_meta_from_class(item_class: type, field_name: str) -> types.MappingProxyTypeAlias foritemadapter.adapter.ItemAdapter.get_field_meta_from_classMetadata supportscrapy.item.Item,dataclass,attrs, andpydanticobjects allow the definition of
arbitrary field metadata. This can be accessed through aMappingProxyTypeobject, which can be retrieved from an item instance withitemadapter.adapter.ItemAdapter.get_field_meta, or from an item class
with theitemadapter.adapter.ItemAdapter.get_field_meta_from_classmethod (or its aliasitemadapter.utils.get_field_meta_from_class).
The source of the data depends on the underlying type (see the docs forItemAdapter.get_field_meta_from_class).scrapy.item.Itemobjects>>>fromscrapy.itemimportItem,Field>>>fromitemadapterimportItemAdapter>>>classInventoryItem(Item):...name=Field(serializer=str)...value=Field(serializer=int,limit=100)...>>>adapter=ItemAdapter(InventoryItem(name="foo",value=10))>>>adapter.get_field_meta("name")mappingproxy({'serializer':<class'str'>})>>>adapter.get_field_meta("value")mappingproxy({'serializer':<class'int'>, 'limit': 100})>>>dataclassobjects>>>fromdataclassesimportdataclass,field>>>@dataclass...classInventoryItem:...name:str=field(metadata={"serializer":str})...value:int=field(metadata={"serializer":int,"limit":100})...>>>adapter=ItemAdapter(InventoryItem(name="foo",value=10))>>>adapter.get_field_meta("name")mappingproxy({'serializer':<class'str'>})>>>adapter.get_field_meta("value")mappingproxy({'serializer':<class'int'>, 'limit': 100})>>>attrsobjects>>>importattr>>>@attr.s...classInventoryItem:...name=attr.ib(metadata={"serializer":str})...value=attr.ib(metadata={"serializer":int,"limit":100})...>>>adapter=ItemAdapter(InventoryItem(name="foo",value=10))>>>adapter.get_field_meta("name")mappingproxy({'serializer':<class'str'>})>>>adapter.get_field_meta("value")mappingproxy({'serializer':<class'int'>, 'limit': 100})>>>pydanticobjects>>>frompydanticimportBaseModel,Field>>>classInventoryItem(BaseModel):...name:str=Field(serializer=str)...value:int=Field(serializer=int,limit=100)...>>>adapter=ItemAdapter(InventoryItem(name="foo",value=10))>>>adapter.get_field_meta("name")mappingproxy({'serializer':<class'str'>})>>>adapter.get_field_meta("value")mappingproxy({'serializer':<class'int'>, 'limit': 100})>>>ExtendingitemadapterThis package allows to handle arbitrary item classes, by implementing an adapter interface:classitemadapter.adapter.AdapterInterface(item: Any)Abstract Base Class for adapters. An adapter that handles a specific type of item must
inherit from this class and implement the abstract methods defined on it.AdapterInterfaceinherits fromcollections.abc.MutableMapping,
so all methods from theMutableMappinginterface must be implemented as well.class methodis_item_class(cls, item_class: type) -> boolReturnTrueif the adapter can handle the given item class,Falseotherwise. Abstract (mandatory).class methodis_item(cls, item: Any) -> boolReturnTrueif the adapter can handle the given item,Falseotherwise.
The default implementation callscls.is_item_class(item.__class__).class methodget_field_meta_from_class(cls, item_class: type) -> boolReturn metadata for the given item class and field name, if available.
By default, this method returns an emptyMappingProxyTypeobject. Please supply your
own method definition if you want to handle field metadata based on custom logic.
See thesection on metadata supportfor additional information.methodget_field_meta(self, field_name: str) -> types.MappingProxyTypeReturn metadata for the given field name, if available. It's usually not necessary to
override this method, since theitemadapter.adapter.AdapterInterfacebase class
provides a default implementation that callsItemAdapter.get_field_meta_from_classwith the wrapped item's class as argument.
See thesection on metadata supportfor additional information.methodfield_names(self) -> collections.abc.KeysView:Return adynamic viewof the item's field names. By default, this method returns the result of callingkeys()on
the current adapter, i.e., its return value depends on the implementation of the methods from theMutableMappinginterface (more specifically, it depends on the return value of__iter__).You might want to override this method if you want a way to get all fields for an item, whether or not
they are populated. For instance, Scrapy uses this method to define column names when exporting items to CSV.Registering an adapterAdd your custom adapter class to theitemadapter.adapter.ItemAdapter.ADAPTER_CLASSESclass attribute in order to handle custom item classes:Example>>>fromitemadapter.adapterimportItemAdapter>>>fromtests.test_interfaceimportBaseFakeItemAdapter,FakeItemClass>>>>>>ItemAdapter.ADAPTER_CLASSES.appendleft(BaseFakeItemAdapter)>>>item=FakeItemClass()>>>adapter=ItemAdapter(item)>>>adapter<ItemAdapterforFakeItemClass()>>>>Multiple adapter classesIf you need to have different handlers and/or priorities for different cases
you can subclass theItemAdapterclass and set theADAPTER_CLASSESattribute as needed:Example>>>fromcollectionsimportdeque>>>fromitemadapter.adapterimport(...ItemAdapter,...AttrsAdapter,...DataclassAdapter,...DictAdapter,...PydanticAdapter,...ScrapyItemAdapter,...)>>>fromscrapy.itemimportItem,Field>>>>>>classBuiltinTypesItemAdapter(ItemAdapter):...ADAPTER_CLASSES=deque([DictAdapter,DataclassAdapter])...>>>classThirdPartyTypesItemAdapter(ItemAdapter):...ADAPTER_CLASSES=deque([AttrsAdapter,PydanticAdapter,ScrapyItemAdapter])...>>>classScrapyItem(Item):...foo=Field()...>>>BuiltinTypesItemAdapter.is_item(dict())True>>>ThirdPartyTypesItemAdapter.is_item(dict())False>>>BuiltinTypesItemAdapter.is_item(ScrapyItem(foo="bar"))False>>>ThirdPartyTypesItemAdapter.is_item(ScrapyItem(foo="bar"))True>>>More examplesscrapy.item.Itemobjects>>>fromscrapy.itemimportItem,Field>>>fromitemadapterimportItemAdapter>>>classInventoryItem(Item):...name=Field()...price=Field()...>>>item=InventoryItem(name="foo",price=10)>>>adapter=ItemAdapter(item)>>>adapter.itemisitemTrue>>>adapter["name"]'foo'>>>adapter["name"]="bar">>>adapter["price"]=5>>>item{'name':'bar','price':5}>>>dict>>>fromitemadapterimportItemAdapter>>>item=dict(name="foo",price=10)>>>adapter=ItemAdapter(item)>>>adapter.itemisitemTrue>>>adapter["name"]'foo'>>>adapter["name"]="bar">>>adapter["price"]=5>>>item{'name':'bar','price':5}>>>dataclassobjects>>>fromdataclassesimportdataclass>>>fromitemadapterimportItemAdapter>>>@dataclass...classInventoryItem:...name:str...price:int...>>>item=InventoryItem(name="foo",price=10)>>>adapter=ItemAdapter(item)>>>adapter.itemisitemTrue>>>adapter["name"]'foo'>>>adapter["name"]="bar">>>adapter["price"]=5>>>itemInventoryItem(name='bar',price=5)>>>attrsobjects>>>importattr>>>fromitemadapterimportItemAdapter>>>@attr.s...classInventoryItem:...name=attr.ib()...price=attr.ib()...>>>item=InventoryItem(name="foo",price=10)>>>adapter=ItemAdapter(item)>>>adapter.itemisitemTrue>>>adapter["name"]'foo'>>>adapter["name"]="bar">>>adapter["price"]=5>>>itemInventoryItem(name='bar',price=5)>>>pydanticobjects>>>frompydanticimportBaseModel>>>fromitemadapterimportItemAdapter>>>classInventoryItem(BaseModel):...name:str...price:int...>>>item=InventoryItem(name="foo",price=10)>>>adapter=ItemAdapter(item)>>>adapter.itemisitemTrue>>>adapter["name"]'foo'>>>adapter["name"]="bar">>>adapter["price"]=5>>>itemInventoryItem(name='bar',price=5)>>>ChangelogSee thefull changelog