Package: dataclasses-avroschema
Dataclasses Avro Schema GeneratorGenerateavro schemasfrom python dataclasses.Code generationfrom avro schemas.Serialize/Deserializepython instances with avro schemasRequirementspython 3.8+Installationwithpiporpoetry:pip install dataclasses-avroschemaorpoetry install dataclasses-avroschemaExtraspydantic:pip install 'dataclasses-avroschema[pydantic]'orpoetry install dataclasses-avroschema --extras "pydantic"faust-streaming:pip install 'dataclasses-avroschema[faust]'orpoetry install dataclasses-avroschema --extras "faust"faker:pip install 'dataclasses-avroschema[faker]'orpoetry install dataclasses-avroschema --extras "faker"Note: You can install all extra dependencies withpip install dataclasses-avroschema[faust, pydantic, faker]orpoetry install dataclasses-avroschema --extras "pydantic faust faker"CLITo installavro schemas cliinstalldc-avropip install 'dataclasses-avroschema[cli]'orpoetry install dataclasses-avroschema --with cliDocumentationhttps://marcosschroh.github.io/dataclasses-avroschema/UsageGenerating the avro schemafromdataclassesimportdataclassimportenumimporttypingfromdataclasses_avroschemaimportAvroModel,typesclassFavoriteColor(enum.Enum):BLUE="BLUE"YELLOW="YELLOW"GREEN="GREEN"@dataclassclassUser(AvroModel):"An User"name:strage:intpets:typing.List[str]accounts:typing.Dict[str,int]favorite_colors:FavoriteColorcountry:str="Argentina"address:str=NoneclassMeta:namespace="User.v1"aliases=["user-v1","super user"]User.avro_schema()'{"type":"record","name":"User","doc":"An User","namespace":"User.v1","aliases":["user-v1","super user"],"fields":[{"name":"name","type":"string"},{"name":"age","type":"long"},{"name":"pets","type":"array","items":"string"},{"name":"accounts","type":"map","values":"long"},{"name":"favorite_color","type":{"type":"enum","name":"FavoriteColor","symbols":["Blue","Yellow","Green"]}}{"name":"country","type":"string","default":"Argentina"},{"name":"address","type":["null","string"],"default":null}]}'User.avro_schema_to_python(){"type":"record","name":"User","doc":"An User","namespace":"User.v1","aliases":["user-v1","super user"],"fields":[{"name":"name","type":"string"},{"name":"age","type":"long"},{"name":"pets","type":{"type":"array","items":"string","name":"pet"}},{"name":"accounts","type":{"type":"map","values":"long","name":"account"}},{"name":"favorite_colors","type":{"type":"enum","name":"FavoriteColor","symbols":["BLUE","YELLOW","GREEN"]}},{"name":"country","type":"string","default":"Argentina"},{"name":"address","type":["null","string"],"default":None}],}Serialization to avro or avro-json and json payloadFor serialization is neccesary to use python class/dataclasses instancefromdataclassesimportdataclassimporttypingfromdataclasses_avroschemaimportAvroModel@dataclassclassAddress(AvroModel):"An Address"street:strstreet_number:int@dataclassclassUser(AvroModel):"User with multiple Address"name:strage:intaddresses:typing.List[Address]address_data={"street":"test","street_number":10,}# create an Address instanceaddress=Address(**address_data)data_user={"name":"john","age":20,"addresses":[address],}# create an User instanceuser=User(**data_user)user.serialize()# >>> b"\x08john(\x02\x08test\x14\x00"user.serialize(serialization_type="avro-json")# >>> b'{"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}'# Get the json from the instanceuser.to_json()# >>> '{"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}'# Get a python dictuser.to_dict()# >>> {"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}DeserializationDeserialization could take place with an instance dataclass or the dataclass itself. Can return the dict representation or a new class instanceimporttypingimportdataclassesfromdataclasses_avroschemaimportAvroModel@dataclasses.dataclassclassAddress(AvroModel):"An Address"street:strstreet_number:int@dataclasses.dataclassclassUser(AvroModel):"User with multiple Address"name:strage:intaddresses:typing.List[Address]avro_binary=b"\x08john(\x02\x08test\x14\x00"avro_json_binary=b'{"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}'# return a new class instance!!User.deserialize(avro_binary)# >>>> User(name='john', age=20, addresses=[Address(street='test', street_number=10)])# return a python dictUser.deserialize(avro_binary,create_instance=False)# >>> {"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}# return a new class instance!!User.deserialize(avro_json_binary,serialization_type="avro-json")# >>>> User(name='john', age=20, addresses=[Address(street='test', street_number=10)])# return a python dictUser.deserialize(avro_json_binary,serialization_type="avro-json",create_instance=False)# >>> {"name": "john", "age": 20, "addresses": [{"street": "test", "street_number": 10}]}Pydantic integrationTo adddataclasses-avroschemafunctionality topydanticyou only need to replaceBaseModelbyAvroBaseModel:importtypingimportenumimportdataclassesfromdataclasses_avroschema.pydanticimportAvroBaseModelfrompydanticimportFieldclassFavoriteColor(str,enum.Enum):BLUE="BLUE"YELLOW="YELLOW"GREEN="GREEN"@dataclasses.dataclassclassUserAdvance(AvroBaseModel):name:strage:intpets:typing.List[str]=Field(default_factory=lambda:["dog","cat"])accounts:typing.Dict[str,int]=Field(default_factory=lambda:{"key":1})has_car:bool=Falsefavorite_colors:FavoriteColor=FavoriteColor.BLUEcountry:str="Argentina"address:str=NoneclassMeta:schema_doc=False# Avro schemaUserAdvance.avro_schema()'{"type":"record","name":"UserAdvance","fields":[{"name":"name","type":"string"},{"name":"age","type":"long"},{"name":"pets","type":{"type":"array","items":"string","name":"pet"},"default":["dog","cat"]},{"name":"accounts","type":{"type":"map","values":"long","name":"account"},"default":{"key":1}},{"name":"has_car","type":"boolean","default":false},{"name":"favorite_colors","type":{"type":"enum","name":"favorite_color","symbols":["BLUE","YELLOW","GREEN"]},"default":"BLUE"},{"name":"country","type":"string","default":"Argentina"},{"name":"address","type":["null","string"],"default":null}]}'# Json schemaUserAdvance.json_schema(){"title":"UserAdvance","description":"UserAdvance(*, name: str, age: int, pets: List[str] = None, ...","type":"object","properties":{"name":{"title":"Name","type":"string"},"age":{"title":"Age","type":"integer"},"pets":{"title":"Pets","type":"array","items":{"type":"string"}},"accounts":{"title":"Accounts","type":"object","additionalProperties":{"type":"integer"}},"has_car":{"title":"Has Car","default":false,"type":"boolean"},"favorite_colors":{"default":"BLUE","allOf":[{"$ref":"#/definitions/FavoriteColor"}]},"country":{"title":"Country","default":"Argentina","type":"string"},"address":{"title":"Address","type":"string"}},"required":["name","age"],"definitions":{"FavoriteColor":{"title":"FavoriteColor","description":"An enumeration.","enum":["BLUE","YELLOW","GREEN"],"type":"string"}}}user=UserAdvance(name="bond",age=50)# pydanticuser.dict()# >>> {'name': 'bond', 'age': 50, 'pets': ['dog', 'cat'], 'accounts': {'key': 1}, 'has_car': False, 'favorite_colors': <FavoriteColor.BLUE: 'BLUE'>, 'country': 'Argentina', 'address': None}# pydanticuser.json()# >>> '{"name": "bond", "age": 50, "pets": ["dog", "cat"], "accounts": {"key": 1}, "has_car": false, "favorite_colors": "BLUE", "country": "Argentina", "address": null}'# pydanticuser=UserAdvance(name="bond")# ValidationError: 1 validation error for UserAdvance# age# field required (type=value_error.missing)# dataclasses-avroschemaevent=user.serialize()print(event)# >>> b'\x08bondd\x04\x06dog\x06cat\x00\x02\x06key\x02\x00\x00\x00\x12Argentina\x00'UserAdvance.deserialize(data=event)# >>> UserAdvance(name='bond', age=50, pets=['dog', 'cat'], accounts={'key': 1}, has_car=False, favorite_colors=<FavoriteColor.BLUE: 'BLUE'>, country='Argentina', address=None)Examples with python streaming drivers (kafka and redis)Underexamplesfolder you can find 3 differents kafka examples, one withaiokafka(async) showing the simplest use case when aAvroModelinstance is serialized and sent it thorught kafka, and the event is consumed.
The other two examples aresyncusing thekafka-pythondriver, where theavro-jsonserialization andschema evolution(FULLcompatibility) is shown.
Also, there are tworedisexamples usingredis streamswithwalrusandredisgears-pyFactory and fixturesDataclasses Avro Schemaalso includes afactoryfeature, so you can generatefastpython instances and use them, for example, to test your data streaming pipelines. Instances can be generated using thefakemethod.Note: This feature is not enabled by default and requires you have thefakerextra installed. You may install it withpip install 'dataclasses-avroschema[faker]'importtypingimportdataclassesfromdataclasses_avroschemaimportAvroModel@dataclasses.dataclassclassAddress(AvroModel):"An Address"street:strstreet_number:int@dataclasses.dataclassclassUser(AvroModel):"User with multiple Address"name:strage:intaddresses:typing.List[Address]Address.fake()# >>>> Address(street='PxZJILDRgbXyhWrrPWxQ', street_number=2067)User.fake()# >>>> User(name='VGSBbOGfSGjkMDnefHIZ', age=8974, addresses=[Address(street='vNpPYgesiHUwwzGcmMiS', street_number=4790)])FeaturesPrimitive types: int, long, double, float, boolean, string and null supportComplex types: enum, array, map, fixed, unions and records supporttyping.Annotatedsupportedtyping.LiteralsupportedLogical Types: date, time (millis and micro), datetime (millis and micro), uuid supportSchema relations (oneToOne, oneToMany)Recursive SchemasGenerate Avro Schemas fromfaust.RecordInstance serialization correspondent toavro schemageneratedData deserialization. Return python dict or class instanceGenerate json from python class instanceCase SchemasGenerate models fromavscfilesExamples of integration withkafkadrivers:aiokafka,kafka-pythonExample of integration  withredisdrivers:walrusandredisgears-pyFactory instancesPydanticintegrationDevelopmentPoetryis needed to install the dependencies and develope locallyInstall dependencies:poetry install --all-extrasCode linting:./scripts/formatRun tests:./scripts/testFor commit messages we usecommitizenin order to standardize a way of committing rules