Package: fastavro-gen
Avro scheme generation:exclamation: This is quite barebones. Not all avro structures/types will be handled correctly since it was made with a specific avro schema catalog in mind. If you find anything missing, please submit a PR or create an issue.This library aims to create dataclasses and/or typed dict definitions from avro schemes, that can be used to type check creation of messages using a static type checker such as mypy.
The aim is to catch bugs before they occurr on runtime, and provide better IDE support.As the name suggests, it was thought of as an optional extension to use with the excellentfastavrolibrary.
Fastavro allows users to read avro schemas, create messages and write them as avro messages, or validate against a schema.Fastavro-gen uses fastavro to read.avscfiles and from the schema object generated, creates classes. Classes are written one per file, using the namespace to create a directory structure.
For example, the following record output class will be created under./example/avro/user.py.{"namespace":"example.avro","type":"record","name":"User","fields":[{"name":"name","type":"string"},{"name":"favorite_number","type":["int","null"]},{"name":"favorite_color","type":["string","null"]}]}Building aUsermessage would normally be done by building a dictionary:{"name":"My User","favorite_number":"1","favorite_color":"green",}Notice that the favorite number field in the schema has type int (or None) but the one we created has a string. This would cause a runtime error when writing or validating the record.
Using the generated dataclass we can get IDE support (screenshot using VSCode with the Pylance language server). Notice the underlined"1". Hovering over shows the relevant error.Mypy will also catch this issue:test.py:9:error:Argument"favorite_number"to"User"hasincompatibletype"str";expected"Optional[int]"Found1errorin1file(checked1sourcefile)OutputThe library offers the user two different output class types,dataclasses andTypedDicts.
Each has it's pros and cons that have to be weighed for the user's use cases.TypedDictTypedDicts are only valuable during type checking, and on runtime they are simply treated as normal dicts.
As such they can be built using common python dict syntax with an added type annotation, or using a class instantiation syntax:classA(TypedDict,total=True):field1:intfield2:str...instance1:A={"field1":1,"field2":"2",}instance2=A(field1=1,field2="2",):heavy_plus_sign:Messages can be built using python dictionary syntax:heavy_plus_sign:Fastavro expects messages as dictionaries:heavy_minus_sign:All fields of the dictionary have to be given at the time of creation, unless thetotaloption is given asFalse.
Havingtotal=Falsehowever restricts some aspects of the type checking e.g. checking if some keys are set or not.
Currently this library has the total option hardcoded asFalsebut that might be configurable at a later time.:heavy_minus_sign:No ability to specify defaults.dataclassDataclasses allow for easy declaration of python classes.:heavy_plus_sign:Can handle default values for fields. As such only non-default fields have to be instantiated initially.:heavy_plus_sign:Easy to transform to dictionaries with the providedfastavro_gen.asdictfunction. It is simply a wrapper arounddataclasses.asdict.:heavy_minus_sign:Complex nested schemas means a lot of objects being created:heavy_minus_sign:Extra overhead transforming messages to dictionaries:heavy_minus_sign:Overhead transforming dictionaries to dataclasses usingfastavro_gen.fromdict.UsageThis is a work in progress but is available on PyPI.pipinstallfastavro-genTo generate classes use the CLI or import thegeneratefunction fromfastavro_gen. The library also exposesfastavro_gen.[asdict, fromdict]to map generated dataclasses to and from dictionaries.:bulb: When the ordered option is specified, the file parameter will be ignored. Instead you can define schemas specified in the file parameter as singletons in the toml file passed to ordered.usage: fastavro_gen [-h] [-o ORDERED] [--class-type {dataclass,TypedDict}] [--no-black] [--prefix PREFIX] [--output-dir OUTPUT_DIR] [file [file ...]]

Generate dataclasses or TypedDicts from avro schemas

positional arguments:
  file                  file(s) to parse, use '-' for stdin

optional arguments:
  -h, --help            show this help message and exit
  -o ORDERED, --ordered ORDERED
                        Path to a .toml file for multiple schemas or ordered schemas. Overwrites 'file' parameter.
  --class-type {dataclass,TypedDict}
  --no-black            Do not run output files through 'black'
  --prefix PREFIX       Removes this prefix from namespace if it is contained
  --output-dir OUTPUT_DIR
                        Specify the output locationThe--orderedoptionThe option allows users to specify an order of files to read throught fastavro'sload_schema_orderedfunction.
This is useful when your files are laid out in a manner that does not follow the structure that the normalload_schemaexpects.The option takes as value a path to a.tomlfile that describes what schemas to read and what their pre-requisites.
For example, creating classes for a schema A that depends on B and C your.tomlwould include:schemaA=["/path/to/C.avsc","/path/to/B.avsc","/path/to/A.avsc",]The toml file can describe multiple schema dependencies, each as their own list.