Package: avro-preprocessor
Avro-PreprocessorThis Python project provides a preprocessor for Avro resources.The preprocessor is made up of a list of modules.
Each module, in the specified order, performs an operation over the entire list of Avro resources.The input is a directory tree (namespace-like) in which each file is either adomain_messages or atype.The output is another directory tree, with the same structure as input, in which Avro resources
underwent the list of preprocessing operations.Input Avro resources (domain_messageortype) haveexavscas file extension.
NEW: they can also beyamlfiles with the same Avro extensions asexavsc!
You can even mixexavscandyamlfiles if passed-ie exavsc,yexavsc.
Output Avro resources haveavscas file extension.Schema definitionTwo different kinds of data structures are used in this project:domain messagesandtypes.Domain messagesDomain messages are Avro schemas that can be sent over a Kafka topic.
In general, multiple messages can be associated to each Kafka topic.
Domain messages are placed inschema/com/example/message_schema/domain/<domain_context>/<record_name>TypesTypes are reusable structures that can be referenced in a domain message.
They are placed inschema/com/example/message_schema/type/<typename>Domain messagescannotbe reused (i.e. referenced by other domain messages).Naming rulesDirectory structure must match the namespaces used inside the message / type definitionsRecords are defined in camel caseProperties are defined in snake caseKafka topics and Avro subjectsEach directory<domain_context>underschema/com/example/message_schema/domain/represents a Kafka topic. All schemas under a specific topic directory (and only them)
will be available to be sent on the corresponding Kafka topic, i.e. they will be registered
on the schema registry.For subject naming we follow the guidelines defined by Confluent toput several event types in the same Kafka topic;
in particular we use the strategyio.confluent.kafka.serializers.subject.TopicRecordNameStrategy.This behaviour is defined inavro_naming.py.Topic namedomain.<domain_context>Avro subject namesdomain.<domain_context>-<fully_qualified_record_name>ExampleFor schema directory:schema/com/example/message_schema/domain/user/Topic name:domain.userList of Avro subjects names:domain.user-com.example.message_schema.domain.user.UserCreated
domain.user-com.example.message_schema.domain.user.UserUpdated
domain.user-com.example.message_schema.domain.user.UserDeletedList of available preprocessing modulesThe following modules should be applied in the order shown as follows.
However, they are all optional.DocumentationCondenser: JSON strings don't support multi lines inside an editor. This makes
it inconvenient to write long documentation for some fields. This module adds support for
lists asdocfield in Avro schemas (they will be joined in the output).NamespaceChecker: Checks if a resource namespace corresponds to the directory tree structure.DocumentationChecker: Checks if the documentation corresponds to specific rules:(sub) schemas with a "name" property MUST have a "doc" property.(sub) schemas without a "name" property MUST NOT have a "doc" property.If two (sub) schemas have the same "doc", they must be identical.DefaultValueNullChecker: Makes sure no default values other than null are used.NamespaceChecker: Checks if a resource namespace corresponds to the directory tree structure.
properties insnake_case.AvroOrderChecker: Check if the input schemas are sorted according to a fixed order
(seeavro_sorter.py). Fails if they are not.MetadataAdder: Adds theMetadatatype(specified as argument on the command line) as first
field of every domain schema. Use the--metadata-excludeoption to exclude a comma-separated list
of namespaces.SchemaDependenciesChecker: Calculates all resources dependencies and assert no cycles are present.NullableOptionalExpander: Allows for a compact definition ofnullable_optionalsfor "partial entity updates", see its specific section below.ReferencesExpander: While Apache'savro-toolsis able to deal with references to other
resources in different files, Confluent's schema registry can only take one JSON as input, for
each schema. Therefore, references totypes in adomain_messagehave to be expanded.
But with a caveat:
only the first reference ("depth-first", NOT "breadth-first"!)mustbe expanded. All subsequent
referencesmust notbe expanded, i.e. they have to remain references.AvroSorter: For the sake of readability and clearness, JSON fields of Avro resources are
rearranged according to a fixed order (seeavro_sorter.py).KeysGenerator: Generates Avro Schemas for Kafka Keys automatically for every topic based
on thepartition-fieldproperty of schema fields.SchemaMappingGenerator: Outputs a JSON mapfully qualified schema -> kafka topicto the fileschema-mapping.json. It also gathers the field names having custom propertypartition-fieldand writes them inschema-mapping.json. This is done so that Kafka producers
know which set of fields to hash to decide which kafka partition to send a message to.
This module also checks that partition keys are consistent within a given Kafka topic.
Furthermore, gathers the field names oflogicalType == user-id-fieldsand stores them inschema-mapping.json. These are meant to be those fields related to users so that they
can be indexed separately (e.g. for GDPR.)DeprecationMappingGenerator: Outputs a JSON map with info about deprecated schemas/fields to the filedeprecation-mapping.json. This is later used for code generation. Also removes "deprecated" and
"deprecated-symbols" properties from the final schemas.JavaClassesCreator: Creates Java classes using Apache'savro-tools. It is also a very
convenient way to check correctness of Avro resources.SchemaRegistryChecker: Check if a schema is compatible with the latest version of the
same schema in the Schema registry.SchemaRegistrar: Register an Avro resources (bothdomain_messages ANDtypes!)
for the appropriate
Kafka topic in the schema registry.types are registered for a dummy topic so that we can
check compatibility for them as well (it would not be possible otherwise).
See the documentation inschema_registrar.pyfor further documentation ofsubjectnaming and topics.
Autogenerated keys or the generic key for all schemas (-koption from the command line) are
registered as well.Installationpipinstallavro-preprocessorExample UsageDownload Avro tools JARshdownload_avro-tools.shFor help:avropreprocessor.py-hIt is possible to define thekey_subject_name_strategywith the-ksnsswitch.
Possible values areRecordNameStrategyandTopicRecordNameStrategy(default).
Only relevant if theSchemaRegistryCheckerand/orSchemaRegistrarmodules are active as well.All modules usageavropreprocessor.py-ischema/-oextended-schema/-v-s'http://localhost:8081'-tcom.jaumo.message_schema.type-dcom.jaumo.message_schema.type.Metadata-pschema-mapping.json-ie'exavsc'-a./avro-tools-1.9.0.jarNo java classes, no schema registryavropreprocessor.py-ischema/-obuild/schema/-v-s'http://localhost:8081'-tcom.jaumo.message_schema.type-dcom.jaumo.message_schema.type.Metadata-pbuild/schema-mapping.json-mDocumentationCondenserNamespaceCheckerDocumentationCheckerDefaultValueNullCheckerNamesCheckerAvroOrderCheckerMetadataAdderSchemaDependenciesCheckerNullableOptionalExpanderReferencesExpanderAvroSorterSchemaMappingGeneratorDeprecationMappingGeneratorSchemaRegistryCheckerOnly register to the schema registryavropreprocessor.py-ibuild/schema/-o/tmp/-v-s'http://localhost:8081'-tcom.jaumo.message_schema.type-ie'avsc'-mSchemaRegistryCheckerSchemaRegistrarReorder theinputschemas according to the order defined insort_avro.pyavropreprocessor.py-ischema/-oschema/-v-tcom.jaumo.message_schema.type-ie'exavsc'-oe'exavsc'-mAvroSorterConvert input schemas fromexavsctoyamlavropreprocessor.py-ischema/-oschema_yaml/-v-tcom.jaumo.message_schema.type-ie'exavsc'-oe'yexavsc'-if'json'-of'yaml'-mDocumentationCondenserThen use the yaml schemas as inputavropreprocessor.py-ischema_yaml/-obuild/schema/-ie'yexavsc'-if'yaml'...nullable_optionals: a convenient solution to theSet-as-nullandnull-because-not-presentissueAsprotobuf,avrodoes not distinguish between these two cases,
unless some sort of wrapper is used.
The simple solution is to send the entire content of an event every time
(i.e. always complete updates, never partial updates).However, since partial updates can still be useful, the moduleNullableOptionalExpanderautomatises
the following process. Let's consider a simple Avro schema for user updates:{"namespace":"com.jaumo","type":"record","name":"UserUpdate","doc":"Update user event","fields":[{"name":"id","doc":"The id","type":"int"},{"name":"name","doc":"The name of the user","type":"string"},{"name":"age","doc":"The age of the user","type":"int"},{"name":"email","doc":"The email of the user","type":"string"}]}This schema does not supportnullvalues at all. Inavro, they can be supported as follows:{"namespace":"com.jaumo","type":"record","name":"UserUpdate","doc":"Update user event","fields":[{"name":"id","doc":"The id","type":"int"},{"name":"name","doc":"The name of the user","type":["null","string"],"default":null},{"name":"age","doc":"The age of the user","type":["null","int"],"default":null},{"name":"email","doc":"The email of the user","type":["null","string"],"default":null}]}With the schema above, bothset-as-nullandnull-because-not-presentare just null,
so they are still ambiguous. To distinguish them, we can wrap around a record type
(which in Java will be a specific class). Showing theemailproperty only, this would be{"name":"email","doc":"The email of the user","default":null,"type":["null",{"name":"OptionalString","type":"record","fields":[{"name":"value","default":null,"type":["null","string"]}]}]}Note how the name of the wrapper record ("OptionalString", above)
comes from the prefixOptionaland the capitalised version of the Avro type field (string).
The name of the inner property is always"value"(this is done automatically, see below).Now we have three casesemailset to null meansnull-because-not-presentemail.valueset to null meansset-as-nullemail.valueset to stringSince there's a lot of boilerplate and thus this is error-prone, thenullable_optionalis simple extension to the Avro syntax that adds the property"nullable_optional": true.
If a field has this property set totrue, the field is expanded with its wrapper. The schema above then becomes:{"name":"email","doc":"The email of the user","nullable_optional":true,"type":"string"}and it is saved to a.exavscfile.Note how this step is completelylanguage independent: its output is a
completely standard Avro schema that can be used by any library or framework.