Package: avro-validator
Avro ValidatorA pure python avro schema validator.The default avro library for Python provide validation of data against the schema, the problem is that the output of
this validation doesn't provide information about the error.
All you get is thethe datum is not an example of the schemaerror message.When working with bigger avro schemas, sometimes is not easy to visually find the field that has an issue.This library provide clearer exceptions when validating data against the avro schema, in order to be easier to
identify the field that is not compliant with the schema and the problem with that field.InstallingInstall using pip:$pipinstall-Uavro_validatorValidating data against Avro schemaThe validator can be used as a console application. It receives a schema file, and a data file, validating the data
and returning the error message in case of failure.The avro_validator can also be used as a library in python code.Console usageIn order to validate thedata_to_validate.jsonfile against theschema.avscusing theavro_validatorcallable, just type:$avro_validatorschema.avscdata_to_valdate.json
OKSince the data is valid according to the schema, the return message isOK.Error validating the dataIf the data is not valid, the program returns an error message:$avro_validatorschema.avscdata_to_valdate.json
Errorvalidatingvalueforfield[data,my_boolean_value]:Thevalue[123]isnotfromoneofthefollowingtypes:[[NullType,BooleanType]]This message indicates that the fieldmy_boolean_valueinside thedatadictionary has value123, which is not
compatible with thebooltype.Command usageIt is possible to get information about usage of theavro_validatorusing the help:$avro_validator-hLibrary usageUsing schema fileWhen using the avr_validator as a library, it is possible to pass the schema as a file:fromavro_validator.schemaimportSchemaschema_file='schema.avsc'schema=Schema(schema_file)parsed_schema=schema.parse()data_to_validate={'name':'My Name'}parsed_schema.validate(data_to_validate)In this example, if thedata_to_validateis valid according to the schema, then theparsed_schema.validate(data_to_validate)call will returnTrue.Using a dict as schemaIt is also possible to provide the schema as a json string:importjsonfromavro_validator.schemaimportSchemaschema=json.dumps({'name':'test schema','type':'record','doc':'schema for testing avro_validator','fields':[{'name':'name','type':'string'}]})schema=Schema(schema)parsed_schema=schema.parse()data_to_validate={'name':'My Name'}parsed_schema.validate(data_to_validate)In this example, theparsed_schema.validate(data_to_validate)call will returnTrue, since the data is valid according to the schema.Invalid dataIf the data is not valid, theparsed_schema.validatewill raise aValueError, with the message containing the error description.importjsonfromavro_validator.schemaimportSchemaschema=json.dumps({'name':'test schema','type':'record','doc':'schema for testing avro_validator','fields':[{'name':'name','type':'string','doc':'Field that stores the name'}]})schema=Schema(schema)parsed_schema=schema.parse()data_to_validate={'my_name':'My Name'}parsed_schema.validate(data_to_validate)The schema defined expects only one field, namedname, but the data contains only the fieldname_2,
making it invalid according to the schema. In this case, thevalidatemethod will return the following error:Traceback (most recent call last):
  File "/Users/leonardo.almeida/.pyenv/versions/avro_validator_venv/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-a5e8ce95d21c>", line 23, in <module>
    parsed_schema.validate(data_to_validate)
  File "/opt/dwh/avro_validator/avro_validator/avro_types.py", line 563, in validate
    raise ValueError(f'The fields from value [{value}] differs from the fields '
ValueError: The fields from value [{'my_name': 'My Name'}] differs from the fields of the record type [{'name': RecordTypeField <name: name, type: StringType, doc: Field that stores the name, default: None, order: None, aliases: None>}]The message detailed enough to enable the developer to pinpoint the error in the data.Invalid schemaIf the schema is not valid according to avro specifications, theparsemethod will also return aValueError.importjsonfromavro_validator.schemaimportSchemaschema=json.dumps({'name':'test schema','type':'record','doc':'schema for testing avro_validator','fields':[{'name':'name','type':'invalid_type','doc':'Field that stores the name'}]})schema=Schema(schema)parsed_schema=schema.parse()Since the schema tries to define thenamefield asinvalid_type, the schema declaration is invalid,
thus the following exception will be raised:Traceback (most recent call last):
  File "/Users/leonardo.almeida/.pyenv/versions/avro_validator_venv/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-7f3f77000f08>", line 18, in <module>
    parsed_schema = schema.parse()
  File "/opt/dwh/avro_validator/avro_validator/schema.py", line 28, in parse
    return RecordType.build(schema)
  File "/opt/dwh/avro_validator/avro_validator/avro_types.py", line 588, in build
    record_type.__fields = {field['name']: RecordTypeField.build(field) for field in json_repr['fields']}
  File "/opt/dwh/avro_validator/avro_validator/avro_types.py", line 588, in <dictcomp>
    record_type.__fields = {field['name']: RecordTypeField.build(field) for field in json_repr['fields']}
  File "/opt/dwh/avro_validator/avro_validator/avro_types.py", line 419, in build
    field.__type = cls.__build_field_type(json_repr)
  File "/opt/dwh/avro_validator/avro_validator/avro_types.py", line 401, in __build_field_type
    raise ValueError(f'Error parsing the field [{fields}]: {actual_error}')
ValueError: Error parsing the field [name]: The type [invalid_type] is not recognized by AvroThe message is clearly indicating that the theinvalid_typeis not recognized by avro.