Package: python-schema-registry-client
Python Rest Client Schema RegistryPython Rest Client to interact againstschema-registryconfluent server to manageAvroandJSONschemas resources.Requirementspython 3.8+Installationpipinstallpython-schema-registry-clientIf you want theFaustfunctionality:pipinstallpython-schema-registry-client[faust]Note that this will automatically add a dependency on thefaust-streamingfork of faust. If you want to use the
old faust version, simply install it manually and then installpython-schema-registry-clientwithout thefaustextra enabled, the functionality will
be the same.Client API, Serializer, Faust Integration and Schema Server descriptionDocumentation:https://marcosschroh.github.io/python-schema-registry-client.ioAvro Schema Usagefromschema_registry.clientimportSchemaRegistryClient,schemaclient=SchemaRegistryClient(url="http://127.0.0.1:8081")deployment_schema={"type":"record","namespace":"com.kubertenes","name":"AvroDeployment","fields":[{"name":"image","type":"string"},{"name":"replicas","type":"int"},{"name":"port","type":"int"},],}avro_schema=schema.AvroSchema(deployment_schema)schema_id=client.register("test-deployment",avro_schema)or asyncfromschema_registry.clientimportAsyncSchemaRegistryClient,schemaasync_client=AsyncSchemaRegistryClient(url="http://127.0.0.1:8081")deployment_schema={"type":"record","namespace":"com.kubertenes","name":"AvroDeployment","fields":[{"name":"image","type":"string"},{"name":"replicas","type":"int"},{"name":"port","type":"int"},],}avro_schema=schema.AvroSchema(deployment_schema)schema_id=awaitasync_client.register("test-deployment",avro_schema)JSON Schema Usagefromschema_registry.clientimportSchemaRegistryClient,schemaclient=SchemaRegistryClient(url="http://127.0.0.1:8081")deployment_schema={"definitions":{"JsonDeployment":{"type":"object","required":["image","replicas","port"],"properties":{"image":{"type":"string"},"replicas":{"type":"integer"},"port":{"type":"integer"}}}},"$ref":"#/definitions/JsonDeployment"}json_schema=schema.JsonSchema(deployment_schema)schema_id=client.register("test-deployment",json_schema)or asyncfromschema_registry.clientimportAsyncSchemaRegistryClient,schemaasync_client=AsyncSchemaRegistryClient(url="http://127.0.0.1:8081")deployment_schema={"definitions":{"JsonDeployment":{"type":"object","required":["image","replicas","port"],"properties":{"image":{"type":"string"},"replicas":{"type":"integer"},"port":{"type":"integer"}}}},"$ref":"#/definitions/JsonDeployment"}json_schema=schema.JsonSchema(deployment_schema)schema_id=awaitasync_client.register("test-deployment",json_schema)Usage with dataclasses-avroschema for avro schemasYou can generate theavro schemadirectely from a python class usingdataclasses-avroschemaand use it in the API forregister schemas,check versionsandtest compatibility:importdataclassesfromdataclasses_avroschemaimportAvroModel,typesfromschema_registry.clientimportSchemaRegistryClientclient=SchemaRegistryClient(url="http://127.0.0.1:8081")@dataclasses.dataclassclassUserAdvance(AvroModel):name:strage:intpets:typing.List[str]=dataclasses.field(default_factory=lambda:["dog","cat"])accounts:typing.Dict[str,int]=dataclasses.field(default_factory=lambda:{"key":1})has_car:bool=Falsefavorite_colors:types.Enum=types.Enum(["BLUE","YELLOW","GREEN"],default="BLUE")country:str="Argentina"address:str=None# register the schemaschema_id=client.register(subject,UserAdvance.avro_schema())print(schema_id)# >>> 12result=client.check_version(subject,UserAdvance.avro_schema())print(result)# >>> SchemaVersion(subject='dataclasses-avroschema-subject-2', schema_id=12, schema=1, version={"type":"record" ...')compatibility=client.test_compatibility(subject,UserAdvance.avro_schema())print(compatibility)# >>> TrueUsage with pydantic for json schemasYou can generate the json schema directely from a python class using pydantic and use it in the API for register schemas, check versions and test compatibility:importtypingfromenumimportEnumfrompydanticimportBaseModelfromschema_registry.clientimportSchemaRegistryClientclient=SchemaRegistryClient(url="http://127.0.0.1:8081")classColorEnum(str,Enum):BLUE="BLUE"YELLOW="YELLOW"GREEN="GREEN"classUserAdvance(BaseModel):name:strage:intpets:typing.List[str]=["dog","cat"]accounts:typing.Dict[str,int]={"key":1}has_car:bool=Falsefavorite_colors:ColorEnum=ColorEnum.BLUEcountry:str="Argentina"address:str=None# register the schemaschema_id=client.register(subject,UserAdvance.schema_json(),schema_type="JSON")print(schema_id)# >>> 12result=client.check_version(subject,UserAdvance.schema_json(),schema_type="JSON")print(result)# >>> SchemaVersion(subject='pydantic-jsonschema-subject', schema_id=12, schema=1, version=<schema_registry.client.schema.JsonSchema object at 0x7f40354550a0>)compatibility=client.test_compatibility(subject,UserAdvance.schema_json(),schema_type="JSON")print(compatibility)# >>> TrueSerializersYou can useAvroMessageSerializerto encode/decode messages inavrofromschema_registry.clientimportSchemaRegistryClient,schemafromschema_registry.serializersimportAvroMessageSerializerclient=SchemaRegistryClient("http://127.0.0.1:8081")avro_message_serializer=AvroMessageSerializer(client)avro_user_schema=schema.AvroSchema({"type":"record","namespace":"com.example","name":"AvroUsers","fields":[{"name":"first_name","type":"string"},{"name":"last_name","type":"string"},{"name":"age","type":"int"},],})# We want to encode the user_record with avro_user_schemauser_record={"first_name":"my_first_name","last_name":"my_last_name","age":20,}# Encode the recordmessage_encoded=avro_message_serializer.encode_record_with_schema("user",avro_user_schema,user_record)print(message_encoded)# >>> b'\x00\x00\x00\x00\x01\x1amy_first_name\x18my_last_name('or withjson schemasfromschema_registry.clientimportSchemaRegistryClient,schemafromschema_registry.serializersimportJsonMessageSerializerclient=SchemaRegistryClient("http://127.0.0.1:8081")json_message_serializer=JsonMessageSerializer(client)json_schema=schema.JsonSchema({"definitions":{"record:python.test.basic.basic":{"description":"basic schema for tests","type":"object","required":["number","name"],"properties":{"number":{"oneOf":[{"type":"integer"},{"type":"null"}]},"name":{"oneOf":[{"type":"string"}]}}}},"$ref":"#/definitions/record:python.test.basic.basic"})# Encode the recordbasic_record={"number":10,"name":"a_name",}message_encoded=json_message_serializer.encode_record_with_schema("basic",json_schema,basic_record)print(message_encoded)# >>> b'\x00\x00\x00\x00\x02{"number": 10, "name": "a_name"}'When use this libraryUsually, we have a situation like this:So, our producers/consumers have to serialize/deserialize messages every time that they send/receive from Kafka topics. In this picture, we can imagine aFaustapplication receiving messages (encoded with an Avro schema) and we want to deserialize them, so we can ask theschema serverto do that for us. In this scenario, theMessageSerializeris perfect.Also, could be a use case that we would like to have an Application only to administrateAvro Schemas(register, update compatibilities, delete old schemas, etc.), so theSchemaRegistryClientis perfect.DevelopmentPoetryis needed to install the dependencies and develope locallyInstall dependencies:poetry install --all-extrasCode linting:./scripts/formatRun tests:./scripts/testFor commit messages we usecommitizenin order to standardize a way of committing rulesNote: The tests are run against theSchema Serverusingdocker compose, so you will needDockerandDocker Composeinstalled.In a terminal rundocker-compose up. Then in a different terminal run the tests:./scripts/testAll additional args will be passed to pytest, for example:./scripts/test./tests/client/Tests usind the python shellTo perform tests using the python shell you can run the project usingdocker-compose.Executedocker-compose up. Then, theschema registry serverwill run onhttp://127.0.0.1:8081, then you can interact against it using theSchemaRegistryClient:Use the python interpreter (get a python shell typingpythonin your command line)Play with theschema serverfromschema_registry.clientimportSchemaRegistryClient,schemaclient=SchemaRegistryClient(url="http://127.0.0.1:8081")# do some operations with the client...deployment_schema={"type":"record","namespace":"com.kubertenes","name":"AvroDeployment","fields":[{"name":"image","type":"string"},{"name":"replicas","type":"int"},{"name":"port","type":"int"},],}avro_schema=schema.AvroSchema(deployment_schema)client.register("test-deployment",avro_schema)# >>>> Out[5]: 1Then, you can check the schema using your browser going to the urlhttp://127.0.0.1:8081/schemas/ids/1