Package: aws-glue-schema-registry
AWS Glue Schema Registry for PythonUse the AWS Glue Schema Registry in Python projects.This library is a partial port ofaws-glue-schema-registrywhich implements a subset of its features with full compatibility.Feature SupportFeatureJava LibraryPython LibraryNotesSerialization and deserialization using schema registry✔️✔️Avro message format✔️✔️JSON Schema message format✔️✔️Kafka Streams support✔️N/A for Python, Kafka Streams is Java-onlyCompression✔️✔️Local schema cache✔️✔️Schema auto-registration✔️✔️Evolution checks✔️✔️Migration from a third party Schema Registry✔️✔️Flink support✔️❌Kafka Connect support✔️N/A for Python, Kafka Connect is Java-onlyInstallationClone this repository and install it:python setup.py install -e .This library includes opt-in extra dependencies that enable support for certain features. For example, to use the schema registry withkafka-python, you should install thekafka-pythonextra:python setup.py install -e .[kafka-python]Extra namePurposekafka-pythonProvides adapter classes to plug intokafka-pythonUsageFirst useboto3to create a low-level AWS Glue client:importboto3# Pass your AWS credentials or profile information heresession=boto3.Session(access_key_id=xxx,secret_access_key=xxx,region_name='us-west-2')glue_client=session.client('glue')Seehttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configurationfor more information on configuring boto3.Send Kafka messages withSchemaRegistrySerializer:fromaws_schema_registryimportDataAndSchema,SchemaRegistryClientfromaws_schema_registry.avroimportAvroSchema# In this example we will use kafka-python as our Kafka client,# so we need to have the `kafka-python` extras installed and use# the kafka adapter.fromaws_schema_registry.adapter.kafkaimportKafkaSerializerfromkafkaimportKafkaConsumer# Create the schema registry client, which is a façade around the boto3 glue clientclient=SchemaRegistryClient(glue_client,registry_name='my-registry')# Create the serializerserializer=KafkaSerializer(client)# Create the producerproducer=KafkaProducer(value_serializer=serializer)# Our producer needs a schema to send along with the data.# In this example we're using Avro, so we'll load an .avsc file.withopen('user.avsc','r')asschema_file:schema=AvroSchema(schema_file.read())# Send message data along with schemadata={'name':'John Doe','favorite_number':6}producer.send('my-topic',value=(data,schema))# the value MUST be a tuple when we're using the KafkaSerializerRead Kafka messages withSchemaRegistryDeserializer:fromaws_schema_registryimportSchemaRegistryClient# In this example we will use kafka-python as our Kafka client,# so we need to have the `kafka-python` extras installed and use# the kafka adapter.fromaws_schema_registry.adapter.kafkaimportKafkaDeserializerfromkafkaimportKafkaConsumer# Create the schema registry client, which is a façade around the boto3 glue clientclient=SchemaRegistryClient(glue_client,registry_name='my-registry')# Create the deserializerdeserializer=KafkaDeserializer(client)# Create the consumerconsumer=KafkaConsumer('my-topic',value_deserializer=deserializer)# Now use the consumer normallyformessageinconsumer:# The deserializer produces DataAndSchema instancesvalue:DataAndSchema=message.value# which are NamedTuples with a `data` and `schema` propertyvalue.data==value[0]value.schema==value[1]# and can be deconstructeddata,schema=valueContributingClone this repository and install development dependencies:pip install -e .[dev]Run the linter and tests with tox before committing. After committing, check Github Actions to see the result of the automated checks.LintingLint the code with:flake8Run the type checker with:mypyTestsTests go under thetests/directory. All tests outside oftests/integrationare unit tests with no external dependencies.Tests undertests/integrationare integration test that interact with external resources and/or real AWS schema registries. They generally run slower and require some additional configuration.Run just the unit tests with:pytest --ignore tests/integrationAll integration tests use the following environment variables:AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_SESSION_TOKENAWS_REGIONAWS_PROFILECLEANUP_REGISTRY: Set to any value to prevent the test from destroying the registry created during the test, allowing you to inspect its contents.If noAWS_environment variables are set,boto3will try to load credentials from your default AWS profile.See individual integration test directories for additional requirements and setup instructions.ToxThis project usesToxto run tests across multiple Python versions.Install Tox with:pip install toxand run it with:toxNote that Tox requires the tested python versions to be installed. One convenient way to manage this is usingpyenv. See the.python-versionsfile for the Python versions that need to be installed.ReleasesAssuming pypi permissions:python -m build
twine upload -r testpypi dist/*
twine upload dist/*