Package: azure-schemaregistry-avroencoder
Azure Schema Registry Avro Encoder client library for PythonAzure Schema Registry is a schema repository service hosted by Azure Event Hubs, providing schema storage, versioning,
and management. This package provides an Avro encoder capable of encoding and decoding payloads containing
Schema Registry schema identifiers and Avro-encoded content.Source code|Package (PyPi)|API reference documentation|Samples|ChangelogDisclaimerAzure SDK Python packages support for Python 2.7 has ended 01 January 2022. For more information and questions, please refer tohttps://github.com/Azure/azure-sdk-for-python/issues/20691Getting startedInstall the packageInstall the Azure Schema Registry Avro Encoder client library for Python withpip:pipinstallazure-schemaregistry-avroencoderPrerequisites:To use this package, you must have:Azure subscription -Create a free accountAzure Schema Registry-Here is the quickstart guideto create a Schema Registry group using the Azure portal.Python 3.6 or later -Install PythonAuthenticate the clientInteraction with the Schema Registry Avro Encoder starts with an instance of AvroEncoder class, which takes the schema group name and theSchema Registry Clientclass. The client constructor takes the Event Hubs fully qualified namespace and and Azure Active Directory credential:The fully qualified namespace of the Schema Registry instance should follow the format:<yournamespace>.servicebus.windows.net.An AAD credential that implements theTokenCredentialprotocol should be passed to the constructor. There are implementations of theTokenCredentialprotocol available in theazure-identity package. To use the credential types provided byazure-identity, please install the Azure Identity client library for Python withpip:pipinstallazure-identityAdditionally, to use the async API, you must first install an async transport, such asaiohttp:pipinstallaiohttpCreate AvroEncoder using the azure-schemaregistry library:importosfromazure.schemaregistryimportSchemaRegistryClientfromazure.schemaregistry.encoder.avroencoderimportAvroEncoderfromazure.identityimportDefaultAzureCredentialcredential=DefaultAzureCredential()# Namespace should be similar to: '<your-eventhub-namespace>.servicebus.windows.net'fully_qualified_namespace=os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']group_name=os.environ['SCHEMAREGISTRY_GROUP']schema_registry_client=SchemaRegistryClient(fully_qualified_namespace,credential)encoder=AvroEncoder(client=schema_registry_client,group_name=group_name)Key conceptsAvroEncoderProvides API to encode to and decode from Avro Binary Encoding plus a
content type with schema ID. UsesSchemaRegistryClientto get schema IDs from schema content or vice versa.Supported message modelsSupport has been added to certain Azure Messaging SDK model classes for interoperability with theAvroEncoder. These models are subtypes of theMessageTypeprotocol defined under theazure.schemaregistry.encoder.avroencodernamespace. Currently, the supported model classes are:azure.eventhub.EventDataforazure-eventhub>=5.9.0Message formatIf a message type that follows the MessageType protocol is provided to the encoder for encoding, it will set the corresponding content and content type properties, where:content: Avro payload (in general, format-specific payload)Avro Binary EncodingNOT Avro Object Container File, which includes the schema and defeats the
purpose of this encoder to move the schema out of the message payload and
into the schema registry.content type: a string of the formatavro/binary+<schema ID>, where:avro/binaryis the format indicator<schema ID>is the hexadecimal representation of GUID, same format and byte order as the string from the Schema Registry service.IfEventDatais passed in as the message type, the following properties will be set on theEventDataobject:Thebodyproperty will be set to the content value.Thecontent_typeproperty will be set to the content type value.If message type is not provided, and by default, the encoder will create the following dict:{"content": <Avro encoded payload>, "content_type": 'avro/binary+<schema ID>' }ExamplesThe following sections provide several code snippets covering some of the most common Schema Registry tasks, including:EncodingDecodingEvent Hubs Sending IntegrationEvent Hubs Receiving IntegrationEncodingUse theAvroEncoder.encodemethod to encode content with the given Avro schema.
The method will use a schema previously registered to the Schema Registry service and keep the schema cached for future encoding usage. In order to avoid pre-registering the schema to the service and automatically register it with theencodemethod, the keyword argumentauto_register=Trueshould be passed to theAvroEncoderconstructor.importosfromazure.schemaregistryimportSchemaRegistryClientfromazure.schemaregistry.encoder.avroencoderimportAvroEncoderfromazure.identityimportDefaultAzureCredentialfromazure.eventhubimportEventDatatoken_credential=DefaultAzureCredential()fully_qualified_namespace=os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']group_name=os.environ['SCHEMAREGISTRY_GROUP']name="example.avro.User"format="Avro"definition="""{"namespace": "example.avro","type": "record","name": "User","fields": [{"name": "name", "type": "string"},{"name": "favorite_number",  "type": ["int", "null"]},{"name": "favorite_color", "type": ["string", "null"]}]}"""schema_registry_client=SchemaRegistryClient(fully_qualified_namespace,token_credential)schema_registry_client.register_schema(group_name,name,definition,format)encoder=AvroEncoder(client=schema_registry_client,group_name=group_name)withencoder:dict_content={"name":"Ben","favorite_number":7,"favorite_color":"red"}event_data=encoder.encode(dict_content,schema=definition,message_type=EventData)# ORmessage_content_dict=encoder.encode(dict_content,schema=definition)event_data=EventData.from_message_content(message_content_dict["content"],message_content_dict["content_type"])DecodingUse theAvroEncoder.decodemethod to decode the Avro-encoded content by either:Passing in a message object that is a subtype of the MessageType protocol.Passing in a dict with keyscontent(type bytes) andcontent_type(type string).
The method automatically retrieves the schema from the Schema Registry Service and keeps the schema cached for future decoding usage.importosfromazure.schemaregistryimportSchemaRegistryClientfromazure.schemaregistry.encoder.avroencoderimportAvroEncoderfromazure.identityimportDefaultAzureCredentialtoken_credential=DefaultAzureCredential()fully_qualified_namespace=os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']group_name="<your-group-name>"schema_registry_client=SchemaRegistryClient(fully_qualified_namespace,token_credential)encoder=AvroEncoder(client=schema_registry_client)withencoder:# event_data is an EventData object with Avro encoded bodydict_content={"name":"Ben","favorite_number":7,"favorite_color":"red"}event_data=encoder.encode(dict_content,schema=definition,message_type=EventData)decoded_content=encoder.decode(event_data)# ORencoded_bytes=b'<content_encoded_by_azure_schema_registry_avro_encoder>'content_type='avro/binary+<schema_id_of_corresponding_schema>'content_dict={"content":encoded_bytes,"content_type":content_type}decoded_content=encoder.decode(content_dict)Event Hubs Sending IntegrationIntegration withEvent Hubsto send anEventDataobject withbodyset to Avro-encoded content and correspondingcontent_type.importosfromazure.eventhubimportEventHubProducerClient,EventDatafromazure.schemaregistryimportSchemaRegistryClientfromazure.schemaregistry.encoder.avroencoderimportAvroEncoderfromazure.identityimportDefaultAzureCredentialtoken_credential=DefaultAzureCredential()fully_qualified_namespace=os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']group_name=os.environ['SCHEMAREGISTRY_GROUP']eventhub_connection_str=os.environ['EVENT_HUB_CONN_STR']eventhub_name=os.environ['EVENT_HUB_NAME']definition="""{"namespace": "example.avro","type": "record","name": "User","fields": [{"name": "name", "type": "string"},{"name": "favorite_number",  "type": ["int", "null"]},{"name": "favorite_color", "type": ["string", "null"]}]}"""schema_registry_client=SchemaRegistryClient(fully_qualified_namespace,token_credential)avro_encoder=AvroEncoder(client=schema_registry_client,group_name=group_name,auto_register=True)eventhub_producer=EventHubProducerClient.from_connection_string(conn_str=eventhub_connection_str,eventhub_name=eventhub_name)witheventhub_producer,avro_encoder:event_data_batch=eventhub_producer.create_batch()dict_content={"name":"Bob","favorite_number":7,"favorite_color":"red"}event_data=avro_encoder.encode(dict_content,schema=definition,message_type=EventData)event_data_batch.add(event_data)eventhub_producer.send_batch(event_data_batch)Event Hubs Receiving IntegrationIntegration withEvent Hubsto receive anEventDataobject and decode the the Avro-encodedbodyvalue.importosfromazure.eventhubimportEventHubConsumerClientfromazure.schemaregistryimportSchemaRegistryClientfromazure.schemaregistry.encoder.avroencoderimportAvroEncoderfromazure.identityimportDefaultAzureCredentialtoken_credential=DefaultAzureCredential()fully_qualified_namespace=os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']group_name=os.environ['SCHEMAREGISTRY_GROUP']eventhub_connection_str=os.environ['EVENT_HUB_CONN_STR']eventhub_name=os.environ['EVENT_HUB_NAME']schema_registry_client=SchemaRegistryClient(fully_qualified_namespace,token_credential)avro_encoder=AvroEncoder(client=schema_registry_client,group_name=group_name)eventhub_consumer=EventHubConsumerClient.from_connection_string(conn_str=eventhub_connection_str,consumer_group='$Default',eventhub_name=eventhub_name,)defon_event(partition_context,event):decoded_content=avro_encoder.decode(event)witheventhub_consumer,avro_encoder:eventhub_consumer.receive(on_event=on_event,starting_position="-1")TroubleshootingGeneralAzure Schema Registry Avro Encoder raises exceptions defined inAzure Coreif errors are encountered when communicating with the Schema Registry service. Errors related to invalid content/content types and invalid schemas will be raised asazure.schemaregistry.encoder.avroencoder.InvalidContentErrorandazure.schemaregistry.encoder.avroencoder.InvalidSchemaError, respectively, where__cause__will contain the underlying exception raised by the Apache Avro library.LoggingThis library uses the standardlogginglibrary for logging.
Basic information about HTTP sessions (URLs, headers, etc.) is logged at INFO
level.Detailed DEBUG level logging, including request/response bodies and unredacted
headers, can be enabled on a client with thelogging_enableargument:importsysimportosimportloggingfromazure.schemaregistryimportSchemaRegistryClientfromazure.schemaregistry.encoder.avroencoderimportAvroEncoderfromazure.identityimportDefaultAzureCredential# Create a logger for the SDKlogger=logging.getLogger('azure.schemaregistry')logger.setLevel(logging.DEBUG)# Configure a console outputhandler=logging.StreamHandler(stream=sys.stdout)logger.addHandler(handler)fully_qualified_namespace=os.environ['SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE']group_name=os.environ['SCHEMAREGISTRY_GROUP']credential=DefaultAzureCredential()schema_registry_client=SchemaRegistryClient(fully_qualified_namespace,credential,logging_enable=True)# This client will log detailed information about its HTTP sessions, at DEBUG levelencoder=AvroEncoder(client=schema_registry_client,group_name=group_name)Similarly,logging_enablecan enable detailed logging for a single operation,
even when it isn't enabled for the client:encoder.encode(dict_content,schema=definition,logging_enable=True)Next stepsMore sample codeFurther examples demonstrating common Azure Schema Registry Avro Encoder scenarios are in thesamplesdirectory.ContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.microsoft.com.When you submit a pull request, a CLA-bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.Release History1.0.0 (2022-05-10)Note:This is the first stable release of our efforts to create a user-friendly Pythonic Avro Encoder library that integrates with the Python client library for Azure Schema Registry.Features AddedAvroEncodersync and async classes provide the functionality to encode and decode content which follows a schema with the RecordSchema format, as defined by the Apache Avro specification. The Apache Avro library is used as the implementation for encoding and decoding.
The encoder will automatically register and retrieve schemas from Azure Schema Registry Service. It provides the following methods:constructor: Ifauto_register=Truekeyword is passed in, will automatically register schemas passed in to theencodemethod. Otherwise, and by default, will require pre-registering of schemas passed toencode. Takes agroup_nameargument that is optional when decoding, but required for encoding.encode: Encodes dict content into bytes according to the given schema and registers schema if needed. Returns either a dict of encoded content and corresponding content type or aMessageTypesubtype object, depending on arguments provided.decode: Decodes bytes content into dict content by automatically retrieving schema from the service.MessageContentTypedDict has been introduced with the following required keys:content: The bytes content.content_type: The string content type, which holds the schema ID and the record format indicator.MessageTypehas been introduced with the following methods:from_message_content: Class method that creates an object with given bytes content and string content type.__message_content__: Returns aMessageContentobject with content and content type values set to their respective properties on the object.Schemas and Schema IDs are cached locally, so that multiple calls with the same schema/schema ID will not trigger multiple service calls.The number of hits, misses, and total entries for the schema/schema ID caches will be logged at an info level when a new entry is added.InvalidContentErrorhas been introduced for errors related to invalid content and content types, where__cause__will contain the underlying exception raised by the Avro library.InvalidSchemaErrorhas been introduced for errors related to invalid schemas, where__cause__will contain the underlying exception raised by the Apache Avro library.Theencodeanddecodemethods onAvroEncodersupport the following message models:azure.eventhub.EventDatainazure-eventhub>=5.9.0Other ChangesThis package is meant to replace the azure-schemaregistry-avroserializer package, which will no longer be supported.group_nameis now an optional parameter in the sync and asyncAvroEncoderconstructors.1.0.0b3 (2022-04-05)Breaking Changesauto_register_schemaskeyword in the sync and asyncAvroEncoderconstructors has been renamedauto_register.SchemaParseError,SchemaEncodeError, andSchemaDecodeErrorhave been replaced withInvalidContentErrorandInvalidSchemaError. The errors have been added under theazure.schemaregistry.encoder.avroencodernamespace.Theexceptionsmodule inazure.schemaregistry.encoder.avroencoderhas been removed.Theencodemethod on the sync and asyncAvroEncoderonly allows subtypes of theMessageTypeprotocol as values to themessage_typeoptional parameter, rather than any callable that has the method signature(content: bytes, content_type: str, **kwargs: Any).The number of hits/misses, in addition to number of entries, for the schema/schema ID caches will be logged at an info level when a new entry is added.Other ChangesThis release and future releases will not have backward compatibility support for decoding data that was encoded with the AvroSerializer.Theencodeanddecodemethods onAvroEncodersupport the following message models:azure.eventhub.EventDatainazure-eventhub==5.9.0b31.0.0b2 (2022-03-09)Features Addedrequest_optionshas been added toencodeanddecodeonAvroEncoderas an optional parameter to be passed into client requests.The size of the current schema/schema ID caches will be logged at an info level when a new entry has been added.Breaking ChangesMessageMetadataDicthas been renamedMessageContent.datainMessageContenthas been renamedcontent.Thedataparameter inencodeanddecodeon the sync and asyncAvroEncoderhas been renamedcontent.Thefrom_message_datamethod in theMessageTypeprotocol has been renamedfrom_message_content. Thedataparameter infrom_message_contenthas been renamedcontent.The__message_data__method in theMessageTypeprotocol has been renamed__message_content__.Other ChangesThis beta release will be backward compatible for decoding data that was encoded with the AvroSerializer.Theencodeanddecodemethods onAvroEncodersupport the following message models:azure.eventhub.EventDatainazure-eventhub==5.9.0b21.0.0b1 (2022-02-09)This version and all future versions will require Python 3.6+. Python 2.7 is no longer supported.Features AddedThis package is meant to replace the azure-schemaregistry-avroserializer.APIs have been updated to allow for encoding directly to and decoding from message type objects, where the data value is the Avro encoded payload.The content type of the message will hold the schema ID and record format indicator.Other ChangesThis beta release will be backward compatible for decoding data that was encoded with the AvroSerializer.Theencodeanddecodemethods onAvroEncodersupport the following message models:azure.eventhub.EventDatainazure-eventhub==5.9.0b1