Package: puavro
puavroThepuavrois a small convenience library enabling usage of theApache Pulsar Python clientwith pre-definedAVRO schemasandPython dictionariesinstead ofAVRO schemas declared as records.In other words, the library provides an interface to the standard Apache Pulsar client
allowing to read/write AVRO messages from/to Python dictionary using AVRO schema, either:declared as Python dictionary (usingfastavro.schema.parse_schema()) orloaded from .avsc file (usingfastavro.schema.load_schema()) orparsed from JSON string (usingfastavro.schema.parse_schema(json.loads())).Thepuavrolibrary consists of just two classes:DictAVROderived from Pythondictand designated to be used instead ofpulsar.schema.Recordclass;DictAvroSchemaderived frompulsar.schema.AvroSchemaand designated to be used instead.See also:How to useSamplesMotivationTo enable usage of Python Pulsar client with AVRO messages generated / received by modules written in other languages and using external AVRO schemas (e.g. stored in .avsc files).Installingpuavrois available on PyPi:pip install puavroDependenciesThe library depends on the following modules:fastavro>=1.4.4pulsar-client>=2.7.0CompatibilityThe library has been run and tested againstPulsar Python clientv. 2.7.0 and 2.8.0. andfastavrov. 1.4.4. and is expected to be compatible with all higher versions also.LicenseThe library is provided under terms of theMIT license.How to useThe samples in this sections assume the following imports:importpulsarimportfastavroimportpuavroimportjsonimportdatetimefrompprintimportppDefining dictionary for AVRO schemaclassSegment(puavro.DictAVRO):SCHEMA=fastavro.schema.load_schema("Segment.avsc")orclassSegment(puavro.DictAVRO):SCHEMA=fastavro.schema.parse_schema(json.loads("""{"type" : "record","name" : "Segment","namespace" : "try","fields" : [ {"name" : "id","type" : "long"}, {"name" : "name","type" : "string"}, {"name" : "when","type" : {"type" : "long","logicalType" : "timestamp-millis"}}, {"name" : "direction","type" : {"type" : "enum","name" : "CardinalDirection","symbols" : [ "north", "south", "east", "west" ]}}, {"name" : "length","type" : [ "null", "long" ]} ]}"""))orclassSegment(puavro.DictAVRO):SCHEMA=fastavro.schema.parse_schema({"type":"record","name":"Segment","namespace":"try","fields":[{"name":"id","type":"long"},{"name":"name","type":"string"},{"name":"when","type":{"type":"long","logicalType":"timestamp-millis"}},{"name":"direction","type":{"type":"enum","name":"CardinalDirection","symbols":["north","south","east","west"]}},{"name":"length","type":["null","long"]}]})orclassSegment(puavro.DictAVRO):passSegment.set_schema(fastavro.schema.load_schema("segment.avsc"))ProducerUsing classSegment(derived frompuavro.DictAVROabove) andpuavro.DictAvroSchemaclass (instead ofpulsar.schema.AvroSchema):PULSAR_SERVICE_URL="pulsar://localhost:6650"TOPIC="try"pulsar_client=pulsar.Client(PULSAR_SERVICE_URL)producer=pulsar_client.create_producer(topic=TOPIC,schema=puavro.DictAvroSchema(Segment))try:segment=Segment(id=99,name="some name",when=datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc),direction="north",length=12345,)producer.send(segment)finally:producer.close()pulsar_client.close()ConsumerUsing classSegment(derived frompuavro.DictAVROabove) andpuavro.DictAvroSchemaclass (instead ofpulsar.schema.AvroSchema):PULSAR_SERVICE_URL="pulsar://localhost:6650"TOPIC="try"WAIT_SECONDS=3pulsar_client=pulsar.Client(PULSAR_SERVICE_URL)consumer=pulsar_client.subscribe(TOPIC,subscription_name="sample",consumer_type=pulsar.ConsumerType.Shared,schema=puavro.DictAvroSchema(Segment))try:whileTrue:msg=consumer.receive(WAIT_SECONDS*1000)segment=msg.value()pp(segment)consumer.acknowledge(msg)exceptExceptionase:ifstr(e)=='Pulsar error: TimeOut':print("END OF DATA")else:raisefinally:consumer.close()pulsar_client.close()SamplesThe complete samples can be found in thesamplesdirectory:Producer:sender.pyConsumer:receiver.pyAVRO schema:Segment.avscAVRO IDL:try.avdl