Package: gcn-kafka
GCN Kafka Client for PythonThis is the official Python client for theGeneral Coordinates Network (GCN). It is a very lightweight wrapper aroundconfluent-kafka-python.To InstallRun this command to install withpip:pip install gcn-kafkaor this command to install with withconda:conda install -c conda-forge gcn-kafkaTo UseCreate a consumer.fromgcn_kafkaimportConsumerconsumer=Consumer(client_id='fill me in',client_secret='fill me in')List all topics:print(consumer.list_topics().topics)Subscribe to topics and receive alerts:consumer.subscribe(['gcn.classic.text.FERMI_GBM_FIN_POS','gcn.classic.text.LVC_INITIAL'])whileTrue:formessageinconsumer.consume(timeout=1):ifmessage.error():print(message.error())continueprint(message.value())Thetimeoutargument toconsume(), given as an integer number of seconds,
will allow the program to exit quickly once it has reached the end of the
existing message buffer. This is useful for users who just want to recover an
older message from the stream.timeoutwill also make thewhile Trueinfinite loop interruptible via the standard ctrl-c key sequence, whichconsume()ignores.Testing and Development Kafka ClustersGCN has three Kafka clusters: production, testing, and an internal development deployment. Use the optionaldomainkeyword argument to select which broker to connect to.# Production (default)consumer=Consumer(client_id='fill me in',client_secret='fill me in',domain='gcn.nasa.gov')# Testingconsumer=Consumer(client_id='fill me in',client_secret='fill me in',domain='test.gcn.nasa.gov')# Development (internal)consumer=Consumer(client_id='fill me in',client_secret='fill me in',domain='dev.gcn.nasa.gov')FAQHow can I keep track of the last read message when restarting a client?A key feature of kafka consumer clients is the ability to perform persistent tracking of which messages have been read. This allows clients to recover missed messages after a   restart by beginning at the earliest unread message rather than the next available message from the stream. In order to enable this feature, you will need to set a client Group ID using the configuration dictionary argument for the Consumer class as well as change the auto offset reset option to the ‘earliest’ setting. Once this is done, every new client with the given Group ID will begin reading the specified topic at the earliest unread message. When doing this, it is recommended to turn OFF the auto commit feature because it can lose track of the last read message if the client crashes before the auto commit interval (5 seconds by default) occurs. Manually committing messages (i.e. storing the state of the last read message) once they are read is the most robust method for tracking the last read message.Example code:fromgcn_kafkaimportConsumerconfig={'group.id':'my group name','auto.offset.reset':'earliest','enable.auto.commit':False}consumer=Consumer(config=config,client_id='fill me in',client_secret='fill me in',domain='gcn.nasa.gov')topics=['gcn.classic.voevent.FERMI_GBM_SUBTHRESH']consumer.subscribe(topics)whileTrue:formessageinconsumer.consume(timeout=1):print(message.value())consumer.commit(message)How can I read messages beginning at the earliest available messages for a given stream?You can begin reading a given topic stream from the earliest message that is present in the stream buffer by setting the Group ID to an empty string and applying the ‘earliest’ setting for the auto offset reset option in the configuration dictionary argument for the Consumer class. This feature allows the user to scan for older messages for testing purposes or to recover messages that may have been missed due to a crash or network outage. Just keep in mind that the stream buffers are finite in size. They currently hold messages from the past few days.Example code:fromgcn_kafkaimportConsumerconfig={'auto.offset.reset':'earliest'}consumer=Consumer(config=config,client_id='fill me in',client_secret='fill me in',domain='gcn.nasa.gov')topics=['gcn.classic.voevent.INTEGRAL_SPIACS']consumer.subscribe(topics)whileTrue:formessageinconsumer.consume(timeout=1):print(message.value())How can I search for messages occurring within a given date range?To search for messages in a given date range, you can use theoffsets_for_times()function from the Consumer class to get the message offsets for the desired date range. You can then assign the starting offset to the Consumer and read the desired number of messages. When doing so, keep in mind that the stream buffers are finite in size. It is not possible to recover messages prior to the start of the stream buffer. The GCN stream buffers are currently set to hold messages from the past few days.Example code:importdatetimefromgcn_kafkaimportConsumerfromconfluent_kafkaimportTopicPartitionconsumer=Consumer(client_id='fill me in',client_secret='fill me in',domain='gcn.nasa.gov')# get messages occurring 3 days agotimestamp1=int((datetime.datetime.now()-datetime.timedelta(days=3)).timestamp()*1000)timestamp2=timestamp1+86400000# +1 daytopic='gcn.classic.voevent.INTEGRAL_SPIACS'start=consumer.offsets_for_times([TopicPartition(topic,0,timestamp1)])end=consumer.offsets_for_times([TopicPartition(topic,0,timestamp2)])consumer.assign(start)formessageinconsumer.consume(end[0].offset-start[0].offset,timeout=1):print(message.value())Known Issuesconfluent-kafka-pythonIf you use confluent-kafka-python v2.1.0 or v2.1.1 with librdkafka v2.1.1 you will encounter a segmentation fault when subscribed to unavailable topics.Please refer tothe confluent-kafka-python github issuefor updates on the issue.