Package: tom-alertstreams
tom-alertstreamstom-alertstreamsis a reusable TOM Toolkit app for listening to kafka streams.tom-alertstreamsprovides a management command,readstreams. There are nourlpatterns,
no Views, and no templates. Thereadstreamsmanagement command reads thesettings.pyALERT_STREAMSconfiguration and starts listening to each configured Kafka stream. It is not expected
to return, and is intended to run along side your TOM's server component. TheALERT_STREAMSconfiguration (see below) tellsreadstreamswhat streams to access, how to access them,
what topics to listen to, and what to do with messages that arrive on a given topic.InstallationInstall the package into your TOM environment:pipinstalltom-alertstreamsIn your projectsettings.py, addtom_alertstreamsto yourINSTALLED_APPSsetting:INSTALLED_APPS=[...'tom_alertstreams',]At this point you can verify the installation by running./manage.pyto list the available
management commands and see[tom_alertstreams]readstreamsin the output.ConfigurationEach Kafka stream that your TOM listens to (viareadstreams) will have a configuration dictionary
in yoursettings.pyALERT_STREAMS.ALERT_STREAMSis a list of configuration dictionaries, one
dictionary for each Kafka stream. Here's an exampleALERT_STREAMSconfiguration for two Kafka streams:SCiMMA HopskotchandGCN Classic over Kafka.ALERT_STREAMS=[{'ACTIVE':True,'NAME':'tom_alertstreams.alertstreams.hopskotch.HopskotchAlertStream','OPTIONS':{'URL':'kafka://kafka.scimma.org/',# The hop-client requires that the GROUP_ID prefix match the SCIMMA_AUTH_USERNAME'GROUP_ID':os.getenv('SCIMMA_AUTH_USERNAME',"")+'-'+'uniqueidforyourapp12345','USERNAME':os.getenv('SCIMMA_AUTH_USERNAME',None),'PASSWORD':os.getenv('SCIMMA_AUTH_PASSWORD',None),'TOPIC_HANDLERS':{'sys.heartbeat':'tom_alertstreams.alertstreams.hopskotch.heartbeat_handler','tomtoolkit.test':'tom_alertstreams.alertstreams.hopskotch.alert_logger','hermes.test':'tom_alertstreams.alertstreams.hopskotch.alert_logger','hermes.*':'regex match public topics here, requires * handler to be defined''*':'default_handler_here'},},},{'ACTIVE':True,'NAME':'tom_alertstreams.alertstreams.gcn.GCNClassicAlertStream',# The keys of the OPTIONS dictionary become (lower-case) properties of the AlertStream instance.'OPTIONS':{# see https://github.com/nasa-gcn/gcn-kafka-python#to-use for configuration details.'GCN_CLASSIC_CLIENT_ID':os.getenv('GCN_CLASSIC_CLIENT_ID',None),'GCN_CLASSIC_CLIENT_SECRET':os.getenv('GCN_CLASSIC_CLIENT_SECRET',None),'DOMAIN':'gcn.nasa.gov',# optional, defaults to 'gcn.nasa.gov''CONFIG':{# optional# 'group.id': 'tom_alertstreams-my-custom-group-id',# 'auto.offset.reset': 'earliest',# 'enable.auto.commit': False},'TOPIC_HANDLERS':{'gcn.classic.text.LVC_INITIAL':'tom_alertstreams.alertstreams.alertstream.alert_logger','gcn.classic.text.LVC_PRELIMINARY':'tom_alertstreams.alertstreams.alertstream.alert_logger','gcn.classic.text.LVC_RETRACTION':'tom_alertstreams.alertstreams.alertstream.alert_logger',},},}]The configuration dictionary for eachAlertStreamsubclass will contain these key-value pairs:ACTIVE: Boolean which tellsreadstreamsto access this stream. Should beTrue, unless you want to
keep a configuration dictionary, but ignore the stream.NAME: The name of theAlertStreamsubclass that implements the interface to this Kafka stream.tom_alertstreamswill provideAlertStreamsubclasses for major astromical Kafka streams. See below for instructions on Subclassing
theAlertStreambase class.OPTIONS: A dictionary of key-value pairs specific to theAlertStreamsubclass given byNAME. The doc string for
theAlertStreamsubclass should document what is expected. Typically, a URL, authentication information, and a
dictionary,TOPIC_HANDLERS, will be required. See "SubclassingAlertStream" below. TheAlertStreamsubclass will
convert the key-value pairs of theOPTIONSdictionary into properties (and values) of theAlertStreamsubclass
instance.The hopskotch alert stream supports a wildcard of*for an alert handler topic name. If specified, ALL public topics will be subscribed and use that handler function. A directly specified topic handler will always be used before the*handler for any topic that is covered twice.Getting Kafka Stream CredentialsAs part of yourOPTIONSfor each Kafka stream, you need to configure access credentials. Visit these links
to get credentials forHopskotchandGCN Classic over Kafka.
Set the environment variables with the username and passwords obtained. Do not check them in to your code repository.Alert HandlingAssuming that anAlertStreamsubclass exists for the Kafka stream of interest,
the keys of theTOPIC_HANDLERSdictionary are the topics that will be subscribed to. The values
of theTOPIC_HANDLERSdictionary specify alert handling methods that will be imported and called
for each alert recieved on that topic. An example is provided,tom_alerts.alertstreams.alertstream.alert_logger, which simply logs the alert.To customize this behaviour according to the needs of your TOM, define an alert handling function for each
topic that you wish to subscribe to. YourTOPIC_HANDLERSdictionary will have a an entry for each topic
whose key is the topic name and whose value is a string indicating the dot-path to the alert handling function.
When theAlertStreamsubclass is instanciated, theOPTIONSdictionary is read and analert_handlerdictionary is created. It is keyed by topic name and it's values are the imported callable functions specified by the
dot-path strings.readstreamswill call the alert handler for each alert that comes in on the topic. The signiture
of the alert handling function is specific to theAlertStreamsubclasss.SubclassingAlertStreamIdeally, As a TOM developer, there is already anAlertStream-subclass for the alert stream that you
want your TOM to listen to. If so, you need only to configure your TOM to use it insettings.pyALERT_STREAMS. If you must implement your ownAlertStreamsubclass, please get in touch. In the meantime, here's a brief outline:Create subclass ofAlertStream.Createrequired_keysandallowed_keysclass variables in yourAlertStream-subclass.These are lists of strings refering to the keys of theOPTIONSdictionary. The purpose of these is to
help TOM developers using yourAlertStream-subclass with the key-value pairs in theirALERT_STREAMSOPTIONSconfiguration dictionary.Implement thelisten()method.This method will be called by thereadstreamsmanagement command and is not expected to return. It
should instanciate your consumer, subscribe to the topics configured inALERT_STREAMS, and start
consuming. The detail of this will depend on the kafka-client used. Seealertstreams.gcn.listen()andalertstreams.hopskotch.listen()for examples to follow.The loop which consumes messages in yourlisten()method should extract the topic from each message
and callself.alert_handler[topic]()with the message or message-derived arguments specific to your
kafka client. Users of yourAlertStream-subclass will write these topic-specific alert handling methods
and configure them in theTOPIC_HANLDERSdictionary of theirALERT_STREAMSconfiguration.
TheAlertStreambase class will set up thealert_handlerdictionary according to your users'
configuration. It helps your users to provide an examplealert_hander()function in your module as
an example. (Again, seealertstreams.gcn.listen()andalertstreams.hopskotch.listen(), their
configurations insettings.py, and thealertstreams.gcn.alert_logger()and
`alertstreams.hopskotch.alert_logger() methods, for example).