Package: balog
balog=====[![Build Status](https://travis-ci.org/balanced/balog.svg?branch=master)](https://travis-ci.org/balanced/balog)Balanced event logging schema and libraryLog schema design goals=======================- Schema version annotated- Provide sufficient information about the event- Open and close (be flexible, avoid unnecessary change to schema)Design rationals================OSI Network Model Structure---------------------------Data in the log can be divided into two major groups, one is the meta data,which is the data about the event. The another group is the log for applicationitself. In most cases, applications should only be interested in theapplication log itself. And the logging facility should only be interested inhow to handle the whole event rather than to consume the content of applicationlog. Also, since schema of application can variant, we don't want change thewhole schema everytime when application log is change. Consider these issues,one idea come to my mind is Internet OSI model is actually dealing with thesame issue. It's like TCP/IP protocol doesn't need to know the content ofapplication layer. Similarly, logging facility doesn't have to know the contentof application log. With this idea in mind, I define two layers for our loggingsystem.- Facility layer- Application layerFacility layer is all about the event, when it is generated, who emited thisevent, what's its routing tag and etc. Application layer is all aboutapplication data, like a dispute is processed, a debit is created and etc.Usage=====Produce logs------------To produce a log, here you can write```pythonfrom balog import get_loggerbalogger = get_logger(__name__)balogger.info('done', payload={'cls_type': 'metrics','values': [{'name': 'total', 'value': 123},{'name': 'succeeded', 'value': 456},{'name': 'failed', 'value': 789},],})```The channel name will be the logger name + the given suffix name, in the above example, the say if the `__name__` is`justitia.scripts.process_results` here, then the channel name will be `justitia.scripts.process_results.done`. If you want to overwrite the channel name, you can also pass `channel` argument to balog logging methods.Consume logs------------To consume events, you can use `consumer_config` like this```pythonfrom balog.consumers import consumer_config@consumer_config(topic='myproj-events-{env}',cls_type='metrics',version='<1.0',)def process_metrics(settings, event):pass```This `consumer_config` decorator is mainly for declaring what this consumer wants, in the example above, since want to subscribe the queue `myproj-events-develop` or `myproj-events-prod`, so we set the topic to `myproj-events-{env}`, for the `{env}` placeholder, we will talk about that later. And then we're only interested in `metrics` type events, so we set `cls_type` to `metrics`. We also don't want to process events that are not compatible, so we set the `version` to `<1.0`.With these configured consumers, to process events, you need to use `ConsumerHub` first. It is basically a collection of consumers. It provides scanning method to make collecting consumers pretty easy. For example, here you can write```pythonimport yourpackagefrom balog.consumers import ConsumerHubhub = ConsumerHub()hub.scan(yourpackage)```By doing that, you have all consuemrs in the hub. Then you can create an event processing engine. We only have Amazon SQS event processing engine for now, but it should be fairly easy to impelement any other engine type as they are needed. Before we create the engine, you need to define a `ConsumerOperator` first. It's basically an object which knows how to operate these consumer. So that you can call these consumer functions in the way you want. Here is an example:```pythonclass ConsumerOperator(object):def __init__(self, settings):self.settings = settingsdef get_topic(self, consumer):env = self.settings['api.sqs.queue_env']return consumer.topic.format(env=env)def process_event(self, consumer, event):return consumer.func(self.settings, event)```It has two methods, the first is `get_topic`, it is for getting topic, namely, the queue name to subscribe. As we saw the topic name in `consumer_config` is `myproj-events-{env}` above, here I explain why it looks like that. Since we may need to subscribe to different event queue in different environment, like integration or production environment, so we cannot just leave a static value there. To make it as flexible as possible, I let the consumer operator decide how to get the topic name. So in our example, you can see it gets `api.sqs.queue_env` from settings and replace the `env` placeholder. By doing that, we can determine which queue to subscribe later. Similar to the deferred topic name, I also want to make it possible to call the consumer function in the way you like, the `process_event` method of consumer operator is for that purpose. Since in our consumer handler function, we may need some extra information other than just the event, in our example, we also need to read settings. So that in our example, we pass an extra argument `settings` to the consumer function.Okay, now let's put it altogether```pythonfrom __future__ import unicode_literalsfrom balog.consumers import ConsumerHubfrom balog.consumers.sqs import SQSEngineimport yourpackageclass ConsumerOperator(object):def __init__(self, settings):self.settings = settingsdef get_topic(self, consumer):env = self.settings['api.sqs.queue_env']return consumer.topic.format(env=env)def process_event(self, consumer, event):return consumer.func(self.settings, event)def process_events(settings):"""Process log events in SQS"""hub = ConsumerHub()hub.scan(yourpackage)consumer_op = ConsumerOperator(settings)engine = SQSEngine(hub=hub,region=settings['api.sqs.region'],aws_access_key_id=settings['api.sqs.aws_access_key_id'],aws_secret_access_key=settings['api.sqs.aws_secret_access_key'],consumer_operator=consumer_op,)engine.run()```