Package: distlog
Distlog=======Is a Python library and a set of applications to facilitate logging in adistributed application.The reason to create yet another logging tool arose while developing anapplication based on (micro) services. Even though each component produces aliberal amount of logging data making sense of it is challenging at times.This is because each log file only tells part of the story.The challenge is to find those parts of the log entries that are relevant to aparticular problem request. I tried to solve this by using correlation IDs.These IDs are assigned when a request first enters the system and is passed onto each service that is invoked to satisfy that request. It sort of works but ittakes a lot of time piecing all the bits together.Hence the desire to let the system sort it out itself.Requirements============My preferred solution should:Interoperate------------Interoperate with Python's native logging library.This will automatically include logging activity from incorporated librarieswhich nearly always use Python's native logging support.Distributed-----------Allow multiple clients to produce log entries.It is after all meant as a solution for a distributed application.Centralize all logs in a single location.This will make it possible to combine entries from different sourcesrelating to the same request.Allow multiple clients to process the log entries.Some will store the log entries on permanent storage (file, database).Others may filter the entries and display them as if tailing a logfile.Structured----------Offer additional structure to the log entries.Normal logs are a sequential but otherwise unstructured list of hopefullysignificant facts. What I'd like to see is that the log entries tell me whatis being achieved and wheter that succeeded or not. Each of these goals mayitself consist of subgoals creating a tree of goals which together reach aconclusion.For many years now we modularize and structure our code, yet none of that isvisible in the log entries that tell us how the program is progressing.The structure in the log entries should transcent process bounderies.When one process invokes the services of another process the structure ofthe log entries should reflect this. A process switch should not break thetree structure described above.Dataset oriented----------------Support record logging.Traditionally log entries are supposed to be a single text string. I want tosee all of Python's LogRecord fields in the log entry. And probably quite afew more fields as well. We can assume the LogRecord fields as a stablebasis set of properties that are expanded by custom properties depending onthe application and function.Oh and the whole thing should be modular off course.- Structured logging- Multi process logging- Record loggingShould be separate functional modules that can be combined to provide all ofthe above.There is just one little thing: it doesn't exist. Yet.The distlog package====================The distlog package exports a couple classes and functions:* :class:`ZmqHandler`This is a handler for the standard Python logging library that transports:class:`LogRecords`. For it to work it needs a serializer that prepares the:class:`LogRecord` instances for transportation. This serializer should bederived from the :class:`~distlog.formatters.Serializer` class. Choose oneof the predefined serializers :class:`JSONFormatter` or:class:`PickleFormatter` and set it as the formatter using:method:`ZmqHandler.setFormatter`.* :class:`JSONFormatter`This is one of the predefined serializers, it converts a :class:`LogRecord`instances to JSON strings that can be sent via 0MQ.* :class:`PickleFormatter`This is the other predefined serializer. it converts a :class:`LogRecord` tothe binary Python specific pickle format. Note that the serialization methodis part of the message so that the receiver knows how to deserialize themessage.* :function:`task`This function returns a :class:`Task` object that implements as a contexthandler and acts as the top level logging context for Pythons normal loggerfunctions.* :function:`to`This function returns a :class:`Task` instance that is a child of anotherlogging context thus constructing a hierarchy of logging contexts.* :function:`import_task`Creates a top level logging context that is subordinate to the loggingcontext in a different process. Using :method:`Task.get_foreign_task` toobtain a task/context id, transferring this id from one process to anotherand passing it as parameter to :function:`import_task` it is possible torelate the activities across processes as belonging to one and the sameactivity.Logging infrastructure======================The process(es) forming the application that is being logged send out their logmessages using :class:`ZmqHandler`. This class sends the message out using a0MQ PUSH socket that binds to the well-known address of the corresponding PULLsocket of the `distlogd` process.The `distlogd` process receives all log messages and filters them using aconfigurable rule set. Based on the rule(s) which triggered, the message isforwarded to one or more processors.The configuration file, probably a YAML file, may instruct `distlogd` to loadprocessors as plugins. A processor will apply one or more filters on a messageand uses the outcome to decide to handle a message or not.