Package: clp-logging
CLP Python Logging LibraryThis is a Pythonlogginglibrary meant to supplementCLP (Compressed Log Processor).
Logs are compressed in a streaming fashion into CLP's Internal Representation (IR) format before written to disk.
More details are described in thisUber's blog.Logs compressed in IR format can be viewed in alog vieweror programmatically analyzed using
APIs provided here. They can also be decompressed back into plain-text log files usingCLP(in a future release).To achieve the best compression ratio, CLP should be used to compress large
batches of logs, one batch at a time. However, individual log
files are generally small and are generated across a long period of time.This logging library helps solve this problem by logging directly in CLP's
Internal Representation (IR). A log created with a CLP logging handler is first
parsed and then appended to a compressed output stream in IR form.
SeeREADME-protocol.mdfor more details on the format of
CLP IR.These log files containing the compressed CLP IR streams can then all be
ingested into CLP together at a later time.Quick StartThe package is hosted with pypi (https://pypi.org/project/clp-logging/), so it
can be installed withpip:python3 -m pip install --upgrade clp-loggingLogger handlersCLPStreamHandlerWrites encoded logs directly to a streamCLPFileHandlerSimple wrapper around CLPStreamHandler that calls openExample: CLPFileHandlerimportloggingfrompathlibimportPathfromclp_logging.handlersimportCLPFileHandlerclp_handler=CLPFileHandler(Path("example.clp.zst"))logger=logging.getLogger(__name__)logger.addHandler(clp_handler)logger.warn("example warning")CLPSockHandler + CLPSockListenerThis library also supports multiple processes writing to the same log file.
In this case, all logging processes write to a listener server process through a TCP socket.
The socket name is the log file path passed to CLPSockHandler with a ".sock" suffix.A CLPSockListener can be explicitly created (and will run as a daemon) by calling:CLPSockListener.fork(log_path, sock_path, timezone, timestamp_format).
Alternatively CLPSockHandlers can transparently start an associated CLPSockListener
by callingCLPSockHandlerwithcreate_listener=True.CLPSockListener must be explicitly stopped once logging is completed.
There are two ways to stop the listener process:Callingstop_listener()from an existing handler, e.g.,clp_handler.stop_listener(), or from a new handler with the same log path, e.g.,CLPSockHandler(Path("example.clp.zst")).stop_listener()Kill the CLPSockListener process with SIGTERMExample: CLPSockHandler + CLPSockListenerIn the handler processes or threads:importloggingfrompathlibimportPathfromclp_logging.handlersimportCLPSockHandlerclp_handler=CLPSockHandler(Path("example.clp.zst"),create_listener=True)logger=logging.getLogger(__name__)logger.addHandler(clp_handler)logger.warn("example warning")In a single process or thread once logging is completed:frompathlibimportPathfromclp_logging.handlersimportCLPSockHandlerCLPSockHandler(Path("example.clp.zst")).stop_listener()CLP readers (decoders)CLPStreamReaderRead/decode any arbitrary streamCan be used as an iterator that returns each log message as an objectCan skip n logs:clp_reader.skip_nlogs(N)Can skip to first log after given time (since unix epoch):clp_reader.skip_to_time(TIME)CLPFileReaderSimple wrapper around CLPStreamHandler that calls openExample code: CLPFileReaderfrompathlibimportPathfromtypingimportListfromclp_logging.readersimportCLPFileReader,Log# create a list of all Log objectslog_objects:List[Log]=[]withCLPFileReader(Path("example.clp.zst"))asclp_reader:forloginclp_reader:log_objects.append(log)CLPSegmentStreamingClasses that inherit from CLPBaseReader can only read a single CLP IR stream from start to finish. This is necessary because, to determine the timestamp of an individual log, the starting timestamp (from the IR stream preamble) and all timestamp deltas up to that log must be known. In scenarios where an IR stream is periodically uploaded in chunks, users would need to either continuously read the entire stream or re-read the entire stream from the start.The CLPSegmentStreaming class has the ability to take an input IR stream and segment it, outputting multiple independent IR streams. This makes it possible to read arbitrary segments of the original input IR stream without needing to decode it from the start.In technical terms, the segment streaming reader allows the read operation to start from a non-zero offset and streams the legally encoded logs from one stream to another.Each read call will return encoded metadata that can be used to resume from the current call.Example code: CLPSegmentStreamingfromclp_logging.readersimportCLPSegmentStreamingfromclp_logging.protocolimportMetadatasegment_idx:int=0segment_max_size:int=8192offset:int=0metadata:Metadata=NonewhileTrue:bytes_read:intwithopen("example.clp","rb")asfin,open(f"{segment_idx}.clp","wb")asfout:bytes_read,metadata=CLPSegmentStreaming.read(fin,fout,offset=offset,max_bytes_to_write=segment_max_size,metadata=metadata)segment_idx+=1offset+=bytes_readifmetadata==None:breakIn the example code provided, "example.clp" is streamed into segments named "0.clp", "1.clp", and so on. Each segment is smaller than 8192 bytes and can be decoded independently.Log level timeout feature: CLPLogLevelTimeoutAll log handlers support a configurable timeout feature. A (user configurable)
timeout will be scheduled based on logs' logging level (verbosity) that will
flush the zstandard frame and allow users to run arbitrary code.
This feature allows users to automatically perform log related tasks, such as
periodically uploading their logs for storage. By setting the timeout in
response to the logs' logging level the responsiveness of a task can be
adjusted based on the severity of logging level seen.
An additional timeout is always triggered on closing the logging handler.See the class documentation for specific details.Example code: CLPLogLevelTimeoutimportloggingfrompathlibimportPathfromclp_logging.handlersimportCLPLogLevelTimeout,CLPSockHandlerclassLogTimeoutUploader:# Store relevent information/objects to carry out the timeout taskdef__init__(self,log_path:Path)->None:self.log_path:Path=log_pathreturn# Create any methods necessary to carry out the timeout taskdefupload_log(self)->None:# upload the logs to the cloudreturndeftimeout(self)->None:self.upload_log()log_path:Path=Path("example.clp.zst")uploader=LogTimeoutUploader(log_path)loglevel_timeout=CLPLogLevelTimeout(uploader.timeout)clp_handler=CLPSockHandler(log_path,create_listener=True,loglevel_timeout=loglevel_timeout)logging.getLogger(__name__).addHandler(clp_handler)CompatibilityTested on Python 3.6, 3.8, and 3.11 (should also work on newer versions).
Built/packaged on Python 3.8 for convenience regarding type annotation.DevelopmentSetup environmentCreate and enter a virtual environment:python3.8 -m venv venv; . ./venv/bin/activateInstall the project in editable mode, the development dependencies, and the test dependencies:pip install -e .[dev,test]Note: you may need to upgrade pip first for-eto work. If so, run:pip install --upgrade pipPackagingTo build a package for distribution run:python -m buildTestingTo run the unit tests run:python -m unittest -bvNote: the baseline comparison logging handler and the CLP handler both get
unique timestamps. It is possible for these timestamps to differ, which will
result in a test reporting a false positive error.ContributingBefore submitting a pull request, run the following error-checking and
formatting tools (found inrequirements-dev.txt):mypy:mypy src testsmypy checks for typing errors. You should resolve all typing errors or if an
error cannot be resolved (e.g., it's due to a third-party library), you
should add a comment# type: ignoretosilencethe error.docformatter:docformatter -i src testsThis formats docstrings. You should review and add any changes to your PR.Black:black src testsThis formats the code according to Black's code-style rules. You should
review and add any changes to your PR.ruff:ruff check --fix src testsThis performs linting according to PEPs. You should review and add any
changes to your PR.Note thatdocformattershould be run beforeblackto give Black thelast
word.