Package: clp-ffi-py
clp-ffi-pyThis module provides Python packages to interface withCLP Core Featuresthrough CLP's FFI (foreign function interface). At present, this library
supplies built-in functions for encoding/decoding log messages usingCLP.Quick StartInstall withpip:# Install the latest versionpython3-mpipinstall--upgradeclp-ffi-pyNote:Python 3.6 or higher is required.Only Linux and macOS are supported at present.To install an older version or download the prebuiltwhlpackage, check the
project homepage on PyPIhere.CompatibilityTested on Python 3.8, 3.9 and 3.10, and it should work on any Python
version >= 3.6.Building/PackagingTo manually build a package for distribution, run the following steps. This
process will generate both .tar.gz package and .whl package under./dist/directory.# 1. Create and enter a virtual environmentpython-mvenvvenv&&../venv/bin/activate# 2. Install development dependenciespipinstall-rrequirements-dev.txt# 3. Pull all submodules in preparation for buildinggitsubmoduleupdate--init--recursive# 4. Buildpython-mbuild--outdirbuild/distCLP IR ReadersCLP IR Readers provide a convenient interface for CLP IR decoding and search
methods.ClpIrStreamReaderRead/decode any arbitrary CLP IR stream (as an instance ofIO[bytes]).Can be used as an iterator that returns each log event as aLogEventobject.Can search target log events by giving a search query:Searching log events within a certain time range.Searching log messages that match certain wildcard queries.ClpIrFileReaderSimple wrapper around CLPIRStreamHandler that callsopenwith a given local
path.Example Code: Using ClpIrFileReader to iterate and print log eventsfrompathlibimportPathfromclp_ffi_py.irimportClpIrFileReaderwithClpIrFileReader(Path("example.clp.zst"))asclp_reader:forlog_eventinclp_reader:# Print the log message with its timestamp properly formatted.print(log_event.get_formatted_message())Each log event is represented by aLogEventobject, which offers methods to
retrieve its underlying details, such as the timestamp and the log message. For
more information, use the following code to see all the available methods and
the associated docstring.fromclp_ffi_py.irimportLogEventhelp(LogEvent)Example Code: Using Query to search log events by specifying a certain time rangefromtypingimportListfromclp_ffi_py.irimportClpIrStreamReader,LogEvent,Query,QueryBuilder# Create a QueryBuilder object to build the search query.query_builder:QueryBuilder=QueryBuilder()# Create a search query that specifies a time range by UNIX epoch timestamp in# milliseconds. It will search from 2016.Nov.28 21:00 to 2016.Nov.29 3:00.time_range_query:Query=(query_builder.set_search_time_lower_bound(1480366800000)# 2016.11.28 21:00.set_search_time_upper_bound(1480388400000)# 2016.11.29 03:00.build())# A list to store all the log events within the search time rangelog_events:List[LogEvent]=[]# Open IRstream compressed log file as a binary file stream, then pass it to# CLpIrStreamReader.withopen("example.clp.zst","rb")ascompressed_log_file:withClpIrStreamReader(compressed_log_file)asclp_reader:forlog_eventinclp_reader.search(time_range_query):log_events.append(log_event)Example Code: Using Query to search log messages of certain pattern(s) specified by wildcard queries.frompathlibimportPathfromtypingimportList,Tuplefromclp_ffi_py.irimportClpIrFileReader,Query,QueryBuilder# Create a QueryBuilder object to build the search query.query_builder:QueryBuilder=QueryBuilder()# Add wildcard patterns to filter log messages:query_builder.add_wildcard_query("*uid=*,status=failed*")query_builder.add_wildcard_query("*UID=*,Status=KILLED*",case_sensitive=True)# Initialize a Query object using the builder:wildcard_search_query:Query=query_builder.build()# Store the log events that match the criteria in the format:# [timestamp, message]matched_log_messages:List[Tuple[int,str]]=[]# A convenience file reader class is also available to interact with a file that# represents an encoded CLP IR stream directly.withClpIrFileReader(Path("example.clp.zst"))asclp_reader:forlog_eventinclp_reader.search(wildcard_search_query):matched_log_messages.append((log_event.get_timestamp(),log_event.get_log_message()))AQueryobject may have both the search time range and the wildcard queries
(WildcardQuery) specified to support more complex search scenarios.QueryBuildercan be used to conveniently construct Query objects. For more
details, use the following code to access the related docstring.fromclp_ffi_py.irimportQuery,QueryBuilderfromclp_ffi_pyimportWildcardQueryhelp(Query)help(QueryBuilder)help(WildcardQuery)Streaming Decode/Search Directly from S3 Remote StorageWhen working with CLP IR files stored on S3-compatible storage systems,smart_opencan be used to open and read the IR stream for the following
benefits:It only performs stream operation and does not download the file to the disk.It only invokes a singleGETrequest so that the API access cost is
minimized.Here is an example:frompathlibimportPathfromclp_ffi_py.irimportClpIrStreamReaderimportboto3importosimportsmart_open# Create a boto3 session by reading AWS credentials from environment variables.session=boto3.Session(aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],)url='s3://clp-example-s3-bucket/example.clp.zst'# Using `smart_open.open` to stream the encoded CLP IR:withsmart_open.open(url,"rb",transport_params={'client':session.client('s3')})asistream:withClpIrStreamReader(istream,allow_incomplete_stream=True)asclp_reader:forlog_eventinclp_reader:# Print the log message with its timestamp properly formatted.print(log_event.get_formatted_message())Note:
Whenallow_incomplete_streamis set to False (default), the reader will raiseclp_ffi_py.ir.IncompleteStreamErrorif the stream is incomplete (it doesn't end
with the byte sequence indicating the stream's end). In practice, this can occur
if you're reading a stream that is still being written or wasn't properly
closed.Parallel ProcessingTheQueryandLogEventclasses can be serialized bypickle. Therefore,
decoding and search can be parallelized across streams/files using libraries
such asmultiprocessingandtqlm.Testing# 1. Create and enter a virtual environmentpython-mvenvvenv&&../venv/bin/activate# 2. Install development dependenciespipinstall-rrequirements-dev.txt# 3. Pull all submodules in preparation for buildinggitsubmoduleupdate--init--recursive# 4. Installpipinstall-e.# 5. Run unit testspython-munittest-bvNote: If the package is installed from awhlfile into the site packages,
rather than installed locally (pip install -e .), the tester cannot be
launched from the project's root directory. Ifunittestis ran from the root
directory, the localclp_ffi_pydirectory will shadow theclp_ffi_pymodule
installed. To run the tester with the installed package, try the following:cdtests
python-munittest-bvBuild and Test with cibuildwheelThis project utilizescibuildwheelconfiguration. Whenever modifications
are made and committed to GitHub, the cibuildwheel Action will automatically
initiate, building this library for several Python environments across diverse
OS and architectures. You can access the build outcomes (wheel files) via the
GitHub Action page. For instructions on customizing the build targets or running
cibuildwheel locally, please refer to the official documentation of
cibuildwheel.ContributingBefore submitting a pull request, run the following error-checking and
formatting tools (found in [pyproject.toml]):mypy:mypy clp_ffi_pymypy checks for typing errors. You should resolve all typing errors or if an
error cannot be resolved (e.g., it's due to a third-party library), you
should add a comment# type: ignoretosilencethe error.docformatter:docformatter -i clp_ffi_py testsThis formats docstrings. You should review and add any changes to your PR.Black:black clp_ffi_pyThis formats the Python code according to Black's code-style rules. You
should review and add any changes to your PR.clang-format:clang-format -i src/clp_ffi_py/**This formats the C++ code according to the code-style rules specified in.clang-format. You should review and add any changes to your PR.ruff:ruff check --fix clp_ffi_py testsThis performs linting according to PEPs. You should review and add any
changes to your PR.Note thatdocformattershould be run beforeblackto give Black thelast.Additionally, the following tools can be useful during development. However,
they cannot be installed usingpip. Developers need to install them using
other package management tools such asapt-get:clang-tidy:clang-tidy --extra-arg=-std=c++17 PATH_TO_THE_FILEThis static analysis tool catches improper coding behaviors based on the
rules specified in.clang-tidy, and sends suggestions corresponding to
each warning. Developers should manually review all the warnings and try
with their best effort to fix the reasonable ones.bear:bear python setup.py buildThis tool generates a JSON compilation database on the project's root
directory namedcompile_commands.json. This file will be used by
clang-tidy to execute the static analysis. It also helps IDEs to configure
code completion (such as VSCode).