Package: logprep
LogprepIntroductionLogprep allows to collect, process and forward log messages from various data sources.
Log messages are being read and written by so-called connectors.
Currently, connectors for Kafka, Opensearch, Opensearch and JSON(L) files exist.
Additionally, an Input Connector for HTTP Input is provided, which starts an uvicorn server and
accepts log message via POST Requests.The log messages are processed step-by-step by a pipeline of processors,
where each processor modifies an event that is being passed through.
The main idea is that each processor performs a simple task that is easy to carry out.
Once the log massage is passed through all processors in the pipeline the resulting
message is sent to a configured output connector.Logprep is designed to be expandable with new connectors and processors.Logprep is primarily designed to process log messages. Generally, Logprep can handle JSON messages,
allowing further applications besides log handling.This readme provides basic information about the following topics:About LogprepGetting StartedDocker QuickstartEvent GenerationDocumentationContributingLicenseMore detailed information can be found in theDocumentation.About LogprepPipelinesLogprep processes incoming log messages with a configured pipeline that can be spawned
multiple times via multiprocessing.
The following chart shows a basic setup that represents this behaviour.
The pipeline consists of three processors: theDissector,Geo-IP Enricherand theDropper.
Each pipeline runs concurrently and takes one event from it'sInput Connector.
Once the log messages is fully processed the result will be forwarded to theOutput Connector,
after which the pipeline will take the next message, repeating the processing cycle.flowchart LR
A1[Input\nConnector] --> B
A2[Input\nConnector] --> C
A3[Input\nConnector] --> D
subgraph Pipeline 1
B[Dissector] --> E[Geo-IP Enricher]
E --> F[Dropper]
end
subgraph Pipeline 2
C[Dissector] --> G[Geo-IP Enricher]
G --> H[Dropper]
end
subgraph Pipeline n
D[Dissector] --> I[Geo-IP Enricher]
I --> J[Dropper]
end
F --> K1[Output\nConnector]
H --> K2[Output\nConnector]
J --> K3[Output\nConnector]ProcessorsEvery processor has one simple task to fulfill.
For example, theDissectorcan split up long message fields into multiple subfields
to facilitate structural normalization.
TheGeo-IP Enricher, for example, takes an ip-address and adds the geolocation of it to the
log message, based on a configured geo-ip database.
Or theDropperdeletes fields from the log message.As detailed overview of all processors can be found in theprocessor documentation.To influence the behaviour of those processors, each can be configured with a set of rules.
These rules define two things.
Firstly, they specify when the processor should process a log message
and secondly they specify how to process the message.
For example which fields should be deleted or to which IP-address the geolocation should be
retrieved.For performance reasons on startup all rules per processor are aggregated to a generic and a specific rule tree, respectively.
Instead of evaluating all rules independently for each log message the message is checked against
the rule tree.
Each node in the rule tree represents a condition that has to be meet,
while the leafs represent changes that the processor should apply.
If no condition is met, the processor will just pass the log event to the next processor.The following chart gives an example of such a rule tree:flowchart TD
A[root]
A-->B[Condition 1]
A-->C[Condition 2]
A-->D[Condition 3]
B-->E[Condition 4]
B-->H(Rule 1)
C-->I(Rule 2)
D-->J(rule 3)
E-->G(Rule 4)To further improve the performance, it is possible to prioritize specific nodes of the rule tree,
such that broader conditions are higher up in the tree.
And specific conditions can be moved further down.
Following json gives an example of such a rule tree configuration.
This configuration will lead to the prioritization oftagsandmessagein the rule tree.{"priority_dict":{"category":"01","message":"02"},"tag_map":{"check_field_name":"check-tag"}}Instead of writing very specific rules that apply to single log messages, it is also possible
to define generic rules that apply to multiple messages.
It is possible to define a set of generic and specific rules for each processor, resulting
in two rule trees.ConnectorsConnectors are responsible for reading the input and writing the result to a desired output.
The main connectors that are currently used and implemented are a kafka-input-connector and a
kafka-output-connector allowing to receive messages from a kafka-topic and write messages into a
kafka-topic. Addionally, you can use the Opensearch or Opensearch output connectors to ship the
messages directly to Opensearch or Opensearch after processing.The details regarding the connectors can be found in theinput connector documentationandoutput connector documentation.ConfigurationTo run Logprep, certain configurations have to be provided. Because Logprep is designed to run in a
containerized environment like Kubernetes, these configurations can be provided via the filesystem or
http. By providing the configuration via http, it is possible to control the configuration change via
a flexible http api. This enables Logprep to quickly adapt to changes in your environment.First, a general configuration is given that describes the pipeline and the connectors,
and lastly, the processors need rules in order to process messages correctly.The following yaml configuration shows an example configuration for the pipeline shown
in the graph above:process_count:3timeout:0.1pipeline:-dissector:type:dissectorspecific_rules:-https://your-api/dissector/generic_rules:-rules/01_dissector/generic/-geoip_enricher:type:geoip_enricherspecific_rules:-https://your-api/geoip/generic_rules:-rules/02_geoip_enricher/generic/tree_config:artifacts/tree_config.jsondb_path:artifacts/GeoDB.mmdb-dropper:type:dropperspecific_rules:-rules/03_dropper/specific/generic_rules:-rules/03_dropper/generic/input:mykafka:type:confluentkafka_inputbootstrapservers:[127.0.0.1:9092]topic:consumergroup:cgroupauto_commit:truesession_timeout:6000offset_reset_policy:smallestoutput:opensearch:type:opensearch_outputhosts:-127.0.0.1:9200default_index:default_indexerror_index:error_indexmessage_backlog_size:10000timeout:10000max_retries:user:the usernamesecret:the passordcert:/path/to/cert.crtThe following yaml represents a dropper rule which according to the previous configuration
should be in therules/03_dropper/generic/directory.filter:"message"drop:-messagedescription:"Dropsthemessagefield"The condition of this rule would check if the fieldmessageexists in the log.
If it does exist then the dropper would delete this field from the log message.Details about the rule language and how to write rules for the processors can be found in therule configuration documentation.Getting StartedInstallationPython should be present on the system, currently supported are the versions 3.10 - 3.12.To install Logprep you have following options:1. Option:Installation via PyPI:This option is recommended if you just want to use the latest release of logprep.pip install logprep2. Option:Installation via Git Repository:This option is recommended if you are interested in the latest developments and might want to
contribute to them.git clone https://github.com/fkie-cad/Logprep.git
cd Logprep
pip install -r requirements.txt3. Option:Installation via Github ReleaseThis option is recommended if you just want to try out the latest developments.pip install git+https://github.com/fkie-cad/Logprep.git@latest4. Option:Docker build from current commitThis option can be used to build a container image from a specific commitdocker build -t logprep .
docker run logprep --helpTestingWe usepytestas our testing framework. We have unittests located in./tests/unitand
acceptance tests located in./tests/acceptance.Run all tests with:pytest ./tests --cov=logprep --cov-report=xml -vvvRunning LogprepDepending on how you have installed Logprep you have different choices to run Logprep as well.
If you have installed it via PyPI or the Github Development release just run:logprep run $CONFIGIf you have installed Logprep via cloning the repository then you should run it via:PYTHONPATH="." python3 logprep/run_logprep.py run $CONFIGWhere$CONFIGis the path or uri to a configuration file (see the documentation about theconfiguration).
The next sections all assume an installation via pipVerifying ConfigurationThe following command can be executed to verify the configuration file without having to run Logprep:logprep test config $CONFIGWhere$CONFIGis the path or uri to a configuration file (see the documentation about theconfiguration).Reload the ConfigurationTo change the configuration of Logprep it is not needed to restart Logprep entirely.
Instead, it can be issued to reload the configuration.
For this, the signalSIGUSR1must be send to the Logprep process.Additionally, aconfig_refresh_intervalcan be set to periodically and automatically refresh the given configuration.
This can be useful in case of containerized environments (such as Kubernetes), when pod volumes often change
on the fly.If the configuration does not pass a consistency check, then an error message is logged and
Logprep keeps running with the previous configuration.
The configuration should be then checked and corrected on the basis of the error message.Logprep Quickstart EnvironmentTo demonstrate the functionality of logprep this repo comes with a completekafka,lokgprepandopensearchstack.
To get it runningdockeranddocker-compose(version >= 1.28) must be first installed.
The docker-compose file is located in the directoryquickstart.
A prerequisite is to runsysctl -w vm.max_map_count=262144, otherwise Opensearch might not
properly start.The environment can either be started with a Logprep container or without one:Run without Logprep Container (default)Run from within thequickstartdirectory:docker-composeup-dIt starts and connectsKafka,logprep,OpensearchandOpensearch Dashboards.Run Logprep against loaded environment from mainLogprepdirectory:logpreprunquickstart/exampledata/config/pipeline.ymlRun with Logprep ContainerRun from within thequickstartdirectory:docker-compose--profilelogprepup-dRun with getting config from http server with basic authenticationRun from within thequickstartdirectory:docker-compose--profilebasic_authup-dRun within the project root directory:exportLOGPREP_CONFIG_AUTH_USERNAME="user"exportLOGPREP_CONFIG_AUTH_PASSWORD="password"logprephttp://localhost:8081/config/pipeline.ymlRun with getting config from FDA with oauth2 authenticationStart logprep by using the oauth2 profile with docker-compose:```bash
docker-compose --profile oauth2 up -d
```In order to run logprep with the FDA configuration it is necessary to set the following environment
variables:LOGPREP_CONFIG_AUTH_METHOD=oauthLOGPREP_CONFIG_AUTH_TOKEN=<TOKEN>(can be intercepted from the FDA<->Keycloak http requests)LOGPREP_INPUT=<INPUT_CONNECTOR_CONFIG>LOGPREP_OUTPUT=<OUTPUT_CONNECTOR_CONFIG>LOGPREP_OPTIONS=<GENERAL_LOGPREP_CONFIG>Once they are set logprep can be started with:logpreprun"http://localhost:8000/api/v1/pipelines?stage=prod&logclass=ExampleClass"Interacting with the Quickstart EnvironmentThe start up takes a few seconds to complete, but once everything is up
and running it is possible to write JSON events into Kafka and read the processed events in
Opensearch Dashboards. Following services are available after start up:ServiceLocationUserPasswordKafka:localhost:9092//Kafka Exporter:localhost:9308//Logprep metrics:localhost:8001//Opensearch:localhost:9200//Opensearch Dashboards:localhost:5601//Grafana Dashboards:localhost:3000adminadminPrometheus:localhost:9090//Nginx:localhost:8081userpasswordKeycloak:localhost:8080adminadminKeycloak Postgres:localhost:5432keycloakbitnamiFDA:localhost:8002logpreplogprepFDA Postgres:localhost:25432fdafdaThe example rules that are used in the docker instance of Logprep can be found
inquickstart/exampledata/rules.
Example events that trigger for the example rules can be found inquickstart/exampledata/input_logdata/logclass/test_input.jsonl.
These events can be added to Kafka with the following command:(dockerexec-ikafkakafka-console-producer.sh--bootstrap-server127.0.0.1:9092--topicconsumer)<exampledata/input_logdata/logclass/test_input.jsonlOnce the events have been processed for the first time, the new indicesprocessed,sreandpseudonymsshould be available in Opensearch Dashboards.The environment can be stopped viadocker-compose down.Event GenerationLogprep has the additional functionality of generating events and sending them to two
different targets.
It can send events to kafka, while also loading events from kafka or reading them from file,
and it can send events to a http endpoint as POST requests.Following sections describe the usage of these event generators.KafkaThe kafka load-tester can send a configurable amount of documents to Kafka.
The documents that are being send can be obtained either from Kafka or from a file with JSON lines.It can be configured how many documents should be retrieved from Kafka (if Kafka is used as source)
and how many documents will be sent.
Documents obtained from Kafka won't be written down to disk.The documents will be sent repeatedly until the desired amount has been sent.
Thetagsfield and the_indexfield of each document will be set toload-tester.
Furthermore, a fieldload-tester-uniquewith a unique value will be added to each document every
time a document is sent.
This is done to prevent that repeatedly sent documents are identical.To find out more about the usage of the kafka load-tester execute:logprepgeneratekafka--helpConfigurationThe kafka load-tester is configured via a YAML file.
It must have the following format:logging_level:LOG_LEVEL# Default: "INFO"source_count:INTERGER# Number of documents to obtain form Kafkacount:INTERGER# Number of documents to sendprocess_count:INTERGER# Number of processes (default: 1)profile:BOOL# Shows profiling data (default: false)target_send_per_sec:INTERGER# Desired number of documents to send per second with each process. Setting it to 0 sends as much as possible (default: 0).kafka:bootstrap_servers:# List of bootstrap servers-URL:PORT# i.e. "127.0.0.1:9092"consumer:# Kafka consumertopic:STRING# Topic to obtain documents fromgroup_id:STRING# Should be different from the group_id of the Logprep Consumer, otherwise the offset in Logprep will be changed!timeout:FLOAT# Timeout for retrieving documents (default: 1.0)producer:# Kafka produceracks:STRING/INTERGER# Determines if sending should be acknowledged (default: 0)compression_type:STRING# Compression type (default: "none")topic:STRING# Topic to send documents toqueue_buffering_max_messages:INTEGER# Batch for sending documents (default: 10000)linger_ms:INTEGER# Time to wait before a batch is sent if the max wasn't reached before (default: 5000)flush_timeout:FLOAT# Timeout to flush the producer (default 30.0)ssl:# SSL authentication (Optional)ca_location:STRINGcertificate_location:STRINGkey:location:STRINGpassword:STRING# OptionalUnused parameters must be removed or commented.HttpThe http endpoint allows for generating events based on templated sample files which are stored
inside a dataset directory.The dataset directory with the sample files has to have the following format:| - Test-Logs-Directory
 | | - Test-Logs-Class-1-Directory
 | | | - config.yaml
 | | | - Test-Logs-1.jsonl
 | | | - Test-Logs-2.jsonl
 | | - Test-Logs-Class-2-Directory
 | | | - config.yaml
 | | | - Test-Logs-A.jsonl
 | | | - Test-Logs-B.jsonlWhile the jsonl event files can have arbitrary names, theconfig.yamlneeds to be called exactly
that. It also needs to follow the following schema:target_path:/endpoint/logsource/pathtimestamps:-key:TIMESTAMP_FIELD_1format:"%Y%m%d"-key:TIMESTAMP_FIELD_1format:"%H%M%S"time_shift:"+0200"# Optional, sets time shift in hours and minutes, if needed ([+-]HHMM)To find out more about the usage of the http event generator execute:logprepgeneratehttp--helpDocumentationThe documentation for Logprep is online athttps://logprep.readthedocs.io/en/latest/or it can
be built locally via:sudo apt install pandoc
cd ./doc/
pip install -r ./requirements.txt
make htmlA HTML documentation can be then found indoc/_build/html/index.html.ContributingEvery contribution is highly appreciated.
If you have ideas or improvements feel free to create a fork and open a pull requests.
Issues and engagement in open discussions are also welcome.LicenseLogprep is distributed under the LGPL-2.1 License. See LICENSE file for more information.