Centralized Logging: Parse Key-Value Logs With FluentD
By: Al-Waleed Shihadeh
Tags: Ruby on Rails, Fluentd, Rails, Ruby, Programming

Implementation

After some investigation, I found that the best way to implement this solution is to build a FluentD Filter plugin with the below configuration items to support the needed functionality.

key : The source key that contains the message to be parsed. The default value is the log .

: The source key that contains the message to be parsed. The default value is the . remove_key : A boolean indicator to remove the source key after extracting the key-value pairs. The default value is false .

: A boolean indicator to remove the source key after extracting the key-value pairs. The default value is . filter_out_lines_without_kesy : A boolean value to enable removing the logline in case the logline does not contain any defined key-value pairs. Default value is false .

: A boolean value to enable removing the logline in case the logline does not contain any defined key-value pairs. Default value is . use_regex : Enable/Disable using regex for extracting key-value pairs. The default value is false .

: Enable/Disable using regex for extracting key-value pairs. The default value is . remove_prefix : A regex that will be used to remove a prefix of the log message. By default the value of this configuration item is empty.

: A regex that will be used to remove a prefix of the log message. By default the value of this configuration item is empty. keys_delimiter : The character used to separate keys from each other. The default value is space.

: The character used to separate keys from each other. The default value is space. kv_delimter_char : The character used to separate keys from their values. The default value is = .

: The character used to separate keys from their values. The default value is . filtered_keys : The whitelisted keys that will be included in the FluentD event.

: The whitelisted keys that will be included in the FluentD event. filtered_keys_regex : A regex to define whitelisted keys.

Integrating the FluentD plugin with the FluentD configuration is pretty straightforward and does not require a lot of effort. The below steps summarize the needed actions for successfully integrating the FluentD filter with the paste scripts.

Install the gem and make it available for your FluentD scripts. If you are using FluentD in a docker container, make sure the gem is included in the docker image. The below example shows how to build a FluentD docker image with the fluent-plugin-filter-kv-parser .

Extend the FluentD configurations to start parsing and filtering the log messages. The minimal configurations that can be added are shown in the snippet below.

With the above Filter definition, the gem will use the default values for each of the defined configurations. You can find the default values here. This means that the log key will be used as the source key for extracting the key-value pairs and it will not be removed from the Fluentd. Keys will be extracted using the default delimiter and not using the defined regex, and all extracted keys will be added to the fluent event.

Extending the configurations of the FluentD filter to change the behavior of the extraction of the keys can be done simply by adding the needed configurations items withing the filter definition. For instance, the below definition will filter the log message and approve only the keys that match a given regex.

As it is shown in the above snippet, only the keys that match the defined regex was extracted and copied to the output data. On the other hand, if you would like to limit the extracted keys to predefined keys, you can use the below configurations.

The key-value FluentD filter also supports defining and mixing configurations. For instance, it is possible to whitelist keys with a regex and exact names at the same time. In that case, the Filter will include all keys that match the regex or the exact keys provided.