Package: pysigma-backend-loki
pySigma Loki BackendThis is the Loki backend for pySigma. It provides the packagesigma.backends.lokiwith theLogQLBackendclass.It supports the following output formats:default: plain Loki LogQL queriesruler: creates Loki LogQL queries in the ruler (YAML) format for generating alertsIt includes the following pipeline transformations insigma.pipelines.loki:SetCustomAttributeTransformation: adds a specified custom attribute to a rule, which can be used to introduce astream selectororparser expressioninto the generated queryTheLokiCustomAttributesenum contains the relevant custom attribute names used by the backendFurther, it contains the processing pipelines insigma.pipelines.loki:loki_log_parser: converts field names to logfmt labels used by Grafanaloki_promtail_sysmon: parse and adjust field names for Windows sysmon data produced by promtailNote: most rules lack thesysmonservice tag, and hence this pipeline should be used in combination with thegeneric sysmon pipelineloki_okta_system_log: parse the Okta System Log event json, adjusting field-names appropriatelyWhen converting rules into queries, the backend has the following optional arguments:add_line_filters(boolean, default:False): ifTrue, attempts to infer and add new line filters to queries without line filters, toimprove Loki query performancecase_insensitive(boolean, default:True): ifFalse, defaults to generating case-sensitive query filters, instead of case-insensitive filters thatthe Sigma specification expects, trading between Loki query performance and potentially missing data with unexpected casingNote: if the generated query will be executed on Loki v2.8.2 or older, this argumentshouldbe set toFalse, as these versions of Loki may contain issues with case-insensitive filters, which cause such queries to fail to match desired dataThis backend is currently maintained by:Nick MooreMostafa MoradianInstallationTo get started developing/testing pySigma-backend-loki, these steps may help you get started:Install poetryClone this repository and open a terminal/shell in the top-level directoryRunpoetry installto install the Python dependenciesRunpoetry shellto activate the poetry environmentCheck it all works by runningpoetry run pytest(Optional) If you wish to validate the generated rules using sigma_backend_tester.py, installLogCLI(Optional, but recommended) To enable the Git hooks, run the following command from the root directory of the repository:gitconfig--localcore.hooksPath.githooks/ReleasingTo release new versions of pySigma-backend-loki, we use GitHub actions to update PyPI. When the main branch is in state that is ready to release, the process is as follows:Determine the correct version number using theSemantic Versioningmethodology. All version numbers should be in the format\d+\.\d+\.\d+(-[0-9A-Za-z-]+)?Updatepyproject.tomlwith the new version numberCommit and push the change to GitHub, and validate that the GitHub actions tests passCreate a signed tag for the release, named the version number prefixed with a v, e.g.,git tag --sign --message="Release vX.X.X" vX.X.XPush the tag to GitHub, e.g.,git push --tags, and validate that the release to the test instance of PyPI is successfulRunpoetry buildto produce distributable versions indist/Create a release in GitHub against the appropriate tag. If the version number starts withv0, or ends with-alpha/betaetc., mark it as a pre-release, and attach the distributable files to the releaseValidate that the release to PyPI GitHub action is successfulIf this release supports a new minor or major version ofpySigma, do a pull request on thepySigma-plugin-directoryto reflect thatWork in progressThese features are currently either WIP or are planned to be implemented in the near future.Various processing pipelines for other applications and log sourcesGenerating more accurate log stream selectors based on logsourceTranslate field names in Sigma signatures into relevant labels for Loki using pipelinesWon't implement (probably)These features are not easily supported by the backend, and hence are unlikely to be implemented.More complex keyword/line filter searches than ANDs of ORs