Elastic SIEM Detection Rules: README

[![Supported Python versions](https://img.shields.io/badge/python-3.12+-yellow.svg)](https://www.python.org/downloads/)
[![Unit Tests](https://github.com/elastic/detection-rules/workflows/Unit%20Tests/badge.svg)](https://github.com/elastic/detection-rules/actions)
[![Chat](https://img.shields.io/badge/chat-%23security--detection--rules-blueviolet)](https://ela.st/slack)
[![ATT&CK navigator coverage](https://img.shields.io/badge/ATT&CK-Navigator-red.svg)](https://ela.st/detection-rules-navigator-trade)

# Detection Rules

Detection Rules is the home for rules used by Elastic Security. This repository is used for the development, maintenance, testing, validation, and release of rules for Elastic Security’s Detection Engine.

This repository was first announced on Elastic's blog post, [Elastic Security opens public detection rules repo](https://elastic.co/blog/elastic-security-opens-public-detection-rules-repo). For additional content, see the accompanying webinar, [Elastic Security: Introducing the public repository for detection rules](https://www.elastic.co/webinars/introducing-the-public-repository-for-detection-rules).


## Table of Contents
- [Detection Rules](#detection-rules)
  - [Table of Contents](#table-of-contents)
  - [Overview of this repository](#overview-of-this-repository)
  - [Getting started](#getting-started)
  - [How to contribute](#how-to-contribute)
  - [Detections as Code (DaC)](#detections-as-code-dac)
  - [RTAs](#rtas)
  - [Licensing](#licensing)
  - [Questions? Problems? Suggestions?](#questions-problems-suggestions)


## Overview of this repository

Detection Rules contains more than just static rule files. This repository also contains code for building Detections-as-code pipelines, unit testing in Python and integrating with the Detection Engine in Kibana.

| folder                                          |  description                                                                        |
|------------------------------------------------ |------------------------------------------------------------------------------------ |
| [`detection_rules/`](detection_rules)           | Python module for rule parsing, validating and packaging                            |
| [`etc/`](detection_rules/etc)                   | Miscellaneous files, such as ECS and Beats schemas and configuration files          |
| [`hunting/`](./hunting/)                        | Root directory where threat hunting package and queries are stored                  |
| [`kibana/`](lib/kibana)                         | Python library for handling the API calls to Kibana and the Detection Engine        |
| [`kql/`](lib/kql)                               | Python library for parsing and validating Kibana Query Language                     |
| [`rules/`](rules)                               | Root directory where rules are stored                                               |
| [`rules_building_block/`](rules_building_block) | Root directory where building block rules are stored                                |
| [`tests/`](tests)                               | Python code for unit testing rules                                                  |


## Getting started

Although rules can be added by manually creating `.toml` files, we don't recommend it. This repository also consists of a python module that aids rule creation and unit testing. Assuming you have Python 3.12+, run the below command to install the dependencies using the makefile:

```console
✗ make
python3.12 -m pip install --upgrade pip setuptools
Looking in indexes: https://pypi.org/simple
Requirement already satisfied: pip in /opt/homebrew/lib/python3.12/site-packages (24.0)
Requirement already satisfied: setuptools in /opt/homebrew/lib/python3.12/site-packages (69.1.1)
python3.12 -m venv ./env/detection-rules-build
./env/detection-rules-build/bin/pip install --upgrade pip setuptools
Looking in indexes: https://pypi.org/simple
Requirement already satisfied: pip in ./env/detection-rules-build/lib/python3.12/site-packages (24.0)
Collecting setuptools
  Using cached setuptools-69.1.1-py3-none-any.whl.metadata (6.2 kB)
Using cached setuptools-69.1.1-py3-none-any.whl (819 kB)
Installing collected packages: setuptools
Successfully installed setuptools-69.1.1
Installing kql and kibana packages...
...
```


Or install the dependencies using the following command:
```console
$ pip3 install ".[dev]"
Collecting jsl==0.2.4
  Downloading jsl-0.2.4.tar.gz (21 kB)
Collecting jsonschema==3.2.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 318 kB/s
Collecting requests==2.22.0
  Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 1.2 MB/s
Collecting Click==7.0
  Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
     |████████████████████████████████| 81 kB 2.6 MB/s
...
```

Note: The `kibana` and `kql` packages are not available on PyPI and must be installed from the `lib` directory. The `hunting` package has optional dependencies to be installed with `pip3 install ".[hunting]`.

```console

# Install from the repository
pip3 install git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kibana
pip3 install git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql

# Or locally for development
pip3 install lib/kibana lib/kql
```

Remember, make sure to activate your virtual environment if you are using one. If installed via `make`, the associated virtual environment is created in `env/detection-rules-build/`.
If you are having trouble using a Python 3.12 environment, please see the relevant section in our [troubleshooting guide](./Troubleshooting.md).

To confirm that everything was properly installed, run with the `--help` flag
```console
$  python -m detection_rules --help

Usage: detection_rules [OPTIONS] COMMAND [ARGS]...

  Commands for detection-rules repository.

Options:
  -D, --debug / -N, --no-debug  Print full exception stacktrace on errors
  -h, --help                    Show this message and exit.

Commands:
  build-limited-rules     Import rules from json, toml, or Kibana exported rule file(s), filter out unsupported ones, and write to output NDJSON file.
  build-threat-map-entry  Build a threat map entry.
  create-rule             Create a detection rule.
  custom-rules            Commands for supporting custom rules.
  dev                     Commands related to the Elastic Stack rules release lifecycle.
  es                      Commands for integrating with Elasticsearch.
  export-rules-from-repo  Export rule(s) and exception(s) into an importable ndjson file.
  generate-rules-index    Generate enriched indexes of rules, based on a KQL search, for indexing/importing into elasticsearch/kibana.
  import-rules-to-repo    Import rules from json, toml, or yaml files containing Kibana exported rule(s).
  kibana                  Commands for integrating with Kibana.
  mass-update