Elastic SIEM Detection Rules: CLI

# Command Line Interface (CLI)

This covers more advanced CLI use cases and workflows. To [get started](README.md#getting-started) with the CLI, reference
the [README](README.md). Basic use of the CLI such as [creating a rule](CONTRIBUTING.md#creating-a-rule-with-the-cli) or
[testing](CONTRIBUTING.md#testing-a-rule-with-the-cli) are referenced in the [contribution guide](CONTRIBUTING.md).


## Using a user config file or environment variables

CLI commands which are tied to Kibana and Elasticsearch are capable of parsing auth-related keyword args from a config
file or environment variables.

If a value is set in multiple places, such as config file and environment variable, the order of precedence will be as
follows:
* explicitly passed args (such as `--user joe`)
* environment variables
* config values
* prompt (this only applies to certain values)

#### Setup a user config file

In the root directory of this repo, create the file `.detection-rules-cfg.json` (or `.yaml`) and add relevant values

Currently supported arguments:
* elasticsearch_url
* kibana_url
* cloud_id
* es_user 
* es_password
* api_key

Authenticating to Kibana is only available using api_key.

#### Using environment variables

Environment variables using the argument format: `DR_<UPPERCASED_ARG_NAME>` will be parsed in commands which expect it.
EX: `DR_USER=joe`


Using the environment variable `DR_BYPASS_NOTE_VALIDATION_AND_PARSE` will bypass the Detection Rules validation on the `note` field in toml files.

Using the environment variable `DR_BYPASS_BBR_LOOKBACK_VALIDATION` will bypass the Detection Rules lookback and interval validation
on the building block rules.

Using the environment variable `DR_BYPASS_TAGS_VALIDATION` will bypass the Detection Rules Unit Tests on the `tags` field in toml files.

Using the environment variable `DR_BYPASS_TIMELINE_TEMPLATE_VALIDATION` will bypass the timeline template id and title validation for rules. 

Using the environment variable `DR_BYPASS_ESQL_KEEP_VALIDATION` will bypass local validation that ES|QL rules include a `keep` command and that non-aggregate queries list `_id`, `_version`, and `_index` in `keep` (other ES|QL checks are unchanged).

Using the environment variable `DR_BYPASS_ESQL_METADATA_VALIDATION` will bypass local validation that non-aggregate ES|QL queries use `FROM ... METADATA _id, _version, _index` or an aggregate `STATS ... BY` pattern (other ES|QL checks are unchanged).

In `_config.yaml`, `bypass_optional_elastic_validation: true` enables all of the above at load time. Alternatively, set any of the top-level booleans `bypass_note_validation_and_parse`, `bypass_bbr_lookback_validation`, `bypass_tags_validation`, `bypass_timeline_template_validation`, `bypass_esql_keep_validation`, or `bypass_esql_metadata_validation` to `true` (see comments in `detection_rules/etc/_config.yaml`).

Using the environment variable `DR_CLI_MAX_WIDTH` will set a custom max width for the click CLI. 
For instance, some users may want to increase the default value in cases where help messages are cut off. 

Using the environment variable `DR_REMOTE_ESQL_VALIDATION` will enable remote ESQL validation for rules that use ESQL queries. This validation will be performed whenever the rule is loaded including for example the view-rule command. This requires the appropriate kibana_url or cloud_id, api_key, and es_url to be set in the config file or as environment variables.

Using the environment variable `DR_SKIP_EMPTY_INDEX_CLEANUP` will disable the cleanup of remote testing indexes that are created as part of the remote ESQL validation. By default, these indexes are deleted after the validation is complete, or upon validation error.

## Importing rules into the repo

You can import rules into the repo using the `create-rule` or `import-rules-to-repo` commands. Both of these commands will
require that the rules are schema-compliant and able to pass full validation. The biggest benefit to using these
commands is that they will strip[*](#note) additional fields[**](#note-2) and prompt for missing required
fields.

Alternatively, you can manually place rule files in the directory and run tests to validate as well.

<a id="note">\* Note</a>: This is currently limited to flat fields and may not apply to nested values.<br>
<a id="note-2">\** Note</a>: Additional fields are based on the current schema at the time the command is used.


#### `create-rule`

```console
Usage: detection_rules create-rule [OPTIONS] PATH

  Create a detection rule.

Options:
  -c, --config FILE               Rule or config file
  --required-only                 Only prompt for required fields
  -t, --rule-type [machine_learning|query|threshold]
                                  Type of rule to create
  -h, --help                      Show this message and exit.
```

This command will allow you to pass a rule file using the `-c/--config` parameter. This is limited to one rule at a time
and will accept any valid rule in the following formats:
* toml
* json
* yaml (yup)
* ndjson (as long as it contains only a single rule and has the extension `.ndjson` or `.jsonl`)

#### `import-rules-to-repo`

```console
Usage: detection_rules import-rules-to-repo [OPTIONS] [INPUT_FILE]...

  Import rules from json, toml, or yaml files containing Kibana exported rule(s).

Options:
  -ac, --action-connector-import  Include action connectors in export
  -e, --exceptions-import         Include exceptions in export
  --required-only                 Only prompt for required fields
  -d, --directory DIRECTORY       Load files from a directory
  -s, --save-directory DIRECTORY  Save imported rules to a directory
  -se, --exceptions-directory DIRECTORY
                                  Save imported exceptions to a directory
  -sa, --action-connectors-directory DIRECTORY
                                  Save imported actions to a directory
  -ske, --skip-errors             Skip rule import errors
  -da, --default-author TEXT      Default author for rules missing one
  -snv, --strip-none-values       Strip None values from the rule
  -lc, --local-creation-date      Preserve the local creation date of the rule
  -lu, --local-updated-date       Preserve the local updated date of the rule
  -lr, --load-rule-loading        Enable arbitrary rule loading from the rules directories (Can be very slow!)
  -h, --help                      Show this message and exit.
```

The primary advantage of using this command is the ability to import multiple rules at once. Multiple rule paths can be
specified explicitly with unlimited arguments, recursively within a directory using `-d/--directory`[*](#note-3), or
a combination of both.

In addition to the formats mentioned using `create-rule`, this will also accept an `.ndjson`/`jsonl` file
containing multiple rules (as would be the case with a bulk export).

The `-s/--save-directory` is an optional parameter to specify a non default directory to place imported rules. If it is not specified,