Elastic SIEM Detection Rules: custom rules management

# Custom Rules

A custom rule is any rule that is not maintained by Elastic under `rules/` or `rules_building_block`. These docs are intended
to show how to manage custom rules using this repository.

For more detailed breakdown and explanation of employing a detections-as-code approach, refer to the
[dac-reference](https://dac-reference.readthedocs.io/en/latest/index.html).


## Defining a custom rule config and directory structure

The simplest way to maintain custom rules alongside the existing prebuilt rules in the repo, is to decouple where the rules
are stored to minimize VCS conflicts and overlap. This is accomplished by defining a custom rules directory using a config file.

### Understanding the structure

```
custom-rules
├── _config.yaml
└── rules
    ├── example_rule_1.toml
    ├── example_rule_2.toml
└── etc
    ├── deprecated_rules.json
    ├── packages.yaml
    ├── stack-schema-map.yaml
    ├── test_config.yaml
    └── version.lock.json
└── actions
    ├── action_1.toml
    ├── action_2.toml
└── action_connectors
    ├── action_connector_1.toml
    └── action_connectors_2.toml    
└── exceptions
    ├── exception_1.toml
    └── exception_2.toml
```

This structure represents a portable set of custom rules. This is just an example, and the exact locations of the files
should be defined in the `_config.yaml` file. Refer to the details in the default
[_config.yaml](../detection_rules/etc/_config.yaml) for more information.

* deprecated_rules.json - tracks all deprecated rules (optional)
* packages.yaml - information for building packages (mostly optional, but the current version is required)
* stack-schema-map.yaml - a mapping of schemas for query validation
* test_config.yaml - a config file for testing (optional)
* version.lock.json - this tracks versioning for rules (optional depending on versioning strategy)

To initialize a custom rule directory, run `python -m detection_rules custom-rules setup-config <directory>`

### Defining a config

```yaml
rule_dirs:
  - rules
files:
  deprecated_rules: deprecated_rules.json
  packages: packages.yaml
  stack_schema_map: stack-schema-map.yaml
  version_lock: version.lock.json
directories:
  action_dir: actions
  action_connector_dir: action_connectors
  exception_dir: exceptions
```

Some notes:

* The paths in this file are relative to the custom rules directory (CUSTOM_RULES_DIR/)
* Refer to each original [source file](../detection_rules/etc/example_test_config.yaml) for purpose and proper formatting
* You can also add an optional `bbr_rules_dirs` section for custom BBR rules.
* To bypass using the version lock versioning strategy (version lock file) you can set the optional `bypass_version_lock` value to be `True`
* To normalize the capitalization KQL keywords in KQL rule queries one can use the optional `normalize_kql_keywords` value set to `True` or `False` as desired.
* To manage exceptions tied to rules one can set an exceptions directory using the optional `exception_dir` value (included above) set to be the desired path. If an exceptions directory is explicitly specified in a CLI command, the config value will be ignored.
* To manage action-connectors tied to rules one can set an action-connectors directory using the optional `action_connector_dir` value (included above) set to be the desired path. If an actions_connector directory is explicitly specified in a CLI command, the config value will be ignored.
* To turn on automatic schema generation for non-ecs fields via custom schemas add `auto_gen_schema_file: <path_to_your_json_file>`. This will generate a schema file in the specified location that will be used to add entries for each field and index combination that is not already in a known schema. This will also automatically add it to your stack-schema-map.yaml file when using a custom rules directory and config.
* For Kibana action items, currently these are included in the rule toml files themselves. At a later date, we may allow for bulk editing of rule action items through separate action toml files. The action_dir config key is left available for this later implementation. For now to bulk update, use the bulk actions add rule actions UI in Kibana.
* To disable optional Elastic validation in bulk, set `bypass_optional_elastic_validation: true` in `_config.yaml`. That sets every `DR_BYPASS_*` environment variable that `set_all_validation_bypass()` controls (note parsing, BBR lookback, tags unit tests, timeline template, ES|QL `keep`, ES|QL `FROM` metadata).
* To enable only some of those bypasses, set the matching top-level booleans in `_config.yaml` (omit `bypass_optional_elastic_validation` or set it to `false`): `bypass_note_validation_and_parse`, `bypass_bbr_lookback_validation`, `bypass_tags_validation`, `bypass_timeline_template_validation`, `bypass_esql_keep_validation`, `bypass_esql_metadata_validation`. Each `true` sets the corresponding `DR_BYPASS_*` variable when the config is loaded. If `bypass_optional_elastic_validation` is `true`, those individual flags are all treated as enabled (the bulk flag wins).


When using the repo, set the environment variable `CUSTOM_RULES_DIR=<directory-with-_config.yaml>`


### Defining a testing config

```yaml
testing:
  config: etc/example_test_config.yaml
```

This points to the testing config file (see example under detection_rules/etc/example_test_config.yaml) and can either
be set in `_config.yaml` or as the environment variable `DETECTION_RULES_TEST_CONFIG`, with precedence going to the
environment variable if both are set. Having both these options allows for configuring testing on prebuilt Elastic rules
without specifying a rules _config.yaml.

Some notes:

* If set in this file, the path should be relative to the location of this config. If passed as an environment variable, it should be the full path
* When using the `--no-tactic-filename` flag for kibana imports and exports, be sure to disable the unit test by using the following line `- tests.test_all_rules.TestRuleFiles.test_rule_file_name_tactic` in your test config file.


### How the config is used and it's designed portability

This repo is designed to operate on certain expectations of structure and config files. By defining the code below, it allows
the design to become portable and based on defined information, rather than the static excpectiations.

```python
RULES_CONFIG = parse_rules_config()

# which then makes the following attribute available for use

@dataclass
class RulesConfig:
    """Detection rules config file."""
    deprecated_rules_file: Path
    deprecated_rules: Dict[str, dict]
    packages_file: Path
    packages: Dict[str, dict]
    rule_dirs: List[Path]
    stack_schema_map_file: Path
    stack_schema_map: Dict[str, dict]
    test_config: TestConfig
    version_lock_file: Path
    version_lock: Dict[str, dict]

    action_dir: Optional[Path] = None
    action_connector_dir: Optional[Path] = None
    auto_gen_schema_file: Optional[Path] = None
    bbr_rules_dirs