Repository: elastic/ecs

Issue:
Error when using usage-example with generator.py
<!--
GitHub is reserved for bug reports and feature requests; it is not the place
for general questions. If you have a general question, please visit the
discuss forums: https://discuss.elastic.co/tag/ecs-elastic-common-schema.

Please fill in the following details to help us reproduce the bug:
-->

**Description of the problem including expected versus actual behavior**:

Trying `script/generator.py` for the first time, I stumble upon a stack trace throwing `KeyError: 'error'`. I initially though I maybe passed wrong parameters but it looks like example documented in `USAGE.md` also have the same issue.

Thanks for writing this tool, I'm eager to use it since it can greatly help me to generate/maintain configuration for custom Beats indices. Let me know if I can somehow help you further.

**Steps to reproduce**:

1. `git clone https://github.com/elastic/ecs && cd ecs`
2. Running `generator.py` as documented in `USAGE.md`:
```sh
$ python3 scripts/generator.py --ref v1.6.0 \
  --subset            usage-example/fields/subset.yml \
  --include           usage-example/fields/custom/ \
  --out               usage-example/ \
  --template-settings usage-example/fields/template-settings.json \
  --mapping-settings  usage-example/fields/mapping-settings.json
```

**Provide logs (if relevant)**:

```
Loading schemas from git ref v1.6.0
Running generator. ECS version 1.6.0
Loading user defined schemas: ['usage-example/fields/custom/']
/Users/florent/Sources/ecs/scripts/schema/cleaner.py:185: UserWarning: Example value for field `header_flags` contains an object or array which must be quoted to avoid YAML interpretation.

This will cause an exception when running in strict mode.
Warning check:
  check_example_value(field, strict=strict_mode)
/Users/florent/Sources/ecs/scripts/schema/cleaner.py:185: UserWarning: Example value for field `resolved_ip` contains an object or array which must be quoted to avoid YAML interpretation.
[...]
Traceback (most recent call last):
  File "/Users/florent/Sources/ecs/scripts/generator.py", line 106, in <module>
    main()
  File "/Users/florent/Sources/ecs/scripts/generator.py", line 60, in main
    es_template.generate_legacy(flat, ecs_version, out_dir, args.template_settings, args.mapping_settings)
  File "/Users/florent/Sources/ecs/scripts/generators/es_template.py", line 109, in generate_legacy
    generate_legacy_template_version(6, ecs_version, mappings_section, out_dir, template_settings_file)
  File "/Users/florent/Sources/ecs/scripts/generators/es_template.py", line 115, in generate_legacy_template_version
    template = template_settings(es_version, ecs_version, mappings_section, template_settings_file)
  File "/Users/florent/Sources/ecs/scripts/generators/es_template.py", line 202, in template_settings
    error_stack_trace_mappings = mappings_section['properties']['error']['properties']['stack_trace']
KeyError: 'error'
```

**Any additional context:**

```
$ git rev-parse HEAD
83ced31c9da6b3e5599b01e319a827a7941e6439

$ python3 --version
Python 3.9.0

$ pip3 list | grep -i -E '(yaml|pep|mock|git|jinja)'
autopep8      1.4.4
gitdb         4.0.5
GitPython     3.1.2
Jinja2        2.11.2
mock          4.0.2
PyYAML        5.3b1
yamllint      1.19.0
```


Discussion:
What a coincidence, I just ran into the same error! However, I only used a `--subset` and get to the same error:

```bash
$ python scripts/generator.py --out ~/tmp --subset subset.yaml --oss
Loading schemas from local files
Running generator. ECS version 2.0.0-dev
Traceback (most recent call last):
  File "scripts/generator.py", line 106, in <module>
    main()
  File "scripts/generator.py", line 60, in main
    es_template.generate_legacy(flat, ecs_version, out_dir, args.template_settings, args.mapping_settings)
  File "/var/home/lwille/src/ext/ecs/scripts/generators/es_template.py", line 109, in generate_legacy
    generate_legacy_template_version(6, ecs_version, mappings_section, out_dir, template_settings_file)
  File "/var/home/lwille/src/ext/ecs/scripts/generators/es_template.py", line 115, in generate_legacy_template_version
    template = template_settings(es_version, ecs_version, mappings_section, template_settings_file)
  File "/var/home/lwille/src/ext/ecs/scripts/generators/es_template.py", line 202, in template_settings
    error_stack_trace_mappings = mappings_section['properties']['error']['properties']['stack_trace']
KeyError: 'error'
```

*subset.yaml*:
```yaml
---
name: json_filebeat
fields:
  base:
    fields:
      "@timestamp": {}
```
```bash
$ git rev-parse HEAD
83ced31c9da6b3e5599b01e319a827a7941e6439
$ python3 --version
Python 3.8.1
$  pip3 list | grep -i -E '(yaml|pep|mock|git|jinja)'
aspy.yaml        1.3.0
autopep8         1.4.4
gitdb            4.0.5
githubrelease    1.5.8
GitPython        3.1.2
Jinja2           2.11.2
mock             4.0.2
PyYAML           5.3b1
yamllint         1.19.0
```
@faudebert you might want to try checking out the `v1.7.0` release tag; at least that works much better for me!

Resolving patch:
diff --git a/scripts/generators/es_template.py b/scripts/generators/es_template.py
--- a/scripts/generators/es_template.py
+++ b/scripts/generators/es_template.py
@@ -198,17 +198,23 @@ def template_settings(es_version, ecs_version, mappings_section, template_settin
         es6_type_fallback(mappings_section['properties'])
 
         # error.stack_trace needs special handling to set
-        # index: false and doc_values: false
-        error_stack_trace_mappings = mappings_section['properties']['error']['properties']['stack_trace']
-        error_stack_trace_mappings.setdefault('index', False)
-        error_stack_trace_mappings.setdefault('doc_values', False)
+        # index: false and doc_values: false if the field
+        # is present in the mappings
+        try:
+            error_stack_trace_mappings = mappings_section['properties']['error']['properties']['stack_trace']
+            error_stack_trace_mappings.setdefault('index', False)
+            error_stack_trace_mappings.setdefault('doc_values', False)
+        except KeyError:
+            pass
 
         template['mappings'] = {'_doc': mappings_section}
     else:
         template['mappings'] = mappings_section
 
     # _meta can't be at template root in legacy templates, so moving back to mappings section
-    mappings_section['_meta'] = template.pop('_meta')
+    # if present
+    if '_meta' in template:
+        mappings_section['_meta'] = template.pop('_meta')
 
     return template
 


Test patch: