Repository: elastic/ecs

Issue:
Generated artifacts for composable index templates
**Summary**

Include support for index template v2 in ECS by generating component templates and using those components to compose a v2 template. Users may also find value in adopting ECS published component templates into their own composable templates.

**Background**

Elasticsearch has introduced the second version of index templates, known as composable templates. A composable template is made up of component templates. Component templates are re-usable configuration building blocks that configure mappings, settings, and aliases. More details around can be found in the [Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) and https://github.com/elastic/elasticsearch/issues/53101.


Discussion:
I wonder if folks have feedback on what level of composability would be most helpful.

Currently we provide a [sample Elasticsearch template](https://github.com/elastic/ecs/tree/master/generated/elasticsearch) that includes all the fields. Folks can adjust it using the scripts in this repo, following [USAGE.md](https://github.com/elastic/ecs/blob/master/USAGE.md) to add custom fields, remove unneeded fields for a given index/data source.

A few different approaches could be taken, in terms of how granular we make these new component templates:

* One component template for core fields, one for extended fields
* One component template per field set
  * Including fields of both levels in the same component template
  * Or one per field set & level (e.g. event-core & event-extended)
* More holistic component templates per use case (e.g. one for web logs, one for firewalls, one for authentications...)

The first two are the low hanging fruits. I think one component template per field set may be the most straightforward one to implement, and easiest to understand for users.

I would be most interested in component templates per field set which in my opinion should include fields from all levels.
How would you approach more holistic component templates? Adding additional tags to each element which mark the use case?
Indeed, more holistic templates would require more work to put together in ECS, and we'd need to e.g. tag each field set or event parts of field sets according to which use cases it's generally used in. Although I get the sense that this approach would be difficult to get right.
Yes, it seems that there would always be some case which would be forgotten with tagging.
I would like to see a component template per top level object in the ECS.  For example, something like this

```json
PUT _component_template/ecs-mapping-base
{
  "template": {
    "mappings": {
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": false,
      "_source": {
        "enabled": true,
        "includes": [],
        "excludes": []
      },
      "_meta": {
        "version": "1.7.0"
      },
      "_routing": {
        "required": false
      },
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "labels": {
          "type": "object"
        },
        "message": {
          "type": "text",
          "norms": false
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        }
      }
    }
  },
  "_meta": {
    "original": "https://github.com/elastic/ecs/blob/1.7/generated/elasticsearch/7/template.json",
    "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-base.html"
  }
}
```

```json
PUT _component_template/ecs-mapping-event
{
  "template": {
    "mappings": {
      "dynamic_templates": [],
      "properties": {
        "event": {
          "type": "object",
          "properties": {
            "action": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "category": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "code": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "created": {
              "type": "date"
            },
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "duration": {
              "type": "long"
            },
            "end": {
              "type": "date"
            },
            "hash": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ingested": {
              "type": "date"
            },
            "kind": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "module": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "original": {
              "type": "keyword",
              "doc_values": false,
              "ignore_above": 1024,
              "index": false
            },
            "outcome": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "provider": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "reason": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "reference": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "risk_score": {
              "type": "float"
            },
            "risk_score_norm": {
              "type": "float"
            },
            "sequence": {
              "type": "long"
            },
            "severity": {
              "type": "long"
            },
            "start": {
              "type": "date"
            },
            "timezone": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "url": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      }
    }
  },
  "_meta": {
    "original": "https://github.com/elastic/ecs/blob/1.7/generated/elasticsearch/7/template.json",
    "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-event.html"
  }
}
```

Then I could take these top level objects to create the `_index_template` for a (custom) given source.  For example, a webserver access log would need the `ecs-mapping-base`, `ecs-mapping-event`, `ecs-mapping-http`, etc component templates.  The top level objects could be added to the index template based on what fields were in the log file.  The next custom source, would use a different sub-set of the ECS component templates.

When I'm doing custom source mappings, I don't pay attention to if a field's level is `core` or `extended`.  I'm just looking at where does this piece of data best fit to achieve (1) standard field names for memorization, (2) multi log source searching, and (3) best use of the prebuilt content for Security and Observability.
Thanks for chiming in @a03nikki! I like these meta doc links too, that's a great idea ❤️ 

The plan for now is indeed to go with component templates per field sets (or top level objects).

It's on our plate to do in the near term. And since these aren't schema changes, we could backport this to the 1.7 branch, even if 1.7.0 is already released. The way we'll do it is these component templates will be new artifacts. The current "monolithic" template will remain as it is.

Do you think it would be helpful to backport even further, like 1.6? How common is it for you and colleagues to work on projects that are one or two ECS versions behind? Or do you simply always use the most recent version?

Resolving patch:
diff --git a/scripts/generator.py b/scripts/generator.py
--- a/scripts/generator.py
+++ b/scripts/generator.py
@@ -41,7 +41,7 @@ def main():
     # ecs_helpers.yaml_dump('ecs.yml', fields)
 
     # Detect usage of experimental changes to tweak artifact version label
-    if loader.EXPERIMENTAL_SCHEMA_DIR in args.include:
+    if args.include and loader.EXPERIMENTAL_SCHEMA_DIR in args.include:
         ecs_version += "+exp"
 
     fields = loader.load_schemas(ref=args.ref, included_files=args.include)
@@ -56,7 +56,8 @@ def main():
         exit()
 
     csv_generator.generate(flat, ecs_version, out_dir)
-    es_template.generate(flat, ecs_version, out_dir, args.template_settings, args.mapping_settings)
+    es_template.generate(nested, ecs_version, out_dir, args.mapping_settings)
+    es_template.generate_legacy(flat, ecs_version, out_dir, args.template_settings, args.mapping_settings)
     beats.generate(nested, ecs_version, out_dir)
     if args.include or args.subset:
         exit()
diff --git a/scripts/generators/es_template.py b/scripts/generators/es_template.py
--- a/scripts/generators/es_template.py
+++ b/scripts/generators/es_template.py
@@ -1,44 +1,136 @@
+import copy
 import json
 import sys
-import copy
 
 from os.path import join
+
 from generators import ecs_helpers
 from schema.cleaner import field_or_multi_field_datatype_defaults
 from schema.oss import TYPE_FALLBACKS
 
 
-def generate(ecs_flat, ecs_version, out_dir, template_settings_file, mapping_settings_file):
+# Composable Template
+
+def generate(ecs_nested, ecs_version, out_dir, mapping_settings_file):
+    """This generates all artifacts for the composable template approach"""
+    all_component_templates(ecs_nested, ecs_version, out_dir)
+    component_names = component_name_convention(ecs_version, ecs_nested)
+    save_composable_template(ecs_version, component_names, out_dir, mapping_settings_file)
+
+
+def save_composable_template(ecs_version, component_names, out_dir, mapping_settings_file):
+    """Generate the master sample composable template"""
+    template = {
+        "index_patterns": ["try-ecs-*"],
+        "composed_of": component_names,
+        "priority": 1,  # Very low, as this is a sample template
+        "_meta": {
+            "ecs_version": ecs_version,
+            "description": "Sample composable template that includes all ECS fields"
+        },
+        "template": {
+            "settings": {
+                "index": {
+                    "mapping": {
+                        "total_fields": {
+                            "limit": 2000
+                        }
+                    }
+                }
+            },
+            "mappings": mapping_settings(mapping_settings_file)
+        }
+    }
+    filename = join(out_dir, "elasticsearch/template.json")
+    save_json(filename, template)
+
+
+def all_component_templates(ecs_nested, ecs_version, out_dir):
+    """Generate one component template per field set"""
+    component_dir = join(out_dir, 'elasticsearch/component')
+    ecs_helpers.make_dirs(component_dir)
+
+    for (fieldset_name, fieldset) in candidate_components(ecs_nested).items():
+        field_mappings = {}
+        for (flat_name, field) in fieldset['fields'].items():
+            name_parts = flat_name.split('.')
+            dict_add_nested(field_mappings, name_parts, entry_for(field))
+
+        save_component_template(fieldset_name, ecs_version, component_dir, field_mappings)
+
+
+def save_component_template(template_name, ecs_version, out_dir, field_mappings):
+    filename = join(out_dir, template_name) + ".json"
+    reference_url = "https://www.elastic.co/guide/en/ecs/current/ecs-{}.html".format(template_name)
+
+    template = {
+        'template': {'mappings': {'properties': field_mappings}},
+        '_meta': {
+            'ecs_version': ecs_version,
+            'documentation': reference_url
+        }
+    }
+    save_json(filename, template)
+
+
+def component_name_convention(ecs_version, ecs_nested):
+    version = ecs_version.replace('+', '-')
+    names = []
+    for (fieldset_name, fieldset) in candidate_components(ecs_nested).items():
+        names.append("ecs_{}_{}".format(version, fieldset_name))
+    return names
+
+
+def candidate_components(ecs_nested):
+    """Returns same structure as ecs_nested, but skips all field sets with reusable.top_level: False"""
+    components = {}
+    for (fieldset_name, fieldset) in ecs_nested.items():
+        if fieldset.get('reusable', None):
+            if not fieldset['reusable']['top_level']:
+                continue
+        components[fieldset_name] = fieldset
+    return components
+
+
+# Legacy template
+
+
+def generate_legacy(ecs_flat, ecs_version, out_dir, template_settings_file, mapping_settings_file):
+    """Generate the legacy index template"""
     field_mappings = {}
     for flat_name in sorted(ecs_flat):
         field = ecs_flat[flat_name]
-        nestings = flat_name.split('.')
-        dict_add_nested(field_mappings, nestings, entry_for(field))
-
-    if mapping_settings_file:
-        with open(mapping_settings_file) as f:
-            mappings_section = json.load(f)
-    else:
-        mappings_section = default_mapping_settings(ecs_version)
+        name_parts = flat_name.split('.')
+        dict_add_nested(field_mappings, name_parts, entry_for(field))
 
+    mappings_section = mapping_settings(mapping_settings_file)
     mappings_section['properties'] = field_mappings
 
-    generate_template_version(6, mappings_section, out_dir, template_settings_file)
-    generate_template_version(7, mappings_section, out_dir, template_settings_file)
+    generate_legacy_template_version(6, ecs_version, mappings_section, out_dir, template_settings_file)
+    generate_legacy_template_version(7, ecs_version, mappings_section, out_dir, template_settings_file)
+
+
+def generate_legacy_template_version(es_version, ecs_version, mappings_section, out_dir, template_settings_file):
+    ecs_helpers.make_dirs(join(out_dir, 'elasticsearch', str(es_version)))
+    template = template_settings(es_version, ecs_version, mappings_section, template_settings_file)
 
-# Field mappings
+    filename = join(out_dir, "elasticsearch/{}/template.json".format(es_version))
+    save_json(filename, template)
+
+
+# Common helpers
 
 
-def dict_add_nested(dct, nestings, value):
-    current_nesting = nestings[0]
-    rest_nestings = nestings[1:]
-    if len(rest_nestings) > 0:
+def dict_add_nested(dct, name_parts, value):
+    current_nesting = name_parts[0]
+    rest_name_parts = name_parts[1:]
+    if len(rest_name_parts) > 0:
         dct.setdefault(current_nesting, {})
         dct[current_nesting].setdefault('properties', {})
 
         dict_add_nested(
             dct[current_nesting]['properties'],
-            rest_nestings,
+            rest_name_parts,
             value)
 
     else:
@@ -84,25 +176,29 @@ def entry_for(field):
         raise ex
     return field_entry
 
-# Generated files
+
+def mapping_settings(mapping_settings_file):
+    if mapping_settings_file:
+        with open(mapping_settings_file) as f:
+            mappings = json.load(f)
+    else:
+        mappings = default_mapping_settings()
+    return mappings
 
 
-def generate_template_version(elasticsearch_version, mappings_section, out_dir, template_settings_file):
-    ecs_helpers.make_dirs(join(out_dir, 'elasticsearch', str(elasticsearch_version)))
+def template_settings(es_version, ecs_version, mappings_section, template_settings_file):
     if template_settings_file:
         with open(template_settings_file) as f:
             template = json.load(f)
     else:
-        template = default_template_settings()
-    if elasticsearch_version == 6:
+        template = default_template_settings(ecs_version)
+    if es_version == 6:
         es6_mappings_section = copy.deepcopy(mappings_section)
         es6_type_fallback(es6_mappings_section['properties'])
         template['mappings'] = {'_doc': es6_mappings_section}
     else:
         template['mappings'] = mappings_section
-
-    filename = join(out_dir, "elasticsearch/{}/template.json".format(elasticsearch_version))
-    save_json(filename, template)
+    return template
 
 
 def save_json(file, data):
@@ -113,9 +209,10 @@ def save_json(file, data):
         jsonfile.write(json.dumps(data, indent=2, sort_keys=True))
 
 
-def default_template_settings():
+def default_template_settings(ecs_version):
     return {
         "index_patterns": ["try-ecs-*"],
+        "_meta": {"version": ecs_version},
         "order": 1,
         "settings": {
             "index": {
@@ -126,14 +223,12 @@ def default_template_settings():
                 },
                 "refresh_interval": "5s"
             }
-        },
-        "mappings": {}
+        }
     }
 
 
-def default_mapping_settings(ecs_version):
+def default_mapping_settings():
     return {
-        "_meta": {"version": ecs_version},
         "date_detection": False,
         "dynamic_templates": [
             {
@@ -145,8 +240,7 @@ def default_mapping_settings(ecs_version):
                     "match_mapping_type": "string"
                 }
             }
-        ],
-        "properties": {}
+        ]
     }
 
 


Test patch: