Repository: elastic/ecs

Issue:
Document how / whether objects should be reused in different places in ECS
In recent conversations we've determined that the following objects should be reusable under specific objects:

- `geo`
  - Expected under `source`, `destination`, `device` and `host`
  - It's not expected at the top level
- `os`
  - Expected under `host`, `device`, `user_agent`
  - It's not expected at the top level
- `user`
  - Expected under `source`, `destination` and `host`
  - It can also be at the top level

Some additional reusable objects have also been discussed but I'm not sure agreement is there yet. Listing them out:

- `threat`
  - Could be nested under the part of the event that's problematic (like source if the source IP is cause for the match)
  - Counterpoint: if we go with a model of saving threat matches to a distinct index, we do not need to nest it. One match per document in this approach. So we can copy the full event data (longer retention on matches than raw events), we can copy the full threat feed indicator that matched, and we can have a field that documents what the matching field is. E.g. `match.field_name: "source.ip"`
- `host`
  - Had a 1-1 discussion with @graphaelli, so the team isn't up to speed yet. But I think this is a good case.
  - Expected at top level or under `agent`
  - It should be under agent when the monitoring agent is deployed on another host (e.g. APM server is the agent, and we also track the host information of the monitored app servers).
  - Other similar case could be Metricbeat monitoring multiple external hosts (e.g. one Mb monitoring lots of RDS instances).

Note that it will be important to "pre-determine" where we expect these objects to be re-used for two reasons:

* To set people's expectations clearly as to what data belongs where
* To properly generate the index template
  * A corollary to this is therefore that this reuseability will have to be represented in the yml files


Add fields for Operating System data
There were some previous discussions around the version field and if it should split up into major, minor, bugfix, patch. I chose to only go with `version` based on the assumption that if someone searches for a specific minor version he will know the major version. So searching for it can be done with a prefix query based on version keyword.


Discussion:
See #72, among others
Sounds similar to https://github.com/elastic/ecs/issues/71 ?
Not really the same thing. And I still need to flesh out the body of this issue, but I think a bit of a free form discussion would be in order before I can do that (about #71 and about this issue).

#71 is about deciding if we let people merge the objects together (compose) or if we add the explicit nesting everywhere we reuse a reuseable object.

What I want to do here is actually have an explanation in the docs about how to do this reuse the right way. The goal being to avoid having tons of tools say they're "ECS compliant" but they're actually all reusing the objects in a way that's different from one source to another.

* An example (probably a bad one) could be that we say "host is reuseable", but also "here are the 3 places where you should put it".
* Another example would be the "endpoint object" (just noticing we don't have a generic name for it yet) that we currently have as `source` an as `destination`, and are considering letting people have as `client` and `server` as well. I think we need guidance on where this object can be reused, in order to avoid having this information spread out in too many places. I could easily see people putting it in: top level, agent, container, device and event (in addition to src/dsv, src/cli).

So this issue is about actually adding the guidance to the doc, once we've figured it out.

Hopefully this clarifies it a bit.
@ruflin @MikePaquette I've fleshed out the body of this issue. Not sure this generates a notification. But I'd like you to check it out and be ready to discuss it at today's ECS meeting, if you can.
@andrewkroh Rebased and https://github.com/elastic/ecs/pull/15 opened to reduce conflicts in the CHANGELOG.

Resolving patch:
diff --git a/scripts/helper.py b/scripts/helper.py
--- a/scripts/helper.py
+++ b/scripts/helper.py
@@ -122,13 +122,31 @@ def get_schema():
     return fields
 
 
-def get_markdown_table(namespace, title_prefix="##", link=False):
+def get_markdown_section(namespace, title_prefix="##", link=False):
+    section_name = namespace["name"]
 
-    output = '{} <a name="{}"></a> {} fields\n\n'.format(title_prefix, namespace["name"], namespace["title"])
+    # Title
+    output = '{} <a name="{}"></a> {} fields\n\n'.format(title_prefix, section_name, namespace["title"])
 
+    # Description
     # Replaces one newlines with two as otherwise double newlines do not show up in markdown
     output += namespace["description"].replace("\n", "\n\n") + "\n"
 
+    # Reuseable object details
+    if "reuseable" in namespace and "expected" in namespace["reuseable"]:
+        sorted_fields = sorted(namespace["reuseable"]["expected"])
+        rendered_fields = map(lambda f: "`{}.{}`".format(f, section_name), sorted_fields)
+        output += "The `{}` fields are expected to be nested at: {}.\n\n".format(
+            section_name, ', '.join(rendered_fields))
+
+        if "top_level" in namespace["reuseable"] and namespace["reuseable"]["top_level"]:
+            template = "Note also that the `{}` fields may be used directly at the top level.\n\n"
+        else:
+            template = "Note also that the `{}` fields are not expected to " + \
+                "be used directly at the top level.\n\n"
+        output += template.format(section_name)
+
+    # Table
     titles = ["Field", "Description", "Level", "Type", "Example"]
 
     for title in titles:
@@ -151,6 +169,7 @@ def get_markdown_table(namespace, title_prefix="##", link=False):
 
     output += "\n\n"
 
+    # Footnote
     if "footnote" in namespace:
         output += namespace["footnote"].replace("\n", "\n\n") + "\n"
 
diff --git a/scripts/schemas.py b/scripts/schemas.py
--- a/scripts/schemas.py
+++ b/scripts/schemas.py
@@ -34,32 +34,19 @@ def create_csv(fields, file):
                 schema_writer.writerow([field["name"], field["type"], field["level"], field["example"]])
 
 
-def create_markdown(fields, file):
-    # Create markdown schema output file
-    output = open(file, 'w')
-
-    for namespace in fields:
-        if len(namespace["fields"]) == 0:
-            continue
-        output.write(get_markdown_table(namespace))
-
-    output.close()
-
-
-def create_markdown_string(fields):
+def create_markdown_document(fields):
     # Create markdown schema output string
-    output = ""
+    tables = ""
 
     links = ""
     for namespace in fields:
         if len(namespace["fields"]) == 0:
             continue
-        # Link list to field prefixes
+        # Links to each namespace / top level object
         links += " * [{} fields](#{})\n".format(namespace["title"], namespace["name"])
-        output += get_markdown_table(namespace)
+        tables += get_markdown_section(namespace)
 
-    output = links + "\n" + output + "\n\n"
-    return output
+    return links + "\n" + tables + "\n\n"
 
 
 def filtered_fields(fields, groups):
@@ -100,11 +87,10 @@ def check_fields(fields):
         groups = [1, 2, 3]
         f_fields = filtered_fields(sortedNamespaces, groups)
         # Print to stdout
-        print(create_markdown_string(f_fields))
+        print(create_markdown_document(f_fields))
 
     # Generates schema.csv
     else:
         groups = [1, 2, 3]
         f_fields = filtered_fields(sortedNamespaces, groups)
-        #create_markdown(f_fields, "schema.md")
         create_csv(f_fields, "schema.csv")
diff --git a/scripts/use-cases.py b/scripts/use-cases.py
--- a/scripts/use-cases.py
+++ b/scripts/use-cases.py
@@ -58,7 +58,7 @@ def write_stdout():
                 fields.append(section_fields)
 
         global_fields = {"name": use_case["name"], "title": use_case["title"], "description": "", "fields": fields}
-        output += get_markdown_table(global_fields, "###", link_prefix) + "\n"
+        output += get_markdown_section(global_fields, "###", link_prefix) + "\n"
 
         # Write output to /use-cases/use_case["name"].md file
         # Adjust links


Test patch: