Repository: elastic/ecs

Issue:
`ValueError` exception using `--ref` 
<!--
GitHub is reserved for bug reports and feature requests; it is not the place
for general questions. If you have a general question, please visit the
[forums](https://discuss.elastic.co/tag/elastic-common-schema).

Please fill in the following details to help us reproduce the bug:
-->

**Description of the problem including expected versus actual behavior**:

Running the `generator.py` with the `--ref` option with a tag or commit prior to d8a5e2a will fail with a `ValueError` exception. Using `git checkout <tag>` continues to work.

**Steps to reproduce**:

```
$ python scripts/generator.py --ref v1.5.0
```

**Provide logs (if relevant)**:

#### Working:

```
$ python scripts/generator.py --ref d8a5e2a
Loading schemas from git ref d8a5e2a
Running generator. ECS version 1.6.0-dev
```

#### Failing:

git tag:

```
$ python scripts/generator.py --ref v1.5.0
Loading schemas from git ref v1.5.0
Running generator. ECS version 1.5.0
Traceback (most recent call last):
  File "scripts/generator.py", line 94, in <module>
    main()
  File "scripts/generator.py", line 44, in main
    cleaner.clean(fields)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 23, in clean
    visitor.visit_fields(fields, fieldset_func=schema_cleanup, field_func=field_cleanup)
  File "<redacted>/ecs/scripts/schema/visitor.py", line 21, in visit_fields
    visit_fields(details['fields'],
  File "<redacted>/ecs/scripts/schema/visitor.py", line 19, in visit_fields
    field_func(details)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 125, in field_cleanup
    field_assertions_and_warnings(field)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 168, in field_assertions_and_warnings
    single_line_short_description(field)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 189, in single_line_short_description
    raise ValueError(msg)
ValueError: Short descriptions must be single line, and under 120 characters (current length: 134).
Offending field or field set: number
Short description:
  Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
```

git commit sha:

```
$ python scripts/generator.py --ref 75da22b
Loading schemas from git ref 75da22b
Running generator. ECS version 1.6.0-dev
Traceback (most recent call last):
  File "scripts/generator.py", line 94, in <module>
    main()
  File "scripts/generator.py", line 44, in main
    cleaner.clean(fields)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 23, in clean
    visitor.visit_fields(fields, fieldset_func=schema_cleanup, field_func=field_cleanup)
  File "<redacted>/ecs/scripts/schema/visitor.py", line 21, in visit_fields
    visit_fields(details['fields'],
  File "<redacted>/ecs/scripts/schema/visitor.py", line 19, in visit_fields
    field_func(details)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 125, in field_cleanup
    field_assertions_and_warnings(field)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 168, in field_assertions_and_warnings
    single_line_short_description(field)
  File "<redacted>/ecs/scripts/schema/cleaner.py", line 189, in single_line_short_description
    raise ValueError(msg)
ValueError: Short descriptions must be single line, and under 120 characters (current length: 134).
Offending field or field set: number
Short description:
  Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
```

**Any additional context:**

The short descriptions were fixed in a91aa11d905dbce2129309e31c074fed5929a448 to align with the new length checking of `short`, but previously fields missing `short` fall back to the `description` field which is failing the new `short` character length check. Need to investigate how to improve `--ref` to handle this change in the generator.


Discussion:
Oh jeez we're getting into meta-land 😂 

As you point out, since #871 ECS tools are now stricter in validating short descriptions. That commit id is the merge commit for that PR. This became necessary after a subtle issue in #762 that had the short description as a full paragraph. Although when working on #871 I did adjust a lot of descriptions, so `--ref` probably doesn't work at all on anything prior to #871.

I'm not 100% sure what the right answer is, here.

I do want the main build to fail if short descriptions are not actually short. But anyone building custom artifacts based on `--ref`, `--include` and `--subset` should be able to get their work done as well...

Here's a few thoughts:

- Approach this simply and disable this specific error when someone is running the tools with `--ref`.
- Add a flag to ignore all validation errors. This sounds a bit blunt; on the other hand, this transition point where we added this validation will fade into the past, as we keep releasing ECS versions.
- Add a flag that turns all validation errors into warnings instead.
- Add a `--strict` flag and use it for the main build. Some of these validation errors could be run only when in strict mode.


And this reminds me that we would benefit from writing a few tests that run the CLI tool, to catch things like that. The idea was floated here a while back https://github.com/elastic/ecs/issues/822#issuecomment-617243627

Resolving patch:
diff --git a/scripts/generator.py b/scripts/generator.py
--- a/scripts/generator.py
+++ b/scripts/generator.py
@@ -41,7 +41,7 @@ def main():
     # ecs_helpers.yaml_dump('ecs.yml', fields)
 
     fields = loader.load_schemas(ref=args.ref, included_files=args.include)
-    cleaner.clean(fields)
+    cleaner.clean(fields, strict=args.strict)
     finalizer.finalize(fields)
     fields = subset_filter.filter(fields, args.subset, out_dir)
     nested, flat = intermediate_files.generate(fields, os.path.join(out_dir, 'ecs'), default_dirs)
@@ -72,6 +72,8 @@ def argument_parser():
                         help='index template settings to use when generating elasticsearch template')
     parser.add_argument('--mapping-settings', action='store',
                         help='mapping settings to use when generating elasticsearch template')
+    parser.add_argument('--strict', action='store_true',
+                        help='enforce stricter checking at schema cleanup')
     args = parser.parse_args()
     # Clean up empty include of the Makefile
     if args.include and [''] == args.include:
diff --git a/scripts/generators/ecs_helpers.py b/scripts/generators/ecs_helpers.py
--- a/scripts/generators/ecs_helpers.py
+++ b/scripts/generators/ecs_helpers.py
@@ -2,6 +2,7 @@
 import os
 import yaml
 import git
+import warnings
 
 from collections import OrderedDict
 from copy import deepcopy
@@ -159,3 +160,17 @@ def list_extract_keys(lst, key_name):
 def is_intermediate(field):
     '''Encapsulates the check to see if a field is an intermediate field or a "real" field.'''
     return ('intermediate' in field['field_details'] and field['field_details']['intermediate'])
+
+
+# Warning helper
+
+
+def strict_warning(msg):
+    """Call warnings.warn(msg) for operations that would throw an Exception
+       if operating in `--strict` mode. Allows a custom message to be passed.
+
+    :param msg: custom text which will be displayed with wrapped boilerplate
+                for strict warning messages.
+    """
+    warn_message = f"{msg}\n\nThis will cause an exception when running in strict mode."
+    warnings.warn(warn_message)
diff --git a/scripts/schema/cleaner.py b/scripts/schema/cleaner.py
--- a/scripts/schema/cleaner.py
+++ b/scripts/schema/cleaner.py
@@ -19,7 +19,9 @@
 # deal with final field names either.
 
 
-def clean(fields):
+def clean(fields, strict=False):
+    global strict_mode
+    strict_mode = strict
     visitor.visit_fields(fields, fieldset_func=schema_cleanup, field_func=field_cleanup)
 
 
@@ -46,7 +48,7 @@ def schema_cleanup(schema):
     else:
         schema['schema_details']['prefix'] = schema['field_details']['name'] + '.'
     normalize_reuse_notation(schema)
-    # Final validity check
+    # Final validity check if in strict mode
     schema_assertions_and_warnings(schema)
 
 
@@ -73,7 +75,7 @@ def schema_mandatory_attributes(schema):
 
 def schema_assertions_and_warnings(schema):
     '''Additional checks on a fleshed out schema'''
-    single_line_short_description(schema)
+    single_line_short_description(schema, strict=strict_mode)
 
 
 def normalize_reuse_notation(schema):
@@ -165,7 +167,8 @@ def field_mandatory_attributes(field):
 def field_assertions_and_warnings(field):
     '''Additional checks on a fleshed out field'''
     if not ecs_helpers.is_intermediate(field):
-        single_line_short_description(field)
+        # check short description length if in strict mode
+        single_line_short_description(field, strict=strict_mode)
         if field['field_details']['level'] not in ACCEPTABLE_FIELD_LEVELS:
             msg = "Invalid level for field '{}'.\nValue: {}\nAcceptable values: {}".format(
                 field['field_details']['name'], field['field_details']['level'],
@@ -178,7 +181,7 @@ def field_assertions_and_warnings(field):
 SHORT_LIMIT = 120
 
 
-def single_line_short_description(schema_or_field):
+def single_line_short_description(schema_or_field, strict=True):
     short_length = len(schema_or_field['field_details']['short'])
     if "\n" in schema_or_field['field_details']['short'] or short_length > SHORT_LIMIT:
         msg = "Short descriptions must be single line, and under {} characters (current length: {}).\n".format(
@@ -186,4 +189,7 @@ def single_line_short_description(schema_or_field):
         msg += "Offending field or field set: {}\nShort description:\n  {}".format(
             schema_or_field['field_details']['name'],
             schema_or_field['field_details']['short'])
-        raise ValueError(msg)
+        if strict:
+            raise ValueError(msg)
+        else:
+            ecs_helpers.strict_warning(msg)


Test patch:
diff --git a/scripts/tests/unit/test_schema_cleaner.py b/scripts/tests/unit/test_schema_cleaner.py
--- a/scripts/tests/unit/test_schema_cleaner.py
+++ b/scripts/tests/unit/test_schema_cleaner.py
@@ -262,6 +262,26 @@ def test_multiline_short_description_raises(self):
         with self.assertRaisesRegex(ValueError, 'single line'):
             cleaner.single_line_short_description(schema)
 
+    def test_very_long_short_description_warns_strict_disabled(self):
+        schema = {'field_details': {
+            'name': 'fake_schema',
+            'short': "Single line but really long. " * 10}}
+        try:
+            with self.assertWarnsRegex(UserWarning, 'under 120 characters \(current length: 290\)'):
+                cleaner.single_line_short_description(schema, strict=False)
+        except Exception:
+            self.fail("cleaner.single_line_short_description() raised Exception unexpectedly.")
+
+    def test_multiline_short_description_warns_strict_disabled(self):
+        schema = {'field_details': {
+            'name': 'fake_schema',
+            'short': "multiple\nlines"}}
+        try:
+            with self.assertWarnsRegex(UserWarning, 'single line'):
+                cleaner.single_line_short_description(schema, strict=False)
+        except Exception:
+            self.fail("cleaner.single_line_short_description() raised Exception unexpectedly.")
+
     def test_clean(self):
         '''A high level sanity test'''
         fields = self.schema_process()