Repository: dev-cafe/parselglossy

Issue:
Create an example with an input template and input validation
[Pasting here notes from our meeting last week]

- While the tool will allow "any" grammar, try to express the input structure as YAML.
- Keywords require to be documented.
- Demonstrate a cross-section validation.
- In validation we can use predicates which allow "any" valid Python.
- Keywords either have defaults or are required.

Discussion:
robertodr: As for complex numbers, I will remove them from the Getkw grammar, as they were not part of the original "specification". We might have to resort to separate real and imaginary parts or have an implicit translation predicate from `str` to `complex`.
bast: So at least from my (validation) perspective, complex numbers are not a problem - I was only "lazy" to solve it today but it's a two-line workaround. But perhaps I misunderstood the comment.
codecov[bot]: # [Codecov](https://codecov.io/gh/dev-cafe/parselglossy/pull/20?src=pr&el=h1) Report
> Merging [#20](https://codecov.io/gh/dev-cafe/parselglossy/pull/20?src=pr&el=desc) into [master](https://codecov.io/gh/dev-cafe/parselglossy/commit/8371424a575c1829c8e33d0328775957fd6d4b02?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.


robertodr: Really looking awesome! Thanks @bast :confetti_ball: :green_apple: 

Patch:
diff --git a/.ci/azure-steps.yml b/.ci/azure-steps.yml
index d82f1ee..801fd63 100644
--- a/.ci/azure-steps.yml
+++ b/.ci/azure-steps.yml
@@ -11,5 +11,5 @@ steps:
   displayName: 'Install dependencies'
 
 - script: |
-    pipenv run py.test -v tests/ --doctest-modules
+    pipenv run py.test -s -v tests/ --doctest-modules
   displayName: 'Running tests'
diff --git a/.travis.yml b/.travis.yml
index 441a99e..33c2368 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -18,7 +18,7 @@ install:
 
 script:
   - pipenv run flake8 parselglossy
-  - pipenv run py.test -v --cov=parselglossy tests/ --doctest-module
+  - pipenv run py.test -s -v --cov=parselglossy tests/ --doctest-module
 
 after_success:
   - pipenv run codecov
diff --git a/Pipfile b/Pipfile
index 12dd159..866dfd1 100644
--- a/Pipfile
+++ b/Pipfile
@@ -22,6 +22,7 @@ twine = "*"
 watchdog = "*"
 wheel = "*"
 yapf = "*"
+pyyaml = "*"
 
 [packages]
 click = "*"
diff --git a/parselglossy/read_yaml.py b/parselglossy/read_yaml.py
new file mode 100644
index 0000000..1e77c9d
--- /dev/null
+++ b/parselglossy/read_yaml.py
@@ -0,0 +1,18 @@
+import yaml
+from typing import Any, Dict
+
+
+JSONDict = Dict[str, Any]
+
+
+def read_yaml_file(file_name: str) -> JSONDict:
+    """Reads a YAML file and returns it as a dictionary.
+    """
+    # convert to str() because
+    # pathlib integrates nicely with open only for python >= 3.6
+    with open(str(file_name), 'r') as f:
+        try:
+            d = yaml.safe_load(f)
+        except yaml.YAMLError as e:
+            print(e)
+    return d
diff --git a/parselglossy/validate.py b/parselglossy/validate.py
new file mode 100644
index 0000000..6f27c54
--- /dev/null
+++ b/parselglossy/validate.py
@@ -0,0 +1,199 @@
+import re
+from typing import Any, Dict
+
+
+JSONDict = Dict[str, Any]
+
+
+class InputError(Exception):
+    pass
+
+
+class TemplateError(Exception):
+    pass
+
+
+def type_matches(value: Any, expected_type: str) -> bool:
+    """Checks whether value has the type expected_type.
+
+    Notes
+    -----
+    Allowed types T are: `str`, `int`, `float`, `complex`, `bool`,
+    as well as `List[T]`.
+
+    Returns
+    -------
+    True if value has the type expected_type, otherwise False.
+
+    Raises
+    ------
+    ValueError
+        If expected_type is not among the allowed types.
+    """
+    allowed_basic_types = ['str', 'int', 'float', 'complex', 'bool']
+    allowed_list_types = ['List[{}]'.format(t) for t in allowed_basic_types]
+    allowed_types = allowed_basic_types + allowed_list_types
+
+    # first verify whether expected_type is allowed
+    if expected_type not in allowed_types:
+        raise ValueError('could not recognize expected_type: {}'.format(expected_type))
+
+    expected_type_is_list = re.match(r"^List\[\w+\]$", expected_type) is not None
+
+    if expected_type_is_list:
+        # make sure that value is actually a list
+        if not type(value).__name__ == 'list':
+            return False
+
+        # iterate over each element of the list
+        # and check whether it matches T
+        list_element_type = re.search(r"^List\[(\w+)\]$", expected_type).group(1)
+        for element in value:
+            # consider replacing type(element).__name__
+            # by isinstance but mind that isinstance(element, list_element_type)
+            # will not work since list_element_type is a string
+            # so if you go this route, there needs to be an extra translation layer
+            if not type(element).__name__ == list_element_type:
+                return False
+
+        return True
+    else:
+        # expected type is a basic type
+        # this is the simpler case
+        return type(value).__name__ == expected_type
+
+
+def validate_node(input_dict: JSONDict, template_dict: JSONDict) -> JSONDict:
+    """Validate a node.
+
+    A node is either the root of the input or it is a section.  The node can
+    contain keywords or other sections which we traverse recursively.
+
+    Parameters
+    ----------
+    input_dict: JSONDict
+        Contains the input to be checked.
+    template_dict: JSONDict
+        Contains the input template to be checked against.
+
+    Returns
+    -------
+    input_dict: JSONDict
+        This function modifies input_dict (e.g. fills in defaults).
+
+    Raises
+    ------
+    InputError
+        This signals an error in the input that is to be parsed.
+    TemplateError
+        This signals an error in the input template.
+    """
+    input_sections = [x for x in input_dict if type(input_dict[x]).__name__ == 'dict']
+    input_keywords = list(filter(lambda x: x not in input_sections, input_dict.keys()))
+
+    if 'sections' in template_dict:
+        template_sections = [x['section'] for x in template_dict['sections']]
+    else:
+        template_sections = []
+
+    if 'keywords' in template_dict:
+        template_keywords = [x['keyword'] for x in template_dict['keywords']]
+    else:
+        template_keywords = []
+
+    # stop if we find a keyword without documentation
+    keywords_no_doc = [x['keyword'] for x in template_dict['keywords']
+                       if 'documentation' not in x or x['documentation'].strip() == '']
+    if len(keywords_no_doc) > 0:
+        raise TemplateError('keyword(s) without any documentation: {}'.format(keywords_no_doc))
+
+    # make sure that we did not receive keywords
+    # or sections which we did not expect
+    difference = set(input_keywords).difference(set(template_keywords))
+    if difference != set():
+        raise InputError('found unexpected keyword(s): {}'.format(difference))
+    difference = set(input_sections).difference(set(template_sections))
+    if difference != set():
+        raise InputError('found unexpected section(s): {}'.format(difference))
+
+    # check that keywords without a default are set in the input
+    keywords_no_default = [x['keyword'] for x in template_dict['keywords'] if 'default' not in x]
+    difference = set(keywords_no_default).difference(set(input_keywords))
+    if difference != set():
+        raise InputError('the following keyword(s) must be set: {}'.format(difference))
+
+    # for each keyword verify the type of the value
+    for keyword in input_keywords:
+        template_keyword = list(filter(lambda x: x['keyword'] == keyword, template_dict['keywords']))[0]
+        _type = template_keyword['type']
+        if not type_matches(input_dict[keyword], _type):
+            raise InputError("incorrect type for keyword: '{0}', expected '{1}' type".format(keyword, _type))
+
+    # fill missing input keywords with template defaults
+    for keyword in set(template_keywords).difference(set(input_keywords)):
+        template_keyword = list(filter(lambda x: x['keyword'] == keyword, template_dict['keywords']))[0]
+        _default = template_keyword['default']
+        input_dict[keyword] = _default
+
+    # if this node contains sections, we descend into these and verify these in turn
+    for section in template_sections:
+        input_section = input_dict[section]
+        template_section = list(filter(lambda x: x['section'] == section, template_dict['sections']))[0]
+        input_dict[section] = validate_node(input_section, template_section)
+
+    return input_dict
+
+
+def check_predicates_node(input_dict: JSONDict,
+                          input_dict_node: JSONDict,
+                          template_dict_node: JSONDict) -> None:
+    """Check keyword predicates in a node.
+
+    A node is either the root of the input or it is a section.  The node can
+    contain keywords or other sections which we traverse recursively.
+
+    This function does not modify input arguments.
+
+    Parameters
+    ----------
+    input_dict: JSONDict
+        Contains the entire input tree.
+    input_dict_node: JSONDict
+        Contains the input to be checked: only current node.
+    template_dict: JSONDict
+        Contains the input template which holds the predicates.
+
+    Raises
+    ------
+    InputError
+        This signals an error in the input that is to be parsed.
+    TemplateError
+        This signals an error in the input template.
+    """
+    input_sections = [x for x in input_dict_node if type(input_dict_node[x]).__name__ == 'dict']
+    input_keywords = list(filter(lambda x: x not in input_sections, input_dict_node.keys()))
+
+    if 'sections' in template_dict_node:
+        template_sections = [x['section'] for x in template_dict_node['sections']]
+    else:
+        template_sections = []
+
+    # go through the predicates
+    for keyword in input_keywords:
+        template_keyword = list(filter(lambda x: x['keyword'] == keyword, template_dict_node['keywords']))[0]
+        # this value is used by eval() so we skip E841 warning of flake8
+        value = input_dict_node[keyword]  # noqa
+        if 'predicates' in template_keyword:
+            for predicate in template_keyword['predicates']:
+                try:
+                    r = eval(predicate)
+                except (SyntaxError, NameError):
+                    raise TemplateError("error in predicate '{}' in keyword '{}'".format(predicate, keyword))
+                if not r:
+                    raise InputError('predicate "{}" failed in keyword "{}"'.format(predicate, keyword))
+
+    # if this node contains sections, we descend into these and verify these in turn
+    for section in template_sections:
+        input_section = input_dict_node[section]
+        template_section = list(filter(lambda x: x['section'] == section, template_dict_node['sections']))[0]
+        check_predicates_node(input_dict, input_section, template_section)
diff --git a/setup.py b/setup.py
index bb459c1..5336323 100644
--- a/setup.py
+++ b/setup.py
@@ -13,6 +13,7 @@ with open('HISTORY.rst') as history_file:
 requirements = [
     'Click>=6.0',
     'pyparsing>=2.2',
+    'pyyaml>=3.13',
 ]
 
 setup_requirements = []


Test patch:
diff --git a/tests/test_types.py b/tests/test_types.py
new file mode 100644
index 0000000..9447ddf
--- /dev/null
+++ b/tests/test_types.py
@@ -0,0 +1,35 @@
+from parselglossy.validate import type_matches
+import pytest
+
+
+def test_type_matches():
+
+    # str
+    assert type_matches('a string', 'str')
+    assert not type_matches(1.0, 'str')
+
+    # float and int
+    assert type_matches(1.0, 'float')
+    assert type_matches(1.0e-8, 'float')
+    assert type_matches(1, 'int')
+    assert not type_matches(1, 'float')
+
+    # complex
+    assert type_matches(1.0+1.0j, 'complex')
+
+    # bool
+    assert type_matches(True, 'bool')
+    assert type_matches(False, 'bool')
+    assert not type_matches(0, 'bool')
+
+    # lists
+    assert type_matches([1, 2, 3], 'List[int]')
+    assert not type_matches((1, 2, 3), 'List[int]')
+    assert not type_matches([1, 2, 3], 'List[float]')
+    assert type_matches([1.0, 2.0, 3.0], 'List[float]')
+    assert not type_matches([1.0, 2.0, 3], 'List[float]')
+
+    # unexpected type input
+    with pytest.raises(ValueError):
+        assert type_matches('example', 'weird')
+        assert type_matches('example', 'List[strange]')
diff --git a/tests/test_validation.py b/tests/test_validation.py
new file mode 100644
index 0000000..c69bf80
--- /dev/null
+++ b/tests/test_validation.py
@@ -0,0 +1,145 @@
+import os
+import pytest
+from pathlib import Path
+from parselglossy.validate import (validate_node,
+                                   check_predicates_node,
+                                   InputError,
+                                   TemplateError)
+from parselglossy.read_yaml import read_yaml_file
+from typing import Dict, Any
+
+
+JSONDict = Dict[str, Any]
+
+
+def _helper(category: str,
+            input_file_name: str,
+            template_file_name: str) -> JSONDict:
+    this_path = Path(os.path.dirname(os.path.realpath(__file__)))
+
+    input_file = this_path / 'validation' / category / input_file_name
+    template_file = this_path / 'validation' / category / template_file_name
+
+    input_dict = read_yaml_file(input_file)
+    template_dict = read_yaml_file(template_file)
+
+    # checks everything except predicates
+    input_dict = validate_node(input_dict, template_dict)
+
+    # now that all keywords have some value, we can check predicates
+    check_predicates_node(input_dict, input_dict, template_dict)
+
+    return input_dict
+
+
+def test_validation():
+    input_dict = _helper('overall',
+                         'input.yml',
+                         'template.yml')
+
+    reference = {
+        'title': 'this is an example',
+        'scf': {
+            'functional': 'B3LYP',
+            'max_num_iterations': 20,
+            'some_acceleration': True,
+            'thresholds': {
+                 'some_integral_screening': 0.0002,
+                 'energy': 1e-05
+             },
+            'another_number': 10,
+            # the complex number is a string here because this is how
+            # pyyaml interprets it
+            # for follow-up please see
+            # https://github.com/dev-cafe/parselglossy/issues/26
+            'some_complex_number': '0.0+0.0j'
+        }
+    }
+
+    assert input_dict == reference
+
+
+def test_template_errors():
+
+    # keyword without doc
+    with pytest.raises(TemplateError) as e:
+        input_dict = _helper('template_errors',
+                             'input.yml',
+                             'template_no_documentation.yml')
+    assert str(e.value) == "keyword(s) without any documentation: ['a_short_string']"
+
+    # keyword with empty doc
+    with pytest.raises(TemplateError) as e:
+        input_dict = _helper('template_errors',
+                             'input.yml',
+                             'template_empty_documentation.yml')
+    assert str(e.value) == "keyword(s) without any documentation: ['a_short_string']"
+
+    # keyword with invalid predicate
+    with pytest.raises(TemplateError) as e:
+        input_dict = _helper('template_errors',
+                             'input.yml',
+                             'template_invalid_predicate.yml')
+    assert str(e.value) == "error in predicate '0 < len(value) <= undefined' in keyword 'a_short_string'"
+
+
+def test_input_errors():
+
+    # unexpected keyword
+    with pytest.raises(InputError) as e:
+        input_dict = _helper('input_errors',
+                             'input_unexpected_keyword.yml',
+                             'template.yml')
+    assert str(e.value) == "found unexpected keyword(s): {'strange'}"
+
+    # unexpected section
+    with pytest.raises(InputError) as e:
+        input_dict = _helper('input_errors',
+                             'input_unexpected_section.yml',
+                             'template.yml')
+    assert str(e.value) == "found unexpected section(s): {'weird'}"
+
+    # keyword which has not default is not set
+    with pytest.raises(InputError) as e:
+        input_dict = _helper('input_errors',
+                             'input_missing_keyword.yml',
+                             'template.yml')
+    assert str(e.value) == "the following keyword(s) must be set: {'a_short_string'}"
+
+    # type errors
+    for file_name, error in [
+        ('input_type_error_bool.yml',
+         "incorrect type for keyword: 'some_feature', expected 'bool' type"),
+        ('input_type_error_float.yml',
+         "incorrect type for keyword: 'some_float', expected 'float' type"),
+        ('input_type_error_int.yml',
+         "incorrect type for keyword: 'some_number', expected 'int' type"),
+        ('input_type_error_list.yml',
+         "incorrect type for keyword: 'some_list', expected 'List[float]' type"),
+        ('input_type_error_str.yml',
+         "incorrect type for keyword: 'a_short_string', expected 'str' type"),
+                        ]:
+        with pytest.raises(InputError) as e:
+            input_dict = _helper('input_errors',
+                                 file_name,
+                                 'template.yml')
+        assert str(e.value) == error
+
+    # intra-keyword predicate validation
+    with pytest.raises(InputError) as e:
+        input_dict = _helper('input_errors',
+                             'input_predicate_intra.yml',
+                             'template.yml')
+    assert str(e.value) == 'predicate "value % 2 == 0" failed in keyword "another_number"'
+
+    # predicate validation across keywords
+    with pytest.raises(InputError) as e:
+        input_dict = _helper('input_errors',
+                             'input_predicate_cross.yml',
+                             'template.yml')
+    # in python < 3.6 the dict order is not guaranteed so we are not sure
+    # which of the two errors we hit first
+    assert str(e.value) in [
+        'predicate "value < input_dict[\'some_section\'][\'another_number\']" failed in keyword "some_number"',
+        'predicate "value > input_dict[\'some_section\'][\'some_number\']" failed in keyword "another_number"'
+    ]
diff --git a/tests/validation/input_errors/input_missing_keyword.yml b/tests/validation/input_errors/input_missing_keyword.yml
new file mode 100644
index 0000000..82778a0
--- /dev/null
+++ b/tests/validation/input_errors/input_missing_keyword.yml
@@ -0,0 +1,7 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_predicate_cross.yml b/tests/validation/input_errors/input_predicate_cross.yml
new file mode 100644
index 0000000..ac8608f
--- /dev/null
+++ b/tests/validation/input_errors/input_predicate_cross.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 30
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_predicate_intra.yml b/tests/validation/input_errors/input_predicate_intra.yml
new file mode 100644
index 0000000..b005861
--- /dev/null
+++ b/tests/validation/input_errors/input_predicate_intra.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 21
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_type_error_bool.yml b/tests/validation/input_errors/input_type_error_bool.yml
new file mode 100644
index 0000000..5907268
--- /dev/null
+++ b/tests/validation/input_errors/input_type_error_bool.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 20
+  some_feature: 1
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_type_error_float.yml b/tests/validation/input_errors/input_type_error_float.yml
new file mode 100644
index 0000000..e37af8c
--- /dev/null
+++ b/tests/validation/input_errors/input_type_error_float.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2
+some_section:
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_type_error_int.yml b/tests/validation/input_errors/input_type_error_int.yml
new file mode 100644
index 0000000..e1aa800
--- /dev/null
+++ b/tests/validation/input_errors/input_type_error_int.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 7.0
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_type_error_list.yml b/tests/validation/input_errors/input_type_error_list.yml
new file mode 100644
index 0000000..77ab5dc
--- /dev/null
+++ b/tests/validation/input_errors/input_type_error_list.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3]
diff --git a/tests/validation/input_errors/input_type_error_str.yml b/tests/validation/input_errors/input_type_error_str.yml
new file mode 100644
index 0000000..e1672ce
--- /dev/null
+++ b/tests/validation/input_errors/input_type_error_str.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 0
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_unexpected_keyword.yml b/tests/validation/input_errors/input_unexpected_keyword.yml
new file mode 100644
index 0000000..d3eb9c6
--- /dev/null
+++ b/tests/validation/input_errors/input_unexpected_keyword.yml
@@ -0,0 +1,9 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  strange: 137
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/input_errors/input_unexpected_section.yml b/tests/validation/input_errors/input_unexpected_section.yml
new file mode 100644
index 0000000..dc896b5
--- /dev/null
+++ b/tests/validation/input_errors/input_unexpected_section.yml
@@ -0,0 +1,10 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
+weird:
+  some_number: 7
diff --git a/tests/validation/input_errors/template.yml b/tests/validation/input_errors/template.yml
new file mode 100644
index 0000000..51b5878
--- /dev/null
+++ b/tests/validation/input_errors/template.yml
@@ -0,0 +1,44 @@
+keywords:
+  - keyword: title
+    type: str
+    documentation: |
+      Title of the calculation.
+  - keyword: some_float
+    type: float
+    default: 1.2
+    documentation: |
+      Some float.
+sections:
+  - section: some_section
+    keywords:
+    - keyword: a_short_string
+      type: str
+      predicates:
+        - '0 < len(value) <= 5'
+      documentation: |
+        Some short string.
+    - keyword: some_number
+      type: int
+      default: 5
+      predicates:
+        - "value < input_dict['some_section']['another_number']"
+      documentation: |
+        Some number.
+    - keyword: another_number
+      type: int
+      default: 10
+      predicates:
+        - '0 <= value <= 40'
+        - 'value % 2 == 0'
+        - "value > input_dict['some_section']['some_number']"
+      documentation: |
+        Another number.
+    - keyword: some_feature
+      type: bool
+      default: False
+      documentation: |
+        Some feature.
+    - keyword: some_list
+      type: List[float]
+      documentation: |
+        A list of floats.
diff --git a/tests/validation/overall/input.yml b/tests/validation/overall/input.yml
new file mode 100644
index 0000000..37784d6
--- /dev/null
+++ b/tests/validation/overall/input.yml
@@ -0,0 +1,8 @@
+title: "this is an example"
+scf:
+  functional: 'B3LYP'
+  max_num_iterations: 20
+  some_acceleration: True
+  thresholds:
+    some_integral_screening: 0.0002
+    energy: 0.00001
diff --git a/tests/validation/overall/template.yml b/tests/validation/overall/template.yml
new file mode 100644
index 0000000..f44cf60
--- /dev/null
+++ b/tests/validation/overall/template.yml
@@ -0,0 +1,53 @@
+keywords:
+  - keyword: title
+    type: str
+    documentation: |
+      Title of the calculation.
+      Since there is no default, this keyword is required.
+sections:
+  - section: scf
+    keywords:
+    - keyword: functional
+      type: str
+      predicates:
+        - 'len(value) <= 80'
+      documentation: |
+        XC functional.
+        We also test that the string contains max 80 characters.
+    - keyword: max_num_iterations
+      type: int
+      default: 20
+      documentation: |
+        Max number of iterations.
+    - keyword: another_number
+      type: int
+      default: 10
+      predicates:
+        - '0 <= value <= 40'
+        - 'value % 2 == 0'
+        - "input_dict['scf']['some_acceleration'] == True"
+      documentation: |
+        Testing an even number in some range.
+    - keyword: some_acceleration
+      type: bool
+      default: False
+      documentation: |
+        Turn on the new amazing acceleration scheme.
+    - keyword: some_complex_number
+      type: complex
+      default: 0.0+0.0j
+      documentation: |
+        Testing input with a complex number.
+    sections:
+    - section: thresholds
+      keywords:
+      - keyword: some_integral_screening
+        type: float
+        default: 0.0001
+        documentation: |
+          Some integral based threshold.
+      - keyword: energy
+        type: float
+        default: 0.001
+        documentation: |
+          Some energy based threshold.
diff --git a/tests/validation/template_errors/input.yml b/tests/validation/template_errors/input.yml
new file mode 100644
index 0000000..eefcd7e
--- /dev/null
+++ b/tests/validation/template_errors/input.yml
@@ -0,0 +1,8 @@
+title: 'this is my title'
+some_float: 2.4
+some_section:
+  a_short_string: 'short'
+  some_number: 7
+  another_number: 20
+  some_feature: True
+  some_list: [1.0, 2.0, 3.0]
diff --git a/tests/validation/template_errors/template_empty_documentation.yml b/tests/validation/template_errors/template_empty_documentation.yml
new file mode 100644
index 0000000..8caf7ba
--- /dev/null
+++ b/tests/validation/template_errors/template_empty_documentation.yml
@@ -0,0 +1,45 @@
+keywords:
+  - keyword: title
+    type: str
+    documentation: |
+      Title of the calculation.
+  - keyword: some_float
+    type: float
+    default: 1.2
+    documentation: |
+      Some float.
+sections:
+  - section: some_section
+    keywords:
+    - keyword: a_short_string
+      type: str
+      predicates:
+        - '0 < len(value) <= 5'
+      documentation: ''
+#     documentation: |
+#       Some short string.
+    - keyword: some_number
+      type: int
+      default: 5
+      predicates:
+        - "value < input_dict['some_section']['another_number']"
+      documentation: |
+        Some number.
+    - keyword: another_number
+      type: int
+      default: 10
+      predicates:
+        - '0 <= value <= 40'
+        - 'value % 2 == 0'
+        - "value > input_dict['some_section']['some_number']"
+      documentation: |
+        Another number.
+    - keyword: some_feature
+      type: bool
+      default: False
+      documentation: |
+        Some feature.
+    - keyword: some_list
+      type: List[float]
+      documentation: |
+        A list of floats.
diff --git a/tests/validation/template_errors/template_invalid_predicate.yml b/tests/validation/template_errors/template_invalid_predicate.yml
new file mode 100644
index 0000000..9b961fc
--- /dev/null
+++ b/tests/validation/template_errors/template_invalid_predicate.yml
@@ -0,0 +1,45 @@
+keywords:
+  - keyword: title
+    type: str
+    documentation: |
+      Title of the calculation.
+  - keyword: some_float
+    type: float
+    default: 1.2
+    documentation: |
+      Some float.
+sections:
+  - section: some_section
+    keywords:
+    - keyword: a_short_string
+      type: str
+      predicates:
+        - '0 < len(value) <= undefined'
+#       - '0 < len(value) <= 5'
+      documentation: |
+        Some short string.
+    - keyword: some_number
+      type: int
+      default: 5
+      predicates:
+        - "value < input_dict['some_section']['another_number']"
+      documentation: |
+        Some number.
+    - keyword: another_number
+      type: int
+      default: 10
+      predicates:
+        - '0 <= value <= 40'
+        - 'value % 2 == 0'
+        - "value > input_dict['some_section']['some_number']"
+      documentation: |
+        Another number.
+    - keyword: some_feature
+      type: bool
+      default: False
+      documentation: |
+        Some feature.
+    - keyword: some_list
+      type: List[float]
+      documentation: |
+        A list of floats.
diff --git a/tests/validation/template_errors/template_no_documentation.yml b/tests/validation/template_errors/template_no_documentation.yml
new file mode 100644
index 0000000..bd367e0
--- /dev/null
+++ b/tests/validation/template_errors/template_no_documentation.yml
@@ -0,0 +1,44 @@
+keywords:
+  - keyword: title
+    type: str
+    documentation: |
+      Title of the calculation.
+  - keyword: some_float
+    type: float
+    default: 1.2
+    documentation: |
+      Some float.
+sections:
+  - section: some_section
+    keywords:
+    - keyword: a_short_string
+      type: str
+      predicates:
+        - '0 < len(value) <= 5'
+#     documentation: |
+#       Some short string.
+    - keyword: some_number
+      type: int
+      default: 5
+      predicates:
+        - "value < input_dict['some_section']['another_number']"
+      documentation: |
+        Some number.
+    - keyword: another_number
+      type: int
+      default: 10
+      predicates:
+        - '0 <= value <= 40'
+        - 'value % 2 == 0'
+        - "value > input_dict['some_section']['some_number']"
+      documentation: |
+        Another number.
+    - keyword: some_feature
+      type: bool
+      default: False
+      documentation: |
+        Some feature.
+    - keyword: some_list
+      type: List[float]
+      documentation: |
+        A list of floats.