Repository: FAIRmat-NFDI/pynxtools
Pull request #638: add a separate error for names with wrong type and remove such keys
Author: login: lukaspie
Created: 2025-05-14T15:42:54Z | Merged: 2025-05-23T16:17:41Z

The idea is to catch the following situation: a field/group is given that has a name that another group/field at the same level has (and which is therefore reserved).

Approach:
1) Throw and log an error on such cases
2) this error should be separate to the existing checks for concept names
3) if this error is thrown, it should result in that field not to be written at all (i.e., it shall be removed from the template)



Diff:
diff --git a/CITATION.cff b/CITATION.cff
index cc9ca7cca..88276769b 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -4,7 +4,7 @@ message:
   If you use this software, please cite it using the
   metadata from this file.
 type: software
-version: 0.10.6
+version: 0.10.7
 authors:
   - given-names: Sherjeel
     family-names: Shabih
diff --git a/src/pynxtools/data/NXtest.nxdl.xml b/src/pynxtools/data/NXtest.nxdl.xml
index 03e6c9b95..c58cc0f8c 100644
--- a/src/pynxtools/data/NXtest.nxdl.xml
+++ b/src/pynxtools/data/NXtest.nxdl.xml
@@ -45,8 +45,8 @@
         </group>
         <group type="NXdata" name="specified_group" nameType="specified">
             <doc>A group with a name and nameType="specified".</doc>
-            <field name="specified_field" type="NX_FLOAT" units="NX_ANY">
-                <attribute name="specified_attr_in_field"/>
+            <field name="specified_field" nameType="specified" type="NX_FLOAT" units="NX_ANY">
+                <attribute name="specified_attr_in_field" nameType="specified"/>
             </field>
             <attribute name="specified_attr"/>
         </group>
diff --git a/src/pynxtools/dataconverter/convert.py b/src/pynxtools/dataconverter/convert.py
index afe273422..508071906 100644
--- a/src/pynxtools/dataconverter/convert.py
+++ b/src/pynxtools/dataconverter/convert.py
@@ -176,18 +176,12 @@ def transfer_data_into_template(
     for entry_name in entry_names:
         helpers.write_nexus_def_to_entry(data, entry_name, nxdl_name)
     if not skip_verify:
-        valid, keys_to_remove = validate_dict_against(
+        valid = validate_dict_against(
             nxdl_name,
             data,
             ignore_undocumented=ignore_undocumented,
         )
 
-        # remove attributes that belong to non-existing fields
-
-        for key in keys_to_remove:
-            # data.__delitem__(key)
-            del data[key]
-
         if fail and not valid:
             raise ValidationFailed(
                 "The data does not match the given NXDL. "
diff --git a/src/pynxtools/dataconverter/helpers.py b/src/pynxtools/dataconverter/helpers.py
index 605847cd0..b3f66578e 100644
--- a/src/pynxtools/dataconverter/helpers.py
+++ b/src/pynxtools/dataconverter/helpers.py
@@ -22,7 +22,7 @@
 import os
 import re
 from datetime import datetime, timezone
-from enum import Enum
+from enum import Enum, auto
 from functools import lru_cache
 from typing import Any, Callable, List, Optional, Tuple, Union, Sequence, cast
 
@@ -46,30 +46,32 @@
 
 
 class ValidationProblem(Enum):
-    UnitWithoutDocumentation = 1
-    InvalidEnum = 2
-    OpenEnumWithNewItem = 3
-    MissingRequiredGroup = 4
-    MissingRequiredField = 5
-    MissingRequiredAttribute = 6
-    InvalidType = 7
-    InvalidDatetime = 8
-    IsNotPosInt = 9
-    ExpectedGroup = 10
-    MissingDocumentation = 11
-    MissingUnit = 12
-    ChoiceValidationError = 13
-    UnitWithoutField = 14
-    AttributeForNonExistingField = 15
-    BrokenLink = 16
-    FailedNamefitting = 17
-    NXdataMissingSignalData = 18
-    NXdataMissingAxisData = 19
-    NXdataAxisMismatch = 20
-    KeyToBeRemoved = 21
-    InvalidConceptForNonVariadic = 22
-    ReservedSuffixWithoutField = 23
-    ReservedPrefixInWrongContext = 24
+    UnitWithoutDocumentation = auto()
+    InvalidEnum = auto()
+    OpenEnumWithNewItem = auto()
+    MissingRequiredGroup = auto()
+    MissingRequiredField = auto()
+    MissingRequiredAttribute = auto()
+    InvalidType = auto()
+    InvalidDatetime = auto()
+    IsNotPosInt = auto()
+    ExpectedGroup = auto()
+    ExpectedField = auto()
+    MissingDocumentation = auto()
+    MissingUnit = auto()
+    ChoiceValidationError = auto()
+    UnitWithoutField = auto()
+    AttributeForNonExistingField = auto()
+    BrokenLink = auto()
+    FailedNamefitting = auto()
+    NXdataMissingSignalData = auto()
+    NXdataMissingAxisData = auto()
+    NXdataAxisMismatch = auto()
+    KeyToBeRemoved = auto()
+    InvalidConceptForNonVariadic = auto()
+    ReservedSuffixWithoutField = auto()
+    ReservedPrefixInWrongContext = auto()
+    InvalidNexusTypeForNamedConcept = auto()
 
 
 class Collector:
@@ -96,9 +98,9 @@ def _log(self, path: str, log_type: ValidationProblem, value: Optional[Any], *ar
                 f"The value at {path} does not match with the enumerated items from the open enumeration: {value}."
             )
         elif log_type == ValidationProblem.MissingRequiredGroup:
-            logger.warning(f"The required group, {path}, hasn't been supplied.")
+            logger.error(f"The required group, {path}, hasn't been supplied.")
         elif log_type == ValidationProblem.MissingRequiredField:
-            logger.warning(
+            logger.error(
                 f"The data entry corresponding to {path} is required "
                 "and hasn't been supplied by the reader.",
             )
@@ -118,9 +120,9 @@ def _log(self, path: str, log_type: ValidationProblem, value: Optional[Any], *ar
                 f"The value at {path} should be a positive int, but is {value}."
             )
         elif log_type == ValidationProblem.ExpectedGroup:
-            logger.warning(
-                f"Expected a group at {path} but found a field or attribute."
-            )
+            logger.error(f"Expected a group at {path} but found a field or attribute.")
+        elif log_type == ValidationProblem.ExpectedField:
+            logger.error(f"Expected a field at {path} but found a group.")
         elif log_type == ValidationProblem.MissingDocumentation:
             if "@" in path.rsplit("/")[-1]:
                 logger.warning(f"Attribute {path} written without documentation.")
@@ -140,7 +142,7 @@ def _log(self, path: str, log_type: ValidationProblem, value: Optional[Any], *ar
                 "but the field does not exist."
             )
         elif log_type == ValidationProblem.BrokenLink:
-            logger.warning(f"Broken link at {path} to {value}")
+            logger.warning(f"Broken link at {path} to {value}.")
         elif log_type == ValidationProblem.FailedNamefitting:
             logger.warning(f"Found no namefit of {path} in {value}.")
         elif log_type == ValidationProblem.NXdataMissingSignalData:
@@ -152,7 +154,7 @@ def _log(self, path: str, log_type: ValidationProblem, value: Optional[Any], *ar
                 f"Length of axis {path} does not match to {value} in dimension {args[0]}"
             )
         elif log_type == ValidationProblem.KeyToBeRemoved:
-            logger.warning(f"The attribute {path} will not be written.")
+            logger.warning(f"The {value} {path} will not be written.")
         elif log_type == ValidationProblem.InvalidConceptForNonVariadic:
             value = cast(Any, value)
             log_text = f"Given {value.type} name '{path}' conflicts with the non-variadic name '{value}'"
@@ -169,6 +171,12 @@ def _log(self, path: str, log_type: ValidationProblem, value: Optional[Any], *ar
             if value != "<unknown>":
                 log_text += f" It is only valid in the context of {value}."
             logger.error(log_text)
+        elif log_type == ValidationProblem.InvalidNexusTypeForNamedConcept:
+            value = cast(Any, value)
+            logger.error(
+                f"The type ('{args[0] if args else '<unknown>'}') of the given concept '{path}' "
+                f"conflicts with another existing concept of the same name, which is of type '{value.type}'."
+            )
 
     def collect_and_log(
         self,
diff --git a/src/pynxtools/dataconverter/validation.py b/src/pynxtools/dataconverter/validation.py
index 1fdefd3c0..bfaf37e1e 100644
--- a/src/pynxtools/dataconverter/validation.py
+++ b/src/pynxtools/dataconverter/validation.py
@@ -17,6 +17,7 @@
 # limitations under the License.
 #
 import re
+import copy
 from collections import defaultdict
 from functools import reduce
 from operator import getitem
@@ -145,7 +146,12 @@ def split_class_and_name_of(name: str) -> Tuple[Optional[str], str]:
     ), f"{name_match.group(2)}{'' if prefix is None else prefix}"
 
 
-def best_namefit_of(name: str, nodes: Iterable[NexusNode]) -> Optional[NexusNode]:
+def best_namefit_of(
+    name: str,
+    nodes: Iterable[NexusNode],
+    expected_types: List[str],
+    check_types: bool = False,
+) -> Optional[NexusNode]:
     """
     Get the best namefit of `name` in `keys`.
 
@@ -166,6 +172,19 @@ def best_namefit_of(name: str, nodes: Iterable[NexusNode]) -> Optional[NexusNode
     for node in nodes:
         if not node.variadic:
             if instance_name == node.name:
+                if node.type not in expected_types and check_types:
+                    expected_types_str = " or ".join(expected_types)
+                    collector.collect_and_log(
+                        name,
+                        ValidationProblem.InvalidNexusTypeForNamedConcept,
+                        node,
+                        expected_types_str,
+                    )
+                    raise TypeError(
+                        f"The type ('{expected_types_str if expected_types else '<unknown>'}') "
+                        f"of the given concept {name} conflicts with another existing concept {node.name} (which is of "
+                        f"type '{node.type}')."
+                    )
                 if concept_name and concept_name != node.name:
                     inherited_names = [
                         name
@@ -209,7 +228,7 @@ def best_namefit_of(name: str, nodes: Iterable[NexusNode]) -> Optional[NexusNode
 
 def validate_dict_against(
     appdef: str, mapping: MutableMapping[str, Any], ignore_undocumented: bool = False
-) -> Tuple[bool, List]:
+) -> bool:
     """
     Validates a mapping against the NeXus tree for application definition `appdef`.
 
@@ -226,7 +245,6 @@ def validate_dict_against(
 
     Returns:
         bool: True if the mapping is valid according to `appdef`, False otherwise.
-        List: list of keys in mapping that correspond to attributes of non-existing fields
     """
 
     def get_variations_of(node: NexusNode, keys: Mapping[str, Any]) -> List[str]:
@@ -398,6 +416,7 @@ def check_nxdata():
             for x in keys
             if x not in [signal, *axes, *indices, *errors, *aux_signals]
         }
+        remaining_keys = _follow_link(remaining_keys, prev_path)
         recurse_tree(
             node,
             remaining_keys,
@@ -437,24 +456,34 @@ def handle_group(node: NexusGroup, keys: Mapping[str, Any], prev_path: str):
             return
 
         for variant in variants:
+            variant_path = f"{prev_path}/{variant}"
             if variant in [node.name for node in node.parent_of]:
                 # Don't process if this is actually a sub-variant of this group
                 continue
             nx_class, _ = split_class_and_name_of(variant)
             if not isinstance(keys[variant], Mapping):
+                # Groups should have subelements
                 if nx_class is not None:
                     collector.collect_and_log(
-                        f"{prev_path}/{variant}",
+                        variant_path,
                         ValidationProblem.ExpectedGroup,
                         None,
                     )
+                    # TODO: decide if we want to remove such keys
+                    # collector.collect_and_log(
+                    #     variant_path,
+                    #     ValidationProblem.KeyToBeRemoved,
+                    #     node.type,
+                    # )
+                    # keys_to_remove.append(not_visited_key)
                 continue
             if node.nx_class == "NXdata":
-                handle_nxdata(node, keys[variant], prev_path=f"{prev_path}/{variant}")
+                handle_nxdata(node, keys[variant], prev_path=variant_path)
             if node.nx_class == "NXcollection":
                 return
             else:
-                recurse_tree(node, keys[variant], prev_path=f"{prev_path}/{variant}")
+                variant_keys = _follow_link(keys[variant], variant_path)
+                recurse_tree(node, variant_keys, prev_path=variant_path)
 
     def remove_from_not_visited(path: str) -> str:
         if path in not_visited:
@@ -462,28 +491,64 @@ def remove_from_not_visited(path: str) -> str:
         return path
 
     def _follow_link(
-        keys: Optional[Mapping[str, Any]], prev_path: str
+        keys: Optional[Mapping[str, Any]], prev_path: str, p=False
     ) -> Optional[Any]:
+        """
+        Resolves internal dictionary "links" by replacing any keys containing a
+        {"link": "/path/to/target"} structure with the actual referenced content.
+
+        Note that the keys are only replaced in copies of the incoming keys, NOT in
+        the gloval mapping. That is, links are resolved for checking, but we still write
+        links into the HDF5 file.
+
+        This function traverses the mapping and recursively resolves any keys that
+        contain a "link" to another path (relative to the global template).
+        If the link cannot be resolved, the issue is logged.
+
+        Args:
+            keys (Optional[Mapping[str, Any]]): The dictionary structure to process.
+                May be None or a non-dict value, in which case it's returned as-is.
+            prev_path (str): The path leading up to the current `keys` context, used
+                for logging and error reporting.
+
+        Returns:
+            Optional[Any]: A dictionary with resolved links, the original value if
+            `keys` is not a dict, or None if `keys` is None.
+        """
         if keys is None:
             return None
-        if len(keys) == 1 and "link" in keys:
-            current_keys = nested_keys
-            link_key = None
-            for path_elem in keys["link"][1:].split("/"):
+
+        if not isinstance(keys, dict):
+            return keys
+
+        resolved_keys = copy.deepcopy(keys)
+        for key, value in keys.copy().items():
+            if isinstance(value, dict) and "link" in value:
+                key_path = f"{prev_path}/{key}" if prev_path else key
+                current_keys = nested_keys
                 link_key = None
-                for dict_path_elem in current_keys:
-                    _, hdf_name = split_class_and_name_of(dict_path_elem)
-                    if hdf_name == path_elem:
-                        link_key = hdf_name
-                        break
+                for path_elem in value["link"][1:].split("/"):
+                    link_key = None
+                    for dict_path_elem in current_keys:
+                        _, hdf_name = split_class_and_name_of(dict_path_elem)
+                        if hdf_name == path_elem:
+                            link_key = hdf_name
+                            current_keys = current_keys[dict_path_elem]
+                            break
+
                 if link_key is None:
                     collector.collect_and_log(
-                        prev_path, ValidationProblem.BrokenLink, keys["link"]
+                        key_path, ValidationProblem.BrokenLink, value["link"]
+                    )
+                    collector.collect_and_log(
+                        key_path,
+                        ValidationProblem.KeyToBeRemoved,
+                        "key",
                     )
-                    return None
-                current_keys = current_keys[dict_path_elem]
-            return current_keys
-        return keys
+                    del resolved_keys[key]
+                else:
+                    resolved_keys[key] = current_keys
+        return resolved_keys
 
     def handle_field(node: NexusNode, keys: Mapping[str, Any], prev_path: str):
         full_path = remove_from_not_visited(f"{prev_path}/{node.name}")
@@ -497,6 +562,26 @@ def handle_field(node: NexusNode, keys: Mapping[str, Any], prev_path: str):
             return
 
         for variant in variants:
+            variant_path = f"{prev_path}/{variant}"
+
+            if isinstance(keys[variant], Mapping) and not all(
+                k.startswith("@") for k in keys[variant]
+            ):
+                # A field should not have a dict of keys that are _not_ all attributes,
+                # i.e. no sub-fields or sub-groups.
+                collector.collect_and_log(
+                    variant_path,
+                    ValidationProblem.ExpectedField,
+                    None,
+                )
+                # TODO: decide if we want to remove such keys
+                # collector.collect_and_log(
+                #     variant_path,
+                #     ValidationProblem.KeyToBeRemoved,
+                #     node.type,
+                # )
+                # keys_to_remove.append(variant_path)
+                continue
             if node.optionality == "required" and isinstance(keys[variant], Mapping):
                 # Check if all fields in the dict are actual attributes (startswith @)
                 all_attrs = True
@@ -506,44 +591,47 @@ def handle_field(node: NexusNode, keys: Mapping[str, Any], prev_path: str):
                         break
                 if all_attrs:
                     collector.collect_and_log(
-                        f"{prev_path}/{variant}", missing_type_err.get(node.type), None
+                        variant_path, missing_type_err.get(node.type), None
                     )
                     collector.collect_and_log(
-                        f"{prev_path}/{variant}",
+                        variant_path,
                         ValidationProblem.AttributeForNonExistingField,
                         None,
                     )
                     return
-            if variant not in keys or mapping.get(f"{prev_path}/{variant}") is None:
+            if variant not in keys or mapping.get(variant_path) is None:
                 continue
 
             # Check general validity
-            mapping[f"{prev_path}/{variant}"] = is_valid_data_field(
-                mapping[f"{prev_path}/{variant}"],
+            mapping[variant_path] = is_valid_data_field(
+                mapping[variant_path],
                 node.dtype,
                 node.items,
                 node.open_enum,
-                f"{prev_path}/{variant}",
+                variant_path,
             )
 
-            _ = check_reserved_suffix(f"{prev_path}/{variant}", mapping)
-            _ = check_reserved_prefix(f"{prev_path}/{variant}", mapping, "field")
+            _ = check_reserved_suffix(variant_path, mapping)
+            _ = check_reserved_prefix(variant_path, mapping, "field")
 
             # Check unit category
             if node.unit is not None:
                 remove_from_not_visited(f"{prev_path}/{variant}/@units")
                 if f"{variant}@units" not in keys:
                     collector.collect_and_log(
-                        f"{prev_path}/{variant}",
+                        variant_path,
                         ValidationProblem.MissingUnit,
                         node.unit,
                     )
                 # TODO: Check unit with pint
 
+            field_attributes = get_field_attributes(variant, keys)
+            field_attributes = _follow_link(field_attributes, variant_path)
+
             recurse_tree(
                 node,
-                get_field_attributes(variant, keys),
-                prev_path=f"{prev_path}/{variant}",
+                field_attributes,
+                prev_path=variant_path,
             )
 
     def handle_attribute(node: NexusNode, keys: Mapping[str, Any], prev_path: str):
@@ -559,18 +647,19 @@ def handle_attribute(node: NexusNode, keys: Mapping[str, Any], prev_path: str):
             return
 
         for variant in variants:
-            mapping[
+            variant_path = (
                 f"{prev_path}/{variant if variant.startswith('@') else f'@{variant}'}"
-            ] = is_valid_data_field(
+            )
+            mapping[variant_path] = is_valid_data_field(
                 mapping[
                     f"{prev_path}/{variant if variant.startswith('@') else f'@{variant}'}"
                 ],
                 node.dtype,
                 node.items,
                 node.open_enum,
-                f"{prev_path}/{variant if variant.startswith('@') else f'@{variant}'}",
+                variant_path,
             )
-            _ = check_reserved_prefix(f"{prev_path}/{variant}", mapping, "attribute")
+            _ = check_reserved_prefix(variant_path, mapping, "attribute")
 
     def handle_choice(node: NexusNode, keys: Mapping[str, Any], prev_path: str):
         global collector
@@ -600,13 +689,34 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
+                if "link" in str(mapping.get(key, "")):
+                    expected_types += ["group"]
+
+            node = best_namefit_of(name, children_to_check, expected_types, check_types)
 
             if node is None:
                 return None
@@ -618,7 +728,18 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             # This value is not really set. Skip checking its documentation.
             return True
 
-        node = add_best_matches_for(key, tree)
+        try:
+            node = add_best_matches_for(key, tree, check_types=True)
+        except TypeError:
+            node = None
+            nx_type = "attribute" if key.split("/")[-1].startswith("@") else "field"
+
+            collector.collect_and_log(
+                key,
+                ValidationProblem.KeyToBeRemoved,
+                nx_type,
+            )
+            keys_to_remove.append(key)
 
         if node is None:
             key_path = key.replace("@", "")
@@ -640,7 +761,49 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             return True
 
         if isinstance(mapping[key], dict) and "link" in mapping[key]:
-            # TODO: Follow link and check consistency with current field
+            resolved_link = _follow_link({key: mapping[key]}, "")
+
+            is_mapping = isinstance(resolved_link[key], Mapping)
+
+            if node.type == "group" and not is_mapping:
+                # Groups must have subelements
+                collector.collect_and_log(
+                    key,
+                    ValidationProblem.ExpectedGroup,
+                    None,
+                )
+                # TODO: decide if we want to remove such keys
+                # collector.collect_and_log(
+                #     key,
+                #     ValidationProblem.KeyToBeRemoved,
+                #     "group",
+                # )
+                # keys_to_remove.append(key)
+                return False
+
+            elif node.type == "field":
+                # A field should not have a dict of keys that are _not_ all attributes,
+                # i.e. no sub-fields or sub-groups.
+                if is_mapping and not all(
+                    k.startswith("@") for k in resolved_link[key]
+                ):
+                    collector.collect_and_log(
+                        key,
+                        ValidationProblem.ExpectedField,
+                        None,
+                    )
+                    # TODO: decide if we want to remove such keys
+                    # collector.collect_and_log(
+                    #     key,
+                    #     ValidationProblem.KeyToBeRemoved,
+                    #     "field",
+                    # )
+                    # keys_to_remove.append(key)
+                    # return False
+                resolved_link[key] = is_valid_data_field(
+                    resolved_link[key], node.dtype, node.items, node.open_enum, key
+                )
+
             return True
 
         if "@" not in key and node.type != "field":
@@ -677,7 +840,6 @@ def recurse_tree(
         for child in node.children:
             if ignore_names is not None and child.name in ignore_names:
                 continue
-            keys = _follow_link(keys, prev_path)
             if keys is None:
                 return
 
@@ -685,24 +847,19 @@ def recurse_tree(
 
     def check_attributes_of_nonexisting_field(
         node: NexusNode,
-    ) -> list:
+    ):
         """
             This method runs through the mapping dictionary and checks if there are any
             attributes assigned to the fields (not groups!) which are not explicitly
             present in the mapping.
             If there are any found, a warning is logged and the corresponding items are
-            added to the list returned by the method.
+            added to the list that stores all keys that shall be removed.
 
         Args:
             node (NexusNode): the tree generated from application definition.
 
-        Returns:
-            list: list of keys in mapping that correspond to attributes of
-            non-existing fields
         """
 
-        keys_to_remove = []
-
         for key in mapping:
             last_index = key.rfind("/")
             if key[last_index + 1] == "@" and key[last_index + 1 :] != "@units":
@@ -741,18 +898,18 @@ def check_attributes_of_nonexisting_field(
                         type_of_parent_from_tree == "group"
                         or type_of_parent_from_tree is None
                     ):
-                        keys_to_remove.append(key)
                         collector.collect_and_log(
                             key[0:last_index],
                             ValidationProblem.AttributeForNonExistingField,
-                            None,
+                            "attribute",
                         )
                         collector.collect_and_log(
                             key,
                             ValidationProblem.KeyToBeRemoved,
-                            None,
+                            "attribute",
                         )
-        return keys_to_remove
+                        keys_to_remove.append(key)
+                        remove_from_not_visited(key)
 
     def check_type_with_tree(
         node: NexusNode,
@@ -1012,13 +1169,16 @@ def check_reserved_prefix(
         "choice": handle_choice,
     }
 
+    keys_to_remove = []
+
     tree = generate_tree_from(appdef)
     collector.clear()
     nested_keys = build_nested_dict_from(mapping)
     not_visited = list(mapping)
+    keys = _follow_link(nested_keys, "")
     recurse_tree(tree, nested_keys)
 
-    keys_to_remove = check_attributes_of_nonexisting_field(tree)
+    check_attributes_of_nonexisting_field(tree)
 
     for not_visited_key in not_visited:
         # TODO: remove again if "@target"/"@reference" is sorted out by NIAC
@@ -1037,7 +1197,7 @@ def check_reserved_prefix(
                 collector.collect_and_log(
                     not_visited_key,
                     ValidationProblem.KeyToBeRemoved,
-                    None,
+                    "attribute",
                 )
                 keys_to_remove.append(not_visited_key)
             else:
@@ -1084,7 +1244,7 @@ def check_reserved_prefix(
                     collector.collect_and_log(
                         not_visited_key,
                         ValidationProblem.KeyToBeRemoved,
-                        None,
+                        "attribute",
                     )
                     keys_to_remove.append(not_visited_key)
                     continue
@@ -1102,7 +1262,7 @@ def check_reserved_prefix(
         if is_documented(not_visited_key, tree):
             continue
 
-        if not ignore_undocumented:
+        if not ignore_undocumented and not_visited_key not in keys_to_remove:
             collector.collect_and_log(
                 not_visited_key, ValidationProblem.MissingDocumentation, None
             )
@@ -1110,7 +1270,11 @@ def check_reserved_prefix(
     # clear lru_cache
     NexusNode.search_add_child_for.cache_clear()
 
-    return (not collector.has_validation_problems(), keys_to_remove)
+    # remove keys that are incorrect
+    for key in set(keys_to_remove):
+        del mapping[key]
+
+    return not collector.has_validation_problems()
 
 
 def populate_full_tree(node: NexusNode, max_depth: Optional[int] = 5, depth: int = 0):
@@ -1144,4 +1308,4 @@ def populate_full_tree(node: NexusNode, max_depth: Optional[int] = 5, depth: int
 def validate_data_dict(
     _: MutableMapping[str, Any], read_data: MutableMapping[str, Any], root: ET._Element
 ) -> bool:
-    return validate_dict_against(root.attrib["name"], read_data)[0]
+    return validate_dict_against(root.attrib["name"], read_data)
diff --git a/src/pynxtools/dataconverter/writer.py b/src/pynxtools/dataconverter/writer.py
index f2a289503..a20183612 100644
--- a/src/pynxtools/dataconverter/writer.py
+++ b/src/pynxtools/dataconverter/writer.py
@@ -250,7 +250,12 @@ def ensure_and_get_parent_node(self, path: str, undocumented_paths) -> h5py.Grou
             attrs = self.__nxdl_to_attrs(parent_path)
 
             if attrs is not None:
-                grp.attrs["NX_class"] = attrs["type"]
+                if nx_class := attrs.get("type"):
+                    grp.attrs["NX_class"] = nx_class
+                else:
+                    logger.error(
+                        f"No attribute 'NX_class' could be written for {parent_path}."
+                    )
             return grp
         return self.output_nexus[parent_path_hdf5]
 
diff --git a/tests/dataconverter/test_readers.py b/tests/dataconverter/test_readers.py
index 11ee48d06..91dc9344c 100644
--- a/tests/dataconverter/test_readers.py
+++ b/tests/dataconverter/test_readers.py
@@ -117,6 +117,6 @@ def test_has_correct_read_func(reader, caplog):
                 with caplog.at_level(logging.WARNING):
                     validate_dict_against(
                         supported_nxdl, read_data, ignore_undocumented=True
-                    )[0]
+                    )
 
                 print(caplog.text)
diff --git a/tests/dataconverter/test_validation.py b/tests/dataconverter/test_validation.py
index e7f2faa53..4769e817b 100644
--- a/tests/dataconverter/test_validation.py
+++ b/tests/dataconverter/test_validation.py
@@ -415,7 +415,11 @@ def listify_template(data_dict: Template):
                 "/ENTRY[my_entry]/NXODD_name[nxodd_name]/int_value",
                 {"link": "/a-link"},
             ),
-            [],
+            [
+                "Broken link at /ENTRY[my_entry]/NXODD_name[nxodd_name]/int_value to /a-link.",
+                "The key /ENTRY[my_entry]/NXODD_name[nxodd_name]/int_value will not be written.",
+                "The data entry corresponding to /ENTRY[my_entry]/NXODD_name[nxodd_name]/int_value is required and hasn't been supplied by the reader.",
+            ],
             id="link-dict-instead-of-int",
         ),
         pytest.param(
@@ -785,6 +789,97 @@ def listify_template(data_dict: Template):
             [],
             id="concept-name-given-for-optional-attribute",
         ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration/identifier_1/some_field",
+                "123",
+            ),
+            [
+                "Expected a field at /ENTRY[my_entry]/identified_calibration/identifier_1 but found a group.",
+                "The type ('group') of the given concept 'identifier_1' conflicts with another "
+                "existing concept of the same name, which is of type 'field'.",
+                "The field /ENTRY[my_entry]/identified_calibration/identifier_1/some_field will not be written.",
+            ],
+            id="group-instead-of-named-field",
+        ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration",
+                "123",
+            ),
+            [
+                "The type ('field') of the given concept 'identified_calibration' conflicts with another "
+                "existing concept of the same name, which is of type 'group'.",
+                "The field /ENTRY[my_entry]/identified_calibration will not be written.",
+            ],
+            id="field-instead-of-named-group",
+        ),
+        pytest.param(
+            alter_dict(
+                alter_dict(
+                    remove_from_dict(
+                        TEMPLATE,
+                        "/ENTRY[my_entry]/required_group/description",
+                        "optional",
+                    ),
+                    "/ENTRY[my_entry]/required_group",
+                    {"link": "/my_entry/required_group2"},
+                ),
+                "/ENTRY[my_entry]/OPTIONAL_group[some_group]/required_field",
+                {"link": "/my_entry/specified_group/specified_field"},
+            ),
+            [],
+            id="appdef-links-with-matching-nexus-types",
+        ),
+        pytest.param(
+            alter_dict(
+                alter_dict(
+                    TEMPLATE,
+                    "/ENTRY[my_entry]/USER[my_user]",
+                    {"link": "/my_entry/my_group/required_field"},
+                ),
+                "/ENTRY[my_entry]/OPTIONAL_group[some_group]/required_field",
+                {"link": "/my_entry/specified_group"},
+            ),
+            [
+                "Expected a field at /ENTRY[my_entry]/OPTIONAL_group[some_group]/required_field but found a group.",
+                "Expected a group at /ENTRY[my_entry]/USER[my_user] but found a field or attribute.",
+                "Field /ENTRY[my_entry]/USER[my_user] written without documentation.",
+            ],
+            id="appdef-links-with-wrong-nexus-types",
+        ),
+        pytest.param(
+            alter_dict(
+                alter_dict(
+                    TEMPLATE,
+                    "/ENTRY[my_entry]/SAMPLE[my_sample]",
+                    {"link": "/my_entry/my_group"},
+                ),
+                "/ENTRY[my_entry]/SAMPLE[my_sample]/name",
+                {"link": "/my_entry/nxodd_name/char_value"},
+            ),
+            [],
+            id="baseclass-links-with-matching-nexus-types",
+        ),
+        pytest.param(
+            alter_dict(
+                alter_dict(
+                    TEMPLATE,
+                    "/ENTRY[my_entry]/SAMPLE[my_sample]",
+                    {"link": "/my_entry/my_group/required_field"},
+                ),
+                "/ENTRY[my_entry]/SAMPLE[my_sample]/name",
+                {"link": "/my_entry/my_group"},
+            ),
+            [
+                "Expected a group at /ENTRY[my_entry]/SAMPLE[my_sample] but found a field or attribute.",
+                "Field /ENTRY[my_entry]/SAMPLE[my_sample] written without documentation.",
+                "Expected a field at /ENTRY[my_entry]/SAMPLE[my_sample]/name but found a group.",
+            ],
+            id="baseclass-links-with-wrong-nexus-types",
+        ),
         pytest.param(
             alter_dict(
                 remove_from_dict(
@@ -1245,7 +1340,7 @@ def listify_template(data_dict: Template):
             [
                 "Reserved suffix '_weights' was used in /ENTRY[my_entry]/OPTIONAL_group[my_group]/FIELDNAME_weights[some_random_field_weights], but there is no associated field some_random_field.",
             ],
-            id="reserved-suffix-from-base-class",
+            id="reserved-suffix-from-baseclass",
         ),
         pytest.param(
             alter_dict(
@@ -1291,7 +1386,7 @@ def format_error_message(msg: str) -> str:
 
     if not error_messages:
         with caplog.at_level(logging.WARNING):
-            assert validate_dict_against("NXtest", data_dict)[0]
+            assert validate_dict_against("NXtest", data_dict)
         assert caplog.text == ""
     else:
         if request.node.callspec.id in (
@@ -1299,15 +1394,13 @@ def format_error_message(msg: str) -> str:
             "baseclass-field-with-illegal-unit",
             "open-enum-with-new-item",
             "baseclass-open-enum-with-new-item",
-            "variadic-nxcollection",
-            "nonvariadic-nxcollection",
         ):
             with caplog.at_level(logging.INFO):
-                assert validate_dict_against("NXtest", data_dict)[0]
+                assert validate_dict_against("NXtest", data_dict)
                 assert error_messages[0] in caplog.text
         else:
             with caplog.at_level(logging.WARNING):
-                assert not validate_dict_against("NXtest", data_dict)[0]
+                assert not validate_dict_against("NXtest", data_dict)
             assert len(caplog.records) == len(error_messages)
             for expected_message, rec in zip(error_messages, caplog.records):
                 assert expected_message == format_error_message(rec.message)


Reviews:
user: login: lukaspie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T16:02:19Z
user: login: lukaspie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T16:11:28Z
user: login: lukaspie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T12:56:21Z
user: login: rettigl
body: Mostly lgtm and seems to do what it should, apart from a few comments
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-20T19:25:56Z
user: login: lukaspie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T08:42:05Z
user: login: lukaspie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T09:07:44Z
user: login: rettigl
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T09:24:17Z
user: login: lukaspie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T09:56:48Z
user: login: rettigl
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T10:02:06Z

Review comments:
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: This is the problematic line for links. We could change it to `else ["field, "group"]`, but then we cannot find the case where a field is given, but a group is defined in the NeXus definitions.
created_at: 2025-05-14T16:02:19Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: I suspect this won't be possible just based on the structure of the key itself, we need additional context (like that there is the str "link" in the value that belongs to this key).
created_at: 2025-05-14T16:11:28Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: Added a workaround that checks if  the str "link" in the value that belongs to this key.
created_at: 2025-05-15T12:56:21Z
author_association: COLLABORATOR
diff_hunk: @@ -169,6 +170,12 @@ def _log(self, path: str, log_type: ValidationProblem, value: Optional[Any], *ar
             if value != "<unknown>":
                 log_text += f" It is only valid in the context of {value}."
             logger.error(log_text)
+        elif log_type == ValidationProblem.InvalidNexusTypeForNamedConcept:
+            value = cast(Any, value)
+            logger.error(
+                f"The type ('{args[0] if args else '<unknown>'}') of the given concept '{path}' "
+                f"conflicts with another existing concept of the same name, which is of type '{value.type}')."
path: src/pynxtools/dataconverter/helpers.py
user: login: rettigl
body: ```suggestion
                f"conflicts with another existing concept of the same name, which is of type '{value.type}'."
```
created_at: 2025-05-20T18:46:04Z
author_association: COLLABORATOR
diff_hunk: @@ -176,6 +194,7 @@ def best_namefit_of(name: str, nodes: Iterable[NexusNode]) -> Optional[NexusNode
                         or (type_attr := elem.attrib.get("type"))
                         and len(type_attr) > 2
                     ]
+
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: remove?
created_at: 2025-05-20T18:50:28Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: Link targets can be fields or groups, but not attributes, I think. Generally, the mathching of link target type to node type should probably be checked separately.
created_at: 2025-05-20T18:59:13Z
author_association: COLLABORATOR
diff_hunk: @@ -785,6 +785,32 @@ def listify_template(data_dict: Template):
             [],
             id="concept-name-given-for-optional-attribute",
         ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration/identifier_1/some_field",
+                "123",
+            ),
+            [
+                "The type ('group') of the given concept 'identifier_1' conflicts with another "
+                "existing concept of the same name, which is of type 'field').",
path: tests/dataconverter/test_validation.py
user: login: rettigl
body: ```suggestion
                "existing concept of the same name, which is of type 'field'.",
```
created_at: 2025-05-20T19:03:56Z
author_association: COLLABORATOR
diff_hunk: @@ -618,7 +658,17 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             # This value is not really set. Skip checking its documentation.
             return True
 
-        node = add_best_matches_for(key, tree)
+        try:
+            node = add_best_matches_for(key, tree, check_types=True)
+        except TypeError:
+            node = None
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: ```suggestion
             type=node.type
            node = None
```
created_at: 2025-05-20T19:06:34Z
author_association: COLLABORATOR
diff_hunk: @@ -618,7 +658,17 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             # This value is not really set. Skip checking its documentation.
             return True
 
-        node = add_best_matches_for(key, tree)
+        try:
+            node = add_best_matches_for(key, tree, check_types=True)
+        except TypeError:
+            node = None
+            keys_to_remove.append(key)
+
+            collector.collect_and_log(
+                key,
+                ValidationProblem.KeyToBeRemoved,
+                None,
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: ```suggestion
                type,
```
created_at: 2025-05-20T19:06:44Z
author_association: COLLABORATOR
diff_hunk: @@ -618,7 +658,17 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             # This value is not really set. Skip checking its documentation.
             return True
 
-        node = add_best_matches_for(key, tree)
+        try:
+            node = add_best_matches_for(key, tree, check_types=True)
+        except TypeError:
+            node = None
+            keys_to_remove.append(key)
+
+            collector.collect_and_log(
+                key,
+                ValidationProblem.KeyToBeRemoved,
+                None,
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: + Adopt warning message to include type
created_at: 2025-05-20T19:07:00Z
author_association: COLLABORATOR
diff_hunk: @@ -785,6 +785,32 @@ def listify_template(data_dict: Template):
             [],
             id="concept-name-given-for-optional-attribute",
         ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration/identifier_1/some_field",
+                "123",
+            ),
+            [
+                "The type ('group') of the given concept 'identifier_1' conflicts with another "
+                "existing concept of the same name, which is of type 'field').",
+                "The attribute /ENTRY[my_entry]/identified_calibration/identifier_1/some_field will not be written.",
+            ],
+            id="group-instead-of-named-field",
+        ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration",
+                "123",
+            ),
+            [
+                "The type ('field') of the given concept 'identified_calibration' conflicts with another "
+                "existing concept of the same name, which is of type 'group').",
path: tests/dataconverter/test_validation.py
user: login: rettigl
body: ```suggestion
                "existing concept of the same name, which is of type 'group'.",
```
created_at: 2025-05-20T19:07:36Z
author_association: COLLABORATOR
diff_hunk: @@ -785,6 +785,32 @@ def listify_template(data_dict: Template):
             [],
             id="concept-name-given-for-optional-attribute",
         ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration/identifier_1/some_field",
+                "123",
+            ),
+            [
+                "The type ('group') of the given concept 'identifier_1' conflicts with another "
+                "existing concept of the same name, which is of type 'field').",
+                "The attribute /ENTRY[my_entry]/identified_calibration/identifier_1/some_field will not be written.",
+            ],
+            id="group-instead-of-named-field",
+        ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration",
+                "123",
+            ),
+            [
+                "The type ('field') of the given concept 'identified_calibration' conflicts with another "
+                "existing concept of the same name, which is of type 'group').",
+                "The attribute /ENTRY[my_entry]/identified_calibration will not be written.",
path: tests/dataconverter/test_validation.py
user: login: rettigl
body: ```suggestion
                "The field /ENTRY[my_entry]/identified_calibration will not be written.",
```
created_at: 2025-05-20T19:07:50Z
author_association: COLLABORATOR
diff_hunk: @@ -785,6 +785,32 @@ def listify_template(data_dict: Template):
             [],
             id="concept-name-given-for-optional-attribute",
         ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration/identifier_1/some_field",
+                "123",
+            ),
+            [
+                "The type ('group') of the given concept 'identifier_1' conflicts with another "
+                "existing concept of the same name, which is of type 'field').",
+                "The attribute /ENTRY[my_entry]/identified_calibration/identifier_1/some_field will not be written.",
path: tests/dataconverter/test_validation.py
user: login: rettigl
body: ```suggestion
                "The field /ENTRY[my_entry]/identified_calibration/identifier_1/some_field will not be written.",
```
created_at: 2025-05-20T19:08:47Z
author_association: COLLABORATOR
diff_hunk: @@ -785,6 +785,32 @@ def listify_template(data_dict: Template):
             [],
             id="concept-name-given-for-optional-attribute",
         ),
+        pytest.param(
+            alter_dict(
+                TEMPLATE,
+                "/ENTRY[my_entry]/identified_calibration/identifier_1/some_field",
+                "123",
+            ),
+            [
+                "The type ('group') of the given concept 'identifier_1' conflicts with another "
+                "existing concept of the same name, which is of type 'field').",
+                "The attribute /ENTRY[my_entry]/identified_calibration/identifier_1/some_field will not be written.",
path: tests/dataconverter/test_validation.py
user: login: rettigl
body: This still does not reflect that we are also removing the group in the end.
created_at: 2025-05-20T19:09:11Z
author_association: COLLABORATOR
diff_hunk: @@ -1299,8 +1325,6 @@ def format_error_message(msg: str) -> str:
             "baseclass-field-with-illegal-unit",
             "open-enum-with-new-item",
             "baseclass-open-enum-with-new-item",
-            "variadic-nxcollection",
-            "nonvariadic-nxcollection",
path: tests/dataconverter/test_validation.py
user: login: rettigl
body: Why did you remove these here?
created_at: 2025-05-20T19:12:33Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: Yes, that's correct. Here we are adding "group" to the ["field"] list if there is a link present.

> the mathching of link target type to node type should probably be checked separately.

Turns out this is surprisingly complicated with the current logic. I will try to do it, but may have to change some other things.
created_at: 2025-05-21T08:42:05Z
author_association: COLLABORATOR
diff_hunk: @@ -1299,8 +1325,6 @@ def format_error_message(msg: str) -> str:
             "baseclass-field-with-illegal-unit",
             "open-enum-with-new-item",
             "baseclass-open-enum-with-new-item",
-            "variadic-nxcollection",
-            "nonvariadic-nxcollection",
path: tests/dataconverter/test_validation.py
user: login: lukaspie
body: They were never needed here, they are covered by the else logic below (i.e., all error messages match).
created_at: 2025-05-21T08:43:21Z
author_association: COLLABORATOR
diff_hunk: @@ -618,7 +658,17 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             # This value is not really set. Skip checking its documentation.
             return True
 
-        node = add_best_matches_for(key, tree)
+        try:
+            node = add_best_matches_for(key, tree, check_types=True)
+        except TypeError:
+            node = None
+            keys_to_remove.append(key)
+
+            collector.collect_and_log(
+                key,
+                ValidationProblem.KeyToBeRemoved,
+                None,
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: changed
created_at: 2025-05-21T08:45:44Z
author_association: COLLABORATOR
diff_hunk: @@ -618,7 +658,17 @@ def is_documented(key: str, tree: NexusNode) -> bool:
             # This value is not really set. Skip checking its documentation.
             return True
 
-        node = add_best_matches_for(key, tree)
+        try:
+            node = add_best_matches_for(key, tree, check_types=True)
+        except TypeError:
+            node = None
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: This doesn't quite work, node is None here. Implemented a slightly different logic.
created_at: 2025-05-21T09:06:50Z
author_association: COLLABORATOR
diff_hunk: @@ -176,6 +194,7 @@ def best_namefit_of(name: str, nodes: Iterable[NexusNode]) -> Optional[NexusNode
                         or (type_attr := elem.attrib.get("type"))
                         and len(type_attr) > 2
                     ]
+
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: done
created_at: 2025-05-21T09:07:24Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: But the list can also be ["attribute"] if we add a link for an attribute key, no? Not that this would make sense anyways...
Maybe we don't tread this at the moment, I am not sure how relevant it is, and how often that case might actually come up.
created_at: 2025-05-21T09:24:17Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: lukaspie
body: Not sure if we can even link attributes, considering one should probably write a `@target` attribute in HDF5. I suggest we don't touch this for now until NIAC has cleared up the discussion about links and targets.
created_at: 2025-05-21T09:56:48Z
author_association: COLLABORATOR
diff_hunk: @@ -588,13 +607,30 @@ def handle_unknown_type(node: NexusNode, keys: Mapping[str, Any], prev_path: str
         # TODO: Raise error or log the issue?
         pass
 
-    def add_best_matches_for(key: str, node: NexusNode) -> Optional[NexusNode]:
-        for name in key[1:].replace("@", "").split("/"):
+    def add_best_matches_for(
+        key: str, node: NexusNode, check_types: bool = False
+    ) -> Optional[NexusNode]:
+        key_components = key[1:].split("/")
+        is_last_attr = key_components[-1].startswith("@")
+        if is_last_attr:
+            key_components[-1] = key_components[-1].replace("@", "")
+
+        key_len = len(key_components)
+
+        expected_types = []
+        for ind, name in enumerate(key_components):
+            index = ind + 1
             children_to_check = [
                 node.search_add_child_for(child)
                 for child in node.get_all_direct_children_names()
             ]
-            node = best_namefit_of(name, children_to_check)
+            if index < key_len - 1:
+                expected_types = ["group"]
+            elif index == key_len - 1:
+                expected_types = ["group"] if not is_last_attr else ["group", "field"]
+            elif index == key_len:
+                expected_types = ["attribute"] if is_last_attr else ["field"]
path: src/pynxtools/dataconverter/validation.py
user: login: rettigl
body: Yes, exactly my point. I still don't understand the link mechanism in Nexus, and how this differs from plain h5 links
created_at: 2025-05-21T10:02:06Z
author_association: COLLABORATOR

PR comments:
user: login: lukaspie
created_at: 2025-05-14T15:49:37Z
body: One problem that we have here (but also in #636) is if there is a link in the template. Then the key is e.g. `/ENTRY[entry]/SAMPLE[sample]/temperature_env/temperature_sensor`, which means that the expected type based on the current logic is a field. But this actually links a whole group.
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-19T15:26:22Z
body: If I add 
```
  "/ENTRY/entry_identifier": {
    "identifier":"@attrs:metadata/loader/scan_path"
  },
```
I get:
```
ERROR: The type ('group') of the given concept 'entry_identifier' conflicts with another existing concept of the same name, which is of type 'field').
WARNING: The attribute /ENTRY[entry]/entry_identifier/identifier will not be written.
```
The validation does not pick up that I try to write a group with a name that is defined as field.
If I add a concept, I get:
```
ERROR: The type ('group') of the given concept 'COLLECTION[entry_identifier]' conflicts with another existing concept of the same name, which is of type 'field').
WARNING: The attribute /ENTRY[entry]/COLLECTION[entry_identifier]/identifier will not be written.
WARNING: Given field name 'COLLECTION' conflicts with the non-variadic name 'entry_identifier (opt)'
```
If I have the conflicting field also in the dict, I get:
```
ERROR: The type ('group') of the given concept 'COLLECTION[entry_identifier]' conflicts with another existing concept of the same name, which is of type 'field').
WARNING: The attribute /ENTRY[entry]/COLLECTION[entry_identifier]/identifier will not be written.
WARNING: Given field name 'COLLECTION' conflicts with the non-variadic name 'entry_identifier (opt)'
```
But the conversion works.
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-19T15:28:52Z
body: Comparing the behavior of #636 and this PR, I would prefer the functionality here. But the reported warnings/consistency can still be improved (i.e. what exactly is being removed, element types etc.).
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-19T15:34:54Z
body: Link checking does not seem to work properly yet. If I add a link
```
    "RESOLUTION[angular_resolution]": {
      "resolution": "@link:/entry/instrument/electronanalyzer/angular_resolution",
      "resolution/@units": "deg",
      "physical_quantity": "angle",
      "type": "estimated"
    },
```
I don't get a warning, even though I linked a group to a defined field name.
If I add the same structure manually
```
    "RESOLUTION[angular_resolution]": {
      "resolution": {
        "RESOLUTION[angular_resolution]": {
          "resolution": 1.0
        }
      },
      "resolution/@units": "deg",
      "physical_quantity": "angle",
      "type": "estimated"
    },
```
, i get:
```
ERROR: The type ('group') of the given concept 'resolution' conflicts with another existing concept of the same name, which is of type 'field').
WARNING: The attribute /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution/RESOLUTION[angular_resolution]/resolution will not be written.
WARNING: Unit /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution/@units in dataset without its field /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution.
WARNING: The attribute /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution/@units will not be written.
```

author_association: COLLABORATOR
user: login: lukaspie
created_at: 2025-05-19T15:51:33Z
body: > Link checking does not seem to work properly yet. If I add a link
> 
> ```
>     "RESOLUTION[angular_resolution]": {
>       "resolution": "@link:/entry/instrument/electronanalyzer/angular_resolution",
>       "resolution/@units": "deg",
>       "physical_quantity": "angle",
>       "type": "estimated"
>     },
> ```
> 
> I don't get a warning, even though I linked a group to a defined field name. If I add the same structure manually
> 
> ```
>     "RESOLUTION[angular_resolution]": {
>       "resolution": {
>         "RESOLUTION[angular_resolution]": {
>           "resolution": 1.0
>         }
>       },
>       "resolution/@units": "deg",
>       "physical_quantity": "angle",
>       "type": "estimated"
>     },
> ```
> 
> , i get:
> 
> ```
> ERROR: The type ('group') of the given concept 'resolution' conflicts with another existing concept of the same name, which is of type 'field').
> WARNING: The attribute /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution/RESOLUTION[angular_resolution]/resolution will not be written.
> WARNING: Unit /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution/@units in dataset without its field /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution.
> WARNING: The attribute /ENTRY[entry]/INSTRUMENT[instrument]/RESOLUTION[angular_resolution]/resolution/@units will not be written.
> ```

Yeah,  this is expected behavior as we don't really follow links currently.

The only thing that I implemented w.r.t. links in this PR is that it picks up that if there is a link, the type you expect to write can be either group or field. Like, if we have `ENTRY[entry]/SAMPLE[sample]/temperature_env/temperature_sensor`, from the key structure we expect a field for temperature_sensor. But if we have "link" in the value, we also allow defining groups like this (and don't throw the newly implemented error). 

To actually follow the link and apply all the same checks as for regular fields is something I would implement in a separate PR. 
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-19T20:20:01Z
body: > The only thing that I implemented w.r.t. links in this PR is that it picks up that if there is a link, the type you expect to write can be either group or field.

I don't understand that. If we want to have a semantically valid file, we should not allow to link a group to a name which is defined as a field (as in my example).
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-20T19:33:58Z
body: Not sure why the tests fail, though
author_association: COLLABORATOR
user: login: lukaspie
created_at: 2025-05-21T08:56:32Z
body: > Not sure why the tests fail, though

I think at least for pynxtools-mpes/igor this is related to #641 and https://github.com/FAIRmat-NFDI/pynxtools-mpes/pull/52.
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-21T09:41:28Z
body: > > Not sure why the tests fail, though
> 
> I think at least for pynxtools-mpes/igor this is related to #641 and [FAIRmat-NFDI/pynxtools-mpes#52](https://github.com/FAIRmat-NFDI/pynxtools-mpes/pull/52).

Indeed this seems to work with merged #641: https://github.com/FAIRmat-NFDI/pynxtools-igor/pull/8
I still don't understand why this only occurs now in this PR.
author_association: COLLABORATOR
user: login: RubelMozumder
created_at: 2025-05-21T12:00:40Z
body: Test from `pynxtools-spm` is failling because of the field and group having the same  name conflict in [NXsample](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/base_classes/NXsample.nxdl.xml#L248) and NXspm is actively using `NXsample_component` group to collect data for `substrate` and `experiment_sample`.
author_association: COLLABORATOR
user: login: lukaspie
created_at: 2025-05-21T15:22:13Z
body: > Test from `pynxtools-spm` is failling because of the field and group having the same name conflict in [NXsample](https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/base_classes/NXsample.nxdl.xml?rgh-link-date=2025-05-21T12%3A00%3A40Z#L248) and NXspm is actively using `NXsample_component` group to collect data for `substrate` and `experiment_sample`.

Yes, that's true. I think if you want to use a `NXsample_component` group, you cannot name it `sample_component` as this is blocked for use in fields. Can you rename it?
author_association: COLLABORATOR
user: login: lukaspie
created_at: 2025-05-22T22:26:27Z
body: @RubelMozumder we now see a bunch of errors and warnings for pynxtools-spm and pynxtools-xrd because we levelled up some of the previous warnings to errors, see [`helpers.py`](https://github.com/FAIRmat-NFDI/pynxtools/pull/638/files#diff-eb8ab2853571e96a900abf9a24b9b7fd9132d48d8974ea6a682dbf05e1669b07). IMO, these should be errors, e.g. if a required field is missing. So I suggest we merge here and then you fix spm and xrd separately. 
author_association: COLLABORATOR
user: login: rettigl
created_at: 2025-05-23T07:06:22Z
body: The logic for attributes with missing fields in the nexus tree still seems to be broken to some extend. If I add (in NXelectron_detector):
```
        "raw_data": {
          "energy/@type": "kinetic"
        }
```
I get:
```
ERROR: The data entry corresponding to /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/raw is required and hasn't been supplied by the reader.
WARNING: Missing attribute: "/ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/@signal"
WARNING: There were attributes set for the field /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/energy, but the field does not exist.
WARNING: There were attributes set for the field /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/energy, but the field does not exist.
WARNING: The attribute /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/energy/@type will not be written.
```
The missing attribute error comes twice, which is not expected.
author_association: COLLABORATOR
user: login: lukaspie
created_at: 2025-05-23T12:10:49Z
body: > The logic for attributes with missing fields in the nexus tree still seems to be broken to some extend. If I add (in NXelectron_detector):
> 
> ```
>         "raw_data": {
>           "energy/@type": "kinetic"
>         }
> ```
> 
> I get:
> 
> ```
> ERROR: The data entry corresponding to /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/raw is required and hasn't been supplied by the reader.
> WARNING: Missing attribute: "/ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/@signal"
> WARNING: There were attributes set for the field /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/energy, but the field does not exist.
> WARNING: There were attributes set for the field /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/energy, but the field does not exist.
> WARNING: The attribute /ENTRY[entry]/INSTRUMENT[instrument]/ELECTRONANALYZER[electronanalyzer]/ELECTRON_DETECTOR[detector]/raw_data/energy/@type will not be written.
> ```
> 
> The missing attribute error comes twice, which is not expected.

Yeah, this occured now because we only remove all collected keys in the end now. For this example, it was special because it is in NXdata and since we do not have signal/axes defined here, we never proceeded to the attributes for axisnames defined in the appdef. So we never removed it from the non-visited list. I have just implemented it such that in the `check_attributes_of_nonexisting_field` function, we also remove such keys from the non-visited list, so we do not need to check them again. Should fix your problem.
author_association: COLLABORATOR