Repository: dandi/dandi-cli
Pull request #1514: Overhauling validation results to get them closer to cover different types of validators
Author: login: yarikoptic
Labels: name: minor
description: Increment the minor version when merged
name: released
Created: 2024-10-18T18:11:43Z | Merged: 2025-04-04T12:56:52Z

Ref:
- https://github.com/con/validation/issues/1

and result of our chat with @candleindark while also reviewing results of linkml validation over dandisets:
- https://github.com/dandi/dandisets-linkml-status/blob/temp-with-error-locations/dandi-reports/summary.md

Notes/TODOs:
- BREAKING CHANGE: 
   - we replaced `ValidationOrigin.bids_version` to more generic `Origin.standard` + `Origin.standard_version`
   - We renamed `ValidationOrigin.name` to `Origin.validator` and `ValidationOrigin.version` to `Origin.validator_version`
   - `Severity.HINT`, `Severity.WARNING`, and `Severity.ERROR` now have value of `20`, `30`, and `40` respectively instead of `1`, `2`, and `3`.
   - Introduce `Origin.type` to indicate the type of the origin of a validation result.
   - We made the `ValidationResult` model into a Pydantic model so that it is readily to serialization to JSON.
- [x] going through the code base and harmonizing construction of those ValidationResults
- [ ] ~~redo #1176 in here~~

Please see individual commit messages for more details of the change.

**Note**:
This PR needs the changes in #1601 to pass the tests. Rebase/merge with master once #1601 is merged into master.


Diff:
diff --git a/dandi/cli/tests/test_cmd_validate.py b/dandi/cli/tests/test_cmd_validate.py
index 66e0cbb0c..d108cf8b4 100644
--- a/dandi/cli/tests/test_cmd_validate.py
+++ b/dandi/cli/tests/test_cmd_validate.py
@@ -6,7 +6,14 @@
 
 from ..cmd_validate import _process_issues, validate
 from ...tests.fixtures import BIDS_ERROR_TESTDATA_SELECTION
-from ...validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ...validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    ValidationResult,
+    Validator,
+)
 
 
 @pytest.mark.parametrize("dataset", BIDS_ERROR_TESTDATA_SELECTION)
@@ -68,13 +75,16 @@ def test_validate_nwb_path_grouping(organized_nwb_dir4: Path) -> None:
 
 
 def test_process_issues(capsys):
+    origin_validation_nwbinspector = Origin(
+        type=OriginType.VALIDATION,
+        validator=Validator.nwbinspector,
+        validator_version="",
+    )
+
     issues = [
         ValidationResult(
             id="NWBI.check_data_orientation",
-            origin=ValidationOrigin(
-                name="nwbinspector",
-                version="",
-            ),
+            origin=origin_validation_nwbinspector,
             scope=Scope.FILE,
             message="Data may be in the wrong orientation.",
             path=Path("dir0/sub-mouse004/sub-mouse004.nwb"),
@@ -82,10 +92,7 @@ def test_process_issues(capsys):
         ),
         ValidationResult(
             id="NWBI.check_data_orientation",
-            origin=ValidationOrigin(
-                name="nwbinspector",
-                version="",
-            ),
+            origin=origin_validation_nwbinspector,
             scope=Scope.FILE,
             message="Data may be in the wrong orientation.",
             path=Path("dir1/sub-mouse001/sub-mouse001.nwb"),
@@ -93,10 +100,7 @@ def test_process_issues(capsys):
         ),
         ValidationResult(
             id="NWBI.check_missing_unit",
-            origin=ValidationOrigin(
-                name="nwbinspector",
-                version="",
-            ),
+            origin=origin_validation_nwbinspector,
             scope=Scope.FILE,
             message="Missing text for attribute 'unit'.",
             path=Path("dir1/sub-mouse001/sub-mouse001.nwb"),
diff --git a/dandi/files/bases.py b/dandi/files/bases.py
index 8db802656..e7d382a83 100644
--- a/dandi/files/bases.py
+++ b/dandi/files/bases.py
@@ -14,6 +14,7 @@
 from xml.etree.ElementTree import fromstring
 
 import dandischema
+from dandischema.consts import DANDI_SCHEMA_VERSION
 from dandischema.digests.dandietag import DandiETag
 from dandischema.models import BareAsset, CommonModel
 from dandischema.models import Dandiset as DandisetMeta
@@ -27,7 +28,17 @@
 from dandi.metadata.core import get_default_metadata
 from dandi.misctypes import DUMMY_DANDI_ETAG, Digest, LocalReadableFile, P
 from dandi.utils import post_upload_size_check, pre_upload_size_check, yaml_load
-from dandi.validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from dandi.validate_types import (
+    ORIGIN_INTERNAL_DANDI,
+    ORIGIN_VALIDATION_DANDI,
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = dandi.get_logger()
 
@@ -201,13 +212,11 @@ def get_validation_errors(
             )
             return [
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandi",
-                        version=dandi.__version__,
-                    ),
+                    origin=ORIGIN_INTERNAL_DANDI,
                     severity=Severity.ERROR,
                     id="dandi.SOFTWARE_ERROR",
                     scope=Scope.FILE,
+                    origin_result=e,
                     # metadata=metadata,
                     path=self.filepath,  # note that it is not relative .path
                     message=f"Failed to read metadata: {e}",
@@ -526,9 +535,10 @@ def get_validation_errors(
         else:
             # make sure that we have some basic metadata fields we require
             try:
-                origin = ValidationOrigin(
-                    name="nwbinspector",
-                    version=str(_get_nwb_inspector_version()),
+                origin_validation_nwbinspector = Origin(
+                    type=OriginType.VALIDATION,
+                    validator=Validator.nwbinspector,
+                    validator_version=str(_get_nwb_inspector_version()),
                 )
 
                 for error in inspect_nwbfile(
@@ -549,10 +559,11 @@ def get_validation_errors(
                         }
                     errors.append(
                         ValidationResult(
-                            origin=origin,
+                            origin=origin_validation_nwbinspector,
                             severity=severity,
                             id=f"NWBI.{error.check_function_name}",
                             scope=Scope.FILE,
+                            origin_result=error,
                             path=Path(error.file_path),
                             message=error.message,
                             # Assuming multiple sessions per multiple subjects,
@@ -694,10 +705,7 @@ def _check_required_fields(
             message = f"Required field {f!r} has no value"
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandischema",
-                        version=dandischema.__version__,  # type: ignore[attr-defined]
-                    ),
+                    origin=ORIGIN_VALIDATION_DANDI,
                     severity=Severity.ERROR,
                     id="dandischema.requred_field",
                     scope=Scope.FILE,
@@ -709,10 +717,7 @@ def _check_required_fields(
             message = f"Required field {f!r} has value {v!r}"
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandischema",
-                        version=dandischema.__version__,  # type: ignore[attr-defined]
-                    ),
+                    origin=ORIGIN_VALIDATION_DANDI,
                     severity=Severity.WARNING,
                     id="dandischema.placeholder_value",
                     scope=Scope.FILE,
@@ -793,13 +798,17 @@ def _pydantic_errors_to_validation_results(
             message = e.get("message", e.get("msg", None))
         out.append(
             ValidationResult(
-                origin=ValidationOrigin(
-                    name="dandischema",
-                    version=dandischema.__version__,  # type: ignore[attr-defined]
+                origin=Origin(
+                    type=OriginType.VALIDATION,
+                    validator=Validator.dandischema,
+                    validator_version=dandischema.__version__,  # type: ignore[attr-defined]
+                    standard=Standard.DANDI_SCHEMA,
+                    standard_version=DANDI_SCHEMA_VERSION,
                 ),
                 severity=Severity.ERROR,
                 id=id,
                 scope=scope,
+                origin_result=e,
                 path=file_path,
                 message=message,
                 # TODO? dataset_path=dataset_path,
diff --git a/dandi/files/bids.py b/dandi/files/bids.py
index 8541b5030..907a14c59 100644
--- a/dandi/files/bids.py
+++ b/dandi/files/bids.py
@@ -111,7 +111,7 @@ def _validate(self) -> None:
                         self._asset_metadata[bids_path] = prepare_metadata(
                             result.metadata
                         )
-                        self._bids_version = result.origin.bids_version
+                        self._bids_version = result.origin.standard_version
 
     def get_asset_errors(self, asset: BIDSAsset) -> list[ValidationResult]:
         """:meta private:"""
diff --git a/dandi/files/zarr.py b/dandi/files/zarr.py
index f2f79cbde..4a19e34bd 100644
--- a/dandi/files/zarr.py
+++ b/dandi/files/zarr.py
@@ -42,11 +42,47 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    ORIGIN_VALIDATION_DANDI_ZARR,
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_format_version(data: Any) -> str:
+    """
+    Get the Zarr storage specification version from a Zarr data object
+
+    Parameters
+    ----------
+    data : zarr.core.Array or zarr.hierarchy.Group
+        The Zarr data object from which to extract the storage specification version
+
+    Returns
+    -------
+    str
+        The Zarr storage specification version,
+        https://zarr-specs.readthedocs.io/en/latest/specs.html, used in the Zarr data
+        object
+    """
+    import zarr  # Delay heavy import
+
+    if isinstance(data, zarr.Group):
+        meta = json.loads(data.store.get(".zgroup"))
+    elif isinstance(data, zarr.Array):
+        meta = json.loads(data.store.get(".zarray"))
+    else:
+        raise TypeError("`data` must be a `zarr.core.Array` or `zarr.hierarchy.Group`")
+    return str(meta["zarr_format"])
+
+
 @dataclass
 class LocalZarrEntry(BasePath):
     """A file or directory within a `ZarrAsset`"""
@@ -209,34 +245,37 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin_internal_zarr: Origin = Origin(
+            type=OriginType.INTERNAL,
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+        )
+
         try:
             data = zarr.open(str(self.filepath))
-        except Exception:
+        except Exception as e:
             if devel_debug:
                 raise
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="zarr",
-                        version=zarr.version.version,
-                    ),
+                    origin=origin_internal_zarr,
                     severity=Severity.ERROR,
                     id="zarr.cannot_open",
                     scope=Scope.FILE,
+                    origin_result=e,
                     path=self.filepath,
                     message="Error opening file.",
                 )
             )
             data = None
+
         if isinstance(data, zarr.Group) and not data:
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="zarr",
-                        version=zarr.version.version,
-                    ),
+                    origin=ORIGIN_VALIDATION_DANDI_ZARR,
                     severity=Severity.ERROR,
-                    id="zarr.empty_group",
+                    id="dandi_zarr.empty_group",
                     scope=Scope.FILE,
                     path=self.filepath,
                     message="Zarr group is empty.",
@@ -248,12 +287,9 @@ def get_validation_errors(
                 raise ValueError(msg)
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="zarr",
-                        version=zarr.version.version,
-                    ),
+                    origin=ORIGIN_VALIDATION_DANDI_ZARR,
                     severity=Severity.ERROR,
-                    id="zarr.tree_depth_exceeded",
+                    id="dandi_zarr.tree_depth_exceeded",
                     scope=Scope.FILE,
                     path=self.filepath,
                     message=msg,
diff --git a/dandi/organize.py b/dandi/organize.py
index 0de237da7..cca5a0e31 100644
--- a/dandi/organize.py
+++ b/dandi/organize.py
@@ -18,7 +18,7 @@
 
 import ruamel.yaml
 
-from . import __version__, get_logger
+from . import get_logger
 from .consts import dandi_layout_fields
 from .dandiset import Dandiset
 from .exceptions import OrganizeImpossibleError
@@ -35,7 +35,12 @@
     move_file,
     yaml_load,
 )
-from .validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from .validate_types import (
+    ORIGIN_VALIDATION_DANDI_LAYOUT,
+    Scope,
+    Severity,
+    ValidationResult,
+)
 
 lgr = get_logger()
 
@@ -1119,7 +1124,7 @@ def validate_organized_path(
         errors.append(
             ValidationResult(
                 id="DANDI.NON_DANDI_FILENAME",
-                origin=ValidationOrigin(name="dandi", version=__version__),
+                origin=ORIGIN_VALIDATION_DANDI_LAYOUT,
                 severity=Severity.ERROR,
                 scope=Scope.FILE,
                 path=filepath,
@@ -1135,7 +1140,7 @@ def validate_organized_path(
         errors.append(
             ValidationResult(
                 id="DANDI.NON_DANDI_FOLDERNAME",
-                origin=ValidationOrigin(name="dandi", version=__version__),
+                origin=ORIGIN_VALIDATION_DANDI_LAYOUT,
                 severity=Severity.ERROR,
                 scope=Scope.FOLDER,
                 path=filepath,
@@ -1151,7 +1156,7 @@ def validate_organized_path(
             errors.append(
                 ValidationResult(
                     id="DANDI.METADATA_MISMATCH_SUBJECT",
-                    origin=ValidationOrigin(name="dandi", version=__version__),
+                    origin=ORIGIN_VALIDATION_DANDI_LAYOUT,
                     severity=Severity.ERROR,
                     scope=Scope.FILE,
                     path=filepath,
diff --git a/dandi/pynwb_utils.py b/dandi/pynwb_utils.py
index 8656ee779..ace9de6f4 100644
--- a/dandi/pynwb_utils.py
+++ b/dandi/pynwb_utils.py
@@ -29,7 +29,15 @@
 )
 from .misctypes import Readable
 from .utils import get_module_version, is_url
-from .validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from .validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
@@ -351,48 +359,6 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
     """
     path = str(path)  # Might come in as pathlib's PATH
     errors: list[ValidationResult] = []
-    try:
-        if Version(pynwb.__version__) >= Version("3.0.0"):
-            error_outputs = pynwb.validate(path=path)
-        elif Version(pynwb.__version__) >= Version(
-            "2.2.0"
-        ):  # Use cached namespace feature
-            # argument get_cached_namespaces is True by default
-            error_outputs, _ = pynwb.validate(paths=[path])
-        else:  # Fallback if an older version
-            with pynwb.NWBHDF5IO(path=path, mode="r", load_namespaces=True) as reader:
-                error_outputs = pynwb.validate(io=reader)
-        for error in error_outputs:
-            errors.append(
-                ValidationResult(
-                    origin=ValidationOrigin(
-                        name="pynwb",
-                        version=pynwb.__version__,
-                    ),
-                    severity=Severity.ERROR,
-                    id=f"pynwb.{error}",
-                    scope=Scope.FILE,
-                    path=Path(path),
-                    message=f"Failed to validate. {error.reason}",
-                    within_asset_paths={path: error.location},
-                )
-            )
-    except Exception as exc:
-        if devel_debug:
-            raise
-        errors.append(
-            ValidationResult(
-                origin=ValidationOrigin(
-                    name="pynwb",
-                    version=pynwb.__version__,
-                ),
-                severity=Severity.ERROR,
-                id="pynwb.GENERIC",
-                scope=Scope.FILE,
-                path=Path(path),
-                message=f"{exc}",
-            )
-        )
 
     # To overcome
     #   https://github.com/NeurodataWithoutBorders/pynwb/issues/1090
@@ -401,6 +367,7 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
         r"general/(experimenter|related_publications)\): "
         r"incorrect shape - expected an array of shape .\[None\]."
     )
+    version = None
     try:
         version = get_nwb_version(path, sanitize=False)
     except Exception:
@@ -414,13 +381,15 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
             for e in nwb_errors:
                 errors.append(
                     ValidationResult(
-                        origin=ValidationOrigin(
-                            name="pynwb",
-                            version=pynwb.__version__,
+                        origin=Origin(
+                            type=OriginType.VALIDATION,
+                            validator=Validator.dandi,
+                            validator_version=__version__,
                         ),
                         severity=Severity.ERROR,
                         id="pynwb.GENERIC",
                         scope=Scope.FILE,
+                        origin_result=e,
                         path=Path(path),
                         message=e,
                     )
@@ -444,6 +413,57 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
                         len(errors_) - len(errors),
                         path,
                     )
+
+    try:
+        if Version(pynwb.__version__) >= Version("3.0.0"):
+            error_outputs = pynwb.validate(path=path)
+        elif Version(pynwb.__version__) >= Version(
+            "2.2.0"
+        ):  # Use cached namespace feature
+            # argument get_cached_namespaces is True by default
+            error_outputs, _ = pynwb.validate(paths=[path])
+        else:  # Fallback if an older version
+            with pynwb.NWBHDF5IO(path=path, mode="r", load_namespaces=True) as reader:
+                error_outputs = pynwb.validate(io=reader)
+    except Exception as exc:
+        if devel_debug:
+            raise
+        errors.append(
+            ValidationResult(
+                origin=Origin(
+                    type=OriginType.INTERNAL,
+                    validator=Validator.pynwb,
+                    validator_version=pynwb.__version__,
+                ),
+                severity=Severity.ERROR,
+                id="pynwb.GENERIC",
+                scope=Scope.FILE,
+                origin_result=exc,
+                path=Path(path),
+                message=f"{exc}",
+            )
+        )
+    else:
+        for error in error_outputs:
+            errors.append(
+                ValidationResult(
+                    origin=Origin(
+                        type=OriginType.VALIDATION,
+                        validator=Validator.pynwb,
+                        validator_version=pynwb.__version__,
+                        standard=Standard.NWB,
+                        standard_version=version,
+                    ),
+                    severity=Severity.ERROR,
+                    id=f"pynwb.{error}",
+                    scope=Scope.FILE,
+                    origin_result=error,
+                    path=Path(path),
+                    message=f"Failed to validate. {error.reason}",
+                    within_asset_paths={path: error.location},
+                )
+            )
+
     return errors
 
 
diff --git a/dandi/tests/test_files.py b/dandi/tests/test_files.py
index 7a26603a0..6f9d6c053 100644
--- a/dandi/tests/test_files.py
+++ b/dandi/tests/test_files.py
@@ -530,7 +530,9 @@ def test_validate_deep_zarr(tmp_path: Path) -> None:
     zf = dandi_file(zarr_path)
     assert zf.get_validation_errors() == []
     mkpaths(zarr_path, "a/b/c/d/e/f/g/h.txt")
-    assert [e.id for e in zf.get_validation_errors()] == ["zarr.tree_depth_exceeded"]
+    assert [e.id for e in zf.get_validation_errors()] == [
+        "dandi_zarr.tree_depth_exceeded"
+    ]
 
 
 def test_validate_zarr_deep_via_excluded_dotfiles(tmp_path: Path) -> None:
diff --git a/dandi/tests/test_validate.py b/dandi/tests/test_validate.py
index a05cfbcf2..f852039b1 100644
--- a/dandi/tests/test_validate.py
+++ b/dandi/tests/test_validate.py
@@ -7,7 +7,15 @@
 from .. import __version__
 from ..consts import dandiset_metadata_file
 from ..validate import validate
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 
 def test_validate_nwb_error(simple3_nwb: Path) -> None:
@@ -31,7 +39,12 @@ def test_validate_empty(tmp_path: Path) -> None:
     assert list(validate(tmp_path)) == [
         ValidationResult(
             id="DANDI.NO_DANDISET_FOUND",
-            origin=ValidationOrigin(name="dandi", version=__version__),
+            origin=Origin(
+                type=OriginType.VALIDATION,
+                validator=Validator.dandi,
+                validator_version=__version__,
+                standard=Standard.DANDI_LAYOUT,
+            ),
             severity=Severity.ERROR,
             scope=Scope.DANDISET,
             path=tmp_path,
diff --git a/dandi/utils.py b/dandi/utils.py
index a5c3fbc16..06e461fbc 100644
--- a/dandi/utils.py
+++ b/dandi/utils.py
@@ -4,6 +4,7 @@
 from collections.abc import Iterable, Iterator
 import datetime
 from email.utils import parsedate_to_datetime
+from enum import Enum
 from functools import lru_cache
 from importlib.metadata import version as importlib_version
 import inspect
@@ -953,3 +954,16 @@ def get_retry_after(response: requests.Response) -> Optional[int]:
                 sleep_amount,
             )
     return sleep_amount
+
+
+class StrEnum(str, Enum):
+    """
+    A variation of Enum that is also a subclass of str, akin to IntEnum
+
+    Member instances of a subclass of this class assigned to `auto()` will have their
+    own name as their value.
+    """
+
+    @staticmethod
+    def _generate_next_value_(name, _start, _count, _last_values):
+        return name
diff --git a/dandi/validate.py b/dandi/validate.py
index 3746434f8..1be68229a 100644
--- a/dandi/validate.py
+++ b/dandi/validate.py
@@ -4,11 +4,19 @@
 import os
 from pathlib import Path
 
-from . import __version__
 from .consts import dandiset_metadata_file
 from .files import find_dandi_files
 from .utils import find_parent_directory_containing
-from .validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from .validate_types import (
+    ORIGIN_VALIDATION_DANDI_LAYOUT,
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 BIDS_TO_DANDI = {
     "subject": "subject_id",
@@ -46,10 +54,12 @@ def validate_bids(
 
     validation_result = validate_bids_(paths, exclude_files=["dandiset.yaml"])
     our_validation_result = []
-    origin = ValidationOrigin(
-        name="bidsschematools",
-        version=bidsschematools.__version__,
-        bids_version=validation_result["bids_version"],
+    origin = Origin(
+        type=OriginType.VALIDATION,
+        validator=Validator.bidsschematools,
+        validator_version=bidsschematools.__version__,
+        standard=Standard.BIDS,
+        standard_version=validation_result["bids_version"],
     )
 
     # Storing variable to not re-compute set paths for each individual file.
@@ -73,6 +83,7 @@ def validate_bids(
                 severity=Severity.ERROR,
                 id="BIDS.NON_BIDS_PATH_PLACEHOLDER",
                 scope=Scope.FILE,
+                origin_result=validation_result,
                 path=Path(path),
                 message="File does not match any pattern known to BIDS.",
                 dataset_path=dataset_path,
@@ -92,6 +103,7 @@ def validate_bids(
                     severity=Severity.ERROR,
                     id="BIDS.MANDATORY_FILE_MISSING_PLACEHOLDER",
                     scope=Scope.DATASET,
+                    origin_result=validation_result,
                     path_regex=pattern["regex"],
                     message="BIDS-required file is not present.",
                 )
@@ -115,6 +127,7 @@ def validate_bids(
                 origin=origin,
                 id="BIDS.MATCH",
                 scope=Scope.FILE,
+                origin_result=validation_result,
                 path=Path(file_path),
                 metadata=meta,
                 dataset_path=dataset_path,
@@ -149,7 +162,7 @@ def validate(
         if dandiset_path is None:
             yield ValidationResult(
                 id="DANDI.NO_DANDISET_FOUND",
-                origin=ValidationOrigin(name="dandi", version=__version__),
+                origin=ORIGIN_VALIDATION_DANDI_LAYOUT,
                 severity=Severity.ERROR,
                 scope=Scope.DANDISET,
                 path=Path(p),
diff --git a/dandi/validate_types.py b/dandi/validate_types.py
index c09740d39..37e9bcfa0 100644
--- a/dandi/validate_types.py
+++ b/dandi/validate_types.py
@@ -1,21 +1,126 @@
 from __future__ import annotations
 
-from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
 
+from pydantic import BaseModel, Field
 
-@dataclass
-class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+import dandi
+from dandi.utils import StrEnum
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bids_validator_deno = "bids-validator-deno"
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    hed_python_validator = "hed-python-validator"
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
+
+
+class OriginType(StrEnum):
+    """Types of validation result origins"""
+
+    INTERNAL = auto()
+    """
+    Validation result is originated from the validator but not necessarily relating
+    to validation of the data"""
+
+    VALIDATION = auto()
+    """Validation result is originated from validation of the data"""
+
+
+class Origin(BaseModel):
+    """
+    Origin of the validation result
+    """
+
+    type: OriginType
+
+    validator: Validator
+    """The validator conducting the validation"""
+
+    validator_version: str
+    """The version of the validator"""
+
+    standard: Standard | None = None
+    """Standard being validated against"""
+
+    standard_version: str | None = None
+    """Version of the standard"""
+
+
+# Some commonly used `Origin` instances
+ORIGIN_VALIDATION_DANDI = Origin(
+    type=OriginType.VALIDATION,
+    validator=Validator.dandi,
+    validator_version=dandi.__version__,
+)
+ORIGIN_VALIDATION_DANDI_LAYOUT = Origin(
+    type=OriginType.VALIDATION,
+    validator=Validator.dandi,
+    validator_version=dandi.__version__,
+    standard=Standard.DANDI_LAYOUT,
+)
+ORIGIN_VALIDATION_DANDI_ZARR = Origin(
+    type=OriginType.VALIDATION,
+    validator=Validator.dandi_zarr,
+    validator_version=dandi.__version__,
+)
+ORIGIN_INTERNAL_DANDI = Origin(
+    type=OriginType.INTERNAL,
+    validator=Validator.dandi,
+    validator_version=dandi.__version__,
+)
+
+
+class Severity(IntEnum):
+    """Severity levels for validation results"""
+
+    INFO = 10
+    """Not an indication of problem but information of status or confirmation"""
+
+    HINT = 20
+    """Data is valid but could be improved"""
+
+    WARNING = 30
+    """Data is not recognized as valid. Changes are needed to ensure validity"""
+
+    ERROR = 40
+    """Data is recognized as invalid"""
+
+    CRITICAL = 50
+    """
+    A serious invalidity in data.
+    E.g., an invalidity that prevents validation of other aspects of the data such
+    as when validating against the BIDS standard, the data is without a `BIDSVersion`
+    field or has an invalid `BIDSVersion` field.
+    """
 
 
 class Scope(Enum):
@@ -25,11 +130,20 @@ class Scope(Enum):
     DATASET = "dataset"
 
 
-@dataclass
-class ValidationResult:
+class ValidationResult(BaseModel):
     id: str
-    origin: ValidationOrigin
+
+    origin: Origin
+    """Origin of the validation result as validator and standard used in producing it"""
+
     scope: Scope
+
+    origin_result: Any | None = Field(None, exclude=True)
+    """
+    The representation of the validation result produced by the used validator,
+    `self.origin.validator`, unchanged
+    """
+
     severity: Severity | None = None
     # asset_paths, if not populated, assumes [.path], but could be smth like
     # {"path": "task-broken_bold.json",
diff --git a/setup.cfg b/setup.cfg
index da9f03d3f..27bba32eb 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -35,6 +35,8 @@ install_requires =
     click-didyoumean
     dandischema >= 0.11.0, < 0.12.0
     etelemetry >= 0.2.2
+    # For pydantic to be able to use type annotations like `X | None`
+    eval_type_backport;  python_version < "3.10"
     fasteners
     fscacher >= 0.3.0
     # 3.14.4: https://github.com/hdmf-dev/hdmf/issues/1186


Reviews:
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-24T04:25:32Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-24T04:30:34Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-24T04:31:56Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-24T04:44:43Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-24T05:16:02Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:18:03Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:20:09Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:24:23Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:31:23Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:46:43Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:48:17Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T18:49:41Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T00:43:09Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T01:49:07Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T04:34:08Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T18:55:38Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T19:15:09Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T19:18:05Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T05:37:04Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T05:55:24Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T05:56:59Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T06:03:38Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T06:04:30Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T23:34:57Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T00:45:13Z
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T02:55:05Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T04:52:16Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T19:48:38Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T19:50:55Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T19:54:42Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T19:58:46Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T20:03:19Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T20:07:33Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T20:28:30Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T20:34:04Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T16:52:53Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T17:44:20Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T18:15:28Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T19:06:23Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T21:10:38Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T06:48:39Z
user: login: jwodder
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T19:30:24Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:06:03Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:09:54Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:28:27Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T21:39:57Z
user: login: yarikoptic
body: a few spots where to define local `origin` to be reused.

But also I wonder if generally we should pre-define all those origins so they get reused across all the paths.  It smells a little as premature optimization to me though, so may be not worth it. But at least I think it is worth renaming them locally so they become a little more descriptive than just `origin` variables
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T15:11:26Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T18:49:31Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T19:47:15Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:20:13Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:20:54Z
user: login: candleindark
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T20:25:48Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T12:51:30Z
user: login: yarikoptic
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T12:54:25Z

Review comments:
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
path: dandi/validate_types.py
user: login: candleindark
body: @yarikoptic Are you talking about naming of the high level entities in this module, such as `Severity` and `ValidationOrigin`, or are you talking about the naming of the enum values within `Severity`?
created_at: 2025-02-24T04:25:32Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
path: dandi/validate_types.py
user: login: candleindark
body: I see three issues with this though they are small issues.

1. We may include all the severity levels in the outputs of all validators, However, for a particular validator, only a subset of these may be used.
2. The same severity level may mean two different things in different validators.
3. Different validator may also use different severity level name to mean the same level.
created_at: 2025-02-24T04:30:34Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
 
 
 class Scope(Enum):
     FILE = "file"
     FOLDER = "folder"
+    # Isaac: make it/add "dandiset-metadata" to signal specific relation to metadata
path: dandi/validate_types.py
user: login: candleindark
body: Need clarification for this.
created_at: 2025-02-24T04:31:56Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
path: dandi/validate_types.py
user: login: candleindark
body: @yarikoptic We can create an enum class to define the supported standards. Do you know what those those standards are ATM?
created_at: 2025-02-24T04:44:43Z
author_association: MEMBER
diff_hunk: @@ -209,17 +209,20 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: ValidationOrigin = ValidationOrigin(
+            name="zarr",
+            version=zarr.__version__,
path: dandi/files/zarr.py
user: login: candleindark
body: Is this line intentional? The original one was `version=zarr.version.version`.
created_at: 2025-02-24T05:16:02Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
path: dandi/validate_types.py
user: login: yarikoptic
body: 1. agree
2. better not
3. better not

But you are pointing to the ambiguous (not clearly defined across validators) semantic . Then ideally we should be able to establish "mapping" if there is difference in agreement.

I think we should just clear up here what **we** mean for each of the levels. E.g.

- `HINT`: data is correct but could be improved
- `INFO`: ... ???
- `WARNING`: when "SHOULD" or "SHOULD NOT" level of requirement is violated
- `ERROR`: when "MUST" or "MUST NOT" level of requirement is violated
- `CRITICAL`: when makes given data completely unusable

may be we might want

- `INTERNAL`: signals about the problem with validator itself. although @candleindark rightfuly notices that it is still an ERROR but about not data, but validator itself... so we might want to explicitly point to the object of the error (not data  but validator)
created_at: 2025-03-03T18:18:03Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
path: dandi/validate_types.py
user: login: yarikoptic
body: Sure. For starters:

- BIDS
- DANDI-LAYOUT  -- our "ad-hoc" naming schema. ATM we validate "explicitly" in dandi-cli code in validate_organized_path, and not as part of the dandi-schema
- DANDI-SCHEMA
- NWB
- HED
- OME-ZARR

Could be also at the level of the "file format" standard:

- JSON
- YAML
- TSV
created_at: 2025-03-03T18:20:09Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
 
 
 class Scope(Enum):
     FILE = "file"
     FOLDER = "folder"
+    # Isaac: make it/add "dandiset-metadata" to signal specific relation to metadata
path: dandi/validate_types.py
user: login: yarikoptic
body: may be forget about it for now -- seems to combine with `ValidationObject` below but we might need to rethink/define some additional levels 
created_at: 2025-03-03T18:24:23Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
 
 
 class Scope(Enum):
     FILE = "file"
     FOLDER = "folder"
+    # Isaac: make it/add "dandiset-metadata" to signal specific relation to metadata
path: dandi/validate_types.py
user: login: yarikoptic
body: as for the folder an example could be an unknown to BIDS folder name under `sub-.../ses-.../` folder.  I am not sure ATM if BIDS validator complains and how
created_at: 2025-03-03T18:31:23Z
author_association: MEMBER
diff_hunk: @@ -209,17 +209,20 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: ValidationOrigin = ValidationOrigin(
+            name="zarr",
+            version=zarr.__version__,
+            standard="zarr",
path: dandi/files/zarr.py
user: login: yarikoptic
body: here to add `standard_version` on what version of zarr it is .
created_at: 2025-03-03T18:46:43Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
 
 
 class Scope(Enum):
     FILE = "file"
     FOLDER = "folder"
+    # Isaac: make it/add "dandiset-metadata" to signal specific relation to metadata
     DANDISET = "dandiset"
     DATASET = "dataset"
 
 
+# new/unused, may be should be gone
path: dandi/validate_types.py
user: login: yarikoptic
body: may be drop this for now indeed
created_at: 2025-03-03T18:48:17Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
path: dandi/validate_types.py
user: login: yarikoptic
body: might want also to create Enum for validators since we also have a set of them.
created_at: 2025-03-03T18:49:41Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
path: dandi/validate_types.py
user: login: candleindark
body: I added the standard `ZARR` as well since you have already started using it.
created_at: 2025-03-18T00:43:09Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
path: dandi/validate_types.py
user: login: candleindark
body: Added `Standard` and `Validator` subclasses of `Enum`. 

Please mark conversation as resolved once confirmed in review.
created_at: 2025-03-18T01:49:07Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
path: dandi/validate_types.py
user: login: candleindark
body: Here are a few more suggestions.

1. Let's define `INFO` as not an indication of problem but information of status or confirmation. However, this `INFO` maybe different than the `INFO` from `LinkML`, but that's perfectly fine (we will have a proper mapping).
2. Using the definition of `INFO` in the last point. `INFO` should be the lowest level of severity amount those listed.
3. Mimic the definition of [logging levels](https://docs.python.org/3/library/logging.html#logging-levels) in Python, have the values of the levels be multiple of 10 instead of 1 to allow greater flexibility (to allow addition of levels in the future or allow custom levels).
4. The above definitions of `WARNING` and `ERROR` are too vague. Together with `CRITICAL`, we can draw inspiration from https://docs.python.org/3/library/logging.html#logging-levels in defining them.

I will provide an implementation of these suggestions the a subsequent commit.
created_at: 2025-03-18T04:34:08Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
path: dandi/validate_types.py
user: login: candleindark
body: Severity levels have now been rearranged and more clearly defined.
created_at: 2025-03-18T18:55:38Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,97 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
+
+from dandi.utils import StrEnum
+
+
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bidsschematools = auto()
path: dandi/validate_types.py
user: login: yarikoptic
body: now soon we would have `bids_validator_deno` or alike 

and we could provision also for `hed_python_validator`
created_at: 2025-03-18T19:15:09Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,97 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
+
+from dandi.utils import StrEnum
+
+
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: Standard | None = None  # Standard being validated against
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+class Severity(IntEnum):
+    """Severity levels for validation results"""
+
+    INFO = 10
+    """Not an indication of problem but information of status or confirmation"""
+
+    HINT = 20
+    """Data is correct but could be improved"""
+
+    WARNING = 30
+    """Data is not recognized as correct. Changes are needed to ensure correctness"""
+
+    ERROR = 40
+    """Data is recognized as incorrect"""
+
+    CRITICAL = 50
+    """
+    A serious incorrectness in data.
+    E.g., an incorrectness that prevents validation of other aspects of the data
path: dandi/validate_types.py
user: login: yarikoptic
body: may be an example could be an absent or misspecified `BIDSVersion` so becomes unclear against which bids version to validate, although typically we just validate against the latest.
created_at: 2025-03-18T19:18:05Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
 
 
 class Scope(Enum):
     FILE = "file"
     FOLDER = "folder"
+    # Isaac: make it/add "dandiset-metadata" to signal specific relation to metadata
path: dandi/validate_types.py
user: login: candleindark
body: I will remove `ValidationObject` for now as well. It is currently unused. It seems to have some overlapping with `Scope`. We can always bring it back when its intent and use is clear.


created_at: 2025-03-19T05:37:04Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
 
 
 class Scope(Enum):
     FILE = "file"
     FOLDER = "folder"
+    # Isaac: make it/add "dandiset-metadata" to signal specific relation to metadata
     DANDISET = "dandiset"
     DATASET = "dataset"
 
 
+# new/unused, may be should be gone
path: dandi/validate_types.py
user: login: candleindark
body: done
created_at: 2025-03-19T05:55:24Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,97 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
+
+from dandi.utils import StrEnum
+
+
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: Standard | None = None  # Standard being validated against
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+class Severity(IntEnum):
+    """Severity levels for validation results"""
+
+    INFO = 10
+    """Not an indication of problem but information of status or confirmation"""
+
+    HINT = 20
+    """Data is correct but could be improved"""
+
+    WARNING = 30
+    """Data is not recognized as correct. Changes are needed to ensure correctness"""
+
+    ERROR = 40
+    """Data is recognized as incorrect"""
+
+    CRITICAL = 50
+    """
+    A serious incorrectness in data.
+    E.g., an incorrectness that prevents validation of other aspects of the data
path: dandi/validate_types.py
user: login: candleindark
body: Example is added to comment to further demo the meaning of `CRITICAL`
created_at: 2025-03-19T05:56:58Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
path: dandi/validate_types.py
user: login: candleindark
body: Marking this as resolved since the new `Enum` subclasses have been reviewed in a meeting.
created_at: 2025-03-19T06:03:38Z
author_association: MEMBER
diff_hunk: @@ -1,35 +1,53 @@
 from __future__ import annotations
 
 from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum
 from pathlib import Path
+from typing import Any
 
 
 @dataclass
 class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+    name: str  # Validator name
+    version: str  # Validator version
+    standard: str | None = (
+        None  # Standard being validated against # TODO: Enum for the standards??
+    )
+    standard_version: str | None = None  # Version of the standard
 
 
-class Severity(Enum):
+# TODO: decide on the naming consistency -- either prepend all with Validation or not
+class Severity(IntEnum):
     HINT = 1
-    WARNING = 2
-    ERROR = 3
+    INFO = 2  # new/unused, available in linkml
+    WARNING = 3
+    ERROR = 4
+    CRITICAL = 5  # new/unused, linkml has FATAL
path: dandi/validate_types.py
user: login: candleindark
body: The new levels have been reviewed in a meeting. Marking resolved.
created_at: 2025-03-19T06:04:30Z
author_association: MEMBER
diff_hunk: @@ -209,17 +209,20 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: ValidationOrigin = ValidationOrigin(
+            name="zarr",
+            version=zarr.__version__,
path: dandi/files/zarr.py
user: login: candleindark
body: They are essentially the same, so it doesn't matter.
created_at: 2025-03-21T23:34:57Z
author_association: MEMBER
diff_hunk: @@ -365,13 +372,16 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
         for error in error_outputs:
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="pynwb",
-                        version=pynwb.__version__,
+                    origin=Origin(
+                        validator=Validator.pynwb,
+                        validator_version=pynwb.__version__,
+                        standard=Standard.NWB,
+                        # TODO: standard_version=...,
path: dandi/pynwb_utils.py
user: login: candleindark
body: What should be the `standard_version`? The NWB standard version coded in the pynwb package or the NWB standard version retrieved from the file in the next try-except statement? If the former, the following code seems to work to.

```python
import yaml
from importlib.resources import files
import pynwb

def get_pynwb_schema_version():
    """
    Return the 'version' (i.e., NWB schema version) declared in PyNWB's included 'nwb.namespace.yaml'.
    """
    # Locate the nwb.namespace.yaml file in the PyNWB install
    yaml_path = files(pynwb.__name__) / "nwb-schema" / "core" / "nwb.namespace.yaml"
    with yaml_path.open("r") as f:
        doc = yaml.safe_load(f)
    # The top-level 'version' in nwb.namespace.yaml typically reflects the NWB 2.x revision
    return doc["namespaces"][0]["version"]

print("PyNWB NWB schema version:", get_pynwb_schema_version())
```
created_at: 2025-03-24T00:45:13Z
author_association: MEMBER
diff_hunk: @@ -6,7 +6,14 @@
 
 from ..cmd_validate import _process_issues, validate
 from ...tests.fixtures import BIDS_ERROR_TESTDATA_SELECTION
-from ...validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ...validate_types import (
+    Origin,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
path: dandi/cli/tests/test_cmd_validate.py
user: login: github-advanced-security[bot]
body: ## Unused import

Import of 'Standard' is not used.

[Show more details](https://github.com/dandi/dandi-cli/security/code-scanning/217)
created_at: 2025-03-24T02:55:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -209,32 +216,44 @@
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: Origin = Origin(
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+            # TODO: standard_version=...,
path: dandi/files/zarr.py
user: login: candleindark
body: Per https://zarr.readthedocs.io/en/stable/user-guide/v3_migration.html#migration-guide, it seems that the Python `zarr` pacakge can support more than one Zarr format specification at the same time. What is the way to determine which Zarr format specification is being used in just opening a file? My bet is that possibly all support format specifications are used, so the `standard_version` field may be irrelevant and should be skipped in this case. Additionally, since the `ValidationResult` is build in case of any error, it may not be an error related to validation of data against a Zarr format specification at all.
created_at: 2025-03-24T04:48:13Z
author_association: MEMBER
diff_hunk: @@ -209,32 +236,43 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: Origin = Origin(
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+            # TODO: standard_version=...,
+        )
+
         try:
             data = zarr.open(str(self.filepath))
path: dandi/files/zarr.py
user: login: yarikoptic
body: does zarr library has some exception which would also provide information about the standard version?
created_at: 2025-03-24T19:48:38Z
author_association: MEMBER
diff_hunk: @@ -42,11 +42,38 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_storage_specification_version(data: Any) -> str:
path: dandi/files/zarr.py
user: login: yarikoptic
body: Let's shorten

```suggestion
def get_zarr_format_version(data: Any) -> str:
```
created_at: 2025-03-24T19:50:55Z
author_association: MEMBER
diff_hunk: @@ -365,13 +372,16 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
         for error in error_outputs:
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="pynwb",
-                        version=pynwb.__version__,
+                    origin=Origin(
+                        validator=Validator.pynwb,
+                        validator_version=pynwb.__version__,
+                        standard=Standard.NWB,
+                        # TODO: standard_version=...,
path: dandi/pynwb_utils.py
user: login: yarikoptic
body: just use `get_nwb_version` from below -- just move it up here so you would already know then what nwb version is in that file.
created_at: 2025-03-24T19:54:42Z
author_association: MEMBER
diff_hunk: @@ -365,13 +372,16 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
         for error in error_outputs:
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="pynwb",
-                        version=pynwb.__version__,
+                    origin=Origin(
+                        validator=Validator.pynwb,
+                        validator_version=pynwb.__version__,
+                        standard=Standard.NWB,
+                        # TODO: standard_version=...,
path: dandi/pynwb_utils.py
user: login: yarikoptic
body: a good point that internally pynwb is actually likely to do migration and then validation against latest. But that is the internal logic which we do not know about unless validation error record would have target version of nwb.   So 
- please check if error record contains a version, if not
   - file an issue requesting NWB or the particular relevant namespace (e.g. from extension) to be added
- if not known from error -- the version of nwb file is the best one we know of 
created_at: 2025-03-24T19:58:46Z
author_association: MEMBER
diff_hunk: @@ -209,32 +236,43 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: Origin = Origin(
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+            # TODO: standard_version=...,
+        )
+
         try:
             data = zarr.open(str(self.filepath))
-        except Exception:
+        except Exception as e:
             if devel_debug:
                 raise
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="zarr",
-                        version=zarr.version.version,
-                    ),
+                    origin=origin,
                     severity=Severity.ERROR,
                     id="zarr.cannot_open",
                     scope=Scope.FILE,
+                    origin_result=e,
path: dandi/files/zarr.py
user: login: yarikoptic
body: we might want to adopt later the [CapturedException](https://github.com/datalad/datalad/blob/HEAD/datalad/support/exceptions.py#L25) to avoid dragging too much of context around from the original location of that exception
created_at: 2025-03-24T20:03:18Z
author_association: MEMBER
diff_hunk: @@ -382,13 +392,15 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
             raise
         errors.append(
             ValidationResult(
-                origin=ValidationOrigin(
-                    name="pynwb",
-                    version=pynwb.__version__,
+                origin=Origin(
+                    validator=Validator.pynwb,
+                    validator_version=pynwb.__version__,
+                    standard=Standard.NWB,
path: dandi/pynwb_utils.py
user: login: yarikoptic
body: per discussion, we might not even be able to claim it has anything to do with NWB:
```suggestion
```
created_at: 2025-03-24T20:07:33Z
author_association: MEMBER
diff_hunk: @@ -1,21 +1,83 @@
 from __future__ import annotations
 
-from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
 
+from pydantic import BaseModel, Field
 
-@dataclass
-class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+from dandi.utils import StrEnum
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bids_validator_deno = "bids-validator-deno"
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    hed_python_validator = "hed-python-validator"
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
+
+
+class Origin(BaseModel):
+    validator: Validator
+    """The validator conducting the validation"""
+
+    validator_version: str
+    """The version of the validator"""
+
+    standard: Standard | None = None
+    """Standard being validated against"""
+
+    standard_version: str | None = None
+    """Version of the standard"""
path: dandi/validate_types.py
user: login: yarikoptic
body: may be here it could be `source` which could be "standard" or "internal" ATM, but then may be smth else too 
created_at: 2025-03-24T20:28:29Z
author_association: MEMBER
diff_hunk: @@ -35,6 +35,7 @@ install_requires =
     click-didyoumean
     dandischema >= 0.11.0, < 0.12.0
     etelemetry >= 0.2.2
+    eval_type_backport
path: setup.cfg
user: login: yarikoptic
body: ```suggestion
    # For pydantic to be able to use type annotations like `X | None`
    eval_type_backport;  python_version < "3.10"
```

created_at: 2025-03-24T20:34:04Z
author_association: MEMBER
diff_hunk: @@ -42,11 +42,38 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_storage_specification_version(data: Any) -> str:
path: dandi/files/zarr.py
user: login: candleindark
body: done
created_at: 2025-03-26T16:52:52Z
author_association: MEMBER
diff_hunk: @@ -1,21 +1,83 @@
 from __future__ import annotations
 
-from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
 
+from pydantic import BaseModel, Field
 
-@dataclass
-class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+from dandi.utils import StrEnum
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bids_validator_deno = "bids-validator-deno"
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    hed_python_validator = "hed-python-validator"
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
+
+
+class Origin(BaseModel):
+    validator: Validator
+    """The validator conducting the validation"""
+
+    validator_version: str
+    """The version of the validator"""
+
+    standard: Standard | None = None
+    """Standard being validated against"""
+
+    standard_version: str | None = None
+    """Version of the standard"""
path: dandi/validate_types.py
user: login: candleindark
body: Since this is a field in the `Origin` object. I think naming the field `source` is a bit confusing.

Since we already use an `Origin` object to represent the "source" of a validation result, and, at this point, we want to indicate rather the "source" is an "external" validator invoked by the `dandi-cli` or some "internal" code of the `dandi-cli` itself, I think we can name the field in the `Origin` object `type` and allow the value of the field to be `internal`, `external`, or `undetermined`. The `undetermined` value is needed because there are cases where source of the error is hard/impossible to determine.
created_at: 2025-03-26T17:44:20Z
author_association: MEMBER
diff_hunk: @@ -1,21 +1,83 @@
 from __future__ import annotations
 
-from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
 
+from pydantic import BaseModel, Field
 
-@dataclass
-class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+from dandi.utils import StrEnum
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bids_validator_deno = "bids-validator-deno"
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    hed_python_validator = "hed-python-validator"
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
+
+
+class Origin(BaseModel):
+    validator: Validator
+    """The validator conducting the validation"""
+
+    validator_version: str
+    """The version of the validator"""
+
+    standard: Standard | None = None
+    """Standard being validated against"""
+
+    standard_version: str | None = None
+    """Version of the standard"""
path: dandi/validate_types.py
user: login: candleindark
body: I have decided on types of the origin to be 

```py
class Type(StrEnum):
    """Types of validation result origins"""

    INTERNAL = auto()
    """Validation result is originated from the code of this project"""

    VALIDATOR = auto()
    """Validation result is originated from a validator"""

    UNDETERMINED = auto()
    """
    Validation result is originated from a source that is not determined to
    be another value in this enum class
    
    This is usually used when the origin of the validation result is unclear
    """
```

I used `VALIDATOR` instead of your suggestion of `STANDARD` because not all validator implements a standard (nwbinspector, for example).
created_at: 2025-03-26T18:15:28Z
author_association: MEMBER
diff_hunk: @@ -1,21 +1,107 @@
 from __future__ import annotations
 
-from dataclasses import dataclass
-from enum import Enum
+from enum import Enum, IntEnum, auto, unique
 from pathlib import Path
+from typing import Any
 
+from pydantic import BaseModel, Field
 
-@dataclass
-class ValidationOrigin:
-    name: str
-    version: str
-    bids_version: str | None = None
+from dandi.utils import StrEnum
 
 
-class Severity(Enum):
-    HINT = 1
-    WARNING = 2
-    ERROR = 3
+@unique
+class Standard(StrEnum):
+    """Standards to validate against"""
+
+    BIDS = auto()
+    DANDI_LAYOUT = "DANDI-LAYOUT"
+    DANDI_SCHEMA = "DANDI-SCHEMA"
+    HED = auto()
+    NWB = auto()
+    OME_ZARR = "OME-ZARR"
+    ZARR = auto()
+
+    # File formats (For denoting validation failures in file format level)
+    JSON = auto()
+    TSV = auto()
+    YAML = auto()
+
+
+@unique
+class Validator(StrEnum):
+    """Validators that are used to do validation"""
+
+    bids_validator_deno = "bids-validator-deno"
+    bidsschematools = auto()
+    dandi = auto()
+    dandi_zarr = "dandi.zarr"
+    dandischema = auto()
+    hed_python_validator = "hed-python-validator"
+    nwbinspector = auto()
+    pynwb = auto()
+    zarr = auto()
+
+
+class Type(StrEnum):
+    """Types of validation result origins"""
+
+    INTERNAL = auto()
+    """Validation result is originated from the code of this project"""
path: dandi/validate_types.py
user: login: yarikoptic
body: ```suggestion
    """Validation result is originated from the validator and not necessarily relating to the standard"""
```
created_at: 2025-03-26T19:06:23Z
author_association: MEMBER
diff_hunk: @@ -209,32 +236,43 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: Origin = Origin(
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+            # TODO: standard_version=...,
+        )
+
         try:
             data = zarr.open(str(self.filepath))
path: dandi/files/zarr.py
user: login: candleindark
body: No, I couldn't find any, https://github.com/zarr-developers/zarr-python/blob/v2.18.2/zarr/errors.py.
created_at: 2025-03-26T21:10:38Z
author_association: MEMBER
diff_hunk: @@ -365,13 +372,16 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
         for error in error_outputs:
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="pynwb",
-                        version=pynwb.__version__,
+                    origin=Origin(
+                        validator=Validator.pynwb,
+                        validator_version=pynwb.__version__,
+                        standard=Standard.NWB,
+                        # TODO: standard_version=...,
path: dandi/pynwb_utils.py
user: login: candleindark
body: Feature request, https://github.com/NeurodataWithoutBorders/pynwb/issues/2062, made.
created_at: 2025-03-27T06:48:39Z
author_association: MEMBER
diff_hunk: @@ -35,6 +35,8 @@ install_requires =
     click-didyoumean
     dandischema >= 0.11.0, < 0.12.0
     etelemetry >= 0.2.2
+    # For pydantic to be able to use type annotations like `X | None`
path: setup.cfg
user: login: jwodder
body: Is there a reason you can't just use `Union` and `Optional` instead?
created_at: 2025-03-31T19:20:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -25,11 +105,20 @@ class Scope(Enum):
     DATASET = "dataset"
 
 
-@dataclass
-class ValidationResult:
+class ValidationResult(BaseModel):
     id: str
-    origin: ValidationOrigin
+
+    origin: Origin
+    """Origin of the validation result as validator and standard used in producing it"""
path: dandi/validate_types.py
user: login: jwodder
body: ```suggestion
    #: Origin of the validation result as validator and standard used in producing it
    origin: Origin
```

And likewise for other attributes.
created_at: 2025-03-31T19:21:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,11 +42,39 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_format_version(data: Any) -> str:
path: dandi/files/zarr.py
user: login: jwodder
body: What strings does this return?  That should be documented.
created_at: 2025-03-31T19:27:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,11 +42,39 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_format_version(data: Any) -> str:
+    """
+    Get the Zarr storage specification version from a Zarr data object
+
+    Parameters
+    ----------
+    data : zarr.core.Array or zarr.hierarchy.Group
+        The Zarr data object from which to extract the storage specification version
+    """
+    import zarr  # Delay heavy import
+
+    if isinstance(data, zarr.Group):
+        meta = json.loads(data.store.get(".zgroup"))
+    elif isinstance(data, zarr.Array):
+        meta = json.loads(data.store.get(".zarray"))
+    else:
+        raise TypeError("`data` must be a `zarr.core.Array` or `zarr.hierarchy.Group`")
+    return str(meta["zarr_format"])
path: dandi/files/zarr.py
user: login: jwodder
body: What type is `meta["zarr_format"]` before you cast it to a string?  If it's already a string and the point of the cast is just to satisfy the type checker, you should instead do:

```python
zf = meta["zarr_formar"]
assert isinstance(zf, str)
return zf
```
created_at: 2025-03-31T19:29:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,6 +35,8 @@ install_requires =
     click-didyoumean
     dandischema >= 0.11.0, < 0.12.0
     etelemetry >= 0.2.2
+    # For pydantic to be able to use type annotations like `X | None`
path: setup.cfg
user: login: candleindark
body: No, but the code base already had many uses of `|` to express `Optional` and `Union` types. I only made the models in `validate_types.py` to Pydantic models, and since Pydantic uses the types at runtime, the `eval_type_backport` package is needed. To avoid the `eval_type_backport` dependency, some code unrelated to the purpose of this PR will need to change.
created_at: 2025-03-31T21:06:02Z
author_association: MEMBER
diff_hunk: @@ -25,11 +105,20 @@ class Scope(Enum):
     DATASET = "dataset"
 
 
-@dataclass
-class ValidationResult:
+class ValidationResult(BaseModel):
     id: str
-    origin: ValidationOrigin
+
+    origin: Origin
+    """Origin of the validation result as validator and standard used in producing it"""
path: dandi/validate_types.py
user: login: candleindark
body: Using a triple-quoted string in this context will gain IDE support and other documentation tool support.
created_at: 2025-03-31T21:09:54Z
author_association: MEMBER
diff_hunk: @@ -42,11 +42,39 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_format_version(data: Any) -> str:
+    """
+    Get the Zarr storage specification version from a Zarr data object
+
+    Parameters
+    ----------
+    data : zarr.core.Array or zarr.hierarchy.Group
+        The Zarr data object from which to extract the storage specification version
+    """
+    import zarr  # Delay heavy import
+
+    if isinstance(data, zarr.Group):
+        meta = json.loads(data.store.get(".zgroup"))
+    elif isinstance(data, zarr.Array):
+        meta = json.loads(data.store.get(".zarray"))
+    else:
+        raise TypeError("`data` must be a `zarr.core.Array` or `zarr.hierarchy.Group`")
+    return str(meta["zarr_format"])
path: dandi/files/zarr.py
user: login: candleindark
body: It is actually an `int`.
created_at: 2025-03-31T21:28:26Z
author_association: MEMBER
diff_hunk: @@ -42,11 +42,39 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_format_version(data: Any) -> str:
path: dandi/files/zarr.py
user: login: candleindark
body: It is the Zarr storage specification version used by the give Zarr data object. The possible versions are documented at https://zarr-specs.readthedocs.io/en/latest/specs.html.

I have added the documentation for the return value.

Thanks.
created_at: 2025-03-31T21:39:57Z
author_association: MEMBER
diff_hunk: @@ -1151,7 +1169,12 @@ def validate_organized_path(
             errors.append(
                 ValidationResult(
                     id="DANDI.METADATA_MISMATCH_SUBJECT",
-                    origin=ValidationOrigin(name="dandi", version=__version__),
+                    origin=Origin(
+                        type=OriginType.VALIDATION,
+                        validator=Validator.dandi,
+                        validator_version=__version__,
+                        standard=Standard.DANDI_LAYOUT,
+                    ),
path: dandi/organize.py
user: login: yarikoptic
body: 3 "uses" for the same `origin` . Similarly to other similar cases -- just define `origin` a tthe top of the function and reuse here?  would make code shorter and thus easier to read.

But then -- this and other functions will be invoked in a loop across paths. So I wonder if we should centrally define some of those origins, like here `origin_dandi_layout` and alike? this would save some cycles/bytes across files. WDYT?

- we should at least define / reuse
created_at: 2025-04-02T15:01:54Z
author_association: MEMBER
diff_hunk: @@ -209,32 +244,43 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: Origin = Origin(
+            type=OriginType.INTERNAL,
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+        )
+
         try:
             data = zarr.open(str(self.filepath))
-        except Exception:
+        except Exception as e:
             if devel_debug:
                 raise
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="zarr",
-                        version=zarr.version.version,
-                    ),
+                    origin=origin,
                     severity=Severity.ERROR,
                     id="zarr.cannot_open",
                     scope=Scope.FILE,
+                    origin_result=e,
                     path=self.filepath,
                     message="Error opening file.",
                 )
             )
             data = None
+
+        origin = Origin(
+            type=OriginType.VALIDATION,
+            validator=Validator.dandi_zarr,
+            validator_version=__version__,
+        )
path: dandi/files/zarr.py
user: login: yarikoptic
body: here we seems to overwrite prior origin which was like `origin_internal_zarr` and this one `origin_dandi_zarr` or alike.  So might better give this somewhat more descriptive name and avoid overwrite so we do not by mistake assign the wrong one?
created_at: 2025-04-02T15:07:10Z
author_association: MEMBER
diff_hunk: @@ -526,9 +537,10 @@ def get_validation_errors(
         else:
             # make sure that we have some basic metadata fields we require
             try:
-                origin = ValidationOrigin(
-                    name="nwbinspector",
-                    version=str(_get_nwb_inspector_version()),
+                origin = Origin(
path: dandi/files/bases.py
user: login: yarikoptic
body: ```suggestion
                origin_nwbinspector = Origin(
```
created_at: 2025-04-02T15:08:04Z
author_association: MEMBER
diff_hunk: @@ -201,13 +210,15 @@ def get_validation_errors(
             )
             return [
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandi",
-                        version=dandi.__version__,
+                    origin=Origin(
path: dandi/files/bases.py
user: login: yarikoptic
body: could be named/reused as origin_internal_dandi
created_at: 2025-04-02T15:08:29Z
author_association: MEMBER
diff_hunk: @@ -71,9 +78,10 @@ def test_process_issues(capsys):
     issues = [
         ValidationResult(
             id="NWBI.check_data_orientation",
-            origin=ValidationOrigin(
-                name="nwbinspector",
-                version="",
+            origin=Origin(
+                type=OriginType.VALIDATION,
+                validator=Validator.nwbinspector,
+                validator_version="",
path: dandi/cli/tests/test_cmd_validate.py
user: login: yarikoptic
body: why no version for nwbinspector?    it seems to be available

```
>>> nwbinspector.__version__
'0.6.2'
```

created_at: 2025-04-02T15:09:24Z
author_association: MEMBER
diff_hunk: @@ -694,9 +707,10 @@ def _check_required_fields(
             message = f"Required field {f!r} has no value"
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandischema",
-                        version=dandischema.__version__,  # type: ignore[attr-defined]
+                    origin=Origin(
+                        type=OriginType.VALIDATION,
+                        validator=Validator.dandi,
+                        validator_version=dandi.__version__,
path: dandi/files/bases.py
user: login: yarikoptic
body: could be `origin_validation_dandi`
created_at: 2025-04-02T15:09:59Z
author_association: MEMBER
diff_hunk: @@ -71,9 +78,10 @@ def test_process_issues(capsys):
     issues = [
         ValidationResult(
             id="NWBI.check_data_orientation",
-            origin=ValidationOrigin(
-                name="nwbinspector",
-                version="",
+            origin=Origin(
+                type=OriginType.VALIDATION,
+                validator=Validator.nwbinspector,
+                validator_version="",
path: dandi/cli/tests/test_cmd_validate.py
user: login: yarikoptic
body: and so -- this could  be `origin_validation_nwbinspector`
created_at: 2025-04-02T15:10:05Z
author_association: MEMBER
diff_hunk: @@ -71,9 +78,10 @@ def test_process_issues(capsys):
     issues = [
         ValidationResult(
             id="NWBI.check_data_orientation",
-            origin=ValidationOrigin(
-                name="nwbinspector",
-                version="",
+            origin=Origin(
+                type=OriginType.VALIDATION,
+                validator=Validator.nwbinspector,
+                validator_version="",
path: dandi/cli/tests/test_cmd_validate.py
user: login: yarikoptic
body: ah, that was in the test, so ok to keep as is I guess
created_at: 2025-04-02T18:49:31Z
author_association: MEMBER
diff_hunk: @@ -526,9 +537,10 @@ def get_validation_errors(
         else:
             # make sure that we have some basic metadata fields we require
             try:
-                origin = ValidationOrigin(
-                    name="nwbinspector",
-                    version=str(_get_nwb_inspector_version()),
+                origin = Origin(
path: dandi/files/bases.py
user: login: candleindark
body: Renamed to `origin_validation_nwbinspector` to be consistent with other names.
created_at: 2025-04-03T19:47:15Z
author_association: MEMBER
diff_hunk: @@ -694,9 +707,10 @@ def _check_required_fields(
             message = f"Required field {f!r} has no value"
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandischema",
-                        version=dandischema.__version__,  # type: ignore[attr-defined]
+                    origin=Origin(
+                        type=OriginType.VALIDATION,
+                        validator=Validator.dandi,
+                        validator_version=dandi.__version__,
path: dandi/files/bases.py
user: login: candleindark
body: Done with a constant
created_at: 2025-04-03T20:20:12Z
author_association: MEMBER
diff_hunk: @@ -201,13 +210,15 @@ def get_validation_errors(
             )
             return [
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="dandi",
-                        version=dandi.__version__,
+                    origin=Origin(
path: dandi/files/bases.py
user: login: candleindark
body: done with a defined constant
created_at: 2025-04-03T20:20:54Z
author_association: MEMBER
diff_hunk: @@ -209,32 +244,43 @@ def get_validation_errors(
         import zarr
 
         errors: list[ValidationResult] = []
+        origin: Origin = Origin(
+            type=OriginType.INTERNAL,
+            validator=Validator.zarr,
+            validator_version=zarr.__version__,
+            standard=Standard.ZARR,
+        )
+
         try:
             data = zarr.open(str(self.filepath))
-        except Exception:
+        except Exception as e:
             if devel_debug:
                 raise
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="zarr",
-                        version=zarr.version.version,
-                    ),
+                    origin=origin,
                     severity=Severity.ERROR,
                     id="zarr.cannot_open",
                     scope=Scope.FILE,
+                    origin_result=e,
                     path=self.filepath,
                     message="Error opening file.",
                 )
             )
             data = None
+
+        origin = Origin(
+            type=OriginType.VALIDATION,
+            validator=Validator.dandi_zarr,
+            validator_version=__version__,
+        )
path: dandi/files/zarr.py
user: login: candleindark
body: Rename the first `origin` to `origin_internal_zarr`. The second `origin` is eliminated entirely because I think the subsequent `ValidationResult` objects are labeled incorrectly. Please see cf401cdd8814b7b670e0423e2736add4ef4063db for details.
created_at: 2025-04-03T20:25:48Z
author_association: MEMBER
diff_hunk: @@ -365,13 +372,16 @@ def validate(path: str | Path, devel_debug: bool = False) -> list[ValidationResu
         for error in error_outputs:
             errors.append(
                 ValidationResult(
-                    origin=ValidationOrigin(
-                        name="pynwb",
-                        version=pynwb.__version__,
+                    origin=Origin(
+                        validator=Validator.pynwb,
+                        validator_version=pynwb.__version__,
+                        standard=Standard.NWB,
+                        # TODO: standard_version=...,
path: dandi/pynwb_utils.py
user: login: yarikoptic
body: followed up on that con/validation issue 1, thanks
created_at: 2025-04-04T12:51:30Z
author_association: MEMBER
diff_hunk: @@ -42,11 +42,47 @@
 )
 
 from .bases import LocalDirectoryAsset
-from ..validate_types import Scope, Severity, ValidationOrigin, ValidationResult
+from ..validate_types import (
+    ORIGIN_VALIDATION_DANDI_ZARR,
+    Origin,
+    OriginType,
+    Scope,
+    Severity,
+    Standard,
+    ValidationResult,
+    Validator,
+)
 
 lgr = get_logger()
 
 
+def get_zarr_format_version(data: Any) -> str:
+    """
+    Get the Zarr storage specification version from a Zarr data object
+
+    Parameters
+    ----------
+    data : zarr.core.Array or zarr.hierarchy.Group
+        The Zarr data object from which to extract the storage specification version
+
+    Returns
+    -------
+    str
+        The Zarr storage specification version,
+        https://zarr-specs.readthedocs.io/en/latest/specs.html, used in the Zarr data
+        object
+    """
+    import zarr  # Delay heavy import
path: dandi/files/zarr.py
user: login: yarikoptic
body: hm, apparently we have no single test on validation of zarrs? that's odd :-/  

filed dedicated
- https://github.com/dandi/dandi-cli/issues/1605

for that
created_at: 2025-04-04T12:54:25Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2024-10-18T18:14:36Z
body: ## [Codecov](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) Report
Attention: Patch coverage is `86.66667%` with `14 lines` in your changes missing coverage. Please review.
> Project coverage is 88.66%. Comparing base [(`b1815da`)](https://app.codecov.io/gh/dandi/dandi-cli/commit/b1815da44ef230b7aa3e595c6a303a1e7014c83c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) to head [(`cf401cd`)](https://app.codecov.io/gh/dandi/dandi-cli/commit/cf401cdd8814b7b670e0423e2736add4ef4063db?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi).
> Report is 110 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | Patch % | Lines |
|---|---|---|
| [dandi/files/zarr.py](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?src=pr&el=tree&filepath=dandi%2Ffiles%2Fzarr.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvZmlsZXMvemFyci5weQ==) | 27.27% | [8 Missing :warning: ](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) |
| [dandi/pynwb\_utils.py](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?src=pr&el=tree&filepath=dandi%2Fpynwb_utils.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#diff-ZGFuZGkvcHlud2JfdXRpbHMucHk=) | 60.00% | [6 Missing :warning: ](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #1514      +/-   ##
==========================================
+ Coverage   88.54%   88.66%   +0.11%     
==========================================
  Files          78       78              
  Lines       10872    10937      +65     
==========================================
+ Hits         9627     9697      +70     
+ Misses       1245     1240       -5     
```

| [Flag](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi) | `88.66% <86.66%> (+0.11%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/dandi/dandi-cli/pull/1514?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dandi).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: candleindark
created_at: 2025-02-24T05:10:10Z
body: In updating the use of `ValidationOrigin`, it is not clear to me what should be provided for the `standard` and `standard_version` attributes of many `ValidationOrigin` objects. For example, this is not clear in `ValidationOrigin` objects in `organized.py`.

https://github.com/dandi/dandi-cli/blob/45db66c34cd6ded26fd1d8c871a7957df73ad38a/dandi/organize.py#L1122

Do we have a dandi standard? If we do, it doesn't seem that we version this standard.

I think we should indeed define an [enum class](https://github.com/dandi/dandi-cli/pull/1514/files#r1967035125) for the supported standard, and also talk about how to retrieve the version of those standards.


author_association: MEMBER
user: login: candleindark
created_at: 2025-03-19T05:53:34Z
body: > In updating the use of `ValidationOrigin`, it is not clear to me what should be provided for the `standard` and `standard_version` attributes of many `ValidationOrigin` objects. For example, this is not clear in `ValidationOrigin` objects in `organized.py`.
> 
> https://github.com/dandi/dandi-cli/blob/45db66c34cd6ded26fd1d8c871a7957df73ad38a/dandi/organize.py#L1122
> 
> Do we have a dandi standard? If we do, it doesn't seem that we version this standard.
> 
> I think we should indeed define an [enum class](https://github.com/dandi/dandi-cli/pull/1514/files#r1967035125) for the supported standard, and also talk about how to retrieve the version of those standards.

We now, with the latest commit, have a subclass of  `Enum`, `Standard`, to represent a list of standards. Some standard is `informal`, e.g., `Standard.DANDI_LAYOUT` and have no versioning. In the case, of a standard with no version, we will just let the `standard_version` attribute of `Origin` to default to `None`.
author_association: MEMBER
user: login: candleindark
created_at: 2025-03-21T05:02:40Z
body: @yarikoptic I think a `ValidationResult` object should represent the result of how well a "data" instance is validated. It should inform area/aspect of invalidity.

https://github.com/dandi/dandi-cli/blob/386e50db53a8fc0236b666927402ec1bee3e7759/dandi/files/bases.py#L193-L217
The above snippet uses `ValidationResult` to represent an error other than a `ValidationError`. The captured error doesn't tell how valid a data instance is. It has nothing to do with the data being validated against but how our software is written. If that error is raised it is either a bug in the software, which should be propagated, or a user's mistake, which should be handled gracefully in another manner. The current arrangement is conflating error handling with the purpose of `ValidationResult`. Case in point, you can't provide the `standard` for the `origin` field in the above example.

Another misuse of `ValidationResult` is below. https://github.com/dandi/dandi-cli/blob/386e50db53a8fc0236b666927402ec1bee3e7759/dandi/pynwb_utils.py#L404-L427
The code snippet above also has other problems. For example, it suppresses errors. The problems should be fixed in an other dedicated PR though.


author_association: MEMBER
user: login: candleindark
created_at: 2025-03-27T00:08:33Z
body: FTR: The following look very problematic though not directly related to this PR. 

https://github.com/dandi/dandi-cli/blob/386e50db53a8fc0236b666927402ec1bee3e7759/dandi/pynwb_utils.py#L406-L408

I don't know its intent either.
author_association: MEMBER
user: login: candleindark
created_at: 2025-04-01T05:05:51Z
body: This PR needs the changes in https://github.com/dandi/dandi-cli/pull/1601 to pass the tests. It will be merged with master once https://github.com/dandi/dandi-cli/pull/1601 is merged into master.
author_association: MEMBER
user: login: candleindark
created_at: 2025-04-02T03:22:28Z
body: The Change from #1601 is merged. This PR is ready for a final review.
author_association: MEMBER
user: login: candleindark
created_at: 2025-04-03T23:56:47Z
body: This one is ready for a final review.
author_association: MEMBER
user: login: yarikoptic
created_at: 2025-04-04T12:39:44Z
body: > FTR: The following look very problematic though not directly related to this PR.
> 
> https://github.com/dandi/dandi-cli/blob/386e50db53a8fc0236b666927402ec1bee3e7759/dandi/pynwb_utils.py#L406-L408
> 
> I don't know its intent either.

yeah, better to be reanalyzed/improved. I think we burnt there a few times already 
author_association: MEMBER
user: login: yarikoptic
created_at: 2025-04-04T12:56:44Z
body: Thank you @candleindark , let's proceed with continuation in
- https://github.com/dandi/dandi-cli/issues/1605
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-04-23T18:55:16Z
body: <!-- GITHUB_RELEASE COMMENT: released -->
:rocket: PR was released in [`0.68.0`](https://github.com/dandi/dandi-cli/releases/tag/0.68.0) :rocket:
author_association: NONE