Repository: SINTEF/dlite (language: python)

Issue:
Validate multiple instances
# Description
Update `dlite-validate` to validate sources with multiple instances, without having to provide the `--id` option. 

This PR build on top of PR #985.

**Update**: Added new Python interface to the jstore implemented in C. Right now this increases the code base, but this change makes it possible to e.g. reduce `utils.instance_from_dict()` and `Instance.asdict()` to a few lines of code. An added benefit would be consistency with JSON parsing (since the json parser in C is used under the hood).

Suggests to take the code removal/replacement in a separate PR to have little more control over changes.

## Type of change
- [x] Bug fix & code cleanup
- [x] New feature
- [ ] Documentation update
- [ ] Test update

## Checklist for the reviewer
This checklist should be used as a help for the reviewer.

- [ ] Is the change limited to one issue?
- [ ] Does this PR close the issue?
- [ ] Is the code easy to read and understand?
- [ ] Do all new feature have an accompanying new test?
- [ ] Has the documentation been updated as necessary?

Ref not checked correctly in dlite-validate
When checking a datamodel, whic has a type $ref, it is not actually checked that there is actuallya @ref which has a uri as value. 

In my opinion dlite-validate should check the whole "cascade" of refs as well.

Example of datamodel that is not checked correctly:

uri: http://onto-ns.com/meta/asmod/vasp/0.1/VaspCalculation
description: Documentation of a VASP calculation.
dimensions: {}
properties:
  vaspinput:
    description: Input required for a vasp-calculation
    type: "ref"
    ref: http://onto-ns.com/meta/asmod/vasp/0.1/VaspInputFiles


The last line should be 
$ref: http://onto-ns.com/meta/asmod/vasp/0.1/VaspInputFiles, and dlite-validate should recognize that.

Fix segfault if dimensions is null
# Description
Avoid that dlite segfaults if "dimensions" is null (empty) in JSON and YAML input.


## Type of change
- [x] Bug fix & code cleanup
- [ ] New feature
- [ ] Documentation update
- [ ] Test update

## Checklist for the reviewer
This checklist should be used as a help for the reviewer.

- [ ] Is the change limited to one issue?
- [ ] Does this PR close the issue?
- [ ] Is the code easy to read and understand?
- [ ] Do all new feature have an accompanying new test?
- [ ] Has the documentation been updated as necessary?



Discussion / hints:

Fix patch:
diff --git a/bindings/python/scripts/CMakeLists.txt b/bindings/python/scripts/CMakeLists.txt
index 0f2a7dad8..6618fca8e 100644
--- a/bindings/python/scripts/CMakeLists.txt
+++ b/bindings/python/scripts/CMakeLists.txt
@@ -25,6 +25,12 @@ function(test_failure name)
   set_property(TEST ${name} PROPERTY WILL_FAIL true)
 endfunction()
 
+# Check for yaml
+execute_process(
+  COMMAND ${Python3_EXECUTABLE} -c "import yaml"
+  RESULT_VARIABLE missing_yaml
+)
+
 
 # Install
 install(
@@ -83,13 +89,35 @@ test_success(
   dlite-validate-empty
   dlite-validate Empty.json
 )
-# Test valid yaml
-test_success(
-  dlite-validate-valid1-yaml
-  dlite-validate ../tests/entities/ValidYaml1.yaml
-)
-# Test invalid yaml
-test_failure(
-  dlite-validate-invalid1-yaml
-  dlite-validate ../tests/entities/InvalidYaml1.yaml
-)
+
+# Tests depending on yaml
+if(NOT ${missing_yaml})
+  # Test valid yaml
+  test_success(
+    dlite-validate-valid1-yaml
+    dlite-validate ../tests/entities/ValidYaml1.yaml
+  )
+  # Test invalid yaml
+  test_failure(
+    dlite-validate-invalid1-yaml
+    dlite-validate ../tests/entities/InvalidYaml1.yaml
+  )
+
+  # Test invalid yaml with ref-type
+  test_failure(
+    dlite-validate-ref-invalid1-yaml
+    dlite-validate ../tests/entities/Ref_invalid1.json
+  )
+
+  # Test valid yaml with invalid ref-type
+  test_failure(
+    dlite-validate-invalidref-yaml
+    dlite-validate ../tests/entities/ValidYaml_invalidref.yaml
+  )
+
+  # Test invalid yaml
+  test_failure(
+    dlite-validate-invalid2-yaml
+    dlite-validate ../tests/entities/InvalidYaml2.yaml
+  )
+endif()
diff --git a/bindings/python/scripts/dlite-validate b/bindings/python/scripts/dlite-validate
index b9ddeb302..f47d7a3bf 100644
--- a/bindings/python/scripts/dlite-validate
+++ b/bindings/python/scripts/dlite-validate
@@ -22,10 +22,23 @@ def parse(url, driver=None, options="mode=r", id=None):
         List of new instances.
     """
     import dlite
-
     loc = url.split("#", 1)[0].split("?",1)[0]
     if driver is None:
         driver = Path(loc).suffix.lstrip('.').lower()
+    
+
+    # Validate all instances if `id` is None
+    if id is None:
+        try:
+            uuids = dlite.Instance.get_uuids(driver, url, options=options)
+        #except dlite.DLiteError:
+        except dlite.DLiteUnsupportedError:
+            pass
+        else:
+            insts = []
+            for uuid in uuids:
+                insts.extend(parse(url, driver=driver, options=options, id=uuid))
+            return insts
 
     # Validate all instances if `id` is None
     if id is None:
@@ -102,8 +115,9 @@ def main():
     parser.add_argument(
         '--storage-path', '-p', action='append', default=[],
         help='URL or path to storage path to look up metadata. '
-        'Only needed when validating data instances. '
-        'This option can be given multiple times.'
+        'Only needed when validating data instances, or datamodels '
+        'referring to other datamodels as properties. '
+        'This option can be given multiple times.' 
     )
     parser.add_argument(
         '--show', '-s', action='store_true',
@@ -140,6 +154,23 @@ def main():
     for inst in insts:
         if inst.is_meta:
             check_dimensions(args.url, inst)
+            # Check that datamodels for properties that refer to
+            # a specific datamodel can be loaded.
+            for prop in inst.properties['properties']:
+                if prop.type == 'ref':
+                    if prop.ref == 'ref':
+                        print(f'Property "{prop.name}" is of type "ref", the value '
+                                'is ref and not a uri. Check that the property '
+                                '"$ref" is given, the "$" is a strict requirement.')
+                        sys.exit(1)
+                    # Add the path of the original url to storage_path
+                    # since it is likely that referenced data models are 
+                    # in the same folder.
+                    dlite.storage_path.append(Path(args.url).resolve().parent)
+                    propmeta = dlite.get_instance(prop.ref)
+                    print(f"Datamodel for property '{prop.name}' was loaded successfully, and "
+                        " not further validated.")
+
 
     print(f'{args.url}: validated')
 
diff --git a/src/dlite-json.c b/src/dlite-json.c
index a7b543d46..705b159bf 100644
--- a/src/dlite-json.c
+++ b/src/dlite-json.c
@@ -725,6 +725,8 @@ static DLiteInstance *parse_instance(const char *src, jsmntok_t *obj,
           if (dlite_property_jscan(src, t, p->name, ptr, p, pdims, 0) < 0)
             goto fail;
         } else {
+          if (!ptr)
+            FAIL1("cannot assign property with NULL destination: %s", p->name);
           if (dlite_type_scan(src+t->start, t->end - t->start, ptr, p->type,
                               p->size, 0) < 0)
             goto fail;


Test patch:
diff --git a/bindings/python/tests/entities/InvalidYaml2.yaml b/bindings/python/tests/entities/InvalidYaml2.yaml
new file mode 100644
index 000000000..27cf46207
--- /dev/null
+++ b/bindings/python/tests/entities/InvalidYaml2.yaml
@@ -0,0 +1,8 @@
+# An invalid version of datamodel Valid1.yaml
+uri: http://onto-ns.com/meta/0.1/InvalidYaml2
+description: A datamodel wiht invalid dimension (empty).
+dimensions:
+properties:
+  name:
+    type: str
+    description: Name of the item.
diff --git a/bindings/python/tests/entities/Ref_invalid1.json b/bindings/python/tests/entities/Ref_invalid1.json
new file mode 100644
index 000000000..a5e5e0be4
--- /dev/null
+++ b/bindings/python/tests/entities/Ref_invalid1.json
@@ -0,0 +1,27 @@
+{
+  "uri": "http://onto-ns.com/meta/0.2/Ref",
+  "description": "A datamodel with ref-types...",
+  "dimensions": {
+    "nitems": "Number of items.",
+    "nrefs": "Number of references to Ref datamodels."
+  },
+  "properties": {
+    "item": {
+      "type": "ref",
+      "ref": "http://onto-ns.com/meta/0.1/Item",
+      "description": "Single item."
+    },
+    "items": {
+      "type": "ref",
+      "$ref": "http://onto-ns.com/meta/0.1/Item",
+      "shape": ["nitems"],
+      "description": "List of items."
+    },
+    "refs": {
+      "type": "ref",
+      "$ref": "http://onto-ns.com/meta/0.1/Ref",
+      "shape": ["nrefs"],
+      "description": "List of references to Ref datamodels.  This may become very cyclic, but DLite should cope with it..."
+    }
+  }
+}
diff --git a/bindings/python/tests/entities/ValidYaml_invalidref.yaml b/bindings/python/tests/entities/ValidYaml_invalidref.yaml
new file mode 100644
index 000000000..eb36c444a
--- /dev/null
+++ b/bindings/python/tests/entities/ValidYaml_invalidref.yaml
@@ -0,0 +1,9 @@
+# An valid version of datamodel Invalid1.yaml
+uri: http://onto-ns.com/meta/0.1/ValidYaml1
+description: An datamodel for an item with float type and invalid shape name.
+dimensions: []
+properties:
+  name:
+    type: ref
+    description: Name of the item.
+    $ref: http://onto-ns.com/meta/0.1/InvalidYaml2