Repository: ResearchObject/ro-crate-py (python)

Issue:
Allow to attach partials to a crate?
Hi,

For Autosubmit, since the workflow configuration doesn't contain the information needed for RO-Crate, I [used](https://earth.bsc.es/gitlab/es/autosubmit/-/merge_requests/317) the exact same approach from [COMPSs](https://github.com/bsc-wdc/compss/blob/9e79542eef60afa9e288e7246e697bd7ac42db08/compss/runtime/scripts/system/provenance/generate_COMPSs_RO-Crate.py) and asked users to provide a YAML file with authors & license.

Then I create the objects and attach/add to the RO-Crate-py object.

The implementation in Autosubmit is similar, but not identical to COMPSs. Other workflow managers with similar need may craft yet another way of doing the same.

It would be nice if there was a way to load RO-Crate-py entities directly from a dictionary/YAML data. Something like

```py

from rocrate.entities.person import Person

crate = ROCrate()

with open('') as f:
  yaml_content = parser.safe_load(f)

for author in yaml_content['authors']:
  crate.add(Person.load_from_dict(author)
```

Not sure how to validate the format of the entities... maybe instead of YAML receive JSON-LD directly, or provide a tool/script to read SPARQL+SHACL, etc.?

Cheers
Bruno

Pull request:
Add methods for adding and updating JSON-LD directly (partials for WMS)
Closes #146 

@simleo I was toying with the code you suggest a little in the Autosubmit code base. But then I thought that if I wrote those functions in Autosubmit, other WMS would still require to do the same.

It occurred to me, then, maybe add these new methods to `ro-crate-py` instead? I made this draft pull request after testing it with Autosubmit. You can see the current implementation here: https://earth.bsc.es/gitlab/es/autosubmit/-/merge_requests/317/diffs

Most of the code in `autosubmit.py` in that merge request is retrieving metadata about the Autosubmit workflow configuration and execution logs and data. But what previously was

```py
        yaml_file = Path(experiment_path, 'conf/rocrate.yml')
        try:
            with open(yaml_file) as f:
                try:
                    yaml_content = YAMLParserFactory().create_parser().load(f)
                except Exception as e:
...
...
# Create RO-Crate and add Autosubmit data
...
...
        # Fill-in the pre-populated data
        crate.license = yaml_content['license']
        # These two sets keep track of the authors and orgs to add to the CRATE
        authors = set()
        organizations = set()
        for author in yaml_content['authors']:
            authors.add(author['orcid'])
            organizations.add(author['ror'])
            crate.add(Person(crate, author['orcid'], {
                'name': author['name'],
                'contactPoint': { '@id': f'mailto: {author["email"]}' },
                'affiliation': { '@id': f'mailto: {author["ror"]}' }
            }))
            crate.add(ContextEntity(crate, f'mailto: {author["email"]}', {
                '@type': 'ContactPoint',
                'contactType': 'Author',
                'email': author['email'],
                'identifier': author['email'],
                'url': author['orcid'],
            }))
            crate.add(ContextEntity(crate, author['ror'], {
                '@type': 'Organization',
                'name': author['organisation_name']
            }))
        crate.creator = { '@id': id for id in authors }
        crate.publisher = { '@id': id for id in organizations }

```

Now became, basically

```py
        json_file = Path(experiment_path, 'conf/rocrate.json')
        try:
            with open(json_file, 'r') as f:
                try:
                    json_content = json.load(f)
                except Exception as e:
                    raise AutosubmitCritical(f'Failed to parse $expid/conf/rocrate.json pre-populated file {json_file}', 7011, e)
        except IOError:
...
...
# Create RO-Crate and add Autosubmit data
# because ro-crate-py's add replaces data.
...
...
            if '@graph' in json_content:
                for jsonld_node in json_content['@graph']:
                    crate.add_or_update_jsonld(jsonld_node)

            # Write RO-Crate ZIP.
            crate.write_zip(Path(experiment_path, "rocrate.zip"))
```

I liked the general structure, and I think it could be applied to other WMS that preferred to go JSON-LD → ro-crate-py → JSON-LD, instead of what I had in the merge request before, YAML → Python → ro-crate-py → JSON-LD (where YAML → Python could possibly vary between WMS's).

If it sounds like a good idea I will add docs & tests :+1: Thank you for the code example.

Also added another commit with comments & f-strings for code that I was having a look, and trying to understand before using it. Let me know if you prefer that I drop that commit and open a separate pull request for it.

Cheers
Bruno


Patch:
diff --git a/CITATION.cff b/CITATION.cff
index b255f9b..6599788 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -16,6 +16,9 @@ authors:
   - family-names: Huber
     given-names: Sebastiaan
     orcid: https://orcid.org/0000-0001-5845-8880
+  - family-names: Kinoshita
+    given-names: Bruno
+    orcid: https://orcid.org/0000-0001-8250-4074
   - family-names: Leo
     given-names: Simone
     orcid: https://orcid.org/0000-0001-8271-5429
diff --git a/README.md b/README.md
index 67b650e..8fe5702 100644
--- a/README.md
+++ b/README.md
@@ -171,6 +171,60 @@ Note that entities can have multiple types, e.g.:
     "@type" = ["File", "SoftwareSourceCode"]
 ```
 
+#### Modifying the crate from JSON-LD dictionaries
+
+The `add_jsonld` method allows to add a contextual entity directly from a
+JSON-LD dictionary containing at least the `@id` and `@type` keys:
+
+```python
+crate.add_jsonld({
+    "@id": "https://orcid.org/0000-0000-0000-0000",
+    "@type": "Person",
+    "name": "Alice Doe"
+})
+```
+
+Existing entities can be updated from JSON-LD dictionaries via `update_jsonld`:
+
+```python
+crate.update_jsonld({
+    "@id": "https://orcid.org/0000-0000-0000-0000",
+    "name": "Alice K. Doe"
+})
+```
+
+There is also an `add_or_update_jsonld` method that adds the entity if it's
+not already in the crate and updates it if it already exists (note that, when
+updating, the `@type` key is ignored). This allows to "patch" an RO-Crate from
+a JSON-LD file. For instance, suppose you have the following `patch.json` file:
+
+```json
+{
+    "@graph": [
+        {
+            "@id": "./",
+            "author": {"@id": "https://orcid.org/0000-0000-0000-0001"}
+        },
+        {
+            "@id": "https://orcid.org/0000-0000-0000-0001",
+            "@type": "Person",
+            "name": "Bob Doe"
+        }
+    ]
+}
+```
+
+Then the following sets Bob as the author of the crate according to the above
+file:
+
+```python
+crate = ROCrate("temp-crate")
+with open("patch.json") as f:
+    json_data = json.load(f)
+for d in json_data.get("@graph", []):
+    crate.add_or_update_jsonld(d)
+```
+
 ### Consuming an RO-Crate
 
 An existing RO-Crate package can be loaded from a directory or zip file:
@@ -278,7 +332,7 @@ Note that data entities (e.g., workflows) must already be present in the directo
 cd test/test-data/ro-crate-galaxy-sortchangecase
 ```
 
-This directory is already an ro-crate. Delete the metadata file to get a plain directory tree:
+This directory is already an RO-Crate. Delete the metadata file to get a plain directory tree:
 
 ```bash
 rm ro-crate-metadata.json
@@ -315,7 +369,6 @@ rocrate add test-instance test1 http://example.com -r jobs -i test1_1
 rocrate add test-definition test1 test/test1/sort-and-change-case-test.yml -e planemo -v '>=0.70'
 ```
 
-
 ## License
 
  * Copyright 2019-2022 The University of Manchester, UK
diff --git a/rocrate/__init__.py b/rocrate/__init__.py
index 2ff3ad1..e19cc6d 100644
--- a/rocrate/__init__.py
+++ b/rocrate/__init__.py
@@ -33,6 +33,7 @@ __author__ = ", ".join((
     'Ignacio Eguinoa',
     'Alban Gaignard',
     'Sebastiaan Huber',
+    'Bruno Kinoshita',
     'Simone Leo',
     'Luca Pireddu',
     'Laura Rodríguez-Navas',
diff --git a/rocrate/model/entity.py b/rocrate/model/entity.py
index ede44d2..6a64501 100644
--- a/rocrate/model/entity.py
+++ b/rocrate/model/entity.py
@@ -32,7 +32,7 @@ class Entity(MutableMapping):
         if identifier:
             self.id = self.format_id(identifier)
         else:
-            self.id = "#%s" % uuid.uuid4()
+            self.id = f"#{uuid.uuid4()}"
         if properties:
             empty = self._empty()
             empty.update(properties)
@@ -46,7 +46,7 @@ class Entity(MutableMapping):
         return str(identifier)
 
     def __repr__(self):
-        return "<%s %s>" % (self.id, self.type)
+        return f"<{self.id} {self.type}>"
 
     def properties(self):
         return self._jsonld
diff --git a/rocrate/rocrate.py b/rocrate/rocrate.py
index 264e894..d50a768 100644
--- a/rocrate/rocrate.py
+++ b/rocrate/rocrate.py
@@ -513,6 +513,76 @@ class ROCrate():
         self.metadata.extra_terms.update(TESTING_EXTRA_TERMS)
         return definition
 
+    def add_jsonld(self, jsonld):
+        """Add a JSON-LD dictionary as a contextual entity to the RO-Crate.
+
+        The `@id` and `@type` keys must be present in the JSON-LD dictionary.
+
+        Args:
+            jsonld: A JSON-LD dictionary containing at least `@id` and `@type`.
+        Return:
+            The entity added to the RO-Crate.
+        Raises:
+            ValueError: if the jsonld object is empty or None or if the
+                entity already exists (found via @id).
+        """
+        required_keys = {"@id", "@type"}
+        if not jsonld or not required_keys.issubset(jsonld):
+            raise ValueError("you must provide a non-empty JSON-LD dictionary with @id and @type set")
+        entity_id = jsonld.pop("@id")
+        if self.get(entity_id):
+            raise ValueError(f"entity {entity_id} already exists in the RO-Crate")
+        return self.add(ContextEntity(
+            self,
+            entity_id,
+            properties=jsonld
+        ))
+
+    def update_jsonld(self, jsonld):
+        """Update an entity in the RO-Crate from a JSON-LD dictionary.
+
+        An `@id` must be present in the JSON-LD dictionary. Any other keys
+        in the JSON-LD dictionary that start with `@` will be removed.
+
+        Args:
+            jsonld: A JSON-LD dictionary.
+        Return:
+            The updated entity.
+        Raises:
+            ValueError: if the jsonld object is empty or None, if @id was not
+              provided, or if the entity was not found.
+        """
+        if not jsonld or "@id" not in jsonld:
+            raise ValueError("you must provide a non-empty JSON-LD dictionary")
+        entity_id = jsonld.pop("@id")
+        entity: Entity = self.get(entity_id)
+        if not entity:
+            raise ValueError(f"entity {entity_id} does not exist in the RO-Crate")
+        jsonld = {k: v for k, v in jsonld.items() if not k.startswith('@')}
+        entity._jsonld.update(jsonld)
+        return entity
+
+    def add_or_update_jsonld(self, jsonld):
+        """Add or update an entity from a JSON-LD dictionary.
+
+        An `@id` must be present in the JSON-LD dictionary.
+
+        Args:
+            jsonld: A JSON-LD dictionary.
+        Return:
+            The added or updated entity.
+        Raises:
+            ValueError: if the jsonld object is empty or None or if @id was not
+              provided.
+        """
+        if not jsonld or "@id" not in jsonld:
+            raise ValueError("you must provide a non-empty JSON-LD dictionary")
+        entity_id = jsonld.get("@id")
+        entity: Entity = self.get(entity_id)
+        if not entity:
+            return self.add_jsonld(jsonld)
+        return self.update_jsonld(jsonld)
+
     def __validate_suite(self, suite):
         if isinstance(suite, TestSuite):
             assert suite.crate is self
diff --git a/setup.py b/setup.py
index 5775239..30a2001 100755
--- a/setup.py
+++ b/setup.py
@@ -61,6 +61,7 @@ setup(
         'Ignacio Eguinoa',
         'Alban Gaignard',
         'Sebastiaan Huber',
+        'Bruno Kinoshita',
         'Simone Leo',
         'Luca Pireddu',
         'Laura Rodríguez-Navas',


Test patch:
diff --git a/test/test_jsonld.py b/test/test_jsonld.py
new file mode 100644
index 0000000..8e29535
--- /dev/null
+++ b/test/test_jsonld.py
@@ -0,0 +1,227 @@
+# Copyright 2019-2022 The University of Manchester, UK
+# Copyright 2020-2022 Vlaams Instituut voor Biotechnologie (VIB), BE
+# Copyright 2020-2022 Barcelona Supercomputing Center (BSC), ES
+# Copyright 2020-2022 Center for Advanced Studies, Research and Development in Sardinia (CRS4), IT
+# Copyright 2022 École Polytechnique Fédérale de Lausanne, CH
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for the JSON-LD methods of the ROCrate object."""
+
+from uuid import uuid4
+
+import pytest
+
+from rocrate.rocrate import ROCrate
+
+
+# --- add
+
+
+def test_add_jsonld_raises_json_is_none():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.add_jsonld(None)
+
+
+def test_add_jsonld_raises_json_is_empty():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.add_jsonld({})
+
+
+def test_add_jsonld_raises_json_missing_id():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.add_jsonld({
+            '@type': 'CreativeWork'
+        })
+
+
+def test_add_jsonld_raises_json_missing_type(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    with pytest.raises(ValueError, match='.*@type.*'):
+        crate.add_jsonld({
+            '@id': './',
+            'license': 'NA'
+        })
+
+
+def test_add_jsonld_raises_json_duplicate_id(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    with pytest.raises(ValueError, match='.*already exists.*'):
+        crate.add_jsonld({
+            '@id': './',
+            '@type': 'Dataset',
+            'license': 'NA'
+        })
+
+
+def test_add_jsonld(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    new_entity_id = f'#{uuid4()}'
+
+    crate.add_jsonld({
+        '@id': new_entity_id,
+        '@type': 'CreativeWork',
+        'name': 'A test entity'
+    })
+    new_entity = crate.get(new_entity_id)
+    assert new_entity.type == 'CreativeWork'
+    assert new_entity["name"] == 'A test entity'
+
+
+# --- update
+
+
+def test_update_jsonld_raises_json_is_none():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.update_jsonld(None)
+
+
+def test_update_jsonld_raises_json_is_empty():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.update_jsonld({})
+
+
+def test_update_jsonld_raises_json_missing_id():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.update_jsonld({
+            '@type': 'CreativeWork'
+        })
+
+
+def test_update_jsonld_raises_id_not_found(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    missing_entity_id = f'#{uuid4()}'
+
+    with pytest.raises(ValueError, match='.*does not exist.*'):
+        crate.update_jsonld({
+            '@id': missing_entity_id,
+            '@type': 'CreativeWork',
+            'name': 'This entity does not exist in the RO-Crate'
+        })
+
+
+def test_update_jsonld(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    new_entity_id = f'#{uuid4()}'
+
+    crate.add_jsonld({
+        '@id': new_entity_id,
+        '@type': 'CreativeWork',
+        'name': 'A test entity'
+    })
+
+    entity_added = crate.get(new_entity_id)
+    assert entity_added.type == 'CreativeWork'
+    assert entity_added['name'] == 'A test entity'
+
+    entity_added['name'] = 'No potatoes today'
+    # N.B.: Properties that start with @ are ignored when updating.
+    update_dict = entity_added._jsonld.copy()
+    update_dict['@type'] = 'Dataset'
+    crate.update_jsonld(update_dict)
+
+    updated_entity = crate.get(new_entity_id)
+    assert entity_added.id == updated_entity.id
+    assert updated_entity.type == 'CreativeWork'
+    assert updated_entity['name'] == 'No potatoes today'
+
+    assert '@type' in update_dict
+
+
+# --- add or update
+
+
+def test_add_or_update_jsonld_raises_json_is_none():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.add_or_update_jsonld(None)
+
+
+def test_add_or_update_jsonld_raises_json_is_empty():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.add_or_update_jsonld({})
+
+
+def test_add_or_update_jsonld_raises_json_missing_id():
+    crate = ROCrate()
+
+    with pytest.raises(ValueError, match='.*non-empty JSON-LD.*'):
+        crate.add_or_update_jsonld({
+            '@type': 'CreativeWork'
+        })
+
+
+def test_add_or_update_add_jsonld(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    new_entity_id = f'#{uuid4()}'
+
+    crate.add_or_update_jsonld({
+        '@id': new_entity_id,
+        '@type': 'CreativeWork',
+        'name': 'A test entity'
+    })
+    new_entity = crate.get(new_entity_id)
+    assert new_entity.type == 'CreativeWork'
+    assert new_entity["name"] == 'A test entity'
+
+
+def test_add_or_update_update_jsonld(test_data_dir):
+    crate_dir = test_data_dir / 'read_crate'
+    crate = ROCrate(crate_dir)
+
+    new_entity_id = f'#{uuid4()}'
+
+    crate.add_jsonld({
+        '@id': new_entity_id,
+        '@type': 'CreativeWork',
+        'name': 'A test entity'
+    })
+
+    entity_added = crate.get(new_entity_id)
+    assert entity_added.type == 'CreativeWork'
+    assert entity_added['name'] == 'A test entity'
+
+    entity_added['name'] = 'No potatoes today'
+    crate.add_or_update_jsonld(entity_added._jsonld.copy())
+
+    updated_entity = crate.get(new_entity_id)
+    assert entity_added.id == updated_entity.id
+    assert entity_added.type == updated_entity.type
+    assert updated_entity['name'] == 'No potatoes today'
diff --git a/test/test_model.py b/test/test_model.py
index 49561ec..fbfa8fd 100644
--- a/test/test_model.py
+++ b/test/test_model.py
@@ -18,6 +18,7 @@
 
 import datetime
 import json
+import os
 import tempfile
 import timeit
 import uuid
@@ -95,6 +96,7 @@ def test_data_entities(test_data_dir):
     assert set(_.id for _ in (file_, dataset, data_entity)) <= part_ids
 
 
+@pytest.mark.skipif(os.name != "posix", reason="CI sometimes fails on macOS")
 def test_data_entities_perf():
     """\
     Test that adding a data entity happens in constant time.