Repository: CONP-PCNO/conp-dataset

Issue:
Zenodo: Add taxonomic "subject" annotation to crawl results
## Purpose
Zenodo offers authors the [option to annotate with "Subjects"](https://developers.zenodo.org/#representation): 
> Specify subjects from a taxonomy or controlled vocabulary. Each term must be uniquely identified (e.g. a URL). For free form text, use the keywords field. Each array element is an object with the attributes:
> * term: Term from taxonomy or controlled vocabulary.
> * identifier: Unique identifier for term.
> * scheme: Persistent identifier scheme for id (automatically detected).

## Context
If the author goes through the trouble of making these connections, it would be good to get them. Here is an example with "subject" annotations: https://zenodo.org/record/4741241

These "subject" annotations are not currently crawled.

## Possible Implementation (optional)
Zenodo currently doesn't seem to expose the "Subject" meta-data in the website explicitly. However, we can get this through their API, e.g.: https://zenodo.org/api/records/4741241?prettyprint=1

See also: https://github.com/zenodo/zenodo/issues/2019



Discussion:

Resolving patch:
diff --git a/scripts/Crawlers/ZenodoCrawler.py b/scripts/Crawlers/ZenodoCrawler.py
--- a/scripts/Crawlers/ZenodoCrawler.py
+++ b/scripts/Crawlers/ZenodoCrawler.py
@@ -1,6 +1,7 @@
 import datetime
 import json
 import os
+import re
 from typing import Callable
 
 import html2markdown
@@ -139,6 +140,26 @@ def get_all_dataset_description(self):
             if "keywords" in metadata.keys():
                 keywords = list(map(lambda x: {"value": x}, metadata["keywords"]))
 
+            # Retrieve subject annotations from Zenodo and clean the annotated
+            # subjects to insert in isAbout of DATS file
+            is_about = []
+            if "subjects" in metadata.keys():
+                for subject in metadata["subjects"]:
+                    if re.match("www.ncbi.nlm.nih.gov/taxonomy", subject["identifier"]):
+                        is_about.append(
+                            {
+                                "identifier": {"identifier": subject["identifier"]},
+                                "name": subject["term"],
+                            }
+                        )
+                    else:
+                        is_about.append(
+                            {
+                                "valueIRI": subject["identifier"],
+                                "value": subject["term"],
+                            }
+                        )
+
             dataset_size, dataset_unit = humanize.naturalsize(
                 sum([filename["size"] for filename in files]),
             ).split(" ")
diff --git a/scripts/dats_validator/validator.py b/scripts/dats_validator/validator.py
--- a/scripts/dats_validator/validator.py
+++ b/scripts/dats_validator/validator.py
@@ -267,7 +267,10 @@ def validate_is_about(dataset):
     if "isAbout" in dataset.keys():
         species_present = False
         for entry in dataset["isAbout"]:
-            if "identifier" in entry.keys():
+            if (
+                "identifier" in entry.keys()
+                and "identifierSource" in entry["identifier"].keys()
+            ):
                 identifier_source = entry["identifier"]["identifierSource"].lower()
                 if identifier_source.startswith(identifier_source_base_url):
                     species_present = True


Test patch: