Repository: CONP-PCNO/conp-dataset

Issue:
Auto-archive script not running on crawled dataset?
All OSF or Zenodo datasets are not being archived when running the script `auto_archive.py` with option `--all`.

The following information systematically shows:
```
SKIPPED: archive not needed for projects/<dataset_name>
```
 but the dataset size is definitely below 20GB.

List of affected datasets:

- CFMM_7T__MP2RAGE_T1_mapping_version-2020-10-02T12:55:03.086599.tar.gz
- Comparing_Perturbation_Modes_for_Evaluating_Instabilities_in_Neuroimaging__Processed_NKI_RS_Subset__08_2019__version-None.tar.gz
- Intracellular_Recordings_of_Murine_Neocortical_Neurons_version-2020-04-18T19:34:10.458118.tar.gz
- Learning_Naturalistic_Structure__Processed_fMRI_dataset_version-0.1.0.tar.gz
- MRI_and_unbiased_averages_of_wild_muskrats__Ondatra_zibethicus__and_red_squirrels__Tamiasciurus_hudsonicus__version-1.0.tar.gz:
- Multi-model_functionalization_of_disease-associated_PTEN_missense_mutations_version-1.0.tar.gz
- Quantifying_Neural_Cognitive_Relationships_Across_the_Brain_version-2020-11-11T02:30:59.429493.tar.gz
- Synthetic_Animated_Mouse__SAM___University_of_British_Columbia__Datasets_and_3D_models_version-2021-04-09T14:12:32.548954.tar.gz:
- Systematic_phenomics_analysis_of_autism-associated_genes_version-3.2.tar.gz
- VFA_T1_mapping___RTHawk__open__vs_Siemens__commercial__version-v1.0.tar.gz



Auto-archive script not running on crawled dataset?
All OSF or Zenodo datasets are not being archived when running the script `auto_archive.py` with option `--all`.

The following information systematically shows:
```
SKIPPED: archive not needed for projects/<dataset_name>
```
 but the dataset size is definitely below 20GB.

List of affected datasets:

- CFMM_7T__MP2RAGE_T1_mapping_version-2020-10-02T12:55:03.086599.tar.gz
- Comparing_Perturbation_Modes_for_Evaluating_Instabilities_in_Neuroimaging__Processed_NKI_RS_Subset__08_2019__version-None.tar.gz
- Intracellular_Recordings_of_Murine_Neocortical_Neurons_version-2020-04-18T19:34:10.458118.tar.gz
- Learning_Naturalistic_Structure__Processed_fMRI_dataset_version-0.1.0.tar.gz
- MRI_and_unbiased_averages_of_wild_muskrats__Ondatra_zibethicus__and_red_squirrels__Tamiasciurus_hudsonicus__version-1.0.tar.gz:
- Multi-model_functionalization_of_disease-associated_PTEN_missense_mutations_version-1.0.tar.gz
- Quantifying_Neural_Cognitive_Relationships_Across_the_Brain_version-2020-11-11T02:30:59.429493.tar.gz
- Synthetic_Animated_Mouse__SAM___University_of_British_Columbia__Datasets_and_3D_models_version-2021-04-09T14:12:32.548954.tar.gz:
- Systematic_phenomics_analysis_of_autism-associated_genes_version-3.2.tar.gz
- VFA_T1_mapping___RTHawk__open__vs_Siemens__commercial__version-v1.0.tar.gz





Discussion:

Resolving patch:
diff --git a/scripts/auto_archive.py b/scripts/auto_archive.py
--- a/scripts/auto_archive.py
+++ b/scripts/auto_archive.py
@@ -64,8 +64,11 @@ def parse_args():
     return parser.parse_args()
 
 
-def get_all_datasets():
-    return {os.path.basename(submodule.path) for submodule in git.Repo().submodules}
+def get_datasets_path():
+    return {
+        os.path.basename(submodule.path): submodule.path
+        for submodule in git.Repo().submodules
+    }
 
 
 def get_modified_datasets(
@@ -129,9 +132,11 @@ def get_modified_datasets(
     return modified_datasets
 
 
-def archive_dataset(dataset_path: str, out_dir: str, version: str) -> None:
+def archive_dataset(
+    dataset_path: str, out_dir: str, archive_name: str, version: str
+) -> None:
     os.makedirs(os.path.dirname(out_dir), mode=0o755, exist_ok=True)
-    out_filename = f"{out_dir}_version-{version}.tar.gz"
+    out_filename = f"{archive_name}_version-{version}.tar.gz"
     logger.info(f"Archiving dataset: {dataset_path} to {out_filename}")
 
     cwd = os.getcwd()
@@ -160,7 +165,8 @@ def archive_dataset(dataset_path: str, out_dir: str, version: str) -> None:
     args = parse_args()
 
     # Only archive the datasets available locally.
-    datasets = get_all_datasets()
+    datasets_path = get_datasets_path()
+    datasets = datasets_path.keys()
     if args.dataset:
         target_datasets = {os.path.basename(os.path.normpath(d)) for d in args.dataset}
         logger.warning(
@@ -176,7 +182,7 @@ def archive_dataset(dataset_path: str, out_dir: str, version: str) -> None:
         datasets &= modified_datasets
 
     for dataset_name in datasets:
-        dataset = "projects/" + dataset_name
+        dataset = datasets_path[dataset_name]
 
         try:
             logger.info(f"Installing dataset: {dataset}")
@@ -189,7 +195,13 @@ def archive_dataset(dataset_path: str, out_dir: str, version: str) -> None:
             with open(os.path.join(dataset, "DATS.json")) as fin:
                 metadata = json.load(fin)
 
-                is_public = metadata.get("privacy") == "open"
+                is_public = (
+                    metadata.get("distributions", [{}])[0]
+                    .get("access", {})
+                    .get("authorizations", [{}])[0]
+                    .get("value")
+                    == "public"
+                )
                 version = metadata.get("version")
 
                 for distribution in metadata.get("distributions", list()):
@@ -205,9 +217,13 @@ def archive_dataset(dataset_path: str, out_dir: str, version: str) -> None:
                 for submodule in get_proper_submodules(dataset):
                     get_dataset(submodule)
 
+                archive_name = "__".join(
+                    os.path.relpath(dataset, "projects").split("/")
+                )
                 archive_dataset(
                     dataset,
                     out_dir=os.path.join(args.out_dir, dataset_name),
+                    archive_name=archive_name,
                     version=version,
                 )
                 logger.info(f"SUCCESS: archive created for {dataset}")


Test patch: