Repository: bids-standard/pybids

Issue:
Native support for Derivative Datasets as `root`
`BIDSLayout` currently expects a valid, BIDS-formatted dataset as `root`. Importantly, at least so far as I can tell, it will only parse valid BIDS entities. So even if the `validate` parameter is set to `False`, invalid entities like `foo`, `bar`, or `desc` will not be indexed.

The BIDS spec allows derivative data to be placed in stand-alone directories. The only requirement is that the `dataset_description.json` be populated with the `GeneratedBy` field. They also recommend that `DatasetType` be set to `derivative`. Thus, if the dataset is correctly formatted, it can be unambiguously identified as a derivative.

Unfortunately, such derivative datasets are not natively supported as `root` datasets by `BIDSLayout`. The workaround is to set the `config` parameter of `BIDSLayout` to `derivatives`, although I could not find this fact mentioned in any public documentation. As described above, though, detecting whether a `root` dataset is a derivative is as simple as reading the `dataset_description.json`.

Would `pybids` consider supporting such automatic detection? (Or is there already an easy way to do this that I'm just missing?)


Discussion:
Yes, we absolutely need to handle this correctly. Any interest in submitting a PR?
Sure, I can take a look. One design decision: the only REQUIRED item for derivatives is the `GeneratedBy` field. So should that be the field to look for (then validate it with existing machinery in `validation.py`)? Other potential fields to check would be `DatasetType` or `SourceDatasets` but neither mandatory.

One other question: I know `pybids` is transitioning from the legacy `PipelineDescription` over to `GeneratedBy`, but I'm not sure how far it actually is. Is it still at a point where it should look for `PipelineDescription`, or can that be left out of consideration?
`PipelineDescription` can be dropped. I have a PR that removes the last bits of it, though that broke some stuff that I need to find time to debug...

Anyway, `DatasetType` has a default value of `raw`, so we should just go by that. Effectively, it's mandatory for derivatives datasets and optional for raw datasets. Using `GeneratedBy` won't work long-term, as there's an effort to allow it for recording the provenance of the raw dataset (this is actually why `DatasetType` was added to the spec).
Ah, I didn't realize that about `DatasetType`. Okay, then that's a really easy field to check.

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -134,6 +134,12 @@ def __init__(self, root=None, validate=True, absolute_paths=True,
 
         # Initialize a completely new layout and index the dataset
         if not load_db:
+            # If root dataset is a derivative, set config accordingly
+            if description and description.get("DatasetType") == "derivative":
+                if validate:
+                    validate_derivative_paths([root], self)
+                config = ["bids", "derivatives"]
+
             init_args = dict(root=root, absolute_paths=absolute_paths,
                              derivatives=derivatives, config=config)
 
diff --git a/bids/layout/validation.py b/bids/layout/validation.py
--- a/bids/layout/validation.py
+++ b/bids/layout/validation.py
@@ -153,6 +153,8 @@ def check_for_description(bids_dir):
                           "``pybids upgrade`` to update your derivative "
                           "dataset.")
             pipeline_name = description["PipelineDescription"].get("Name")
+        else:
+            pipeline_name = None
         if pipeline_name is None:
             raise BIDSDerivativesValidationError(
                                 "Every valid BIDS-derivatives dataset must "


Test patch:
diff --git a/bids/layout/tests/conftest.py b/bids/layout/tests/conftest.py
--- a/bids/layout/tests/conftest.py
+++ b/bids/layout/tests/conftest.py
@@ -54,7 +54,7 @@ def layout_ds005_multi_derivs(request, db_dir):
 
     layout = BIDSLayout(data_dir,
                         database_path=database_path)
-    deriv_dir1 = join(get_test_data_path(), 'ds005_derivs')
+    deriv_dir1 = join(get_test_data_path(), 'ds005_derivs', 'dummy')
     deriv_dir2 = join(data_dir, 'derivatives', 'events')
     layout.add_derivatives([deriv_dir1, deriv_dir2])
     return layout
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -18,6 +18,7 @@
 from bids.utils import natural_sort
 
 from bids.exceptions import (
+    BIDSDerivativesValidationError,
     BIDSValidationError,
     NoMatchError,
     TargetError,
@@ -141,6 +142,47 @@ def test_get_file(layout_ds005_derivs):
     assert not layout.get_file('/absolute/bleargh')
 
 
+class TestDerivativeAsRoot:
+    def test_dataset_without_datasettype_parsed_as_raw(self):
+        dataset_path = Path("ds005_derivs", "format_errs", "no_dataset_type")
+        unvalidated = BIDSLayout(
+            Path(get_test_data_path())/dataset_path,
+            validate=False
+        )
+        assert len(unvalidated.get()) == 4
+        with pytest.raises(ValueError):
+            unvalidated.get(desc="preproc")
+
+        validated = BIDSLayout(Path(get_test_data_path())/dataset_path)
+        assert len(validated.get()) == 1
+
+    def test_dataset_missing_generatedby_fails_validation(self):
+        dataset_path = Path("ds005_derivs", "format_errs", "no_pipeline_description")
+        with pytest.raises(BIDSDerivativesValidationError):
+            BIDSLayout(Path(get_test_data_path())/dataset_path)
+
+
+    def test_correctly_formatted_derivative_loads_as_derivative(self):
+        dataset_path = Path("ds005_derivs", "dummy")
+        layout = BIDSLayout(Path(get_test_data_path())/dataset_path)
+        assert len(layout.get()) == 4
+        assert len(layout.get(desc="preproc")) == 3
+
+    @pytest.mark.parametrize(
+        "dataset_path",
+        [
+            Path("ds005_derivs", "dummy"),
+            Path("ds005_derivs", "format_errs", "no_pipeline_description")
+        ]
+    )
+    def test_derivative_datasets_load_with_no_validation(self, dataset_path):
+        layout = BIDSLayout(
+            Path(get_test_data_path())/dataset_path,
+            validate=False
+        )
+        assert len(layout.get()) == 4
+        assert len(layout.get(desc="preproc")) == 3
+
 def test_get_metadata(layout_7t_trt):
     target = 'sub-03/ses-2/func/sub-03_ses-2_task-' \
              'rest_acq-fullbrain_run-2_bold.nii.gz'
@@ -722,3 +764,30 @@ def test_load_layout(layout_synthetic_nodb, db_dir):
     cm2 = reloaded.connection_manager
     for attr in ['root', 'absolute_paths', 'config', 'derivatives']:
         assert getattr(cm1.layout_info, attr) == getattr(cm2.layout_info, attr)
+
+
+def test_load_layout_config_not_overwritten(layout_synthetic_nodb, tmpdir):
+    modified_dataset_path = tmpdir/"modified"
+    shutil.copytree(layout_synthetic_nodb.root, modified_dataset_path)
+
+    # Save index
+    db_path = str(tmpdir / 'tmp_db')
+    BIDSLayout(modified_dataset_path).save(db_path)
+
+    # Update dataset_description.json
+    dataset_description = modified_dataset_path/"dataset_description.json"
+    with dataset_description.open('r') as f:
+        description = json.load(f)
+    description["DatasetType"] = "derivative"
+    with dataset_description.open('w') as f:
+        json.dump(description, f)
+
+    # Reload
+    db_layout = BIDSLayout(modified_dataset_path, database_path=db_path)
+    fresh_layout = BIDSLayout(modified_dataset_path, validate=False)
+    cm1 = db_layout.connection_manager
+    cm2 = fresh_layout.connection_manager
+    for attr in ['root', 'absolute_paths', 'derivatives']:
+        assert getattr(cm1.layout_info, attr) == getattr(cm2.layout_info, attr)
+
+    assert cm1.layout_info.config != cm2.layout_info.config
\ No newline at end of file
diff --git a/bids/tests/data/ds005_derivs/dummy/dataset_description.json b/bids/tests/data/ds005_derivs/dummy/dataset_description.json
--- a/bids/tests/data/ds005_derivs/dummy/dataset_description.json
+++ b/bids/tests/data/ds005_derivs/dummy/dataset_description.json
@@ -4,5 +4,6 @@
       },
     "BIDSVersion": "1.0.0rc2",
     "Name": "Mixed-gambles task -- dummy derivative",
-    "ReferencesAndLinks": "Tom, S.M., Fox, C.R., Trepel, C., Poldrack, R.A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315(5811):515-8"
+    "ReferencesAndLinks": "Tom, S.M., Fox, C.R., Trepel, C., Poldrack, R.A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315(5811):515-8",
+    "DatasetType": "derivative"
 }
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/dataset_description.json b/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/dataset_description.json
new file mode 100644
--- /dev/null
+++ b/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/dataset_description.json
@@ -0,0 +1,8 @@
+{
+    "PipelineDescription": {
+        "Name": "dummy"
+      },
+    "BIDSVersion": "1.0.0rc2",
+    "Name": "Mixed-gambles task -- dummy derivative",
+    "ReferencesAndLinks": "Tom, S.M., Fox, C.R., Trepel, C., Poldrack, R.A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315(5811):515-8"
+}
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/sub-01/func/sub-01_task-mixedgamblestask_run-01_desc-preproc_bold.nii.gz b/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/sub-01/func/sub-01_task-mixedgamblestask_run-01_desc-preproc_bold.nii.gz
new file mode 100644
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/sub-01/func/sub-01_task-mixedgamblestask_run-02_desc-preproc_bold.nii.gz b/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/sub-01/func/sub-01_task-mixedgamblestask_run-02_desc-preproc_bold.nii.gz
new file mode 100644
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/sub-01/func/sub-01_task-mixedgamblestask_run-03_desc-preproc_bold.nii.gz b/bids/tests/data/ds005_derivs/format_errs/no_dataset_type/sub-01/func/sub-01_task-mixedgamblestask_run-03_desc-preproc_bold.nii.gz
new file mode 100644
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/dataset_description.json b/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/dataset_description.json
new file mode 100644
--- /dev/null
+++ b/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/dataset_description.json
@@ -0,0 +1,6 @@
+{
+    "BIDSVersion": "1.0.0rc2",
+    "Name": "Mixed-gambles task -- dummy derivative",
+    "ReferencesAndLinks": "Tom, S.M., Fox, C.R., Trepel, C., Poldrack, R.A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315(5811):515-8",
+    "DatasetType": "derivative"
+}
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/sub-01/func/sub-01_task-mixedgamblestask_run-01_desc-preproc_bold.nii.gz b/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/sub-01/func/sub-01_task-mixedgamblestask_run-01_desc-preproc_bold.nii.gz
new file mode 100644
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/sub-01/func/sub-01_task-mixedgamblestask_run-02_desc-preproc_bold.nii.gz b/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/sub-01/func/sub-01_task-mixedgamblestask_run-02_desc-preproc_bold.nii.gz
new file mode 100644
diff --git a/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/sub-01/func/sub-01_task-mixedgamblestask_run-03_desc-preproc_bold.nii.gz b/bids/tests/data/ds005_derivs/format_errs/no_pipeline_description/sub-01/func/sub-01_task-mixedgamblestask_run-03_desc-preproc_bold.nii.gz
new file mode 100644