Repository: bids-standard/pybids

Issue:
Derivatives indexing fails silently for deriv directories that lack a `dataset_description.json` file
I have a set of derivatives files that look like this:

derivatives/rt/sub-02/func/sub-02_task-dts_meanRT.tsv

they seem to pass the validator fine, but I get nothing when I use .get(scope='derivatives'):

```
In [35]: layout2 = BIDSLayout(args.bids_dir, derivatives=True)

In [36]: layout2.get(scope='derivatives')
Out[36]: []

```


Discussion:
this is using the pip installed version
@poldrack Can you post the contents of this file?
> more derivatives/rt/sub-01/func/sub-01_task-dts_meanRT.tsv
subject run     meanRT
01      1       2.0384300241666664
01      2       1.9368950080000007
01      3       1.9037270379555555
01      4       1.7877016932790697
01      5       1.9219444336888891
01      6       1.9910011718095237
01      7       2.0799579712558143
01      8       2.0416789709487184
01      9       1.7422352943695651
01      10      1.8070772854791664
01      11      1.6951297532765965
01      12      1.6897267177708333
@poldrack when you say it passes the validator, do you mean the JS validator, or the Python `BIDSValidator.is_bids()` method?

Can you try initializing the `BIDSLayout` with `validate=False` and see if that fixes the problem?
`meanRT` may be ignored as a suffix. 
ah, is there a dictionary of accepted suffixes?

On Wed, Mar 27, 2019 at 7:28 AM Chris Markiewicz <notifications@github.com>
wrote:

> meanRT may be ignored as a suffix.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/bids-standard/pybids/issues/426#issuecomment-477180834>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA1KkMlGP6Yns6cStich4LjsDGp3JmMjks5va3_3gaJpZM4cN1Ll>
> .
>


-- 
Russell A. Poldrack
Albert Ray Lang Professor of Psychology
Professor (by courtesy) of Computer Science
Bldg. 420, Jordan Hall
Stanford University
Stanford, CA 94305

poldrack@stanford.edu
http://www.poldracklab.org/

I meant that it passed the JS validator. just tried with validate=False and
it still doesn't see the derivatives
cheers
rp

On Wed, Mar 27, 2019 at 7:27 AM Tal Yarkoni <notifications@github.com>
wrote:

> @poldrack <https://github.com/poldrack> when you say it passes the
> validator, do you mean the JS validator, or the Python
> BIDSValidator.is_bids() method?
>
> Can you try initializing the BIDSLayout with validate=False and see if
> that fixes the problem?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/bids-standard/pybids/issues/426#issuecomment-477180445>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA1KkN1IcnO32fGcddg1u9okJxaX6itmks5va3-0gaJpZM4cN1Ll>
> .
>


-- 
Russell A. Poldrack
Albert Ray Lang Professor of Psychology
Professor (by courtesy) of Computer Science
Bldg. 420, Jordan Hall
Stanford University
Stanford, CA 94305

poldrack@stanford.edu
http://www.poldracklab.org/

Probably [this](https://github.com/bids-standard/bids-validator/blob/master/bids-validator/bids_validator/rules/file_level_rules.json). It may not be completely in sync with the draft derivatives spec though.

That said, if the files still aren't visible with `validate=False`, then it doesn't appear to be a validator issue.

Can you see if `layout.derivatives` actually contains anything? If it does, then check the `.files` attribute in any derivative `BIDSLayout` contained in the `.derivatives` dictionary and see if there's anything there. That'll help me diagnose where it's failing. (I can't reproduce it otherwise; querying for derivatives works in the test datasets.)
I just changed the suffix to beh and reran - no change.  

> In [2]: layout2 = BIDSLayout(args.bids_dir, derivatives=True,validate=False)
> 
> In [3]: layout2.get(scope='derivatives')
> Out[3]: []
> 
> In [4]: layout2.derivatives
> Out[4]: {}

I am attaching the full derivatives directory here - the associated dataset is https://openneuro.org/datasets/ds001715/versions/1.0.0
[derivs.zip](https://github.com/bids-standard/pybids/files/3013671/derivs.zip)


It's definitely not the filename itself; I just dropped a file with the same name into one of the test datasets and it's returned fine via `get()` (actually, it returns fine even with `validate=True`, which suggests that the validator isn't currently restricting suffixes after all).
EDIT: Oh, sorry. I'm thinking of this in terms of loading the variable in the analysis submodule. I think I'm being unhelpful for the actual problem at hand.

----

So the new derivatives suffix is `_timeseries.tsv`, but I don't think this really fits, conceptually. It doesn't really fit `participants.tsv` or `sessions.tsv`, but perhaps `scans.tsv`?

https://bids-specification.readthedocs.io/en/stable/03-modality-agnostic-files.html#scans-file

In which case it might be:

```
filename meanRT
func/sub-01_task-dts_run-1_bold.nii.gz 2.0384300241666664
func/sub-01_task-dts_run-2_bold.nii.gz 1.9368950080000007
func/sub-01_task-dts_run-3_bold.nii.gz 1.9037270379555555
func/sub-01_task-dts_run-4_bold.nii.gz 1.7877016932790697
func/sub-01_task-dts_run-5_bold.nii.gz 1.9219444336888891
func/sub-01_task-dts_run-6_bold.nii.gz 1.9910011718095237
func/sub-01_task-dts_run-7_bold.nii.gz 2.0799579712558143
func/sub-01_task-dts_run-8_bold.nii.gz 2.0416789709487184
func/sub-01_task-dts_run-9_bold.nii.gz 1.7422352943695651
func/sub-01_task-dts_run-10_bold.nii.gz 1.8070772854791664
func/sub-01_task-dts_run-11_bold.nii.gz 1.6951297532765965
func/sub-01_task-dts_run-12_bold.nii.gz 1.6897267177708333
```
@poldrack is the derivatives folder stored in `derivatives/` directly below `args.bids_dir`? (Note: I'm not sure if symlinking will work here; if it's symlinked, that *may* be the issue.)
If by "below" you mean "within", yes

On Wed, Mar 27, 2019, 7:43 AM Tal Yarkoni <notifications@github.com> wrote:

> @poldrack <https://github.com/poldrack> is the derivatives folder stored
> in derivatives/ directly below args.bids_dir?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/bids-standard/pybids/issues/426#issuecomment-477187697>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA1KkPs48vzW9m3pEir1hGsw81kZFAUuks5va4N6gaJpZM4cN1Ll>
> .
>

Ah, the problem is probably that you don't have a `dataset_description.json` in your derivatives directory. That's mandatory, because every derivatives dataset must also be a valid raw dataset.

What's unclear to me is why you're not getting an error message. Looking into that now. (And also, why it's still failing even when `validate=False`.)
But I do have a creeping feeling that I must be doing something stupid. I'm
afk for next hour, will have another look after that

On Wed, Mar 27, 2019, 7:45 AM Russ Poldrack <poldrack@gmail.com> wrote:

> If by "below" you mean "within", yes
>
> On Wed, Mar 27, 2019, 7:43 AM Tal Yarkoni <notifications@github.com>
> wrote:
>
>> @poldrack <https://github.com/poldrack> is the derivatives folder stored
>> in derivatives/ directly below args.bids_dir?
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/bids-standard/pybids/issues/426#issuecomment-477187697>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AA1KkPs48vzW9m3pEir1hGsw81kZFAUuks5va4N6gaJpZM4cN1Ll>
>> .
>>
>

Ah, yes, I do not

On Wed, Mar 27, 2019, 7:48 AM Tal Yarkoni <notifications@github.com> wrote:

> Ah, the problem is probably that you don't have a dataset_description.json
> in your derivatives directory. That's mandatory, because every derivatives
> dataset must also be a valid raw dataset.
>
> What's unclear to me is why you're not getting an error message. Looking
> into that now.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/bids-standard/pybids/issues/426#issuecomment-477190399>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AA1KkJfCFt2eGd2aqHLEr1Mw0AI0L3aPks5va4TWgaJpZM4cN1Ll>
> .
>

Ah, okay, so yeah, this is a bug. Currently the derivative-scanning routine is mandatorily checking if a description is found, which lives in `dataset_description.json`, and skipping silently if not found. There should be some error message or at least a warning. Will fix, thanks!

As far as your problem goes, adding a `dataset_description.json` (ideally with a `PipelineDescription` dict that contains its own `Name` field) should fix it.
On further examination, I realize why I implemented it this way. The problem is that a user could potentially pass in either a directory that contains multiple derivatives datasets in subdirectories, or an actual derivative directory. I.e., the following are both valid initialization patterns:

```python
#  Option 1
layout = BIDSLayout('/bids/root', derivatives='/tmp/derivs')
# Option 2
layout = BIDSLayout('/bids/root', derivatives=['/tmp/derivs/deriv1', '/tmp/derivs/deriv2'])
```

The problem is that if there's no `dataset_description.json` file in a folder, we can't simply assume the user made a mistake—it could be that the container folder contains under subfolders that the user didn't mean to index. Things get even hairier in the case where `validate=False`, because if we can't rely on `dataset_description.json` as a diagnostic, we now really have no way of knowing whether the user wants to index the specific subdirectory they passed, or [some] subdirectories below that.

Rather than adding convoluted logic to make educated guesses about what the user wants, I think a better approach is probably just to be a hard-ass about this and beef up the documentation. I propose to just say in the docstring that if you want to use pybids to index your derivatives, they *must* have a valid `dataset_description.json` file. Any objections, @effigies, @yarikoptic?
That seems sensible, but I'd still raise a warning by default.
We could do that, but we need to be smart about how we do it. I don't think we can do it on a per-directory basis. Consider case 1 in my snippet above: should we raise a warning for the `/tmp/derivs/` container? Probably not if there's at least one subdirectory containing the description file. In case 2, if we find a `dataset_description.json` for some subdirectories but not others, should we raise a warning for each directory that *doesn't* have it? Unclear.

How about we just raise a warning if, after the current scanning process, *no* candidate directories have been found? This seems reasonable, since you don't get derivative indexing unless you explicitly set `derivatives=True` in the layout init, or explicitly call `add_derivatives()`. So what about a single warning saying "derivative indexing requested, but none were found; please make sure all derivative datasets contain a `dataset_description.json` file as described in the BIDS specification"?

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -6,6 +6,7 @@
 from functools import reduce, partial
 from itertools import chain
 import copy
+import warnings
 
 from bids_validator import BIDSValidator
 from ..utils import listify, natural_sort, check_path_matches_patterns
@@ -117,6 +118,8 @@ class BIDSLayout(object):
             the paths to one or more derivatives directories to index. If False
             or None, the derivatives/ directory is ignored during indexing, and
             derivatives will have to be added manually via add_derivatives().
+            Note: derivatives datasets MUST contain a dataset_description.json
+            file in order to be indexed.
         config (str, list): Optional name(s) of configuration file(s) to use.
             By default (None), uses 'bids'.
         sources (BIDLayout, list): Optional BIDSLayout(s) from which the
@@ -356,6 +359,10 @@ def add_derivatives(self, path, **kwargs):
                 pipeline directory (e.g., derivatives/fmriprep).
             kwargs (dict): Optional keyword arguments to pass on to
                 BIDSLayout() when initializing each of the derivative datasets.
+
+        Note: Every derivatives directory intended for indexing MUST contain a
+            valid dataset_description.json file. See the BIDS-Derivatives
+            specification for details.
         '''
         paths = listify(path)
         deriv_dirs = []
@@ -378,6 +385,14 @@ def check_for_description(dir):
                         if check_for_description(sd):
                             deriv_dirs.append(sd)
 
+        if not deriv_dirs:
+            warnings.warn("Derivative indexing was enabled, but no valid "
+                          "derivatives datasets were found in any of the "
+                          "provided or default locations. Please make sure "
+                          "all derivatives datasets you intend to index "
+                          "contain a 'dataset_description.json' file, as "
+                          "described in the BIDS-derivatives specification.")
+
         for deriv in deriv_dirs:
             dd = os.path.join(deriv, 'dataset_description.json')
             with open(dd, 'r', encoding='utf-8') as ddfd:


Test patch:
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -448,6 +448,21 @@ def test_parse_file_entities_from_layout(layout_synthetic):
     assert target == layout.parse_file_entities(filename, config='derivatives')
 
 
+def test_deriv_indexing():
+    data_dir = join(get_test_data_path(), 'ds005')
+    deriv_dir = join(data_dir, 'derivatives', 'bbr')
+
+    # missing dataset_description.json
+    with pytest.warns(UserWarning):
+        layout = BIDSLayout(data_dir, derivatives=deriv_dir)
+
+    # Should work fine
+    deriv_dir = join(data_dir, 'derivatives', 'events')
+    layout = BIDSLayout(data_dir, derivatives=deriv_dir)
+    assert layout.get(scope='derivatives')
+    assert layout.get(scope='events')
+    assert not layout.get(scope='nonexistent')
+
 def test_add_config_paths():
     bids_dir = dirname(bids.__file__)
     bids_json = os.path.join(bids_dir, 'layout', 'config', 'bids.json')