Repository: bids-standard/pybids

Issue:
Dangling symlinks to JSON files crashes BIDSLayout indexing
Reported in https://github.com/datalad/datalad/issues/4662.

With very large datalad datasets becoming more common, we should increasingly expect this. I would suggest that a JSON file is acceptable to be missing if no applicable data files are present.


Discussion:
cc @mih
Makes sense to me.
Just to preserve #632:

> In a DataLad dataset where I want to process only a single participant with fmriprep in an HTC environement, I need to either delete all unfulfilled annex symlinks to .json files, or fetch all content for them in order to pacify pybids.
> 
> In my concrete case I went for delete, because otherwise I would have to fetch 15k files that I do not need.
> 
> It would be nice, if pybids would have a way to not make this an issue. It would also be nice, if it would not search .git/
Are there by any chance any updates on this?
No, but it should not be a big lift. Need to make a test case and follow the traceback.
> Need to make a test case and follow the traceback

If helpful, I've pushed a real-life test case to https://github.com/adswa/workflow-testing (fmriprep-based).

You can clone the repository, and then rerun a single subject computation (e.g., ``datalad rerun 6085a99f40fd254480``). This recomputes an fmriprep ``--anat-only`` run on a single subject (pulls in quite a bit of data, 5GB) and fails like this:

```
datalad rerun 6085a99f40fd254480
[INFO   ] run commit 6085a99; (Compute sub-16) 
[INFO   ] Making sure inputs are available (this may take some time) 
Clone attempt:   0%|                                                 | 0.00/3.00 [00:00<?, ? Candidate locations/s]Enter passphrase for key '/home/adina/.ssh/id_rsa': 
[INFO   ] Scanning for unlocked files (this may take some time)                                                    
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore                                              
get(ok): inputs/data/sub-16/anat/sub-16_SWI.json (file) [from mddatasrc...]                                        
get(ok): inputs/data/sub-16/anat/sub-16_SWI_defacemask.nii.gz (file) [from mddatasrc...]                           
get(ok): inputs/data/sub-16/anat/sub-16_SWImag.nii.gz (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_SWIphase.nii.gz (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_T1w.json (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_T1w.nii.gz (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_T1w_defacemask.nii.gz (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_T2w.json (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_T2w.nii.gz (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat/sub-16_T2w_defacemask.nii.gz (file) [from mddatasrc...]
get(ok): inputs/data/sub-16/anat (directory)
[INFO   ] Scanning for unlocked files (this may take some time)                                                    
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore                                              
get(ok): code/pipeline/images/bids/bids-fmriprep--20.2.0.sing (file) [from datasets.datalad.org...]                
[INFO   ] == Command start (output follows) ===== ██████████████████████▉| 4.91G/4.92G [03:19<00:00, 26.1M Bytes/s]
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
  warnings.warn("The ability to pass arguments to BIDSLayout that control "
You are using fMRIPrep-20.2.0, and a newer version of fMRIPrep is available: 20.2.1.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 17, in main
    parse_args()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/parser.py", line 592, in parse_args
    config.from_dict(vars(opts))
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 612, in from_dict
    execution.load(settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 218, in load
    cls.init()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 448, in init
    re.compile(r"^\."),
  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 155, in __init__
    indexer(self)
  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 111, in __call__
    self._index_metadata()
  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 278, in _index_metadata
    with open(bf.path, 'r') as handle:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/merge/workflow-testing/inputs/data/sub-01/anat/sub-01_SWI.json'
CommandError: 'singularity run -B /tmp/merge/workflow-testing --cleanenv code/pipeline/images/bids/bids-fmriprep--20.2.0.sing inputs/data . participant --participant-label sub-16 --anat-only -w .git/tmp/wdir --fs-no-reconall --skip-bids-validation --fs-license-file code/license.txt' failed with exitcode 1 under /tmp/merge/workflow-testing

```

Resolving patch:
diff --git a/bids/layout/index.py b/bids/layout/index.py
--- a/bids/layout/index.py
+++ b/bids/layout/index.py
@@ -4,6 +4,7 @@
 import json
 from collections import defaultdict
 from pathlib import Path
+from functools import partial, lru_cache
 
 from bids_validator import BIDSValidator
 
@@ -264,6 +265,18 @@ def _index_metadata(self):
         # The payload is left empty for non-JSON files.
         file_data = {}
 
+        # Memoizing JSON loader
+        # Use as a function to allow lazy loading so only read JSON files
+        # if they correspond to data files that are indexed
+        @lru_cache(maxsize=None)
+        def load_json(path):
+            with open(path, 'r', encoding='utf-8') as handle:
+                try:
+                    return json.load(handle)
+                except json.JSONDecodeError as e:
+                    msg = f"Error occurred while trying to decode JSON from file {path}"
+                    raise IOError(msg) from e
+
         for bf in all_files:
             file_ents = bf.entities.copy()
             suffix = file_ents.pop('suffix', None)
@@ -274,16 +287,9 @@ def _index_metadata(self):
                 if key not in file_data:
                     file_data[key] = defaultdict(list)
 
+                payload = None
                 if ext == dot + 'json':
-                    with open(bf.path, 'r', encoding='utf-8') as handle:
-                        try:
-                            payload = json.load(handle)
-                        except json.JSONDecodeError as e:
-                            msg = ("Error occurred while trying to decode JSON"
-                                   " from file '{}'.".format(bf.path))
-                            raise IOError(msg) from e
-                else:
-                    payload = None
+                    payload = partial(load_json, bf.path)
 
                 to_store = (file_ents, payload, bf.path)
                 file_data[key][bf.dirname].append(to_store)
@@ -359,6 +365,10 @@ def create_association_pair(src, dst, kind, kind2=None):
             if not payloads:
                 continue
 
+            # Missing data files can tolerate absent metadata files,
+            # but we will try to load it anyway
+            virtual_datafile = not os.path.exists(bf.path)
+
             # Create DB records for metadata associations
             js_file = payloads[0][1]
             create_association_pair(js_file, bf.path, 'Metadata')
@@ -366,7 +376,15 @@ def create_association_pair(src, dst, kind, kind2=None):
             # Consolidate metadata by looping over inherited JSON files
             file_md = {}
             for pl, js_file in payloads[::-1]:
-                file_md.update(pl)
+                try:
+                    file_md.update(pl())
+                except FileNotFoundError:
+                    if not virtual_datafile:
+                        raise
+                    # Drop metadata if any files are missing
+                    # Otherwise missing overrides could give misleading metadata
+                    file_md = {}
+                    break
 
             # Create FileAssociation records for JSON inheritance
             n_pl = len(payloads)


Test patch: