Repository: nipy/heudiconv

Issue:
FileNotFoundError: [Errno 2] No such file or directory: '_task-rest_bold.json'
<!-- DO NOT DELETE THIS! 
This template is used to facilitate issue resolution.
All text in <!-> tags will not be displayed.
-->

### Summary
<!-- If you are having conversion troubles, please share as much
relevant information as possible. This includes, but is not limited to:
- log of conversion
- heuristic
-->I try to implement bids conversion with heudiconv on our local HPC using a singularity container ( heudiconv-0.9.0). As I will work with a large cohort (>2000 subjects) I am currently implementing a parallelization across nodes via SLURM and within each node with GNU parallel with a test dataset (5 times the same subject). In doing so, a test run fails with the following error.

```
`local:5/0/100%/0.0s 0: Traceback (most recent call last):
0:   File "/opt/miniconda-latest/bin/heudiconv", line 33, in <module>
0:     sys.exit(load_entry_point('heudiconv', 'console_scripts', 'heudiconv')())
0:   File "/src/heudiconv/heudiconv/cli/run.py", line 24, in main
0:     workflow(**kwargs)
0:   File "/src/heudiconv/heudiconv/main.py", line 351, in workflow
0:     grouping=grouping,)
0:   File "/src/heudiconv/heudiconv/convert.py", line 238, in prep_conversion
0:     getattr(heuristic, 'DEFAULT_FIELDS', {}))
0:   File "/src/heudiconv/heudiconv/bids.py", line 94, in populate_bids_templates
0:     populate_aggregated_jsons(path)
0:   File "/src/heudiconv/heudiconv/bids.py", line 126, in populate_aggregated_jsons
0:     json_ = load_json(fpath)
0:   File "/src/heudiconv/heudiconv/utils.py", line 177, in load_json
0:     with open(filename, 'r') as fp:
0: FileNotFoundError: [Errno 2] No such file or directory: '/bids/sub-ewgenia001/ses-1/func/sub-ewgenia001_ses-1_task-rest_bold.json'` 
```

Interestingly this affects only 4 of 5 subjects with the remaining one (seemingly always the first subject completing) without issues. All this sounds a little bit like race-condition as discussed in #362. However, as far as I understand a fix has been implemented and I am working with datalad creating a ephemeral clone of the dataset  on a scratch partition for each subject before applying heudiconv to it (scripts below). Maybe I am misunderstanding something but shouldn't the latter somehow address the race condition with the processes writing to a separate/cloned top-level file? 

Any input would be highly appreciated. Happy to provide further details.

[heudiconv_heuristic.txt](https://github.com/nipy/heudiconv/files/6776972/heudiconv_heuristic.txt)
[pipelines_parallelization.txt](https://github.com/nipy/heudiconv/files/6776973/pipelines_parallelization.txt) (batch script parallelizing pipelines_processing across subjects with GNU parallel)
[pipelines_processing.txt](https://github.com/nipy/heudiconv/files/6776974/pipelines_processing.txt)


### Platform details:

Choose one:
- [ x] Container (heudiconv 0.9.0)
<!-- If selected, please provide container name and tag"-->

Comments:

Patch:
diff --git a/heudiconv/bids.py b/heudiconv/bids.py
index 714e4a3..fa5d260 100644
--- a/heudiconv/bids.py
+++ b/heudiconv/bids.py
@@ -157,7 +157,7 @@ def populate_aggregated_jsons(path):
         # TODO: if we are to fix it, then old ones (without _acq) should be
         # removed first
         task = re.sub('.*_(task-[^_\.]*(_acq-[^_\.]*)?)_.*', r'\1', fpath)
-        json_ = load_json(fpath)
+        json_ = load_json(fpath, retry=100)
         if task not in tasks:
             tasks[task] = json_
         else:
@@ -212,7 +212,7 @@ def populate_aggregated_jsons(path):
             "CogAtlasID": "http://www.cognitiveatlas.org/task/id/TODO",
         }
         if op.lexists(task_file):
-            j = load_json(task_file)
+            j = load_json(task_file, retry=100)
             # Retain possibly modified placeholder fields
             for f in placeholders:
                 if f in j:
diff --git a/heudiconv/utils.py b/heudiconv/utils.py
index 3dc402f..eb2756b 100644
--- a/heudiconv/utils.py
+++ b/heudiconv/utils.py
@@ -14,6 +14,7 @@ from collections import namedtuple
 from glob import glob
 from subprocess import check_output
 from datetime import datetime
+from time import sleep
 
 from nipype.utils.filemanip import which
 
@@ -147,24 +148,40 @@ def write_config(outfile, info):
         fp.writelines(PrettyPrinter().pformat(info))
 
 
-def load_json(filename):
+def load_json(filename, retry=0):
     """Load data from a json file
 
     Parameters
     ----------
     filename : str
         Filename to load data from.
+    retry: int, optional
+        Number of times to retry opening/loading the file in case of
+        failure.  Code will sleep for 0.1 seconds between retries.
+        Could be used in code which is not sensitive to order effects
+        (e.g. like populating bids templates where the last one to
+        do it, would make sure it would be the correct/final state).
 
     Returns
     -------
     data : dict
     """
-    try:
-        with open(filename, 'r') as fp:
-            data = json.load(fp)
-    except JSONDecodeError:
-        lgr.error("{fname} is not a valid json file".format(fname=filename))
-        raise
+    assert retry >= 0
+    for i in range(retry + 1):  # >= 10 sec wait
+        try:
+            try:
+                with open(filename, 'r') as fp:
+                    data = json.load(fp)
+                    break
+            except JSONDecodeError:
+                lgr.error("{fname} is not a valid json file".format(fname=filename))
+                raise
+        except (JSONDecodeError, FileNotFoundError) as exc:
+            if i >= retry:
+                raise
+            lgr.warning("Caught %s. Will retry again", exc)
+            sleep(0.1)
+            continue
 
     return data
     


Test patch:
diff --git a/heudiconv/tests/test_utils.py b/heudiconv/tests/test_utils.py
index b8c8c16..f43a09b 100644
--- a/heudiconv/tests/test_utils.py
+++ b/heudiconv/tests/test_utils.py
@@ -2,6 +2,8 @@ import json
 import os
 import os.path as op
 
+import mock
+
 from heudiconv.utils import (
     get_known_heuristics_with_descriptions,
     get_heuristic_description,
@@ -77,6 +79,13 @@ def test_load_json(tmpdir, caplog):
     with pytest.raises(JSONDecodeError):
         load_json(str(invalid_json_file))
 
+    # and even if we ask to retry a few times -- should be the same
+    with pytest.raises(JSONDecodeError):
+        load_json(str(invalid_json_file), retry=3)
+
+    with pytest.raises(FileNotFoundError):
+        load_json("absent123not.there", retry=3)
+
     assert ifname in caplog.text
 
     # test valid json
@@ -87,6 +96,23 @@ def test_load_json(tmpdir, caplog):
     
     assert load_json(valid_json_file) == vcontent
 
+    calls = [0]
+    json_load = json.load
+
+    def json_load_patched(fp):
+        calls[0] += 1
+        if calls[0] == 1:
+            # just reuse bad file
+            load_json(str(invalid_json_file))
+        elif calls[0] == 2:
+            raise FileNotFoundError()
+        else:
+            return json_load(fp)
+
+    with mock.patch.object(json, 'load', json_load_patched):
+        assert load_json(valid_json_file, retry=3) == vcontent
+
+
 
 def test_get_datetime():
     """