Repository: bids-standard/pybids

Issue:
Provide more informative errors whenever reading of some .json file fails
I was left with a backtrace without a hint on actually which json file reading has failed.  I had to write minimal script and run it via pdb to actually get an idea:

```shell
(git)smaug:/mnt/btrfs/datasets/datalad/crawl/openneuro/ds000224[master]git
$> python -m pdb /tmp/1.py
> /tmp/1.py(1)<module>()
-> from bids import BIDSLayout, __version__; print(__version__); b=BIDSLayout(".", derivatives=False); b.get_collections(level="dataset")
(Pdb) r
0.9.3+3.g6f77de6
--Return--
> /tmp/1.py(1)<module>()->None
-> from bids import BIDSLayout, __version__; print(__version__); b=BIDSLayout(".", derivatives=False); b.get_collections(level="dataset")
(Pdb) c
Traceback (most recent call last):
  File "/usr/lib/python3.7/pdb.py", line 1701, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.7/pdb.py", line 1570, in _runscript
    self.run(statement)
  File "/usr/lib/python3.7/bdb.py", line 585, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/tmp/1.py", line 1, in <module>
    from bids import BIDSLayout, __version__; print(__version__); b=BIDSLayout(".", derivatives=False); b.get_collections(level="dataset")
  File "/home/yoh/proj/datalad/datalad-crawler/venvs/dev3/lib/python3.7/site-packages/bids/layout/layout.py", line 212, in __init__
    indexer.index_metadata()
  File "/home/yoh/proj/datalad/datalad-crawler/venvs/dev3/lib/python3.7/site-packages/bids/layout/index.py", line 208, in index_metadata
    payload = json.load(handle)
  File "/usr/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 6 column 5 (char 214)
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /usr/lib/python3.7/json/decoder.py(353)raw_decode()
-> obj, end = self.scan_once(s, idx)
(Pdb) up
> /usr/lib/python3.7/json/decoder.py(337)decode()
-> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
(Pdb)
> /usr/lib/python3.7/json/__init__.py(348)loads()
-> return _default_decoder.decode(s)
(Pdb)
> /usr/lib/python3.7/json/__init__.py(296)load()
-> parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
(Pdb)
> /home/yoh/proj/datalad/datalad-crawler/venvs/dev3/lib/python3.7/site-packages/bids/layout/index.py(208)index_metadata()
-> payload = json.load(handle)
(Pdb) p handle
<_io.TextIOWrapper name='/mnt/btrfs/datasets/datalad/crawl/openneuro/ds000224/sourcedata/phenotype/BAS_BIS.json' mode='r' encoding='UTF-8'>
```

so probably try/except around
```
packages/bids/layout/index.py", line 208, in index_metadata
    payload = json.load(handle)
```
with a more informative exception message would be of help here.

NB why is it readying from `sourcedata/` -- is that intended?

`jq` confirms bad .json:
```
$> jq . < sourcedata/phenotype/BAS_BIS.json
parse error: Invalid numeric literal at line 6, column 20
```

and the reason is the use of a fancy unicode `“` and `”` instead of a simple `"`
```shell
$> cat sourcedata/phenotype/BAS_BIS.json
{
    "MeasurementToolMetadata": {
        "Description": "Behavioral Approach System / Behavioral Inhibition System scales",
        "TermURL": "http://www.cognitiveatlas.org/term/id/trm_56a9137d9dce1"
    },
    “BAS_drive”: {
        "Description": “Persistent pursuit of goals”,
        "Units": “summed Likert scores”
    },
    “BAS_fun_seeking”: {
        "Description": “Desire for new rewards”,
        "Units": “summed Likert scores”
    },
    “BAS_reward_responsiveness”: {
        "Description": “Positive response to reward”,
        "Units": “summed Likert scores”
    },
    “BIS”: {
        "Description": “Sensitivity to negative events / anxiety”,
        "Units": “summed Likert scores”
    }
}%         
```

@effigies -- may be worth a wishlist issue against bids-validator to scream on any illegit .json file found around?



Discussion:
Sorry... Comcast went out around 4 yesterday.

----

I agree that we should wrap the `json.load` call with a `try`/`except`, and that the validator should complain about unicode quotes or other issues. I'm a little surprised it doesn't...

> NB why is it readying from `sourcedata/` -- is that intended?

I don't think we're supposed to index it by default... @tyarkoni or @adelavega would know better though.
It should be ignored by default,  yes...

https://github.com/bids-standard/pybids/blob/6f77de6b3474fb95c2a7a2c01a3275891be43f16/bids/layout/layout.py#L164
Unless I'm missing something, I see no reference anywhere in the code base to the variable `_default_ignore`. Looks like `self.ignore` defaults to `[]` now. 

Resolving patch:
diff --git a/bids/layout/index.py b/bids/layout/index.py
--- a/bids/layout/index.py
+++ b/bids/layout/index.py
@@ -205,7 +205,12 @@ def index_metadata(self):
 
                 if ext == 'json':
                     with open(bf.path, 'r') as handle:
-                        payload = json.load(handle)
+                        try:
+                            payload = json.load(handle)
+                        except json.JSONDecodeError as e:
+                            msg = ("Error occurred while trying to decode JSON"
+                                   " from file '{}'.".format(bf.path))
+                            six.raise_from(IOError(msg), e)
                 else:
                     payload = None
 
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -132,7 +132,9 @@ class BIDSLayout(object):
             absolute path, or be relative to the BIDS project root. If an
             SRE_Pattern is passed, the contained regular expression will be
             matched against the full (absolute) path of all files and
-            directories.
+            directories. By default, indexing ignores all files in 'code/',
+            'stimuli/', 'sourcedata/', 'models/', and any hidden files/dirs
+            beginning with '.' at root level.
         force_index (str, SRE_Pattern, list): Path(s) to forcibly index in the
             BIDSLayout, even if they would otherwise fail validation. See the
             documentation for the ignore argument for input format details.
@@ -161,8 +163,8 @@ class BIDSLayout(object):
             indexing will be faster).
     """
 
-    _default_ignore = {"code", "stimuli", "sourcedata", "models",
-                       "derivatives", re.compile(r'^\.')}
+    _default_ignore = ("code", "stimuli", "sourcedata", "models",
+                       re.compile(r'^\.'))
 
     def __init__(self, root, validate=True, absolute_paths=True,
                  derivatives=False, config=None, sources=None, ignore=None,
@@ -184,6 +186,9 @@ def __init__(self, root, validate=True, absolute_paths=True,
         # Do basic BIDS validation on root directory
         self._validate_root()
 
+        if ignore is None:
+            ignore = self._default_ignore
+
         # Instantiate after root validation to ensure os.path.join works
         self.ignore = [os.path.abspath(os.path.join(self.root, patt))
                        if isinstance(patt, six.string_types) else patt


Test patch:
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -247,6 +247,23 @@ def test_get_return_sorted(layout_7t_trt):
     assert files == paths
 
 
+def test_ignore_files(layout_ds005):
+    data_dir = join(get_test_data_path(), 'ds005')
+    target1 = join(data_dir, 'models', 'ds-005_type-test_model.json')
+    target2 = join(data_dir, 'models', 'extras', 'ds-005_type-test_model.json')
+    layout1 = BIDSLayout(data_dir, validate=False)
+    assert target1 not in layout_ds005.files
+    assert target1 not in layout1.files
+    assert target2 not in layout1.files
+    # now the models/ dir should show up, because passing ignore explicitly
+    # overrides the default - but 'model/extras/' should still be ignored because
+    # of the regex.
+    ignore = [re.compile('xtra'), 'dummy']
+    layout2 = BIDSLayout(data_dir, validate=False, ignore=ignore)
+    assert target1 in layout2.files
+    assert target2 not in layout2.files
+
+
 def test_force_index(layout_ds005):
     data_dir = join(get_test_data_path(), 'ds005')
     target= join(data_dir, 'models', 'ds-005_type-test_model.json')
@@ -570,4 +587,4 @@ def test_get_with_wrong_dtypes(layout_7t_trt):
     assert (l.get(run=1) == l.get(run='1') == l.get(run=np.int64(1)) ==
             l.get(run=[1, '15']))
     assert not l.get(run='not_numeric')
-    assert l.get(session=1) == l.get(session='1')
\ No newline at end of file
+    assert l.get(session=1) == l.get(session='1')