Repository: bids-standard/pybids

Issue:
MacOS Finder `._*` hidden metadata files cause pybids to crash
## Example
<details><summary><code>UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte</code> … <code>while trying to decode JSON from file […]/._sub-PA069_ses-V1W1_task-poke_run-2_bold.json</code></summary>

```Python
Traceback (most recent call last):
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/bids/layout/index.py", line 303, in load_json
    return json.load(handle)
  File "/opt/conda/envs/sdcflows/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/opt/conda/envs/sdcflows/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/sdcflows/bin/sdcflows", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/sdcflows/cli/main.py", line 39, in main
    parse_args(argv)
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/sdcflows/cli/parser.py", line 281, in parse_args
    config.from_dict(vars(opts))
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/sdcflows/config.py", line 589, in from_dict
    execution.load(settings)
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/sdcflows/config.py", line 249, in load
    cls.init()
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/sdcflows/config.py", line 476, in init
    cls._layout = BIDSLayout(
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/bids/layout/layout.py", line 177, in __init__
    _indexer(self)
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/bids/layout/index.py", line 154, in __call__
    self._index_metadata()
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/bids/layout/index.py", line 415, in _index_metadata
    file_md.update(pl())
  File "/opt/conda/envs/sdcflows/lib/python3.10/site-packages/bids/layout/index.py", line 305, in load_json
    raise OSError(
OSError: Error occurred while trying to decode JSON from file /ocean/projects/med220004p/shared/data_raw/vannucci/bids_raw/sub-PA069/ses-V1W1/func/._sub-PA069_ses-V1W1_task-poke_run-2_bold.json
```

</details>

## Proposed Solution

I think [these types of files (`._*` and `.DS_Store`)](https://lists.apple.com/archives/applescript-users/2006/Jun/msg00180.html) can be safely ignored.

## Context

In analyzing someone else's read-only (to me) data, I hit this issue. I worked around it by creating a symlinked recreation of the data directory without the MacOS hidden metadata files, but I don't think that should have been necessary.

[bids-validator](https://github.com/bids-standard/bids-validator) raised errors and warnings for the dataset, but none related to these hidden metadata files as far as I can tell:

> ```
> 1: [ERR] Files with such naming scheme are not part of BIDS specification. This error is most commonly caused by typos in file names that make them not BIDS compatible. Please consult the specification and make sure your files are named correctly. If this is not a file naming issue (for example when including files not yet covered by the BIDS specification) you should include a ".bidsignore" file in your dataset (see https://github.com/bids-standard/bids-validator#bidsignore for details). Please note that derived (processed) data should be placed in /derivatives folder and source data (such as DICOMS or behavioural logs in proprietary formats) should be placed in the /sourcedata folder. (code: 1 - NOT_INCLUDED)
>		./sub-PA028/ses-V2W2/files.txt
>			Evidence: files.txt
>		./sub-PA070/ses-V1W1/anat/sub-PA070_ses-V2W2_acq-MPR_rec-vNavNorm_T1w.nii.gz
>			Evidence: sub-PA070_ses-V2W2_acq-MPR_rec-vNavNorm_T1w.nii.gz
> 2: [ERR] 'IntendedFor' field needs to point to an existing file. (code: 37 - INTENDED_FOR)
> 3: [ERR] You have to define 'TaskName' for this file. (code: 50 - TASK_NAME_MUST_DEFINE)
> 4: [ERR] Session label in the filename doesn't match with the path of the file. File seems to be saved in incorrect session directory. (code: 65 - SESSION_LABEL_IN_FILENAME_DOESNOT_MATCH_DIRECTORY)
> 5: [ERR] _T1w.nii[.gz] files must have exactly three dimensions.  (code: 95 - T1W_FILE_WITH_TOO_MANY_DIMENSIONS)
> ```
> 
> ```
> 1: [WARN] Task scans should have a corresponding events.tsv file. If this is a resting state scan you can ignore this warning or rename the task to include the word "rest". (code: 25 - EVENTS_TSV_MISSING)
> 2: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)
> 3: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
> 4: [WARN] NIfTI file's header field for pixel dimension information empty or too short. (code: 42 - NIFTI_PIXDIM)
> 5: [WARN] There are files in the /stimuli directory that are not utilized in any _events.tsv file. (code: 77 - UNUSED_STIMULUS)
> 6: [WARN] Tabular file contains custom columns not described in a data dictionary (code: 82 - CUSTOM_COLUMN_WITHOUT_DESCRIPTION)
> 7: [WARN] The onset of the last event is after the total duration of the corresponding scan. This design is suspiciously long.  (code: 85 - SUSPICIOUSLY_LONG_EVENT_DESIGN)
> 8: [WARN] Not all subjects contain the same sessions. (code: 97 - MISSING_SESSION)
> 9: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)
> 10: [WARN] The Authors field of dataset_description.json should contain an array of fields - with one author per field. This was triggered because there are no authors, which will make DOI registration from dataset metadata impossible. (code: 113 - NO_AUTHORS)
> ```

I get the same errors and warnings in my workaround data directory but avoid the issue in PyBIDS.

Hints from issue comments:
@effigies: I agree that we should generally ignore dotfiles.

Solution patch:
diff --git a/src/bids/layout/validation.py b/src/bids/layout/validation.py
index 50732811..cbf4def9 100644
--- a/src/bids/layout/validation.py
+++ b/src/bids/layout/validation.py
@@ -32,9 +32,12 @@
 
 DEFAULT_LOCATIONS_TO_IGNORE = {
     re.compile(r"^/(code|models|sourcedata|stimuli)"),
-    re.compile(r'/\.'),
 }
 
+ALWAYS_IGNORE = (
+    re.compile(r'/\.'),  # dotfiles should never be indexed
+)
+
 def absolute_path_deprecation_warning():
     warnings.warn("The absolute_paths argument will be removed from PyBIDS "
                   "in 0.14. You can easily access the relative path of "
@@ -156,9 +159,11 @@ def _sort_patterns(patterns, root):
 
 def validate_indexing_args(ignore, force_index, root):
     if ignore is None:
-        ignore = list(
-            DEFAULT_LOCATIONS_TO_IGNORE - set(force_index or [])
-        )
+        ignore = DEFAULT_LOCATIONS_TO_IGNORE - set(force_index or [])
+
+    ignore = list(ignore)
+
+    ignore.extend(ALWAYS_IGNORE)
 
     # root has already been validated to be a directory
     ignore = _sort_patterns(ignore, root)


Test patch:
diff --git a/src/bids/layout/tests/conftest.py b/src/bids/layout/tests/conftest.py
index d5959045..6451f893 100644
--- a/src/bids/layout/tests/conftest.py
+++ b/src/bids/layout/tests/conftest.py
@@ -1,4 +1,5 @@
 from os.path import join
+import shutil
 
 import pytest
 
@@ -101,3 +102,10 @@ def layout_synthetic(tests_dir, request, db_dir):
 def layout_synthetic_nodb(tests_dir, request, db_dir):
     path = tests_dir / 'data' / 'synthetic'
     return BIDSLayout(path, derivatives=True)
+
+
+@pytest.fixture
+def temporary_dataset(tmp_path, tests_dir):
+    path = tests_dir / 'data' / 'ds005'
+    shutil.copytree(path, tmp_path / 'ds005')
+    return tmp_path / 'ds005'
diff --git a/src/bids/layout/tests/test_layout.py b/src/bids/layout/tests/test_layout.py
index c0c7089a..737db4a1 100644
--- a/src/bids/layout/tests/test_layout.py
+++ b/src/bids/layout/tests/test_layout.py
@@ -1170,3 +1170,38 @@ def test_symlinks_in_path(tests_dir, tmp_path):
     os.symlink(src_sub, link_sub)
 
     assert "Subjects: 1 | Sessions: 2 | Runs: 2" in str(BIDSLayout(tmp_path / "7t_trt"))
+
+
+def test_ignore_dotfiles(temporary_dataset):
+    arbitrary_dotfile = temporary_dataset / '.dotfile'
+    ds_store = temporary_dataset / 'sub-01' / '.DS_Store'
+    osx_companion_file = temporary_dataset / '._task-mixedgamblestask_bold.json'
+
+    arbitrary_dotfile.touch()
+    ds_store.touch()
+    osx_companion_file.touch()
+
+    # Default behavior
+    layout = BIDSLayout(temporary_dataset, validate=False)
+    assert str(temporary_dataset / 'dataset_description.json') in layout.files
+    assert str(arbitrary_dotfile) not in layout.files
+    assert str(ds_store) not in layout.files
+    assert str(osx_companion_file) not in layout.files
+
+    # Explicit ignores do not disable dotfile filtering
+    indexer = BIDSLayoutIndexer(ignore=['some_ignore'])
+    layout = BIDSLayout(temporary_dataset, validate=False, indexer=indexer)
+    assert str(temporary_dataset / 'dataset_description.json') in layout.files
+    assert str(arbitrary_dotfile) not in layout.files
+    assert str(ds_store) not in layout.files
+    assert str(osx_companion_file) not in layout.files
+
+
+def test_empty_directory(temporary_dataset):
+    anat = temporary_dataset / 'sub-01' / 'anat'
+    shutil.rmtree(anat)
+    anat.mkdir()
+
+    layout = BIDSLayout(temporary_dataset)
+
+    assert layout.get(subject='01', datatype='anat') == []