Repository: mne-tools/mne-bids-pipeline

Issue:
BUG: Hash sidecars properly
@hoechenberger we have this comment regarding our hashing

https://github.com/mne-tools/mne-bids-pipeline/blob/eb37cf9682da60441ddfd55c6efc21d25fd0b0d2/mne_bids_pipeline/_run.py#L185

For this, do you think it makes sense to call `bids_path.find_matching_sidecar(...)` on all `BIDSPath` instances in `in_files` and add them to the list of files we hash (if such sidecars exist / it returns non-`None`)? If so, should I just iterate over extensions `.json` and `.tsv` to check, or would you recommend something else?

I assume this is the idea behind the comment at least. Then there will be specific other places we'll want to do this, like currently implemented in `_01_find_empty_room`, and probably needed but not implemented for the fiducial matching stuff in coreg (but maybe a BIDSPath to the T1 will already take care of this using the above idea?).

*EDIT*: There are several TODOs in `mne_bids_pipeline/_import_data.py` that might be relevant as well


Discussion:

Resolving patch:
diff --git a/mne_bids_pipeline/_run.py b/mne_bids_pipeline/_run.py
--- a/mne_bids_pipeline/_run.py
+++ b/mne_bids_pipeline/_run.py
@@ -165,25 +165,44 @@ def wrapper(*args, **kwargs):
             # If this is ever true, we'll need to improve the logic below
             assert not (unknown_inputs and force_run)
 
-            hashes = []
-            for k, v in in_files.items():
+            def hash_(k, v):
                 if isinstance(v, BIDSPath):
                     v = v.fpath
-                assert isinstance(v, pathlib.Path), f'{type(v)}: {v}'
+                assert isinstance(v, pathlib.Path), \
+                    f'Bad type {type(v)}: in_files["{k}"] = {v}'
                 assert v.exists(), f'missing in_files["{k}"] = {v}'
                 if self.memory_file_method == 'mtime':
                     this_hash = v.lstat().st_mtime
                 else:
                     assert self.memory_file_method == 'hash'  # guaranteed
                     this_hash = hash_file_path(v)
-                hashes.append((str(v), this_hash))
+                return (str(v), this_hash)
+
+            hashes = []
+            for k, v in in_files.items():
+                hashes.append(hash_(k, v))
+                # also hash the sidecar files if this is a BIDSPath and
+                # MNE-BIDS is new enough
+                if not hasattr(v, 'find_matching_sidecar'):
+                    continue
+                # from mne_bids/read.py
+                # The v.datatype is maybe not right, might need to use
+                # _infer_datatype like in read.py...
+                for suffix, extension in (('events', '.tsv'),
+                                          ('channels', '.tsv'),
+                                          ('electrodes', '.tsv'),
+                                          ('coordsystem', '.json'),
+                                          (v.datatype, '.json')):
+                    sidecar = v.find_matching_sidecar(
+                        suffix=suffix, extension=extension, on_error='ignore')
+                    if sidecar is None:
+                        continue
+                    hashes.append(hash_(k, sidecar))
 
             kwargs['cfg'] = copy.deepcopy(kwargs['cfg'])
             kwargs['cfg'].hashes = hashes
             del in_files  # will be modified by func call
 
-            # XXX we should also hash the sidecar files
-
             # Someday we could modify the joblib API to combine this with the
             # call (https://github.com/joblib/joblib/issues/1342), but our hash
             # should be plenty fast so let's not bother for now.


Test patch: