Repository: mne-tools/mne-bids

Issue:
scans.tsv for split files
Hi, 
I was running the bids-validator on my data and stumbled upon a BIDS-incompatibility. The information in the `scans.tsv` does not match with the actual scans present in the directory, when the  scans consist of split files. To illustrate, Ihere is the `scans.tsv` file:
~~~
filename	                                        acq_time
meg/sub-01_ses-02_task-ml_meg.fif	                2020-10-23T12:20:27
meg/sub-01_ses-02_task-rdm_meg.fif	                2020-10-23T11:25:01
meg/sub-01_ses-02_task-restingstate_meg.fif	        2020-10-23T10:30:32
meg/sub-01_ses-02_task-vbm_meg.fif	                2020-10-23T10:49:34
~~~

 and a screenshot of the corresponding directory:
![image](https://user-images.githubusercontent.com/8819465/106287432-10dc3880-6247-11eb-9b55-255eb5e358e6.png)

For all the split files, the bids-validator had following issue: 
~~~
Error 2: [Code 129] SCANS_FILENAME_NOT_MATCH_DATASET
The filename in scans.tsv file does not match what is present in the BIDS dataset.
~~~

Not sure whether this is or the bids-validator-repo would be the place to report this. If more appropriate there, I could move the issue. 

Thanks


Discussion:
Thanks for the report! This is an interesting edge case indeed. @sappelhoff do you know whether the spec defines what to do in a situation like this? Otherwise I can have a look myself later
Thanks for raising this issue @eort I think this is a good place to discuss it.

I guess we never discussed whether split parts of a scan are to be considered "one scan", or "multiple scans" --> I cannot find anything about this in the specification ... but the validator apparently expects each "split" that is in the data to also be listed in scans.tsv --> whereas in MNE-BIDS we go the other way (split parts are still one scan, albeit split across files for technical limitations, so only one scan is listed)

Overall I think it would be *easier* to consider each split part a separate scan file ... i.e., adjust MNE-BIDS ... it may also be a better overview for someone inspecting the scans.tsv.

However, I can also see it the other way around ... it's sensible to think of the split parts as *one single scan*.

Either way, there are these options:
1. fix mne-bids to comply with the validator and move on with our lives
2. start a discussion on the bids-specification repo which road we wanna go (A or B), and then:
    3. update the specification
    4. update EITHER validator or mne-bids


links to the parts in the spec:

- https://bids-specification.readthedocs.io/en/latest/99-appendices/06-meg-file-formats.html#split-files
- https://bids-specification.readthedocs.io/en/latest/99-appendices/09-entities.html#split
> * fix mne-bids to comply with the validator and move on with our lives

I think that's the best approach for now. Looking at `scants.tsv` will always easily reveal whether a file listed there is actually a "complete" scan or simply a partial scan (split file), as the split file naming is standardized.
I agree. Also, the split scans will have the same `acq-time`, right?

> I agree. Also, the split scans will have the same acq-time, right?

I would make that a requirement, *all* metadata should be the same, except that in the name, they have `split-<index>`
@eort do you want to raise an issue about this in the BIDS specification repo? https://github.com/bids-standard/bids-specification/

We should clarify the spec with what we agreed on here. (And get feedback from others, who may or may not think differently)
> @eort do you want to raise an issue about this in the BIDS specification repo? 

Will do.

But to be clear, the plan is to change the way `_scans.tsv` is being written (so separately listing split files), and to suggest at the BIDS repo, that this procedure is made explicit?


Exactly:

1. Make a pull request (PR) to mne-bids, so that all `_split-*` files are also listed in `scans.tsv`
1. Make a PR to https://github.com/bids-standard/bids-specification/ adding a note in an appropriate place, that different `_split-*` files should (MUST) still be listed as individual scans in scans.tsv

Let me know if you need help with this - especially if you have never made a PR --> I don't know what your background is.

The sections to look at in the BIDS specification are these ones (search for "split"):
- https://bids-specification.readthedocs.io/en/latest/04-modality-specific-files/02-magnetoencephalography.html#meg-recording-data
- https://bids-specification.readthedocs.io/en/latest/99-appendices/06-meg-file-formats.html#split-files
- https://bids-specification.readthedocs.io/en/latest/99-appendices/09-entities.html#split
Ok, I'll have a look over the weekend. 


Resolving patch:
diff --git a/mne_bids/read.py b/mne_bids/read.py
--- a/mne_bids/read.py
+++ b/mne_bids/read.py
@@ -9,6 +9,7 @@
 import os.path as op
 import glob
 import json
+import re
 from datetime import datetime, timezone
 
 import numpy as np
@@ -192,12 +193,7 @@ def _handle_scans_reading(scans_fname, raw, bids_path, verbose=False):
     scans_tsv = _from_tsv(scans_fname)
     fname = bids_path.fpath.name
 
-    if '_split-' in fname:
-        # for split files, scans only stores the filename without ``split``
-        extension = bids_path.fpath.suffix
-        bids_path.update(split=None, extension=extension)
-        fname = bids_path.basename
-    elif fname.endswith('.pdf'):
+    if fname.endswith('.pdf'):
         # for BTI files, the scan is an entire directory
         fname = fname.split('.')[0]
 
@@ -209,6 +205,21 @@ def _handle_scans_reading(scans_fname, raw, bids_path, verbose=False):
     acq_times = scans_tsv['acq_time']
     row_ind = fnames.index(data_fname)
 
+    # check whether all split files have the same acq_time
+    # and throw an error if they don't
+    if '_split-' in fname:
+        split_idx = fname.find('split-')
+        pattern = re.compile(bids_path.datatype + '/' +
+                             bids_path.basename[:split_idx] +
+                             r'split-\d+_' + bids_path.datatype +
+                             bids_path.fpath.suffix)
+        split_fnames = list(filter(pattern.match, fnames))
+        split_acq_times = []
+        for split_f in split_fnames:
+            split_acq_times.append(acq_times[fnames.index(split_f)])
+        if len(set(split_acq_times)) != 1:
+            raise ValueError("Split files must have the same acq_time.")
+
     # extract the acquisition time from scans file
     acq_time = acq_times[row_ind]
     if acq_time != 'n/a':
@@ -542,6 +553,7 @@ def read_raw_bids(bids_path, extra_params=None, verbose=True):
         suffix='scans', extension='.tsv',
         root=bids_path.root
     ).fpath
+
     if scans_fname.exists():
         raw = _handle_scans_reading(scans_fname, raw, bids_path,
                                     verbose=verbose)
diff --git a/mne_bids/write.py b/mne_bids/write.py
--- a/mne_bids/write.py
+++ b/mne_bids/write.py
@@ -413,8 +413,28 @@ def _scans_tsv(raw, raw_fname, fname, overwrite=False, verbose=True):
         # for MNE >= v0.20
         acq_time = meas_date.strftime('%Y-%m-%dT%H:%M:%S.%fZ')
 
-    data = OrderedDict([('filename', ['%s' % raw_fname.replace(os.sep, '/')]),
-                        ('acq_time', [acq_time])])
+    # for fif files check whether raw file is likely to be split
+    raw_fnames = [raw_fname]
+    if raw_fname.endswith('.fif'):
+        # check whether fif files were split when saved
+        # use the files in the target directory what should be written
+        # to scans.tsv
+        datatype, basename = raw_fname.split(os.sep)
+        raw_dir = op.join(op.dirname(fname), datatype)
+        raw_files = [f for f in os.listdir(raw_dir) if f.endswith('.fif')]
+        if basename not in raw_files:
+            raw_fnames = []
+            split_base = basename.replace('_meg.fif', '_split-{}')
+            for raw_f in raw_files:
+                if len(raw_f.split('_split-')) == 2:
+                    if split_base.format(raw_f.split('_split-')[1]) == raw_f:
+                        raw_fnames.append(op.join(datatype, raw_f))
+            raw_fnames.sort()
+
+    data = OrderedDict(
+        [('filename', ['{:s}'.format(raw_f.replace(os.sep, '/'))
+          for raw_f in raw_fnames]),
+            ('acq_time', [acq_time] * len(raw_fnames))])
 
     if os.path.exists(fname):
         orig_data = _from_tsv(fname)


Test patch:
diff --git a/mne_bids/tests/test_read.py b/mne_bids/tests/test_read.py
--- a/mne_bids/tests/test_read.py
+++ b/mne_bids/tests/test_read.py
@@ -769,10 +769,14 @@ def test_bads_reading():
 @pytest.mark.filterwarnings(warning_str['channel_unit_changed'])
 def test_write_read_fif_split_file():
     """Test split files are read correctly."""
+    # load raw test file, extend it to be larger than 2gb, and save it
     bids_root = _TempDir()
     tmp_dir = _TempDir()
     bids_path = _bids_path.copy().update(root=bids_root, datatype='meg')
     raw = _read_raw_fif(raw_fname, verbose=False)
+    bids_path.update(acquisition=None)
+    write_raw_bids(raw, bids_path, verbose=False)
+    bids_path.update(acquisition='01')
     n_channels = len(raw.ch_names)
     n_times = int(2.2e9 / (n_channels * 4))  # enough to produce a split
     data = np.empty((n_channels, n_times), dtype=np.float32)
@@ -782,9 +786,9 @@ def test_write_read_fif_split_file():
     raw = _read_raw_fif(big_fif_fname, verbose=False)
     write_raw_bids(raw, bids_path, verbose=False)
 
+    # test whether split raw files were read correctly
     raw1 = read_raw_bids(bids_path=bids_path)
     assert 'split-01' in str(bids_path.fpath)
-
     bids_path.update(split='01')
     raw2 = read_raw_bids(bids_path=bids_path)
     bids_path.update(split='02')
@@ -793,6 +797,33 @@ def test_write_read_fif_split_file():
     assert len(raw) == len(raw2)
     assert len(raw) > len(raw3)
 
+    # check that split files both appear in scans.tsv
+    scans_tsv = BIDSPath(
+        subject=subject_id, session=session_id,
+        suffix='scans', extension='.tsv',
+        root=bids_root)
+    scan_data = _from_tsv(scans_tsv)
+    scan_fnames = scan_data['filename']
+    scan_acqtime = scan_data['acq_time']
+
+    assert len(scan_fnames) == 3
+    assert 'split-01' in scan_fnames[0] and 'split-02' in scan_fnames[1]
+    # check that the acq_times in scans.tsv are the same
+    assert scan_acqtime[0] == scan_acqtime[1]
+    # check the recordings are in the correct order
+    assert raw2.first_time < raw3.first_time
+
+    # check whether non-matching acq_times are caught
+    scan_data['acq_time'][0] = scan_acqtime[0].split('.')[0]
+    _to_tsv(scan_data, scans_tsv)
+    with pytest.raises(ValueError,
+                       match='Split files must have the same acq_time.'):
+        read_raw_bids(bids_path)
+
+    # reset scans.tsv file for downstream tests
+    scan_data['acq_time'][0] = scan_data['acq_time'][1]
+    _to_tsv(scan_data, scans_tsv)
+
 
 @pytest.mark.filterwarnings(warning_str['channel_unit_changed'])
 def test_ignore_exclude_param():