Repository: mne-tools/mne-bids

Issue:
write_raw_bids() throws an IndexError when overwriting participants.tsv with missing "default" columns
### Description of the problem

Dear MNE-BIDS team, 

First of all, thank you for such a beautifully practical piece of software 👍 

Unfortunately, I've discovered a strange behaviour when converting a dataset. Some "default" data in the Participants file, such as **height** and **weight**, wasn't necessary to collect, therefore it made little sense to keep these empty columns in the `participants.tsv` file. So I deleted the columns after the conversion and added some extra columns from the experiment logs.

However, `write_raw_bids()` doesn't handle missing columns well. When I want to overwrite the data (that is, I use `write_raw_bids(..., overwrite=True)`), when the missing default columns are added to the data frame in the overwriting process, they are not populated with placeholders through the whole length of the dataset, which later throws an exception when trying to iterate through participants. You can find the error with the traceback below.

Adding custom columns to `participants.tsv` runs as expected and keeps custom columns in place.

I would appreciate if you take a quick look into it. Thanks!

### Steps to reproduce

```Python
import os.path as op
import pandas as pd
import mne
from mne_bids import write_raw_bids, BIDSPath

# I'll take the script folder as a default.
BIDS_ROOT = op.dirname(op.realpath(__file__))

# Download and read sample data. WILL DOWNLOAD 1.65GB OF DATA INTO THE FOLDER WITH THE SCRIPT
sample_data_folder = mne.datasets.sample.data_path(path=BIDS_ROOT)
sample_data_raw_file = op.join(sample_data_folder, 'MEG', 'sample', 'sample_audvis_raw.fif')
raw = mne.io.read_raw_fif(sample_data_raw_file)

# Let's say I have data for three subjects (I use the same sample data for all of them). The loop doesn't fail if
# executed twice on its own (no columns deleted; adding columns doesn't throw an error). It just rewrites the
# participant data, as supposed to.
for subject_id in range(1, 4):
    # Use mne-bids to expand the existing data
    bids_path = BIDSPath(subject=f'{subject_id:02}', task='test', root=BIDS_ROOT)
    write_raw_bids(raw, bids_path, overwrite=True)

# Say, we didn't measure weight and height of participants - we don't need this information.
participants = pd.read_csv(op.join(BIDS_ROOT, "participants.tsv"), sep='\t')
participants.drop(columns=['weight', 'height'], inplace=True)
participants.to_csv(op.join(BIDS_ROOT, "participants.tsv"), index=False, na_rep="n/a", sep="\t")

# Wait, we want to update info for one participant! Let's "convert" again with overwriting the data.
subject_id = 2
bids_path = BIDSPath(subject=f'{subject_id:02}', task='sample', root=BIDS_ROOT)

# The next line of code throws an error:
write_raw_bids(raw, bids_path, overwrite=True)
```


### Expected results

`participants.tsv` overwritten with deleted columns added again

### Actual results

Missing columns are added to the `participants.tsv` data frame but not populated with values 

Traceback (with shortened paths):
```
Traceback (most recent call last):
  File "bug/bug.py", line 33, in <module>
    write_raw_bids(raw, bids_path, overwrite=True)
  File "<decorator-gen-623>", line 12, in write_raw_bids
  File "mne_bids/write.py", line 1801, in write_raw_bids
    _participants_tsv(
  File "mne_bids/write.py", line 448, in _participants_tsv
    data = _combine_rows(orig_data, data, 'participant_id')
  File "mne_bids/tsv_handler.py", line 45, in _combine_rows
    data[key] = [data[key][n_rows - 1 - idx] for idx in idxs]
  File "mne_bids/tsv_handler.py", line 45, in <listcomp>
    data[key] = [data[key][n_rows - 1 - idx] for idx in idxs]
IndexError: list index out of range
```

### Additional information

MacBook Air 2020 with Apple M1 chip

```
Platform:         macOS-13.0.1-arm64-arm-64bit
Python:           3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ]
Executable:       /Users/alisenz/opt/anaconda3/envs/mne/bin/python
CPU:              arm: 8 cores
Memory:           8.0 GB
mne:              1.2.0
numpy:            1.23.1 {OpenBLAS 0.3.21 with 8 threads}
scipy:            1.9.1
matplotlib:       3.6.1Backend MacOSX is interactive backend. Turning interactive mode on.
 {backend=MacOSX}
sklearn:          1.1.2
numba:            0.56.2
nibabel:          4.0.2
nilearn:          0.9.2
dipy:             1.5.0
openmeeg:         Not found
cupy:             Not found
pandas:           1.5.0
pyvista:          0.36.1 {OpenGL 4.1 Metal - 83 via Apple M1}
pyvistaqt:        0.9.0
ipyvtklink:       0.2.3
vtk:              9.2.2
qtpy:             2.2.1 {PyQt5=5.15.6}
ipympl:           Not found
pyqtgraph:        0.13.1
pooch:            v1.6.0
mne_bids:         0.11
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   0.4.0
mne_connectivity: Not found
mne_icalabel:     Not found
```
write_raw_bids() throws an IndexError when overwriting participants.tsv with missing "default" columns
### Description of the problem

Dear MNE-BIDS team, 

First of all, thank you for such a beautifully practical piece of software 👍 

Unfortunately, I've discovered a strange behaviour when converting a dataset. Some "default" data in the Participants file, such as **height** and **weight**, wasn't necessary to collect, therefore it made little sense to keep these empty columns in the `participants.tsv` file. So I deleted the columns after the conversion and added some extra columns from the experiment logs.

However, `write_raw_bids()` doesn't handle missing columns well. When I want to overwrite the data (that is, I use `write_raw_bids(..., overwrite=True)`), when the missing default columns are added to the data frame in the overwriting process, they are not populated with placeholders through the whole length of the dataset, which later throws an exception when trying to iterate through participants. You can find the error with the traceback below.

Adding custom columns to `participants.tsv` runs as expected and keeps custom columns in place.

I would appreciate if you take a quick look into it. Thanks!

### Steps to reproduce

```Python
import os.path as op
import pandas as pd
import mne
from mne_bids import write_raw_bids, BIDSPath

# I'll take the script folder as a default.
BIDS_ROOT = op.dirname(op.realpath(__file__))

# Download and read sample data. WILL DOWNLOAD 1.65GB OF DATA INTO THE FOLDER WITH THE SCRIPT
sample_data_folder = mne.datasets.sample.data_path(path=BIDS_ROOT)
sample_data_raw_file = op.join(sample_data_folder, 'MEG', 'sample', 'sample_audvis_raw.fif')
raw = mne.io.read_raw_fif(sample_data_raw_file)

# Let's say I have data for three subjects (I use the same sample data for all of them). The loop doesn't fail if
# executed twice on its own (no columns deleted; adding columns doesn't throw an error). It just rewrites the
# participant data, as supposed to.
for subject_id in range(1, 4):
    # Use mne-bids to expand the existing data
    bids_path = BIDSPath(subject=f'{subject_id:02}', task='test', root=BIDS_ROOT)
    write_raw_bids(raw, bids_path, overwrite=True)

# Say, we didn't measure weight and height of participants - we don't need this information.
participants = pd.read_csv(op.join(BIDS_ROOT, "participants.tsv"), sep='\t')
participants.drop(columns=['weight', 'height'], inplace=True)
participants.to_csv(op.join(BIDS_ROOT, "participants.tsv"), index=False, na_rep="n/a", sep="\t")

# Wait, we want to update info for one participant! Let's "convert" again with overwriting the data.
subject_id = 2
bids_path = BIDSPath(subject=f'{subject_id:02}', task='sample', root=BIDS_ROOT)

# The next line of code throws an error:
write_raw_bids(raw, bids_path, overwrite=True)
```


### Expected results

`participants.tsv` overwritten with deleted columns added again

### Actual results

Missing columns are added to the `participants.tsv` data frame but not populated with values 

Traceback (with shortened paths):
```
Traceback (most recent call last):
  File "bug/bug.py", line 33, in <module>
    write_raw_bids(raw, bids_path, overwrite=True)
  File "<decorator-gen-623>", line 12, in write_raw_bids
  File "mne_bids/write.py", line 1801, in write_raw_bids
    _participants_tsv(
  File "mne_bids/write.py", line 448, in _participants_tsv
    data = _combine_rows(orig_data, data, 'participant_id')
  File "mne_bids/tsv_handler.py", line 45, in _combine_rows
    data[key] = [data[key][n_rows - 1 - idx] for idx in idxs]
  File "mne_bids/tsv_handler.py", line 45, in <listcomp>
    data[key] = [data[key][n_rows - 1 - idx] for idx in idxs]
IndexError: list index out of range
```

### Additional information

MacBook Air 2020 with Apple M1 chip

```
Platform:         macOS-13.0.1-arm64-arm-64bit
Python:           3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ]
Executable:       /Users/alisenz/opt/anaconda3/envs/mne/bin/python
CPU:              arm: 8 cores
Memory:           8.0 GB
mne:              1.2.0
numpy:            1.23.1 {OpenBLAS 0.3.21 with 8 threads}
scipy:            1.9.1
matplotlib:       3.6.1Backend MacOSX is interactive backend. Turning interactive mode on.
 {backend=MacOSX}
sklearn:          1.1.2
numba:            0.56.2
nibabel:          4.0.2
nilearn:          0.9.2
dipy:             1.5.0
openmeeg:         Not found
cupy:             Not found
pandas:           1.5.0
pyvista:          0.36.1 {OpenGL 4.1 Metal - 83 via Apple M1}
pyvistaqt:        0.9.0
ipyvtklink:       0.2.3
vtk:              9.2.2
qtpy:             2.2.1 {PyQt5=5.15.6}
ipympl:           Not found
pyqtgraph:        0.13.1
pooch:            v1.6.0
mne_bids:         0.11
mne_nirs:         Not found
mne_features:     Not found
mne_qt_browser:   0.4.0
mne_connectivity: Not found
mne_icalabel:     Not found
```


Discussion:
Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️


Resolving patch:
diff --git a/mne_bids/write.py b/mne_bids/write.py
--- a/mne_bids/write.py
+++ b/mne_bids/write.py
@@ -413,34 +413,30 @@ def _participants_tsv(raw, subject_id, fname, overwrite=False):
         # if the subject data provided is different to the currently existing
         # data and overwrite is not True raise an error
         if (sid_included and not exact_included) and not overwrite:
-            raise FileExistsError(f'"{subject_id}" already exists in '  # noqa: E501 F821
+            raise FileExistsError(f'"{subject_id}" already exists in '
                                   f'the participant list. Please set '
                                   f'overwrite to True.')
 
-        # Append any columns the original data did not have
-        # that mne-bids is trying to write. This handles
-        # the edge case where users write participants data for
-        # a subset of `hand`, `age` and `sex`.
+        # Append any columns the original data did not have, and fill them with
+        # n/a's.
         for key in data.keys():
             if key in orig_data:
                 continue
 
-            # add 'n/a' if any missing columns
-            orig_data[key] = ['n/a'] * len(next(iter(data.values())))
+            orig_data[key] = ['n/a'] * len(orig_data['participant_id'])
 
         # Append any additional columns that original data had.
         # Keep the original order of the data by looping over
         # the original OrderedDict keys
-        col_name = 'participant_id'
         for key in orig_data.keys():
             if key in data:
                 continue
 
             # add original value for any user-appended columns
             # that were not handled by mne-bids
-            p_id = data[col_name][0]
-            if p_id in orig_data[col_name]:
-                row_idx = orig_data[col_name].index(p_id)
+            p_id = data['participant_id'][0]
+            if p_id in orig_data['participant_id']:
+                row_idx = orig_data['participant_id'].index(p_id)
                 data[key] = [orig_data[key][row_idx]]
 
         # otherwise add the new data as new row


Test patch:
diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml
--- a/.github/workflows/unit_tests.yml
+++ b/.github/workflows/unit_tests.yml
@@ -154,7 +154,7 @@ jobs:
     - name: Install MNE (stable)
       if: "matrix.mne-version == 'mne-stable'"
       run: |
-        git clone --depth 1 --single-branch --branch maint/1.2 https://github.com/mne-tools/mne-python.git
+        git clone --depth 1 --single-branch --branch maint/1.3 https://github.com/mne-tools/mne-python.git
         python -m pip install -e ./mne-python
 
     - name: Install MNE (main)
diff --git a/mne_bids/tests/test_write.py b/mne_bids/tests/test_write.py
--- a/mne_bids/tests/test_write.py
+++ b/mne_bids/tests/test_write.py
@@ -202,21 +202,26 @@ def test_write_participants(_bids_validate, tmp_path):
     data = _from_tsv(participants_tsv)
     assert data['age'][data['participant_id'].index('sub-01')] == '1'
 
-    # if we remove some fields, they should be filled back in upon
-    # re-writing with 'n/a'
+    # Removing some columns from participants.tsv should not prevent us from
+    # writing additional participants later on. Before running this test,
+    # ensure we have at least 2 participants in the dataset already – this is
+    # a regression test for GH-1104.
+    bids_path.update(subject='02')
+    write_raw_bids(raw, bids_path, verbose=False)
     data = _from_tsv(participants_tsv)
     data.pop('hand')
     _to_tsv(data, participants_tsv)
 
     # write in now another subject
-    bids_path.update(subject='02')
+    bids_path.update(subject='03')
     write_raw_bids(raw, bids_path, verbose=False)
     data = _from_tsv(participants_tsv)
 
-    # hand should have been written properly with now 'n/a' for sub-01
-    # but 'L' for sub-02
+    # hand should have been written properly with now 'n/a' for sub-01 and
+    # sub-03, but 'L' for sub-03
     assert data['hand'][data['participant_id'].index('sub-01')] == 'n/a'
-    assert data['hand'][data['participant_id'].index('sub-02')] == 'L'
+    assert data['hand'][data['participant_id'].index('sub-02')] == 'n/a'
+    assert data['hand'][data['participant_id'].index('sub-03')] == 'L'
 
     # check to make sure participant data is overwritten, but keeps the fields
     # if there are extra fields that were user defined