Repository: mne-tools/mne-python

Issue:
Enable mne.concatenate_raws(raws) to handle varying bad channels in raws
### Describe the new feature or enhancement

Maximization of available data is very important for many machine learning tasks. Therefore it is better to concatenate three similar recordings instead of using only one. However, what if the bad channels don't match? Currently, there are only two easy options:

1) Mark the bad channel as bad in all recordings. This means throwing away valuable data from that channel (for example in the case of deep brain stimulation LFPs where channel interpolation is difficult). 
2) Ignore the recording with bad channels. This also means throwing away valuable data from the other channels.

### Describe your proposed implementation

I would suggest to enhance `mne.concatenate_raws` to add the option of setting bad channels as `np.nan`. In that case, the maximum of available data can be used since most algorithms handle nan-values well.

For example, 
`mne.concatenate_raws(raws, preload=None, events_list=None, *, on_mismatch='raise', verbose=None)`
with the option `on_mismatch='set_nan'`.  Or to add another kwarg `on_bads_mismatch='set_nan'`.

### Describe possible alternatives

[Example code for enhancement](https://github.com/mne-tools/mne-python/files/10747126/mne_concatenate_channels.pdf)


### Additional context

_No response_


Discussion:
> I would suggest to enhance mne.concatenate_raws to add the option of setting bad channels as np.nan. In that case, the maximum of available data can be used since most algorithms handle nan-values well.

Rather than doing this it might be easier and more general to make a function like `mne.preprocessing.bads_to_nan` (the name is not good but you get the idea) that takes any bad channels and sets the data to NaN, maybe also clearing `info['bads']` (though this would just be one more line for the end user). Then `concatenate_raws` doesn't have to be changed at all.

Actually this could almost trivially be wrapped in a kind of ugly (naming-wise) way into `raw.interpolate_bads(..., method='nan')` or something that just replaces bad channel data with NaN. That's already where one should probably look into dealing with bad channels across recordings/sessions/runs/subjects, so it's maybe an okay place...
it would be a bit odd if `raw.interpolate_bads(...)` might result in no interpolation at all (but rather just masking the bad channels)...  but the more I think about it the more I like the idea.

But this raises the question of why current/intended usage of `interpolate_bads()` is not a viable option for @moritz-gerster ?  I.e., why not interpolate rather than setting as `nan` (obviously `nan` is faster, but are there other reasons)?
In my case, I work on deep brain stimulation leads. I think these leads are very difficult to reasonably interpolate because they are in a tiny inhomogeneous structure (the subthalamic nucleus) very close to the neuronal sources and in addition they have various leads with various impedances (2 ring leads and 6 smaller directional leads). I attach a figure @drammock. 

Even if interpolation would work well, it would not improve my Common Spatial Patterns algorithm anyways as feeding in linear dependent data does not really improve the training sample size I guess. 
![image](https://user-images.githubusercontent.com/45031224/219276222-9dfec2de-aab4-4619-acd9-041fd08da033.png)
[1] shows the 8 different leads
There is another functionality that `raw.interpolate_bads(...)` does not provide @drammock, @larsoner. It could be the case, that even the number of channels might not agree across recordings. Maybe one or two channels went missing completely. 

At the moment, one would have to create fake channels with nan values that match the recording length and copy the info from the other recordings where the channel is present. This requires tremendous coding. 

Wouldn't it be nice if their is one simple function that compares channels across recordings and
- sets bad channels as nan (only if they are good in some recordings and otherwise keeps them as bad throughout)
- and adds empty channels with matching info if some channels are missing?

I think this could be added to `mne.concatenate_raws()` and it would probably make it much easier to maximize available data even if it is inhomogeneous.
I currently do this:
```Python
try:
    # all channels match, including bad channels
    raw_concat = concatenate_raws(raws)
except ValueError:
    # set bad channels as nan and remove from bad channels
    for raw in raws:
        _interpolate_bads_nan(raw)

    # add missing channels as nan
    raws = _add_missing_channels_as_nan(raws)
    
    # now concatenate raws
    raw_concat = concatenate_raws(raws)


def _interpolate_bads_nan(raw):
    """Set bad channels to np.nan."""
    set_nan = lambda x: np.full(x.shape, np.nan)
    raw.apply_function(set_nan, picks=raw.info["bads"])
    raw.info["bads"] = []


def _add_missing_channels_as_nan(raws):
    # all channels across all raws
    all_chs = [raw.ch_names for raw in raws]  # as list
    all_chs = {el for list in all_chs for el in list}  # as set

    missing_channels = []
    for ch in all_chs:
        for raw in raws:
            if ch not in raw.ch_names:
                missing_channels.append(ch)

    # add missing channels as nan
    for ch in missing_channels:
        for raw in raws:
            if ch not in raw.ch_names:
                # need to add fake channel to raw
                ch_good = False
                while not ch_good:
                    # loop again through raws to get the very same channel
                    # in a recording where it is good
                    for sub_raw in raws:
                        if ch in sub_raw.ch_names:
                            ch_info = sub_raw.copy().pick(ch).info
                            data = np.empty([1, raw.get_data().shape[1]])
                            data[:] = np.nan
                            empty_channel = RawArray(data, ch_info)
                            ch_good = True
                raw.add_channels([empty_channel], force_update_info=True)
    return raws
```

This is pretty ugly. Is there a more elegant way to do this? Don't you think that many other researchers need the same functionality?
> In my case, I work on deep brain stimulation leads. I think these leads are very difficult to reasonably interpolate because they are in a tiny inhomogeneous structure (the subthalamic nucleus) very close to the neuronal sources and in addition they have various leads with various impedances (2 ring leads and 6 smaller directional leads).

OK that makes sense. I'm +1 on adapting `interpolate_bads` to have a `method='nan'` option.

> Maybe one or two channels went missing completely. At the moment, one would have to create fake channels with nan values that match the recording length and copy the info from the other recordings where the channel is present. This requires tremendous coding.

see #11106; maybe your use case is enough to convince @larsoner to [change his mind](https://github.com/mne-tools/mne-python/issues/11106#issuecomment-1248396420). 

As an aside, if I'm understanding your code correctly, it looks like when a channel is missing you're using the `info` for that channel *from a different recording*. I'm guessing you thought of this already, but if that recording is from a different subject the `pos` info will almost surely be wrong. In the case of EEG even the same subject in different sessions could be wrong if you're using real (digitized) sensor locations rather than a template montage (though IDK if that applies to depth electrodes; I would hope that they don't shift around much day-to-day!)
> see #11106; maybe your use case is enough to convince @larsoner to change his mind.
>
> As an aside, if I'm understanding your code correctly, it looks like when a channel is missing you're using the info for that channel from a different recording. I'm guessing you thought of this already, but...

It's considerations like this -- that there are a lot of underlying variables to consider -- that make me hesitate to add some new simple one-liner to do this stuff. In some sense the cumbersome nature of needing to create new `RawArray` and then `add_channels` and/or `reorder_channels` them helps see what's really necessary to do what you want, and hopefully implicitly see concerns like what @drammock brings up.

However, I am a bit more convinced seeing that there are more use cases like that of @hoechenberger in #11106 in the wild. Perhaps we can add something new and document sufficiently all the caveats.

We handled a somewhat similar problem in #10533 / #10537 to create a `prepare_empty_room` function. Rather than making `concatenate_raws` any smarter, though, I think I'd rather have a dedicated function for coercing recordings with different channels (and/or different bads) to have the same set of (good) channels. After all you might want this *without* concatenating the recordings as well, just to get the same sort of / compatible Evoked instance from multiple participants. Maybe a new `mne.preprocessing.unify_channels(inst_list, add_method='nan', bad_method='interpolate')` or something to start? I don't love the name and am not sure about the necessary arguments yet, but the idea is that you are somehow unifying all channels across instances so that any subsequent concatenation and processing (epoching, evoking, PSD, etc.) afterward give reasonable results.
I thought we already had a solution for that but I cannot remember the name anymore. I believe @agramfort implemented this...  I did make a reference to it without ever mentioning the name of the function, see:

https://github.com/mne-tools/mne-bids-pipeline/issues/538

> As an aside, if I'm understanding your code correctly, it looks like when a channel is missing you're using the `info` for that channel _from a different recording_. I'm guessing you thought of this already, but if that recording is from a different subject the `pos` info will almost surely be wrong. In the case of EEG even the same subject in different sessions could be wrong if you're using real (digitized) sensor locations rather than a template montage (though IDK if that applies to depth electrodes; I would hope that they don't shift around much day-to-day!)

Yes, thank you! It's always the same subject and its intracranial electrodes which hopefully don't move 😊


I like the idea of having a function :

mne.preprocessing.unify_bad_channels

there are indeed some places eg the mne bids pipeline where there is indeed
some logic like this.

On Sat, Feb 18, 2023 at 3:25 PM Moritz Gerster ***@***.***>
wrote:

> As an aside, if I'm understanding your code correctly, it looks like when
> a channel is missing you're using the info for that channel *from a
> different recording*. I'm guessing you thought of this already, but if
> that recording is from a different subject the pos info will almost
> surely be wrong. In the case of EEG even the same subject in different
> sessions could be wrong if you're using real (digitized) sensor locations
> rather than a template montage (though IDK if that applies to depth
> electrodes; I would hope that they don't shift around much day-to-day!)
>
> Yes, thank you! It's always the same subject and its intracranial
> electrodes which hopefully don't move 😊
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/11481#issuecomment-1435688316>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABHKHBVSGYMVS2W42RRWLDWYDLUXANCNFSM6AAAAAAU5JIIYU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Resolving patch:
diff --git a/mne/channels/channels.py b/mne/channels/channels.py
--- a/mne/channels/channels.py
+++ b/mne/channels/channels.py
@@ -830,7 +830,8 @@ def interpolate_bads(
             .. versionadded:: 0.17
         method : dict | None
             Method to use for each channel type.
-            Currently only the key ``"eeg"`` has multiple options:
+            All channel types support "nan".
+            The key ``"eeg"`` has two additional options:
 
             - ``"spline"`` (default)
                 Use spherical spline interpolation.
@@ -838,9 +839,10 @@ def interpolate_bads(
                 Use minimum-norm projection to a sphere and back.
                 This is the method used for MEG channels.
 
-            The value for ``"meg"`` is ``"MNE"``, and the value for
-            ``"fnirs"`` is ``"nearest"``. The default (None) is thus an alias
-            for::
+            The default value for ``"meg"`` is ``"MNE"``, and the default value
+            for ``"fnirs"`` is ``"nearest"``.
+
+            The default (None) is thus an alias for::
 
                 method=dict(meg="MNE", eeg="spline", fnirs="nearest")
 
@@ -858,6 +860,10 @@ def interpolate_bads(
         Notes
         -----
         .. versionadded:: 0.9.0
+
+        .. warning::
+            Be careful when using ``method="nan"``; the default value
+            ``reset_bads=True`` may not be what you want.
         """
         from .interpolation import (
             _interpolate_bads_eeg,
@@ -869,9 +875,31 @@ def interpolate_bads(
         method = _handle_default("interpolation_method", method)
         for key in method:
             _check_option("method[key]", key, ("meg", "eeg", "fnirs"))
-        _check_option("method['eeg']", method["eeg"], ("spline", "MNE"))
-        _check_option("method['meg']", method["meg"], ("MNE",))
-        _check_option("method['fnirs']", method["fnirs"], ("nearest",))
+        _check_option(
+            "method['eeg']",
+            method["eeg"],
+            (
+                "spline",
+                "MNE",
+                "nan",
+            ),
+        )
+        _check_option(
+            "method['meg']",
+            method["meg"],
+            (
+                "MNE",
+                "nan",
+            ),
+        )
+        _check_option(
+            "method['fnirs']",
+            method["fnirs"],
+            (
+                "nearest",
+                "nan",
+            ),
+        )
 
         if len(self.info["bads"]) == 0:
             warn("No bad channels to interpolate. Doing nothing...")
@@ -884,11 +912,18 @@ def interpolate_bads(
         else:
             eeg_mne = True
         _interpolate_bads_meeg(
-            self, mode=mode, origin=origin, eeg=eeg_mne, exclude=exclude
+            self, mode=mode, origin=origin, eeg=eeg_mne, exclude=exclude, method=method
         )
-        _interpolate_bads_nirs(self, exclude=exclude)
+        _interpolate_bads_nirs(self, exclude=exclude, method=method["fnirs"])
 
         if reset_bads is True:
+            if "nan" in method.values():
+                warn(
+                    "interpolate_bads was called with method='nan' and "
+                    "reset_bads=True. Consider setting reset_bads=False so that the "
+                    "nan-containing channels can be easily excluded from later "
+                    "computations."
+                )
             self.info["bads"] = [ch for ch in self.info["bads"] if ch in exclude]
 
         return self
diff --git a/mne/channels/interpolation.py b/mne/channels/interpolation.py
--- a/mne/channels/interpolation.py
+++ b/mne/channels/interpolation.py
@@ -1,4 +1,5 @@
 # Authors: Denis Engemann <denis.engemann@gmail.com>
+#          Ana Radanovic <radanovica@protonmail.com>
 #
 # License: BSD-3-Clause
 
@@ -191,10 +192,14 @@ def _interpolate_bads_meeg(
     eeg=True,
     ref_meg=False,
     exclude=(),
+    *,
+    method=None,
     verbose=None,
 ):
     from ..forward import _map_meg_or_eeg_channels
 
+    if method is None:
+        method = {"meg": "MNE", "eeg": "MNE"}
     bools = dict(meg=meg, eeg=eeg)
     info = _simplify_info(inst.info)
     for ch_type, do in bools.items():
@@ -210,6 +215,12 @@ def _interpolate_bads_meeg(
             continue
         # select the bad channels to be interpolated
         picks_bad = pick_channels(inst.info["ch_names"], bads_type, exclude=[])
+
+        if method[ch_type] == "nan":
+            inst._data[picks_bad] = np.nan
+            continue
+
+        # do MNE based interpolation
         if ch_type == "eeg":
             picks_to = picks_type
             bad_sel = np.isin(picks_type, picks_bad)
@@ -243,7 +254,7 @@ def _interpolate_bads_nirs(inst, method="nearest", exclude=(), verbose=None):
     chs = [inst.info["chs"][i] for i in picks_nirs]
     locs3d = np.array([ch["loc"][:3] for ch in chs])
 
-    _check_option("fnirs_method", method, ["nearest"])
+    _check_option("fnirs_method", method, ["nearest", "nan"])
 
     if method == "nearest":
         dist = pdist(locs3d)
@@ -258,7 +269,10 @@ def _interpolate_bads_nirs(inst, method="nearest", exclude=(), verbose=None):
             # Find closest remaining channels for same frequency
             closest_idx = np.argmin(dists_to_bad) + (bad % 2)
             inst._data[bad] = inst._data[closest_idx]
-
-        inst.info["bads"] = [ch for ch in inst.info["bads"] if ch in exclude]
+    else:
+        assert method == "nan"
+        inst._data[picks_bad] = np.nan
+    # TODO: this seems like a bug because it does not respect reset_bads
+    inst.info["bads"] = [ch for ch in inst.info["bads"] if ch in exclude]
 
     return inst


Test patch:
diff --git a/mne/channels/tests/test_interpolation.py b/mne/channels/tests/test_interpolation.py
--- a/mne/channels/tests/test_interpolation.py
+++ b/mne/channels/tests/test_interpolation.py
@@ -17,6 +17,7 @@
 from mne._fiff.proj import _has_eeg_average_ref_proj
 from mne.utils import _record_warnings
 
+
 base_dir = Path(__file__).parent.parent.parent / "io" / "tests" / "data"
 raw_fname = base_dir / "test_raw.fif"
 event_name = base_dir / "test-eve.fif"
@@ -324,3 +325,27 @@ def test_interpolation_nirs():
     assert raw_haemo.info["bads"] == ["S1_D2 hbo", "S1_D2 hbr"]
     raw_haemo.interpolate_bads()
     assert raw_haemo.info["bads"] == []
+
+
+def test_nan_interpolation(raw):
+    """Test 'nan' method for interpolating bads."""
+    ch_to_interp = [raw.ch_names[1]]  # don't use channel 0 (type is IAS not MEG)
+    raw.info["bads"] = ch_to_interp
+
+    # test that warning appears for reset_bads = True
+    with pytest.warns(RuntimeWarning, match="Consider setting reset_bads=False"):
+        raw.interpolate_bads(method="nan", reset_bads=True)
+
+    # despite warning, interpolation still happened, make sure the channel is NaN
+    bad_chs = raw.get_data(ch_to_interp)
+    assert np.isnan(bad_chs).all()
+
+    # make sure reset_bads=False works as expected
+    raw.info["bads"] = ch_to_interp
+    raw.interpolate_bads(method="nan", reset_bads=False)
+    assert raw.info["bads"] == ch_to_interp
+
+    # make sure other channels are untouched
+    raw.drop_channels(ch_to_interp)
+    good_chs = raw.get_data()
+    assert np.isfinite(good_chs).all()