Repository: mne-tools/mne-python

Issue:
raw.add_channels  'buffer_size_sec' error
I'm trying to add a `stim` channel (a clean combination of multiple already existing stim channels) to a raw object:

```
# read raw
raw = read_raw_edf(fname, preload=True)

# clean stim
raw_stim = raw.pick_channels(chs_stim, copy=True)
data = clean_stim(raw_stim._data)

# append data
info = create_info(ch_names=['stim_clean'], sfreq=raw.info['sfreq'], ch_types=['stim'])
raw_stim = RawArray(data, info=info, first_samp=raw._first_samps[0])
raw.add_channels([raw_stim])
```

But I get the following error:

`RuntimeError: Don't know how to merge 'buffer_size_sec'. Make sure values are compatible.`

What shall I do?

cc @choldgraf @Eric89GXL 



Discussion:
Set `buffer_size_sec` in `create_info` to whatever it is in `raw`, i.e.:

```
create_info(..., buffer_size_sec=raw.info['buffer_size_sec'])
```

TypeError: create_info() got an unexpected keyword argument 'buffer_size_sec'

```
info = create_info(...)
info['buffer_size_sec'] = raw.info['buffer_size_sec']
```

However I can do:
`info['buffer_size_sec'] = raw.info['buffer_size_sec']`

But that doesn't feel very clean

It's a fine way to do it. We don't expose all options in `create_info` but you can set them afterward. We could change this behavior.

And now I have:
`RuntimeError: Don't know how to merge 'filename'. Make sure values are compatible.`

when I add the channel `raw.add_channels([raw_stim])`

Overall, it seems like an aweful lot of steps to just add a stim channel..

It works with

```
        info['buffer_size_sec'] = raw.info['buffer_size_sec']
        info['filename'] = raw.info['filename']
```

I'm keeping the issue opened. I think we should have a less hacky way. WDYT?

I would expect to be able to do
`Raw.add_channels(data=np.array, ch_type=['stim', 'misc'], ch_names=['foo', 'bar'])`

```
for key in ('buffer_size_sec', 'filename'):
    ...
```

is how I would do it to make it DRY.

I know it is more work, but it is also more explicit. It forces the user to think about what keys are being merged and what the values should actually be, which isn't necessarily a bad thing. I'm open to ideas about how to make it more automated. However, with automation there is the risk of getting burned by not understanding what's going on under the hood. As I mentioned in another issue, `info` actually stores a lot of information, and thinking of it as just containing channel types and names can cause problems.

If adding a stim channel to data that does not have one is a common enough use case, maybe we should add a function specific for that. Stim channels in general have less information than, say, EEG or MEG. Then the user won't need to care about `info` at all.

> Stim channels in general have less information than, say, EEG or MEG. Then the user won't need to care about info at all.

+1

If you don't have a problem totally obliterating what's in your info channel (e.g., if you're doing this with ecog or something that doesn't have montage metadata etc), then what I've done is first re-create my dataset using `RawArray`, and then append the stim channel to it. Agreed that it removes a bunch of information.

That said, I think it'd be helpful to have documentation about what fields will result in an error if you try to merge a manually-created info object with one read from an EDF file or something.

What if there were a function to merge `info` objects that behaved similarly to how pandas merges dataframes? E.g., it could have flags to either throw an error with a field mismatch, or to replace values using one or the other info object. Then any time that a field was overwritten it could write that to the log.

I think it's better to not create a RawArray from scratch, just in case we forget some info.

This is what I currently do

```
def add_channels(inst, data, ch_names, ch_types):
    from mne.io import _BaseRaw, RawArray
    from mne.epochs import _BaseEpochs, EpochsArray
    from mne import create_info
    if 'meg' in ch_types or 'eeg' in ch_types:
        return NotImplementedError('Can only add misc, stim and ieeg channels')
    info = create_info(ch_names=ch_names, sfreq=inst.info['sfreq'],
                       ch_types=ch_types)
    if isinstance(inst, _BaseRaw):
        for key in ('buffer_size_sec', 'filename'):
            info[key] = inst.info[key]
        new_inst = RawArray(data, info=info, first_samp=inst._first_samps[0])
    elif isinstance(inst, _BaseEpochs):
        new_inst = EpochsArray(data, info=info)
    else:
        raise ValueError('unknown inst type')
    return inst.add_channels([new_inst], copy=True)
```

how about:

# read raw

raw = read_raw_edf(fname, preload=True)

# clean stim

raw_stim = raw.pick_channels(chs_stim, copy=True)raw_stim._data =
clean_stim(raw_stim._data)

# append data

raw.add_channels([raw_stim])

?

The `raw_stim` has multiple chans (one for each binary TTL) whereas the clean_stim is the combination (similar to STI 101 in elekta defaults).

One could however follow your solution and add an additional `pick_channels`, and perhaps a `rename_channels` afterward.

However, if one (i.e. me) wants to add, say, an auditory misc channel, then you need to change the ch_type (or already have a `misc` in your `raw`). 

Overall, there's more than one hack that'll do the trick, I was mainly asking what should be the cleanest way. 

(I don't want to reopen a fight on small API changes; I'm personally just going to use the code above, that seems clean and generic enough to me)

I would be fine adding a "force" param to raw.add_channels to drop any
offending key in the Info and be therefore more permissive

That sounds good to me - it'd just force any overlapping keys w/ different values to take on the value of the `self` object, and raise a warning? Can anybody think of an instance where this might break things?

That's okay with me as long as it's opt-in by kwarg, not default

PR welcome then :)

can do this, but not right now...it will go on my list of "fun things to do
once I resubmit my paper" :)

On Mon, Feb 29, 2016 at 1:41 PM, Alexandre Gramfort <
notifications@github.com> wrote:

> PR welcome then :)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/mne-tools/mne-python/issues/2948#issuecomment-190408411
> .

> I would be fine adding a "force" param to raw.add_channels to drop any offending key in the Info and be therefore more permissive

+1 

> can do this, but not right now...it will go on my list of "fun things to do once I resubmit my paper" :)

Ok, good luck ;)

@choldgraf certainly your paper is resubmitted by now? :)

:(

let me put it this way: I'm not shaving or getting a haircut until I resubmit the damn paper, and yesterday my labmate said that I had a nice "The Revenant" thing going on...

I'm going to move this to 0.13 but if you get to it by mid/end of April then it's fine to put in 0.12

OK - the tentative plan is to resubmit by the end of march, so that does seem reasonable. This thread is on my to-do list so I won't forget about it

Looking through this now. I'm not quite sure the best way to add a "force" option. The reason is that there's a lot of machinery under the hood to make sure that values are able to be merged between info objects. So here are a couple of thoughts:
1. Hard-code some fields that we know are going to be problematic for adding stim channels, and if a "force" parameter is given, then automatically add those fields to new info objects before merging (similar to what @kingjr and @Eric89GXL do above)
2. Separate from the merging code, write some kind of `make_info_mergeable` function that takes a base info object and a list of other info objects, and does something like 1 above for the to-be-appended info objects.
3. Write a more general function that has a `force` parameter which will overwrite _all_ inconsistencies between the base / to-append info with those of the base info object.

If we think that the problematic fields won't change much in the future, I think that 1 might be the most verbose and easy to implement, but it might be a pain if we think that the fields which throw this error will change over time.

WDYT? 

Would this work?
1. The force option takes any conflicting keys and uses the value from the
   first dictionary (easy to explain, hopefully easy to implement). Call
   _check_consistency at the end, and beef up that function of necessary to
   hopefully catch any traps.


Resolving patch:
diff --git a/mne/channels/channels.py b/mne/channels/channels.py
--- a/mne/channels/channels.py
+++ b/mne/channels/channels.py
@@ -560,7 +560,7 @@ def inst_has(attr):
         elif isinstance(self, Evoked):
             self.data = self.data.take(idx, axis=0)
 
-    def add_channels(self, add_list, copy=None):
+    def add_channels(self, add_list, copy=None, force_update_info=False):
         """Append new channels to the instance.
 
         Parameters
@@ -572,6 +572,12 @@ def add_channels(self, add_list, copy=None):
             This parameter has been deprecated and will be removed in 0.13.
             Use inst.copy() instead.
             Whether to return a new instance or modify in place.
+        force_update_info : bool
+            If True, force the info for objects to be appended to match the
+            values in `self`. This should generally only be used when adding
+            stim channels for which important metadata won't be overwritten.
+
+            .. versionadded:: 0.12
 
         Returns
         -------
@@ -614,7 +620,7 @@ def add_channels(self, add_list, copy=None):
         # Create final data / info objects
         data = np.concatenate(data, axis=con_axis)
         infos = [self.info] + [inst.info for inst in add_list]
-        new_info = _merge_info(infos)
+        new_info = _merge_info(infos, force_update_to_first=force_update_info)
 
         # Now update the attributes
         setattr(out, data_name, data)
diff --git a/mne/io/__init__.py b/mne/io/__init__.py
--- a/mne/io/__init__.py
+++ b/mne/io/__init__.py
@@ -7,7 +7,7 @@
 
 from .open import fiff_open, show_fiff, _fiff_get_fid
 from .meas_info import (read_fiducials, write_fiducials, read_info, write_info,
-                        _empty_info, _merge_info, Info)
+                        _empty_info, _merge_info, _force_update_info, Info)
 
 from .proj import make_eeg_average_ref_proj, Projection
 from .tag import _loc_to_coil_trans, _coil_trans_to_loc, _loc_to_eeg_loc
diff --git a/mne/io/meas_info.py b/mne/io/meas_info.py
--- a/mne/io/meas_info.py
+++ b/mne/io/meas_info.py
@@ -1289,7 +1289,7 @@ def _where_isinstance(values, kind):
 
 
 @verbose
-def _merge_info(infos, verbose=None):
+def _merge_info(infos, force_update_to_first=False, verbose=None):
     """Merge multiple measurement info dictionaries.
 
      - Fields that are present in only one info object will be used in the
@@ -1305,6 +1305,10 @@ def _merge_info(infos, verbose=None):
     ----------
     infos | list of instance of Info
         Info objects to merge into one info object.
+    force_update_to_first : bool
+        If True, force the fields for objects in `info` will be updated
+        to match those in the first item. Use at your own risk, as this
+        may overwrite important metadata.
     verbose : bool, str, int, or NonIe
         If not None, override default verbose level (see mne.verbose).
 
@@ -1315,6 +1319,9 @@ def _merge_info(infos, verbose=None):
     """
     for info in infos:
         info._check_consistency()
+    if force_update_to_first is True:
+        infos = deepcopy(infos)
+        _force_update_info(infos[0], infos[1:])
     info = Info()
     info['chs'] = []
     for this_info in infos:
@@ -1477,3 +1484,32 @@ def _empty_info(sfreq):
     info._update_redundant()
     info._check_consistency()
     return info
+
+
+def _force_update_info(info_base, info_target):
+    """Update target info objects with values from info base.
+
+    Note that values in info_target will be overwritten by those in info_base.
+    This will overwrite all fields except for: 'chs', 'ch_names', 'nchan'.
+
+    Parameters
+    ----------
+    info_base : mne.Info
+        The Info object you want to use for overwriting values
+        in target Info objects.
+    info_target : mne.Info | list of mne.Info
+        The Info object(s) you wish to overwrite using info_base. These objects
+        will be modified in-place.
+    """
+    exclude_keys = ['chs', 'ch_names', 'nchan']
+    info_target = np.atleast_1d(info_target).ravel()
+    all_infos = np.hstack([info_base, info_target])
+    for ii in all_infos:
+        if not isinstance(ii, Info):
+            raise ValueError('Inputs must be of type Info. '
+                             'Found type %s' % type(ii))
+    for key, val in info_base.items():
+        if key in exclude_keys:
+            continue
+        for i_targ in info_target:
+            i_targ[key] = val


Test patch:
diff --git a/mne/io/fiff/tests/test_raw_fiff.py b/mne/io/fiff/tests/test_raw_fiff.py
--- a/mne/io/fiff/tests/test_raw_fiff.py
+++ b/mne/io/fiff/tests/test_raw_fiff.py
@@ -1056,6 +1056,17 @@ def test_add_channels():
     assert_array_equal(raw_new[:, :][1], raw[:, :][1])
     assert_true(all(ch not in raw_new.ch_names for ch in raw_stim.ch_names))
 
+    # Testing force updates
+    raw_arr_info = create_info(['1', '2'], raw_meg.info['sfreq'], 'eeg')
+    orig_head_t = raw_arr_info['dev_head_t']
+    raw_arr = np.random.randn(2, raw_eeg.n_times)
+    raw_arr = RawArray(raw_arr, raw_arr_info)
+    # This should error because of conflicts in Info
+    assert_raises(ValueError, raw_meg.copy().add_channels, [raw_arr])
+    raw_meg.copy().add_channels([raw_arr], force_update_info=True)
+    # Make sure that values didn't get overwritten
+    assert_true(raw_arr.info['dev_head_t'] is orig_head_t)
+
     # Now test errors
     raw_badsf = raw_eeg.copy()
     raw_badsf.info['sfreq'] = 3.1415927
diff --git a/mne/io/tests/test_meas_info.py b/mne/io/tests/test_meas_info.py
--- a/mne/io/tests/test_meas_info.py
+++ b/mne/io/tests/test_meas_info.py
@@ -12,7 +12,7 @@
 from mne.io.constants import FIFF
 from mne.io.meas_info import (Info, create_info, _write_dig_points,
                               _read_dig_points, _make_dig_points, _merge_info,
-                              RAW_INFO_FIELDS)
+                              _force_update_info, RAW_INFO_FIELDS)
 from mne.utils import _TempDir, run_tests_if_main
 from mne.channels.montage import read_montage, read_dig_montage
 
@@ -267,6 +267,19 @@ def test_merge_info():
     assert_equal(info_merged['ch_names'], ['a', 'b', 'c', 'd', 'e', 'f'])
     assert_raises(ValueError, _merge_info, [info_a, info_a])
 
+    # Testing for force updates before merging
+    info_c = create_info(ch_names=['g', 'h', 'i'], sfreq=500., ch_types=None)
+    # This will break because sfreq is not equal
+    assert_raises(RuntimeError, _merge_info, [info_a, info_c])
+    _force_update_info(info_a, info_c)
+    assert_true(info_c['sfreq'] == info_a['sfreq'])
+    assert_true(info_c['ch_names'][0] != info_a['ch_names'][0])
+    # Make sure it works now
+    _merge_info([info_a, info_c])
+    # Check that you must supply Info
+    assert_raises(ValueError, _force_update_info, info_a,
+                  dict([('sfreq', 1000.)]))
+
 
 def test_check_consistency():
     """Test consistency check of Info objects"""