Repository: mne-tools/mne-python

Issue:
Reading CTF Evokeds fails: missing compensation channel(s)
On master, the attached file was created with `evoked.save(fname)`, but can't be read with `mne.read_evokeds(fname)`:

```
/Users/jona/tools/mne-python/mne/io/meas_info.py in read_meas_info(fid, tree, clean_bads, verbose)
    775
    776     #   Load the CTF compensation data
--> 777     comps = read_ctf_comp(fid, meas_info, chs)
    778
    779     #   Load the bad channel list

/Users/jona/tools/mne-python/mne/io/ctf_comp.py in read_ctf_comp(fid, node, chs, verbose)

/Users/jona/tools/mne-python/mne/utils.py in verbose(function, *args, **kwargs)
    705         with use_log_level(verbose_level):
    706             return function(*args, **kwargs)
--> 707     return function(*args, **kwargs)
    708
    709

/Users/jona/tools/mne-python/mne/io/ctf_comp.py in read_ctf_comp(fid, node, chs, verbose)
    109         if not calibrated:
    110             #   Calibrate...
--> 111             _calibrate_comp(one, chs, mat['row_names'], mat['col_names'])
    112         else:
    113             one['rowcals'] = np.ones(mat['data'].shape[0], dtype=np.float)

/Users/jona/tools/mne-python/mne/io/ctf_comp.py in _calibrate_comp(comp, chs, row_names, col_names, mult_keys, flip)
     42             if p != 1:
     43                 raise RuntimeError('Channel %s does not appear exactly once '
---> 44                                    'in data' % names[ii])
     45             idx = ch_names.index(names[ii])
     46             val = chs[idx][mult_keys[0]] * chs[idx][mult_keys[1]]

RuntimeError: Channel BG1-3609 does not appear exactly once in data
```

The instance was created by `linear_regression_raw` out of a Raw instance.

[HA0710_demo-ave.fif.zip](https://github.com/mne-tools/mne-python/files/974780/HA0710_demo-ave.fif.zip)

@jaeilepp ..?

Edit: the problem stems from picking the channels and removing the ref_meg.

It also makes `mne.make_forward_solution` fail with a similar error.


Discussion:
Can you see if you can reproduce using one of the CTF datasets in `testing`?

If not, can you at least see if this happens without using `linear_regression_raw` (i.e., using plain `Epochs` and `epochs.average()`? My guess is that there is some interaction with `picks` where `linear_regression_raw` chooses non-reference channels, and the resulting `Evoked` instance doesn't have the channels necessary for `comps`, and this is only noticed on read.
It only depends on picking `ref_meg = False`. E.g., if I write the raw file's info to the disk after picking, I can't read it due to this error. If I don't pick, it works.
I'm not sure what's the right fix for this. I see 3 options:

1. Don't save the comps info from dropped channels.
2. Save the comps info, but ignore them while reading, throwing a warning.
3. Not allow saving when required channels are missing.

And BTW the issue heading is incorrect. The problem is that the compensation channel is not there.

	
		
		
	
		
		At a minimum, if the error message read "missing compensation channel" instead of saying that a specific weirdly named channel appears more or less than 1 times, it would be a much easier problem :)
And a warning when removing the comp channels about what consequences this will have - that it precludes file saving/reading, and if it worsens the data later on. (If I understand it, the compensation has already been appplied and they can in principle be safely dropped?)


Comps are also necessary for fwd/inv.

My proposed fix is that when picking is done (e.g., in `raw.pick_types`, or in `linear_regresison_raw`, etc.):

1. If compensation has been applied but not all used comp channels are picked, raise an error.

2. If no compensation has been applied, discard the projectors (alongside a `logger.info` that this is being done).

	
		
		
	You have to manually turn off compensation right? If yes - seems sensible to me; if you've manually turned off comps and pick the channels, you probably know what you're doing. Otherwise I would prefer a warning. 


Hi, is there a quick workaround before it gets fixed? I got the same error when I try to read my epoched data, so I assume it's related.
If you can, it would probably be easiest to recreate the epochs in a what that can be read. 
Yes, you're right. Thanks.

So in my case skipping  `picks=picks_meg` argument, while creating epochs, helped.
yes the easy fix is to avoid storing the data with picked channels.


Resolving patch:
diff --git a/mne/channels/channels.py b/mne/channels/channels.py
--- a/mne/channels/channels.py
+++ b/mne/channels/channels.py
@@ -594,11 +594,13 @@ def anonymize(self):
 class UpdateChannelsMixin(object):
     """Mixin class for Raw, Evoked, Epochs, AverageTFR."""
 
+    @verbose
     def pick_types(self, meg=True, eeg=False, stim=False, eog=False,
                    ecg=False, emg=False, ref_meg='auto', misc=False,
                    resp=False, chpi=False, exci=False, ias=False, syst=False,
                    seeg=False, dipole=False, gof=False, bio=False, ecog=False,
-                   fnirs=False, include=(), exclude='bads', selection=None):
+                   fnirs=False, include=(), exclude='bads', selection=None,
+                   verbose=None):
         """Pick some channels by type and names.
 
         Parameters
@@ -657,6 +659,10 @@ def pick_types(self, meg=True, eeg=False, stim=False, eog=False,
             in ``info['bads']``.
         selection : list of string
             Restrict sensor channels (MEG, EEG) to this list of channel names.
+        verbose : bool, str, int, or None
+            If not None, override default verbose level (see
+            :func:`mne.verbose` and :ref:`Logging documentation <tut_logging>`
+            for more).
 
         Returns
         -------
@@ -804,6 +810,31 @@ def _pick_drop_channels(self, idx):
         if hasattr(self, '_cals'):
             self._cals = self._cals[idx]
 
+        if len(self.info['comps']) > 0:
+            current_comp = get_current_comp(self.info)
+            # Check and possibly remove comps
+            comp_names = sorted(set(
+                comp_name for comp in self.info['comps']
+                for comp_name in comp['data']['col_names']))
+            comp_picks = pick_channels(self.ch_names, comp_names)
+            assert len(comp_picks) == len(comp_names)
+            missing = [comp_names[ii]
+                       for ii in np.where(~np.in1d(comp_picks, idx))[0]]
+            if len(missing) > 0:
+                names = ', '.join(missing)
+                names = names[:20] + '...' if len(names) > 20 else names
+                if current_comp != 0:
+                    raise RuntimeError(
+                        'Compensation grade %d has been applied, but '
+                        'compensation channels are missing: %s\n'
+                        'Either remove compensation or pick compensation '
+                        'channels' % (current_comp, names))
+                else:
+                    logger.info('Removing %d compensators from info because '
+                                'not all compensation channels were picked'
+                                % (len(self.info['comps']),))
+                    self.info['comps'] = []
+
         pick_info(self.info, idx, copy=False)
 
         if getattr(self, '_projector', None) is not None:


Test patch:
diff --git a/mne/io/ctf/tests/test_ctf.py b/mne/io/ctf/tests/test_ctf.py
--- a/mne/io/ctf/tests/test_ctf.py
+++ b/mne/io/ctf/tests/test_ctf.py
@@ -9,7 +9,6 @@
 import warnings
 
 import numpy as np
-from nose.tools import assert_raises, assert_true, assert_false
 from numpy import array_equal
 from numpy.testing import assert_allclose, assert_array_equal, assert_equal
 import pytest
@@ -18,8 +17,9 @@
 # from mne.tests.common import assert_dig_allclose
 from mne.transforms import apply_trans
 from mne.io import read_raw_fif, read_raw_ctf
+from mne.io.compensator import get_current_comp
 from mne.io.tests.test_raw import _test_raw_reader
-from mne.utils import _TempDir, run_tests_if_main, _clean_names
+from mne.utils import _TempDir, run_tests_if_main, _clean_names, catch_logging
 from mne.datasets import testing, spm_face
 from mne.io.constants import FIFF
 
@@ -60,14 +60,14 @@ def test_read_ctf():
     shutil.copytree(op.join(ctf_dir, ctf_fname_catch), ctf_eeg_fname)
     with warnings.catch_warnings(record=True) as w:  # reclassified ch
         raw = _test_raw_reader(read_raw_ctf, directory=ctf_eeg_fname)
-    assert_true(all('MISC channel' in str(ww.message) for ww in w))
+    assert all('MISC channel' in str(ww.message) for ww in w)
     picks = pick_types(raw.info, meg=False, eeg=True)
     pos = np.random.RandomState(42).randn(len(picks), 3)
     fake_eeg_fname = op.join(ctf_eeg_fname, 'catch-alp-good-f.eeg')
     # Create a bad file
     with open(fake_eeg_fname, 'wb') as fid:
         fid.write('foo\n'.encode('ascii'))
-    assert_raises(RuntimeError, read_raw_ctf, ctf_eeg_fname)
+    pytest.raises(RuntimeError, read_raw_ctf, ctf_eeg_fname)
     # Create a good file
     with open(fake_eeg_fname, 'wb') as fid:
         for ii, ch_num in enumerate(picks):
@@ -77,11 +77,11 @@ def test_read_ctf():
     pos_read_old = np.array([raw.info['chs'][p]['loc'][:3] for p in picks])
     with warnings.catch_warnings(record=True) as w:  # reclassified channel
         raw = read_raw_ctf(ctf_eeg_fname)  # read modified data
-    assert_true(all('MISC channel' in str(ww.message) for ww in w))
+    assert all('MISC channel' in str(ww.message) for ww in w)
     pos_read = np.array([raw.info['chs'][p]['loc'][:3] for p in picks])
     assert_allclose(apply_trans(raw.info['ctf_head_t'], pos), pos_read,
                     rtol=1e-5, atol=1e-5)
-    assert_true((pos_read == pos_read_old).mean() < 0.1)
+    assert (pos_read == pos_read_old).mean() < 0.1
     shutil.copy(op.join(ctf_dir, 'catch-alp-good-f.ds_randpos_raw.fif'),
                 op.join(temp_dir, 'randpos', 'catch-alp-good-f.ds_raw.fif'))
 
@@ -92,7 +92,7 @@ def test_read_ctf():
     remove_base = op.join(ctf_no_hc_fname, op.basename(ctf_fname_catch[:-3]))
     os.remove(remove_base + '.hc')
     with warnings.catch_warnings(record=True):  # no coord tr
-        assert_raises(RuntimeError, read_raw_ctf, ctf_no_hc_fname)
+        pytest.raises(RuntimeError, read_raw_ctf, ctf_no_hc_fname)
     os.remove(remove_base + '.eeg')
     shutil.copy(op.join(ctf_dir, 'catch-alp-good-f.ds_nohc_raw.fif'),
                 op.join(temp_dir, 'no_hc', 'catch-alp-good-f.ds_raw.fif'))
@@ -103,7 +103,7 @@ def test_read_ctf():
         raw_c = read_raw_fif(fname + '_raw.fif', preload=True)
         with warnings.catch_warnings(record=True) as w:  # reclassified ch
             raw = read_raw_ctf(fname)
-        assert_true(all('MISC channel' in str(ww.message) for ww in w))
+        assert all('MISC channel' in str(ww.message) for ww in w)
 
         # check info match
         assert_array_equal(raw.ch_names, raw_c.ch_names)
@@ -114,7 +114,7 @@ def test_read_ctf():
         py_time = raw.info['meas_id']['secs']
         c_time = raw_c.info['meas_id']['secs']
         max_offset = 24 * 60 * 60  # probably overkill but covers timezone
-        assert_true(c_time - max_offset <= py_time <= c_time)
+        assert c_time - max_offset <= py_time <= c_time
         for t in ('dev_head_t', 'dev_ctf_t', 'ctf_head_t'):
             assert_allclose(raw.info[t]['trans'], raw_c.info[t]['trans'],
                             rtol=1e-4, atol=1e-7)
@@ -208,13 +208,13 @@ def test_read_ctf():
         # all data / preload
         with warnings.catch_warnings(record=True) as w:  # reclassified ch
             raw = read_raw_ctf(fname, preload=True)
-        assert_true(all('MISC channel' in str(ww.message) for ww in w))
+        assert all('MISC channel' in str(ww.message) for ww in w)
         assert_allclose(raw[:][0], raw_c[:][0])
-    assert_raises(TypeError, read_raw_ctf, 1)
-    assert_raises(ValueError, read_raw_ctf, ctf_fname_continuous + 'foo.ds')
+    pytest.raises(TypeError, read_raw_ctf, 1)
+    pytest.raises(ValueError, read_raw_ctf, ctf_fname_continuous + 'foo.ds')
     # test ignoring of system clock
     read_raw_ctf(op.join(ctf_dir, ctf_fname_continuous), 'ignore')
-    assert_raises(ValueError, read_raw_ctf,
+    pytest.raises(ValueError, read_raw_ctf,
                   op.join(ctf_dir, ctf_fname_continuous), 'foo')
 
 
@@ -260,17 +260,43 @@ def test_read_spm_ctf():
     raw = read_raw_ctf(raw_fname)
     extras = raw._raw_extras[0]
     assert_equal(extras['n_samp'], raw.n_times)
-    assert_false(extras['n_samp'] == extras['n_samp_tot'])
+    assert extras['n_samp'] != extras['n_samp_tot']
 
     # Test that LPA, nasion and RPA are correct.
     coord_frames = np.array([d['coord_frame'] for d in raw.info['dig']])
-    assert_true(np.all(coord_frames == FIFF.FIFFV_COORD_HEAD))
+    assert np.all(coord_frames == FIFF.FIFFV_COORD_HEAD)
     cardinals = {d['ident']: d['r'] for d in raw.info['dig']}
-    assert_true(cardinals[1][0] < cardinals[2][0] < cardinals[3][0])  # x coord
-    assert_true(cardinals[1][1] < cardinals[2][1])  # y coord
-    assert_true(cardinals[3][1] < cardinals[2][1])  # y coord
+    assert cardinals[1][0] < cardinals[2][0] < cardinals[3][0]  # x coord
+    assert cardinals[1][1] < cardinals[2][1]  # y coord
+    assert cardinals[3][1] < cardinals[2][1]  # y coord
     for key in cardinals.keys():
         assert_allclose(cardinals[key][2], 0, atol=1e-6)  # z coord
 
 
+@testing.requires_testing_data
+def test_saving_picked():
+    """Test saving picked CTF instances."""
+    temp_dir = _TempDir()
+    out_fname = op.join(temp_dir, 'test_py_raw.fif')
+    raw = read_raw_ctf(op.join(ctf_dir, ctf_fname_1_trial))
+    raw.crop(0, 1).load_data()
+    assert raw.compensation_grade == get_current_comp(raw.info) == 0
+    assert len(raw.info['comps']) == 5
+    pick_kwargs = dict(meg=True, ref_meg=False, verbose=True)
+    with catch_logging() as log:
+        raw_pick = raw.copy().pick_types(**pick_kwargs)
+    assert len(raw.info['comps']) == 5
+    assert len(raw_pick.info['comps']) == 0
+    log = log.getvalue()
+    assert 'Removing 5 compensators' in log
+    raw_pick.save(out_fname)  # should work
+    read_raw_fif(out_fname)
+    read_raw_fif(out_fname, preload=True)
+    # If comp is applied, picking should error
+    raw.apply_gradient_compensation(1)
+    assert raw.compensation_grade == get_current_comp(raw.info) == 1
+    with pytest.raises(RuntimeError, match='Compensation grade 1 has been'):
+        raw.copy().pick_types(**pick_kwargs)
+
+
 run_tests_if_main()