Repository: mne-tools/mne-python

Issue:
Maxwell filter scaling.
When I use ``st_only=True`` the scaling on mag channels seem off. Script to reproduce.

```
import mne
import os.path as op
from mne.datasets import testing

data_path = testing.data_path(download=False)
sss_path = op.join(data_path, 'SSS')
fine_cal_fname = op.join(sss_path, 'sss_cal_3053.dat')
ctc_fname = op.join(sss_path, 'ct_sparse.fif')
pre = op.join(sss_path, 'test_move_anon_')
raw_fname = pre + 'raw.fif'
raw = mne.io.Raw(raw_fname, allow_maxshield=True)

raw_sss = mne.preprocessing.maxwell_filter(raw, calibration=fine_cal_fname,
                                           st_duration=2, st_only=True,
                                           cross_talk=ctc_fname)
raw_sss.plot()
```
I also see similar looking issues with reconstructed bad grad channels on [multi-subject, multi-modal human neuroimaging dataset](https://openfmri.org/dataset/ds000117/) subject 10, but I'm unable to reproduce it with the testing dataset.


Discussion:
Hmm... I think that within any given window, the result actually looks reasonable. And tSSS is reducing power, as expected:
```
import mne
import os.path as op
from mne.datasets import testing
import numpy as np
from nose.tools import assert_true

data_path = testing.data_path(download=False)
sss_path = op.join(data_path, 'SSS')
fine_cal_fname = op.join(sss_path, 'sss_cal_3053.dat')
ctc_fname = op.join(sss_path, 'ct_sparse.fif')
pre = op.join(sss_path, 'test_move_anon_')
raw_fname = pre + 'raw.fif'
raw = mne.io.Raw(raw_fname, allow_maxshield='yes', preload=True)

raw_sss = mne.preprocessing.maxwell_filter(raw, calibration=fine_cal_fname,
                                           st_duration=2, st_only=True,
                                           cross_talk=ctc_fname)
raw.pick_types(meg='mag')
raw_sss.pick_types(meg='mag')
raw_sss.plot()
assert_true((np.linalg.norm(raw_sss[:][0], axis=-1) <
             np.linalg.norm(raw[:][0], axis=-1)).all())
```
But window-by-window the projection changes, so you get over multiple windows an up-down sort of behavior, which I assume is what you're pointing out? This might be fixed by doing overlap-add windowing, which is something I've discussed previously with Samu. I hope to get to it soon.
Oh wait... I think there might also something odd going on due to the DC value in the signal. I'll keep digging.
BTW anyone to lazy to run code, here's an example before/after tSSS, the 2-second-ness of the data processing is visible:
![screenshot from 2017-02-01 12-03-14](https://cloud.githubusercontent.com/assets/2365790/22517224/78b48e0c-e876-11e6-8967-0224af5199a7.png)
![screenshot from 2017-02-01 12-03-17](https://cloud.githubusercontent.com/assets/2365790/22517227/7a64da40-e876-11e6-9ec1-c2f5b7cc1b11.png)

The bad channel reconstruction was fixed by setting ``origin`` and ``destination`` manually, but the mag channels are still noisy with ``st_only=True``. On multimodal dataset it's even more prominent.
![mags](https://cloud.githubusercontent.com/assets/3456414/22539843/fe6a78f8-e91c-11e6-9936-1936cb8e9e58.png)

Can you paste the updated command lines so I can look at the same thing
locally?

That is from the multimodal dataset sub010. So just
```
raw = mne.preprocessing.maxwell_filter(raw, calibration=cal,
                                                   cross_talk=ctc,
                                                   st_duration=10.,
                                                   st_only=True,
                                                   origin=(0., 0., 0.04),
                                                   destination=destination)
```
what is `destination` and which run is that?
actually I think `destination` shouldn't matter for this, and I guess I can check any run and they'll probably display this behavior
The original raw data with no projectors looks about the same. `st_only` removes projectors, so it will look different unless you disable proj.

I guess in `st_only` mode we don't need to remove projectors, since tSSS by itself doesn't/shouldn't change the spatial configuration of the data.

Resolving patch:
diff --git a/mne/preprocessing/maxwell.py b/mne/preprocessing/maxwell.py
--- a/mne/preprocessing/maxwell.py
+++ b/mne/preprocessing/maxwell.py
@@ -49,12 +49,10 @@ def maxwell_filter(raw, origin='auto', int_order=8, ext_order=3,
 
     .. warning:: Automatic bad channel detection is not currently implemented.
                  It is critical to mark bad channels before running Maxwell
-                 filtering, so data should be inspected and marked accordingly
-                 prior to running this algorithm.
+                 filtering to prevent artifact spreading.
 
-    .. warning:: Not all features of Elekta MaxFilter™ are currently
-                 implemented (see Notes). Maxwell filtering in mne-python
-                 is not designed for clinical use.
+    .. warning:: Maxwell filtering in MNE is not designed or certified
+                 for clinical use.
 
     Parameters
     ----------
@@ -161,8 +159,9 @@ def maxwell_filter(raw, origin='auto', int_order=8, ext_order=3,
 
     See Also
     --------
-    mne.epochs.average_movements
+    mne.chpi.filter_chpi
     mne.chpi.read_head_pos
+    mne.epochs.average_movements
 
     Notes
     -----
@@ -172,34 +171,46 @@ def maxwell_filter(raw, origin='auto', int_order=8, ext_order=3,
     permission from Jussi Nurminen. These algorithms are based on work
     from [1]_ and [2]_.
 
-    Compared to Elekta's MaxFilter™ software, our Maxwell filtering
-    algorithm currently provides the following features:
-
-        * Bad channel reconstruction
-        * Cross-talk cancellation
-        * Fine calibration correction
-        * tSSS
-        * Coordinate frame translation
-        * Regularization of internal components using information theory
-        * Raw movement compensation
-          (using head positions estimated by MaxFilter)
-        * cHPI subtraction (see :func:`mne.chpi.filter_chpi`)
-
-    The following features are not yet implemented:
-
-        * **Not certified for clinical use**
-        * Automatic bad channel detection
-        * Head position estimation
-
-    Our algorithm has the following enhancements:
-
-        * Double floating point precision
-        * Handling of 3D (in addition to 1D) fine calibration files
-        * Automated processing of split (-1.fif) and concatenated files
-        * Epoch-based movement compensation as described in [1]_ through
-          :func:`mne.epochs.average_movements`
-        * **Experimental** processing of data from (un-compensated)
-          non-Elekta systems
+    Compared to Elekta's MaxFilter™ software, the MNE Maxwell filtering
+    routines currently provide the following features:
+
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Feature                                                                     | MNE | MaxFilter |
+    +=============================================================================+=====+===========+
+    | Maxwell filtering software shielding                                        | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Bad channel reconstruction                                                  | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Cross-talk cancellation                                                     | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Fine calibration correction (1D)                                            | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Fine calibration correction (3D)                                            | X   |           |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Spatio-temporal SSS (tSSS)                                                  | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Coordinate frame translation                                                | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Regularization using information theory                                     | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Movement compensation (raw)                                                 | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Movement compensation (:func:`epochs <mne.epochs.average_movements>`)       | X   |           |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | :func:`cHPI subtraction <mne.chpi.filter_chpi>`                             | X   | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Double floating point precision                                             | X   |           |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Seamless processing of split (``-1.fif``) and concatenated files            | X   |           |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Certified for clinical use                                                  |     | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Automatic bad channel detection                                             |     | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+    | Head position estimation                                                    |     | X         |
+    +-----------------------------------------------------------------------------+-----+-----------+
+
+    Epoch-based movement compensation is described in [1]_.
 
     Use of Maxwell filtering routines with non-Elekta systems is currently
     **experimental**. Worse results for non-Elekta systems are expected due
@@ -225,6 +236,11 @@ def maxwell_filter(raw, origin='auto', int_order=8, ext_order=3,
               These patents likely preclude the use of Maxwell filtering code
               in commercial applications. Consult a lawyer if necessary.
 
+    Currently, in order to perform Maxwell filtering, the raw data must not
+    have any projectors applied. During Maxwell filtering, the spatial
+    structure of the data is modified, so projectors are discarded (unless
+    in ``st_only=True`` mode).
+
     References
     ----------
     .. [1] Taulu S. and Kajola M. "Presentation of electromagnetic
@@ -238,7 +254,7 @@ def maxwell_filter(raw, origin='auto', int_order=8, ext_order=3,
            Physics in Medicine and Biology, vol. 51, pp. 1759-1768, 2006.
 
            http://lib.tkk.fi/Diss/2008/isbn9789512295654/article3.pdf
-    """
+    """  # noqa: E501
     # There are an absurd number of different possible notations for spherical
     # coordinates, which confounds the notation for spherical harmonics.  Here,
     # we purposefully stay away from shorthand notation in both and use
@@ -294,7 +310,9 @@ def maxwell_filter(raw, origin='auto', int_order=8, ext_order=3,
     raw_sss, pos_picks = _copy_preload_add_channels(
         raw, add_channels=add_channels)
     del raw
-    _remove_meg_projs(raw_sss)  # remove MEG projectors, they won't apply now
+    if not st_only:
+        # remove MEG projectors, they won't apply now
+        _remove_meg_projs(raw_sss)
     info = raw_sss.info
     meg_picks, mag_picks, grad_picks, good_picks, mag_or_fine = \
         _get_mf_picks(info, int_order, ext_order, ignore_ref)
@@ -905,7 +923,8 @@ def _check_regularize(regularize):
 def _check_usable(inst):
     """Ensure our data are clean."""
     if inst.proj:
-        raise RuntimeError('Projectors cannot be applied to data.')
+        raise RuntimeError('Projectors cannot be applied to data during '
+                           'Maxwell filtering.')
     current_comp = inst.compensation_grade
     if current_comp not in (0, None):
         raise RuntimeError('Maxwell filter cannot be done on compensated '


Test patch:
diff --git a/mne/preprocessing/tests/test_maxwell.py b/mne/preprocessing/tests/test_maxwell.py
--- a/mne/preprocessing/tests/test_maxwell.py
+++ b/mne/preprocessing/tests/test_maxwell.py
@@ -554,6 +554,7 @@ def test_spatiotemporal_only():
     power = np.sqrt(np.sum(raw[picks][0] ** 2))
     # basics
     raw_tsss = maxwell_filter(raw, st_duration=1., st_only=True)
+    assert_equal(len(raw.info['projs']), len(raw_tsss.info['projs']))
     assert_equal(raw_tsss.estimate_rank(), 366)
     _assert_shielding(raw_tsss, power, 10)
     # temporal proj will actually reduce spatial DOF with small windows!