Repository: mne-tools/mne-python

Issue:
Auto-scaling plot.raw in combination with remove_dc and/or filter
I often work with EEG data recorded without a highpass filter, which means that channels have large offsets different from zero. In a first step, I usually want to create a quick `raw.plot` to look at the raw data. Unfortunately, using `raw.plot(scalings='auto', remove_dc=True)` generally produces flat lines. This is despite setting `remove_dc=True`, the `scalings` are still inferred from the original data containing large offsets. Here's a short example to illustrate what I mean:

```
import numpy as np
import mne


data = np.random.normal(loc=500, scale=1, size=(10, 10000))
info = mne.create_info(ch_names=data.shape[0], sfreq=100)
raw = mne.io.RawArray(data, info)
raw.plot(scalings="auto", remove_dc=True)
```

<img width="1511" alt="Screen Shot 2019-06-04 at 11 20 38" src="https://user-images.githubusercontent.com/4377312/58867483-d2315c00-86ba-11e9-9697-c10501b302a8.png">

It would be really useful if `scalings` were determined from the offset-corrected values, which would result in the following plot (simulated with `np.random.normal(loc=0, scale=1, size=(10, 10000))` here):

<img width="1555" alt="Screen Shot 2019-06-04 at 11 22 14" src="https://user-images.githubusercontent.com/4377312/58867599-073dae80-86bb-11e9-9d57-01f5543c073c.png">

The same is true for the `highpass` and `lowpass` arguments. It would be useful if the automatic scalings were calculated on the filtered signals. Otherwise, I have to supply manually chosen values for `scalings` in order for the plot to actually show the signals (e.g. `raw.plot(scalings=2, remove_dc=True)` or channel type specific `raw.plot(scalings={'misc': 2}, remove_dc=True)`), because the default e.g. for EEG channels is not appropriate for many EEG signals I use.

What do you think? If this is a useful change, I'd be happy if someone could give me a pointer to where this should be implemented. Currently, we compute automatic scalings at the very start of `raw.plot`, but `remove_dc` and filtering is implemented in the `_update_raw_data` function which works only on the currently visible data.


Discussion:
+1 for autoscaling after filtering/dc correct

>

For filtering you'll need to load all data into memory, filter it, and compute.

For DC correction, the operation is done *on your current view of the data*. But we probably still get a good value by (hopefully quickly) using `make_fixed_length_events`, creating Epochs from this baselined with `(None, None)`, then getting the data and computing the scaling factors from that.
What if we add a new autoscale feature which works based on the current view of the data only?
To me it seems like a misleading (and challenging) way to look at the data to have the scale change as a function of which time window you've cropped to. If the primary motivation is just to avoid the technical challenge of loading into memory it doesn't seem like it's worth it.
But don't both filter and remove_dc arguments to `raw.plot` work only on the currently visible data segment? DC can vary tremendously if there is a large drift and you calculate the global mean vs. the mean within the visible segment.
Yes they both operate only on the given segment. Removing frequency content (what they do) on a per-window basis is different from changing the effective scale bar / pixel-to-units scaling on a per-window basis (what you propose).
Yes, I'm getting ahead of myself. Both options would be useful, but I'm OK with tackling the scaling based on the whole signal first. However, how do we consolidate `remove_dc` with this approach then? One is based on the whole signal while the other is based on the visible segment...
>  how do we consolidate remove_dc with this approach then?

See above -- construct an `Epochs` object with `baseline=(None, None)` using events from `make_fixed_length_events`, get the data from it, (effecitively) concatenate the epochs to be raw-like `(n_channels, n_times)`, and use this to compute the scale factors (using `std` or however we currently do it for the original data, and will do it for the filtered data)
Which epoch duration would you use?
The (starting) `raw.plot` duration
OK, I got myself into circular dependency hell. I'm trying to use `mne.make_fixed_length_events`, but if I do:

```
from .. import make_fixed_length_events
```

at the top of the module I get

```
ImportError: cannot import name 'make_fixed_length_events' from 'mne' (/Users/clemens/Repositories/mne-python/mne/__init__.py)`
```

It works if I do

```
from ..event import make_fixed_length_events
```

However, I have other imports that I need such as `RawArray` and `create_info`, and there is no way I can import these at the top without getting an `ImportError`.

Is it OK if I just import these functions within the function I need them? I know there are several approaches how to handle this situation, so I wanted to know what you guys preferred.
Turns out it is much easier to achieve a segment-based offset correction using plain old numpy, so I don't need the additional imports after all. I'm still interested on how you handle such cases though.

Also, it depends a lot on the actual data whether or not this new autoscale works well. For the toy example, I get a nice output:

<img width="1369" alt="Screen Shot 2019-06-05 at 12 20 38" src="https://user-images.githubusercontent.com/4377312/58949287-5bfd2a00-878c-11e9-9167-e511f408e57f.png">

But with one of my real-world EEG data sets, I still don't really see the signals, but the scaling factor determined by the autoscale procedure did improve significantly:

Old auto-scale: 0.2596253484038322
New auto-scale: 0.004032760298876942

What I'd consider a good value (where I can see the actual variations in the signals): 0.00005

So although this change improves the auto-estimate, it still doesn't work at least for the data set I've tested, probably because there are some huge artifacts within the 10s segments.

I don't know if it's worth adding this feature then. I guess as long as it doesn't make things worse it'll probably be OK to add it.


Resolving patch:
diff --git a/mne/viz/raw.py b/mne/viz/raw.py
--- a/mne/viz/raw.py
+++ b/mne/viz/raw.py
@@ -270,7 +270,8 @@ def plot_raw(raw, events=None, duration=10.0, start=0.0, n_channels=20,
     from scipy.signal import butter
     from ..io.base import BaseRaw
     color = _handle_default('color', color)
-    scalings = _compute_scalings(scalings, raw)
+    scalings = _compute_scalings(scalings, raw, remove_dc=remove_dc,
+                                 duration=duration)
     _validate_type(raw, BaseRaw, 'raw', 'Raw')
     n_channels = min(len(raw.info['chs']), n_channels)
     _check_option('clipping', clipping, [None, 'clamp', 'transparent'])
diff --git a/mne/viz/utils.py b/mne/viz/utils.py
--- a/mne/viz/utils.py
+++ b/mne/viz/utils.py
@@ -1638,7 +1638,7 @@ def _plot_sensors(pos, colors, bads, ch_names, title, show_names, ax, show,
     return fig
 
 
-def _compute_scalings(scalings, inst):
+def _compute_scalings(scalings, inst, remove_dc=False, duration=10):
     """Compute scalings for each channel type automatically.
 
     Parameters
@@ -1652,6 +1652,14 @@ def _compute_scalings(scalings, inst):
         The data for which you want to compute scalings. If data
         is not preloaded, this will read a subset of times / epochs
         up to 100mb in size in order to compute scalings.
+    remove_dc : bool
+        Whether to remove the mean (DC) before calculating the scalings. If
+        True, the mean will be computed and subtracted for short epochs in
+        order to compensate not only for global mean offset, but also for slow
+        drifts in the signals.
+    duration : float
+        If remove_dc is True, the mean will be computed and subtracted on
+        segments of length ``duration`` seconds.
 
     Returns
     -------
@@ -1703,9 +1711,17 @@ def _compute_scalings(scalings, inst):
                                  % (key, value))
             continue
         this_data = data[ch_types[key]]
-        scale_factor = np.percentile(this_data.ravel(), [0.5, 99.5])
-        scale_factor = np.max(np.abs(scale_factor))
-        scalings[key] = scale_factor
+        if remove_dc and (this_data.shape[1] / inst.info["sfreq"] >= duration):
+            length = int(duration * inst.info["sfreq"])  # segment length
+            # truncate data so that we can divide into segments of equal length
+            this_data = this_data[:, :this_data.shape[1] // length * length]
+            shape = this_data.shape  # original shape
+            this_data = this_data.T.reshape(-1, length, shape[0])  # segment
+            this_data -= this_data.mean(0)  # subtract segment means
+            this_data = this_data.T.reshape(shape)  # reshape into original
+
+        iqr = np.diff(np.percentile(this_data.ravel(), [25, 75]))
+        scalings[key] = iqr.item()
     return scalings
 
 


Test patch:
diff --git a/mne/preprocessing/tests/test_ica.py b/mne/preprocessing/tests/test_ica.py
--- a/mne/preprocessing/tests/test_ica.py
+++ b/mne/preprocessing/tests/test_ica.py
@@ -270,7 +270,7 @@ def test_ica_core(method):
         for line in fig.axes[0].lines:
             y = line.get_ydata()
             if len(y) > 2:  # actual data, not markers
-                assert np.ptp(y) < 10
+                assert np.ptp(y) < 15
         plt.close('all')
 
         sources = raw_sources[:, :][0]