Repository: mne-tools/mne-python

Issue:
EEG (re-)referencing is confusing
Hello, we were struggling with the issue of (re-)referencing EEG data in MNE, and I would like to share with you what caused our trouble, and suggest and discuss potential solutions. Everything presented here is based on the `master` branch.

Loosely following [the re-referencing example](http://martinos.org/mne/dev/auto_examples/preprocessing/plot_rereference_eeg.html), we did:

```python
import numpy as np
import matplotlib
import os
import mne
from mne.datasets import sample

matplotlib.use('Qt5Agg')

data_path = sample.data_path()
raw_fname = os.path.join(data_path, 'MEG/sample/sample_audvis_filt-0-40_raw.fif')
event_fname = os.path.join(data_path, 'MEG/sample/sample_audvis_filt-0-40_raw-eve.fif')
event_id, tmin, tmax = 1, -0.2, 0.5

raw = mne.io.read_raw_fif(raw_fname, preload=True)
events = mne.read_events(event_fname)

picks = mne.pick_types(raw.info, meg=False, eeg=True, eog=True, exclude='bads')
reject = dict(eeg=180e-6, eog=150e-6)
epochs_params = dict(events=events, event_id=event_id, tmin=tmin, tmax=tmax,
                     picks=picks, reject=reject, add_eeg_ref=False, preload=True)

# Let's remove all existing projectors from the example data set,
# since in "real life", we start with just the raw data.
raw.del_proj()

epochs = mne.Epochs(raw, **epochs_params)
```
During data cleaning, we'd sometimes switch between different reference electrodes.

```python
epochs.set_eeg_reference(['EEG 001']).plot()
epochs.set_eeg_reference(['EEG 010']).plot()
```
After the data is cleaned, we'd like to apply an average reference and view the evoked potentials. Thanks to #3756, one can easily switch to an average EEG reference, even if another "custom" reference had been applied earlier. 
```python
evoked = epochs.set_eeg_reference().average()
evoked.plot()
```
What we expected to look like a butterfly plot actually looked more like a butterfly that barely survived a freak accident. We were at a loss as to what went wrong. Later, I looked a bit closer at the output produced by the first line:
```python
Adding average EEG reference projection.
1 projection items deactivated
```
Until then, we had simply managed to entirely ignore these messages:
*Projection? Never heard of it. Must be an MEG thing that doesn't concern us!*

However, I discovered [the SSP manual](http://martinos.org/mne/stable/manual/preprocessing/ssp.html), and the following actually did do the trick:
```python
evoked = epochs.set_eeg_reference().apply_proj().average()
evoked.plot()
```

Neat! Maybe let's swtich back to another custom reference just to check something:
```python
epochs.set_eeg_reference(['EEG 002'])
```
Let's check out `epochs.info` for the sake of it (I'm showing only the relevant lines here):
```python
In[65]: epochs.info
Out[65]: 

<Info | 19 non-empty fields
    custom_ref_applied : 'bool | True
    projs : 'list | Average EEG reference: on
>
```
So now we somehow have both, an average reference (applied projection) and our custom reference (`EEG 002`). The projection cannot be removed, so now I'm stuck forever and always with the average reference.

But wait, there might be a workaround!

```python
epochs = mne.Epochs(raw, **epochs_params)
evoked = epochs.set_eeg_reference(epochs.info['ch_names'][:-1]).average()
evoked.plot()
```
Looks good!

Let's take a peek at `epochs.info` again:
```python
In[69]: epochs.info
Out[69]: 

<Info | 19 non-empty fields
    custom_ref_applied : 'bool | True
    projs : 'list | 0 items
>
```
Although we effectively use an average reference, apparently no projector was added this time.

We can now also easily switch back and forth between average and custom references:
```python
In[83]: np.allclose((epochs
   ...:              .copy()
   ...:              .set_eeg_reference(['EEG 002'])
   ...:              .get_data()),
   ...:             (epochs
   ...:              .copy()
   ...:              .set_eeg_reference(epochs.info['ch_names'][:-1])
   ...:              .set_eeg_reference(['EEG 002'])
   ...:              .get_data()))
   ...: 
Applying a custom EEG reference.
Applying a custom EEG reference.
Applying a custom EEG reference.
Out[83]: 
True
```
This works exactly as I would have expected.

I find this behavior somewhat confusing, because it appears kind of inconsistent to me: sometimes, `Epochs.set_eeg_reference()` will add what I would commonly consider an *ordninary* EEG reference; but on some occasions, it it would create this weird thing called a *projector* that takes an additional step to activate, and then it's forever stuck to the data. This is nothing I've ever seen in any EEG papers I've read so far.

In my opinion, `Epochs.set_eeg_reference()` should never add a projector unless explicitly requested. `Epochs.set_eeg_reference(ref_channels=None)` should effectively translate into `Epochs.set_eeg_reference(ref_channels=epochs.info['ch_names'][:-1])` instead of adding a projector. 

In the  [example code](http://martinos.org/mne/dev/auto_examples/preprocessing/plot_rereference_eeg.html) I mentioned earlier, the (average reference) projectors are implicitly automatically applied when constructing the `Epochs` objects (since `proj=True` by default in `Epochs.__init__()`). For that reason, we entirely overlooked the need to call `apply_proj()`: In our workflow, average references
are added to an existing `Epochs` instance once all the data cleaning has been done, immediately before calculating GFP etc. However, `Epochs.average()` does not apply the projectors automatically, so we ended up with weirdly looking "butterfly" plots.

I've asked several researchers who'd been working with EEG data for several years now whether they could spontaneously associate the words *EEG*, *projector*, and *projection* in the context of EEG *pre-processing* and, potentially, *artifact removal*. The responses were all negative.

Even if you assume I might be kind of ignorant or stupid by now, I do believe that either small API or implementation changes (as proposed here), enhanced documentation, and "improved" / new example code could make MNE much more approachable to those who have no experience with MEG so far. I'd be willing to help working on this, but first I will have to finish #3967, which has been stalled for a while already...


Discussion:
> What we expected to look like a butterfly plot actually looked more like a butterfly that barely survived a freak accident.

:)

> In my opinion, Epochs.set_eeg_reference() should never add a projector unless explicitly requested. 
> ... I find this behavior somewhat confusing, because it appears kind of inconsistent to me

It is indeed inconsistent. From a source imaging (and historical) perspective, projections are preferred, so we use them whenever possible. So for our purposes let's assume we need to stick with it, and figure out how to best make the situation better for EEG folks.

Can you think of a good way to modify the docstring for `set_eeg_reference` that makes this approach clearer? It would be especially useful if you could open a PR to add better text to that function. We should also change [the reference example](https://mne-tools.github.io/stable/auto_examples/preprocessing/plot_rereference_eeg.html) into a tutorial at some point. Feel free to do that if you want, but even without doing that, can you look to see if there are things there that could be made clearer?

> projectors are implicitly automatically applied when constructing the Epochs objects

This is something that has come up a few times as being problematic, actually: #3664, #3410, #2310... still trying to figure out the best solution.

> Until then, we had simply managed to entirely ignore these messages:
> Projection? Never heard of it. Must be an MEG thing that doesn't concern us!

How about rephrasing to:
```
Adding average EEG reference as a projection operator.
Use apply_proj() to apply the average projection.
```
Would that have worked? If so, I can add that.

> So now we somehow have both, an average reference (applied projection) and our custom reference (EEG 002). The projection cannot be removed, so now I'm stuck forever and always with the average reference.

That's a bug. @wmvanvliet do you have time to try to fix it? The projector should be removed if a new reference is set.

> Even if you assume I might be kind of ignorant or stupid by now

I [agree with Don Norman](https://www.amazon.com/gp/product/0465050654) that things like this should not be so hard, and that user struggles usually due to API and documentation. In this case I think the API is pretty settled (beyond the bug mentioned above), but hopefully we can make the docs better.

>  first I will have to finish #3967, which has been stalled for a while already...

Looking at that issue, I think it needs a couple of (hopefully) small changes. So feel free to tackle that one first if you want. But most contributors are guilty of having multiple PRs open at once, so don't let #3967 stop you from tackling this, especially since the considerations here are fresh in your mind.
Hi Eric, thanks for your response!

> It is indeed inconsistent. From a source imaging (and historical) perspective, projections are preferred, so we use them whenever possible.

Ah, I already suspected that! What weird guys you are! ;) (Just kidding, of course. Very interesting how approaches can differ!)

> So for our purposes let's assume we need to stick with it, and figure out how to best make the situation better for EEG folks.
> 
> Can you think of a good way to modify the docstring for set_eeg_reference that makes this approach clearer? It would be especially useful if you could open a PR to add better text to that function. We should also change the reference example into a tutorial at some point. Feel free to do that if you want, but even without doing that, can you look to see if there are things there that could be made clearer?

I do have some ideas on my mind, but I think a (backward-compatible) API change could be a better approach. I will explain below.

> How about rephrasing to:
> 
> Adding average EEG reference as a projection operator.
> Use apply_proj() to apply the average projection.
> Would that have worked? If so, I can add that.

This would be a more clear message indeed, yes. Still I wonder if it would actually have helped many EEG folks: I believe the major issue is non-familiarity with a specific concept (projections) instead of actual phrasing. **Remember, I actually somehow managed to completely ignore the whole projector messages at first!** So we're dealing with a large amount of persistent ignorance here; this is not to be taken lightly! No but seriously: assuming we insisted on always doing average referencing via a projection no matter what, the message should be even more extensive, something along the lines of:
```
Adding average EEG reference as an INACTIVE projection operator.
!!! The EEG data is not actually referenced to average yet !!!
You must call apply_proj() to apply the projection and reference to average.
```
... and add some blinking exclamation marks and such ;)

> So now we somehow have both, an average reference (applied projection) and our custom reference (EEG 002). The projection cannot be removed, so now I'm stuck forever and always with the average reference.
> That's a bug. @wmvanvliet do you have time to try to fix it? The projector should be removed if a new reference is set.

I was referring to a different issue here :) Namely that you cannot reverse a projection operation:

```python
In[7]: (epochs
  ...:  .copy()
  ...:  .set_eeg_reference()
  ...:  .apply_proj()
  ...:  .del_proj())
Adding average EEG reference projection.
1 projection items deactivated
Created an SSP operator (subspace dimension = 1)
1 projection items activated
SSP projectors applied...
Traceback (most recent call last):
  File "/Users/hoechenberger/anaconda3/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-95a9933f4fb3>", line 1, in <module>
    (epochs
  File "/Users/hoechenberger/Development/mne-python/mne/io/proj.py", line 200, in del_proj
    raise ValueError('Cannot remove projectors that have already '
ValueError: Cannot remove projectors that have already been applied
```
Which brings me back to my idea regarding a potential API change.

Currently, `set_eeg_reference()` accepts its argument `ref_channels` to be either `None`, an empty list (which will probably be removed, according to #3981), or a list of channel names (actually, it also accepts strings and then tries to treat them like lists, which usually won't work... but that's another issue :D).

How about we additionally allow `ref_channels='average_proj'` (which would behave like `ref_channels=None`) and `ref_channels='average'`. The latter would *not* create a projection, but do something like
```python
set_eeg_reference(inst.info['ch_names'][first_eeg_chan:last_eeg_chan+1])
```
as I demonstrated in my earlier posting. 
*(Please note that there was a bug in my original posting: it would also include the EOG channel in the calculation. This has been corrected now. BTW is there any easy way to figure out which channels belong to MEG, EEG, EOG etc. without parsing `info['ch_names']`? **edit** just discovered `io.pick.channel_type()`, which should do the trick...)*

This approach
- would remove the need of having to apply the average projector and
- therefore allow users to switch back and forth between "custom" and average references as desired,
- thereby making the behavior much more consistent
- would make the API very explicit for both EEG and MEG users
- would make the API and behavior much more approachable to EEG folks and avoid confusion, as they could simply entirely ignore the projector parts.

What do you think?

> first I will have to finish #3967, which has been stalled for a while already...
> Looking at that issue, I think it needs a couple of (hopefully) small changes. 

Yes, my first approach wasn't tackling the actual problem, I assume. But I think I've figured it out in the meantime; just didn't have the time to implement the changes :)

Thanks for your support!
> I agree with Don Norman that things like this should not be so hard

Wow, thanks for the reference! I just ordered this book.
can someone take care of the bug fix reported here? thx

I'll take a look
I verified that the following works as expected:

```python
epochs.set_eeg_reference().apply_proj()  # Set average reference
epochs.set_eeg_reference(['EEG 001'])  # Re-reference to channel EEG 001
```

There is no bug here. When you look at the info structure, you indeed see this:

```python
In[65]: epochs.info
Out[65]: 

<Info | 19 non-empty fields
    custom_ref_applied : 'bool | True
    projs : 'list | Average EEG reference: on
>
```

An "average" projector that has been applied and a custom reference that has been applied. Projectors are never removed from the info structure. When they have been applied to the data, they are flagged as such and thereafter become inert. I agree that the term "active" and it's value "on" are confusing. It conveys the message that the projection is actively being applied (hence overriding the custom reference) but it's not, it indicates that the projection has been applied somewhere in the past.

We're not likely to change the behavior of the projection code. We can tweak the printing of `info` a bit if we can agree on less confusing terminology, but most of all, this is a documentation issue. 
@hoechenberger 's initial code is completely right, does exactly what he hoped to achieve, but still managed to confuse him :) I agree with @Eric89GXL that it would be good to expand the referencing example in a comprehensive tutorial.
the behavior of projectors is actually more complicated then I led on. When `Epochs` are created with `preload=False`, an active projector is indeed actively being applied when data is being accessed. There may actually be a bug here. When doing `epochs.set_eeg_reference()`, and the data is preloaded, the average ref projector should be applied straight away, because the `Epochs` object will not do so automatically in `preload=False` mode.

You know what? I'll take a shot at writing a tutorial on the behavior of projectors first. They still confuse me even after reading through the code 5+ times.
well when changing the EEG ref manually you undo the EEG Average Ref so
technically the proj is not On anymore. I think the EEG average ref shoud
be marked as off as soon as you undo it.

that would cause the average reference to be re-applied if you ever called `.apply_proj()` again
to make it actually be gone, it should be completely removed from the projections list.
>
> that would cause the average reference to be re-applied if you ever called
> .apply_proj() again
>
yes but is it a problem?

yes, because some functions (such as plotting evokeds) will call `inst.apply_proj`, expecting it to be a null operation of projections have already been applied.
or at least expecting that the user would like all the projections to be applied. The code currently cannot deal with projections that the user *doesn't* want to be applied.
It should be okay if we also remove it from the list

sounds like a plan


Resolving patch:
diff --git a/examples/preprocessing/plot_rereference_eeg.py b/examples/preprocessing/plot_rereference_eeg.py
--- a/examples/preprocessing/plot_rereference_eeg.py
+++ b/examples/preprocessing/plot_rereference_eeg.py
@@ -31,7 +31,7 @@
 picks = mne.pick_types(raw.info, meg=False, eeg=True, eog=True, exclude='bads')
 
 ###############################################################################
-# Apply different EEG referencing schemes and plot the resulting evokeds
+# Apply different EEG referencing schemes and plot the resulting evokeds.
 
 reject = dict(eeg=180e-6, eog=150e-6)
 epochs_params = dict(events=events, event_id=event_id, tmin=tmin, tmax=tmax,
@@ -41,25 +41,21 @@
 
 # No reference. This assumes that the EEG has already been referenced properly.
 # This explicitly prevents MNE from adding a default EEG reference.
-raw_no_ref, _ = mne.io.set_eeg_reference(raw, [])
-evoked_no_ref = mne.Epochs(raw_no_ref, **epochs_params).average()
-del raw_no_ref  # Free memory
+raw.set_eeg_reference([])
+evoked_no_ref = mne.Epochs(raw, **epochs_params).average()
 
 evoked_no_ref.plot(axes=ax1, titles=dict(eeg='EEG Original reference'))
 
 # Average reference. This is normally added by default, but can also be added
 # explicitly.
-raw_car, _ = mne.io.set_eeg_reference(raw)
-evoked_car = mne.Epochs(raw_car, **epochs_params).average()
-del raw_car
+raw.set_eeg_reference()
+evoked_car = mne.Epochs(raw, **epochs_params).average()
 
 evoked_car.plot(axes=ax2, titles=dict(eeg='EEG Average reference'))
 
-# Use the mean of channels EEG 001 and EEG 002 as a reference
-raw_custom, _ = mne.io.set_eeg_reference(raw, ['EEG 001', 'EEG 002'])
-evoked_custom = mne.Epochs(raw_custom, **epochs_params).average()
-del raw_custom
+# Re-reference from an average reference to the mean of channels EEG 001 and
+# EEG 002.
+raw.set_eeg_reference(['EEG 001', 'EEG 002'])
+evoked_custom = mne.Epochs(raw, **epochs_params).average()
 
 evoked_custom.plot(axes=ax3, titles=dict(eeg='EEG Custom reference'))
-
-mne.viz.tight_layout()
diff --git a/mne/io/reference.py b/mne/io/reference.py
--- a/mne/io/reference.py
+++ b/mne/io/reference.py
@@ -74,25 +74,31 @@ def _apply_reference(inst, ref_from, ref_to=None):
         ref_to = [inst.ch_names[i] for i in eeg_idx]
 
     # After referencing, existing SSPs might not be valid anymore.
+    projs_to_remove = []
     for i, proj in enumerate(inst.info['projs']):
-        if (not proj['active'] and
-            len([ch for ch in (ref_from + ref_to)
-                 if ch in proj['data']['col_names']]) > 0):
-
-            # Remove any average reference projections, apply any other types
-            if proj['desc'] == 'Average EEG reference' or \
-                    proj['kind'] == FIFF.FIFFV_MNE_PROJ_ITEM_EEG_AVREF:
-                logger.info('Removing existing average EEG reference '
-                            'projection.')
-                del inst.info['projs'][i]
-            else:
-                logger.info(
-                    'Inactive signal space projection (SSP) operators are '
-                    'present that operate on sensors involved in the current '
-                    'referencing scheme. Applying them now. Be aware that '
-                    'after re-referencing, these operators will be invalid.')
-                inst.apply_proj()
-            break
+        # Remove any average reference projections
+        if proj['desc'] == 'Average EEG reference' or \
+                proj['kind'] == FIFF.FIFFV_MNE_PROJ_ITEM_EEG_AVREF:
+            logger.info('Removing existing average EEG reference '
+                        'projection.')
+            # Don't remove the projection right away, but do this at the end of
+            # this loop.
+            projs_to_remove.append(i)
+
+        # Inactive SSPs may block re-referencing
+        elif (not proj['active'] and
+              len([ch for ch in (ref_from + ref_to)
+                   if ch in proj['data']['col_names']]) > 0):
+
+            raise RuntimeError(
+                'Inactive signal space projection (SSP) operators are '
+                'present that operate on sensors involved in the desired '
+                'referencing scheme. These projectors need to be applied '
+                'using the apply_proj() method function before the desired '
+                'reference can be set.'
+            )
+    for i in projs_to_remove:
+        del inst.info['projs'][i]
 
     ref_from = [inst.ch_names.index(ch) for ch in ref_from]
     ref_to = [inst.ch_names.index(ch) for ch in ref_to]
@@ -233,22 +239,51 @@ def add_reference_channels(inst, ref_channels, copy=True):
 
 @verbose
 def set_eeg_reference(inst, ref_channels=None, copy=True, verbose=None):
-    """Rereference EEG channels to new reference channel(s).
+    """Specify which reference to use for EEG data.
+
+    By default, MNE-Python will automatically re-reference the EEG signal to
+    use an average reference (see below). Use this function to explicitly
+    specify the desired reference for EEG. This can be either an existing
+    electrode or a new virtual channel. This function will re-reference the
+    data according to the desired reference and prevent MNE-Python from
+    automatically adding an average reference.
+
+    Some common referencing schemes and the corresponding value for the
+    ``ref_channels`` parameter:
+
+    No re-referencing:
+        If the EEG data is already using the proper reference, set
+        ``ref_channels=[]``. This will prevent MNE-Python from automatically
+        re-referencing the data to an average reference.
+
+    Average reference:
+        A new virtual reference electrode is created by averaging the current
+        EEG signal. Make sure that all bad EEG channels are properly marked
+        and set ``ref_channels=None``.
+
+    A single electrode:
+        Set ``ref_channels`` to the name of the channel that will act as the
+        new reference.
+
+    The mean of multiple electrodes:
+        A new virtual reference electrode is created by computing the average
+        of the current EEG signal recorded from two or more selected channels.
+        Set ``ref_channels`` to a list of channel names, indicating which
+        channels to use. For example, to apply an average mastoid reference,
+        when using the 10-20 naming scheme, set ``ref_channels=['M1', 'M2']``.
 
-    If multiple reference channels are specified, they will be averaged. If
-    no reference channels are specified, an average reference will be applied.
 
     Parameters
     ----------
     inst : instance of Raw | Epochs | Evoked
         Instance of Raw or Epochs with EEG channels and reference channel(s).
     ref_channels : list of str | None
-        The names of the channels to use to construct the reference. If
-        None (default), an average reference will be added as an SSP
-        projector but not immediately applied to the data. If an empty list
-        is specified, the data is assumed to already have a proper reference
-        and MNE will not attempt any re-referencing of the data. Defaults
-        to an average reference (None).
+        The names of the channels to use to construct the reference. To apply
+        an average reference, specify ``None`` here (default). It will be added
+        as an SSP projector, so it is not immediately applied to the data. If
+        an empty list is specified, the data is assumed to already have a
+        proper reference and MNE will not attempt any re-referencing of the
+        data. Defaults to an average reference (None).
     copy : bool
         Specifies whether the data will be copied (True) or modified in place
         (False). Defaults to True.
@@ -277,6 +312,11 @@ def set_eeg_reference(inst, ref_channels=None, copy=True, verbose=None):
     3. In order to apply a reference other than an average reference, the data
        must be preloaded.
 
+    4. Re-referencing to an average reference is done with an SSP projector.
+       This allows applying this reference without preloading the data. Be
+       aware that on preloaded data, SSP projectors are not automatically
+       applied. Use the ``apply_proj()`` method to apply them.
+
     .. versionadded:: 0.9.0
 
     See Also
@@ -284,6 +324,10 @@ def set_eeg_reference(inst, ref_channels=None, copy=True, verbose=None):
     set_bipolar_reference : Convenience function for creating bipolar
                             references.
     """
+    if not isinstance(inst, (BaseRaw, BaseEpochs, Evoked)):
+        raise ValueError('Setting a reference is only supported for instances '
+                         'of Raw, Epochs or Evoked.')
+
     if ref_channels is None:
         # CAR requested
         if _has_eeg_average_ref_proj(inst.info['projs']):
@@ -298,15 +342,28 @@ def set_eeg_reference(inst, ref_channels=None, copy=True, verbose=None):
                 inst.info['custom_ref_applied'] = False
                 inst.add_proj(make_eeg_average_ref_proj(inst.info,
                               activate=False))
+                # If the data has been preloaded, projections will no longer be
+                # automatically applied.
+                if isinstance(inst, Evoked) or inst.preload:
+                    logger.info('Average reference projection was added, but '
+                                "hasn't been applied yet. Use the "
+                                '.apply_proj() method function to apply '
+                                'projections.')
             except:
                 inst.info['custom_ref_applied'] = custom_ref_applied
                 raise
 
             return inst, None
+
+    if ref_channels == []:
+        logger.info('EEG data marked as already having the desired reference. '
+                    'Preventing automatic future re-referencing to an average '
+                    'reference.')
     else:
         logger.info('Applying a custom EEG reference.')
-        inst = inst.copy() if copy else inst
-        return _apply_reference(inst, ref_channels)
+
+    inst = inst.copy() if copy else inst
+    return _apply_reference(inst, ref_channels)
 
 
 def set_bipolar_reference(inst, anode, cathode, ch_name=None, ch_info=None,


Test patch:
diff --git a/mne/io/tests/test_reference.py b/mne/io/tests/test_reference.py
--- a/mne/io/tests/test_reference.py
+++ b/mne/io/tests/test_reference.py
@@ -17,7 +17,7 @@
 from mne.epochs import BaseEpochs
 from mne.io import read_raw_fif
 from mne.io.constants import FIFF
-from mne.io.proj import _has_eeg_average_ref_proj
+from mne.io.proj import _has_eeg_average_ref_proj, Projection
 from mne.io.reference import _apply_reference
 from mne.datasets import testing
 from mne.utils import run_tests_if_main
@@ -49,7 +49,8 @@ def _test_reference(raw, reref, ref_data, ref_from):
         _reref = reref._data
 
     # Check that the ref has been properly computed
-    assert_array_equal(ref_data, _data[..., picks_ref, :].mean(-2))
+    if ref_data is not None:
+        assert_array_equal(ref_data, _data[..., picks_ref, :].mean(-2))
 
     # Get the raw EEG data and other channel data
     raw_eeg_data = _data[..., picks_eeg, :]
@@ -59,16 +60,17 @@ def _test_reference(raw, reref, ref_data, ref_from):
     reref_eeg_data = _reref[..., picks_eeg, :]
     reref_other_data = _reref[..., picks_other, :]
 
-    # Undo rereferencing of EEG channels
-    if isinstance(raw, BaseEpochs):
-        unref_eeg_data = reref_eeg_data + ref_data[:, np.newaxis, :]
-    else:
-        unref_eeg_data = reref_eeg_data + ref_data
-
-    # Check that both EEG data and other data is the same
-    assert_allclose(raw_eeg_data, unref_eeg_data, 1e-6, atol=1e-15)
+    # Check that non-EEG channels are untouched
     assert_allclose(raw_other_data, reref_other_data, 1e-6, atol=1e-15)
 
+    # Undo rereferencing of EEG channels if possible
+    if ref_data is not None:
+        if isinstance(raw, BaseEpochs):
+            unref_eeg_data = reref_eeg_data + ref_data[:, np.newaxis, :]
+        else:
+            unref_eeg_data = reref_eeg_data + ref_data
+        assert_allclose(raw_eeg_data, unref_eeg_data, 1e-6, atol=1e-15)
+
 
 @testing.requires_testing_data
 def test_apply_reference():
@@ -110,10 +112,33 @@ def test_apply_reference():
     assert_true(reref.info['custom_ref_applied'])
     _test_reference(evoked, reref, ref_data, ['EEG 001', 'EEG 002'])
 
-    # Test invalid input
+    # Referencing needs data to be preloaded
     raw_np = read_raw_fif(fif_fname, preload=False)
     assert_raises(RuntimeError, _apply_reference, raw_np, ['EEG 001'])
 
+    # Test having inactive SSP projections that deal with channels involved
+    # during re-referencing
+    raw = read_raw_fif(fif_fname, preload=True)
+    raw.add_proj(
+        Projection(
+            active=False,
+            data=dict(
+                col_names=['EEG 001', 'EEG 002'],
+                row_names=None,
+                data=[[1, 1]],
+                ncol=2,
+                nrow=1
+            ),
+            desc='test',
+            kind=1,
+        )
+    )
+    # Projection concerns channels mentioned in projector
+    assert_raises(RuntimeError, _apply_reference, raw, ['EEG 001'])
+
+    # Projection does not concern channels mentioned in projector, no error
+    _apply_reference(raw, ['EEG 003'], ['EEG 004'])
+
 
 @testing.requires_testing_data
 def test_set_eeg_reference():
@@ -125,13 +150,26 @@ def test_set_eeg_reference():
     assert_true(not _has_eeg_average_ref_proj(raw.info['projs']))
     reref, ref_data = set_eeg_reference(raw)
     assert_true(_has_eeg_average_ref_proj(reref.info['projs']))
+    assert_true(not reref.info['projs'][0]['active'])
     assert_true(ref_data is None)
+    reref.apply_proj()
+    eeg_chans = [raw.ch_names[ch]
+                 for ch in pick_types(raw.info, meg=False, eeg=True)]
+    _test_reference(raw, reref, ref_data,
+                    [ch for ch in eeg_chans if ch not in raw.info['bads']])
 
     # Test setting an average reference when one was already present
-    with warnings.catch_warnings(record=True):  # weight tables
+    with warnings.catch_warnings(record=True):
         reref, ref_data = set_eeg_reference(raw, copy=False)
     assert_true(ref_data is None)
 
+    # Test setting an average reference on non-preloaded data
+    raw_nopreload = read_raw_fif(fif_fname, preload=False)
+    raw_nopreload.info['projs'] = []
+    reref, ref_data = set_eeg_reference(raw_nopreload)
+    assert_true(_has_eeg_average_ref_proj(reref.info['projs']))
+    assert_true(not reref.info['projs'][0]['active'])
+
     # Rereference raw data by creating a copy of original data
     reref, ref_data = set_eeg_reference(raw, ['EEG 001', 'EEG 002'], copy=True)
     assert_true(reref.info['custom_ref_applied'])
@@ -156,11 +194,19 @@ def test_set_eeg_reference():
     assert_raises(ValueError, set_eeg_reference, reref)
     assert_true(reref.info['custom_ref_applied'])
 
+    # Test moving from average to custom reference
+    reref, ref_data = set_eeg_reference(raw)
+    reref, _ = set_eeg_reference(reref, ['EEG 001', 'EEG 002'])
+    assert_true(not _has_eeg_average_ref_proj(reref.info['projs']))
+    assert_equal(reref.info['custom_ref_applied'], True)
+
 
 @testing.requires_testing_data
 def test_set_bipolar_reference():
     """Test bipolar referencing."""
     raw = read_raw_fif(fif_fname, preload=True)
+    raw.apply_proj()
+
     reref = set_bipolar_reference(raw, 'EEG 001', 'EEG 002', 'bipolar',
                                   {'kind': FIFF.FIFFV_EOG_CH,
                                    'extra': 'some extra value'})