Repository: mne-tools/mne-python

Issue:
Defect montages shipped with MNE?
As mentioned in #4400, these two montages seem to have duplicate channel positions:

- `mne/channels/montages/standard_postfixed.elc`
- `mne/channels/montages/standard_primed.elc`

Furthermore, we currently have no description for `mne/channels/montages/10-5-System_Mastoids_EGI129.csd`, which contains a total of 465 locations. The last 129 locations are just numbers. Besides the cumbersome file name, we should make sure that this is a valid and useful montage (this file was also not documented in the docstring of the `read_montage` function).

EDIT by @larsoner: We should:

1. Deprecate `transform=False` -> `True` for internal consistency.
2. Make `montage.plot` do something smart with overlapping locations (newline separation?)


Discussion:
@dengemann any insight on these montages? You added them ages ago (2014)
Something else I've noticed. The elements in the `ch_names` list have different types, especially the ones with byte elements could be problematic:

```
10-5-System_Mastoids_EGI129 <class 'numpy.bytes_'>
biosemi128 <class 'numpy.str_'>
biosemi16 <class 'numpy.str_'>
biosemi160 <class 'numpy.str_'>
biosemi256 <class 'numpy.str_'>
biosemi32 <class 'numpy.str_'>
biosemi64 <class 'numpy.str_'>
easycap-M1 <class 'numpy.str_'>
easycap-M10 <class 'numpy.str_'>
EGI_256 <class 'numpy.bytes_'>
GSN-HydroCel-128 <class 'str'>
GSN-HydroCel-129 <class 'str'>
GSN-HydroCel-256 <class 'str'>
GSN-HydroCel-257 <class 'str'>
GSN-HydroCel-32 <class 'str'>
GSN-HydroCel-64_1.0 <class 'str'>
GSN-HydroCel-65_1.0 <class 'str'>
standard_1005 <class 'str'>
standard_1020 <class 'str'>
standard_alphabetic <class 'str'>
standard_postfixed <class 'str'>
standard_prefixed <class 'str'>
standard_primed <class 'str'>
```

This was fixed in #4427.
Yes we should also fix those
not fixed by #4427 ?
Not entirely, there's still the issue with duplicate locations and the weird montage.
I guess the things in the original post are still unanswered.
@dengemann any time to look in the next week or two to see if you can make sense of these files you added a few years back? It would be good to fix this for 0.15
ping @dengemann any insight on these?
I think we can remove those. Early attempts to support csd transform that
got stuck.
On Wed, 16 Aug 2017 at 22:11, Eric Larson <notifications@github.com> wrote:

> ping @dengemann <https://github.com/dengemann> any insight on these?
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/4403#issuecomment-322884809>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AB0fivoHQtnZAHsPAH9g5NgeQebdK-KAks5sY0zdgaJpZM4OczN->
> .
>

So remove both or just one of these:

* `mne/channels/montages/standard_postfixed.elc`
* `mne/channels/montages/standard_primed.elc`

Ah wait these I don't know. I think they must be from fieldtrip.
On Wed, 16 Aug 2017 at 22:27, Eric Larson <notifications@github.com> wrote:

> So remove both or just one of these:
>
>    - mne/channels/montages/standard_postfixed.elc
>    - mne/channels/montages/standard_primed.elc
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/4403#issuecomment-322889053>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AB0fioMnab32PJDHm6bhbsOOWxWXrhq-ks5sY1DNgaJpZM4OczN->
> .
>

The mysterious `10-5-System_Mastoids_EGI129.csd` file is already gone from our repo, so I'll close this then
@larsoner but what about the other two? We should either correct the duplicate locations or remove the montages (because we have other standard montages).
I am almost certain they are from Fieldtrip. Maybe we can see if they are
still the same there and pull their updates in case they received
corrections.
On Thu, 17 Aug 2017 at 08:46, Clemens Brunner <notifications@github.com>
wrote:

> @larsoner <https://github.com/larsoner> but what about the other two? We
> should either correct the duplicate locations or remove the montages
> (because we have other standard montages).
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/4403#issuecomment-322983692>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AB0fionWn6bnxfYympyvdOzBF_w75IBRks5sY-HZgaJpZM4OczN->
> .
>

Right, it looks like we use all Fieldtrip montages in MNE 😄 : https://github.com/fieldtrip/fieldtrip/tree/master/template/electrode

They haven't been updated in years, so they're identical to what we're using.
I assume it is useful to have them (even if the physical locations are identical) because the channel names differ, which could make them easier to apply to your data.
You think the identical locations really have two alternative names? Even if this is true I'd still find it ugly, because then why bother with having four different montages for the standard locations when we could put all names into one?
> You think the identical locations really have two alternative names?

Look at the files, the channel names they contain are different

> why bother with having four different montages for the standard locations when we could put all names into one?

This would require re-working the reading functions, and probably wouldn't otherwise change the user-facing API, so I don't see much point
> Look at the files, the channel names they contain are different

I actually looked at the montages from FieldTrip, and they look weird. Not only do some of them have channels with identical locations, but they look wrong in general. For example, the Cx channels should be the central channels from left to right ears - they are too posterior.

Compare one of the FieldTrip montages:
![fieldtrip_1020](https://user-images.githubusercontent.com/4377312/29446929-4f4aaa14-83ef-11e7-90b7-f49f550517cc.png)

And here's a 10-20 montage from EEGLAB:
![eeglab_1020](https://user-images.githubusercontent.com/4377312/29446936-60780aac-83ef-11e7-8a13-c9d07281dee0.png)

The Biosemi montages shipped with MNE look correct (i.e. just like the one from EEGLAB).
This looks like a coordinate frame problem -- the `FieldTrip` one looks suspiciously like it's not in head coords (a tilt forward of the head would make it look at least closer to the other one). Can you check to see if they are in head coords? (You can tell quickly because they RPA, LPA, and Nasion should each have two zero-valued coordinates.)

If that doesn't make them equivalent, then we might want to bother the FieldTrip folks to sort this out.
Here are the coordinates for LPA, RPA, and Nz:

```
[ -8.60761000e-02,  -1.99897000e-02,  -4.79860000e-02]
[  8.57939000e-02,  -2.00093000e-02,  -4.80310000e-02]
[  8.30000000e-06,   8.68110000e-02,  -3.99830000e-02]
```
Yeah those are incorrect (not Neuromag head coords). Next up would be to try `get_ras_to_neuromag_trans` to get the transformation, and `mne.transforms.apply_trans` to the points (double-checking that LPA/Nasion/RPA are now in reasonable locations), setting those as `montage.pos`, and then re-plotting the positions. Feel free to do it or I can.

In any case, we'll almost certainly want to have our montages in head coords, so we should probably fix it.
Oh actually we have it already. You just need `transform=True` in `read_montage`. Can you try it to see if it fixes it?
@teonbrooks any idea why we default to `transform=False` in our montage-reading code? Everything else in MNE is in Neuromag head coords, seems like we'd want to (at least by default) make them in head coords, too.
@larsoner yeah, this looks better:

![screen shot 2017-08-18 at 15 42 55](https://user-images.githubusercontent.com/4377312/29461413-e8ea2122-842b-11e7-994b-778cca228d1a.png)

```
array([[ -8.60701216e-02,   3.46944695e-18,   6.93889390e-18],
       [  8.57998854e-02,   0.00000000e+00,   0.00000000e+00],
       [ -2.40049137e-18,   1.07111604e-01,   6.93889390e-18]])
```

So are all montages shipped with MNE in other coordinates? In other words, should we always set `transform=True`? I noticed that it doesn't make a huge difference for e.g. `biosemi64` other than the transformed locations are a bit more condensed toward the center.
Still I don't like the two montages with duplicate channel names...
> In other words, should we always set transform=True? 

Yes I think it's worth a deprecation cycle to go to `transform=True`, we assume EEG coords are Neuromag head coords in various places.

> Still I don't like the two montages with duplicate channel names...

You mean montages with duplicate positions and different channel names, right? What is your proposed alternative? Let's say we have two people, one whose channel names match those of `standard_postfixed.elc` and another whose match those of `standard_primed.elc`. Currently each just does `montage = read_montage(...)` each using a different name, and everything else works.

How do you propose this should change from the user-facing API?
> Yes I think it's worth a deprecation cycle to go to `transform=True`, we assume EEG coords are Neuromag head coords in various places.

OK, but maybe it's `False` by default because it fails if the three fiducials are not in the montage? Why don't we ship all montages in head coordinates? Then `transform` could remain `False` for custom montage files.

> You mean montages with duplicate positions and different channel names, right? What is your proposed alternative? Let's say we have two people, one whose channel names match those of `standard_postfixed.elc` and another whose match those of `standard_primed.elc.` Currently each just does `montage = read_montage(...)` each using a different name, and everything else works.

This is not quite what I meant. Currently we have overlapping channel names within one montage. I would just remove all duplicate names - these are all non-standard names anyway, and it is not that hard to rename channels in a raw file so that they match names in a given montage.
OK, but maybe it's False by default because it fails if the three fiducials
are not in the montage? Why don't we ship all montages in head coordinates?
Then transform could remain False for custom montage files.

This would make it harder to copy paste files between packages and doesn't
buy us much over using transform.

One thing to note is that, for sensor positions defined on a sphere (with
fiducials that are or should also be on that sphere), knowing what the head
coordinate frame is ends up being equivalent to knowing the locations of
the three fiducials. So in order to get to your proposal (ship all in head
frame) we essentially need to know the fiducials anyway. Thus as soon as we
make it possible to get to head coordinates, we obviate the need to
actually do the transformation to the points saved on disc.

And for the files that don't have the fiducials, the diff to other packages
or standard montage files is still smaller by just adding the fiducials. So
it still makes maintenance and interoperability easier.

This is not quite what I meant. Currently we have overlapping channel names
within one montage.

I somehow missed this and it's not easy to check ATM (on mobile), do each
of the two listed in the top comment have this problem?

it is not that hard to rename channels in a raw file so that they match
names in a given montage.

I think we'd rather let people rename the montage names and apply than
change channel names then apply. You shouldn't need to change your Raw
instance channel names (which could be meaningful or standard for your
system) to get positions added. But this is perhaps a different issue.

You're right, but maybe setting `transform=True` is not such a good idea if fiducials are required.

> I somehow missed this and it's not easy to check ATM (on mobile), do each
of the two listed in the top comment have this problem?

Yes, each of the two montages have this problem.
> maybe setting transform=True is not such a good idea if fiducials are required.

My proposal is to add fiducials to any montage that doesn't currently have them, and do `transform=True` as default via deprecation. In the long run this will facilitate a lot of things, such as template-based EEG localization.

Regarding the duplicate channel names, I guess we could split them up or deduplicate somehow. But I wonder if it's actually doing any damage or creating problems beyond not being the most elegant solution. If it's not causing problems, my vote would be just to document the redundancies.
> My proposal is to add fiducials to any montage that doesn't currently have them, and do `transform=True` as default via deprecation. In the long run this will facilitate a lot of things, such as template-based EEG localization.

+1 if there is an informative error message in case the transform fails (such as "Required fiducials LPA, RPA, and Nz not found in montage. You can still load the montage by setting transform=False.")

> Regarding the duplicate channel names, I guess we could split them up or deduplicate somehow. But I wonder if it's actually doing any damage or creating problems beyond not being the most elegant solution. If it's not causing problems, my vote would be just to document the redundancies.

It's just ugly/unprofessional if you visualize them. If you happen to use these (non-standard) location names in your recordings, you should be prepared to use a custom montage. Or let's split it up into `standard_postfixed_a` and `standard_postfixed_b`.
> @teonbrooks any idea why we default to transform=False in our montage-reading code? Everything else in MNE is in Neuromag head coords, seems like we'd want to (at least by default) make them in head coords, too.
I agree, I think not all of the montages had fiducials so we left it there by default. also I think we just left it because we assumed the neuromag transformation was only needed when going to source space and that the user might want the electrode displayed in their native space. 

@larsoner, separate issue, but I think we have finally gotten to the point where we need to scope out a plan for restructuring the montage/digmontage class. also, make a straightforward procedure to go from a standard template to an average source brain

> +1 if there is an informative error message in case the transform fails

If we do the point-adding I'm advocating, then the transform really shouldn't ever fail, at least for our builtin montages (the way `read_montage` is really meant for public consumption as far as I can tell). But we can add it just to be safe. I think the check is there, we'll just need to tweak the message.

> It's just ugly/unprofessional if you visualize them.

Ahh, I hadn't thought about `montage.plot`. This would happen there. However, renaming is a rough API break.

Regardless of what we do with the montages, we should probably do something intelligent with numerically-identical location plotting (e.g., separate by newlines). Even for MEG sensor plotting like Neuromag, plotting their actual physical locations you'd want to do something like this. And maybe it would be good enough to make the `montage.plot` look okay with these FieldTrip montage files.

> @larsoner, separate issue, but I think we have finally gotten to the point where we need to scope out a plan for restructuring the montage/digmontage class.

There has been [a lot of discussion on this](https://github.com/mne-tools/mne-python/issues/3893#issuecomment-271925975) and I think people are mostly happy with the API, [you included based on a quick look](https://github.com/mne-tools/mne-python/issues/3893#issuecomment-272071754). Feel free to look it over again though.

> also, make a straightforward procedure to go from a standard template to an average source brain

Yep this is part of the motivation for #3893. That plus `mne.coreg` should get you there, and we can hopefully build a tutorial out of it.
> If we do the point-adding I'm advocating, then the transform really shouldn't ever fail, at least for our builtin montages (the way `read_montage` is really meant for public consumption as far as I can tell). But we can add it just to be safe. I think the check is there, we'll just need to tweak the message.

I think `read_montage` is also meant to read custom montage files, and without a clear error message it will be difficult to figure out what's going on when reading a montage without the required fiducials.

> Regardless of what we do with the montages, we should probably do something intelligent with numerically-identical location plotting (e.g., separate by newlines).

You mean `montage.plot` should check for identical locations and only plot one? Which label should it use then? The first? I guess this is feasible, and it could also throw a warning if it tries to visualize a montage with duplicate locations.
>  Which label should it use then? The first?

I was thinking of showing all names, separated by newlines. We'd have to do some testing to figure out the best option.
Ah OK, might get a bit crowded then. What if there are more than 2 names, like 3 or 4? I guess we could also just leave it as it is. The plot then indicates that there's something wrong with the montage (which I still think is the case, at least for EEG there shouldn't be montages with duplicate names).
I guess setting `transform=True` by default can be done in a new PR.

Besides that, what do we do with the two montages that have multiple labels for identical locations? People will probably start plotting montages because there now is a method for that and notice that something is wrong. I'd either remove these two montages or de-duplicate the labels.
I am against those for the same reasons as before (maintainability and
backward compatibility), and still in favor of the less invasive change of
making the plotting function smarter to handle overlaps.

Then I propose to issue a warning if plotting such a montage and only plotting the first label. Handling more than one label per position will be difficult since there is already overlap with only one label per location.
That sounds reasonable to me.
Checking for duplicate labels is giving me a headache. First, I need to use `scipy.spatial.distance.pdist` and `scipy.spatial.distance.squareform` - MNE does not require SciPy, so how should this be handled? Second, figuring out if there are duplicate labels is easy, but finding their indices and plotting only one label is pretty hard. I'm really not sure if these two defect montages are really worth this effort. In the end, they are defective and I still think we should either remove them completely or remove the duplicate labels from the montages.
> First, I need to use scipy.spatial.distance.pdist and scipy.spatial.distance.squareform

We have a standard fast pairwise distance wrapper `_compute_nearest(..., return_dists=True)`.

> MNE does not require SciPy, so how should this be handled?

SciPy is a hard dependency of MNE

> Second, figuring out if there are duplicate labels is easy, but finding their indices and plotting only one label is pretty hard.

Does this work?
```
use_pos = []
use_labels = []
usable = np.ones(len(pos), bool)
dists = _compute_nearest(pos, pos, return_dists=True)[1]
for ii in range(len(pos)):
    if usable[ii]:
        use_pos.append(pos[ii])
        use_labels.append(labels[ii])
    usable[dists[ii] < 1e-5] = False
logger.info('Plotting %d/%d deduplicated labels' % (len(use_pos), len(pos)))
assert not usable.any()
```
Thanks @larsoner. If SciPy is a hard dependency then I just use it. MNE's `_compute_nearest` returns a vector of all zeros, whereas `scipy.spatial.distance.cdist` works. I think I just found a solution without loops.
> _compute_nearest returns a vector of all zeros

Oh right because it gives the neighbor as zero. `cdist` should be fine
One non-(explicit-)loop would be to turn the square distance matrix into a connectivity matrix and do:
```
_, components = sparse.csgraph.connected_components(cdist(pos, pos) < 1e-5)
```
And then unpack the first element of each item in `components`. Is that what you're doing?

Here's what I'm doing right now:

```
dists = cdist(montage.pos, montage.pos)
dists[np.tril_indices(dists.shape[0])] = np.nan
dupes = np.argwhere(np.isclose(dists, 0))
```
This gives me all duplicates in the rows, column 0 is the index of the first label and column 1 the index of the second (duplicate) label. Now I'm working on integrating all of this into the plot function (which is not that straightforward because `Montage` objects don't support removal of channels out of the box, but anyway I'll do it manually).

Resolving patch:
diff --git a/mne/viz/montage.py b/mne/viz/montage.py
--- a/mne/viz/montage.py
+++ b/mne/viz/montage.py
@@ -1,5 +1,7 @@
 """Functions to plot EEG sensor montages or digitizer montages."""
-from ..utils import check_version
+from copy import deepcopy
+import numpy as np
+from ..utils import check_version, logger
 from . import plot_sensors
 
 
@@ -25,8 +27,10 @@ def plot_montage(montage, scale_factor=20, show_names=True, kind='topomap',
     fig : Instance of matplotlib.figure.Figure
         The figure object.
     """
+    from scipy.spatial.distance import cdist
     from ..channels import Montage, DigMontage
     from .. import create_info
+
     if isinstance(montage, Montage):
         ch_names = montage.ch_names
         title = montage.kind
@@ -39,6 +43,26 @@ def plot_montage(montage, scale_factor=20, show_names=True, kind='topomap',
                         "mne.channels.montage.DigMontage")
     if kind not in ['topomap', '3d']:
         raise ValueError("kind must be 'topomap' or '3d'")
+
+    if isinstance(montage, Montage):  # check for duplicate labels
+        dists = cdist(montage.pos, montage.pos)
+        # only consider upper triangular part by setting the rest to np.nan
+        dists[np.tril_indices(dists.shape[0])] = np.nan
+        dupes = np.argwhere(np.isclose(dists, 0))
+        if dupes.any():
+            montage = deepcopy(montage)
+            n_chans = montage.pos.shape[0]
+            n_dupes = dupes.shape[0]
+            idx = np.setdiff1d(montage.selection, dupes[:, 1]).tolist()
+            logger.info("{} duplicate electrode labels found:".format(n_dupes))
+            logger.info(", ".join([ch_names[d[0]] + "/" + ch_names[d[1]]
+                                   for d in dupes]))
+            logger.info("Plotting {} unique labels.".format(n_chans - n_dupes))
+            montage.ch_names = [montage.ch_names[i] for i in idx]
+            ch_names = montage.ch_names
+            montage.pos = montage.pos[idx, :]
+            montage.selection = np.arange(n_chans - n_dupes)
+
     info = create_info(ch_names, sfreq=256, ch_types="eeg", montage=montage)
     fig = plot_sensors(info, kind=kind, show_names=show_names, show=show,
                        title=title)


Test patch:
diff --git a/mne/viz/tests/test_montage.py b/mne/viz/tests/test_montage.py
--- a/mne/viz/tests/test_montage.py
+++ b/mne/viz/tests/test_montage.py
@@ -22,7 +22,8 @@
 
 
 def test_plot_montage():
-    """Test plotting montages."""
+    """Test plotting montages.
+    """
     import matplotlib.pyplot as plt
     m = read_montage('easycap-M1')
     m.plot()
@@ -42,3 +43,18 @@ def test_plot_montage():
     plt.close('all')
     d.plot(kind='3d', show_names=True)
     plt.close('all')
+
+
+def test_plot_defect_montage():
+    """Test plotting defect montages (i.e. with duplicate labels).
+    """
+    # montage name and number of unique labels
+    montages = [('standard_1005', 342), ('standard_postfixed', 85),
+                ('standard_primed', 85), ('standard_1020', 93)]
+    for name, n in montages:
+        m = read_montage(name)
+        fig = m.plot()
+        collection = fig.axes[0].collections[0]
+        assert collection._edgecolors.shape[0] == n
+        assert collection._facecolors.shape[0] == n
+        assert collection._offsets.shape[0] == n