Repository: mne-tools/mne-python

Issue:
Failing tests because head surface not found
I get several failing tests because the head surface from the sample data can't be found. The function `get_head_surf` tries to load "sample-5120-bem.fif" which does not contain the head surface, so I get errors of the form "ValueError: surface with id 4 not found". This causes a total of 4 tests to fail. Can anyone reproduce this?



Discussion:
I have not experienced this problem, but I know Jenkins complains about something similar fairly frequently. @agramfort?

@mluessi That's what typically happens when you try to extract the head from a single layer bem. In the latter case the source parameter should point to 'head', not 'bem'. Wondering why the tests actually care about this. Maybe a problem with anatomy scripts or dataset packakging?

indeed a classic failure on jenkins when a fresh sample dataset is
downloaded. I always have to manually add sample-bem.fif. If you can fix
without adding the sample-bem.fif it would be great. This pb never occured
with py3k for some reason...

An obvious fix would be to make the change the default from `source='bem'` to `source='head'`. I think `sample-head.fif` is a much more typical file name for the head surface file than `sample-bem.fif`. If we keep `sample-bem.fif` we need to update mne-scripts to generate that file. The problem occurs for me because my nightly build first builds the sample dataset and then runs the tests.. so if it is missing from mne-scripts the failure will keep happening.

One problem with that approach is there is no guarantee of the sampling of the `-head.fif` surface. For example, our `-head.fif` is the dense head, with hundreds of thousands of vertices. This will make the interpolation calculations take ~100x longer for us and other users like us. Using the BEM better guarantees we interpolate to the low-res (~2500 vertex) surface.

I see. Another option would be to add `source='5120-5120-5120-bem'` to the tests, so it would get the right BEM file.

I suggest we look for sample-5120-5120-5120-bem.fif before sample-5120-bem.fif
for example by looking at the longest filename. I also think the
sample-bem.fif should be added to the sample data.

@agramfort in the function `get_head_surf`, we could just put `read_bem_surfaces` inside the loop use try/except to find the first BEM file that contains a head surface. WDYT?

> @agramfort in the function get_head_surf, we could just put read_bem_surfaces inside the loop use try/except to find the first BEM file that contains a head surface. WDYT?

that's maybe too much magic. +1 for source='5120-5120-5120-bem'
as well as adding the sample-bem.fif to the sample data.

ok?

I think adding source='5120-5120-5120-bem' makes sense. I'm -1 for adding a sample-bem.fif file to the sample data.. it's very uncommon to have this file, so we shouldn't start relying on it.

> I think adding source='5120-5120-5120-bem' makes sense. I'm -1 for adding
> a sample-bem.fif file to the sample data.. it's very uncommon to have this
> file, so we shouldn't start relying on it.
> 
> fine with me

Hmm.. unfortunately this fix  doesn't work because the `get_head_surf` is used internally, e.g., in `mne.make_field_map` without the option to specify the source of the head mesh.. so more magic is needed.. 

Arfff ok then ... Try to keep it simple


Resolving patch:
diff --git a/mne/surface.py b/mne/surface.py
--- a/mne/surface.py
+++ b/mne/surface.py
@@ -9,10 +9,11 @@
 from os import path as op
 import sys
 from struct import pack
+from glob import glob
+
 import numpy as np
 from scipy.spatial.distance import cdist
 from scipy import sparse
-from fnmatch import fnmatch
 
 from .io.constants import FIFF
 from .io.open import fiff_open
@@ -304,7 +305,9 @@ def get_head_surf(subject, source='bem', subjects_dir=None):
     source : str
         Type to load. Common choices would be `'bem'` or `'head'`. We first
         try loading `'$SUBJECTS_DIR/$SUBJECT/bem/$SUBJECT-$SOURCE.fif'`, and
-        then look for `'$SUBJECT*$SOURCE.fif'` in the same directory.
+        then look for `'$SUBJECT*$SOURCE.fif'` in the same directory by going
+        through all files matching the pattern. The head surface will be read
+        from the first file containing a head surface.
     subjects_dir : str, or None
         Path to the SUBJECTS_DIR. If None, the path is obtained by using
         the environment variable SUBJECTS_DIR.
@@ -319,23 +322,29 @@ def get_head_surf(subject, source='bem', subjects_dir=None):
     # use realpath to allow for linked surfaces (c.f. MNE manual 196-197)
     this_head = op.realpath(op.join(subjects_dir, subject, 'bem',
                                     '%s-%s.fif' % (subject, source)))
-    if not op.isfile(this_head):
+    if op.exists(this_head):
+        surf = read_bem_surfaces(this_head, True,
+                                 FIFF.FIFFV_BEM_SURF_ID_HEAD)
+    else:
         # let's do a more sophisticated search
         this_head = None
         path = op.join(subjects_dir, subject, 'bem')
         if not op.isdir(path):
             raise IOError('Subject bem directory "%s" does not exist'
                           % path)
-        files = os.listdir(path)
-        for fname in files:
-            if fnmatch(fname, '%s*%s.fif' % (subject, source)):
-                this_head = op.join(path, fname)
+        files = sorted(glob(op.join(path, '%s*%s.fif' % (subject, source))))
+        for this_head in files:
+            try:
+                surf = read_bem_surfaces(this_head, True,
+                                         FIFF.FIFFV_BEM_SURF_ID_HEAD)
                 break
+            except ValueError:
+                # the file does not contain a head surface
+                this_head = None
+
         if this_head is None:
-            raise IOError('No file matching "%s*%s" found'
-                          % (subject, source))
-    surf = read_bem_surfaces(this_head, True,
-                             FIFF.FIFFV_BEM_SURF_ID_HEAD)
+            raise IOError('No file matching "%s*%s" and containing a head surface '
+                          'found' % (subject, source))
     return surf
 
 


Test patch:
diff --git a/mne/tests/test_label.py b/mne/tests/test_label.py
--- a/mne/tests/test_label.py
+++ b/mne/tests/test_label.py
@@ -29,7 +29,7 @@
 v1_label_fname = op.join(subjects_dir, 'sample', 'label', 'lh.V1.label')
 
 fwd_fname = op.join(data_path, 'MEG', 'sample',
-                    'sample_audvis-eeg-oct-6p-fwd.fif')
+                    'sample_audvis-eeg-oct-6-fwd.fif')
 src_bad_fname = op.join(data_path, 'subjects', 'fsaverage', 'bem',
                         'fsaverage-ico-5-src.fif')