Repository: mne-tools/mne-python

Issue:
BUG: GAT.plot_diagonal with predict_proba and AUC scoring fails b/c calculating chance levels doesn't work
```
import mne
from mne.datasets import spm_face
from mne.decoding import GeneralizationAcrossTime

# Preprocess data
data_path = spm_face.data_path()
# Load and filter data, set up epochs
raw_fname = data_path + '/MEG/spm/SPM_CTF_MEG_example_faces%d_3D_raw.fif'

raw = mne.io.Raw(raw_fname % 1, preload=True)  # Take first run

picks = mne.pick_types(raw.info, meg=True, exclude='bads')
raw.filter(1, 45, method='iir')

events = mne.find_events(raw, stim_channel='UPPT001')
event_id = {"faces": 1, "scrambled": 2}
tmin, tmax = -0.1, 0.5

decim = 4
epochs = mne.Epochs(raw, events, event_id, tmin, tmax, proj=True,
                    picks=picks, baseline=None, preload=True,
                    reject=dict(mag=1.5e-12), decim=decim, verbose=False)

def auc_scorer(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    from sklearn.preprocessing import LabelBinarizer
    le = LabelBinarizer()
    y_true = le.fit_transform(y_true)
    return roc_auc_score(y_true, y_pred[:, 1])

# Define decoder. The decision function is employed to use cross-validation
gat = GeneralizationAcrossTime(predict_mode='cross-validation', n_jobs=4,
                               predict_method='predict_proba',
                               scorer=auc_scorer)

# fit and score
gat.fit(epochs)
gat.score(epochs)
gat.plot_diagonal()  # plot decoding across time (correspond to GAT diagonal)
```

gives
`AttributeError: 'function' object has no attribute 'func_name'`

but 
`gat.plot_diagonal(chance=.5)`
works.

I think that counts as a bug? Or should the user-defined scorer be different?

We've discussed this in the context of #2755 , but I think there should be an issue.

Also, maybe we should consider adding a scorer that can inherently deal with `predict_proba`, ie this one (by Jean-Remi).

@kingjr 



Discussion:
It's a user-specific scorer, so I think it's ok to ask the user to specify chance level. But perhaps a warning would suffice.

Also, I think the scorer's chance level should be set in the sklearn API. I'm not aware of this ATM.

> It's a user-specific scorer, so I think it's ok to ask the user to specify chance level. But perhaps a warning would suffice.

There could be a clearer error message that the problem is in determining the chance level. As-is, the error is rather opaque.

agreed.

Let's wait until the current beast is merged, so we can do small PR again.

How about, when #2856 is done, I tackle this one? At the very least by providing an informative error when chance is not a float.

Sure! Let's just throw a warning though, and avoid plotting the dashed line.

I'm also wondering whether we shouldn't ask sklearn people whether a `get_chance(scorer, clf)` function wouldn't be worth implementing. Although I can see how this may be tough e.g. 

```
y_train = [0, 1, 2, 2]
# prior-informed chance level
print np.max([np.mean(y_train == c) for c in np.unique(y_train)])

# mean accuracy of prior-uninformed chance level
print np.mean([np.mean(np.random.randint(3, size=4)==y_train) for i in range(1000)])
```

@jona-sassenhagen are you working on this? If not, I would like to try this one.

I'd start there once #2975 is merged, so if you want to start now, it's yours!

@jona-sassenhagen I think you should work on this. I'll work on some other issue! :)


Resolving patch:
diff --git a/mne/viz/decoding.py b/mne/viz/decoding.py
--- a/mne/viz/decoding.py
+++ b/mne/viz/decoding.py
@@ -151,8 +151,10 @@ def plot_gat_times(gat, train_time='diagonal', title=None, xmin=None,
     if chance is not False:
         if chance is True:
             chance = _get_chance_level(gat.scorer_, gat.y_train_)
-        ax.axhline(float(chance), color='k', linestyle='--',
-                   label="Chance level")
+        chance = float(chance)
+        if np.isfinite(chance):  # don't plot nan chance level
+            ax.axhline(chance, color='k', linestyle='--',
+                       label="Chance level")
     ax.axvline(0, color='k', label='')
 
     if isinstance(train_time, (str, float)):
@@ -232,5 +234,5 @@ def _get_chance_level(scorer, y_train):
     else:
         chance = np.nan
         warn('Cannot find chance level from %s, specify chance level'
-             % scorer.func_name)
+             % scorer.__name__)
     return chance


Test patch: