F1_score is good. Thé standard accuracy score -by default- doesn't work if se do simple per-category scoring?eickenberg <notifications@github.com> a écrit :In plot_haxby_full_analysis.py:

> +    all_timecourses = brain_masker.fit_transform(data_files.func[subject_id])
> +    from sklearn.feature_selection import f_classif, SelectKBest
> +    from sklearn.pipeline import Pipeline
> +    feature_selection = SelectKBest(f_classif, 500)
> +    pipeline = Pipeline([("Feature selection", feature_selection),
> +                         ("Classifier", classifier)])
> +
> +    scores_anova_svm = cross_val_score(pipeline,
> +                                       all_timecourses[resting_state == False], 
> +                                       labels['labels'][resting_state == False],
> +                                       cv=12, n_jobs=1,
> +                                       verbose=True)
> +    # With resting removed state this scores at .91 (with RS: .87)
> +    print "ANOVA + Linear SVM C=1 on full brain"
> +    print "Score: %1.2f +- %1.2f" % (scores_anova_svm.mean(),
> +                                     scores_anova_svm.std())
This may be related to different scoring standards in possibly different versions of sklearn. 

Not sure though.

In any case, I will have to deal with a strong class imbalance problem while doing OvR for each label. Chance level for 0-1 scoring is at 87.5%
I am now going for f1_score. Any reason why I shouldn't?
If I go for AUC, I will have to dig into the classifier and move an offset around, which may not be ideal for a simple example

—
Reply to this email directly or view it on GitHub.