Repository: nilearn/nilearn

Issue:
[BUG] Decoding docs improvements 
Following #2415, two points that were not tackled inside #2476 in the interest of short cycles

- [x]  Add a section in 2.1 on how to determine chance with a Decoder API (waiting for #2147) EDIT : Adressed #2622
- [ ]  Rethink 2.2 section on OVO vs OVA, not sure it is useful as Decoder doesn't provide this flexibility easily. But show how to plot a confusion matrix using Decoder on a simple example.


Discussion:
- Regarding chance determination with Decoder: this is performed in `plot_haxby_full_analysis.py` (drvtion 9.3.11) we just need to refer to that, right ?
- I would propose to leave the OVO vs OVA thing aside for the moment.
- Confusion matrix is presented in 2.2. This is enough imho. Agreed  ?
First part adressed by #2622 

> I would propose to leave the OVO vs OVA thing aside for the moment

Agreed
> `Confusion` matrix is presented in 2.2. This is enough imho. Agreed ?

I think it would be helpful to show in an example how to make a confusion matrix from decoder object since it does not seem straight forward IMHO

If you don't agree we can close
+1 for confusion matrix for decoder object example !
Hi, I'd be interested in adding an example for plotting a confusion matrix from a decoder object, if you still think this would be a good idea.

Should this be a new example or get folded into an existing one where there is a train/test split? For example https://nilearn.github.io/stable/auto_examples/02_decoding/plot_haxby_frem.html.
Thanks @michellewang ! yes I think that would still be a good idea. If possible it would be better to include that in an existing example

Resolving patch:
diff --git a/examples/02_decoding/plot_haxby_frem.py b/examples/02_decoding/plot_haxby_frem.py
--- a/examples/02_decoding/plot_haxby_frem.py
+++ b/examples/02_decoding/plot_haxby_frem.py
@@ -1,9 +1,9 @@
 """
-Decoding with FREM: face vs house object recognition
-=========================================================
+Decoding with FREM: face vs house vs chair object recognition
+===============================================================
 
 This example uses fast ensembling of regularized models (FREM) to decode
-a face vs house discrimination task from Haxby 2001 study. FREM uses an
+a face vs house vs chair discrimination task from Haxby 2001 study. FREM uses an
 implicit spatial regularization through fast clustering and aggregates a
 high number of estimators trained on various splits of the training set,
 thus returning a very robust decoder at a lower computational cost than
@@ -24,9 +24,9 @@
 
 behavioral = pd.read_csv(data_files.session_target[0], sep=" ")
 
-# Restrict to face and house conditions
+# Restrict to face, house, and chair conditions
 conditions = behavioral["labels"]
-condition_mask = conditions.isin(["face", "house"])
+condition_mask = conditions.isin(["face", "house", "chair"])
 
 # Split data into train and test samples, using the chunks
 condition_mask_train = (condition_mask) & (behavioral["chunks"] <= 6)
@@ -61,16 +61,50 @@
 accuracy = (y_pred == y_test).mean() * 100.0
 print(f"FREM classification accuracy : {accuracy:g}%")
 
+#############################################################################
+# Plot confusion matrix
+# ------------------------------------
+
+import numpy as np
+from sklearn.metrics import confusion_matrix
+from nilearn import plotting
+
+# Calculate the confusion matrix
+matrix = confusion_matrix(
+    y_test, 
+    y_pred, 
+    normalize='true',
+)
+
+# Plot the confusion matrix
+im = plotting.plot_matrix(
+    matrix,
+    labels=sorted(np.unique(y_test)),
+    vmin=0,
+    cmap='hot_r',
+)
+
+# Add x/y-axis labels
+ax = im.axes
+ax.set_ylabel('True label')
+ax.set_xlabel('Predicted label')
+
+# Adjust figure to make labels fit
+ax.get_figure().tight_layout()
+
+plotting.show()
+
 #############################################################################
 # Visualization of FREM weights
 # ------------------------------------
 from nilearn import plotting
 
+
 plotting.plot_stat_map(
     decoder.coef_img_["face"],
     background_img,
     title=f"FREM: accuracy {accuracy:g}%, 'face coefs'",
-    cut_coords=(-52, -5),
+    cut_coords=(-50, -4),
     display_mode="yz",
 )
 plotting.show()


Test patch: