Repository: nilearn/nilearn

Issue:
[BUG] second-level contrasts in string format in non_parametric_permutation()
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Operating system

- [X] Linux
- [ ] Mac
- [ ] Windows

### Operating system version

Linux Ubuntu 22.04



### Python version

- [ ] 3.11
- [X] 3.10
- [ ] 3.9
- [ ] 3.8
- [ ] 3.7

### nilearn version

0.10.0

### Expected behavior

It is possible to define second_level_contrast as string using column names and +/- signs for computing second level contrast, but not when running non_parametric_permutation(). It seems that the _get_contrast() function that the permutation function is using, does not check the contrast input in a appropriate manner (it checks if it matches a column in design matrix, but it does not as it consists of two columns subtracted from each other - e.g., second_level_contrast='female-male'). 
As this contrast format works for computing contrasts, it should also work in permutation, so that we can run non parametric corrections of the p-values. 

### Current behavior & error messages

This is what I got:



```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 17, in permutation_test_and_figure
  File "/users/line/miniconda3/envs/env_bidscoin3/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py", line 921, in non_parametric_inference
    contrast = _get_contrast(second_level_contrast, design_matrix)
  File "/users/line/miniconda3/envs/env_bidscoin3/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py", line 239, in _get_contrast
    raise ValueError(
ValueError: "female-male" is not a valid contrast name
```


### Steps and code to reproduce bug

```python
from nilearn.datasets import fetch_localizer_contrasts

n_subjects = 16
data = fetch_localizer_contrasts(
    ["left vs right button press"],
    n_subjects,
    get_tmaps=True,
    legacy_format=False,
)

import pandas as pd
import random 

k = random.randint(0, 1) # decide on k once
male = []
for i in range(0,len(second_level_input)):
    male.append(random.randint(0,1))
female = male 
female[male==1] = 0
female[male==0] = 1

second_level_input = data["cmaps"]
design_matrix = pd.DataFrame(
    [1] * len(second_level_input),
    male, 
    female, 
    columns=["intercept", "male", "female"],
)

from nilearn.glm.second_level import SecondLevelModel

second_level_model = SecondLevelModel(smoothing_fwhm=8.0)
second_level_model = second_level_model.fit(
    second_level_input,
    design_matrix=design_matrix,
)

z_map = second_level_model.compute_contrast(
    second_level_contrast="female-male",
    output_type="z_score",
)

import numpy as np 
from nilearn.image import get_data, math_img 
from nilearn.glm.second_level import non_parametric_inference

p_val = second_level_model.compute_contrast(output_type='p_value', second_level_contrast=sl_contrast)
#Get the number of voxels in the image 
n_voxels = np.sum(get_data(slm.masker_.mask_img_))
#Making an image with significant values, correcting the p-values for multiple testing and taking negative log 
neg_log_pval = math_img(
        '-np.log10(np.minimum(1, img * {}))'.format(str(n_voxels)),
        img=p_val)

n_perm = 1000 
#Calculate p-values using permutations 
out_dict = non_parametric_inference(
        second_level_input,
        design_matrix=design_matrix,
        second_level_contrast="female-male",
        model_intercept=True,
        n_perm=n_perm,  
        two_sided_test=False,
        smoothing_fwhm=8.0,
        n_jobs=1  
        )

```

[BUG]  Trouble running arithmetic in second_level_contrast of non_parametric_inference
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Operating system

- [X] Linux
- [ ] Mac
- [ ] Windows

### Operating system version

Rocky Linux release 8.8 (Green Obsidian)


### Python version

- [ ] 3.12
- [ ] 3.11
- [X] 3.10
- [ ] 3.9
- [ ] 3.8

### nilearn version

0.10.2

### Expected behavior

I would like to a run a paired t-test using `nilearn.glm.second_level.non_parametric_inference`. The documentation states that 

> Basically one can use the name of the conditions as they appear in the design matrix of the fitted model combined with operators +- and combined with numbers with operators +-*/. "

design_matrix_df
    H1   H2  Subject 1  Subject 2
0  1.0  0.0        1.0        0.0
1  0.0  1.0        1.0        0.0
2  1.0  0.0        0.0        1.0
3  0.0  1.0        0.0        1.0

```
out_dict = non_parametric_inference(
    searchlight_results,
    design_matrix=design_matrix_df,
    second_level_contrast="H1",
    model_intercept=True,
    n_perm=500,  # 500 for the sake of time. Ideally, this should be 10,000.
    two_sided_test=False,
    smoothing_fwhm=None,
    n_jobs=-1,
    verbose=3,
    threshold=0.01,
)

out_dict = non_parametric_inference(
    searchlight_results,
    design_matrix=design_matrix_df,
    second_level_contrast="H2",
    model_intercept=True,
    n_perm=500,  # 500 for the sake of time. Ideally, this should be 10,000.
    two_sided_test=False,
    smoothing_fwhm=None,
    n_jobs=-1,
    verbose=3,
    threshold=0.01,
)

out_dict = non_parametric_inference(
    searchlight_results,
    design_matrix=design_matrix_df,
    second_level_contrast="H1-H2",
    model_intercept=True,
    n_perm=500,  # 500 for the sake of time. Ideally, this should be 10,000.
    two_sided_test=False,
    smoothing_fwhm=None,
    n_jobs=-1,
    verbose=3,
    threshold=0.01,
)

```

### Current behavior & error messages

I was able to run second_level_contrast="H1", second_level_contrast="H2", but not second_level_contrast="H1-H2"
```python
{
	"name": "ValueError",
	"message": "\"H1-H2\" is not a valid contrast name",
	"stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[80], line 1
----> 1 out_dict = non_parametric_inference(
      2     searchlight_results,
      3     design_matrix=design_matrix_df,
      4     second_level_contrast=\"H1-H2\",
      5     model_intercept=True,
      6     n_perm=500,  # 500 for the sake of time. Ideally, this should be 10,000.
      7     two_sided_test=False,
      8     smoothing_fwhm=None,
      9     n_jobs=-1,
     10     verbose=3,
     11     threshold=0.01,
     12 )

File ~/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py:969, in non_parametric_inference(second_level_input, confounds, design_matrix, second_level_contrast, first_level_contrast, mask, smoothing_fwhm, model_intercept, n_perm, two_sided_test, random_state, n_jobs, verbose, threshold, tfce)
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=962'>963</a>     sys.stderr.write(
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=963'>964</a>         \"\
Computation of second level model done in \"
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=964'>965</a>         f\"{time.time() - t0} seconds\
\"
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=965'>966</a>     )
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=967'>968</a> # Check and obtain the contrast
--> <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=968'>969</a> contrast = _get_contrast(second_level_contrast, design_matrix)
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=969'>970</a> # Get first-level effect_maps
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=970'>971</a> effect_maps = _infer_effect_maps(second_level_input, first_level_contrast)

File ~/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py:287, in _get_contrast(second_level_contrast, design_matrix)
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=284'>285</a>         contrast = second_level_contrast
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=285'>286</a>     else:
--> <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=286'>287</a>         raise ValueError(
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=287'>288</a>             f'\"{second_level_contrast}\" is not a valid contrast name'
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=288'>289</a>         )
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=289'>290</a> else:
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=290'>291</a>     # Check contrast definition
    <a href='file:///home/saptaf1/anaconda3/envs/spy310/lib/python3.10/site-packages/nilearn/glm/second_level/second_level.py?line=291'>292</a>     if second_level_contrast is None:

ValueError: \"H1-H2\" is not a valid contrast name"
}
```


### Steps and code to reproduce bug

```python

import numpy as np
import nibabel as nib

def create_random_nii(filename, shape=(100, 100, 100), affine=np.eye(4)):
    data = np.random.rand(*shape)
    img = nib.Nifti1Image(data, affine)
    nib.save(img, filename)
    return filename

# Create four random .nii.gz files and add them to a list
nii_files = [create_random_nii(f'dataset{i}.nii.gz') for i in range(1, 5)]

print(nii_files)

# Number of subjects and conditions
num_subjects = len(searchlight_results_h1)
num_conditions = 2

def generate_file_names(num_conditions, num_participants):
    files = []
    for i in range(1, num_participants + 1):
        for j in range(1, num_conditions + 1):
            files.append(f'sub-{i}_cond-{j}.nii.gz')
    return files

files = generate_file_names(num_conditions, num_subjects)

# Initialize the design matrix
design_matrix = np.zeros((len(files), num_conditions + num_subjects))

# Iterate over the files
for i, file in enumerate(files):
    # Split the file name to get the subject and condition
    parts = file.split('_')
    subject = int(parts[0].split('-')[1])
    condition = int(parts[1].split('-')[1].split('.')[0])
    # Assign a value of 1 for the corresponding condition
    design_matrix[i, condition-1] = 1
    # Assign a value of 1 for the corresponding subject
    design_matrix[i, num_conditions + subject - 1] = 1

print("Design matrix:")
print(design_matrix)

# turn design matrix into a dataframe
import pandas as pd

# Convert design matrix to dataframe
design_matrix_df = pd.DataFrame(design_matrix)
# name first column H1 and second column H2 and rest of columns subject numbers
design_matrix_df.columns = ['H1', 'H2'] + [f'Subject {i}' for i in range(1, num_subjects + 1)]

out_dict = non_parametric_inference(
    nii_files,
    design_matrix=design_matrix_df,
    second_level_contrast="H1-H2",
    model_intercept=True,
    n_perm=500,  # 500 for the sake of time. Ideally, this should be 10,000.
    two_sided_test=False,
    smoothing_fwhm=None,
    n_jobs=-1,
    verbose=3,
    threshold=0.01,
)


```



Discussion:
thanks for the bug report

had to tweak a couple of lines on the code you provided but I can reproduce the issue you described

```python
from nilearn.datasets import fetch_localizer_contrasts

n_subjects = 16
data = fetch_localizer_contrasts(
    ["left vs right button press"],
    n_subjects,
    get_tmaps=True,
    legacy_format=False,
)

import pandas as pd
import random 

second_level_input = data["cmaps"]

k = random.randint(0, 1) # decide on k once
male = []
for i in range(0,len(second_level_input)):
    male.append(random.randint(0,1))
female = male 
female[male==1] = 0
female[male==0] = 1

data = {"intercept": [1] * len(second_level_input), 
         "male": male,
         "female": female}

design_matrix = pd.DataFrame(data)

from nilearn.glm.second_level import SecondLevelModel

second_level_model = SecondLevelModel(smoothing_fwhm=8.0)
second_level_model = second_level_model.fit(
    second_level_input,
    design_matrix=design_matrix,
)

sl_contrast = "female-male"

z_map = second_level_model.compute_contrast(
    second_level_contrast=sl_contrast,
    output_type="z_score",
)

import numpy as np 
from nilearn.image import get_data, math_img 
from nilearn.glm.second_level import non_parametric_inference

p_val = second_level_model.compute_contrast(output_type='p_value', second_level_contrast=sl_contrast)
#Get the number of voxels in the image 
n_voxels = np.sum(get_data(second_level_model.masker_.mask_img_))
#Making an image with significant values, correcting the p-values for multiple testing and taking negative log 
neg_log_pval = math_img(
        '-np.log10(np.minimum(1, img * {}))'.format(str(n_voxels)),
        img=p_val)

n_perm = 1000 
#Calculate p-values using permutations 
out_dict = non_parametric_inference(
        second_level_input,
        design_matrix=design_matrix,
        second_level_contrast="female-male",
        model_intercept=True,
        n_perm=n_perm,  
        two_sided_test=False,
        smoothing_fwhm=8.0,
        n_jobs=1  
        )

```
@Remi-Gau any update on this? Happy to take over
Can't remember why I stopped digging on this. From the top of my (failing) head, it was either that there was some refactoring ongoing for that code I wanted to get done first (yeah right... 😏) or that it was less trivial than I thought...

If you have the bandwidth go for it


Resolving patch:
diff --git a/nilearn/glm/second_level/second_level.py b/nilearn/glm/second_level/second_level.py
--- a/nilearn/glm/second_level/second_level.py
+++ b/nilearn/glm/second_level/second_level.py
@@ -274,9 +274,13 @@ def _get_con_val(second_level_contrast, design_matrix):
         else:
             raise ValueError("No second-level contrast is specified.")
     if not isinstance(second_level_contrast, str):
-        con_val = second_level_contrast
-        if np.all(con_val == 0):
-            raise ValueError("Contrast is null")
+        con_val = np.array(second_level_contrast)
+        if np.all(con_val == 0) or len(con_val) == 0:
+            raise ValueError(
+                "Contrast is null. Second_level_contrast must be a valid "
+                "contrast vector, a list/array of 0s and 1s, a string, or a "
+                "string expression."
+            )
     else:
         design_columns = design_matrix.columns.tolist()
         con_val = expression_to_contrast_vector(
@@ -285,31 +289,6 @@ def _get_con_val(second_level_contrast, design_matrix):
     return con_val
 
 
-def _get_contrast(second_level_contrast, design_matrix):
-    """Check and return contrast when testing one contrast at the time."""
-    if isinstance(second_level_contrast, str):
-        if second_level_contrast in design_matrix.columns.tolist():
-            contrast = second_level_contrast
-        else:
-            raise ValueError(
-                f'"{second_level_contrast}" is not a valid contrast name'
-            )
-    else:
-        # Check contrast definition
-        if second_level_contrast is None:
-            if design_matrix.shape[1] == 1:
-                second_level_contrast = np.ones([1])
-            else:
-                raise ValueError("No second-level contrast is specified.")
-        elif (np.nonzero(second_level_contrast)[0]).size != 1:
-            raise ValueError(
-                "second_level_contrast must be a list of 0s and 1s."
-            )
-        con_val = np.asarray(second_level_contrast, dtype=bool)
-        contrast = np.asarray(design_matrix.columns.tolist())[con_val][0]
-    return contrast
-
-
 def _infer_effect_maps(second_level_input, contrast_def):
     """Deal with the different possibilities of second_level_input."""
     # Build the design matrix X and list of imgs Y for GLM fit
@@ -970,7 +949,7 @@ def non_parametric_inference(
         )
 
     # Check and obtain the contrast
-    contrast = _get_contrast(second_level_contrast, design_matrix)
+    contrast = _get_con_val(second_level_contrast, design_matrix)
     # Get first-level effect_maps
     effect_maps = _infer_effect_maps(second_level_input, first_level_contrast)
 
@@ -981,9 +960,11 @@ def non_parametric_inference(
     var_names = design_matrix.columns.tolist()
 
     # Obtain tested_var
-    tested_var = np.asarray(design_matrix[contrast])
+    column_mask = [bool(val) for val in contrast]
+    tested_var = np.dot(design_matrix, contrast)
+
     # Remove tested var from remaining var names
-    var_names.remove(contrast)
+    var_names = [var for var, mask in zip(var_names, column_mask) if not mask]
 
     # Obtain confounding vars
     if len(var_names) == 0:


Test patch:
diff --git a/nilearn/glm/tests/test_second_level.py b/nilearn/glm/tests/test_second_level.py
--- a/nilearn/glm/tests/test_second_level.py
+++ b/nilearn/glm/tests/test_second_level.py
@@ -29,7 +29,6 @@
     _check_input_as_first_level_model,
     _check_output_type,
     _check_second_level_input,
-    _get_contrast,
     _infer_effect_maps,
     _process_second_level_input_as_dataframe,
     _process_second_level_input_as_firstlevelmodels,
@@ -420,32 +419,6 @@ def test_check_effect_maps():
         _check_effect_maps([1, 2], np.array([[1, 2], [3, 4], [5, 6]]))
 
 
-def test_get_contrast():
-    design_matrix = pd.DataFrame([1, 2, 3], columns=["conf"])
-    assert _get_contrast("conf", design_matrix) == "conf"
-
-    design_matrix = pd.DataFrame({"conf1": [1, 2, 3], "conf2": [4, 5, 6]})
-    assert _get_contrast([0, 1], design_matrix) == "conf2"
-    assert _get_contrast([1, 0], design_matrix) == "conf1"
-
-
-def test_get_contrast_errors():
-    design_matrix = pd.DataFrame([1, 2, 3], columns=["conf"])
-    with pytest.raises(ValueError, match='"foo" is not a valid contrast name'):
-        _get_contrast("foo", design_matrix)
-
-    design_matrix = pd.DataFrame({"conf1": [1, 2, 3], "conf2": [4, 5, 6]})
-    with pytest.raises(
-        ValueError, match="No second-level contrast is specified."
-    ):
-        _get_contrast(None, design_matrix)
-    with pytest.raises(
-        ValueError,
-        match="second_level_contrast must be a list of 0s and 1s",
-    ):
-        _get_contrast([0, 0], design_matrix)
-
-
 def test_infer_effect_maps(tmp_path):
     shapes, rk = (SHAPE, (7, 8, 7, 16)), 3
     mask, fmri_data, design_matrices = write_fake_fmri_data_and_design(
@@ -1217,7 +1190,9 @@ def test_non_parametric_inference_contrast_computation(tmp_path):
     )
 
 
-@pytest.mark.parametrize("second_level_contrast", [[1, 0], "r1"])
+@pytest.mark.parametrize(
+    "second_level_contrast", [[1, 0], "r1", "r1-r2", [-1, 1]]
+)
 def test_non_parametric_inference_contrast_formula(
     tmp_path, second_level_contrast, rng
 ):
@@ -1253,7 +1228,7 @@ def test_non_parametric_inference_contrast_computation_errors(tmp_path, rng):
     # passing null contrast should give back a value error
     with pytest.raises(
         ValueError,
-        match=("second_level_contrast must be a list of 0s and 1s."),
+        match=("Second_level_contrast must be a valid"),
     ):
         non_parametric_inference(
             second_level_input=Y,
@@ -1263,7 +1238,7 @@ def test_non_parametric_inference_contrast_computation_errors(tmp_path, rng):
         )
     with pytest.raises(
         ValueError,
-        match=("second_level_contrast must be a list of 0s and 1s."),
+        match=("Second_level_contrast must be a valid"),
     ):
         non_parametric_inference(
             second_level_input=Y,