Repository: nilearn/nilearn

Issue:
design matrix columns name checking is excessive
<!--Provide a brief description of the bug.-->
Currently, a string can only a design matrix column if it fits the  `isidentifier()` condition, which looks excessive.
For instance, strings started with a number are not accepted, while there is no problem to create dataframes with such strings.
Thus I suggest to remove the lines below from the codebase

https://github.com/nilearn/nilearn/blob/2fd96b86cb53304f2f0b0c978d0bcd670f3795d2/nilearn/glm/first_level/hemodynamic_models.py#L443
```
# Check that all names look like proper pandas.DataFrame column names
    if not all([name.isidentifier() for name in names]):
        raise ValueError("At least one regressor name can't be used "
                         f"as a column identifier: {names}")
```
WDYT ?


Discussion:
Thanks for raising @bthirion but I think that is there in order to allow access to columns as attributes (using such syntax as `df.A`). This does not work with invalid identifiers (which you can only access using `[]` syntax). For more details see this section and relevant warning block in pandas docs: https://pandas.pydata.org/docs/user_guide/indexing.html#attribute-access. We could of course remove that and let the user deal with pandas error if they try to index an invalid identifier with the attribute syntax.
It's perfectly fine and quite common to have pandas column names that are not
valid identifiers and it's better to always index dataframes with `[]` (or
`loc`, `iloc`, `at`) anyway.

In any case, the function chooses the names so if they are not satisfying it
should make a different choice (as it already tries to do by removing spaces,
non-ascii and non-word characters) rather than fail and complain to the user
about its own error. BTW the same applies to making sure the names are all
different.
Thx ! I had the concrete case of a column name starting with a digit. What would be the replacement in that case ?
IMO the best is to do what you suggest in your first comment -- keep the original name, ie have a column name in the design matrix that is not a valid identifier, and access it with standard indexing eg `design_matrix["1): my name"]`.

but if we do want to go for transforming it in a valid identifier we could prefix it with some arbitrary string that starts with a letter, eg `reg_1_my_name`.
> IMO the best is to do what you suggest in your first comment -- keep the original name, ie have a column name in the design matrix that is not a valid identifier, and access it with standard indexing eg `design_matrix["1): my name"]`.

I'm ok with this. An error raised relating to the indexing I mentioned is a non nilearn issue anyways. 



Resolving patch:
diff --git a/nilearn/glm/contrasts.py b/nilearn/glm/contrasts.py
--- a/nilearn/glm/contrasts.py
+++ b/nilearn/glm/contrasts.py
@@ -40,7 +40,15 @@ def expression_to_contrast_vector(expression, design_columns):
         contrast_vector[list(design_columns).index(expression)] = 1.
         return contrast_vector
     df = pd.DataFrame(np.eye(len(design_columns)), columns=design_columns)
-    contrast_vector = df.eval(expression, engine="python").values
+    try:
+        contrast_vector = df.eval(expression, engine="python").values
+    except Exception:
+        raise ValueError(
+            'The expression (%s) is not valid. This could be due to '
+            'defining the contrasts using design matrix columns that are '
+            'invalid python identifiers. '
+            % expression
+        )
     return contrast_vector
 
 
diff --git a/nilearn/glm/first_level/first_level.py b/nilearn/glm/first_level/first_level.py
--- a/nilearn/glm/first_level/first_level.py
+++ b/nilearn/glm/first_level/first_level.py
@@ -657,11 +657,11 @@ def compute_contrast(self, contrast_def, stat_type=None,
             where ``n_col`` is the number of columns of the design matrix,
             (one array per run). If only one array is provided when there
             are several runs, it will be assumed that the same contrast is
-            desired for all runs. The string can be a formula compatible with
-            `pandas.DataFrame.eval`. Basically one can use the name of the
-            conditions as they appear in the design matrix of the fitted model
-            combined with operators +- and combined with numbers
-            with operators +-`*`/.
+            desired for all runs. One can use the name of the conditions as
+            they appear in the design matrix of the fitted model combined with
+            operators +- and combined with numbers with operators +-`*`/. In
+            this case, the string defining the contrasts must be a valid
+            expression for compatibility with :meth:`pandas.DataFrame.eval`.
 
         stat_type : {'t', 'F'}, optional
             Type of the contrast.
diff --git a/nilearn/glm/first_level/hemodynamic_models.py b/nilearn/glm/first_level/hemodynamic_models.py
--- a/nilearn/glm/first_level/hemodynamic_models.py
+++ b/nilearn/glm/first_level/hemodynamic_models.py
@@ -435,16 +435,6 @@ def _regressor_names(con_name, hrf_model, fir_delays=None):
     if len(np.unique(names)) != len(names):
         raise ValueError(f"Computed regressor names are not unique: {names}")
 
-    # Replace spaces with underscores
-    names = [re.sub(" +", "_", str(name)) for name in names]
-    # Remove any non-word character
-    names = [re.sub("[^a-zA-Z0-9_]+", "", str(name)) for name in names]
-
-    # Check that all names look like proper pandas.DataFrame column names
-    if not all([name.isidentifier() for name in names]):
-        raise ValueError("At least one regressor name can't be used "
-                         f"as a column identifier: {names}")
-
     return names
 
 


Test patch:
diff --git a/nilearn/glm/tests/test_contrasts.py b/nilearn/glm/tests/test_contrasts.py
--- a/nilearn/glm/tests/test_contrasts.py
+++ b/nilearn/glm/tests/test_contrasts.py
@@ -1,4 +1,5 @@
 import numpy as np
+import pytest
 
 from numpy.testing import assert_almost_equal
 from sklearn.datasets import make_regression
@@ -25,6 +26,13 @@ def test_expression_to_contrast_vector():
     cols = ["column_1"]
     contrast = expression_to_contrast_vector("column_1", cols)
     assert np.allclose(contrast, [1.])
+    cols = ['0', '1']
+    exp = '0-1'
+    with pytest.raises(
+            ValueError,
+            match='invalid python identifiers'
+    ):
+        expression_to_contrast_vector(exp, cols)
 
 
 def test_Tcontrast():
diff --git a/nilearn/glm/tests/test_dmtx.py b/nilearn/glm/tests/test_dmtx.py
--- a/nilearn/glm/tests/test_dmtx.py
+++ b/nilearn/glm/tests/test_dmtx.py
@@ -247,32 +247,6 @@ def test_design_matrix6():
     assert len(names) == 10
 
 
-def test_design_matrix7():
-    """
-    idem test_design_matrix1, but odd experimental paradigm;
-    code should raise an exception as condition names are not valid
-    pandas.DataFrame column names (and hence the computed design matrix
-    won't be able to call pd.eval when computing contrasts for instance).
-    """
-    tr = 1.0
-    frame_times = np.linspace(0, 127 * tr, 128)
-    conditions = [0, 0, 0, 1, 1, 1, 3, 3, 3]
-    durations = 1 * np.ones(9)
-    # no condition 'c2'
-    onsets = [30, 70, 100, 10, 30, 90, 30, 40, 60]
-    events = pd.DataFrame({'trial_type': conditions,
-                           'onset': onsets,
-                           'duration': durations})
-    hrf_model = 'glover'
-
-    with pytest.raises(
-        ValueError,
-        match="At least one regressor name can't be used"
-    ):
-        design_matrix_light(frame_times, events, hrf_model=hrf_model,
-                            drift_model='polynomial', drift_order=3)
-
-
 def test_design_matrix8():
     # basic test based on basic_paradigm and FIR
     tr = 1.0
diff --git a/nilearn/glm/tests/test_first_level.py b/nilearn/glm/tests/test_first_level.py
--- a/nilearn/glm/tests/test_first_level.py
+++ b/nilearn/glm/tests/test_first_level.py
@@ -881,8 +881,9 @@ def test_first_level_hrf_model(hrf_model, spaces):
     Ensure that FirstLevelModel runs without raising errors
     for different values of hrf_model. In particular, one checks that it runs
     without raising errors when given a custom response function.
-    Also ensure that it computes contrasts without raising errors,
-    even when event (ie condition) names have spaces.
+    When :meth:`~nilearn.glm.first_level.FirstLevelModel.compute_contrast`
+    is used errors should be raised when event (ie condition) names are not
+    valid identifiers.
     """
     shapes, rk = [(10, 10, 10, 25)], 3
     mask, fmri_data, _ =\
@@ -897,7 +898,15 @@ def test_first_level_hrf_model(hrf_model, spaces):
     model.fit(fmri_data, events)
 
     columns = model.design_matrices_[0].columns
-    model.compute_contrast(f"{columns[0]}-{columns[1]}")
+    exp = f"{columns[0]}-{columns[1]}"
+    try:
+        model.compute_contrast(exp)
+    except Exception:
+        with pytest.raises(
+                ValueError,
+                match='invalid python identifiers'
+        ):
+            model.compute_contrast(exp)
 
 
 def test_glm_sample_mask():
diff --git a/nilearn/glm/tests/test_hemodynamic_models.py b/nilearn/glm/tests/test_hemodynamic_models.py
--- a/nilearn/glm/tests/test_hemodynamic_models.py
+++ b/nilearn/glm/tests/test_hemodynamic_models.py
@@ -217,10 +217,6 @@ def custom_rf(tr, ov):
         [f"{name}_{custom_rf.__name__}"]
     assert _regressor_names(name, [custom_rf]) == \
         [f"{name}_{custom_rf.__name__}"]
-    assert _regressor_names(name, lambda tr, ov: np.ones(int(tr * ov))) == \
-        [f"{name}_lambda"]
-    assert _regressor_names(name, [lambda tr, ov: np.ones(int(tr * ov))]) == \
-        [f"{name}_lambda"]
 
     with pytest.raises(ValueError,
                        match="Computed regressor names are not unique"):
@@ -316,6 +312,13 @@ def test_make_regressor_3():
     assert_array_equal(reg, reg_)
 
 
+def test__regressor_names():
+    """ test that function allows invalid column identifier
+    """
+    reg_names = _regressor_names('1_cond', 'glover')
+    assert reg_names[0] == '1_cond'
+
+
 def test_design_warnings():
     """
     test that warnings are correctly raised upon weird design specification