Repository: matthewwardrop/formulaic

Issue:
Linear constraint parsing without advanced rules
It is possible to use a simpler parser when parsing linear constraints?  I am having problems with existing code that has variable names that were produced from formulaic, e.g., 

```python
import numpy as np
from formulaic.utils.constraints import LinearConstraints

cons = "ylocus_of_control_prog[T.general]"

variable_names = [
    "ylocus_of_control_Intercept",
    "ylocus_of_control_read",
    "ylocus_of_control_write",
    "ylocus_of_control_science",
    "ylocus_of_control_prog[T.general]",
    "ylocus_of_control_prog[T.vocational]",
    "yself_concept_Intercept",
    "yself_concept_read",
    "yself_concept_write",
    "yself_concept_science",
    "yself_concept_prog[T.general]",
    "yself_concept_prog[T.vocational]",
    "ymotivation_Intercept",
    "ymotivation_read",
    "ymotivation_write",
    "ymotivation_science",
    "ymotivation_prog[T.general]",
    "ymotivation_prog[T.vocational]",
]

lc = LinearConstraints.from_spec(cons, variable_names=variable_names)

```


The raises due to the presence of the ```[T.general]``` from the categorical expansion.

Discussion:
@matthewwardrop: Hi @bashtage !

Thanks for bring this up again here to make sure I don't lose sight of it. IIUC, the request here is to remove the requirement for having to quote variables names in the linear constraint formula (as discussed elsewhere quoting the name of the column with backticks would work fine).

I took a quick look at this, and it actually looks like this is caused by an overzealous check in the tokenizer, the original motivation for which I have forgotten. Patching this resulted in the above code working as written. I'll push up this patch tomorrow :).

Thanks again for continuing to push on things on the statsmodels side!@bashtage: Thanks for this.  It is one of about 3 major stumbling blocks left.  The otherS are trying to match patsy ordering, which triggered the other PR, and some missing value handling, which needs some more investigation.  

Solution patch:
diff --git a/formulaic/parser/algos/tokenize.py b/formulaic/parser/algos/tokenize.py
index f7c1607e..8c73bfe1 100644
--- a/formulaic/parser/algos/tokenize.py
+++ b/formulaic/parser/algos/tokenize.py
@@ -88,9 +88,9 @@ def tokenize(
                 yield token
                 token = Token(source=formula)
             continue
-        if quote_context and quote_context[-1] in ('"', "'", "`", ")", "}", "%"):
-            if char in "(`" and quote_context[-1] in "})":
-                quote_context.append(char.replace("(", ")"))
+        if quote_context and quote_context[-1] in ('"', "'", "`", ")", "]", "}", "%"):
+            if char in "`([" and quote_context[-1] in "})]":
+                quote_context.append(char.replace("(", ")").replace("[", "]"))
             token.update(char, i)
             continue
 


Test patch:
diff --git a/tests/parser/algos/test_tokenize.py b/tests/parser/algos/test_tokenize.py
index 2c297700..512884ae 100644
--- a/tests/parser/algos/test_tokenize.py
+++ b/tests/parser/algos/test_tokenize.py
@@ -61,6 +61,12 @@
         "operator:in",
         "operator:custom op",
     ],
+    "A[T.a]": [
+        "python:A[T.a]",
+    ],
+    "A(T.a)": [
+        "python:A(T.a)",
+    ],
 }
 
 TOKEN_ERRORS = {
@@ -69,6 +75,14 @@
         FormulaSyntaxError,
         "Formula ended before quote context was closed. Expected: `",
     ],
+    "A[(T.a]": [
+        FormulaSyntaxError,
+        r"Formula ended before quote context was closed. Expected: \)",
+    ],
+    "A([T.a)": [
+        FormulaSyntaxError,
+        "Formula ended before quote context was closed. Expected: ]",
+    ],
 }
 
 
diff --git a/tests/utils/test_constraints.py b/tests/utils/test_constraints.py
index 23eff7db..e530b74b 100644
--- a/tests/utils/test_constraints.py
+++ b/tests/utils/test_constraints.py
@@ -39,6 +39,12 @@ class TestLinearConstraints:
                     {"a + b + c = 10": -10}, variable_names=["a", "b", "c"]
                 ),
             ),
+            (
+                1,
+                LinearConstraints.from_spec(
+                    {"A[T.a] + b + c = 10": -10}, variable_names=["A[T.a]", "b", "c"]
+                ),
+            ),
             (1, LinearConstraints.from_spec([1, 1, 1], variable_names=["a", "b", "c"])),
             (
                 1,
@@ -88,6 +94,13 @@ class TestLinearConstraints:
                     {"a + b + c = 5": 5, "a - c - 5": 5}, variable_names=["a", "b", "c"]
                 ),
             ),
+            (
+                2,
+                LinearConstraints.from_spec(
+                    {"A[T.a] + b + c = 5": 5, "A[T.a] - c - 5": 5},
+                    variable_names=["A[T.a]", "b", "c"],
+                ),
+            ),
             (
                 2,
                 LinearConstraints.from_spec(