Repository: pantsbuild/pants

Issue:
Pants processing argv after --
just ran into this on 2.2, we were trying to do a `pants run path/to/binary -- binary’s arguments`. Now the binary’s arguments happened to have brackets [] in them and pants was aborting with:
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.2.0_py38/lib/python3.8/site-packages/pants/util/eval.py", line 47, in parse_expression
    raise raise_type(
pants.option.errors.ParseError: The value cannot be evaluated as a literal expression: SyntaxError('unexpected EOF while parsing', ('<string>', 1, 8, '[test] quoting'))
Given raw value:
1: [test] quoting

In v1 we made heavy use of pants run with -- expecting pants to stop processing argv after that. I tried looking at --run-args but it seems that the brackets still cause it to bail.


Discussion:
Thanks for reporting! Can you post a precise example command-line (redacted if necessary), in backticks?


All good:
```
$ ./pants test --force --output=all src/python/pants/util/ordered_set_test.py -- -vvsk test_xor[FrozenOrderedSet]
09:39:51.01 [INFO] Completed: test - src/python/pants/util/ordered_set_test.py:tests succeeded.
============================= test session starts ==============================
collecting ... collected 36 items / 35 deselected / 1 selected

src/python/pants/util/ordered_set_test.py::test_xor[FrozenOrderedSet] PASSED

======================= 1 passed, 35 deselected in 0.04s =======================



✓ src/python/pants/util/ordered_set_test.py:tests succeeded.
```

Still good:
```
$ ./pants test --force --output=all src/python/pants/util/ordered_set_test.py -- -vvsk r[FrozenOrderedSet]
09:40:04.83 [INFO] Completed: test - src/python/pants/util/ordered_set_test.py:tests succeeded.
============================= test session starts ==============================
collecting ... collected 36 items / 33 deselected / 3 selected

src/python/pants/util/ordered_set_test.py::test_stable_order[FrozenOrderedSet] PASSED
src/python/pants/util/ordered_set_test.py::test_repr[FrozenOrderedSet] PASSED
src/python/pants/util/ordered_set_test.py::test_xor[FrozenOrderedSet] PASSED

======================= 3 passed, 33 deselected in 0.04s =======================



✓ src/python/pants/util/ordered_set_test.py:tests succeeded.
```

Broken:
```
$ ./pants test --force --output=all src/python/pants/util/ordered_set_test.py -- -vvsk [FrozenOrderedSet]
09:40:10.99 [ERROR] Error computing value for --args in scope 'pytest' (may also be from PANTS_* environment variables).
Caused by:
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/util/eval.py", line 47, in parse_expression
    parsed_value = eval(val)
  File "<string>", line 1, in <module>
NameError: name 'FrozenOrderedSet' is not defined

...

pants.option.errors.ParseError: The value cannot be evaluated as a literal expression: NameError("name 'FrozenOrderedSet' is not defined")
Given raw value:
1: [FrozenOrderedSet]
```
Then this works:

```
./pants test --force --output=all src/python/pants/util/ordered_set_test.py -- "-vvsk [FrozenOrderedSet]"
```


That's a good workaround, but it seems to me we really should be able to stop parsing after the -- like all other tools that I know of that support --.
When I add this test:

```
diff --git c/tests/python/BUILD.echo i/tests/python/BUILD.echo
new file mode 100644
index 000000000..4ff7bc919
--- /dev/null
+++ i/tests/python/BUILD.echo
@@ -0,0 +1,9 @@
+python_library(
+    name="echolib",
+    sources=["echolib.py"]
+)
+
+pex_binary(
+    name="echolibbin",
+    entry_point="echolib:main",
+)
diff --git c/tests/python/echolib.py i/tests/python/echolib.py
new file mode 100644
index 000000000..8187b8d83
--- /dev/null
+++ i/tests/python/echolib.py
@@ -0,0 +1,4 @@
+import sys
+
+def main():
+    print(sys.argv)
```

I then run `pants run tests/python:echolibbin -- "[test] foo"` and it fails.

This works though, `pants run tests/python:echolibbin -- "foo [test]"

So the workaround only seems to work when the brackets aren't at the start of the argv.

Resolving patch:
diff --git a/src/python/pants/option/parser.py b/src/python/pants/option/parser.py
--- a/src/python/pants/option/parser.py
+++ b/src/python/pants/option/parser.py
@@ -669,7 +669,13 @@ def expand(val_or_str):
         # Get value from cmd-line flags.
         flag_vals = [to_value_type(expand(x)) for x in flag_val_strs]
         if kwargs.get("passthrough"):
-            flag_vals.extend(to_value_type(x) for x in passthru_arg_strs)
+            # NB: Passthrough arguments are either of type `str` or `shell_str`
+            # (see self._validate): the former never need interpretation, and the latter do not
+            # need interpretation when they have been provided directly via `sys.argv` as the
+            # passthrough args have been.
+            flag_vals.append(
+                ListValueComponent(ListValueComponent.MODIFY, [*passthru_arg_strs], [])
+            )
 
         if is_list_option(kwargs):
             # Note: It's important to set flag_val to None if no flags were specified, so we can


Test patch:
diff --git a/src/python/pants/option/options_test.py b/src/python/pants/option/options_test.py
--- a/src/python/pants/option/options_test.py
+++ b/src/python/pants/option/options_test.py
@@ -1046,6 +1046,27 @@ def test_at_most_one_goal_with_passthru_args(self):
             "(args after `--`) is ambiguous."
         ) in str(exc.value)
 
+    def test_passthru_args_not_interpreted(self):
+        # Test that passthrough args are not interpreted.
+        options = Options.create(
+            env={},
+            config=self._create_config(),
+            known_scope_infos=[global_scope(), task("test"), subsystem("consumer")],
+            args=["./pants", "--consumer-shlexed=a", "--consumer-string=b", "test", "--", "[bar]"],
+        )
+        options.register(
+            "consumer", "--shlexed", passthrough=True, type=list, member_type=shell_str
+        )
+        options.register("consumer", "--string", passthrough=True, type=list, member_type=str)
+        self.assertEqual(
+            ["a", "[bar]"],
+            options.for_scope("consumer").shlexed,
+        )
+        self.assertEqual(
+            ["b", "[bar]"],
+            options.for_scope("consumer").string,
+        )
+
     def test_global_scope_env_vars(self):
         def check_pants_foo(expected_val, env):
             val = self._parse(env=env).for_global_scope().pants_foo