Repository: pypa/cibuildwheel

Issue:
`sh: No such file or directory` in `before_all` for linux builds with cibuildwheel v2.10.0
### Description

After updating from `2.9.0` to `v2.10.0`, I get the following error for linux builds using a simple `before-all` command like `before-all = "ls"`:
```       
Running before_all...
      + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
    + sh -c ls
env: sh: No such file or directory
```
This seems to be related to the fact that we're updating the `PATH` environment variable:
```
[tool.cibuildwheel]
before-all = "ls"
environment = { PATH="$PATH:/usr/local/bin" }
```


Discussion:
The host `$PATH` isn't valid inside the container. So it must have been expanded inside the container. Hmm, what I expect is happening is `shlex.quote` is protecting the envvars there; we added that to have better string quoting in https://github.com/pypa/cibuildwheel/pull/1244, didn't realize it would change envvar usage
@joerick two easy options:

We could call `os.path.expandvars` inside the container, which would expand these usages, but unfortunately, would ignore quoting protection choices (which is also fortunate, actually, since shlex.quote is adding single quotes - it just means there would be no way to pass a string with a valid environment variable as a string. Though, thinking about it, you might be able to use PASSENV in that case).

We could remove the quoting, and just trust users not to produce malformed input. It wasn't supported before 2.10 anyway.

A final "option" that I'm not calling an option would be to restructure this so it is not passed through as strings, but is passed through as a dict (making the toml version the "true" method and strings just a way to build that dict). But this is a larger architectural change.

Quick example:

```py
>>> shlex.quote("$HOME")
"'$HOME'"
```

This is quoted on the host, but needs to be expanded on the target.
I wonder if this is related to why https://github.com/scikit-build/cmake-python-distributions/pull/282 is breaking. I don't see an expansion, but maybe the quoting is not working as expected: https://github.com/scikit-build/cmake-python-distributions/blob/d8404ad6bcb294f2b95b0ceba26c9b35d248706f/pyproject.toml#L42

The only thing unique to that pair of jobs is the part: `-DRUN_CMAKE_TEST_EXCLUDE:STRING='BootstrapTest|ExportImport|RunCMake.install|RunCMake.file-GET_RUNTIME_DEPENDENCIES'`
Ah. Hmm. So it seems to be that the problem is that unlike the config-settings option, we're not calling `shlex.split` before these values are processed further. `shlex.split` would remove the quotes that `shlex.quote` added. But we use bashlex to parse these options, when the bashlex parser sees a single-quote, it's getting more from that than just tokenisation. It changes how env vars are processed.
Could we call `shlex.split`?
I think we can't - we can't call `shlex.split` on these values because they will sometimes contain single-quotes that really do mean something.
So, let's think what we want...

1. 
    ```toml
    environment = { PATH="$PATH:/usr/local/bin" }
    ```
    should translate to `PATH="$PATH:/usr/local/bin"` (or `PATH=$PATH:/usr/local/bin`)

2.
    ```toml
    environment = { SOME_STRING="a string with spaces" }
    ```
    should translate to `SOME_STRING="a string with spaces"` (or `SOME_STRING='a string with spaces'`)

3.
    ```toml
    environment = { SOME_STRING="an evil string with\" some strange punctuation" }
    ```
    should translate to `SOME_STRING="an evil string with\" some strange punctuation"`
4. 
    ```toml
    environment = { LITERAL_PASSTHROUGH="'this is not expanded: $PATH'" }
    ```
    should translate to `LITERAL_PASSTHROUGH='this is not expanded: $PATH'`

Though 3 might not be a huge priority... I'm not even sure I care about 4. much, tbh.
One idea I had was, when parsing the option from TOML, we could run `shlex.split` on it at option parse time, and see if it is more than one token. If it is, we run `shlex.quote` on it, to ensure that it's a valid env var assignment. Otherwise we don't quote these values at all.
☝️ I _think_ this might be a good approach - it gives users the most control - if they specify one token (i.e. get their bash quoting right) then cibuildwheel doesn't touch it. Only if the option value would form an invalid bash assignment do things get quoted. We _could_ raise a warning in this case, too?

~Or, the other option would be to just raise an error if the value is more than one bash token, and get the user to fix their config~ Nah, based on how you're using it above this would be pretty user-hostile I feel!
@henryiii and I have just been chatting on Discord. There is some concern that my proposed fix above would make the quoting behaviour even more confusing than it was before (which I kinda agree with!)

Based on @henryiii's build logs above, it also seems like there's might be an incompatibility between shlex and bashlex somewhere-

```
>>> import shlex
>>> shlex.quote("a string with 'single quotes'")
'\'a string with \'"\'"\'single quotes\'"\'"\'\''
>>> shlex.split(shlex.quote("a string with 'single quotes'"))
["a string with 'single quotes'"]
>>> import bashlex
>>> list(bashlex.split(shlex.quote("a string with 'single quotes'")))
['a string with \'"\'"\'single quotes\'"\'"\'']
```

We don't actually use bashlex.split anywhere though, so maybe it's not actually a problem. Needs more investigation!

> We could call `os.path.expandvars` inside the container, which would expand these usages, but unfortunately, would ignore quoting protection choices (which is also fortunate, actually, since shlex.quote is adding single quotes - it just means there would be no way to pass a string with a valid environment variable as a string. Though, thinking about it, you might be able to use PASSENV in that case).

I think this is only papering over the problem... correct quoting is also required for the bash parser to do command substitution with `$()` or backticks. It could also make it impossible to pass a literal `$` in an option.

> We could remove the quoting, and just trust users not to produce malformed input. It wasn't supported before 2.10 anyway.

Yeah, this is what I'm leaning towards, at least temporarily.

> A final "option" that I'm not calling an option would be to restructure this so it is not passed through as strings, but is passed through as a dict (making the toml version the "true" method and strings just a way to build that dict). But this is a larger architectural change.

Perhaps. Though we'd still have to figure out the tricky problem of how to translate between a dict and a bash string of assignments somewhere e.g. how do we reconcile this:

> @henryiii: I'm just not fond of this working:
> 
> ```
> PATH=$PATH:/local/bin
> PATH=/windows loves spaces/bin
> ```
> 
> But not this:
> ```
> PATH=$PATH:/windows loves spaces/bin
> ```

So I'm kinda leaning towards reverting to the previous behaviour for now, until we can figure out what the right policy is here.

Resolving patch:
diff --git a/cibuildwheel/environment.py b/cibuildwheel/environment.py
--- a/cibuildwheel/environment.py
+++ b/cibuildwheel/environment.py
@@ -4,6 +4,7 @@
 from typing import Any, Mapping, Sequence
 
 import bashlex
+import bashlex.errors
 
 from cibuildwheel.typing import Protocol
 
@@ -33,7 +34,11 @@ def split_env_items(env_string: str) -> list[str]:
     if not env_string:
         return []
 
-    command_node = bashlex.parsesingle(env_string)
+    try:
+        command_node = bashlex.parsesingle(env_string)
+    except bashlex.errors.ParsingError as e:
+        raise EnvironmentParseError(env_string) from e
+
     result = []
 
     for word_node in command_node.parts:
diff --git a/cibuildwheel/options.py b/cibuildwheel/options.py
--- a/cibuildwheel/options.py
+++ b/cibuildwheel/options.py
@@ -10,7 +10,7 @@
 from contextlib import contextmanager
 from dataclasses import asdict, dataclass
 from pathlib import Path
-from typing import Any, Dict, Generator, Iterator, List, Mapping, Union, cast
+from typing import Any, Callable, Dict, Generator, Iterator, List, Mapping, Union, cast
 
 if sys.version_info >= (3, 11):
     import tomllib
@@ -23,7 +23,7 @@
 from .environment import EnvironmentParseError, ParsedEnvironment, parse_environment
 from .oci_container import ContainerEngine
 from .projectfiles import get_requires_python_str
-from .typing import PLATFORMS, Literal, PlatformName, TypedDict
+from .typing import PLATFORMS, Literal, NotRequired, PlatformName, TypedDict
 from .util import (
     MANYLINUX_ARCHS,
     MUSLLINUX_ARCHS,
@@ -123,6 +123,7 @@ class Override:
 class TableFmt(TypedDict):
     item: str
     sep: str
+    quote: NotRequired[Callable[[str], str]]
 
 
 class ConfigOptionError(KeyError):
@@ -329,7 +330,7 @@ def get(
             if table is None:
                 raise ConfigOptionError(f"{name!r} does not accept a table")
             return table["sep"].join(
-                item for k, v in result.items() for item in _inner_fmt(k, v, table["item"])
+                item for k, v in result.items() for item in _inner_fmt(k, v, table)
             )
 
         if isinstance(result, list):
@@ -343,14 +344,16 @@ def get(
         return result
 
 
-def _inner_fmt(k: str, v: Any, table_item: str) -> Iterator[str]:
+def _inner_fmt(k: str, v: Any, table: TableFmt) -> Iterator[str]:
+    quote_function = table.get("quote", lambda a: a)
+
     if isinstance(v, list):
         for inner_v in v:
-            qv = shlex.quote(inner_v)
-            yield table_item.format(k=k, v=qv)
+            qv = quote_function(inner_v)
+            yield table["item"].format(k=k, v=qv)
     else:
-        qv = shlex.quote(v)
-        yield table_item.format(k=k, v=qv)
+        qv = quote_function(v)
+        yield table["item"].format(k=k, v=qv)
 
 
 class Options:
@@ -449,13 +452,13 @@ def build_options(self, identifier: str | None) -> BuildOptions:
 
             build_frontend_str = self.reader.get("build-frontend", env_plat=False)
             environment_config = self.reader.get(
-                "environment", table={"item": "{k}={v}", "sep": " "}
+                "environment", table={"item": '{k}="{v}"', "sep": " "}
             )
             environment_pass = self.reader.get("environment-pass", sep=" ").split()
             before_build = self.reader.get("before-build", sep=" && ")
             repair_command = self.reader.get("repair-wheel-command", sep=" && ")
             config_settings = self.reader.get(
-                "config-settings", table={"item": "{k}={v}", "sep": " "}
+                "config-settings", table={"item": "{k}={v}", "sep": " ", "quote": shlex.quote}
             )
 
             dependency_versions = self.reader.get("dependency-versions")
diff --git a/cibuildwheel/typing.py b/cibuildwheel/typing.py
--- a/cibuildwheel/typing.py
+++ b/cibuildwheel/typing.py
@@ -10,6 +10,10 @@
 else:
     from typing import Final, Literal, OrderedDict, Protocol, TypedDict
 
+if sys.version_info < (3, 11):
+    from typing_extensions import NotRequired
+else:
+    from typing import NotRequired
 
 __all__ = (
     "Final",
@@ -26,6 +30,7 @@
     "OrderedDict",
     "Union",
     "assert_never",
+    "NotRequired",
 )
 
 


Test patch:
diff --git a/unit_test/options_test.py b/unit_test/options_test.py
--- a/unit_test/options_test.py
+++ b/unit_test/options_test.py
@@ -1,11 +1,14 @@
 from __future__ import annotations
 
+import os
 import platform as platform_module
 import textwrap
+from pathlib import Path
 
 import pytest
 
 from cibuildwheel.__main__ import get_build_identifiers
+from cibuildwheel.bashlex_eval import local_environment_executor
 from cibuildwheel.environment import parse_environment
 from cibuildwheel.options import Options, _get_pinned_container_images
 
@@ -59,7 +62,7 @@ def test_options_1(tmp_path, monkeypatch):
 
     default_build_options = options.build_options(identifier=None)
 
-    assert default_build_options.environment == parse_environment("FOO=BAR")
+    assert default_build_options.environment == parse_environment('FOO="BAR"')
 
     all_pinned_container_images = _get_pinned_container_images()
     pinned_x86_64_container_image = all_pinned_container_images["x86_64"]
@@ -119,30 +122,75 @@ def test_passthrough_evil(tmp_path, monkeypatch, env_var_value):
     assert parsed_environment.as_dictionary(prev_environment={}) == {"ENV_VAR": env_var_value}
 
 
+xfail_env_parse = pytest.mark.xfail(
+    raises=SystemExit, reason="until we can figure out the right way to quote these values"
+)
+
+
 @pytest.mark.parametrize(
     "env_var_value",
     [
         "normal value",
-        '"value wrapped in quotes"',
-        'an unclosed double-quote: "',
+        pytest.param('"value wrapped in quotes"', marks=[xfail_env_parse]),
+        pytest.param('an unclosed double-quote: "', marks=[xfail_env_parse]),
         "string\nwith\ncarriage\nreturns\n",
-        "a trailing backslash \\",
+        pytest.param("a trailing backslash \\", marks=[xfail_env_parse]),
     ],
 )
 def test_toml_environment_evil(tmp_path, monkeypatch, env_var_value):
     args = get_default_command_line_arguments()
     args.package_dir = tmp_path
 
-    with tmp_path.joinpath("pyproject.toml").open("w") as f:
-        f.write(
-            textwrap.dedent(
-                f"""\
-        [tool.cibuildwheel.environment]
-        EXAMPLE='''{env_var_value}'''
-        """
-            )
+    tmp_path.joinpath("pyproject.toml").write_text(
+        textwrap.dedent(
+            f"""\
+            [tool.cibuildwheel.environment]
+            EXAMPLE='''{env_var_value}'''
+            """
         )
+    )
 
     options = Options(platform="linux", command_line_arguments=args)
     parsed_environment = options.build_options(identifier=None).environment
     assert parsed_environment.as_dictionary(prev_environment={}) == {"EXAMPLE": env_var_value}
+
+
+@pytest.mark.parametrize(
+    "toml_assignment,result_value",
+    [
+        ('TEST_VAR="simple_value"', "simple_value"),
+        # spaces
+        ('TEST_VAR="simple value"', "simple value"),
+        # env var
+        ('TEST_VAR="$PARAM"', "spam"),
+        ('TEST_VAR="$PARAM $PARAM"', "spam spam"),
+        # env var extension
+        ('TEST_VAR="before:$PARAM:after"', "before:spam:after"),
+        # env var extension with spaces
+        ('TEST_VAR="before $PARAM after"', "before spam after"),
+        # literal $ - this test is just for reference, I'm not sure if this
+        # syntax will work if we change the TOML quoting behaviour
+        (r'TEST_VAR="before\\$after"', "before$after"),
+    ],
+)
+def test_toml_environment_quoting(tmp_path: Path, toml_assignment, result_value):
+    args = get_default_command_line_arguments()
+    args.package_dir = tmp_path
+
+    tmp_path.joinpath("pyproject.toml").write_text(
+        textwrap.dedent(
+            f"""\
+            [tool.cibuildwheel.environment]
+            {toml_assignment}
+            """
+        )
+    )
+
+    options = Options(platform="linux", command_line_arguments=args)
+    parsed_environment = options.build_options(identifier=None).environment
+    environment_values = parsed_environment.as_dictionary(
+        prev_environment={**os.environ, "PARAM": "spam"},
+        executor=local_environment_executor,
+    )
+
+    assert environment_values["TEST_VAR"] == result_value