Repository: readthedocs/readthedocs.org

Issue:
Build: `build.jobs.post_build` produces mysterious shell syntax errors
## Details

* Read the Docs project URL: https://readthedocs.org/projects/mmref/
* Build URL (if applicable): https://readthedocs.org/projects/mmref/builds/19671469/ and https://readthedocs.org/projects/mmref/builds/19671433/ and most previous  builds
* Read the Docs username (if applicable): https://readthedocs.org/profiles/rptb1/

## Expected Result

A post_build script `if test -n "A"; then echo "B"; fi` should not produce a syntax error.
A post_build script with `sh -c "foo"` in it should run the shell with the argument.

## Actual Result

`/bin/sh: 1: Syntax error: "then" unexpected`
`sh: 0: -c requires an argument`

I'm following the manual at ["Extend the build process"](https://docs.readthedocs.io/en/stable/build-customization.html#extend-the-build-process) and trying to insert a simple shell script at the post_build step.

A script that works locally (Ubuntu 22) is somehow mangled and causes syntax errors when inserted into the .readthedocs.yml at the post_build stage.  I have tried many ways of writing the script and quoting it in YAML, to try to eliminate YAML as the culprit:

1. https://github.com/Ravenbrook/mps/pull/187/commits/4146715a1253a9ba7dcb8e528997545d52bb7ab2
2. https://github.com/Ravenbrook/mps/pull/187/commits/9fbc6224dbe1b645568fd32a938c77f84ef67d3a
3. https://github.com/Ravenbrook/mps/pull/187/commits/2a642594a525a3cbcd0d9c10f686bf6b6e43307c
4. https://github.com/Ravenbrook/mps/pull/187/commits/7fdbbd3ff01544484ba5a518b46c96dc9c98f1f3
5. https://github.com/Ravenbrook/mps/pull/187/commits/82f0d180a1bb0a7909d94eee22ab4a7b71a7dea3
6. https://github.com/Ravenbrook/mps/pull/187/commits/044e790f24217ad709dbc8ced9c35febaf02f0e9
7. https://github.com/Ravenbrook/mps/pull/187/commits/4a07b8418a4e5b1036dd50f13552098308972fb2
8. https://github.com/Ravenbrook/mps/pull/187/commits/1106c9cd066c9638ccf2f7772083384bb526673f

The [last one](https://github.com/Ravenbrook/mps/pull/187/commits/1106c9cd066c9638ccf2f7772083384bb526673f) is especially mysterious.

Is there some sort of mangling of the script going on, or am I just going blind?

None of these scripts produce syntax errors when I feed them to /bin/sh on my local Ubuntu.  I have written similar scripts for multiple CI systems without coming across this problem before.

Thanks.


Discussion:
A *literal copy of your own example* produces a syntax error.

I coped the text of the script from https://docs.readthedocs.io/en/stable/build-customization.html#id3 exactly in https://github.com/Ravenbrook/mps/pull/187/commits/b08bcf32bcd8d0e38e3e4bcc474820c16eda53e6 and it causes the syntax error seen at https://readthedocs.org/projects/mmref/builds/19671564/
Thanks for reporting this issue. We have been dealing with this already at https://github.com/pypi/warehouse/pull/12953 and I wasn't able to find the cause of it at that time. We will need to dig deeper into the code to understand what's going on here.
> We will need to dig deeper into the code to understand what's going on here.

This split looks highly suspicious. https://github.com/readthedocs/readthedocs.org/blob/f4215c5516eaa07c15afe49abdf1bb7a1a34655a/readthedocs/doc_builder/director.py#L333

From a 10 minute scan of your code, it looks to me like there might be a confusion between executing a single command and executing a script.

On a single command, where you want to split argv and do something like `run(['ls', '-l'])`, because you're basically doing something like a call to [exec(3)](https://linux.die.net/man/3/exec) and spawing a single process.

But a *script* is a single string that you want to pass *complete* to a shell, like `run(['/bin/sh', '-c', SCRIPT])` (or similar).  Splitting the script makes no sense at all.

```
>>> post_build = """
... if test -n "${MMREF}"; then \
...          mv "${READTHEDOCS_OUTPUT}/html" "${READTHEDOCS_OUTPUT}/mmref.in" && \
...          python manual/make-mmref.py "${READTHEDOCS_OUTPUT}/mmref.in" "${READTHEDOCS_OUTPUT}/html" \
...     fi
... """
>>> post_build.split()
['if', 'test', '-n', '"${MMREF}";', 'then', 'mv', '"${READTHEDOCS_OUTPUT}/html"', '"${READTHEDOCS_OUTPUT}/mmref.in"', '&&', 'python', 'manual/make-mmref.py', '"${READTHEDOCS_OUTPUT}/mmref.in"', '"${READTHEDOCS_OUTPUT}/html"', 'fi']
```

Again, I am in no way sure this is what's wrong, but your splits don't look right to me.
@rptb1 I'm curious if it works if you put the script in a file, and just run that from the post-build step? Not ideal, but I don't think our indention with this feature was having full-length scripts here, but having commands or scripts that could be run. We can definitely change that design, but curious if it works that way. 

That said, if our docs examples aren't working, we should probably fix them :)
That might be a workaround for now, thanks.

The intention is clear from the documentation: currently the documented examples don't work.

You'll need to carefully document the subset of shell language you *do* support. Why not just pass everything to the shell? Is there a reason not to?
I opened a PR that removes the `.split()` on those command and interprets the text as a whole. I did some small QA locally and it worked. I'd appreciate some review on that code since I'm not 100% sure that it's not gonna cause other issues.
I'm re-opening this issue because the PR that was merged doesn't solve all the issues.
I'm still not sure what's the best way to solve this problem 🤷🏼 . This case, for example, is not solved:

![Screenshot_2023-03-09_10-41-14](https://user-images.githubusercontent.com/244656/223982456-2a45cb79-43da-47a4-a545-355440b9b7df.png)

That exact same code works properly in my local terminal. Even wrapping it in `/bin/sh -c '{cmd}'` as we are doing in our code. It seems we are doing some manipulation of the command that breaks it, or, there is a problem when communicating with Docker.

This requires more researching and testing. Suggestions and ideas welcomed!
I think the main problem here is the `PATH=` environment variable prefix we are adding:

It works without the prefix:
```
▶ sh -c 'if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- docs/ .readthedocs.yaml;
then
 exit 183;
fi'
```

It *fails* with the prefix
```
▶ sh -c 'PATH=/bin:$PATH if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- docs/ .readthedocs.yaml;
then
 exit 183;
fi'
sh: line 1: if: command not found
sh: -c: line 2: syntax error near unexpected token `then'
sh: -c: line 2: `then'
```

This code is at https://github.com/readthedocs/readthedocs.org/blob/dd07a3e9d467d9fd0047f6faf40d08fe0afed019/readthedocs/doc_builder/environments.py#L356-L384
Another simplified example that breaks because of the environment variable prefix:

```
▶ sh -c 'if [ "1" ]; then echo 1; fi'
1

▶ sh -c 'FOO=bar if [ "1" ]; then echo 1; fi'
sh: -c: line 1: syntax error near unexpected token `then'
sh: -c: line 1: `FOO=bar if [ "1" ]; then echo 1; fi'
```

Resolving patch:
diff --git a/readthedocs/doc_builder/environments.py b/readthedocs/doc_builder/environments.py
--- a/readthedocs/doc_builder/environments.py
+++ b/readthedocs/doc_builder/environments.py
@@ -14,7 +14,7 @@
 from docker.errors import APIError as DockerAPIError
 from docker.errors import DockerException
 from docker.errors import NotFound as DockerNotFoundError
-from requests.exceptions import ConnectionError, ReadTimeout
+from requests.exceptions import ConnectionError, ReadTimeout  # noqa
 from requests_toolbelt.multipart.encoder import MultipartEncoder
 
 from readthedocs.api.v2.client import api as api_v2
@@ -73,7 +73,7 @@ def __init__(
         bin_path=None,
         record_as_success=False,
         demux=False,
-        **kwargs,
+        **kwargs,  # pylint: disable=unused-argument
     ):
         self.command = command
         self.shell = shell
@@ -252,8 +252,8 @@ def save(self):
                 {key: str(value) for key, value in data.items()}
             )
             resource = api_v2.command
-            resp = resource._store['session'].post(
-                resource._store['base_url'] + '/',
+            resp = resource._store["session"].post(  # pylint: disable=protected-access
+                resource._store["base_url"] + "/",  # pylint: disable=protected-access
                 data=encoder,
                 headers={
                     'Content-Type': encoder.content_type,
@@ -301,11 +301,40 @@ def run(self):
 
         self.start_time = datetime.utcnow()
         client = self.build_env.get_client()
+
+        # Create a copy of the environment to update PATH variable
+        environment = self._environment.copy()
+        # Default PATH variable
+        # This default comes from our Docker image:
+        #
+        # $ docker run --user docs -it --rm readthedocs/build:ubuntu-22.04 /bin/bash
+        # docs@bfe702e31cdd:~$ echo $PATH
+        # /home/docs/.asdf/shims:/home/docs/.asdf/bin
+        #  :/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+        # docs@bfe702e31cdd:~$
+        asdf_paths = (
+            f"/home/{settings.RTD_DOCKER_USER}/.asdf/shims"
+            f":/home/{settings.RTD_DOCKER_USER}/.asdf/bin"
+        )
+        if settings.RTD_DOCKER_COMPOSE:
+            asdf_paths += ":/root/.asdf/shims:/root/.asdf/bin"
+
+        default_paths = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+        environment["PATH"] = f"{asdf_paths}:{default_paths}"
+
+        # Prepend the BIN_PATH if it's defined
+        if self.bin_path:
+            original_path = environment.get("PATH")
+            escaped_bin_path = self._escape_command(self.bin_path)
+            environment["PATH"] = escaped_bin_path
+            if original_path:
+                environment["PATH"] = f"{escaped_bin_path}:{original_path}"
+
         try:
             exec_cmd = client.exec_create(
                 container=self.build_env.container_id,
                 cmd=self.get_wrapped_command(),
-                environment=self._environment,
+                environment=environment,
                 user=self.user,
                 workdir=self.cwd,
                 stdout=True,
@@ -357,31 +386,18 @@ def get_wrapped_command(self):
         """
         Wrap command in a shell and optionally escape special bash characters.
 
-        In order to set the current working path inside a docker container, we
-        need to wrap the command in a shell call manually.
-
         Some characters will be interpreted as shell characters without
         escaping, such as: ``pip install requests<0.8``. When passing
         ``escape_command=True`` in the init method this escapes a good majority
         of those characters.
         """
-        prefix = ''
-        if self.bin_path:
-            bin_path = self._escape_command(self.bin_path)
-            prefix += f'PATH={bin_path}:$PATH '
-
         command = (
             ' '.join(
                 self._escape_command(part) if self.escape_command else part
                 for part in self.command
             )
         )
-        return (
-            "/bin/sh -c '{prefix}{cmd}'".format(
-                prefix=prefix,
-                cmd=command,
-            )
-        )
+        return f"/bin/bash -c '{command}'"
 
     def _escape_command(self, cmd):
         r"""Escape the command by prefixing suspicious chars with `\`."""
@@ -524,14 +540,14 @@ class BuildEnvironment(BaseEnvironment):
     """
 
     def __init__(
-            self,
-            project=None,
-            version=None,
-            build=None,
-            config=None,
-            environment=None,
-            record=True,
-            **kwargs,
+        self,
+        project=None,
+        version=None,
+        build=None,
+        config=None,
+        environment=None,
+        record=True,
+        **kwargs,  # pylint: disable=unused-argument
     ):
         super().__init__(project, environment)
         self.version = version
@@ -557,7 +573,7 @@ def run(self, *cmd, **kwargs):
         })
         return super().run(*cmd, **kwargs)
 
-    def run_command_class(self, *cmd, **kwargs):  # pylint: disable=arguments-differ
+    def run_command_class(self, *cmd, **kwargs):  # pylint: disable=signature-differs
         kwargs.update({
             'build_env': self,
         })


Test patch: