Repository: readthedocs/readthedocs.org

Issue:
Build: `build.jobs.post_build` produces mysterious shell syntax errors
## Details

* Read the Docs project URL: https://readthedocs.org/projects/mmref/
* Build URL (if applicable): https://readthedocs.org/projects/mmref/builds/19671469/ and https://readthedocs.org/projects/mmref/builds/19671433/ and most previous  builds
* Read the Docs username (if applicable): https://readthedocs.org/profiles/rptb1/

## Expected Result

A post_build script `if test -n "A"; then echo "B"; fi` should not produce a syntax error.
A post_build script with `sh -c "foo"` in it should run the shell with the argument.

## Actual Result

`/bin/sh: 1: Syntax error: "then" unexpected`
`sh: 0: -c requires an argument`

I'm following the manual at ["Extend the build process"](https://docs.readthedocs.io/en/stable/build-customization.html#extend-the-build-process) and trying to insert a simple shell script at the post_build step.

A script that works locally (Ubuntu 22) is somehow mangled and causes syntax errors when inserted into the .readthedocs.yml at the post_build stage.  I have tried many ways of writing the script and quoting it in YAML, to try to eliminate YAML as the culprit:

1. https://github.com/Ravenbrook/mps/pull/187/commits/4146715a1253a9ba7dcb8e528997545d52bb7ab2
2. https://github.com/Ravenbrook/mps/pull/187/commits/9fbc6224dbe1b645568fd32a938c77f84ef67d3a
3. https://github.com/Ravenbrook/mps/pull/187/commits/2a642594a525a3cbcd0d9c10f686bf6b6e43307c
4. https://github.com/Ravenbrook/mps/pull/187/commits/7fdbbd3ff01544484ba5a518b46c96dc9c98f1f3
5. https://github.com/Ravenbrook/mps/pull/187/commits/82f0d180a1bb0a7909d94eee22ab4a7b71a7dea3
6. https://github.com/Ravenbrook/mps/pull/187/commits/044e790f24217ad709dbc8ced9c35febaf02f0e9
7. https://github.com/Ravenbrook/mps/pull/187/commits/4a07b8418a4e5b1036dd50f13552098308972fb2
8. https://github.com/Ravenbrook/mps/pull/187/commits/1106c9cd066c9638ccf2f7772083384bb526673f

The [last one](https://github.com/Ravenbrook/mps/pull/187/commits/1106c9cd066c9638ccf2f7772083384bb526673f) is especially mysterious.

Is there some sort of mangling of the script going on, or am I just going blind?

None of these scripts produce syntax errors when I feed them to /bin/sh on my local Ubuntu.  I have written similar scripts for multiple CI systems without coming across this problem before.

Thanks.


Discussion:
A *literal copy of your own example* produces a syntax error.

I coped the text of the script from https://docs.readthedocs.io/en/stable/build-customization.html#id3 exactly in https://github.com/Ravenbrook/mps/pull/187/commits/b08bcf32bcd8d0e38e3e4bcc474820c16eda53e6 and it causes the syntax error seen at https://readthedocs.org/projects/mmref/builds/19671564/
Thanks for reporting this issue. We have been dealing with this already at https://github.com/pypi/warehouse/pull/12953 and I wasn't able to find the cause of it at that time. We will need to dig deeper into the code to understand what's going on here.
> We will need to dig deeper into the code to understand what's going on here.

This split looks highly suspicious. https://github.com/readthedocs/readthedocs.org/blob/f4215c5516eaa07c15afe49abdf1bb7a1a34655a/readthedocs/doc_builder/director.py#L333

From a 10 minute scan of your code, it looks to me like there might be a confusion between executing a single command and executing a script.

On a single command, where you want to split argv and do something like `run(['ls', '-l'])`, because you're basically doing something like a call to [exec(3)](https://linux.die.net/man/3/exec) and spawing a single process.

But a *script* is a single string that you want to pass *complete* to a shell, like `run(['/bin/sh', '-c', SCRIPT])` (or similar).  Splitting the script makes no sense at all.

```
>>> post_build = """
... if test -n "${MMREF}"; then \
...          mv "${READTHEDOCS_OUTPUT}/html" "${READTHEDOCS_OUTPUT}/mmref.in" && \
...          python manual/make-mmref.py "${READTHEDOCS_OUTPUT}/mmref.in" "${READTHEDOCS_OUTPUT}/html" \
...     fi
... """
>>> post_build.split()
['if', 'test', '-n', '"${MMREF}";', 'then', 'mv', '"${READTHEDOCS_OUTPUT}/html"', '"${READTHEDOCS_OUTPUT}/mmref.in"', '&&', 'python', 'manual/make-mmref.py', '"${READTHEDOCS_OUTPUT}/mmref.in"', '"${READTHEDOCS_OUTPUT}/html"', 'fi']
```

Again, I am in no way sure this is what's wrong, but your splits don't look right to me.
@rptb1 I'm curious if it works if you put the script in a file, and just run that from the post-build step? Not ideal, but I don't think our indention with this feature was having full-length scripts here, but having commands or scripts that could be run. We can definitely change that design, but curious if it works that way. 

That said, if our docs examples aren't working, we should probably fix them :)
That might be a workaround for now, thanks.

The intention is clear from the documentation: currently the documented examples don't work.

You'll need to carefully document the subset of shell language you *do* support. Why not just pass everything to the shell? Is there a reason not to?

Resolving patch:
diff --git a/readthedocs/doc_builder/director.py b/readthedocs/doc_builder/director.py
--- a/readthedocs/doc_builder/director.py
+++ b/readthedocs/doc_builder/director.py
@@ -330,7 +330,7 @@ def run_build_job(self, job):
 
         commands = getattr(self.data.config.build.jobs, job, [])
         for command in commands:
-            environment.run(*command.split(), escape_command=False, cwd=cwd)
+            environment.run(command, escape_command=False, cwd=cwd)
 
     def run_build_commands(self):
         reshim_commands = (
@@ -344,7 +344,7 @@ def run_build_commands(self):
         cwd = self.data.project.checkout_path(self.data.version.slug)
         environment = self.build_environment
         for command in self.data.config.build.commands:
-            environment.run(*command.split(), escape_command=False, cwd=cwd)
+            environment.run(command, escape_command=False, cwd=cwd)
 
             # Execute ``asdf reshim python`` if the user is installing a
             # package since the package may contain an executable


Test patch:
diff --git a/readthedocs/projects/tests/test_build_tasks.py b/readthedocs/projects/tests/test_build_tasks.py
--- a/readthedocs/projects/tests/test_build_tasks.py
+++ b/readthedocs/projects/tests/test_build_tasks.py
@@ -916,10 +916,11 @@ def test_build_jobs(self, load_yaml_config):
 
         self.mocker.mocks["environment.run"].assert_has_calls(
             [
-                mock.call(
-                    "git", "fetch", "--unshallow", escape_command=False, cwd=mock.ANY
-                ),
-                mock.call("echo", "`date`", escape_command=False, cwd=mock.ANY),
+                # NOTE: when running commands from `build.jobs` or
+                # `build.commands` they are not split to allow multi-line
+                # scripts
+                mock.call("git fetch --unshallow", escape_command=False, cwd=mock.ANY),
+                mock.call("echo `date`", escape_command=False, cwd=mock.ANY),
             ],
             any_order=True,
         )
@@ -1035,10 +1036,11 @@ def test_build_commands(self, load_yaml_config):
                     "virtualenv",
                     "setuptools<58.3.0",
                 ),
+                # NOTE: when running commands from `build.jobs` or
+                # `build.commands` they are not split to allow multi-line
+                # scripts
                 mock.call(
-                    "pip",
-                    "install",
-                    "pelican[markdown]",
+                    "pip install pelican[markdown]",
                     escape_command=False,
                     cwd=mock.ANY,
                 ),
@@ -1051,12 +1053,7 @@ def test_build_commands(self, load_yaml_config):
                     cwd=mock.ANY,
                 ),
                 mock.call(
-                    "pelican",
-                    "--settings",
-                    "docs/pelicanconf.py",
-                    "--output",
-                    "$READTHEDOCS_OUTPUT/html/",
-                    "docs/",
+                    "pelican --settings docs/pelicanconf.py --output $READTHEDOCS_OUTPUT/html/ docs/",
                     escape_command=False,
                     cwd=mock.ANY,
                 ),