Repository: zulip/zulip

Issue:
upgrade-zulip-from-git: Provide useful error messages when `update-prod-static` fails
In https://github.com/zulip/zulip/issues/14615, we have `upgrade-zulip-from-git` failing inside `update-prod-static`, and the error message is terrible:

```
2020-04-17 21:23:41,590 upgrade-zulip-stage-2: Building static assets...
Cached version not found! Installing node modules.
+ /srv/zulip-yarn/bin/yarn install --non-interactive --frozen-lockfile --prod

Error running a subcommand of ./tools/update-prod-static: /srv/zulip-yarn/bin/yarn install --non-interactive --frozen-lockfile --prod
Actual error output for the subcommand is just above this.

Traceback (most recent call last):
  File "./tools/update-prod-static", line 37, in <module>
    setup_node_modules(production=True, stdout=fp, stderr=fp)
  File "./tools/../scripts/lib/node_cache.py", line 66, in setup_node_modules
    stderr=stderr)
  File "./tools/../scripts/lib/node_cache.py", line 91, in do_yarn_install
    cwd=target_path, stdout=stdout, stderr=stderr)
  File "./tools/../scripts/lib/zulip_tools.py", line 200, in run
    subprocess.check_call(args, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/srv/zulip-yarn/bin/yarn', 'install', '--non-interactive', '--frozen-lockfile', '--prod']' returned non-zero exit status 1
Traceback (most recent call last):
  File "/home/zulip/deployments/2020-04-18-01-53-13/scripts/lib/upgrade-zulip-stage-2", line 122, in <module>
    preexec_fn=su_to_zulip)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./tools/update-prod-static', '--authors-not-required', '--prev-deploy', '/home/zulip/deployments/current']' returned non-zero exit status 1
Traceback (most recent call last):
  File "/home/zulip/deployments/current/scripts/lib/upgrade-zulip-from-git", line 69, in <module>
    deploy_path, "--from-git"] + deploy_options)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/zulip/deployments/2020-04-18-01-53-13/scripts/lib/upgrade-zulip-stage-2', '/home/zulip/deployments/2020-04-18-01-53-13', '--from-git']' returned non-zero exit status 1
```

Because `update-prod-static` is failing, the actual error messages is in `/home/zulip/deployments/next/var/log/upgrade-prod-static.log`.

`update-prod-static` itself has very verbose output, so I think there's two paths forward:
* Simplest is to copy `build-release-tarball` in at least printing the path to `upgrade-prod-static.log`, and perhaps catching and not emitting much of that unhelpful traceback.
* Better would be to make `update-prod-static` have more sane output.
upgrade-zulip-from-git: Provide useful error messages when `update-prod-static` fails 
This just prints the path to the prod static log file. 


Discussion:
Hello @zulip/server-production members, this issue was labeled with the "area: production", "area: production installer" labels, so you may want to check it out!

<!-- areaLabelAddition -->
Hello @zulip/server-production members, this pull request was labeled with the "area: production", "area: production installer" labels, so you may want to check it out!

<!-- areaLabelAddition -->
Posted one comment; can you also post example output?
This pr is broken sorry.
@timabbott updated and this works now.

Sample error output:
```
2020-04-22 17:59:23,483 upgrade-zulip-stage-2: Building static assets...
Using cached node modules from /srv/zulip-npm-cache/c18d6cee3155e6013068b5a7f785e30f09ca6a0a/node_modules
+ ./tools/setup/emoji/build_emoji
+ ./scripts/setup/inline_email_css.py
+ ./tools/setup/generate_zulip_bots_static_files.py

Error running a subcommand of ./tools/update-prod-static: ./tools/setup/generate_zulip_bots_static_files.py
Actual error output for the subcommand is just above this.

Traceback (most recent call last):
  File "./tools/update-prod-static", line 49, in <module>
    run(['./tools/setup/generate_zulip_bots_static_files.py'], stdout=fp, stderr=fp)
  File "./tools/../scripts/lib/zulip_tools.py", line 189, in run
    subprocess.check_call(args, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./tools/setup/generate_zulip_bots_static_files.py']' returned non-zero exit status 1.
2020-04-22 17:59:25,233 upgrade-zulip-stage-2: Building static assets failed.
build_emoji: Using cached emojis from /srv/zulip-emoji-cache/933bee7b5c0c76b88258c010e6d2ef597f251825/emoji
Traceback (most recent call last):
  File "./tools/setup/generate_zulip_bots_static_files.py", line 10, in <module>
    assert 1==2
AssertionError

2020-04-22 17:59:25,234 upgrade-zulip-stage-2: Please check /home/zulip/deployments/next/var/log/update-prod-static.log for the complete
error.
```

I was testing to get the error output above the `Actual error output for the subcommand is just above this.` line but that's some trouble and not easy for me to do.
I think we probably want to just post the log file path and maybe its last ~50 lines at most; if `update-prod-static` fails halfway through the `webpack` stage, the `update-prod-static` output that you're pasting here will be like 50 screens long, which isn't a good experience.  
Oh! Updated to just print the file path.
This pre-dates the PR, but I wonder if this would be nicer to read in `update-prod-static` (and then helpful for cross-checking against places where we document more manual installs for things like half-supported OSes):

~~~
tools = [
    './tools/setup/emoji/build_emoji',
    './scripts/setup/inline_email_css.py',
    './tools/setup/generate_zulip_bots_static_files.py',
    './tools/setup/build_pygments_data',
    './tools/webpack',
]

for tool in tools:
    run([tool], stdout=fp, stderr=fp)
~~~


Hmm, thinking about this more, I think we might want to try modifying `update-prod-static` so that only  `manage.py collectstatic` doesn't get logged to stdout; that's rarely the place things fail, and I think it's also the only place that produces a million lines of output.
OK, I think with some small changes we can make the output like this with everything but `manage.py collectstatic` going to stdout.  And that bit basically never fails:
```
+ ./tools/update-prod-static
Using cached node modules from /srv/zulip-npm-cache/c18d6cee3155e6013068b5a7f785e30f09ca6a0a/node_modules
+ ./tools/setup/emoji/build_emoji
build_emoji: Using cached emojis from /srv/zulip-emoji-cache/2a8798306d68e75ed7d6e2f23f2376b0f048ec88/emoji
+ ./scripts/setup/inline_email_css.py
+ ./tools/setup/generate_zulip_bots_static_files.py
+ ./tools/setup/build_pygments_data
+ ./tools/webpack --quiet
Starting webpack compilation
+ ./manage.py collectstatic --no-default-ignore --noinput -i assets -i emoji-styles -i html -i js -i styles -i templates
+ ./manage.py compilemessages -v0
```

Resolving patch:
diff --git a/scripts/lib/node_cache.py b/scripts/lib/node_cache.py
--- a/scripts/lib/node_cache.py
+++ b/scripts/lib/node_cache.py
@@ -3,7 +3,7 @@
 import json
 import shutil
 
-from typing import Optional, List, IO, Any
+from typing import Optional, List
 from scripts.lib.zulip_tools import subprocess_text_output, run
 
 ZULIP_PATH = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
@@ -49,8 +49,6 @@ def generate_sha1sum_node_modules(
 
 def setup_node_modules(
     production: bool = DEFAULT_PRODUCTION,
-    stdout: Optional[IO[Any]] = None,
-    stderr: Optional[IO[Any]] = None,
     prefer_offline: bool = False,
 ) -> None:
     yarn_args = get_yarn_args(production=production)
@@ -64,9 +62,7 @@ def setup_node_modules(
     if not os.path.exists(success_stamp):
         do_yarn_install(target_path,
                         yarn_args,
-                        success_stamp,
-                        stdout=stdout,
-                        stderr=stderr)
+                        success_stamp)
 
     print("Using cached node modules from %s" % (cached_node_modules,))
     if os.path.islink('node_modules'):
@@ -78,9 +74,7 @@ def setup_node_modules(
 def do_yarn_install(
     target_path: str,
     yarn_args: List[str],
-    success_stamp: str,
-    stdout: Optional[IO[Any]] = None,
-    stderr: Optional[IO[Any]] = None,
+    success_stamp: str
 ) -> None:
     os.makedirs(target_path, exist_ok=True)
     shutil.copy('package.json', target_path)
@@ -93,9 +87,8 @@ def do_yarn_install(
     if os.path.exists("node_modules") and not os.path.exists(cached_node_modules):
         shutil.copytree("node_modules/", cached_node_modules, symlinks=True)
     if os.environ.get('CUSTOM_CA_CERTIFICATES'):
-        run([YARN_BIN, "config", "set", "cafile", os.environ['CUSTOM_CA_CERTIFICATES']],
-            stdout=stdout, stderr=stderr)
+        run([YARN_BIN, "config", "set", "cafile", os.environ['CUSTOM_CA_CERTIFICATES']])
     run([YARN_BIN, "install", "--non-interactive", "--frozen-lockfile"] + yarn_args,
-        cwd=target_path, stdout=stdout, stderr=stderr)
+        cwd=target_path)
     with open(success_stamp, 'w'):
         pass


Test patch: