Repository: spack/spack

Issue:
Adding --log-format to spack install alway returns exit code 0
Here is an example:

```
$ spack install --log-format=junit --dirty coreneuron +gpu %pgi
.....
==> No patches needed for coreneuron
==> Building coreneuron [CMakePackage]
==> Executing phase: 'cmake'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'
See build log for details:
.....

$ echo $?
0

```

Without `--log-format`:

```
$ spack install --dirty coreneuron +gpu %pgi
....
==> Executing phase: 'cmake'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16'
See build log for details:
....

$ echo $?
1
```


Discussion:
@tgamblin : I forgot to mention this issue in #5388. `--show-log-on-error` also remains silent when we add `--log-format=junit` to install, for example:

```
$ spack install --show-log-on-error --log-format=junit coreneuron %gcc@4.9.0
==> Installing mod2c
==> Already fetched /some/path/spack/var/spack/stage/mod2c-develop-xxxx/mod2c
==> No checksum needed when fetching with git
==> Already staged mod2c-develop-xxxx in /some/path/spack/var/spack/stage/mod2c-develop-xxxx
==> No patches needed for mod2c
==> Building mod2c [CMakePackage]
==> Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    'cmake' '/some/path/spack/var/spack/stage/mod2c-develop-xxxx/mod2c' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:PATH=/some/xxxx/mod2c-develop-ion2w7' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE' '-DCMAKE_INSTALL_RPATH:STRING=/some/xxxx/mod2c-develop-ion2w7/lib:/some/xxxx/mod2c-develop-ion2w7/lib64:/gpfs/bbp.cscs.ch/apps/viz/tools/mpc/mpc-1.0.2/install' '-DCMAKE_PREFIX_PATH:STRING=/some/zzzz/dev/cmake/3.6.2'
See build log for details:
  /some/path/spack/var/spack/stage/mod2c-develop-xxxx/mod2c/spack-build.out

==> Installing coreneuron
==> Already fetched /some/path/spack/var/spack/stage/coreneuron-develop-rhg6imgqungzdsee42fzcwdvmiitcjmz/CoreNeuron
==> No checksum needed when fetching with git
==> Already staged coreneuron-develop-rhg6imgqungzdsee42fzcwdvmiitcjmz in /some/path/spack/var/spack/stage/coreneuron-develop-rhg6imgqungzdsee42fzcwdvmiitcjmz
==> No patches needed for coreneuron
==> Building coreneuron [CMakePackage]
==> Executing phase: 'cmake'
==> Error: ProcessError: Command exited with status 1:
    'cmake' '/some/path/spack/var/spack/stage/coreneuron-develop-rhg6imgqungzdsee42fzcwdvmiitcjmz/CoreNeuron' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:PATH=/some/xxxx/coreneuron-develop-rhg6im' '-DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo' '-DCMAKE_VERBOSE_MAKEFILE:BOOL…….NAL_MECHPATH=;/some/xxxx/neurodamus-develop-uy5woo/lib/modlib'
See build log for details:
  /some/path/spack/var/spack/stage/coreneuron-develop-rhg6imgqungzdsee42fzcwdvmiitcjmz/CoreNeuron/spack-build.out
```
@tgamblin Can you take a look at this while you're preparing CDash support? Otherwise I can take a stab at it if modifying that part of the code doesn't get in your way.
Because of this bug, `--log-format junit|cdash` is kind of unusable for me.

In addition to not exiting with an error when a build fails, worse, with `--log-format`, spack still goes on building the dependen**ts** of failed builds.

And it also ignores the `--fail-fast` options.

Here is a quick mocked test with `--fail-fast`, if that helps:

```sh
$ spack --mock install --fail-fast failing-build b ; echo exit=$?
...(failed failing-build)...
exit=1
#
# vs
#
$ spack --mock install --log-format junit --fail-fast failing-build b ; echo exit=$?
...(failed failing-build)...
...(still installs b)...
exit=0
```

Resolving patch:
diff --git a/lib/spack/spack/report.py b/lib/spack/spack/report.py
--- a/lib/spack/spack/report.py
+++ b/lib/spack/spack/report.py
@@ -151,6 +151,22 @@ def wrapper(instance, *args, **kwargs):
                     'installed_from_binary_cache': False
                 }
 
+                # Append the package to the correct spec report. In some
+                # cases it may happen that a spec that is asked to be
+                # installed explicitly will also be installed as a
+                # dependency of another spec. In this case append to both
+                # spec reports.
+                for s in llnl.util.lang.dedupe([pkg.spec.root, pkg.spec]):
+                    name = name_fmt.format(s.name, s.dag_hash(length=7))
+                    try:
+                        item = next((
+                            x for x in self.specs
+                            if x['name'] == name
+                        ))
+                        item['packages'].append(package)
+                    except StopIteration:
+                        pass
+
                 start_time = time.time()
                 value = None
                 try:
@@ -170,6 +186,7 @@ def wrapper(instance, *args, **kwargs):
                     package['stdout'] = fetch_log(pkg, do_fn, self.dir)
                     package['stdout'] += package['message']
                     package['exception'] = e.traceback
+                    raise
 
                 except (Exception, BaseException) as e:
                     # Everything else is an error (the installation
@@ -178,26 +195,11 @@ def wrapper(instance, *args, **kwargs):
                     package['stdout'] = fetch_log(pkg, do_fn, self.dir)
                     package['message'] = str(e) or 'Unknown error'
                     package['exception'] = traceback.format_exc()
+                    raise
 
                 finally:
                     package['elapsed_time'] = time.time() - start_time
 
-                # Append the package to the correct spec report. In some
-                # cases it may happen that a spec that is asked to be
-                # installed explicitly will also be installed as a
-                # dependency of another spec. In this case append to both
-                # spec reports.
-                for s in llnl.util.lang.dedupe([pkg.spec.root, pkg.spec]):
-                    name = name_fmt.format(s.name, s.dag_hash(length=7))
-                    try:
-                        item = next((
-                            x for x in self.specs
-                            if x['name'] == name
-                        ))
-                        item['packages'].append(package)
-                    except StopIteration:
-                        pass
-
                 return value
 
             return wrapper


Test patch:
diff --git a/lib/spack/spack/test/cmd/install.py b/lib/spack/spack/test/cmd/install.py
--- a/lib/spack/spack/test/cmd/install.py
+++ b/lib/spack/spack/test/cmd/install.py
@@ -393,9 +393,14 @@ def test_junit_output_with_failures(tmpdir, exc_typename, msg):
             '--log-format=junit', '--log-file=test.xml',
             'raiser',
             'exc_type={0}'.format(exc_typename),
-            'msg="{0}"'.format(msg)
+            'msg="{0}"'.format(msg),
+            fail_on_error=False,
         )
 
+    assert isinstance(install.error, spack.build_environment.ChildError)
+    assert install.error.name == exc_typename
+    assert install.error.pkg.name == 'raiser'
+
     files = tmpdir.listdir()
     filename = tmpdir.join('test.xml')
     assert filename in files
@@ -407,18 +412,22 @@ def test_junit_output_with_failures(tmpdir, exc_typename, msg):
     assert 'failures="1"' in content
     assert 'errors="0"' in content
 
+    # Nothing should have succeeded
+    assert 'tests="0"' not in content
+    assert 'failures="0"' not in content
+
     # We want to have both stdout and stderr
     assert '<system-out>' in content
     assert msg in content
 
 
 @pytest.mark.disable_clean_stage_check
-@pytest.mark.parametrize('exc_typename,msg', [
-    ('RuntimeError', 'something weird happened'),
-    ('KeyboardInterrupt', 'Ctrl-C strikes again')
+@pytest.mark.parametrize('exc_typename,expected_exc,msg', [
+    ('RuntimeError', spack.installer.InstallError, 'something weird happened'),
+    ('KeyboardInterrupt', KeyboardInterrupt, 'Ctrl-C strikes again')
 ])
 def test_junit_output_with_errors(
-        exc_typename, msg,
+        exc_typename, expected_exc, msg,
         mock_packages, mock_archive, mock_fetch, install_mockery,
         config, tmpdir, monkeypatch):
 
@@ -429,11 +438,11 @@ def just_throw(*args, **kwargs):
     monkeypatch.setattr(spack.installer.PackageInstaller, '_install_task',
                         just_throw)
 
-    # TODO: Why does junit output capture appear to swallow the exception
-    # TODO: as evidenced by the two failing packages getting tagged as
-    # TODO: installed?
     with tmpdir.as_cwd():
-        install('--log-format=junit', '--log-file=test.xml', 'libdwarf')
+        install('--log-format=junit', '--log-file=test.xml', 'libdwarf',
+                fail_on_error=False)
+
+    assert isinstance(install.error, expected_exc)
 
     files = tmpdir.listdir()
     filename = tmpdir.join('test.xml')
@@ -441,10 +450,14 @@ def just_throw(*args, **kwargs):
 
     content = filename.open().read()
 
-    # Count failures and errors correctly: libdwarf _and_ libelf
-    assert 'tests="2"' in content
+    # Only libelf error is reported (through libdwarf root spec). libdwarf
+    # install is skipped and it is not an error.
+    assert 'tests="1"' in content
     assert 'failures="0"' in content
-    assert 'errors="2"' in content
+    assert 'errors="1"' in content
+
+    # Nothing should have succeeded
+    assert 'errors="0"' not in content
 
     # We want to have both stdout and stderr
     assert '<system-out>' in content
@@ -877,7 +890,7 @@ def test_install_help_cdash(capsys):
 
 
 @pytest.mark.disable_clean_stage_check
-def test_cdash_auth_token(tmpdir, install_mockery, capfd):
+def test_cdash_auth_token(tmpdir, mock_fetch, install_mockery, capfd):
     # capfd interferes with Spack's capturing
     with tmpdir.as_cwd():
         with capfd.disabled():
diff --git a/lib/spack/spack/test/cmd/test.py b/lib/spack/spack/test/cmd/test.py
--- a/lib/spack/spack/test/cmd/test.py
+++ b/lib/spack/spack/test/cmd/test.py
@@ -133,7 +133,8 @@ def test_junit_output_with_failures(tmpdir, mock_test_stage, pkg_name, msgs):
     with tmpdir.as_cwd():
         spack_test('run',
                    '--log-format=junit', '--log-file=test.xml',
-                   pkg_name)
+                   pkg_name,
+                   fail_on_error=False)
 
     files = tmpdir.listdir()
     filename = tmpdir.join('test.xml')
@@ -160,7 +161,8 @@ def test_cdash_output_test_error(
         spack_test('run',
                    '--log-format=cdash',
                    '--log-file=cdash_reports',
-                   'test-error')
+                   'test-error',
+                   fail_on_error=False)
         report_dir = tmpdir.join('cdash_reports')
         print(tmpdir.listdir())
         assert report_dir in tmpdir.listdir()