Repository: spack/spack

Issue:
Zero exit code if dependency fails to build
I'm trying to build a package A with many dependencies B, C, D, etc. I noticed that if B fails, I see:
```
==> Warning: skipping build of A since B failed
```
then Spack continues to install C and D. This is fine, but the exit code of this process is 0 even though the package I want to build (A) did not build. I would also expect an error message at the end of this informing me that the package I requested could not be built.

@tldahlgren I believe this may be related to the parallel build stuff you added a while back, maybe you can take a look. I've seen reports of this bug on Slack, but I don't think anyone has opened an issue on GitHub yet.

### Information on your system

* **Spack:** 0.16.0-1537-29201814cc
* **Python:** 3.8.7
* **Platform:** darwin-catalina-ivybridge
* **Concretizer:** clingo

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output


Discussion:
https://github.com/spack/spack/issues/21319
https://github.com/spack/spack/issues/20510

Yeah, this needs to be fixed...
Thanks! I knew I had seen this bug before. Closing as a duplicate. 
how can we raise the priority of that issue? it's reported many times now
Be more annoying 😁 
Duplicate of #21319
> @tldahlgren I believe this may be related to the parallel build stuff you added a while back, maybe you can take a look. I've seen reports of this bug on Slack, but I don't think anyone has opened an issue on GitHub yet.
> 

Yes, it is related to the parallel build support changes.

There is (still) code at the bottom of build queue processing that is *supposed* to be raising an `InstallError` with the following message when a spec that is provided on the command line fails to install:

```
Installation request failed.  Refer to recent errors for specific package(s).
```

The package and associated exception should also be being logged as an ERROR.

I'm wondering if some of the follow-on work for parallelizing environments, perhaps, masks the error or it's an issue with how the particular failure is detected.
> > @tldahlgren I believe this may be related to the parallel build stuff you added a while back, maybe you can take a look. I've seen reports of this bug on Slack, but I don't think anyone has opened an issue on GitHub yet.
> 
> Yes, it is related to the parallel build support changes.
> 
> There is (still) code at the bottom of build queue processing that is _supposed_ to be raising an `InstallError` with the following message when a spec that is provided on the command line fails to install:
> 
> ```
> Installation request failed.  Refer to recent errors for specific package(s).
> ```
> 
> The package and associated exception should also be being logged as an ERROR.
> 
> I'm wondering if some of the follow-on work for parallelizing environments, perhaps, masks the error or it's an issue with how the particular failure is detected.

Ah.  Just realized.  The issue here isn't that requested spec failed to install due to an exception; rather, it is skipped because its dependency(ies) failed to install!  


Resolving patch:
diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py
--- a/lib/spack/spack/installer.py
+++ b/lib/spack/spack/installer.py
@@ -1609,15 +1609,22 @@ def install(self):
         self._cleanup_all_tasks()
 
         # Ensure we properly report if one or more explicit specs failed
-        if exists_errors or failed_explicits:
+        # or were not installed when should have been.
+        missing = [request.pkg_id for request in self.build_requests if
+                   request.install_args.get('install_package') and
+                   request.pkg_id not in self.installed]
+        if exists_errors or failed_explicits or missing:
             for pkg_id, err in exists_errors:
                 tty.error('{0}: {1}'.format(pkg_id, err))
 
             for pkg_id, err in failed_explicits:
                 tty.error('{0}: {1}'.format(pkg_id, err))
 
+            for pkg_id in missing:
+                tty.error('{0}: Package was not installed'.format(pkg_id))
+
             raise InstallError('Installation request failed.  Refer to '
-                               'recent errors for specific package(s).')
+                               'reported errors for failing package(s).')
 
 
 def build_process(pkg, kwargs):


Test patch:
diff --git a/lib/spack/spack/test/cmd/dev_build.py b/lib/spack/spack/test/cmd/dev_build.py
--- a/lib/spack/spack/test/cmd/dev_build.py
+++ b/lib/spack/spack/test/cmd/dev_build.py
@@ -89,7 +89,7 @@ def test_dev_build_until_last_phase(tmpdir, mock_packages, install_mockery):
     assert os.path.exists(str(tmpdir))
 
 
-def test_dev_build_before_until(tmpdir, mock_packages, install_mockery):
+def test_dev_build_before_until(tmpdir, mock_packages, install_mockery, capsys):
     spec = spack.spec.Spec('dev-build-test-install@0.0.0 dev_path=%s' % tmpdir)
     spec.concretize()
 
@@ -103,13 +103,18 @@ def test_dev_build_before_until(tmpdir, mock_packages, install_mockery):
 
         bad_phase = 'phase_that_does_not_exist'
         not_allowed = 'is not a valid phase'
-        out = dev_build('-u', bad_phase, 'dev-build-test-install@0.0.0')
+        not_installed = 'was not installed'
+        out = dev_build('-u', bad_phase, 'dev-build-test-install@0.0.0',
+                        fail_on_error=False)
         assert bad_phase in out
         assert not_allowed in out
+        assert not_installed in out
 
-        out = dev_build('-b', bad_phase, 'dev-build-test-install@0.0.0')
+        out = dev_build('-b', bad_phase, 'dev-build-test-install@0.0.0',
+                        fail_on_error=False)
         assert bad_phase in out
         assert not_allowed in out
+        assert not_installed in out
 
 
 def print_spack_cc(*args):
diff --git a/lib/spack/spack/test/cmd/install.py b/lib/spack/spack/test/cmd/install.py
--- a/lib/spack/spack/test/cmd/install.py
+++ b/lib/spack/spack/test/cmd/install.py
@@ -629,10 +629,11 @@ def test_cache_only_fails(tmpdir, mock_fetch, install_mockery, capfd):
     # libelf from cache fails to install, which automatically removes the
     # the libdwarf build task
     with capfd.disabled():
-        out = install('--cache-only', 'libdwarf')
+        out = install('--cache-only', 'libdwarf', fail_on_error=False)
 
     assert 'Failed to install libelf' in out
     assert 'Skipping build of libdwarf' in out
+    assert 'was not installed' in out
 
     # Check that failure prefix locks are still cached
     failure_lock_prefixes = ','.join(spack.store.db._prefix_failures.keys())
diff --git a/lib/spack/spack/test/installer.py b/lib/spack/spack/test/installer.py
--- a/lib/spack/spack/test/installer.py
+++ b/lib/spack/spack/test/installer.py
@@ -879,7 +879,8 @@ def test_install_failed(install_mockery, monkeypatch, capsys):
     # Make sure the package is identified as failed
     monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
 
-    installer.install()
+    with pytest.raises(inst.InstallError, match='request failed'):
+        installer.install()
 
     out = str(capsys.readouterr())
     assert installer.build_requests[0].pkg_id in out
@@ -894,7 +895,8 @@ def test_install_failed_not_fast(install_mockery, monkeypatch, capsys):
     # Make sure the package is identified as failed
     monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
 
-    installer.install()
+    with pytest.raises(inst.InstallError, match='request failed'):
+        installer.install()
 
     out = str(capsys.readouterr())
     assert 'failed to install' in out
@@ -1046,7 +1048,9 @@ def _requeued(installer, task):
     # Ensure don't continually requeue the task
     monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
 
-    installer.install()
+    with pytest.raises(inst.InstallError, match='request failed'):
+        installer.install()
+
     out = capfd.readouterr()[0]
     expected = ['write locked', 'read locked', 'requeued']
     for exp, ln in zip(expected, out.split('\n')):
@@ -1077,7 +1081,9 @@ def _requeued(installer, task):
     # Ensure don't continually requeue the task
     monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
 
-    installer.install()
+    with pytest.raises(inst.InstallError, match='request failed'):
+        installer.install()
+
     assert b_pkg_id not in installer.installed
 
     out = capfd.readouterr()[0]
@@ -1113,7 +1119,9 @@ def _requeued(installer, task):
     const_arg = installer_args(['b'], {})
     installer = create_installer(const_arg)
 
-    installer.install()
+    with pytest.raises(inst.InstallError, match='request failed'):
+        installer.install()
+
     assert 'b' not in installer.installed
 
     out = capfd.readouterr()[0]