Repository: spack/spack

Issue:
Question: How to tell spack to use bzip2 and zlib from system?
With the current `develop` branch (5320268033570d93d40b431fc4d93893572bac0b) I get:

```
$ spack spec fairlogger@1.5.0+pretty
Input spec
--------------------------------
fairlogger@1.5.0+pretty

Concretized
--------------------------------
fairlogger@1.5.0%gcc@9.2.1 build_type=RelWithDebInfo cxxstd=default +pretty arch=linux-fedora31-skylake
    ^boost@1.72.0%gcc@9.2.1+atomic+chrono~clanglibcpp~container~context~coroutine cxxstd=98 +date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout visibility=hidden +wave arch=linux-fedora31-skylake
        ^bzip2@1.0.8%gcc@9.2.1+shared arch=linux-fedora31-skylake
            ^diffutils@3.7%gcc@9.2.1 arch=linux-fedora31-skylake
                ^libiconv@1.16%gcc@9.2.1 arch=linux-fedora31-skylake
        ^zlib@1.2.11%gcc@9.2.1+optimize+pic+shared arch=linux-fedora31-skylake
    ^cmake@3.17.0%gcc@9.2.1~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora31-skylake
        ^ncurses@6.2%gcc@9.2.1~symlinks+termlib arch=linux-fedora31-skylake
            ^pkgconf@1.6.3%gcc@9.2.1 arch=linux-fedora31-skylake
        ^openssl@1.1.1f%gcc@9.2.1+systemcerts arch=linux-fedora31-skylake
            ^perl@5.30.1%gcc@9.2.1+cpanm+shared+threads arch=linux-fedora31-skylake
                ^gdbm@1.18.1%gcc@9.2.1 arch=linux-fedora31-skylake
                    ^readline@8.0%gcc@9.2.1 arch=linux-fedora31-skylake
```

Now marking several packages to be picked up from the system
```
$ cat ~/.spack/packages.yaml
packages:
  bzip2:
    paths:
      bzip2 os=fedora31: /usr
  cmake:
    paths:
      cmake os=fedora31: /usr
  zlib:
    paths:
      zlib os=fedora31: /usr
```
results in
```
$ spack spec fairlogger@1.5.0+pretty
Input spec
--------------------------------
fairlogger@1.5.0+pretty

Concretized
--------------------------------
fairlogger@1.5.0%gcc@9.2.1 build_type=RelWithDebInfo cxxstd=default +pretty arch=linux-fedora31-skylake
    ^boost@1.72.0%gcc@9.2.1+atomic+chrono~clanglibcpp~container~context~coroutine cxxstd=98 +date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout visibility=hidden +wave arch=linux-fedora31-skylake
        ^bzip2@1.0.8%gcc@9.2.1+shared arch=linux-fedora31-skylake
            ^diffutils@3.7%gcc@9.2.1 arch=linux-fedora31-skylake
                ^libiconv@1.16%gcc@9.2.1 arch=linux-fedora31-skylake
        ^zlib@1.2.11%gcc@9.2.1+optimize+pic+shared arch=linux-fedora31-skylake
    ^cmake@3.17.0%gcc@9.2.1~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora31-skylake
```

Somehow this only works for `cmake`. However, tweaking the `packages.yaml` to be explicit about the target, achieves my goal:

```
$ cat ~/.spack/packages.yaml
packages:
  bzip2:
    paths:
      bzip2 os=fedora31 target=skylake: /usr
  cmake:
    paths:
      cmake os=fedora31: /usr
  zlib:
    paths:
      zlib os=fedora31 target=skylake: /usr
```
```
$ spack spec fairlogger@1.5.0+pretty
Input spec
--------------------------------
fairlogger@1.5.0+pretty

Concretized
--------------------------------
fairlogger@1.5.0%gcc@9.2.1 build_type=RelWithDebInfo cxxstd=default +pretty arch=linux-fedora31-skylake
    ^boost@1.72.0%gcc@9.2.1+atomic+chrono~clanglibcpp~container~context~coroutine cxxstd=98 +date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout visibility=hidden +wave arch=linux-fedora31-skylake
        ^bzip2@1.0.8%gcc@9.2.1+shared arch=linux-fedora31-skylake
        ^zlib@1.2.11%gcc@9.2.1+optimize+pic+shared arch=linux-fedora31-skylake
    ^cmake@3.17.0%gcc@9.2.1~doc+ncurses+openssl+ownlibs~qt arch=linux-fedora31-skylake
```
```
$ spack install fairlogger@1.5.0+pretty
==> bzip2@1.0.8 : externally installed in /usr
==> bzip2@1.0.8 : generating module file
==> bzip2@1.0.8 : registering into DB
[+] /usr
==> zlib@1.2.11 : externally installed in /usr
==> zlib@1.2.11 : generating module file
==> zlib@1.2.11 : registering into DB
[+] /usr
==> cmake@3.17.0 : externally installed in /usr
==> cmake@3.17.0 : generating module file
==> cmake@3.17.0 : registering into DB
[+] /usr
==> 2093636: Installing boost
(...)
```

*  Why do I need to be explicit about the target for `bzip2` and `zlib` and not for `cmake` (dependency type?)?
*  If this is not a bug, how can I mark `bzip2` and `zlib` external without being explicit about the target?


Discussion:
Not specifying any of the arch fields also works:

```
$ cat ~/.spack/packages.yaml
packages:
  bzip2:
    paths:
      bzip2: /usr
  cmake:
    paths:
      cmake: /usr
  zlib:
    paths:
      zlib: /usr
```

Unfortunately, I eventually have to provide different paths depending on the os field.

Resolving patch:
diff --git a/lib/spack/spack/abi.py b/lib/spack/spack/abi.py
--- a/lib/spack/spack/abi.py
+++ b/lib/spack/spack/abi.py
@@ -18,10 +18,13 @@ class ABI(object):
     """This class provides methods to test ABI compatibility between specs.
        The current implementation is rather rough and could be improved."""
 
-    def architecture_compatible(self, parent, child):
-        """Return true if parent and child have ABI compatible targets."""
-        return not parent.architecture or not child.architecture or \
-            parent.architecture == child.architecture
+    def architecture_compatible(self, target, constraint):
+        """Return true if architecture of target spec is ABI compatible
+           to the architecture of constraint spec. If either the target
+           or constraint specs have no architecture, target is also defined
+           as architecture ABI compatible to constraint."""
+        return not target.architecture or not constraint.architecture or \
+            target.architecture.satisfies(constraint.architecture)
 
     @memoized
     def _gcc_get_libstdcxx_version(self, version):
@@ -107,8 +110,8 @@ def compiler_compatible(self, parent, child, **kwargs):
                     return True
         return False
 
-    def compatible(self, parent, child, **kwargs):
-        """Returns true iff a parent and child spec are ABI compatible"""
+    def compatible(self, target, constraint, **kwargs):
+        """Returns true if target spec is ABI compatible to constraint spec"""
         loosematch = kwargs.get('loose', False)
-        return self.architecture_compatible(parent, child) and \
-            self.compiler_compatible(parent, child, loose=loosematch)
+        return self.architecture_compatible(target, constraint) and \
+            self.compiler_compatible(target, constraint, loose=loosematch)


Test patch:
diff --git a/lib/spack/spack/test/abi.py b/lib/spack/spack/test/abi.py
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/abi.py
@@ -0,0 +1,66 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+""" Test ABI compatibility helpers"""
+
+import pytest
+
+from spack.abi import ABI
+from spack.spec import Spec
+
+
+@pytest.mark.parametrize(
+    'target,constraint,expected',
+    [
+        ('foo', 'bar', True),
+        ('platform=linux', 'foo', True),
+        ('foo', 'arch=linux-fedora31-x86_64', True),
+        ('arch=linux-fedora31-skylake', 'arch=linux-fedora31-skylake', True),
+        ('arch=linux-fedora31-skylake', 'arch=linux-fedora31-x86_64', False),
+        ('platform=linux os=fedora31', 'arch=linux-fedora31-x86_64', True),
+        ('platform=linux', 'arch=linux-fedora31-x86_64', True),
+        ('platform=linux os=fedora31', 'platform=linux', True),
+        ('platform=darwin', 'arch=linux-fedora31-x86_64', False),
+        ('os=fedora31', 'platform=linux', False),  # TODO should be true ?
+    ])
+def test_architecture_compatibility(target, constraint, expected):
+    assert ABI().architecture_compatible(Spec(target),
+                                         Spec(constraint)) == expected
+
+
+@pytest.mark.parametrize(
+    'target,constraint,loose,expected',
+    [
+        ('foo', 'bar', False, True),
+        ('%gcc', 'foo', False, True),
+        ('foo', '%gcc', False, True),
+        ('%gcc', '%gcc', False, True),
+        ('%gcc', '%intel', False, False),
+        ('%gcc', '%clang', False, False),
+        ('%gcc@9.1', '%gcc@9.2', False, False),  # TODO should be true ?
+        ('%gcc@9.2.1', '%gcc@9.2.2', False, False),  # TODO should be true ?
+        ('%gcc@4.9', '%gcc@9.2', False, False),
+        ('%clang@5', '%clang@6', False, False),
+        ('%gcc@9.1', '%gcc@9.2', True, True),
+        ('%gcc@9.2.1', '%gcc@9.2.2', True, True),
+        ('%gcc@4.9', '%gcc@9.2', True, True),
+        ('%clang@5', '%clang@6', True, True),
+    ])
+def test_compiler_compatibility(target, constraint, loose, expected):
+    assert ABI().compiler_compatible(Spec(target),
+                                     Spec(constraint),
+                                     loose=loose) == expected
+
+
+@pytest.mark.parametrize('target,constraint,loose,expected', [
+    ('foo', 'bar', False, True),
+    ('%gcc', 'platform=linux', False, True),
+    ('%gcc@9.2.1', '%gcc@8.3.1 platform=linux', False, False),
+    ('%gcc@9.2.1', '%gcc@8.3.1 platform=linux', True, True),
+    ('%gcc@9.2.1 arch=linux-fedora31-skylake', '%gcc@9.2.1 platform=linux',
+     False, True),
+])
+def test_compatibility(target, constraint, loose, expected):
+    assert ABI().compatible(Spec(target), Spec(constraint),
+                            loose=loose) == expected