Repository: spack/spack

Issue:
Various issues with python packages
I am running into a few issues. This all stems from my attempt to deprecate the py-pil package, which is the Python Imaging Library, and instead create a py-pillow package, with pillow being a fork/replacement of PIL. I also updated packages that depended on PIL and this led to errors building mesa because it didn't properly depend on py-libxml2. I noticed, however, that the libxml2 package (which py-libxml2 depends on) can build the python bindings, thus deprecating py-libxml2. libxml2+python appears to build OK on it's own, but I have found 2 issues:

1) libxml2+python will install the python bindings directly into the python installation rather than the libxml2 prefix. Furthermore, activating libxml2+python will link all the libxml2 files, even those not necessary for python. The latter can probably be fixed easily with some ignore directives.

2) when installing the py-basemap package I get the following error:

==> Building libxml2.
Traceback (most recent call last):
  File "/nfs/tmp2/lee218/spack/lib/spack/spack/build_environment.py", line 278, in fork
    function()
  File "/nfs/tmp2/lee218/spack/lib/spack/spack/package.py", line 812, in real_work
    self.install(self.spec, self.prefix)
  File "/nfs/tmp2/lee218/spack/var/spack/packages/libxml2/package.py", line 20, in install
    python_arg = "--with-python=%s" % spec['python'].prefix
  File "/nfs/tmp2/lee218/spack/lib/spack/spack/spec.py", line 1419, in **getitem**
    raise KeyError("No spec with name %s in %s" % (name, self))
KeyError: 'No spec with name python in libxml2@2.9.2%gcc@4.4.7+python=chaos_5_x86_64_ib^xz@5.2.2%gcc@4.4.7=chaos_5_x86_64_ib^zlib@1.2.8%gcc@4.4.7=chaos_5_x86_64_ib'
==> Error: Installation process had nonzero exit code.

The dependence chain to get to the error above is fairly convoluted. It looks like I can install libxml2+python directly, I can install a package (i.e. mesa) that depends_on('libxml2+python'), and I can even install a package (i.e., py-pandas) that depends on mesa, but py-matplotlib+gui fails with the above error.



Discussion:
I will take a first crack at some of these issues.

Hi, I was trying to build py-pandas on my systems, I get the following stack trace back when trying to activate it.

==> No patches needed for py-pandas.
==> Building py-pandas.
==> Successfully installed py-pandas.
Fetch: 0.28s. Build: 2m 36.64s. Total: 2m 36.92s.
[+] /gpfs/apps/haswell-EL7/gcc-4.8.3/py-pandas-0.16.1-xixmqcjmrdjynky4akccphmj6wbyi2oj
[~ 10:54] $ spack activate py-pandas ^python@3.5.0 ^openssl@1.0.2d
Traceback (most recent call last):
File "/gpfs/sysapps/spack/bin/spack", line 163, in <module>
main()
File "/gpfs/sysapps/spack/bin/spack", line 144, in main
return_val = command(parser, args)
File "/gpfs/sysapps/spack/lib/spack/spack/cmd/activate.py", line 52, in activate
spec.package.do_activate()
File "/gpfs/sysapps/spack/lib/spack/spack/package.py", line 978, in do_activate
if not spec.package.activated:
File "/gpfs/sysapps/spack/lib/spack/spack/package.py", line 518, in activated
exts = spack.install_layout.extension_map(self.extendee_spec)
File "/gpfs/sysapps/spack/lib/spack/spack/directory_layout.py", line 343, in extension_map
_check_concrete(spec)
File "/gpfs/sysapps/spack/lib/spack/spack/directory_layout.py", line 44, in _check_concrete
raise ValueError('Specs passed to a DirectoryLayout must be concrete!')
ValueError: Specs passed to a DirectoryLayout must be concrete!

I haven't found the root cause of that issue, but I believe if you revert var/spack/packages/libxml2/package.py to the previous version, make it extend Python without the variant, then it should work around the problem. Let me know if that does the trick for you. 

@sjtbham were you able to work around this? I also committed a change to the mesa package in the develop branch that makes it depend on libxml2+python, which may help.

@tgamblin I am trying to find the best way to fix this issue. I think the underlying problem is that the extends clause isn't working properly with variants. In particular, if the requested variant does not actually extend another package, the other package is being recorded as an extendee. Should we maintain extends conditions in a similar manner as with dependencies?

@lee218llnl: let me check this out.  I think i can get to it tonight or tomorrow -- extends should work similarly to dependencies so this may have been an oversight in the implementation.  Or it's a function of the fact that variants were added _after_ Python merge support.

I committed some changes to the libxml2 package. The changes point libxml2 to install the python module into the libxml2 prefix instead of directly installing to the python prefix. I also have the package ignore all the files that aren't related to the python module when activating.


Resolving patch:
diff --git a/lib/spack/spack/packages.py b/lib/spack/spack/packages.py
--- a/lib/spack/spack/packages.py
+++ b/lib/spack/spack/packages.py
@@ -67,27 +67,28 @@ def get(self, spec, **kwargs):
         if spec.virtual:
             raise UnknownPackageError(spec.name)
 
+        key = hash(spec)
         if kwargs.get('new', False):
-            if spec in self.instances:
-                del self.instances[spec]
+            if key in self.instances:
+                del self.instances[key]
 
-        if not spec in self.instances:
+        if not key in self.instances:
             package_class = self.get_class_for_package_name(spec.name)
             try:
-                copy = spec.copy()
-                self.instances[copy] = package_class(copy)
+                copy = spec.copy() # defensive copy.  Package owns its spec.
+                self.instances[key] = package_class(copy)
             except Exception, e:
                 if spack.debug:
                     sys.excepthook(*sys.exc_info())
                 raise FailedConstructorError(spec.name, e)
 
-        return self.instances[spec]
+        return self.instances[key]
 
 
     @_autospec
     def delete(self, spec):
         """Force a package to be recreated."""
-        del self.instances[spec]
+        del self.instances[spec.dag_hash()]
 
 
     def purge(self):
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1481,8 +1481,11 @@ def ne_dag(self, other):
 
     def _cmp_node(self):
         """Comparison key for just *this node* and not its deps."""
-        return (self.name, self.versions, self.variants,
-                self.architecture, self.compiler)
+        return (self.name,
+                self.versions,
+                self.variants,
+                self.architecture,
+                self.compiler)
 
 
     def eq_node(self, other):
@@ -1496,11 +1499,15 @@ def ne_node(self, other):
 
 
     def _cmp_key(self):
-        """Comparison key for this node and all dependencies *without*
-           considering structure.  This is the default, as
-           normalization will restore structure.
+        """This returns a key for the spec *including* DAG structure.
+
+        The key is the concatenation of:
+          1. A tuple describing this node in the DAG.
+          2. The hash of each of this node's dependencies' cmp_keys.
         """
-        return self._cmp_node() + (self.sorted_deps(),)
+        return self._cmp_node() + (
+            tuple(hash(self.dependencies[name])
+                  for name in sorted(self.dependencies)),)
 
 
     def colorized(self):


Test patch:
diff --git a/lib/spack/spack/test/spec_dag.py b/lib/spack/spack/test/spec_dag.py
--- a/lib/spack/spack/test/spec_dag.py
+++ b/lib/spack/spack/test/spec_dag.py
@@ -340,16 +340,18 @@ def test_normalize_mpileaks(self):
         self.assertEqual(spec, expected_flat)
         self.assertTrue(spec.eq_dag(expected_flat))
 
-        self.assertEqual(spec, expected_normalized)
+        # Normalized has different DAG structure, so NOT equal.
+        self.assertNotEqual(spec, expected_normalized)
         self.assertFalse(spec.eq_dag(expected_normalized))
 
-        self.assertEqual(spec, non_unique_nodes)
+        # Again, different DAG structure so not equal.
+        self.assertNotEqual(spec, non_unique_nodes)
         self.assertFalse(spec.eq_dag(non_unique_nodes))
 
         spec.normalize()
 
         # After normalizing, spec_dag_equal should match the normalized spec.
-        self.assertEqual(spec, expected_flat)
+        self.assertNotEqual(spec, expected_flat)
         self.assertFalse(spec.eq_dag(expected_flat))
 
         self.assertEqual(spec, expected_normalized)