Repository: spack/spack

Issue:
installed package jpeg not found
Hi, I am not sure if this happens to be related to #5281 #5190 but R packages that require jpeg no longer work:

```
[root@node141 spack]# git log | head
commit feefdedadf224315d69ef2f6da1fafd6338bba5a
Author: Christoph Junghans <christoph.junghans@gmail.com>
Date:   Tue Sep 5 13:15:08 2017 -0600

    nauty: shared libraries (#5207)

commit 41d8981ab5b5e912d38e201b71b981ef0f17e20e
Author: Massimiliano Culpo <massimiliano.culpo@googlemail.com>
Date:   Tue Sep 5 17:15:25 2017 +0200
```

```
[root@node141 spack]# spack load -r r-zoo
==> Error: Package jpeg not found.
```

```
[root@node141 spack]# spack find | grep jpeg
libjpeg-turbo@1.5.0
jpeg@9b
libjpeg-turbo@1.5.0
r-jpeg@0.1-8
```
Concreteization works though I see:

```
[root@node141 spack]# spack -d load -r r-zoo                                                                                                                                                                                                  
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> Error: Package jpeg not found.
Traceback (most recent call last):
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/main.py", line 440, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/main.py", line 356, in _invoke_spack_command
    return_val = command(parser, args)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/cmd/module.py", line 267, in module
    callbacks[args.subparser_name](module_type, specs, args)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/cmd/module.py", line 134, in loads
    for spec in specs if os.path.exists(module_cls(spec).file_name)]
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/modules.py", line 265, in __init__
    self.pkg = spec.package  # Just stored for convenience
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/spec.py", line 1291, in package
    return spack.repo.get(self)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 74, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 317, in get
    return self.repo_for_pkg(spec).get(spec)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 74, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 567, in get
    raise UnknownPackageError(spec.name)
spack.repository.UnknownPackageError: Package jpeg not found.
```

This still works on my testing branch which is from way back in June.


```
[root@node141 spack]# git branch
  develop
  master
* testing
[root@node141 spack]# git log | head

commit 7d59197b7b056ce4525b3faccdcaa7f06519c886
Author: Douglas Duckworth <quackmaster@protonmail.com>
Date:   Thu Jun 29 14:42:56 2017 -0400

    added numpy 1.13 - modified albacore to use 1.13 (#4639)
    
    * albacore 1.24 requires numpy 1.13 - added numpy 1.13
    
    * added numpy 1.13
```





Discussion:
On the develop branch:

```
[root@node141 spack]# spack find -p -l -v jpeg
==> 3 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.6 -------------------------------
rue73bc    libjpeg-turbo@1.5.0  /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.6/libjpeg-turbo-1.5.0-rue73bcgjpn4m4ctt4wicsqxc7tgm2zj

-- linux-rhel6-x86_64 / gcc@6.3.0 -------------------------------
xm3nnrp    jpeg@9b              /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-6.3.0/jpeg-9b-xm3nnrpn2eclrcfzsfhkacyt6etnzi4w
56ibsvd    libjpeg-turbo@1.5.0  /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-6.3.0/libjpeg-turbo-1.5.0-56ibsvdkdzl55qmgordmlvsgvjvlqwx7
```


```
[root@node141 spack]# ls -ltr /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-6.3.0/libjpeg-turbo-1.5.0-56ibsvdkdzl55qmgordmlvsgvjvlqwx7
total 16
drwxr-xr-x 4 root root 4096 Sep  5 12:34 share
drwxr-xr-x 2 root root 4096 Sep  5 12:34 include
drwxr-xr-x 3 root root 4096 Sep  5 12:34 lib
drwxr-xr-x 2 root root 4096 Sep  5 12:34 bin

[root@node141 spack]# ls -ltr /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/linux-rhel6-x86_64/gcc-6.3.0/jpeg-9b-xm3nnrpn2eclrcfzsfhkacyt6etnzi4w
total 16
drwxr-xr-x 3 root root 4096 Apr 21 15:35 share
drwxr-xr-x 2 root root 4096 Apr 21 15:35 include
drwxr-xr-x 2 root root 4096 Apr 21 15:35 lib
drwxr-xr-x 2 root root 4096 Apr 21 15:35 bin

```



On the testing branch:

```
[root@node141 spack]# spack find jpeg                                                                                                                                                                                                  
==> 1 installed packages.
-- linux-rhel6-x86_64 / gcc@6.3.0 -------------------------------
jpeg@9b

```
So it seems when I installed a few things like `libjpeg-turbo` on the develop branch, that caused problems?  I don't understand why though since R depends upon `jpeg` and not `libjpeg-turbo?`

OK

I switched to the develop branch and removed all of the programs which were installed within that branch including `libjpeg-turbo` however

```
[root@node141 spack]# spack find -l -v jpeg
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.6 -------------------------------
rue73bc libjpeg-turbo@1.5.0

-- linux-rhel6-x86_64 / gcc@6.3.0 -------------------------------
xm3nnrp jpeg@9b
[root@node141 spack]# spack load -r r-zoo r-yaml
==> Error: Package jpeg not found.
```
So `jpeg` seems to be present but not found?  Do I need to `reindex?`

`reindex` did not work.
#5281 points to https://github.com/LLNL/spack/pull/5190 which made jpeg a virtual dependency, so that could have affected packages built with jpeg. Deleting everything and doing a reindex ought to have worked at least (to be sure - you deleted everything under `spack/opt`?). ~Also I'm not sure why~ 

```
spack load -r r-zoo
```

~would have worked since it isn't present in `spack find`. Are those packages installed in `spack/opt`?~

(EDIT: just noticed all instances of `spack find` in your examples were constrained to `jpeg`)
Hi @scheibelp!

So I need to reinstall everything that depends upon `jpeg`?  That's **many** R packages as well as a few important Python ones.  

There must be a better way to address this issue.
@JusticeForMikeBrown FYI I think #5281 is a different issue than this (although both stem from #5190) - `spack spec r-zoo` runs fine.

I think the issue in this case is that Spack tries to find jpeg in the package repository but only searches the central repository. #5297 updates this to also check the repositories associated with the installation prefix. I'm curious if that helps you. I'd like to get it in working condition by the end of 9/8.
Hi @scheibelp 

Yes, `spec` works fine for any packages that depend upon `jpeg` though `load` fails entirely while the error shown in `spack -d load blah` does mention failure with `repository.py`.  If #5297 fixes the issue then I'll try it once approved.  What do you think @tgamblin?
As a note: I ran into a similar issue (besides others in #5228). At some point spack tried to find a package ```jpeg``` even though it was a virtual dependency. Cleaning out all caches helped for me.
@JusticeForMikeBrown did you try ```spack clean --all```?
hi @healther 

Sadly that does not work:
```

[root@node141 spack]# git checkout develop
Checking out files: 100% (748/748), done.
Switched to branch 'develop'
                                                                                                                                              
[root@node141 spack]# spack clean --all
==> Removing all temporary build stages
==> Removing cached downloads
==> Removing cached information on repositories
[root@node141 spack]# spack load -r r-zoo r-yaml py-matplotlib py-pillow
==> Error: Package jpeg not found.

[root@node141 spack]# spack -d load -r r-zoo r-yaml py-matplotlib py-pillow                                                                                                                                                                   
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /pbtech_mounts/softlib001/apps/EL6/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> Error: Package jpeg not found.
Traceback (most recent call last):
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/main.py", line 440, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/main.py", line 356, in _invoke_spack_command
    return_val = command(parser, args)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/cmd/module.py", line 267, in module
    callbacks[args.subparser_name](module_type, specs, args)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/cmd/module.py", line 134, in loads
    for spec in specs if os.path.exists(module_cls(spec).file_name)]
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/modules.py", line 265, in __init__
    self.pkg = spec.package  # Just stored for convenience
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/spec.py", line 1291, in package
    return spack.repo.get(self)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 74, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 317, in get
    return self.repo_for_pkg(spec).get(spec)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 74, in converter
    return function(self, spec_like, *args, **kwargs)
  File "/pbtech_mounts/softlib001/apps/EL6/spack/lib/spack/spack/repository.py", line 567, in get
    raise UnknownPackageError(spec.name)
spack.repository.UnknownPackageError: Package jpeg not found.
```

Again, `spec` still works:

```
Concretized
--------------------------------
r-zoo@1.7-14%gcc@4.4.6 arch=linux-rhel6-x86_64 
    ^r@3.4.1%gcc@4.4.6~X~external-lapack arch=linux-rhel6-x86_64 
        ^bzip2@1.0.6%gcc@4.4.6+shared arch=linux-rhel6-x86_64 
        ^cairo@1.14.8%gcc@4.4.6~X arch=linux-rhel6-x86_64 
            ^fontconfig@2.12.3%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^font-util@1.3.1%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^pkg-config@0.29.2%gcc@4.4.6+internal_glib arch=linux-rhel6-x86_64 
                    ^util-macros@1.19.1%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^freetype@2.7.1%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^libpng@1.6.29%gcc@4.4.6 arch=linux-rhel6-x86_64 
                        ^zlib@1.2.11%gcc@4.4.6+pic+shared arch=linux-rhel6-x86_64 
                ^gperf@3.0.4%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^libxml2@2.9.4%gcc@4.4.6~python arch=linux-rhel6-x86_64 
                    ^xz@5.2.3%gcc@4.4.6 arch=linux-rhel6-x86_64 
            ^glib@2.53.1%gcc@4.4.6~libmount arch=linux-rhel6-x86_64 
                ^gettext@0.19.8.1%gcc@4.4.6+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-rhel6-x86_64 
                    ^ncurses@6.0%gcc@4.4.6~symlinks arch=linux-rhel6-x86_64 
                    ^tar@1.29%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^libffi@3.2.1%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^pcre@8.40%gcc@4.4.6+utf arch=linux-rhel6-x86_64 
            ^pixman@0.34.0%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^curl@7.54.0%gcc@4.4.6 arch=linux-rhel6-x86_64 
            ^openssl@1.0.2k%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^icu4c@58.2%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^jdk@8u141-b15%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^libjpeg-turbo@1.5.0%gcc@4.4.6 arch=linux-rhel6-x86_64 
            ^nasm@2.11.06%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^libtiff@4.0.8%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^pango@1.40.3%gcc@4.4.6~X arch=linux-rhel6-x86_64 
            ^gobject-introspection@1.49.2%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^bison@3.0.4%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^m4@1.4.18%gcc@4.4.6+sigsegv arch=linux-rhel6-x86_64 
                        ^libsigsegv@2.11%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^flex@2.6.4%gcc@4.4.6+lex arch=linux-rhel6-x86_64 
                    ^help2man@1.47.4%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^python@2.7.13%gcc@4.4.6+shared~tk~ucs4 arch=linux-rhel6-x86_64 
                    ^readline@7.0%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^sqlite@3.20.0%gcc@4.4.6 arch=linux-rhel6-x86_64 
            ^harfbuzz@1.4.6%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^tcl@8.6.6%gcc@4.4.6 arch=linux-rhel6-x86_64 
        ^tk@8.6.6%gcc@4.4.6 arch=linux-rhel6-x86_64 
            ^libx11@1.6.5%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^inputproto@2.3.2%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^kbproto@1.0.7%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^libxcb@1.12%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^libpthread-stubs@0.4%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^libxau@1.0.8%gcc@4.4.6 arch=linux-rhel6-x86_64 
                        ^xproto@7.0.31%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^libxdmcp@1.1.2%gcc@4.4.6 arch=linux-rhel6-x86_64 
                    ^xcb-proto@1.12%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^xextproto@7.3.0%gcc@4.4.6 arch=linux-rhel6-x86_64 
                ^xtrans@1.3.5%gcc@4.4.6 arch=linux-rhel6-x86_64 
    ^r-lattice@0.20-34%gcc@4.4.6 arch=linux-rhel6-x86_64 
```
Hello,

I have a similar error when I try to regenerate my tcl modules :
`
spack module refresh --module-type tcl --delete-tree -y

==> Regenerating tcl module files

==> Error: Package jpeg not found.
`

Seems to be linked with that commit : 
"
commit a48b5a627cb4fd18f7bd145e6bb944d5a315cfdb
Author: Sergey Kosukhin <skosukhin@gmail.com>
Date:   Thu Aug 24 10:30:35 2017 +0200

    Make jpeg a virtual dependency. (#5190)

    * Make jpeg a virtual dependency.

    * Make 'libjpeg-turbo' the default implementation of 'jpeg'.
"
Regards,

XL

EDIT : unistalling jpeg and its dependencies solved the issue.
Now fixed!  Closing.
Some `r-package` still has this issue with `jpeg`

I am loading each one to find out the cuprit.


Resolving patch:
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1888,7 +1888,7 @@ def _mark_concrete(self, value=True):
         unless there is a need to force a spec to be concrete.
         """
         for s in self.traverse(deptype_query=all):
-            if s.concrete and s.package.installed:
+            if (not value) and s.concrete and s.package.installed:
                 continue
             s._normal = value
             s._concrete = value


Test patch:
diff --git a/lib/spack/spack/test/database.py b/lib/spack/spack/test/database.py
--- a/lib/spack/spack/test/database.py
+++ b/lib/spack/spack/test/database.py
@@ -32,6 +32,7 @@
 import pytest
 import spack
 import spack.store
+from spack.test.conftest import MockPackageMultiRepo
 from spack.util.executable import Executable
 from llnl.util.tty.colify import colify
 
@@ -380,6 +381,22 @@ def fail_while_writing():
         assert install_db.query('cmake', installed=any) == []
 
 
+def test_115_reindex_with_packages_not_in_repo(database, refresh_db_on_exit):
+    install_db = database.mock.db
+
+    saved_repo = spack.repo
+    # Dont add any package definitions to this repository, the idea is that
+    # packages should not have to be defined in the repository once they are
+    # installed
+    mock_repo = MockPackageMultiRepo([])
+    try:
+        spack.repo = mock_repo
+        spack.store.db.reindex(spack.store.layout)
+        _check_db_sanity(install_db)
+    finally:
+        spack.repo = saved_repo
+
+
 def test_external_entries_in_db(database):
     install_db = database.mock.db