Repository: spack/spack

Issue:
package version auto-location exceptions and mis-identification
Working with gnu package directories, which are pretty regularly structured and I expected to work well, it seems that some oddities come up with the current spider approach.  This actually turned up two separate but related issues.

The first is that loading a directory from an FTP url appears to cause issues, second is that `.sig` files are being recognized as potential archives and causing failures when spack attempts to treat them as tar archives.

See errors below:

```
rzalastor1@scogland:EDITOR='emacs -nw' spack create ftp://ftp.gnu.org/pub/gnu/emacs/emacs-24.5.tar.xz
==> This looks like a URL for emacs version 24.5.
==> Creating template for package emacs
Traceback (most recent call last):
  File "/g/g12/scogland/spack/bin/spack", line 168, in <module>
    main()
  File "/g/g12/scogland/spack/bin/spack", line 146, in main
    return_val = command(parser, args)
  File "/g/g12/scogland/spack/lib/spack/spack/cmd/create.py", line 169, in create
    versions = spack.util.web.find_versions_of_archive(url)
  File "/g/g12/scogland/spack/lib/spack/spack/util/web.py", line 214, in find_versions_of_archive
    p, l = spider(lurl, depth=list_depth)
  File "/g/g12/scogland/spack/lib/spack/spack/util/web.py", line 176, in spider
    pages, links =  _spider((root_url, set(), root_url, None, 1, max_depth, False))
ValueError: need more than 0 values to unpack
rzalastor1@scogland:EDITOR='emacs -nw' spack create http://ftp.gnu.org/pub/gnu/emacs/emacs-24.5.tar.xz
==> This looks like a URL for emacs version 24.5.
==> Creating template for package emacs
==> Found 8 versions of emacs:
  24.5.tar.xz.sighttp://ftp.gnu.org/pub/gnu/emacs/emacs-24.5.tar.xz.sig
  24.5      http://ftp.gnu.org/pub/gnu/emacs/emacs-24.5.tar.xz
  24.4.tar.xz.sighttp://ftp.gnu.org/pub/gnu/emacs/emacs-24.4.tar.xz.sig
  24.4      http://ftp.gnu.org/pub/gnu/emacs/emacs-24.4.tar.xz
  24.3.tar.xz.sighttp://ftp.gnu.org/pub/gnu/emacs/emacs-24.3.tar.xz.sig
  24.3      http://ftp.gnu.org/pub/gnu/emacs/emacs-24.3.tar.xz
  24.2.tar.xz.sighttp://ftp.gnu.org/pub/gnu/emacs/emacs-24.2.tar.xz.sig
  24.2      http://ftp.gnu.org/pub/gnu/emacs/emacs-24.2.tar.xz

Include how many checksums in the package file? (default is 5, q to abort)
==> Downloading...
==> Trying to fetch from http://ftp.gnu.org/pub/gnu/emacs/emacs-24.5.tar.xz.sig
######################################################################## 100.0%
/bin/tar: This does not look like a tar archive
/bin/tar: Skipping to next header
/bin/tar: Exiting with failure status due to previous errors
==> Error: Command exited with status 2:
/bin/tar --exclude=*/*/* -tf /g/g12/scogland/spack/var/spack/stage/spack-stage-aC986I/emacs-24.5.tar.xz.sig
```



Discussion:
I tried to figure out how to address this today, but ran into a bit of a conundrum.  The current setup builds a regex to match the package, but does nothing about restricting what comes after the end of the package string.  I was temped to just add a `$` at the end, but I'm guessing this is meant to support urls with embedded packages and arguments.  Can anyone think of a case we care about where adding a suffix of `([^.].*|)$` or `([?/&].*|)` would bite us? It should just disallow extra extensions or name components being added without screwing with complex URLs, but I'm not sure I'm thinking all of the cases through.

This issue also occurs with Bash:

``` bash
$ spack create ftp://ftp.gnu.org/gnu/bash/bash-4.3.tar.gz
==> This looks like a URL for bash version 4.3.
==> Creating template for package bash
Traceback (most recent call last):
  File "/soft/spack/bin/spack", line 176, in <module>
    main()
  File "/soft/spack/bin/spack", line 154, in main
    return_val = command(parser, args)
  File "/blues/gpfs/home/software/spack/lib/spack/spack/cmd/create.py", line 259, in create
    versions, urls = fetch_tarballs(url, name, args)
  File "/blues/gpfs/home/software/spack/lib/spack/spack/cmd/create.py", line 218, in fetch_tarballs
    versions = spack.util.web.find_versions_of_archive(url)
  File "/blues/gpfs/home/software/spack/lib/spack/spack/util/web.py", line 214, in find_versions_of_archive
    p, l = spider(lurl, depth=list_depth)
  File "/blues/gpfs/home/software/spack/lib/spack/spack/util/web.py", line 176, in spider
    pages, links =  _spider((root_url, set(), root_url, None, 1, max_depth, False))
ValueError: need more than 0 values to unpack
```

The bash source repository also contains .sig files.

The latter issue still exists, I encountered it when creating packages for recent X11 libs/protos[1].

It's easy to work around (just remove the versions pointing to .sig files), but it's still inconvenient.

@trws Shouldn't it be possible to test such a change against existing packages somehow? For example, compare output of `spack checksum` for all packages without and with the change.

[1]: Sources at: https://www.x.org/archive/individual/lib/, https://www.x.org/archive/individual/proto/

That makes sense to me, but I haven't thought of a great way to wire it up.  If you have an idea @tobbez it could be useful.

@trws You can put the following in a file:

``` python
import spack

from spack.url import UndetectableVersionError, UndetectableNameError, UndetectableNameError
from spack.package import VersionFetchError

for p in sorted(spack.repo.all_package_names()):
  try:
    for k,v in sorted(spack.repo.get(p).fetch_remote_versions().items(), key=lambda x:x[0]):
      print('{} {} {}'.format(p, str(k), v))
  except UndetectableVersionError:
      print('{} UndetectableVersionError'.format(p))
  except VersionFetchError:
      print('{} VersionFetchError'.format(p))
  except UndetectableNameError:
      print('{} UndetectableNameError'.format(p))
```

And then call it by running `spack python script.py`.

Run it one time without any changes to `spack`, and then one time each for every change you want to test, piping the output of each run into separate files.

When finished you can run `diff` on the "before" and "after" files to see what effect your change(s) had.

This bug has been biting me for a while now, and was particularly painful while packaging #1740. I'm going to take a second look at this and see what I can do.

Here's another method of testing. Since the `*.sig` versions show up when you run `spack versions`, you can do this for all packages and diff the output. Here is a short script:

``` bash
#!/usr/bin/env bash

output=versions-before.txt
rm -f $output

# Without this, Ctrl+C will kill Spack, not the script,
# meaning that the script continues running forever.
trap "exit" SIGINT

for package in $(spack list)
do
    echo $package | tee -a $output
    spack versions $package 2>&1 | tee -a $output
done
```

Other common non-tarball files that are being picked up:

```
tar.bz2.asc
tar.bz2.sha256.asc
tar.bz2.sha256sum
tar.bz2.sig
tar.gz.asc
tar.gz.back
tar.gz.md5.txt
tar.gz.mds
tar.gz.minisig
tar.gz.old
tar.gz.sha1.asc
tar.gz.sha1.txt
tar.gz.sha256.txt
tar.gz.sig
tar.gz.sign
tar.gz.smime
tar.gz.torrent
tar.xz.sha1.asc
tar.xz.sig
tgz.asc
```


Resolving patch:
diff --git a/lib/spack/spack/url.py b/lib/spack/spack/url.py
--- a/lib/spack/spack/url.py
+++ b/lib/spack/spack/url.py
@@ -323,7 +323,7 @@ def parse_name_and_version(path):
 
 def insensitize(string):
     """Change upper and lowercase letters to be case insensitive in
-       the provided string.  e.g., 'a' because '[Aa]', 'B' becomes
+       the provided string.  e.g., 'a' becomes '[Aa]', 'B' becomes
        '[bB]', etc.  Use for building regexes."""
     def to_ins(match):
         char = match.group(1)
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -227,7 +227,16 @@ def find_versions_of_archive(*archive_urls, **kwargs):
 
         # We'll be a bit more liberal and just look for the archive
         # part, not the full path.
-        regexes.append(os.path.basename(url_regex))
+        url_regex = os.path.basename(url_regex)
+
+        # We need to add a $ anchor to the end of the regex to prevent
+        # Spack from picking up signature files like:
+        #   .asc
+        #   .md5
+        #   .sha256
+        #   .sig
+        # However, SourceForge downloads still need to end in '/download'.
+        regexes.append(url_regex + '(\/download)?$')
 
     # Build a dict version -> URL from any links that match the wildcards.
     versions = {}


Test patch: