Repository: spack/spack

Issue:
r-xml: Spack version does not match upstream version
The r-xml package seems to have changed.  This updates the digest
value.  I was worried that the curl was flapping between different
archive sites (see below) but it's been stable for 3 days.

Here's today's fetch output (with the updated digest value)

```
==> Fetching https://cran.r-project.org/src/contrib/XML_3.98-1.4.tar.gz

curl: (22) The requested URL returned error: 404 Not Found
==> Fetching from https://cran.r-project.org/src/contrib/XML_3.98-1.4.tar.gz failed.
==> Fetching https://cran.r-project.org/src/contrib/Archive/XML/XML_3.98-1.2.tar.gz
```


Discussion:
I agree with @adamjstewart; random changes in checksums should be investigated.  If there is no known cause, it could indicate a security breach.
> random changes in checksums should be investigated. If there is no known cause, it could indicate a security breach.

I'm not particularly worried. If it's been stable for 3+ days, the tarball was probably just updated to backport a patch. Anyway, no one made sure the package wasn't hacked when it was added. I don't personally see the value in checksumming if you're just going to download it willy-nilly and declare that checksum to be safe without confirming that in any way.
> Anyway, no one made sure the package wasn't hacked when it was added. I don't personally see the value in checksumming if you're just going to download it willy-nilly and declare that checksum to be safe without confirming that in any way.

If a package is hacked, eventually someone will notice and discussion will happen on-line.  The longer the time is since a package was put up and nobody's heard a peep, the better chances you have that the package is what the authors intended (and that the authors are benevolent).

By putting a checksum in the Spack package, we protect against problems that might happen AFTER the checksum is created.

This is NOT a case of changing hashes, which makes my discussion with @adamjstewart somewhat off-topic.

The real problem is the main CRAN archive directory (listed in the package) only lists the LATEST version of packages.  Once a package is no longer latest, it is moved to the `00Archive` directory, where it lives permanently.  Here is where you can get all past versions of `XML`:
https://cran.r-project.org/src/contrib/00Archive/XML/

We don't want to use URLs that will change the next time a version comes out.  My recommendation would be to change the URL to the `00Archive` directory, and include one or more versions that you might think are interesting.  The Spack versions should match up exactly with the CRAN versions (i.e. `xml@3.98-1.4`, not `xml@3.91-1`).

Unfortunately, the latest/greatest XML is NOT in the `00Archive` directory.  If you don't really need it, the easiest is to just not list it in the Spack package.  If you HAVE to have it, then we should get around to something like #2281 

@citibeth: I believe that this is actually handled by the `list_url` for this package.  See #2309.
Hold on a second here. Did the hash even change? Look closely at the error message. It is downloading `XML_3.98-1.2.tar.gz`, not `XML_3.98-1.4.tar.gz`. All the more reason we need to change the version number to be more specific.
Confirmed that the checksum didn't actually change, it's just picking up the wrong version.

@tgamblin This is out of my area of expertise, but when trying to fix this, I encountered a problem with versions and urls. When the package was created, it saw `XML_3.98-1.4.tar.gz` and decided the version was `3.98-1`. When I change the version to `3.98-1.4`, it tries to download `XML_3.98-1.4.4.tar.gz`. This is definitely a bug, but I don't know enough about it to fix it.
I'm bringing up the rear of the pack on this.  I didn't catch the sub-sub-version change until it was called out above.

For what it's worth: `spack versions` doesn't list any of them as possible choices, just the top level `3.98-1`.

```
[hartzelg@spack-0 spack-20161204]$ spack versions r-xml
==> Safe versions (already checksummed):
  3.98-1
==> Remote versions (not yet checksummed):
  3.96-1  3.7-4  3.4-2  2.5-3   1.98-1  1.93-1  1.6-3  1.2-0    0.99-5  0.97-3  0.94-1  0.92-2  0.7-4  0.6-1
  3.96-0  3.7-3  3.4-0  2.5-1   1.98-0  1.92-1  1.6-2  1.1-1    0.99-4  0.97-0  0.94-0  0.92-1  0.7-3  0.3-3
  3.95-0  3.7-2  3.2-0  2.5-0   1.96-0  1.92-0  1.6-0  0.99-93  0.99-2  0.95-6  0.93-4  0.9-1   0.7-2  0.2
  3.9-4   3.6-2  3.1-1  2.3-0   1.95-3  1.9-0   1.5-1  0.99-92  0.97-8  0.95-4  0.93-3  0.9-0   0.7-1
  3.9-2   3.6-1  3.1-0  2.1-0   1.95-2  1.7-3   1.4-1  0.99-8   0.97-7  0.95-3  0.93-2  0.8-2   0.7-0
  3.9-0   3.6-0  2.8-1  2.0.7   1.95-1  1.7-2   1.4-0  0.99-7   0.97-6  0.95-1  0.93-1  0.8-0   0.6-3
  3.8-0   3.4-3  2.6-0  1.99-0  1.93-2  1.7-1   1.3-2  0.99-6   0.97-4  0.95-0  0.93-0  0.7-5   0.6-2
```


Yes, Spack is definitely incorrectly parsing the version from the URL. In the meantime, you can write a `url_for_version` that returns the correct URL. But we should probably fix the source of the problem.

Resolving patch:
diff --git a/lib/spack/spack/cmd/url_parse.py b/lib/spack/spack/cmd/url_parse.py
--- a/lib/spack/spack/cmd/url_parse.py
+++ b/lib/spack/spack/cmd/url_parse.py
@@ -53,15 +53,19 @@ def print_name_and_version(url):
 def url_parse(parser, args):
     url = args.url
 
-    ver,  vs, vl = spack.url.parse_version_offset(url)
-    name, ns, nl = spack.url.parse_name_offset(url, ver)
+    ver,  vs, vl = spack.url.parse_version_offset(url, debug=True)
+    name, ns, nl = spack.url.parse_name_offset(url, ver, debug=True)
+    print
 
-    tty.msg("Parsing URL:")
+    tty.msg("Detected:")
     try:
         print_name_and_version(url)
     except spack.url.UrlParseError as e:
         tty.error(str(e))
 
+    print '    name:     %s' % name
+    print '    version:  %s' % ver
+
     print
     tty.msg("Substituting version 9.9.9b:")
     newurl = spack.url.substitute_version(url, '9.9.9b')
diff --git a/lib/spack/spack/url.py b/lib/spack/spack/url.py
--- a/lib/spack/spack/url.py
+++ b/lib/spack/spack/url.py
@@ -163,7 +163,7 @@ def determine_url_file_extension(path):
     return ext
 
 
-def parse_version_offset(path):
+def parse_version_offset(path, debug=False):
     """Try to extract a version string from a filename or URL.  This is taken
        largely from Homebrew's Version class."""
     original_path = path
@@ -209,8 +209,8 @@ def parse_version_offset(path):
         # e.g. lame-398-1
         (r'-((\d)+-\d)', stem),
 
-        # e.g. foobar_1.2-3
-        (r'_((\d+\.)+\d+(-\d+)?[a-z]?)', stem),
+        # e.g. foobar_1.2-3 or 3.98-1.4
+        (r'_((\d+\.)+\d+(-(\d+(\.\d+)?))?[a-z]?)', stem),
 
         # e.g. foobar-4.5.1
         (r'-((\d+\.)*\d+)$', stem),
@@ -246,6 +246,10 @@ def parse_version_offset(path):
         regex, match_string = vtype
         match = re.search(regex, match_string)
         if match and match.group(1) is not None:
+            if debug:
+                tty.msg("Parsing URL: %s" % path,
+                        "  Matched regex %d: r'%s'" % (i, regex))
+
             version = match.group(1)
             start   = match.start(1)
 
@@ -266,9 +270,9 @@ def parse_version(path):
     return Version(ver)
 
 
-def parse_name_offset(path, v=None):
+def parse_name_offset(path, v=None, debug=False):
     if v is None:
-        v = parse_version(path)
+        v = parse_version(path, debug=debug)
 
     path, ext, suffix = split_url_extension(path)
 
diff --git a/var/spack/repos/builtin/packages/r-xml/package.py b/var/spack/repos/builtin/packages/r-xml/package.py
--- a/var/spack/repos/builtin/packages/r-xml/package.py
+++ b/var/spack/repos/builtin/packages/r-xml/package.py
@@ -34,7 +34,8 @@ class RXml(Package):
     url      = "https://cran.r-project.org/src/contrib/XML_3.98-1.4.tar.gz"
     list_url = "https://cran.r-project.org/src/contrib/Archive/XML"
 
-    version('3.98-1', '1a7f3ce6f264eeb109bfa57bedb26c14')
+    version('3.98-1.5', 'd1cfcd56f7aec96a84ffca91aea507ee')
+    version('3.98-1.4', '1a7f3ce6f264eeb109bfa57bedb26c14')
 
     extends('R')
 


Test patch:
diff --git a/lib/spack/spack/test/url_parse.py b/lib/spack/spack/test/url_parse.py
--- a/lib/spack/spack/test/url_parse.py
+++ b/lib/spack/spack/test/url_parse.py
@@ -326,3 +326,8 @@ def test_github_raw_url(self):
         self.check(
             'powerparser', '2.0.7',
             'https://github.com/losalamos/CLAMR/blob/packages/PowerParser_v2.0.7.tgz?raw=true')
+
+    def test_r_xml_version(self):
+        self.check(
+            'xml', '3.98-1.4',
+            'https://cran.r-project.org/src/contrib/XML_3.98-1.4.tar.gz')