Repository: spack/spack

Issue:
spack versions and spack checksum only using list_url
I noticed that when looking for updates to R CRAN packages, which have a `url` and a `list_url` setting, that `spack versions/checksum` will only use the `list_url`. The expected, or preferred, behavior is to use both the parent directory of the file in the `url` *and* `list_url`.

### Steps to reproduce the issue

```console
$ spack versions r-broom
==> Safe versions (already checksummed):
  0.5.1  0.4.2
==> Remote versions (not yet checksummed):
  0.5.0  0.4.5  0.4.4  0.4.3  0.4.1  0.4.0  0.3.7  0.3.6  0.3.5  0.3.4
...
```
The current version on CRAN is actually 0.5.2 but that does not show up. That version would be in the parent directory of the file in `url` and not where `list_url` points to.

Both the parent directory of the file in `url` and `list_url` should be used. Alternatively, allow `list_url` to have multiple entries. 

A workaround for R CRAN packages would be to make the root of `list_url` higher up to catch everything but that will mean more spidering.



Discussion:
So the issue here is that Spack is trying to be too smart. `spack.cmd.versions` calls `spack.package.fetch_remote_versions`, which calls `spack.util.web.find_versions_of_archive`, which calls `spack.url.find_list_url` on the package's `url` attribute. Normally, this function simply returns the `dirname` of the URL, as you would expect. However, since Spack knows that this URL belongs to CRAN, it instead returns the Archive URL.

The simplest fix would be to return both the Archive URL _and_ the `dirname` of the URL. I'll have to make sure this change doesn't break anything else.

Resolving patch:
diff --git a/lib/spack/spack/url.py b/lib/spack/spack/url.py
--- a/lib/spack/spack/url.py
+++ b/lib/spack/spack/url.py
@@ -43,8 +43,8 @@
 # work on paths and URLs.  There's not a good word for both of these, but
 # "path" seemed like the most generic term.
 #
-def find_list_url(url):
-    r"""Finds a good list URL for the supplied URL.
+def find_list_urls(url):
+    r"""Find good list URLs for the supplied URL.
 
     By default, returns the dirname of the archive path.
 
@@ -62,7 +62,7 @@ def find_list_url(url):
         url (str): The download URL for the package
 
     Returns:
-        str: The list URL for the package
+        set: One or more list URLs for the package
     """
 
     url_types = [
@@ -93,12 +93,14 @@ def find_list_url(url):
          lambda m: m.group(1) + '/Archive/' + m.group(2)),
     ]
 
+    list_urls = set([os.path.dirname(url)])
+
     for pattern, fun in url_types:
         match = re.search(pattern, url)
         if match:
-            return fun(match)
-    else:
-        return os.path.dirname(url)
+            list_urls.add(fun(match))
+
+    return list_urls
 
 
 def strip_query_and_fragment(path):
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -270,20 +270,18 @@ def find_versions_of_archive(archive_urls, list_url=None, list_depth=0):
     """Scrape web pages for new versions of a tarball.
 
     Arguments:
-      archive_urls:
-          URL or sequence of URLs for different versions of a
-          package. Typically these are just the tarballs from the package
-          file itself.  By default, this searches the parent directories
-          of archives.
+        archive_urls (str or list or tuple): URL or sequence of URLs for
+            different versions of a package. Typically these are just the
+            tarballs from the package file itself. By default, this searches
+            the parent directories of archives.
 
     Keyword Arguments:
-      list_url:
-          URL for a listing of archives.  Spack wills scrape these
-          pages for download links that look like the archive URL.
-
-      list_depth:
-          Max depth to follow links on list_url pages. Default 0.
+        list_url (str or None): URL for a listing of archives.
+            Spack will scrape these pages for download links that look
+            like the archive URL.
 
+        list_depth (int): Max depth to follow links on list_url pages.
+            Defaults to 0.
     """
     if not isinstance(archive_urls, (list, tuple)):
         archive_urls = [archive_urls]
@@ -291,17 +289,17 @@ def find_versions_of_archive(archive_urls, list_url=None, list_depth=0):
     # Generate a list of list_urls based on archive urls and any
     # explicitly listed list_url in the package
     list_urls = set()
-    if list_url:
+    if list_url is not None:
         list_urls.add(list_url)
     for aurl in archive_urls:
-        list_urls.add(spack.url.find_list_url(aurl))
+        list_urls |= spack.url.find_list_urls(aurl)
 
     # Add '/' to the end of the URL. Some web servers require this.
     additional_list_urls = set()
     for lurl in list_urls:
         if not lurl.endswith('/'):
             additional_list_urls.add(lurl + '/')
-    list_urls.update(additional_list_urls)
+    list_urls |= additional_list_urls
 
     # Grab some web pages to scrape.
     pages = {}


Test patch: