Repository: spack/spack

Issue:
create (via find_versions_of_archive()) does wrong thing for Go projects on GitHub with GoPackage
I'm working on a GoPackage build system (see #13023) in a Spack tree at commit `94c8bf032`.

**TL;DR**:  

I'm coincidentally/unfortunately prototyping with projects (e.g. [lazygit](https://github.com/jesseduffield/lazygit), [lazydocker](https://github.com/jesseduffield/lazydocker)) that have uploaded a number of "assets" for a release (so many "releases/download" URLs are discovered), *but* for which the source itself is at an "archive" URL.  Spack's heuristic of preferring "download" URLs causes it to get the wrong tarball for a release, which causes the package type guesser to fail and/or to insert an incorrect checksum for a version.

Perhaps there should there be an option to `spack create` that tells it to just use the provided URL and not try to guess additional versions.

---

I'd like `spack create` to Do The Right Thing for Go packages on GitHub, e.g.

```
spack create https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
```

should use the GoPackage template that I've created.  Instead of offering to use the URL that I provided it offers me a series of ".../releases/download/" URLS that point to the *binaries* that have been attached to the releases.  It does, eventually, offers a couple of older "archive" tarballs.

I believe that for the recent versions, it's choosing one of the binaries that have been attached to the release (with a `.../releases/download/...` URL) in preference to the `archive` URL that I provided, driven by code following [this comment in `spack/spack/util/web.py`](https://github.com/spack/spack/blob/develop/lib/spack/spack/util/web.py#L567).  The older releases don't have any attached "assets" and so the Right Thing happens.

I believe that the intent is to prefer maintainer uploaded release source tarballs (which e.g. may already have been autoconfed) in preference to the automagically generated tarball of the repo at that particular tag.  In this/my case that turns out to be The Wrong Thing.

Those binary-containing release tarballs don't contain the source tree, so the build system guesser can't do anything useful and I end up with a default package.

My workaround is to 

```
(alice)[11:27:30]spack>>spack create -t go https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
==> This looks like a URL for lazygit
==> Found 10 versions of lazygit:

  0.11.3  https://github.com/jesseduffield/lazygit/releases/download/v0.11.3/lazygit_0.11.3_freebsd_x86_64.tar.gz
  0.11.2  https://github.com/jesseduffield/lazygit/releases/download/v0.11.2/lazygit_0.11.2_freebsd_x86_64.tar.gz
  0.11.1  https://github.com/jesseduffield/lazygit/releases/download/v0.11.1/lazygit_0.11.1_freebsd_x86_64.tar.gz
  0.10.6  https://github.com/jesseduffield/lazygit/releases/download/v0.10.6/lazygit_0.10.6_freebsd_x86_64.tar.gz
  0.10.5  https://github.com/jesseduffield/lazygit/releases/download/v0.10.5/lazygit_0.10.5_freebsd_x86_64.tar.gz
  0.10.4  https://github.com/jesseduffield/lazygit/releases/download/v0.10.4/lazygit_0.10.4_freebsd_x86_64.tar.gz
  0.10.3  https://github.com/jesseduffield/lazygit/releases/download/v0.10.3/lazygit_0.10.3_freebsd_x86_64.tar.gz
  0.10.2  https://github.com/jesseduffield/lazygit/releases/download/v0.10.2/lazygit_0.10.2_freebsd_x86_64.tar.gz
  0.10.1  https://github.com/jesseduffield/lazygit/archive/v0.10.1.tar.gz
  0.10    https://github.com/jesseduffield/lazygit/archive/v0.10.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://github.com/jesseduffield/lazygit/releases/download/v0.11.3/lazygit_0.11.3_freebsd_x86_64.tar.gz
######################################################################## 100.0%
==> Checksummed 1 version of lazygit:
==> Using specified package template: 'go'
==> Created template for lazygit package
==> Created package file: /Users/hartzell/tmp/spack/var/spack/repos/builtin/packages/lazygit/package.py
(alice)[11:28:00]spack>>
```

which results in the `package.py` I expect albeit with an inappropriate sha256 checksum for the tarball (because it calculated the signature on a different file than it's fetching and unpacking).

Then, either by hand or after the build fails, I update the checksum.

```
(alice)[11:28:42]spack>>spack install lazygit
==> Warning: clang@10.0.0-apple cannot build optimized binaries for "haswell". Using best target possible: "x86_64"
==> libsigsegv is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libsigsegv-2.12-5rgsrc5khc7wfva6oqq6dnchzcy23c44
==> m4 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/m4-1.4.18-5qz6n7qmauidmhx6qfm7yqjof6n5pxek
==> pkgconf is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/pkgconf-1.6.3-emiquv3wq32427k4htpjastyyxcibey2
==> ncurses is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/ncurses-6.1-3zvai2464mpmryqtefjdkzn3udofiwux
==> readline is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/readline-8.0-hwrhaf6j4ixq5tf5m5tixatg3llazzpo
==> gdbm is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/gdbm-1.18.1-atkqjart2dfjnori2tgjznimitqqphwu
==> perl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/perl-5.30.0-wmlonvwujujfe3wvknnbikfwl4nqf3zx
==> autoconf is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/autoconf-2.69-7zvd6iq5hqj6biefv3etpp2ojxqoora7
==> automake is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/automake-1.16.1-jz53ixq46xbx2creygmha2kew4pkpctl
==> zlib is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/zlib-1.2.11-caosjr5fcruvzyocttnqqkpackpwyzpz
==> curl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/curl-7.63.0-wewrogc25man2k5nu2klgjk6w3qkkyue
==> expat is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/expat-2.2.9-ou2ln2zskxs2jlevzifqijhbj4gz64t5
==> libiconv is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libiconv-1.16-3cgb7j3myogfqhkfiwubqfoqea5g6j6i
==> diffutils is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/diffutils-3.7-btjr4gj77c3eiyzfirrxiw5mgmr3hm6l
==> bzip2 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/bzip2-1.0.8-rlvwpbm5a5faacq255eko6kz5kdbcl5h
==> xz is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/xz-5.2.4-n6uufvkzz3jrhiyjylh3ggu3vujnlbto
==> libxml2 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libxml2-2.9.9-bh3u7tmbkhzgveodvcrztnkxbmhfw2nr
==> tar is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/tar-1.32-eexlhwrs7ipvnxzvplytgs2aytoxknzv
==> gettext is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/gettext-0.20.1-r6ksjrlxvr2xuezholiysmpcguhpuaxm
==> libtool is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libtool-2.4.6-75s5xqeyblajfff44vqgkcz73nogfsx4
==> openssl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/openssl-1.1.1d-zdnx7cdmqrclhe7qvozjsgvyrt24nldd
==> pcre is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/pcre-8.42-oxcmvw6kmk4n5k7dc6jkcj2gkamoe2g3
==> git is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/git-2.21.0-44ozdyhctqq7viiudvmzvxlnm6ryaisd
==> go-bootstrap is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-bootstrap-1.4-bootstrap-20171003-t6yornwalmj7lw4ysglchwg7x2lkhcuw
==> go is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-1.13.4-w3obiflkwgwd5usciec7qyujpm4556ez
==> Installing lazygit
==> Searching for binary cache of lazygit
==> Warning: No Spack mirrors are currently configured
==> No binary for lazygit found: installing from source
==> Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check clang@10.0.0-apple for further details]
==> Fetching https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
######################################################################## 100.0%
==> Error: ChecksumError: sha256 checksum failed for /var/folders/f7/fcb363190nz0jbypndp3nj_m0000gn/T/hartzell/spack-stage/spack-stage-lazygit-0.11.3-nsecxd6wynzqxlx4m7kcvmqeqj2bjku5/v0.11.3.tar.gz
    Expected 5580ece2191fb9a52555e7f5dba38e62451749697889e440002705ecb015c9ec but got b3c503de6b34fd4284fd70655e7f42feafc07f090e7f7cc00db261f56c583c46

/Users/hartzell/tmp/spack/lib/spack/spack/package.py:1083, in do_fetch:
       1080        self.stage.fetch(mirror_only)
       1081        self._fetch_time = time.time() - start_time
       1082
  >>   1083        if checksum and self.version in self.versions:
       1084            self.stage.check()
       1085
       1086        self.stage.cache_local()


(alice)[11:30:42]spack>>
```

I'm wondering if there's a better way around this.  Perhaps the fix, if it's useful enough, is an option to `spack create` that avoids the fancy guesswork and just tells it to do what it's told.

### Steps to reproduce the issue

```console
(alice)[10:44:03]spack>>spack -d create https://github.com/jesseduffield/lazygit/archive/v0.11.3.tar.gz
==> [2019-11-30-10:44:26.091159] Imported create from built-in commands
==> [2019-11-30-10:44:26.092401] Imported create from built-in commands
==> [2019-11-30-10:44:26.094375] Reading config file /Users/hartzell/tmp/spack/etc/spack/defaults/modules.yaml
==> [2019-11-30-10:44:26.104312] Reading config file /Users/hartzell/tmp/spack/etc/spack/defaults/darwin/modules.yaml
==> [2019-11-30-10:44:26.107297] Reading config file /Users/hartzell/tmp/spack/etc/spack/modules.yaml
==> [2019-11-30-10:44:26.113038] Reading config file /Users/hartzell/.spack/darwin/modules.yaml
==> [2019-11-30-10:44:26.118919] Reading config file /Users/hartzell/tmp/spack/etc/spack/defaults/config.yaml
==> [2019-11-30-10:44:26.153879] This looks like a URL for lazygit
==> [2019-11-30-10:44:26.419630] HTTPError: HTTP Error 404: Not Found
==> [2019-11-30-10:44:26.772969] HTTPError: HTTP Error 404: Not Found
==> [2019-11-30-10:44:29.634677] Found 10 versions of lazygit:

  0.11.3  https://github.com/jesseduffield/lazygit/releases/download/v0.11.3/lazygit_0.11.3_freebsd_x86_64.tar.gz
  0.11.2  https://github.com/jesseduffield/lazygit/releases/download/v0.11.2/lazygit_0.11.2_freebsd_x86_64.tar.gz
  0.11.1  https://github.com/jesseduffield/lazygit/releases/download/v0.11.1/lazygit_0.11.1_freebsd_x86_64.tar.gz
  0.10.6  https://github.com/jesseduffield/lazygit/releases/download/v0.10.6/lazygit_0.10.6_freebsd_x86_64.tar.gz
  0.10.5  https://github.com/jesseduffield/lazygit/releases/download/v0.10.5/lazygit_0.10.5_freebsd_x86_64.tar.gz
  0.10.4  https://github.com/jesseduffield/lazygit/releases/download/v0.10.4/lazygit_0.10.4_freebsd_x86_64.tar.gz
  0.10.3  https://github.com/jesseduffield/lazygit/releases/download/v0.10.3/lazygit_0.10.3_freebsd_x86_64.tar.gz
  0.10.2  https://github.com/jesseduffield/lazygit/releases/download/v0.10.2/lazygit_0.10.2_freebsd_x86_64.tar.gz
  0.10.1  https://github.com/jesseduffield/lazygit/archive/v0.10.1.tar.gz
  0.10    https://github.com/jesseduffield/lazygit/archive/v0.10.tar.gz

==> [2019-11-30-10:44:29.634843] How many would you like to checksum? (default is 1, q to abort) q
==> [2019-11-30-10:45:50.387245] Error: Aborted.
```

and notice that it offers "releases/download" links in preference to the link you provided.



Discussion:
Ps.  It's worth noting that other Go projects (e.g. go-md2man) use archive links and do not upload the confounding binary assets:

```console
(alice)[11:59:47]spack>>spack create -f https://github.com/cpuguy83/go-md2man/archive/v1.0.10.tar.gz
==> This looks like a URL for go-md2man
==> Found 10 versions of go-md2man:

  2.0.0   https://github.com/cpuguy83/go-md2man/archive/v2.0.0.tar.gz
  1.0.10  https://github.com/cpuguy83/go-md2man/archive/v1.0.10.tar.gz
  1.0.9   https://github.com/cpuguy83/go-md2man/archive/v1.0.9.tar.gz
  1.0.8   https://github.com/cpuguy83/go-md2man/archive/v1.0.8.tar.gz
  1.0.7   https://github.com/cpuguy83/go-md2man/archive/v1.0.7.tar.gz
  1.0.6   https://github.com/cpuguy83/go-md2man/archive/v1.0.6.tar.gz
  1.0.5   https://github.com/cpuguy83/go-md2man/archive/v1.0.5.tar.gz
  1.0.4   https://github.com/cpuguy83/go-md2man/archive/v1.0.4.tar.gz
  1.0.3   https://github.com/cpuguy83/go-md2man/archive/v1.0.3.tar.gz
  1.0.2   https://github.com/cpuguy83/go-md2man/archive/v1.0.2.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) 2
==> Downloading...
==> Fetching https://github.com/cpuguy83/go-md2man/archive/v2.0.0.tar.gz
######################################################################## 100.0%
==> Fetching https://github.com/cpuguy83/go-md2man/archive/v1.0.10.tar.gz
######################################################################## 100.0%
==> Checksummed 2 versions of go-md2man:
==> This package looks like it uses the go build system
==> Created template for go-md2man package
==> Created package file: /Users/hartzell/tmp/spack/var/spack/repos/builtin/packages/go-md2man/package.py
(alice)[12:00:44]spack>>spack install go-md2man
==> Warning: clang@10.0.0-apple cannot build optimized binaries for "haswell". Using best target possible: "x86_64"
==> libsigsegv is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libsigsegv-2.12-5rgsrc5khc7wfva6oqq6dnchzcy23c44
==> m4 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/m4-1.4.18-5qz6n7qmauidmhx6qfm7yqjof6n5pxek
==> pkgconf is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/pkgconf-1.6.3-emiquv3wq32427k4htpjastyyxcibey2
==> ncurses is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/ncurses-6.1-3zvai2464mpmryqtefjdkzn3udofiwux
==> readline is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/readline-8.0-hwrhaf6j4ixq5tf5m5tixatg3llazzpo
==> gdbm is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/gdbm-1.18.1-atkqjart2dfjnori2tgjznimitqqphwu
==> perl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/perl-5.30.0-wmlonvwujujfe3wvknnbikfwl4nqf3zx
==> autoconf is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/autoconf-2.69-7zvd6iq5hqj6biefv3etpp2ojxqoora7
==> automake is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/automake-1.16.1-jz53ixq46xbx2creygmha2kew4pkpctl
==> zlib is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/zlib-1.2.11-caosjr5fcruvzyocttnqqkpackpwyzpz
==> curl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/curl-7.63.0-wewrogc25man2k5nu2klgjk6w3qkkyue
==> expat is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/expat-2.2.9-ou2ln2zskxs2jlevzifqijhbj4gz64t5
==> libiconv is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libiconv-1.16-3cgb7j3myogfqhkfiwubqfoqea5g6j6i
==> diffutils is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/diffutils-3.7-btjr4gj77c3eiyzfirrxiw5mgmr3hm6l
==> bzip2 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/bzip2-1.0.8-rlvwpbm5a5faacq255eko6kz5kdbcl5h
==> xz is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/xz-5.2.4-n6uufvkzz3jrhiyjylh3ggu3vujnlbto
==> libxml2 is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libxml2-2.9.9-bh3u7tmbkhzgveodvcrztnkxbmhfw2nr
==> tar is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/tar-1.32-eexlhwrs7ipvnxzvplytgs2aytoxknzv
==> gettext is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/gettext-0.20.1-r6ksjrlxvr2xuezholiysmpcguhpuaxm
==> libtool is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/libtool-2.4.6-75s5xqeyblajfff44vqgkcz73nogfsx4
==> openssl is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/openssl-1.1.1d-zdnx7cdmqrclhe7qvozjsgvyrt24nldd
==> pcre is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/pcre-8.42-oxcmvw6kmk4n5k7dc6jkcj2gkamoe2g3
==> git is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/git-2.21.0-44ozdyhctqq7viiudvmzvxlnm6ryaisd
==> go-bootstrap is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-bootstrap-1.4-bootstrap-20171003-t6yornwalmj7lw4ysglchwg7x2lkhcuw
==> go is already installed in /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-1.13.4-w3obiflkwgwd5usciec7qyujpm4556ez
==> Installing go-md2man
==> Searching for binary cache of go-md2man
==> Warning: No Spack mirrors are currently configured
==> No binary for go-md2man found: installing from source
==> Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check clang@10.0.0-apple for further details]
==> Fetching https://github.com/cpuguy83/go-md2man/archive/v2.0.0.tar.gz
######################################################################## 100.0%
==> Staging archive: /var/folders/f7/fcb363190nz0jbypndp3nj_m0000gn/T/hartzell/spack-stage/spack-stage-go-md2man-2.0.0-fzjlakrsi5dwvvzyxnf3qjjpehvbppw5/v2.0.0.tar.gz
==> Created stage in /var/folders/f7/fcb363190nz0jbypndp3nj_m0000gn/T/hartzell/spack-stage/spack-stage-go-md2man-2.0.0-fzjlakrsi5dwvvzyxnf3qjjpehvbppw5
==> No patches needed for go-md2man
==> Building go-md2man [GoPackage]
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed go-md2man
  Fetch: 0.85s.  Build: 0.99s.  Total: 1.84s.
[+] /Users/hartzell/tmp/spack/opt/spack/darwin-highsierra-x86_64/clang-10.0.0-apple/go-md2man-2.0.0-fzjlakrsi5dwvvzyxnf3qjjpehvbppw5
```
Another example project that suffers from this issue: 

```
spack create https://github.com/wagoodman/dive/archive/v0.9.1.tar.gz
```

Resolving patch:
diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py
--- a/lib/spack/spack/package.py
+++ b/lib/spack/spack/package.py
@@ -2564,7 +2564,11 @@ def fetch_remote_versions(self, concurrency=128):
 
         try:
             return spack.util.web.find_versions_of_archive(
-                self.all_urls, self.list_url, self.list_depth, concurrency
+                self.all_urls,
+                self.list_url,
+                self.list_depth,
+                concurrency,
+                reference_package=self,
             )
         except spack.util.web.NoNetworkConnectionError as e:
             tty.die("Package.fetch_versions couldn't connect to:", e.url,
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -562,7 +562,7 @@ def _urlopen(req, *args, **kwargs):
 
 
 def find_versions_of_archive(
-        archive_urls, list_url=None, list_depth=0, concurrency=32
+    archive_urls, list_url=None, list_depth=0, concurrency=32, reference_package=None
 ):
     """Scrape web pages for new versions of a tarball.
 
@@ -577,6 +577,10 @@ def find_versions_of_archive(
         list_depth (int): max depth to follow links on list_url pages.
             Defaults to 0.
         concurrency (int): maximum number of concurrent requests
+        reference_package (spack.package.Package or None): a spack package
+            used as a reference for url detection.  Uses the url_for_version
+            method on the package to produce reference urls which, if found,
+            are preferred.
     """
     if not isinstance(archive_urls, (list, tuple)):
         archive_urls = [archive_urls]
@@ -638,11 +642,26 @@ def find_versions_of_archive(
     # Walk through archive_url links first.
     # Any conflicting versions will be overwritten by the list_url links.
     versions = {}
+    matched = set()
     for url in archive_urls + sorted(links):
         if any(re.search(r, url) for r in regexes):
             try:
                 ver = spack.url.parse_version(url)
+                if ver in matched:
+                    continue
                 versions[ver] = url
+                # prevent this version from getting overwritten
+                if url in archive_urls:
+                    matched.add(ver)
+                elif reference_package is not None:
+                    if url == reference_package.url_for_version(ver):
+                        matched.add(ver)
+                else:
+                    extrapolated_urls = [
+                        spack.url.substitute_version(u, ver) for u in archive_urls
+                    ]
+                    if url in extrapolated_urls:
+                        matched.add(ver)
             except spack.url.UndetectableVersionError:
                 continue
 


Test patch: