Repository: spack/spack

Issue:
spack checksum prefers version-specific url over global url
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->

I wanted to update `py-setuptools-rust` to the newest version and ran into something that might be a bug regarding the chosen download path.

I updated the package to use pypi as default but wanted to keep the existing checksum for the old package (to avoid breaking anything), so my `package.py` looks like this:

```python
from spack import *


class PySetuptoolsRust(PythonPackage):
    """Setuptools rust extension plugin."""

    homepage = "https://github.com/PyO3/setuptools-rust"
    pypi = "setuptools-rust/setuptools-rust-0.12.1.tar.gz"

    version('0.12.1', sha256='647009e924f0ae439c7f3e0141a184a69ad247ecb9044c511dabde232d3d570e')
    version('0.10.6', sha256='1446d3985e4aaf4cc679fda8a48a73ac1390b627c8ae1bebe7d9e08bb3b33769',
            url="https://github.com/PyO3/setuptools-rust/archive/v0.10.6.tar.gz")

    depends_on('python@3.6:', when='@0.12:', type=('build', 'run'))
    depends_on('py-setuptools@46.1:', when='@0.12:', type='build')
    depends_on('py-setuptools', when='@:0.11', type='build')
    depends_on('py-semantic-version@2.6.0:', type=('build', 'run'))
    depends_on('py-toml@0.9.0:', type=('build', 'run'))
    depends_on('rust', type='run')
```

When running `spack checksum py-setuptools-rust` for the latest package, spack still uses the github url
```
==> How many would you like to checksum? (default is 1, q to abort)  
==> Fetching https://github.com/PyO3/setuptools-rust/archive/refs/tags/v0.12.1.tar.gz
```

Wheras if I remove `url=....` from version 0.10.6, version 0.12.1 is correctly fetched from pypi. But this means that 0.10.6 is broken since it does not exist there (and if so the checksum would be wrong).
```
==> How many would you like to checksum? (default is 1, q to abort)
==> Fetching https://files.pythonhosted.org/packages/12/22/6ba3031e7cbd6eb002e13ffc7397e136df95813b6a2bd71ece52a8f89613/setuptools-rust-0.12.1.tar.gz#sha256=647009e924f0ae439c7f3e0141a184a69ad247ecb9044c511dabde232d3d570e
```

I thought the download mechanism works like this:
1. try the default url (here pypi)
2. if a version specific url is defined use this one instead only for that version

Do I understand something wrong here or is this a bug?


A workaround is to define `url_for_version` in this case but I still wanted to report the behaviour

### Information on your system

<!-- Please include the output of `spack debug report` -->
* **Spack:** 0.13.4-8544-c8868f1922
* **Python:** 3.5.3
* **Platform:** linux-debian9-piledriver
* **Concretizer:** clingo


<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
Spack's URL parsing stuff is very complicated. Basically, there are two separate tasks and two separate mechanisms to handle them:

1. downloading a specific version (`spack fetch`, `url_for_version`)
2. finding new versions (`spack versions`, `spack checksum`)

These two tasks share little-to-no code. When you use `pypi`, this just sets `url` for you. So this bug has nothing to do with `pypi`, and can be reproduced by using different `url` arguments for each version with or without a global `url` attribute. The bug you've found affects task 2 but not task 1.

@tgamblin fixed this bug for task 1 a while back. Not sure exactly what is causing it to choose the version-specific `url` argument for `spack versions/checksum` instead of the global `url` attribute. Maybe @tgamblin knows.

Resolving patch:
diff --git a/lib/spack/spack/cmd/checksum.py b/lib/spack/spack/cmd/checksum.py
--- a/lib/spack/spack/cmd/checksum.py
+++ b/lib/spack/spack/cmd/checksum.py
@@ -63,6 +63,16 @@ def checksum(parser, args):
         if not url_dict:
             tty.die("Could not find any versions for {0}".format(pkg.name))
 
+        # And ensure the specified version URLs take precedence, if available
+        try:
+            explicit_dict = {}
+            for v in pkg.versions:
+                if not v.isdevelop():
+                    explicit_dict[v] = pkg.url_for_version(v)
+            url_dict.update(explicit_dict)
+        except spack.package.NoURLError:
+            pass
+
     version_lines = spack.stage.get_checksums_for_versions(
         url_dict, pkg.name, keep_stage=args.keep_stage,
         batch=(args.batch or len(args.versions) > 0 or len(url_dict) == 1),


Test patch: