Repository: spack/spack

Issue:
SZ: bad source fetch url for older versions
Unable to fetch older versions of `sz` using `spack@develop` (6b70597271232e442b9e729b45adaee02bccf8c0 from 6-Oct-2020):

* `sz@2.0.2.0`
* `sz@1.4.12.3`

Probably introduced via eabfea2e49f6db473401b29ca80966f41a7e7c23

```
$> spack install sz@2.0.2.0
...
==> Installing sz
==> No binary for sz found: installing from source
==> Error: FetchError: All fetchers failed

/opt/spack/lib/spack/spack/package.py:1262, in do_fetch:
       1259                                 self.spec.format('{name}{@version}'), ck_msg)
       1260
       1261        self.stage.create()
  >>   1262        err_msg = None if not self.manual_download else self.download_instr
       1263        self.stage.fetch(mirror_only, err_msg=err_msg)
       1264        self._fetch_time = time.time() - start_time
       1265
```

Same outcome for `sz@1.4.12.3`

@scottwittenburg @disheng222 @robertu94 


Discussion:
Hi,
I checked that the .tar.gz package on the SZ website is available to
download and its checksum in the package.py is also correct for 1.4.12.3.
I have no idea what I should do.

Best,
Sheng

On Tue, Oct 6, 2020 at 4:30 PM eugeneswalker <notifications@github.com>
wrote:

> Unable to fetch older versions of sz using spack@develop (6b70597
> <https://github.com/spack/spack/commit/6b70597271232e442b9e729b45adaee02bccf8c0>
> from 6-Oct-2020):
>
>    - sz@1.4.12.3
>    - sz@2.0.2.0
>
> Probably introduced via eabfea2
> <https://github.com/spack/spack/commit/eabfea2e49f6db473401b29ca80966f41a7e7c23>
>
> $> spack install sz@1.4.12.3
> ...
> ==> Installing sz
> ==> No binary for sz found: installing from source
> ==> Error: FetchError: All fetchers failed
>
> /opt/spack/lib/spack/spack/package.py:1262, in do_fetch:
>        1259                                 self.spec.format('{name}{@version}'), ck_msg)
>        1260
>        1261        self.stage.create()
>   >>   1262        err_msg = None if not self.manual_download else self.download_instr
>        1263        self.stage.fetch(mirror_only, err_msg=err_msg)
>        1264        self._fetch_time = time.time() - start_time
>        1265
>
> @scottwittenburg <https://github.com/scottwittenburg> @disheng222
> <https://github.com/disheng222> @robertu94 <https://github.com/robertu94>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/19213>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACK3KSMF5PXCOWBJZULYFFTSJOEBFANCNFSM4SGSK62Q>
> .
>

@eugeneswalker it works for me.  Perhaps a spurious failure?

```console
[runderwood@voyager experiments]$ spack install sz@2.0.2.0
==> cmake@3.18.1 : externally installed in /usr
[+] /usr
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-gentoo2-skylake/gcc-10.1.0/zlib-1.2.11-x5whpcbwrkid4b7itid7euanufvg4pyy
==> Installing zstd
==> No binary for zstd found: installing from source
==> Fetching https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/_source-cache/archive/73/734d1f565c42f691f8420c8d06783ad818060fc390dee43ae0a89f86d0a4f8c2.tar.gz
########################################################################################################################################### 100.0%
==> zstd: Executing phase: 'edit'
==> zstd: Executing phase: 'build'
==> zstd: Executing phase: 'install'
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-gentoo2-skylake/gcc-10.1.0/zstd-1.4.5-ymj3vwmm6hk6n2otwsdl4blvprtbxxr5
==> Installing sz
==> No binary for sz found: installing from source
==> Fetching https://github.com/disheng222/SZ/archive/v2.0.2.0.tar.gz
########################################################################################################################################### 100.0%            -=O=-              #         #         #          #                                                                                  
==> sz: Executing phase: 'cmake'
==> sz: Executing phase: 'build'
==> sz: Executing phase: 'install'
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-gentoo2-skylake/gcc-10.1.0/sz-2.0.2.0-qekt6lyx3dzot7ggcm7vkjmmigyorvrq
[runderwood@voyager experiments]$ spack install sz@1.4.12.3 
==> cmake@3.18.1 : externally installed in /usr
[+] /usr
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-gentoo2-skylake/gcc-10.1.0/zlib-1.2.11-x5whpcbwrkid4b7itid7euanufvg4pyy
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-gentoo2-skylake/gcc-10.1.0/zstd-1.4.5-ymj3vwmm6hk6n2otwsdl4blvprtbxxr5
==> Installing sz
==> No binary for sz found: installing from source
==> Fetching https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz
########################################################################################################################################### 100.0%########################################################################################################################################### 100.0%       -=O=-#      #       #         #                                                                                                           
==> sz: Executing phase: 'cmake'
==> sz: Executing phase: 'build'
==> sz: Executing phase: 'install'
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-gentoo2-skylake/gcc-10.1.0/sz-1.4.12.3-mlsltrfajlmq6ph4el5cjavfeyfn35zo
```
> ==> Installing sz
> ==> No binary for sz found: installing from source
> ==> Fetching https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz

Your Spack instance is fetching from:
* `https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz`

That source url was changed to:
* https://github.com/szcompressor/SZ/releases/download/

via https://github.com/spack/spack/commit/eabfea2e49f6db473401b29ca80966f41a7e7c23

I think if you make sure your Spack instance is checked out at a point past this commit then you will observe the problem.


   - https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz should be
   available too, because github will automatically resolve the redirecting.

Best,
Sheng

On Tue, Oct 6, 2020 at 4:59 PM eugeneswalker <notifications@github.com>
wrote:

> ==> Installing sz
> ==> No binary for sz found: installing from source
> ==> Fetching https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz
>
> You're Spack is fetching from:
>
>    - https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz
>
> That source url was changed to:
>
>    - https://github.com/szcompressor/SZ/releases/download/
>
> via eabfea2
> <https://github.com/spack/spack/commit/eabfea2e49f6db473401b29ca80966f41a7e7c23>
>
> I think if you make sure your Spack instance is checked out at a point
> past this commit then you will observe the problem.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/issues/19213#issuecomment-704575905>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACK3KSJQMG77VWPCUD2LVVTSJOHNZANCNFSM4SGSK62Q>
> .
>

> - https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz should be available too, because github will automatically resolve the redirecting

@adamjstewart Do you know anything about this behavior? It isn't working for me and I've tried on:
* RHEL7 x86_64 and ppc64le
* Ubuntu 18.04 x86_64 and ppc64le

Also the versions of `SZ` needed in the E4S `gitlab.spack.io` pipeline are failing because of this fetch error.

Perhaps Spack is not doing the fetch in a way that can handle the redirection @disheng222 mentions?
> - https://github.com/disheng222/SZ/archive/v1.4.12.3.tar.gz should be available too, because github will automatically resolve the redirecting.

```
$> spack fetch sz@1.4.12.3
...
==> [2020-10-06-21:56:29.213582] Fetching from https://github.com/szcompressor/SZ/releases/download/v1.4.12.3/sz-1.4.12.3.tar.gz failed.
==> [2020-10-06-21:56:29.213785] FetchError: All fetchers failed
==> [2020-10-06-21:56:29.213911] Error: All fetchers failed
Traceback (most recent call last):
  File "/opt/spack/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/opt/spack/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/opt/spack/lib/spack/spack/cmd/fetch.py", line 53, in fetch
    package.do_fetch()
  File "/opt/spack/lib/spack/spack/package.py", line 1263, in do_fetch
    self.stage.fetch(mirror_only, err_msg=err_msg)
  File "/opt/spack/lib/spack/spack/util/pattern.py", line 68, in getter
    getattr(item, self.name)(*args, **kwargs)
  File "/opt/spack/lib/spack/spack/stage.py", line 490, in fetch
    raise fs.FetchError(err_msg or 'All fetchers failed', None)
spack.fetch_strategy.FetchError: All fetchers failed
```

Notice the URL used is `https://github.com/szcompressor/SZ/releases/download/v1.4.12.3/sz-1.4.12.3.tar.gz` -- this is the URL being tried, and it is failing. 

Trying manually:
```
$> wget https://github.com/szcompressor/SZ/releases/download/v1.4.12.3/sz-1.4.12.3.tar.gz
--2020-10-06 22:07:04--  https://github.com/szcompressor/SZ/releases/download/v1.4.12.3/sz-1.4.12.3.tar.gz
Resolving github.com (github.com)... 192.30.255.112
Connecting to github.com (github.com)|192.30.255.112|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-10-06 22:07:05 ERROR 404: Not Found.
```

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/sz/package.py b/var/spack/repos/builtin/packages/sz/package.py
--- a/var/spack/repos/builtin/packages/sz/package.py
+++ b/var/spack/repos/builtin/packages/sz/package.py
@@ -10,12 +10,19 @@ class Sz(CMakePackage):
     """Error-bounded Lossy Compressor for HPC Data"""
 
     homepage = "https://szcompressor.org"
-    url      = "https://github.com/szcompressor/SZ/releases/download/v2.1.10/sz-2.1.10.tar.gz"
+    url      = "https://github.com/szcompressor/SZ/archive/v2.1.10.tar.gz"
     git      = "https://github.com/szcompressor/sz"
     maintainers = ['disheng222', 'robertu94']
 
+    def url_for_version(self, version):
+        """provide url to ensure that download counting via github releases
+        works accurately"""
+        url = "https://github.com/szcompressor/SZ/releases/download/v{0}/SZ-{0}.tar.gz"
+        return url.format(version)
+
     version('master', branch='master')
-    version('2.1.10', sha256='3aba7619bdb5412218f162696f946c9d3a3df5acf128ddc685b21e45c11f6ae3')    
+    version('2.1.10', sha256='3aba7619bdb5412218f162696f946c9d3a3df5acf128ddc685b21e45c11f6ae3',
+            url="https://github.com/szcompressor/SZ/releases/download/v2.1.10/sz-2.1.10.tar.gz")
     version('2.1.9', sha256='491724ff1c0eaaab5e1a7a28e36aba6da9dcbeddb29d8d21a6d1388383d4891e')
     version('2.1.8.3', sha256='be94f3c8ab03d6849c59a98e0ebf80816a6b8d07a1d762a4b285498acb2f3871')
     version('2.1.8.1', sha256='a27c9c9da16c9c4232c54813ba79178945f47609043f11501d49a171e47d3f46')


Test patch: