Repository: spack/spack

Issue:
`spack checksum` support for GitHub URLs is broken
### Steps to reproduce

As requested by @adamjstewart in #34162, open this as separate issue:

When trying to see the different versions available upstream for a package on github, `spack checksum` does not detect GitHub URLs automatically.

Reprocuce e.g. with package `dcm2niix`:
```
$ spack checksum dcm2niix
==> Found 1 version of dcm2niix:
  
  1.0.20210317  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20210317.tar.gz

==> Fetching https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20210317.tar.gz

    version("1.0.20210317", sha256="42fb22458ebfe44036c3d6145dacc6c1dc577ebbb067bedc190ed06f546ee05a")
```
Expected output:
```
» spack checksum dcm2niix    
==> Found 10 versions of dcm2niix:
  
  1.0.20220720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz
  1.0.20211006  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20211006.tar.gz
  1.0.20210317  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20210317.tar.gz
  1.0.20201102  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20201102.tar.gz
  1.0.20200331  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20200331.tar.gz
  1.0.20190902  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190902.tar.gz
  1.0.20190720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190720.tar.gz
  1.0.20181125  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181125.tar.gz
  1.0.20181114  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181114.tar.gz
  1.0.20180622  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20180622.tar.gz
...
```


### Error message

_No response_

### Information on your system

* **Spack:** 0.20.0.dev0 (066ec316047cb6dfc92a3fef9980718ade47d4aa)
* **Python:** 3.11.0
* **Platform:** linux-fedora37-skylake
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I think @tldahlgren recently edited these files and may know which commit might have broken this. If not let me know and I can do some digging after my prelims.
After some digging, I noted `list_depth` defaulting to `0` is an issue.  Per https://spack.readthedocs.io/en/latest/packaging_guide.html#list-depth, I added `list_url` and `list_depth` and `spack checksum` and it worked BUT *not consistently* as shown below:

```
$ spack checksum dcm2niix
==> Found 10 versions of dcm2niix:
  
  1.0.20220720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz
  1.0.20211006  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20211006.tar.gz
  1.0.20210317  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20210317.tar.gz
  1.0.20201102  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20201102.tar.gz
  1.0.20200331  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20200331.tar.gz
  1.0.20190902  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190902.tar.gz
  1.0.20190720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190720.tar.gz
  1.0.20181125  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181125.tar.gz
  1.0.20181114  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181114.tar.gz
  1.0.20180622  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20180622.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) q
==> Error: Aborted.
$ spack checksum dcm2niix
==> Found 10 versions of dcm2niix:
  
  1.0.20220720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz
  1.0.20211006  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20211006.tar.gz
  1.0.20210317  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20210317.tar.gz
  1.0.20201102  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20201102.tar.gz
  1.0.20200331  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20200331.tar.gz
  1.0.20190902  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190902.tar.gz
  1.0.20190720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190720.tar.gz
  1.0.20181125  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181125.tar.gz
  1.0.20181114  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181114.tar.gz
  1.0.20180622  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20180622.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) q
==> Error: Aborted.
$ spack checksum dcm2niix
==> Found 10 versions of dcm2niix:
  
  1.0.20220720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz
  1.0.20211006  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20211006.tar.gz
  1.0.20210317  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20210317.tar.gz
  1.0.20201102  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20201102.tar.gz
  1.0.20200331  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20200331.tar.gz
  1.0.20190902  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190902.tar.gz
  1.0.20190720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20190720.tar.gz
  1.0.20181125  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181125.tar.gz
  1.0.20181114  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20181114.tar.gz
  1.0.20180622  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20180622.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) 1
==> Fetching https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz

    version("1.0.20220720", sha256="a095545d6d70c5ce2efd90dcd58aebe536f135410c12165a9f231532ddab8991")

$ spack checksum dcm2niix
==> Found 1 version of dcm2niix:
  
  1.0.20220720  https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz

==> Fetching https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz

    version("1.0.20220720", sha256="a095545d6d70c5ce2efd90dcd58aebe536f135410c12165a9f231532ddab8991")
```
`list_depth` shouldn't matter, https://github.com/rordenlab/dcm2niix/releases lists all release tarballs, there's no need to spider down through different pages.
> `list_depth` shouldn't matter, https://github.com/rordenlab/dcm2niix/releases lists all release tarballs, there's no need to spider down through different pages.

It apparently does based on the URLs that are being `spider`ed.  At least that's what I'm seeing for this example.  And it's the https://github.com/rordenlab/dcm2niix/archive/refs/tags/ that show more versions.
Note the behavior above may also be related to switching `_spider` to use `ThreadPool` as a context manager.
> It apparently does based on the URLs that are being spidered.

The https://github.com/rordenlab/dcm2niix/releases page contains the links we are looking for, spidering shouldn't be necessary. I think what's happening is that it's going from https://github.com/rordenlab/dcm2niix/releases to https://github.com/rordenlab/dcm2niix/tags and getting the links we want from there instead. I'm not sure why that would be necessary.

> And it's the https://github.com/rordenlab/dcm2niix/archive/refs/tags/ that show more versions.

We usually try to prefer releases over tags (especially for Autotools packages) since tags are auto-generated and don't contain built files like `configure`. Otherwise, we could modify the logic in `lib/spack/spack/url.py` to prefer `tags` over `releases`.

> Note the behavior above may also be related to switching _spider to use ThreadPool as a context manager.

What commit was this? Should be easy to test before and after.
> > It apparently does based on the URLs that are being spidered.
> 
> The https://github.com/rordenlab/dcm2niix/releases page contains the links we are looking for, spidering shouldn't be necessary. I think what's happening is that it's going from https://github.com/rordenlab/dcm2niix/releases to https://github.com/rordenlab/dcm2niix/tags and getting the links we want from there instead. I'm not sure why that would be necessary.
> 

The links to the .tar.gz files scraped from the releases page point to the tags page.  For example, https://github.com/rordenlab/dcm2niix/archive/refs/tags/v1.0.20220720.tar.gz.

> 
> What commit was this? Should be easy to test before and after.

I agree it should be easy to test pre-commit.  I got a bit caught up in experimenting with changes related to use of `ThreadPool`.
Any updates on this? Would love to be able to use `spack checksum` again for GitHub packages.
Hey all, was there any movement on this issue? I'm noticing that as more packages (e.g. abyss) move from GitHub archive tarballs to release tarballs that `spack version` is no longer finding versions for the packages. I believe this is going to be a growing issue as more packages move away from archive tarballs due to the lack of a permanent checksum.
Ran into this here too. The releases page has the asset links encapsulated in a fragment, e.g. `<include-fragment loading="lazy" src="https://github.com/VcDevel/Vc/releases/expanded_assets/1.4.3" >` at https://github.com/VcDevel/Vc/releases is linking to https://github.com/VcDevel/Vc/releases/expanded_assets/1.4.3. More at https://github.github.io/include-fragment-element/.

Resolving patch:
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -75,7 +75,7 @@ class LinkParser(HTMLParser):
     links.  Good enough for a really simple spider."""
 
     def __init__(self):
-        HTMLParser.__init__(self)
+        super().__init__()
         self.links = []
 
     def handle_starttag(self, tag, attrs):
@@ -85,6 +85,21 @@ def handle_starttag(self, tag, attrs):
                     self.links.append(val)
 
 
+class IncludeFragmentParser(HTMLParser):
+    """This parser takes an HTML page and selects the include-fragments,
+    used on GitHub, https://github.github.io/include-fragment-element."""
+
+    def __init__(self):
+        super().__init__()
+        self.links = []
+
+    def handle_starttag(self, tag, attrs):
+        if tag == "include-fragment":
+            for attr, val in attrs:
+                if attr == "src":
+                    self.links.append(val)
+
+
 def read_from_url(url, accept_content_type=None):
     if isinstance(url, str):
         url = urllib.parse.urlparse(url)
@@ -550,9 +565,38 @@ def _spider(url, collect_nested):
             page = codecs.getreader("utf-8")(response).read()
             pages[response_url] = page
 
-            # Parse out the links in the page
+            # Parse out the include-fragments in the page
+            # https://github.github.io/include-fragment-element
+            include_fragment_parser = IncludeFragmentParser()
+            include_fragment_parser.feed(page)
+
+            fragments = set()
+            while include_fragment_parser.links:
+                raw_link = include_fragment_parser.links.pop()
+                abs_link = url_util.join(response_url, raw_link.strip(), resolve_href=True)
+
+                try:
+                    # This seems to be text/html, though text/fragment+html is also used
+                    fragment_response_url, _, fragment_response = read_from_url(
+                        abs_link, "text/html"
+                    )
+                except Exception as e:
+                    msg = f"Error reading fragment: {(type(e), str(e))}:{traceback.format_exc()}"
+                    tty.debug(msg)
+
+                if not fragment_response_url or not fragment_response:
+                    continue
+
+                fragment = codecs.getreader("utf-8")(fragment_response).read()
+                fragments.add(fragment)
+
+                pages[fragment_response_url] = fragment
+
+            # Parse out the links in the page and all fragments
             link_parser = LinkParser()
             link_parser.feed(page)
+            for fragment in fragments:
+                link_parser.feed(fragment)
 
             while link_parser.links:
                 raw_link = link_parser.links.pop()


Test patch:
diff --git a/lib/spack/spack/test/data/web/fragment.html b/lib/spack/spack/test/data/web/fragment.html
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/data/web/fragment.html
@@ -0,0 +1 @@
+<a href="foo-5.0.0.tar.gz">foo-5.0.0.tar.gz</a>
diff --git a/lib/spack/spack/test/data/web/index_with_fragment.html b/lib/spack/spack/test/data/web/index_with_fragment.html
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/data/web/index_with_fragment.html
@@ -0,0 +1,13 @@
+<html>
+  <head>
+    This is the root page.
+  </head>
+  <body>
+    This is a page with an include-fragment element.
+
+    <script type="module" src="https://unpkg.com/@github/include-fragment-element@latest?module"></script>
+    <include-fragment src="fragment.html">
+      <p>Loading...</p>
+    </include-fragment>
+  </body>
+</html>
diff --git a/lib/spack/spack/test/web.py b/lib/spack/spack/test/web.py
--- a/lib/spack/spack/test/web.py
+++ b/lib/spack/spack/test/web.py
@@ -31,6 +31,8 @@ def _create_url(relative_url):
 page_3 = _create_url("3.html")
 page_4 = _create_url("4.html")
 
+root_with_fragment = _create_url("index_with_fragment.html")
+
 
 @pytest.mark.skipif(sys.platform == "win32", reason="Not supported on Windows (yet)")
 @pytest.mark.parametrize(
@@ -143,6 +145,14 @@ def test_find_exotic_versions_of_archive_3():
     assert ver("4.5-rc5") in versions
 
 
+@pytest.mark.skipif(sys.platform == "win32", reason="Not supported on Windows (yet)")
+def test_find_versions_of_archive_with_fragment():
+    versions = spack.util.web.find_versions_of_archive(
+        root_tarball, root_with_fragment, list_depth=0
+    )
+    assert ver("5.0.0") in versions
+
+
 def test_get_header():
     headers = {"Content-type": "text/plain"}