Repository: easybuilders/easybuild-framework

Issue:
ElementTree cannot handle package entries from PyPI with attribute data-yanked
Downloading the sources of a Python package from PyPI will fail if some of the entries listed in the HTML file retrieved from PyPI's simple API have the extra attribute `data-yanked`.

More info on yanked packages: https://www.python.org/dev/peps/pep-0592/

This happens with
* Python v2.7.5 from CentOS 7.7 
* Python v2.7.16 from Easybuild
* Python v3.7.4 from Easybuild

At the present time, this can be tested with IPython-7.13.0-foss-2020a-Python-3.8.2.eb from `2020a` branch
```
$ eb4 --debug IPython-7.13.0-foss-2020a-Python-3.8.2.eb 
== temporary log file in case of crash /tmp/eb-wh9MvS/easybuild-lPPWux.log
== resolving dependencies ...
== processing EasyBuild easyconfig /theia/home/brussel/101/vsc10122/src/easybuild-easyconfigs/easybuild/easyconfigs/IPython-7.13.0-foss-2020a-Python-3.8.2.eb
== building and installing IPython/7.13.0-foss-2020a-Python-3.8.2...
== fetching files...
/tmp/eb-wh9MvS/tmp2GjUvY/ipython_urls.html
ERROR: Traceback (most recent call last):
  File "/theia/home/brussel/101/vsc10122/src/easybuild-framework/easybuild/main.py", line 114, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 3225, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 3129, in run_all_steps
    self.run_step(step_name, step_methods)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 2968, in run_step
    step_method(self)()
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 1795, in fetch_step
    self.exts = self.fetch_extension_sources(skip_checksums=skip_checksums)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 520, in fetch_extension_sources
    src_fn = self.obtain_file(fn, extension=True, urls=source_urls, force_download=force_download)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/framework/easyblock.py", line 727, in obtain_file
    alt_url = derive_alt_pypi_url(fullurl)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/tools/filetools.py", line 520, in derive_alt_pypi_url
    cand_urls = pypi_source_urls(pkg_name)
  File "/user/brussel/101/vsc10122/src/easybuild-framework/easybuild/tools/filetools.py", line 499, in pypi_source_urls
    parsed_html = ElementTree.parse(urls_html)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1182, in parse
    tree.parse(source, parser)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
    parser.feed(data)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1642, in feed
    self._raiseerror(v)
  File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 237, column 267
```

Line 237 of the HTML file is the first entry with the `data-yanked` attribute
```
[...]
    <a href="https://files.pythonhosted.org/packages/5b/e3/4b3082bd7f6908af828561b0129b5064bff4a13e6acadb321fc2d939a605/ipython-7.0.1.tar.gz#sha256=9fed506c3772c875a3048bc134a25e6f5e997b1569b2636f6a5d891f34cbfd46" data-requires-python="&gt;=3.5">ipython-7.0.1.tar.gz</a><br/>
    <a href="https://files.pythonhosted.org/packages/ea/56/d75d25b30936bff3f6b4d7b80cfd399b703e2162f31afee926f66ff77b13/ipython-7.1.0-py3-none-any.whl#sha256=107859b062f6360d7bf0063c275511c638b633622954ba9a401ffa4a0c6aba1b" data-requires-python="&gt;=3.5" data-yanked>ipython-7.1.0-py3-none-any.whl</a><br/>
```



Discussion:
@lexming I'm a bit surprised that I'm not seeing this...

Can you share the output of `rpm -qf /usr/lib64/python2.7/xml/etree/ElementTree.py`?
OK, nevermind, easy to reproduce locally...

This occurs for Python packages that have yanked releases, see https://pypi.org/help/#yanked .

Resolving patch:
diff --git a/easybuild/tools/filetools.py b/easybuild/tools/filetools.py
--- a/easybuild/tools/filetools.py
+++ b/easybuild/tools/filetools.py
@@ -495,7 +495,13 @@ def pypi_source_urls(pkg_name):
         _log.debug("Failed to download %s to determine available PyPI URLs for %s", simple_url, pkg_name)
         res = []
     else:
-        parsed_html = ElementTree.parse(urls_html)
+        urls_txt = read_file(urls_html)
+
+        # ignore yanked releases (see https://pypi.org/help/#yanked)
+        # see https://github.com/easybuilders/easybuild-framework/issues/3301
+        urls_txt = re.sub(r'<a.*?data-yanked.*?</a>', '', urls_txt)
+
+        parsed_html = ElementTree.ElementTree(ElementTree.fromstring(urls_txt))
         if hasattr(parsed_html, 'iter'):
             res = [a.attrib['href'] for a in parsed_html.iter('a')]
         else:


Test patch:
diff --git a/test/framework/filetools.py b/test/framework/filetools.py
--- a/test/framework/filetools.py
+++ b/test/framework/filetools.py
@@ -1303,6 +1303,15 @@ def test_pypi_source_urls(self):
         # more than 50 releases at time of writing test, which always stay there
         self.assertTrue(len(res) > 50)
 
+        # check for Python package that has yanked releases,
+        # see https://github.com/easybuilders/easybuild-framework/issues/3301
+        res = ft.pypi_source_urls('ipython')
+        self.assertTrue(isinstance(res, list) and res)
+        prefix = 'https://pypi.python.org/packages'
+        for entry in res:
+            self.assertTrue(entry.startswith(prefix), "'%s' should start with '%s'" % (entry, prefix))
+            self.assertTrue('ipython' in entry, "Pattern 'ipython' should be found in '%s'" % entry)
+
     def test_derive_alt_pypi_url(self):
         """Test derive_alt_pypi_url() function."""
         url = 'https://pypi.python.org/packages/source/e/easybuild/easybuild-2.7.0.tar.gz'