Repository: streamlink/streamlink

Issue:
Remove `streamlink.plugin.api.utils.itertags`
[`streamlink.plugin.api.utils.itertags`](https://github.com/streamlink/streamlink/blob/3.2.0/src/streamlink/plugin/api/utils.py#L16-L28) has become obsolete ever since `lxml` was added as a dependency to Streamlink for parsing HTML.

`itertags` is a hacky implementation via regexes, which is not only slow, but it's also impossible to correctly parse HTML nodes with regular expressions, so it shouldn't be used when better and much faster solutions are available. It also always requires unescaping tag values, which is annoying.

We've already updated and replaced lots of plugins which were previously using it, but there are still some left:
```
$ GIT_PAGER=cat git grep -F 'from streamlink.plugin.api.utils import' a1ce471f
a1ce471f:src/streamlink/plugins/cdnbg.py:from streamlink.plugin.api.utils import itertags
a1ce471f:src/streamlink/plugins/facebook.py:from streamlink.plugin.api.utils import itertags
a1ce471f:src/streamlink/plugins/funimationnow.py:from streamlink.plugin.api.utils import itertags
a1ce471f:src/streamlink/plugins/senategov.py:from streamlink.plugin.api.utils import itertags
a1ce471f:src/streamlink/plugins/vrtbe.py:from streamlink.plugin.api.utils import itertags
a1ce471f:tests/test_plugin_utils.py:from streamlink.plugin.api.utils import itertags
```

- [x] cdnbg
- [x] facebook
- [x] funimationnow
- [x] senategov
- [x] vrtbe

Once every last plugin has been updated, the entire `streamlink.plugin.api.utils` module can be removed, as it only contains the `itertags` function and some other useless export aliases which are not even used anymore in Streamlink's codebase.

If we care about plugin-API stability (something which has never been discussed), removing this would be considered a breaking change. Since we've just dropped py36, that's something which could be included in the 4.0.0 release.


Discussion:
Regarding facebook, as I already posted on Gitter/Matrix, I have a dev branch with a plugin rewrite here:
https://github.com/streamlink/streamlink/compare/master...bastimeyer:plugins/facebook/rewrite
I lost interest though, because I don't have a FB account, and all new ones I've created got blocked pretty much immediately. That's why my rewrite is incomplete and basically just a translation and improvement of the old logic, which means that the changes are most likely not working.

----

Regarding VRTbe, this site uses DRM for both HLS and DASH streams. The last plugin update was in June 2018 via e70053511331db579e4adde085ebf049d34e4ecf, and the commit message says that it only applies to high quality streams. I can't confirm this, for both live and vod streams. And from what I can see, [the current plugin implementation](https://github.com/streamlink/streamlink/blob/3.2.0/src/streamlink/plugins/vrtbe.py#L44) is not working (no HTMLElement with a `vrtvideo` class). The livestream page also lists all live streams at once, which makes it pretty much useless.

- https://www.vrt.be/vrtnu/livestream/
- https://www.vrt.be/vrtnu/kanalen/een/

```
#EXT-X-SESSION-KEY:METHOD=SAMPLE-AES,URI="skd://...",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1"
```
```xml
      <!-- Common Encryption -->
      <ContentProtection
        xmlns="urn:mpeg:dash:schema:mpd:2011"
        xmlns:cenc="urn:mpeg:cenc:2013"
        schemeIdUri="urn:mpeg:dash:mp4protection:2011"
        value="cenc"
        cenc:default_KID="F0A275F8-F9A1-EDD5-546C-4ABB1DB127AA">
      </ContentProtection>
      <!-- PlayReady -->
      <ContentProtection
        xmlns="urn:mpeg:dash:schema:mpd:2011"
        schemeIdUri="urn:uuid:9A04F079-9840-4286-AB92-E65BE0885F95"
        value="MSPR 2.0">
      </ContentProtection>
      <!-- Widevine -->
      <ContentProtection
        xmlns="urn:mpeg:dash:schema:mpd:2011"
        schemeIdUri="urn:uuid:EDEF8BA9-79D6-4ACE-A3C8-27DCD51D21ED">
      </ContentProtection>
      <!-- Marlin -->
      <ContentProtection
        xmlns="urn:mpeg:dash:schema:mpd:2011"
        schemeIdUri="urn:uuid:5E629AF5-38DA-4063-8977-97FFBD9902D4">
        <MarlinContentIds xmlns="urn:marlin:mas:1-0:services:schemas:mpd">
          <MarlinContentId>urn:marlin:kid:f0a275f8f9a1edd5546c4abb1db127aa</MarlinContentId>
        </MarlinContentIds>
      </ContentProtection>
```
For VRTbe I've merged the PR removing it. For FB there's nothing I can do either here as I don't use FB nor do I have an account. It's been broken for quite some time and the number of issues we get related to FB not working is relatively low so I'd be fine with removing it. If someone who regularly uses FB is willing to develop and maintain that plugin we can reconsider adding it in the future.

Resolving patch:
diff --git a/src/streamlink/plugin/api/utils.py b/src/streamlink/plugin/api/utils.py
deleted file mode 100644
--- a/src/streamlink/plugin/api/utils.py
+++ /dev/null
@@ -1,28 +0,0 @@
-"""Useful wrappers and other tools."""
-import re
-from collections import namedtuple
-
-from streamlink.utils.parse import parse_json, parse_qsd as parse_query, parse_xml
-
-__all__ = ["parse_json", "parse_xml", "parse_query"]
-
-
-tag_re = re.compile(r'''(?=<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?)''',
-                    re.MULTILINE | re.DOTALL)
-attr_re = re.compile(r'''\s*(?P<key>[\w-]+)\s*(?:=\s*(?P<quote>["']?)(?P<value>.*?)(?P=quote)\s*)?''')
-Tag = namedtuple("Tag", "tag attributes text")
-
-
-def itertags(html, tag):
-    """
-    Brute force regex based HTML tag parser. This is a rough-and-ready searcher to find HTML tags when
-    standards compliance is not required. Will find tags that are commented out, or inside script tag etc.
-
-    :param html: HTML page
-    :param tag: tag name to find
-    :return: generator with Tags
-    """
-    for match in tag_re.finditer(html):
-        if match.group("tag") == tag:
-            attrs = {a.group("key").lower(): a.group("value") for a in attr_re.finditer(match.group("attr"))}
-            yield Tag(match.group("tag"), attrs, match.group("inner"))


Test patch:
diff --git a/tests/test_plugin_utils.py b/tests/test_plugin_utils.py
deleted file mode 100644
--- a/tests/test_plugin_utils.py
+++ /dev/null
@@ -1,97 +0,0 @@
-import sys
-import unittest
-
-from streamlink.plugin.api.utils import itertags
-
-
-def unsupported_versions_1979():
-    """Unsupported python versions for itertags
-       3.7.0 - 3.7.2 and 3.8.0a1
-       - https://github.com/streamlink/streamlink/issues/1979
-       - https://bugs.python.org/issue34294
-    """
-    v = sys.version_info
-    return (v.major == 3) and (
-        # 3.7.0 - 3.7.2
-        (v.minor == 7 and v.micro <= 2)
-        # 3.8.0a1
-        or (v.minor == 8 and v.micro == 0 and v.releaselevel == 'alpha' and v.serial <= 1)
-    )
-
-
-class TestPluginUtil(unittest.TestCase):
-    test_html = """
-<!doctype html>
-<html lang="en" class="no-js">
-<title>Title</title>
-<meta property="og:type" content= "website" />
-<meta property="og:url" content="http://test.se/"/>
-<meta property="og:site_name" content="Test" />
-<script src="https://test.se/test.js"></script>
-<link rel="stylesheet" type="text/css" href="https://test.se/test.css">
-<script>Tester.ready(function () {
-alert("Hello, world!"); });</script>
-<p>
-<a 
-href="http://test.se/foo">bar</a>
-</p>
-</html>
-        """  # noqa: W291
-
-    def test_itertags_single_text(self):
-        title = list(itertags(self.test_html, "title"))
-        self.assertTrue(len(title), 1)
-        self.assertEqual(title[0].tag, "title")
-        self.assertEqual(title[0].text, "Title")
-        self.assertEqual(title[0].attributes, {})
-
-    def test_itertags_attrs_text(self):
-        script = list(itertags(self.test_html, "script"))
-        self.assertTrue(len(script), 2)
-        self.assertEqual(script[0].tag, "script")
-        self.assertEqual(script[0].text, "")
-        self.assertEqual(script[0].attributes, {"src": "https://test.se/test.js"})
-
-        self.assertEqual(script[1].tag, "script")
-        self.assertEqual(script[1].text.strip(), """Tester.ready(function () {\nalert("Hello, world!"); });""")
-        self.assertEqual(script[1].attributes, {})
-
-    @unittest.skipIf(unsupported_versions_1979(),
-                     "python3.7 issue, see bpo-34294")
-    def test_itertags_multi_attrs(self):
-        metas = list(itertags(self.test_html, "meta"))
-        self.assertTrue(len(metas), 3)
-        self.assertTrue(all(meta.tag == "meta" for meta in metas))
-
-        self.assertEqual(metas[0].text, None)
-        self.assertEqual(metas[1].text, None)
-        self.assertEqual(metas[2].text, None)
-
-        self.assertEqual(metas[0].attributes, {"property": "og:type", "content": "website"})
-        self.assertEqual(metas[1].attributes, {"property": "og:url", "content": "http://test.se/"})
-        self.assertEqual(metas[2].attributes, {"property": "og:site_name", "content": "Test"})
-
-    def test_multi_line_a(self):
-        anchor = list(itertags(self.test_html, "a"))
-        self.assertTrue(len(anchor), 1)
-        self.assertEqual(anchor[0].tag, "a")
-        self.assertEqual(anchor[0].text, "bar")
-        self.assertEqual(anchor[0].attributes, {"href": "http://test.se/foo"})
-
-    @unittest.skipIf(unsupported_versions_1979(),
-                     "python3.7 issue, see bpo-34294")
-    def test_no_end_tag(self):
-        links = list(itertags(self.test_html, "link"))
-        self.assertTrue(len(links), 1)
-        self.assertEqual(links[0].tag, "link")
-        self.assertEqual(links[0].text, None)
-        self.assertEqual(links[0].attributes, {"rel": "stylesheet",
-                                               "type": "text/css",
-                                               "href": "https://test.se/test.css"})
-
-    def test_tag_inner_tag(self):
-        links = list(itertags(self.test_html, "p"))
-        self.assertTrue(len(links), 1)
-        self.assertEqual(links[0].tag, "p")
-        self.assertEqual(links[0].text.strip(), '<a \nhref="http://test.se/foo">bar</a>')
-        self.assertEqual(links[0].attributes, {})