Repository: streamlink/streamlink

Issue:
plugins.tviplayer: unable to handle CNN Portugal
### Checklist

- [X] This is a plugin issue and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed plugin issues](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22plugin+issue%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest stable release

### Description

- issue:
  - the new `tviplayer` plugin is unable to handle https://tviplayer.iol.pt/direto/CNN 
  - of note, the previous TVI 24 became CNN Portugal after #4199.

- to reproduce:
  ```sh
  streamlink https://tviplayer.iol.pt/direto/CNN
  ```
  ```sh
  [cli][info] Found matching plugin tviplayer for URL https://tviplayer.iol.pt/direto/CNN
  error: Unable to validate response text: Unable to parse HTML: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration. ('<?xml version=\'1.0\' encoding=\'U ...)
  ```




### Debug log

```text
streamlink --loglevel debug https://tviplayer.iol.pt/direto/CNN
[cli][debug] OS:         Linux-5.10.0-9-amd64-x86_64-with-glibc2.31
[cli][debug] Python:     3.9.2
[cli][debug] Streamlink: 3.0.2
[cli][debug] Requests(2.26.0), Socks(1.7.1), Websocket(1.2.1)
[cli][debug] Arguments:
[cli][debug]  url=https://tviplayer.iol.pt/direto/CNN
[cli][debug]  --loglevel=debug
[cli][info] Found matching plugin tviplayer for URL https://tviplayer.iol.pt/direto/CNN
error: Unable to validate response text: Unable to parse HTML: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration. ('<?xml version=\'1.0\' encoding=\'U ...)
```



Discussion:
because of https://github.com/streamlink/streamlink/pull/4201
works fine without it.
> Unable to parse HTML: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration. ('<?xml version=\'1.0\' encoding=\'U ...)

This is what the site returns, which is XHTML5, a subset of XML, not HTML5. However, they also return invalid HTTP headers and invalid XML syntax, so web browsers will just ignore the XML declaration and parse it as regular HTML5. lxml on the other hand will raise an error due to the invalid XML declaration in the HTML5 content.

```html
<?xml version='1.0' encoding='UTF-8' ?>
<!DOCTYPE html>
<html lang="pt" xmlns="http://www.w3.org/1999/xhtml"><head id="j_idt2">
            <meta charset="utf-8" />
```

- https://en.wikipedia.org/wiki/HTML5#XHTML_5_(XML-serialized_HTML_5)
- https://validator.w3.org/nu/?doc=https%3A%2F%2Ftviplayer.iol.pt%2Fdireto%2FCNN
- https://www.w3.org/International/questions/qa-html-encoding-declarations.en#xml

> XHTML5: An XHTML5 document is served as XML and has XML syntax. XML parsers do not recognise the encoding declarations in meta elements. They only recognise the XML declaration.
>
> The XML declaration is only required if the page is not being served as UTF-8 (or UTF-16)

This means that we need to detect this edge case in the `parse_html` method, because `parse_xml` can't be used here due to the invalid XML syntax. When the input are bytes, like it was previously, the HTML parser simply ignores the XML declaration for some reason.

Resolving patch:
diff --git a/src/streamlink/utils/parse.py b/src/streamlink/utils/parse.py
--- a/src/streamlink/utils/parse.py
+++ b/src/streamlink/utils/parse.py
@@ -48,8 +48,12 @@ def parse_html(
     """Wrapper around lxml.etree.HTML with some extras.
 
     Provides these extra features:
+     - Removes XML declarations of invalid XHTML5 documents
      - Wraps errors in custom exception with a snippet of the data in the message
     """
+    if isinstance(data, str) and data.lstrip().startswith("<?xml"):
+        data = re.sub(r"^\s*<\?xml.+?\?>", "", data)
+
     return _parse(HTML, data, name, exception, schema, *args, **kwargs)
 
 


Test patch:
diff --git a/tests/utils/test_parse.py b/tests/utils/test_parse.py
--- a/tests/utils/test_parse.py
+++ b/tests/utils/test_parse.py
@@ -89,6 +89,12 @@ def test_parse_html_encoding(self):
         tree = parse_html(b"""<!DOCTYPE html><html><body>\xC3\xA4</body></html>""")
         self.assertEqual(tree.xpath(".//body/text()"), ["Ã¤"])
 
+    def test_parse_html_xhtml5(self):
+        tree = parse_html("""<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE html><html><body>ä?></body></html>""")
+        self.assertEqual(tree.xpath(".//body/text()"), ["ä?>"])
+        tree = parse_html(b"""<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE html><html><body>\xC3\xA4?></body></html>""")
+        self.assertEqual(tree.xpath(".//body/text()"), ["ä?>"])
+
     def test_parse_qsd(self):
         self.assertEqual(
             {"test": "1", "foo": "bar"},