Repository: streamlink/streamlink

Issue:
raise AttributeError with raiplay plugin
## Plugin Issue

- [x] This is a plugin issue and I have read the contribution guidelines.
- [x] I am using the latest development version from the master branch.

### Description

I use the `net-misc/streamlink-9999` package (git version) on a stable Gentoo Linux system.
I can successfully use streamlink with the Rainews 24 channel, but it doesn't work with Rai 1, Rai 2 or Rai 3.

### Reproduction steps / Explicit stream URLs to test

1. Run `streamlink https://www.raiplay.it/dirette/rai1`

### Log output

```
$ streamlink --loglevel debug https://www.raiplay.it/dirette/rai1
[cli][debug] OS:         Linux-5.4.80-gentoo-r1-2-x86_64-Intel-R-_Core-TM-2_Duo_CPU_E8400_@_3.00GHz-with-glibc2.2.5
[cli][debug] Python:     3.8.7
[cli][debug] Streamlink: 2.0.0+25.g7a74e18
[cli][debug] Requests(2.25.1), Socks(1.7.1), Websocket(0.57.0)
[cli][info] Found matching plugin raiplay for URL https://www.raiplay.it/dirette/rai1
[plugins.raiplay][debug] Found JSON URL: https://www.raiplay.it/dirette/rai1.json
[plugins.raiplay][debug] Found stream URL: https://mediapolis.rai.it/relinker/relinkerServlet.htm?cont=2606803
[utils.l10n][debug] Language code: en_US
Traceback (most recent call last):
  File "/usr/bin/streamlink", line 33, in <module>
    sys.exit(load_entry_point('streamlink==2.0.0+25.g7a74e18', 'console_scripts', 'streamlink')())
  File "/usr/lib/python3.8/site-packages/streamlink_cli/main.py", line 1052, in main
    handle_url()
  File "/usr/lib/python3.8/site-packages/streamlink_cli/main.py", line 580, in handle_url
    streams = fetch_streams(plugin)
  File "/usr/lib/python3.8/site-packages/streamlink_cli/main.py", line 459, in fetch_streams
    return plugin.streams(stream_types=args.stream_types,
  File "/usr/lib/python3.8/site-packages/streamlink/plugin/plugin.py", line 323, in streams
    ostreams = list(ostreams)
  File "/usr/lib/python3.8/site-packages/streamlink/plugins/raiplay.py", line 49, in _get_streams
    yield from HLSStream.parse_variant_playlist(self.session, stream_url).items()
  File "/usr/lib/python3.8/site-packages/streamlink/stream/hls.py", line 494, in parse_variant_playlist
    if not default_audio and (media.autoselect and locale.equivalent(language=media.language)):
  File "/usr/lib/python3.8/site-packages/streamlink/utils/l10n.py", line 155, in equivalent
    equivalent = equivalent and (not language or self.language == self.get_language(language))
  File "/usr/lib/python3.8/site-packages/streamlink/utils/l10n.py", line 169, in get_language
    return Language.get(language)
  File "/usr/lib/python3.8/site-packages/streamlink/utils/l10n.py", line 73, in get
    return Language(lang.alpha_2, lang.alpha_3, lang.name, getattr(lang, "bibliographic", None))
  File "/usr/lib/python3.8/site-packages/pycountry/db.py", line 19, in __getattr__
    raise AttributeError
AttributeError
```

### Additional comments, etc.

I have `dev-python/pycountry-20.7.3`.

I can easily add patches to the `/etc/portage/patches/net-misc/streamlink` directory and then reinstall `streamlink`, but unfortunately I don't know which patch I should apply. Patch from pull #3057 cannot be applied to `net-misc/streamlink-9999` as far as I know.


Discussion:
Could you please post the content of the HLS playlist?
```bash
curl -sSL "$(streamlink --stream-url 'https://www.raiplay.it/dirette/rai1')"
```

The `HLSStream` is trying to select the right audio stream based on your locale (which is `en_US` according to the log), as the media is annotated with `AUTOSELECT`:
https://github.com/streamlink/streamlink/blob/7a74e1848d9af9dad9f6fdb8fa4935049efb9a62/src/streamlink/stream/hls.py#L494

The reason why it fails is that there's a bug in one of the recent pycountry versions where it looks up generic/unknown language lookups incorrectly and a workaround had to be added in the PR you are referring to. It doesn't catch `AttributeError`s on invalid data though:
https://github.com/streamlink/streamlink/blob/7a74e1848d9af9dad9f6fdb8fa4935049efb9a62/src/streamlink/utils/l10n.py#L71-L73
> I don't know which patch I should apply

There is nothing to apply, because you are already using the latest version.
You can disable pycountry though when building with having `STREAMLINK_USE_PYCOUNTRY` unset:
https://github.com/streamlink/streamlink/blob/7a74e1848d9af9dad9f6fdb8fa4935049efb9a62/setup.py#L28
> Could you please post the content of the HLS playlist?
> 
> ```shell
> curl -sSL "$(streamlink --stream-url 'https://www.raiplay.it/dirette/rai1')"

The `curl` command fails because the `streamlink` command fails to produce a valid URL.

```
$ streamlink --stream-url https://www.raiplay.it/dirette/rai1
Traceback (most recent call last):
  File "/usr/bin/streamlink", line 33, in <module>
    sys.exit(load_entry_point('streamlink==2.0.0+25.g7a74e18', 'console_scripts', 'streamlink')())
  File "/usr/lib/python3.8/site-packages/streamlink_cli/main.py", line 1052, in main
    handle_url()
  File "/usr/lib/python3.8/site-packages/streamlink_cli/main.py", line 580, in handle_url
    streams = fetch_streams(plugin)
  File "/usr/lib/python3.8/site-packages/streamlink_cli/main.py", line 459, in fetch_streams
    return plugin.streams(stream_types=args.stream_types,
  File "/usr/lib/python3.8/site-packages/streamlink/plugin/plugin.py", line 323, in streams
    ostreams = list(ostreams)
  File "/usr/lib/python3.8/site-packages/streamlink/plugins/raiplay.py", line 49, in _get_streams
    yield from HLSStream.parse_variant_playlist(self.session, stream_url).items()
  File "/usr/lib/python3.8/site-packages/streamlink/stream/hls.py", line 494, in parse_variant_playlist
    if not default_audio and (media.autoselect and locale.equivalent(language=media.language)):
  File "/usr/lib/python3.8/site-packages/streamlink/utils/l10n.py", line 155, in equivalent
    equivalent = equivalent and (not language or self.language == self.get_language(language))
  File "/usr/lib/python3.8/site-packages/streamlink/utils/l10n.py", line 169, in get_language
    return Language.get(language)
  File "/usr/lib/python3.8/site-packages/streamlink/utils/l10n.py", line 73, in get
    return Language(lang.alpha_2, lang.alpha_3, lang.name, getattr(lang, "bibliographic", None))
  File "/usr/lib/python3.8/site-packages/pycountry/db.py", line 19, in __getattr__
    raise AttributeError
AttributeError
```

For comparison, Rainews 24 works instead:

```
$ streamlink --stream-url https://www.raiplay.it/dirette/rainews24
https://streamcdnm24-8e7439fdb1694c8da3a0fd63e4dda518.msvdn.net/rainews1/hls/rainews_2400/chunklist.m3u8?baseuri=%2Frainews1%2Fhls%2F&tstart=0&tend=1611684671&tof=86400&tk2=14c9908814ca5b9318f0f72f000ab1656d2829d3c7842c466b614ddb3a509f3a
```
Can you check it via your webbrowser's dev tools? Go to the network tab, refresh the site and look for the first m3u8 request that their web player will make. That would help me understand why it's raising the AttributeError.

The issue is `getattr(lang, "bibliographic", None)`, but without knowing what `lang` is from the statement above, it's hard to tell why it's raising the error.
there is no `alpha_2` for `LANGUAGE="des"`
```
language=ita
lang=Language(alpha_2='it', alpha_3='ita', name='Italian', scope='I', type='L')
---
language=ita
lang=Language(alpha_2='it', alpha_3='ita', name='Italian', scope='I', type='L')
---
language=V.O
---
language=V.O
---
language=des
lang=Language(alpha_3='des', name='Desano', scope='I', type='L')
Traceback (most recent call last):
...
AttributeError
```

```
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",LANGUAGE="ita",NAME="Italiano",DEFAULT=YES,AUTOSELECT=YES,URI="itarai1_160/"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",LANGUAGE="V.O",NAME="Lingua Originale",DEFAULT=NO,AUTOSELECT=YES,URI="engrai1_160/"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac",LANGUAGE="des",NAME="Audiodescrizione",DEFAULT=NO,AUTOSELECT=YES,URI="desrai1_160/"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2793078,CODECS="avc1.77.41,mp4a.40.2",RESOLUTION=1024x576,AUDIO="aac"
rai1_2400/chunklist.m3u8
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2793078,CODECS="avc1.77.42",URI="rai1_2400/chunklist.m3u8"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2137033,CODECS="avc1.77.31,mp4a.40.2",RESOLUTION=928x522,AUDIO="aac"
rai1_1800/chunklist.m3u8
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2137033,CODECS="avc1.77.42",URI="rai1_1800/chunklist.m3u8"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1365331,CODECS="avc1.77.31,mp4a.40.2",RESOLUTION=704x396,AUDIO="aac"
rai1_1200/chunklist.m3u8
#EXT-X-I-FRAME-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1365331,CODECS="avc1.77.31,mp4a.40.2",URI="rai1_1200/chunklist.m3u8"
```
Replacing `languages.lookup` in favor of multiple `languages.get` calls with specific attributes and raising a `KeyError` if none was found should fix this issue. We shouldn't be using the unknown lookup, as was already suggested in my pycountry bugreport thread, so catching its `AttributeError` is unnecessary.

If one can confirm that this patch is working, I'll open a PR.

```diff
diff --git a/src/streamlink/utils/l10n.py b/src/streamlink/utils/l10n.py
index de2c8892..05113db8 100644
--- a/src/streamlink/utils/l10n.py
+++ b/src/streamlink/utils/l10n.py
@@ -68,8 +68,13 @@ class Language:
     def get(cls, language):
         try:
             if PYCOUNTRY:
-                # lookup workaround for alpha_2 language codes
-                lang = languages.get(alpha_2=language) if re.match(r"^[a-z]{2}$", language) else languages.lookup(language)
+                lang = languages.get(alpha_2=language) \
+                       or languages.get(alpha_3=language) \
+                       or languages.get(bibliographic=language) \
+                       or languages.get(name=language) \
+                       or languages.get(common_name=language)
+                if not lang:
+                    raise KeyError(language)
                 return Language(lang.alpha_2, lang.alpha_3, lang.name, getattr(lang, "bibliographic", None))
             else:
                 lang = None
```
The patch does not work.

lang is not None, it won't raise the KeyError there.

`lang=Language(alpha_3='des', name='Desano', scope='I', type='L')`

`lang.alpha_2` is invalid

```py
Language(alpha_3='des', name='Desano', scope='I', type='L')
-- pycountry.db self._fields --
{'alpha_3': 'des', 'name': 'Desano', 'scope': 'I', 'type': 'L'}
-- pycountry.db key --
alpha_2
```

---

it is also probably `Italian` not `Desano`,
but there is not much we can do about it.

```
LANGUAGE="des",NAME="Audiodescrizione"
```


Resolving patch:
diff --git a/src/streamlink/utils/l10n.py b/src/streamlink/utils/l10n.py
--- a/src/streamlink/utils/l10n.py
+++ b/src/streamlink/utils/l10n.py
@@ -1,6 +1,5 @@
 import locale
 import logging
-import re
 
 
 try:
@@ -68,9 +67,19 @@ def __init__(self, alpha2, alpha3, name, bibliographic=None):
     def get(cls, language):
         try:
             if PYCOUNTRY:
-                # lookup workaround for alpha_2 language codes
-                lang = languages.get(alpha_2=language) if re.match(r"^[a-z]{2}$", language) else languages.lookup(language)
-                return Language(lang.alpha_2, lang.alpha_3, lang.name, getattr(lang, "bibliographic", None))
+                lang = (languages.get(alpha_2=language)
+                        or languages.get(alpha_3=language)
+                        or languages.get(bibliographic=language)
+                        or languages.get(name=language))
+                if not lang:
+                    raise KeyError(language)
+                return Language(
+                    # some languages don't have an alpha_2 code
+                    getattr(lang, "alpha_2", ""),
+                    lang.alpha_3,
+                    lang.name,
+                    getattr(lang, "bibliographic", "")
+                )
             else:
                 lang = None
                 if len(language) == 2:


Test patch:
diff --git a/tests/test_localization.py b/tests/test_localization.py
--- a/tests/test_localization.py
+++ b/tests/test_localization.py
@@ -109,6 +109,14 @@ def test_language_compare(self):
         b = l10n.Language("AA", None, "Test")
         self.assertNotEqual(a, b)
 
+    # issue #3517: language lookups without alpha2 but with alpha3 codes should not raise
+    def test_language_a3_no_a2(self):
+        a = l10n.Localization.get_language("des")
+        self.assertEqual(a.alpha2, "")
+        self.assertEqual(a.alpha3, "des")
+        self.assertEqual(a.name, "Desano")
+        self.assertEqual(a.bibliographic, "")
+
 
 @unittest.skipIf(not ISO639, "iso639+iso3166 modules are required to test iso639+iso3166 Localization")
 class TestLocalization(LocalizationTestsMixin, unittest.TestCase):
@@ -131,3 +139,14 @@ def setUp(self):
 
     def test_pycountry(self):
         self.assertEqual(True, l10n.PYCOUNTRY)
+
+    # issue #3057: generic "en" lookups via pycountry yield the "En" language, but not "English"
+    def test_language_en(self):
+        english_a = l10n.Localization.get_language("en")
+        english_b = l10n.Localization.get_language("eng")
+        english_c = l10n.Localization.get_language("English")
+        for lang in [english_a, english_b, english_c]:
+            self.assertEqual(lang.alpha2, "en")
+            self.assertEqual(lang.alpha3, "eng")
+            self.assertEqual(lang.name, "English")
+            self.assertEqual(lang.bibliographic, "")