Repository: yt-dlp/yt-dlp

Issue:
[Broken] PolskieRadio and PolskieRadioCategory
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.09.02. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.09.02**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.09.02
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
**PolskieRadio**

yt-dlp -v https://www.polskieradio.pl/8/406/Artykul/2773032,Surowy-autoportret-Jan-van-Eyck
[debug] Command-line config: ['-v', 'https://www.polskieradio.pl/8/406/Artykul/2773032,Surowy-autoportret-Jan-van-Eyck']
[debug] Encodings: locale cp1250, fs utf-8, out utf-8, pref cp1250
[debug] yt-dlp version 2021.09.02
[debug] Python version 3.8.3 (CPython 64bit) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg 4.2.3, ffprobe git-2020-06-28-4cfcfb3
[debug] Optional libraries: mutagen, pycryptodome, sqlite, websockets
[debug] Proxy map: {}
[debug] [PolskieRadio] Extracting URL: https://www.polskieradio.pl/8/406/Artykul/2773032,Surowy-autoportret-Jan-van-Eyck
[PolskieRadio] 2773032: Downloading webpage
[download] Downloading playlist: Surowy autoportret. Jan van Eyck
[PolskieRadio] playlist Surowy autoportret. Jan van Eyck: Collected 0 videos; downloading 0 of them
[download] Finished downloading playlist: Surowy autoportret. Jan van Eyck

And 
**PolskieRadioCategory**

yt-dlp -v https://www.polskieradio.pl/8/406
[debug] Command-line config: ['-v', 'https://www.polskieradio.pl/8/406']
[debug] Encodings: locale cp1250, fs utf-8, out utf-8, pref cp1250
[debug] yt-dlp version 2021.09.02
[debug] Python version 3.8.3 (CPython 64bit) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg 4.2.3, ffprobe git-2020-06-28-4cfcfb3
[debug] Optional libraries: mutagen, pycryptodome, sqlite, websockets
[debug] Proxy map: {}
[debug] [PolskieRadioCategory] Extracting URL: https://www.polskieradio.pl/8/406
[PolskieRadioCategory] 406: Downloading webpage
[download] Downloading playlist: Jest taki obraz
[PolskieRadioCategory] playlist Jest taki obraz: Downloading 12 videos
[download] Downloading video 1 of 12
[debug] [PolskieRadio] Extracting URL: https://www.polskieradio.pl/8/406/Artykul/2807852,Milosc-ziemska-i-niebianska-wedlug-Tycjana
[PolskieRadio] 2807852: Downloading webpage
[download] Downloading playlist: "Miłość ziemska i niebiańska" według Tycjana
[PolskieRadio] playlist "Miłość ziemska i niebiańska" według Tycjana: Collected 0 videos; downloading 0 of them
[download] Finished downloading playlist: "Miłość ziemska i niebiańska" według Tycjana
[download] Downloading video 2 of 12
[debug] [PolskieRadio] Extracting URL: https://www.polskieradio.pl/8/406/Artykul/2803302,Meduzy-Potwory-Rubensa-i-Caravaggia
[PolskieRadio] 2803302: Downloading webpage
[download] Downloading playlist: Meduzy. Potwory Rubensa i Caravaggia
[PolskieRadio] playlist Meduzy. Potwory Rubensa i Caravaggia: Collected 0 videos; downloading 0 of them
[download] Finished downloading playlist: Meduzy. Potwory Rubensa i Caravaggia
[...]
[download] Downloading video 12 of 12
[debug] [PolskieRadio] Extracting URL: https://www.polskieradio.pl/8/406/Artykul/2765844,Matka-Boska-Czula-Obraz-Andrei-Mantegni
[PolskieRadio] 2765844: Downloading webpage
[download] Downloading playlist: Matka Boska Czuła. Obraz Andrei Mantegni
[PolskieRadio] playlist Matka Boska Czuła. Obraz Andrei Mantegni: Collected 0 videos; downloading 0 of them
[download] Finished downloading playlist: Matka Boska Czuła. Obraz Andrei Mantegni
[download] Finished downloading playlist: Jest taki obraz

```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials, please provide them or explain how one can obtain them.
-->

Broken download possibility, due to change in site structure. It is listing etc but not downloading.
It was reported already in 2020 - and here is a fix contribution from original author: 
https://github.com/ytdl-org/youtube-dl/pull/26069 
But unfortunately the fix was never added to youtube-dl. :( 
Please help.


Discussion:

Patch:
diff --git a/README.md b/README.md
index a14880282..d13eb4dc1 100644
--- a/README.md
+++ b/README.md
@@ -964,6 +964,7 @@ # OUTPUT TEMPLATE
 1. **Alternatives**: Alternate fields can be specified seperated with a `,`. Eg: `%(release_date>%Y,upload_date>%Y|Unknown)s`
 1. **Default**: A literal default value can be specified for when the field is empty using a `|` seperator. This overrides `--output-na-template`. Eg: `%(uploader|Unknown)s`
 1. **More Conversions**: In addition to the normal format types `diouxXeEfFgGcrs`, `B`, `j`, `l`, `q` can be used for converting to **B**ytes, **j**son, a comma seperated **l**ist and a string **q**uoted for the terminal respectively
+1. **Unicode normalization**: The format type `U` can be used for NFC [unicode normalization](https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize). The alternate form flag (`#`) changes the normalization to NFD and the conversion flag `+` can be used for NFKC/NFKD compatibility equivalence normalization. Eg: `%(title)+.100U` is NFKC
 
 To summarize, the general syntax for a field is:
 ```
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
index 11371fa86..a6eddd7f7 100644
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -25,6 +25,7 @@
 import tokenize
 import traceback
 import random
+import unicodedata
 
 from string import ascii_letters
 
@@ -908,7 +909,7 @@ def escape_outtmpl(outtmpl):
     def validate_outtmpl(cls, outtmpl):
         ''' @return None or Exception object '''
         outtmpl = re.sub(
-            STR_FORMAT_RE_TMPL.format('[^)]*', '[ljqB]'),
+            STR_FORMAT_RE_TMPL.format('[^)]*', '[ljqBU]'),
             lambda mobj: f'{mobj.group(0)[:-1]}s',
             cls._outtmpl_expandpath(outtmpl))
         try:
@@ -940,7 +941,7 @@ def prepare_outtmpl(self, outtmpl, info_dict, sanitize=None):
         }
 
         TMPL_DICT = {}
-        EXTERNAL_FORMAT_RE = re.compile(STR_FORMAT_RE_TMPL.format('[^)]*', f'[{STR_FORMAT_TYPES}ljqB]'))
+        EXTERNAL_FORMAT_RE = re.compile(STR_FORMAT_RE_TMPL.format('[^)]*', f'[{STR_FORMAT_TYPES}ljqBU]'))
         MATH_FUNCTIONS = {
             '+': float.__add__,
             '-': float.__sub__,
@@ -1031,21 +1032,26 @@ def create_key(outer_mobj):
             value = default if value is None else value
 
             str_fmt = f'{fmt[:-1]}s'
-            if fmt[-1] == 'l':
+            if fmt[-1] == 'l':  # list
                 value, fmt = ', '.join(variadic(value)), str_fmt
-            elif fmt[-1] == 'j':
+            elif fmt[-1] == 'j':  # json
                 value, fmt = json.dumps(value, default=_dumpjson_default), str_fmt
-            elif fmt[-1] == 'q':
+            elif fmt[-1] == 'q':  # quoted
                 value, fmt = compat_shlex_quote(str(value)), str_fmt
-            elif fmt[-1] == 'B':
+            elif fmt[-1] == 'B':  # bytes
                 value = f'%{str_fmt}'.encode('utf-8') % str(value).encode('utf-8')
                 value, fmt = value.decode('utf-8', 'ignore'), 's'
+            elif fmt[-1] == 'U':  # unicode normalized
+                opts = outer_mobj.group('conversion') or ''
+                value, fmt = unicodedata.normalize(
+                    # "+" = compatibility equivalence, "#" = NFD
+                    'NF%s%s' % ('K' if '+' in opts else '', 'D' if '#' in opts else 'C'),
+                    value), str_fmt
             elif fmt[-1] == 'c':
-                value = str(value)
-                if value is None:
-                    value, fmt = default, 's'
+                if value:
+                    value = str(value)[0]
                 else:
-                    value = value[0]
+                    fmt = str_fmt
             elif fmt[-1] not in 'rs':  # numeric
                 value = float_or_none(value)
                 if value is None:
diff --git a/yt_dlp/cookies.py b/yt_dlp/cookies.py
index 1409e6799..049ec9fb1 100644
--- a/yt_dlp/cookies.py
+++ b/yt_dlp/cookies.py
@@ -353,7 +353,7 @@ def decrypt(self, encrypted_value):
 class MacChromeCookieDecryptor(ChromeCookieDecryptor):
     def __init__(self, browser_keyring_name, logger):
         self._logger = logger
-        password = _get_mac_keyring_password(browser_keyring_name)
+        password = _get_mac_keyring_password(browser_keyring_name, logger)
         self._v10_key = None if password is None else self.derive_key(password)
 
     @staticmethod
@@ -546,7 +546,7 @@ def _parse_safari_cookies_record(data, jar, logger):
         p.skip_to(value_offset)
         value = p.read_cstring()
     except UnicodeDecodeError:
-        logger.warning('failed to parse cookie because UTF-8 decoding failed', only_once=True)
+        logger.warning('failed to parse Safari cookie because UTF-8 decoding failed', only_once=True)
         return record_size
 
     p.skip_to(record_size, 'space at the end of the record')
@@ -592,11 +592,13 @@ def _get_linux_keyring_password(browser_keyring_name):
     return password.encode('utf-8')
 
 
-def _get_mac_keyring_password(browser_keyring_name):
+def _get_mac_keyring_password(browser_keyring_name, logger):
     if KEYRING_AVAILABLE:
+        logger.debug('using keyring to obtain password')
         password = keyring.get_password('{} Safe Storage'.format(browser_keyring_name), browser_keyring_name)
         return password.encode('utf-8')
     else:
+        logger.debug('using find-generic-password to obtain password')
         proc = subprocess.Popen(['security', 'find-generic-password',
                                  '-w',  # write password to stdout
                                  '-a', browser_keyring_name,  # match 'account'
@@ -605,8 +607,11 @@ def _get_mac_keyring_password(browser_keyring_name):
                                 stderr=subprocess.DEVNULL)
         try:
             stdout, stderr = process_communicate_or_kill(proc)
+            if stdout[-1:] == b'\n':
+                stdout = stdout[:-1]
             return stdout
-        except BaseException:
+        except BaseException as e:
+            logger.warning(f'exception running find-generic-password: {type(e).__name__}({e})')
             return None
 
 
@@ -640,7 +645,7 @@ def _decrypt_aes_cbc(ciphertext, key, logger, initialization_vector=b' ' * 16):
     try:
         return plaintext[:-padding_length].decode('utf-8')
     except UnicodeDecodeError:
-        logger.warning('failed to decrypt cookie because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
+        logger.warning('failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
         return None
 
 
@@ -648,13 +653,13 @@ def _decrypt_aes_gcm(ciphertext, key, nonce, authentication_tag, logger):
     try:
         plaintext = aes_gcm_decrypt_and_verify_bytes(ciphertext, key, authentication_tag, nonce)
     except ValueError:
-        logger.warning('failed to decrypt cookie because the MAC check failed. Possibly the key is wrong?', only_once=True)
+        logger.warning('failed to decrypt cookie (AES-GCM) because the MAC check failed. Possibly the key is wrong?', only_once=True)
         return None
 
     try:
         return plaintext.decode('utf-8')
     except UnicodeDecodeError:
-        logger.warning('failed to decrypt cookie because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
+        logger.warning('failed to decrypt cookie (AES-GCM) because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
         return None
 
 
diff --git a/yt_dlp/extractor/polskieradio.py b/yt_dlp/extractor/polskieradio.py
index 978d6f813..53fe0340a 100644
--- a/yt_dlp/extractor/polskieradio.py
+++ b/yt_dlp/extractor/polskieradio.py
@@ -15,12 +15,13 @@
     int_or_none,
     strip_or_none,
     unified_timestamp,
+    unescapeHTML,
 )
 
 
 class PolskieRadioIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?polskieradio\.pl/\d+/\d+/Artykul/(?P<id>[0-9]+)'
-    _TESTS = [{
+    _TESTS = [{  # Old-style single broadcast.
         'url': 'http://www.polskieradio.pl/7/5102/Artykul/1587943,Prof-Andrzej-Nowak-o-historii-nie-da-sie-myslec-beznamietnie',
         'info_dict': {
             'id': '1587943',
@@ -39,14 +40,41 @@ class PolskieRadioIE(InfoExtractor):
                 'thumbnail': r're:^https?://static\.prsa\.pl/images/.*\.jpg$'
             },
         }],
-    }, {
-        'url': 'http://www.polskieradio.pl/265/5217/Artykul/1635803,Euro-2016-nie-ma-miejsca-na-blad-Polacy-graja-ze-Szwajcaria-o-cwiercfinal',
+    }, {  # New-style single broadcast.
+        'url': 'https://www.polskieradio.pl/8/2382/Artykul/2534482,Zagarysci-Poezja-jak-spoiwo',
+        'info_dict': {
+            'id': '2534482',
+            'title': 'Żagaryści. Poezja jak spoiwo',
+            'description': 'md5:f18d95d5dcba747a09b635e21a4c0695',
+        },
+        'playlist': [{
+            'md5': 'd07559829f61d5a93a75755987ded760',
+            'info_dict': {
+                'id': '2516679',
+                'ext': 'mp3',
+                'title': 'md5:c6e1234e0b747ad883cb91b7ad06b98c',
+                'timestamp': 1592654400,
+                'upload_date': '20200620',
+                'duration': 1430,
+                'thumbnail': r're:^https?://static\.prsa\.pl/images/.*\.jpg$'
+            },
+        }],
+    }, {  # Old-style multiple broadcast playlist.
+        'url': 'https://www.polskieradio.pl/8/4346/Artykul/2487823,Marek-Kondrat-czyta-Mistrza-i-Malgorzate',
+        'info_dict': {
+            'id': '2487823',
+            'title': 'Marek Kondrat czyta "Mistrza i Małgorzatę"',
+            'description': 'md5:8422a95cc83834f2aaeff9d82e9c8f39',
+        },
+        'playlist_mincount': 50,
+    }, {  # New-style multiple broadcast playlist.
+        'url': 'https://www.polskieradio.pl/8/4346/Artykul/2541317,Czytamy-Kalendarz-i-klepsydre-Tadeusza-Konwickiego',
         'info_dict': {
-            'id': '1635803',
-            'title': 'Euro 2016: nie ma miejsca na błąd. Polacy grają ze Szwajcarią o ćwierćfinał',
-            'description': 'md5:01cb7d0cad58664095d72b51a1ebada2',
+            'id': '2541317',
+            'title': 'Czytamy "Kalendarz i klepsydrę" Tadeusza Konwickiego',
+            'description': 'md5:0baeaa46d877f1351fb2eeed3e871f9f',
         },
-        'playlist_mincount': 12,
+        'playlist_mincount': 15,
     }, {
         'url': 'http://polskieradio.pl/9/305/Artykul/1632955,Bardzo-popularne-slowo-remis',
         'only_matching': True,
@@ -78,8 +106,8 @@ def _real_extract(self, url):
 
         media_urls = set()
 
-        for data_media in re.findall(r'<[^>]+data-media=({[^>]+})', content):
-            media = self._parse_json(data_media, playlist_id, fatal=False)
+        for data_media in re.findall(r'<[^>]+data-media="?({[^>]+})"?', content):
+            media = self._parse_json(data_media, playlist_id, transform_source=unescapeHTML, fatal=False)
             if not media.get('file') or not media.get('desc'):
                 continue
             media_url = self._proto_relative_url(media['file'], 'http:')
@@ -98,6 +126,7 @@ def _real_extract(self, url):
 
         title = self._og_search_title(webpage).strip()
         description = strip_or_none(self._og_search_description(webpage))
+        description = description.replace('\xa0', ' ') if description is not None else None
 
         return self.playlist_result(entries, playlist_id, title, description)
 
diff --git a/yt_dlp/utils.py b/yt_dlp/utils.py
index 141d2c9cc..770d7feb9 100644
--- a/yt_dlp/utils.py
+++ b/yt_dlp/utils.py
@@ -4474,12 +4474,12 @@ def q(qid):
 STR_FORMAT_RE_TMPL = r'''(?x)
     (?<!%)(?P<prefix>(?:%%)*)
     %
-    (?P<has_key>\((?P<key>{0})\))?  # mapping key
+    (?P<has_key>\((?P<key>{0})\))?
     (?P<format>
-        (?:[#0\-+ ]+)?  # conversion flags (optional)
-        (?:\d+)?  # minimum field width (optional)
-        (?:\.\d+)?  # precision (optional)
-        [hlL]?  # length modifier (optional)
+        (?P<conversion>[#0\-+ ]+)?
+        (?P<min_width>\d+)?
+        (?P<precision>\.\d+)?
+        (?P<len_mod>[hlL])?  # unused in python
         {1}  # conversion type
     )
 '''


Test patch:
diff --git a/test/test_YoutubeDL.py b/test/test_YoutubeDL.py
index 6feca2ce2..f6483575f 100644
--- a/test/test_YoutubeDL.py
+++ b/test/test_YoutubeDL.py
@@ -649,7 +649,7 @@ def test_add_extra_info(self):
         'title2': '%PATH%',
         'title3': 'foo/bar\\test',
         'title4': 'foo "bar" test',
-        'title5': 'áéí',
+        'title5': 'áéí 𝐀',
         'timestamp': 1618488000,
         'duration': 100000,
         'playlist_index': 1,
@@ -769,6 +769,10 @@ def expect_same_infodict(out):
         test('%(formats.:.id) 15l', '  id1, id2, id3')
         test('%(formats)j', (json.dumps(FORMATS), sanitize(json.dumps(FORMATS))))
         test('%(title5).3B', 'á')
+        test('%(title5)U', 'áéí 𝐀')
+        test('%(title5)#U', 'a\u0301e\u0301i\u0301 𝐀')
+        test('%(title5)+U', 'áéí A')
+        test('%(title5)+#U', 'a\u0301e\u0301i\u0301 A')
         if compat_os_name == 'nt':
             test('%(title4)q', ('"foo \\"bar\\" test"', "'foo _'bar_' test'"))
         else: