Repository: ytdl-org/youtube-dl

Issue:
Option --restrict-filenames should implicitly de-zalgo the text.
## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
- Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
- Finally, put x into all relevant boxes (like this [x])
-->

- [x] I'm reporting a feature request
- [x] I've verified that I'm running youtube-dl version **2021.12.17**
- [x] I've searched the bugtracker for similar feature requests including closed ones


## Problem

I had to manually fix the output filename for a video whose title contains lots of character decorations ([Zalgo](https://en.wikipedia.org/wiki/Zalgo_text)):

```text
$ ytdl-rg3 --format=18 --get-title -- i41M0no6ff4
Film Theory: There Is Only DAD DaD ͈͘DAd ̒̇͞d̻̈́̚aD dAD

$ ytdl-rg3 --format=18 --output='%(title)s' --get-filename -- i41M0no6ff4
Film_Theory_-_There_Is_Only_DAD_D_a_D_DA_d_d_a_D_d_A_D
```

(Why I need `--format=` to read the title can be left for another day.)


## Proposed solution

Let's automatically dezalgo the input before restricting filename. On PyPI I found [zalgolib](https://pypi.org/project/zalgolib/) but it may be easier to just regexp-replace the unicode ranges mentioned there.

Fortunately, for this specific video I found a good enough work-around:

```text
$ ytdl-rg3 --format=18 --get-title i41M0no6ff4 | tr -cd 'A-Za-z \n' | tr ' ' _
Film_Theory_There_Is_Only_DAD_DaD_DAd_daD_dAD
```


Discussion:
Maybe I should deploy [this Unicode processing](https://github.com/yt-dlp/yt-dlp/issues/7079#issuecomment-1555132638), assuming for the moment that the algorithm would effectively strip the decorations?  
Decoding confusable characters sounds like a neat feature, but I'm undecided whether it's worth the disk space for the character equivalence table. I feel that for `--restrict-filenames` it would be more in its spirit to just strip everything with codepoint >= 127. Might be overreach though, thus my idea of using the ranges from zalgolib.
Everything with codepoint >= 127 is stripped and replaced by `_`, which explains your example output. You're suggesting that the Unicode codes representing decorations be dropped instead, no?

My hope was that the confusable remapping would also strip the combining decoration codes, and then any resulting non-ASCII codes would be mapped to `_` as now. I'll try this out when I get a chance.
Indeed confusable processing doesn't make any difference.

But this apparently reasonable change (skip Control and Mark codes) in the `replace_insane()` local function does:
```diff
         if restricted and ord(char) > 127:
-            return '_'
+            return '' if unicodedata.category(char)[0] in 'CM' else '_'
```
Then the filename as above is `Film_Theory_-_There_Is_Only_DAD_DaD_DAd_daD_dAD`.
I don't understand enough details to comment on the implementation, but the new output looks like it solves my problem. :+1:
The logic uses the General Category assigned to each Unicode code point (https://en.wikipedia.org/wiki/Unicode#General_Category_property). The Major category (`L`, `M`, `N`, `P`, `S`, `Z`, `C`), indicating Letter, Mark, Number, Punctuation, Symbol, Separator, or Other, is the first character of the category name (all two-letter values). I made the executive decision that for code points above 127 Marks and Others (eg, combining marks) should be omitted rather than mapped to `_` for `--restrict_filename`.

Fortunately Python offers us the `unicodedata` library module to help with this, so it's a (less than) one-line change.
While you're at it anyway, would it be cumbersome to have an option to use a custom character instead of underline?
I'm inclined to think that it would be, unless you have a very compelling use case, especially since this option exists only to help file systems that don't understand Unicode (so there isn't a large choice of alternate characters).

There are utilities, depending on the platform, that could be used with `--exec ...` to rename downloaded files.

_yt-dl_ could acquire some of the output template formatting enhancements from _yt-dlp_, which includes a string replacement syntax. I would favour that over a custom option, but it involves a lot of code to replace/extend the built-in Python formatting syntax currently used.
Yeah ok that indeed sounds too cumbersome. My benefit would have been to simplify the renaming tool I `--exec` but I see now that the impact on your codebase here would be disproportional.

Resolving patch:
diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -260,16 +260,10 @@ def _initialize_consent(self):
         cookies = self._get_cookies('https://www.youtube.com/')
         if cookies.get('__Secure-3PSID'):
             return
-        consent_id = None
-        consent = cookies.get('CONSENT')
-        if consent:
-            if 'YES' in consent.value:
-                return
-            consent_id = self._search_regex(
-                r'PENDING\+(\d+)', consent.value, 'consent', default=None)
-        if not consent_id:
-            consent_id = random.randint(100, 999)
-        self._set_cookie('.youtube.com', 'CONSENT', 'YES+cb.20210328-17-p0.en+FX+%s' % consent_id)
+        socs = cookies.get('SOCS')
+        if socs and not socs.value.startswith('CAA'):  # not consented
+            return
+        self._set_cookie('.youtube.com', 'SOCS', 'CAI', secure=True)  # accept all (required for mixes)
 
     def _real_initialize(self):
         self._initialize_consent()
diff --git a/youtube_dl/utils.py b/youtube_dl/utils.py
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@@ -2121,7 +2121,8 @@ def replace_insane(char):
         if restricted and (char in '!&\'()[]{}$;`^,#' or char.isspace()):
             return '_'
         if restricted and ord(char) > 127:
-            return '_'
+            return '' if unicodedata.category(char)[0] in 'CM' else '_'
+
         return char
 
     # Replace look-alike Unicode glyphs


Test patch: