Repository: yt-dlp/yt-dlp

Issue:
CrunchyRoll HardSub videos are not available to download when using a premium account
### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.04.08** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

US

### Description

Since support has been added for the beta.crunchyroll.com site, HardSub video options only show up for free user options. This is demonstrated by the first section of the verbose log output. When using a premium account, the HardSub video formats no longer show up as an option to download. This is demonstrated by the second section of the verbose log output.

I simply used a blank cookie file for the first section and replaced it with a valid cookie file for the second section.  This happens when using the English Dub episode links. I have not tried others. The HardSub formats are not available for all dubbed videos but are available for most newer releases. This is not a difference between premium content versus free content. The same episode link was used in both sections, so in this case, it's the same content.

You can see that it downloads all content information in the first section

     [crunchyroll] 842078: Downloading webpage
     [crunchyroll] 842078: Downloading adaptive_hls-audio-enUS information
     [crunchyroll] 842078: Downloading adaptive_hls-audio-enUS-hardsub-enUS information

But it only downloads some of the content information in the second section using a premium account

     [crunchyroll:beta] Retrieving main page
     [crunchyroll:beta] Authenticating with cookie
     [crunchyroll:beta] Retrieving signed policy
     [crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Retrieving episode metadata
     [crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Retrieving stream info
     [crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Downloading adaptive_hls information

I tried to be as thorough as possible, but this is my first time reporting here. Please let me know if you need any additional information. Attached is a screenshot of a non-verbose output for an example. The verbose output was pasted in the box provided

![Image 090](https://user-images.githubusercontent.com/28385138/162831580-bae7db94-3dd6-43f3-8fd7-fb2efdec74ea.png)
.

### Verbose log

```shell
[debug] Command-line config: ['-vU', '-F', '--cookies', 'cookies-crunchyroll-com.txt', '-a', 'Links.txt']
[debug] Batch file urls: ['https://beta.crunchyroll.com/watch/GZ7UV0315/bear-hunting-a-queen-a-bath-and-monsters']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.04.08 [7884ade] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.4-full_build-www.gyan.dev (setts), ffprobe 4.4-full_build-www.gyan.dev
[debug] Optional libraries: brotli, certifi, Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.04.08, Current version: 2022.04.08
yt-dlp is up to date (2022.04.08)
[debug] [crunchyroll:beta] Extracting URL: https://beta.crunchyroll.com/watch/GZ7UV0315/bear-hunting-a-queen-a-bath-and-monsters
[crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Downloading webpage
[crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Not logged in. Redirecting to non-beta site - https://www.crunchyroll.com/in-the-land-of-leadale/bear-hunting-a-queen-a-bath-and-monsters-842078
[debug] [crunchyroll] Extracting URL: https://www.crunchyroll.com/in-the-land-of-leadale/bear-hunting-a-queen-a-bath-and-monsters-842078
[crunchyroll] 842078: Downloading webpage
[crunchyroll] 842078: Downloading adaptive_hls-audio-enUS information
[crunchyroll] 842078: Downloading adaptive_hls-audio-enUS-hardsub-enUS information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[crunchyroll] 842078: Downloading media info
WARNING: [crunchyroll] Unable to download XML: HTTP Error 404: Not Found
[info] Available formats for 842078:
ID                                          EXT RESOLUTION FPS │   TBR PROTO  │ VCODEC        VBR ACODEC    ABR MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
adaptive_hls-audio-enUS-hardsub-enUS-561-0  mp4 428x240     23 │  561k m3u8_n │ avc1.42c015  561k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-561-1  mp4 428x240     23 │  561k m3u8_n │ avc1.42c015  561k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-1092-0 mp4 640x360     23 │ 1092k m3u8_n │ avc1.4d401e 1092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-1092-1 mp4 640x360     23 │ 1092k m3u8_n │ avc1.4d401e 1092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-2092-0 mp4 848x480     23 │ 2092k m3u8_n │ avc1.4d401f 2092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-2092-1 mp4 848x480     23 │ 2092k m3u8_n │ avc1.4d401f 2092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-4108-0 mp4 1280x720    23 │ 4108k m3u8_n │ avc1.640028 4108k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-4108-1 mp4 1280x720    23 │ 4108k m3u8_n │ avc1.640028 4108k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-8102-0 mp4 1920x1080   23 │ 8102k m3u8_n │ avc1.640028 8102k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-hardsub-enUS-8102-1 mp4 1920x1080   23 │ 8102k m3u8_n │ avc1.640028 8102k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-561-0               mp4 428x240     23 │  561k m3u8_n │ avc1.42c015  561k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-561-1               mp4 428x240     23 │  561k m3u8_n │ avc1.42c015  561k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-1092-0              mp4 640x360     23 │ 1092k m3u8_n │ avc1.4d401e 1092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-1092-1              mp4 640x360     23 │ 1092k m3u8_n │ avc1.4d401e 1092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-2092-0              mp4 848x480     23 │ 2092k m3u8_n │ avc1.4d401f 2092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-2092-1              mp4 848x480     23 │ 2092k m3u8_n │ avc1.4d401f 2092k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-4108-0              mp4 1280x720    23 │ 4108k m3u8_n │ avc1.640028 4108k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-4108-1              mp4 1280x720    23 │ 4108k m3u8_n │ avc1.640028 4108k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-8103-0              mp4 1920x1080   23 │ 8103k m3u8_n │ avc1.640028 8103k mp4a.40.2  0k [enUS]
adaptive_hls-audio-enUS-8103-1              mp4 1920x1080   23 │ 8103k m3u8_n │ avc1.640028 8103k mp4a.40.2  0k [enUS]





[debug] Command-line config: ['-vU', '-F', '--cookies', 'cookies-crunchyroll-com.txt', '-a', 'Links.txt']
[debug] Batch file urls: ['https://beta.crunchyroll.com/watch/GZ7UV0315/bear-hunting-a-queen-a-bath-and-monsters']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.04.08 [7884ade] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.4-full_build-www.gyan.dev (setts), ffprobe 4.4-full_build-www.gyan.dev
[debug] Optional libraries: brotli, certifi, Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.04.08, Current version: 2022.04.08
yt-dlp is up to date (2022.04.08)
[debug] [crunchyroll:beta] Extracting URL: https://beta.crunchyroll.com/watch/GZ7UV0315/bear-hunting-a-queen-a-bath-and-monsters
[crunchyroll:beta] Retrieving main page
[crunchyroll:beta] Authenticating with cookie
[crunchyroll:beta] Retrieving signed policy
[crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Retrieving episode metadata
[crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Retrieving stream info
[crunchyroll:beta] bear-hunting-a-queen-a-bath-and-monsters: Downloading adaptive_hls information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for GZ7UV0315:
ID                  EXT RESOLUTION FPS │   FILESIZE   TBR PROTO  │ VCODEC        VBR ACODEC    ABR MORE INFO
────────────────────────────────────────────────────────────────────────────────────────────────────────────
adaptive_hls-195-0  mp4 144x80      23 │ ~ 33.88MiB  195k m3u8_n │ avc1.42c00c  195k mp4a.40.2  0k [en-US]
adaptive_hls-195-1  mp4 144x80      23 │ ~ 33.88MiB  195k m3u8_n │ avc1.42c00c  195k mp4a.40.2  0k [en-US]
adaptive_hls-561-0  mp4 428x240     23 │ ~ 97.42MiB  561k m3u8_n │ avc1.42c015  561k mp4a.40.2  0k [en-US]
adaptive_hls-561-1  mp4 428x240     23 │ ~ 97.42MiB  561k m3u8_n │ avc1.42c015  561k mp4a.40.2  0k [en-US]
adaptive_hls-1092-0 mp4 640x360     23 │ ~189.36MiB 1092k m3u8_n │ avc1.4d401e 1092k mp4a.40.2  0k [en-US]
adaptive_hls-1092-1 mp4 640x360     23 │ ~189.36MiB 1092k m3u8_n │ avc1.4d401e 1092k mp4a.40.2  0k [en-US]
adaptive_hls-2092-0 mp4 848x480     23 │ ~362.70MiB 2092k m3u8_n │ avc1.4d401f 2092k mp4a.40.2  0k [en-US]
adaptive_hls-2092-1 mp4 848x480     23 │ ~362.70MiB 2092k m3u8_n │ avc1.4d401f 2092k mp4a.40.2  0k [en-US]
adaptive_hls-4108-0 mp4 1280x720    23 │ ~712.12MiB 4108k m3u8_n │ avc1.640028 4108k mp4a.40.2  0k [en-US]
adaptive_hls-4108-1 mp4 1280x720    23 │ ~712.12MiB 4108k m3u8_n │ avc1.640028 4108k mp4a.40.2  0k [en-US]
adaptive_hls-8103-0 mp4 1920x1080   23 │ ~  1.37GiB 8103k m3u8_n │ avc1.640028 8103k mp4a.40.2  0k [en-US]
adaptive_hls-8103-1 mp4 1920x1080   23 │ ~  1.37GiB 8103k m3u8_n │ avc1.640028 8103k mp4a.40.2  0k [en-US]
```



Discussion:
@tejing1, could u have a look
- The beta extractor doesn't list hardsub formats by default. They can be included using the `hardsub` extractor argument. See https://github.com/yt-dlp/yt-dlp/blob/master/README.md#extractor-arguments.
- When not logged in, the beta extractor just determines the url for the non-beta site and continues as if you had specified that, so it shows whatever the non-beta extractor would have shown. You're only *really* using the beta extractor if you're logged in.
So you can get english hardsub formats with `--extractor-args crunchyrollbeta:hardsub=en-US`, though generally I would use `--sub-lang en-US --embed-subs` to get softsubs instead, unless you have a player that deals poorly with the soft subs for some reason.
> The beta extractor doesn't list hardsub formats by default. They can be included using the hardsub extractor argument. See [https://github.com/yt-dlp/yt-dlp/blob/master/README.md#extractor-arguments](https://github.com/yt-dlp/yt-dlp/blob/master/README.md?rgh-link-date=2022-04-16T11%3A13%3A35Z#extractor-arguments).

Totally forgot about that... This is why I initially insisted all formats be listed by default (#2506) 😞 
I don't think it would be nearly so much of a problem if the extractor in use was clearer. The redirects confuse the issue. That's why I wanted a prominent message about switching extractors, but I'm starting to wonder if It'd be better to just fail rather than redirect, so users are clear on which extractor they're actually using.
The extractor switching is clear from the logs (even without verbose). Not that many users understand enough of (or care about) yt-dlp's internals to understand what a redirect implies for them.

The issue is that the normal expectation is for all formats available in the browser to also be available in `-F`.

I'm not saying we should extract all formats here - that has it's own issues like we discussed in the PR. Perhaps we can add a warning that hard-subs are being ignored?
Actually, I don't think the hardsubs *are* available in the browser. If they are, I can't find them. People just expect them to be there because they're there in the non-beta extractor's formats. (EDIT: to be clear, I mean on the beta site)
I suspect the hardsubs are only in the api at all to support apps on platforms that have less featureful subtitle rendering abilities, since crunchyroll places subs over top of written text in the source material, matching the color and font and such. Subtitle renderers on some platforms may just throw all that text at the bottom of the screen, which would be a pretty bad viewing experience.
Yeah, that's why I prefer to get the hardsub versions because they get viewed on different boxes/devices connected to televisions. If it was only for watching on a PC then it wouldn't be a big deal. But, as you mentioned with the varied support problems, android tv boxes have different levels of subtitle support depending on the version used, and apple tv devices just... well apple does whatever apple does. And since CR goes through the trouble of adding such nice-looking subtitles, it would be a shame to not retain that experience.

Thank you all for looking into this and for all the effort you've put into this project. I'll give the above mentioned extractor args a shot.
I'm surprised you bring up Android TV and Apple TV specifically, those should have the least issues with subtitles compared to crappy smart TVs like Samsung's ones. I use Infuse on Apple TV and it handles them great, but Plex or VLC may also work well on Android TV.
Does this still need to be open, or is "use the extractor args" a sufficient resolution?
https://github.com/yt-dlp/yt-dlp/issues/4886 gives more reasons for why this should be fixed
Unless someone has a better idea, we should just set the default to "all" like it is for the non-beta page
We could assume formats with the same audio language would have almost the same format data as others. Here is a comparison of two `m3u8` annotations between hardsub languages:
```
#EXT-X-STREAM-INF:BANDWIDTH=4115377,RESOLUTION=1280x720,FRAME-RATE=23.974,CODECS="avc1.640028,mp4a.40.2"
#EXT-X-STREAM-INF:BANDWIDTH=4114464,RESOLUTION=1280x720,FRAME-RATE=23.974,CODECS="avc1.640028,mp4a.40.2"
```
Extract one format, sort by that, then do actual extraction lazily if we access the `url` attribute.
Since this is inherently unsafe it would make sense to add another extractor argument (`unsafe_speedup`?) where this would be performed.
This may be more trouble than help, but perhaps the extractor should fall back to hardsub formats if it doesn't find any other ones? It would clear up cases like #4886 without burdening the normal use, but it might also turn into another annoying quirk later on.

EDIT: It would also match the browser behavior afaict. The browser doesn't normally give you a way to show hardsubs, but appears to use them if softsubs aren't available.
What about wrapping hardsubs in `InfoExtractor._m3u8_meta_format`, unless the `hardsub` extractor argument is specified (Or, like you said, if only hardsubs exist)? That will not make any more requests from the basics and should solve basic cases.

Resolving patch:
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -846,7 +846,7 @@ def to_stdout(self, message, skip_eol=False, quiet=None):
                                      'Use "YoutubeDL.to_screen" instead')
         self._write_string(f'{self._bidi_workaround(message)}\n', self._out_files.out)
 
-    def to_screen(self, message, skip_eol=False, quiet=None):
+    def to_screen(self, message, skip_eol=False, quiet=None, only_once=False):
         """Print message to screen if not in quiet mode"""
         if self.params.get('logger'):
             self.params['logger'].debug(message)
@@ -855,7 +855,7 @@ def to_screen(self, message, skip_eol=False, quiet=None):
             return
         self._write_string(
             '%s%s' % (self._bidi_workaround(message), ('' if skip_eol else '\n')),
-            self._out_files.screen)
+            self._out_files.screen, only_once=only_once)
 
     def to_stderr(self, message, only_once=False):
         """Print message to stderr"""
diff --git a/yt_dlp/extractor/crunchyroll.py b/yt_dlp/extractor/crunchyroll.py
--- a/yt_dlp/extractor/crunchyroll.py
+++ b/yt_dlp/extractor/crunchyroll.py
@@ -779,7 +779,28 @@ class CrunchyrollBetaIE(CrunchyrollBetaBaseIE):
             'episode_number': 73,
             'thumbnail': r're:^https://beta.crunchyroll.com/imgsrv/.*\.jpeg$',
         },
-        'params': {'skip_download': 'm3u8'},
+        'params': {'skip_download': 'm3u8', 'format': 'all[format_id~=hardsub]'},
+    }, {
+        'url': 'https://beta.crunchyroll.com/watch/GYE5WKQGR',
+        'info_dict': {
+            'id': 'GYE5WKQGR',
+            'ext': 'mp4',
+            'duration': 366.459,
+            'timestamp': 1476788400,
+            'description': 'md5:74b67283ffddd75f6e224ca7dc031e76',
+            'title': 'SHELTER Episode  – Porter Robinson presents Shelter the Animation',
+            'upload_date': '20161018',
+            'series': 'SHELTER',
+            'series_id': 'GYGG09WWY',
+            'season': 'SHELTER',
+            'season_id': 'GR09MGK4R',
+            'season_number': 1,
+            'episode': 'Porter Robinson presents Shelter the Animation',
+            'episode_number': 0,
+            'thumbnail': r're:^https://beta.crunchyroll.com/imgsrv/.*\.jpeg$',
+        },
+        'params': {'skip_download': True},
+        'skip': 'Video is Premium only',
     }, {
         'url': 'https://beta.crunchyroll.com/watch/GY2P1Q98Y',
         'only_matching': True,
@@ -807,30 +828,48 @@ def _real_extract(self, url):
         hardsub_preference = qualities(requested_hardsubs[::-1])
         requested_formats = self._configuration_arg('format') or ['adaptive_hls']
 
-        formats = []
+        available_formats = {}
         for stream_type, streams in get_streams('streams'):
             if stream_type not in requested_formats:
                 continue
             for stream in streams.values():
-                hardsub_lang = stream.get('hardsub_locale') or ''
-                if hardsub_lang.lower() not in requested_hardsubs:
-                    continue
-                format_id = join_nonempty(stream_type, format_field(stream, 'hardsub_locale', 'hardsub-%s'))
                 if not stream.get('url'):
                     continue
-                if stream_type.endswith('hls'):
+                hardsub_lang = stream.get('hardsub_locale') or ''
+                format_id = join_nonempty(stream_type, format_field(stream, 'hardsub_locale', 'hardsub-%s'))
+                available_formats[hardsub_lang] = (stream_type, format_id, hardsub_lang, stream['url'])
+
+        if '' in available_formats and 'all' not in requested_hardsubs:
+            full_format_langs = set(requested_hardsubs)
+            self.to_screen(
+                'To get all formats of a hardsub language, use '
+                '"--extractor-args crunchyrollbeta:hardsub=<language_code or all>". '
+                'See https://github.com/yt-dlp/yt-dlp#crunchyrollbeta for more info',
+                only_once=True)
+        else:
+            full_format_langs = set(map(str.lower, available_formats))
+
+        formats = []
+        for stream_type, format_id, hardsub_lang, stream_url in available_formats.values():
+            if stream_type.endswith('hls'):
+                if hardsub_lang.lower() in full_format_langs:
                     adaptive_formats = self._extract_m3u8_formats(
-                        stream['url'], display_id, 'mp4', m3u8_id=format_id,
+                        stream_url, display_id, 'mp4', m3u8_id=format_id,
                         fatal=False, note=f'Downloading {format_id} HLS manifest')
-                elif stream_type.endswith('dash'):
-                    adaptive_formats = self._extract_mpd_formats(
-                        stream['url'], display_id, mpd_id=format_id,
-                        fatal=False, note=f'Downloading {format_id} MPD manifest')
-                for f in adaptive_formats:
-                    if f.get('acodec') != 'none':
-                        f['language'] = stream_response.get('audio_locale')
-                    f['quality'] = hardsub_preference(hardsub_lang.lower())
-                formats.extend(adaptive_formats)
+                else:
+                    adaptive_formats = (self._m3u8_meta_format(stream_url, ext='mp4', m3u8_id=format_id),)
+            elif stream_type.endswith('dash'):
+                adaptive_formats = self._extract_mpd_formats(
+                    stream_url, display_id, mpd_id=format_id,
+                    fatal=False, note=f'Downloading {format_id} MPD manifest')
+            else:
+                self.report_warning(f'Encountered unknown stream_type: {stream_type!r}', display_id, only_once=True)
+                continue
+            for f in adaptive_formats:
+                if f.get('acodec') != 'none':
+                    f['language'] = stream_response.get('audio_locale')
+                f['quality'] = hardsub_preference(hardsub_lang.lower())
+            formats.extend(adaptive_formats)
         self._sort_formats(formats)
 
         return {


Test patch: