Repository: yt-dlp/yt-dlp

Issue:
[tvplay] Extractor deprecation
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.10.10. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.10.10**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones
- [x] I've read the opening an issue section in CONTRIBUTING.md
- [x] I have given an appropriate title to the issue


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.10.10
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
gustaf@tuxbox-tuxnet yt-dlp$ yt-dlp -v https://play.tv3.lt/series/moterys-meluoja-geriau-n-7,serial-2574652/serija-25,episode-3284937
[debug] Command-line config: ['-v', 'https://play.tv3.lt/series/moterys-meluoja-geriau-n-7,serial-2574652/serija-25,episode-3284937']
[debug] Encodings: locale UTF-8, fs utf-8, stdout utf-8, stderr utf-8, pref UTF-8
[debug] yt-dlp version 2021.10.10
[debug] Python version 3.9.7 (CPython 64bit) - Linux-5.14.10-200.fc34.x86_64-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] ANSI escape support: stdout = True, stderr = True
[debug] Proxy map: {}
[debug] [TVPlayHome] Extracting URL: https://play.tv3.lt/series/moterys-meluoja-geriau-n-7,serial-2574652/serija-25,episode-3284937
[TVPlayHome] 3284937: Downloading JSON metadata
ERROR: [TVPlayHome] 3284937: 3284937: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 972, in _parse_json
    return json.loads(json_string)
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 590, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/tvplay.py", line 474, in _real_extract
    asset = self._download_json(
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 961, in _download_json
    res = self._download_json_handle(
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 947, in _download_json_handle
    return self._parse_json(
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 976, in _parse_json
    raise ExtractorError(errmsg, cause=ve)
yt_dlp.utils.ExtractorError: 3284937: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 972, in _parse_json
    return json.loads(json_string)
  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 590, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/tvplay.py", line 474, in _real_extract
    asset = self._download_json(
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 961, in _download_json
    res = self._download_json_handle(
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 947, in _download_json_handle
    return self._parse_json(
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 976, in _parse_json
    raise ExtractorError(errmsg, cause=ve)
yt_dlp.utils.ExtractorError: 3284937: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/YoutubeDL.py", line 1265, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/YoutubeDL.py", line 1290, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp-2021.10.10-py3.9.egg/yt_dlp/extractor/common.py", line 606, in extract
    raise ExtractorError(
yt_dlp.utils.ExtractorError: [TVPlayHome] 3284937: 3284937: Failed to parse JSON  (caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.



```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

I made an issue in the youtube-dl repo in regards to how deprecated the tvplay extractor is. It is less deprecated here but still needs some refactoring.

I have made the changes I believe are needed here: https://github.com/18928172992817182/yt-dlp/commit/8f4091cf8aeae0fdeba24de494698f47e673b05c

I have cleaned up the MTG extractor since all of those I have removed now either directs to the viaplay extractor for all Scandinavian services or the tvplayhome extractor for the Baltics. There is one service left in the original tvplay extractor. I don't have enough knowledge to decide on anything there. 

I also fixed the tvplayhome extractor for the Baltics since it is currently broken after their URL and API change it seems. I used the website and the endpoints used there to update the extractor which does feel a bit forced compared to the single endpoint that was used before. But that endpoint no longer seems available. Now it works for live streamed content and vods.

Though the deprecation and fixing the baltics might be separate issues. What do you think @pukkandan ? For releases and versioning it might be safer to have them be separate?

Now I am just wondering if you want to discuss the changes and perhaps merge them?

I am not sure how deprecation like this should be handled. It always feels scary to remove stuff and perhaps that is why the MTG (tvplay) extractor has contained data deprecated since 2017?


Discussion:
Since both changes are to the same extractor, I wouldn't mind either a single PR or 2 seperate PRs. You can go with whichever is easier for you

> I am not sure how deprecation like this should be handled. It always feels scary to remove stuff and perhaps that is why the MTG (tvplay) extractor has contained data deprecated since 2017?

You can remove extraction steps that you are sure no longer work for any supported URL. For an extractor that handles a lot of regional variants of a site, this can be difficult to determine. In that case, leave the existing code as fallback to any new code you write

Resolving patch:
diff --git a/yt_dlp/extractor/tvplay.py b/yt_dlp/extractor/tvplay.py
--- a/yt_dlp/extractor/tvplay.py
+++ b/yt_dlp/extractor/tvplay.py
@@ -12,9 +12,9 @@
     determine_ext,
     ExtractorError,
     int_or_none,
-    parse_duration,
     parse_iso8601,
     qualities,
+    traverse_obj,
     try_get,
     update_url_query,
     url_or_none,
@@ -431,77 +431,96 @@ def _real_extract(self, url):
 
 
 class TVPlayHomeIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:tv3?)?play\.(?:tv3\.lt|skaties\.lv|tv3\.ee)/(?:[^/]+/)*[^/?#&]+-(?P<id>\d+)'
+    _VALID_URL = r'''(?x)
+            https?://
+            (?:tv3?)?
+            play\.(?:tv3|skaties)\.(?P<country>lv|lt|ee)/
+            (?P<live>lives/)?
+            [^?#&]+(?:episode|programme|clip)-(?P<id>\d+)
+    '''
     _TESTS = [{
-        'url': 'https://tvplay.tv3.lt/aferistai-n-7/aferistai-10047125/',
+        'url': 'https://play.tv3.lt/series/gauju-karai-karveliai,serial-2343791/serija-8,episode-2343828',
         'info_dict': {
-            'id': '366367',
+            'id': '2343828',
             'ext': 'mp4',
-            'title': 'Aferistai',
-            'description': 'Aferistai. Kalėdinė pasaka.',
-            'series': 'Aferistai [N-7]',
-            'season': '1 sezonas',
+            'title': 'Gaujų karai. Karveliai (2021) | S01E08: Serija 8',
+            'description': 'md5:f6fcfbb236429f05531131640dfa7c81',
+            'duration': 2710,
+            'season': 'Gaujų karai. Karveliai',
             'season_number': 1,
-            'duration': 464,
-            'timestamp': 1394209658,
-            'upload_date': '20140307',
-            'age_limit': 18,
+            'release_year': 2021,
+            'episode': 'Serija 8',
+            'episode_number': 8,
         },
         'params': {
-            'skip_download': True,
+            'skip_download': 'm3u8',
         },
     }, {
-        'url': 'https://tvplay.skaties.lv/vinas-melo-labak/vinas-melo-labak-10280317/',
-        'only_matching': True,
+        'url': 'https://play.tv3.lt/series/moterys-meluoja-geriau-n-7,serial-2574652/serija-25,episode-3284937',
+        'info_dict': {
+            'id': '3284937',
+            'ext': 'mp4',
+            'season': 'Moterys meluoja geriau [N-7]',
+            'season_number': 14,
+            'release_year': 2021,
+            'episode': 'Serija 25',
+            'episode_number': 25,
+            'title': 'Moterys meluoja geriau [N-7] (2021) | S14|E25: Serija 25',
+            'description': 'md5:c6926e9710f1a126f028fbe121eddb79',
+            'duration': 2440,
+        },
+        'skip': '404'
     }, {
-        'url': 'https://tvplay.tv3.ee/cool-d-ga-mehhikosse/cool-d-ga-mehhikosse-10044354/',
+        'url': 'https://play.tv3.lt/lives/tv6-lt,live-2838694/optibet-a-lygos-rungtynes-marijampoles-suduva--vilniaus-riteriai,programme-3422014',
         'only_matching': True,
     }, {
-        'url': 'https://play.tv3.lt/aferistai-10047125',
+        'url': 'https://tv3play.skaties.lv/series/women-lie-better-lv,serial-1024464/women-lie-better-lv,episode-1038762',
         'only_matching': True,
     }, {
-        'url': 'https://tv3play.skaties.lv/vinas-melo-labak-10280317',
+        'url': 'https://play.tv3.ee/series/_,serial-2654462/_,episode-2654474',
         'only_matching': True,
     }, {
-        'url': 'https://play.tv3.ee/cool-d-ga-mehhikosse-10044354',
+        'url': 'https://tv3play.skaties.lv/clips/tv3-zinas-valsti-lidz-15novembrim-bus-majsede,clip-3464509',
         'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        video_id = self._match_id(url)
+        country, is_live, video_id = self._match_valid_url(url).groups()
 
-        asset = self._download_json(
-            urljoin(url, '/sb/public/asset/' + video_id), video_id)
+        api_path = 'lives/programmes' if is_live else 'vods'
+        data = self._download_json(
+            urljoin(url, f'/api/products/{api_path}/{video_id}?platform=BROWSER&lang={country.upper()}'),
+            video_id)
 
-        m3u8_url = asset['movie']['contentUrl']
-        video_id = asset['assetId']
-        asset_title = asset['title']
-        title = asset_title['title']
-
-        formats = self._extract_m3u8_formats(
-            m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls')
+        video_type = 'CATCHUP' if is_live else 'MOVIE'
+        stream_id = data['programRecordingId'] if is_live else video_id
+        stream = self._download_json(
+            urljoin(url, f'/api/products/{stream_id}/videos/playlist?videoType={video_type}&platform=BROWSER'), video_id)
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(
+            stream['sources']['HLS'][0]['src'], video_id, 'mp4', 'm3u8_native', m3u8_id='hls')
         self._sort_formats(formats)
 
-        thumbnails = None
-        image_url = asset.get('imageUrl')
-        if image_url:
-            thumbnails = [{
-                'url': urljoin(url, image_url),
-                'ext': 'jpg',
-            }]
-
-        metadata = asset.get('metadata') or {}
+        thumbnails = set(traverse_obj(
+            data, (('galary', 'images', 'artworks'), ..., ..., ('miniUrl', 'mainUrl')), expected_type=url_or_none))
 
         return {
             'id': video_id,
-            'title': title,
-            'description': asset_title.get('summaryLong') or asset_title.get('summaryShort'),
-            'thumbnails': thumbnails,
-            'duration': parse_duration(asset_title.get('runTime')),
-            'series': asset.get('tvSeriesTitle'),
-            'season': asset.get('tvSeasonTitle'),
-            'season_number': int_or_none(metadata.get('seasonNumber')),
-            'episode': asset_title.get('titleBrief'),
-            'episode_number': int_or_none(metadata.get('episodeNumber')),
+            'title': self._resolve_title(data),
+            'description': traverse_obj(data, 'description', 'lead'),
+            'duration': int_or_none(data.get('duration')),
+            'season': traverse_obj(data, ('season', 'serial', 'title')),
+            'season_number': int_or_none(traverse_obj(data, ('season', 'number'))),
+            'episode': data.get('title'),
+            'episode_number': int_or_none(data.get('episode')),
+            'release_year': int_or_none(traverse_obj(data, ('season', 'serial', 'year'))),
+            'thumbnails': [{'url': url, 'ext': 'jpg'} for url in thumbnails],
             'formats': formats,
+            'subtitles': subtitles,
         }
+
+    @staticmethod
+    def _resolve_title(data):
+        return try_get(data, lambda x: (
+            f'{data["season"]["serial"]["title"]} ({data["season"]["serial"]["year"]}) | '
+            f'S{data["season"]["number"]:02d}E{data["episode"]:02d}: {data["title"]}'
+        )) or data.get('title')


Test patch: