Repository: yt-dlp/yt-dlp

Issue:
[Broken] Mediaset [ITALY]
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.05.20**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.05.20
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
D:\TEMP\YDL\YDLP>yt-dlp -v -F https://www.mediasetplay.mediaset.it/video/loveisintheair/episodio-1_F310769301000104
[debug] Command-line config: ['-v', '-F', 'https://www.mediasetplay.mediaset.it/video/loveisintheair/episodio-1_F310769301000104']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.05.20 (exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg N-91266-g8c20ea8ee0, ffprobe N-91266-g8c20ea8ee0
[debug] Proxy map: {}
[debug] [Mediaset] Extracting URL: https://www.mediasetplay.mediaset.it/video/loveisintheair/episodio-1_F310769301000104
[Mediaset] F310769301000104: Downloading JSON metadata
[Mediaset] F310769301000104: Downloading MPEG4 SD SMIL data
[Mediaset] F310769301000104: Downloading MPEG4 HD SMIL data
ERROR: This content is not available in your location.
You might want to use a VPN or a proxy server (with --proxy) to workaround.
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1126, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1147, in __extract_info
  File "yt_dlp\extractor\common.py", line 564, in extract
  File "yt_dlp\extractor\mediaset.py", line 157, in _real_extract
  File "yt_dlp\extractor\mediaset.py", line 142, in _real_extract
  File "yt_dlp\extractor\theplatform.py", line 47, in _extract_theplatform_smil
  File "yt_dlp\extractor\common.py", line 1001, in raise_geo_restricted
yt_dlp.utils.GeoRestrictedError: This content is not available in your location.

```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

Trying to download a video from mediaset.it results in a "Geo Restricted Error" even though I'm attempting the download from Italy, so without the use of a VPN or proxy.

This does not occur with all videos, for example this works right now (https://www.mediasetplay.mediaset.it/video/mrwronglezionidamore/episodio-1_F310575103000102), but it also seem to work fine if the video is not brand new, but not always.

In all cases, videos can be downloaded using the mpd urls instead.
[Broken] Mediaset [ITALY]
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running yt-dlp version **2021.05.20**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.05.20
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
D:\TEMP\YDL\YDLP>yt-dlp -v -F https://www.mediasetplay.mediaset.it/video/loveisintheair/episodio-1_F310769301000104
[debug] Command-line config: ['-v', '-F', 'https://www.mediasetplay.mediaset.it/video/loveisintheair/episodio-1_F310769301000104']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.05.20 (exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg N-91266-g8c20ea8ee0, ffprobe N-91266-g8c20ea8ee0
[debug] Proxy map: {}
[debug] [Mediaset] Extracting URL: https://www.mediasetplay.mediaset.it/video/loveisintheair/episodio-1_F310769301000104
[Mediaset] F310769301000104: Downloading JSON metadata
[Mediaset] F310769301000104: Downloading MPEG4 SD SMIL data
[Mediaset] F310769301000104: Downloading MPEG4 HD SMIL data
ERROR: This content is not available in your location.
You might want to use a VPN or a proxy server (with --proxy) to workaround.
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1126, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1147, in __extract_info
  File "yt_dlp\extractor\common.py", line 564, in extract
  File "yt_dlp\extractor\mediaset.py", line 157, in _real_extract
  File "yt_dlp\extractor\mediaset.py", line 142, in _real_extract
  File "yt_dlp\extractor\theplatform.py", line 47, in _extract_theplatform_smil
  File "yt_dlp\extractor\common.py", line 1001, in raise_geo_restricted
yt_dlp.utils.GeoRestrictedError: This content is not available in your location.

```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

Trying to download a video from mediaset.it results in a "Geo Restricted Error" even though I'm attempting the download from Italy, so without the use of a VPN or proxy.

This does not occur with all videos, for example this works right now (https://www.mediasetplay.mediaset.it/video/mrwronglezionidamore/episodio-1_F310575103000102), but it also seem to work fine if the video is not brand new, but not always.

In all cases, videos can be downloaded using the mpd urls instead.


Discussion:
Note for devs: they seem to block VPNs/proxies.
> 
> 
> Note for devs: they seem to block VPNs/proxies.

VPN (Nord) works for me, both viewing and downloading.
Related: https://github.com/ytdl-org/youtube-dl/issues/29218
I have the same issue, connecting via Nord Italy servers.

https://www.mediasetplay.mediaset.it/video/masantoniosezionescomparsi/pt1--cap1_F309911301001C01

All episodes of the above show can be played in the browser, but youtube-dl version 2021.06.06 can only download episodes 2, 3, 4, 7, 9 and 10.  Episodes 1, 5, 6 and 8 cannot be downloaded.

I'd be very grateful if a dev could look at this.
@nixxo may be interested
ok, got it.

I think I fixed it. I'll do some more tests to see if it works on other links as well.
Note for devs: they seem to block VPNs/proxies.
> 
> 
> Note for devs: they seem to block VPNs/proxies.

VPN (Nord) works for me, both viewing and downloading.
Related: https://github.com/ytdl-org/youtube-dl/issues/29218
I have the same issue, connecting via Nord Italy servers.

https://www.mediasetplay.mediaset.it/video/masantoniosezionescomparsi/pt1--cap1_F309911301001C01

All episodes of the above show can be played in the browser, but youtube-dl version 2021.06.06 can only download episodes 2, 3, 4, 7, 9 and 10.  Episodes 1, 5, 6 and 8 cannot be downloaded.

I'd be very grateful if a dev could look at this.
@nixxo may be interested
ok, got it.

I think I fixed it. I'll do some more tests to see if it works on other links as well.

Resolving patch:
diff --git a/yt_dlp/extractor/mediaset.py b/yt_dlp/extractor/mediaset.py
--- a/yt_dlp/extractor/mediaset.py
+++ b/yt_dlp/extractor/mediaset.py
@@ -30,20 +30,20 @@ class MediasetIE(ThePlatformBaseIE):
                     '''
     _TESTS = [{
         # full episode
-        'url': 'https://www.mediasetplay.mediaset.it/video/hellogoodbye/quarta-puntata_FAFU000000661824',
-        'md5': '9b75534d42c44ecef7bf1ffeacb7f85d',
+        'url': 'https://www.mediasetplay.mediaset.it/video/mrwronglezionidamore/episodio-1_F310575103000102',
+        'md5': 'a7e75c6384871f322adb781d3bd72c26',
         'info_dict': {
-            'id': 'FAFU000000661824',
+            'id': 'F310575103000102',
             'ext': 'mp4',
-            'title': 'Quarta puntata',
+            'title': 'Episodio 1',
             'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
             'thumbnail': r're:^https?://.*\.jpg$',
-            'duration': 1414.26,
-            'upload_date': '20161107',
-            'series': 'Hello Goodbye',
-            'timestamp': 1478532900,
-            'uploader': 'Rete 4',
-            'uploader_id': 'R4',
+            'duration': 2682.0,
+            'upload_date': '20210530',
+            'series': 'Mr Wrong - Lezioni d\'amore',
+            'timestamp': 1622413946,
+            'uploader': 'Canale 5',
+            'uploader_id': 'C5',
         },
     }, {
         'url': 'https://www.mediasetplay.mediaset.it/video/matrix/puntata-del-25-maggio_F309013801000501',
@@ -54,10 +54,10 @@ class MediasetIE(ThePlatformBaseIE):
             'title': 'Puntata del 25 maggio',
             'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
             'thumbnail': r're:^https?://.*\.jpg$',
-            'duration': 6565.007,
-            'upload_date': '20180526',
+            'duration': 6565.008,
+            'upload_date': '20200903',
             'series': 'Matrix',
-            'timestamp': 1527326245,
+            'timestamp': 1599172492,
             'uploader': 'Canale 5',
             'uploader_id': 'C5',
         },
@@ -135,36 +135,38 @@ def _real_extract(self, url):
         formats = []
         subtitles = {}
         first_e = None
-        for asset_type in ('SD', 'HD'):
-            # TODO: fixup ISM+none manifest URLs
-            for f in ('MPEG4', 'MPEG-DASH+none', 'M3U+none'):
-                try:
-                    tp_formats, tp_subtitles = self._extract_theplatform_smil(
-                        update_url_query('http://link.theplatform.%s/s/%s' % (self._TP_TLD, tp_path), {
-                            'mbr': 'true',
-                            'formats': f,
-                            'assetTypes': asset_type,
-                        }), guid, 'Downloading %s %s SMIL data' % (f.split('+')[0], asset_type))
-                except ExtractorError as e:
-                    if not first_e:
-                        first_e = e
-                    break
-                for tp_f in tp_formats:
-                    tp_f['quality'] = 1 if asset_type == 'HD' else 0
-                formats.extend(tp_formats)
-                subtitles = self._merge_subtitles(subtitles, tp_subtitles)
+        asset_type = 'HD,browser,geoIT|SD,browser,geoIT|geoNo:HD,browser,geoIT|geoNo:SD,browser,geoIT|geoNo'
+        # TODO: fixup ISM+none manifest URLs
+        for f in ('MPEG4', 'MPEG-DASH+none', 'M3U+none'):
+            try:
+                tp_formats, tp_subtitles = self._extract_theplatform_smil(
+                    update_url_query('http://link.theplatform.%s/s/%s' % (self._TP_TLD, tp_path), {
+                        'mbr': 'true',
+                        'formats': f,
+                        'assetTypes': asset_type,
+                    }), guid, 'Downloading %s SMIL data' % (f.split('+')[0]))
+            except ExtractorError as e:
+                if not first_e:
+                    first_e = e
+                break
+            formats.extend(tp_formats)
+            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
         if first_e and not formats:
             raise first_e
         self._sort_formats(formats)
 
-        fields = []
-        for templ, repls in (('tvSeason%sNumber', ('', 'Episode')), ('mediasetprogram$%s', ('brandTitle', 'numberOfViews', 'publishInfo'))):
-            fields.extend(templ % repl for repl in repls)
         feed_data = self._download_json(
-            'https://feed.entertainment.tv.theplatform.eu/f/PR1GhC/mediaset-prod-all-programs/guid/-/' + guid,
-            guid, fatal=False, query={'fields': ','.join(fields)})
+            'https://feed.entertainment.tv.theplatform.eu/f/PR1GhC/mediaset-prod-all-programs-v2/guid/-/' + guid,
+            guid, fatal=False)
         if feed_data:
             publish_info = feed_data.get('mediasetprogram$publishInfo') or {}
+            thumbnails = feed_data.get('thumbnails') or {}
+            thumbnail = None
+            for key, value in thumbnails.items():
+                if key.startswith('image_keyframe_poster-'):
+                    thumbnail = value.get('url')
+                    break
+
             info.update({
                 'episode_number': int_or_none(feed_data.get('tvSeasonEpisodeNumber')),
                 'season_number': int_or_none(feed_data.get('tvSeasonNumber')),
@@ -172,6 +174,7 @@ def _real_extract(self, url):
                 'uploader': publish_info.get('description'),
                 'uploader_id': publish_info.get('channel'),
                 'view_count': int_or_none(feed_data.get('mediasetprogram$numberOfViews')),
+                'thumbnail': thumbnail,
             })
 
         info.update({


Test patch: