Repository: yt-dlp/yt-dlp

Issue:
goplay extractor broken
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2023.01.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Belgium

### Provide a description that is worded well enough to be understood

goplay.be now loads newer videos in a new format. (DASH MPD)
which means the current extractor is broken. 


Extracted the following information so far; 

``` BASH
curl -X POST https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10150078534976527/streams
```

``` JSON
{
  "stream_id": "091f9127-c615-4354-9006-af3fb09ac261",
  "duration": 2687.64,
  "stream_type": "on_demand",
  "stream_manifest": "https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10150078534976527/GRQ/streams/091f9127-c615-4354-9006-af3fb09ac261/manifest.mpd",
  "manifest_format": "dash",
  "time_events_url": "https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10150078534976527/GRQ/streams/091f9127-c615-4354-9006-af3fb09ac261/time-events.json"
}
```

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['--verbose', '-o', '/media/data2/series/Big Brother/Season 3/%(series)s - S0%(season_number)sE%(episode_number)s.%(ext)s', '--no-hls-use-mpegts', '-u', 'PRIVATE', '-p', 'PRIVATE', '-N', '8', '-f', 'bestvideo+bestaudio', '--merge-output-format', 'mp4', 'https://www.goplay.be/video/big-brother/big-brother-s3/big-brother-s3-aflevering-32']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version 2023.01.06 [6becd2508] (linux_exe)
[debug] Python 3.10.6 (CPython x86_64 64bit) - Linux-5.15.0-60-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-3.3.1, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1767 extractors
[GoPlay] Logging in
[GoPlay] Authenticating username
[GoPlay] Authenticating password
[GoPlay] Extracting URL: https://www.goplay.be/video/big-brother/big-brother-s3/big-brother-s3-aflevering-32
[GoPlay] big-brother-s3-aflevering-32: Downloading webpage
[GoPlay] 5bd06f78-6e20-439b-8d67-cfe152c0cd68: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError('manifestUrls')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp/extractor/common.py", line 693, in extract
  File "yt_dlp/extractor/goplay.py", line 83, in _real_extract
KeyError: 'manifestUrls'
```



Discussion:
Doesn't look like it's just newer videos - for example, I could download De Rechtbank S12E05 just fine last sunday using the "old" URL format, but S12E02 right now has the new format too.

Thanks for your investigation though, I thought I was going crazy. I just verified your findings for another video 😊 

the api of goplay.be returns following:

`GET https://api.goplay.be/web/v1/videos/long-form/26e19146-57ef-4ea4-ab41-e247061d8407`
```
{
    "videoUuid": "26e19146-57ef-4ea4-ab41-e247061d8407",
    "title": "Aflevering 3",
    "brand": "PLAY4",
    "duration": 2643,
    "dates": {
        "publishDate": 1673985600,
        "unpublishDate": 1684619940
    },
    "adType": "SSAI",
    "ads": {
        "enablePreroll": true
    },
    "flags": {
        "hasProductPlacement": false,
        "isProtected": true,
        "isSeekable": true,
        "isStreaming": false,
        "isDrm": false
    },
    "position": 0,
    "streamTypes": [],
    "ssai": {
        "contentSourceID": "2615619",
        "videoID": "10149680614757527"
    },
    "parentalRating": "12+",
    "tracking": {
        "cimTag": "vid.tvi.ep.vod.free"
    }
}
```

It then does a POST to `https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10149680614692527/streams` (`2615619` being the `contentSourceID` and `10149680614692527` the `videoId`

Doubleclick returns the `stream_manifest` which contains the video including ads and prerolls.
```
{
    "stream_id": "3ecc516e-14e7-4276-a97e-35968b3b17ac",
    "duration": 2750.36,
    "stream_type": "on_demand",
    "stream_manifest": "https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10149680614692527/GRQ/streams/3ecc516e-14e7-4276-a97e-35968b3b17ac/manifest.mpd",
    "manifest_format": "dash",
    "time_events_url": "https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10149680614692527/GRQ/streams/3ecc516e-14e7-4276-a97e-35968b3b17ac/time-events.json",
    "subtitles": [
        {
            "language": "nl",
            "ttml": "https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10149680614692527/GRQ/streams/3ecc516e-14e7-4276-a97e-35968b3b17ac/ttml/e743427a-605e-45e1-a53a-1c35557523af.ttml",
            "webvtt": "https://pubads.g.doubleclick.net/ondemand/dash/content/2615619/vid/10149680614692527/GRQ/streams/3ecc516e-14e7-4276-a97e-35968b3b17ac/vtt/e743427a-605e-45e1-a53a-1c35557523af.vtt"
        }
    ]
}
```
Downloading the MPD file directly via yt-dlp {stream_manifest} gives me very mixed results - it seems the audio and video are now split up in 3 parts...
```
WARNING: [generic] Falling back on generic information extractor
[generic] manifest: Extracting information
[info] Available formats for manifest:
ID                                     EXT RESOLUTION FPS │   TBR PROTO │ VCODEC        VBR ACODEC      ABR ASR MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
f14a8946-45cc-4c18-a38b-0030d1578d78-0 m4a audio only     │  128k dash  │ audio only        mp4a.40.2  128k 48k [nl] DASH audio, m4a_dash
fd37a468-7700-4068-b941-6df53a510710   m4a audio only     │  128k dash  │ audio only        mp4a.40.2  128k 48k [nl] DASH audio, m4a_dash
fdfc6f74-68ca-4933-bc6d-a19a6881772c   m4a audio only     │  128k dash  │ audio only        mp4a.40.2  128k 48k [nl] DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-1 m4a audio only     │  129k dash  │ audio only        mp4a.40.2  129k 48k DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-2 m4a audio only     │  129k dash  │ audio only        mp4a.40.2  129k 48k DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-3 m4a audio only     │  129k dash  │ audio only        mp4a.40.2  129k 48k DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-4 m4a audio only     │  129k dash  │ audio only        mp4a.40.2  129k 48k DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-5 m4a audio only     │  129k dash  │ audio only        mp4a.40.2  129k 48k DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-6 m4a audio only     │  130k dash  │ audio only        mp4a.40.2  130k 48k DASH audio, m4a_dash
f14a8946-45cc-4c18-a38b-0030d1578d78-7 m4a audio only     │  131k dash  │ audio only        mp4a.40.2  131k 48k DASH audio, m4a_dash
98304c2b-76c0-453b-85dd-99a3df729a06-0 mp4 640x360     25 │  200k dash  │ avc1.64001e  200k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-1 mp4 640x360     25 │ 1004k dash  │ avc1.64001e 1004k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-2 mp4 640x360     25 │ 1004k dash  │ avc1.64001e 1004k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-3 mp4 640x360     25 │ 1004k dash  │ avc1.64001e 1004k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-4 mp4 640x360     25 │ 1016k dash  │ avc1.64001e 1016k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-5 mp4 640x360     25 │ 1053k dash  │ avc1.64001e 1053k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-6 mp4 640x360     25 │ 1053k dash  │ avc1.64001e 1053k video only          DASH video, mp4_dash
793e68e8-9bef-44cf-8832-08f5a9385356   mp4 768x432     25 │ 1000k dash  │ avc1.64001E 1000k video only          DASH video, mp4_dash
98304c2b-76c0-453b-85dd-99a3df729a06-7 mp4 768x432     25 │ 1000k dash  │ avc1.64001E 1000k video only          DASH video, mp4_dash
fdd8c87e-2e2d-41c7-a291-45ec95cf14a8   mp4 768x432     25 │ 1000k dash  │ avc1.64001E 1000k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-0 mp4 1024x576    25 │  357k dash  │ avc1.4d401f  357k video only          DASH video, mp4_dash
368b7c6c-6c67-45a3-8879-8f52f442a025   mp4 1024x576    25 │ 1500k dash  │ avc1.64001F 1500k video only          DASH video, mp4_dash
60987e7d-a6cc-491c-91a4-99a60a49e26b   mp4 1024x576    25 │ 1500k dash  │ avc1.64001F 1500k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-1 mp4 1024x576    25 │ 1500k dash  │ avc1.64001F 1500k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-2 mp4 1024x576    25 │ 1604k dash  │ avc1.4d401f 1604k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-3 mp4 1024x576    25 │ 1604k dash  │ avc1.4d401f 1604k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-4 mp4 1024x576    25 │ 1604k dash  │ avc1.4d401f 1604k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-5 mp4 1024x576    25 │ 1631k dash  │ avc1.4d401f 1631k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-6 mp4 1024x576    25 │ 1678k dash  │ avc1.4d401f 1678k video only          DASH video, mp4_dash
9860ee25-eddc-4f30-b3fc-dd443846c6ec-7 mp4 1024x576    25 │ 1678k dash  │ avc1.4d401f 1678k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-0 mp4 1280x720    25 │  479k dash  │ avc1.64001f  479k video only          DASH video, mp4_dash
0fc32a6c-169e-467e-a9e8-f50863c9634b   mp4 1280x720    25 │ 2300k dash  │ avc1.64001F 2300k video only          DASH video, mp4_dash
6aeaa3a7-0ad2-4b24-adba-4fcf994dbb83   mp4 1280x720    25 │ 2300k dash  │ avc1.64001F 2300k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-1 mp4 1280x720    25 │ 2300k dash  │ avc1.64001F 2300k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-2 mp4 1280x720    25 │ 2511k dash  │ avc1.64001f 2511k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-3 mp4 1280x720    25 │ 2511k dash  │ avc1.64001f 2511k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-4 mp4 1280x720    25 │ 2511k dash  │ avc1.64001f 2511k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-5 mp4 1280x720    25 │ 2566k dash  │ avc1.64001f 2566k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-6 mp4 1280x720    25 │ 2615k dash  │ avc1.64001f 2615k video only          DASH video, mp4_dash
809f3f8a-2d21-4b2d-992d-b18c871c0d83-7 mp4 1280x720    25 │ 2615k dash  │ avc1.64001f 2615k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-0 mp4 1920x1080   25 │  808k dash  │ avc1.640028  808k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-1 mp4 1920x1080   25 │ 3848k dash  │ avc1.640028 3848k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-2 mp4 1920x1080   25 │ 3848k dash  │ avc1.640028 3848k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-3 mp4 1920x1080   25 │ 3848k dash  │ avc1.640028 3848k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-4 mp4 1920x1080   25 │ 3858k dash  │ avc1.640028 3858k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-5 mp4 1920x1080   25 │ 3858k dash  │ avc1.640028 3858k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-6 mp4 1920x1080   25 │ 3958k dash  │ avc1.640028 3958k video only          DASH video, mp4_dash
1f8fb6c1-37cb-44ef-a233-344a7d0e1bc2-7 mp4 1920x1080   25 │ 4300k dash  │ avc1.640032 4300k video only          DASH video, mp4_dash
351adfcf-f794-4d2a-b0ad-a7f7ab9321bc   mp4 1920x1080   25 │ 4300k dash  │ avc1.640032 4300k video only          DASH video, mp4_dash
b9ffb2aa-e9de-4d6b-bcc9-258abee80305   mp4 1920x1080   25 │ 4300k dash  │ avc1.640032 4300k video only          DASH video, mp4_dash
```



@jeroenj do you have any insight?
@michaelarnauts @cnugteren do any of you have an idea how to fix this issue, since you were involved in older patches? De Mol is going to start again soon and I am desperate :(
> @CNugteren do any of you have an idea how to fix this issue, since you were involved in older patches?

Don't count on me too much, all I did last time was apply patches from different people together in a new branch, I didn't actually go into the details of how the extractor works.

Hopefully someone can fix it in time!
Same for me, GoPlay moved over for some streams to the doubleclick service, where ads are injected directly in the stream. It's a totally different stream compared with the usual streams. 

You might find more info when searching about https://pubads.g.doubleclick.net/ondemand/dash

Maybe there is another service that is using doubleclick that had some more documentation.
I haven't looked into this yet either. Since the GoPlay app now supports AirPlay I'm using that instead of downloading it first.
Currently receiving errors on https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/goplay.py#L83

I checked it and indeed the request returns no object with 'manifestUrls' key.

--

I checked the network tab and found a 'https://pubads.g.doubleclick.net/ondemand/dash/content/' url and when downloading that using yt-dlp it downloads a 1080p file but only the first 18-19min..
> Currently receiving errors on https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/goplay.py#L83
> 
> I checked it and indeed the request returns no object with 'manifestUrls' key.
> 
> --
> 
> I checked the network tab and found a 'https://pubads.g.doubleclick.net/ondemand/dash/content/' url and when downloading that using yt-dlp it downloads a 1080p file but only the first 18-19min..

I noticed the same. As @b12e said, the audio and video is now split up in 3 parts, probably to have 2 blocks of ads in-between. There should be a way to download the 3 parts and then merge them, but I'm not that knowledgeable about ffmpeg and did not have time yet
the 18min clip had video and sound. I opened the manifest file using VLC and after 18m an ad got shown, and then restarted from 0 and showed the next part of the video. Any idea how I can download all the parts using yt-dlp?
you can download all parts according to the wiki:
`yt-dlp -F <url to manifest>` will give a list of all formats, each leading with a tag
`yt-dlp -f <tag> <url to manifest>` will download that specific video

The problem that I have is that I don't know how to select what audio file to download together with the video, so I have the wrong audio file stitched to the video unfortunately. I also don't know how to merge all video parts
I figured it out manually how to match the video + audio and indeed download every part. I guess stitching can be down using ffmpeg.. I can explain how I matched them on discord if you want, maybe someone can figure out how to do it automaticly and update the extractor
Would it be possible to explain it here? That way other people can also have a workaround and as you said, it might inspire people to write a workaround in the extractor
I downloaded the manifest file and checked it

![image](https://user-images.githubusercontent.com/17174443/226605573-6847144b-2dfb-4071-9df0-23211dad045b.png)

if we remove the preroll + ads
- 3 parts left

![image](https://user-images.githubusercontent.com/17174443/226605854-2f76690d-e4ef-49e1-a0dd-3325e19042ec.png)


Every part has 1080p 720p ... and 1 audio link
![image](https://user-images.githubusercontent.com/17174443/226606097-7d277a98-6dff-43ee-bf54-701bdafdefdb.png)

Then I downloaded them using

part1: yt-dlp -f  qualityID+audioID <url to manifest>
part2: yt-dlp -f  qualityID+audioID <url to manifest>
part3: yt-dlp -f  qualityID+audioID <url to manifest>
Thanks! Currently at work so I'll check myself later, much appreciated! How did you figure out what audio goes with what video?

EDIT nvm doesn't seem to be relevant if you actually download the manifest. Using the -F flag gives me everything at once, so 3 audio files, and 3 video files per resolution
Indeed, but this is a ton of manual work to download 1 episode.. still need to figure out the stitching part
I figured out how to merge the files successfully with the help of [this stackoverflow article](https://stackoverflow.com/questions/7333232/how-to-concatenate-two-mp4-files-using-ffmpeg): first make a document with all names of the files that looks like this
```
$ cat mylist.txt
file '/path/to/file1'
file '/path/to/file2'
file '/path/to/file3'
```
and then execute the following ffmpeg commando
    
`$ ffmpeg -f concat -safe 0 -i mylist.txt -c copy output.mp4`
just tested the ffmpeg stitching works great :D 

now someone has to automate it 

Resolving patch:
diff --git a/yt_dlp/extractor/goplay.py b/yt_dlp/extractor/goplay.py
--- a/yt_dlp/extractor/goplay.py
+++ b/yt_dlp/extractor/goplay.py
@@ -40,6 +40,22 @@ class GoPlayIE(InfoExtractor):
             'title': 'A Family for the Holidays',
         },
         'skip': 'This video is only available for registered users'
+    }, {
+        'url': 'https://www.goplay.be/video/de-mol/de-mol-s11/de-mol-s11-aflevering-1#autoplay',
+        'info_dict': {
+            'id': '03eb8f2f-153e-41cb-9805-0d3a29dab656',
+            'ext': 'mp4',
+            'title': 'S11 - Aflevering 1',
+            'episode': 'Episode 1',
+            'series': 'De Mol',
+            'season_number': 11,
+            'episode_number': 1,
+            'season': 'Season 11'
+        },
+        'params': {
+            'skip_download': True
+        },
+        'skip': 'This video is only available for registered users'
     }]
 
     _id_token = None
@@ -77,16 +93,39 @@ def _real_extract(self, url):
 
         api = self._download_json(
             f'https://api.goplay.be/web/v1/videos/long-form/{video_id}',
-            video_id, headers={'Authorization': 'Bearer %s' % self._id_token})
+            video_id, headers={
+                'Authorization': 'Bearer %s' % self._id_token,
+                **self.geo_verification_headers(),
+            })
+
+        if 'manifestUrls' in api:
+            formats, subtitles = self._extract_m3u8_formats_and_subtitles(
+                api['manifestUrls']['hls'], video_id, ext='mp4', m3u8_id='HLS')
 
-        formats, subs = self._extract_m3u8_formats_and_subtitles(
-            api['manifestUrls']['hls'], video_id, ext='mp4', m3u8_id='HLS')
+        else:
+            if 'ssai' not in api:
+                raise ExtractorError('expecting Google SSAI stream')
+
+            ssai_content_source_id = api['ssai']['contentSourceID']
+            ssai_video_id = api['ssai']['videoID']
+
+            dai = self._download_json(
+                f'https://dai.google.com/ondemand/dash/content/{ssai_content_source_id}/vid/{ssai_video_id}/streams',
+                video_id, data=b'{"api-key":"null"}',
+                headers={'content-type': 'application/json'})
+
+            periods = self._extract_mpd_periods(dai['stream_manifest'], video_id)
+
+            # skip pre-roll and mid-roll ads
+            periods = [p for p in periods if '-ad-' not in p['id']]
+
+            formats, subtitles = self._merge_mpd_periods(periods)
 
         info_dict.update({
             'id': video_id,
             'formats': formats,
+            'subtitles': subtitles,
         })
-
         return info_dict
 
 


Test patch: