Repository: yt-dlp/yt-dlp

Issue:
extractor/europarl no longer works - new video provider (arbor.nl)
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2023.02.17** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

_No response_

### Provide a description that is worded well enough to be understood

The European Parliament has started using **arbor.nl** for video streaming. As a result, the extractor in place that handles their old provider, **vuplay**, is no longer finding the video stream.

You can see an example here:
`https://multimedia.europarl.europa.eu/en/webstreaming/committee-on-culture-and-education_20230301-1130-COMMITTEE-CULT`

The new **arbor.nl** service appears to be using **Azure** for the video streaming, from looking at the payload, with **theoplayer** as the player in the browser. The `webstreaming` page still calls a `connectedviews.eu `iframe for the player, per the example above:
`https://acs.europarl.connectedviews.eu/embed/meeting/?refid=20230301-1130-COMMITTEE-CULT`

This appears to be referencing an API call as follows:
`https://acs-api.europarl.connectedviews.eu/api/FullMeeting?api-version=1.0&tenantId=bae646ca-1fc8-4363-80ba-2c04f06b4968&externalReference=20230301-1130-COMMITTEE-CULT`

where:

- **tenantId** is constant like a client id or key
- **externalReference** is the key using the refId from the iFrame

That API call returns a json payload where there's a ton of meeting info, including an array with the videos, but it calls out a primary in `meetingVideo` -> `hlsUrl` for events, both live and pre-recorded. The `status` indicates if the meeting is:

- Upcoming `(none)`
- Live `(started)`
- Complete / VOD `(stopped)`

```
{
	"id": "f5ce345a-4184-462d-34c6-08db134184b0",
	"tenantId": "bae646ca-1fc8-4363-80ba-2c04f06b4968",
...
	"meetingVideo": {
		"id": "2220a771-4501-42fa-9e6b-08db14b5005b",
		"meetingId": "f5ce345a-4184-462d-34c6-08db134184b0",
		"label": "Video NorthEurope",
		"startDateTime": "2023-03-02T10:29:05.3215897+00:00",
		"status": "Started",
		"hlsUrl": "https://manifest-backup.europarl.streaming.arbor.nl/live/ch12bxl/index.m3u8",
		"thumbnailUrl": "",
		"streamingRegionId": "4cfaca02-3001-40f3-a611-146a630b5c13",
		"streamingRegion": {
			"id": "4cfaca02-3001-40f3-a611-146a630b5c13",
			"tenantId": "bae646ca-1fc8-4363-80ba-2c04f06b4968",
			"name": "NorthEurope",
			"priority": 2,
			"enabled": true
		},
		"encoderChannelId": "24508f5f-2fa8-48ea-827f-9b2dec035468"
	},
	...
}
```



### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
hal9000:$ yt-dlp -F --verbose https://multimedia.europarl.europa.eu/en/webstreaming/plenary-session_20220914-0900-PLENARY
[debug] Command-line config: ['-F', '--verbose', 'https://multimedia.europarl.europa.eu/en/webstreaming/plenary-session_20220914-0900-PLENARY']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.9.16 (CPython x86_64 64bit) - Linux-5.15.0-1028-aws-x86_64-with-glibc2.31 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.12.07, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[EuroParlWebstream] Extracting URL: https://multimedia.europarl.europa.eu/en/webstreaming/plenary-session_20220914-0900-PLENARY
[EuroParlWebstream] 20220914-0900-PLENARY: Downloading JSON metadata
ERROR: [EuroParlWebstream] 20220914-0900-PLENARY: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by <HTTPError 400: 'Bad Request'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 693, in extract
    ie_result = self._real_extract(url)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/europa.py", line 152, in _real_extract
    json_info = self._download_json(
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 1047, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 1011, in download_handle
    res = self._download_webpage_handle(
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 881, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 838, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 820, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "/home/ubuntu/.local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 3726, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request



The non-verbose:
hal9000:$ yt-dlp -F https://multimedia.europarl.europa.eu/en/webstreaming/plenary-session_20220914-0900-PLENARY
[EuroParlWebstream] Extracting URL: https://multimedia.europarl.europa.eu/en/webstreaming/plenary-session_20220914-0900-PLENARY
[EuroParlWebstream] 20220914-0900-PLENARY: Downloading JSON metadata
ERROR: [EuroParlWebstream] 20220914-0900-PLENARY: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by <HTTPError 400: 'Bad Request'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```



Discussion:

Resolving patch:
diff --git a/yt_dlp/extractor/europa.py b/yt_dlp/extractor/europa.py
--- a/yt_dlp/extractor/europa.py
+++ b/yt_dlp/extractor/europa.py
@@ -6,6 +6,7 @@
     parse_iso8601,
     parse_qs,
     qualities,
+    traverse_obj,
     unified_strdate,
     xpath_text
 )
@@ -92,82 +93,81 @@ def get_item(type_, preference):
 
 class EuroParlWebstreamIE(InfoExtractor):
     _VALID_URL = r'''(?x)
-        https?://(?:multimedia|webstreaming)\.europarl\.europa\.eu/[^/#?]+/
-        (?:embed/embed\.html\?event=|(?!video)[^/#?]+/[\w-]+_)(?P<id>[\w-]+)
+        https?://multimedia\.europarl\.europa\.eu/[^/#?]+/
+        (?:(?!video)[^/#?]+/[\w-]+_)(?P<id>[\w-]+)
     '''
     _TESTS = [{
         'url': 'https://multimedia.europarl.europa.eu/pl/webstreaming/plenary-session_20220914-0900-PLENARY',
         'info_dict': {
-            'id': 'bcaa1db4-76ef-7e06-8da7-839bd0ad1dbe',
+            'id': '62388b15-d85b-4add-99aa-ba12ccf64f0d',
             'ext': 'mp4',
-            'release_timestamp': 1663137900,
             'title': 'Plenary session',
+            'release_timestamp': 1663139069,
             'release_date': '20220914',
         },
         'params': {
             'skip_download': True,
         }
     }, {
-        'url': 'https://multimedia.europarl.europa.eu/pl/webstreaming/eu-cop27-un-climate-change-conference-in-sharm-el-sheikh-egypt-ep-delegation-meets-with-ngo-represen_20221114-1600-SPECIAL-OTHER',
+        # live webstream
+        'url': 'https://multimedia.europarl.europa.eu/en/webstreaming/euroscola_20221115-1000-SPECIAL-EUROSCOLA',
         'info_dict': {
-            'id': 'a8428de8-b9cd-6a2e-11e4-3805d9c9ff5c',
             'ext': 'mp4',
-            'release_timestamp': 1668434400,
-            'release_date': '20221114',
-            'title': 'md5:d3550280c33cc70e0678652e3d52c028',
+            'id': '510eda7f-ba72-161b-7ee7-0e836cd2e715',
+            'release_timestamp': 1668502800,
+            'title': 'Euroscola 2022-11-15 19:21',
+            'release_date': '20221115',
+            'live_status': 'is_live',
         },
-        'params': {
-            'skip_download': True,
-        }
+        'skip': 'not live anymore'
     }, {
-        # embed webpage
-        'url': 'https://webstreaming.europarl.europa.eu/ep/embed/embed.html?event=20220914-0900-PLENARY&language=en&autoplay=true&logo=true',
+        'url': 'https://multimedia.europarl.europa.eu/en/webstreaming/committee-on-culture-and-education_20230301-1130-COMMITTEE-CULT',
         'info_dict': {
-            'id': 'bcaa1db4-76ef-7e06-8da7-839bd0ad1dbe',
+            'id': '7355662c-8eac-445e-4bb9-08db14b0ddd7',
             'ext': 'mp4',
-            'title': 'Plenary session',
-            'release_date': '20220914',
-            'release_timestamp': 1663137900,
-        },
-        'params': {
-            'skip_download': True,
+            'release_date': '20230301',
+            'title': 'Committee on Culture and Education',
+            'release_timestamp': 1677666641,
         }
     }, {
-        # live webstream
-        'url': 'https://multimedia.europarl.europa.eu/en/webstreaming/euroscola_20221115-1000-SPECIAL-EUROSCOLA',
+        # live stream
+        'url': 'https://multimedia.europarl.europa.eu/en/webstreaming/committee-on-environment-public-health-and-food-safety_20230524-0900-COMMITTEE-ENVI',
         'info_dict': {
+            'id': 'e4255f56-10aa-4b3c-6530-08db56d5b0d9',
             'ext': 'mp4',
-            'id': '510eda7f-ba72-161b-7ee7-0e836cd2e715',
-            'release_timestamp': 1668502800,
-            'title': 'Euroscola 2022-11-15 19:21',
-            'release_date': '20221115',
+            'release_date': '20230524',
+            'title': r're:Committee on Environment, Public Health and Food Safety \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}',
+            'release_timestamp': 1684911541,
             'live_status': 'is_live',
         },
-        'skip': 'not live anymore'
+        'skip': 'Not live anymore'
     }]
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
+
+        webpage_nextjs = self._search_nextjs_data(webpage, display_id)['props']['pageProps']
 
         json_info = self._download_json(
-            'https://vis-api.vuplay.co.uk/event/external', display_id,
+            'https://acs-api.europarl.connectedviews.eu/api/FullMeeting', display_id,
             query={
-                'player_key': 'europarl|718f822c-a48c-4841-9947-c9cb9bb1743c',
-                'external_id': display_id,
+                'api-version': 1.0,
+                'tenantId': 'bae646ca-1fc8-4363-80ba-2c04f06b4968',
+                'externalReference': display_id
             })
 
-        formats, subtitles = self._extract_mpd_formats_and_subtitles(json_info['streaming_url'], display_id)
-        fmts, subs = self._extract_m3u8_formats_and_subtitles(
-            json_info['streaming_url'].replace('.mpd', '.m3u8'), display_id)
-
-        formats.extend(fmts)
-        self._merge_subtitles(subs, target=subtitles)
+        formats, subtitles = [], {}
+        for hls_url in traverse_obj(json_info, ((('meetingVideo'), ('meetingVideos', ...)), 'hlsUrl')):
+            fmt, subs = self._extract_m3u8_formats_and_subtitles(hls_url, display_id)
+            formats.extend(fmt)
+            self._merge_subtitles(subs, target=subtitles)
 
         return {
             'id': json_info['id'],
-            'title': json_info.get('title'),
+            'title': traverse_obj(webpage_nextjs, (('mediaItem', 'title'), ('title', )), get_all=False),
             'formats': formats,
             'subtitles': subtitles,
-            'release_timestamp': parse_iso8601(json_info.get('published_start')),
-            'is_live': 'LIVE' in json_info.get('state', '')
+            'release_timestamp': parse_iso8601(json_info.get('startDateTime')),
+            'is_live': traverse_obj(webpage_nextjs, ('mediaItem', 'mediaSubType')) == 'Live'
         }


Test patch: