Repository: yt-dlp/yt-dlp
Pull request: [on24] Add new extractor
## Please follow the guide below

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes [ ] relevant to your *pull request* (like that [x])
- Use *Preview* tab to see how your *pull request* will actually look like

---

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [adding new extractor tutorial](https://github.com/ytdl-org/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/ytdl-org/youtube-dl#youtube-dl-coding-conventions) sections
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [x] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [ ] Bug fix
- [ ] Improvement
- [x] New extractor
- [ ] New feature

---

### Description of your *pull request* and other information

This PR adds a new extractor for ON24 events.

(Fixes ytdl-org/youtube-dl#24775)

Problem statement:
Add on24 downloader
## Checklist

- [x] I'm reporting a feature request
- [x] I've verified that I'm running youtube-dl version **2020.03.24**
- [x] I've searched the bugtracker for similar feature requests including closed ones


## Description

ON24 is used for online trainings like from the O'Reilly learning platform. Replays are available after the training but it seems they cannot be downloaded. I'd like to ask for a download functionality.

Hints from issue discussion:
@chbndrhnns you need to find **stream.mpd** file using chrome developer tool.
then use the command 
`youtube-dl -f bestvideo+bestaudio https://dashod.akamaized.net/media/cv/events/XYZ/stream.mpd`
you need ffmpeg for merging formats.
see this stackoverflow [link](https://stackoverflow.com/a/55071769)
I've used this method and it works on on24 platform.
@VioletLightning thanks for sharing! If it's possible like that, I would not even need a native implementation.
I think that the main issue there is that they are separating video with audio with actually slides.
Of course, they are providing pdfs, but it would be cool to have a video, audio with actual slides at a particular moment at the time
@chbndrhnns You don't absolutely need it, but then again we don't /need/ most of the extractors either. They just make downloads of common media easier. Please change the label of this request to "site-support-request"

I would also like to plus +1 this request. As an example for why this would be useful: on24 is the streaming media platform behind many Microsoft learning materials.
@remitamine Examples can be found with this google search: `site:event.on24.com`. Many require a registration. Here is one that is available without registration:

https://event.on24.com/wcc/r/1886661/005F80F25EA62BDC10904B02AE9FD43D
Thank you, @remitamine.  Here is another example of a recent Microsoft online learning event URL:

> https://ve.on24.com/vshow/MSFTWindowsVirtual/exhibits/WVDMC?regPageId=18844

If I remember correctly, registration for this event is/was required, but is/was open and basic.
I'm interested. How can I help with this effort?
> @chbndrhnns you need to find **stream.mpd** file using chrome developer tool.
> then use the command
> `youtube-dl -f bestvideo+bestaudio https://dashod.akamaized.net/media/cv/events/XYZ/stream.mpd`
> you need ffmpeg for merging formats.
> see this stackoverflow [link](https://stackoverflow.com/a/55071769)
> I've used this method and it works on on24 platform.

The StackOverflow comment just after the one that @chbndrhnns mentioned also works perfectly for O'Reilly Live Events hosted on ON24. https://stackoverflow.com/a/55071769/5559588

Commit message: Remove whitespace

Fixes Flake8 warning

Patch to review:
diff --git a/yt_dlp/extractor/extractors.py b/yt_dlp/extractor/extractors.py
index 8c5b8b16077a..7ed8fb450cb3 100644
--- a/yt_dlp/extractor/extractors.py
+++ b/yt_dlp/extractor/extractors.py
@@ -983,6 +983,7 @@
 from .odnoklassniki import OdnoklassnikiIE
 from .oktoberfesttv import OktoberfestTVIE
 from .olympics import OlympicsReplayIE
+from .on24 import On24IE
 from .ondemandkorea import OnDemandKoreaIE
 from .onet import (
     OnetIE,
diff --git a/yt_dlp/extractor/on24.py b/yt_dlp/extractor/on24.py
new file mode 100644
index 000000000000..3c0a4fd2241c
--- /dev/null
+++ b/yt_dlp/extractor/on24.py
@@ -0,0 +1,105 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
+from ..utils import strip_or_none
+
+
+class On24IE(InfoExtractor):
+    IE_NAME = 'on24'
+    IE_DESC = 'ON24'
+
+    _VALID_URL = r'''(?x)
+                (?:
+                    https?://event\.on24\.com/
+                        (?:
+                            wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
+                            eventRegistration/
+                                (?:
+                                    console/EventConsoleApollo\.jsp\?|
+                                    EventLobbyServlet\?target=lobby30\.jsp&
+                                )
+                                (?:uimode=nextgeneration&)?(?:&)?eventid=(?P<id_2>\d{7})(?:&sessionid=1)?.*&key=(?P<key_2>[0-9A-F]{32})
+                        )
+                )
+                '''
+
+    _TESTS = [{
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
+        'info_dict': {
+            'id': '2197467',
+            'ext': 'wav',
+            'title': 'Pearson Test of English General/Pearson English International Certificate Teacher Training Guide',
+            'upload_date': '20200219',
+            'timestamp': 1582149600.0,
+        }
+    }, {
+        'url': 'https://event.on24.com/wcc/r/2639291/82829018E813065A122363877975752E?mode=login&email=johnsmith@gmail.com',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _is_absolute(url):
+        return bool(compat_urlparse.urlparse(url).netloc)
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        event_id = mobj.group('id_1') or mobj.group('id_2')
+        event_key = mobj.group('key_1') or mobj.group('key_2')
+
+        event_data = self._download_json(
+            'https://event.on24.com/apic/utilApp/EventConsoleCachedServlet',
+            event_id, query={
+                'eventId': event_id,
+                'displayProfile': 'player',
+                'key': event_key,
+                'contentType': 'A'
+            })
+
+        event_id = compat_str(event_data.get('presentationLogInfo', {}).get('eventid')) or event_id
+
+        info_media = event_data.get('mediaUrlInfo', {})
+        info_media.sort(key=lambda m: m['id'])
+
+        formats = []
+        for m in info_media:
+            media_url = compat_str(m['url'])
+            if not self._is_absolute(media_url):
+                media_url = 'https://event.on24.com/media/news/corporatevideo/events/%s' % media_url
+            if m['code'] == 'fhvideo1':
+                formats.append({
+                    'format_id': 'video',
+                    'url': media_url,
+                    'ext': 'mp4',
+                    'vcodec': 'avc1.640020',
+                    'acodec': 'mp4a.40.2',
+                })
+            elif m['code'] == 'audio':
+                formats.append({
+                    'format_id': 'audio',
+                    'url': media_url,
+                    'ext': 'wav',
+                    'vcodec': 'none',
+                    'acodec': 'wav'
+                })
+        self._sort_formats(formats)
+
+        return {
+            'id': event_id,
+            'title': strip_or_none(event_data.get('description')),
+            'language': event_data.get('localelanguagecode'),
+            'timestamp': (event_data.get('session', {}).get('startdate') / 1000
+                          if event_data.get('session', {}).get('startdate') is not None else None),
+            'webpage_url': 'https://event.on24.com/wcc/r/%s/%s' % (event_id, event_key),
+            'view_count': event_data.get('registrantcount'),
+            'formats': formats,
+        }


Code review comments:
diff_hunk: @@ -0,0 +1,105 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
+from ..utils import strip_or_none
+
+
+class On24IE(InfoExtractor):
+    IE_NAME = 'on24'
+    IE_DESC = 'ON24'
+
+    _VALID_URL = r'''(?x)
+                (?:
+                    https?://event\.on24\.com/
+                        (?:
+                            wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
+                            eventRegistration/
+                                (?:
+                                    console/EventConsoleApollo\.jsp\?|
+                                    EventLobbyServlet\?target=lobby30\.jsp&
+                                )
+                                (?:uimode=nextgeneration&)?(?:&)?eventid=(?P<id_2>\d{7})(?:&sessionid=1)?.*&key=(?P<key_2>[0-9A-F]{32})
+                        )
+                )
+                '''
+
+    _TESTS = [{
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
+        'info_dict': {
+            'id': '2197467',
+            'ext': 'wav',
+            'title': 'Pearson Test of English General/Pearson English International Certificate Teacher Training Guide',
+            'upload_date': '20200219',
+            'timestamp': 1582149600.0,
+        }
+    }, {
+        'url': 'https://event.on24.com/wcc/r/2639291/82829018E813065A122363877975752E?mode=login&email=johnsmith@gmail.com',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _is_absolute(url):
+        return bool(compat_urlparse.urlparse(url).netloc)
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
path: yt_dlp/extractor/on24.py
text: @user1:
use self._match_valid_url

diff_hunk: @@ -0,0 +1,105 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
+from ..utils import strip_or_none
+
+
+class On24IE(InfoExtractor):
+    IE_NAME = 'on24'
+    IE_DESC = 'ON24'
+
+    _VALID_URL = r'''(?x)
+                (?:
+                    https?://event\.on24\.com/
+                        (?:
+                            wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
+                            eventRegistration/
+                                (?:
+                                    console/EventConsoleApollo\.jsp\?|
+                                    EventLobbyServlet\?target=lobby30\.jsp&
+                                )
+                                (?:uimode=nextgeneration&)?(?:&)?eventid=(?P<id_2>\d{7})(?:&sessionid=1)?.*&key=(?P<key_2>[0-9A-F]{32})
+                        )
+                )
+                '''
+
+    _TESTS = [{
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
+        'info_dict': {
+            'id': '2197467',
+            'ext': 'wav',
+            'title': 'Pearson Test of English General/Pearson English International Certificate Teacher Training Guide',
+            'upload_date': '20200219',
+            'timestamp': 1582149600.0,
+        }
+    }, {
+        'url': 'https://event.on24.com/wcc/r/2639291/82829018E813065A122363877975752E?mode=login&email=johnsmith@gmail.com',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _is_absolute(url):
+        return bool(compat_urlparse.urlparse(url).netloc)
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        event_id = mobj.group('id_1') or mobj.group('id_2')
+        event_key = mobj.group('key_1') or mobj.group('key_2')
+
+        event_data = self._download_json(
+            'https://event.on24.com/apic/utilApp/EventConsoleCachedServlet',
+            event_id, query={
+                'eventId': event_id,
+                'displayProfile': 'player',
+                'key': event_key,
+                'contentType': 'A'
+            })
+
+        event_id = compat_str(event_data.get('presentationLogInfo', {}).get('eventid')) or event_id
path: yt_dlp/extractor/on24.py
text: @user1:
I personally wouldn't use `compat_str` just so that the code can't be blindly ported back to py2 😈 
But it's okay to use it if u want

@user1:
Also don't use .get chains.
Use try_get with expected type set

diff_hunk: @@ -0,0 +1,105 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
+from ..utils import strip_or_none
+
+
+class On24IE(InfoExtractor):
+    IE_NAME = 'on24'
+    IE_DESC = 'ON24'
+
+    _VALID_URL = r'''(?x)
+                (?:
+                    https?://event\.on24\.com/
+                        (?:
+                            wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
+                            eventRegistration/
+                                (?:
+                                    console/EventConsoleApollo\.jsp\?|
+                                    EventLobbyServlet\?target=lobby30\.jsp&
+                                )
+                                (?:uimode=nextgeneration&)?(?:&)?eventid=(?P<id_2>\d{7})(?:&sessionid=1)?.*&key=(?P<key_2>[0-9A-F]{32})
+                        )
+                )
+                '''
+
+    _TESTS = [{
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
+        'info_dict': {
+            'id': '2197467',
+            'ext': 'wav',
+            'title': 'Pearson Test of English General/Pearson English International Certificate Teacher Training Guide',
+            'upload_date': '20200219',
+            'timestamp': 1582149600.0,
+        }
+    }, {
+        'url': 'https://event.on24.com/wcc/r/2639291/82829018E813065A122363877975752E?mode=login&email=johnsmith@gmail.com',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _is_absolute(url):
+        return bool(compat_urlparse.urlparse(url).netloc)
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        event_id = mobj.group('id_1') or mobj.group('id_2')
+        event_key = mobj.group('key_1') or mobj.group('key_2')
+
+        event_data = self._download_json(
+            'https://event.on24.com/apic/utilApp/EventConsoleCachedServlet',
+            event_id, query={
+                'eventId': event_id,
+                'displayProfile': 'player',
+                'key': event_key,
+                'contentType': 'A'
+            })
+
+        event_id = compat_str(event_data.get('presentationLogInfo', {}).get('eventid')) or event_id
+
+        info_media = event_data.get('mediaUrlInfo', {})
+        info_media.sort(key=lambda m: m['id'])
+
+        formats = []
+        for m in info_media:
+            media_url = compat_str(m['url'])
+            if not self._is_absolute(media_url):
+                media_url = 'https://event.on24.com/media/news/corporatevideo/events/%s' % media_url
+            if m['code'] == 'fhvideo1':
+                formats.append({
+                    'format_id': 'video',
+                    'url': media_url,
+                    'ext': 'mp4',
+                    'vcodec': 'avc1.640020',
+                    'acodec': 'mp4a.40.2',
+                })
+            elif m['code'] == 'audio':
+                formats.append({
+                    'format_id': 'audio',
+                    'url': media_url,
+                    'ext': 'wav',
+                    'vcodec': 'none',
+                    'acodec': 'wav'
+                })
+        self._sort_formats(formats)
+
+        return {
+            'id': event_id,
+            'title': strip_or_none(event_data.get('description')),
+            'language': event_data.get('localelanguagecode'),
+            'timestamp': (event_data.get('session', {}).get('startdate') / 1000
+                          if event_data.get('session', {}).get('startdate') is not None else None),
path: yt_dlp/extractor/on24.py
text: @user1:
if you going to inline return, better keep it 1 line imo

therefore move timestamp above

Also don't use .get chains

diff_hunk: @@ -0,0 +1,105 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_str,
+    compat_urlparse,
+)
+from ..utils import strip_or_none
+
+
+class On24IE(InfoExtractor):
+    IE_NAME = 'on24'
+    IE_DESC = 'ON24'
+
+    _VALID_URL = r'''(?x)
+                (?:
+                    https?://event\.on24\.com/
+                        (?:
+                            wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
+                            eventRegistration/
+                                (?:
+                                    console/EventConsoleApollo\.jsp\?|
+                                    EventLobbyServlet\?target=lobby30\.jsp&
+                                )
+                                (?:uimode=nextgeneration&)?(?:&)?eventid=(?P<id_2>\d{7})(?:&sessionid=1)?.*&key=(?P<key_2>[0-9A-F]{32})
+                        )
+                )
+                '''
+
+    _TESTS = [{
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
+        'info_dict': {
+            'id': '2197467',
+            'ext': 'wav',
+            'title': 'Pearson Test of English General/Pearson English International Certificate Teacher Training Guide',
+            'upload_date': '20200219',
+            'timestamp': 1582149600.0,
+        }
+    }, {
+        'url': 'https://event.on24.com/wcc/r/2639291/82829018E813065A122363877975752E?mode=login&email=johnsmith@gmail.com',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
+        'only_matching': True,
+    }]
+
+    @staticmethod
+    def _is_absolute(url):
+        return bool(compat_urlparse.urlparse(url).netloc)
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        event_id = mobj.group('id_1') or mobj.group('id_2')
+        event_key = mobj.group('key_1') or mobj.group('key_2')
+
+        event_data = self._download_json(
+            'https://event.on24.com/apic/utilApp/EventConsoleCachedServlet',
+            event_id, query={
+                'eventId': event_id,
+                'displayProfile': 'player',
+                'key': event_key,
+                'contentType': 'A'
+            })
+
+        event_id = compat_str(event_data.get('presentationLogInfo', {}).get('eventid')) or event_id
+
+        info_media = event_data.get('mediaUrlInfo', {})
+        info_media.sort(key=lambda m: m['id'])
path: yt_dlp/extractor/on24.py
text: @user1:
Why sort?

@author:
The sorting could be removed.