Repository: ytdl-org/youtube-dl

Issue:
Blerp
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
- Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
- Finally, put x into all relevant boxes (like this [x])
-->

- [x] I'm reporting a new site support request
- [x] I've verified that I'm running youtube-dl version **2021.12.17**
- [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that none of provided URLs violate any copyrights
- [x] I've searched the bugtracker for similar site support requests including closed ones


## Example URLs

<!--
Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
-->

- Sound: https://blerp.com/soundbites/6320fe8745636cb4dd677a5a
- Another Sound: https://blerp.com/soundbites/5bc94ef4796001000498429f
- User: https://blerp.com/u/the_mariocrafter_csed5x
- Search: https://blerp.com/search?q=Friday-Night-Funkin-&r=R
- Category: https://blerp.com/categories/sound-effects

## Description

<!--
Provide any additional information.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

Blerp is a popular soundboard service. This site actually use json+ld too but the generic extract twitter:player (maybe because it listed first ?). Not tested yet, but i assume backporting https://github.com/yt-dlp/yt-dlp/pull/4517 will work (with filtering the ie).
json ld:
```
 <script type="application/ld+json">
      [{
        "@context": "http://schema.org",
        "@type": ["MediaObject", "AudioObject", "CreativeWork", "MusicRecording"],
        "url": "https://blerp.com/soundbites/5bc94ef4796001000498429f",
        "byArtist": {
          "@type": "MusicGroup",
          "name": "179617322678353920",
          "@id": "https://blerp.com/user/5ba99cf71386730004552c42"
        },
        "embedUrl": "https://blerp.com/embed/5bc94ef4796001000498429f",
        "encodingFormat": "audio/mpeg",
        "name": "Yee",
        "image": "https://cdn.blerp.com/thumbnails/4cd92b90-195b-11e9-944f-470c2f71db8e",
        "keywords": "YEE,YEET,wo ha haah catchy tune yee,yee",
        "contentUrl": "https://cdn.blerp.com/normalized/cd4e4a10-d34e-11e8-a867-4741c3fed662"
      }]
    </script>
```
This probably should be being picked up by generic, and preferred over the twitter:player. There might be an issue with the core JSON LD extraction (might be to do with it being a list?)


Discussion:
Picked up by https://github.com/yt-dlp/yt-dlp/commit/0f60ba6e656516ec24d619d20d61249be6296105 , however metadata and playlists remain
>... an issue with the core JSON LD extraction (might be to do with it being a list?)

Definitely. Besides which, AudioObject derives from MediaObject, and MediaObject and MusicRecording both derive from CreativeWork; `byArtist` is a property only of MusicRecording, and all the other properties belong to MediaObject. I'd like to say that they're mad, but the snippet passes https://validator.schema.org/.

I have a WIP ld+json rewrite so this is good test fodder.

There may be more detailed stuff in the gigantic hydration JSON but the memory of the laptop is too small to contain it.
Hello everyone! I am a first-time contributor to this project.

If this issue is open, I would like to take a look at it.

Edit: I see that the downloaded file has no extension. Would it cause problems? If possible, can you link a reference PR that handles such cases?
The issue is definitely open and should be fixed by a rewrite of the ld+json extraction in `extractor/common.py`. As this may take a while, and yt-dlp, with which I'd like to keep alignment, has also been changing the same code (as noted above), an interim update could be justified.

An alternative approach that would probably get more metadata would be to create a site-specific extractor that processes the hydration JSON. It would be preferable to make the generic extractor work well enough.

@upintheairsheep, please complete the remaining checkbox items in the issue template.

These problems need to be handled:
* the ld+json has an `@type` that is a list
* none of the list is `VideoObject` which is targeted by the generic extractor
* as noted, the media link should be given a type or extension

The changes below against the git master give what looks like an adequate result. @clueless-skywatcher, you could certainly work up a PR if you wished.
```diff
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -70,6 +70,7 @@ from ..utils import (
     str_or_none,
     str_to_int,
     strip_or_none,
+    traverse_obj,
     try_get,
     unescapeHTML,
     unified_strdate,
@@ -79,6 +80,7 @@ from ..utils import (
     urljoin,
     url_basename,
     url_or_none,
+    variadic,
     xpath_element,
     xpath_text,
     xpath_with_ns,
@@ -1229,6 +1231,11 @@ class InfoExtractor(object):
         if isinstance(json_ld, dict):
             json_ld = [json_ld]
 
+        MEDIA_OBJECTS = ('VideoObject', 'AudioObject', 'MediaObject', 'MusicVideoObject')
+
+        def disjoint(a, b):
+            return all(x not in a for x in b)
+
         INTERACTION_TYPE_MAP = {
             'CommentAction': 'comment',
             'AgreeAction': 'like',
@@ -1274,8 +1281,8 @@ class InfoExtractor(object):
                     continue
                 info[count_key] = interaction_count
 
-        def extract_video_object(e):
-            assert e['@type'] == 'VideoObject'
+        def extract_media_object(e):
+            assert not disjoint(MEDIA_OBJECTS, variadic(e['@type'] or []))
             author = e.get('author')
             info.update({
                 'url': url_or_none(e.get('contentUrl')),
@@ -1283,7 +1290,9 @@ class InfoExtractor(object):
                 'description': unescapeHTML(e.get('description')),
                 'thumbnail': url_or_none(e.get('thumbnailUrl') or e.get('thumbnailURL')),
                 'duration': parse_duration(e.get('duration')),
+                'ext': mimetype2ext(e.get('encodingFormat')),
                 'timestamp': unified_timestamp(e.get('uploadDate')),
+                'tags': list(filter(None, map(lambda x: x.strip() or None, (e.get('keywords') or '').split(',')))) or None,
                 # author can be an instance of 'Organization' or 'Person' types.
                 # both types can have 'name' property(inherited from 'Thing' type). [1]
                 # however some websites are using 'Text' type instead.
@@ -1299,10 +1308,10 @@ class InfoExtractor(object):
 
         for e in json_ld:
             if '@context' in e:
-                item_type = e.get('@type')
-                if expected_type is not None and expected_type != item_type:
+                item_type = variadic(e.get('@type') or [])
+                if disjoint(item_type, variadic(expected_type or [])):
                     continue
-                if item_type in ('TVEpisode', 'Episode'):
+                if not disjoint(item_type, ('TVEpisode', 'Episode')):
                     episode_name = unescapeHTML(e.get('name'))
                     info.update({
                         'episode': episode_name,
@@ -1320,28 +1329,32 @@ class InfoExtractor(object):
                     part_of_series = e.get('partOfSeries') or e.get('partOfTVSeries')
                     if isinstance(part_of_series, dict) and part_of_series.get('@type') in ('TVSeries', 'Series', 'CreativeWorkSeries'):
                         info['series'] = unescapeHTML(part_of_series.get('name'))
-                elif item_type == 'Movie':
+                if 'Movie' in item_type:
                     info.update({
                         'title': unescapeHTML(e.get('name')),
                         'description': unescapeHTML(e.get('description')),
                         'duration': parse_duration(e.get('duration')),
                         'timestamp': unified_timestamp(e.get('dateCreated')),
                     })
-                elif item_type in ('Article', 'NewsArticle'):
+                if not disjoint(item_type, ('Article', 'NewsArticle')):
                     info.update({
                         'timestamp': parse_iso8601(e.get('datePublished')),
                         'title': unescapeHTML(e.get('headline')),
                         'description': unescapeHTML(e.get('articleBody')),
                     })
-                elif item_type == 'VideoObject':
-                    extract_video_object(e)
+                if 'MusicRecording' in item_type:
+                    info.update({
+                        'creator': unescapeHTML(traverse_obj(e, 'byArtist', ('byArtist', 'name'), expected_type=compat_str)),
+                         })
+                if not disjoint(item_type, MEDIA_OBJECTS):
+                    extract_media_object(e)
                     if expected_type is None:
                         continue
                     else:
                         break
-                video = e.get('video')
-                if isinstance(video, dict) and video.get('@type') == 'VideoObject':
-                    extract_video_object(video)
+                media = traverse_obj(e, 'video', 'audio', expected_type=dict)
+                if media and not disjoint(MEDIA_OBJECTS, variadic(media.get('@type') or [])):
+                    extract_media_object(media)
                 if expected_type is None:
                     continue
                 else:

--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@@@ -3556,10 +3594,10 @@@
                  self._extract_kvs(url, webpage, video_id),
                  info_dict)
  
-        # Looking for http://schema.org/VideoObject
+        # Looking for http://schema.org/MediaObject
          json_ld = self._search_json_ld(
-            webpage, video_id, default={}, expected_type='VideoObject')
-        if json_ld.get('url'):
+            webpage, video_id, default={}, expected_type=('VideoObject', 'AudioObject', 'MediaObject'))
+        if json_ld.get('url') not in (None, url):
              return merge_dicts(json_ld, info_dict)
  
          def check_video(vurl):

```
@dirkf Thank you for the advice. I can assimilate these changes in the integration PR right?
Please!
I have a PR ready, but probably due to a lack of extensions, the tests are failing like in the following error. Is my assumption correct or there any other possible reasons as to why this is happening?

```[debug] Loaded 1732 extractors
[debug] [blerp] Extracting URL: https://blerp.com/soundbites/6320fe8745636cb4dd677a5a
[blerp] 6320fe8745636cb4dd677a5a: Downloading webpage
ERROR: [blerp] 6320fe8745636cb4dd677a5a: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
Traceback (most recent call last):
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1495, in wrapper
    return func(self, *args, **kwargs)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1592, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1651, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 2600, in process_video_result
    self.raise_no_formats(info_dict)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1036, in raise_no_formats
    raise ExtractorError(msg, video_id=info['id'], ie=info['extractor'],
yt_dlp.utils.ExtractorError: [blerp] 6320fe8745636cb4dd677a5a: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

E
======================================================================
ERROR: test_Blerp (__main__.TestDownload):
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1495, in wrapper
    return func(self, *args, **kwargs)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1592, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1651, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 2600, in process_video_result
    self.raise_no_formats(info_dict)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1036, in raise_no_formats
    raise ExtractorError(msg, video_id=info['id'], ie=info['extractor'],
yt_dlp.utils.ExtractorError: [blerp] 6320fe8745636cb4dd677a5a: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/d/GitHub/yt-dlp/test/test_download.py", line 160, in test_template
    res_dict = ydl.extract_info(
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1484, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1513, in wrapper
    self.report_error(str(e), e.format_traceback())
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 1005, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "/mnt/d/GitHub/yt-dlp/yt_dlp/YoutubeDL.py", line 945, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: [blerp] 6320fe8745636cb4dd677a5a: No video formats found!; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

----------------------------------------------------------------------
Ran 1 test in 5.122s

FAILED (errors=1)```
1. Base your PR on yt-dl, not yt-dlp, or take it to yt-dlp.
2. If your extractor finds no formats, it should raise `ExtractorError('No video formats found')` itself and not just return `None`.
3. The error means that your code has failed to find any formats, so whatever regex or other method is being used isn't working, leaving the extractor in the situation 2 above.
4. As always, you will be surprised by what unexpected thing the code is doing so interactive debugging is the best way forward. You may have some preferred Python IDE that supports it: otherwise just insert the magic line `import pdb; pdb.set_trace()` at some point in the main line of your extractor to break at that line, after which you can set further breakpoints, step the code, display variables, etc.

@dirkf Are there any reference IEs where such cases are handled (where the media file has no extension)?
In my example code I made the ld+json extraction pick up the type of the media from the `encodingType` member, which happens to be set by the problem site.
```
                'ext': mimetype2ext(e.get('encodingFormat')),
```

Facing a new problem. Blerp's webpage doesn't render the soundbite instantly when loaded, so it takes some time to fetch the soundbite from their CDN. As a result, the ld+json fails to load, and the `self._download_webpage` function only fetches the webpage before the ld+json has loaded, resulting in a failure to read the soundbite. Is there any way to make the function fetch the full page at once?
@dirkf After a few more digging into the sources and network traffic, and several unsuccessful attempts to fetch the ld+json, I stumbled upon this website's GraphQL API. Fetching the query data I eventually concocted an IE that runs the tests fine. Should we proceed with this approach or stick to the ld+json solution?

Resolving patch:
diff --git a/youtube_dl/extractor/blerp.py b/youtube_dl/extractor/blerp.py
new file mode 100644
--- /dev/null
+++ b/youtube_dl/extractor/blerp.py
@@ -0,0 +1,173 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+
+from ..utils import (
+    strip_or_none,
+    traverse_obj,
+)
+from .common import InfoExtractor
+
+
+class BlerpIE(InfoExtractor):
+    IE_NAME = 'blerp'
+    _VALID_URL = r'https?://(?:www\.)?blerp\.com/soundbites/(?P<id>[0-9a-zA-Z]+)'
+    _TESTS = [{
+        'url': 'https://blerp.com/soundbites/6320fe8745636cb4dd677a5a',
+        'info_dict': {
+            'id': '6320fe8745636cb4dd677a5a',
+            'title': 'Samsung Galaxy S8 Over the Horizon Ringtone 2016',
+            'uploader': 'luminousaj',
+            'uploader_id': '5fb81e51aa66ae000c395478',
+            'ext': 'mp3',
+            'tags': ['samsung', 'galaxy', 's8', 'over the horizon', '2016', 'ringtone'],
+        }
+    }, {
+        'url': 'https://blerp.com/soundbites/5bc94ef4796001000498429f',
+        'info_dict': {
+            'id': '5bc94ef4796001000498429f',
+            'title': 'Yee',
+            'uploader': '179617322678353920',
+            'uploader_id': '5ba99cf71386730004552c42',
+            'ext': 'mp3',
+            'tags': ['YEE', 'YEET', 'wo ha haah catchy tune yee', 'yee']
+        }
+    }]
+
+    _GRAPHQL_OPERATIONNAME = "webBitePageGetBite"
+    _GRAPHQL_QUERY = (
+        '''query webBitePageGetBite($_id: MongoID!) {
+            web {
+                biteById(_id: $_id) {
+                    ...bitePageFrag
+                    __typename
+                }
+                __typename
+            }
+        }
+
+        fragment bitePageFrag on Bite {
+            _id
+            title
+            userKeywords
+            keywords
+            color
+            visibility
+            isPremium
+            owned
+            price
+            extraReview
+            isAudioExists
+            image {
+                filename
+                original {
+                    url
+                    __typename
+                }
+                __typename
+            }
+            userReactions {
+                _id
+                reactions
+                createdAt
+                __typename
+            }
+            topReactions
+            totalSaveCount
+            saved
+            blerpLibraryType
+            license
+            licenseMetaData
+            playCount
+            totalShareCount
+            totalFavoriteCount
+            totalAddedToBoardCount
+            userCategory
+            userAudioQuality
+            audioCreationState
+            transcription
+            userTranscription
+            description
+            createdAt
+            updatedAt
+            author
+            listingType
+            ownerObject {
+                _id
+                username
+                profileImage {
+                    filename
+                    original {
+                        url
+                        __typename
+                    }
+                    __typename
+                }
+                __typename
+            }
+            transcription
+            favorited
+            visibility
+            isCurated
+            sourceUrl
+            audienceRating
+            strictAudienceRating
+            ownerId
+            reportObject {
+                reportedContentStatus
+                __typename
+            }
+            giphy {
+                mp4
+                gif
+                __typename
+            }
+            audio {
+                filename
+                original {
+                    url
+                    __typename
+                }
+                mp3 {
+                    url
+                    __typename
+                }
+                __typename
+            }
+            __typename
+        }
+
+        ''')
+
+    def _real_extract(self, url):
+        audio_id = self._match_id(url)
+
+        data = {
+            'operationName': self._GRAPHQL_OPERATIONNAME,
+            'query': self._GRAPHQL_QUERY,
+            'variables': {
+                '_id': audio_id
+            }
+        }
+
+        headers = {
+            'Content-Type': 'application/json'
+        }
+
+        json_result = self._download_json('https://api.blerp.com/graphql',
+                                          audio_id, data=json.dumps(data).encode('utf-8'), headers=headers)
+
+        bite_json = json_result['data']['web']['biteById']
+
+        info_dict = {
+            'id': bite_json['_id'],
+            'url': bite_json['audio']['mp3']['url'],
+            'title': bite_json['title'],
+            'uploader': traverse_obj(bite_json, ('ownerObject', 'username'), expected_type=strip_or_none),
+            'uploader_id': traverse_obj(bite_json, ('ownerObject', '_id'), expected_type=strip_or_none),
+            'ext': 'mp3',
+            'tags': list(filter(None, map(strip_or_none, (traverse_obj(bite_json, 'userKeywords', expected_type=list) or []))) or None)
+        }
+
+        return info_dict
diff --git a/youtube_dl/extractor/extractors.py b/youtube_dl/extractor/extractors.py
--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@@ -138,6 +138,7 @@
     BleacherReportIE,
     BleacherReportCMSIE,
 )
+from .blerp import BlerpIE
 from .bloomberg import BloombergIE
 from .bokecc import BokeCCIE
 from .bongacams import BongaCamsIE


Test patch: