Repository: ytdl-org/youtube-dl

Issue:
Unable to extract mp3Form form for InfoQ video
## Checklist

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running youtube-dl version **2021.12.17**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

```
$ youtube-dl  --verbose  'https://www.infoq.com/presentations/problems-async-arch/'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.infoq.com/presentations/problems-async-arch/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.5 (CPython) - macOS-12.5-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.1, ffprobe 5.1
[debug] Proxy map: {}
[InfoQ] problems-async-arch: Downloading webpage
ERROR: Unable to extract mp3Form form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/infoq.py", line 128, in _real_extract
    + self._extract_http_audio(webpage, video_id))
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/infoq.py", line 93, in _extract_http_audio
    fields = self._form_hidden_inputs('mp3Form', webpage)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1367, in _form_hidden_inputs
    form = self._search_regex(
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract mp3Form form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
```


## Description

I can watch the video in a browser when I'm logged in or not. When downloading the video with youtube-dl, I get the error, "Unable to extract mp3Form form".



Discussion:
The page doesn't have the `<form...>` with `id="mp3Form"` that the extractor expects. This shouldn't be a crashing error. The [`http_video` format](https://videoh.infoq.com/presentations/21-nov-unblockeddesign.mp4?Signature=a83fgFGAqJAFhPYYUvVs72T4J5jxajpL1LKMqSNDRCzqkUWJBelQwyZhIIjBNTTyXb27NkpI5kyJD3iw5LnfSTtDU9Mn8zj%7ES0-j7TJFeS7ojSNqYPb5v4vq0BVd534MuKEoS0KKMJuJhzG9BDMK305YsKmAs8-njJ--S9R5qJk_&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6IioiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2NTk0ODM5Mjd9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiMC4wLjAuMC8wIn19fV19&Key-Pair-Id=APKAIMZVI7QH4C5YKH6Q) has audio.
I was able to work around this problem by modifying the `_real_extract()` method in `youtube_dl/extractor/infoq.py` and commenting out the `self._extract_http_audio(webpage, video_id)` call in line 128.
That would work in this case.

More generally, let's assume that the targeted form is sometimes present. We can either
* trap the failing code inside `_extract_http_audio()` with a `try:`, or 
* extend the `__form_hidden_inputs()` method signature with `**kwargs` passed into its `_search_regex()` call so that `fatal=False` can be passed down.
The first option occurred to me to minimize the number of things touched (infoq.py vs infoq.py and common.py) and just deal with a smaller blast radius.

The second option would better if other extractors might need to deal with a similar situation. This would expose more of the facilities that `_search_regex()` provides.

Thanks for looking into this!
I tried extending the `_form_hidden_inputs()` method to pass `fatal=False` and it does download the video from InfoQ but with `WARNING: unable to extract mp3Form form`.

```
$ youtube-dl -v https://www.infoq.com/presentations/problems-async-arch/
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.infoq.com/presentations/problems-async-arch/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.5 (CPython) - macOS-12.5-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.1, ffprobe 5.1
[debug] Proxy map: {}
[InfoQ] problems-async-arch: Downloading webpage
WARNING: unable to extract mp3Form form; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://videoh.infoq.com/presentations/21-nov-unblockeddesign.mp4?Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6IioiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2NTk0OTQ0MDZ9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiMC4wLjAuMC8wIn19fV19&Signature=KUA9-Ak19p9ZPMBvwk1KTESuNeZXi~nLl8HSlcyOEUmRP6CRa~5LXQsBxOqSRgyHeKDMd9OFidkSyysMkwbav3msMuV6nfR8P1KdbUKSfX-c980~KvPQn51X15IxLQpYrPjfoU-TMiGp232JwL3i5vizxcX-8MN3KLuIHmqY1RQ_&Key-Pair-Id=APKAIMZVI7QH4C5YKH6Q'
[download] Destination: Unblocked by Design-problems-async-arch.mp4
[download]   1.8% of 217.90MiB at  7.89MiB/s ETA 00:27^C
ERROR: Interrupted by user
```

This is what I did:
```
$ diff -u extractor/infoq.py-orig extractor/infoq.py
--- extractor/infoq.py-orig	2021-12-16 09:02:01.000000000 -1000
+++ extractor/infoq.py	2022-08-02 14:31:51.000000000 -1000
@@ -90,7 +90,7 @@
         }]
 
     def _extract_http_audio(self, webpage, video_id):
-        fields = self._form_hidden_inputs('mp3Form', webpage)
+        fields = self._form_hidden_inputs('mp3Form', webpage, fatal=False)
         http_audio_url = fields.get('filename')
         if not http_audio_url:
             return []

$ diff -u extractor/common.py-orig extractor/common.py
--- extractor/common.py-orig	2021-12-16 09:02:01.000000000 -1000
+++ extractor/common.py	2022-08-02 14:33:30.000000000 -1000
@@ -1363,10 +1363,12 @@
                 hidden_inputs[name] = value
         return hidden_inputs
 
-    def _form_hidden_inputs(self, form_id, html):
+    def _form_hidden_inputs(self, form_id, html, fatal=True):
         form = self._search_regex(
             r'(?is)<form[^>]+?id=(["\'])%s\1[^>]*>(?P<form>.+?)</form>' % form_id,
-            html, '%s form' % form_id, group='form')
+            html, '%s form' % form_id, group='form', fatal=fatal)
+        if not form:
+            form = ''
         return self._hidden_inputs(form)
 
     def _sort_formats(self, formats, field_preference=None):
```

The warning comes from the `_search_regex()` method in `common.py`.  When the fatal parameter is passed as False and no match is found, things end up in `else` clause  and the warning message is printed with the `bug_reports_message()` text. No exception is raised.

To have the warning message not printed when passing `fatal=False` to the `_form_hidden_inputs()` method, a check for no match object and not fatal is needed in the `else` clause of the `_search_regex()` method.

```
$ diff -u extractor/common.py-orig extractor/common.py
--- extractor/common.py-orig	2021-12-16 09:02:01.000000000 -1000
+++ extractor/common.py	2022-08-02 14:51:27.000000000 -1000
@@ -1011,6 +1011,8 @@
         elif fatal:
             raise RegexNotFoundError('Unable to extract %s' % _name)
         else:
+            if not mobj and not fatal:
+                return None
             self._downloader.report_warning('unable to extract %s' % _name + bug_reports_message())
             return None
 
@@ -1363,10 +1365,12 @@
                 hidden_inputs[name] = value
         return hidden_inputs
 
-    def _form_hidden_inputs(self, form_id, html):
+    def _form_hidden_inputs(self, form_id, html, fatal=True):
         form = self._search_regex(
             r'(?is)<form[^>]+?id=(["\'])%s\1[^>]*>(?P<form>.+?)</form>' % form_id,
-            html, '%s form' % form_id, group='form')
+            html, '%s form' % form_id, group='form', fatal=fatal)
+        if not form:
+            form = ''
         return self._hidden_inputs(form)
 
     def _sort_formats(self, formats, field_preference=None):
```
You can use `default=None` (no warning log message) or `fatal=False` (warning message) depending on whether the missing form is expected or its absence indicates that more work is needed on the extractor.  There isn't, though there could be, an `expected` parameter along with `fatal` that would suppress the `bug_reports_message()` in the warning when `True`, if the user should be able to see why no separate audio was extracted.

So I would (and did, just like your diff above) use `fatal=False` initially. Then, depending on tests:
* if the form is never there in current pages, I'd just remove the `_extract_http_audio()` method;
* if the form is still served in old pages but not in new ones, I'd switch to `default=None`;

Having a look at yt-dlp's extractor, the case is already handled and ignored there. Equivalent yt-dl code would be:
```diff
     def _extract_http_audio(self, webpage, video_id):
+        try:
-        fields = self._form_hidden_inputs('mp3Form', webpage)
+            fields = self._form_hidden_inputs('mp3Form', webpage)
+        except ExtractorError:
+            fields = {}
         http_audio_url = fields.get('filename')
...
```
This code assumes that any kind of `ExtractorError` should be ignored  here, not just `RegexNotFoundError`. For now I'll probably just push this for compatibility.
Awesome! Thanks, again!

Resolving patch:
diff --git a/youtube_dl/extractor/infoq.py b/youtube_dl/extractor/infoq.py
--- a/youtube_dl/extractor/infoq.py
+++ b/youtube_dl/extractor/infoq.py
@@ -1,6 +1,9 @@
 # coding: utf-8
 
 from __future__ import unicode_literals
+from ..utils import (
+    ExtractorError,
+)
 
 from ..compat import (
     compat_b64decode,
@@ -90,7 +93,11 @@ def _extract_http_video(self, webpage):
         }]
 
     def _extract_http_audio(self, webpage, video_id):
-        fields = self._form_hidden_inputs('mp3Form', webpage)
+        try:
+            fields = self._form_hidden_inputs('mp3Form', webpage)
+        except ExtractorError:
+            fields = {}
+
         http_audio_url = fields.get('filename')
         if not http_audio_url:
             return []


Test patch: