Repository: yt-dlp/yt-dlp

Issue:
Big memory consumption spike
### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2022.01.21**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Description

Calling `extract_info` with url `https://redd.it/sdl0bi` in parameters will cause big memory consumption spike, up to 400 MB for me. For other links all work as intended.

### Verbose log

```shell
yt-dlp -vU 'https://redd.it/sdl0bi'
[debug] Command-line config: ['-vU', 'https://redd.it/sdl0bi']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0]
[debug] Python version 3.9.9 (CPython 64bit) - macOS-10.15.7-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.01.21, Current version: 2022.01.21
yt-dlp is up to date (2022.01.21)
[debug] [generic] Extracting URL: https://redd.it/sdl0bi
[generic] sdl0bi: Requesting header
[redirect] Following redirect to https://www.reddit.com/comments/sdl0bi
[debug] [generic] Extracting URL: https://www.reddit.com/comments/sdl0bi
[generic] sdl0bi: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] sdl0bi: Downloading webpage
[generic] sdl0bi: Extracting information
[debug] Looking for video embeds
ERROR: Unsupported URL: https://www.reddit.com/comments/sdl0bi
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1367, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1437, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 612, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/extractor/generic.py", line 3966, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.reddit.com/comments/sdl0bi
```



Discussion:
The spike happens at this code https://github.com/yt-dlp/yt-dlp/blob/fa8fd951184017b9ba164dfa5e6d95d44ff6edd4/yt_dlp/extractor/generic.py#L3533-L3537

@zmousm, could you take a look
https://github.com/yt-dlp/yt-dlp/blob/e26f9cc1e58a298b00f1b9747981a05b3bbe919f/yt_dlp/extractor/glomex.py#L183-L199

All those `(?:(?!(?P=X)).)+)(?P=X)` patterns look pretty heavy to me. When I replace them with simple `.+?(?P=X)`, it seems to help.

Of course, a real solution would be to use an HTML parser.
> When I replace them with simple .+?(?P=X), it seems to help.

This could cause catastrophic backtracking in some webpages

> Of course, a real solution would be to use an HTML parser.

We had considered writing one (not for this), but put it off. If the regex is indeed the culprit, it looks like the best way forward

Resolving patch:
diff --git a/yt_dlp/extractor/glomex.py b/yt_dlp/extractor/glomex.py
--- a/yt_dlp/extractor/glomex.py
+++ b/yt_dlp/extractor/glomex.py
@@ -198,8 +198,13 @@ def _extract_urls(cls, webpage, origin_url):
             )+</script>
         )''' % {'quot_re': r'["\']', 'url_re': VALID_SRC}
 
-        for mobj in re.finditer(EMBED_RE, webpage):
-            mdict = mobj.groupdict()
+        for mtup in re.findall(EMBED_RE, webpage):
+            # re.finditer causes a memory spike. See https://github.com/yt-dlp/yt-dlp/issues/2512
+            mdict = dict(zip((
+                'url', '_',
+                'html_tag', '_', 'integration_html', '_', 'id_html', '_', 'glomex_player',
+                'script_tag', '_', '_', 'integration_js', '_', 'id_js',
+            ), mtup))
             if mdict.get('url'):
                 url = unescapeHTML(mdict['url'])
                 if not cls.suitable(url):


Test patch: