Repository: yt-dlp/yt-dlp

Issue:
Big memory consumption spike
### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2022.01.21**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Description

Calling `extract_info` with url `https://redd.it/sdl0bi` in parameters will cause big memory consumption spike, up to 400 MB for me. For other links all work as intended.

### Verbose log

```shell
yt-dlp -vU 'https://redd.it/sdl0bi'
[debug] Command-line config: ['-vU', 'https://redd.it/sdl0bi']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0]
[debug] Python version 3.9.9 (CPython 64bit) - macOS-10.15.7-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.01.21, Current version: 2022.01.21
yt-dlp is up to date (2022.01.21)
[debug] [generic] Extracting URL: https://redd.it/sdl0bi
[generic] sdl0bi: Requesting header
[redirect] Following redirect to https://www.reddit.com/comments/sdl0bi
[debug] [generic] Extracting URL: https://www.reddit.com/comments/sdl0bi
[generic] sdl0bi: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] sdl0bi: Downloading webpage
[generic] sdl0bi: Extracting information
[debug] Looking for video embeds
ERROR: Unsupported URL: https://www.reddit.com/comments/sdl0bi
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1367, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 1437, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 612, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.9/site-packages/yt_dlp/extractor/generic.py", line 3966, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.reddit.com/comments/sdl0bi
```



Discussion:
The spike happens at this code https://github.com/yt-dlp/yt-dlp/blob/fa8fd951184017b9ba164dfa5e6d95d44ff6edd4/yt_dlp/extractor/generic.py#L3533-L3537

@zmousm, could you take a look
https://github.com/yt-dlp/yt-dlp/blob/e26f9cc1e58a298b00f1b9747981a05b3bbe919f/yt_dlp/extractor/glomex.py#L183-L199

All those `(?:(?!(?P=X)).)+)(?P=X)` patterns look pretty heavy to me. When I replace them with simple `.+?(?P=X)`, it seems to help.

Of course, a real solution would be to use an HTML parser.
> When I replace them with simple .+?(?P=X), it seems to help.

This could cause catastrophic backtracking in some webpages

> Of course, a real solution would be to use an HTML parser.

We had considered writing one (not for this), but put it off. If the regex is indeed the culprit, it looks like the best way forward
I ran this through a memory profiler:

NB: There is an issue with profiling generator functions, hence I temporarily removed yield statements; it does not matter in this context as no match is returned anyway.

```
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   180     89.4 MiB     89.4 MiB           1       @classmethod
   181                                             @profile
   182                                             def _extract_urls(cls, webpage, origin_url):
   183     89.4 MiB      0.0 MiB           1           VALID_SRC = rf'(?:https?:)?{cls._BASE_PLAYER_URL_RE}\?(?:(?!(?P=_q1)).)+'
   184                                         
   185                                                 # https://docs.glomex.com/publisher/video-player-integration/javascript-api/
   186     89.4 MiB      0.0 MiB           2           EMBED_RE = r'''(?x)(?:
   187                                                     <iframe[^>]+?src=(?P<_q1>%(quot_re)s)(?P<url>%(url_re)s)(?P=_q1)|
   188                                                     <(?P<html_tag>glomex-player|div)(?:
   189                                                         data-integration-id=(?P<_q2>%(quot_re)s)(?P<integration_html>(?:(?!(?P=_q2)).)+)(?P=_q2)|
   190                                                         data-playlist-id=(?P<_q3>%(quot_re)s)(?P<id_html>(?:(?!(?P=_q3)).)+)(?P=_q3)|
   191                                                         data-glomex-player=(?P<_q4>%(quot_re)s)(?P<glomex_player>true)(?P=_q4)|
   192                                                         [^>]*?
   193                                                     )+>|
   194                                                     # naive parsing of inline scripts for hard-coded integration parameters
   195                                                     <(?P<script_tag>script)[^<]*?>(?:
   196                                                         (?P<_stjs1>dataset\.)?integrationId\s*(?(_stjs1)=|:)\s*
   197                                                             (?P<_q5>%(quot_re)s)(?P<integration_js>(?:(?!(?P=_q5)).)+)(?P=_q5)\s*(?(_stjs1);|,)?|
   198                                                         (?P<_stjs2>dataset\.)?playlistId\s*(?(_stjs2)=|:)\s*
   199                                                             (?P<_q6>%(quot_re)s)(?P<id_js>(?:(?!(?P=_q6)).)+)(?P=_q6)\s*(?(_stjs2);|,)?|
   200                                                         (?:\s|.)*?
   201                                                     )+</script>
   202     89.4 MiB      0.0 MiB           1           )''' % {'quot_re': r'["\']', 'url_re': VALID_SRC}
   203                                         
   204    429.0 MiB    339.5 MiB         558           for mobj in re.finditer(EMBED_RE, webpage):
   205    429.0 MiB    339.8 MiB         557               mdict = mobj.groupdict()
   206    429.0 MiB    339.8 MiB         557               if mdict.get('url'):
   207                                                         url = unescapeHTML(mdict['url'])
   208                                                         if not cls.suitable(url):
   209                                                             continue
   210                                                         # yield cls._smuggle_origin_url(url, origin_url)
   211    429.0 MiB    339.8 MiB         557               elif mdict.get('html_tag'):
   212     89.5 MiB      0.0 MiB         521                   if mdict['html_tag'] == 'div' and not mdict.get('glomex_player'):
   213     89.5 MiB      0.0 MiB         521                       continue
   214                                                         if not mdict.get('video_id_html') or not mdict.get('integration_html'):
   215                                                             continue
   216                                                         # yield cls.build_player_url(mdict['video_id_html'], mdict['integration_html'], origin_url)
   217    429.0 MiB    339.8 MiB          36               elif mdict.get('script_tag'):
   218    429.0 MiB    339.8 MiB          36                   if not mdict.get('video_id_js') or not mdict.get('integration_js'):
   219                                                             continue
   220                                                         # yield cls.build_player_url(mdict['video_id_js'], mdict['integration_js'], origin_url)
```

* There are 521 html (div, probably) and 36 script tags matching the regex.
* However the memory usage stems from the match object (`mobj`), which also holds a copy (?) of `webpage` etc, while we only care for the named groups.

If I replace `re.finditer` with `re.findall`, which returns a list of tuples for matched groups, rather than match objects, then memory usage does not increase as much anymore:

```
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   180     87.0 MiB     87.0 MiB           1       @classmethod
   181                                             @profile
   182                                             def _extract_urls(cls, webpage, origin_url):
   183     87.0 MiB      0.0 MiB           1           VALID_SRC = rf'(?:https?:)?{cls._BASE_PLAYER_URL_RE}\?(?:(?!(?P=_q1)).)+'
   184                                         
   185                                                 # https://docs.glomex.com/publisher/video-player-integration/javascript-api/
   186     87.0 MiB      0.0 MiB           2           EMBED_RE = r'''(?x)(?:
   187                                                     <iframe[^>]+?src=(?P<_q1>%(quot_re)s)(?P<url>%(url_re)s)(?P=_q1)|
   188                                                     <(?P<html_tag>glomex-player|div)(?:
   189                                                         data-integration-id=(?P<_q2>%(quot_re)s)(?P<integration_html>(?:(?!(?P=_q2)).)+)(?P=_q2)|
   190                                                         data-playlist-id=(?P<_q3>%(quot_re)s)(?P<id_html>(?:(?!(?P=_q3)).)+)(?P=_q3)|
   191                                                         data-glomex-player=(?P<_q4>%(quot_re)s)(?P<glomex_player>true)(?P=_q4)|
   192                                                         [^>]*?
   193                                                     )+>|
   194                                                     # naive parsing of inline scripts for hard-coded integration parameters
   195                                                     <(?P<script_tag>script)[^<]*?>(?:
   196                                                         (?P<_stjs1>dataset\.)?integrationId\s*(?(_stjs1)=|:)\s*
   197                                                             (?P<_q5>%(quot_re)s)(?P<integration_js>(?:(?!(?P=_q5)).)+)(?P=_q5)\s*(?(_stjs1);|,)?|
   198                                                         (?P<_stjs2>dataset\.)?playlistId\s*(?(_stjs2)=|:)\s*
   199                                                             (?P<_q6>%(quot_re)s)(?P<id_js>(?:(?!(?P=_q6)).)+)(?P=_q6)\s*(?(_stjs2);|,)?|
   200                                                         (?:\s|.)*?
   201                                                     )+</script>
   202     87.0 MiB      0.0 MiB           1           )''' % {'quot_re': r'["\']', 'url_re': VALID_SRC}
   203                                         
   204     93.6 MiB      6.7 MiB         558           for mtup in re.findall(EMBED_RE, webpage):
   205     93.6 MiB      0.0 MiB        1114               mdict = dict(zip(
   206     93.6 MiB      0.0 MiB         557                   (
   207                                                             'url', '_',
   208                                                             'html_tag', '_', 'integration_html', '_', 'id_html', '_', 'glomex_player',
   209                                                             'script_tag', '_', '_', 'integration_js', '_', 'id_js',
   210                                                         ),
   211     93.6 MiB      0.0 MiB         557                   mtup))
   212     93.6 MiB      0.0 MiB         557               if mdict.get('url'):
   213                                                         url = unescapeHTML(mdict['url'])
   214                                                         if not cls.suitable(url):
   215                                                             continue
   216                                                         # yield cls._smuggle_origin_url(url, origin_url)
   217     93.6 MiB      0.0 MiB         557               elif mdict.get('html_tag'):
   218     93.6 MiB      0.0 MiB         521                   if mdict['html_tag'] == 'div' and not mdict.get('glomex_player'):
   219     93.6 MiB      0.0 MiB         521                       continue
   220                                                         if not mdict.get('video_id_html') or not mdict.get('integration_html'):
   221                                                             continue
   222                                                         # yield cls.build_player_url(mdict['video_id_html'], mdict['integration_html'], origin_url)
   223     93.6 MiB      0.0 MiB          36               elif mdict.get('script_tag'):
   224     93.6 MiB      0.0 MiB          36                   if not mdict.get('video_id_js') or not mdict.get('integration_js'):
   225                                                             continue
   226                                                         # yield cls.build_player_url(mdict['video_id_js'], mdict['integration_js'], origin_url)
```

Therefore I believe the issue is mainly with the use of match objects, rather than the regex itself.
If you are happy with this solution, please check #2535 for the patch.
Even bigger spike(up to 1GB) with such link(yt-dlp updated from master) https://accounts.google.com/ServiceLogin?service=wise&passive=1209600&continue=https://docs.google.com/forms/d/e/1FAIpQLSeNnjEancwnmDKba_vbS3BbNo1rNmJwcqmE2ewq7oA8_U1nlA/viewform?usp%3Dsend_form&followup=https://docs.google.com/forms/d/e/1FAIpQLSeNnjEancwnmDKba_vbS3BbNo1rNmJwcqmE2ewq7oA8_U1nlA/viewform?usp%3Dsend_form&ltmpl=forms


Resolving patch:
diff --git a/yt_dlp/extractor/glomex.py b/yt_dlp/extractor/glomex.py
--- a/yt_dlp/extractor/glomex.py
+++ b/yt_dlp/extractor/glomex.py
@@ -7,6 +7,7 @@
 from .common import InfoExtractor
 from ..utils import (
     determine_ext,
+    extract_attributes,
     ExtractorError,
     int_or_none,
     parse_qs,
@@ -177,49 +178,38 @@ def build_player_url(cls, video_id, integration, origin_url=None):
 
     @classmethod
     def _extract_urls(cls, webpage, origin_url):
-        VALID_SRC = rf'(?:https?:)?{cls._BASE_PLAYER_URL_RE}\?(?:(?!(?P=_q1)).)+'
-
         # https://docs.glomex.com/publisher/video-player-integration/javascript-api/
-        EMBED_RE = r'''(?x)(?:
-            <iframe[^>]+?src=(?P<_q1>%(quot_re)s)(?P<url>%(url_re)s)(?P=_q1)|
-            <(?P<html_tag>glomex-player|div)(?:
-                data-integration-id=(?P<_q2>%(quot_re)s)(?P<integration_html>(?:(?!(?P=_q2)).)+)(?P=_q2)|
-                data-playlist-id=(?P<_q3>%(quot_re)s)(?P<id_html>(?:(?!(?P=_q3)).)+)(?P=_q3)|
-                data-glomex-player=(?P<_q4>%(quot_re)s)(?P<glomex_player>true)(?P=_q4)|
-                [^>]*?
-            )+>|
-            # naive parsing of inline scripts for hard-coded integration parameters
-            <(?P<script_tag>script)[^<]*?>(?:
-                (?P<_stjs1>dataset\.)?integrationId\s*(?(_stjs1)=|:)\s*
-                    (?P<_q5>%(quot_re)s)(?P<integration_js>(?:(?!(?P=_q5)).)+)(?P=_q5)\s*(?(_stjs1);|,)?|
-                (?P<_stjs2>dataset\.)?playlistId\s*(?(_stjs2)=|:)\s*
-                    (?P<_q6>%(quot_re)s)(?P<id_js>(?:(?!(?P=_q6)).)+)(?P=_q6)\s*(?(_stjs2);|,)?|
-                (?:\s|.)*?
-            )+</script>
-        )''' % {'quot_re': r'["\']', 'url_re': VALID_SRC}
-
-        for mtup in re.findall(EMBED_RE, webpage):
-            # re.finditer causes a memory spike. See https://github.com/yt-dlp/yt-dlp/issues/2512
-            mdict = dict(zip((
-                'url', '_',
-                'html_tag', '_', 'integration_html', '_', 'id_html', '_', 'glomex_player',
-                'script_tag', '_', '_', 'integration_js', '_', 'id_js',
-            ), mtup))
-            if mdict.get('url'):
-                url = unescapeHTML(mdict['url'])
-                if not cls.suitable(url):
-                    continue
+        quot_re = r'["\']'
+
+        regex = fr'''(?x)
+            <iframe[^>]+?src=(?P<q>{quot_re})(?P<url>
+                (?:https?:)?{cls._BASE_PLAYER_URL_RE}\?(?:(?!(?P=q)).)+
+            )(?P=q)'''
+        for mobj in re.finditer(regex, webpage):
+            url = unescapeHTML(mobj.group('url'))
+            if cls.suitable(url):
                 yield cls._smuggle_origin_url(url, origin_url)
-            elif mdict.get('html_tag'):
-                if mdict['html_tag'] == 'div' and not mdict.get('glomex_player'):
-                    continue
-                if not mdict.get('video_id_html') or not mdict.get('integration_html'):
-                    continue
-                yield cls.build_player_url(mdict['video_id_html'], mdict['integration_html'], origin_url)
-            elif mdict.get('script_tag'):
-                if not mdict.get('video_id_js') or not mdict.get('integration_js'):
-                    continue
-                yield cls.build_player_url(mdict['video_id_js'], mdict['integration_js'], origin_url)
+
+        regex = fr'''(?x)
+            <glomex-player [^>]+?>|
+            <div[^>]* data-glomex-player=(?P<q>{quot_re})true(?P=q)[^>]*>'''
+        for mobj in re.finditer(regex, webpage):
+            attrs = extract_attributes(mobj.group(0))
+            if attrs.get('data-integration-id') and attrs.get('data-playlist-id'):
+                yield cls.build_player_url(attrs['data-playlist-id'], attrs['data-integration-id'], origin_url)
+
+        # naive parsing of inline scripts for hard-coded integration parameters
+        regex = fr'''(?x)
+            (?P<is_js>dataset\.)?%s\s*(?(is_js)=|:)\s*
+            (?P<q>{quot_re})(?P<id>(?:(?!(?P=q)).)+)(?P=q)\s'''
+        for mobj in re.finditer(r'(?x)<script[^<]*>.+?</script>', webpage):
+            script = mobj.group(0)
+            integration_id = re.search(regex % 'integrationId', script)
+            if not integration_id:
+                continue
+            playlist_id = re.search(regex % 'playlistId', script)
+            if playlist_id:
+                yield cls.build_player_url(playlist_id, integration_id, origin_url)
 
     def _real_extract(self, url):
         url, origin_url = self._unsmuggle_origin_url(url)


Test patch: