Repository: ytdl-org/youtube-dl

Issue:
[YouTube] TypeError: '>' not supported between instances of 'int' and 'NoneType' 
## Checklist

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running youtube-dl version **2021.12.17**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

```
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: […]
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: e52e8b811
[debug] Python version 3.8.10 (CPython) - Linux-5.8.0-44-lowlatency-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Proxy map: {…}
[youtube] t-hR-TZJT2U: Downloading webpage
[youtube] t-hR-TZJT2U: Downloading MPD manifest
WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (Failed to evaluate 0 > None (caused by TypeError("'>' not supported between instances of 'int' and 'NoneType'")); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last):
  File "/mnt/…/youtube_dl/jsinterp.py", line 203, in _operator
    return opfunc(left_val, right_val)
TypeError: '>' not supported between instances of 'int' and 'NoneType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
…
  File "/mnt/…/youtube_dl/jsinterp.py", line 590, in interpret_expression
    ret, should_return = self.interpret_statement(expr, local_vars, allow_recursion)
  File "/mnt/…/youtube_dl/jsinterp.py", line 454, in interpret_statement
    return self._operator(op, 0 if left_val is None else left_val,
  File "/mnt/…/youtube_dl/jsinterp.py", line 205, in _operator
    raise self.Exception('Failed to evaluate {left_val!r} {op} {right_val!r}'.format(**locals()), expr, cause=e)
youtube_dl.jsinterp.JSInterpreter.Exception: Failed to evaluate 0 > None (caused by TypeError("'>' not supported between instances of 'int' and 'NoneType'")); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
)
```


## Description
Download seems to work nonetheless.


Discussion:
Apparently you just need to reinstall Python 2.7!

Python 3.5 has this message:
```
...
  File "/home/df/Documents/src/youtube-dl/youtube_dl/jsinterp.py", line 205, in _operator
    raise self.Exception('Failed to evaluate {left_val!r} {op} {right_val!r}'.format(**locals()), expr, cause=e)
youtube_dl.jsinterp.JSInterpreter.Exception: Failed to evaluate 0 > None (caused by TypeError('unorderable types: int() > NoneType()',));  ...
```
However today's dev version fixes whatever problem there was:
```shellsession
$ python3.9 -m youtube_dl -F -v 'https://www.youtube.com/watch?v=t-hR-TZJT2U'
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-v', 'https://www.youtube.com/watch?v=t-hR-TZJT2U']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: e52e8b811
[debug] Python version 3.9.13 (CPython) - Linux-4.4.0-210-generic-i686-with-glibc2.23
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[youtube] t-hR-TZJT2U: Downloading webpage
[youtube] t-hR-TZJT2U: Downloading MPD manifest
[debug] [youtube] Decrypted nsig BnLAR5umBf9R48MhQ => y5mzqXT4vmPGow
[debug] [youtube] Decrypted nsig oxC0xXr4o8rSUj1L9 => mfQBGpVcJ9-_fQ
[info] Available formats for t-hR-TZJT2U:
format code  extension  resolution note
139          m4a        audio only DASH audio   49k , m4a_dash container, mp4a.40.5 (22050Hz), 513.68KiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 1.33MiB
251          webm       audio only tiny  141k , webm_dash container, opus @141k (48000Hz), 1.45MiB
278          webm       256x144    DASH video   95k , webm_dash container, vp9, 30fps, video only
160          mp4        256x144    DASH video  108k , mp4_dash container, avc1.4d400b, 30fps, video only
242          webm       426x240    DASH video  220k , webm_dash container, vp9, 30fps, video only
133          mp4        426x240    DASH video  242k , mp4_dash container, avc1.4d400c, 30fps, video only
134          mp4        640x360    360p  315k , mp4_dash container, avc1.4d401e@ 315k, 30fps, video only, 3.23MiB
243          webm       640x360    DASH video  405k , webm_dash container, vp9, 30fps, video only
244          webm       854x480    DASH video  752k , webm_dash container, vp9, 30fps, video only
135          mp4        854x480    DASH video 1155k , mp4_dash container, avc1.4d4014, 30fps, video only
136          mp4        1280x720   720p 1092k , mp4_dash container, avc1.64001f@1092k, 30fps, video only, 11.20MiB
247          webm       1280x720   DASH video 1505k , webm_dash container, vp9, 30fps, video only
302          webm       1280x720   DASH video 2646k , webm_dash container, vp9, 60fps, video only
298          mp4        1280x720   DASH video 3465k , mp4_dash container, avc1.4d400d, 60fps, video only
299          mp4        1920x1080  1080p60 3021k , mp4_dash container, avc1.64002a@3021k, 60fps, video only, 30.98MiB
303          webm       1920x1080  DASH video 4410k , webm_dash container, vp9, 60fps, video only
18           mp4        640x360    360p  443k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz)
22           mp4        1280x720   720p 1220k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)
$
``` 
Coming in  this version:
* operator `??`
* operator `?.`
* operator `**`
* accurate operator functions ([Douglas Crockford special](https://devclass.com/2022/08/04/retire_javascript_says-json-creator-douglas-crockford/): `undefined ** 0 === 1`)
* `undefined` handling
* object literals `{a: 1, "b": expr}`

The problem is with this line 264 of the pretty-printed nsig code:
```
      6 >= c[104] ? (0, c[22]) ((0, c[1]) ((0, c[5]) (c[13], c[44]), c[76], c[12], c[75]), c[2], (0, c[0]) (((0, c[82]) (c[12]), (0, c[98]) (c[72], c[12])), c[106], (0, c[31]) (c[63], c[35]) > (0, c[102]) (c[ - 120 * Math.pow(7, 2) + 5969], c[82]), c[12], c[10]), c[29], c[21])  : (0, c[81]) ((0, c[40]) (c[10], c[11], (0, c[20]) ()), c[53], (0, c[new Date('31 December 1969 13:15:53 -1045') / 1000]) ((0, c[53]) ((0, c[12]) (c[26]), c[35], c[10], c[275 % Math.pow(new Date('1969-12-31T17:00:02.000-07:00') / 1000, 5) + 9]), c[40], c[67], c[74], (0, c[21]) ()), (0, c[53]) ((0, c[35]) (c[42], c[104]), c[35], c[67], c[48]), (0, c[86]) (c[Math.pow(5, 5) + 17176 + - 20291], c[51]), c[49], c[64], c[42]),
```
yt-dl is trying to evaluate this part of the line, and the RHS of the '>' is evaluating to `None`: 
```
(0,c[0])(((0,c[82])(c[12]),(0,c[98])(c[72],c[12])),c[106],(0,c[31])(c[63],c[35])>(0,c[102])(c[-120*Math.pow(7,2)+5969],c[82]),c[12],c[10])
```
Here `c[102]` is a number, so something went wrong way back, something that behaves differently in Py2.7 (no error, though I didn't check the download speed), Py3.5, Py 3.8+.
> Apparently you just need to reinstall Python 2.7!

I tried it, but the error persists. Maybe Ubuntu mirrors didn't get the update yet.
Sorry, that wasn't a serious suggestion. You'd have to go through more hoops to get Python 2.7 running your yt-dl, anyway.

Please wait for the new version.
> You'd have to go through more hoops to get Python 2.7 running your yt-dl, anyway.

Indeed, I had to edit the shebang line of my wrapper script. I didn't remember I had upgraded it to use python3 yet. Now the warning seems gone and download speed seems to be back to normal. So thanks for reminding me of that extra step!

Resolving patch:
diff --git a/youtube_dl/jsinterp.py b/youtube_dl/jsinterp.py
--- a/youtube_dl/jsinterp.py
+++ b/youtube_dl/jsinterp.py
@@ -7,6 +7,7 @@
 import re
 
 from .utils import (
+    error_to_compat_str,
     ExtractorError,
     js_to_json,
     remove_quotes,
@@ -130,7 +131,7 @@ def wrapped(a, b):
 _OPERATOR_RE = '|'.join(map(lambda x: re.escape(x[0]), _OPERATORS + _LOG_OPERATORS))
 
 _MATCHING_PARENS = dict(zip(*zip('()', '{}', '[]')))
-_QUOTES = '\'"'
+_QUOTES = '\'"/'
 
 
 def _ternary(cndn, if_true=True, if_false=False):
@@ -155,6 +156,12 @@ def __init__(self):
         ExtractorError.__init__(self, 'Invalid continue')
 
 
+class JS_Throw(ExtractorError):
+    def __init__(self, e):
+        self.error = e
+        ExtractorError.__init__(self, 'Uncaught exception ' + error_to_compat_str(e))
+
+
 class LocalNameSpace(ChainMap):
     def __getitem__(self, key):
         try:
@@ -172,6 +179,17 @@ def __setitem__(self, key, value):
     def __delitem__(self, key):
         raise NotImplementedError('Deleting is not supported')
 
+    # except
+    def pop(self, key, *args):
+        try:
+            off = self.__getitem__(key)
+            super(LocalNameSpace, self).__delitem__(key)
+            return off
+        except KeyError:
+            if len(args) > 0:
+                return args[0]
+            raise
+
     def __contains__(self, key):
         try:
             super(LocalNameSpace, self).__getitem__(key)
@@ -188,9 +206,29 @@ class JSInterpreter(object):
 
     undefined = _UNDEFINED
 
+    RE_FLAGS = {
+        # special knowledge: Python's re flags are bitmask values, current max 128
+        # invent new bitmask values well above that for literal parsing
+        # TODO: new pattern class to execute matches with these flags
+        'd': 1024,  # Generate indices for substring matches
+        'g': 2048,  # Global search
+        'i': re.I,  # Case-insensitive search
+        'm': re.M,  # Multi-line search
+        's': re.S,  # Allows . to match newline characters
+        'u': re.U,  # Treat a pattern as a sequence of unicode code points
+        'y': 4096,  # Perform a "sticky" search that matches starting at the current position in the target string
+    }
+
+    _EXC_NAME = '__youtube_dl_exception__'
+    _OBJ_NAME = '__youtube_dl_jsinterp_obj'
+
+    OP_CHARS = None
+
     def __init__(self, code, objects=None):
         self.code, self._functions = code, {}
         self._objects = {} if objects is None else objects
+        if type(self).OP_CHARS is None:
+            type(self).OP_CHARS = self.OP_CHARS = self.__op_chars()
 
     class Exception(ExtractorError):
         def __init__(self, msg, *args, **kwargs):
@@ -199,32 +237,64 @@ def __init__(self, msg, *args, **kwargs):
                 msg = '{0} in: {1!r}'.format(msg.rstrip(), expr[:100])
             super(JSInterpreter.Exception, self).__init__(msg, *args, **kwargs)
 
+    @classmethod
+    def __op_chars(cls):
+        op_chars = set(';,')
+        for op in cls._all_operators():
+            for c in op[0]:
+                op_chars.add(c)
+        return op_chars
+
     def _named_object(self, namespace, obj):
         self.__named_object_counter += 1
-        name = '__youtube_dl_jsinterp_obj%d' % (self.__named_object_counter, )
+        name = '%s%d' % (self._OBJ_NAME, self.__named_object_counter)
         namespace[name] = obj
         return name
 
-    @staticmethod
-    def _separate(expr, delim=',', max_split=None, skip_delims=None):
+    @classmethod
+    def _regex_flags(cls, expr):
+        flags = 0
+        if not expr:
+            return flags, expr
+        for idx, ch in enumerate(expr):
+            if ch not in cls.RE_FLAGS:
+                break
+            flags |= cls.RE_FLAGS[ch]
+        return flags, expr[idx:] if idx > 0 else expr
+
+    @classmethod
+    def _separate(cls, expr, delim=',', max_split=None, skip_delims=None):
         if not expr:
             return
         counters = {k: 0 for k in _MATCHING_PARENS.values()}
-        start, splits, pos, skipping, delim_len = 0, 0, 0, 0, len(delim) - 1
-        in_quote, escaping = None, False
+        start, splits, pos, delim_len = 0, 0, 0, len(delim) - 1
+        in_quote, escaping, skipping = None, False, 0
+        after_op, in_regex_char_group, skip_re = True, False, 0
+
         for idx, char in enumerate(expr):
+            if skip_re > 0:
+                skip_re -= 1
+                continue
             if not in_quote:
                 if char in _MATCHING_PARENS:
                     counters[_MATCHING_PARENS[char]] += 1
                 elif char in counters:
                     counters[char] -= 1
-            if not escaping:
-                if char in _QUOTES and in_quote in (char, None):
-                    in_quote = None if in_quote else char
-                else:
-                    escaping = in_quote and char == '\\'
-            else:
-                escaping = False
+            if not escaping and char in _QUOTES and in_quote in (char, None):
+                if in_quote or after_op or char != '/':
+                    in_quote = None if in_quote and not in_regex_char_group else char
+                    if in_quote is None and char == '/' and delim != '/':
+                        # regexp flags
+                        n_idx = idx + 1
+                        while n_idx < len(expr) and expr[n_idx] in cls.RE_FLAGS:
+                            n_idx += 1
+                        skip_re = n_idx - idx - 1
+                        if skip_re > 0:
+                            continue
+            elif in_quote == '/' and char in '[]':
+                in_regex_char_group = char == '['
+            escaping = not escaping and in_quote and char == '\\'
+            after_op = not in_quote and char in cls.OP_CHARS or (char == ' ' and after_op)
 
             if char != delim[pos] or any(counters.values()) or in_quote:
                 pos = skipping = 0
@@ -313,16 +383,23 @@ def interpret_statement(self, stmt, local_vars, allow_recursion=100):
             if should_return:
                 return ret, should_return
 
-        m = re.match(r'(?P<var>(?:var|const|let)\s)|return(?:\s+|$)', stmt)
+        m = re.match(r'(?P<var>(?:var|const|let)\s)|return(?:\s+|(?=["\'])|$)|(?P<throw>throw\s+)', stmt)
         if m:
             expr = stmt[len(m.group(0)):].strip()
+            if m.group('throw'):
+                raise JS_Throw(self.interpret_expression(expr, local_vars, allow_recursion))
             should_return = not m.group('var')
         if not expr:
             return None, should_return
 
         if expr[0] in _QUOTES:
             inner, outer = self._separate(expr, expr[0], 1)
-            inner = json.loads(js_to_json(inner + expr[0]))  # , strict=True))
+            if expr[0] == '/':
+                flags, _ = self._regex_flags(outer)
+                inner, outer = inner.replace('"', r'\"'), ''
+                inner = re.compile(js_to_json(inner + expr[0]), flags=flags)  # , strict=True))
+            else:
+                inner = json.loads(js_to_json(inner + expr[0]))  # , strict=True))
             if not outer:
                 return inner, should_return
             expr = self._named_object(local_vars, inner) + outer
@@ -374,22 +451,37 @@ def interpret_statement(self, stmt, local_vars, allow_recursion=100):
                 for item in self._separate(inner)])
             expr = name + outer
 
-        m = re.match(r'(?P<try>try|finally)\s*|(?:(?P<catch>catch)|(?P<for>for)|(?P<switch>switch))\s*\(', expr)
+        m = re.match(r'''(?x)
+            (?P<try>try|finally)\s*|
+            (?P<catch>catch\s*(?P<err>\(\s*{_NAME_RE}\s*\)))|
+            (?P<switch>switch)\s*\(|
+            (?P<for>for)\s*\(|'''.format(**globals()), expr)
         md = m.groupdict() if m else {}
         if md.get('try'):
             if expr[m.end()] == '{':
                 try_expr, expr = self._separate_at_paren(expr[m.end():], '}')
             else:
                 try_expr, expr = expr[m.end() - 1:], ''
-            ret, should_abort = self.interpret_statement(try_expr, local_vars, allow_recursion)
-            if should_abort:
-                return ret, True
+            try:
+                ret, should_abort = self.interpret_statement(try_expr, local_vars, allow_recursion)
+                if should_abort:
+                    return ret, True
+            except JS_Throw as e:
+                local_vars[self._EXC_NAME] = e.error
+            except Exception as e:
+                # XXX: This works for now, but makes debugging future issues very hard
+                local_vars[self._EXC_NAME] = e
             ret, should_abort = self.interpret_statement(expr, local_vars, allow_recursion)
             return ret, should_abort or should_return
 
         elif md.get('catch'):
-            # We ignore the catch block
-            _, expr = self._separate_at_paren(expr, '}')
+            catch_expr, expr = self._separate_at_paren(expr[m.end():], '}')
+            if self._EXC_NAME in local_vars:
+                catch_vars = local_vars.new_child({m.group('err'): local_vars.pop(self._EXC_NAME)})
+                ret, should_abort = self.interpret_statement(catch_expr, catch_vars, allow_recursion)
+                if should_abort:
+                    return ret, True
+
             ret, should_abort = self.interpret_statement(expr, local_vars, allow_recursion)
             return ret, should_abort or should_return
 
@@ -503,7 +595,7 @@ def interpret_statement(self, stmt, local_vars, allow_recursion=100):
                 raise self.Exception('List index %s must be integer' % (idx, ), expr=expr)
             idx = int(idx)
             left_val[idx] = self._operator(
-                m.group('op'), left_val[idx], m.group('expr'), expr, local_vars, allow_recursion)
+                m.group('op'), self._index(left_val, idx), m.group('expr'), expr, local_vars, allow_recursion)
             return left_val[idx], should_return
 
         elif expr.isdigit():


Test patch:
diff --git a/test/test_jsinterp.py b/test/test_jsinterp.py
--- a/test/test_jsinterp.py
+++ b/test/test_jsinterp.py
@@ -9,6 +9,7 @@
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 import math
+import re
 
 from youtube_dl.jsinterp import JSInterpreter
 undefined = JSInterpreter.undefined
@@ -316,19 +317,39 @@ def test_object(self):
         function x() { return {}; }
         ''')
         self.assertEqual(jsi.call_function('x'), {})
+
         jsi = JSInterpreter('''
         function x() { let a = {m1: 42, m2: 0 }; return [a["m1"], a.m2]; }
         ''')
         self.assertEqual(jsi.call_function('x'), [42, 0])
+
         jsi = JSInterpreter('''
         function x() { let a; return a?.qq; }
         ''')
         self.assertIs(jsi.call_function('x'), undefined)
+
         jsi = JSInterpreter('''
         function x() { let a = {m1: 42, m2: 0 }; return a?.qq; }
         ''')
         self.assertIs(jsi.call_function('x'), undefined)
 
+    def test_regex(self):
+        jsi = JSInterpreter('''
+        function x() { let a=/,,[/,913,/](,)}/; }
+        ''')
+        self.assertIs(jsi.call_function('x'), None)
+
+        jsi = JSInterpreter('''
+        function x() { let a=/,,[/,913,/](,)}/; return a; }
+        ''')
+        # Pythons disagree on the type of a pattern
+        self.assertTrue(isinstance(jsi.call_function('x'), type(re.compile(''))))
+
+        jsi = JSInterpreter('''
+        function x() { let a=/,,[/,913,/](,)}/i; return a; }
+        ''')
+        self.assertEqual(jsi.call_function('x').flags & re.I, re.I)
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/test/test_youtube_signature.py b/test/test_youtube_signature.py
--- a/test/test_youtube_signature.py
+++ b/test/test_youtube_signature.py
@@ -106,6 +106,10 @@
         'https://www.youtube.com/s/player/c81bbb4a/player_ias.vflset/en_US/base.js',
         'gre3EcLurNY2vqp94', 'Z9DfGxWP115WTg',
     ),
+    (
+        'https://www.youtube.com/s/player/1f7d5369/player_ias.vflset/en_US/base.js',
+        'batNX7sYqIJdkJ', 'IhOkL_zxbkOZBw',
+    ),
 ]