CVE-2023-40581 (Python, published 2023-09-25T19:15Z)
Description: value: yt-dlp is a youtube-dl fork with additional features and fixes. yt-dlp allows the user to provide shell command lines to be executed at various stages in its download steps through the `--exec` flag. This flag allows output template expansion in its argument, so that metadata values may be used in the shell commands. The metadata fields can be combined with the `%q` conversion, which is intended to quote/escape these values so they can be safely passed to the shell. However, the escaping used for `cmd` (the shell used by Python's `subprocess` on Windows) does not properly escape special characters, which can allow for remote code execution if `--exec` is used directly with maliciously crafted remote data. This vulnerability only impacts `yt-dlp` on Windows, and the vulnerability is present regardless of whether `yt-dlp` is run from `cmd` or from `PowerShell`. Support for output template expansion in `--exec`, along with this vulnerable behavior, was added to `yt-dlp` in version 2021.04.11. yt-dlp version 2023.09.24 fixes this issue by properly escaping each special character. `\n` will be replaced by `\r` as no way of escaping it has been found. It is recommended to upgrade yt-dlp to version 2023.09.24 as soon as possible. Also, always be careful when using --exec, because while this specific vulnerability has been patched, using unvalidated input in shell commands is inherently dangerous. For Windows users who are not able to upgrade: 1. Avoid using any output template expansion in --exec other than {} (filepath). 2. If expansion in --exec is needed, verify the fields you are using do not contain ", | or &. 3. Instead of using --exec, write the info json and load the fields from it instead.

Severity: nan (CVSS v3 base score: 7.8)
Weakness: CWE-78 \u2014 Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')
The product constructs all or part of an OS command using externally-influenced input from an upstream component, but it does not neutralize or incorrectly neutralizes special elements that could modify the intended OS command when it is sent to a downstream component.
Repository: https://github.com/yt-dlp/yt-dlp
Files changed:
devscripts/changelog_override.json
test/test_YoutubeDL.py
test/test_utils.py
yt_dlp/compat/__init__.py
yt_dlp/postprocessor/exec.py
yt_dlp/utils/_utils.py
Commit message: [core] Prevent RCE when using `--exec` with `%q` (CVE-2023-40581)

The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.

Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.

Authored by: Grub4K


Vulnerable code:
            test('%(title4)q', ('"foo \\"bar\\" test"', "＂foo ⧹＂bar⧹＂ test＂"))
            test('%(formats.:.id)#q', ('"id 1" "id 2" "id 3"', '＂id 1＂ ＂id 2＂ ＂id 3＂'))
            test('%(formats.0.id)#q', ('"id 1"', '＂id 1＂'))
        return s if re.match(r'^[-_\w./]+$', s) else '"%s"' % s.replace('"', '\\"')
import subprocess

from ..utils import PostProcessingError, encodeArgument, variadic
            self.to_screen('Executing command: %s' % cmd)
            retCode = subprocess.call(encodeArgument(cmd), shell=True)
            if retCode != 0:
                raise PostProcessingError('Command returned error code %d' % retCode)
    def __init__(self, *args, env=None, text=False, **kwargs):
        super().__init__(*args, env=env, **kwargs, startupinfo=self._startupinfo)

Fixed code:
    },
    {
        "action": "add",
        "when": "61bdf15fc7400601c3da1aa7a43917310a5bf391",
        "short": "[priority] Security: [[CVE-2023-40581](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40581)] [Prevent RCE when using `--exec` with `%q` on Windows](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg)\n    - The shell escape function is now using `\"\"` instead of `\\\"`.\n    - `utils.Popen` has been patched to properly quote commands."
            test('%(title4)q', ('"foo ""bar"" test"', None))
            test('%(formats.:.id)#q', ('"id 1" "id 2" "id 3"', None))
            test('%(formats.0.id)#q', ('"id 1"', None))
import subprocess
    Popen,
    @unittest.skipUnless(compat_os_name == 'nt', 'Only relevant on Windows')
    def test_Popen_windows_escaping(self):
        def run_shell(args):
            stdout, stderr, error = Popen.run(
                args, text=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            assert not stderr
            assert not error
            return stdout

        # Test escaping
        assert run_shell(['echo', 'test"&']) == '"test""&"\n'
        # Test if delayed expansion is disabled
        assert run_shell(['echo', '^!']) == '"^!"\n'
        assert run_shell('echo "^!"') == '"^!"\n'
        return s if re.match(r'^[-_\w./]+$', s) else s.replace('"', '""').join('""')
from ..utils import Popen, PostProcessingError, variadic
            self.to_screen(f'Executing command: {cmd}')
            _, _, return_code = Popen.run(cmd, shell=True)
            if return_code != 0:
                raise PostProcessingError(f'Command returned error code {return_code}')
    def __init__(self, args, *remaining, env=None, text=False, shell=False, **kwargs):

        if shell and compat_os_name == 'nt' and kwargs.get('executable') is None:
            if not isinstance(args, str):
                args = ' '.join(compat_shlex_quote(a) for a in args)
            shell = False
            args = f'{self.__comspec()} /Q /S /D /V:OFF /C "{args}"'

        super().__init__(args, *remaining, env=env, shell=shell, **kwargs, startupinfo=self._startupinfo)

    def __comspec(self):
        comspec = os.environ.get('ComSpec') or os.path.join(
            os.environ.get('SystemRoot', ''), 'System32', 'cmd.exe')
        if os.path.isabs(comspec):
            return comspec
        raise FileNotFoundError('shell not found: neither %ComSpec% nor %SystemRoot% is set')