Repository: yt-dlp/yt-dlp

Issue:
UnicodeDecodeError when configuration saved as UTF-8 and OS default encoding is GBK
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.08.10. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
- Read bugs section in FAQ: https://github.com/yt-dlp/yt-dlp
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->

- [x] I'm reporting a bug unrelated to a specific site
- [x] I've verified that I'm running yt-dlp version **2021.08.10**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] The provided URLs do not contain any DRM to the best of my knowledge
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar bug reports including closed ones
- [x] I've read bugs section in FAQ

## Verbose log

<!--
Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] yt-dlp version 2021.08.10
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
Traceback (most recent call last):
  File "yt_dlp\__main__.py", line 19, in <module>
  File "yt_dlp\__init__.py", line 750, in main
  File "yt_dlp\__init__.py", line 73, in _real_main
  File "yt_dlp\options.py", line 1496, in parseOpts
  File "yt_dlp\options.py", line 1476, in get_configs
  File "yt_dlp\options.py", line 1471, in read_options
  File "yt_dlp\options.py", line 60, in _readOptions
UnicodeDecodeError: 'gbk' codec can't decode byte 0xa8 in position 16: illegal multibyte sequence
[40668] Failed to execute script '__main__' due to unhandled exception!
```
<!--
Do not remove the above ```
-->


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

I'm a Chinese user and I tried to write comments in my native language in the configuration file `yt-dlp.conf`, putting the file along with the executable:

```bash
# 代理服务器 (which means Proxy Server)
--proxy 127.0.0.1:29970
```
Then I saved the file as UTF-8. While using yt-dlp with or without any arguments, it reports `UnicodeDecodeError`.

Because I'm using Chinese as my display language, the default encoding of my system is GBK. It seems that yt-dlp tries to decode the configuration file as GBK, regardless of the actual encoding.

Changing the code page to 65001 (UTF-8) with `chcp 65001` in `cmd` doesn't work. Removing CJK characters or changing the file encoding into GBK solves the problem, but I think saving the file as UTF-8 might be more reasonable.




Discussion:
Related? https://github.com/yt-dlp/yt-dlp/pull/710
> Related? #710

Sorry for not searching in pull requests. Closing the issue.
Issue should still be open as the PR has not been merged yet
The current behavior is to use the system codepage to read the config files (see https://github.com/ytdl-org/youtube-dl/commit/dfe5fa49aed02cf36ba9f743b11b0903554b5e56). #710 tries to use utf8 as default, but this is not an acceptable solution as it will break many people's existing configs. We did discuss adding a switch for it, but no implementation has been made yet
Will this be implemented via a switch of some sort, or the presence of a UTF-8 signature at the beginning of the file, or some other kind of indicator that the file should be taken as UTF-8? I'm on Windows and have Emoji in my config for some filter expressions, and there's no way I can encode that as anything but Unicode. Will it recognize a UTF-16 BOM and read a file as that in the meantime?
I'd suggest to specify the encoding by writing preamble like `# coding: utf-8` to the beginning of the file
Since GBK and UTF-8 (and most of other regional encodings like EUC-JP but UTF>=16 and EBCDIC) map 0x00-0x7f to the ASCII equivalent, it works on most platforms without breaking anything
No preamble? Use system locale
> I'd suggest to specify the encoding by writing preamble like `# coding: utf-8` to the beginning of the file Since GBK and UTF-8 (and most of other regional encodings like EUC-JP but UTF>=16 and EBCDIC) map 0x00-0x7f to the ASCII equivalent, it works on most platforms without breaking anything No preamble? Use system locale

Good idea! I can't wait for this.
Read the first line **in bytes** should work.
> I'd suggest to specify the encoding by writing preamble like `# coding: utf-8` to the beginning of the file Since GBK and UTF-8 (and most of other regional encodings like EUC-JP but UTF>=16 and EBCDIC) map 0x00-0x7f to the ASCII equivalent, it works on most platforms without breaking anything No preamble? Use system locale

I agree that sounds like a reasonable way to do it. IIRC python doesn't even support EBCDIC so that's that taken care of, and as long as you also detect a UTF-16 or UTF-32 BOM as well as the # coding tag to change the encoding on the fly that won't be a problem. You could theoretically disallow UTF-16 and UTF-32 entirely, since they can be expressed in UTF-8 and they're rarely used in files, but you would at least want to detect the BOM and give an error in that case, since otherwise the interleaved null bytes would probably break who knows what.

Resolving patch:
diff --git a/yt_dlp/utils.py b/yt_dlp/utils.py
--- a/yt_dlp/utils.py
+++ b/yt_dlp/utils.py
@@ -3485,17 +3485,19 @@ def age_restricted(content_limit, age_limit):
     return age_limit < content_limit
 
 
+BOMS = [
+    (b'\xef\xbb\xbf', 'utf-8'),
+    (b'\x00\x00\xfe\xff', 'utf-32-be'),
+    (b'\xff\xfe\x00\x00', 'utf-32-le'),
+    (b'\xff\xfe', 'utf-16-le'),
+    (b'\xfe\xff', 'utf-16-be'),
+]
+""" List of known byte-order-marks (BOM) """
+
+
 def is_html(first_bytes):
     """ Detect whether a file contains HTML by examining its first bytes. """
 
-    BOMS = [
-        (b'\xef\xbb\xbf', 'utf-8'),
-        (b'\x00\x00\xfe\xff', 'utf-32-be'),
-        (b'\xff\xfe\x00\x00', 'utf-32-le'),
-        (b'\xff\xfe', 'utf-16-le'),
-        (b'\xfe\xff', 'utf-16-be'),
-    ]
-
     encoding = 'utf-8'
     for bom, enc in BOMS:
         while first_bytes.startswith(bom):
@@ -5394,6 +5396,41 @@ def read_stdin(what):
     return sys.stdin
 
 
+def determine_file_encoding(data):
+    """
+    From the first 512 bytes of a given file,
+    it tries to detect the encoding to be used to read as text.
+
+    @returns (encoding, bytes to skip)
+    """
+
+    for bom, enc in BOMS:
+        # matching BOM beats any declaration
+        # BOMs are skipped to prevent any errors
+        if data.startswith(bom):
+            return enc, len(bom)
+
+    # strip off all null bytes to match even when UTF-16 or UTF-32 is used
+    # endians don't matter
+    data = data.replace(b'\0', b'')
+
+    PREAMBLES = [
+        # "# -*- coding: utf-8 -*-"
+        # "# coding: utf-8"
+        rb'(?m)^#(?:\s+-\*-)?\s*coding\s*:\s*(?P<encoding>\S+)(?:\s+-\*-)?\s*$',
+        # "# vi: set fileencoding=utf-8"
+        rb'^#\s+vi\s*:\s+set\s+fileencoding=(?P<encoding>[^\s,]+)'
+    ]
+    for pb in PREAMBLES:
+        mobj = re.match(pb, data)
+        if not mobj:
+            continue
+        # preambles aren't skipped since they're just ignored when reading as config
+        return mobj.group('encoding').decode(), 0
+
+    return None, 0
+
+
 class Config:
     own_args = None
     parsed_args = None
@@ -5445,12 +5482,17 @@ def __str__(self):
     @staticmethod
     def read_file(filename, default=[]):
         try:
-            optionf = open(filename)
+            optionf = open(filename, 'rb')
         except OSError:
             return default  # silently skip if file is not present
+        try:
+            enc, skip = determine_file_encoding(optionf.read(512))
+            optionf.seek(skip, io.SEEK_SET)
+        except OSError:
+            enc = None  # silently skip read errors
         try:
             # FIXME: https://github.com/ytdl-org/youtube-dl/commit/dfe5fa49aed02cf36ba9f743b11b0903554b5e56
-            contents = optionf.read()
+            contents = optionf.read().decode(enc or preferredencoding())
             res = shlex.split(contents, comments=True)
         except Exception as err:
             raise ValueError(f'Unable to parse "{filename}": {err}')


Test patch:
diff --git a/test/test_utils.py b/test/test_utils.py
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -39,6 +39,7 @@
     datetime_from_str,
     detect_exe_version,
     determine_ext,
+    determine_file_encoding,
     dfxp2srt,
     dict_get,
     encode_base_n,
@@ -1822,6 +1823,33 @@ def test_locked_file(self):
             with contextlib.suppress(OSError):
                 os.remove(FILE)
 
+    def test_determine_file_encoding(self):
+        self.assertEqual(determine_file_encoding(b''), (None, 0))
+        self.assertEqual(determine_file_encoding(b'--verbose -x --audio-format mkv\n'), (None, 0))
+
+        self.assertEqual(determine_file_encoding(b'\xef\xbb\xbf'), ('utf-8', 3))
+        self.assertEqual(determine_file_encoding(b'\x00\x00\xfe\xff'), ('utf-32-be', 4))
+        self.assertEqual(determine_file_encoding(b'\xff\xfe'), ('utf-16-le', 2))
+
+        self.assertEqual(determine_file_encoding(b'# -*- coding: cp932 -*-'), ('cp932', 0))
+        self.assertEqual(determine_file_encoding(b'# -*- coding: cp932 -*-\n'), ('cp932', 0))
+        self.assertEqual(determine_file_encoding(b'# -*- coding: cp932 -*-\r\n'), ('cp932', 0))
+
+        self.assertEqual(determine_file_encoding(b'# coding: utf-8\n--verbose'), ('utf-8', 0))
+        self.assertEqual(determine_file_encoding(b'# coding: someencodinghere-12345\n--verbose'), ('someencodinghere-12345', 0))
+
+        self.assertEqual(determine_file_encoding(b'# vi: set fileencoding=cp932'), ('cp932', 0))
+        self.assertEqual(determine_file_encoding(b'# vi: set fileencoding=cp932\n'), ('cp932', 0))
+        self.assertEqual(determine_file_encoding(b'# vi: set fileencoding=cp932\r\n'), ('cp932', 0))
+        self.assertEqual(determine_file_encoding(b'# vi: set fileencoding=cp932,euc-jp\r\n'), ('cp932', 0))
+
+        self.assertEqual(determine_file_encoding(
+            b'\0\0\0#\0\0\0 \0\0\0c\0\0\0o\0\0\0d\0\0\0i\0\0\0n\0\0\0g\0\0\0:\0\0\0 \0\0\0u\0\0\0t\0\0\0f\0\0\0-\0\0\x003\0\0\x002\0\0\0-\0\0\0b\0\0\0e'),
+            ('utf-32-be', 0))
+        self.assertEqual(determine_file_encoding(
+            b'#\0 \0c\0o\0d\0i\0n\0g\0:\0 \0u\0t\0f\0-\x001\x006\0-\0l\0e\0'),
+            ('utf-16-le', 0))
+
 
 if __name__ == '__main__':
     unittest.main()