Repository: commitizen-tools/commitizen

Issue:
`UnicodeDecodeError` if commit messages contain Unicode characters
### Description

If I run
```
cz changelog
```
and the commit messages contain Unicode characters like 🤦🏻‍♂️ (which is an eight-byte utf-8 sequence: `\xf0\x9f\xa4\xa6 \xf0\x9f\x8f\xbb`) then I get the following traceback
```
Traceback (most recent call last):
  File "/.../.venv/bin/cz", line 8, in <module>
    sys.exit(main())
  File "/.../.venv/lib/python3.10/site-packages/commitizen/cli.py", line 389, in main
    args.func(conf, vars(args))()
  File "/.../.venv/lib/python3.10/site-packages/commitizen/commands/changelog.py", line 143, in __call__
    commits = git.get_commits(
  File "/.../.venv/lib/python3.10/site-packages/commitizen/git.py", line 98, in get_commits
    c = cmd.run(command)
  File "/.../.venv/lib/python3.10/site-packages/commitizen/cmd.py", line 32, in run
    stdout.decode(chardet.detect(stdout)["encoding"] or "utf-8"),
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/encodings/cp1254.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 1689: character maps to <undefined>
```
The result of `chardet.detect()` here

https://github.com/commitizen-tools/commitizen/blob/2ff9f155435b487057ce5bd8e32e1ab02fd57c94/commitizen/cmd.py#L26

is:
```
{'encoding': 'Windows-1254', 'confidence': 0.6864215607255395, 'language': 'Turkish'}
```
An interesting character encoding prediction with a low confidence, which in turn picks the incorrect codec and then decoding the `bytes` fails. Using `decode("utf-8")` works fine. It looks like issue https://github.com/chardet/chardet/issues/148 is related to this.

I think the fix would be something like this to replace [these lines of code](https://github.com/commitizen-tools/commitizen/blob/2ff9f155435b487057ce5bd8e32e1ab02fd57c94/commitizen/cmd.py#L23-L31):
```python
stdout, stderr = process.communicate()
return_code = process.returncode
try:
    stdout_s = stdout.decode("utf-8")  # Try this one first.
except UnicodeDecodeError:
    result = chardet.detect(stdout)  # Final result of the UniversalDetector’s prediction.
    # Consider checking confidence value of the result?
    stdout_s = stdout.decode(result["encoding"])
try:
    stderr_s = stderr.decode("utf-8")  # Try this one first.
except UnicodeDecodeError:
    result = chardet.detect(stderr)  # Final result of the UniversalDetector’s prediction.
    # Consider checking confidence value of the result?
    stderr_s = stderr.decode(result["encoding"])
return Command(stdout_s, stderr_s, stdout, stderr, return_code)
```

### Steps to reproduce

Well I suppose you can add a few commits to a local branch an go crazy with much text and funky unicode characters (emojis with skin tones, flags, etc.), and then attempt to create a changelog.

### Current behavior

`cz` throws an exception.

### Desired behavior

`cz` creates a changelog.

### Screenshots

_No response_

### Environment

```
> cz version
2.29.3
> python --version
Python 3.10.5
> uname -a
Darwin pooh 18.7.0 Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASE_X86_64 x86_64 i386 Darwin
```

Comments:

Patch:
diff --git a/commitizen/cmd.py b/commitizen/cmd.py
index 7f4efb6a..656dea07 100644
--- a/commitizen/cmd.py
+++ b/commitizen/cmd.py
@@ -3,6 +3,8 @@ from typing import NamedTuple
 
 from charset_normalizer import from_bytes
 
+from commitizen.exceptions import CharacterSetDecodeError
+
 
 class Command(NamedTuple):
     out: str
@@ -12,6 +14,19 @@ class Command(NamedTuple):
     return_code: int
 
 
+def _try_decode(bytes_: bytes) -> str:
+    try:
+        return bytes_.decode("utf-8")
+    except UnicodeDecodeError:
+        charset_match = from_bytes(bytes_).best()
+        if charset_match is None:
+            raise CharacterSetDecodeError()
+        try:
+            return bytes_.decode(charset_match.encoding)
+        except UnicodeDecodeError as e:
+            raise CharacterSetDecodeError() from e
+
+
 def run(cmd: str) -> Command:
     process = subprocess.Popen(
         cmd,
@@ -23,8 +38,8 @@ def run(cmd: str) -> Command:
     stdout, stderr = process.communicate()
     return_code = process.returncode
     return Command(
-        str(from_bytes(stdout).best()),
-        str(from_bytes(stderr).best()),
+        _try_decode(stdout),
+        _try_decode(stderr),
         stdout,
         stderr,
         return_code,
diff --git a/commitizen/exceptions.py b/commitizen/exceptions.py
index a95ab3b5..16869b5a 100644
--- a/commitizen/exceptions.py
+++ b/commitizen/exceptions.py
@@ -26,6 +26,7 @@ class ExitCode(enum.IntEnum):
     INVALID_CONFIGURATION = 19
     NOT_ALLOWED = 20
     NO_INCREMENT = 21
+    UNRECOGNIZED_CHARACTERSET_ENCODING = 22
 
 
 class CommitizenException(Exception):
@@ -148,3 +149,7 @@ class InvalidConfigurationError(CommitizenException):
 
 class NotAllowed(CommitizenException):
     exit_code = ExitCode.NOT_ALLOWED
+
+
+class CharacterSetDecodeError(CommitizenException):
+    exit_code = ExitCode.UNRECOGNIZED_CHARACTERSET_ENCODING
diff --git a/docs/exit_codes.md b/docs/exit_codes.md
index f4c2fa82..b3996460 100644
--- a/docs/exit_codes.md
+++ b/docs/exit_codes.md
@@ -28,4 +28,5 @@ These exit codes can be found in `commitizen/exceptions.py::ExitCode`.
 | InvalidCommandArgumentError | 18        | The argument provide to command is invalid (e.g. `cz check -commit-msg-file filename --rev-range master..`) |
 | InvalidConfigurationError   | 19        | An error was found in the Commitizen Configuration, such as duplicates in `change_type_order`               |
 | NotAllowed                  | 20        | `--incremental` cannot be combined with a `rev_range`                                                       |
-| NoneIncrementExit           | 21        | The commits found are not elegible to be bumped                                                             |
+| NoneIncrementExit           | 21        | The commits found are not eligible to be bumped                                                             |
+| CharacterSetDecodeError     | 22        | The character encoding of the command output could not be determined                                        |


Test patch:
diff --git a/tests/test_cmd.py b/tests/test_cmd.py
new file mode 100644
index 00000000..e8a869e0
--- /dev/null
+++ b/tests/test_cmd.py
@@ -0,0 +1,53 @@
+import pytest
+
+from commitizen import cmd
+from commitizen.exceptions import CharacterSetDecodeError
+
+
+# https://docs.python.org/3/howto/unicode.html
+def test_valid_utf8_encoded_strings():
+    valid_strings = (
+        "",
+        "ascii",
+        "🤦🏻‍♂️",
+        "﷽",
+        "\u0000",
+    )
+    assert all(s == cmd._try_decode(s.encode("utf-8")) for s in valid_strings)
+
+
+# A word of caution: just because an encoding can be guessed for a given
+# sequence of bytes and because that guessed encoding may yield a decoded
+# string, does not mean that that string was the original! For more, see:
+# https://docs.python.org/3/library/codecs.html#standard-encodings
+
+
+# Pick a random, non-utf8 encoding to test.
+def test_valid_cp1250_encoded_strings():
+    valid_strings = (
+        "",
+        "ascii",
+        "äöüß",
+        "ça va",
+        "jak se máte",
+    )
+    for s in valid_strings:
+        assert cmd._try_decode(s.encode("cp1250")) or True
+
+
+def test_invalid_bytes():
+    invalid_bytes = (b"\x73\xe2\x9d\xff\x00",)
+    for s in invalid_bytes:
+        with pytest.raises(CharacterSetDecodeError):
+            cmd._try_decode(s)
+
+
+def test_always_fail_decode():
+    class _bytes(bytes):
+        def decode(self, encoding="utf-8", errors="strict"):
+            raise UnicodeDecodeError(
+                encoding, self, 0, 0, "Failing intentionally for testing"
+            )
+
+    with pytest.raises(CharacterSetDecodeError):
+        cmd._try_decode(_bytes())