Repository: PyCQA/doc8

Issue:
Do not use chardet to detect encoding because of poor accuracy
Please do not use chardet to detect document encoding. For UTF-8 texts it works more or less reliably only for Latin-1 and Latin-1 Supplement unicode blocks, for Latin Extended-A and Extended-B it fails in about 50% cases wrongly detecting Windows encodings, eg. for UTF-8 document with Latin Extended-A content:

```
$ chardet docs/index.rst 
docs/index.rst: Windows-1252 with confidence 0.594336283186
$ file docs/index.rst 
docs/index.rst: UTF-8 Unicode text
```

In fact chardet reports low confidence in this case.

Discussion:

Patch:
diff --git a/README.rst b/README.rst
index 76113f7..c8536a6 100644
--- a/README.rst
+++ b/README.rst
@@ -78,9 +78,7 @@ Usage
                             default file extension to use when a file is found
                             without a file extension.
       --file-encoding encoding
-                            override encoding to use when attempting to determine
-                            an input files text encoding (providing this avoids
-                            using `chardet` to automatically detect encoding/s)
+                            set input files text encoding
       --max-line-length int
                             maximum allowed line length (default: 79).
       -e extension, --extension extension
diff --git a/doc8/main.py b/doc8/main.py
index 1962f6a..fe604ba 100644
--- a/doc8/main.py
+++ b/doc8/main.py
@@ -437,10 +437,7 @@ def main():
     parser.add_argument(
         "--file-encoding",
         action="store",
-        help="override encoding to use when attempting"
-        " to determine an input files text encoding "
-        "(providing this avoids using `chardet` to"
-        " automatically detect encoding/s)",
+        help="set input files text encoding",
         default=defaults["file_encoding"],
         dest="file_encoding",
         metavar="encoding",
diff --git a/doc8/parser.py b/doc8/parser.py
index e223212..ae7e411 100644
--- a/doc8/parser.py
+++ b/doc8/parser.py
@@ -16,7 +16,6 @@ import errno
 import os
 import threading
 
-import chardet
 from docutils import frontend
 from docutils import parsers as docutils_parser
 from docutils import utils
@@ -110,10 +109,7 @@ class ParsedFile(object):
     @property
     def encoding(self):
         if not self._encoding:
-            encoding = chardet.detect(self.raw_contents)["encoding"]
-            if not encoding:
-                encoding = self.FALLBACK_ENCODING
-            self._encoding = encoding
+            self._encoding = self.FALLBACK_ENCODING
         return self._encoding
 
     @property
diff --git a/requirements.txt b/requirements.txt
index 74b6ee2..5592636 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -2,7 +2,6 @@
 # of appearance. Changing the order has an impact on the overall integration
 # process, which may cause wedges in the gate later.
 
-chardet
 docutils
 restructuredtext-lint>=0.7
 stevedore


Test patch:
diff --git a/doc8/tests/test_main.py b/doc8/tests/test_main.py
index a9e43a9..00b6f93 100644
--- a/doc8/tests/test_main.py
+++ b/doc8/tests/test_main.py
@@ -41,7 +41,7 @@ OUTPUT_CMD_VERBOSE = """\
 Scanning...
   Selecting '{path}/invalid.rst'
 Validating...
-Validating {path}/invalid.rst (ascii, 10 chars, 1 lines)
+Validating {path}/invalid.rst (utf-8, 10 chars, 1 lines)
   Running check 'doc8.checks.CheckValidity'
   Running check 'doc8.checks.CheckTrailingWhitespace'
     - {path}/invalid.rst:1: D002 Trailing whitespace