Repository: spyder-ide/spyder

Issue:
Spyder not conform PEP3120
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

Spyder 5.3.0 on MS Windows 10. Open a UTF-8 encoded source file (without -\*- coding: utf-8 -\*- encoding declaration). 

**Spyder DOES NOT RESPECT PEP3120**: the file will be opened with Windows-1250 encoding. (It may/may not depends on BOM.) **This is a bug.** 

The proper way (for Python 3, see PEP3120) is to open each file (by default) as UTF-8 encoding.

Obviously one can enforce UTF-8 encoding (see PEP263). It is a workaround.

BTW, PyCharm and MS Visual Code, the same PC/OS, behave as expected: open .py files with UTF-8 as default encodings.


## Versions

* Spyder version: 5.3.0 
* Python version: 3.9.12
* Qt version: 5.12.9
* PyQt5 version: 5.12.3
* Operating System: Windows 10

### Dependencies

```
# Mandatory:
atomicwrites >=1.2.0          :  1.4.0 (OK)
chardet >=2.0.0               :  4.0.0 (OK)
cloudpickle >=0.5.0           :  2.0.0 (OK)
cookiecutter >=1.6.0          :  1.7.3 (OK)
diff_match_patch >=20181111   :  20200713 (OK)
intervaltree >=3.0.2          :  3.0.2 (OK)
IPython >=7.31.1;<8.0.0       :  7.33.0 (OK)
jedi >=0.17.2;<0.19.0         :  0.18.1 (OK)
jellyfish >=0.7               :  0.9.0 (OK)
jsonschema >=3.2.0            :  4.4.0 (OK)
keyring >=17.0.0              :  23.4.0 (OK)
nbconvert >=4.0               :  6.5.0 (OK)
numpydoc >=0.6.0              :  1.3.1 (OK)
paramiko >=2.4.0              :  2.10.4 (OK)
parso >=0.7.0;<0.9.0          :  0.8.3 (OK)
pexpect >=4.4.0               :  4.8.0 (OK)
pickleshare >=0.4             :  0.7.5 (OK)
psutil >=5.3                  :  5.9.0 (OK)
pygments >=2.0                :  2.12.0 (OK)
pylint >=2.5.0                :  2.13.7 (OK)
pyls_spyder >=0.4.0           :  0.4.0 (OK)
pylsp >=1.4.1;<1.5.0          :  1.4.1 (OK)
pylsp_black >=1.2.0           :  1.2.1 (OK)
qdarkstyle >=3.0.2;<3.1.0     :  3.0.3 (OK)
qstylizer >=0.1.10            :  0.2.1 (OK)
qtawesome >=1.0.2             :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0      :  5.3.0 (OK)
qtpy >=2.0.1                  :  2.0.1 (OK)
rtree >=0.9.7                 :  1.0.0 (OK)
setuptools >=49.6.0           :  62.1.0 (OK)
sphinx >=0.6.6                :  4.5.0 (OK)
spyder_kernels >=2.3.0;<2.4.0 :  2.3.0 (OK)
textdistance >=4.2.0          :  4.2.2 (OK)
three_merge >=0.1.1           :  0.1.1 (OK)
watchdog >=0.10.3             :  2.1.7 (OK)
zmq >=17                      :  22.3.0 (OK)

# Optional:
cython >=0.21                 :  None (NOK)
matplotlib >=3.0.0            :  3.5.1 (OK)
numpy >=1.7                   :  1.22.3 (OK)
pandas >=1.1.1                :  None (NOK)
scipy >=0.17.0                :  1.8.0 (OK)
sympy >=0.7.3                 :  None (NOK)
```



Discussion:
Hi @slawomirmarczynski thank you for the feedback! I think I was able to reproduce this. We will try to fix it in a future release :+1:
@ccordoba12 @dalthviz I have a similar issue opening UTF-8 encoded `.md` files (no BOM), that are detected as `MACROMAN`. Can you give me a hint of where the autodetecting code is, so I try to fix this?

Resolving patch:
diff --git a/spyder/utils/encoding.py b/spyder/utils/encoding.py
--- a/spyder/utils/encoding.py
+++ b/spyder/utils/encoding.py
@@ -35,6 +35,7 @@
 
 PREFERRED_ENCODING = locale.getpreferredencoding()
 
+
 def transcode(text, input=PREFERRED_ENCODING, output=PREFERRED_ENCODING):
     """Transcode a text string"""
     try:
@@ -45,10 +46,11 @@ def transcode(text, input=PREFERRED_ENCODING, output=PREFERRED_ENCODING):
         except UnicodeError:
             return text
 
-#------------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------------
 #  Functions for encoding and decoding bytes that come from
 #  the *file system*.
-#------------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
 
 # The default encoding for file paths and environment variables should be set
 # to match the default encoding that the OS is using.
@@ -63,13 +65,15 @@ def getfilesystemencoding():
         encoding = PREFERRED_ENCODING
     return encoding
 
+
 FS_ENCODING = getfilesystemencoding()
 
+
 def to_unicode_from_fs(string):
     """
     Return a unicode version of string decoded using the file system encoding.
     """
-    if not is_string(string): # string is a QString
+    if not is_string(string):  # string is a QString
         string = to_text_string(string.toUtf8(), 'utf-8')
     else:
         if is_binary_string(string):
@@ -81,6 +85,7 @@ def to_unicode_from_fs(string):
                 return unic
     return string
 
+
 def to_fs_from_unicode(unic):
     """
     Return a byte string version of unic encoded using the file
@@ -95,23 +100,34 @@ def to_fs_from_unicode(unic):
             return string
     return unic
 
-#------------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------------
 #  Functions for encoding and decoding *text data* itself, usually originating
 #  from or destined for the *contents* of a file.
-#------------------------------------------------------------------------------
+# -----------------------------------------------------------------------------
 
 # Codecs for working with files and text.
 CODING_RE = re.compile(r"coding[:=]\s*([-\w_.]+)")
-CODECS = ['utf-8', 'iso8859-1',  'iso8859-15', 'ascii', 'koi8-r', 'cp1251',
-          'koi8-u', 'iso8859-2', 'iso8859-3', 'iso8859-4', 'iso8859-5',
-          'iso8859-6', 'iso8859-7', 'iso8859-8', 'iso8859-9',
-          'iso8859-10', 'iso8859-13', 'iso8859-14', 'latin-1',
-          'utf-16']
+CODECS = [
+    'utf-8', 'iso8859-1',  'iso8859-15', 'ascii', 'koi8-r', 'cp1251',
+    'koi8-u', 'iso8859-2', 'iso8859-3', 'iso8859-4', 'iso8859-5',
+    'iso8859-6', 'iso8859-7', 'iso8859-8', 'iso8859-9',
+    'iso8859-10', 'iso8859-13', 'iso8859-14', 'latin-1', 'utf-16'
+]
 
 
-def get_coding(text, force_chardet=False):
+def get_coding(text, force_chardet=False, default_codec=None):
     """
     Function to get the coding of a text.
+
+    By default, first look for declarations of the form "coding=xxx" in the
+    first two lines of text. If none is found, then use the chardet library to
+    guess. If that also fails, then return None.
+
+    If force_chardet is set, then do not look for "coding=xxx" and only use
+    the chardet library. If default_codec is set, then do not use the chardet
+    library and return default_codec if there is no "coding=xxx" declaration.
+
     @param text text to inspect (string)
     @return coding string
     """
@@ -133,18 +149,20 @@ def get_coding(text, force_chardet=False):
                         return codec
 
     # Fallback using chardet
-    if is_binary_string(text):
+    if is_binary_string(text) and (force_chardet or default_codec is None):
         detector = UniversalDetector()
         for line in text.splitlines()[:2]:
             detector.feed(line)
-            if detector.done: break
+            if detector.done:
+                break
 
         detector.close()
         return detector.result['encoding']
 
-    return None
+    return default_codec
 
-def decode(text):
+
+def decode(text, default_codec=None):
     """
     Function to decode a text.
     @param text text to decode (string)
@@ -160,7 +178,7 @@ def decode(text):
         elif text.startswith(BOM_UTF32):
             # UTF-32 with BOM
             return to_text_string(text[len(BOM_UTF32):], 'utf-32'), 'utf-32'
-        coding = get_coding(text)
+        coding = get_coding(text, default_codec=default_codec)
         if coding:
             return to_text_string(text, coding), coding
     except (UnicodeError, LookupError):
@@ -173,6 +191,7 @@ def decode(text):
     # Assume Latin-1 (behaviour before 3.7.1)
     return to_text_string(text, "latin-1"), 'latin-1-guessed'
 
+
 def encode(text, orig_coding):
     """
     Function to encode a text.
@@ -197,8 +216,8 @@ def encode(text, orig_coding):
             return text.encode(coding), coding
         except (UnicodeError, LookupError):
             raise RuntimeError("Incorrect encoding (%s)" % coding)
-    if orig_coding and orig_coding.endswith('-default') or \
-      orig_coding.endswith('-guessed'):
+    if (orig_coding and orig_coding.endswith('-default')
+            or orig_coding.endswith('-guessed')):
         coding = orig_coding.replace("-default", "")
         coding = orig_coding.replace("-guessed", "")
         try:
@@ -209,6 +228,7 @@ def encode(text, orig_coding):
     # Save as UTF-8 without BOM
     return text.encode('utf-8'), 'utf-8'
 
+
 def to_unicode(string):
     """Convert a string to unicode"""
     if not is_text_string(string):
@@ -296,14 +316,21 @@ def writelines(lines, filename, encoding='utf-8', mode='wb'):
     """
     return write(os.linesep.join(lines), filename, encoding, mode)
 
+
 def read(filename, encoding='utf-8'):
     """
     Read text from file ('filename')
     Return text and encoding
     """
-    text, encoding = decode( open(filename, 'rb').read() )
+    if filename.endswith(('.py', '.pyw', '.ipy', '.pyi')):
+        default_codec = 'utf-8'  # Per PEP3120
+    else:
+        default_codec = None
+    contents = open(filename, 'rb').read()
+    text, encoding = decode(contents, default_codec=default_codec)
     return text, encoding
 
+
 def readlines(filename, encoding='utf-8'):
     """
     Read lines from file ('filename')


Test patch:
diff --git a/spyder/utils/tests/copyright.py b/spyder/utils/tests/copyright.py
new file mode 100644
--- /dev/null
+++ b/spyder/utils/tests/copyright.py
@@ -0,0 +1,4 @@
+# -----------------------------------------------------------------------------
+# Copyright © 2014-2015 Colin Duquesnoy
+# Copyright © 2009- The Spyder Development Team
+# -----------------------------------------------------------------------------
diff --git a/spyder/utils/tests/copyright.txt b/spyder/utils/tests/copyright.txt
new file mode 100644
--- /dev/null
+++ b/spyder/utils/tests/copyright.txt
@@ -0,0 +1,4 @@
+# -----------------------------------------------------------------------------
+# Copyright © 2014-2015 Colin Duquesnoy
+# Copyright © 2009- The Spyder Development Team
+# -----------------------------------------------------------------------------
diff --git a/spyder/utils/tests/iso8859-9.py b/spyder/utils/tests/iso8859-9.py
new file mode 100644
--- /dev/null
+++ b/spyder/utils/tests/iso8859-9.py
@@ -0,0 +1,3 @@
+#!/usr/bin/env python3
+# -*- coding: iso8859-9 -*-
+answer = 42
diff --git a/spyder/utils/tests/test_encoding.py b/spyder/utils/tests/test_encoding.py
--- a/spyder/utils/tests/test_encoding.py
+++ b/spyder/utils/tests/test_encoding.py
@@ -12,7 +12,7 @@
 from flaky import flaky
 import pytest
 
-from spyder.utils.encoding import is_text_file, get_coding, write
+from spyder.utils.encoding import is_text_file, read, write
 from spyder.py3compat import to_text_string
 
 __location__ = os.path.realpath(os.path.join(os.getcwd(),
@@ -89,7 +89,7 @@ def test_timestamp(tmpdir):
 def test_is_text_file(tmpdir):
     p = tmpdir.mkdir("sub").join("random_text.txt")
     p.write("Some random text")
-    assert is_text_file(str(p)) == True
+    assert is_text_file(str(p)) is True
 
 
 @pytest.mark.parametrize(
@@ -99,11 +99,14 @@ def test_is_text_file(tmpdir):
      ('ascii', 'ascii.txt'),
      ('Big5', 'Big5.txt'),
      ('KOI8-R', 'KOI8-R.txt'),
+     ('iso-8859-1', 'copyright.txt'),
+     ('utf-8', 'copyright.py'),  # Python files are UTF-8 by default
+     ('iso8859-9', 'iso8859-9.py')  # Encoding declared in file
      ])
 def test_files_encodings(expected_encoding, text_file):
-    with open(os.path.join(__location__, text_file), 'rb') as f:
-        text = f.read()
-        assert get_coding(text).lower() == expected_encoding.lower()
+    file_path = os.path.join(__location__, text_file)
+    text, encoding = read(file_path)
+    assert encoding.lower() == expected_encoding.lower()
 
 
 if __name__ == '__main__':