Repository: spyder-ide/spyder

Issue:
Realtime code analysis stops working when non-ASCII characters are used in strings
Hi everyone. Searched for the problem I'm about to report, and didn't found anything. Please excuse me about any possible mistakes, as I'm not used to Github issue reporting.

## Problem Description

Code highlighting and analysis suddenly stops as soon as you type any latin1 character inside the editor. Removing the character resumes the normal process.

(PLEASE NOTICE THE DIFFERENCE IN THE HIGHLIGHTED "ê")
As a Brazilian user (many ê, â, ã etc needs), this is somewhat frustrating.

![with_mark](https://user-images.githubusercontent.com/9609651/38823615-3b8e38c8-417d-11e8-82a7-4cc985563106.JPG)

![without_mark](https://user-images.githubusercontent.com/9609651/38823627-40cac900-417d-11e8-91ec-82be031b04f4.JPG)


## Package Versions

* Spyder: 3.2.8
* Python: 3.6
* PyQt: 5.9.2
* Operating System: W10

### Dependencies

```
IPython >=4.0    :  6.3.0 (OK)
cython >=0.21    :  0.28.2 (OK)
jedi >=0.9.0     :  0.11.1 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
numpy >=1.7      :  1.14.2 (OK)
pandas >=0.13.1  :  None (NOK)
pycodestyle >=2.3:  2.3.1 (OK)
pyflakes >=0.6.0 :  1.6.0 (OK)
pygments >=2.0   :  2.2.0 (OK)
pylint >=0.25    :  1.8.3 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
rope >=0.9.4     :  0.10.7 (OK)
sphinx >=0.6.6   :  1.7.2 (OK)
sympy >=0.7.3    :  None (NOK)

```



Discussion:
Thanks for reporting. I reproduced the above error by setting a file to ISO 8859-1 (Latin-1) in Np++ and then opening it in Spyder and entering/removing the character above in a string. Furthermore, I tried with proper UTF-8 (which of course should always be used whenever possible instead of archaic legacy encoding) and reproduced it as well. Interestingly, the first item in the test file, a wildcard import like ``numpy import *`` was sometimes still flagged, but nothing else was. @ccordoba12 any ideas?
Most probably we're not sending the file as utf-8 to Pyflakes. I'd have to take a look at it to confirm.
@dalthviz, please work on this one. It seems a Windows only issue (because I can't reproduced it on Linux).
> Most probably we're not sending the file as utf-8 to Pyflakes.

>  It seems a Windows only issue

That would seem plausible, since reading and writing with ``file()`` on Windows defaults to the current locale default encoding, e.g. Windows-1252 for Western European locales, etc., rather than UTF-8 like all (sane) operating systems do, unless UTF-8 is explicitly specified.
By setting `SPYDER_DEBUG = 3` the problem seems like an encoding error (tested with a file with ISO 8859-1):

```python
Traceback (most recent call last):
  File "C:\Users\Daniel\Desktop\Spyder\spyder\spyder\utils\codeanalysis.py", lin
e 174, in check_with_pep8
    results = check(args, source_code, filename=filename, options=['-r'])
  File "C:\Users\Daniel\Desktop\Spyder\spyder\spyder\utils\codeanalysis.py", lin
e 161, in check
    text = to_text_string(lines[lineno-1], coding)
  File "C:\Users\Daniel\Desktop\Spyder\spyder\spyder\py3compat.py", line 145, in
 to_text_string
    return str(obj, encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xea in position 32: unexpec
ted end of data
```
Furthermore, this only happens for me when the file has [the comment for its encoding at the beging of the file](https://www.python.org/dev/peps/pep-0263/) and it doesn't match with the actual file encoding. 

@dfrison01 could you give us a bit of information about the enconding you are using in your file, or even an example file where you are having this difficulty to test it and look if the problem is the same as the one I describe above?
> Furthermore, this only happens for me when the file has the comment for its encoding at the beging of the file and it doesn't match with the actual file encoding.

@dalthviz, then we should catch this error and fallback to detect the encoding with `encoding.get_coding` and pass the result to `to_text_string(lines[lineno-1], coding)`
@ccordoba12 we are already using `enconding.get_coding` [here](https://github.com/spyder-ide/spyder/blob/bd86ec132ce06e9342186817a2a5e95b1177646c/spyder/utils/codeanalysis.py#L156), should I change the behavior of the function to make possible the use of the chardet fallback directly and handle the expection [here](https://github.com/spyder-ide/spyder/blob/bd86ec132ce06e9342186817a2a5e95b1177646c/spyder/utils/codeanalysis.py#L161)?
Yes, please.

Resolving patch:
diff --git a/spyder/utils/codeanalysis.py b/spyder/utils/codeanalysis.py
--- a/spyder/utils/codeanalysis.py
+++ b/spyder/utils/codeanalysis.py
@@ -161,6 +161,11 @@ def check(args, source_code, filename=None, options=None):
             text = to_text_string(lines[lineno-1], coding)
         except TypeError:
             text = to_text_string(lines[lineno-1])
+        except UnicodeDecodeError:
+            # Needed to handle UnicodeDecodeError and force the use
+            # of chardet to detect enconding. See issue 6970
+            coding = encoding.get_coding(source_code, force_chardet=True)
+            text = to_text_string(lines[lineno-1], coding)
         if 'analysis:ignore' not in text:
             message = line[line.find(': ')+2:]
             results.append((message, lineno))
diff --git a/spyder/utils/encoding.py b/spyder/utils/encoding.py
--- a/spyder/utils/encoding.py
+++ b/spyder/utils/encoding.py
@@ -102,25 +102,29 @@ def to_fs_from_unicode(unic):
           'iso8859-10', 'iso8859-13', 'iso8859-14', 'latin-1',
           'utf-16']
 
-def get_coding(text):
+
+def get_coding(text, force_chardet=False):
     """
     Function to get the coding of a text.
     @param text text to inspect (string)
     @return coding string
     """
-    for line in text.splitlines()[:2]:
-        try:
-            result = CODING_RE.search(to_text_string(line))
-        except UnicodeDecodeError:
-            # This could fail because to_text_string assume the text is utf8-like
-            # and we don't know the encoding to give it to to_text_string
-            pass
-        else:
-            if result:
-                codec = result.group(1)
-                # sometimes we find a false encoding that can result in errors
-                if codec in CODECS:
-                    return codec
+    if not force_chardet:
+        for line in text.splitlines()[:2]:
+            try:
+                result = CODING_RE.search(to_text_string(line))
+            except UnicodeDecodeError:
+                # This could fail because to_text_string assume the text
+                # is utf8-like and we don't know the encoding to give
+                # it to to_text_string
+                pass
+            else:
+                if result:
+                    codec = result.group(1)
+                    # sometimes we find a false encoding that can
+                    # result in errors
+                    if codec in CODECS:
+                        return codec
 
     # Fallback using chardet
     if is_binary_string(text):


Test patch:
diff --git a/spyder/tests/test_dont_use.py b/spyder/tests/test_dont_use.py
--- a/spyder/tests/test_dont_use.py
+++ b/spyder/tests/test_dont_use.py
@@ -14,15 +14,16 @@
 
 
 @pytest.mark.parametrize("pattern,exclude_patterns,message", [
-    ("isinstance\(.*,.*str\)", ['py3compat.py'],
+    ("isinstance\(.*,.*str\)", ['py3compat.py', 'example_latin1.py'],
      ("Don't use builtin isinstance() function,"
       "use spyder.py3compat.is_text_string() instead")),
 
-    (r"^[\s\#]*\bprint\(((?!file=).)*\)", ['.*test.*', 'example.py', 'binaryornot'],
+    (r"^[\s\#]*\bprint\(((?!file=).)*\)", ['.*test.*', 'example.py',
+                                           'example_latin1.py', 'binaryornot'],
      ("Don't use print() functions, ",
       "for debuging you could use debug_print instead")),
 
-    (r"^[\s\#]*\bprint\s+(?!>>)((?!#).)*", ['.*test.*'],
+    (r"^[\s\#]*\bprint\s+(?!>>)((?!#).)*", ['.*test.*', 'example_latin1.py'],
      ("Don't use print __builtin__, ",
       "for debuging you could use debug_print instead")),
 ])
diff --git a/spyder/utils/tests/data/example_latin1.py b/spyder/utils/tests/data/example_latin1.py
new file mode 100644
--- /dev/null
+++ b/spyder/utils/tests/data/example_latin1.py
@@ -0,0 +1,3 @@
+# -*- coding: utf-8 -*-
+#�
+foo
diff --git a/spyder/utils/tests/test_codeanalysis.py b/spyder/utils/tests/test_codeanalysis.py
--- a/spyder/utils/tests/test_codeanalysis.py
+++ b/spyder/utils/tests/test_codeanalysis.py
@@ -10,6 +10,7 @@
 
 # Standard library imports
 import os
+import io
 
 # Test library imports
 import pytest
@@ -20,6 +21,22 @@
 from spyder.py3compat import PY2
 
 TEST_FILE = os.path.join(os.path.dirname(__file__), 'data/example.py')
+TEST_FILE_LATIN = os.path.join(os.path.dirname(__file__),
+                               'data/example_latin1.py')
+
+
+def test_codeanalysis_latin():
+    """Test codeanalysis with pyflakes and pep8."""
+    code = io.open(TEST_FILE_LATIN, encoding="iso-8859-1").read()
+    check_results = (check_with_pyflakes(code, TEST_FILE_LATIN)
+                     + check_with_pep8(code, TEST_FILE_LATIN)
+                     + find_tasks(code))
+    if PY2:
+        num_results = 1
+    else:
+        num_results = 2
+    assert len(check_results) == num_results
+
 
 def test_codeanalysis():
     """Test codeanalysis with pyflakes and pep8."""