Repository: spyder-ide/spyder

Issue:
Syntax highlighting for numeric literals containing underscores
<!--- **PLEASE READ:** Before submitting here, please carefully consult our *Troubleshooting Guide*: https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ and search the issues page for your error/problem, as most posted bugs are duplicates or easy fixes.

If you don't find anything, please provide a detailed step-by-step description (in English) of the problem and what led up to it below. Issue reports without a clear way to reproduce them will be closed. Thanks! --->

## Problem Description
I am using the [numeric literals with underscores](https://www.python.org/dev/peps/pep-0515/) new in Python 3.6, because I need to deal with large integers. My Spyder version does not do syntax highlighting correctly with these numbers.

### What steps will reproduce the problem?
Enter the integer `x = 1_000_000` in the editor. The number does not get highlighted as literal like in `x = 1000000`.


## Package Versions

* Spyder version: 3.2.8 
* Python version: 3.6.5
* Qt version: 5.9.3
* PyQt5 version: 5.9.2
* Operating system: Windows 7

### Dependencies

```
pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.22.0 (OK)
numpy >=1.7      :  1.13.3 (OK)
sphinx >=0.6.6   :  1.7.2 (OK)
rope >=0.9.4     :  0.10.5 (OK)
jedi >=0.9.0     :  0.11.1 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
cython >=0.21    :  0.28.1 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.2.1 (OK)
pylint >=0.25    :  1.8.3 (OK)
```



Discussion:
Thanks for reporting. We'll add this functionality in a future release.
@dalthviz, please work on this one.

Resolving patch:
diff --git a/spyder/py3compat.py b/spyder/py3compat.py
--- a/spyder/py3compat.py
+++ b/spyder/py3compat.py
@@ -24,6 +24,7 @@
 
 PY2 = sys.version[0] == '2'
 PY3 = sys.version[0] == '3'
+PY36_OR_MORE = sys.version_info[0] >= 3 and sys.version_info[1] >= 6
 
 #==============================================================================
 # Data types
diff --git a/spyder/utils/syntaxhighlighters.py b/spyder/utils/syntaxhighlighters.py
--- a/spyder/utils/syntaxhighlighters.py
+++ b/spyder/utils/syntaxhighlighters.py
@@ -25,7 +25,8 @@
 from spyder import dependencies
 from spyder.config.base import _
 from spyder.config.main import CONF
-from spyder.py3compat import builtins, is_text_string, to_text_string
+from spyder.py3compat import (builtins, is_text_string, to_text_string,
+                              PY36_OR_MORE)
 from spyder.utils.sourcecode import CELL_LANGUAGES
 from spyder.utils.workers import WorkerManager
 
@@ -294,12 +295,30 @@ def make_python_patterns(additional_keywords=[], additional_builtins=[]):
                                 r"\bcls\b",
                                 (r"^\s*@([a-zA-Z_][a-zA-Z0-9_]*)"
                                      r"(\.[a-zA-Z_][a-zA-Z0-9_]*)*")])
-    number = any("number",
-                 [r"\b[+-]?[0-9]+[lLjJ]?\b",
-                  r"\b[+-]?0[xX][0-9A-Fa-f]+[lL]?\b",
-                  r"\b[+-]?0[oO][0-7]+[lL]?\b",
-                  r"\b[+-]?0[bB][01]+[lL]?\b",
-                  r"\b[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?[jJ]?\b"])
+    number_regex = [r"\b[+-]?[0-9]+[lLjJ]?\b",
+                    r"\b[+-]?0[xX][0-9A-Fa-f]+[lL]?\b",
+                    r"\b[+-]?0[oO][0-7]+[lL]?\b",
+                    r"\b[+-]?0[bB][01]+[lL]?\b",
+                    r"\b[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?[jJ]?\b"]
+    # Needed to achieve correct highlighting in Python 3.6+
+    # See issue 7324
+    if PY36_OR_MORE:
+        # Based on
+        # https://github.com/python/cpython/blob/
+        # 81950495ba2c36056e0ce48fd37d514816c26747/Lib/tokenize.py#L117
+        # In order: Hexnumber, Binnumber, Octnumber, Decnumber,
+        # Pointfloat + Exponent, Expfloat, Imagnumber
+        number_regex = [
+                r"\b[+-]?0[xX](?:_?[0-9A-Fa-f])+[lL]?\b",
+                r"\b[+-]?0[bB](?:_?[01])+[lL]?\b",
+                r"\b[+-]?0[oO](?:_?[0-7])+[lL]?\b",
+                r"\b[+-]?(?:0(?:_?0)*|[1-9](?:_?[0-9])*)[lL]?\b",
+                r"\b((\.[0-9](?:_?[0-9])*')|\.[0-9](?:_?[0-9])*)"
+                "([eE][+-]?[0-9](?:_?[0-9])*)?[jJ]?\b",
+                r"\b[0-9](?:_?[0-9])*([eE][+-]?[0-9](?:_?[0-9])*)?[jJ]?\b",
+                r"\b[0-9](?:_?[0-9])*[jJ]\b"]
+    number = any("number", number_regex)
+
     sqstring =     r"(\b[rRuU])?'[^'\\\n]*(\\.[^'\\\n]*)*'?"
     dqstring =     r'(\b[rRuU])?"[^"\\\n]*(\\.[^"\\\n]*)*"?'
     uf_sqstring =  r"(\b[rRuU])?'[^'\\\n]*(\\.[^'\\\n]*)*(\\)$(?!')$"


Test patch: