Repository: spyder-ide/spyder

Issue:
Matching bracket highlighting index error
## Problem Description
When the final closing bracket (`)`, `]` or `}`) of a bracket pair is the first character in its line, any empty lines between the bracket pair are discarded during the search for the opening bracket leading to an indexing error of what character is being highlighted.

The error and a method of reproduction should be apparent from this image:
![spyder-highlight-bug](https://user-images.githubusercontent.com/17409078/101279560-56d6cc00-37c3-11eb-8dfe-ef09e9f3eb47.png)


## Versions
This happens in 4.2.0. See permalinks to code below for more details.

## Root cause
The bracket finder searches text that it gets from `BaseEditMixin.get_text` (lines 286 and 291)
https://github.com/spyder-ide/spyder/blob/436290ba71bb16c5928676467a358b70f4f38c10/spyder/plugins/editor/widgets/base.py#L282-L293

However, `BaseEditMixin.get_text` removes newlines under some conditions, resulting in the aberrant behaviour (lines 907-912)
https://github.com/spyder-ide/spyder/blob/436290ba71bb16c5928676467a358b70f4f38c10/spyder/widgets/mixins.py#L900-L913

## Possible fixes
I encountered this bug while working on #1354 and I plan to include a fix for this issue in the same pull request. But there are several alternatives for fixing it and I'd like some input on which is best.

1. Remove lines 907-912 from `BaseEditMixin.get_text`. I've tested it and it solves the problem, but it will cause other errors if the line removal behaviour is depended on elsewhere.
2. Make `all_text` an optional parameter with default value `None`. If `None`, it will be computed as previously.
3. Write a new function identical to `get_text` except without the line removal.

Ideally I would like to go with option 1, but that requires figuring out why the newlines are removed in the first place. If this behaviour is relied on elsewhere I think the newline removal should be moved there instead of kept in `get_text`. I don't think it's suitable to have a function called `get_text` that filters the text before returning it without having any parameters to control that behaviour.

Option 2 is safer and requires no investigation into side effects. So that's my second choice.

And while I'm here: How would I go about writing a regression test for a GUI feature like this? Where should I look for examples?


Discussion:

Resolving patch:
diff --git a/spyder/plugins/editor/widgets/base.py b/spyder/plugins/editor/widgets/base.py
--- a/spyder/plugins/editor/widgets/base.py
+++ b/spyder/plugins/editor/widgets/base.py
@@ -278,46 +278,117 @@ def unhighlight_current_cell(self):
         """Unhighlight current cell"""
         self.clear_extra_selections('current_cell')
 
-    #------Brace matching
-    def find_brace_match(self, position, brace, forward):
+    def in_comment(self, cursor=None, position=None):
+        """Returns True if the given position is inside a comment.
+
+        Trivial default implementation. To be overridden by subclass.
+        This function is used to define the default behaviour of
+        self.find_brace_match.
+        """
+        return False
+
+    def in_string(self, cursor=None, position=None):
+        """Returns True if the given position is inside a string.
+
+        Trivial default implementation. To be overridden by subclass.
+        This function is used to define the default behaviour of
+        self.find_brace_match.
+        """
+        return False
+
+    def find_brace_match(self, position, brace, forward,
+                         ignore_brace=None, stop=None):
+        """Returns position of matching brace.
+
+        Parameters
+        ----------
+        position : int
+            The position of the brace to be matched.
+        brace : {'[', ']', '(', ')', '{', '}'}
+            The brace character to be matched.
+            [ <-> ], ( <-> ), { <-> }
+        forward : boolean
+            Whether to search forwards or backwards for a match.
+        ignore_brace : callable taking int returning boolean, optional
+            Whether to ignore a brace (as function of position).
+        stop : callable taking int returning boolean, optional
+            Whether to stop the search early (as function of position).
+
+        If both *ignore_brace* and *stop* are None, then brace matching
+        is handled differently depending on whether *position* is
+        inside a string, comment or regular code. If in regular code,
+        then any braces inside strings and comments are ignored. If in a
+        string/comment, then only braces in the same string/comment are
+        considered potential matches. The functions self.in_comment and
+        self.in_string are used to determine string/comment/code status
+        of characters in this case.
+
+        If exactly one of *ignore_brace* and *stop* is None, then it is
+        replaced by a function returning False for every position. I.e.:
+            lambda pos: False
+
+        Returns
+        -------
+        The position of the matching brace. If no matching brace
+        exists, then None is returned.
+        """
+
+        if ignore_brace is None and stop is None:
+            if self.in_string(position=position):
+                # Only search inside the current string
+                def stop(pos):
+                    return not self.in_string(position=pos)
+            elif self.in_comment(position=position):
+                # Only search inside the current comment
+                def stop(pos):
+                    return not self.in_comment(position=pos)
+            else:
+                # Ignore braces inside strings and comments
+                def ignore_brace(pos):
+                    return (self.in_string(position=pos) or
+                            self.in_comment(position=pos))
+
+        # Deal with search range and direction
         start_pos, end_pos = self.BRACE_MATCHING_SCOPE
         if forward:
-            bracemap = {'(': ')', '[': ']', '{': '}'}
-            text = self.get_text(position, end_pos)
-            i_start_open = 1
-            i_start_close = 1
+            closing_brace = {'(': ')', '[': ']', '{': '}'}[brace]
+            text = self.get_text(position, end_pos, remove_newlines=False)
         else:
-            bracemap = {')': '(', ']': '[', '}': '{'}
-            text = self.get_text(start_pos, position)
-            i_start_open = len(text)-1
-            i_start_close = len(text)-1
-
+            # Handle backwards search with the same code as forwards
+            # by reversing the string to be searched.
+            closing_brace = {')': '(', ']': '[', '}': '{'}[brace]
+            text = self.get_text(start_pos, position+1, remove_newlines=False)
+            text = text[-1::-1]  # reverse
+
+        def ind2pos(index):
+            """Computes editor position from search index."""
+            return (position + index) if forward else (position - index)
+
+        # Search starts at the first position after the given one
+        # (which is assumed to contain a brace).
+        i_start_close = 1
+        i_start_open = 1
         while True:
-            if forward:
-                i_close = text.find(bracemap[brace], i_start_close)
-            else:
-                i_close = text.rfind(bracemap[brace], 0, i_start_close+1)
-            if i_close > -1:
-                if forward:
-                    i_start_close = i_close+1
+            i_close = text.find(closing_brace, i_start_close)
+            i_start_close = i_close+1  # next potential start
+            if i_close == -1:
+                return  # no matching brace exists
+            elif ignore_brace is None or not ignore_brace(ind2pos(i_close)):
+                while True:
                     i_open = text.find(brace, i_start_open, i_close)
-                else:
-                    i_start_close = i_close-1
-                    i_open = text.rfind(brace, i_close, i_start_open+1)
-                if i_open > -1:
-                    if forward:
-                        i_start_open = i_open+1
-                    else:
-                        i_start_open = i_open-1
-                else:
-                    # found matching brace
-                    if forward:
-                        return position+i_close
-                    else:
-                        return position-(len(text)-i_close)
-            else:
-                # no matching brace
-                return
+                    i_start_open = i_open+1  # next potential start
+                    if i_open == -1:
+                        # found matching brace, but should we have
+                        # stopped before this point?
+                        if stop is not None:
+                            # There's room for optimization here...
+                            for i in range(1, i_close+1):
+                                if stop(ind2pos(i)):
+                                    return
+                        return ind2pos(i_close)
+                    elif (ignore_brace is None or
+                          not ignore_brace(ind2pos(i_open))):
+                        break  # must find new closing brace
 
     def __highlight(self, positions, color=None, cancel=False):
         if cancel:
@@ -338,23 +409,32 @@ def __highlight(self, positions, color=None, cancel=False):
         self.update_extra_selections()
 
     def cursor_position_changed(self):
-        """Brace matching"""
+        """Handle brace matching."""
+        # Clear last brace highlight (if any)
         if self.bracepos is not None:
             self.__highlight(self.bracepos, cancel=True)
             self.bracepos = None
+
+        # Get the current cursor position, check if it is at a brace,
+        # and, if so, determine the direction in which to search for able
+        # matching brace.
         cursor = self.textCursor()
         if cursor.position() == 0:
             return
         cursor.movePosition(QTextCursor.PreviousCharacter,
                             QTextCursor.KeepAnchor)
         text = to_text_string(cursor.selectedText())
-        pos1 = cursor.position()
         if text in (')', ']', '}'):
-            pos2 = self.find_brace_match(pos1, text, forward=False)
+            forward = False
         elif text in ('(', '[', '{'):
-            pos2 = self.find_brace_match(pos1, text, forward=True)
+            forward = True
         else:
             return
+
+        pos1 = cursor.position()
+        pos2 = self.find_brace_match(pos1, text, forward=forward)
+
+        # Set a new brace highlight
         if pos2 is not None:
             self.bracepos = (pos1, pos2)
             self.__highlight(self.bracepos, color=self.matched_p_color)
diff --git a/spyder/plugins/editor/widgets/codeeditor.py b/spyder/plugins/editor/widgets/codeeditor.py
--- a/spyder/plugins/editor/widgets/codeeditor.py
+++ b/spyder/plugins/editor/widgets/codeeditor.py
@@ -3965,13 +3965,18 @@ def __get_current_color(self, cursor=None):
         else:
             return None
 
-    def in_comment_or_string(self, cursor=None):
-        """Is the cursor inside or next to a comment or string?"""
+    def in_comment_or_string(self, cursor=None, position=None):
+        """Is the cursor or position inside or next to a comment or string?
+
+        If *cursor* is None, *position* is used instead. If *position* is also
+        None, then the current cursor position is used.
+        """
         if self.highlighter:
             if cursor is None:
-                current_color = self.__get_current_color()
-            else:
-                current_color = self.__get_current_color(cursor=cursor)
+                cursor = self.textCursor()
+                if position:
+                    cursor.setPosition(position)
+            current_color = self.__get_current_color(cursor=cursor)
 
             comment_color = self.highlighter.get_color_name('comment')
             string_color = self.highlighter.get_color_name('string')
@@ -4019,6 +4024,18 @@ def __has_colon_not_in_brackets(self, text):
                 return True
         return False
 
+    def __has_unmatched_opening_bracket(self):
+        """
+        Checks if there are any unmatched opening brackets before the current
+        cursor position.
+        """
+        position = self.textCursor().position()
+        for brace in [']', ')', '}']:
+            match = self.find_brace_match(position, brace, forward=False)
+            if match is not None:
+                return True
+        return False
+
     def autoinsert_colons(self):
         """Decide if we want to autoinsert colons"""
         bracket_ext = self.editor_extensions.get(CloseBracketsExtension)
@@ -4036,6 +4053,8 @@ def autoinsert_colons(self):
             return False
         elif self.__has_colon_not_in_brackets(line_text):
             return False
+        elif self.__has_unmatched_opening_bracket():
+            return False
         else:
             return True
 
@@ -4046,25 +4065,53 @@ def next_char(self):
         next_char = to_text_string(cursor.selectedText())
         return next_char
 
-    def in_comment(self, cursor=None):
+    def in_comment(self, cursor=None, position=None):
+        """Returns True if the given position is inside a comment.
+
+        Parameters
+        ----------
+        cursor : QTextCursor, optional
+            The position to check.
+        position : int, optional
+            The position to check if *cursor* is None. This parameter
+            is ignored when *cursor* is not None.
+
+        If both *cursor* and *position* are none, then the position returned
+        by self.textCursor() is used instead.
+        """
         if self.highlighter:
+            if cursor is None:
+                cursor = self.textCursor()
+                if position is not None:
+                    cursor.setPosition(position)
             current_color = self.__get_current_color(cursor)
             comment_color = self.highlighter.get_color_name('comment')
-            if current_color == comment_color:
-                return True
-            else:
-                return False
+            return (current_color == comment_color)
         else:
             return False
 
-    def in_string(self, cursor=None):
+    def in_string(self, cursor=None, position=None):
+        """Returns True if the given position is inside a string.
+
+        Parameters
+        ----------
+        cursor : QTextCursor, optional
+            The position to check.
+        position : int, optional
+            The position to check if *cursor* is None. This parameter
+            is ignored when *cursor* is not None.
+
+        If both *cursor* and *position* are none, then the position returned
+        by self.textCursor() is used instead.
+        """
         if self.highlighter:
+            if cursor is None:
+                cursor = self.textCursor()
+                if position is not None:
+                    cursor.setPosition(position)
             current_color = self.__get_current_color(cursor)
             string_color = self.highlighter.get_color_name('string')
-            if current_color == string_color:
-                return True
-            else:
-                return False
+            return (current_color == string_color)
         else:
             return False
 
diff --git a/spyder/widgets/mixins.py b/spyder/widgets/mixins.py
--- a/spyder/widgets/mixins.py
+++ b/spyder/widgets/mixins.py
@@ -897,15 +897,25 @@ def get_text_region(self, start_line, end_line):
         start_cursor.setPosition(end_position, mode=QTextCursor.KeepAnchor)
         return self.get_selected_text(start_cursor)
 
-    def get_text(self, position_from, position_to):
-        """
-        Return text between *position_from* and *position_to*
-        Positions may be positions or 'sol', 'eol', 'sof', 'eof' or 'cursor'
+    def get_text(self, position_from, position_to, remove_newlines=True):
+        """Returns text between *position_from* and *position_to*.
+
+        Positions may be integers or 'sol', 'eol', 'sof', 'eof' or 'cursor'.
+
+        Unless position_from='sof' and position_to='eof' any trailing newlines
+        in the string are removed. This was added as a workaround for
+        spyder-ide/spyder#1546 and later caused spyder-ide/spyder#14374.
+        The behaviour can be overridden by setting the optional parameter
+        *remove_newlines* to False.
+
+        TODO: Evaluate if this is still a problem and if the workaround can
+              be moved closer to where the problem occurs.
         """
         cursor = self.__select_text(position_from, position_to)
         text = to_text_string(cursor.selectedText())
-        all_text = position_from == 'sof' and position_to == 'eof'
-        if text and not all_text:
+        if remove_newlines:
+            remove_newlines = position_from != 'sof' or position_to != 'eof'
+        if text and remove_newlines:
             while text.endswith("\n"):
                 text = text[:-1]
             while text.endswith(u"\u2029"):


Test patch:
diff --git a/spyder/plugins/editor/widgets/tests/assets/braces.py b/spyder/plugins/editor/widgets/tests/assets/braces.py
new file mode 100644
--- /dev/null
+++ b/spyder/plugins/editor/widgets/tests/assets/braces.py
@@ -0,0 +1,16 @@
+b = [ [x,x*2]
+     for x in [1,[2,3],[4,[5],6,[7,8,9]]]]
+
+a = [
+"""
+(((((()))))
+
+[[[[]]]]]
+""", 'x)', 'b('
+
+# ){[(
+# abcdef
+# )]} (((([[[{{{
+
+
+]
diff --git a/spyder/plugins/editor/widgets/tests/test_autocolon.py b/spyder/plugins/editor/widgets/tests/test_autocolon.py
--- a/spyder/plugins/editor/widgets/tests/test_autocolon.py
+++ b/spyder/plugins/editor/widgets/tests/test_autocolon.py
@@ -75,6 +75,15 @@ def test_no_auto_colon_in_listcomp_over_two_lines():
     editor = construct_editor("ns = [ n for ns in range(10) \n if n < 5 ]")
     assert editor.autoinsert_colons() == False
 
+def test_no_auto_colon_in_listcomp_over_three_lines():
+    """Tests spyder-ide/spyder#1354"""
+    editor = construct_editor("ns = [ n \n for ns in range(10) \n if n < 5 ]")
+    cursor = editor.textCursor()
+    cursor.movePosition(QTextCursor.Up)
+    cursor.movePosition(QTextCursor.EndOfLine)
+    editor.setTextCursor(cursor)
+    assert not editor.autoinsert_colons()
+
 
 # --- Failing tests
 # -----------------------------------------------------------------------------
@@ -83,15 +92,6 @@ def test_auto_colon_even_if_colon_inside_quotes():
     editor = construct_editor("if text == ':'")
     assert editor.autoinsert_colons() == True
 
-@pytest.mark.xfail
-def test_no_auto_colon_in_listcomp_over_three_lines():
-    editor = construct_editor("ns = [ n \n for ns in range(10) \n if n < 5 ]")
-    cursor = editor.textCursor()
-    cursor.movePosition(QTextCursor.Up)
-    cursor.movePosition(QTextCursor.EndOfLine)
-    editor.setTextCursor(cursor)
-    assert editor.autoinsert_colons() == False
-
 @pytest.mark.xfail
 def test_auto_colon_in_two_if_statements_on_one_line():
     editor = construct_editor("if x < 0: x = 0; if x == 0")
diff --git a/spyder/plugins/editor/widgets/tests/test_codeeditor.py b/spyder/plugins/editor/widgets/tests/test_codeeditor.py
--- a/spyder/plugins/editor/widgets/tests/test_codeeditor.py
+++ b/spyder/plugins/editor/widgets/tests/test_codeeditor.py
@@ -4,6 +4,9 @@
 # Licensed under the terms of the MIT License
 #
 
+# Standard library imports
+import os.path as osp
+
 # Third party imports
 from qtpy.QtCore import Qt, QEvent
 from qtpy.QtGui import QFont, QTextCursor, QMouseEvent
@@ -11,9 +14,12 @@
 import pytest
 
 # Local imports
+from spyder.utils.qthelpers import qapplication
 from spyder.plugins.editor.widgets.editor import codeeditor
 from spyder.py3compat import PY2, PY3
 
+HERE = osp.dirname(osp.abspath(__file__))
+ASSETS = osp.join(HERE, 'assets')
 
 # --- Fixtures
 # -----------------------------------------------------------------------------
@@ -264,5 +270,101 @@ def test_undo_return(editorbot):
     assert editor.toPlainText() == text
 
 
+def test_brace_match(editorbot):
+    """Tests for the highlighting of matching parenthesis, braces and brackets.
+
+    Specifically provides regression tests for issues
+     * spyder-ide/spyder#2965
+     * spyder-ide/spyder#9179
+     * spyder-ide/spyder#14374
+
+    If this test fails the best way to investigate is probably to open
+    assets/braces.py in Spyder, step through the file and visually
+    observe brace matching.
+
+    Some caveats for brace matching can be found in pull request
+    spyder-ide/spyder#14376
+
+    The functions being tested are essentially:
+     * TextEditBaseWidget.find_brace_match
+     * CodeEditor.in_comment
+     * CodeEditor.in_string
+    """
+    # Create editor with contents loaded from assets/brackets.py
+    qtbot, editor = editorbot
+    editor.setup_editor(language='Python')
+    with open(osp.join(ASSETS, 'braces.py'), 'r') as file:
+        editor.set_text(file.read())
+
+    # Each element of *positions* is a two element list:
+    #  [position, expected]
+    # Here *position* is the position at which to place the cursor and
+    # *expected* is what editor.bracepos should be at that location if
+    # the brace matching works correctly. Specifically if at *position* ...
+    # a) ... there is no brace, then *expected* should be None.
+    # b) ... there is an unmatched brace, then *expected* should be a
+    #        1-tuple containing position-1
+    # c) ... there is a matched brace then *expected* should be a
+    #        2-tuple with the first element being position-1 and the
+    #        second element being the position of the mathing brace.
+    # At the end of each row, a comment has been added that attempts to
+    # illustrate in what part of 'braces.py' the cursor is placed in
+    # that test case.
+    positions = [
+        [0, None],       # b
+        [5,  (4, 55)],   # b = [
+        [56, (55,  4)],  # ]
+        [7,  (6, 12)],   # [x
+        [13, (12,  6)],  # x*2]
+        [29, (28, 54)],  # [1
+        [55, (54, 28)],  # ]
+        [32, (31, 35)],  # [2
+        [36, (35, 31)],  # 3]
+        [38, (37, 53)],  # [4
+        [54, (53, 37)],  # ]
+        [41, (40, 42)],  # [5
+        [42, None],      # 5
+        [43, (42, 40)],  # 5]
+        [47, (46, 52)],  # [7
+        [53, (52, 46)],  # 8]
+        [63, (62, 143)],  # a = [
+        [144, (143, 62)],  # ]
+        [69, (68, )],    # """(
+        [70, (69, 78)],  # (
+        [71, (70, 77)],  # (
+        [72, (71, 76)],  # (
+        [73, (72, 75)],  # (
+        [74, (73, 74)],  # (
+        [75, (74, 73)],  # )
+        [76, (75, 72)],  # )
+        [77, (76, 71)],  # )
+        [78, (77, 70)],  # )
+        [79, (78, 69)],  # )
+        [82, (81, 88)],  # [
+        [83, (82, 87)],  # [
+        [84, (83, 86)],  # [
+        [85, (84, 85)],  # [
+        [86, (85, 84)],  # ]
+        [87, (86, 83)],  # ]
+        [88, (87, 82)],  # ]
+        [89, (88, 81)],  # ]
+        [90, (89, )],    # ]"""
+        [99, (98, )],    # 'x)'
+        [105, (104, )],  # 'b('
+        [111, (110, )],  # # )
+        [112, (111, 128)],  # {[
+        [129, (128, 111)],  # ]}
+        [113, (112, 127)],  # [(
+        [128, (127, 112)],  # )]
+        [114, (113, 126)],  # (
+        [127, (126, 113)],  # )
+    ]
+    cursor = editor.textCursor()
+    for position, expected in positions:
+        cursor.setPosition(position)
+        editor.setTextCursor(cursor)
+        assert editor.bracepos == expected
+
+
 if __name__ == '__main__':
     pytest.main(['test_codeeditor.py'])