Repository: spyder-ide/spyder

Issue:
Spyder comments (Ctrl+1 and Ctrl+4) generate pep8 warnings
_From [bentkows...@gmail.com](https://code.google.com/u/114523486630528179401/) on 2014-04-07T06:47:08Z_

Spyder Version:  2.2.5
Python Version:  2.7.5+
Qt Version    :  4.8.4, PyQt4 (API v2) 4.10.3 on Linux
IPython >=0.13  :  0.13.2 (OK)
matplotlib >=1.0:  1.2.1 (OK)
pep8 >=0.6      :  1.5.4 (OK)
pyflakes >=0.5.0:  0.8.1 (OK)
pylint >=0.25   :  1.1.0 (OK)
rope >=0.9.2    :  0.9.4 (OK)
sphinx >=0.6.6  :  1.2.2 (OK)
sympy >=0.7.0   :  0.7.3 (OK) 

* What steps will reproduce the problem?

  1. I updated Spyder2 and the whole bundle around it (rope, pylint, sphinx etc.) with PIP -U [...]
  2. It seems there as some update to PEP8 guidelines regarding commenting with #  (PEP8 E265).
  3. Say, I have commented a line:

         #somefunc(arg1, arg2)

     I get a warning "E265 block comment should start with '# '". Adding a space:

         # somefunc(arg1, arg2)

     removes the warning. Ctr+4 adds a line:

         #===================================

     at the beginning and end of the block commented out. This brings a warning as well.

* What is the expected output? What do you see instead?

  No false positive error message. 

* Please provide any additional information below

  Probably it's not a problem for people using software deposited in repos, but with time it will come up.

_Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1785_



Discussion:
_From [contrebasse](https://code.google.com/u/contrebasse/) on 2014-04-11T06:25:16Z_

The warning is correct, the problem lies with spyder's autocomment, it should add a space after the #.

The problem is to know if the space should be removed when removing the comment.

_From [bentkows...@gmail.com](https://code.google.com/u/114523486630528179401/) on 2014-04-11T06:42:27Z_

A workaround would be adding not just the '#' sign but a 'special Spyder comment sequence', e.g. "# %" ("hash-space-proc"). But on the other hand that's looks just dirty :-(

Spyder can warn about a gap being not a multiplication of 4. That could be used to count the gap size and auto adjust it, if the removed hash was the first character in the line.

 P.

_From [contrebasse](https://code.google.com/u/contrebasse/) on 2014-04-11T14:25:22Z_

Having a number of space multiple of 4 (or 2 since it is an configuration option) would not work for continuation lines.

The issue can be fixed for Ctrl+4, however.

_From [ccordoba12](https://code.google.com/u/ccordoba12/) on 2014-04-16T09:13:50Z_

Joseph, if you can help us to solve this one it'd be great.

**Status:** HelpNeeded  
  **Labels:** Cat-Editor  

_From [ccordoba12](https://code.google.com/u/ccordoba12/) on 2015-01-01T15:21:16Z_

**Status:** Accepted  
  **Labels:** MS-v2.3.3  

_From [ccordoba12](https://code.google.com/u/ccordoba12/) on 2015-01-08T15:16:16Z_

**Labels:** -MS-v2.3.3 MS-v2.4  

Why can't we just make Ctrl-1 add and remove '# ' (with a space)? If there is no space, just remove '#', but always add '# '. I think that would work well with most code that follows PEP8 conventions. 

Ctrl-1 also leads to problems when you are commenting out indented blocks:

``` python
for i in range(10):
#    print(i)  # this is how the line looks when commented out using Ctrl-1
    # print(i)  # this is how the line should look if following PEP8
    print(i)  # so that the indent makes sense
```

Using Ctrl-1 above gives a PEP8 error: `E115 expected an indented block (comment)`.

---

So if Ctrl-1 worked by adding / removing '# ' before the first non-whitespace character in the line, that would fix both problems?

The problem is not adding `#`, but to know how to remove it without breaking the code. It should work even if the user added a comment manually at any place.

Imagine that the user manually added `#`(with no additional space) at the beginning of an indented line. Removing `#` at the beginning of the line with would be wrong because print(2) would be indented by 3 spaces instead of 4:

``` python
if True:
    print(1)
#    print(2)
```

We could check the surrounding block and try to guess the correct behavior, but I'm not sure we could do it without doing it wrong in some tricky corner cases.
But we are open to suggestions !

But according to PEP8, block comments should be indented [at the same level as the code](https://www.python.org/dev/peps/pep-0008/#block-comments). So the only time this would create problems is when people are not following PEP8 conventions to begin with. Maybe adding "Use '# ' for comments" as an option somewhere in configurations would work?

For the problem that you point out above, it would be useful if `Shift - Tab` worked even when lines are not multiples of four (it works this way in Geany for example). That way you could fix indentation with `Shift - Tab`, `Tab`.

Spyder shouldn't break code even if people are not following pep8.

I agree about Shift-Tab, it could be improved, but I don't think it's a solution for the uncommenting problem.

Do you know what other python editors do in this case ?

Just tested this with PyCharm. 

For the most part, it indents and unindents at the level of the code using '# ' (with a space) for indentation:

``` python
if True:
    print(1)
    # print(2)
```

If you select code that spans several different levels of indentation, it adds '# ' at the first level:

``` python
if True:
    print(1)
    # if True:
    #     print(2)
```

In both cases, it unindents by removing '# '.

For the case you pointed out above, it gives you a PEP8 warning, and unindenting still works by removing '# ', resulting in:

``` python
if True:
    print(1)
  print(2)
```

Thanks for testing.
The last case is strange, I'd expect to get 3 spaces before `print(2)` instead of 2, is it really like this in pycharm or just a typo when copying the result ?

@spyder-ide/core-developers Any idea ?

It's really like this. I think it always removes '# ' (with a space), if there is a space.

I actually think that that is the easiest to implement, cause we dont check for corner cases at all.. simply `#` is removed and inserted always. 

@Nodd I would prefer doing something like that cause is predictable, if we start playing with how to understand other cases we will end with weird code...


Resolving patch:
diff --git a/spyder/widgets/sourcecode/codeeditor.py b/spyder/widgets/sourcecode/codeeditor.py
--- a/spyder/widgets/sourcecode/codeeditor.py
+++ b/spyder/widgets/sourcecode/codeeditor.py
@@ -1509,7 +1509,6 @@ def add_prefix(self, prefix):
             first_pos = min([start_pos, end_pos])
             first_cursor = self.textCursor()
             first_cursor.setPosition(first_pos)
-            begins_at_block_start = first_cursor.atBlockStart()
 
             cursor.beginEditBlock()
             cursor.setPosition(end_pos)
@@ -1519,35 +1518,75 @@ def add_prefix(self, prefix):
                 cursor.movePosition(QTextCursor.PreviousBlock)
                 if cursor.position() < start_pos:
                     cursor.setPosition(start_pos)
+            move_number = self.__spaces_for_prefix()
 
             while cursor.position() >= start_pos:
                 cursor.movePosition(QTextCursor.StartOfBlock)
-                cursor.insertText(prefix)
+                line_text = to_text_string(cursor.block().text())
+                if (self.get_character(cursor.position()) == ' '
+                        and '#' in prefix and not line_text.isspace()
+                        or (not line_text.startswith(' ')
+                            and line_text != '')):
+                    cursor.movePosition(QTextCursor.Right,
+                                        QTextCursor.MoveAnchor,
+                                        move_number)
+                    cursor.insertText(prefix)
                 if start_pos == 0 and cursor.blockNumber() == 0:
                     # Avoid infinite loop when indenting the very first line
                     break
                 cursor.movePosition(QTextCursor.PreviousBlock)
                 cursor.movePosition(QTextCursor.EndOfBlock)
             cursor.endEditBlock()
-            if begins_at_block_start:
-                # Extending selection to prefix:
-                cursor = self.textCursor()
-                start_pos = cursor.selectionStart()
-                end_pos = cursor.selectionEnd()
-                if start_pos < end_pos:
-                    start_pos -= len(prefix)
-                else:
-                    end_pos -= len(prefix)
-                cursor.setPosition(start_pos, QTextCursor.MoveAnchor)
-                cursor.setPosition(end_pos, QTextCursor.KeepAnchor)
-                self.setTextCursor(cursor)
         else:
             # Add prefix to current line
             cursor.beginEditBlock()
             cursor.movePosition(QTextCursor.StartOfBlock)
+            if self.get_character(cursor.position()) == ' ' and '#' in prefix:
+                cursor.movePosition(QTextCursor.NextWord)
             cursor.insertText(prefix)
             cursor.endEditBlock()
 
+    def __spaces_for_prefix(self):
+        """Find the less indented level of text."""
+        cursor = self.textCursor()
+        if self.has_selected_text():
+            # Add prefix to selected line(s)
+            start_pos, end_pos = cursor.selectionStart(), cursor.selectionEnd()
+
+            # Let's see if selection begins at a block start
+            first_pos = min([start_pos, end_pos])
+            first_cursor = self.textCursor()
+            first_cursor.setPosition(first_pos)
+
+            cursor.beginEditBlock()
+            cursor.setPosition(end_pos)
+            # Check if end_pos is at the start of a block: if so, starting
+            # changes from the previous block
+            if cursor.atBlockStart():
+                cursor.movePosition(QTextCursor.PreviousBlock)
+                if cursor.position() < start_pos:
+                    cursor.setPosition(start_pos)
+
+            number_spaces = -1
+            while cursor.position() >= start_pos:
+                cursor.movePosition(QTextCursor.StartOfBlock)
+                line_text = to_text_string(cursor.block().text())
+                start_with_space = line_text.startswith(' ')
+                left_number_spaces = self.__number_of_spaces(line_text)
+                if not start_with_space:
+                    left_number_spaces = 0
+                if ((number_spaces == -1
+                        or number_spaces > left_number_spaces)
+                        and not line_text.isspace() and line_text != ''):
+                    number_spaces = left_number_spaces
+                if start_pos == 0 and cursor.blockNumber() == 0:
+                    # Avoid infinite loop when indenting the very first line
+                    break
+                cursor.movePosition(QTextCursor.PreviousBlock)
+                cursor.movePosition(QTextCursor.EndOfBlock)
+            cursor.endEditBlock()
+        return number_spaces
+
     def __is_cursor_at_start_of_block(self, cursor):
         cursor.movePosition(QTextCursor.StartOfBlock)
 
@@ -1590,28 +1629,68 @@ def remove_prefix(self, prefix):
                 else:
                     old_pos = new_pos
                 line_text = to_text_string(cursor.block().text())
-                if (prefix.strip() and line_text.lstrip().startswith(prefix)
-                    or line_text.startswith(prefix)):
-                    cursor.movePosition(QTextCursor.Right,
-                                        QTextCursor.MoveAnchor,
-                                        line_text.find(prefix))
-                    cursor.movePosition(QTextCursor.Right,
-                                        QTextCursor.KeepAnchor, len(prefix))
-                    cursor.removeSelectedText()
+                self.__remove_prefix(prefix, cursor, line_text)
                 cursor.movePosition(QTextCursor.PreviousBlock)
             cursor.endEditBlock()
         else:
             # Remove prefix from current line
             cursor.movePosition(QTextCursor.StartOfBlock)
             line_text = to_text_string(cursor.block().text())
-            if (prefix.strip() and line_text.lstrip().startswith(prefix)
-                or line_text.startswith(prefix)):
+            self.__remove_prefix(prefix, cursor, line_text)
+
+    def __remove_prefix(self, prefix, cursor, line_text):
+        """Handle the removal of the prefix for a single line."""
+        start_with_space = line_text.startswith(' ')
+        if start_with_space:
+            left_spaces = self.__even_number_of_spaces(line_text)
+        else:
+            left_spaces = False
+        if start_with_space:
+            right_number_spaces = self.__number_of_spaces(line_text, group=1)
+        else:
+            right_number_spaces = self.__number_of_spaces(line_text)
+        # Handle prefix remove for comments with spaces
+        if (prefix.strip() and line_text.lstrip().startswith(prefix + ' ')
+                or line_text.startswith(prefix + ' ') and '#' in prefix):
+            cursor.movePosition(QTextCursor.Right,
+                                QTextCursor.MoveAnchor,
+                                line_text.find(prefix))
+            if (right_number_spaces == 1
+                    and (left_spaces or not start_with_space)
+                    or (not start_with_space and right_number_spaces % 2 != 0)
+                    or (left_spaces and right_number_spaces % 2 != 0)):
+                # Handle inserted '# ' with the count of the number of spaces
+                # at the right and left of the prefix.
                 cursor.movePosition(QTextCursor.Right,
-                                    QTextCursor.MoveAnchor,
-                                    line_text.find(prefix))
+                                    QTextCursor.KeepAnchor, len(prefix + ' '))
+            else:
+                # Handle manual insertion of '#'
                 cursor.movePosition(QTextCursor.Right,
                                     QTextCursor.KeepAnchor, len(prefix))
-                cursor.removeSelectedText()
+            cursor.removeSelectedText()
+        # Check for prefix without space
+        elif (prefix.strip() and line_text.lstrip().startswith(prefix)
+                or line_text.startswith(prefix)):
+            cursor.movePosition(QTextCursor.Right,
+                                QTextCursor.MoveAnchor,
+                                line_text.find(prefix))
+            cursor.movePosition(QTextCursor.Right,
+                                QTextCursor.KeepAnchor, len(prefix))
+            cursor.removeSelectedText()
+
+    def __even_number_of_spaces(self, line_text, group=0):
+        """
+        Get if there is a correct indentation from a group of spaces of a line.
+        """
+        spaces = re.findall('\s+', line_text)
+        if len(spaces) - 1 >= group:
+            return len(spaces[group]) % 4 == 0
+
+    def __number_of_spaces(self, line_text, group=0):
+        """Get the number of spaces from a group of spaces in a line."""
+        spaces = re.findall('\s+', line_text)
+        if len(spaces) - 1 >= group:
+            return len(spaces[group])
 
 
     def fix_indent(self, *args, **kwargs):
@@ -1960,14 +2039,24 @@ def is_comment(self, block):
 
     def comment(self):
         """Comment current line or selection."""
-        self.add_prefix(self.comment_string)
+        self.add_prefix(self.comment_string + ' ')
 
     def uncomment(self):
         """Uncomment current line or selection."""
-        self.remove_prefix(self.comment_string)
-
-    def __blockcomment_bar(self):
-        return self.comment_string + ' ' + '=' * (78 - len(self.comment_string))
+        blockcomment = self.unblockcomment()
+        if not blockcomment:
+            self.remove_prefix(self.comment_string)
+
+    def __blockcomment_bar(self, compatibility=False):
+        """Handle versions of blockcomment bar for backwards compatibility."""
+        # Blockcomment bar in Spyder version >= 4
+        blockcomment_bar = self.comment_string + ' ' + '=' * (
+                                    79 - len(self.comment_string + ' '))
+        if compatibility:
+            # Blockcomment bar in Spyder version < 4
+            blockcomment_bar = self.comment_string + '=' * (
+                                    79 - len(self.comment_string))
+        return blockcomment_bar
 
     def transform_to_uppercase(self):
         """Change to uppercase current line or selection."""
@@ -2031,10 +2120,21 @@ def blockcomment(self):
         cursor.endEditBlock()
 
     def unblockcomment(self):
-        """Un-block comment current line or selection"""
+        """Un-block comment current line or selection."""
+        # Needed for backward compatibility with Spyder previous blockcomments.
+        # See issue 2845
+        unblockcomment = self.__unblockcomment()
+        if not unblockcomment:
+            unblockcomment =  self.__unblockcomment(compatibility=True)
+        else:
+            return unblockcomment
+
+    def __unblockcomment(self, compatibility=False):
+        """Un-block comment current line or selection helper."""
         def __is_comment_bar(cursor):
             return to_text_string(cursor.block().text()
-                           ).startswith(self.__blockcomment_bar())
+                           ).startswith(
+                         self.__blockcomment_bar(compatibility=compatibility))
         # Finding first comment bar
         cursor1 = self.textCursor()
         if __is_comment_bar(cursor1):
@@ -2044,7 +2144,7 @@ def __is_comment_bar(cursor):
             if cursor1.atStart():
                 break
         if not __is_comment_bar(cursor1):
-            return
+            return False
         def __in_block_comment(cursor):
             cs = self.comment_string
             return to_text_string(cursor.block().text()).startswith(cs)
@@ -2056,7 +2156,7 @@ def __in_block_comment(cursor):
             if cursor2.block() == self.document().lastBlock():
                 break
         if not __is_comment_bar(cursor2):
-            return
+            return False
         # Removing block comment
         cursor3 = self.textCursor()
         cursor3.beginEditBlock()
@@ -2068,8 +2168,9 @@ def __in_block_comment(cursor):
             if not cursor3.atBlockEnd():
                 # standard commenting inserts '# ' but a trailing space on an
                 # empty line might be stripped.
-                cursor3.movePosition(QTextCursor.NextCharacter,
-                                     QTextCursor.KeepAnchor)
+                if not compatibility:
+                    cursor3.movePosition(QTextCursor.NextCharacter,
+                                         QTextCursor.KeepAnchor)
             cursor3.removeSelectedText()
             cursor3.movePosition(QTextCursor.NextBlock)
         for cursor in (cursor2, cursor1):
@@ -2077,6 +2178,7 @@ def __in_block_comment(cursor):
             cursor3.select(QTextCursor.BlockUnderCursor)
             cursor3.removeSelectedText()
         cursor3.endEditBlock()
+        return True
 
     #------Kill ring handlers
     # Taken from Jupyter's QtConsole


Test patch:
diff --git a/spyder/widgets/sourcecode/tests/test_comments.py b/spyder/widgets/sourcecode/tests/test_comments.py
new file mode 100644
--- /dev/null
+++ b/spyder/widgets/sourcecode/tests/test_comments.py
@@ -0,0 +1,161 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright © Spyder Project Contributors
+# Licensed under the terms of the MIT License
+#
+
+"""
+Tests for the comment features
+"""
+
+# Third party imports
+import pytest
+from qtpy.QtGui import QTextCursor
+
+# Local imports
+from spyder.py3compat import to_text_string
+from spyder.widgets.sourcecode.codeeditor import CodeEditor
+
+
+# --- Helper methods
+# -----------------------------------------------------------------------------
+def toggle_comment(editor, single_line=True, start_line=1):
+    """Toggle comment and return code."""
+    editor.go_to_line(start_line)
+    if single_line:
+        editor.toggle_comment()
+    else:
+        editor.moveCursor(QTextCursor.End, mode=QTextCursor.KeepAnchor)
+        editor.toggle_comment()
+    text = editor.toPlainText()
+    return to_text_string(text)
+
+
+# --- Fixtures
+# -----------------------------------------------------------------------------
+@pytest.fixture
+def code_editor_bot(qtbot):
+    """
+    Setup CodeEditor with some text useful for folding related tests.
+    """
+    editor = CodeEditor(parent=None)
+    indent_chars = " " * 4
+    tab_stop_width_spaces = 4
+    language = "Python"
+    editor.setup_editor(language=language, indent_chars=indent_chars,
+                        tab_stop_width_spaces=tab_stop_width_spaces)
+
+    return editor, qtbot
+
+
+# --- Tests
+# -----------------------------------------------------------------------------
+def test_single_line_comment(code_editor_bot):
+    """Test toggle comment in a single line."""
+    editor, qtbot = code_editor_bot
+    text = ("#class a():\n"
+            "#    self.b = 1\n"
+            " #   print(self.b)\n"
+            "#    \n"
+            )
+    editor.set_text(text)
+    # Toggle comment without spaces from the prefix and manually inserted
+    text = toggle_comment(editor)
+    assert text == ("class a():\n"
+                    "#    self.b = 1\n"
+                    " #   print(self.b)\n"
+                    "#    \n"
+                    )
+    # Toggle comment with space insertion
+    text = toggle_comment(editor)
+    assert text == ("# class a():\n"
+                    "#    self.b = 1\n"
+                    " #   print(self.b)\n"
+                    "#    \n"
+                    )
+    # Toggle comment deleting the insert space
+    text = toggle_comment(editor)
+    assert text == ("class a():\n"
+                    "#    self.b = 1\n"
+                    " #   print(self.b)\n"
+                    "#    \n"
+                    )
+    # Toggle comment with space at the right of prefix but manually inserted
+    text = toggle_comment(editor, start_line=2)
+    assert text == ("class a():\n"
+                    "    self.b = 1\n"
+                    " #   print(self.b)\n"
+                    "#    \n"
+                    )
+    # Toggle comment with space insertion
+    text = toggle_comment(editor, start_line=2)
+    assert text == ("class a():\n"
+                    "    # self.b = 1\n"
+                    " #   print(self.b)\n"
+                    "#    \n"
+                    )
+    # Toggle comment deleting inserted space
+    text = toggle_comment(editor, start_line=2)
+    assert text == ("class a():\n"
+                    "    self.b = 1\n"
+                    " #   print(self.b)\n"
+                    "#    \n"
+                    )
+    # Toggle comment with space at the right and left of prefix
+    # but manually inserted
+    text = toggle_comment(editor, start_line=3)
+    assert text == ("class a():\n"
+                    "    self.b = 1\n"
+                    "    print(self.b)\n"
+                    "#    \n"
+                    )
+
+
+def test_selection_comment(code_editor_bot):
+    """Test toggle comments with selection of more tha one line."""
+    editor, qtbot = code_editor_bot
+    text = ("#class a():\n"
+            "#    self.b = 1\n"
+            " #   print(self.b)\n"
+            "#    \n"
+            )
+    editor.set_text(text)
+    # Toggle manually commented code
+    text = toggle_comment(editor, single_line=False)
+    assert text == ("class a():\n"
+                    "    self.b = 1\n"
+                    "    print(self.b)\n"
+                    "    \n"
+                    )
+    # Toggle comment inserting prefix and space
+    text = toggle_comment(editor, single_line=False)
+    assert text == ("# class a():\n"
+                    "#     self.b = 1\n"
+                    "#     print(self.b)\n"
+                    "    \n"
+                    )
+    # Toggle comment deleting inserted prefix and space
+    text = toggle_comment(editor, single_line=False)
+    assert text == ("class a():\n"
+                    "    self.b = 1\n"
+                    "    print(self.b)\n"
+                    "    \n"
+                    )
+    # Test compatibility with Spyder 3 commenting structure
+    text = ("#class a():\n"
+            "#    self.b = 1\n"
+            "#    print(self.b)\n"
+            "#    \n"
+            )
+    editor.set_text(text)
+    # Toggle comment deleting inserted prefix (without space)
+    text = toggle_comment(editor, single_line=False)
+    assert text == ("class a():\n"
+                    "    self.b = 1\n"
+                    "    print(self.b)\n"
+                    "    \n"
+                    )
+
+
+if __name__ == "__main__":
+    pytest.main()