Repository: spyder-ide/spyder

Issue:
Grouping cells in Outline without code indentation
## Problem Description

Spyders Outline works well most of the time with cells and grouping by indentation to represent the code. Sometimes, manual setting would be advantageous (e.g. group functions desfined at top). See example..


### What steps reproduce the problem?

Add the following to spyder. See Outline. 

```
#%%a simple cell with some code
def withAFunction(x):
    return x
y = withAFunction(x)

#%%another cell with a function which gets grouped, but produces indentation error
    def withAnotherFunction(x):
        return x
    y = withAFunction(x)
```


### What is the expected output? What do you see instead?
First one expected, second would be good sometimes I think. However, getting second forces a wrong ident in code.

### Paste Traceback/Error Below (if applicable)
No traceback

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->
Python 3.6.3 64bits, Qt 5.9.4, PyQt5 5.9.2 on Windows 
* Spyder version:  3.2.8
* Python version: 3.6.3
* Qt version: 5.9.4
* PyQt version: 5.9.2
* Operating System name/version: Win10 x64


Discussion:
So you want to define cells with indented code? Sorry, but I don't follow exactly what you mean.
@ccordoba12 I believe what the OP is suggesting is a way to get functions (etc.) grouped into collapsible sections by cell, like Cell 2 in the screenshot, without having to manually indent the code. Perhaps an option to that effect ("Group by cell") in the drop down outline menu to turn on/off the behavior would be the best way to accomplish that, assuming someone had time to implement it. @max3-2 , if you're interested and @ccordoba12 agrees, perhaps that someone could be you. Otherwise, due to our current funding situation, it might not get done anytime soon, unfortunately.

Screenshot of the resulting behavior, for reference.

![image](https://user-images.githubusercontent.com/17051931/39796922-0c0567b2-531f-11e8-850c-98c2fdb59d3b.png)

> I believe what the OP is suggesting is a way to get functions (etc.) grouped into collapsible sections by cell

Ok, but it doesn't seem easy to implement.
Yeah that was the idea. If you have a large function library in front of your code or in an ext. file this would help grouping and organizing. I'll leave this open and check if I find the time to take a closer look myself. Havent worked / contributed to spyder yet so its starting from 0...
Thanks @max3-2 , we'd be glad to look at your contribution! Check out our [Contributing Guide](https://github.com/spyder-ide/spyder/blob/master/CONTRIBUTING.md) to get started, and let us know if you'd like pointers.
@max3-2 This would help organize the outline.

As for syntax, you shouldn't need to play with the code in the file at all, so the grouping needs to be done in the block comment.

I would suggest this because it's sort of like markdown header hierarchy: 

```python
# %% Top level cell
def withAFunction(x):
    return x
y = withAFunction(x)

# %%% A grouped cell
def withAnotherFunction(x):
    return x
y = withAFunction(x)
```
Top level cells have 2 `%` and grouped cell have 3 `%`. You could even go down another level to 4 `%`

Grouped cells also open up the possibility to run the whole group or just the one cell.  That would get quite complicated.
> I would suggest this because it's sort of like markdown header hierarchy

Really nice suggestion! Would you like to implement it @bcolsen?
I don't want to take it away from @max3-2, but I'll have a look at after the sympy console stuff

Resolving patch:
diff --git a/spyder/utils/syntaxhighlighters.py b/spyder/utils/syntaxhighlighters.py
--- a/spyder/utils/syntaxhighlighters.py
+++ b/spyder/utils/syntaxhighlighters.py
@@ -478,6 +478,14 @@ def highlight_block(self, text):
                                 self.found_cell_separators = True
                                 oedata = OutlineExplorerData()
                                 oedata.text = to_text_string(text).strip()
+                                # cell_head: string contaning the first group
+                                # of '%'s in the cell header
+                                cell_head = re.search(r"%+|$",
+                                                      text.lstrip()).group()
+                                if cell_head == '':
+                                    oedata.cell_level = 0
+                                else:
+                                    oedata.cell_level = len(cell_head) - 2
                                 oedata.fold_level = start
                                 oedata.def_type = OutlineExplorerData.CELL
                                 oedata.def_name = text.strip()
diff --git a/spyder/widgets/editortools.py b/spyder/widgets/editortools.py
--- a/spyder/widgets/editortools.py
+++ b/spyder/widgets/editortools.py
@@ -182,10 +182,11 @@ def remove_from_tree_cache(tree_cache, line=None, item=None):
 
 class OutlineExplorerTreeWidget(OneColumnTree):
     def __init__(self, parent, show_fullpath=False,
-                 show_all_files=True, show_comments=True):
+                 show_all_files=True, show_comments=True, group_cells=True):
         self.show_fullpath = show_fullpath
         self.show_all_files = show_all_files
         self.show_comments = show_comments
+        self.group_cells = group_cells
         OneColumnTree.__init__(self, parent)
         self.freeze = False # Freezing widget to avoid any unwanted update
         self.editor_items = {}
@@ -211,7 +212,11 @@ def get_actions_from_items(self, items):
         comment_act = create_action(self, text=_('Show special comments'),
                         toggled=self.toggle_show_comments)
         comment_act.setChecked(self.show_comments)
-        actions = [fullpath_act, allfiles_act, comment_act, fromcursor_act]
+        group_cells_act = create_action(self, text=_('Group code cells'),
+                                        toggled=self.toggle_group_cells)
+        group_cells_act.setChecked(self.group_cells)
+        actions = [fullpath_act, allfiles_act, group_cells_act, comment_act,
+                   fromcursor_act]
         return actions
 
     @Slot(bool)
@@ -242,6 +247,11 @@ def toggle_show_comments(self, state):
         self.show_comments = state
         self.update_all()
 
+    @Slot(bool)
+    def toggle_group_cells(self, state):
+        self.group_cells = state
+        self.update_all()
+
     @Slot()
     def go_to_cursor_position(self):
         if self.current_editor is not None:
@@ -339,9 +349,12 @@ def populate_branch(self, editor, root_item, tree_cache=None):
                     remove_from_tree_cache(tree_cache, line=_l)
                     
         ancestors = [(root_item, 0)]
+        cell_ancestors = [(root_item, 0)]
         previous_item = None
         previous_level = None
-        
+        prev_cell_level = None
+        prev_cell_item = None
+
         oe_data = editor.highlighter.get_outlineexplorer_data()
         editor.has_cell_separators = oe_data.get('found_cell_separators', False)
         for block_nb in range(editor.get_line_count()):
@@ -373,6 +386,48 @@ def populate_branch(self, editor, root_item, tree_cache=None):
                     remove_from_tree_cache(tree_cache, line=line_nb)
                 continue
 
+            if citem is not None:
+                cname = to_text_string(citem.text(0))
+                cparent = citem.parent
+
+            # Blocks for Cell Groups.
+            if (data is not None and data.def_type == data.CELL and
+                    self.group_cells):
+                preceding = (root_item if previous_item is None
+                             else previous_item)
+                cell_level = data.cell_level
+                if prev_cell_level is not None:
+                    if cell_level == prev_cell_level:
+                        pass
+                    elif cell_level > prev_cell_level:
+                        cell_ancestors.append((prev_cell_item,
+                                               prev_cell_level))
+                    else:
+                        while (len(cell_ancestors) > 1 and
+                               cell_level <= prev_cell_level):
+                            cell_ancestors.pop(-1)
+                            _item, prev_cell_level = cell_ancestors[-1]
+                parent, _level = cell_ancestors[-1]
+                if citem is not None:
+                    if data.text == cname and level == clevel:
+                        previous_level = clevel
+                        previous_item = citem
+                        continue
+                    else:
+                        remove_from_tree_cache(tree_cache, line=line_nb)
+                item = CellItem(data.text, line_nb, parent, preceding)
+                item.setup()
+                debug = "%s -- %s/%s" % (str(item.line).rjust(6),
+                                         to_text_string(item.parent().text(0)),
+                                         to_text_string(item.text(0)))
+                tree_cache[line_nb] = (item, level, debug)
+                ancestors = [(item, 0)]
+                prev_cell_level = cell_level
+                prev_cell_item = item
+                previous_item = item
+                continue
+
+            # Blocks for Code Groups.
             if previous_level is not None:
                 if level == previous_level:
                     pass
@@ -385,10 +440,7 @@ def populate_branch(self, editor, root_item, tree_cache=None):
                         ancestors.pop(-1)
                         _item, previous_level = ancestors[-1]
             parent, _level = ancestors[-1]
-            
-            if citem is not None:
-                cname = to_text_string(citem.text(0))
-                
+
             preceding = root_item if previous_item is None else previous_item
             if not_class_nor_function and data.is_comment():
                 if not self.show_comments:
@@ -408,7 +460,8 @@ def populate_branch(self, editor, root_item, tree_cache=None):
                     item = CommentItem(data.text, line_nb, parent, preceding)
             elif class_name is not None:
                 if citem is not None:
-                    if class_name == cname and level == clevel:
+                    if (class_name == cname and level == clevel and
+                            parent is cparent):
                         previous_level = clevel
                         previous_item = citem
                         continue
@@ -417,7 +470,8 @@ def populate_branch(self, editor, root_item, tree_cache=None):
                 item = ClassItem(class_name, line_nb, parent, preceding)
             else:
                 if citem is not None:
-                    if func_name == cname and level == clevel:
+                    if (func_name == cname and level == clevel and
+                            parent is cparent):
                         previous_level = clevel
                         previous_item = citem
                         continue


Test patch:
diff --git a/spyder/widgets/tests/test_editortools.py b/spyder/widgets/tests/test_editortools.py
--- a/spyder/widgets/tests/test_editortools.py
+++ b/spyder/widgets/tests/test_editortools.py
@@ -7,6 +7,8 @@
 """
 Tests for editortool.py
 """
+# Standard Libray Imports
+from textwrap import dedent
 
 # Third party imports
 import pytest
@@ -58,6 +60,71 @@
         "        except AssertionError:\n"
         "            pass")
 
+code = """# -*- coding: utf-8 -*-
+
+    def function0(x):
+        return x
+
+    # %% Top level 1
+    def function1(x):
+        return x
+
+    x = function1(x)
+
+    # %%% Cell Level 1-1
+    q = 3
+    w = 'word'
+
+    # %%% Cell Level 1-2
+    def function2(x):
+        def inside(x):
+            return x
+        return x
+
+    y = function2(x)
+
+    # %%%% Cell Level 2
+    class Class2(x):
+        def __init__(x):
+            return x
+        def medthod1(x):
+            if x:
+                return x
+
+    # %%%%%% Cell level 4
+    def function4(x):
+        return x
+
+    # %%%%% Cell Level 3
+    def function5(x):
+        return x
+
+    # %%%%%%%% Cell Level 6
+    class Class3(x):
+        def __init__(x):
+            return x
+        def medthod1(x):
+            if x:
+                return x
+
+    # %% Top level 2
+    class Class4(x):
+        def __init__(x):
+            return x
+        def medthod1(x):
+            if x:
+                return x
+
+    z = Class1(x)
+
+    if __name__ == "__main__":
+
+    # %% MGroup3
+        def function6(x):
+            return x
+    # %%% MGroup4
+        x = 'test'
+"""
 
 # Qt Test Fixtures
 # --------------------------------
@@ -119,6 +186,129 @@ def test_outline_explorer(outline_explorer_bot):
             assert item.is_method() == expected_result[2]
 
 
+# Qt Test Fixtures
+# --------------------------------
+@pytest.fixture
+def outline_explorer_bot2(qtbot):
+    editor = CodeEditor()
+    editor = CodeEditor(None)
+    editor.set_language('py', 'test_file.py')
+    editor.set_text(dedent(code))
+
+    outline_explorer = OutlineExplorerWidget()
+    outline_explorer.set_current_editor(
+            editor, 'test_file.py', False, False)
+
+    qtbot.addWidget(outline_explorer)
+
+    return outline_explorer, qtbot
+
+
+# Test OutlineExplorerWidget
+# -------------------------------
+def test_code_cell_grouping(outline_explorer_bot2):
+    """
+    Test to assert the outline explorer is initializing correctly and
+    is showing the expected number of items, the expected type of items, and
+    the expected text for each item. In addition this tests ancestry, code
+    cells comments, code cell grouping and disabling this feature.
+    """
+    outline_explorer, qtbot = outline_explorer_bot2
+    outline_explorer.show()
+    outline_explorer.setFixedSize(400, 350)
+    assert outline_explorer
+
+    expected_results = [
+        ('test_file.py', FileRootItem),
+        ('function0', FunctionItem, 'test_file.py', 'test_file.py', False),
+        ('Top level 1', CellItem, 'test_file.py', 'test_file.py'),
+        ('function1', FunctionItem, 'Top level 1', 'test_file.py', False),
+        ('Cell Level 1-1', CellItem, 'Top level 1', 'test_file.py'),
+        ('Cell Level 1-2', CellItem, 'Top level 1', 'test_file.py'),
+        ('function2', FunctionItem, 'Cell Level 1-2', 'test_file.py', False),
+        ('inside', FunctionItem, 'function2', 'function2', False),
+        ('Cell Level 2', CellItem, 'Cell Level 1-2', 'test_file.py'),
+        ('Class2', ClassItem, 'Cell Level 2', 'test_file.py'),
+        ('__init__', FunctionItem, 'Class2', 'Class2', True),
+        ('medthod1', FunctionItem, 'Class2', 'Class2', True),
+        ('Cell level 4', CellItem, 'Cell Level 2', 'test_file.py'),
+        ('function4', FunctionItem, 'Cell level 4', 'test_file.py', False),
+        ('Cell Level 3', CellItem, 'Cell Level 1-2', 'test_file.py'),
+        ('function5', FunctionItem, 'Cell Level 3', 'test_file.py', False),
+        ('Cell Level 6', CellItem, 'Cell Level 3', 'test_file.py'),
+        ('Class3', ClassItem, 'Cell Level 6', 'test_file.py'),
+        ('__init__', FunctionItem, 'Class3', 'Class3', True),
+        ('medthod1', FunctionItem, 'Class3', 'Class3', True),
+        ('Top level 2', CellItem, 'test_file.py', 'test_file.py'),
+        ('Class4', ClassItem, 'Top level 2', 'test_file.py'),
+        ('__init__', FunctionItem, 'Class4', 'Class4', True),
+        ('medthod1', FunctionItem, 'Class4', 'Class4', True),
+        ('MGroup3', CellItem, 'test_file.py', 'test_file.py'),
+        ('function6', FunctionItem, 'MGroup3', 'MGroup3', False),
+        ('MGroup4', CellItem, 'MGroup3', 'test_file.py')
+        ]
+
+    outline_explorer.treewidget.expandAll()
+    tree_widget = outline_explorer.treewidget
+    cell_items = tree_widget.get_top_level_items() + tree_widget.get_items()
+
+    # Assert that the expected number, text, ancestry and type of cell items is
+    # displayed in the tree.
+    assert len(cell_items) == len(expected_results)
+    for item, expected_result in zip(cell_items, expected_results):
+        assert item.text(0) == expected_result[0]
+        assert type(item) == expected_result[1]
+        if type(item) != FileRootItem:
+            assert item.parent().text(0) == expected_result[2]
+        if type(item) == FunctionItem:
+            assert item.is_method() == expected_result[4]
+
+    # Disable cell groups
+    tree_widget.toggle_group_cells(False)
+    tree_widget.expandAll()
+    flat_items = tree_widget.get_top_level_items() + tree_widget.get_items()
+
+    # Assert that the expected number, text, ancestry and type of flat items is
+    # displayed in the tree.
+    assert len(flat_items) == len(expected_results)
+    for item, expected_result in zip(flat_items, expected_results):
+        assert item.text(0) == expected_result[0]
+        assert type(item) == expected_result[1]
+        if type(item) != FileRootItem:
+            assert item.parent().text(0) == expected_result[3]
+        if type(item) == FunctionItem:
+            assert item.is_method() == expected_result[4]
+
+    # Change back to cell groups
+    tree_widget.toggle_group_cells(True)
+    tree_widget.expandAll()
+    cell_items2 = tree_widget.get_top_level_items() + tree_widget.get_items()
+
+    # Assert that the expected number, text, ancestry and type of flat items is
+    # displayed in the tree.
+    assert len(cell_items2) == len(expected_results)
+    for item, expected_result in zip(cell_items2, expected_results):
+        assert item.text(0) == expected_result[0]
+        assert type(item) == expected_result[1]
+        if type(item) != FileRootItem:
+            assert item.parent().text(0) == expected_result[2]
+        if type(item) == FunctionItem:
+            assert item.is_method() == expected_result[4]
+
+# Code used to create expected_results
+# =============================================================================
+#     for item in cell_items2:
+#         if type(item) == FunctionItem:
+#             print(f"('{item.text(0)}', {type(item).__name__}, "
+#                   f"'{item.parent().text(0)}', {item.is_method()}),")
+#         elif type(item) == FileRootItem:
+#             print(f"('{item.text(0)}', {type(item).__name__}),")
+#         else:
+#             print(f"('{item.text(0)}', {type(item).__name__}, "
+#                   f"'{item.parent().text(0)}'),")
+# =============================================================================
+
+
 if __name__ == "__main__":
     import os
     pytest.main(['-x', os.path.basename(__file__), '-v', '-rw'])