Repository: spyder-ide/spyder
Pull request: PR: Show group cells in Outline Explorer
### Pull Request Checklist

* [x] Read and followed this repo's [Contributing Guidelines](https://github.com/spyder-ide/spyder/blob/master/CONTRIBUTING.md)
* [x] Based your PR on the latest version of the correct branch (master or 3.x)
* [x] Followed [PEP8](https://www.python.org/dev/peps/pep-0008/) for code style
* [x] Ensured your pull request hasn't eliminated unrelated blank lines/spaces, modified the ``spyder/defaults`` directory, or added new icons/assets
* [x] Wrote at least one-line docstrings for any new functions
* [x] Added at least one unit test covering the changes, if at all possible
* [x] Described your changes and the motivation for them below
* [x] Noted what issue(s) this pull request resolves, creating one if needed
* [x] Included a screenshot, if this PR makes any visible changes to the UI


## Description of Changes

**Goal:** Being able to use code cells to organize and group the code blocks in the outline panel.

1. The first necessary change is to group all code blocks into code cells.  When present, code cells are now the top level objects in the outliner and all code block will be grouped under them.

2. Second the ability to use `# %%%` as a grouped cell that will be place under the previous top level `# %%` cell.

See it in action:
![outline_cell_folding](https://user-images.githubusercontent.com/10513354/39934369-cabee63a-5502-11e8-877d-3d3493b8b103.gif)

I didn't actually change much code and seems to cover most cases.  If you put a code cell comment in a code block like a class or a function the out line misses the rest of the block.  I don't see a use case of a code cell in a function or class. Unfortunately this simple code doesn't know about function or classes in defined in a code block such as an if statement so I will have to do something more sophisticated with cell blocks in code blocks 

### Issue(s) Resolved

Fixes #7086 

### Things to do

* [x] Deal with cell blocks in code blocks
* [x] More group levels? (I think it's unnecessary/messy, but possible)
* [x] Fix updating the tree_cache
* [x] More manual testing
* [x] Fix bug in "previous"
* [x] Unit testing functions



Problem statement:
Grouping cells in Outline without code indentation
## Problem Description

Spyders Outline works well most of the time with cells and grouping by indentation to represent the code. Sometimes, manual setting would be advantageous (e.g. group functions desfined at top). See example..


### What steps reproduce the problem?

Add the following to spyder. See Outline. 

```
#%%a simple cell with some code
def withAFunction(x):
    return x
y = withAFunction(x)

#%%another cell with a function which gets grouped, but produces indentation error
    def withAnotherFunction(x):
        return x
    y = withAFunction(x)
```


### What is the expected output? What do you see instead?
First one expected, second would be good sometimes I think. However, getting second forces a wrong ident in code.

### Paste Traceback/Error Below (if applicable)
No traceback

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->
Python 3.6.3 64bits, Qt 5.9.4, PyQt5 5.9.2 on Windows 
* Spyder version:  3.2.8
* Python version: 3.6.3
* Qt version: 5.9.4
* PyQt version: 5.9.2
* Operating System name/version: Win10 x64

Hints from issue discussion:
So you want to define cells with indented code? Sorry, but I don't follow exactly what you mean.
@ccordoba12 I believe what the OP is suggesting is a way to get functions (etc.) grouped into collapsible sections by cell, like Cell 2 in the screenshot, without having to manually indent the code. Perhaps an option to that effect ("Group by cell") in the drop down outline menu to turn on/off the behavior would be the best way to accomplish that, assuming someone had time to implement it. @max3-2 , if you're interested and @ccordoba12 agrees, perhaps that someone could be you. Otherwise, due to our current funding situation, it might not get done anytime soon, unfortunately.

Screenshot of the resulting behavior, for reference.

![image](https://user-images.githubusercontent.com/17051931/39796922-0c0567b2-531f-11e8-850c-98c2fdb59d3b.png)

> I believe what the OP is suggesting is a way to get functions (etc.) grouped into collapsible sections by cell

Ok, but it doesn't seem easy to implement.
Yeah that was the idea. If you have a large function library in front of your code or in an ext. file this would help grouping and organizing. I'll leave this open and check if I find the time to take a closer look myself. Havent worked / contributed to spyder yet so its starting from 0...
Thanks @max3-2 , we'd be glad to look at your contribution! Check out our [Contributing Guide](https://github.com/spyder-ide/spyder/blob/master/CONTRIBUTING.md) to get started, and let us know if you'd like pointers.
@max3-2 This would help organize the outline.

As for syntax, you shouldn't need to play with the code in the file at all, so the grouping needs to be done in the block comment.

I would suggest this because it's sort of like markdown header hierarchy: 

```python
# %% Top level cell
def withAFunction(x):
    return x
y = withAFunction(x)

# %%% A grouped cell
def withAnotherFunction(x):
    return x
y = withAFunction(x)
```
Top level cells have 2 `%` and grouped cell have 3 `%`. You could even go down another level to 4 `%`

Grouped cells also open up the possibility to run the whole group or just the one cell.  That would get quite complicated.
> I would suggest this because it's sort of like markdown header hierarchy

Really nice suggestion! Would you like to implement it @bcolsen?
I don't want to take it away from @max3-2, but I'll have a look at after the sympy console stuff

Commit message: Group fuctions in cells and make sub cell

Patch to review:
diff --git a/spyder/utils/syntaxhighlighters.py b/spyder/utils/syntaxhighlighters.py
index 14849892e04..ec317b39daf 100644
--- a/spyder/utils/syntaxhighlighters.py
+++ b/spyder/utils/syntaxhighlighters.py
@@ -420,6 +420,7 @@ def __init__(self, parent, font=None, color_scheme='Spyder'):
         BaseSH.__init__(self, parent, font, color_scheme)
         self.import_statements = {}
         self.found_cell_separators = False
+        self.cell_offset = 0
         self.cell_separators = CELL_LANGUAGES['Python']
 
     def highlight_block(self, text):
@@ -441,7 +442,7 @@ def highlight_block(self, text):
         else:
             offset = 0
             prev_state = self.NORMAL
-        
+
         oedata = None
         import_stmt = None
 
@@ -478,7 +479,12 @@ def highlight_block(self, text):
                                 self.found_cell_separators = True
                                 oedata = OutlineExplorerData()
                                 oedata.text = to_text_string(text).strip()
-                                oedata.fold_level = start
+                                if text.lstrip().startswith(("# %%%", "#%%%")):
+                                    self.cell_offset = 8
+                                    oedata.fold_level = 4
+                                else:
+                                    self.cell_offset = 4
+                                    oedata.fold_level = 0
                                 oedata.def_type = OutlineExplorerData.CELL
                                 oedata.def_name = text.strip()
                             elif self.OECOMMENT.match(text.lstrip()):
@@ -496,7 +502,8 @@ def highlight_block(self, text):
                                                    self.formats["definition"])
                                     oedata = OutlineExplorerData()
                                     oedata.text = to_text_string(text)
-                                    oedata.fold_level = start
+                                    oedata.fold_level = (start +
+                                                         self.cell_offset)
                                     oedata.def_type = self.DEF_TYPES[
                                                         to_text_string(value)]
                                     oedata.def_name = text[start1:end1]
@@ -507,7 +514,8 @@ def highlight_block(self, text):
                                 if text.lstrip().startswith(value):
                                     oedata = OutlineExplorerData()
                                     oedata.text = to_text_string(text).strip()
-                                    oedata.fold_level = start
+                                    oedata.fold_level = (start +
+                                                         self.cell_offset)
                                     oedata.def_type = \
                                         OutlineExplorerData.STATEMENT
                                     oedata.def_name = text.strip()


Code review comments:
diff_hunk: @@ -478,7 +479,12 @@ def highlight_block(self, text):
                                 self.found_cell_separators = True
                                 oedata = OutlineExplorerData()
                                 oedata.text = to_text_string(text).strip()
-                                oedata.fold_level = start
+                                if text.lstrip().startswith(("# %%%", "#%%%")):
path: spyder/utils/syntaxhighlighters.py
text: @user1:
You should be able to allow for an unlimited number of levels and make your code cleaner to boot if you just count the number of ``"%"``s after the initial ``# %%`` or ``#%%`` and set your offset/fold level to ``base_value`` + ``count_%`` * 4. 

@author:
I agree.  The code could be cleaner here.  I'll see what I need to do to fix "cell blocks in code blocks" and then clean it up.  Even the `4` could get messy with those "2-space" people ;-)

After looking at the "cell blocks in code blocks" issue, I think more than one subgroup level would be a testing nightmare with lots of corner cases.  What would happen if some one decorates the code block with `# %%%%%%%%%%% Cell Name %%%%%%%%%%%%`? This currently makes a code block, but you can see the problem.