Repository: spyder-ide/spyder
Pull request: PR: Improve cell name detection and run cells with the right namespace while debugging
<!--- Make sure to read the Contributing Guidelines:                   --->
<!--- https://github.com/spyder-ide/spyder/blob/master/CONTRIBUTING.md --->
<!--- and follow PEP 8, PEP 257 and Spyder's code style:               --->
<!--- https://github.com/spyder-ide/spyder/wiki/Dev:-Coding-Style      --->

## Description of Changes


* [x] Wrote at least one-line docstrings (for any new functions)
* [x] Added unit test(s) covering the changes (if testable)
<!--- Remember that an image/animation is worth a thousand words! --->
* [ ] Included a screenshot or animation (if affecting the UI, see [Licecap](https://www.cockos.com/licecap/))


<!--- Explain what you've done and why --->
Fixed the way spyder gets the cell name and fixed the namespace runcell uses while debugging



### Issue(s) Resolved

<!--- List the issue(s) below, in the form "Fixes #1234"; one per line --->

Fixes #10563
Depends on spyder-ide/spyder-kernels#178


### Affirmation

By submitting this Pull Request or typing my (user)name below,
I affirm the [Developer Certificate of Origin](https://developercertificate.org)
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

<!--- TYPE YOUR USER/NAME AFTER THE FOLLOWING: --->
I certify the above statement is true and correct:
impact27
<!--- Thanks for your help making Spyder better for everyone! --->


Problem statement:
runcell is buggy
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

Create a file with this content (Exactly 3 lines):
```
if True:
    a = 1
#%%
```
The first cells fails with `RuntimeError: Cell 'if True:' not found.` and the next one creates the traceback below.

### Traceback
```python-traceback
  File "/Users/quentinpeter/pyscripts/spyder/spyder/plugins/editor/widgets/editor.py", line 2649, in run_cell_and_advance
    self.run_cell()
  File "/Users/quentinpeter/pyscripts/spyder/spyder/plugins/editor/widgets/editor.py", line 2638, in run_cell
    cell_name = self._get_cell_name(block)
  File "/Users/quentinpeter/pyscripts/spyder/spyder/plugins/editor/widgets/editor.py", line 2702, in _get_cell_name
    raise RuntimeError('Not a cell?')
RuntimeError: Not a cell?
```

## Versions

* Spyder version: 4.0.0.dev0 49e0a22d5
* Python version: 3.7.3
* Qt version: 5.12.4
* PyQt5 version: 5.12.3
* Operating System: Darwin 19.0.0

### Dependencies

```
cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.4.2 (OK)
qtconsole >=4.5.5            :  4.5.5 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  2.2.1 (OK)
pylint >=0.25                :  2.4.3 (OK)
psutil >=0.3                 :  5.6.3 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.5.0;<2.0.0:  1.7.0.dev0 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  1.4 (OK)
cython >=0.21                :  0.29.13 (OK)
IPython >=4.0                :  7.9.0 (OK)
matplotlib >=2.0.0           :  3.1.1 (OK)
pandas >=0.13.1              :  0.25.2 (OK)
numpy >=1.7                  :  1.17.3 (OK)
scipy >=0.17.0               :  1.3.1 (OK)
pyls >=0.29.2;<0.30.0        :  0.29.3 (OK)
applaunchservices >=0.1.7    :  0.2.1 (OK)
```

Hints from issue discussion:

Commit message: fix Py2 tests

Patch to review:
diff --git a/continuous_integration/azure/install.bat b/continuous_integration/azure/install.bat
index 35ec304b212..61954a7c91c 100644
--- a/continuous_integration/azure/install.bat
+++ b/continuous_integration/azure/install.bat
@@ -34,7 +34,7 @@ if %USE_CONDA% == yes (
 )
 
 :: Install spyder-kernels from master
-pip install -q --no-deps git+https://github.com/spyder-ide/spyder-kernels
+pip install -q --no-deps git+https://github.com/spyder-ide/spyder-kernels.git@refs/pull/178/head
 if errorlevel 1 exit 1
 
 :: Install python-language-server from master
diff --git a/continuous_integration/azure/install.sh b/continuous_integration/azure/install.sh
index bed492c4afd..4f5c40ba594 100755
--- a/continuous_integration/azure/install.sh
+++ b/continuous_integration/azure/install.sh
@@ -20,7 +20,7 @@ if [ "$USE_CONDA" = "yes" ]; then
     conda install -q -y openssl=1.1.1c
 
     # Install spyder-kernels from Github with no deps
-    pip install -q --no-deps git+https://github.com/spyder-ide/spyder-kernels
+    pip install -q --no-deps git+https://github.com/spyder-ide/spyder-kernels.git@refs/pull/178/head
 
     # Install python-language-server from Github with no deps
     pip install -q --no-deps git+https://github.com/palantir/python-language-server
@@ -44,7 +44,7 @@ else
     pip install git+https://github.com/jupyter/qtconsole.git
 
     # Install spyder-kernels from Github
-    pip install -q git+https://github.com/spyder-ide/spyder-kernels
+    pip install -q git+https://github.com/spyder-ide/spyder-kernels.git@refs/pull/178/head
 
     # Install python-language-server from Github
     pip install -q git+https://github.com/palantir/python-language-server
diff --git a/continuous_integration/travis/install.sh b/continuous_integration/travis/install.sh
index 9f13de40f5c..dafab0a502e 100755
--- a/continuous_integration/travis/install.sh
+++ b/continuous_integration/travis/install.sh
@@ -35,7 +35,7 @@ if [ "$USE_CONDA" = "yes" ]; then
     pip install -q coveralls
 
     # Install spyder-kernels from Github with no deps
-    pip install -q --no-deps git+https://github.com/spyder-ide/spyder-kernels
+    pip install -q --no-deps git+https://github.com/spyder-ide/spyder-kernels.git@refs/pull/178/head
 
     # Install python-language-server from Github with no deps
     pip install -q --no-deps git+https://github.com/palantir/python-language-server
@@ -68,7 +68,7 @@ else
     pip install git+https://github.com/jupyter/qtconsole.git
 
     # Install spyder-kernels from Github
-    pip install -q git+https://github.com/spyder-ide/spyder-kernels
+    pip install -q git+https://github.com/spyder-ide/spyder-kernels.git@refs/pull/178/head
 
     # Install python-language-server from Github
     pip install -q git+https://github.com/palantir/python-language-server
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
index 73b6067aba1..b1024691bc4 100644
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -2322,5 +2322,84 @@ def test_runcell(main_window, qtbot, tmpdir, debug):
         pass
 
 
+@pytest.mark.slow
+@flaky(max_runs=3)
+def test_runcell_edge_cases(main_window, qtbot, tmpdir):
+    """Test the runcell command."""
+    # Write code with a cell to a file
+    code = ('if True:\n'
+            '    a = 1\n'
+            '#%%')
+    p = tmpdir.join("test.py")
+    p.write(code)
+    main_window.editor.load(to_text_string(p))
+    shell = main_window.ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+    code_editor = main_window.editor.get_focus_widget()
+    # call runcell
+    with qtbot.waitSignal(shell.executed):
+        qtbot.keyClick(code_editor, Qt.Key_Return, modifier=Qt.ShiftModifier)
+    assert 'runcell(0' in shell._control.toPlainText()
+    assert 'cell is empty' not in shell._control.toPlainText()
+    with qtbot.waitSignal(shell.executed):
+        qtbot.keyClick(code_editor, Qt.Key_Return, modifier=Qt.ShiftModifier)
+    assert 'runcell(1' in shell._control.toPlainText()
+    assert 'Error' not in shell._control.toPlainText()
+    assert 'cell is empty' in shell._control.toPlainText()
+
+
+@pytest.mark.slow
+@flaky(max_runs=3)
+def test_runcell_pdb(main_window, qtbot):
+    """Test the runcell command in pdb."""
+    # Write code with a cell to a file
+    code = ("if 'abba' in dir():\n"
+            "    print('abba {}'.format(abba))\n"
+            "else:\n"
+            "    def foo():\n"
+            "        abba = 27\n"
+            "    foo()\n")
+    # Wait until the window is fully up
+    shell = main_window.ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+
+    # Main variables
+    debug_action = main_window.debug_toolbar_actions[0]
+    debug_button = main_window.debug_toolbar.widgetForAction(debug_action)
+
+    # Clear all breakpoints
+    main_window.editor.clear_all_breakpoints()
+
+    # create new file
+    main_window.editor.new()
+    code_editor = main_window.editor.get_focus_widget()
+    code_editor.set_text(code)
+
+    # Start debugging
+    qtbot.mouseClick(debug_button, Qt.LeftButton)
+
+    for key in ['n', 'n', 's', 'n', 'n']:
+        qtbot.waitUntil(
+            lambda: shell._control.toPlainText().split()[-1] == 'ipdb>')
+        qtbot.keyClick(shell._control, key)
+        qtbot.keyClick(shell._control, Qt.Key_Enter)
+
+    qtbot.waitUntil(
+        lambda: shell._control.toPlainText().split()[-1] == 'ipdb>')
+    assert shell.get_value('abba') == 27
+
+    code_editor.setFocus()
+    # call runcell
+    qtbot.keyClick(code_editor, Qt.Key_Return, modifier=Qt.ShiftModifier)
+    qtbot.waitUntil(lambda: "!runcell" in shell._control.toPlainText())
+    qtbot.waitUntil(
+        lambda: shell._control.toPlainText().split()[-1] == 'ipdb>')
+
+    # Make sure the local variables are detected
+    assert "abba 27" in shell._control.toPlainText()
+
+
 if __name__ == "__main__":
     pytest.main()
diff --git a/spyder/plugins/editor/widgets/base.py b/spyder/plugins/editor/widgets/base.py
index a565ba83936..c69794c7685 100644
--- a/spyder/plugins/editor/widgets/base.py
+++ b/spyder/plugins/editor/widgets/base.py
@@ -31,6 +31,7 @@
 from spyder.plugins.editor.api.decoration import TextDecoration, DRAW_ORDERS
 from spyder.plugins.editor.utils.decoration import TextDecorationsManager
 from spyder.plugins.editor.widgets.completion import CompletionWidget
+from spyder.plugins.outlineexplorer.api import is_cell_header
 
 
 class TextEditBaseWidget(QPlainTextEdit, BaseEditMixin):
@@ -436,24 +437,24 @@ def get_selection_as_executable_code(self, cursor=None):
 
         return leading_lines_str + ls.join(lines)
 
-    def __exec_cell(self, cursor=None):
-        """Get text and line number from cursor or current position."""
+    def get_cell_as_executable_code(self, cursor=None):
+        """Return cell contents as executable code."""
         if cursor is None:
             cursor = self.textCursor()
         ls = self.get_line_separator()
         cursor, whole_file_selected = self.select_current_cell(cursor)
         line_from, line_to = self.get_selection_bounds(cursor)
-        block = self.get_selection_first_block(cursor)
+        # Get the block for the first cell line
+        start = cursor.selectionStart()
+        block = self.document().findBlock(start)
+        if not is_cell_header(block) and start > 0:
+            block = self.document().findBlock(start - 1)
+        # Get text
         text = self.get_selection_as_executable_code(cursor)
-
         if text is not None:
             text = ls * line_from + text
         return text, block
 
-    def get_cell_as_executable_code(self, cursor=None):
-        """Return cell contents as executable code."""
-        return self.__exec_cell(cursor)
-
     def is_cell_separator(self, cursor=None, block=None):
         """Return True if cursor (or text block) is on a block separator"""
         assert cursor is not None or block is not None
diff --git a/spyder/plugins/editor/widgets/editor.py b/spyder/plugins/editor/widgets/editor.py
index 091baaee0a7..100d93d66ae 100644
--- a/spyder/plugins/editor/widgets/editor.py
+++ b/spyder/plugins/editor/widgets/editor.py
@@ -59,6 +59,7 @@
 from spyder.config.manager import CONF
 from spyder.plugins.explorer.widgets.explorer import (
     show_in_external_file_explorer)
+from spyder.plugins.outlineexplorer.api import cell_name
 
 
 logger = logging.getLogger(__name__)
@@ -2635,10 +2636,10 @@ def run_cell(self, debug=False):
         text, block = self.get_current_editor().get_cell_as_executable_code()
         finfo = self.get_current_finfo()
         editor = self.get_current_editor()
-        cell_name = self._get_cell_name(block)
+        name = cell_name(block)
         filename = finfo.filename
 
-        self._run_cell_text(text, editor, (filename, cell_name), debug)
+        self._run_cell_text(text, editor, (filename, name), debug)
 
     def debug_cell(self):
         """Debug current cell."""
@@ -2686,21 +2687,6 @@ def re_run_last_cell(self):
 
         self._run_cell_text(text, editor, (filename, cell_name))
 
-    def _get_cell_name(self, block):
-        """Get the cell name from the block."""
-        oe_data = block.userData()
-        if oe_data and oe_data.oedata:
-            if oe_data.oedata.has_name():
-                cell_name = oe_data.oedata.def_name
-            else:
-                cell_name = oe_data.oedata.cell_index()
-        else:
-            if block.blockNumber() == 0:
-                # There is no name for the first cell, refer by cell number
-                cell_name = 0
-            else:
-                raise RuntimeError('Not a cell?')
-        return cell_name
 
     def _run_cell_text(self, text, editor, cell_id, debug=False):
         """Run cell code in the console.
@@ -2717,7 +2703,7 @@ def _run_cell_text(self, text, editor, cell_id, debug=False):
             The starting line number of the cell in the file.
         """
         (filename, cell_name) = cell_id
-        if editor.is_python() and text:
+        if editor.is_python():
             args = (text, cell_name, filename, self.run_cell_copy)
             if debug:
                 self.debug_cell_in_ipyclient.emit(*args)
diff --git a/spyder/plugins/outlineexplorer/api.py b/spyder/plugins/outlineexplorer/api.py
index e2471cbf95a..d86708a5e72 100644
--- a/spyder/plugins/outlineexplorer/api.py
+++ b/spyder/plugins/outlineexplorer/api.py
@@ -160,13 +160,13 @@ def check_match(oedata):
 
         # Count cells
         N_prev = 0
-        for oedata in self._document_cells(forward=False):
+        for oedata in document_cells(self.block, forward=False):
             if check_match(oedata):
                 N_prev += 1
         N_fix_previous = len(existing_numbers)
 
         N_next = 0
-        for oedata in self._document_cells(forward=True):
+        for oedata in document_cells(self.block, forward=True):
             if check_match(oedata):
                 N_next += 1
 
@@ -186,33 +186,6 @@ def def_name(self, value):
         """Set name."""
         self._def_name = value
 
-    def _document_cells(self, forward=True):
-        """
-        Get all cells oedata in the document.
-
-        Parameters
-        ----------
-        forward : bool, optional
-            Whether to iterate forward or backward from the current block.
-        """
-        if not self.is_valid():
-            # Avoid calling blockNumber if not a valid block
-            return
-
-        if forward:
-            block = self.block.next()
-        else:
-            block = self.block.previous()
-
-        while block.isValid():
-            data = block.userData()
-            if data and data.oedata and data.oedata.def_type == self.CELL:
-                yield data.oedata
-            if forward:
-                block = block.next()
-            else:
-                block = block.previous()
-
     def update(self, other):
         """Try to update to avoid reloading everything."""
         if (self.def_type == other.def_type and
@@ -226,7 +199,7 @@ def update(self, other):
                 if self.cell_level != other.cell_level:
                     return False
                 # Must update all other cells whose name has changed.
-                for oedata in self._document_cells(forward=True):
+                for oedata in document_cells(self.block, forward=True):
                     if oedata._def_name in [self._def_name, old_def_name]:
                         oedata.sig_update.emit()
             return True
@@ -242,13 +215,6 @@ def is_valid(self):
                 and block.userData().oedata == self
                 )
 
-    def cell_index(self):
-        """Get the cell index."""
-        if self.def_type != self.CELL:
-            raise RuntimeError("This is not a cell.")
-        # Cell 0 has no header
-        return len(list(self._document_cells(forward=False))) + 1
-
     def has_name(self):
         """Check if cell has a name."""
         if self._def_name:
@@ -262,3 +228,72 @@ def get_block_number(self):
             # Avoid calling blockNumber if not a valid block
             return None
         return self.block.blockNumber()
+
+
+def document_cells(block, forward=True):
+    """
+    Get cells oedata before or after block in the document.
+
+    Parameters
+    ----------
+    forward : bool, optional
+        Whether to iterate forward or backward from the current block.
+    """
+    if not block.isValid():
+        # Not a valid block
+        return
+
+    if forward:
+        block = block.next()
+    else:
+        block = block.previous()
+
+    while block.isValid():
+        data = block.userData()
+        if (data
+                and data.oedata
+                and data.oedata.def_type == OutlineExplorerData.CELL):
+            yield data.oedata
+        if forward:
+            block = block.next()
+        else:
+            block = block.previous()
+
+
+def is_cell_header(block):
+    """Check if the given block is a cell header."""
+    if not block.isValid():
+        return False
+    data = block.userData()
+    return (data
+            and data.oedata
+            and data.oedata.def_type == OutlineExplorerData.CELL)
+
+
+def cell_index(block):
+    """Get the cell index of the given block."""
+    index = len(list(document_cells(block, forward=False)))
+    if is_cell_header(block):
+        return index + 1
+    return index
+
+
+def cell_name(block):
+    """
+    Get the cell name the block is in.
+
+    If the cell is unnamed, return the cell index instead.
+    """
+    if is_cell_header(block):
+        header = block.userData().oedata
+    else:
+        try:
+            header = next(document_cells(block, forward=False))
+        except StopIteration:
+            # This cell has no header, so it is the first cell.
+            return 0
+    if header.has_name():
+        return header.def_name
+    else:
+        # No name, return the index
+        return cell_index(block)
diff --git a/spyder/widgets/mixins.py b/spyder/widgets/mixins.py
index 8afbdced712..ab80bc4732e 100644
--- a/spyder/widgets/mixins.py
+++ b/spyder/widgets/mixins.py
@@ -1018,16 +1018,6 @@ def get_selection_start_end(self, cursor=None):
         end_position = self.get_cursor_line_column(end_cursor)
         return start_position, end_position
 
-    def get_selection_first_block(self, cursor=None):
-        """Return the first block of the selection."""
-        if cursor is None:
-            cursor = self.textCursor()
-        start = cursor.selectionStart()
-        if start > 0:
-            start = start - 1
-        return self.document().findBlock(start)
-
-
     #------Text selection
     def has_selected_text(self):
         """Returns True if some text is selected."""


Code review comments:
diff_hunk: @@ -262,3 +228,72 @@ def get_block_number(self):
             # Avoid calling blockNumber if not a valid block
             return None
         return self.block.blockNumber()
+
+
+def document_cells(block, forward=True):
path: spyder/plugins/outlineexplorer/api.py
text: @user1:
Please move these functions to the beginning of the file because they are auxiliary functions, used here and by other classes around our codebase.