Repository: spyder-ide/spyder

Issue:
Shift-Enter to run a cell is not cached
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

If you have two consecutive code cells (starting with a #%%) and you try to run them both by pressing Shift-Enter twice, only the first cell will be run.

This happens when the first cell takes some time to run, and it's still running when you press the second shift-enter. The second shift-enter is ignored.

In version 3.X it works fine, running the two cells one after the other.



## Versions

* Spyder version: 4.0.0b3 
* Python version: 3.7.3
* Qt version: 5.12.3
* PyQt5 version: 5.12.2
* Operating System: Windows 10

### Dependencies

```
pygments >=2.0    :  2.4.2 (OK)
qdarkstyle >=2.6.4:  2.7 (OK)
sphinx >=0.6.6    :  2.1.0 (OK)
pyls >=0.27.0     :  0.27.0 (OK)
nbconvert >=4.0   :  5.5.0 (OK)
pandas >=0.13.1   :  0.24.2 (OK)
numpy >=1.7       :  1.16.4 (OK)
sympy >=0.7.3     :  None (NOK)
cython >=0.21     :  0.29.10 (OK)
qtconsole >=4.5.0 :  4.5.1 (OK)
IPython >=4.0     :  7.5.0 (OK)
matplotlib >=2.0.0:  3.1.0 (OK)
pylint >=0.25     :  2.3.1 (OK)
```



Discussion:
Thanks for reporting.

@bcolsen, could you take a look at this one?
This behavior was introduced in `runfile` with #7294 in version 3.3 to fix #7293. This allowed the stop button to stop all executing tasks by not allowing additional calls to the kernel while it's executing.

When I made the `codecell` function for spyder4 I included this fix for code cell execution.

There is also a ux issue here with the cell being advanced to the next cell without cell execution.

I for one regularly used this unofficial queuing behavior for code cells.


## Some possible resolutions:

### Solution 1
Don't allow cell advancement without cell execution.

This might need some sort of error or users might think the functionality is broken.

### Solution 2 
Make a runcell queue.

I personally like this idea but it would be quite work to implement.

### Solution 3
Remove the fix for #7293 from `runcell` and regain the feature from spyder3

This reintroduces #7293, and gives broken output in the console:
```
In [14]: runcell('Cell at line 0','/home/bcolsen/Documents/python/runcell_queue.py')

In [15]: runcell('Long cell','/home/bcolsen/Documents/python/runcell_queue.py')
running long...


Ran long
running 1...
Ran 1

In [16]: runcell('short cell 2','/home/bcolsen/Documents/python/runcell_queue.py')Ran 2


In [17]:
```
Testing Code:

```python
from time import sleep

# %% Long cell
print('running long...')
sleep(5)
print('Ran long')
# %% short cell 1
print('running 1...')
sleep(3)
print('Ran 1')
# %% short cell 2
print('Ran 2')
```
I don't agree with reverting the fix for #7293, so the only possible solution would be to create a queue for cells (and files too). But I don't think if that's a good idea because each evaluation could change the current namespace in significant ways, which could produce odd results for future evaluations.
> I don't agree with reverting the fix for #7293

Yes that solution is isn't good.

> But I don't think if that's a good idea because each evaluation could change the current namespace in significant ways, which could produce odd results for future evaluations.

I can see that this could be a problem with files and cells in the same queue, but it would be less of a problem if we only queue cell execution. I think that it would cover most of the use cases for this and users would expect that the commands would be run sequetially

There will be problems with queuing the new implementation of the `runcell` function @impact27 is working on. It currently gets the cell from editor at runtime in the kernel using the API. If the user queues up a few long running cells and makes changes to the file in the editor the queued commands would take the changed code from the file and not the code from the cell at that time it was queued. It's likely that most users would find this unexpected. The same thing would happen if you queued a couple of files for execution and then changed them. 
> There will be problems with queuing the new implementation of the `runcell` function @impact27 is working on. It currently gets the cell from editor at runtime in the kernel using the API. If the user queues up a few long running cells and makes changes to the file in the editor the queued commands would take the changed code from the file and not the code from the cell at that time it was queued. It's likely that most users would find this unexpected. The same thing would happen if you queued a couple of files for execution and then changed them.

The solution would be to capture the code when Shift-Enter is pressed, and cache it in some way. The logic would then be: 
- If there is code cached, use it. (verify the cell name and file name matches, maybe a `dict`)
- else, use the api to get the code

This poses the problem of cache invalidation of course. How do we know if the cache is still valid (What to do if the user cancelled execution of one of the cached cells?). 

An alternative would be to add an argument to `runcell` to explicitly fetch from cache. something like:
`runcell(cellname, filename, cached=True)`
This way it would be clear if `runcell` should use the cache (It would use the api anyway, but the frontend should know if the code should be from the editor of from the cache).
>The solution would be to capture the code when Shift-Enter is pressed, and cache it in some way. The logic would then be:
>
>   - If there is code cached, use it. (verify the cell name and file name matches, maybe a dict)
>   - else, use the api to get the code

An edge case would be running the same cell 5 times with 5 different edits.

One way around this and cache invalidation is to cache the code directly into the queue. 

- While the kernel is busy, commands(and cached code) should be added to the queue every time a command(like `runcell` and `runline`) is given in Spyder. 
- When the kernel is finished, the next command in the queue is run.
    - This can include sending cache code to the server 
- If at any point the user clicks the stop button the current command is stopped and the queue is flushed. 
- This queue would be part of the Spyder IPython plugin so it would be unique to each kernel and still be modified while the kernel is busy.
 
Moving this here because it's not a regression and whether we save or not might not make a difference. 

@impact27 said:
>Why? Do you want to let the user run a cell and then modify it while it is in the queue? This will already be difficult, as the code is fetched at execution time.

I want to modify cell and execute a cell while a previous version of the cell is in the queue but I don't want to modify the code that is already in the queue.

The front end(spyder) queue needs to contain cached code because a user would expect that when they press a button on the front end on a cell or a file that the code at that time will be the code that eventually runs.

One solution would be to have a comms queue handler in spyder kernels that gets a command and cached code from the spyder queue(a list of dictionaries with commands and code caches). The queue handler could populate a code variable in ipython that if populated would be used as the code instead of asking spyder what the code should be.  There should be a few other ways as well. 

> Moving this here because it's not a regression and whether we save or not might not make a difference.
> 
> @impact27 <https://github.com/impact27> said:
> 
> Why? Do you want to let the user run a cell and then modify it while it is in the queue? This will already be difficult, as the code is fetched at execution time.
> 
> I want to modify cell and execute a cell while a previous version of the cell is in the queue but I don't want to modify the code that is already in the queue.
> 
> The front end(spyder) queue needs to contain cached code because a user would expect that when they press a button on the front end on a cell or a file that the code at that time will be the code that eventually runs.
> 
> One solution would be to have a comms queue handler in spyder kernels that gets a command and cached code from the spyder queue(a list of dictionaries with commands and code caches). The queue handler could populate a code variable in ipython that if populated would be used as the code instead of asking spyder what the code should be. There should be a few other ways as well.
> 
One way I can think of would be to add something to the cell name: if we want to run the cell "do stuff", and the kernel is busy, we could cache the cell content and give it a name like "do Stuff @ cache:1234". When the line is executed, the run_cell function will ask the frontend for the code for cell "do Stuff @ cache:1234", and the frontend could pop the cache. This way we don't need to modify the kernel.
>One way I can think of would be to add something to the cell name: if we want to run the cell "do stuff", and the kernel is busy, we could cache the cell content and give it a name like "do Stuff @ cache:1234". When the line is executed, the run_cell function will ask the frontend for the code for cell "do Stuff @ cache:1234", and the frontend could pop the cache. This way we don't need to modify the kernel.

Yeah that's great idea. Definitely the way to go. The queue would then be a simple list of command strings and the cache would be a dictionary of code and hashes. We can have Spyder do all the code caching separately when it adds to the queue.

We could add an optional parameter to the runcell and runfile like `cache=None` which would be a little cleaner than adding it to the cellname and then having to parse it. Then we would be be able to  cache whole files for runcell as well(the imported modules could still change but we have to stop somewhere).



> Yeah that's great idea. Definitely the way to go. The queue would then be a simple list of command strings and the cache would be a dictionary of code and hashes. We can have Spyder do all the code caching separately when it adds to the queue.
> 
> We could add an optional parameter to the runcell and runfile like `cache=None` which would be a little cleaner than adding it to the cellname and then having to parse it. Then we would be be able to cache whole files for runcell as well(the imported modules could still change but we have to stop somewhere).

I think this is the correct way of doing this. I modified `runfile` in https://github.com/spyder-ide/spyder/pull/10115 to get the code from the editor, so we could even cache `runfile`. When this PR is merged, a `cache` parameter can be added to `_handle_get_file_code` and `handle_run_cell`.

Resolving patch:
diff --git a/spyder/plugins/ipythonconsole/plugin.py b/spyder/plugins/ipythonconsole/plugin.py
--- a/spyder/plugins/ipythonconsole/plugin.py
+++ b/spyder/plugins/ipythonconsole/plugin.py
@@ -522,13 +522,7 @@ def norm(text):
                 line = code.strip()
 
             try:
-                if client.shellwidget._executing:
-                    # Don't allow multiple executions when there's
-                    # still an execution taking place
-                    # Fixes spyder-ide/spyder#7293.
-                    pass
-                else:
-                    self.execute_code(line)
+                self.execute_code(line)
             except AttributeError:
                 pass
             self._visibility_changed(True)
diff --git a/spyder/plugins/ipythonconsole/widgets/debugging.py b/spyder/plugins/ipythonconsole/widgets/debugging.py
--- a/spyder/plugins/ipythonconsole/widgets/debugging.py
+++ b/spyder/plugins/ipythonconsole/widgets/debugging.py
@@ -444,12 +444,10 @@ def execute(self, source=None, hidden=False, interactive=False):
             else:
                 if self._reading_callback:
                     self._reading_callback()
-
             return
-        if not self._executing:
-            # Only execute if not executing
-            return super(DebuggingWidget, self).execute(
-                source, hidden, interactive)
+
+        return super(DebuggingWidget, self).execute(
+            source, hidden, interactive)
 
     def _pdb_readline_callback(self, line):
         """Callback used when the user inputs text in pdb."""
diff --git a/spyder/plugins/ipythonconsole/widgets/shell.py b/spyder/plugins/ipythonconsole/widgets/shell.py
--- a/spyder/plugins/ipythonconsole/widgets/shell.py
+++ b/spyder/plugins/ipythonconsole/widgets/shell.py
@@ -115,6 +115,9 @@ def __init__(self, ipyclient, additional_options, interpreter_versions,
             self.spyder_kernel_comm.register_call_handler(
                 request_id, handlers[request_id])
 
+        self._execute_queue = []
+        self.executed.connect(self.pop_execute_queue)
+
     def __del__(self):
         """Avoid destroying shutdown_thread."""
         if (self.shutdown_thread is not None
@@ -195,7 +198,29 @@ def set_kernel_client_and_manager(self, kernel_client, kernel_manager):
         # Redefine the complete method to work while debugging.
         self._redefine_complete_for_dbg(self.kernel_client)
 
+    def pop_execute_queue(self):
+        """Pop one waiting instruction."""
+        if self._execute_queue:
+            self.execute(*self._execute_queue.pop(0))
+
     # ---- Public API ---------------------------------------------------------
+    def interrupt_kernel(self):
+        """Attempts to interrupt the running kernel."""
+        # Empty queue when interrupting
+        # Fixes spyder-ide/spyder#7293.
+        self._execute_queue = []
+        super(ShellWidget, self).interrupt_kernel()
+
+    def execute(self, source=None, hidden=False, interactive=False):
+        """
+        Executes source or the input buffer, possibly prompting for more
+        input.
+        """
+        if self._executing:
+            self._execute_queue.append((source, hidden, interactive))
+            return
+        super(ShellWidget, self).execute(source, hidden, interactive)
+
     def request_interrupt_eventloop(self):
         """Send a message to the kernel to interrupt the eventloop."""
         self.call_kernel()._interrupt_eventloop()


Test patch:
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -2931,6 +2931,43 @@ def test_runcell_pdb(main_window, qtbot):
     assert "abba 27" in shell._control.toPlainText()
 
 
+@pytest.mark.slow
+@flaky(max_runs=3)
+@pytest.mark.parametrize(
+    "debug", [False, True])
+def test_runcell_cache(main_window, qtbot, debug):
+    """Test the runcell command cache."""
+    # Write code with a cell to a file
+    code = ("import time\n"
+            "time.sleep(.5)\n"
+            "# %%\n"
+            "print('Done')\n")
+    # Wait until the window is fully up
+    shell = main_window.ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+
+    # create new file
+    main_window.editor.new()
+    code_editor = main_window.editor.get_focus_widget()
+    code_editor.set_text(code)
+
+    if debug:
+        # Start debugging
+        with qtbot.waitSignal(shell.executed):
+            shell.execute("%debug print()")
+
+    # Run the two cells
+    code_editor.setFocus()
+    code_editor.move_cursor(0)
+    qtbot.keyClick(code_editor, Qt.Key_Return, modifier=Qt.ShiftModifier)
+    qtbot.wait(100)
+    qtbot.keyClick(code_editor, Qt.Key_Return, modifier=Qt.ShiftModifier)
+    qtbot.wait(500)
+
+    qtbot.waitUntil(lambda: "Done" in shell._control.toPlainText())
+
+
 # --- Path manager
 # ----------------------------------------------------------------------------
 @pytest.mark.slow
diff --git a/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py b/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
--- a/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
+++ b/spyder/plugins/ipythonconsole/tests/test_ipythonconsole.py
@@ -1863,5 +1863,63 @@ def test_stop_pdb(ipyconsole, qtbot):
     assert control.toPlainText().split()[-1] != 'ipdb>'
 
 
+
+@flaky(max_runs=3)
+@pytest.mark.skipif(sys.platform == 'nt', reason="Times out on Windows")
+def test_code_cache(ipyconsole, qtbot):
+    """
+    Test that code sent to execute is properly cached
+    and that the cache is empited on interrupt.
+    """
+    shell = ipyconsole.get_current_shellwidget()
+    qtbot.waitUntil(lambda: shell._prompt_html is not None,
+                    timeout=SHELL_TIMEOUT)
+
+    # Give focus to the widget that's going to receive clicks
+    control = ipyconsole.get_focus_widget()
+    control.setFocus()
+
+    def check_value(name, value):
+        try:
+            return shell.get_value(name) == value
+        except KeyError:
+            return False
+
+    # Send two execute requests and make sure the second one is executed
+    shell.execute('import time; time.sleep(.5)')
+    shell.execute('var = 142')
+    qtbot.wait(500)
+    qtbot.waitUntil(lambda: check_value('var', 142))
+    assert shell.get_value('var') == 142
+
+    # Send two execute requests and cancel the second one
+    shell.execute('import time; time.sleep(.5)')
+    shell.execute('var = 1000')
+    shell.interrupt_kernel()
+    qtbot.wait(1000)
+    # Make sure the value of var didn't change
+    assert shell.get_value('var') == 142
+
+    # Same for debugging
+    with qtbot.waitSignal(shell.executed):
+        shell.execute('%debug print()')
+
+    assert 'ipdb>' in shell._control.toPlainText()
+    # Send two execute requests and make sure the second one is executed
+    shell.execute('time.sleep(.5)')
+    shell.execute('var = 318')
+    qtbot.wait(500)
+    qtbot.waitUntil(lambda: check_value('var', 318))
+    assert shell.get_value('var') == 318
+
+    # Send two execute requests and cancel the second one
+    shell.execute('import time; time.sleep(.5)')
+    shell.execute('var = 1000')
+    shell.interrupt_kernel()
+    qtbot.wait(1000)
+    # Make sure the value of var didn't change
+    assert shell.get_value('var') == 318
+
+
 if __name__ == "__main__":
     pytest.main()