Repository: spyder-ide/spyder

Issue:
Spyder 4.0.0 unbearably slow when adding or removing lines
## Problem Description

Spyder is very very slow everytime I add (with Enter) or remove (with shortcut set to Ctrl+D) lines in the Editor. It takes on average something close to a second for every line. When holding, the line commands will buffer and then they will be added or deleted as slow as maybe two seconds per line. 

Same issue after spyder --reset. Also tried toggling auto completion. This has been happening since the beta/release candidate and I havent been using it because of this. Spyder is unusable now like this for me.

### What steps reproduce the problem?

1. Press Enter or Ctrl+D

### What is the expected output? What do you see instead?

I expect it to not take so long like older versions.

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.0.0
* Python version: 3.7.5
* Qt version: 5.9.6
* PyQt version: 5.9.2
* Operating System name/version: Win 10

### Dependencies
cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.5.2 (OK)
qtconsole >=4.6.0            :  4.6.0 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  2.2.2 (OK)
pylint >=0.25                :  2.4.4 (OK)
psutil >=0.3                 :  5.6.7 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.8.1 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  1.4 (OK)
cython >=0.21                :  0.29.14 (OK)
IPython >=4.0                :  7.10.1 (OK)
matplotlib >=2.0.0           :  3.1.1 (OK)
pandas >=0.13.1              :  0.25.3 (OK)
numpy >=1.7                  :  1.17.4 (OK)
scipy >=0.17.0               :  1.3.1 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.2 (OK)
rtree >=0.8.3                :  0.8.3 (OK)


Discussion:
@gabrielclow, your description is not enough. Please post:

1. What kind of files are you editing (Python or other)?
2. How many lines you file have?
3. Do you have things like "Indent guides" enabled?

Finally, please post a video (animated gif with the Licecap) problem to see what you're exactly referring to.
> @gabrielclow, your description is not enough. Please post:
> 
> 1. What kind of files are you editing (Python or other)?
> 2. How many lines you file have?
> 3. Do you have things like "Indent guides" enabled?
> 
> Finally, please post a video (animated gif with the Licecap) problem to see what you're exactly referring to.

1. I'm editing only .py files
2. Doesn't look like it depends on the number of lines. Happens in small files with 50 lines to big ones with 5k lines.
3. I only use auto-complete (config in screenshot). The rest is turned off (no linting, no code style linting, no docstring linting, no indent guides). I have no other languages set-up and I also tried toggling Kite on and off.

![autocomplete](https://user-images.githubusercontent.com/9113853/70472803-836fdf00-1aae-11ea-8d50-c53ebd5add0a.png)

In the gif, I hold Enter, wait for the new lines to be created and then hold Ctrl+Z to remove them (the same effect as Ctrl+D, so I'm guessing it also doesn't have anything to do with the shortcut). Spyder then stops and adds or deletes the lines very slowly instead of doing it smoothly:

![stopwatch](https://user-images.githubusercontent.com/9113853/70474806-02b2e200-1ab2-11ea-89a8-71500a9ac32d.gif)

**update**: apparently, if I close some files the slow down is less noticeable!
**update 2**: also, the amount of slowing down is apparently related to the number of open files + number of lines in each file, but it happens regardless of the one currently selected. If I keep only a few smaller files and close the rest, it's much smoother
**update 3**: if I hold down too long or press too fast, Spyder can even hang for a while
> also, the amount of slowing down is apparently related to the number of open files

How many files do you have opened when you saw the worst slowdown?
> > also, the amount of slowing down is apparently related to the number of open files
> 
> How many files do you have opened when you saw the worst slowdown?

I mostly work with 6-8 files open. Usually about two of them are bigger (4k-5k lines) and the rest are smaller files with 100-500 lines.
@CAM-Gerlach, have you seen something like this before?
If I may comment on this, I am having the same problem as well, which actually made me uninstall Spyder 4 and go back to Spyder 3.3.6.
I have only the settings enabled that are enabled in Spyder 3 as well, so PEP8 checking; code completion and details are enabled; but no docstring checking; code snippets; hover descriptions; kite; etc.
The slowdown is as shown in @gabrielclow's GIF, although it can sometimes take as much as 2 seconds for a single line to be inserted.

As I build large packages, it is not uncommon for me to have like 30-50 Python files open simultaneously, some of which can easily contain over 3k lines of code (I had these open when I was noticing the massive slowdown).
Since I use 3 monitors simultaneously to edit my scripts most of the time, which can hold a maximum of 8 panels, and I need to switch between files frequently, closing most of these files will not work for me.

I personally get the feeling that the main reason for this slowdown, is that there are quite a few databases and scripts that need to be updated/ran whenever an open file is modified.
Is there, by any chance, a script that is ran on all lines in all open files whenever a file is modified?
It would explain why the slowdown seems to scale with the combined total number of lines in all open files, and why it seems to be independent of the file being edited. 
I can confirm this bug. It's not so bad(didn't really notice until now) on my computer but adding/removing line is much slower than adding text. The enter problem is really noticeable if you hold it down.
@bcolsen, how can we reproduce it?
Open about 40 files and it seems to slow pretty good. It looks like there might be something in docdidchange that is checking all the files not just the active one. Maybe the outliner.
@ccordoba12 As far as I can tell, this behavior is basically the same as #8864 , and sure enough I can repro it with clean settings on Spyder 4 final with just `mainwindow.py` open so long as indent guides are enabled (having split panes makes the lag double or more). Deleting lines has the same major lag (exactly as described here; with a ~1s latency and a ~1s repeat delay on a moderately powerful machine, and scales proportionally to file length and the number of split panes open) as the other things I tested there (moving lines, duplicating lines, etc), and I can confirm it still happens on Spyder 4 final with only the temp file and `mainwindow.py` open with clean settings.

However, I thought indent guides were not enabled by default, and indeed they don't appear to be enabled after I reset prefs, but I can't seem to repro anywhere near this amount of lag with any other setting (real-time code analysis, OTF completion, underline errors and warnings, show blank spaces, etc,); though with all of them enabled save indent guides it does lag quite a bit.

> Is there, by any chance, a script that is ran on all lines in all open files whenever a file is modified?

LSP gets called, and I initially though that was it but based on its behavior with split panes, specific displayed items but not others, etc. I'm thinking it may be redraw-related instead (drawing the whole file instead of just the visible area, etc). But that's way above my pay grade.
> @ccordoba12 As far as I can tell, this behavior is basically the same as #8864 , and sure enough I can repro it with clean settings on Spyder 4 final with just `mainwindow.py` open so long as indent guides are enabled (having split panes makes the lag double or more).

If I also enable the indent guides together with having my 50 files open and 8 split panes (which is how I usually edit my files), the lag increases to anywhere between 6 to 10 seconds per new line.
The computer I am using is a high-end gaming desktop, so it is not really because of the power of the machine.
@dalthviz and I tried to reproduce #8864 unsuccessfully. So sorry to say, but until we're able to reproduce it, there's nothing we can do about it.
> Open about 40 files and it seems to slow pretty good. It looks like there might be something in docdidchange that is checking all the files not just the active one. Maybe the outliner.

Ok, that's a good start. We'll check this one out.
@ccordoba12 I've figured out the source of this lag, which is in fact appears to be the main "baseline" lag felt in Spyder 4 vs. Spyder 3 that I've talked about for a while: As @bcolsen suggested, it is indeed the outline pane, although it actually occurs regardless of whether said pane is open or closed. On a lower-end machine, it is actually visible (though nowhere near as bad) with only `temp.py` and `mainwindow.py` open, nothing else enabled and no outline pane open. Here's how to repro from clean settings:

1. Open `mainwindow.py` and a split editor panel, and set the right panel to show temp.py and the left to show mainwindow.
2. Hold down delete/duplicate/move line in the left panel (mainwindow), undo/redo, etc. for the same. A small but noticeable lag is apparent.
3. Open the Outline pane (which was hidden by default). This (due to a minor but useful apparent bug) de-syncs the file shown in the outline pane from the one focused, due to it switching to show the file active in the right-most split panel rather than the active split panel when opened/unhidden.
4. Retry deleting/etc/ lines in mainwindow. Very little lag is present and repeat rate is higher (likely will be more noticeable with lower performance machines)
5. Click on right panel (temp.py), and then back to left panel (mainwindow) to make the outline explorer properly show the outline of the left panel.
6. Retry deleting/etc., and the lag is back, same as in step 2 (when outline explorer wasn't visible at all)

To test it with many files, I searched `import os` in the `spyder` directory, opened the first 40-50 files in a split panel view (fewer larger files should produce the equivalent effect) and repeated steps 2-6 above. Even on a short file, there was a far larger amount of lag (similar to indent guide level on Mainwindow) when deleting lines in both steps 2 and 6 (outline explorer hidden, and shown on the active file), while there was essentially no lag (as if no files were open, or Spyder 3 level) on step 4 (outline explorer open and de-synced from active file).

Meanwhile, enabling indent guides for both times, deleting/etc lines was very slow regardless of the number of files open, and the resulting lag appeared to only be substantially affected by the length of the current file being modified and the number of split panes showing the file.
> 3. Open the Outline pane (which was hidden by default). This (due to a minor but useful apparent bug) de-syncs the file shown in the outline pane from the one focused, due to it switching to show the file active in the right-most split panel rather than the active split panel when opened/unhidden.

This really does sound like what I'm experiencing but just adding another detail: I work with one separate editor window on another display and I close the editor pane on Spyder everytime I launch it. So I still experience this bug with a single editor window without any split panes, if that's relevant. Indent guides also make it take much longer.

I hope you can find the answer and this gets fixed soon so I can use the great dark mode in 4.0.0
I can confirm that every time a line is added or removed the outliner updates all of the files that are open. This leads to increases in time for line removal /addition that increases with the number of files open. I have made a simple "fix" in the above PR that seems to remedy the problem with many open files. @impact27 Thoughts?

A separate but obviously associated issue is that large files (>2000 lines) are also slow to update with line changes. Having 4 or 5 of these large files open currently make things even slower with update_all makes. With the above patch, the problem that @CAM-Gerlach is describing above will still happen on the large file but the small file should always be fast. 
>A separate but obviously associated issue is that large files (>2000 lines) are also slow to update with line changes. Having 4 or 5 of these large files open currently make things even slower with update_all makes. With the above patch, the problem that @CAM-Gerlach is describing above will still happen on the large file but the small file should always be fast.

It seems we need to move the UI blocking process to a thread.
> It seems we need to move the UI blocking process to a thread.

This would be best. I think it's associated with the syntax highlighter at that point however.
>I think it's associated with the syntax highlighter at that point however.

Yeah that is a problem since the syntax highlighter infrastructure of Qt runs on the main thread :-\



Resolving patch:
diff --git a/spyder/plugins/outlineexplorer/widgets.py b/spyder/plugins/outlineexplorer/widgets.py
--- a/spyder/plugins/outlineexplorer/widgets.py
+++ b/spyder/plugins/outlineexplorer/widgets.py
@@ -338,12 +338,12 @@ def connect_current_editor(self, state):
         sig_update = editor.sig_outline_explorer_data_changed
         sig_move = editor.sig_cursor_position_changed
         if state:
-            sig_update.connect(self.update_all)
+            sig_update.connect(self.update_current)
             sig_move.connect(self.do_follow_cursor)
             self.do_follow_cursor()
         else:
             try:
-                sig_update.disconnect(self.update_all)
+                sig_update.disconnect(self.update_current)
                 sig_move.disconnect(self.do_follow_cursor)
             except TypeError:
                 # This catches an error while performing
@@ -399,14 +399,40 @@ def file_renamed(self, editor, new_filename):
 
     @Slot()
     def update_all(self):
+        """
+        Update the outline explorer for all editors tree preserving the tree
+        state
+        """
         self.save_expanded_state()
         for editor, editor_id in list(self.editor_ids.items()):
-            item = self.editor_items[editor_id]
-            tree_cache = self.editor_tree_cache[editor_id]
-            self.populate_branch(editor, item, tree_cache)
+            self.__do_update(editor, editor_id)
         self.restore_expanded_state()
         self.do_follow_cursor()
 
+    @Slot()
+    def update_current(self):
+        """
+        Update the outline explorer for the current editor tree preserving the
+        tree state
+        """
+        plugin_base = self.parent().parent()
+        if getattr(plugin_base, "_isvisible", True):
+            self.save_expanded_state()
+            editor = self.current_editor
+            editor_id = editor.get_id()
+            self.__do_update(editor, editor_id)
+            self.restore_expanded_state()
+            self.do_follow_cursor()
+
+    def __do_update(self, editor, editor_id):
+        """
+        Recalculate the and update the tree items in the Outliner for a
+        given editor
+        """
+        item = self.editor_items[editor_id]
+        tree_cache = self.editor_tree_cache[editor_id]
+        self.populate_branch(editor, item, tree_cache)
+
     def remove_editor(self, editor):
         if editor in self.editor_ids:
             if self.current_editor is editor:


Test patch: