Repository: spyder-ide/spyder

Issue:
Spyder is slow to start due to big history file
When starting, spyder reads a 10Mo text file in ~/.spyder-py3 called history.py.
That takes ~15s, which is a painfully noticeable start-up time. 
Is there an option to limit how far back I want to load the history?

The easy work-around is for me to edit manually this file, but if an user is not aware of this, he might get annoyed by the start-up time.
I can go from 22s to 6s start-up time, which is not fast but way less annoying.

Also closing spyder takes almost 9s. I couldn't figure out why. If I have to restart spyder, usually I just kill it to avoid having to wait.

Python 3.6.5 64bits, Qt 5.10.1, PyQt5 5.10.1 on Darwin 
OSX 10.13.4



Discussion:
For closing, I can go down to 2s  by not having any ipython console opened, and down to <1s by changing `self.process.waitForFinished(1000)` to `self.process.waitForFinished(100)` in the close function of `AsyncClient` in `plugin_client.py`.
Upon further investigation, I would say that when a new client is added to ipythonconsole, a new QtWebEngineProcess is added but not closed when the client is removed. 
If I open and then close 3 iPython tabs:
![screen shot 2018-05-07 at 15 29 31](https://user-images.githubusercontent.com/6740194/39707283-e30c6194-520b-11e8-8169-bff0fe0a546e.png)
First I have 1 main Python process, 2 other python process and 2 QtWebEngineProcess.
If I close spyder now it will be fast.
![screen shot 2018-05-07 at 15 30 39](https://user-images.githubusercontent.com/6740194/39707282-e2f63766-520b-11e8-9f6e-eb3467518d93.png)
Then I open 3 tabs. I get 3 extra python process, and 3 extra QtWebEngineProcess. Spyder will need to wait for some kind of timeout before closing (5-15 s usually).
![screen shot 2018-05-07 at 15 31 23](https://user-images.githubusercontent.com/6740194/39707281-e2df47e0-520b-11e8-93dc-45722047ea0c.png)
The three tabs are closed. The 3 extra python process disappeared, but not the QtWebEngineProcess. Spyder is still slow to close.



> Python 3.6.5 64bits, Qt 5.10.1, PyQt5 5.10.1 on Darwin

Please downgrade to Qt 5.9 and try again. There are multiple reports that PyQt apps don't work well with this version.
> When starting, spyder reads a 10Mo text file in ~/.spyder-py3 called history.py.
That takes ~15s, which is a painfully noticeable start-up time.
Is there an option to limit how far back I want to load the history?

We have plans to improve this for Spyder 4, but I'm unsure what to do about Spyder 3. There's a config option to limit the number of lines of `history.py`, so I guess you're evaluating very long lines. Is that the case?
I can reproduce with:
Python 3.6.5 64bits, Qt 5.9.1, PyQt5 5.9 on Darwin 



My problem is that until I spent time looking into that, I didn't know why spyder was so slow to start. So this is not a good experience for an everyday user. I would probably try to show a message like: "Spyder is slow to start, here what you can do"?

My history is 225k lines, but the setting says 100 max. My understanding of that is:

When spyder is starting, `mainwindow.py` `run_spyder` is called. Somewhere in `main.setup()`, `shell.py`'s `load_history` is called. This history is loaded from `'/Users/quentinpeter/.spyder-py3/history_internal.py'`. This file is empty, so the length is 0. Then the limit is applied, but 0 < 100 so it keeps on going.
Later, `main.post_visible_setup()` is called. When loading ipython, `history.py` `add_history` is called. This loads 
`'/Users/quentinpeter/.spyder-py3/history.py'`, which is 200k lines long. The length is never compared to the `CONF.get('historylog', 'max_entries')` settings, so this file gets to grow as much as it wants. 

So either  `'/Users/quentinpeter/.spyder-py3/history_internal.py'` and `'/Users/quentinpeter/.spyder-py3/history.py'` should be the same thing, or `CONF.get('historylog', 'max_entries')` should be applied on `'/Users/quentinpeter/.spyder-py3/history.py'`. I don't know which one is best.
This seems to be two unrelated issues. Should I open a new issue?
> My history is 225k lines, but the setting says 100 max. My understanding of that is:

Your understanding seems to be correct. Could you open a PR for this? It'd be greatly appreciated!

> Should I open a new issue?

Yes, please open a new issue about the WebEngineProcess'es.
What does the `history_internal.py` file stores? Is the bug that `history_internal.py` should be the same as `history.py` or should I just apply the `max_entries` to `history.py` as well?
I opened #7091 
> What does the history_internal.py file stores? Is the bug that history_internal.py should be the same as history.py or should I just apply the max_entries to history.py as well?

It's not my code, but I'm pretty sure that `history_internal.py` stores the history of the internal console (called `InternalShell` in code). I think the `max_entries` setting shold be appled to `history.py` as well.
Yes, @jitseniesen is right.
The history is duplicated. Why is the history not just loaded from the ipython "history.sqlite"? Is there anyone not using the ipython console? Also this is a bit confusing because the "history log" is not related to what you can actually access by pressing "up" in the ipython console.

Resolving patch:
diff --git a/spyder/plugins/history.py b/spyder/plugins/history.py
--- a/spyder/plugins/history.py
+++ b/spyder/plugins/history.py
@@ -9,6 +9,7 @@
 # Standard library imports
 import os.path as osp
 import sys
+import re
 
 # Third party imports
 from qtpy import PYQT5
@@ -18,14 +19,15 @@
 
 
 # Local imports
-from spyder.utils import encoding
 from spyder.config.base import _
 from spyder.plugins import SpyderPluginWidget
 from spyder.plugins.configdialog import PluginConfigPage
 from spyder.py3compat import is_text_string, to_text_string
+from spyder.utils import encoding
 from spyder.utils import icon_manager as ima
 from spyder.utils.qthelpers import (add_actions, create_action,
                                     create_toolbutton)
+from spyder.utils.sourcecode import normalize_eols
 from spyder.widgets.tabs import Tabs
 from spyder.widgets.sourcecode import codeeditor
 from spyder.widgets.findreplace import FindReplace
@@ -230,6 +232,12 @@ def add_history(self, filename):
         editor.toggle_wrap_mode( self.get_option('wrap') )
 
         text, _ = encoding.read(filename)
+        text = normalize_eols(text)
+        linebreaks = [m.start() for m in re.finditer('\n', text)]
+        maxNline = self.get_option('max_entries')
+        if len(linebreaks) > maxNline:
+            text = text[linebreaks[-maxNline - 1] + 1:]
+            encoding.write(text, filename)
         editor.set_text(text)
         editor.set_cursor_position('eof')
         
diff --git a/spyder/utils/sourcecode.py b/spyder/utils/sourcecode.py
--- a/spyder/utils/sourcecode.py
+++ b/spyder/utils/sourcecode.py
@@ -71,6 +71,14 @@ def has_mixed_eol_chars(text):
     return repr(correct_text) != repr(text)
 
 
+def normalize_eols(text, eol='\n'):
+    """Use the same eol's in text"""
+    for eol_char, _ in EOL_CHARS:
+        if eol_char != eol:
+            text = text.replace(eol_char, eol)
+    return text
+
+
 def fix_indentation(text):
     """Replace tabs by spaces"""
     return text.replace('\t', ' '*4)


Test patch:
diff --git a/spyder/plugins/tests/test_history.py b/spyder/plugins/tests/test_history.py
new file mode 100644
--- /dev/null
+++ b/spyder/plugins/tests/test_history.py
@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# -----------------------------------------------------------------------------
+# Copyright © Spyder Project Contributors
+#
+# Licensed under the terms of the MIT License
+# (see spyder/__init__.py for details)
+# -----------------------------------------------------------------------------
+
+"""
+Tests for the history log
+"""
+
+# Standard library imports
+try:
+    from unittest.mock import Mock
+except ImportError:
+    from mock import Mock
+
+# Third party imports
+from qtpy.QtWidgets import QWidget
+import pytest
+
+# Local imports
+from spyder.plugins.history import HistoryLog
+from spyder.py3compat import to_text_string
+
+
+# =============================================================================
+# Fixtures
+# =============================================================================
+@pytest.fixture
+def history_plugin(qtbot):
+    """History plugin fixture"""
+
+    class MainMock(QWidget):
+        def __getattr__(self, attr):
+            return Mock()
+
+    history = HistoryLog(parent=MainMock())
+    qtbot.addWidget(history)
+    return history
+
+
+# =============================================================================
+# Tests
+# =============================================================================
+def test_max_entries(history_plugin, tmpdir):
+    """Test that history is truncated at max_entries."""
+    max_entries = history_plugin.get_option('max_entries')
+
+    # Write more than max entries in a test file
+    history = ''
+    for i in range(max_entries + 1):
+        history = history + '{}\n'.format(i)
+
+    history_file = tmpdir.join('history.py')
+    history_file.write(history)
+
+    # Load test file in plugin
+    history_plugin.add_history(to_text_string(history_file))
+
+    # Assert that we have max_entries after loading history and
+    # that there's no 0 in the first line
+    assert len(history_file.readlines()) == max_entries
+    assert '0' not in history_file.readlines()[0]
+
+
+if __name__ == "__main__":
+    pytest.main()
diff --git a/spyder/utils/tests/test_sourcecode.py b/spyder/utils/tests/test_sourcecode.py
--- a/spyder/utils/tests/test_sourcecode.py
+++ b/spyder/utils/tests/test_sourcecode.py
@@ -13,6 +13,11 @@
 from spyder.utils import sourcecode
 
 
+def test_normalize_eols():
+    text = "a\nb\r\nc\rd"
+    assert sourcecode.normalize_eols(text) == "a\nb\nc\nd"
+
+
 def test_get_primary_at():
     code = 'import functools\nfunctools.partial'
     assert sourcecode.get_primary_at(code, len(code)) == 'functools.partial'