Repository: spyder-ide/spyder

Issue:
Variable explorer slow to open in spyder 4 with large dataframes



## Problem Description

Moderately large pandas data frames load very slowly in spyder 4 variable explorer.

### What steps reproduce the problem?

`import numpy as np`
`import pandas as pd`

`df=pd.DataFrame(np.arange(400000).reshape(40000,10))`



### What is the expected output? What do you see instead?

The above dataframe (df) takes more than 30 seconds to load  in spyder 4 variable explorer
while it is instantaneous in spyder 3.


## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.0.0
* Python version: 3.7.4
* Qt version: 5.9.7
* PyQt version: 5.9.2
* Operating System name/version: windows 10 x64 1903 

### Dependencies
<!--- Please go to the menu entry Help > Dependencies,
press the Copy to clipboard button and paste below --->

```
cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.5.2 (OK)
qtconsole >=4.6.0            :  4.6.0 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  2.2.2 (OK)
pylint >=0.25                :  2.4.4 (OK)
psutil >=0.3                 :  5.6.7 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.8.1 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  1.4 (OK)
cython >=0.21                :  0.29.14 (OK)
IPython >=4.0                :  7.10.1 (OK)
matplotlib >=2.0.0           :  3.1.1 (OK)
pandas >=0.13.1              :  0.25.3 (OK)
numpy >=1.7                  :  1.17.4 (OK)
scipy >=0.17.0               :  1.3.2 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.2 (OK)
rtree >=0.8.3                :  0.8.3 (OK)

```







Discussion:
I can confirm. ~30 seconds here as well on linux latest 4.x
Thoughts @impact27 ?
looks like `DataFrameEditor`'s `_sizeHintForColumn` is taking a lot of time. Not sure how to optimise that though.
@dalthviz, please take a look at this one.
I second this.  my data loads instantly on v3, but v4 it's taking around 20-30secs.  I like rest of V4 especially Kite's integration.  but this is slow wait is driving me toward downgrading.

Resolving patch:
diff --git a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
@@ -897,7 +897,9 @@ def setup_and_check(self, data, title=''):
 
         # autosize columns on-demand
         self._autosized_cols = set()
-        self._max_autosize_ms = None
+        # Set limit time to calculate column sizeHint to 300ms,
+        # See spyder-ide/spyder#11060
+        self._max_autosize_ms = 300
         self.dataTable.installEventFilter(self)
 
         avg_width = self.fontMetrics().averageCharWidth()
@@ -1096,8 +1098,8 @@ def _reset_model(self, table, model):
         if old_sel_model:
             del old_sel_model
 
-    def setAutosizeLimit(self, limit_ms):
-        """Set maximum size for columns."""
+    def setAutosizeLimitTime(self, limit_ms):
+        """Set maximum time to calculate size hint for columns."""
         self._max_autosize_ms = limit_ms
 
     def setModel(self, model, relayout=True):


Test patch: