Repository: spyder-ide/spyder

Issue:
Variable Explorer very slow with large dataframe
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->

### Issue Report Checklist

* [X] Searched the [issues page](https://github.com/spyder-ide/spyder/issues?q=is%3Aissue) for similar reports
* [X] Read the relevant sections of the [Spyder Troubleshooting Guide](https://github.com/spyder-ide/spyder/wiki/Troubleshooting-Guide-and-FAQ) and followed its advice
* [X] Reproduced the issue after updating with ``conda update spyder`` (or ``pip``, if not using Anaconda)
* [ ] Could not reproduce inside ``jupyter qtconsole`` (if console-related)
* [ ] Tried basic troubleshooting (if a bug/error)
    * [X] Restarted Spyder
    * [X] Reset preferences with ``spyder --reset``
    * [X] Reinstalled the latest version of [Anaconda](https://www.anaconda.com/download/)
    * [ ] Tried the other applicable steps from the Troubleshooting Guide
* [X] Completed the **Problem Description**, **Steps to Reproduce** and **Version** sections below


## Problem Description

Variable Explorer is extremely slow in a dataframe containing 30k+ rows.
It's nearly nstantaneous in Spyder 3.3.3.
Scrolling is extremely laggy, window freezes frequently.

### What steps reproduce the problem?

Run the code following:
```
import pandas as pd
import numpy as np
from datetime import datetime

bdates = pd.bdate_range(start = datetime(1900, 1, 1), end = datetime.today())
vect_random = np.random.normal(size=len(bdates))
df = pd.DataFrame(vect_random, index = bdates)
```

And open the df variable in the Variable Explorer, and scroll.

### What is the expected output? What do you see instead?

Should be as fast to open and scroll as in 3.3.3

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 4.1.2
* Python version: 3.7.7
* Qt version: 5.9.6
* PyQt version: 5.9.2
* Operating System name/version: Windows 10




Discussion:
Hi @Skullnick, thanks for the report.

I can confirm that this is indeed an issue with the provided example. The scrolling is really slow.
Hi @dalthviz it seems we have some performance degradation. Please take a look at this one to see what might be caiusing the slowdowns.
Note: Seems like the problem is the handling of the datetime index. With a numeric index no performance degradation is experienced (even testing with more than 30k rows):

No degradation case:
```python
import pandas as pd
import numpy as np

bdates = range(50000)
vect_random = np.random.normal(size=len(bdates))
df = pd.DataFrame(vect_random, index = bdates)
```
Indeed @dalthviz it is the conversion we are doing to the datetime index.
> Note: Seems like the problem is the handling of the datetime index

How is this conversion performed?
See: https://github.com/spyder-ide/spyder/pull/11210

Resolving patch:
diff --git a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
@@ -124,6 +124,8 @@ def __init__(self, dataFrame, format=DEFAULT_FORMAT, parent=None):
         QAbstractTableModel.__init__(self)
         self.dialog = parent
         self.df = dataFrame
+        self.df_columns_list = None
+        self.df_index_list = None
         self._format = format
         self.complex_intran = None
         self.display_error_idxs = []
@@ -166,6 +168,21 @@ def _axis(self, axis):
         """
         return self.df.columns if axis == 0 else self.df.index
 
+    def _axis_list(self, axis):
+        """
+        Return the corresponding labels as a list taking into account the axis.
+
+        The axis could be horizontal (0) or vertical (1).
+        """
+        if axis == 0:
+            if self.df_columns_list is None:
+                self.df_columns_list = self.df.columns.tolist()
+            return self.df_columns_list
+        else:
+            if self.df_index_list is None:
+                self.df_index_list = self.df.index.tolist()
+            return self.df_index_list
+
     def _axis_levels(self, axis):
         """
         Return the number of levels in the labels taking into account the axis.
@@ -198,8 +215,11 @@ def header(self, axis, x, level=0):
         given level.
         """
         ax = self._axis(axis)
-        return ax.tolist()[x] if not hasattr(ax, 'levels') \
-            else ax.values[x][level]
+        if not hasattr(ax, 'levels'):
+            ax = self._axis_list(axis)
+            return ax[x]
+        else:
+            return ax.values[x][level]
 
     def name(self, axis, level):
         """Return the labels of the levels if any."""


Test patch: