Repository: spyder-ide/spyder

Issue:
Larger dataframes with columns on the far right doesnt display max values when clicking on their column names twice + Other bugs
### Issue Report Checklist

Checklist checked.

## Problem Description
It is hard to describe this problem so I screen-recorded the main problem, together with 1-2 other bugs (which can also been seen in the video). So if i have a larger data frame with many columns, pressing one of the column names will sort the values from min to max, and by pressing again, it will sort the values from max to min. However, if i scroll somewhat to the right (beyond what one can see when opening up a df), i can only sort by min to max, but not max to min. Also,  it takes me back to column 40 (out of 80) and sometimes doesnt let me scroll to the right, unless i scroll to the left and then to the right. However, it is a bit tedious to always scroll back to column 80 again. An improvement beside fixing the max/min bug would be if it just let you stay at column 80 when clicking the column name. 


https://github.com/spyder-ide/spyder/assets/123987501/2e7df699-e80f-4bb3-a97f-77ad812ccae9

### What steps reproduce the problem?
```import pandas as pd
import numpy as np

np.random.seed(42)

num_columns = 80
num_rows = 5000

data = np.random.rand(num_rows, num_columns)
df = pd.DataFrame(data, columns=[f'Column{i+1}' for i in range(num_columns)])
```
### What is the expected output? What do you see instead?
That i can sort from max to min on the later columns

## Versions
<!--- You can get this information from Help > About Spyder...
or (if Spyder won't launch) the "conda list" command
from the Anaconda Prompt/Terminal/command line. --->

* Spyder version: 5.4.5 (Doesnt work in 5.5 either)
* Python version: 3.12
* Qt version: 5.15.2
* PyQt version: 5.15.10
* Operating System name/version: Win10 x64

### Dependencies

```
# Mandatory:
atomicwrites >=1.2.0                                                        :  1.4.1 (OK)
chardet >=2.0.0                                                             :  5.2.0 (OK)
cloudpickle >=0.5.0                                                         :  3.0.0 (OK)
cookiecutter >=1.6.0                                                        :  2.5.0 (OK)
diff_match_patch >=20181111                                                 :  20230430 (OK)
intervaltree >=3.0.2                                                        :  3.1.0 (OK)
IPython >=7.31.1,<9.0.0,!=8.8.0,!=8.9.0,!=8.10.0,!=8.11.0,!=8.12.0,!=8.12.1 :  8.18.1 (OK)
jedi >=0.17.2,<0.19.0                                                       :  0.18.2 (OK)
jellyfish >=0.7                                                             :  1.0.3 (OK)
jsonschema >=3.2.0                                                          :  4.20.0 (OK)
keyring >=17.0.0                                                            :  24.3.0 (OK)
nbconvert >=4.0                                                             :  7.12.0 (OK)
numpydoc >=0.6.0                                                            :  1.6.0 (OK)
paramiko >=2.4.0                                                            :  3.3.1 (OK)
parso >=0.7.0,<0.9.0                                                        :  0.8.3 (OK)
pexpect >=4.4.0                                                             :  4.9.0 (OK)
pickleshare >=0.4                                                           :  0.7.5 (OK)
psutil >=5.3                                                                :  5.9.7 (OK)
pygments >=2.0                                                              :  2.17.2 (OK)
pylint >=2.5.0,<3.0                                                         :  2.17.7 (OK)
pylint_venv >=3.0.2                                                         :  3.0.3 (OK)
pyls_spyder >=0.4.0                                                         :  0.4.0 (OK)
pylsp >=1.7.4,<1.8.0                                                        :  1.7.4 (OK)
pylsp_black >=1.2.0,<3.0.0                                                  :  1.3.0 (OK)
qdarkstyle >=3.0.2,<3.2.0                                                   :  3.1 (OK)
qstylizer >=0.2.2                                                           :  0.2.2 (OK)
qtawesome >=1.2.1                                                           :  1.3.0 (OK)
qtconsole >=5.4.2,<5.5.0                                                    :  5.4.4 (OK)
qtpy >=2.1.0                                                                :  2.4.1 (OK)
rtree >=0.9.7                                                               :  1.1.0 (OK)
setuptools >=49.6.0                                                         :  68.2.2 (OK)
sphinx >=0.6.6                                                              :  7.2.6 (OK)
spyder_kernels >=2.4.4,<2.5.0                                               :  2.4.4 (OK)
textdistance >=4.2.0                                                        :  4.6.0 (OK)
three_merge >=0.1.1                                                         :  0.1.1 (OK)
watchdog >=0.10.3                                                           :  3.0.0 (OK)
zmq >=22.1.0                                                                :  25.1.2 (OK)

# Optional:
cython >=0.21                                                               :  None (NOK)
matplotlib >=3.0.0                                                          :  3.8.2 (OK)
numpy >=1.7                                                                 :  1.26.2 (OK)
pandas >=1.1.1                                                              :  2.1.4 (OK)
scipy >=0.17.0                                                              :  1.11.4 (OK)
sympy >=0.7.3                                                               :  None (NOK)

```



Discussion:
Hi @Chuck321123 thank you for the video and the example code! I think I was able to reproduce this, in fact seems like from column 41 things start to work strangely if you try to use the column header to order entries or even try to resize the column. My guess is that something is happening inside the logic that handles fetching more data when scrolling vs interacting with the new fetched information (since column 40 seems like the last column loaded by default for the example code) 🤔

What do you think @ccordoba12 @jitseniesen ?
I had a look at what this looks like in the upcoming 6.0 release and there is another issue: the scrollbars have disappeared. This bug was introduced when I added the Refresh functionality. I'll have a look at what is going on and see what we can do about both the scrollbars and the issues mentioned in the original report.

Resolving patch:
diff --git a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/dataframeeditor.py
@@ -658,6 +658,10 @@ def __init__(self, parent, model, header, hscroll, vscroll,
         self.resize_action = None
         self.resize_columns_action = None
 
+        self.menu = self.setup_menu()
+        self.menu_header_h = self.setup_menu_header()
+        self.config_shortcut(self.copy, 'copy', self)
+
         self.setModel(model)
         self.setHorizontalScrollBar(hscroll)
         self.setVerticalScrollBar(vscroll)
@@ -671,14 +675,9 @@ def __init__(self, parent, model, header, hscroll, vscroll,
             self.show_header_menu)
         self.header_class.sectionClicked.connect(self.sortByColumn)
         self.data_function = data_function
-        self.menu = self.setup_menu()
-        self.menu_header_h = self.setup_menu_header()
-        self.config_shortcut(self.copy, 'copy', self)
         self.horizontalScrollBar().valueChanged.connect(
             self._load_more_columns)
         self.verticalScrollBar().valueChanged.connect(self._load_more_rows)
-        self.selectionModel().selectionChanged.connect(self.refresh_menu)
-        self.refresh_menu()
 
     def _load_more_columns(self, value):
         """Load more columns to display."""
@@ -713,6 +712,17 @@ def load_more_data(self, value, rows=False, columns=False):
             # See spyder-ide/spyder#7880.
             pass
 
+    def setModel(self, model: DataFrameModel) -> None:
+        """
+        Set the model for the view to present.
+
+        This overrides the function in QTableView so that we can enable or
+        disable actions when appropriate if the selection changes.
+        """
+        super().setModel(model)
+        self.selectionModel().selectionChanged.connect(self.refresh_menu)
+        self.refresh_menu()
+
     def sortByColumn(self, index):
         """Implement a column sort."""
         if self.sort_old == [None]:
@@ -1417,22 +1427,8 @@ def __init__(self, model, axis, use_monospace_font=False):
 
         self.total_rows = self.model.shape[0]
         self.total_cols = self.model.shape[1]
-        size = self.total_rows * self.total_cols
-
-        # Use paging when the total size, number of rows or number of
-        # columns is too large
-        if size > LARGE_SIZE:
-            self.rows_loaded = ROWS_TO_LOAD
-            self.cols_loaded = COLS_TO_LOAD
-        else:
-            if self.total_cols > LARGE_COLS:
-                self.cols_loaded = COLS_TO_LOAD
-            else:
-                self.cols_loaded = self.total_cols
-            if self.total_rows > LARGE_NROWS:
-                self.rows_loaded = ROWS_TO_LOAD
-            else:
-                self.rows_loaded = self.total_rows
+        self.cols_loaded = self.model.cols_loaded
+        self.rows_loaded = self.model.rows_loaded
 
         if self.axis == 0:
             self.total_cols = self.model.shape[1]
@@ -1687,6 +1683,10 @@ def data(self, index, role):
         return None
 
 
+class EmptyDataFrame:
+    shape = (0, 0)
+
+
 class DataFrameEditor(BaseDialog, SpyderWidgetMixin):
     """
     Dialog for displaying and editing DataFrame and related objects.
@@ -1790,9 +1790,23 @@ def setup_ui(self, title: str) -> None:
         # Create menu to allow edit index
         self.menu_header_v = self.setup_menu_header(self.table_index)
 
+        # Create the model and view of the data
+        empty_data = EmptyDataFrame()
+        self.dataModel = DataFrameModel(empty_data, parent=self)
+        self.dataModel.dataChanged.connect(self.save_and_close_enable)
+        self.create_data_table()
+
         self.glayout.addWidget(self.hscroll, 2, 0, 1, 2)
         self.glayout.addWidget(self.vscroll, 0, 2, 2, 1)
 
+        # autosize columns on-demand
+        self._autosized_cols = set()
+
+        # Set limit time to calculate column sizeHint to 300ms,
+        # See spyder-ide/spyder#11060
+        self._max_autosize_ms = 300
+        self.dataTable.installEventFilter(self)
+
         avg_width = self.fontMetrics().averageCharWidth()
         self.min_trunc = avg_width * 12  # Minimum size for columns
         self.max_width = avg_width * 64  # Maximum size for columns
@@ -1853,7 +1867,7 @@ def set_data_and_check(self, data) -> bool:
         # Create the model and view of the data
         self.dataModel = DataFrameModel(data, parent=self)
         self.dataModel.dataChanged.connect(self.save_and_close_enable)
-        self.create_data_table()
+        self.dataTable.setModel(self.dataModel)
 
         # autosize columns on-demand
         self._autosized_cols = set()
@@ -2034,11 +2048,6 @@ def create_table_index(self):
 
     def create_data_table(self):
         """Create the QTableView that will hold the data model."""
-        if self.dataTable:
-            self.layout.removeWidget(self.dataTable)
-            self.dataTable.deleteLater()
-            self.dataTable = None
-
         self.dataTable = DataFrameView(self, self.dataModel,
                                        self.table_header.horizontalHeader(),
                                        self.hscroll, self.vscroll,


Test patch:
diff --git a/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py b/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
--- a/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
+++ b/spyder/plugins/variableexplorer/widgets/tests/test_dataframeeditor.py
@@ -34,7 +34,7 @@
 from spyder.utils.test import close_message_box
 from spyder.plugins.variableexplorer.widgets import dataframeeditor
 from spyder.plugins.variableexplorer.widgets.dataframeeditor import (
-    DataFrameEditor, DataFrameModel)
+    DataFrameEditor, DataFrameModel, COLS_TO_LOAD, LARGE_COLS)
 
 
 # =============================================================================
@@ -150,6 +150,77 @@ def test_dataframe_to_type(qtbot):
     assert editor.btn_save_and_close.isEnabled()
 
 
+def test_dataframe_editor_shows_scrollbar(qtbot):
+    """
+    Test the dataframe editor shows a scrollbar when opening a large dataframe.
+    Regression test for spyder-ide/spyder#21627 .
+    """
+    df = DataFrame(numpy.zeros((100, 100)))
+    editor = DataFrameEditor()
+    editor.setup_and_check(df)
+    with qtbot.waitExposed(editor):
+        editor.show()
+
+    assert editor.dataTable.horizontalScrollBar().isVisible()
+
+
+def test_dataframe_editor_scroll(qtbot):
+    """
+    Test that when opening a "large" dataframe, only a part of it is initially
+    loaded in the editor window. When scrolling past that part, the rest is
+    loaded. When moving to the right-most column and sorting it, the view
+    stay scrolled to the right end.
+
+    Regression test for spyder-ide/spyder#21627 .
+    """
+
+    # Make DataFrame with LARGE_COLS + 5 columns
+    df = DataFrame(numpy.zeros((10, LARGE_COLS + 5)))
+    editor = DataFrameEditor()
+    editor.setup_and_check(df)
+    model = editor.dataModel
+    with qtbot.waitExposed(editor):
+        editor.show()
+
+    # Check that initially, COLS_TO_LOAD columns are loaded in the editor
+    assert model.rowCount() == 10
+    assert model.columnCount() == COLS_TO_LOAD
+
+    # Press the End key to move to the right and wait
+    view = editor.dataTable
+    view.setCurrentIndex(view.model().index(0, 0))
+    qtbot.keyPress(view, Qt.Key_End)
+
+    # Check that now all the columns are loaded in the editor
+    def check_column_count():
+        assert model.columnCount() == LARGE_COLS + 5
+
+    qtbot.waitUntil(check_column_count)
+
+    # Press the End key to move to the right and wait
+    qtbot.keyPress(view, Qt.Key_End)
+    scrollbar = editor.dataTable.horizontalScrollBar()
+
+    # Check that we are at the far right
+    def check_at_far_right():
+        assert scrollbar.value() == scrollbar.maximum()
+
+    qtbot.waitUntil(check_at_far_right)
+
+    # Sort the rightmost column
+    old_index_model = editor.table_index.model()
+    view.sortByColumn(model.columnCount() - 1)
+
+    # Wait until the model for the index is updated
+    def check_index_model_updated():
+        assert editor.table_index.model() != old_index_model
+
+    qtbot.waitUntil(check_index_model_updated)
+
+    # Check that we are at the far right
+    assert scrollbar.value() == scrollbar.maximum()
+
+
 def test_dataframe_datetimeindex(qtbot):
     """Regression test for spyder-ide/spyder#11129 ."""
     ds = Series(