Repository: spyder-ide/spyder

Issue:
DataFrame editor should first sort up, then down
## Description of your problem

When clicking a column header in the DataFrame editor, the column is sorted from high to low. A second click sorts it from low to high. Since sorting from low to high is needed much more often, the effects of the first and the second click should be reversed.

**What steps will reproduce the problem?**
1. Open a Pandas DataFrame in the editor.
2. Click on a column header.

**What is the expected output?**
The DataFrame column is sorted s.t. the lowest value is on top.

**What do you see instead?**
The DataFrame column is sorted s.t. the lowest value is on bottom.

**Please provide any additional information below**
N/A
## Versions and main components
- Spyder Version: 2.3.5.2 from WinPython-64bit-3.4.3.5.
- Python Version: 3.4.3
- Operating system: Windows 7 64-bit
## Dependencies

Please go to the menu entry `Help > Optional Dependencies` (or
`Help > Dependencies`), press the button `Copy to clipboard`
and paste the contents below:

IPython >=1.0   :  3.2.1 (OK)
jedi >=0.8.1    :  0.9.0 (OK)
matplotlib >=1.0:  1.4.3 (OK)
pandas >=0.13.1 :  0.16.2 (OK)
pep8 >=0.6      :  1.6.2 (OK)
pyflakes >=0.6.0:  0.9.2 (OK)
pygments >=1.6  :  2.0.2 (OK)
pylint >=0.25   :  1.4.3 (OK)
rope >=0.9.2    :  0.9.4-1 (OK)
sphinx >=0.6.6  :  1.3.1 (OK)
sympy >=0.7.3   :  0.7.6 (OK)
zmq >=2.1.11    :  14.7.0 (OK)

DataFrame editor should do a stable sort
I am using Spyder 2.3.5.2 from WinPython-64bit-3.4.3.5.

I noticed that clicking column headers in the DataFrame editor does not sort the table [in a stable way](https://en.wikipedia.org/wiki/Sorting_algorithm#Stability). This especially means that one cannot reach a hierarchical sorting.



Discussion:

I don't understand what you mean. Please post examples about what problem are you seeing and what's the expected output, as it's explicitly demanded in our issue template.

Also, for next time, please don't dismiss the template contents. That saves us a lot of time.

> please don't dismiss the template contents

You are right. Strangely, I cannot come up with a small example. I will reopen this when I find an example again.

Ok, I've got it now:
- Define `df = pd.DataFrame([[2,14],[2,13],[2,16],[1,3],[2,9],[1,15],[1,17],[2,2],[2,10],[1,6],[2,5],[2,8],[1,11],[1,1],[1,12],[1,4],[2,7]])`
- Open df in the DataFrame editor.
- Try to sort hierarchically. For this, click the header of column '1', then the header of column '0'.

Expected result: df is hierarchically sorted by the columns '0' and '1'.

Observed result: Within the value '2' of column '0', the values of column '1' have become unsorted again.

Interestingly, the hierarchical sorting works for smaller DataFrames, e.g. when `[1,17]` is removed from the above example.

I think this is quite easy to implement by using telling DataFrame.sort to use mergesort (which is stable) in the dataframe editor. A potential problem is that mergesort has a higher memory cost according to the [numpy manual](http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.sort.html), but I don't think that's likely to be an issue for us.

PR coming? ;-)

@ccordoba12 Admit it, it's an automated answer ! ;o)


Resolving patch:
diff --git a/spyderlib/widgets/variableexplorer/dataframeeditor.py b/spyderlib/widgets/variableexplorer/dataframeeditor.py
--- a/spyderlib/widgets/variableexplorer/dataframeeditor.py
+++ b/spyderlib/widgets/variableexplorer/dataframeeditor.py
@@ -269,12 +269,20 @@ def sort(self, column, order=Qt.AscendingOrder):
                                      "relation is defined for complex numbers")
                 return False
         try:
+            ascending = order == Qt.AscendingOrder
             if column > 0:
-                self.df.sort(columns=self.df.columns[column-1],
-                             ascending=order, inplace=True)
+                try:
+                    self.df.sort_values(by=self.df.columns[column-1],
+                                        ascending=ascending, inplace=True,
+                                        kind='mergesort')
+                except AttributeError:
+                    # for pandas version < 0.17
+                    self.df.sort(columns=self.df.columns[column-1],
+                                 ascending=ascending, inplace=True,
+                                 kind='mergesort')
                 self.update_df_index()
             else:
-                self.df.sort_index(inplace=True, ascending=order)
+                self.df.sort_index(inplace=True, ascending=ascending)
                 self.update_df_index()
         except TypeError as e:
             QMessageBox.critical(self.dialog, "Error",


Test patch:
diff --git a/spyderlib/widgets/variableexplorer/tests/test_dataframeeditor.py b/spyderlib/widgets/variableexplorer/tests/test_dataframeeditor.py
new file mode 100644
--- /dev/null
+++ b/spyderlib/widgets/variableexplorer/tests/test_dataframeeditor.py
@@ -0,0 +1,57 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright © 2009- The Spyder Development Team
+# Licensed under the terms of the MIT License
+
+"""
+Tests for dataframeeditor.py
+"""
+
+# Third party imports
+from pandas import DataFrame
+import pytest
+
+# Local imports
+from spyderlib.widgets.variableexplorer.dataframeeditor import DataFrameModel
+
+
+# --- Tests
+# -----------------------------------------------------------------------------
+def test_dataframemodel_basic():
+    df = DataFrame({'colA': [1, 3], 'colB': ['c', 'a']})
+    dfm = DataFrameModel(df)
+    assert dfm.rowCount() == 2
+    assert dfm.columnCount() == 3
+    assert dfm.data(dfm.createIndex(0, 0)) == '0'
+    assert dfm.data(dfm.createIndex(0, 1)) == '1'
+    assert dfm.data(dfm.createIndex(0, 2)) == 'c'
+    assert dfm.data(dfm.createIndex(1, 0)) == '1'
+    assert dfm.data(dfm.createIndex(1, 1)) == '3'
+    assert dfm.data(dfm.createIndex(1, 2)) == 'a'
+    
+def test_dataframemodel_sort():
+    df = DataFrame({'colA': [1, 3], 'colB': ['c', 'a']})
+    dfm = DataFrameModel(df)
+    dfm.sort(2)
+    assert dfm.data(dfm.createIndex(0, 0)) == '1'
+    assert dfm.data(dfm.createIndex(0, 1)) == '3'
+    assert dfm.data(dfm.createIndex(0, 2)) == 'a'
+    assert dfm.data(dfm.createIndex(1, 0)) == '0'
+    assert dfm.data(dfm.createIndex(1, 1)) == '1'
+    assert dfm.data(dfm.createIndex(1, 2)) == 'c'
+
+def test_dataframemodel_sort_is_stable():   # cf. issue 3010
+    df = DataFrame([[2,14], [2,13], [2,16], [1,3], [2,9], [1,15], [1,17],
+                    [2,2], [2,10], [1,6], [2,5], [2,8], [1,11], [1,1],
+                    [1,12], [1,4], [2,7]])
+    dfm = DataFrameModel(df)
+    dfm.sort(2)
+    dfm.sort(1)
+    col2 = [dfm.data(dfm.createIndex(i, 2)) for i in range(len(df))]
+    assert col2 == [str(x) for x in [1, 3, 4, 6, 11, 12, 15, 17,
+                                     2, 5, 7, 8, 9, 10, 13, 14, 16]]
+
+
+if __name__ == "__main__":
+    pytest.main()
+