Repository: spyder-ide/spyder

Issue:
Sorting by column in variable explorer
## Description

**What steps will reproduce the problem?**

1.  Double click on a pandas Series variable in Variable Explorer
2.  Clicked on the column heading for the Series data in the worksheet containing the Series values, to sort it
3.  Results (repeatedly) in error below

**What is the expected output? What do you see instead?**


**Please provide any additional information below**

  File "D:\Users\803375527\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\dataframeeditor.py", line 562, in sortByColumn
    if not self.model().sort(index, sort_order):
  File "D:\Users\803375527\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\dataframeeditor.py", line 312, in sort
    kind='mergesort')
  File "D:\Users\803375527\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\frame.py", line 3201, in sort_values
    na_position=na_position)
  File "D:\Users\803375527\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\sorting.py", line 247, in nargsort
    indexer = non_nan_idx[non_nans.argsort(kind=kind)]
SystemError: <method-wrapper '__gt__' of CategoricalDtype object at 0x000001A5AF1A0B70> returned a result with an error set


## Version and main components

* Spyder Version: 3.2.3 
* Python Version: 3.6.2
* Qt Versions: 5.6.2, PyQt5 5.6 on Windows

## Dependencies
```
pyflakes >=0.6.0 :  1.5.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.20.3 (OK)
numpy >=1.7      :  1.13.1 (OK)
sphinx >=0.6.6   :  1.6.3 


Discussion:
Thanks @lifescholar. Do you have a simple example to reproduce this issue? 
Sure. Looks like a problem with categorical data types. The following code results in this error when you click on the column heading for the series in the variable explorer.

```python
import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3, 4],
                   'B': ['a', 'b', 'c', 'd']})
df = df.astype(dtype={'B': 'category'})
df_cols = df.dtypes
```
Thanks @lifescholar. I will investigate this issue. 
Thanks @Prikers for looking into it!

Just one word of advice about our milestones: if an issue is tagged with a `3.2.x` milestone, that means that a PR to fix it needs to be created against our `3.x` branch. Else, they need to go against master.

You most probably noticed that, but I wanted to clarify it for you ;-)
Just to clarify, the error occurs when inspecting df_cols, not df, in my code example.
@lifescholar This seems to be an error in pandas. Probably related to [this issue](https://github.com/pandas-dev/pandas/issues/14368). 
@ccordoba12 should we add an error message as we did in #5327? It is a `SystemError` and seems to be a very specific error.
@ccordoba12 yes I noticed about the branches 😄. For my defense the only time I mixed-up the branches, there was no label on the issue 😇 
> should we add an error message as we did in #5327?

Yes, please.

> For my defense the only time I mixed-up the branches, there was no label on the issue

Don't worry about it, my comment was just to let things crystal clear for you ;-)

Resolving patch:
diff --git a/spyder/widgets/variableexplorer/dataframeeditor.py b/spyder/widgets/variableexplorer/dataframeeditor.py
--- a/spyder/widgets/variableexplorer/dataframeeditor.py
+++ b/spyder/widgets/variableexplorer/dataframeeditor.py
@@ -319,6 +319,10 @@ def sort(self, column, order=Qt.AscendingOrder):
                     # Not possible to sort on duplicate columns #5225
                     QMessageBox.critical(self.dialog, "Error",
                                          "ValueError: %s" % to_text_string(e))
+                except SystemError as e:
+                    # Not possible to sort on category dtypes #5361
+                    QMessageBox.critical(self.dialog, "Error",
+                                         "SystemError: %s" % to_text_string(e))
                 self.update_df_index()
             else:
                 self.df.sort_index(inplace=True, ascending=ascending)


Test patch:
diff --git a/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py b/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
--- a/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
+++ b/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
@@ -246,5 +246,19 @@ def test_sort_dataframe_with_duplicate_column(qtbot):
     assert [data(dfm, row, 2) for row in range(len(df))] == ['4', '5', '6']
 
 
+def test_sort_dataframe_with_category_dtypes(qtbot):  # cf. issue 5361
+    df = DataFrame({'A': [1, 2, 3, 4],
+                    'B': ['a', 'b', 'c', 'd']})
+    df = df.astype(dtype={'B': 'category'})
+    df_cols = df.dtypes
+    editor = DataFrameEditor(None)
+    editor.setup_and_check(df_cols)
+    dfm = editor.dataModel
+    QTimer.singleShot(1000, lambda: close_message_box(qtbot))
+    editor.dataModel.sort(1)
+    assert data(dfm, 0, 1) == 'int64'
+    assert data(dfm, 1, 1) == 'category'
+
+
 if __name__ == "__main__":
     pytest.main()