Repository: spyder-ide/spyder

Issue:
DataFrame Editor raises TypeError when editing any complex type
## Description

When editing a column of a complex type (``complex64``, ``complex128`` and the built in ``complex``), the DataFrame Editor raises a ``TypeError`` when attempting to confirm/commit any edits to any of the values in the column. This seems to occur regardless of what value is entered, whether the original (displayed as “###j), a tuple, a single integer, etc. However, it works just fine for arrays, so perhaps that logic can just be ported over?

**What steps will reproduce the problem?**

1. Create a DataFrame with at least one column being of a complex type. E.g.
```
import pandas as pd
import numpy as np
df = pd.DataFrame([0, 0, 0], dtype=np.complex128)
```
2. Open the DataFrame in the Data Frame editor
3. Doubleclick one of the complex values to edit and click away again to commit

**What is the expected output? What do you see instead?**

Expected: Either a functioning ability to input a complex value as works perfectly well for arrays, or at the very least a "The type of the cell is not a supported type" warning or just setting the cell to read only.

Instead, an error dialog with :

```
File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\dataframeeditor.py", line 368, in setData
self.df.iloc[row, column-1] = current_value.__class__(val)
TypeError: must be real number, not str
```

**Proposed solution**

Port whatever logic is currently used for ArrayEditor to handle working with complex types over to the DataFrame Editor, if that's practicable. If not, and there's no other straightforward way to fix this, just catch the exception and pop up a "The type of the cell is not a supported type" warning dialog on attempting to edit instead for the time being, or disable editing completely if possible (so a double click action is just ignored).


## Version and main components

* Spyder Version: 3.2.5 
* Python Version: 3.6.3
* Qt Versions: 5.6.2, PyQt5 5.6 on Windows

## Dependencies
```
pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.21.1 (OK)
numpy >=1.7      :  1.13.3 (OK)
sphinx >=0.6.6   :  1.6.3 (OK)
rope >=0.9.4     :  0.10.5 (OK)
jedi >=0.9.0     :  0.10.2 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
cython >=0.21    :  0.26.1 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.1.0 (OK)
pylint >=0.25    :  1.7.4 (OK)

```

DataFrame Editor raises TypeError when editing any complex type
## Description

When editing a column of a complex type (``complex64``, ``complex128`` and the built in ``complex``), the DataFrame Editor raises a ``TypeError`` when attempting to confirm/commit any edits to any of the values in the column. This seems to occur regardless of what value is entered, whether the original (displayed as “###j), a tuple, a single integer, etc. However, it works just fine for arrays, so perhaps that logic can just be ported over?

**What steps will reproduce the problem?**

1. Create a DataFrame with at least one column being of a complex type. E.g.
```
import pandas as pd
import numpy as np
df = pd.DataFrame([0, 0, 0], dtype=np.complex128)
```
2. Open the DataFrame in the Data Frame editor
3. Doubleclick one of the complex values to edit and click away again to commit

**What is the expected output? What do you see instead?**

Expected: Either a functioning ability to input a complex value as works perfectly well for arrays, or at the very least a "The type of the cell is not a supported type" warning or just setting the cell to read only.

Instead, an error dialog with :

```
File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\dataframeeditor.py", line 368, in setData
self.df.iloc[row, column-1] = current_value.__class__(val)
TypeError: must be real number, not str
```

**Proposed solution**

Port whatever logic is currently used for ArrayEditor to handle working with complex types over to the DataFrame Editor, if that's practicable. If not, and there's no other straightforward way to fix this, just catch the exception and pop up a "The type of the cell is not a supported type" warning dialog on attempting to edit instead for the time being, or disable editing completely if possible (so a double click action is just ignored).


## Version and main components

* Spyder Version: 3.2.5 
* Python Version: 3.6.3
* Qt Versions: 5.6.2, PyQt5 5.6 on Windows

## Dependencies
```
pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.21.1 (OK)
numpy >=1.7      :  1.13.3 (OK)
sphinx >=0.6.6   :  1.6.3 (OK)
rope >=0.9.4     :  0.10.5 (OK)
jedi >=0.9.0     :  0.10.2 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
cython >=0.21    :  0.26.1 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.1.0 (OK)
pylint >=0.25    :  1.7.4 (OK)

```



Discussion:
Agreed with the proposed solution. I think someone fixed this for arrays a while ago.
Agreed with the proposed solution. I think someone fixed this for arrays a while ago.

Resolving patch:
diff --git a/spyder/widgets/variableexplorer/dataframeeditor.py b/spyder/widgets/variableexplorer/dataframeeditor.py
--- a/spyder/widgets/variableexplorer/dataframeeditor.py
+++ b/spyder/widgets/variableexplorer/dataframeeditor.py
@@ -42,7 +42,7 @@
 REAL_NUMBER_TYPES = (float, int, np.int64, np.int32)
 COMPLEX_NUMBER_TYPES = (complex, np.complex64, np.complex128)
 # Used to convert bool intrance to false since bool('False') will return True
-_bool_false = ['false', '0']
+_bool_false = ['false', 'f', '0', '0.', '0.0', ' ']
 
 # Default format for data frames with floats
 DEFAULT_FORMAT = '%.6g'
@@ -359,9 +359,9 @@ def setData(self, index, value, role=Qt.EditRole, change_type=None):
         else:
             val = from_qvariant(value, str)
             current_value = self.get_value(row, column-1)
-            if isinstance(current_value, bool):
+            if isinstance(current_value, (bool, np.bool_)):
                 val = bool_false_check(val)
-            supported_types = (bool,) + REAL_NUMBER_TYPES + COMPLEX_NUMBER_TYPES
+            supported_types = (bool, np.bool_) + REAL_NUMBER_TYPES
             if (isinstance(current_value, supported_types) or
                     is_text_string(current_value)):
                 try:
@@ -372,8 +372,8 @@ def setData(self, index, value, role=Qt.EditRole, change_type=None):
                     return False
             else:
                 QMessageBox.critical(self.dialog, "Error",
-                                     "The type of the cell is not a supported "
-                                     "type")
+                                     "Editing dtype {0!s} not yet supported."
+                                     .format(type(current_value).__name__))
                 return False
         self.max_min_col_update()
         return True


Test patch:
diff --git a/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py b/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
--- a/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
+++ b/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
@@ -292,9 +292,8 @@ def test_dataframemodel_set_data_overflow(monkeypatch):
 
 
 @flaky(max_runs=3)
-@pytest.mark.skipif(os.environ.get('CI', None) is not None or
-                    platform.startswith('linux'),
-                    reason="Fails on Travis CI for unknown reasons.")
+@pytest.mark.skipif(platform.startswith('linux'),
+                    reason="Fails on some Linux platforms locally and Travis.")
 def test_dataframeeditor_edit_overflow(qtbot, monkeypatch):
     """Test #6114: entry of an overflow int is caught and handled properly"""
     MockQMessageBox = Mock()
@@ -323,6 +322,7 @@ def test_dataframeeditor_edit_overflow(qtbot, monkeypatch):
         qtbot.keyClicks(view, '5')
         qtbot.keyPress(view, Qt.Key_Down)
         qtbot.keyPress(view, Qt.Key_Space)
+        qtbot.keyPress(view.focusWidget(), Qt.Key_Backspace)
         qtbot.keyClicks(view.focusWidget(), str(int(2 ** bit_exponet)))
         qtbot.keyPress(view.focusWidget(), Qt.Key_Down)
         MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
@@ -336,5 +336,125 @@ def test_dataframeeditor_edit_overflow(qtbot, monkeypatch):
                          dialog.get_value().as_matrix()) == len(expected_df)
 
 
+def test_dataframemodel_set_data_complex(monkeypatch):
+    """Unit test #6115: editing complex dtypes raises error in df editor"""
+    MockQMessageBox = Mock()
+    attr_to_patch = ('spyder.widgets.variableexplorer' +
+                     '.dataframeeditor.QMessageBox')
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    test_params = [(1, numpy.complex128), (2, numpy.complex64), (3, complex)]
+
+    for count, complex_type in test_params:
+        test_df = DataFrame(numpy.arange(10, 15), dtype=complex_type)
+        model = DataFrameModel(test_df.copy())
+        index = model.createIndex(2, 1)
+        assert not model.setData(index, '42')
+        MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
+        assert MockQMessageBox.critical.call_count == count
+        assert numpy.sum(test_df[0].as_matrix() ==
+                         model.df.as_matrix()) == len(test_df)
+
+
+@flaky(max_runs=3)
+@pytest.mark.skipif(os.environ.get('CI', None) is not None and
+                    platform.startswith('linux'),
+                    reason="Fails on Travis for no good reason.")
+def test_dataframeeditor_edit_complex(qtbot, monkeypatch):
+    """Test for #6115: editing complex dtypes raises error in df editor"""
+    MockQMessageBox = Mock()
+    attr_to_patch = ('spyder.widgets.variableexplorer' +
+                     '.dataframeeditor.QMessageBox')
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    test_params = [(1, numpy.complex128), (2, numpy.complex64), (3, complex)]
+
+    for count, complex_type in test_params:
+        test_df = DataFrame(numpy.arange(10, 15), dtype=complex_type)
+        dialog = DataFrameEditor()
+        assert dialog.setup_and_check(test_df, 'Test Dataframe')
+        dialog.show()
+        qtbot.waitForWindowShown(dialog)
+        view = dialog.dataTable
+
+        qtbot.keyPress(view, Qt.Key_Right)
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyPress(view, Qt.Key_Space)
+        qtbot.keyPress(view.focusWidget(), Qt.Key_Backspace)
+        qtbot.keyClicks(view.focusWidget(), "42")
+        qtbot.keyPress(view.focusWidget(), Qt.Key_Down)
+        MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
+        assert MockQMessageBox.critical.call_count == count * 2 - 1
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyClick(view, '1')
+        qtbot.keyPress(view.focusWidget(), Qt.Key_Down)
+        MockQMessageBox.critical.assert_called_with(
+            ANY, "Error", ("Editing dtype {0!s} not yet supported."
+                           .format(type(test_df.iloc[1, 0]).__name__)))
+        assert MockQMessageBox.critical.call_count == count * 2
+        qtbot.keyPress(view, Qt.Key_Return)
+        qtbot.wait(1000)
+        assert numpy.sum(test_df[0].as_matrix() ==
+                         dialog.get_value().as_matrix()) == len(test_df)
+
+
+def test_dataframemodel_set_data_bool(monkeypatch):
+    """Unit test that bools are editible in df and false-y strs are detected"""
+    MockQMessageBox = Mock()
+    attr_to_patch = ('spyder.widgets.variableexplorer' +
+                     '.dataframeeditor.QMessageBox')
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    test_params = [numpy.bool_, numpy.bool, bool]
+    test_strs = ['foo', 'false', 'f', '0', '0.', '0.0', '', ' ']
+    expected_df = DataFrame([1, 0, 0, 0, 0, 0, 0, 0, 0], dtype=bool)
+
+    for bool_type in test_params:
+        test_df = DataFrame([0, 1, 1, 1, 1, 1, 1, 1, 0], dtype=bool_type)
+        model = DataFrameModel(test_df.copy())
+        for idx, test_str in enumerate(test_strs):
+            assert model.setData(model.createIndex(idx, 1), test_str)
+            assert not MockQMessageBox.critical.called
+        assert numpy.sum(expected_df[0].as_matrix() ==
+                         model.df.as_matrix()[:, 0]) == len(expected_df)
+
+
+@flaky(max_runs=3)
+@pytest.mark.skipif(os.environ.get('CI', None) is not None and
+                    platform.startswith('linux'),
+                    reason="Fails on Travis for no good reason.")
+def test_dataframeeditor_edit_bool(qtbot, monkeypatch):
+    """Unit test that bools are editible in df and false-y strs are detected"""
+    MockQMessageBox = Mock()
+    attr_to_patch = ('spyder.widgets.variableexplorer' +
+                     '.dataframeeditor.QMessageBox')
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    test_params = [numpy.bool_, numpy.bool, bool]
+    test_strs = ['foo', 'false', 'f', '0', '0.', '0.0', '', ' ']
+    expected_df = DataFrame([1, 0, 0, 0, 0, 0, 0, 0, 0], dtype=bool)
+
+    for bool_type in test_params:
+        test_df = DataFrame([0, 1, 1, 1, 1, 1, 1, 1, 0], dtype=bool_type)
+        dialog = DataFrameEditor()
+        assert dialog.setup_and_check(test_df, 'Test Dataframe')
+        dialog.show()
+        qtbot.waitForWindowShown(dialog)
+        view = dialog.dataTable
+
+        qtbot.keyPress(view, Qt.Key_Right)
+        for test_str in test_strs:
+            qtbot.keyPress(view, Qt.Key_Space)
+            qtbot.keyPress(view.focusWidget(), Qt.Key_Backspace)
+            qtbot.keyClicks(view.focusWidget(), test_str)
+            qtbot.keyPress(view.focusWidget(), Qt.Key_Down)
+            assert not MockQMessageBox.critical.called
+        qtbot.keyPress(view, Qt.Key_Return)
+        qtbot.wait(1000)
+        assert (numpy.sum(expected_df[0].as_matrix() ==
+                          dialog.get_value().as_matrix()[:, 0]) ==
+                len(expected_df))
+
+
 if __name__ == "__main__":
     pytest.main()