Repository: spyder-ide/spyder

Issue:
Fix errors when entering a value for a cell larger than the max int in Array and Dataframe editors
## Description

Upon entering/pasting a value into a cell in a dataframe or array editor opened from Variable Explorer that is large enough to cause an overflow of that type, Spyder doesn't catch the resulting `OverflowError` and handle it appropriately, instead displaying an error dialog when the cell is clicked off to commit the value (which is, of course, left unchanged). Additionally, for arrays, something in the exception handling isn't right, such that this situation crashes the entire program with the standard "Python has stopped working error"...*except* if ``SPYDER_DEBUG=3``, in which case the error message is just printed to the shell spyder is started from and everything else proceeds normally.

**What steps will reproduce the problem?**

1. Create a pandas DataFrame of type ``int32`` or ``int64`` (other integer types aren't supported at all), e.g. with
```
import pandas as pd
df = pd.DataFrame([0, 0, 0], dtype=np.int32)
```
or an array, with e.g.
```
import numpy as np
test_array = np.array([0, 0, 0], dtype=np.int32)
``` 
2. Open the dataframe or array from Variable Explorer in a DataFrame/Array Editor 
3. Paste in a value larger than the data type can hold, e.g. ``17179869184`` for ``int32`` or ``73786976294838206464`` for ``int64``, and hit enter or click off.

**What is the expected output? What do you see instead?**

Expected: No hard crash with arrays, and a popup warning dialog similar to the behavior entering a text string in a float column.


When entering a value too large for the current type (int32) but not larger than 64 bit (int64) in an array:
 
```
Traceback (most recent call last):
  File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 318, in setData
    self.test_array[0] = val # will raise an Exception eventually
OverflowError: Python int too large to convert to C long
```
 
When entering a value greater than an int64 into an int64 cell in an array:
 
```
Traceback (most recent call last):
  File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 318, in setData
    self.test_array[0] = val # will raise an Exception eventually
OverflowError: int too big to convert
```
 
Further, both cases then print the following (followed by a hard crash, “Python has stopped working” etc., except in debug mode (SPYDER_DEBUG=3), where everything seems to work fine other than printing the error to the debug console, and of course not actually updating the array with the new value):
 
```
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 320, in setData
    print(type(e.message))  # spyder: test-skip
AttributeError: 'OverflowError' object has no attribute 'message'
```
 
 
 
For Dataframes, similar occurs but the error is caught by Spyder before it crashes.
 
When entering a value too large for the current type (int32) but not larger than 64 bit (int64) in a df:
 
```
  File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\dataframeeditor.py", line 368, in setData
    self.df.iloc[row, column-1] = current_value.__class__(val)
OverflowError: Python int too large to convert to C long
```
 
When entering a value greater than an int64 into an int64 cell in a df:
 
```
File "C:\Users\christopher.gerlach\Anaconda3\lib\site-packages\spyder\widgets\variableexplorer\dataframeeditor.py", line 368, in setData
self.df.iloc[row, column-1] = current_value.__class__(val)
OverflowError: int too big to convert
```

**Please provide any additional information below**

**Proposed Solution:**

First of all, fix the exception handling that allows the application hard crash to occur. Then, catch the error in each case, and generate a popup warning dialog with the error similar to the behavior when entering e.g. a value that cannot be converted to the specified data type, like a text string in a numeric column. Ideally, both this case and the “cannot convert” one mentioned should abort while leaving the user with the edit-mode text field with the value they entered so they can more easily correct it, rather than just kicking them out to non-edit mode, showing the previous value, but this would probably take more work than the current behavior.
 

## Version and main components

* Spyder Version: 3.2.5 
* Python Version: 3.6.3
* Qt Versions: 5.6.2, PyQt5 5.6 on Windows

## Dependencies
```
pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.21.1 (OK)
numpy >=1.7      :  1.13.3 (OK)
sphinx >=0.6.6   :  1.6.3 (OK)
rope >=0.9.4     :  0.10.5 (OK)
jedi >=0.9.0     :  0.10.2 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
cython >=0.21    :  0.26.1 (OK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.1.0 (OK)
pylint >=0.25    :  1.7.4 (OK)

```

  


Discussion:
I agree with the proposed solution Please try to fix it for 3.2.7 because this is a critical bug.

Resolving patch:
diff --git a/spyder/widgets/variableexplorer/arrayeditor.py b/spyder/widgets/variableexplorer/arrayeditor.py
--- a/spyder/widgets/variableexplorer/arrayeditor.py
+++ b/spyder/widgets/variableexplorer/arrayeditor.py
@@ -315,13 +315,13 @@ def setData(self, index, value, role=Qt.EditRole):
                                      "Value error: %s" % str(e))
                 return False
         try:
-            self.test_array[0] = val # will raise an Exception eventually
+            self.test_array[0] = val  # will raise an Exception eventually
         except OverflowError as e:
-            print(type(e.message))  # spyder: test-skip
+            print("OverflowError: " + str(e))  # spyder: test-skip
             QMessageBox.critical(self.dialog, "Error",
-                                 "Overflow error: %s" % e.message)
+                                 "Overflow error: %s" % str(e))
             return False
-        
+
         # Add change to self.changes
         self.changes[(i, j)] = val
         self.dataChanged.emit(index, index)
diff --git a/spyder/widgets/variableexplorer/dataframeeditor.py b/spyder/widgets/variableexplorer/dataframeeditor.py
--- a/spyder/widgets/variableexplorer/dataframeeditor.py
+++ b/spyder/widgets/variableexplorer/dataframeeditor.py
@@ -362,13 +362,13 @@ def setData(self, index, value, role=Qt.EditRole, change_type=None):
             if isinstance(current_value, bool):
                 val = bool_false_check(val)
             supported_types = (bool,) + REAL_NUMBER_TYPES + COMPLEX_NUMBER_TYPES
-            if (isinstance(current_value, supported_types) or 
+            if (isinstance(current_value, supported_types) or
                     is_text_string(current_value)):
                 try:
                     self.df.iloc[row, column-1] = current_value.__class__(val)
-                except ValueError as e:
+                except (ValueError, OverflowError) as e:
                     QMessageBox.critical(self.dialog, "Error",
-                                         "Value error: %s" % str(e))
+                                         str(type(e).__name__) + ": " + str(e))
                     return False
             else:
                 QMessageBox.critical(self.dialog, "Error",


Test patch:
diff --git a/spyder/widgets/variableexplorer/tests/test_arrayeditor.py b/spyder/widgets/variableexplorer/tests/test_arrayeditor.py
--- a/spyder/widgets/variableexplorer/tests/test_arrayeditor.py
+++ b/spyder/widgets/variableexplorer/tests/test_arrayeditor.py
@@ -7,17 +7,23 @@
 Tests for arrayeditor.py
 """
 
-# Stdlib imports
+# Standard library imports
 import os
+from sys import platform
+try:
+    from unittest.mock import Mock, ANY
+except ImportError:
+    from mock import Mock, ANY  # Python 2
 
 # Third party imports
 import numpy as np
 from numpy.testing import assert_array_equal
 import pytest
 from qtpy.QtCore import Qt
+from flaky import flaky
 
 # Local imports
-from spyder.widgets.variableexplorer.arrayeditor import ArrayEditor
+from spyder.widgets.variableexplorer.arrayeditor import ArrayEditor, ArrayModel
 
 
 def launch_arrayeditor(data, title="", xlabels=None, ylabels=None):
@@ -179,5 +185,70 @@ def test_arrayeditor_edit_2d_array(qtbot):
     assert np.sum(diff_arr != dlg.get_value()) == 2
 
 
+def test_arraymodel_set_data_overflow(monkeypatch):
+    """Unit test #6114: entry of an overflow int caught and handled properly"""
+    MockQMessageBox = Mock()
+    attr_to_patch = 'spyder.widgets.variableexplorer.arrayeditor.QMessageBox'
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    # Numpy doesn't raise OverflowError on Linux for ints smaller than 64 bits
+    if platform.startswith('linux'):
+        int32_bit_exponent = 66
+    else:
+        int32_bit_exponent = 34
+    test_parameters = [(1, np.int32, int32_bit_exponent), (2, np.int64, 66)]
+
+    for idx, int_type, bit_exponent in test_parameters:
+        test_array = np.array([[5], [6], [7], [3], [4]], dtype=int_type)
+        model = ArrayModel(test_array.copy())
+        index = model.createIndex(0, 2)
+        assert not model.setData(index, str(int(2 ** bit_exponent)))
+        MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
+        assert MockQMessageBox.critical.call_count == idx
+        assert np.sum(test_array == model._data) == len(test_array)
+
+
+@flaky(max_runs=3)
+def test_arrayeditor_edit_overflow(qtbot, monkeypatch):
+    """Int. test #6114: entry of an overflow int caught and handled properly"""
+    MockQMessageBox = Mock()
+    attr_to_patch = 'spyder.widgets.variableexplorer.arrayeditor.QMessageBox'
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    # Numpy doesn't raise the OverflowError for ints smaller than 64 bits
+    if platform.startswith('linux'):
+        int32_bit_exponent = 66
+    else:
+        int32_bit_exponent = 34
+    test_parameters = [(1, np.int32, int32_bit_exponent), (2, np.int64, 66)]
+    expected_array = np.array([5, 6, 7, 3, 4])
+
+    for idx, int_type, bit_exponent in test_parameters:
+        test_array = np.arange(0, 5).astype(int_type)
+        dialog = ArrayEditor()
+        assert dialog.setup_and_check(test_array, '1D array',
+                                      xlabels=None, ylabels=None)
+        dialog.show()
+        qtbot.waitForWindowShown(dialog)
+        view = dialog.arraywidget.view
+
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyPress(view, Qt.Key_Up)
+        qtbot.keyClicks(view, '5')
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyPress(view, Qt.Key_Space)
+        qtbot.keyClicks(view.focusWidget(), str(int(2 ** bit_exponent)))
+        qtbot.keyPress(view.focusWidget(), Qt.Key_Down)
+        MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
+        assert MockQMessageBox.critical.call_count == idx
+        qtbot.keyClicks(view, '7')
+        qtbot.keyPress(view, Qt.Key_Up)
+        qtbot.keyClicks(view, '6')
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyPress(view, Qt.Key_Return)
+        assert np.sum(expected_array ==
+                      dialog.get_value()) == len(expected_array)
+
+
 if __name__ == "__main__":
     pytest.main()
diff --git a/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py b/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
--- a/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
+++ b/spyder/widgets/variableexplorer/tests/test_dataframeeditor.py
@@ -10,10 +10,11 @@
 from __future__ import division
 
 # Standard library imports
+from sys import platform
 try:
-    from unittest.mock import Mock
+    from unittest.mock import Mock, ANY
 except ImportError:
-    from mock import Mock # Python 2
+    from mock import Mock, ANY  # Python 2
 import os
 
 # Third party imports
@@ -23,6 +24,7 @@
 from qtpy.QtCore import Qt, QTimer
 import numpy
 import pytest
+from flaky import flaky
 
 # Local imports
 from spyder.utils.programs import is_module_installed
@@ -263,5 +265,76 @@ def test_sort_dataframe_with_category_dtypes(qtbot):  # cf. issue 5361
     assert data(dfm, 1, 1) == 'category'
 
 
+def test_dataframemodel_set_data_overflow(monkeypatch):
+    """Unit test #6114: entry of an overflow int caught and handled properly"""
+    MockQMessageBox = Mock()
+    attr_to_patch = ('spyder.widgets.variableexplorer' +
+                     '.dataframeeditor.QMessageBox')
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    # Numpy doesn't raise the OverflowError for ints smaller than 64 bits
+    if platform.startswith('linux'):
+        int32_bit_exponent = 66
+    else:
+        int32_bit_exponent = 34
+    test_parameters = [(1, numpy.int32, int32_bit_exponent),
+                       (2, numpy.int64, 66)]
+
+    for idx, int_type, bit_exponent in test_parameters:
+        test_df = DataFrame(numpy.arange(7, 11), dtype=int_type)
+        model = DataFrameModel(test_df.copy())
+        index = model.createIndex(2, 1)
+        assert not model.setData(index, str(int(2 ** bit_exponent)))
+        MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
+        assert MockQMessageBox.critical.call_count == idx
+        assert numpy.sum(test_df[0].as_matrix() ==
+                         model.df.as_matrix()) == len(test_df)
+
+
+@flaky(max_runs=3)
+@pytest.mark.skipif(os.environ.get('CI', None) is not None and
+                    platform.startswith('linux'),
+                    reason="Fails on Travis CI for unknown reasons.")
+def test_dataframeeditor_edit_overflow(qtbot, monkeypatch):
+    """Test #6114: entry of an overflow int is caught and handled properly"""
+    MockQMessageBox = Mock()
+    attr_to_patch = ('spyder.widgets.variableexplorer' +
+                     '.dataframeeditor.QMessageBox')
+    monkeypatch.setattr(attr_to_patch, MockQMessageBox)
+
+    # Numpy doesn't raise the OverflowError for ints smaller than 64 bits
+    if platform.startswith('linux'):
+        int32_bit_exponent = 66
+    else:
+        int32_bit_exponent = 34
+    test_parameters = [(1, numpy.int32, int32_bit_exponent),
+                       (2, numpy.int64, 66)]
+    expected_df = DataFrame([5, 6, 7, 3, 4])
+
+    for idx, int_type, bit_exponet in test_parameters:
+        test_df = DataFrame(numpy.arange(0, 5), dtype=int_type)
+        dialog = DataFrameEditor()
+        assert dialog.setup_and_check(test_df, 'Test Dataframe')
+        dialog.show()
+        qtbot.waitForWindowShown(dialog)
+        view = dialog.dataTable
+
+        qtbot.keyPress(view, Qt.Key_Right)
+        qtbot.keyClicks(view, '5')
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyPress(view, Qt.Key_Space)
+        qtbot.keyClicks(view.focusWidget(), str(int(2 ** bit_exponet)))
+        qtbot.keyPress(view.focusWidget(), Qt.Key_Down)
+        MockQMessageBox.critical.assert_called_with(ANY, "Error", ANY)
+        assert MockQMessageBox.critical.call_count == idx
+        qtbot.keyClicks(view, '7')
+        qtbot.keyPress(view, Qt.Key_Up)
+        qtbot.keyClicks(view, '6')
+        qtbot.keyPress(view, Qt.Key_Down)
+        qtbot.keyPress(view, Qt.Key_Return)
+        assert numpy.sum(expected_df[0].as_matrix() ==
+                         dialog.get_value().as_matrix()) == len(expected_df)
+
+
 if __name__ == "__main__":
     pytest.main()