Repository: spyder-ide/spyder

Issue:
Spyder encountered a problem when opening numpy array
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

build 120x12 array of float64

double click to open in variable explorer

### Traceback
```python-traceback
  File "C:\Users\CZ3QFP\miniconda2\envs\gmrpg_dev\lib\site-packages\spyder\widgets\variableexplorer\collectionseditor.py", line 482, in createEditor
    if not editor.setup_and_check(value, title=key, readonly=readonly):
  File "C:\Users\CZ3QFP\miniconda2\envs\gmrpg_dev\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 620, in setup_and_check
    self.data.flags.writeable = True
ValueError: cannot set WRITEABLE flag to True of this array
```

## Versions

* Spyder version: 3.3.2 
* Python version: 3.7.2
* Qt version: 5.9.6
* PyQt5 version: 5.9.2
* Operating System: Windows 10

### Dependencies

```
pyflakes >=0.6.0  :  2.0.0 (OK)
pycodestyle >=2.3 :  2.4.0 (OK)
pygments >=2.0    :  2.3.1 (OK)
sphinx >=0.6.6    :  1.8.2 (OK)
rope >=0.9.4      :  0.11.0 (OK)
jedi >=0.9.0      :  0.13.2 (OK)
nbconvert >=4.0   :  5.3.1 (OK)
pandas >=0.13.1   :  None (NOK)
numpy >=1.7       :  1.16.0rc2 (OK)
sympy >=0.7.3     :  None (NOK)
cython >=0.21     :  None (NOK)
qtconsole >=4.2.0 :  4.4.3 (OK)
IPython >=4.0     :  7.2.0 (OK)
matplotlib >=2.0.0:  3.0.2 (OK)
pylint >=0.25     :  2.2.2 (OK)
```

Spyder encountered a problem when opening numpy array
## Description

### What steps will reproduce the problem?

<!--- You can use Markdown here --->

build 120x12 array of float64

double click to open in variable explorer

### Traceback
```python-traceback
  File "C:\Users\CZ3QFP\miniconda2\envs\gmrpg_dev\lib\site-packages\spyder\widgets\variableexplorer\collectionseditor.py", line 482, in createEditor
    if not editor.setup_and_check(value, title=key, readonly=readonly):
  File "C:\Users\CZ3QFP\miniconda2\envs\gmrpg_dev\lib\site-packages\spyder\widgets\variableexplorer\arrayeditor.py", line 620, in setup_and_check
    self.data.flags.writeable = True
ValueError: cannot set WRITEABLE flag to True of this array
```

## Versions

* Spyder version: 3.3.2 
* Python version: 3.7.2
* Qt version: 5.9.6
* PyQt5 version: 5.9.2
* Operating System: Windows 10

### Dependencies

```
pyflakes >=0.6.0  :  2.0.0 (OK)
pycodestyle >=2.3 :  2.4.0 (OK)
pygments >=2.0    :  2.3.1 (OK)
sphinx >=0.6.6    :  1.8.2 (OK)
rope >=0.9.4      :  0.11.0 (OK)
jedi >=0.9.0      :  0.13.2 (OK)
nbconvert >=4.0   :  5.3.1 (OK)
pandas >=0.13.1   :  None (NOK)
numpy >=1.7       :  1.16.0rc2 (OK)
sympy >=0.7.3     :  None (NOK)
cython >=0.21     :  None (NOK)
qtconsole >=4.2.0 :  4.4.3 (OK)
IPython >=4.0     :  7.2.0 (OK)
matplotlib >=2.0.0:  3.0.2 (OK)
pylint >=0.25     :  2.2.2 (OK)
```



Discussion:
I can post the array if needed. If desired, what format do you guys want? There is nothing special about it, just a 120x12 numpy arrray. each float64 is 4 decimal places, no extended precision weird stuff or anything like that.

tried restarting spyder, same non-special array opens fine in spyder 2.3.8, which seems to have the most stable variable explorer (loads properly, updates values properly etc), but this version has issues I have mostly worked around regarding per-file run settings.
> If desired, what format do you guys want?

Please post the command you're using to generate it.
it is captured data, that's why I asked if you'd like me to post it. but there's nothing special, it was imported from a tdms file but printing it in the ipython window works just fine, it seems valid.
Then an npz file should be enough.
Alright, it's attached as a zip because GH got mad at npz. Rename 'zip' to 'npz' then open with np.load (as I'm sure you're aware)

I opened with np.load, and verified the incoming array also produces the same issue when double clicking in the variable explorer. Like I said, nothing is special about it. If you find this to be not true, please let me know.. I'm curious! Thanks for helping make spyder better.

[issue_array.zip](https://github.com/spyder-ide/spyder/files/2765455/issue_array.zip)


Thanks for your help!
No problem, thanks for supporting. This piece of software made it easy for me to leave Matlab in the dust, eventually leading me to use Git and properly version control my Python code and older C# projects, and taught me how to bug report. So I am happy to help you out with at least decent bug reports :D
Is this problem fixed ? 
No, it hasn't.
This is popping up a lot lately. Is this bug related to #6213 as well?
I think this is caused by the new Numpy 1.16. That should be why we're getting so much reports about this problem lately.
I'm not sure what goes on under the hood, but I have not found the variable explorer to be 100% reliable since spyder 2.3.8, but 2.3.8 chokes up on spaces in the path for per-file input args which is so 90's. It wouldn't always update the values after a run. The issue not updating values was not reproduceable enough for me to report. Now with it not opening at all, AND it putting a nice template right in front of my face I couldn't not report. The variable explorer is probably my favorite feature of Spyder, seems like it needs to be made more robust (probably easier said than done). Wish I could help more, please let me know if I can provide anything else.
> spyder 2.3.8

@sayboltm The Variable Explorer in 2.3.8 only supported a limited set of types, while in recent 3.2.x versions it can now (theoretically, issues like this aside) display almost all types, which results in many more edge/corner cases that need to be accounted for. We're working to improve this further for Spyder 4, including a dedicated arbitrary object viewer and support for Numpy object types.

@ccordoba12 According to numpy/numpy#12791 , setting the flag manually is a dangerous practice and accordingly has been removed in 1.16, and if we've made a copy of the original array (which should always be the case, AFAIK) it should be writable already in any modern Numpy version. If its not for some very special-case reason, we shouldn't try to modify it, so we should set ``readonly = True`` to prevent this.

Should I go ahead and implement this accordingly? Although its not a clean merge, should we consider including this in 3.3.3, since as people start upgrading to Numpy 1.16 *en masse* we're going to get a flood of reports and it will completely break viewing/editing any arrays in the Variable Explorer?
> Should I go ahead and implement this accordingly?

Ok, but please try to have it for today or tomorrow. It should be a really simple fix.

> Although its not a clean merge, should we consider including this in 3.3.3

Yes. That's why this issue is marked for that milestone.
Yes, numpy version 1.16 shows this problem. When downgrade numpy version to 1.15.0, variable explorer working. Spyder version 3.3.2. 
But, with numpy version 1.15.0 in some scripts i has this error:
_ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject_
@meteorolog90 That's unrelated to both this issue and Spyder, and should be fixed in numpy/numpy#11788 (in Numpy 1.16.1).

> Ok, but please try to have it for today or tomorrow. It should be a really simple fix.

In work as we speak.

> Yes. That's why this issue is marked for that milestone.

D'oh, how did I miss that?
@CAM-Gerlach Okay, thanks for the answer
I can post the array if needed. If desired, what format do you guys want? There is nothing special about it, just a 120x12 numpy arrray. each float64 is 4 decimal places, no extended precision weird stuff or anything like that.

tried restarting spyder, same non-special array opens fine in spyder 2.3.8, which seems to have the most stable variable explorer (loads properly, updates values properly etc), but this version has issues I have mostly worked around regarding per-file run settings.
> If desired, what format do you guys want?

Please post the command you're using to generate it.
it is captured data, that's why I asked if you'd like me to post it. but there's nothing special, it was imported from a tdms file but printing it in the ipython window works just fine, it seems valid.
Then an npz file should be enough.
Alright, it's attached as a zip because GH got mad at npz. Rename 'zip' to 'npz' then open with np.load (as I'm sure you're aware)

I opened with np.load, and verified the incoming array also produces the same issue when double clicking in the variable explorer. Like I said, nothing is special about it. If you find this to be not true, please let me know.. I'm curious! Thanks for helping make spyder better.

[issue_array.zip](https://github.com/spyder-ide/spyder/files/2765455/issue_array.zip)


Thanks for your help!
No problem, thanks for supporting. This piece of software made it easy for me to leave Matlab in the dust, eventually leading me to use Git and properly version control my Python code and older C# projects, and taught me how to bug report. So I am happy to help you out with at least decent bug reports :D
Is this problem fixed ? 
No, it hasn't.
This is popping up a lot lately. Is this bug related to #6213 as well?
I think this is caused by the new Numpy 1.16. That should be why we're getting so much reports about this problem lately.
I'm not sure what goes on under the hood, but I have not found the variable explorer to be 100% reliable since spyder 2.3.8, but 2.3.8 chokes up on spaces in the path for per-file input args which is so 90's. It wouldn't always update the values after a run. The issue not updating values was not reproduceable enough for me to report. Now with it not opening at all, AND it putting a nice template right in front of my face I couldn't not report. The variable explorer is probably my favorite feature of Spyder, seems like it needs to be made more robust (probably easier said than done). Wish I could help more, please let me know if I can provide anything else.
> spyder 2.3.8

@sayboltm The Variable Explorer in 2.3.8 only supported a limited set of types, while in recent 3.2.x versions it can now (theoretically, issues like this aside) display almost all types, which results in many more edge/corner cases that need to be accounted for. We're working to improve this further for Spyder 4, including a dedicated arbitrary object viewer and support for Numpy object types.

@ccordoba12 According to numpy/numpy#12791 , setting the flag manually is a dangerous practice and accordingly has been removed in 1.16, and if we've made a copy of the original array (which should always be the case, AFAIK) it should be writable already in any modern Numpy version. If its not for some very special-case reason, we shouldn't try to modify it, so we should set ``readonly = True`` to prevent this.

Should I go ahead and implement this accordingly? Although its not a clean merge, should we consider including this in 3.3.3, since as people start upgrading to Numpy 1.16 *en masse* we're going to get a flood of reports and it will completely break viewing/editing any arrays in the Variable Explorer?
> Should I go ahead and implement this accordingly?

Ok, but please try to have it for today or tomorrow. It should be a really simple fix.

> Although its not a clean merge, should we consider including this in 3.3.3

Yes. That's why this issue is marked for that milestone.
Yes, numpy version 1.16 shows this problem. When downgrade numpy version to 1.15.0, variable explorer working. Spyder version 3.3.2. 
But, with numpy version 1.15.0 in some scripts i has this error:
_ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject_
@meteorolog90 That's unrelated to both this issue and Spyder, and should be fixed in numpy/numpy#11788 (in Numpy 1.16.1).

> Ok, but please try to have it for today or tomorrow. It should be a really simple fix.

In work as we speak.

> Yes. That's why this issue is marked for that milestone.

D'oh, how did I miss that?
@CAM-Gerlach Okay, thanks for the answer

Resolving patch:
diff --git a/spyder/widgets/variableexplorer/arrayeditor.py b/spyder/widgets/variableexplorer/arrayeditor.py
--- a/spyder/widgets/variableexplorer/arrayeditor.py
+++ b/spyder/widgets/variableexplorer/arrayeditor.py
@@ -617,7 +617,7 @@ def setup_and_check(self, data, title='', readonly=False,
         return False if data is not supported, True otherwise
         """
         self.data = data
-        self.data.flags.writeable = True
+        readonly = readonly or not self.data.flags.writeable
         is_record_array = data.dtype.names is not None
         is_masked_array = isinstance(data, np.ma.MaskedArray)
 


Test patch:
diff --git a/spyder/app/tests/test_mainwindow.py b/spyder/app/tests/test_mainwindow.py
--- a/spyder/app/tests/test_mainwindow.py
+++ b/spyder/app/tests/test_mainwindow.py
@@ -580,33 +580,6 @@ def test_connection_to_external_kernel(main_window, qtbot):
     km.shutdown_kernel(now=True)
 
 
-@pytest.mark.slow
-@flaky(max_runs=3)
-@pytest.mark.skipif(os.name == 'nt', reason="It times out sometimes on Windows")
-def test_np_threshold(main_window, qtbot):
-    """Test that setting Numpy threshold doesn't make the Variable Explorer slow."""
-    # Set Numpy threshold
-    shell = main_window.ipyconsole.get_current_shellwidget()
-    qtbot.waitUntil(lambda: shell._prompt_html is not None, timeout=SHELL_TIMEOUT)
-    with qtbot.waitSignal(shell.executed):
-        shell.execute('import numpy as np; np.set_printoptions(threshold=np.nan)')
-
-    # Create a big Numpy array
-    with qtbot.waitSignal(shell.executed):
-        shell.execute('x = np.random.rand(75000,5)')
-
-    # Wait a very small time to see the array in the Variable Explorer
-    main_window.variableexplorer.visibility_changed(True)
-    nsb = main_window.variableexplorer.get_focus_widget()
-    qtbot.waitUntil(lambda: nsb.editor.model.rowCount() == 1, timeout=500)
-
-    # Assert that NumPy threshold remains the same as the one
-    # set by the user
-    with qtbot.waitSignal(shell.executed):
-        shell.execute("t = np.get_printoptions()['threshold']")
-    assert np.isnan(shell.get_value('t'))
-
-
 @pytest.mark.slow
 @flaky(max_runs=3)
 @pytest.mark.skipif(os.name == 'nt', reason="It times out sometimes on Windows")