Repository: spyder-ide/spyder

Issue:
Saving large workspace on network drive crashes Spyder on Windows
_From [rockspor...@gmail.com](https://code.google.com/u/107947666923439381005/) on 2012-01-24T12:08:38Z_

If I use the variable explorer to store large objects on a network drive spyder crashes. 

The reason is: http://stackoverflow.com/questions/4226941/python-ioerror-errno-22-invalid-argument-when-using-cpickle-to-write-large If I modify the mode flags in iofuncs.py as

   cPickle.dump(...file(..., "w+b"))

and

   cPickle.load(... file(..., "rb"))

the problem disappears.

_Original issue: http://code.google.com/p/spyderlib/issues/detail?id=919_

Saving large workspace on network drive crashes Spyder on Windows
_From [rockspor...@gmail.com](https://code.google.com/u/107947666923439381005/) on 2012-01-24T12:08:38Z_

If I use the variable explorer to store large objects on a network drive spyder crashes. 

The reason is: http://stackoverflow.com/questions/4226941/python-ioerror-errno-22-invalid-argument-when-using-cpickle-to-write-large If I modify the mode flags in iofuncs.py as

   cPickle.dump(...file(..., "w+b"))

and

   cPickle.load(... file(..., "rb"))

the problem disappears.

_Original issue: http://code.google.com/p/spyderlib/issues/detail?id=919_

Saving session data (spydata) deletes all arrays nested inside other data structures
## Problem Description

In fixing #919 and writing more comprehensive tests for Spyder's ``save_dictionary`` and ``load_dictionary`` session spydata save/load functions, I discovered that saving one's session will delete all arrays that are not top-level, i.e. those inside lists, dictionaries, etc from the user's active session, during the process of removing them to be serialized separately. The fix is a simple one (just deepcopying the data passed to the save_dictionary function and a PR should be up shortly, but I thought I'd open this issue to document the problem.

### What steps will reproduce the problem?

1. Create a list or dictionary with a numpy array inside of it
2. Save the session
3. View the list/dictionary again

### What is the expected output? What do you see instead?

Expected: Array is not removed. Actual: Array is.

## Package Versions

* Spyder: 3.2.7
* Python: 3.6.4
* Qt: 5.6.2
* PyQt: 5.6
* Operating System: Win 8.1 Pro x64



Discussion:
@ccordoba12 Currently, both calls to ``pickle.load()`` in ``iofuncs.py`` are on files set to the ``rb`` mode, while the one call to ``pickle.dump`` is to a file with the ``wb`` (create new file for writing in binary mode, overwriting any previous) rather than ``w+b`` (open for reading and writing as binary, first truncating to 0 bytes). Apparently; the latter gets around the Windows OS limitation causing this error, and I wouldn't think it would otherwise impact functionality. 

Mind if I submit a quick PR? Aside from possibly a basic test, it should be a one line fix...
> Mind if I submit a quick PR?

Not at all, please do :-)
@ccordoba12 Currently, both calls to ``pickle.load()`` in ``iofuncs.py`` are on files set to the ``rb`` mode, while the one call to ``pickle.dump`` is to a file with the ``wb`` (create new file for writing in binary mode, overwriting any previous) rather than ``w+b`` (open for reading and writing as binary, first truncating to 0 bytes). Apparently; the latter gets around the Windows OS limitation causing this error, and I wouldn't think it would otherwise impact functionality. 

Mind if I submit a quick PR? Aside from possibly a basic test, it should be a one line fix...
> Mind if I submit a quick PR?

Not at all, please do :-)


Resolving patch:
diff --git a/spyder/utils/iofuncs.py b/spyder/utils/iofuncs.py
--- a/spyder/utils/iofuncs.py
+++ b/spyder/utils/iofuncs.py
@@ -1,8 +1,10 @@
-# -*- coding:utf-8 -*-
-#
+# -*- coding: utf-8 -*-
+# -----------------------------------------------------------------------------
 # Copyright © Spyder Project Contributors
+#
 # Licensed under the terms of the MIT License
 # (see spyder/__init__.py for details)
+# ----------------------------------------------------------------------------
 
 """
 Input/Output Utilities
@@ -13,17 +15,20 @@
 
 from __future__ import print_function
 
+# Standard library imports
 import sys
 import os
+import os.path as osp
 import tarfile
 import tempfile
-import os.path as osp
 import shutil
 import warnings
 import json
 import inspect
 import dis
+import copy
 
+# Third party imports
 # - If pandas fails to import here (for any reason), Spyder
 #   will crash at startup (e.g. see Issue 2300)
 # - This also prevents Spyder to start IPython kernels
@@ -34,7 +39,7 @@
     pd = None            #analysis:ignore
 
 # Local imports
-from spyder.config.base import _, get_conf_path
+from spyder.config.base import _, STDERR
 from spyder.py3compat import pickle, to_text_string, PY2
 from spyder.utils.misc import getcwd_or_home
 
@@ -285,6 +290,16 @@ def save_dictionary(data, filename):
     old_cwd = getcwd_or_home()
     os.chdir(osp.dirname(filename))
     error_message = None
+
+    # Copy dictionary before modifying to fix #6689
+    try:
+        data = copy.deepcopy(data)
+    except NotImplementedError:
+        try:
+            data = copy.copy(data)
+        except Exception:
+            data = data
+
     try:
         saved_arrays = {}
         if load_array is not None:
@@ -315,7 +330,7 @@ def save_dictionary(data, filename):
             if saved_arrays:
                 data['__saved_arrays__'] = saved_arrays
         pickle_filename = osp.splitext(filename)[0]+'.pickle'
-        with open(pickle_filename, 'wb') as fdesc:
+        with open(pickle_filename, 'w+b') as fdesc:
             pickle.dump(data, fdesc, 2)
         tar = tarfile.open(filename, "w")
         for fname in [pickle_filename]+[fn for fn in list(saved_arrays.values())]:


Test patch:
diff --git a/spyder/utils/tests/test_export.spydata b/spyder/utils/tests/export_data.spydata
similarity index 73%
rename from spyder/utils/tests/test_export.spydata
rename to spyder/utils/tests/export_data.spydata
Binary files a/spyder/utils/tests/test_export.spydata and b/spyder/utils/tests/export_data.spydata differ
diff --git a/spyder/utils/tests/test_iofuncs.py b/spyder/utils/tests/test_iofuncs.py
--- a/spyder/utils/tests/test_iofuncs.py
+++ b/spyder/utils/tests/test_iofuncs.py
@@ -1,35 +1,50 @@
 # -*- coding: utf-8 -*-
-#
+# -----------------------------------------------------------------------------
 # Copyright © Spyder Project Contributors
-# Licensed under the terms of the MIT License
 #
+# Licensed under the terms of the MIT License
+# (see spyder/__init__.py for details)
+# ----------------------------------------------------------------------------
 
 """
 Tests for iofuncs.py
 """
 
+# Standard library imports
 import io
 import os
+
+# Third party imports
 import pytest
 import numpy as np
+
+# Local imports
 import spyder.utils.iofuncs as iofuncs
 
+
+# Full path to this file's parent directory for loading data
 LOCATION = os.path.realpath(os.path.join(os.getcwd(),
                                          os.path.dirname(__file__)))
 
+
+# =============================================================================
+# Fixtures
+# =============================================================================
 @pytest.fixture
 def spydata_values():
     """
     Define spydata file ground truth values.
 
-    The file test_export.spydata contains four variables to be loaded, this
-    function declares those variables in a static way.
+    The file export_data.spydata contains five variables to be loaded.
+    This fixture declares those variables in a static way.
     """
     A = 1
     B = 'ham'
     C = np.eye(3)
-    D = {'a':1}
-    return {'A':A, 'B':B, 'C':C, 'D':D}
+    D = {'a': True, 'b': np.eye(4, dtype=np.complex)}
+    E = [np.eye(2, dtype=np.int64), 42.0, np.eye(3, dtype=np.bool_)]
+    return {'A': A, 'B': B, 'C': C, 'D': D, 'E': E}
+
 
 @pytest.fixture
 def real_values():
@@ -52,8 +67,12 @@ def real_values():
     C = file_s['C']
     D = file_s['D'].item()
     E = file_s['E']
-    return {'A':A, 'B':B, 'C':C, 'D':D, 'E':E}
+    return {'A': A, 'B': B, 'C': C, 'D': D, 'E': E}
+
 
+# =============================================================================
+# Tests
+# =============================================================================
 @pytest.mark.skipif(iofuncs.load_matlab is None, reason="SciPy required")
 def test_matlab_import(real_values):
     """
@@ -72,6 +91,7 @@ def test_matlab_import(real_values):
         valid = valid and bool(np.mean(real_values[var] == inf[var]))
     assert valid
 
+
 def test_spydata_import(spydata_values):
     """
     Test spydata handling and variable importing.
@@ -79,13 +99,19 @@ def test_spydata_import(spydata_values):
     This test loads all the variables contained inside a spydata tar
     container and compares them against their static values.
     """
-    path = os.path.join(LOCATION, 'test_export.spydata')
+    path = os.path.join(LOCATION, 'export_data.spydata')
     data, error = iofuncs.load_dictionary(path)
+    assert error is None
     valid = True
     for var in sorted(spydata_values.keys()):
-        valid = valid and bool(np.mean(spydata_values[var] == data[var]))
+        try:
+            valid = valid and bool(np.mean(spydata_values[var] == data[var]))
+        except ValueError:
+            valid = valid and all([np.all(obj1 == obj2) for obj1, obj2 in
+                                   zip(spydata_values[var], data[var])])
     assert valid
 
+
 @pytest.mark.skipif(iofuncs.load_matlab is None, reason="SciPy required")
 def test_matlabstruct():
     """Test support for matlab stlye struct."""
@@ -100,10 +126,40 @@ def test_matlabstruct():
     buf = io.BytesIO()
     iofuncs.save_matlab(a, buf)
     buf.seek(0)
-    data, err = iofuncs.load_matlab(buf)
+    data, error = iofuncs.load_matlab(buf)
+
+    assert error is None
     assert data['b'] == 'spam'
     assert data['c'].d == 'eggs'
     assert data['d'].tolist() == [[1, 2, 3]]
 
+
+def test_spydata_export(spydata_values):
+    """
+    Test spydata export and re-import.
+
+    This test saves the variables in ``spydata`` format and then
+    reloads and checks them to make sure they save/restore properly
+    and no errors occur during the process.
+    """
+    path = os.path.join(LOCATION, 'export_data_copy.spydata')
+    export_error = iofuncs.save_dictionary(spydata_values, path)
+    assert export_error is None
+    data, import_error = iofuncs.load_dictionary(path)
+    assert import_error is None
+    valid = True
+    for var in sorted(spydata_values.keys()):
+        try:
+            valid = valid and bool(np.mean(spydata_values[var] == data[var]))
+        except ValueError:
+            valid = valid and all([np.all(obj1 == obj2) for obj1, obj2 in
+                                   zip(spydata_values[var], data[var])])
+    assert valid
+    try:
+        os.remove(path)
+    except (IOError, OSError, PermissionError):
+        pass
+
+
 if __name__ == "__main__":
     pytest.main()