Repository: joblib/joblib
Pull request #1181 (2021-05-05 10:03:31)

Fixing byte-order consistency/mismatch for cross-endian platform
- Addressing https://github.com/joblib/joblib/issues/1123

pradghos:  Circle CI failed at ./continuous_integration/build_doc.sh

  ```
  result = self._call_chain(self.handle_open, protocol, protocol +
  File "/home/circleci/miniconda/envs/testenv/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/home/circleci/miniconda/envs/testenv/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
  urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>

  Makefile:11: recipe for target 'html' failed
```

Before this commit `https://github.com/joblib/joblib/pull/1181/commits/d9e5adbcc5478159310ebd58b3c070c5feea7e7d` all were green except `ci/circleci: build` 

Now `test_nested_loop_error_in_grandchild_resource_tracker_silent` failed which looks like unrelated to this PR.

Any pointers would really help . Thank you ! 
pradghos: @ogrisel It would be great if you can take a look. Thanks in advance ! 
lesteve: Thanks a lot for this PR! What would be great is to have a test for this.

Is there a way to store a pickle string obtained from a big endian machine and test that it can be loaded fine in the CI (little endian machine)?

I think we can ignore the CI failures as they do seem spurious.
ogrisel: Alternatively could you just please add a couple of unit tests for `_is_numpy_array_byte_order_mismatch` alone (independently of `jolib.load` / `dump`)?

Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
deleted file mode 100644
index 7668179..0000000
--- a/.circleci/config.yml
+++ /dev/null
@@ -1,16 +0,0 @@
-version: 2
-
-jobs:
-  build:
-    docker:
-      - image: circleci/python:3.6.2
-    steps:
-      - checkout
-      - run:
-          command: ./continuous_integration/build_doc.sh
-          environment:
-            MINICONDA_PATH: ~/miniconda
-            CONDA_ENV_NAME: testenv
-      - store_artifacts:
-          path: doc/_build/html
-          destination: doc
diff --git a/CHANGES.rst b/CHANGES.rst
index 93144b3..f9ad25f 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -4,6 +4,12 @@ Latest changes
 Development version
 -------------------
 
+- Fix byte order inconsistency issue during deserialization using joblib.load
+  in cross-endian environment: the numpy arrays are now always loaded to
+  use the system byte order, independently of the byte order of the system
+  that serialized the pickle.
+  https://github.com/joblib/joblib/pull/1181
+
 - Fix joblib.Memory bug with the ``ignore`` parameter when the cached function
   is a decorated function.
   https://github.com/joblib/joblib/pull/1165
@@ -336,8 +342,8 @@ Thomas Moreau
 
     Implement the ``'loky'`` backend with @ogrisel. This backend relies on
     a robust implementation of ``concurrent.futures.ProcessPoolExecutor``
-    with spawned processes that can be reused accross the ``Parallel``
-    calls. This fixes the bad interation with third paty libraries relying on
+    with spawned processes that can be reused across the ``Parallel``
+    calls. This fixes the bad integration with third paty libraries relying on
     thread pools, described in https://pythonhosted.org/joblib/parallel.html#bad-interaction-of-multiprocessing-and-third-party-libraries
 
     Limit the number of threads used in worker processes by C-libraries that
@@ -397,7 +403,7 @@ Alexandre Abadie
 
     Add ``register_compressor`` function for extending available compressors.
 
-    Allow passing a string to ``compress`` parameter in ``dump`` funtion. This
+    Allow passing a string to ``compress`` parameter in ``dump`` function. This
     string should correspond to the compressor used (e.g. zlib, gzip, lz4,
     etc). The default compression level is used in this case.
 
@@ -447,7 +453,7 @@ Loïc Estève
 Loïc Estève
 
     Fix handling of memmap objects with offsets greater than
-    mmap.ALLOCATIONGRANULARITY in ``joblib.Parrallel``. See
+    mmap.ALLOCATIONGRANULARITY in ``joblib.Parallel``. See
     https://github.com/joblib/joblib/issues/451 for more details.
 
 Loïc Estève
@@ -1046,7 +1052,7 @@ Release 0.5.3
 2011-06-25
 Gael varoquaux
 
-   API: All the usefull symbols in the __init__
+   API: All the useful symbols in the __init__
 
 
 Release 0.5.2
diff --git a/README.rst b/README.rst
index f1aac98..d659dbd 100644
--- a/README.rst
+++ b/README.rst
@@ -1,4 +1,4 @@
-|PyPi| |Azure| |Codecov|
+|PyPi| |Azure| |ReadTheDocs| |Codecov| 
 
 .. |PyPi| image:: https://badge.fury.io/py/joblib.svg
    :target: https://badge.fury.io/py/joblib
@@ -6,7 +6,11 @@
 
 .. |Azure| image:: https://dev.azure.com/joblib/joblib/_apis/build/status/joblib.joblib?branchName=master
    :target: https://dev.azure.com/joblib/joblib/_build?definitionId=3&_a=summary&branchFilter=40
-   :alt: Codecov coverage
+   :alt: Azure CI status
+
+.. |ReadTheDocs| image:: https://readthedocs.org/projects/pip/badge/?version=latest
+   :target: https://joblib.readthedocs.io/en/latest/
+   :alt: Doc status
 
 .. |Codecov| image:: https://codecov.io/gh/joblib/joblib/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/joblib/joblib
diff --git a/continuous_integration/build_doc.sh b/continuous_integration/build_doc.sh
deleted file mode 100755
index 0c1e6e2..0000000
--- a/continuous_integration/build_doc.sh
+++ /dev/null
@@ -1,24 +0,0 @@
-#!/usr/bin/env bash
-set -x
-set -e
-
-# deactivate circleci virtualenv and setup a miniconda env instead
-if [[ `type -t deactivate` ]]; then
-    deactivate
-fi
-
-wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
-     -O miniconda.sh
-chmod +x miniconda.sh && ./miniconda.sh -b -p $MINICONDA_PATH
-export PATH="$MINICONDA_PATH/bin:$PATH"
-conda update --yes --quiet conda
-
-conda create -n $CONDA_ENV_NAME --yes --quiet python=3
-source activate $CONDA_ENV_NAME
-
-conda install --yes --quiet pip numpy sphinx matplotlib pillow dask distributed
-pip install sphinx-gallery numpydoc lz4
-
-python setup.py develop
-
-make doc 2>&1
diff --git a/doc/conf.py b/doc/conf.py
index fa9e928..05e3aa1 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -253,7 +253,7 @@ try:
     shutil.copyfile('../README.rst', 'README.rst')
 except IOError:
     pass
-    # This fails during the tesing, as the code is ran in a different
+    # This fails during the testing, as the code is ran in a different
     # directory
 
 numpydoc_show_class_members = False
diff --git a/examples/serialization_and_wrappers.py b/examples/serialization_and_wrappers.py
index 8e7ccf9..4e9967d 100644
--- a/examples/serialization_and_wrappers.py
+++ b/examples/serialization_and_wrappers.py
@@ -78,7 +78,7 @@ if sys.platform != 'win32':
 # POSIX specification and can have bad interaction with compiled extensions
 # that use ``openmp``. Also, it is not possible to start processes with
 # ``fork`` on windows where only ``spawn`` is available. The ``loky`` backend
-# has been developped to mitigate these issues.
+# has been developed to mitigate these issues.
 #
 # To have fast pickling with ``loky``, it is possible to rely on ``pickle`` to
 # serialize all communications between the main process and the workers with
diff --git a/joblib/_dask.py b/joblib/_dask.py
index 009ddc6..e759b7f 100644
--- a/joblib/_dask.py
+++ b/joblib/_dask.py
@@ -51,7 +51,7 @@ class _WeakKeyDictionary:
     such as large numpy arrays or pandas dataframes that are not hashable and
     therefore cannot be used as keys of traditional python dicts.
 
-    Futhermore using a dict with id(array) as key is not safe because the
+    Furthermore using a dict with id(array) as key is not safe because the
     Python is likely to reuse id of recently collected arrays.
     """
 
diff --git a/joblib/externals/loky/backend/queues.py b/joblib/externals/loky/backend/queues.py
index 62735db..5671464 100644
--- a/joblib/externals/loky/backend/queues.py
+++ b/joblib/externals/loky/backend/queues.py
@@ -209,7 +209,7 @@ class SimpleQueue(mp_SimpleQueue):
             else:
                 self._wlock = ctx.Lock()
 
-        # Add possiblity to use custom reducers
+        # Add possibility to use custom reducers
         self._reducers = reducers
 
     def close(self):
diff --git a/joblib/externals/loky/backend/resource_tracker.py b/joblib/externals/loky/backend/resource_tracker.py
index 95dff35..48e49be 100644
--- a/joblib/externals/loky/backend/resource_tracker.py
+++ b/joblib/externals/loky/backend/resource_tracker.py
@@ -36,7 +36,7 @@
 # Note that this behavior differs from CPython's resource_tracker, which only
 # implements list of shared resources, and not a proper refcounting scheme.
 # Also, CPython's resource tracker will only attempt to cleanup those shared
-# resources once all procsses connected to the resouce tracker have exited.
+# resources once all procsses connected to the resource tracker have exited.
 
 
 import os
diff --git a/joblib/externals/loky/backend/spawn.py b/joblib/externals/loky/backend/spawn.py
index 2a16c84..d4a6a8c 100644
--- a/joblib/externals/loky/backend/spawn.py
+++ b/joblib/externals/loky/backend/spawn.py
@@ -105,7 +105,7 @@ def get_preparation_data(name, init_main_module=True):
             _resource_tracker as mp_resource_tracker
         )
         # multiprocessing's resource_tracker must be running before loky
-        # process is created (othewise the child won't be able to use it if it
+        # process is created (otherwise the child won't be able to use it if it
         # is created later on)
         mp_resource_tracker.ensure_running()
         d["mp_tracker_args"] = {
diff --git a/joblib/externals/loky/backend/utils.py b/joblib/externals/loky/backend/utils.py
index dc1b82a..5342204 100644
--- a/joblib/externals/loky/backend/utils.py
+++ b/joblib/externals/loky/backend/utils.py
@@ -117,7 +117,7 @@ def _recursive_terminate(pid):
 
 
 def get_exitcodes_terminated_worker(processes):
-    """Return a formated string with the exitcodes of terminated workers.
+    """Return a formatted string with the exitcodes of terminated workers.
 
     If necessary, wait (up to .25s) for the system to correctly set the
     exitcode of one terminated worker.
diff --git a/joblib/memory.py b/joblib/memory.py
index 76daf52..93124ee 100644
--- a/joblib/memory.py
+++ b/joblib/memory.py
@@ -848,7 +848,7 @@ class Memory(Logger):
 
         Parameters
         ----------
-        location: str or None
+        location: str, pathlib.Path or None
             The path of the base directory to use as a data store
             or None. If None is given, no caching is done and
             the Memory object is completely transparent. This option
diff --git a/joblib/numpy_pickle.py b/joblib/numpy_pickle.py
index 93e5537..cc593af 100644
--- a/joblib/numpy_pickle.py
+++ b/joblib/numpy_pickle.py
@@ -20,6 +20,7 @@ from .compressor import (ZlibCompressorWrapper, GzipCompressorWrapper,
 from .numpy_pickle_utils import Unpickler, Pickler
 from .numpy_pickle_utils import _read_fileobject, _write_fileobject
 from .numpy_pickle_utils import _read_bytes, BUFFER_SIZE
+from .numpy_pickle_utils import _ensure_native_byte_order
 from .numpy_pickle_compat import load_compatibility
 from .numpy_pickle_compat import NDArrayWrapper
 # For compatibility with old versions of joblib, we need ZNDArrayWrapper
@@ -147,7 +148,8 @@ class NumpyArrayWrapper(object):
             else:
                 array.shape = self.shape
 
-        return array
+        # Detect byte order mis-match and swap as needed.
+        return _ensure_native_byte_order(array)
 
     def read_mmap(self, unpickler):
         """Read an array using numpy memmap."""
diff --git a/joblib/numpy_pickle_compat.py b/joblib/numpy_pickle_compat.py
index 6541a06..096acbc 100644
--- a/joblib/numpy_pickle_compat.py
+++ b/joblib/numpy_pickle_compat.py
@@ -9,7 +9,7 @@ from io import BytesIO
 
 from .numpy_pickle_utils import _ZFILE_PREFIX
 from .numpy_pickle_utils import Unpickler
-
+from .numpy_pickle_utils import _ensure_native_byte_order
 
 def hex_str(an_int):
     """Convert an int to an hexadecimal string."""
@@ -105,6 +105,9 @@ class NDArrayWrapper(object):
             kwargs["allow_pickle"] = True
         array = unpickler.np.load(filename, **kwargs)
 
+        # Detect byte order mis-match and swap as needed.
+        array = _ensure_native_byte_order(array)
+
         # Reconstruct subclasses. This does not work with old
         # versions of numpy
         if (hasattr(array, '__array_prepare__') and
diff --git a/joblib/numpy_pickle_utils.py b/joblib/numpy_pickle_utils.py
index a501055..02a88ff 100644
--- a/joblib/numpy_pickle_utils.py
+++ b/joblib/numpy_pickle_utils.py
@@ -6,6 +6,7 @@
 
 import pickle
 import io
+import sys
 import warnings
 import contextlib
 
@@ -48,6 +49,30 @@ def _get_prefixes_max_len():
     return max(prefixes)
 
 
+def _is_numpy_array_byte_order_mismatch(array):
+    """Check if numpy array is having byte order mis-match"""
+    return ((sys.byteorder == 'big' and
+             (array.dtype.byteorder == '<' or
+              (array.dtype.byteorder == '|' and array.dtype.fields and
+               all(e[0].byteorder == '<'
+                   for e in array.dtype.fields.values())))) or
+            (sys.byteorder == 'little' and
+             (array.dtype.byteorder == '>' or
+              (array.dtype.byteorder == '|' and array.dtype.fields and
+               all(e[0].byteorder == '>'
+                   for e in array.dtype.fields.values())))))
+
+
+def _ensure_native_byte_order(array):
+    """Use the byte order of the host while preserving values
+
+    Does nothing if array already uses the system byte order.
+    """
+    if _is_numpy_array_byte_order_mismatch(array):
+        array = array.byteswap().newbyteorder('=')
+    return array
+
+
 ###############################################################################
 # Cache file utilities
 def _detect_compressor(fileobj):


Test patch:
diff --git a/joblib/test/test_hashing.py b/joblib/test/test_hashing.py
index 37d9480..5682d8f 100644
--- a/joblib/test/test_hashing.py
+++ b/joblib/test/test_hashing.py
@@ -260,8 +260,8 @@ def test_numpy_scalar():
 
 
 def test_dict_hash(tmpdir):
-    # Check that dictionaries hash consistently, eventhough the ordering
-    # of the keys is not garanteed
+    # Check that dictionaries hash consistently, even though the ordering
+    # of the keys is not guaranteed
     k = KlassWithCachedMethod(tmpdir.strpath)
 
     d = {'#s12069__c_maps.nii.gz': [33],
diff --git a/joblib/test/test_memory.py b/joblib/test/test_memory.py
index e917871..89d61d4 100644
--- a/joblib/test/test_memory.py
+++ b/joblib/test/test_memory.py
@@ -365,7 +365,7 @@ def test_memory_eval(tmpdir):
 def count_and_append(x=[]):
     """ A function with a side effect in its arguments.
 
-        Return the lenght of its argument and append one element.
+        Return the length of its argument and append one element.
     """
     len_x = len(x)
     x.append(None)
@@ -1157,7 +1157,7 @@ def test_register_invalid_store_backends_object():
 
 
 def test_memory_default_store_backend():
-    # test an unknow backend falls back into a FileSystemStoreBackend
+    # test an unknown backend falls back into a FileSystemStoreBackend
     with raises(TypeError) as excinfo:
         Memory(location='/tmp/joblib', backend='unknown')
     excinfo.match(r"Unknown location*")
diff --git a/joblib/test/test_numpy_pickle.py b/joblib/test/test_numpy_pickle.py
index db130b1..7cdd90f 100644
--- a/joblib/test/test_numpy_pickle.py
+++ b/joblib/test/test_numpy_pickle.py
@@ -5,6 +5,7 @@ import os
 import random
 import re
 import io
+import sys
 import warnings
 import gzip
 import zlib
@@ -30,6 +31,8 @@ from joblib.test import data
 
 from joblib.numpy_pickle_utils import _IO_BUFFER_SIZE
 from joblib.numpy_pickle_utils import _detect_compressor
+from joblib.numpy_pickle_utils import _is_numpy_array_byte_order_mismatch
+from joblib.numpy_pickle_utils import _ensure_native_byte_order
 from joblib.compressor import (_COMPRESSORS, _LZ4_PREFIX, CompressorWrapper,
                                LZ4_NOT_INSTALLED_ERROR, BinaryZlibFile)
 
@@ -355,6 +358,7 @@ def test_compressed_pickle_dump_and_load(tmpdir):
     result_list = numpy_pickle.load(fname)
     for result, expected in zip(result_list, expected_list):
         if isinstance(expected, np.ndarray):
+            expected = _ensure_native_byte_order(expected)
             assert result.dtype == expected.dtype
             np.testing.assert_equal(result, expected)
         else:
@@ -394,6 +398,7 @@ def _check_pickle(filename, expected_list):
                         "pickle file.".format(filename))
             for result, expected in zip(result_list, expected_list):
                 if isinstance(expected, np.ndarray):
+                    expected = _ensure_native_byte_order(expected)
                     assert result.dtype == expected.dtype
                     np.testing.assert_equal(result, expected)
                 else:
@@ -457,6 +462,47 @@ def test_joblib_pickle_across_python_versions():
         _check_pickle(fname, expected_list)
 
 
+@with_numpy
+def test_numpy_array_byte_order_mismatch_detection():
+    # List of numpy arrays with big endian byteorder.
+    be_arrays = [np.array([(1, 2.0), (3, 4.0)],
+                          dtype=[('', '>i8'), ('', '>f8')]),
+                 np.arange(3, dtype=np.dtype('>i8')),
+                 np.arange(3, dtype=np.dtype('>f8'))]
+
+    # Verify the byteorder mismatch is correctly detected.
+    for array in be_arrays:
+        if sys.byteorder == 'big':
+            assert not _is_numpy_array_byte_order_mismatch(array)
+        else:
+            assert _is_numpy_array_byte_order_mismatch(array)
+        converted = _ensure_native_byte_order(array)
+        if converted.dtype.fields:
+            for f in converted.dtype.fields.values():
+                f[0].byteorder == '='
+        else:
+            assert converted.dtype.byteorder == "="
+
+    # List of numpy arrays with little endian byteorder.
+    le_arrays = [np.array([(1, 2.0), (3, 4.0)],
+                          dtype=[('', '<i8'), ('', '<f8')]),
+                 np.arange(3, dtype=np.dtype('<i8')),
+                 np.arange(3, dtype=np.dtype('<f8'))]
+
+    # Verify the byteorder mismatch is correctly detected.
+    for array in le_arrays:
+        if sys.byteorder == 'little':
+            assert not _is_numpy_array_byte_order_mismatch(array)
+        else:
+            assert _is_numpy_array_byte_order_mismatch(array)
+        converted = _ensure_native_byte_order(array)
+        if converted.dtype.fields:
+            for f in converted.dtype.fields.values():
+                f[0].byteorder == '='
+        else:
+            assert converted.dtype.byteorder == "="
+
+
 @parametrize('compress_tuple', [('zlib', 3), ('gzip', 3)])
 def test_compress_tuple_argument(tmpdir, compress_tuple):
     # Verify the tuple is correctly taken into account.
diff --git a/joblib/test/test_parallel.py b/joblib/test/test_parallel.py
index 2458ed9..7edeb85 100644
--- a/joblib/test/test_parallel.py
+++ b/joblib/test/test_parallel.py
@@ -1347,7 +1347,7 @@ def test_invalid_backend_hinting_and_constraints():
         # requiring shared memory semantics.
         Parallel(prefer='processes', require='sharedmem')
 
-    # It is inconsistent to ask explictly for a process-based parallelism
+    # It is inconsistent to ask explicitly for a process-based parallelism
     # while requiring shared memory semantics.
     with raises(ValueError):
         Parallel(backend='loky', require='sharedmem')
@@ -1563,7 +1563,7 @@ def _parent_max_num_threads_for(child_module, parent_info):
 
 def check_child_num_threads(workers_info, parent_info, num_threads):
     # Check that the number of threads reported in workers_info is consistent
-    # with the expectation. We need to be carefull to handle the cases where
+    # with the expectation. We need to be careful to handle the cases where
     # the requested number of threads is below max_num_thread for the library.
     for child_threadpool_info in workers_info:
         for child_module in child_threadpool_info: