Repository: automl/auto-sklearn

Issue:
BufferError: memoryview: underlying buffer is not C-contiguous
Hi devs,

I am experimenting with automl, and ran into an issue while using LFW face features dataset.  It keeps throwing memory buffer error.

Here is the GIST:
https://gist.github.com/campbx/133a2e1eea6ac8b384a810a9d904071d

Thanks!



Discussion:
Thanks for reporting this issue. In order to allow us reproducing this, could you please copy us the output of `pip freeze` and post the link where we can download the version of the LFW dataset you've used (there is more than one on the LFW website).

I am running default Jupyter/scipy-notebook Docker image, with TPOT and XGBoost added.
`pip freeze`output: 

auto-sklearn==0.0.1
backports.shutil-get-terminal-size==1.0.0
bokeh==0.11.1
cloudpickle==0.1.1
conda==4.1.11
conda-env==2.5.0a0
ConfigArgParse==0.10.0
ConfigSpace==0.1.1
cycler==0.10.0
Cython==0.23.5
deap==1.0.2
decorator==4.0.10
dill==0.2.5
docutils==0.12
fastcache==1.0.2
h5py==2.6.0
ipykernel==4.3.1
ipython==5.0.0
ipython-genutils==0.1.0
ipywidgets==5.1.5
Jinja2==2.8
jsonschema==2.5.1
jupyter-client==4.3.0
jupyter-core==4.1.0
jupyterhub==0.5.0
liac-arff==2.1.0
llvmlite==0.8.0
lockfile==0.12.2
MarkupSafe==0.23
matplotlib==1.5.1
mistune==0.7.3
mpmath==0.19
nbconvert==4.2.0
nbformat==4.0.1
networkx==1.11
notebook==4.2.2
numba==0.23.1
numexpr==2.5.2
numpy==1.10.4
pamela==0.2.1
pandas==0.18.1
patsy==0.4.1
pexpect==4.2.0
pickleshare==0.7.3
Pillow==3.2.0
prompt-toolkit==1.0.4
psutil==4.3.0
ptyprocess==0.5.1
pycosat==0.6.1
pycrypto==2.6.1
Pygments==2.1.3
pynisher==0.4.2
pyparsing==2.1.7
pyrfr==0.2.0
python-dateutil==2.5.3
pytz==2016.6.1
PyYAML==3.11
pyzmq==15.3.0
requests==2.10.0
ruamel-yaml===-VERSION
scikit-image==0.11.3
scikit-learn==0.17.1
scipy==0.17.1
seaborn==0.7.1
simplegeneric==0.8.1
six==1.10.0
smac==0.0.1
SQLAlchemy==1.0.13
statsmodels==0.6.1
sympy==1.0
terminado==0.6
tornado==4.4.1
TPOT==0.4.1
tqdm==4.8.3
traitlets==4.2.2
update-checker==0.12
wcwidth==0.1.7
widgetsnbextension==1.2.6
xgboost==0.6a2

I was able to reproduce the issue with a much simpler dataset.  Attaching the Indonesia contraceptive study dataset and descriptors here:

[dtcmc.data.txt](https://github.com/automl/auto-sklearn/files/418834/dtcmc.data.txt)
[dtcmc.names.txt](https://github.com/automl/auto-sklearn/files/418835/dtcmc.names.txt)

And the code to run the model along with output is in this gist:

https://gist.github.com/campbx/133a2e1eea6ac8b384a810a9d904071d 

**Some more tests**

Looking at the flags, X dataset is not C_Contiguous which is causing the crash in fit(). 

```
X_train.shape, y_train.shape, X_test.shape, y_test.shape
((1104, 9), (1104,), (369, 9), (369,))

X_train.dtype, y_train.dtype, X_test.dtype, y_test.dtype
(dtype('int64'), dtype('int64'), dtype('int64'), dtype('int64'))

X_train.flags, y_train.flags, X_test.flags, y_test.flags
(  C_CONTIGUOUS : False
   F_CONTIGUOUS : True
   OWNDATA : False
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False,   

   C_CONTIGUOUS : True
   F_CONTIGUOUS : True
   OWNDATA : True
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False,   

   C_CONTIGUOUS : False
   F_CONTIGUOUS : True
   OWNDATA : False
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False,   

   C_CONTIGUOUS : True
   F_CONTIGUOUS : True
   OWNDATA : True
   WRITEABLE : True
   ALIGNED : True
   UPDATEIFCOPY : False
)
```

As a temporary solutions, I reshaped the array to be C_Contiguous.  You can remove this line from the GIST to reproduce the crash.

`X_train=np.ascontig‌​uousarray(X_train)`

This time automl finished fit(), but threw following errors in score():

```
IndexErrorTraceback (most recent call last)
/opt/conda/lib/python3.5/site-packages/autosklearn/metrics/util.py in create_multiclass_solution(solution, prediction)
    167         try:
--> 168             solution_binary[i, int(solution[i])] = 1
    169         except IndexError as e:

IndexError: index 3 is out of bounds for axis 1 with size 3

During handling of the above exception, another exception occurred:

IndexErrorTraceback (most recent call last)
<ipython-input-18-1c98ea7597b0> in <module>()
----> 1 automl.score(X_test, y_test)

/opt/conda/lib/python3.5/site-packages/autosklearn/automl.py in score(self, X, y)
    534         return calculate_score(y, prediction, self._task,
    535                                self._metric, self._label_num,
--> 536                                logger=self._logger)
    537 
    538     def show_models(self):

/opt/conda/lib/python3.5/site-packages/autosklearn/evaluation/util.py in calculate_score(solution, prediction, task_type, metric, num_classes, all_scoring_functions, logger)
     37         else:
     38             score = classification_metrics.calculate_score(
---> 39                 metric, solution, prediction, task=task_type)
     40     return score

/opt/conda/lib/python3.5/site-packages/autosklearn/metrics/classification_metrics.py in calculate_score(metric, solution, prediction, task)
     22 
     23     metric = METRIC_TO_STRING[metric]
---> 24     return globals()[metric](solution, prediction, task)
     25 
     26 

/opt/conda/lib/python3.5/site-packages/autosklearn/metrics/classification_metrics.py in acc_metric(solution, prediction, task)
     62     elif task == MULTICLASS_CLASSIFICATION:
     63         if len(solution.shape) == 1:
---> 64             solution = create_multiclass_solution(solution, prediction)
     65         elif len(solution.shape ) == 2:
     66             if solution.shape[1] > 1:

/opt/conda/lib/python3.5/site-packages/autosklearn/metrics/util.py in create_multiclass_solution(solution, prediction)
    170             raise IndexError('too many indices to array. array has shape %s, '
    171                              'indices are "%s %s"' %
--> 172                              (solution_binary.shape, str(i), solution[i]))
    173     return solution_binary

IndexError: too many indices to array. array has shape (369, 3), indices are "1 3"
```

I think the second error with `score()` is fixed in the development branch (#46). What worked for me is using the scikit-learn accuracy method (as you had in the cell below): 

```
predictions = automl.predict(X_test)
sklearn.metrics.accuracy_score(y_test, predictions + 1))
```

The `+ 1` is necessary due to the aforementioned bug. I'll now be looking into the initial error. Hopefully, we can soon release the next version of auto-sklearn which fixes this bug.

The only resource on the actual problem I can find is this issue: [https://github.com/joblib/joblib/issues/94](https://github.com/joblib/joblib/issues/94) and I think the best solution is the one you already gave, make the array c-contiguous. I'll leave this open, so we can fix this issue. 

I also got the same problem when running with anaconda python3.


Resolving patch:
diff --git a/autosklearn/automl.py b/autosklearn/automl.py
--- a/autosklearn/automl.py
+++ b/autosklearn/automl.py
@@ -25,6 +25,7 @@
     pipeline
 from autosklearn.ensemble_builder import EnsembleBuilder
 from autosklearn.smbo import AutoMLSMBO
+from autosklearn.util.hash import hash_numpy_array
 
 
 class AutoML(BaseEstimator):
@@ -148,9 +149,7 @@ def fit(self, X, y,
         self._backend.context.create_directories()
 
         if dataset_name is None:
-            m = hashlib.md5()
-            m.update(X.data)
-            dataset_name = m.hexdigest()
+            dataset_name = hash_numpy_array(X)
 
         self._backend.save_start_time(self._seed)
         self._stopwatch = StopWatch()
diff --git a/autosklearn/util/hash.py b/autosklearn/util/hash.py
new file mode 100644
--- /dev/null
+++ b/autosklearn/util/hash.py
@@ -0,0 +1,13 @@
+import hashlib
+
+
+def hash_numpy_array(X):
+    m = hashlib.md5()
+
+    if X.flags['C_CONTIGUOUS']:
+        m.update(X.data)
+    else:
+        m.update(X.T.data)
+
+    hash = m.hexdigest()
+    return hash
\ No newline at end of file


Test patch:
diff --git a/test/test_automl/test_estimators.py b/test/test_automl/test_estimators.py
--- a/test/test_automl/test_estimators.py
+++ b/test/test_automl/test_estimators.py
@@ -227,8 +227,7 @@ def test_cv_results(self):
         del cls
         self._tearDown(output)
 
-
-
+        
 class AutoMLClassifierTest(unittest.TestCase):
 
     def test_multiclass_prediction(self):
diff --git a/test/test_util/test_hash.py b/test/test_util/test_hash.py
new file mode 100644
--- /dev/null
+++ b/test/test_util/test_hash.py
@@ -0,0 +1,51 @@
+import unittest
+
+import numpy as np
+
+from autosklearn.util.hash import hash_numpy_array
+
+
+class HashTests(unittest.TestCase):
+
+    def test_c_contiguous_array(self):
+        array = np.array([[1, 2], [3, 4]])
+
+        hash = hash_numpy_array(array)
+
+        self.assertIsNotNone(hash)
+
+    def test_f_contiguous_array(self):
+        array = np.array([[1, 2], [3, 4]])
+        array = np.asfortranarray(array)
+
+        hash = hash_numpy_array(array)
+
+        self.assertIsNotNone(hash)
+
+    def test_transpose_arrays(self):
+        c_array = np.array([[1, 2], [3, 4]])
+        f_array = np.array([[1, 3], [2, 4]])
+        f_array = np.asfortranarray(f_array)
+
+        c_hash = hash_numpy_array(c_array)
+        f_hash = hash_numpy_array(f_array)
+
+        self.assertEqual(c_hash, f_hash)
+
+    def test_same_data_arrays(self):
+        first_array = np.array([[1, 2], [3, 4]])
+        second_array = np.array([[1, 2], [3, 4]])
+
+        first_hash = hash_numpy_array(first_array)
+        second_hash = hash_numpy_array(second_array)
+
+        self.assertEqual(first_hash, second_hash)
+
+    def test_different_data_arrays(self):
+        first_array = np.array([[1, 2], [3, 4]])
+        second_array = np.array([[1, 3], [2, 4]])
+
+        first_hash = hash_numpy_array(first_array)
+        second_hash = hash_numpy_array(second_array)
+
+        self.assertNotEqual(first_hash, second_hash)
\ No newline at end of file