Repository: openml/openml-python

Issue:
openml-python does not recognize corrupt dataset.pkl.py3 file
People are experiencing issues downloading dataset `41081` [here](https://github.com/ML-course/ML-course/issues/19).
Investigation is on-going, but it looks like the Python process gets killed when pickling the dataset to disk.
Subsequent loads of the cached data fail because the `dataset.pkl.py3` file is created (and sits at 0 bytes). Because the file exists it is attempted to be loaded without further sanity checks. I think the correct behavior would to check for this faulty file and if it exists either (1) raise an error or (2) delete the file, attempt to pickle is again, and raise an error only if this fails. I would prefer option 2.

edit: to clarify, this issue is about trying to mitigate the effect of corrupt files in cache. Not about the fact that the process gets killed (possibly during pickling). That warrants a separate issue which I will open after we have some more information from the linked issue.

Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=h1) Report
> Merging [#787](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=desc) into [develop](https://codecov.io/gh/openml/openml-python/commit/dcac17e6ba947e892159c430e5ed95cf2b039a9c?src=pr&el=desc) will **decrease** coverage by `0.12%`.
> The diff coverage is `68.65%`.

[![Impacted file tree graph](https://codecov.io/gh/openml/openml-python/pull/787/graphs/tree.svg?width=650&token=IlImE4jvlP&height=150&src=pr)](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop     #787      +/-   ##
===========================================
- Coverage    87.74%   87.62%   -0.13%     
===========================================
  Files           36       36              
  Lines         4187     4202      +15     
===========================================
+ Hits          3674     3682       +8     
- Misses         513      520       +7
```


| [Impacted Files](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [openml/datasets/dataset.py](https://codecov.io/gh/openml/openml-python/pull/787/diff?src=pr&el=tree#diff-b3Blbm1sL2RhdGFzZXRzL2RhdGFzZXQucHk=) | `82.76% <68.65%> (-1.31%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=footer). Last update [dcac17e...5ea3d07](https://codecov.io/gh/openml/openml-python/pull/787?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

PGijsbers: I reverted the change to shortcut the conversion warning.
The warning is now written to log instead of raised.
The problem is that it is also called in dataset#648 to convert the `y` data from pd.Series to pd.DataFrame, which it wouldn't do.

Patch:
diff --git a/examples/20_basic/README.txt b/examples/20_basic/README.txt
new file mode 100644
index 00000000..29c78711
--- /dev/null
+++ b/examples/20_basic/README.txt
@@ -0,0 +1,4 @@
+Introductory Examples
+=====================
+
+Introductory examples to the usage of the OpenML python connector.
diff --git a/examples/introduction_tutorial.py b/examples/20_basic/introduction_tutorial.py
similarity index 97%
rename from examples/introduction_tutorial.py
rename to examples/20_basic/introduction_tutorial.py
index 9cd88ceb..c54bb7b9 100644
--- a/examples/introduction_tutorial.py
+++ b/examples/20_basic/introduction_tutorial.py
@@ -1,8 +1,8 @@
 """
-Introduction
-============
+Setup
+=====
 
-An introduction to OpenML, followed up by a simple example.
+An example how to set up OpenML-Python followed up by a simple example.
 """
 ############################################################################
 # OpenML is an online collaboration platform for machine learning which allows
diff --git a/examples/20_basic/simple_datasets_tutorial.py b/examples/20_basic/simple_datasets_tutorial.py
new file mode 100644
index 00000000..6239cbb1
--- /dev/null
+++ b/examples/20_basic/simple_datasets_tutorial.py
@@ -0,0 +1,29 @@
+"""
+========
+Datasets
+========
+
+A basic tutorial on how to list and download datasets.
+"""
+############################################################################
+import openml
+
+############################################################################
+# List datasets
+# =============
+
+datasets_df = openml.datasets.list_datasets(output_format='dataframe')
+print(datasets_df.head(n=10))
+
+############################################################################
+# Download a dataset
+# ==================
+
+first_dataset_id = int(datasets_df['did'].iloc[0])
+dataset = openml.datasets.get_dataset(first_dataset_id)
+
+# Print a summary
+print("This is dataset '%s', the target feature is '%s'" %
+      (dataset.name, dataset.default_target_attribute))
+print("URL: %s" % dataset.url)
+print(dataset.description[:500])
diff --git a/examples/20_basic/simple_flows_and_runs_tutorial.py b/examples/20_basic/simple_flows_and_runs_tutorial.py
new file mode 100644
index 00000000..e3f02841
--- /dev/null
+++ b/examples/20_basic/simple_flows_and_runs_tutorial.py
@@ -0,0 +1,48 @@
+"""
+Flows and Runs
+==============
+
+A simple tutorial on how to train/run a model and how to upload the results.
+"""
+
+import openml
+from sklearn import ensemble, neighbors
+
+############################################################################
+# Train a machine learning model
+# ==============================
+#
+# .. warning:: This example uploads data. For that reason, this example
+#   connects to the test server at test.openml.org. This prevents the main
+#   server from crowding with example datasets, tasks, runs, and so on.
+
+openml.config.start_using_configuration_for_example()
+
+# NOTE: We are using dataset 20 from the test server: https://test.openml.org/d/20
+dataset = openml.datasets.get_dataset(20)
+X, y, categorical_indicator, attribute_names = dataset.get_data(
+    dataset_format='array',
+    target=dataset.default_target_attribute
+)
+clf = neighbors.KNeighborsClassifier(n_neighbors=3)
+clf.fit(X, y)
+
+############################################################################
+# Running a model on a task
+# =========================
+
+task = openml.tasks.get_task(119)
+clf = ensemble.RandomForestClassifier()
+run = openml.runs.run_model_on_task(clf, task)
+print(run)
+
+############################################################################
+# Publishing the run
+# ==================
+
+myrun = run.publish()
+print("Run was uploaded to http://test.openml.org/r/" + str(myrun.run_id))
+print("The flow can be found at http://test.openml.org/f/" + str(myrun.flow_id))
+
+############################################################################
+openml.config.stop_using_configuration_for_example()
diff --git a/examples/20_basic/simple_studies_tutorial.py b/examples/20_basic/simple_studies_tutorial.py
new file mode 100644
index 00000000..5198edf6
--- /dev/null
+++ b/examples/20_basic/simple_studies_tutorial.py
@@ -0,0 +1,7 @@
+"""
+=======
+Studies
+=======
+
+This is only a placeholder so far.
+"""
diff --git a/examples/30_extended/README.txt b/examples/30_extended/README.txt
new file mode 100644
index 00000000..432fa68f
--- /dev/null
+++ b/examples/30_extended/README.txt
@@ -0,0 +1,4 @@
+In-Depth Examples
+=================
+
+Extended examples for the usage of the OpenML python connector.
\ No newline at end of file
diff --git a/examples/create_upload_tutorial.py b/examples/30_extended/create_upload_tutorial.py
similarity index 100%
rename from examples/create_upload_tutorial.py
rename to examples/30_extended/create_upload_tutorial.py
diff --git a/examples/datasets_tutorial.py b/examples/30_extended/datasets_tutorial.py
similarity index 99%
rename from examples/datasets_tutorial.py
rename to examples/30_extended/datasets_tutorial.py
index 70da03d1..e94476c2 100644
--- a/examples/datasets_tutorial.py
+++ b/examples/30_extended/datasets_tutorial.py
@@ -14,6 +14,7 @@ import pandas as pd
 # **********
 #
 # * List datasets
+#
 #   * Use the output_format parameter to select output type
 #   * Default gives 'dict' (other option: 'dataframe')
 
diff --git a/examples/fetch_evaluations_tutorial.py b/examples/30_extended/fetch_evaluations_tutorial.py
similarity index 100%
rename from examples/fetch_evaluations_tutorial.py
rename to examples/30_extended/fetch_evaluations_tutorial.py
diff --git a/examples/flows_and_runs_tutorial.py b/examples/30_extended/flows_and_runs_tutorial.py
similarity index 99%
rename from examples/flows_and_runs_tutorial.py
rename to examples/30_extended/flows_and_runs_tutorial.py
index d65abdf2..6603b662 100644
--- a/examples/flows_and_runs_tutorial.py
+++ b/examples/30_extended/flows_and_runs_tutorial.py
@@ -132,7 +132,7 @@ run = openml.runs.run_model_on_task(
 # The run may be stored offline, and the flow will be stored along with it:
 run.to_filesystem(directory='myrun')
 
-# They made later be loaded and uploaded
+# They may be loaded and uploaded at a later time
 run = openml.runs.OpenMLRun.from_filesystem(directory='myrun')
 run.publish()
 
diff --git a/examples/run_setup_tutorial.py b/examples/30_extended/run_setup_tutorial.py
similarity index 100%
rename from examples/run_setup_tutorial.py
rename to examples/30_extended/run_setup_tutorial.py
diff --git a/examples/tasks_tutorial.py b/examples/30_extended/tasks_tutorial.py
similarity index 87%
rename from examples/tasks_tutorial.py
rename to examples/30_extended/tasks_tutorial.py
index c54ecdbd..f7a28ef9 100644
--- a/examples/tasks_tutorial.py
+++ b/examples/30_extended/tasks_tutorial.py
@@ -13,15 +13,14 @@ import pandas as pd
 # Tasks are identified by IDs and can be accessed in two different ways:
 #
 # 1. In a list providing basic information on all tasks available on OpenML.
-# This function will not download the actual tasks, but will instead download
-# meta data that can be used to filter the tasks and retrieve a set of IDs.
-# We can filter this list, for example, we can only list tasks having a
-# special tag or only tasks for a specific target such as
-# *supervised classification*.
-#
+#    This function will not download the actual tasks, but will instead download
+#    meta data that can be used to filter the tasks and retrieve a set of IDs.
+#    We can filter this list, for example, we can only list tasks having a
+#    special tag or only tasks for a specific target such as
+#    *supervised classification*.
 # 2. A single task by its ID. It contains all meta information, the target
-# metric, the splits and an iterator which can be used to access the
-# splits in a useful manner.
+#    metric, the splits and an iterator which can be used to access the
+#    splits in a useful manner.
 
 ############################################################################
 # Listing tasks
@@ -148,9 +147,9 @@ print(tasks[0])
 #
 # You can also create new tasks. Take the following into account:
 #
-# * You can only create tasks on _active_ datasets
+# * You can only create tasks on *active* datasets
 # * For now, only the following tasks are supported: classification, regression,
-# clustering, and learning curve analysis.
+#   clustering, and learning curve analysis.
 # * For now, tasks can only be created on a single dataset.
 # * The exact same task must not already exist.
 #
@@ -158,10 +157,9 @@ print(tasks[0])
 #
 # * task_type_id: The task type ID, required (see below). Required.
 # * dataset_id: The dataset ID. Required.
-# * target_name: The name of the attribute you aim to predict.
-# Optional.
+# * target_name: The name of the attribute you aim to predict. Optional.
 # * estimation_procedure_id : The ID of the estimation procedure used to create train-test
-# splits. Optional.
+#   splits. Optional.
 # * evaluation_measure: The name of the evaluation measure. Optional.
 # * Any additional inputs for specific tasks
 #
@@ -178,7 +176,7 @@ print(tasks[0])
 #
 # Let's create a classification task on a dataset. In this example we will do this on the
 # Iris dataset (ID=128 (on test server)). We'll use 10-fold cross-validation (ID=1),
-# and _predictive accuracy_ as the predefined measure (this can also be left open).
+# and *predictive accuracy* as the predefined measure (this can also be left open).
 # If a task with these parameters exist, we will get an appropriate exception.
 # If such a task doesn't exist, a task will be created and the corresponding task_id
 # will be returned.
@@ -212,8 +210,7 @@ openml.config.stop_using_configuration_for_example()
 
 
 ############################################################################
-# [Complete list of task types](https://www.openml.org/search?type=task_type)
-# [Complete list of model estimation procedures](
-# https://www.openml.org/search?q=%2520measure_type%3Aestimation_procedure&type=measure)
-# [Complete list of evaluation measures](
-# https://www.openml.org/search?q=measure_type%3Aevaluation_measure&type=measure)
+# * `Complete list of task types <https://www.openml.org/search?type=task_type>`_.
+# * `Complete list of model estimation procedures <https://www.openml.org/search?q=%2520measure_type%3Aestimation_procedure&type=measure>`_.
+# * `Complete list of evaluation measures <https://www.openml.org/search?q=measure_type%3Aevaluation_measure&type=measure>`_.
+#
diff --git a/examples/40_paper/2015_neurips_feurer_example.py b/examples/40_paper/2015_neurips_feurer_example.py
new file mode 100644
index 00000000..106d120d
--- /dev/null
+++ b/examples/40_paper/2015_neurips_feurer_example.py
@@ -0,0 +1,18 @@
+"""
+Feurer et al. (2015)
+====================
+
+A tutorial on how to get the datasets used in the paper introducing *Auto-sklearn* by Feurer et al..
+
+Auto-sklearn website: https://automl.github.io/auto-sklearn/master/
+
+Publication
+~~~~~~~~~~~
+
+| Efficient and Robust Automated Machine Learning
+| Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum and Frank Hutter
+| In *Advances in Neural Information Processing Systems 28*, 2015
+| Available at http://papers.nips.cc/paper/5872-efficient-and-robust-automated-machine-learning.pdf
+
+This is currently a placeholder.
+"""
diff --git a/examples/40_paper/2018_ida_strang_example.py b/examples/40_paper/2018_ida_strang_example.py
new file mode 100644
index 00000000..21165a8f
--- /dev/null
+++ b/examples/40_paper/2018_ida_strang_example.py
@@ -0,0 +1,17 @@
+"""
+Strang et al. (2018)
+====================
+
+A tutorial on how to reproduce the analysis conducted for *Don't Rule Out Simple Models
+Prematurely: A Large Scale Benchmark Comparing Linear and Non-linear Classifiers in OpenML*.
+
+Publication
+~~~~~~~~~~~
+
+| Don't Rule Out Simple Models Prematurely: A Large Scale Benchmark Comparing Linear and Non-linear Classifiers in OpenML
+| Benjamin Strang, Pieter Putten, Jan van Rijn and Frank Hutter
+| In *Advances in Intelligent Data Analysis XVII 17th International Symposium*, 2018
+| Available at https://link.springer.com/chapter/10.1007%2F978-3-030-01768-2_25
+
+This is currently a placeholder.
+"""
diff --git a/examples/40_paper/2018_kdd_rijn_example.py b/examples/40_paper/2018_kdd_rijn_example.py
new file mode 100644
index 00000000..5e03f09a
--- /dev/null
+++ b/examples/40_paper/2018_kdd_rijn_example.py
@@ -0,0 +1,16 @@
+"""
+van Rijn and Hutter (2018)
+==========================
+
+A tutorial on how to reproduce the paper *Hyperparameter Importance Across Datasets*.
+
+Publication
+~~~~~~~~~~~
+
+| Hyperparameter importance across datasets
+| Jan van Rijn and Frank Hutter
+| In *Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, 2018
+| Available at https://dl.acm.org/citation.cfm?id=3220058
+
+This is currently a placeholder.
+"""
diff --git a/examples/40_paper/2018_neurips_fusi_example.py b/examples/40_paper/2018_neurips_fusi_example.py
new file mode 100644
index 00000000..656f617f
--- /dev/null
+++ b/examples/40_paper/2018_neurips_fusi_example.py
@@ -0,0 +1,17 @@
+"""
+Fusi et al. (2018)
+==================
+
+A tutorial on how to get the datasets used in the paper introducing *Probabilistic Matrix
+Factorization for Automated Machine Learning* by Fusi et al..
+
+Publication
+~~~~~~~~~~~
+
+| Probabilistic Matrix Factorization for Automated Machine Learning
+| Nicolo Fusi and Rishit Sheth and Melih Elibol
+| In *Advances in Neural Information Processing Systems 31*, 2018
+| Available at http://papers.nips.cc/paper/7595-probabilistic-matrix-factorization-for-automated-machine-learning.pdf
+
+This is currently a placeholder.
+"""
diff --git a/examples/40_paper/2018_neurips_perrone_example.py b/examples/40_paper/2018_neurips_perrone_example.py
new file mode 100644
index 00000000..3262ee4a
--- /dev/null
+++ b/examples/40_paper/2018_neurips_perrone_example.py
@@ -0,0 +1,17 @@
+"""
+Perrone et al. (2018)
+=====================
+
+A tutorial on how to build a surrogate model based on OpenML data as done for *Scalable
+Hyperparameter Transfer Learning* by Perrone et al..
+
+Publication
+~~~~~~~~~~~
+
+| Scalable Hyperparameter Transfer Learning
+| Valerio Perrone and Rodolphe Jenatton and Matthias Seeger and Cedric Archambeau
+| In *Advances in Neural Information Processing Systems 31*, 2018
+| Available at http://papers.nips.cc/paper/7917-scalable-hyperparameter-transfer-learning.pdf
+
+This is currently a placeholder.
+"""
diff --git a/examples/40_paper/README.txt b/examples/40_paper/README.txt
new file mode 100644
index 00000000..9b571d55
--- /dev/null
+++ b/examples/40_paper/README.txt
@@ -0,0 +1,5 @@
+Usage in research papers
+========================
+
+These examples demonstrate how OpenML-Python can be used for research purposes by re-implementing
+its use in recent publications.
diff --git a/examples/README.txt b/examples/README.txt
index e41bfd4f..b90c0e1c 100644
--- a/examples/README.txt
+++ b/examples/README.txt
@@ -1,4 +1,3 @@
-Introductory Examples
-=====================
-
-General examples for OpenML usage.
+========
+Examples
+========
diff --git a/examples/sklearn/README.txt b/examples/sklearn/README.txt
deleted file mode 100644
index d61578cf..00000000
--- a/examples/sklearn/README.txt
+++ /dev/null
@@ -1,4 +0,0 @@
-Experiment Examples
-===================
-
-OpenML experiment examples using a sklearn classifier/pipeline.
diff --git a/examples/sklearn/openml_run_example.py b/examples/sklearn/openml_run_example.py
deleted file mode 100644
index 195a0aa7..00000000
--- a/examples/sklearn/openml_run_example.py
+++ /dev/null
@@ -1,40 +0,0 @@
-"""
-OpenML Run Example
-==================
-
-An example of an automated machine learning experiment.
-"""
-import openml
-from sklearn import impute, tree, pipeline
-
-############################################################################
-# .. warning:: This example uploads data. For that reason, this example
-#   connects to the test server at test.openml.org. This prevents the main
-#   server from crowding with example datasets, tasks, runs, and so on.
-
-openml.config.start_using_configuration_for_example()
-############################################################################
-
-# Uncomment and set your OpenML key. Don't share your key with others.
-# openml.config.apikey = 'YOURKEY'
-
-# Define a scikit-learn pipeline
-clf = pipeline.Pipeline(
-    steps=[
-        ('imputer', impute.SimpleImputer()),
-        ('estimator', tree.DecisionTreeClassifier())
-    ]
-)
-############################################################################
-# Download the OpenML task for the german credit card dataset.
-task = openml.tasks.get_task(97)
-############################################################################
-# Run the scikit-learn model on the task (requires an API key).
-run = openml.runs.run_model_on_task(clf, task)
-# Publish the experiment on OpenML (optional, requires an API key).
-run.publish()
-
-print('URL for run: %s/run/%d' % (openml.config.server, run.run_id))
-
-############################################################################
-openml.config.stop_using_configuration_for_example()
diff --git a/openml/datasets/dataset.py b/openml/datasets/dataset.py
index 630fac35..92cf63f0 100644
--- a/openml/datasets/dataset.py
+++ b/openml/datasets/dataset.py
@@ -4,7 +4,7 @@ import io
 import logging
 import os
 import pickle
-from typing import List, Optional, Union, Tuple, Iterable
+from typing import List, Optional, Union, Tuple, Iterable, Dict
 
 import arff
 import numpy as np
@@ -168,7 +168,7 @@ class OpenMLDataset(object):
         self.qualities = _check_qualities(qualities)
 
         if data_file is not None:
-            self.data_pickle_file = self._data_arff_to_pickle(data_file)
+            self.data_pickle_file = self._create_pickle_in_cache(data_file)
         else:
             self.data_pickle_file = None
 
@@ -202,26 +202,111 @@ class OpenMLDataset(object):
         body = '\n'.join(field_line_format.format(name, value) for name, value in fields)
         return header + body
 
-    def _data_arff_to_pickle(self, data_file):
-        data_pickle_file = data_file.replace('.arff', '.pkl.py3')
-        if os.path.exists(data_pickle_file):
-            with open(data_pickle_file, "rb") as fh:
-                data, categorical, attribute_names = pickle.load(fh)
+    def __eq__(self, other):
 
-            # Between v0.8 and v0.9 the format of pickled data changed from
-            # np.ndarray to pd.DataFrame. This breaks some backwards compatibility,
-            # e.g. for `run_model_on_task`. If a local file still exists with
-            # np.ndarray data, we reprocess the data file to store a pickled
-            # pd.DataFrame blob. See also #646.
-            if isinstance(data, pd.DataFrame) or scipy.sparse.issparse(data):
-                logger.debug("Data pickle file already exists.")
-                return data_pickle_file
+        if type(other) != OpenMLDataset:
+            return False
+
+        server_fields = {
+            'dataset_id',
+            'version',
+            'upload_date',
+            'url',
+            'dataset',
+            'data_file',
+        }
 
+        # check that the keys are identical
+        self_keys = set(self.__dict__.keys()) - server_fields
+        other_keys = set(other.__dict__.keys()) - server_fields
+        if self_keys != other_keys:
+            return False
+
+        # check that values of the common keys are identical
+        return all(self.__dict__[key] == other.__dict__[key]
+                   for key in self_keys)
+
+    def _download_data(self) -> None:
+        """ Download ARFF data file to standard cache directory. Set `self.data_file`. """
+        # import required here to avoid circular import.
+        from .functions import _get_dataset_arff
+        self.data_file = _get_dataset_arff(self)
+
+    def _get_arff(self, format: str) -> Dict:
+        """Read ARFF file and return decoded arff.
+
+        Reads the file referenced in self.data_file.
+
+        Parameters
+        ----------
+        format : str
+            Format of the ARFF file.
+            Must be one of 'arff' or 'sparse_arff' or a string that will be either of those
+            when converted to lower case.
+
+
+        Returns
+        -------
+        dict
+            Decoded arff.
+
+        """
+
+        # TODO: add a partial read method which only returns the attribute
+        # headers of the corresponding .arff file!
+        import struct
+
+        filename = self.data_file
+        bits = (8 * struct.calcsize("P"))
+        # Files can be considered too large on a 32-bit system,
+        # if it exceeds 120mb (slightly more than covtype dataset size)
+        # This number is somewhat arbitrary.
+        if bits != 64 and os.path.getsize(filename) > 120000000:
+            raise NotImplementedError("File {} too big for {}-bit system ({} bytes)."
+                                      .format(filename, os.path.getsize(filename), bits))
+
+        if format.lower() == 'arff':
+            return_type = arff.DENSE
+        elif format.lower() == 'sparse_arff':
+            return_type = arff.COO
+        else:
+            raise ValueError('Unknown data format {}'.format(format))
+
+        def decode_arff(fh):
+            decoder = arff.ArffDecoder()
+            return decoder.decode(fh, encode_nominal=True,
+                                  return_type=return_type)
+
+        if filename[-3:] == ".gz":
+            with gzip.open(filename) as fh:
+                return decode_arff(fh)
+        else:
+            with io.open(filename, encoding='utf8') as fh:
+                return decode_arff(fh)
+
+    def _parse_data_from_arff(
+            self,
+            arff_file_path: str
+    ) -> Tuple[Union[pd.DataFrame, scipy.sparse.csr_matrix], List[bool], List[str]]:
+        """ Parse all required data from arff file.
+
+        Parameters
+        ----------
+        arff_file_path : str
+            Path to the file on disk.
+
+        Returns
+        -------
+        Tuple[Union[pd.DataFrame, scipy.sparse.csr_matrix], List[bool], List[str]]
+            DataFrame or csr_matrix: dataset
+            List[bool]: List indicating which columns contain categorical variables.
+            List[str]: List of column names.
+        """
         try:
             data = self._get_arff(self.format)
         except OSError as e:
-            logger.critical("Please check that the data file %s is "
-                            "there and can be read.", data_file)
+            logger.critical("Please check that the data file {} is "
+                            "there and can be read.".format(arff_file_path))
             raise e
 
         ARFF_DTYPES_TO_PD_DTYPE = {
@@ -282,7 +367,6 @@ class OpenMLDataset(object):
             X = scipy.sparse.coo_matrix(
                 (X[0], (X[1], X[2])), shape=X_shape, dtype=np.float32)
             X = X.tocsr()
-
         elif self.format.lower() == 'arff':
             X = pd.DataFrame(data['data'], columns=attribute_names)
 
@@ -295,17 +379,73 @@ class OpenMLDataset(object):
                 else:
                     col.append(X[column_name])
             X = pd.concat(col, axis=1)
+        else:
+            raise ValueError("Dataset format '{}' is not a valid format.".format(self.format))
+
+        return X, categorical, attribute_names
+
+    def _create_pickle_in_cache(self, data_file: str) -> str:
+        """ Parse the arff and pickle the result. Update any old pickle objects. """
+        data_pickle_file = data_file.replace('.arff', '.pkl.py3')
+        if os.path.exists(data_pickle_file):
+            # Load the data to check if the pickle file is outdated (i.e. contains numpy array)
+            with open(data_pickle_file, "rb") as fh:
+                try:
+                    data, categorical, attribute_names = pickle.load(fh)
+                except EOFError:
+                    # The file is likely corrupt, see #780.
+                    # We deal with this when loading the data in `_load_data`.
+                    return data_pickle_file
+
+            # Between v0.8 and v0.9 the format of pickled data changed from
+            # np.ndarray to pd.DataFrame. This breaks some backwards compatibility,
+            # e.g. for `run_model_on_task`. If a local file still exists with
+            # np.ndarray data, we reprocess the data file to store a pickled
+            # pd.DataFrame blob. See also #646.
+            if isinstance(data, pd.DataFrame) or scipy.sparse.issparse(data):
+                logger.debug("Data pickle file already exists and is up to date.")
+                return data_pickle_file
+
+        # At this point either the pickle file does not exist, or it had outdated formatting.
+        # We parse the data from arff again and populate the cache with a recent pickle file.
+        X, categorical, attribute_names = self._parse_data_from_arff(data_file)
 
-        # Pickle the dataframe or the sparse matrix.
         with open(data_pickle_file, "wb") as fh:
-            pickle.dump((X, categorical, attribute_names), fh, -1)
+            pickle.dump((X, categorical, attribute_names), fh, pickle.HIGHEST_PROTOCOL)
         logger.debug("Saved dataset {did}: {name} to file {path}"
                      .format(did=int(self.dataset_id or -1),
                              name=self.name,
                              path=data_pickle_file)
                      )
+
         return data_pickle_file
 
+    def _load_data(self):
+        """ Load data from pickle or arff. Download data first if not present on disk. """
+        if self.data_pickle_file is None:
+            if self.data_file is None:
+                self._download_data()
+            self.data_pickle_file = self._create_pickle_in_cache(self.data_file)
+
+        try:
+            with open(self.data_pickle_file, "rb") as fh:
+                data, categorical, attribute_names = pickle.load(fh)
+        except EOFError:
+            logging.warning(
+                "Detected a corrupt cache file loading dataset %d: '%s'. "
+                "We will continue loading data from the arff-file, "
+                "but this will be much slower for big datasets. "
+                "Please manually delete the cache file if you want openml-python "
+                "to attempt to reconstruct it."
+                "" % (self.dataset_id, self.data_pickle_file)
+            )
+            data, categorical, attribute_names = self._parse_data_from_arff(self.data_file)
+        except FileNotFoundError:
+            raise ValueError("Cannot find a pickle file for dataset {} at "
+                             "location {} ".format(self.name, self.data_pickle_file))
+
+        return data, categorical, attribute_names
+
     def push_tag(self, tag):
         """Annotates this data set with a tag on the server.
 
@@ -326,73 +466,6 @@ class OpenMLDataset(object):
         """
         _tag_entity('data', self.dataset_id, tag, untag=True)
 
-    def __eq__(self, other):
-
-        if type(other) != OpenMLDataset:
-            return False
-
-        server_fields = {
-            'dataset_id',
-            'version',
-            'upload_date',
-            'url',
-            'dataset',
-            'data_file',
-        }
-
-        # check that the keys are identical
-        self_keys = set(self.__dict__.keys()) - server_fields
-        other_keys = set(other.__dict__.keys()) - server_fields
-        if self_keys != other_keys:
-            return False
-
-        # check that values of the common keys are identical
-        return all(self.__dict__[key] == other.__dict__[key]
-                   for key in self_keys)
-
-    def _get_arff(self, format):
-        """Read ARFF file and return decoded arff.
-
-        Reads the file referenced in self.data_file.
-
-        Returns
-        -------
-        dict
-            Decoded arff.
-
-        """
-
-        # TODO: add a partial read method which only returns the attribute
-        # headers of the corresponding .arff file!
-        import struct
-
-        filename = self.data_file
-        bits = (8 * struct.calcsize("P"))
-        # Files can be considered too large on a 32-bit system,
-        # if it exceeds 120mb (slightly more than covtype dataset size)
-        # This number is somewhat arbitrary.
-        if bits != 64 and os.path.getsize(filename) > 120000000:
-            return NotImplementedError("File too big")
-
-        if format.lower() == 'arff':
-            return_type = arff.DENSE
-        elif format.lower() == 'sparse_arff':
-            return_type = arff.COO
-        else:
-            raise ValueError('Unknown data format %s' % format)
-
-        def decode_arff(fh):
-            decoder = arff.ArffDecoder()
-            return decoder.decode(fh, encode_nominal=True,
-                                  return_type=return_type)
-
-        if filename[-3:] == ".gz":
-            with gzip.open(filename) as fh:
-                return decode_arff(fh)
-        else:
-            with io.open(filename, encoding='utf8') as fh:
-                return decode_arff(fh)
-
     @staticmethod
     def _convert_array_format(data, array_format, attribute_names):
         """Convert a dataset to a given array format.
@@ -417,6 +490,7 @@ class OpenMLDataset(object):
                 else returns data as is
 
         """
+
         if array_format == "array" and not scipy.sparse.issparse(data):
             # We encode the categories such that they are integer to be able
             # to make a conversion to numeric for backward compatibility
@@ -445,7 +519,10 @@ class OpenMLDataset(object):
             return pd.SparseDataFrame(data, columns=attribute_names)
         else:
             data_type = "sparse-data" if scipy.sparse.issparse(data) else "non-sparse data"
-            warn("Cannot convert {} to '{}'. Returning input data.".format(data_type, array_format))
+            logging.warning(
+                "Cannot convert %s (%s) to '%s'. Returning input data."
+                % (data_type, type(data), array_format)
+            )
         return data
 
     @staticmethod
@@ -461,12 +538,6 @@ class OpenMLDataset(object):
         raw_cat = pd.Categorical(col, ordered=True, categories=categories)
         return pd.Series(raw_cat, index=series.index, name=series.name)
 
-    def _download_data(self) -> None:
-        """ Download ARFF data file to standard cache directory. Set `self.data_file`. """
-        # import required here to avoid circular import.
-        from .functions import _get_dataset_arff
-        self.data_file = _get_dataset_arff(self)
-
     def get_data(
             self,
             target: Optional[Union[List[str], str]] = None,
@@ -507,18 +578,7 @@ class OpenMLDataset(object):
         attribute_names : List[str]
             List of attribute names.
         """
-        if self.data_pickle_file is None:
-            if self.data_file is None:
-                self._download_data()
-            self.data_pickle_file = self._data_arff_to_pickle(self.data_file)
-
-        path = self.data_pickle_file
-        if not os.path.exists(path):
-            raise ValueError("Cannot find a pickle file for dataset %s at "
-                             "location %s " % (self.name, path))
-        else:
-            with open(path, "rb") as fh:
-                data, categorical, attribute_names = pickle.load(fh)
+        data, categorical, attribute_names = self._load_data()
 
         to_exclude = []
         if not include_row_id and self.row_id_attribute is not None:
diff --git a/setup.cfg b/setup.cfg
index 726c8fa7..156baa3b 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -4,3 +4,11 @@ description-file = README.md
 [tool:pytest]
 filterwarnings =
     ignore:the matrix subclass:PendingDeprecationWarning
+
+[flake8]
+exclude =
+    # the following file and directory can be removed when the descriptions
+    # are shortened. More info at:
+    # https://travis-ci.org/openml/openml-python/jobs/590382001
+    examples/30_extended/tasks_tutorial.py
+    examples/40_paper
\ No newline at end of file


Test patch:
diff --git a/tests/test_datasets/test_dataset.py b/tests/test_datasets/test_dataset.py
index cabad956..132cf458 100644
--- a/tests/test_datasets/test_dataset.py
+++ b/tests/test_datasets/test_dataset.py
@@ -26,6 +26,7 @@ class OpenMLDatasetTest(TestBase):
         # these datasets have some boolean features
         self.pc4 = openml.datasets.get_dataset(1049, download_data=False)
         self.jm1 = openml.datasets.get_dataset(1053, download_data=False)
+        self.iris = openml.datasets.get_dataset(61, download_data=False)
 
     def test_get_data_array(self):
         # Basic usage
@@ -190,6 +191,18 @@ class OpenMLDatasetTest(TestBase):
         # Check that no label is mapped to 3, since it is reserved for label '4'.
         self.assertEqual(np.sum(y == 3), 0)
 
+    def test_get_data_corrupt_pickle(self):
+        # Lazy loaded dataset, populate cache.
+        self.iris.get_data()
+        # Corrupt pickle file, overwrite as empty.
+        with open(self.iris.data_pickle_file, "w") as fh:
+            fh.write("")
+        # Despite the corrupt file, the data should be loaded from the ARFF file.
+        # A warning message is written to the python logger.
+        xy, _, _, _ = self.iris.get_data()
+        self.assertIsInstance(xy, pd.DataFrame)
+        self.assertEqual(xy.shape, (150, 5))
+
 
 class OpenMLDatasetTestOnTestServer(TestBase):
     def setUp(self):