Repository: openml/openml-python

Issue:
Cannot load sparse datasets with categorical variables any more
Can be reproduced with:

```
import openml

openml.datasets.get_dataset(395)
```

Discussion:
codecov-io: # [Codecov](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=h1) Report
> Merging [#823](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=desc) into [develop](https://codecov.io/gh/openml/openml-python/commit/4853d7cfa279e5fb348cc96471c4cf61fdaf8b23?src=pr&el=desc) will **increase** coverage by `1.23%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/openml/openml-python/pull/823/graphs/tree.svg?width=650&token=IlImE4jvlP&height=150&src=pr)](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop     #823      +/-   ##
===========================================
+ Coverage    88.05%   89.28%   +1.23%     
===========================================
  Files           36       36              
  Lines         4243     4768     +525     
===========================================
+ Hits          3736     4257     +521     
- Misses         507      511       +4
```


| [Impacted Files](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [openml/datasets/dataset.py](https://codecov.io/gh/openml/openml-python/pull/823/diff?src=pr&el=tree#diff-b3Blbm1sL2RhdGFzZXRzL2RhdGFzZXQucHk=) | `90.15% <100%> (+2.68%)` | :arrow_up: |
| [openml/flows/flow.py](https://codecov.io/gh/openml/openml-python/pull/823/diff?src=pr&el=tree#diff-b3Blbm1sL2Zsb3dzL2Zsb3cucHk=) | `94.24% <0%> (+0.37%)` | :arrow_up: |
| [openml/extensions/sklearn/extension.py](https://codecov.io/gh/openml/openml-python/pull/823/diff?src=pr&el=tree#diff-b3Blbm1sL2V4dGVuc2lvbnMvc2tsZWFybi9leHRlbnNpb24ucHk=) | `94.01% <0%> (+2.73%)` | :arrow_up: |
| [openml/exceptions.py](https://codecov.io/gh/openml/openml-python/pull/823/diff?src=pr&el=tree#diff-b3Blbm1sL2V4Y2VwdGlvbnMucHk=) | `97.67% <0%> (+4.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=footer). Last update [4853d7c...b06d348](https://codecov.io/gh/openml/openml-python/pull/823?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/openml/datasets/dataset.py b/openml/datasets/dataset.py
index fb760583..fde45cc5 100644
--- a/openml/datasets/dataset.py
+++ b/openml/datasets/dataset.py
@@ -109,17 +109,18 @@ class OpenMLDataset(object):
                  paper_url=None, update_comment=None,
                  md5_checksum=None, data_file=None, features=None,
                  qualities=None, dataset=None):
-        if description and not re.match("^[\x00-\x7F]*$", description):
-            # not basiclatin (XSD complains)
-            raise ValueError("Invalid symbols in description: {}".format(
-                description))
-        if citation and not re.match("^[\x00-\x7F]*$", citation):
-            # not basiclatin (XSD complains)
-            raise ValueError("Invalid symbols in citation: {}".format(
-                citation))
-        if not re.match("^[a-zA-Z0-9_\\-\\.\\(\\),]+$", name):
-            # regex given by server in error message
-            raise ValueError("Invalid symbols in name: {}".format(name))
+        if dataset_id is None:
+            if description and not re.match("^[\x00-\x7F]*$", description):
+                # not basiclatin (XSD complains)
+                raise ValueError("Invalid symbols in description: {}".format(
+                    description))
+            if citation and not re.match("^[\x00-\x7F]*$", citation):
+                # not basiclatin (XSD complains)
+                raise ValueError("Invalid symbols in citation: {}".format(
+                    citation))
+            if not re.match("^[a-zA-Z0-9_\\-\\.\\(\\),]+$", name):
+                # regex given by server in error message
+                raise ValueError("Invalid symbols in name: {}".format(name))
         # TODO add function to check if the name is casual_string128
         # Attributes received by querying the RESTful API
         self.dataset_id = int(dataset_id) if dataset_id is not None else None
@@ -257,6 +258,7 @@ class OpenMLDataset(object):
             when converted to lower case.
 
 
+
         Returns
         -------
         dict
@@ -331,13 +333,15 @@ class OpenMLDataset(object):
         attribute_names = []
         categories_names = {}
         categorical = []
-        for name, type_ in data['attributes']:
+        for i, (name, type_) in enumerate(data['attributes']):
             # if the feature is nominal and the a sparse matrix is
             # requested, the categories need to be numeric
             if (isinstance(type_, list)
                     and self.format.lower() == 'sparse_arff'):
                 try:
-                    np.array(type_, dtype=np.float32)
+                    # checks if the strings which should be the class labels
+                    # can be encoded into integers
+                    pd.factorize(type_)[0]
                 except ValueError:
                     raise ValueError(
                         "Categorical data needs to be numeric when "


Test patch:
diff --git a/tests/test_datasets/test_dataset.py b/tests/test_datasets/test_dataset.py
index 5388c4bb..9d107637 100644
--- a/tests/test_datasets/test_dataset.py
+++ b/tests/test_datasets/test_dataset.py
@@ -9,6 +9,7 @@ from scipy import sparse
 import openml
 from openml.testing import TestBase
 from openml.exceptions import PyOpenMLError
+from openml.datasets import OpenMLDataset, OpenMLDataFeature
 
 
 class OpenMLDatasetTest(TestBase):
@@ -341,6 +342,16 @@ class OpenMLDatasetTestSparse(TestBase):
         self.assertListEqual(categorical, [False] * 19998)
         self.assertEqual(y.shape, (600, ))
 
+    def test_get_sparse_categorical_data_id_395(self):
+        dataset = openml.datasets.get_dataset(395, download_data=True)
+        feature = dataset.features[3758]
+        self.assertTrue(isinstance(dataset, OpenMLDataset))
+        self.assertTrue(isinstance(feature, OpenMLDataFeature))
+        self.assertEqual(dataset.name, 're1.wc')
+        self.assertEqual(feature.name, 'CLASS_LABEL')
+        self.assertEqual(feature.data_type, 'nominal')
+        self.assertEqual(len(feature.nominal_values), 25)
+
 
 class OpenMLDatasetQualityTest(TestBase):
     def test__check_qualities(self):