Repository: huggingface/datasets
Pull request: Fix ClassLabel to/from dict when passed names_file
Currently, `names_file` is a field of the data class `ClassLabel`, thus appearing when transforming it to dict (when saving infos). Afterwards, when trying to read it from infos, it conflicts with the other field `names`.

This PR, removes `names_file` as a field of the data class `ClassLabel`.
- it is only used at instantiation to generate the `labels` field

Fix #3631.

Problem statement:
Labels conflict when loading a local CSV file.
## Describe the bug
I am trying to load a local CSV file with a separate file containing label names. It is successfully loaded for the first time, but when I try to load it again, there is a conflict between provided labels and the cached dataset info. Disabling caching globally and/or using `download_mode="force_redownload"` did not help.

## Steps to reproduce the bug
```python
load_dataset('csv', data_files='data/my_data.csv',
                        features=Features(text=Value(dtype='string'),
                                          label=ClassLabel(names_file='data/my_data_labels.txt')))
```
`my_data.csv` file has the following structure:
```
text,label
"example1",0
"example2",1
...
```
and the `my_data_labels.txt` looks like this:
```
label1
label2
...
```

## Expected results
Successfully loaded dataset.

## Actual results
```python
  File "/usr/local/lib/python3.8/site-packages/datasets/load.py", line 1706, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.8/site-packages/datasets/builder.py", line 766, in as_dataset
    datasets = utils.map_nested(
  File "/usr/local/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 261, in map_nested
    mapped = [
  File "/usr/local/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 262, in <listcomp>
    _single_map_nested((function, obj, types, None, True))
  File "/usr/local/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 197, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.8/site-packages/datasets/builder.py", line 797, in _build_single_dataset
    ds = self._as_dataset(
  File "/usr/local/lib/python3.8/site-packages/datasets/builder.py", line 872, in _as_dataset
    return Dataset(fingerprint=fingerprint, **dataset_kwargs)
  File "/usr/local/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 638, in __init__
    inferred_features = Features.from_arrow_schema(arrow_table.schema)
  File "/usr/local/lib/python3.8/site-packages/datasets/features/features.py", line 1242, in from_arrow_schema
    return Features.from_dict(metadata["info"]["features"])
  File "/usr/local/lib/python3.8/site-packages/datasets/features/features.py", line 1271, in from_dict
    obj = generate_from_dict(dic)
  File "/usr/local/lib/python3.8/site-packages/datasets/features/features.py", line 1076, in generate_from_dict
    return {key: generate_from_dict(value) for key, value in obj.items()}
  File "/usr/local/lib/python3.8/site-packages/datasets/features/features.py", line 1076, in <dictcomp>
    return {key: generate_from_dict(value) for key, value in obj.items()}
  File "/usr/local/lib/python3.8/site-packages/datasets/features/features.py", line 1083, in generate_from_dict
    return class_type(**{k: v for k, v in obj.items() if k in field_names})
  File "<string>", line 7, in __init__
  File "/usr/local/lib/python3.8/site-packages/datasets/features/features.py", line 776, in __post_init__
    raise ValueError("Please provide either names or names_file but not both.")
ValueError: Please provide either names or names_file but not both.
```

## Environment info
- `datasets` version: 1.18.0
- Python version: 3.8.2

Hints from issue discussion:

Commit message: Fix ClassLabel to/from dict with names_file

Patch to review:
diff --git a/src/datasets/features/features.py b/src/datasets/features/features.py
index 05212b718db..fab4e531e3d 100644
--- a/src/datasets/features/features.py
+++ b/src/datasets/features/features.py
@@ -19,7 +19,7 @@
 import re
 import sys
 from collections.abc import Iterable
-from dataclasses import _asdict_inner, dataclass, field, fields
+from dataclasses import InitVar, _asdict_inner, dataclass, field, fields
 from functools import reduce
 from operator import mul
 from typing import Any, ClassVar, Dict, List, Optional
@@ -761,7 +761,7 @@ class ClassLabel:
 
     num_classes: int = None
     names: List[str] = None
-    names_file: Optional[str] = None
+    names_file: InitVar[Optional[str]] = None
     id: Optional[str] = None
     # Automatically constructed
     dtype: ClassVar[str] = "int64"
@@ -770,7 +770,8 @@ class ClassLabel:
     _int2str: ClassVar[Dict[int, int]] = None
     _type: str = field(default="ClassLabel", init=False, repr=False)
 
-    def __post_init__(self):
+    def __post_init__(self, names_file):
+        self.names_file = names_file
         if self.names_file is not None and self.names is not None:
             raise ValueError("Please provide either names or names_file but not both.")
         # Set self.names
diff --git a/tests/features/test_features.py b/tests/features/test_features.py
index 0e966169af6..f30437ca70e 100644
--- a/tests/features/test_features.py
+++ b/tests/features/test_features.py
@@ -17,6 +17,7 @@
     _cast_to_python_objects,
     cast_to_python_objects,
     encode_nested_example,
+    generate_from_dict,
     string_to_arrow,
 )
 from datasets.info import DatasetInfo
@@ -269,6 +270,20 @@ def test_classlabel_int2str():
         classlabel.int2str(len(names))
 
 
+@pytest.mark.parametrize("class_label_arg", ["names", "names_file"])
+def test_class_label_to_and_from_dict(class_label_arg, tmp_path_factory):
+    names = ["negative", "positive"]
+    names_file = str(tmp_path_factory.mktemp("features") / "labels.txt")
+    with open(names_file, "w", encoding="utf-8") as f:
+        f.write("\n".join(names))
+    if class_label_arg == "names":
+        class_label = ClassLabel(names=names)
+    elif class_label_arg == "names_file":
+        class_label = ClassLabel(names_file=names_file)
+    generated_class_label = generate_from_dict(asdict(class_label))
+    assert generated_class_label == class_label
+
+
 def test_encode_nested_example_sequence_with_none():
     schema = Sequence(Value("int32"))
     obj = None


Code review comments:
diff_hunk: @@ -761,7 +761,7 @@ class ClassLabel:
 
     num_classes: int = None
     names: List[str] = None
-    names_file: Optional[str] = None
+    names_file: InitVar[Optional[str]] = None
path: src/datasets/features/features.py
text: @user1:
```suggestion
    names_file: InitVar[Optional[str]] = None  # use InitVar to ignore this field when converting to dict format
```