Repository: huggingface/datasets (Python)

Issue:
Dict feature is non-nullable while nested dict feature is
When i try to create a `Dataset` object with None values inside a dict column, like this: 
```python
from datasets import Dataset, Features, Value

Dataset.from_dict(
    {
        "dict": [{"a": 0, "b": 0}, None],
    }, features=Features(
        {"dict": {"a": Value("int16"), "b": Value("int16")}}
    )
)
```
i get `ValueError: Got None but expected a dictionary instead`.


At the same time, having None in _nested_ dict feature works, for example, this doesn't throw any errors:

```python
from datasets import Dataset, Features, Value, Sequence

dataset = Dataset.from_dict(
    {
        "list_dict": [[{"a": 0, "b": 0}], None],
        "sequence_dict": [[{"a": 0, "b": 0}], None],
    }, features=Features({
        "list_dict": [{"a": Value("int16"), "b": Value("int16")}],
        "sequence_dict": Sequence({"a": Value("int16"), "b": Value("int16")}),
    })
)
```
Other types of features also seem to be nullable (but I haven't checked all of them).

Version of `datasets` is the latest atm (2.18.0)


Is this an expected behavior or a bug?


Discussion:
It looks like a bug, by default every feature should be nullable.
I've linked a PR with a fix :)
@mariosasko awesome thank you!



Patch:
diff --git a/src/datasets/features/features.py b/src/datasets/features/features.py
index 5aedd42c4b2..893c9f9a1b0 100644
--- a/src/datasets/features/features.py
+++ b/src/datasets/features/features.py
@@ -1937,7 +1937,7 @@ def encode_column(self, column, column_name: str):
             `list[Any]`
         """
         column = cast_to_python_objects(column)
-        return [encode_nested_example(self[column_name], obj) for obj in column]
+        return [encode_nested_example(self[column_name], obj, level=1) for obj in column]
 
     def encode_batch(self, batch):
         """
@@ -1955,7 +1955,7 @@ def encode_batch(self, batch):
             raise ValueError(f"Column mismatch between batch {set(batch)} and features {set(self)}")
         for key, column in batch.items():
             column = cast_to_python_objects(column)
-            encoded_batch[key] = [encode_nested_example(self[key], obj) for obj in column]
+            encoded_batch[key] = [encode_nested_example(self[key], obj, level=1) for obj in column]
         return encoded_batch
 
     def decode_example(self, example: dict, token_per_repo_id: Optional[Dict[str, Union[str, bool, None]]] = None):


Test patch:
diff --git a/tests/features/test_features.py b/tests/features/test_features.py
index 8c14f157817..b1ba7e9745f 100644
--- a/tests/features/test_features.py
+++ b/tests/features/test_features.py
@@ -405,6 +405,16 @@ def test_encode_batch_with_example_with_empty_first_elem():
     assert encoded_batch == {"x": [[[0], [1]], [[], [1]]]}
 
 
+def test_encode_column_dict_with_none():
+    features = Features(
+        {
+            "x": {"a": ClassLabel(names=["a", "b"]), "b": Value("int32")},
+        }
+    )
+    encoded_column = features.encode_column([{"a": "a", "b": 1}, None], "x")
+    assert encoded_column == [{"a": 0, "b": 1}, None]
+
+
 @pytest.mark.parametrize(
     "feature",
     [