GitHub issue #6522: Loading HF Hub Dataset (private org repo) fails to load all features
State: open | Created: 2023-12-21T12:26:35Z

### Describe the bug

When pushing a `Dataset` with multiple `Features` (`input`, `output`, `tags`) to Huggingface Hub (private org repo), and later downloading the `Dataset`, only `input` and `output` load - I believe the expected behavior is for all `Features` to be loaded by default?

### Steps to reproduce the bug

Pushing the data. `data_concat` is a `list` of `dict`s.
```python
for datum in data_concat:
    datum_tags = {d["key"]: d["value"] for d in datum["tags"]}
    split_fraction = # some logic that generates a train/test split number
    if split_faction < test_fraction:
        data_test.append(datum)
    else:
        data_train.append(datum)

dataset = DatasetDict(
    {
        "train": Dataset.from_list(data_train),
        "test": Dataset.from_list(data_test),
        "full": Dataset.from_list(data_concat),
    },
)

dataset_shuffled = dataset.shuffle(seed=shuffle_seed)

dataset_shuffled.push_to_hub(
    repo_id=hf_repo_id,
    private=True,
    config_name=m,
    revision=revision,
    token=hf_token,
)
```

Loading it later:
```python
        dataset = datasets.load_dataset(
            path=hf_repo_id,
            name=name,
            token=hf_token,
        )
```

Produces:
```
DatasetDict({
    train: Dataset({
        features: ['input', 'output'],
        num_rows: <obfuscated>
    })
    test: Dataset({
        features: ['input', 'output'],
        num_rows: <obfuscated>
    })
    full: Dataset({
        features: ['input', 'output'],
        num_rows: <obfuscated>
    })
})
```

### Expected behavior

The expected result is below:
```
DatasetDict({
    train: Dataset({
        features: ['input', 'output', 'tags'],
        num_rows: <obfuscated>
    })
    test: Dataset({
        features: ['input', 'output', 'tags'],
        num_rows: <obfuscated>
    })
    full: Dataset({
        features: ['input', 'output', 'tags'],
        num_rows: <obfuscated>
    })
})
```

My workaround is as follows:
```python
dsinfo = datasets.get_dataset_config_info(
    path=data_files,
    config_name=data_config,
    token=hf_token,
)

allfeatures = dsinfo.features.copy()
if "tags" not in allfeatures:
    allfeatures["tags"] = [{"key": Value(dtype="string", id=None), "value": Value(dtype="string", id=None)}]

dataset = datasets.load_dataset(
    path=data_files,
    name=data_config,
    features=allfeatures,
    token=hf_token,
)
```

Interestingly enough (and perhaps a related bug?), if I don't add the `tags` to `allfeatures` above (i.e. only loading `input` and `output`), it throws an error when executing `load_dataset`:

```
ValueError: Couldn't cast
tags: list<element: struct<key: string, value: string>>
  child 0, element: struct<key: string, value: string>
      child 0, key: string
      child 1, value: string
input: <obfuscated>
output: <obfuscated>
-- schema metadata --
huggingface: '{"info": {"features": {"tags": [{"key": {"dtype": "string",' + 532
to
{'input': <obfuscated>, 'output': <obfuscated>
because column names don't match
```

Traceback for this:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/load.py", line 2152, in load_dataset
    builder_instance.download_and_prepare(
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 948, in download_and_prepare
    self._download_and_prepare(
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 1043, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 1805, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/Users/bt/github/core/.venv/lib/python3.11/site-packages/datasets/builder.py", line 1950, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Environment info

- `datasets` version: 2.15.0
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0