Many things broken since the new 4.0.0 release

### Describe the bug

The new changes in 4.0.0 are breaking many datasets, including those from lm-evaluation-harness. 
I am trying to revert back to older versions, like 3.6.0 to make the eval work but I keep getting:
``` Python
File /venv/main/lib/python3.12/site-packages/datasets/features/features.py:1474, in generate_from_dict(obj)
   1471 class_type = _FEATURE_TYPES.get(_type, None) or globals().get(_type, None)
   1473 if class_type is None:
-> 1474     raise ValueError(f"Feature type '{_type}' not found. Available feature types: {list(_FEATURE_TYPES.keys())}")
   1476 if class_type == LargeList:
   1477     feature = obj.pop("feature")

ValueError: Feature type 'List' not found. Available feature types: ['Value', 'ClassLabel', 'Translation', 'TranslationVariableLanguages', 'LargeList', 'Sequence', 'Array2D', 'Array3D', 'Array4D', 'Array5D', 'Audio', 'Image', 'Video', 'Pdf']

```



### Steps to reproduce the bug

``` Python
import lm_eval
model_eval = lm_eval.models.huggingface.HFLM(pretrained=model, tokenizer=tokenizer)
lm_eval.evaluator.simple_evaluate(model_eval, tasks=["winogrande"], num_fewshot=5, batch_size=1)
```

### Expected behavior

Older `datasets` versions should work just fine as before

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-60-generic-x86_64-with-glibc2.39
- Python version: 3.12.11
- `huggingface_hub` version: 0.33.1
- PyArrow version: 20.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0


Happy to take a look, do you have a list of impacted datasets ?

Thanks @lhoestq , related to lm-eval, at least `winogrande`, `mmlu` and `hellaswag`, based on my tests yesterday. But many others like <a href="https://huggingface.co/datasets/lukaemon/bbh">bbh</a>, most probably others too. 

Hi @mobicham ,

I was having the same issue `ValueError: Feature type 'List' not found` yesterday, when I tried to load my dataset using the `load_dataset()` function.
By updating to `4.0.0`, I don't see this error anymore.

p.s. I used `Sequence` in replace of list when building my dataset (see below)
```
features = Features({
    ...
    "objects": Sequence({
        "id": Value("int64"),
        "bbox": Sequence(Value("float32"), length=4),
        "category": Value("string")
    }),
    ...
})
dataset = Dataset.from_dict(data_dict)
dataset = dataset.cast(features)

``` 


The issue comes from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train), [allenai/winogrande](https://huggingface.co/datasets/allenai/winogrande), [lukaemon/bbh](https://huggingface.co/datasets/lukaemon/bbh) and [Rowan/hellaswag](https://huggingface.co/datasets/Rowan/hellaswag) which are all unsupported in `datasets` 4.0 since they are based on python scripts. Fortunately there are PRs to fix those datasets (I did some of them a year ago but dataset authors haven't merged yet... will have to ping people again about it and update here):

- https://huggingface.co/datasets/hails/mmlu_no_train/discussions/2 ⚙️ 
- https://huggingface.co/datasets/allenai/winogrande/discussions/6 merged ! ✅ 
- https://huggingface.co/datasets/Rowan/hellaswag/discussions/7 merged ! ✅ 
- https://huggingface.co/datasets/lukaemon/bbh/discussions/2 merged ! ✅ 

Thank you very much @lhoestq , I will try next week 👍 

I get this error when using datasets 3.5.1 to load a dataset saved with datasets 4.0.0. If you are hitting this issue, make sure that both dataset saving code and the loading code are <4.0.0 or >=4.0.0.

This broke several lm-eval-harness workflows for me and reverting to older versions of datasets is not fixing the issue, does anyone have a workaround?

> I get this error when using datasets 3.5.1 to load a dataset saved with datasets 4.0.0. If you are hitting this issue, make sure that both dataset saving code and the loading code are <4.0.0 or >=4.0.0.

`datasets` 4.0 can load datasets saved using any older version. But the other way around is not always true: if you save a dataset with `datasets` 4.0 it may use the new `List` type that requires 4.0 and raise `ValueError: Feature type 'List' not found.`

However issues with lm eval harness seem to come from another issue: unsupported dataset scripts (see https://github.com/huggingface/datasets/issues/7676#issuecomment-3057550659)

> This broke several lm-eval-harness workflows for me and reverting to older versions of datasets is not fixing the issue, does anyone have a workaround?

when reverting to an old `datasets` version I'd encourage you to clear your cache (by default it is located at `~/.cache/huggingface/datasets`) otherwise it might try to load a `List` type that didn't exist in old versions