Cannot load the dataset go_emotions
Opened by arame (open)

### Describe the bug

When I run the following code I get an exception;
`go_emotions = load_dataset("go_emotions")`

> AttributeError                            Traceback (most recent call last)
Cell In[6], [line 1](vscode-notebook-cell:?execution_count=6&line=1)
----> [1](vscode-notebook-cell:?execution_count=6&line=1) go_emotions = load_dataset("go_emotions")
      [2](vscode-notebook-cell:?execution_count=6&line=2) data = go_emotions.data

File [c:\Users\hijik\anaconda3\Lib\site-packages\datasets\load.py:2523](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2523), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   [2518](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2518) verification_mode = VerificationMode(
   [2519](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2519)     (verification_mode or VerificationMode.BASIC_CHECKS) if not save_infos else VerificationMode.ALL_CHECKS
   [2520](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2520) )
   [2522](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2522) # Create a dataset builder
-> [2523](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2523) builder_instance = load_dataset_builder(
   [2524](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2524)     path=path,
   [2525](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2525)     name=name,
   [2526](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2526)     data_dir=data_dir,
   [2527](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2527)     data_files=data_files,
   [2528](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2528)     cache_dir=cache_dir,
   [2529](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2529)     features=features,
   [2530](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2530)     download_config=download_config,
   [2531](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2531)     download_mode=download_mode,
   [2532](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2532)     revision=revision,
   [2533](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2533)     token=token,
   [2534](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2534)     storage_options=storage_options,
   [2535](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2535)     trust_remote_code=trust_remote_code,
   [2536](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/load.py:2536)     _require_default_config_name=name is None,
...
---> [63](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/utils/_dill.py:63)         if issubclass(obj_type, transformers.PreTrainedTokenizerBase):
     [64](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/utils/_dill.py:64)             pklregister(obj_type)(_save_transformersPreTrainedTokenizerBase)
     [66](file:///C:/Users/hijik/anaconda3/Lib/site-packages/datasets/utils/_dill.py:66) # Unwrap `torch.compile`-ed functions

AttributeError: module 'transformers' has no attribute 'PreTrainedTokenizerBase'
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?10bc0728-6947-456e-9a3e-f056872b04c6) or open in a [text editor](command:workbench.action.openLargeOutput?10bc0728-6947-456e-9a3e-f056872b04c6). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

### Steps to reproduce the bug

```
from datasets import load_dataset
go_emotions = load_dataset("go_emotions")
```

### Expected behavior

Should simply load the variable with the data from the file

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.16.1
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.4
- `huggingface_hub` version: 0.20.3
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0

Comments:
Thanks for reporting, @arame.

I guess you have an old version of `transformers` (that submodule is present in `transformers` since version 3.0.1, since nearly 4 years ago). If you update it, the error should disappear:
```shell
pip install -U transformers
```

On the other hand, I am wondering: does it make sense to use `transformers` in this case, even if we don't need it to load the `go_emotions` dataset (already converted to Parquet files)?
- Maybe @mariosasko can give some insight, as he included these code lines:
  - #6454

https://github.com/huggingface/datasets/blob/9751fb14594d354e952f0ebdfaf31cb203b011e7/src/datasets/utils/_dill.py#L60-L63


The linked code lazily registers a custom reducer for `transformers.PreTrainedTokenizerBase` only if `transformers` have already been imported (imports are expensive, so we check `sys.modules`).

However, the logic does not account for `transformers<3`, so we should add a version check to fix that.

> The linked code lazily registers a custom reducer for `transformers.PreTrainedTokenizerBase` only if `transformers` have already been imported (imports are expensive, so we check `sys.modules`).
> 
> However, the logic does not account for `transformers<3`, so we should add a version check to fix that.

Thank you for that Mario. Would this fix solve the problem and do you have any idea when it will be done? 
I tried the pip install suggested by Albert and it made no difference.

I tried running the code today and the problem appears to be fixed.