load_dataset('emotion')    UnicodeDecodeError
Opened by L-Block-C (open)

### Describe the bug

**emotions = load_dataset('emotion')**

_UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte_

### Steps to reproduce the bug

load_dataset('emotion') 

### Expected behavior

succese

### Environment info

py3.10
transformers  4.41.0.dev0
datasets     2.19.0

Comments:
I met the same problem, here is my code:
```
from datasets import load_dataset

ds_name = "togethercomputer/RedPajama-Data-1T"
ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
```
And output error is:
```
Traceback (most recent call last):
  File "/home/yatorho/doc/projs/TransformerEngine/local/download_redpajama.py", line 10, in <module>
    ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2277, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1923, in dataset_module_factory
    raise e1 from None
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1875, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
                                                                       ^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
```
My `datasets` version is 2.21.0. Any help here would be appreciated!


> I met the same problem, here is my code:
> 
> ```
> from datasets import load_dataset
> 
> ds_name = "togethercomputer/RedPajama-Data-1T"
> ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
> ```
> 
> And output error is:
> 
> ```
> Traceback (most recent call last):
>   File "/home/yatorho/doc/projs/TransformerEngine/local/download_redpajama.py", line 10, in <module>
>     ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
>          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2606, in load_dataset
>     builder_instance = load_dataset_builder(
>                        ^^^^^^^^^^^^^^^^^^^^^
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2277, in load_dataset_builder
>     dataset_module = dataset_module_factory(
>                      ^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1923, in dataset_module_factory
>     raise e1 from None
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1875, in dataset_module_factory
>     can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
>                                                                        ^^^^^^^^
>   File "<frozen codecs>", line 322, in decode
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
> ```
> 
> My `datasets` version is 2.21.0. Any help here would be appreciated!

I passed encoding="utf-16" to the `load_dataset` call and now it works for me.
```
ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD, encoding="utf-16")
```