UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)
Author: andreybond
State: closed
Labels: name: bug

## Describe the bug
Error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)" is thrown when downloading dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset 
datasets = load_dataset("nielsr/XFUN", "xfun.ja")
```

## Expected results
Dataset should be downloaded without exceptions

## Actual results
Stack trace (for the second-time execution):
Downloading and preparing dataset xfun/xfun.ja to /root/.cache/huggingface/datasets/nielsr___xfun/xfun.ja/0.0.0/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477...
Downloading data files: 100%
2/2 [00:00<00:00, 88.48it/s]
Extracting data files: 100%
2/2 [00:00<00:00, 79.60it/s]

UnicodeDecodeErrorTraceback (most recent call last)
<ipython-input-31-79c26bd1109c> in <module>
      1 from datasets import load_dataset
      2 
----> 3 datasets = load_dataset("nielsr/XFUN", "xfun.ja")

/usr/local/lib/python3.6/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    604         )
    605 
--> 606         # By default, return all splits
    607         if split is None:
    608             split = {s: s for s in self.info.splits}

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    692         Args:
    693             split: `datasets.Split` which subset of the data to read.
--> 694 
    695         Returns:
    696             `Dataset`

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys)

/usr/local/lib/python3.6/dist-packages/tqdm/notebook.py in __iter__(self)
    252         if not self.disable:
    253             self.display(check_delay=False)
--> 254 
    255     def __iter__(self):
    256         try:

/usr/local/lib/python3.6/dist-packages/tqdm/std.py in __iter__(self)
   1183             for obj in iterable:
   1184                 yield obj
-> 1185             return
   1186 
   1187         mininterval = self.mininterval

~/.cache/huggingface/modules/datasets_modules/datasets/nielsr--XFUN/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477/XFUN.py in _generate_examples(self, filepaths)
    140             logger.info("Generating examples from = %s", filepath)
    141             with open(filepath[0], "r") as f:
--> 142                 data = json.load(f)
    143 
    144             for doc in data["documents"]:

/usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    294 
    295     """
--> 296     return loads(fp.read(),
    297         cls=cls, object_hook=object_hook,
    298         parse_float=parse_float, parse_int=parse_int,

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27 
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)



## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0 (but reproduced with many previous versions)
- Platform: Docker: Linux da5b74136d6b 5.3.0-1031-azure #32~18.04.1-Ubuntu SMP Mon Jun 22 15:27:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ; Base docker image is : huggingface/transformers-pytorch-cpu
- Python version: 3.6.9
- PyArrow version:   6.0.1


Comments: Hi @andreybond, thanks for reporting.

Unfortunately, I'm not able to able to reproduce your issue:
```python
In [4]: from datasets import load_dataset
   ...: datasets = load_dataset("nielsr/XFUN", "xfun.ja")

In [5]: datasets
Out[5]: 
DatasetDict({
    train: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 194
    })
    validation: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 71
    })
})
```

The only reason I can imagine this issue may arise is if your default encoding is not "UTF-8" (and it is ASCII instead). This is usually the case on Windows machines; but you say your environment is a Linux machine. Maybe you change your machine default encoding?

Could you please check this?
```python
In [6]: import sys

In [7]: sys.getdefaultencoding()
Out[7]: 'utf-8'
```

I opened a PR in the original dataset loading script:
-  microsoft/unilm#677

and fixed the corresponding dataset script on the Hub:
- https://huggingface.co/datasets/nielsr/XFUN/commit/73ba5e026621e05fb756ae0f267eb49971f70ebd

import sys
sys.getdefaultencoding()

returned: 'utf-8'

---------------------

I've just cloned master branch - your fix works! Thank you!