UnicodeDecodeError while loading PAN-X task of XTREME dataset
State: closed | Author: lewtun (MEMBER)

Hi 🤗  team!

## Description of the problem
I'm running into a `UnicodeDecodeError` while trying to load the PAN-X subset the XTREME dataset: 

```
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-5-1d61f439b843> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    528     ignore_verifications = ignore_verifications or save_infos
    529     # Download/copy dataset processing script
--> 530     module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
    531 
    532     # Get dataset builder class from the processing script

/usr/local/lib/python3.6/dist-packages/nlp/load.py in prepare_module(path, download_config, dataset, force_local_path, **download_kwargs)
    265 
    266     # Download external imports if needed
--> 267     imports = get_imports(local_path)
    268     local_imports = []
    269     library_imports = []

/usr/local/lib/python3.6/dist-packages/nlp/load.py in get_imports(file_path)
    156     lines = []
    157     with open(file_path, mode="r") as f:
--> 158         lines.extend(f.readlines())
    159 
    160     logger.info("Checking %s for additional imports.", file_path)

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27 
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 111: ordinal not in range(128)
```

## Steps to reproduce
Install from nlp's master branch
```python
pip install git+https://github.com/huggingface/nlp.git
```
then run
```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
```

## OS / platform details

- `nlp` version: latest from master
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

## Proposed solution
Either change [line 762](https://github.com/huggingface/nlp/blob/7ada00b1d62f94eee22a7df38c6b01e3f27194b7/datasets/xtreme/xtreme.py#L762) in `xtreme.py` to include UTF-8 encoding:

```
# old
with open(filepath) as f
# new
with open(filepath, encoding='utf-8') as f
```

or raise a warning that suggests setting the locale explicitly, e.g.
```python
import locale
locale.setlocale(locale.LC_ALL, 'C.UTF-8')
```
I have a preference for the first solution. Let me know if you agree and I'll be happy to implement the simple fix!

Comments: Indeed. Solution 1 is the simplest.

This is actually a recurring problem.
I think we should scan all the datasets with regexpr to fix the use of `open()` without encodings.
And probably add a test in the CI to forbid using this in the future.

I'm happy to tackle the broader problem - will open a PR when it's ready!

That would be awesome!

I've created a simple function that seems to do the trick:

```python
def apply_encoding_on_file_open(filepath: str):
    """Apply UTF-8 encoding for all instances where a non-binary file is opened."""
    
    with open(filepath, 'r', encoding='utf-8') as input_file:
        regexp = re.compile(r"""
                            (?!.*\b(?:encoding|rb|wb|wb+|ab|ab+)\b)
                            (open)
                            \((.*)\)
                            """)
        input_text = input_file.read()
        match = regexp.search(input_text)
        
        if match:
            print('Found match!', match.group())
            # append utf-8 encoding to matching groups in-place
            output = regexp.sub(lambda m: m.group()[:-1]+', encoding="utf-8")', input_text)
            with open(filepath, 'w', encoding='utf-8') as output_file:
                output_file.write(output)
        else:
            print("No match found!")
```

The regexp does a negative lookahead to avoid matching on cases where the encoding is already specified or when binary files are involved.

From an implementation perspective:

* Would it make sense to include this function in `nlp-cli` so that we can run something like
```
nlp-cli fix_encoding path/to/folder
```
and the command recursively fixes all files in the target?
* What is the desired behaviour in the CI test? Here we could either have a simple script that we run as a `job` in the CI and raises an error if a missing encoding is detected. Alternatively we could incorporate this behaviour into the CLI and run that in the CI.

Please let me know what you prefer among the alternatives.


I realised I was overthinking the problem, so decided to just run the regexp over the codebase and make the PR. In other words, we can ignore my comments about using the CLI 😸