Apply utf-8 encoding to all datasets
State: closed | Author: lewtun (MEMBER)

## Description
This PR applies utf-8 encoding for all instances of `with open(...) as f` to all Python files in `datasets/`. As suggested by @thomwolf in #468 , we use regular expressions and the following function

```python
def apply_encoding_on_file_open(filepath: str):
    """Apply UTF-8 encoding for all instances where a non-binary file is opened."""
    
    with open(filepath, 'r', encoding='utf-8') as input_file:
        regexp = re.compile(r"(?!.*\b(?:encoding|rb|w|wb|w+|wb+|ab|ab+)\b)(?<=\s)(open)\((.*)\)")
        input_text = input_file.read()
        match = regexp.search(input_text)
        
        if match:
            output = regexp.sub(lambda m: m.group()[:-1]+', encoding="utf-8")', input_text)
            with open(filepath, 'w', encoding='utf-8') as output_file:
                output_file.write(output)
```

to perform the replacement. 

Note:

1. I excluded all _**binary files**_ from the search since it's possible some objects are opened for which the encoding doesn't make sense. Please correct me if I'm wrong and I'll tweak the regexp accordingly
2. There were two edge cases where the regexp failed (e.g. two `open` instances on a single line), but I decided to just fix these manually in the interest of time.
3. I only applied the replacement to files in `datasets/`. Let me know if this should be extended to other places like `metrics/`
4. I have implemented a unit test that should catch missing encodings in future CI runs

Closes #468  and possibly #347 

Comments: Not sure why the AWS test is failing - perhaps I made too many concurrent CI builds 😢. Can someone please rerun the CI to check the error is not on my end?

I pushed an improved docstring and the unit tests now pass, which suggests the previous failure on AWS was simply a timeout error. 

For some reason the docs are now failing to build, but does not seem related to my changes:
```
Warning, treated as error:
/home/circleci/nlp/src/nlp/dataset_dict.py:docstring of nlp.DatasetDict.filter:27:Inline interpreted text or phrase reference start-string without end-string.
make: *** [Makefile:20: html] Error 2
```

Any ideas what's going wrong?

The build_doc fail has been fixed on master.
It was due to the latest update of sphinx that has some issues, so I pinned the previous version for now.

I noticed that you also changed the Apache Beam `open` to also use utf-8. However it doesn't have an `encoding` parameter.
Therefore you should ignore lines like

```python
beam.io.filesystems.FileSystems.open(filepath)
```

I guess you could add a rule to your regex to only include the `open` call that have a space right before it.

Good catch @lhoestq! Your suggestion to match on `open(...)` with a whitespace was a great idea - it allowed me to simplify the regexp considerably 😄.

I fixed the Apache Beam false positives and also caught a few problems in `json.load()`, e.g.
```python
relation_name_map = json.load(open(rel_info), encoding='utf-8')
```

I've tested that the new regexp doesn't reintroduce these false positives, so I think the PR is ready for another review.

Ok to merge this @lhoestq ?