Repository: OpenNMT/OpenNMT-py

Issue:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 0: invalid continuation byt
with the current V2.0.0rc2, on Windows 10, Python 3.7.7.  onmt_build_vocab doesn't save utf-8 src/tgt files.  

I build a data set and it's fine, but when I try to train I get

```PS C:\Users\Admin\Documents\pytorch\bugrepo\utf-8-bug-report-for-opennmt> onmt_train -config .\config.yaml       
[2020-11-26 11:08:30,632 INFO] Missing transforms field for corpus_1 data, set to default: [].
[2020-11-26 11:08:30,633 WARNING] Corpus corpus_1's weight should be given. We default it to 1 for you.
[2020-11-26 11:08:30,633 INFO] Missing transforms field for valid data, set to default: [].
[2020-11-26 11:08:30,633 INFO] Parsed 2 corpora from -data.
[2020-11-26 11:08:30,634 INFO] Get special vocabs from Transforms: {'src': set(), 'tgt': set()}.
[2020-11-26 11:08:30,640 INFO] Loading vocab from text file...
[2020-11-26 11:08:30,641 INFO] Loading src vocabulary from ./out.vocab.src
Traceback (most recent call last):
  File "c:\users\admin\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python37\Scripts\onmt_train.exe\__main__.py", line 7, in <module>
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\bin\train.py", line 169, in main
    train(opt)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\bin\train.py", line 103, in train
    checkpoint, fields, transforms_cls = _init_train(opt)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\bin\train.py", line 80, in _init_train
    fields, transforms_cls = prepare_fields_transforms(opt)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\bin\train.py", line 34, in prepare_fields_transforms
    opt, src_specials=specials['src'], tgt_specials=specials['tgt'])
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\inputters\fields.py", line 33, in build_dynamic_fields
    min_freq=opts.src_words_min_frequency)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\inputters\inputter.py", line 272, in _load_vocab   
    vocab, has_count = _read_vocab_file(vocab_path, name)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\inputters\inputter.py", line 475, in _read_vocab_file
    lines = [line.strip() for line in f if line.strip()]
  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\onmt\inputters\inputter.py", line 475, in <listcomp>    
    lines = [line.strip() for line in f if line.strip()]
  File "c:\users\admin\appdata\local\programs\python\python37\lib\codecs.py", line 714, in __next__
    return next(self.reader)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\codecs.py", line 645, in __next__
    line = self.readline()
  File "c:\users\admin\appdata\local\programs\python\python37\lib\codecs.py", line 558, in readline
    data = self.read(readsize, firstline=True)
  File "c:\users\admin\appdata\local\programs\python\python37\lib\codecs.py", line 504, in read
    newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 0: invalid continuation byte
```

Repro case here:

https://github.com/increpare/utf-8-bug-report-for-opennmt

Adding in utf8 encoding to save_counter in build_vocab fixes it (about to make pull request).


Discussion:

Resolving patch:
diff --git a/onmt/bin/build_vocab.py b/onmt/bin/build_vocab.py
--- a/onmt/bin/build_vocab.py
+++ b/onmt/bin/build_vocab.py
@@ -40,7 +40,7 @@ def build_vocab_main(opts):
 
     def save_counter(counter, save_path):
         check_path(save_path, exist_ok=opts.overwrite, log=logger.warning)
-        with open(save_path, "w") as fo:
+        with open(save_path, "w",encoding="utf8") as fo:
             for tok, count in counter.most_common():
                 fo.write(tok + "\t" + str(count) + "\n")
 


Test patch: