Bugfix for encoding error during GPT2Tokenizer.from_pretrained('local…
Author: DrStoop
Pull request: yes

…/path/to/mode')

BUG DESCRIPTION: Loading GPT2-tokenizer from local path with
GPT2Tokenizer.from_pretrained(pretrained_model_name_or_path='local/path/to/model')
returns following error due to encoding error for json.load():
Traceback (most recent call last):
  File "/opt/pycharm-2019.1.3/helpers/pydev/pydevd.py", line 1758, in <module>
    main()
  File "/opt/pycharm-2019.1.3/helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/opt/pycharm-2019.1.3/helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/pycharm-2019.1.3/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/developer/AmI/transfer-learning-conv-ai/pytorch_transformer_evaluation.py", line 24, in <module>
    cache_dir=None)
  File "/home/developer/AmI/pytorch-transformers/pytorch_transformers/tokenization_utils.py", line 151, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/developer/AmI/pytorch-transformers/pytorch_transformers/tokenization_utils.py", line 240, in _from_pretrained
    tokenizer = cls(*inputs, **kwargs)
  File "/home/developer/AmI/pytorch-transformers/pytorch_transformers/tokenization_gpt2.py", line 110, in __init__
    self.encoder = json.load(open(vocab_file))
  File "/conda/envs/rapids/lib/python3.6/json/__init__.py", line 296, in load
    return loads(fp.read(),
  File "/conda/envs/rapids/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 840: ordinal not in range(128)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=h1) Report
> Merging [#905](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/46cc9dd2b51a152b2e262ec12e40dddd13235aba?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #905      +/-   ##
==========================================
+ Coverage   79.03%   79.03%   +<.01%     
==========================================
  Files          34       34              
  Lines        6234     6235       +1     
==========================================
+ Hits         4927     4928       +1     
  Misses       1307     1307
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `96.69% <100%> (+0.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=footer). Last update [46cc9dd...f8d9977](https://codecov.io/gh/huggingface/pytorch-transformers/pull/905?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).