regarding #1026 pull request 
Author: rabeehk
Pull request: yes

Dear Thomas,
This is regarding my #1026 pull request, here is my understanding of  the reproducibility issue I was getting:

- on line 451,  in the codes tokenizer is reloaded without setting do_lower_case, then if you use both do_train+do_eval, you will get different results than if you do do_eval only on the same directory since if you use only do_eval only, tokenizer is read from line 408 where do_lower_case is considered.

- the second issue I see is that if you do both do_train and do_eval you read tokenizer from the output_dir, but if you do only do_eval you read tokenizer from args.model_name_or_path which can be different and could results in different results, so this is better to reload the tokenizer once from output_dir during the evaluation and remove it from training part. 

thanks.
Best regards,
Rabeeh


Comments:
Oh I see what you mean, indeed that's a more general issue with saving and loading tokenizer with specific configuration parameters. This is actually also relevant to our work on XLM's tokenizer in #1092

Dear Thomas,
The pull request #1026 does not work unfortunately when using eval_all_check_points, and I was wondering if you could undo that merge, sorry for this, this new pull request here works for me.
thanks. 

Ok let's do that for now and I'll think about a more general way to save tokenizer configurations.

awesome. thanks 

# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=h1) Report
> Merging [#1158](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e0caab0cf052c86e456bc4b4fdac5788433ed935?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1158   +/-   ##
======================================
  Coverage    80.7%   80.7%           
======================================
  Files          46      46           
  Lines        7411    7411           
======================================
  Hits         5981    5981           
  Misses       1430    1430
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=footer). Last update [e0caab0...0a2fecd](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=h1) Report
> Merging [#1158](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/e0caab0cf052c86e456bc4b4fdac5788433ed935?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1158   +/-   ##
======================================
  Coverage    80.7%   80.7%           
======================================
  Files          46      46           
  Lines        7411    7411           
======================================
  Hits         5981    5981           
  Misses       1430    1430
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=footer). Last update [e0caab0...0a2fecd](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Addressing this up-stream with #1092