Tokenizer.from_pretrained: fetch all possible files remotely
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=h1) Report
> Merging [#2535](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/eb59e9f70513b538d2174d4ea1efea7ba8554b58?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2535/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2535      +/-   ##
==========================================
- Coverage   74.67%   74.61%   -0.07%     
==========================================
  Files          87       87              
  Lines       14798    14798              
==========================================
- Hits        11050    11041       -9     
- Misses       3748     3757       +9
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2535/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.39% <100%> (+0.03%)` | :arrow_up: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2535/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `96.2% <100%> (ø)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2535/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2535/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `85.64% <65%> (-1.92%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2535/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `71.69% <80%> (+1.64%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=footer). Last update [eb59e9f...a08b24d](https://codecov.io/gh/huggingface/transformers/pull/2535?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


They’re not *super* slow (meaning each download is a few hundred kB at most)

But maybe we should have a slow and a super_slow decorator at some point

On Thu, Jan 16, 2020 at 3:15 AM Thomas Wolf <notifications@github.com>
wrote:

> *@thomwolf* commented on this pull request.
> ------------------------------
>
> In tests/test_tokenization_auto.py
> <https://github.com/huggingface/transformers/pull/2535#discussion_r367280674>
> :
>
> > @@ -56,3 +56,17 @@ def test_tokenizer_from_model_type(self):
>          tokenizer = AutoTokenizer.from_pretrained(DUMMY_UNKWOWN_IDENTIFIER)
>          self.assertIsInstance(tokenizer, RobertaTokenizer)
>          self.assertEqual(len(tokenizer), 20)
> +
> +    def test_tokenizer_identifier_with_correct_config(self):
>
> Should we decorate these tests (and the previous ones) with @slow?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/2535?email_source=notifications&email_token=AACPXMJDS2SCYQ66THBT7LTQ6AJSFA5CNFSM4KHJ6ZEKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCR6PMZY#pullrequestreview-343733863>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACPXMIDZWRFCQ4EGPSQTNLQ6AJSFANCNFSM4KHJ6ZEA>
> .
>


> They’re not *super* slow (meaning each download is a few hundred kB at most) But maybe we should have a slow and a super_slow decorator at some point
> […](#)
> On Thu, Jan 16, 2020 at 3:15 AM Thomas Wolf ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tests/test_tokenization_auto.py <[#2535 (comment)](https://github.com/huggingface/transformers/pull/2535#discussion_r367280674)> : > @@ -56,3 +56,17 @@ def test_tokenizer_from_model_type(self): tokenizer = AutoTokenizer.from_pretrained(DUMMY_UNKWOWN_IDENTIFIER) self.assertIsInstance(tokenizer, RobertaTokenizer) self.assertEqual(len(tokenizer), 20) + + def test_tokenizer_identifier_with_correct_config(self): Should we decorate these tests (and the previous ones) with @slow? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2535?email_source=notifications&email_token=AACPXMJDS2SCYQ66THBT7LTQ6AJSFA5CNFSM4KHJ6ZEKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCR6PMZY#pullrequestreview-343733863>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACPXMIDZWRFCQ4EGPSQTNLQ6AJSFANCNFSM4KHJ6ZEA> .

Oh yes you're right, tokenizer vocabs are pretty small indeed. Ok for no `@slow`!