Add standardized get_vocab method to tokenizers
Author: joeddav
Pull request: yes

This PR adds a `get_vocab` method to the `PretrainedTokenizers` to standardize extracting vocabularies from tokenizers.

Comments:
- I didn't do anything with fast tokenizers. cc'ing @mfuntowicz for his thoughts there.
- I opted to keep it a method rather than a `@property` in order to encourage users to primarily use existing methods like `convert_tokens_to_ids` for general encoding/decoding purposes and use `get_vocab` only when they need the entire vocabulary.
- For tokenizers which rely on `sentencepiece`, I was unable to figure out a better way to get the vocabs than to loop through it. If someone knows a better way, please let me know.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=h1) Report
> Merging [#2939](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ea8eba35e2984882c3cd522ff669eb8060941a94?src=pr&el=desc) will **decrease** coverage by `1.02%`.
> The diff coverage is `89.65%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2939/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2939      +/-   ##
==========================================
- Coverage   75.35%   74.32%   -1.03%     
==========================================
  Files          94       94              
  Lines       15445    15474      +29     
==========================================
- Hits        11638    11501     -137     
- Misses       3807     3973     +166
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.71% <100%> (+1.77%)` | :arrow_up: |
| [src/transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtLnB5) | `83.39% <100%> (+0.13%)` | :arrow_up: |
| [src/transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fY3RybC5weQ==) | `96.19% <100%> (+0.07%)` | :arrow_up: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `97.02% <100%> (+0.02%)` | :arrow_up: |
| [src/transformers/tokenization\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYWxiZXJ0LnB5) | `89.52% <100%> (+0.41%)` | :arrow_up: |
| [src/transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fb3BlbmFpLnB5) | `82.5% <100%> (+0.22%)` | :arrow_up: |
| [src/transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZ3B0Mi5weQ==) | `96.87% <100%> (+0.04%)` | :arrow_up: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.61% <100%> (+0.16%)` | :arrow_up: |
| [src/transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxuZXQucHk=) | `90.17% <100%> (+0.36%)` | :arrow_up: |
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `38.22% <100%> (+0.31%)` | :arrow_up: |
| ... and [8 more](https://codecov.io/gh/huggingface/transformers/pull/2939/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=footer). Last update [ea8eba3...197d74f](https://codecov.io/gh/huggingface/transformers/pull/2939?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@sshleifer thanks for the review! The trouble is that different tokenizers store their vocabs pretty differently (thus this PR) – only BERT-inhereted tokenizers currently have a `self.vocab`, for example. I'd argue it's better to make it explicit that subclasses need to implement it rather than risk a silent error (i.e. if a subclass defines a `self.vocab` property differently than BERT's tokenizer does).