Return tokens from tokenizer.__call__()
Author: jarednielsen
Pull request: yes

Currently, pre-tokenizing (not encode tokens to ids, just generating the tokens) a batch of strings requires a manual for-loop. I'm adding a method to call the underlying Rust batch_encode implementation, which runs about 2x faster than a Python for-loop. I was expecting an even greater speedup, so if there's any way this could be made more efficient I would love to hear it.

Before:
```python
batch = ["Sentence 1", "Sentence 2"]
tokenized_batch = [tokenizer.tokenize(ex) for ex in batch]
# [["Sen", "##tence", "1"], ["Sen", "##tence", "2"]
```

Now:
```python
tokenized_batch = tokenizer.tokenize_batch(batch)
# [["Sen", "##tence", "1"], ["Sen", "##tence", "2"]
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=h1) Report
> Merging [#5728](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c3c61ea01733403210a1d159114e8c3d042dabb7&el=desc) will **increase** coverage by `1.18%`.
> The diff coverage is `50.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5728/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5728      +/-   ##
==========================================
+ Coverage   77.22%   78.41%   +1.18%     
==========================================
  Files         146      146              
  Lines       26012    26014       +2     
==========================================
+ Hits        20088    20399     +311     
+ Misses       5924     5615     -309     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/5728/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `93.60% <ø> (ø)` | |
| [src/transformers/tokenization\_utils\_fast.py](https://codecov.io/gh/huggingface/transformers/pull/5728/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfZmFzdC5weQ==) | `93.57% <50.00%> (-0.64%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5728/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `43.98% <0.00%> (-49.38%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5728/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `84.96% <0.00%> (-1.26%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5728/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.36% <0.00%> (+0.40%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5728/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.77% <0.00%> (+73.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=footer). Last update [c3c61ea...d91c382](https://codecov.io/gh/huggingface/transformers/pull/5728?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi Jared,

We are currently trying to reduce the number of user-facing methods in the tokenizer so I would be in favor of extending `tokenize` to accept batches as well. This can be done by having `tokenize` call `__call__` which accept both single examples and batches and extracting the tokens from the results.

It's already what happening in `encode` if you dive in the code (`encode` is calling the full tokenization pipeline in `encode_plus` and filtering the output to keep only the tokens).

Thanks for explaining your thinking! What about adding a `return_tokens` argument to `tokenizer.__call__`? `encoding.tokens` is the only attribute in an `Encoding` that can't be accessed directly from that method.

Oh yes, actually you can already do `.tokens(index_in_the_batch)` on the output of `__call__` but I see the docstring was missed and it is thus not in the doc currently, we will add it (and the similar method `.words(index_in_the_batch)`).

It's here in the `BatchEncoding` class (which is the output of the encoding methods): https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L221

So you can do:
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased', use_fast=True)
batch = ['hello how are you?', 'good morning everyone']
encodings = tokenizer(batch)
>>> encodings.tokens(1)
['[CLS]', 'good', 'morning', 'everyone', '[SEP]']
>>> encodings.tokens(0)
['[CLS]', 'hello', 'how', 'are', 'you', '?', '[SEP]']
```

Is it what you were looking for?

It currently only work for "fast" tokenizers (i.e. most of the tokenizers except the sentencepiece ones but they should be added not too far in the future)

Somewhat. I still have to do `[encodings.tokens(i) for i in range(len(encodings))]`, but that's fine. I also got direct access to the underlying Rust tokenizer via `tokenizer._tokenizer.encode_batch(batch["sentences"])`, so that will do. Thanks!