FillMaskPipeline very slow when provided with a large `targets`
Author: EtaoinWu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Linux-5.4.0-67-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1 (False)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik @Narsil 

## Information

The model I am using: `ethanyt/guwenbert-base`, with a `RoBERTa` model and a `BertTokenizerFast` tokenizer.

## To reproduce

Steps to reproduce the behavior:

1. Initialize a `fill-mask` pipeline with the model and the tokenizer mentioned above
2. Call it with any sentence and a large `targets` (with a length of ~10k single words)

## Problem

The call would be much slower than a similar call without a `targets` argument. A call without a `targets` argument costs ~0.1s, while a call with a `targets` argument costs ~0.3s.

The following code is present in `src/transformers/pipelines/fill_mask.py`:

```python
class FillMaskPipeline(Pipeline):
    # ...
    def __call__(self, *args, targets=None, top_k: Optional[int] = None, **kwargs):
        # ...
        if targets is not None:
            # ...
            targets_proc = []
            for target in targets:
                target_enc = self.tokenizer.tokenize(target)
                # ...
                targets_proc.append(target_enc[0])
```

This function iterates through targets, rather than sending it directly to `tokenize`, which does not utilize the batch processing optimization of `TokenizerFast`s, hence the slow speed.

Comments:
Do you have an example to reproduce the issue ? Benchmarking is sometimes suprising and hardware dependant.

I can imagine that this is indeed a slowdown as Python - Rust communication is not free.
However the omitted part of your comment is error detection, which is important and we need to keep it.

```python
            if len(targets) == 0 or len(targets[0]) == 0:
                raise ValueError("At least one target must be provided when passed.")
            if isinstance(targets, str):
                targets = [targets]

            targets_proc = []
            for target in targets:
                target_enc = self.tokenizer.tokenize(target)
                if len(target_enc) > 1 or target_enc[0] == self.tokenizer.unk_token:
                    logger.warning(
                        f"The specified target token `{target}` does not exist in the model vocabulary. "
                        f"Replacing with `{target_enc[0]}`."
                    )
                targets_proc.append(target_enc[0])
            target_inds = np.array(self.tokenizer.convert_tokens_to_ids(targets_proc))
```

I think we can get away with encoding every target at once, then iterating through the whole array to do the error detection.

However, as this is a performance problem, I think we realistically need to test that improving performance on 10K targets, does not reduce performance significantly on 10targets (which is a more common usage).

Caveat: when a target is going to be very long (like 20 tokens) with 10k targets, the resulting array will be 20 x 10k for ids, that can pile up quite fast memory usage. In that context, it could be much slower to pass everything at once. We need to benchmark that too.

We don't have benchmarking tests right now, but if a PR goes in I think a test should demonstrate the usage and have a clear comment at leat about this specific decision.

An example code can be found [in this colab example](https://colab.research.google.com/gist/EtaoinWu/0cf5b37882bd18bcc554d3da717a3974/fillmaskpipeline-test.ipynb). On the default Google machine that I wrote this notebook on, the version with a `targets` argument slows down significantly (100ish ms to 600ish ms).

> Caveat: when a target is going to be very long (like 20 tokens) with 10k targets, the resulting array will be 20 x 10k for ids, that can pile up quite fast memory usage. In that context, it could be much slower to pass everything at once. We need to benchmark that too.

The current behavior of `FillMaskPipeline` is that when a multi-token string is passed, only the first token is used. I doubt anyone would actually need this, because if someone want to choose a token from a subset of the vocabulary to fill into a mask, they usually know the subset exactly. Deliberately passing multi-token strings into `FillMaskPipeline` (and expecting it to tokenize them and drop all-but-first tokens) does not make much sense.

### Another discovery

When coding my example, I just discovered the bottleneck of the performance problem. When provided with a `targets` argument, `FillMaskPipeline` ignores its `top_k` parameter, which means that it has to output a whole list proportional to `len(targets)`, and that's the bottleneck (at least in my test). The code example above actually respects `top_k` parameter when a `targets` is present, hence much faster when constructing the return value. After the optimization, the code costs 200ish milliseconds.

> An example code can be found [in this colab example](https://colab.research.google.com/gist/EtaoinWu/0cf5b37882bd18bcc554d3da717a3974/fillmaskpipeline-test.ipynb). On the default Google machine that I wrote this notebook on, the version with a `targets` argument slows down significantly (100ish ms to 600ish ms).

Thanks, this will help ! 
> 
> > Caveat: when a target is going to be very long (like 20 tokens) with 10k targets, the resulting array will be 20 x 10k for ids, that can pile up quite fast memory usage. In that context, it could be much slower to pass everything at once. We need to benchmark that too.
> 
> The current behavior of `FillMaskPipeline` is that when a multi-token string is passed, only the first token is used. I doubt anyone would actually need this, because if someone want to choose a token from a subset of the vocabulary to fill into a mask, they usually know the subset exactly. Deliberately passing multi-token strings into `FillMaskPipeline` (and expecting it to tokenize them and drop all-but-first tokens) does not make much sense.

As a maintainer of a live product, I can tell you not everyone is aware of what happens behind a pipeline (and it is exactly why they exist, so we can abstract away all nitty gritty details of transformers). So it will happen that some users will try out those examples and be surprised at slowness. 
It's something that `pipelines` should try to address if possible. 

> ### Another discovery
> 
> When coding my example, I just discovered the bottleneck of the performance problem. When provided with a `targets` argument, `FillMaskPipeline` ignores its `top_k` parameter, which means that it has to output a whole list proportional to `len(targets)`, and that's the bottleneck (at least in my test). The code example above actually respects `top_k` parameter when a `targets` is present, hence much faster when constructing the return value. After the optimization, the code costs 200ish milliseconds.

Ok, I think I remember the `targets` being added and the decision was that if `top_k` > `len(targets)` we were not obliged of honoring `top_k` because it wouldn't make any sense. `top_k` < `len(targets)` should be honored though.



I was able to reproduce and optimize away most of the performance, now any example should run at roughly the same speed.

Slowdown will happen when you miss the vocabulary, but the warnings should help users figure it out.

Thanks a lot. As a background, I found the issue when reproducing the following paper:

> Deng, Liming, et al. "An Iterative Polishing Framework Based on Quality Aware Masked Language Model for Chinese Poetry Generation." _Proceedings of the AAAI Conference on Artificial Intelligence_. Vol. 34. No. 05. 2020.

which involves calling `FillMaskPipeline` iteratively 10 times at most for each API call, which depending on the input, may or may not have the `targets` parameter. The time difference in the two types of API calls made me find this issue.