[Performance improvement] "Bad tokens ids" optimization
Author: guillaume-be
Pull request: yes

Running some benchmarks I noticed that the generation pipeline was varying quite a bit in terms of execution time. Especially the banned token masking seems to be fairly expensive (I ran some experiments where up to 30% of the time for an entire generation process was spent in this step - which seems too high considering its expected simplicity).

This PR accelerates the entire generation pipeline for models using a `bad_words_ids` in their configuration by around 20% on a GPU-enabled node (this includes for example translation using the Marian models). 

The following changes contribute to the performance improvement:
- Single conversion from tensor to list. Previous approach was accessing the GPU buffer for every banned token and every batch element, causing this operation to be slower than the entire forward pass through the model
- Vectorized update of the banned tokens using a masked fill
- Skipping the EOS token for the banned tokens (avoiding a potential duplicate masking)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=h1) Report
> Merging [#6064](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8a8ae27617e3c4dafb34bcbbaadf4ceee28583bd&el=desc) will **decrease** coverage by `0.10%`.
> The diff coverage is `27.77%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6064/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6064      +/-   ##
==========================================
- Coverage   78.49%   78.38%   -0.11%     
==========================================
  Files         146      147       +1     
  Lines       26335    26384      +49     
==========================================
+ Hits        20671    20681      +10     
- Misses       5664     5703      +39     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/test\_generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6064/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Rlc3RfZ2VuZXJhdGlvbl91dGlscy5weQ==) | `0.00% <0.00%> (ø)` | |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6064/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `96.64% <93.75%> (-0.19%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=footer). Last update [8a8ae27...7d9767a](https://codecov.io/gh/huggingface/transformers/pull/6064?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@sshleifer Thank you very much for the review. I have added unit tests for the modified method that hopefully aligns with what you had in mind. I have re-run the Marian integration tests that run without issue. I somehow have issues running the BART integration tests (even on master) due to an `ImportError` and unable to see if these still run:
```python
from .test_configuration_common import ConfigTester
ImportError: attempted relative import with no known parent package
```

regarding point 2) could you please clarify? The `calc_banned_bad_words_ids` still exists (and is used) in the proposed PR. Would you recommend making a copy of it instead of changing its behaviour? Then the original `calc_banned_bad_words_ids`  would no longer be used anywhere

@JetRunner I have added a comment to clarify the mask tensor generation

I am currently running into issues with Tensorflow test failing - but I do not see how it relates to the proposed changes

Thank you!

I misread your code, sorry.
My point 2 should be that it feels like the new masking logic could be put into a helper method like
```python
def set_scores_to_inf_for_banned_tokens(self, scores, bad_words_ids) -> None:

```
just for the sake of namespace control.
You could also test that method without running `generate`.

Also, how significant is the speedup here?

@sshleifer This makes sense, just pushed a few more changes:
- Moved the masking to a utility function
- Updated the unit test to let it fail if it hits timeout. As this is configuration dependent, the limit was increased to 10 if the CI compute power available fluctuates. In general I am not sure if unit tests are the best way to perform performance regression tests
- I have created a gist to share the performance difference between the current and the proposed approach: https://gist.github.com/guillaume-be/e335b099005e9bf38448d0e2eb02f74f . On this simple example with a GPU on Colab, the proposed approach is twice as fast. This actually has a significant impact on the entire generation process, but I did not manage to create a good example on Colab (the resources fluctuate too much from notebook to notebook, and not aware of a way to change a library version within a same notebook). Running locally with a consumer-grade Turing GPU (2070), I observe a time reduction of around 20% for the end-to-end generation process. 

@sshleifer Thank you again for the thorough review! Tried to address the latest comments - I believe it cleans it up quite a bit thank you for the suggestions

This is ready to be merged @LysandreJik !