(Will make a PR) `BeamScorer` is super slow and takes 2x time compared with model itself, and can speed up by 1000%
Author: fzyzcjy
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.23
- Python version: 3.10.0
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run `.generate(num_beams=3)` thus use beam search.

### Expected behavior

Should be fast, but it is super slow.

I am working on a fix and will soon make a PR (today?). Just want to open this issue first so that I can get some early feedbacks (e.g. do you welcome PRs?).

One cause I realize is that, the BeamScorer.process etc is working on torch tensors on gpu, *one by one*. I refactored it so it works on numpy arrays on cpu, and it is 3x faster.

---

If you are interested, here are some *very early* results:

before (scorer takes 7.37s)

![image](https://user-images.githubusercontent.com/5236035/208365686-8d00f36a-364a-4310-a4fc-a2f71a7015b7.png)

after (scorer 2.72s, still slow but better)

![image](https://user-images.githubusercontent.com/5236035/208366916-b3773eb1-1f4d-4d9d-be9d-ea2a286b7b1e.png)


Comments:
cc @gante 

Update: I made a vectorized version of BeamScorer.

## Performance results: 1000% faster

<details>

### MyBeamSearchScorer: 0.45s

![image](https://user-images.githubusercontent.com/5236035/208571326-1502c8a3-fc7d-429a-a3c5-3addf8aab556.png)

### BeamSearchScorer: 5.72-6.28s

![image](https://user-images.githubusercontent.com/5236035/208571462-fdd1ce15-7eff-4aee-ab20-26a028d707d5.png)

</details>

## Code

Quite messy currently, but the core is beam_search.py and the rest are glues and benchmarks

https://gist.github.com/fzyzcjy/fab4bf82c62f23b3432123c84f14a2c6

If you are interested please tell me and I will make a PR :)

Hey @fzyzcjy 👋 

We thrive on external contributions, so you're more than welcome to open a PR. In general, these would be the requirements:
1. Has a significant speedup on either CPU or GPU
2. All existing tests pass
3. The code maintains its readability, such that beam search is easy to understand

Looking forward to the PR :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

bump, will submit PR when having time (indeed all code are already completed - see link above, just no time to sit down and PR)

PR: https://github.com/huggingface/transformers/pull/21234

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.