[WIP][Speculative Decoding] Enable batch size > 1
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR enables speculative decoding when batch size is > 1.

This PR has been tested to work correctly for Whisper. You can try it as follows:

```py
from transformers import pipeline, AutoModelForCausalLM, AutoModelForSpeechSeq2Seq, AutoProcessor
import torch
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

assistant_model_id = "distil-whisper/distil-large-v2"

assistant_model = AutoModelForCausalLM.from_pretrained(
    assistant_model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
assistant_model.to(device)

model_id = "openai/whisper-large-v2"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    generate_kwargs={"assistant_model": assistant_model},
    torch_dtype=torch_dtype,
    chunk_length_s=15,
    batch_size=4,
    device=device,
)

dataset = load_dataset("distil-whisper/librispeech_long", "default", split="validation")
sample = dataset[0]["audio"]

result = pipe(sample)
print(result["text"])
```

## TODOS
- [x] Test that it works correctly for Whisper
- [ ] Test for Llama
- [ ] Write nice tests
- [ ] Clean code
- [ ] Write docs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26875). All of your documentation changes will be reflected on that endpoint.

Hi, good work! I am curious about how to deal with the wrong kv cache token rolling back for draft model, or you just mask them?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@patrickvonplaten Is there any plan to merge this PR sometimes or is it canceled ?

@Hubert-Bonisseur The PR has closed due to inactivity. I'm not sure of the current status of batch > 1 for speculative decoding, but I'm sure @gante will know :) 

It is still not supported : 
`ValueError: assisted generate is only supported for batch_size = 1`

But I see that Patrick has left HF since writing this PR, which probably explains why it stalled.
If there is still interest in this PR, I'm willing to help with finish the work of Patrick. It seems like it is very close to be ready and I'm vert interested in this feature.

Maybe your whisper expert @sanchit-gandhi also knows the status of this feature ?

Hey @Hubert-Bonisseur - handing this one over to @kamilakesbi who will take over the integration!

Hey @kamilakesbi, just checking in to see if there are any updates on this. Thanks!