Implementation of Speculative Cascades
Author: radia78
Pull request: no
Labels: name: Feature request

### Feature request

I would like to modify the current  _speculative_sampling method to accommodate an implementation of speculative cascades. It would allow for the option to use regular speculative decoding or a combination of tokens from the assistant model and target model with the user's own specified rule for combining both tokens. 

The implementation is based on [this paper](https://arxiv.org/abs/2405.19261) and currently there are no official implementations, but a rough implementation would look like the [this](https://github.com/radia78/speccasc/blob/main/evaluation/speculative_cascades.py)

### Motivation

I was working on a class project that was experimenting with different deferral rules for speculative execution. Currently, the generate method only allows for normal speculative decoding, so it will be helpful for the community if we can tweak the _speculative_sampling function so that users can feed their own rule to experiment further.

### Your contribution

I can work on the full implementation based on the implementation I had for my project and will work from that point.

Comments:
Hi @radia78, this sounds good.

One architectural constraint **How do you plan to expose the "rule" in the API?**

The core team is usually cautious about adding arbitrary "callable" arguments (like `def my_rule(...)`) to `generate()` because it complicates serialization (Export/TorchScript) and adds to the already large signature.

Instead of passing a raw function, would it make sense to implement this as a `CandidateGenerator` subclass or a `LogitsProcessor`-style class? If you can share a small pseudocode snippet of how a user would call this, we can help ensure the design fits the library's philosophy before you invest time in the full implementation.

> Hi [@radia78](https://github.com/radia78), this sounds good.
> 
> One architectural constraint **How do you plan to expose the "rule" in the API?**
> 
> The core team is usually cautious about adding arbitrary "callable" arguments (like `def my_rule(...)`) to `generate()` because it complicates serialization (Export/TorchScript) and adds to the already large signature.
> 
> Instead of passing a raw function, would it make sense to implement this as a `CandidateGenerator` subclass or a `LogitsProcessor`-style class? If you can share a small pseudocode snippet of how a user would call this, we can help ensure the design fits the library's philosophy before you invest time in the full implementation.

I guess if users can't some arbitrary function, it makes sense to do a LogitsProcessor subclass.

> it will be helpful for the community if we can tweak the _speculative_sampling function so that users can feed their own rule to experiment further

hey @radia78 , we already have a way to do that. [Custom generate](https://huggingface.co/docs/transformers/en/generation_strategies#custom-generation-methods) gives community freedom to experiment with various generation strategies and then share it on the hub. I think you can share your own implem of cascade decoding as custom generate first, it can easily be shared with community to gather feedback and keep experimenting. If the decoding method gets traction and we can add it in the core library :)

Hi all thank you for your feedback and suggestions. I've managed to implement it on https://huggingface.co/radia/speculative-cascades. I made this suggestion it would be a small tweak to the speculative-sampling function with the option to do the default speculative sampling. Regardless, I hope I can contribute further in the future.

Nice, feel free to communicate and share with the community. I added a small PR with `custom_generate` tag to increase discoverability

Ig we can close this issue now :)