[Continuous batching] Tracker for issues and feature requests
Author: remi-or
Pull request: no
Labels: name: Performance

name: Inference

# Continuous batching issue and feature tracker

## Introduction

### Core idea
Continuous batching is the throughput-focused generation API for the `transformers` library, available through the `generate_batch` method.
It aims to provide a highly optimized generation API to a wide range of `transformers` models across various hardware. Since this is a non-trivial goal to achieve, we are doing this in an incremental manner. This is because we would rather roll out support slowly, ensuring a decent level of performance, rather than promising support for models that are not correctly optimized.

### Current status
Continuous batching currently supports text-only models deployed on a single device or a single node through TP.

## Motivation
This document's goal is to centralize:
- current issues with continuous batching, related to either performance or accuracy. The latter is considered the worse of the two
- features that are requested by the community
- features that are already on the roadmap
- improvements that could be made to the existing API, for performance or user-friendliness reasons
and track related issues and/or PRs.

### Contributing
This document is also a good starting point for anyone wanting to contribute to continuous batching. Please note that some bullet points are marked as **MR**, which stands for **maintainer-reserved**: please do not try to submit a PR for those, as it will probably be rejected. The rest is fair game! Some bullet points are marked as **Tricky**: these may take more time to properly fix / ship than others, be warned!

And of course, please read the [transformers guidelines](https://github.com/huggingface/transformers?tab=contributing-ov-file#readme) before contributing.

### A note on AI
I encourage all contributors to read the part on agent-written PRs in the [transformers guidelines](https://github.com/huggingface/transformers?tab=contributing-ov-file#readme).
Since reviewing a PR takes time, any PR that was fully written by an agent will be closed for that reason alone.
For instance, PRs with an AI-written summary have a higher chance of not passing that filter. So **please write your PR summaries yourself**! It shows that you clearly understand the issue and the path that was taken to solve it. The quality of your English does **not** matter: broken English is better than LLM English.
**The same applies to comments.**

## Issues, features, experiments

### Issues
- **(High prio)** Rewrite `get_result` to be more efficient
- **(Low hanging fruit)** Change the temporary token id to 0 so we can simplify carry-over
- Consolidate the schedulers into two near-identical classes
- Switch back to Python `hash` for TP processes if that does not break TP
- Re-enable FA2 for block table path
- Fix CPU block leak when H2D copy crashes during request reloading
- Fix CPU request reloading when async batching is turned on
- Address low-priority # TODOs

### Features
- **(MR)** #46376 
- **(MR)** Add speculative decoding support in continuous batching (abstract mechanism first)
- **(MR)** Support exotic caches like DeepseekV4 (mostly about generalizing the current cache)
- **(Tricky)** Support models with a multi-device `device_map` object
- **(Tricky)** Switch the synchronous inputs_outputs to a HostDevicePair-based system
- Add CB logits processors that are not per-request processors
- Add a temperature + top-p + top-k fused logits processor
- Estimate KV head count by running the model instead of relying on TP keys
- Support sliding window in the decode fast path

### Experiments
- **(High prio)** Is there a way to speed up compile for models?
- Is there a scheduling method that increases the throughput over benchmarks like GSM8K?
- Are there workloads where `generate` is still faster than `generate_batch`?

Comments:
Hi @remi-or, I'd like to take a look at the high priority 'Rewrite get_result' task. Could you assign this to me or let me know if anyone is currently working on it? I can draft an optimized approach.

Hey @shreyescodes ! You can just open a PR, no one is working on it right now as far as I know. Looking forward to your contribution!

Thanks @remi-or! I have just opened PR #46714 with the fix.

I spent some time debugging the shared output_queue behavior and implemented a dictionary of private sync_queues inside the OutputRouter to completely eliminate the heavy busy-waiting loop.

Looking forward to your review, and please let me know if you'd like me to make any adjustments to the approach!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.