Support for efficient Triton Server deployment (TensorRT-LLM / vLLM / ONNX) for Sooktam-2
#10
by ayush220 - opened
Feature Request / Inquiry
I am looking to deploy the Sooktam-2 Indic TTS model in a production environment using NVIDIA Triton Inference Server, and I want to know if there is an officially supported, efficient way to do so.
Current Limitation
Right now, deploying Sooktam-2 on Triton requires using the Triton Python Backend (trust_remote_code=True) paired with Hugging Face pipelines. While this works as a quick Proof of Concept, it is highly inefficient for production due to:
- Python GIL Bottlenecks: Stalls GPU utilization when scaling concurrent requests.
- Lack of Continuous Batching: Standard PyTorch pipelines handle varying audio/text lengths sequentially rather than dynamically stacking them.
- ODE/SDE Solver Overhead: The underlying F5-TTS architecture relies on iterative flow-matching steps that introduce massive CPU-GPU data transfer overhead inside a standard Python wrapper.
Proposed Solution / Request
I would love to see guidance, scripts, or export pipelines to serve Sooktam-2 via a highly optimized C++ engine backend. Specifically:
- TensorRT / TensorRT-LLM Export: An official script to compile the custom F5-TTS transformer layers into an optimized
.engineformat. - vLLM Backend Compatibility: Configuration templates to utilize PagedAttention and continuous batching with Triton's vLLM backend.
- ONNX Graph Definitions: Standardized ONNX weights that natively export the custom remote code layers, bypassing the need for
trust_remote_code=True.
Are there any existing optimization paths, scripts, or benchmarks for low-latency serving of this model? Any help or direction would be greatly appreciated!