GitHub repository: microsoft/sarathi-serve
Primary language: Python
Description: A low-latency & high-throughput serving engine for LLMs