microsoft/llmlingua: To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss. 
Category: llm
Language: Python
Topics: inference, performance
Homepage: https://llmlingua.com/