Package: aphrodite-engine
Breathing Life into LanguageAphrodite is the official backend engine for PygmalionAI. It is designed to serve as the inference endpoint for the PygmalionAI website, and to allow serving thePygmalionmodels to a large number of users with blazing fast speeds (thanks to FasterTransformer and vLLM).Aphrodite builds upon and integrates the exceptional work fromvarious projects.The compute necessary for Aphrodite's development is provided byArc Compute.FeaturesContinuous BatchingEfficient K/V management withPagedAttentionOptimized CUDA kernels for improved inferenceQuantization support via GPTQ, GGUF, AWQ, QuIP#, and SqueezeLLM.Distributed inferenceVariety of sampling methods (Mirostat,Locally Typical Sampling, Tail-Free Sampling, etc)8-bit KV Cache for higher context lengths and throughput.Quickstartpipinstallaphrodite-engine

python-maphrodite.endpoints.openai.api_server--modelPygmalionAI/pygmalion-2-7b[!CAUTION]
If the installation reports CUDA kernel errors, please runpip install aphrodite-engine=0.4.5instead.This will create aOpenAI-compatible API server that can be accessed at port 2242 of the localhost. You can plug in the API into a UI that supports Kobold, such asSillyTavern.Dockersudo mkdir ~/workspace
sudo docker run --gpus '"all"' --shm-size 10g -p 2242:2242 -p 7650:7650 -p 8080:8080 --rm -it -v '/home/workspace' alignmentlabai/aphrodite-engine:latestPerformanceSpeeds vary with different GPUs, model sizes, quantization schemes, batch sizes, etc. Here are some baseline benchmarks conducted by requesting as many completions as possible from theAPI server. Keep in mind that these are the theoritical peak throughput with parallel decoding, with as high a batch size as possible.Per-request generation speed is a fraction of this, at 30-40 t/s.High Batch Size Performance[!NOTE]The numbers below are the theoritical peak achieved byonlyrequesting output tokens at very high batch sizes. At lower batch sizes with much larger prompts, the results will be vastly different.
Throughput refers to output tokens per second.ModelQuantizationbitsGPUThroughput (T/s)Mistral 7BNone16RTX 40905489.3AWQ4RTX 40904078.8GPTQ4RTX 40907850.48RTX 40907658.0GGUFQ8RTX 40905141.2Q6KMRTX 40905791.7Q5KMRTX 40905786.2Q4KMRTX 40905815.8SqueezeLLM4RTX 4090549.5Llama-2 7BNone16RTX 40902576.2AWQ4RTX 40903551.3GPTQ4RTX 40902919.1GGUFQ4KMRTX 40902726.6Q5KMRTX 40902763.4Q6KMRTX 40902694.7Q8RTX 40902647.0SqueezeLLM4RTX 4090580.3Batch Size 1These are the speeds a user would normally get if they request a single output with a sizable prompt and output length. Essentially, normal chatting experience.The following results were gathered by sending a request with 2000 prompt tokens and requesting 1024 tokens withignore_eos=True.ModelQuantizationbitsGPUThroughput (T/s)Mistral 7BNone16RTX 409054.0AWQ4RTX 4090128.2GPTQ8RTX 409092.84RTX 4090146.8GGUFQ8RTX 409091.0Q6KMRTX 4090105.4Q5KMRTX 4090117.8Q4KMRTX 4090128.9Llama-2 7BNone16RTX 409055.2GPTQ8RTX 409090.24RTX 4090128.0AWQ4RTX 4090116.3GGUFQ8RTX 409088.1Q6KMRTX 409099.4Q5KMRTX 4090109.9Q4KMRTX 4090118.9RequirementsOperating System: Linux (or WSL for Windows)Python: at least 3.8Build Requirements:CUDA >=12For supported GPUs, seehere.InstallationFrom PyPiBuild from sourceUsageFor usage, please refer to thewiki pagefor detailed instructions. Aphrodite provides many different options for LLM inference, so please read through the list of optionshere.NotesBy design, Aphrodite takes up 90% of your GPU's VRAM. If you're not serving an LLM at scale, you may want to limit the amount of memory it takes up. You can do this in the API example by launching the server with the--gpu-memory-utilization 0.6(0.6 means 60%).You can view the full list of commands by runningpython -m aphrodite.endpoints.openai.api_server --help.Context Length extension via the RoPE method is supported for most models. Use the command-line flag--max-model-lento specify a desired context length and the engine will adjust the RoPE scaling accordingly.Please refer to theFAQ & Issuesif you run into problems. If you don't find an answer there, please make anissue.AcknowledgementsAphrodite Engine would have not been possible without the phenomenal work of other open-source projects. Credits go to:vLLM(CacheFlow)FasterTransformerxFormersAutoAWQAutoGPTQSqueezeLLMExLlamaExllamav2KoboldAIText Generation WebUIMegatron-LMFastChatRaySkyPilotOpenAI Python LibraryContributingEveryone is welcome to contribute. You can support the project by opening Pull Requests for new features, fixes, or general UX improvements.