Can VLLM be used for inference acceleration?

by obtion - opened Jan 11

obtion

Jan 11

Can VLLM be used for inference acceleration?

Owner Jan 11

"architectures": [
"MixtralForCausalLM"
],
you need to check whether vllm support "MixtralForCausalLM"

Jan 11

yeah VLLM supports that

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment