will there be a gptq int4 version?

by tutu329 - opened Apr 23

Apr 23

vllm only supports gptq or awq format.
gptq is much faster than awq.

thanks a lot if there will be a llama-3-70B-instruct-uncensored-gptq-int4.

Dogge

Owner Apr 23

I'll think about it, thank you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment