Intel
/

gemma-4-31B-it-int4-AutoRound

text-generation-inference

4-bit precision

Model card Files Files and versions

INT8 version for TP=2 / dual Ampere GPUs?

#6

by mancub - opened May 6

May 6

Would you please make an INT8 version for Ampere owners with dual GPUs?

Intel org May 25

sorry for the later reply. For INT8, you could leverage pure RTN mode in AutoRound which is very fast.

mancub changed discussion status to closed May 25

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment