VRAM

#53

by DataSoul - opened Apr 18

Apr 18

I totally agree this is a great model, but I'm wondering, why it requires significantly more VRAM when running compared to other models with similar parameter sizes. It's to the point where I can't use longer contexts on my setup. ( I am using the Q4 version of gguf )

shivi

Cohere For AI org Jun 18

Hey @DataSoul

Thanks for the feedback. Could you tell me which models did you find consuming lesser VRAM in comparison ?

10100101j

Jul 31

this is because of no group query attention right?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment