HuggingFaceM4

/

idefics2-8b-AWQ

like 22

Hi all,

using the sample code from the non-AWQ version of the model, I get OOME:

from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image

DEVICE = "cuda"
MODEL = "HuggingFaceM4/idefics2-8b-AWQ"

# Note that passing the image urls (instead of the actual pil images) to the processor is also possible
image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")
processor = AutoProcessor.from_pretrained(MODEL)
model = AutoModelForVision2Seq.from_pretrained(MODEL,
    torch_dtype="auto",
    device_map="auto")

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.52 GiB. GPU 0 has a total capacity of 15.71 GiB of which 4.31 GiB is free. Process 9715 has 15.40 MiB memory in use. Including non-PyTorch memory, this process has 11.32 GiB memory in use. Of the allocated memory 10.70 GiB is allocated by PyTorch, and 474.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Memory usage is around 6G after loading model, then it crashes when i am calling model.generate().

Is this expected? How much RAM should I have to be able to use this model?