How to use gguf model without ComfyUI?
As the title says..
The original model is too big for me. But I only found how to use the existing quantitative model in comfyui.
Also i want to know why 32GB RAM still not enough to inference.
I've tried quantize the tokenizer2 as "https://huggingface.co/black-forest-labs/FLUX.1-dev/discussions/50"
But it still occur error:CUDA out of memory
packages/diffusers/models/attention_processor.py", line 1911, in call hidden_states = F.scaled_dot_product_attention( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 6.26 GiB. GPU 0 has a total capacity of 31.74 GiB of which 4.11 GiB is free. Process 207695 has 27.62 GiB memory in use. Of the allocated memory 26.09 GiB is allocated by PyTorch, and 440.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation
Were you able to use gguf model without comfyUI?