GPU requirements
What are the GPU instance requirements for using this model on RunPod?
Will this model work with HF Inference API? The widget on the model page doesn't work. It doesn't say that I don't have access just failed to fetch.
Also do your scripts work for HF ZeroGPU? If so, how many A100s are needed?
Howdy.
This can run in 16 bits on 4x A6000s (48 GB each) or 2X A100s (80GB each). You can also half that requirement by adding --quantize eetq to the command box on the template (see the model card for the one click template).
Ah yeah, the widget doesn't work because I haven't turned on inference, I've just done that now, but I'm not confident it will work because this is a big model requiring quit ea bit of compute.
Seems like ZeroGPU [only works with Gradio SDK)[https://huggingface.co/zero-gpu-explorers], so in principle yes this model could work, but it would need to be prepared and wrapped with a gradio ui.