How do I know what kind of machine do I need to run this model?
#2
by
adaboese
- opened
This is newbie question, but I am browsing top models listed on https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, and the main thing that I cannot answer is how do I know what kind of machine I need to run the model, how fast it will be and how expensive?
Hi,
If you load it with normal precision, it should take around 25 GB vram or ram if you are loading to cpu. However, you can load the model in 8 or 4 bits to reduce this amount. There are also quantized models available for this purpose.
Weyaxi
changed discussion status to
closed