OpenLemur

/

lemur-70b-v1

like 45

@glad4enkonm Thanks for your interest! I think the minimum hardware should be 1 x A100 (80G) card with the option load_in_8_bit=True. Otherwise, you may try the framework of TGI and etc.

@SivilTaram Thanks a lot for your quick reply!

There are 2 docker deploy scripts in the Lemur github repo.
One of them is using vllm with 4 GPUs (gpus='"device=0,1,2,3"')
what are hardware requirements for that GPUs? Are requirements the same to run
tgi and vllm scripts?

@glad4enkonm We deploy it using 4 x 80 GB A100, but I think it should also work on 8 x 40 GB A100 cards. Yes the requirements are the same to run tgi and vllm scripts.

Thanks a lot!