TheBloke

/

Llama-2-70B-Chat-GPTQ

like 257

@msteele same problem, figure it out? i can get the quantized 13b running, but not 70...

Hi, see https://github.com/huggingface/text-generation-inference/pull/648

Fantastic, thanks!

@TheBloke @fxmarty @msteele Finally got it to run, ended up having to use a older text-gen-inference server dockerfile combined with fxmarty's changes and the updated transformers version in the server requirements.txt.

Something is wrong with the current dockerfile and cargo chef keeps running me out of memory (>64gb) when trying to docker build

Great, thanks for letting us know. Hopefully that PR will be merged soon and then it will work with the TGI provided Dockerfile