Thanks for the ggml model - Docker integration

#1
by flyingkiwiguy - opened

I've integrated it into a Docker build for "Open Llama-in-a-box" REST API server. Please keep the open-llama-3b-q5_1.bin file available for download. Full details here:

https://github.com/abetlen/llama-cpp-python/pull/310

I will probably update the same repo when newer checkpoints are released, would that be alright?

The conversion itself is not anything complex. I will add the Makefile here someday.

I could maybe leave a branch for this specific one.

Sorry, didn't see your message.

@abetlen accepted my PR, so you can find the latest stable "open llama in-a-box" here:

https://github.com/abetlen/llama-cpp-python/tree/main/docker/open_llama

I just smoke tested your updated 3B repo and it looks to work without any changes with the above code. Once you have the 7B model converted, it a single line change in build.sh to d/l and package that, instead.

Sign up or log in to comment