Text Generation
Transformers
English
llama
text-generation-inference

How to use ctransfomers with GGML files?

#1
by rfinn - opened

Tom, I have not been able to get GGML models to work with ctransformers or llama-cpp-python. They do work when I use LM Studio. I've tried several different ones and always get a nondescript error: "Failed to create LLM 'llama' from 'stablebeluga-7b.ggmlv3.q5_1.bin'." The .bin file is in the same directory and it shows up on a listdir(). What step am I missing?

code:
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained('stablebeluga-7b.ggmlv3.q5_1.bin', model_type='llama')

Tom, I have not been able to get GGML models to work with ctransformers or llama-cpp-python. They do work when I use LM Studio. I've tried several different ones and always get a nondescript error: "Failed to create LLM 'llama' from 'stablebeluga-7b.ggmlv3.q5_1.bin'." The .bin file is in the same directory and it shows up on a listdir(). What step am I missing?

code:
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained('stablebeluga-7b.ggmlv3.q5_1.bin', model_type='llama')

Transformers load Transformers models, GGML models are loaded my llama.cpp and its variants.

Sign up or log in to comment