TheBloke

/

MPT-7B-GGML

like 21

Hi, thanks for uploading the models.

I created Python bindings for the GGML models https://github.com/marella/ctransformers which currently supports MPT, LLaMA and many other models (see Supported Models).

It provides a unified interface for all models, supports LangChain and can be used with Hugging Face Hub models:

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained('TheBloke/MPT-7B-GGML', model_type='mpt', model_file='mpt-7b.ggmlv3.q4_0.bin')

print(llm('AI is going to'))

If you add config.json file to your model repo with { "model_type": "mpt" } then model_type can be omitted:

llm = AutoModelForCausalLM.from_pretrained('TheBloke/MPT-7B-GGML', model_file='mpt-7b.ggmlv3.q4_0.bin')

If there is only one model file in repo then model_file can be omitted:

llm = AutoModelForCausalLM.from_pretrained('TheBloke/MPT-7B-GGML')

Please see marella/gpt-2-ggml for reference.