TheBloke

/

Llama-2-7B-Chat-GGML

like 805

from ctransformers import AutoModelForCausalLM

MODEL_NAME = "TheBloke/Llama-2-7B-chat-GGML"
config = {
'max_new_tokens': 490,

'temperature': 0.7,
'stream': True

}
llm = AutoModelForCausalLM.from_pretrained(MODEL_NAME, **config)

Trying to deploy above codes in gradio but getting JSON input error.

Can someone guide me for this how to overcome with this error.