Package: locallm
LocallmAn api to query local language models using different backends. Supported backends:Llama.cpp Python: the local Python bindings for Llama.cppKobold.cpp: the Koboldcpp api serverOllama: the Ollama api serverQuickstartpipinstalllocallmLocalfromlocallmimportLocalLm,InferenceParams,LmParamslm=LocalLm(LmParams(models_dir="/home/me/my/models/dir"))lm.load_model("mistral-7b-instruct-v0.1.Q4_K_M.gguf",8192)template="

{prompt}
"lm.infer("list the planets in the solar system",InferenceParams(template=template,temperature=0.2,stream=True,max_tokens=512,),)KoboldcppfromlocallmimportKoboldcppLm,LmParams,InferenceParamslm=KoboldcppLm(LmParams(is_verbose=True))lm.load_model("",8192)# sets the context window size to 8196 tokenstemplate="

{prompt}
"lm.infer("list the planets in the solar system",InferenceParams(template=template,stream=True,max_tokens=512,),)OllamafromlocallmimportOllamaLm,LmParams,InferenceParamslm=Ollama(LmParams(is_verbose=True))lm.load_model("mistral-7b-instruct-v0.1.Q4_K_M.gguf",8192)template="

{prompt}
"lm.infer("list the planets in the solar system",InferenceParams(stream=True,template=template,temperature=0.5,),)ExamplesProviders:Llama.cpp PythonproviderKobold.cppproviderOllamaproviderOther:Cli: a Python terminal clientAutodoc: generate docstrings from codeApiLmProviderAn abstract base class to describe a language model provider. All the
providers implement this apiAttributesllmOptional[Llama]: the language model.models_dirstr: the directory where the models are stored.api_keystr: the API key for the language model.server_urlstr: the URL of the language model server.is_verbosebool: whether to print more information.threadsOptional[int]: the numbers of threads to use.gpu_layersOptional[int]: the numbers of layers to offload to the GPU.embeddingOptional[bool]: use embeddings or not.on_tokenOnTokenType: the function to be called when a token is generated. Default: outputs the token to the terminal.on_start_emitOnStartEmitType: the function to be called when the model starts emitting tokens.Examplelm=OllamaLm(LmParams(is_verbose=True))Methods:__init__Constructs all the necessary attributes for the LmProvider object.ParametersparamsLmParams: the parameters for the language model.Examplelm=KoboldcppLm(LmParams())load_modelLoads a language model.Parametersmodel_namestr: The name of the model to load.ctxint: The context window size for the model.gpu_layersOptional[int]: The number of layers to offload to the GPU for the model.Examplelm.load_model("my_model.gguf",2048,32)inferRun an inference query.Parameterspromptstr: the prompt to generate text from.paramsInferenceParams: the parameters for the inference query.ReturnsresultInferenceResult: the generated text and statsExample>>>lm.infer("

 List the planets in the solar system [/INST>")Theplanetsinthesolarsystemare:Mercury,Venus,Earth,Mars,Jupiter,Saturn,Uranus,andNeptune.TypesInferenceParamsParameters for inference.Argsstreambool, Optional: Whether to stream the output.templatestr, Optional: The template to use for the inference.threadsint, Optional: The number of threads to use for the inference.max_tokensint, Optional: The maximum number of tokens to generate.temperaturefloat, Optional: The temperature for the model.top_pfloat, Optional: The probability cutoff for the top k tokens.top_kint, Optional: The top k tokens to generate.min_pfloat, Optional: The minimum probability for a token to be considered.stopList[str], Optional: A list of words to stop the model from generating.frequency_penaltyfloat, Optional: The frequency penalty for the model.presence_penaltyfloat, Optional: The presence penalty for the model.repeat_penaltyfloat, Optional: The repeat penalty for the model.tfsfloat, Optional: The temperature for the model.grammarstr, Optional: A gbnf grammar to constraint the model's outputExampleInferenceParams(stream=True,template="

{prompt}[/INST>"){"stream":True,"template":"

{prompt}[/INST>"}LmParamsParameters for language model.Argsmodels_dirstr, Optional: The directory containing the language model.api_keystr, Optional: The API key for the language model.server_urlstr, Optional: The server URL for the language model.is_verbosebool, Optional: Whether to enable verbose output.on_tokenCallable[[str], None], Optional: A callback function to be called on each token generated. If not provided the default will output tokens to the command line as they arriveon_start_emitCallable[[Optional[Any]], None], Optional: A callback function to be called on the start of the emission.ExampleLmParams(models_dir="/home/me/models",api_key="abc123",)TestsTo configure the tests create atests/localconf.pycontaining the some local config info to
run the tests:# absolute path to your models dirMODELS_DIR="/home/me/my/models/dir"# the model to use in the testsMODEL="q5_1-gguf-mamba-gpt-3B_v4.gguf"# the context window size for the testsCTX=2048Be sure to have the corresponding backend up before running a test.