Package: ctransformers-langdash
C TransformersFork of CTransformers to support Langdash functionsPython bindings for the Transformer models implemented in C/C++ usingGGMLlibrary.Also seeChatDocsSupported ModelsInstallationUsageHugging Face HubLangChainGPUDocumentationLicenseSupported ModelsModelsModel TypeGPT-2gpt2GPT-J, GPT4All-JgptjGPT-NeoX, StableLMgpt_neoxLLaMA, LLaMA 2llamaMPTmptDolly V2dolly-v2ReplitreplitStarCoder, StarChatstarcoderFalcon (Experimental)falconInstallationpipinstallctransformersUsageIt provides a unified interface for all models:fromctransformersimportAutoModelForCausalLMllm=AutoModelForCausalLM.from_pretrained('/path/to/ggml-gpt-2.bin',model_type='gpt2')print(llm('AI is going to'))Run in Google ColabIf you are gettingillegal instructionerror, try usinglib='avx'orlib='basic':llm=AutoModelForCausalLM.from_pretrained('/path/to/ggml-gpt-2.bin',model_type='gpt2',lib='avx')It provides a generator interface for more control:tokens=llm.tokenize('AI is going to')fortokeninllm.generate(tokens):print(llm.detokenize(token))It can be used with a custom or Hugging Face tokenizer:fromtransformersimportAutoTokenizertokenizer=AutoTokenizer.from_pretrained('gpt2')tokens=tokenizer.encode('AI is going to')fortokeninllm.generate(tokens):print(tokenizer.decode(token))It also provides access to the low-level C API. SeeDocumentationsection below.Hugging Face HubIt can be used with models hosted on the Hub:llm=AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml')If a model repo has multiple model files (.binfiles), specify a model file using:llm=AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml',model_file='ggml-model.bin')It can be used with your own models uploaded on the Hub. For better user experience, upload only one model per repo.To use it with your own model, addconfig.jsonfile to your model repo specifying themodel_type:{"model_type":"gpt2"}You can also specify additional parameters undertask_specific_params.text-generation.Seemarella/gpt-2-ggmlfor a minimal example andmarella/gpt-2-ggml-examplefor a full example.LangChainIt is integrated into LangChain. SeeLangChain docs.GPUNote:Currently only LLaMA and Falcon models have GPU support.To run some of the model layers on GPU, set thegpu_layersparameter:llm=AutoModelForCausalLM.from_pretrained('/path/to/ggml-llama.bin',model_type='llama',gpu_layers=50)CUDAMake sure you have installedCUDA 12and latestNVIDIA Drivers.Show instructions for CUDA 11To use with CUDA 11, install thectransformerspackage using:CT_CUBLAS=1pipinstallctransformers--no-binaryctransformersOn Windows PowerShell run:$env:CT_CUBLAS=1pipinstallctransformers--no-binaryctransformersOn Windows Command Prompt run:setCT_CUBLAS=1pipinstallctransformers--no-binaryctransformersRun in Google ColabMetalTo enable Metal support, install thectransformerspackage using:CT_METAL=1pipinstallctransformers--no-binaryctransformersDocumentationConfigParameterTypeDescriptionDefaulttop_kintThe top-k value to use for sampling.40top_pfloatThe top-p value to use for sampling.0.95temperaturefloatThe temperature to use for sampling.0.8repetition_penaltyfloatThe repetition penalty to use for sampling.1.1last_n_tokensintThe number of last tokens to use for repetition penalty.64seedintThe seed value to use for sampling tokens.-1max_new_tokensintThe maximum number of new tokens to generate.256stopList[str]A list of sequences to stop generation when encountered.NonestreamboolWhether to stream the generated text.FalseresetboolWhether to reset the model state before generating text.Truebatch_sizeintThe batch size to use for evaluating tokens.8threadsintThe number of threads to use for evaluating tokens.-1context_lengthintThe maximum context length to use.-1gpu_layersintThe number of layers to run on GPU.0Note:Currently only LLaMA, MPT, Falcon models support thecontext_lengthparameter and only LLaMA, Falcon models support thegpu_layersparameter.classAutoModelForCausalLMclassmethodAutoModelForCausalLM.from_pretrainedfrom_pretrained(model_path_or_repo_id:str,model_type:Optional[str]=None,model_file:Optional[str]=None,config:Optional[ctransformers.hub.AutoConfig]=None,lib:Optional[str]=None,local_files_only:bool=False,**kwargs)→LLMLoads the language model from a local file or remote repo.Args:model_path_or_repo_id: The path to a model file or directory or the name of a Hugging Face Hub model repo.model_type: The model type.model_file: The name of the model file in repo or directory.config:AutoConfigobject.lib: The path to a shared library or one ofavx2,avx,basic.local_files_only: Whether or not to only look at local files (i.e., do not try to download the model).Returns:LLMobject.classLLMmethodLLM.__init____init__(model_path:str,model_type:str,config:Optional[ctransformers.llm.Config]=None,lib:Optional[str]=None)Loads the language model from a local file.Args:model_path: The path to a model file.model_type: The model type.config:Configobject.lib: The path to a shared library or one ofavx2,avx,basic.propertyLLM.configThe config object.propertyLLM.context_lengthThe context length of model.propertyLLM.embeddingsThe input embeddings.propertyLLM.eos_token_idThe end-of-sequence token.propertyLLM.logitsThe unnormalized log probabilities.propertyLLM.model_pathThe path to the model file.propertyLLM.model_typeThe model type.propertyLLM.vocab_sizeThe number of tokens in vocabulary.methodLLM.detokenizedetokenize(tokens:Sequence[int],decode:bool=True)→Union[str,bytes]Converts a list of tokens to text.Args:tokens: The list of tokens.decode: Whether to decode the text as UTF-8 string.Returns:The combined text of all tokens.methodLLM.embedembed(input:Union[str,Sequence[int]],batch_size:Optional[int]=None,threads:Optional[int]=None)→List[float]Computes embeddings for a text or list of tokens.Note:Currently only LLaMA and Falcon models support embeddings.Args:input: The input text or list of tokens to get embeddings for.batch_size: The batch size to use for evaluating tokens. Default:8threads: The number of threads to use for evaluating tokens. Default:-1Returns:The input embeddings.methodLLM.evaleval(tokens:Sequence[int],batch_size:Optional[int]=None,threads:Optional[int]=None)→NoneEvaluates a list of tokens.Args:tokens: The list of tokens to evaluate.batch_size: The batch size to use for evaluating tokens. Default:8threads: The number of threads to use for evaluating tokens. Default:-1methodLLM.generategenerate(tokens:Sequence[int],top_k:Optional[int]=None,top_p:Optional[float]=None,temperature:Optional[float]=None,repetition_penalty:Optional[float]=None,last_n_tokens:Optional[int]=None,seed:Optional[int]=None,batch_size:Optional[int]=None,threads:Optional[int]=None,reset:Optional[bool]=None)→Generator[int,NoneType,NoneType]Generates new tokens from a list of tokens.Args:tokens: The list of tokens to generate tokens from.top_k: The top-k value to use for sampling. Default:40top_p: The top-p value to use for sampling. Default:0.95temperature: The temperature to use for sampling. Default:0.8repetition_penalty: The repetition penalty to use for sampling. Default:1.1last_n_tokens: The number of last tokens to use for repetition penalty. Default:64seed: The seed value to use for sampling tokens. Default:-1batch_size: The batch size to use for evaluating tokens. Default:8threads: The number of threads to use for evaluating tokens. Default:-1reset: Whether to reset the model state before generating text. Default:TrueReturns:The generated tokens.methodLLM.is_eos_tokenis_eos_token(token:int)→boolChecks if a token is an end-of-sequence token.Args:token: The token to check.Returns:Trueif the token is an end-of-sequence token elseFalse.methodLLM.resetreset()→NoneResets the model state.methodLLM.samplesample(top_k:Optional[int]=None,top_p:Optional[float]=None,temperature:Optional[float]=None,repetition_penalty:Optional[float]=None,last_n_tokens:Optional[int]=None,seed:Optional[int]=None)→intSamples a token from the model.Args:top_k: The top-k value to use for sampling. Default:40top_p: The top-p value to use for sampling. Default:0.95temperature: The temperature to use for sampling. Default:0.8repetition_penalty: The repetition penalty to use for sampling. Default:1.1last_n_tokens: The number of last tokens to use for repetition penalty. Default:64seed: The seed value to use for sampling tokens. Default:-1Returns:The sampled token.methodLLM.tokenizetokenize(text:str)→List[int]Converts a text into list of tokens.Args:text: The text to tokenize.Returns:The list of tokens.methodLLM.__call____call__(prompt:str,max_new_tokens:Optional[int]=None,top_k:Optional[int]=None,top_p:Optional[float]=None,temperature:Optional[float]=None,repetition_penalty:Optional[float]=None,last_n_tokens:Optional[int]=None,seed:Optional[int]=None,batch_size:Optional[int]=None,threads:Optional[int]=None,stop:Optional[Sequence[str]]=None,stream:Optional[bool]=None,reset:Optional[bool]=None)→Union[str,Generator[str,NoneType,NoneType]]Generates text from a prompt.Args:prompt: The prompt to generate text from.max_new_tokens: The maximum number of new tokens to generate. Default:256top_k: The top-k value to use for sampling. Default:40top_p: The top-p value to use for sampling. Default:0.95temperature: The temperature to use for sampling. Default:0.8repetition_penalty: The repetition penalty to use for sampling. Default:1.1last_n_tokens: The number of last tokens to use for repetition penalty. Default:64seed: The seed value to use for sampling tokens. Default:-1batch_size: The batch size to use for evaluating tokens. Default:8threads: The number of threads to use for evaluating tokens. Default:-1stop: A list of sequences to stop generation when encountered. Default:Nonestream: Whether to stream the generated text. Default:Falsereset: Whether to reset the model state before generating text. Default:TrueReturns:The generated text.LicenseMIT