Package: ctransformers
CTransformersPython bindings for the Transformer models implemented in C/C++ usingGGMLlibrary.Also seeChatDocsSupported ModelsInstallationUsage🤗 TransformersLangChainGPUGPTQDocumentationLicenseSupported ModelsModelsModel TypeCUDAMetalGPT-2gpt2GPT-J, GPT4All-JgptjGPT-NeoX, StableLMgpt_neoxFalconfalcon✅LLaMA, LLaMA 2llama✅✅MPTmpt✅StarCoder, StarChatgpt_bigcode✅Dolly V2dolly-v2ReplitreplitInstallationpipinstallctransformersUsageIt provides a unified interface for all models:fromctransformersimportAutoModelForCausalLMllm=AutoModelForCausalLM.from_pretrained("/path/to/ggml-model.bin",model_type="gpt2")print(llm("AI is going to"))Run in Google ColabTo stream the output, setstream=True:fortextinllm("AI is going to",stream=True):print(text,end="",flush=True)You can load models from Hugging Face Hub directly:llm=AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml")If a model repo has multiple model files (.binor.gguffiles), specify a model file using:llm=AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml",model_file="ggml-model.bin")🤗 TransformersNote:This is an experimental feature and may change in the future.To use it with 🤗 Transformers, create model and tokenizer using:fromctransformersimportAutoModelForCausalLM,AutoTokenizermodel=AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml",hf=True)tokenizer=AutoTokenizer.from_pretrained(model)Run in Google ColabYou can use 🤗 Transformers text generation pipeline:fromtransformersimportpipelinepipe=pipeline("text-generation",model=model,tokenizer=tokenizer)print(pipe("AI is going to",max_new_tokens=256))You can use 🤗 Transformers generationparameters:pipe("AI is going to",max_new_tokens=256,do_sample=True,temperature=0.8,repetition_penalty=1.1)You can use 🤗 Transformers tokenizers:fromctransformersimportAutoModelForCausalLMfromtransformersimportAutoTokenizermodel=AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml",hf=True)# Load model from GGML model repo.tokenizer=AutoTokenizer.from_pretrained("gpt2")# Load tokenizer from original model repo.LangChainIt is integrated into LangChain. SeeLangChain docs.GPUTo run some of the model layers on GPU, set thegpu_layersparameter:llm=AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-GGML",gpu_layers=50)Run in Google ColabCUDAInstall CUDA libraries using:pipinstallctransformers[cuda]ROCmTo enable ROCm support, install thectransformerspackage using:CT_HIPBLAS=1pipinstallctransformers--no-binaryctransformersMetalTo enable Metal support, install thectransformerspackage using:CT_METAL=1pipinstallctransformers--no-binaryctransformersGPTQNote:This is an experimental feature and only LLaMA models are supported usingExLlama.Install additional dependencies using:pipinstallctransformers[gptq]Load a GPTQ model using:llm=AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-GPTQ")Run in Google ColabIf model name or path doesn't contain the wordgptqthen specifymodel_type="gptq".It can also be used with LangChain. Low-level APIs are not fully supported.DocumentationConfigParameterTypeDescriptionDefaulttop_kintThe top-k value to use for sampling.40top_pfloatThe top-p value to use for sampling.0.95temperaturefloatThe temperature to use for sampling.0.8repetition_penaltyfloatThe repetition penalty to use for sampling.1.1last_n_tokensintThe number of last tokens to use for repetition penalty.64seedintThe seed value to use for sampling tokens.-1max_new_tokensintThe maximum number of new tokens to generate.256stopList[str]A list of sequences to stop generation when encountered.NonestreamboolWhether to stream the generated text.FalseresetboolWhether to reset the model state before generating text.Truebatch_sizeintThe batch size to use for evaluating tokens in a single prompt.8threadsintThe number of threads to use for evaluating tokens.-1context_lengthintThe maximum context length to use.-1gpu_layersintThe number of layers to run on GPU.0Note:Currently only LLaMA, MPT and Falcon models support thecontext_lengthparameter.classAutoModelForCausalLMclassmethodAutoModelForCausalLM.from_pretrainedfrom_pretrained(model_path_or_repo_id:str,model_type:Optional[str]=None,model_file:Optional[str]=None,config:Optional[ctransformers.hub.AutoConfig]=None,lib:Optional[str]=None,local_files_only:bool=False,revision:Optional[str]=None,hf:bool=False,**kwargs)→LLMLoads the language model from a local file or remote repo.Args:model_path_or_repo_id: The path to a model file or directory or the name of a Hugging Face Hub model repo.model_type: The model type.model_file: The name of the model file in repo or directory.config:AutoConfigobject.lib: The path to a shared library or one ofavx2,avx,basic.local_files_only: Whether or not to only look at local files (i.e., do not try to download the model).revision: The specific model version to use. It can be a branch name, a tag name, or a commit id.hf: Whether to create a Hugging Face Transformers model.Returns:LLMobject.classLLMmethodLLM.__init____init__(model_path:str,model_type:Optional[str]=None,config:Optional[ctransformers.llm.Config]=None,lib:Optional[str]=None)Loads the language model from a local file.Args:model_path: The path to a model file.model_type: The model type.config:Configobject.lib: The path to a shared library or one ofavx2,avx,basic.propertyLLM.bos_token_idThe beginning-of-sequence token.propertyLLM.configThe config object.propertyLLM.context_lengthThe context length of model.propertyLLM.embeddingsThe input embeddings.propertyLLM.eos_token_idThe end-of-sequence token.propertyLLM.logitsThe unnormalized log probabilities.propertyLLM.model_pathThe path to the model file.propertyLLM.model_typeThe model type.propertyLLM.pad_token_idThe padding token.propertyLLM.vocab_sizeThe number of tokens in vocabulary.methodLLM.detokenizedetokenize(tokens:Sequence[int],decode:bool=True)→Union[str,bytes]Converts a list of tokens to text.Args:tokens: The list of tokens.decode: Whether to decode the text as UTF-8 string.Returns:The combined text of all tokens.methodLLM.embedembed(input:Union[str,Sequence[int]],batch_size:Optional[int]=None,threads:Optional[int]=None)→List[float]Computes embeddings for a text or list of tokens.Note:Currently only LLaMA and Falcon models support embeddings.Args:input: The input text or list of tokens to get embeddings for.batch_size: The batch size to use for evaluating tokens in a single prompt. Default:8threads: The number of threads to use for evaluating tokens. Default:-1Returns:The input embeddings.methodLLM.evaleval(tokens:Sequence[int],batch_size:Optional[int]=None,threads:Optional[int]=None)→NoneEvaluates a list of tokens.Args:tokens: The list of tokens to evaluate.batch_size: The batch size to use for evaluating tokens in a single prompt. Default:8threads: The number of threads to use for evaluating tokens. Default:-1methodLLM.generategenerate(tokens:Sequence[int],top_k:Optional[int]=None,top_p:Optional[float]=None,temperature:Optional[float]=None,repetition_penalty:Optional[float]=None,last_n_tokens:Optional[int]=None,seed:Optional[int]=None,batch_size:Optional[int]=None,threads:Optional[int]=None,reset:Optional[bool]=None)→Generator[int,NoneType,NoneType]Generates new tokens from a list of tokens.Args:tokens: The list of tokens to generate tokens from.top_k: The top-k value to use for sampling. Default:40top_p: The top-p value to use for sampling. Default:0.95temperature: The temperature to use for sampling. Default:0.8repetition_penalty: The repetition penalty to use for sampling. Default:1.1last_n_tokens: The number of last tokens to use for repetition penalty. Default:64seed: The seed value to use for sampling tokens. Default:-1batch_size: The batch size to use for evaluating tokens in a single prompt. Default:8threads: The number of threads to use for evaluating tokens. Default:-1reset: Whether to reset the model state before generating text. Default:TrueReturns:The generated tokens.methodLLM.is_eos_tokenis_eos_token(token:int)→boolChecks if a token is an end-of-sequence token.Args:token: The token to check.Returns:Trueif the token is an end-of-sequence token elseFalse.methodLLM.prepare_inputs_for_generationprepare_inputs_for_generation(tokens:Sequence[int],reset:Optional[bool]=None)→Sequence[int]Removes input tokens that are evaluated in the past and updates the LLM context.Args:tokens: The list of input tokens.reset: Whether to reset the model state before generating text. Default:TrueReturns:The list of tokens to evaluate.methodLLM.resetreset()→NoneDeprecated since 0.2.27.methodLLM.samplesample(top_k:Optional[int]=None,top_p:Optional[float]=None,temperature:Optional[float]=None,repetition_penalty:Optional[float]=None,last_n_tokens:Optional[int]=None,seed:Optional[int]=None)→intSamples a token from the model.Args:top_k: The top-k value to use for sampling. Default:40top_p: The top-p value to use for sampling. Default:0.95temperature: The temperature to use for sampling. Default:0.8repetition_penalty: The repetition penalty to use for sampling. Default:1.1last_n_tokens: The number of last tokens to use for repetition penalty. Default:64seed: The seed value to use for sampling tokens. Default:-1Returns:The sampled token.methodLLM.tokenizetokenize(text:str,add_bos_token:Optional[bool]=None)→List[int]Converts a text into list of tokens.Args:text: The text to tokenize.add_bos_token: Whether to add the beginning-of-sequence token.Returns:The list of tokens.methodLLM.__call____call__(prompt:str,max_new_tokens:Optional[int]=None,top_k:Optional[int]=None,top_p:Optional[float]=None,temperature:Optional[float]=None,repetition_penalty:Optional[float]=None,last_n_tokens:Optional[int]=None,seed:Optional[int]=None,batch_size:Optional[int]=None,threads:Optional[int]=None,stop:Optional[Sequence[str]]=None,stream:Optional[bool]=None,reset:Optional[bool]=None)→Union[str,Generator[str,NoneType,NoneType]]Generates text from a prompt.Args:prompt: The prompt to generate text from.max_new_tokens: The maximum number of new tokens to generate. Default:256top_k: The top-k value to use for sampling. Default:40top_p: The top-p value to use for sampling. Default:0.95temperature: The temperature to use for sampling. Default:0.8repetition_penalty: The repetition penalty to use for sampling. Default:1.1last_n_tokens: The number of last tokens to use for repetition penalty. Default:64seed: The seed value to use for sampling tokens. Default:-1batch_size: The batch size to use for evaluating tokens in a single prompt. Default:8threads: The number of threads to use for evaluating tokens. Default:-1stop: A list of sequences to stop generation when encountered. Default:Nonestream: Whether to stream the generated text. Default:Falsereset: Whether to reset the model state before generating text. Default:TrueReturns:The generated text.LicenseMIT