Package: llm-rs
llm-rs-python: Python Bindings for Rust's llm LibraryWelcome tollm-rs, an unofficial Python interface for the Rust-basedllmlibrary, made possible throughPyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍❤️🦀Withllm-rs, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU or GPU.For a detailed overview of all the supported architectures, visit thellmproject page.Integrations:🦜️🔗LangChain🌾🔱HaystackInstallationSimply install it via pip:pip install llm-rsInstallation with GPU Acceleration Supportllm-rsincorporates support for various GPU-accelerated backends to facilitate enhanced inference times. To enable GPU-acceleration theuse_gpuparameter of yourSessionConfigmust be set toTrue. Thellm documentationlists all model architectures, which are currently accelerated. We distribute prebuilt binaries for the following operating systems and graphics APIs:MacOS (Using Metal)For MacOS users, the Metal-supported version ofllm-rscan be easily installed via pip:pip install llm-rs-metalWindows/Linux (Using CUDA for Nvidia GPUs)Due to the significant file size, CUDA-supported packages cannot be directly uploaded topip. To install them, download the appropriate*.whlfile from the latestReleaseand install it using pip as follows:pip install [wheelname].whlWindows/Linux (Using OpenCL for All GPUs)For universal GPU support on Windows and Linux, we offer an OpenCL-supported version. It can be installed via pip:pip install llm-rs-openclUsageRunning local GGML models:Models can be loaded via theAutoModelinterface.fromllm_rsimportAutoModel,KnownModels#load the modelmodel=AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)#generateprint(model.generate("The meaning of life is"))Streaming TextText can be yielded from a generator via thestreamfunction:fromllm_rsimportAutoModel,KnownModels#load the modelmodel=AutoModel.from_pretrained("path/to/model.bin",model_type=KnownModels.Llama)#generatefortokeninmodel.stream("The meaning of life is"):print(token)Running GGML models from the Hugging Face HubGGML converted models can be directly downloaded and run from the hub.fromllm_rsimportAutoModelmodel=AutoModel.from_pretrained("rustformers/mpt-7b-ggml",model_file="mpt-7b-q4_0-ggjt.bin")If there are multiple models in a repo themodel_filehas to be specified.
If you want to load repositories which were not created throught this library, you have to specify themodel_typeparameter as the metadata files needed to infer the architecture are missing.Running Pytorch Transfomer models from the Hugging Face Hubllm-rssupports automatic conversion of all supported transformer architectures on the Huggingface Hub.To run covnersions additional dependencies are needed which can be installed viapip install llm-rs[convert].The models can then be loaded and automatically converted via thefrom_pretrainedfunction.fromllm_rsimportAutoModelmodel=AutoModel.from_pretrained("mosaicml/mpt-7b")Convert Huggingface Hub ModelsThe following example shows how aPythiamodel can be covnverted, quantized and run.fromllm_rs.convertimportAutoConverterfromllm_rsimportAutoModel,AutoQuantizerimportsys#define the model which should be converted and an output directoryexport_directory="path/to/directory"base_model="EleutherAI/pythia-410m"#convert the modelconverted_model=AutoConverter.convert(base_model,export_directory)#quantize the model (this step is optional)quantized_model=AutoQuantizer.quantize(converted_model)#load the quantized modelmodel=AutoModel.load(quantized_model,verbose=True)#generate textdefcallback(text):print(text,end="")sys.stdout.flush()model.generate("The meaning of life is",callback=callback)🦜️🔗 LangChain UsageUtilizingllm-rs-pythonthrough langchain requires additional dependencies. You can install these usingpip install llm-rs[langchain]. Once installed, you gain access to theRustformersLLMmodel through thellm_rs.langchainmodule. This particular model offers features for text generation and embeddings.Consider the example below, demonstrating a straightforward LLMchain implementation with MPT-Instruct:fromllm_rs.langchainimportRustformersLLMfromlangchainimportPromptTemplatefromlangchain.chainsimportLLMChainfromlangchain.callbacks.streaming_stdoutimportStreamingStdOutCallbackHandlertemplate="""Below is an instruction that describes a task. Write a response that appropriately completes the request.### Instruction:{instruction}### Response:Answer:"""prompt=PromptTemplate(input_variables=["instruction"],template=template,)llm=RustformersLLM(model_path_or_repo_id="rustformers/mpt-7b-ggml",model_file="mpt-7b-instruct-q5_1-ggjt.bin",callbacks=[StreamingStdOutCallbackHandler()])chain=LLMChain(llm=llm,prompt=prompt)chain.run("Write a short post congratulating rustformers on their new release of their langchain integration.")🌾🔱 Haystack UsageUtilizingllm-rs-pythonthrough haystack requires additional dependencies. You can install these usingpip install llm-rs[haystack]. Once installed, you gain access to theRustformersInvocationLayermodel through thellm_rs.haystackmodule. This particular model offers features for text generation.Consider the example below, demonstrating a straightforward Haystack-Pipeline implementation with OpenLLama-3B:fromhaystack.nodesimportPromptNode,PromptModelfromllm_rs.haystackimportRustformersInvocationLayermodel=PromptModel("rustformers/open-llama-ggml",max_length=1024,invocation_layer_class=RustformersInvocationLayer,model_kwargs={"model_file":"open_llama_3b-q5_1-ggjt.bin"})pn=PromptNode(model,max_length=1024)pn("Write me a short story about a lama riding a crab.",stream=True)DocumentationFor in-depth information on customizing the loading and generation processes, refer to our detaileddocumentation.