Package: onprem
OnPrem.LLMA tool for running large language models on-premises using non-public
dataOnPrem.LLMis a simple Python
package that makes it easier to run large language models (LLMs) on your
own machines using non-public data (possibly behind corporate
firewalls). Inspired largely by theprivateGPTGitHub repo,OnPrem.LLMis intended to help integrate local LLMs into practical
applications.The full documentation ishere.A Google Colab demo of installing and usingOnPrem.LLMishere.InstallOnce you haveinstalled
PyTorchandinstalled
llama-cpp-python,
you can installOnPrem.LLMwith:pipinstallonpremFor fast GPU-accelerated inference, seeadditional instructions
below.
Seethe FAQ, if you experience
issues withllama-cpp-pythoninstallation.Note:Thepip install onpremcommand will install PyTorch and
llama-cpp-python automatically if not already installed, but we
recommend visting the links above to install these packages in a way
that is optimized for your system (e.g., with GPU support).How to UseSetupfromonpremimportLLMllm=LLM()By default, a 7B-parameter model is downloaded and used. Ifuse_larger=True, a 13B-parameter is used. You can also supply the URL
to an LLM of your choosing toLLM(see thecode
generation section
belowfor an
example). Any extra parameters supplied toLLMare forwarded
directly tollama-cpp-python. As of v0.0.20,OnPrem.LLMsupports
the newer GGUF format.Send Prompts to the LLM to Solve ProblemsThis is an example of few-shot prompting, where we provide an example of
what we want the LLM to do.prompt="""Extract the names of people in the supplied sentences. Here is an example:Sentence: James Gandolfini and Paul Newman were great actors.People:James Gandolfini, Paul NewmanSentence:I like Cillian Murphy's acting. Florence Pugh is great, too.People:"""saved_output=llm.prompt(prompt)Cillian Murphy, Florence PughAdditional prompt examples areshown
here.Talk to Your DocumentsAnswers are generated from the content of your documents (i.e.,retrieval augmented generationor
RAG). Here, we will supplyuse_larger=Trueto use the larger default
model better suited to this use case in addition to usingGPU
offloadingto speed up answer generation. However, the smaller Zephyr-7B model may
perform even better, responds faster, and is used in ourexample
notebook.fromonpremimportLLMllm=LLM(use_larger=True,n_gpu_layers=35)Step 1: Ingest the Documents into a Vector Databasellm.ingest("./sample_data")Creating new vectorstore at /home/amaiya/onprem_data/vectordb
Loading documents from ./sample_data
Loaded 12 new documents from ./sample_data
Split into 153 chunks of text (max. 500 chars each)
Creating embeddings. May take some minutes...
Ingestion complete! You can now query your documents using the LLM.ask method

Loading new documents: 100%|██████████████████████| 3/3 [00:00<00:00, 25.52it/s]Step 2: Answer Questions About the Documentsquestion="""What is  ktrain?"""result=llm.ask(question)ktrain is a low-code platform designed to facilitate the full machine learning workflow, from preprocessing inputs to training, tuning, troubleshooting, and applying models. It focuses on automating other aspects of the ML workflow in order to augment and complement human engineers rather than replacing them. Inspired by fastai and ludwig, ktrain is intended to democratize machine learning for beginners and domain experts with minimal programming or data science experience.The sources used by the model to generate the answer are stored inresult['source_documents']:print("\nSources:\n")fori,documentinenumerate(result["source_documents"]):print(f"\n{i+1}.> "+document.metadata["source"]+":")print(document.page_content)Sources:


1.> ./sample_data/ktrain_paper.pdf:
lection (He et al., 2019). By contrast, ktrain places less emphasis on this aspect of au-
tomation and instead focuses on either partially or fully automating other aspects of the
machine learning (ML) workﬂow. For these reasons, ktrain is less of a traditional Au-
2

2.> ./sample_data/ktrain_paper.pdf:
possible, ktrain automates (either algorithmically or through setting well-performing de-
faults), but also allows users to make choices that best ﬁt their unique application require-
ments. In this way, ktrain uses automation to augment and complement human engineers
rather than attempting to entirely replace them. In doing so, the strengths of both are
better exploited. Following inspiration from a blog post1 by Rachel Thomas of fast.ai

3.> ./sample_data/ktrain_paper.pdf:
with custom models and data formats, as well.
Inspired by other low-code (and no-
code) open-source ML libraries such as fastai (Howard and Gugger, 2020) and ludwig
(Molino et al., 2019), ktrain is intended to help further democratize machine learning by
enabling beginners and domain experts with minimal programming or data science experi-
4. http://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups
6

4.> ./sample_data/ktrain_paper.pdf:
ktrain: A Low-Code Library for Augmented Machine Learning
toML platform and more of what might be called a “low-code” ML platform. Through
automation or semi-automation, ktrain facilitates the full machine learning workﬂow from
curating and preprocessing inputs (i.e., ground-truth-labeled training data) to training,
tuning, troubleshooting, and applying models. In this way, ktrain is well-suited for domain
experts who may have less experience with machine learning and software coding. WhereGuided PromptsYou can useOnPrem.LLMwith theGuidancepackage to guide the
LLM to generate outputs based on your conditions and constraints. We’ll
show a couple of examples here, but seeour documentation on guided
promptsfor more information.Structured Outputs withonprem.guider.GuiderHere, we’ll use aGuiderinstance
to generate fictional D&D-type characters that conform to the precise
structure we want (i.e., JSON):fromonprem.guiderimportGuiderguider=Guider(llm)# create the Guider instancefromonprem.guiderimportGuiderfromguidanceimportgen,selectguider=Guider(llm)# this is a function that generates a Guidance prompt that will be fed to Guidersample_weapons=["sword","axe","mace","spear","bow","crossbow"]sample_armour=["leather","chainmail","plate"]defgenerate_character_prompt(character_one_liner,weapons:list[str]=sample_weapons,armour:list[str]=sample_armour,n_items:int=3):prompt=''prompt+="{"prompt+=f'"description" : "{character_one_liner}",'prompt+='"name" : "'+gen(name="character_name",stop='"')+'",'prompt+='"age" : '+gen(name="age",regex="[0-9]+")+','prompt+='"armour" : "'+select(armour,name="armour")+'",'prompt+='"weapon" : "'+select(weapons,name="weapon")+'",'prompt+='"class" : "'+gen(name="character_class",stop='"')+'",'prompt+='"mantra" : "'+gen(name="mantra",stop='"')+'",'prompt+='"strength" : '+gen(name="age",regex="[0-9]+")+','prompt+='"quest_items" : [ 'foriinrange(n_items):prompt+='"'+gen(name="items",list_append=True,stop='"')+'"'ifi<n_items-1:prompt+=','prompt+="]"prompt+="}"returnprompt# feed prompt to Guider and extract JSONimportjsond=guider.prompt(generate_character_prompt("A quick and nimble fighter"),echo=False)print('Generated JSON:')print(json.dumps(d,indent=4))Generated JSON:
{
    "items": [
        "Quest Item 3",
        "Quest Item 2",
        "Quest Item 1"
    ],
    "age": "10",
    "mantra": "I am the blade of justice.",
    "character_class": "fighter",
    "weapon": "sword",
    "armour": "leather",
    "character_name": "Katana"
}Using Regular Expressions to Control LLM Generationprompt=f"""Question: Luke has ten balls. He gives three to his brother. How many balls does he have left?Answer: """+gen(name='answer',regex='\d+')guider.prompt(prompt,echo=False){'answer': '7'}prompt='19, 18,'+gen(name='output',max_tokens=50,stop_regex='[^\d]7[^\d]')guider.prompt(prompt)19, 18,17,16,15,14,13,12,11,10,9,8,{'output': ' 17, 16, 15, 14, 13, 12, 11, 10, 9, 8,'}Seethe
documentationfor more examples of how to useGuidancewithOnPrem.LLM.Summarization PipelineSummarize your raw documents (e.g., PDFs, MS Word) with an LLM.fromonpremimportLLMllm=LLM(n_gpu_layers=35,verbose=False,mute_stream=True)# disabling viewing of intermediate summarization prompts/inferencesfromonprem.pipelinesimportSummarizersumm=Summarizer(llm)text=summ.summarize('sample_data/1/ktrain_paper.pdf',max_chunks_to_use=5)# omit max_chunks_to_use parameter to consider entire documentprint(text)The KTrain library provides an easy-to-use framework for building and training machine learning models using low-code techniques for various data types (text, image, graph, tabular) and tasks (classification, regression). It can be used to fine-tune pretrained models in text classification and image classification tasks respectively. Additionally, it reduces cognitive load by providing a unified interface to various and disparate machine learning tasks, allowing users to focus on more important tasks that may require domain expertise or are less amenable to automation.Text to Code GenerationWe’ll use the CodeUp LLM by supplying the URL and employing the
particular prompt format this model expects.fromonpremimportLLMurl="https://huggingface.co/TheBloke/CodeUp-Llama-2-13B-Chat-HF-GGUF/resolve/main/codeup-llama-2-13b-chat-hf.Q4_K_M.gguf"llm=LLM(url,n_gpu_layers=43)# see below for GPU informationSetup the prompt based on whatthis model
expects(this is important):template="""Below is an instruction that describes a task. Write a response that appropriately completes the request.### Instruction:{prompt}### Response:"""answer=llm.prompt("Write Python code to validate an email address.",prompt_template=template)Here is an example of Python code that can be used to validate an email address:
```
import re

def validate_email(email):
    # Use a regular expression to check if the email address is in the correct format
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    if re.match(pattern, email):
        return True
    else:
        return False

# Test the validate_email function with different inputs
print("Email address is valid:", validate_email("example@example.com"))  # Should print "True"
print("Email address is invalid:", validate_email("example@"))  # Should print "False"
print("Email address is invalid:", validate_email("example.com"))  # Should print "False"
```
The code defines a function `validate_email` that takes an email address as input and uses a regular expression to check if the email address is in the correct format. The regular expression checks for an email address that consists of one or more letters, numbers, periods, hyphens, or underscores followed by the `@` symbol, followed by one or more letters, periods, hyphens, or underscores followed by a `.` and two to three letters.
The function returns `True` if the email address is valid, and `False` otherwise. The code also includes some test examples to demonstrate how to use the function.Let’s try out the code generated above.importredefvalidate_email(email):# Use a regular expression to check if the email address is in the correct formatpattern=r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"ifre.match(pattern,email):returnTrueelse:returnFalseprint(validate_email("sam@@openai.com"))# bad email addressprint(validate_email("sam@openai"))# bad email addressprint(validate_email("sam@openai.com"))# good email addressFalse
False
TrueThe generated code may sometimes need editing, but this one worked
out-of-the-box.Using OpenAI Models with OnPrem.LLMEven when using on-premises language models, it can sometimes be useful
to have easy access to non-local, cloud-based models (e.g., OpenAI) for
testing, producing baselines for comparison, and generating synthetic
examples for fine-tuning. For these reasons, in spite of the name,OnPrem.LLMnow includes support for OpenAI chat models:fromonpremimportLLMllm=LLM(model_url='openai://gpt-3.5-turbo',temperature=0)# ChatGPT/home/amaiya/projects/ghub/onprem/onprem/core.py:138: UserWarning: The model you supplied is gpt-3.5-turbo, an external service (i.e., not on-premises). Use with caution, as your data and prompts will be sent externally.
  warnings.warn(f'The model you supplied is {self.model_name}, an external service (i.e., not on-premises). '+\saved_result=llm.prompt('List three cute  names for a cat and explain why each is cute.')1. Whiskers: Whiskers is a cute name for a cat because it perfectly describes one of the most adorable features of a feline - their long, delicate whiskers. It's a playful and endearing name that captures the essence of a cat's charm.

2. Pudding: Pudding is an incredibly cute name for a cat because it evokes a sense of softness and sweetness. Just like a bowl of creamy pudding, this name brings to mind a cat's cuddly and lovable nature. It's a name that instantly makes you want to snuggle up with your furry friend.

3. Muffin: Muffin is an adorable name for a cat because it conjures up images of something small, round, and irresistibly cute - just like a cat! This name is playful and charming, and it perfectly captures the delightful and lovable nature of our feline companions.Built-In Web AppOnPrem.LLMincludes a built-in Web app to access the LLM. To start
it, run the following command after installation:onprem--port8000Then, enterlocalhost:8000(or<domain_name>:8000if running on
remote server) in a Web browser to access the application:For more information,see the corresponding
documentation.Speeding Up Inference Using a GPUThe above example employed the use of a CPU. If you have a GPU (even an
older one with less VRAM), you can speed up responses. Seethe
LangChain docs on
LLama.cppfor installingllama-cpp-pythonwith GPU support for your system.The steps below describe installing and usingllama-cpp-pythonwithcuBLASsupport and can be employed for GPU acceleration on systems
with NVIDIA GPUs (e.g., Linux, WSL2, Google Colab).Step 1: Installllama-cpp-pythonwith cuBLAS supportCMAKE_ARGS="-DLLAMA_CUBLAS=on"FORCE_CMAKE=1pipinstall--upgrade--force-reinstallllama-cpp-python--no-cache-dir# For Mac users replace above with:# CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dirStep 2: Use then_gpu_layersargument withLLMllm=LLM(n_gpu_layers=35)The value forn_gpu_layersdepends on your GPU memory and the model
you’re using (e.g., max of 35 for default 7B model). You can reduce the
value if you get an error (e.g.,CUDA error: out-of-memory). For
instance, using two old NVDIDIA TITAN V GPUs each with 12GB of VRAM, 59
out 83 layers in aquantized Llama-2 70B
modelcan be offloaded to the GPUs (i.e., 60 layers or more results in a “CUDA
out of memory” error).With the steps above, calls to methods likellm.promptwill offload
computation to your GPU and speed up responses from the LLM.The above assumes that NVIDIA drivers and the CUDA toolkit are already
installed. On Ubuntu Linux systems, this can be accomplishedwith a
single
command.FAQHow do I use other models with OnPrem.LLM?You can supply the URL to other models to theLLMconstructor,
as we did above in the code generation example.As of v0.0.20, we support models in GGUF format, which supersedes
the older GGML format. You can find llama.cpp-supported models
withGGUFin the file name onhuggingface.co.Make sure you are pointing to the URL of the actual GGUF model
file, which is the “download” link on the model’s page. An example
forMistral-7Bis shown below:Note that some models have specific prompt formats. For instance,
the prompt template required forZephyr-7B, as described on
themodel’s
page, is:
\n
\n
\n{prompt}
\n
So, to use theZephyr-7Bmodel, you must supply theprompt_templateargument to theLLMconstructor (or specify it
in thewebapp.ymlconfiguration for the Web app).# how to use Zephyr-7B with OnPrem.LLMllm=LLM(model_url='https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/resolve/main/zephyr-7b-beta.Q4_K_M.gguf',prompt_template="
\n
\n
\n{prompt}
\n
",n_gpu_layers=33)llm.prompt("List three cute names for a cat.")I’m behind a corporate firewall and am receiving an SSL error when
trying to download the model?Try this:fromonpremimportLLMLLM.download_model(url,ssl_verify=False)How do I use this on a machine with no internet access?Use theLLM.download_modelmethod to download the model files to<your_home_directory>/onprem_dataand transfer them to the same
location on the air-gapped machine.For theingestandaskmethods, you will need to also download
and transfer the embedding model files:fromsentence_transformersimportSentenceTransformermodel=SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')model.save('/some/folder')Copy thesome/folderfolder to the air-gapped machine and supply
the path toLLMvia theembedding_model_nameparameter.When installingonprem, I’m getting errors related tollama-cpp-pythonon Windows/Mac/Linux?Seethis LangChain documentation on
LLama.cppfor help on installing thellama-cpp-pythonpackage for your
system. Additional tips for different operating systems are shown
below:ForLinuxsystems like Ubuntu, try this:sudo apt-get install build-essential g++ clang. Other tips arehere.ForWindowssystems, either useWindows Subsystem for Linux
(WSL)or
installMicrosoft Visual Studio build
toolsand
ensure the selections shown inthis
postare installed. WSL is recommended.ForMacs, try followingthese
tips.If you still have problems, there are various other tips for each
of the above OSes inthis privateGPT repo
thread. Of
course, you can alsoeasily
useOnPrem.LLMon Google Colab.llama-cpp-pythonis failing to load my model from the model path
on Google Colab.For reasons that are unclear, newer versions ofllama-cpp-pythonfail to load models on Google Colab unless you supplyverbose=Trueto theLLMconstructor (which is passed directly
tollama-cpp-python). If you experience this problem locally,
try supplyingverbose=TruetoLLM.I’m getting an“Illegal instruction (core dumped)error when
instantiating alangchain.llms.Llamacpporonprem.LLMobject?Your CPU may not support instructions thatcmakeis using for
one reason or another (e.g.,due to Hyper-V in VirtualBox
settings).
You can try turning them off when building and installingllama-cpp-python:# exampleCMAKE_ARGS="-DLLAMA_CUBLAS=ON -DLLAMA_AVX2=OFF -DLLAMA_AVX=OFF -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF"FORCE_CMAKE=1pipinstall--force-reinstallllama-cpp-python--no-cache-dirHow can I speed upLLM.ingestusing my GPU?Try using theembedding_model_kwargsargument:fromonpremimportLLMllm=LLM(embedding_model_kwargs={'device':'cuda'})