Package: py-llm-core
PyLLMCoreOverviewPyLLMCore is a light-weighted structured interface with Large Language Models
with native support forllama.cppand OpenAI API.The design decisions behind PyLLMCore are:Sane defaultsClear abstractions and terminologyOut of the box utility classesMain benefits of using PyLLMCorePythonic APISimple to useYou need structureseverywhere(provided by the standard librarydataclassesmodule)High-level API with theassistantsmoduleSwitching between models has never been easierWhy you shouldn't use PyLLMCoreYou need a lot of external integrations: Take a look atlangchainYou need tremendous performance: Take a look atvllmYou don't need OpenAI: Take a look allama-cpp-python(which is integrated in PyLLMCore)You use Pydantic and don't use the dataclasses moduleModels supportedBesides OpenAI API, the following models are supported for local inference using thellama.cpp:LLaMALLaMA 2FalconAlpacaGPT4AllChinese LLaMA / AlpacaandChinese LLaMA-2 / Alpaca-2Vigogne (French)VicunaKoalaOpenBuddy 🐶 (Multilingual)Pygmalion 7B / Metharme 7BWizardLMBaichuan-7Band its derivations (such asbaichuan-7b-sft)Aquila-7B/AquilaChat-7BStarcoder modelsMistral AI v0.1Use casesPyLLMCore covers a narrow range of use cases and serves as a building brick:Parsing: see theparsersmoduleSummarizing: see theassistants.summarizersmoduleQuestion answering: see theassistants.analystmoduleHallucinations reduction: see theassistants.verifiersmoduleContext size management: see thesplittersmoduleTokenizing, encoding, decoding: see thetoken_codecsmoduleChangelog2.8.3: Raised timeout2.8.1: Fixed bug when deserializing instances2.8.0: Added support for native type annotation (pep585) for lists and sets2.7.0: Fixed bug when function_call was set at None2.6.1: Add dynamic max_tokens computation for OpenAI2.6.0: Add support for Azure OpenAI2.5.1: Fix bug on system prompt format2.5.0: Add support for LLaVA models2.4.0:Set timeouts on OpenAI API2.2.0:Default settings on ARM64 MacOS modified (1 thread / offloading everything on the GPU)Addedcompletion_kwargsfor Assistants to set temperature2.1.0:Added support for Enum to provide better support for classification tasksAdded example in the documentation2.0.0:Refactored codeDynamically enable GPU offloading on MacOSAdded configuration option for storing local models (MODELS_CACHE_DIR)Updated documentation1.4.0: Free up resources in LLamaParser when exiting the context manager1.3.0: Support for LLaMA based models (llama, llama2, Mistral Instruct)1.2.0: Chain of density prompting implemented with OpenAI1.1.0: Chain of Verification implemented with OpenAI1.0.0: Initial versionInstallQuick startpipinstallpy-llm-core# Add you OPENAI_API_KEY to the environmentexportOPENAI_API_KEY=sk-<replacewithyouractualapikey># For local inference with GGUF models, store your models in MODELS_CACHE_DIRmkdir-p~/.cache/py-llm-core/modelscd~/.cache/py-llm-core/models
wgethttps://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q4_K_M.ggufTroubleshootingThellama-cpp-pythondependency may improperly detects the architecture and raise an erroran incompatible architecture (have 'x86_64', need 'arm64')).If that's the case, run the following in your virtual env:CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64"pip3install--upgrade--verbose--force-reinstall--no-cache-dirllama-cpp-pythonDocumentationParsingUse Parser classesAvailable parsers:parsers.OpenAIParserparsers.LLaMACPPParserUsing a local model : Mistral AI Instructfromdataclassesimportdataclassfromllm_core.parsersimportLLaMACPPParser@dataclassclassBook:title:strsummary:strauthor:strpublished_year:inttext="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy (laterexpanded into the Foundation series). Foundation is a cycle of fiveinterrelated short stories, first published as a single book by Gnome Pressin 1951. Collectively they tell the early story of the Foundation,an institute founded by psychohistorian Hari Seldon to preserve the bestof galactic civilization after the collapse of the Galactic Empire."""model="mistral-7b-instruct-v0.1.Q4_K_M.gguf"withLLaMACPPParser(Book,model=model)asparser:book=parser.parse(text)print(book)Book(title='Foundation',summary="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy(later expanded into the Foundation series). Foundation is acycle of five interrelated short stories, first published as asingle book by Gnome Press in 1951. Collectively they tell theearly story of the Foundation, an institute founded bypsychohistorian Hari Seldon to preserve the best of galacticcivilization after the collapse of the Galactic Empire.""",author='Isaac Asimov',published_year=1951)Using OpenAIfromdataclassesimportdataclassfromllm_core.parsersimportOpenAIParser@dataclassclassBook:title:strsummary:strauthor:strpublished_year:inttext="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy (laterexpanded into the Foundation series). Foundation is a cycle of fiveinterrelated short stories, first published as a single book by Gnome Pressin 1951. Collectively they tell the early story of the Foundation,an institute founded by psychohistorian Hari Seldon to preserve the bestof galactic civilization after the collapse of the Galactic Empire."""withOpenAIParser(Book)asparser:book=parser.parse(text)print(book)Book(title='Foundation',summary="""Foundation is a cycle of five interrelatedshort stories, first published as a single book by Gnome Press in 1951.Collectively they tell the early story of the Foundation, an institutefounded by psychohistorian Hari Seldon to preserve the best of galacticcivilization after the collapse of the Galactic Empire.""",author='Isaac Asimov',published_year=1951)Using Azure OpenAIUsing OpenAI Azure services can be enabled by following the steps:Create an Azure accountEnable Azure OpenAI cognitive servicesGet the API key and the API endpoint providedSet the environment variable USE_AZURE_OPENAI to True (export USE_AZURE_OPENAI=True)Set the environment variable AZURE_OPENAI_ENDPOINT (see step 3)Set the environment variable AZURE_OPENAI_API_KEY (see step 3)Create a deployment where you use the model name from OpenAI. You'll need to remove dot signs, i.e. for the modelgpt-3.5-turbo-0613create a deployment namedgpt-35-turbo-0613.PyLLMCore will take care of removing the dot sign for you so you can use the same code base for both OpenAI and Azure.When calling Parser or Assistant classes, specify the modelThe following example uses Azure OpenAI:exportUSE_AZURE_OPENAI=TrueexportAZURE_OPENAI_API_KEY=<yourapikey>exportAZURE_OPENAI_ENDPOINT=https://<yourendpoint>.openai.azure.com/fromdataclassesimportdataclassfromllm_core.parsersimportOpenAIParser@dataclassclassBook:title:strsummary:strauthor:strpublished_year:inttext="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy (laterexpanded into the Foundation series). Foundation is a cycle of fiveinterrelated short stories, first published as a single book by Gnome Pressin 1951. Collectively they tell the early story of the Foundation,an institute founded by psychohistorian Hari Seldon to preserve the bestof galactic civilization after the collapse of the Galactic Empire."""withOpenAIParser(Book,model="gpt-3.5-turbo-0613")asparser:book=parser.parse(text)print(book)Book(title='Foundation',summary="""Foundation is a cycle of five interrelatedshort stories, first published as a single book by Gnome Press in 1951.Collectively they tell the early story of the Foundation, an institutefounded by psychohistorian Hari Seldon to preserve the best of galacticcivilization after the collapse of the Galactic Empire.""",author='Isaac Asimov',published_year=1951)Perform advanced tasksOverviewTo perform generic tasks, you will use theassistantsmodule that provides generic assistants:assistants.OpenAIAssistantassistants.LLaMACPPAssistantUsing these assistants, you can take a look at how the utilities are built:assistants.analysts.Analystassistants.verifiers.Doubterassistants.verifiers.ConsistencyVerifierassistants.summarizers.SummarizerCreate your own utilityThere are 3 items required to build and run a utility:A language model (any compatible model)An assistant class: This is where your logic is writtenA results class: This is the structure you need. It also contains the prompt.Here is an example whereRecipeis the results class. We'll use the
Mistral AI Instruct model.fromtypingimportListfromdataclassesimportdataclass# LLaMACPPAssistant is needed to instanciate Mistral Instructfromllm_core.assistantsimportLLaMACPPAssistant# Make sure that ~/.cache/py-llm-core/models contains the following filemodel="mistral-7b-instruct-v0.1.Q4_K_M.gguf"@dataclassclassRecipeStep:step_title:strstep_instructions:str@dataclassclassRecipe:system_prompt="You are a world-class chef"prompt="Write a detailed step-by-step recipe to make{dish}"title:strsteps:List[RecipeStep]ingredients:List[str]classChef:defgenerate_recipe(self,dish):withLLaMACPPAssistant(Recipe,model=model)asassistant:recipe=assistant.process(dish=dish)returnrecipechef=Chef()recipe=chef.generate_recipe("Boeuf bourguignon")print(recipe)Recipe(title="Boeuf Bourguignon Recipe",steps=[RecipeStep(step_title="Preheat the Oven",step_instructions="Preheat the oven to 350°F.",),RecipeStep(step_title="Brown the Brisket",step_instructions="In a large pot, heat the olive oil over me...",),RecipeStep(step_title="Cook the Onions and Garlic",step_instructions="Remove the brisket from the pot and set it...",),RecipeStep(step_title="Simmer the Wine",step_instructions="Add the red wine to the pot and stir to sc...",),RecipeStep(step_title="Bake in the Oven",step_instructions="Return the brisket to the pot, along with ...",),RecipeStep(step_title="Finish Cooking",step_instructions="After 2 hours, remove the aluminum foil an...",),RecipeStep(step_title="Serve",step_instructions="Remove the brisket from the pot and let it...",),],ingredients=["1 pound beef brisket","2 tablespoons olive oil","1 large onion, chopped","3 cloves garlic, minced","1 cup red wine","4 cups beef broth","2 cups heavy cream","1 teaspoon dried thyme","1 teaspoon dried rosemary","Salt and pepper to taste",],)Summarizingimportwikipediafromllm_core.assistantsimportSummarizer,LLaMACPPAssistantsummarizer=Summarizer(model="mistral-7b-instruct-v0.1.Q4_K_M.gguf",assistant_cls=LLaMACPPAssistant)text=wikipedia.page("Foundation from Isaac Asimov").content# To summarize only with 50% of the model context sizepartial_summary=summarizer.fast_summarize(text)# Iterative summaries on the whole contentforsummaryinsummarizer.summarize(text):print(summary)The partial summary generated is:SimpleSummary(content="""The Foundation series is a science fiction book series writtenby Isaac Asimov. It was first published as a series of short stories andnovellas in 1942-50, and subsequently in three collections in 1951-53....""")Reduce hallucinations using the verifiers moduleThis example implements loosely the Chain of Verification (CoVe).To reduce hallucinations in the LLM completions, you can use the following example
as a starting point:importrequestsfromllm_core.splittersimportTokenSplitterfromllm_core.assistantsimport(Analyst,Doubter,ConsistencyVerifier,LLaMACPPAssistant,)pizza_dough_recipe_url=("https://raw.githubusercontent.com/hendricius/pizza-dough/main/README.md")model="mistral-7b-instruct-v0.1.Q4_K_M.gguf"assistant_cls=LLaMACPPAssistant# Utilitiesanalyst=Analyst(model,assistant_cls)doubter=Doubter(model,assistant_cls)verifier=ConsistencyVerifier(model,assistant_cls)# Fetch some contentsplitter=TokenSplitter(model=model,chunk_size=3_000)pizza_dough_recipe=requests.get(pizza_dough_recipe_url).textcontext=splitter.first_extract(pizza_dough_recipe)query="Write 3 advices when making pizza dough."analyst_response=analyst.ask(query,context)question_collection=doubter.verify(query,analyst_response.content)questions=question_collection.questionsanswers=[]forquestioninquestions:response=analyst.ask(question,context=context)answers.append(response.content)forquestion,answerinzip(questions,answers):verifications=verifier.verify(question=question,context=context,answer=response.content)Here is a summary of what's been printed:> Baseline answer:

When making pizza dough, it is important to choose high-protein flour such as bread or all-purpose flour.
The dough should be mixed and kneaded for a long time to develop flavor and gluten.
It is also important to let the dough rest and rise before shaping it into pizza balls.

> Questions

1. Is bread or all-purpose flour a good choice for making pizza dough?
2. How long should the dough be mixed and kneaded for flavor development and gluten formation?
3. Should the dough be allowed to rest and rise before shaping it into pizza balls?
4. What is the purpose of mixing and kneading the dough?
5. Is there a specific step in making pizza dough that can be skipped?

> Consistency checks

1.

Bread or all-purpose flour is a good choice for making pizza dough.
The rule of thumb is to pick a flour that has high protein content.

AnswerConsistency(is_consistent=True, is_inferred_from_context=True)


2.

The dough should be mixed and kneaded for around 5 minutes.
The mixing process starts the germination of the flour, which develops the flavor of the dough.
Kneading helps to form the gluten network that gives the dough its elasticity and structure.

AnswerConsistency(is_consistent=True, is_inferred_from_context=True)

...From there, you can further process answers to remove any hallucinations or inconsistencies.Using the assistants moduleThe following example using theassistants.analystsmodule shows how to use assistants to generate a simple recommendation.fromdataclassesimportdataclassfromllm_core.assistantsimportAnalyst,Answer,LLaMACPPAssistantcontext="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy (laterexpanded into the Foundation series). Foundation is a cycle of fiveinterrelated short stories, first published as a single book by Gnome Pressin 1951. Collectively they tell the early story of the Foundation,an institute founded by psychohistorian Hari Seldon to preserve the bestof galactic civilization after the collapse of the Galactic Empire.----The user likes the movie Interstellar"""@dataclassclassRecommendation(Answer):is_recommended:boolanalyst=Analyst(model="mistral-7b-instruct-v0.1.Q4_K_M.gguf",assistant_cls=LLaMACPPAssistant,results_cls=Recommendation,)response=analyst.ask("Should we recommend Foundation ?",context=context)print(response)Recommendation(content='Foundation is a science fiction novel by Isaac Asimov that tells the early story of the Foundation, an institute founded by psychohistorian Hari Seldon to preserve the best of galactic civilization after the collapse of the Galactic Empire. The user has not mentioned any specific reasons for liking or disliking the movie Interstellar, so it is difficult to determine if they would also enjoy Foundation. However, if the user enjoys science fiction and exploring complex ideas about the future of humanity, then Foundation may be a good recommendation.',is_recommended=True)TokenizerTokenizers are registered as a codecs within the Python
codecs registry :importllm_coreimportcodecstext="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy (laterexpanded into the Foundation series). Foundation is a cycle of fiveinterrelated short stories, first published as a single book by Gnome Pressin 1951. Collectively they tell the early story of the Foundation,an institute founded by psychohistorian Hari Seldon to preserve the bestof galactic civilization after the collapse of the Galactic Empire."""# You can encode the text into tokens like that:# tokens = codecs.encode(text, 'gpt-3.5-turbo')tokens=codecs.encode(text,'mistral-7b-instruct-v0.1.Q4_K_M.gguf')print(tokens)[19137,374,264,8198,...627]print(len(tokens))100Chunking and splittingfromllm_core.splittersimportTokenSplittertext="""Foundation is a science fiction novel by American writerIsaac Asimov. It is the first published in his Foundation Trilogy (laterexpanded into the Foundation series). Foundation is a cycle of fiveinterrelated short stories, first published as a single book by Gnome Pressin 1951. Collectively they tell the early story of the Foundation,an institute founded by psychohistorian Hari Seldon to preserve the bestof galactic civilization after the collapse of the Galactic Empire."""splitter=TokenSplitter(model="mistral-7b-instruct-v0.1.Q4_K_M.gguf",chunk_size=50,chunk_overlap=0)forchunkinsplitter.chunkify(text):print(chunk)Classification and using enumsOne useful use case when interacting with LLMs is their ability to understand
what a user wants to achieve using natural language.Here's a simplified example :fromdataclassesimportdataclassfromllm_core.assistantsimportLLaMACPPAssistantfromenumimportEnumclassTargetItem(Enum):PROJECT=1TASK=2COMMENT=3MEETING=4classCRUDOperation(Enum):CREATE=1READ=2UPDATE=3DELETE=4@dataclassclassUserQuery:system_prompt="You are a helpful assistant."prompt="""Analyze the user's query and convert his intent to:- an operation (among CRUD)- a target itemQuery:{prompt}"""operation:CRUDOperationtarget:TargetItemdefask(prompt):withLLaMACPPAssistant(UserQuery,model="mistral")asassistant:user_query=assistant.process(prompt=prompt)returnuser_queryIn[2]:ask('Cancel all my meetings for the week')Out[2]:UserQuery(operation=<CRUDOperation.DELETE:4>,target=<TargetItem.MEETING:4>)In[3]:ask('What is the agenda ?')Out[3]:UserQuery(operation=<CRUDOperation.READ:2>,target=<TargetItem.MEETING:4>)In[4]:ask('Schedule meeting for next monday')Out[4]:UserQuery(operation=<CRUDOperation.CREATE:1>,target=<TargetItem.MEETING:4>)In[5]:ask('When is my next meeting ?')Out[5]:UserQuery(operation=<CRUDOperation.READ:2>,target=<TargetItem.MEETING:4>)# The classification went wrong here, so I tried a different formulationIn[6]:ask('Todo: read the final report on the project LLMCore')Out[6]:UserQuery(operation=<CRUDOperation.READ:2>,target=<TargetItem.TASK:2>)# Still no joyIn[7]:ask('Task: read the final report on the project LLMCore')Out[7]:UserQuery(operation=<CRUDOperation.READ:2>,target=<TargetItem.PROJECT:1>)# Being just a little more specific and voilà !In[8]:ask('Add to my todo: read the final report on the project LLMCore')Out[8]:UserQuery(operation=<CRUDOperation.CREATE:1>,target=<TargetItem.TASK:2>)Synthetic dataset generation examplefromtypingimportListfromenumimportEnumfromdataclassesimportdataclassfromllm_core.assistantsimportLLaMACPPAssistantclassItem(Enum):CALENDAR=1EVENT=2TASK=3REMINDER=4INVITEE=5classCRUDOperation(Enum):CREATE=1READ=2UPDATE=3DELETE=4@dataclassclassUserQueryGenerator:system_prompt="You are a helpful assistant."prompt="""# GoalsWe are developing a new business calendar software that is able to understand plain english.# ExamplesCancel all my meetings of the weekWhat is my next meeting ?What is on the agenda for the meeting at 1 pm ?{queries}# TodoWrite{queries_count}new examples of what a user could have asked."""user_queries:List[str]@classmethoddefgenerate(cls,queries_count=10,existing_queries=()):withLLaMACPPAssistant(cls,model="mistral")asassistant:existing_queries_str='\n'.join(existing_queries)batch=assistant.process(queries_count=queries_count,queries=existing_queries_str)returnbatch.user_queries@dataclassclassUserQueryClassification:system_prompt="You are a helpful assistant."prompt="""Analyze the user's query and convert his intent to:- an operation (among CRUD)- a target itemQuery:{prompt}"""operation:CRUDOperationitem:Item@classmethoddefask(cls,prompt):withLLaMACPPAssistant(cls,model="mistral")asassistant:user_query=assistant.process(prompt=prompt)returnuser_queryArgument analysis using Toulmin's methodSee the code inexamples/toulmin-model-argument-analysis.pypython3examples/toulmin-model-argument-analysis.py**Claim**: All forms of CMC should be studied in order to fully understand how online communication effects relationships**Grounds**: Numerous studies have been conducted on various facets of Internet relationships, focusing on the levels of intimacy, closeness, different communication modalities, and the frequency of use of computer-mediated communication (CMC). However, contradictory results are suggested within this research mostly because only certain aspects of CMC are investigated.**Warrant**: CMC is defined and used as ‘email’ in creating feelings of closeness or intimacy. Other articles define CMC differently and, therefore, offer different results.**Qualifier**: The strength of the relationship was predicted best by FtF and phone communication, as participants rated email as an inferior means of maintaining personal relationships as compared to FtF and phone contacts.**Rebuttal**: Other studies define CMC differently and, therefore, offer different results.**Backing**: Cummings et al.'s (2002) research in relation to three other research articlesLLaVA - Multi modalities - Mistral VisionWe can use a quantized version of the BakLLaVA model fromSkunkworksAIto process images.DownloadBakLLaVA-1-Q4_K_M.ggufandBakLLaVA-1-clip-model.gguffiles fromhttps://huggingface.co/advanced-stack/bakllava-mistral-v1-gguf/tree/mainTo run inference:fromllm_core.llmimportLLaVACPPModelmodel="BakLLaVA-1-Q4_K_M.gguf"llm=LLaVACPPModel(name=model,llama_cpp_kwargs={"logits_all":True,"n_ctx":8000,"verbose":False,"n_gpu_layers":100,# Set to 0 if you don't have a GPU"n_threads":1,"clip_model_path":"BakLLaVA-1-clip-model.gguf"})llm.load_model()history=[{'role':'user','content':[{'type':'image_url','image_url':'http://localhost:8000/adv.png'}]}]llm.ask('Describe the image as accurately as possible',history=history)ChatCompletion(id='chatcmpl-c1e49a42-fe96-49ba-a47b-991479f7d672',object='chat.completion',created=1699476989,model='/Users/pas/.cache/py-llm-core/models/BakLLaVA-1-Q4_K_M.gguf',choices=[ChatCompletionChoice(index=0,message=Message(role='assistant',content='''The image features a brown background with large,bold text that reads"Understand, Learn, Build and Deploy LLM Projects."This text is yellow.The words "Leverage AI Power Without Disclosing Your Data"are also written on the background, adding more information to the scene.'''),finish_reason='stop')],usage=Usage(prompt_tokens=630,completion_tokens=62,total_tokens=692),system_fingerprint=None)