Package: typegpt
TypeGPT - Making GPT Safe for ProductionIt is inherently difficult to produce outputs from LLMs in a consistent structure. TypeGPT simplifies this process to be as easy as defining a class in Python.Powering our own projects, such asBoostSEOInstallationpipinstalltypegptUsageDefine your prompt and desired output schema as a subclass in Python:fromtypegptimportBaseLLMResponse,PromptTemplateclassExamplePrompt(PromptTemplate):def__init__(self,sentence:str):self.sentence=sentencedefsystem_prompt(self)->str:return"Given a sentence, extract sentence parts."defuser_prompt(self)->str:returnself.sentenceclassOutput(BaseLLMResponse):num_sentences:intadjectives:list[str]nouns:list[str]verbs:list[str]If you are using OpenAI as your LLM provider, simply replace the OpenAI client class name with the subclassTypeOpenAI(for async useAsyncTypeOpenAI, or for Azure useTypeAzureOpenAI/AsyncTypeAzureOpenAI) to make it safe. You can still use it as you would have before, but you can now also call thegenerate_outputfunction for chat completions like this to generate the output object:fromtypegpt.openaiimportTypeOpenAIprompt=ExamplePrompt("The young athlete demonstrated exceptional skill and agility on the field.")client=TypeOpenAI(api_key="<your api key>")# subclass of `OpenAI`output=client.chat.completions.generate_output(model="gpt-3.5-turbo",prompt=prompt,max_output_tokens=1000)And you get a nice output like this:Output(num_sentences=1,adjectives=['young','exceptional'],nouns=['athlete','skill','agility','field'],verbs=['demonstrated'])Output TypesYour output type can contain string, integer, float, boolean, or lists of these. It is also possible to mark elements as optional. Default values can be provided as well.Example 1classOutput(BaseLLMResponse):title:str="My Recipe"description:str|Nonenum_ingredients:intingredients:list[int]estimated_time:floatis_oven_required:boolHere, the parser will parsedescriptionif the LLM returns it, but won't require it. It isNoneby default. The same holds fortitle, as it has a default value.Example 2You can also define more restrictions or give the LLM more information for some elements:classOutput(BaseLLMResponse):title:str=LLMOutput(instruction="The title for the recipe.")description:str|None=LLMOutput(instruction="An optional description for the recipe.")num_ingredients:intingredients:list[int]=LLMArrayOutput(expected_count=(1,5),instruction=lambdapos:f"The id of the{pos.ordinal}ingredient")# between 1 and 5 ingredients expected (and required at parse time)estimated_time:float=LLMOutput(instruction="The estimated time to cook")is_oven_required:boolExample 3By default, the library always expects only one line response per element. You can override this by settingmultiline=TrueinLLMOutput:classOutput(BaseLLMResponse):description:str=LLMOutput(instruction="A description for the recipe.",multiline=True)items:list[str]=LLMArrayOutput(expected_count=5,instruction=lambdapos:f"The{pos.ordinal}item in the list",multiline=True)Example 4You can nest response types. Note that you need to useBaseLLMArrayElementfor classes that you want to nest inside a list. To add instructions inside an element ofBaseLLMArrayElement, you must useLLMArrayElementOutputinstead ofLLMOutput.classOutput(BaseLLMResponse):classItem(BaseLLMArrayElement):classDescription(BaseLLMResponse):short:str|Nonelong:strtitle:strdescription:Descriptionprice:float=LLMArrayElementOutput(instruction=lambdapos:f"The price of the{pos.ordinal}item")items:list[Item]count:intAdvanced UsageAutomatic Prompt ReductionYou might have a prompt that uses an unpredictably large number of tokens due to potentially large dependencies. To ensure your prompt always fits within the LLM's token limit, you can implement the functionreduce_if_possibleinside your prompt class:classSummaryPrompt(PromptTemplate):def__init__(self,article:str):self.article=articledefsystem_prompt(self)->str:return"Summarize the given news article"defuser_prompt(self)->str:returnf"ARTICLE:{self.article}"defreduce_if_possible(self)->bool:iflen(self.article)>100:# remove last 100 characters at a timeself.article=self.article[:-100]returnTruereturnFalseclassOutput(BaseLLMResponse):summary:strInside thereduce_if_possiblefunction, you should reduce the size of your prompt in small steps and returnTrueif successfully reduced. The function is called repeatedly until the prompt fits. When calling the OpenAIgenerate_outputfunction, this automatically ensures the prompt is suitable for the given models. Additionally, you can specify a custom input token limit with the same effect to save costs:client.chat.completions.generate_output(..., max_input_tokens=2000).Automatic RetryingIn some cases, GPT might still return an output that does not follow the schema correctly. When this occurs, the OpenAI client throws anLLMParseException. To automatically retry when the output does not meet your schema, you can setretry_on_parse_errorto the number of retries you want to allow:out=client.chat.completions.generate_output("gpt-3.5-turbo",prompt=prompt,...,retry_on_parse_error=3)Now, the library will attempt to call GPT three times before throwing an error. However, ensure you only use this when the temperature is not zero.Full Static Type Safetyprompt=ExamplePrompt(...)output=client.chat.completions.generate_output(model="gpt-4",prompt=prompt,...)Due to Python's limited type system, the output type is of typeBaseLLMResponseinstead of the explicit subclassExamplePrompt.Output. To achieve full type safety in your code, simply add the parameteroutput_type=ExamplePrompt.Output:prompt=ExamplePrompt(...)output=client.chat.completions.generate_output(model="gpt-4",prompt=prompt,output_type=ExamplePrompt.Output,...)This parameter is not merely a type decorator. It can also be used to overwrite the actual output type that GPT attempts to predict.Azuree sure to use theAzureChatModelas the model when generating the output, which consists of the deployment_id and the corresponding base model (this is used for automatically reducing prompts if needed).fromtypegpt.openaiimportAzureChatModel,TypeAzureOpenAIclient=TypeAzureOpenAI(azure_endpoint="<your azure endpoint>",api_key="<your api key>",api_version="2023-05-15",)out=client.chat.completions.generate_output(model=AzureChatModel(deployment_id="gpt-35-turbo",base_model="gpt-3.5-turbo"),prompt=prompt,max_output_tokens=1000)Non-OpenAI LLM supportAny LLM that has a notion of system and user prompts can use this library. Generate the system and user messages (including the schema prompt) like this:messages=prompt.generate_messages(token_limit=max_prompt_length,token_counter=lambdamessages:num_tokens_from_messages(messages))wheremax_prompt_lengthis the maximum number of tokens the prompt is allowed to use, andnum_tokens_from_messagesneeds to be a function that counts the predicted token usage for a given list of messages. Return0here if you do not want to automatically reduce the size of a prompt.Use the generated messages to call your LLM. Parse the completion string you receive back into the desired output class like this:out=ExamplePrompt.Output.parse_response(completion)How it worksThis library automatically generates an LLM-compatible schema from your defined output class and adds instructions to the end of the system prompt to adhere to this schema.
For example, for the following abstract prompt:classDemoPrompt(PromptTemplate):defsystem_prompt(self)->str:return"This is a system prompt"defuser_prompt(self)->str:return"This is a user prompt"classOutput(BaseLLMResponse):title:strdescription:str=LLMOutput("Custom instruction")mice:list[str]The following system prompt will be generated:This is a system prompt

Always return the answer in the following format:
"""
TITLE: <Put the title here>
DESCRIPTION: <Custom instruction>
MOUSE 1: <Put the first mouse here>
MOUSE 2: <Put the second mouse here>
...
"""Notice how the plural "mice" is automatically converted to the singular "mouse" to avoid confusing the language model.