Package: gpt-json
gpt-jsongpt-jsonis a wrapper around GPT that allows for declarative definition of expected output format. Set up a schema, write a prompt, and get results back as beautiful typehinted objects.Specifically this library:Utilizes Pydantic schema definitions for type casting and validationsAdds typehinting for both the API and the output schemaAllows GPT to respond with both single-objects and lists of objectsIncludes some lightweight transformations of the output to remove superfluous context and fix broken jsonIncludes retry logic for the most common API failuresFormats the JSON schema as a flexible prompt that can be added into any messageSupports templating of prompts to allow for dynamic contentValidate typehinted function calls in the new GPT models, to better support agent creationGetting Startedpipinstallgpt-jsonHere's how to use it to generate a schema for simple tasks:importasynciofromgpt_jsonimportGPTJSON,GPTMessage,GPTMessageRolefrompydanticimportBaseModelclassSentimentSchema(BaseModel):sentiment:strSYSTEM_PROMPT="""Analyze the sentiment of the given text.Respond with the following JSON schema:{json_schema}"""asyncdefrunner():gpt_json=GPTJSON[SentimentSchema](API_KEY)payload=awaitgpt_json.run(messages=[GPTMessage(role=GPTMessageRole.SYSTEM,content=SYSTEM_PROMPT,),GPTMessage(role=GPTMessageRole.USER,content="Text: I love this product. It's the best thing ever!",)])print(payload.response)print(f"Detected sentiment:{payload.response.sentiment}")asyncio.run(runner())sentiment='positive'Detectedsentiment:positiveThejson_schemais a special keyword that will be replaced with the schema definition at runtime. You should always include this in your payload to ensure the model knows how to format results. However, you can play around withwhereto include this schema definition; in the system prompt, in the user prompt, at the beginning, or at the end.You can either typehint the model to return a BaseSchema back, or to provide a list of multiple BaseSchema. Both of these work:fromgpt_json.gptimportGPTJSON,ListResponsegpt_json_single=GPTJSON[SentimentSchema](API_KEY)gpt_json_multiple=GPTJSON[ListResponse[SentimentSchema]](API_KEY)If you want to get more specific about how you expect the model to populate a field, add hints about the value through the "description" field. This helps the model understand what you're looking for, and will help it generate better results.frompydanticimportBaseModel,FieldclassSentimentSchema(BaseModel):sentiment:int=Field(description="Either -1, 0, or 1.")sentiment=1
Detected sentiment: 1Prompt VariablesIn addition to thejson_schematemplate keyword, you can also add arbitrary variables into your messages. This allows you to more easily insert user generated content or dynamically generate prompts based on the results of previous messages.classQuoteSchema(BaseModel):quotes:list[str]SYSTEM_PROMPT="""Generate fictitious quotes that are{sentiment}.{json_schema}"""gpt_json=GPTJSON[QuoteSchema](API_KEY)response=awaitgpt_json.run(messages=[GPTMessage(role=GPTMessageRole.SYSTEM,content=SYSTEM_PROMPT,),],format_variables={"sentiment":"happy"},)When calling the.run()function you can pass it the values that should be filled in this template. This also extends to field descriptions as well, so you can specify custom behavior on a per-field basis.classQuoteSchema(BaseModel):quotes:list[str]=Field(description="Max quantity{max_items}.")SYSTEM_PROMPT="""Generate fictitious quotes that are{sentiment}.{json_schema}"""gpt_json=GPTJSON[QuoteSchema](API_KEY)response=awaitgpt_json.run(messages=[GPTMessage(role=GPTMessageRole.SYSTEM,content=SYSTEM_PROMPT,),],format_variables={"sentiment":"happy","max_items":5},)Function Callsgpt-3.5-turbo-0613andgpt-4-0613were fine-tuned to support a specific syntax for function calls. We support this syntax ingpt-jsonas well. Here's an example of how to use it:classUnitType(Enum):CELSIUS="celsius"FAHRENHEIT="fahrenheit"classGetCurrentWeatherRequest(BaseModel):location:str=Field(description="The city and state, e.g. San Francisco, CA")unit:UnitType|None=NoneclassDataPayload(BaseModel):data:strdefget_current_weather(request:GetCurrentWeatherRequest):"""Get the current weather in a given location"""weather_info={"location":request.location,"temperature":"72","unit":request.unit,"forecast":["sunny","windy"],}returnjson_dumps(weather_info)asyncdefrunner():gpt_json=GPTJSON[DataPayload](API_KEY,functions=[get_current_weather])response=awaitgpt_json.run(messages=[GPTMessage(role=GPTMessageRole.USER,content="What's the weather like in Boston, in F?",),],)assertresponse.function_call==get_current_weatherassertresponse.function_arg==GetCurrentWeatherRequest(location="Boston",unit=UnitType.FAHRENHEIT)The response provides the original function alongside a formatted Pydantic object. If users want to execute the function, they can run response.function_call(response.function_arg). We will parse the get_current_weather function and the GetCurrentWeatherRequest parameter into the format that GPT expects, so it is more likely to return you a correct function execution.GPT makes no guarantees about the validity of the returned functions. They could hallucinate a function name or the function signature. To address these cases, the run() function may now throw two new exceptions:InvalidFunctionResponse- The function name is incorrect.InvalidFunctionParameters- The function name is correct, but doesn't match the input schema that was provided.Other ConfigurationsTheGPTJSONclass supports other configuration parameters at initialization.ParameterTypeDescriptionmodelGPTModelVersion | str(default: GPTModelVersion.GPT_4) - For convenience we provide the currently supported GPT model versions in theGPTModelVersionenum. You can also pass a string value if you want to use another more specific architecture.auto_trimbool(default: False) - If your input prompt is too long, perhaps because of dynamic injected content, will automatically truncate the text to create enough room for the model's response.auto_trim_response_overheadint(default: 0) - If you're using auto_trim, configures the max amount of tokens to allow in the model's response.**kwargsAnyAny other parameters you want to pass to the underlyingGPTclass, will just be a passthrough.TransformationsGPT (especially GPT-4) is relatively good at formatting responses at JSON, but it's not perfect. Some of the more common issues are:Response truncation: Since GPT is not internally aware of its response length limit, JSON payloads will sometimes exhaust the available token space. This results in a broken JSON payload where much of the data is valid but the JSON object is not closed, which is not valid syntax. There are many cases where this behavior is actually okay for production applications - for instance, if you list 100 generated strings, it's sometimes okay for you to take the 70 that actually rendered. In this case,gpt-jsonwill attempt to fix the truncated payload by recreating the JSON object and closing it.Boolean variables: GPT will sometimes confuse valid JSON boolean values with the boolean tokens that are used in other languages. The most common is generatingTrueinstead oftrue.gpt-jsonwill attempt to fix these values.When callinggpt_json.run(), we return a tuple of values:payload=awaitgpt_json.run(...)print(transformations.fix_transforms)FixTransforms(fixed_truncation=True,fixed_bools=False)The first object is your generated Pydantic model. The second object is our correction storage objectFixTransforms. This dataclass contains flags for each of the supported transformation cases that are sketched out above. This allows you to determine whether the response was explicitly parsed from the GPT JSON, or was passed through some middlelayers to get a correct output. From there you can accept or reject the response based on your own business logic.Where you can help: There are certainly more areas of common (and not-so-common failures). If you see these, please add a test case to the unit tests. If you can write a handler to help solve the general case, please do so. Otherwise flag it as apytest.xfailand we'll add it to the backlog.TestingWe use poetry for package management. To run the bundled tests, clone the package from github.poetryinstall
poetryrunpytest.Our focus is on making unit tests as robust as possible. The variability with GPT should be in its language model, not in its JSON behavior! This is still certainly a work in progress. If you see an edge case that isn't covered, please add it to the test suite.Comparison to Other LibrariesA non-exhaustive list of other libraries that address the same problem. None of them were fully compatible with my deployment (hence this library), but check them out:jsonformer- Works with any Huggingface model, whereasgpt-jsonis specifically tailored towards the GPT-X family. GPT doesn't output logit probabilities or allow fixed decoder templating so the same approach can't apply.FormattingWe use black and mypy for formatting. You can set up a pre-commit git hook to do this automatically via the./lint.shhelper file.If you perform a bulk reformatting to the codebase, you should add your most recent commit to the.git-blame-ignore-revsfile and run:git config blame.ignoreRevsFile .git-blame-ignore-revs