Let’s create a simple environment in a Flask application. This environment will have various functions, such as sleeping, writing poems, using a computer with real-time data retrieval from Wikipedia and Wikipedia Commons, and note-taking and musing.

We’ll give an LLM the ability to call these functions and build a world model of the current situation, time, and recent actions. This model will be based on language information and must be effective enough to navigate and manipulate its environment without overwhelming it.

The LLM will control a character in this environment over time. It will have a memory of recent interactions as direct context. Every time the LLM sleeps, we’ll generate a summary of the last day and append it to its memories. At the end of a week, we’ll generate a weekly summary and remove the previous days.

The LLM must be aware of the current time and the total time it’s spent there.

When interacting with something, the LLM will first suggest thought actions to dedicate to the received information. Then, it will generate generations of thoughts that build on that data. When the limit is reached, we’ll summarize these thoughts and append them to the daily memory.

While idle, the AI can choose to remain idle or move to any area of the house. It should know where things are but can only access them if it’s in the correct room.

The note-taking and musing areas will have memory functions. If the AI interacts with them, it will generate random ideas, compress or create knowledge based on its memories, and make these ideas forever accessible. It can select to read previous notes or make new ones. 
This is how you make a simple call for OpenAI LLM:



from openai import OpenAI
client = OpenAI()
completion = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ]
)
print(completion.choices[0].message)
----
Here is the documentation for function calling and structured outputs 
use pydantic for structured outputs, please use both on the project.


the llm should have complete autonomy on its actions and decisions, make an html that accompanies it so we can see it move around and interact with the house




Function calling connects language models to external tools and systems. It empowers AI assistants with capabilities and builds deep integrations between applications and models.

In August 2024, Structured Outputs was launched. When enabled, it ensures that the arguments generated by the model match the JSON Schema provided in the function definition.

Function calling is useful for various use cases, such as:

- Enabling assistants to fetch data, take actions, or perform computations.
- Building rich workflows, like data extraction pipelines.
- Modifying applications’ UI based on user input.

When using the OpenAI API with function calling, the model generates parameters that can be used to call your function, allowing your code to handle the function’s execution. Your application remains in full control.

Function calling is supported in the Chat Completions API, Assistants API, and Batch API. This guide focuses on function calling using the Chat Completions API. For a conversational assistant to help users with delivery orders, we want it to look up orders and reply with real data.

Step 1: Choose a function in your codebase for the model to call.
Select a function in your codebase that the model can generate arguments for.

For this example, let’s imagine you want to enable the model to call the get_delivery_date function, which accepts an order_id and queries the database for the delivery date of a given package. Your function might look like this:

python

# This is the function that we want the model to be able to call
def get_delivery_date(order_id: str) -> datetime:
    # Connect to the database
    conn = sqlite3.connect('ecommerce.db')
    cursor = conn.cursor()
    # ...
Step 2: Describe your function to the model.

We’ve decided what function we want the model to call. Now, we’ll create a “function definition” that describes the function to the model. This definition includes what the function does, when it should be called, and the required parameters.

The parameters section should be described using JSON Schema. The model will use this information to generate arguments according to the schema when it makes a function call.

Here’s an example:

{
    "name": "get_delivery_date",
    "description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'",
    "parameters": {
        "type": "object",
        "properties": {
            "order_id": {
                "type": "string",
                "description": "The customer's order ID."
            }
        },
        "required": ["order_id"],
        "additionalProperties": false
    }
}
Step 3: Provide function definitions and messages as “tools” to the model.

When calling the Chat Completions API, we’ll provide an array of “messages,” which could contain your prompt or a conversation between the user and the assistant.

This example shows how to call the API with relevant functions and messages for an assistant that handles customer inquiries for a store.

python

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_delivery_date",
            "description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID.",
                    },
                },
                "required": ["order_id"],
                "additionalProperties": False,
            },
        }
    }
]

messages = [
{"role": "system", "content": "You are a helpful customer support assistant. Use the supplied tools to assist the user."},
{"role": "user", "content": "Hi, can you tell me the delivery date for my order?"}
]

response = openai.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
)
Step 4: Receive and handle the model response
If the model decides that no function should be called
If the model does not generate a function call, then the response will contain a direct reply to the user in the normal way that Chat Completions does.

For example, in this case chat_response.choices[0].message may contain:

python

chat.completionsMessage(content='Hi there! I can help with that. Can you please provide your order ID?', role='assistant', function_call=None, tool_calls=None)
In an assistant use case you will typically want to show this response to the user and let them respond to it, in which case you will call the API again (with both the latest responses from the assistant and user appended to the messages).

Let's assume our user responded with their order id, and we sent the following request to the API.

python

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_delivery_date",
            "description": "Get the delivery date for a customer's order. Call this whenever you need to know the delivery date, for example when a customer asks 'Where is my package'",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "The customer's order ID."
                    }
                },
                "required": ["order_id"],
                "additionalProperties": False
            }
        }
    }
]

messages = []
messages.append({"role": "system", "content": "You are a helpful customer support assistant. Use the supplied tools to assist the user."})
messages.append({"role": "user", "content": "Hi, can you tell me the delivery date for my order?"})
messages.append({"role": "assistant", "content": "Hi there! I can help with that. Can you please provide your order ID?"})
messages.append({"role": "user", "content": "i think it is order_12345"})

response = client.chat.completions.create(
model='gpt-4o',
messages=messages,
tools=tools
)
If the model generated a function call
If the model generated a function call, it will generate the arguments for the call (based on the parameters definition you provided).

Here is an example response showing this:

python

Choice(
    finish_reason='tool_calls', 
    index=0, 
    logprobs=None, 
    message=chat.completionsMessage(
        content=None, 
        role='assistant', 
        function_call=None, 
        tool_calls=[
            chat.completionsMessageToolCall(
                id='call_62136354', 
                function=Function(
                    arguments='{"order_id":"order_12345"}', 
                    name='get_delivery_date'), 
                type='function')
        ])
)
Handling the model response indicating that a function should be called
Assuming the response indicates that a function should be called, your code will now handle this:

python

# Extract the arguments for get_delivery_date
# Note this code assumes we have already determined that the model generated a function call. See below for a more production ready example that shows how to check if the model generated a function call
tool_call = response.choices[0].message.tool_calls[0]
arguments = json.loads(tool_call['function']['arguments'])

order_id = arguments.get('order_id')

# Call the get_delivery_date function with the extracted order_id

delivery_date = get_delivery_date(order_id)
Step 5: Provide the function call result back to the model
Now we have executed the function call locally, we need to provide the result of this function call back to the Chat Completions API so the model can generate the actual response that the user should see:

python

# Simulate the order_id and delivery_date
order_id = "order_12345"
delivery_date = datetime.now()

# Simulate the tool call response

response = {
"choices": [
{
"message": {
"role": "assistant",
"tool_calls": [
{
"id": "call_62136354",
"type": "function",
"function": {
"arguments": "{'order_id': 'order_12345'}",
"name": "get_delivery_date"
}
}
]
}
}
]
}

# Create a message containing the result of the function call

function_call_result_message = {
"role": "tool",
"content": json.dumps({
"order_id": order_id,
"delivery_date": delivery_date.strftime('%Y-%m-%d %H:%M:%S')
}),
"tool_call_id": response['choices'][0]['message']['tool_calls'][0]['id']
}

# Prepare the chat completion call payload

completion_payload = {
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "You are a helpful customer support assistant. Use the supplied tools to assist the user."},
{"role": "user", "content": "Hi, can you tell me the delivery date for my order?"},
{"role": "assistant", "content": "Hi there! I can help with that. Can you please provide your order ID?"},
{"role": "user", "content": "i think it is order_12345"},
response['choices'][0]['message'],
function_call_result_message
]
}

# Call the OpenAI API's chat completions endpoint to send the tool call result back to the model

response = openai.chat.completions.create(
model=completion_payload["model"],
messages=completion_payload["messages"]
)

# Print the response from the API. In this case it will typically contain a message such as "The delivery date for your order #12345 is xyz. Is there anything else I can help you with?"

print(response)
That’s all you need to give gpt-4o access to your functions.

Handling edge cases

We recommend using the SDK to handle the edge cases described below. If for any reason you cannot use the SDK, you should handle these cases in your code.
When you receive a response from the API, if you're not using the SDK, there are a number of edge cases that production code should handle.

In general, the API will return a valid function call, but there are some edge cases when this won’t happen, such as when you have specified max_tokens and the model’s response is cut off as a result.

This sample explains them:

python


    
# Check if the model has made a tool_call. This is the case either if the "finish_reason" is "tool_calls" or if the "finish_reason" is "stop" and our API request had forced a function call
if (response['choices'][0]['message']['finish_reason'] == "tool_calls" or 
    # This handles the edge case where if we forced the model to call one of our functions, the finish_reason will actually be "stop" instead of "tool_calls"
    (our_api_request_forced_a_tool_call and response['choices'][0]['message']['finish_reason'] == "stop")):
    # Handle tool call
    print("Model made a tool call.")
    # Your code to handle tool calls
    handle_tool_call(response)
    
# Else finish_reason is "stop", in which case the model was just responding directly to the user
elif response['choices'][0]['message']['finish_reason'] == "stop":
    # Handle the normal stop case
    print("Model responded directly to the user.")
    # Your code to handle normal responses
    handle_normal_response(response)
    
# Catch any other case, this is unexpected
else:
    print("Unexpected finish_reason:", response['choices'][0]['message']['finish_reason'])
    # Handle unexpected cases as needed
    handle_unexpected_case(response)


Function calling with Structured Outputs
By default, when you use function calling, the API will offer best-effort matching for your parameters, which means that occasionally the model may miss parameters or get their types wrong when using complicated schemas.

Structured Outputs is a feature that ensures model outputs for function calls will exactly match your supplied schema.

Structured Outputs for function calling can be enabled with a single parameter, just by supplying strict: true.

python

from enum import Enum
from typing import Union
from pydantic import BaseModel
import openai
from openai import OpenAI

client = OpenAI()

class GetDeliveryDate(BaseModel):
order_id: str

tools = [openai.pydantic_function_tool(GetDeliveryDate)]

messages = []
messages.append({"role": "system", "content": "You are a helpful customer support assistant. Use the supplied tools to assist the user."})
messages.append({"role": "user", "content": "Hi, can you tell me the delivery date for my order #12345?"})

response = client.chat.completions.create(
model='gpt-4o-2024-08-06',
messages=messages,
tools=tools
)

print(response.choices[0].message.tool_calls[0].function)

When enabling Structured Outputs with strict: true, the OpenAI API pre-processes your schema and constrains the model to it.

The model always follows your schema, except in cases like:

- When the response is truncated due to max_tokens, stop tokens, or maximum context length
- When the model refuses
- When there’s a content_filter finish reason

Note that the first request with a new schema incurs additional latency due to schema processing, but subsequent requests are free.

Supported schemas are a subset of JSON Schema. For more details, see the Structured Outputs guide.

Function calling supports advanced features like forcing function calls and parallel function calling.

Configuring parallel function calling:

- Models released after Nov 6, 2023, may generate multiple function calls in a single response, indicating parallel execution.

This is useful for long-running functions, like getting weather data from multiple locations simultaneously. The tool_calls array will contain 3 function calls in such cases.

Example response:

python

response = Choice(
    finish_reason='tool_calls', 
    index=0, 
    logprobs=None, 
    message=chat.completionsMessage(
        content=None, 
        role='assistant', 
        function_call=None, 
        tool_calls=[
            chat.completionsMessageToolCall(
                id='call_62136355', 
                function=Function(
                    arguments='{"city":"New York"}', 
                    name='check_weather'), 
                type='function'),
            chat.completionsMessageToolCall(
                id='call_62136356', 
                function=Function(
                    arguments='{"city":"London"}', 
                    name='check_weather'), 
                type='function'),
            chat.completionsMessageToolCall(
                id='call_62136357', 
                function=Function(
                    arguments='{"city":"Tokyo"}', 
                    name='check_weather'), 
                type='function')
        ])
)

# Iterate through tool calls to handle each weather check

for tool_call in response.message.tool_calls:
arguments = json.loads(tool_call.function.arguments)
city = arguments['city']
weather_info = check_weather(city)
print(f"Weather in {city}: {weather_info}")
Each function call in the array has a unique id.

Once you've executed these function calls in your application, you can provide the result back to the model by adding one new message to the conversation for each function call, each containing the result of one function call, with a tool_call_id referencing the id from tool_calls, for example:

python

# Assume we have fetched the weather data from somewhere
weather_data = {
    "New York": {"temperature": "22°C", "condition": "Sunny"},
    "London": {"temperature": "15°C", "condition": "Cloudy"},
    "Tokyo": {"temperature": "25°C", "condition": "Rainy"}
}
    
# Prepare the chat completion call payload with inline function call result creation
completion_payload = {
    "model": "gpt-4o",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant providing weather updates."},
        {"role": "user", "content": "Can you tell me the weather in New York, London, and Tokyo?"},
        # Append the original function calls to the conversation
        response['message'],
        # Include the result of the function calls
        {
            "role": "tool",
            "content": json.dumps({
                "city": "New York",
                "weather": weather_data["New York"]
            }),
            # Here we specify the tool_call_id that this result corresponds to
            "tool_call_id": response['message']['tool_calls'][0]['id']
        },
        {
            "role": "tool",
            "content": json.dumps({
                "city": "London",
                "weather": weather_data["London"]
            }),
            "tool_call_id": response['message']['tool_calls'][1]['id']
        },
        {
            "role": "tool",
            "content": json.dumps({
                "city": "Tokyo",
                "weather": weather_data["Tokyo"]
            }),
            "tool_call_id": response['message']['tool_calls'][2]['id']
        }
    ]
}
    
# Call the OpenAI API's chat completions endpoint to send the tool call result back to the model
response = openai.chat.completions.create(
    model=completion_payload["model"],
    messages=completion_payload["messages"]
)
    
# Print the response from the API, which will return something like "In New York the weather is..."
print(response)
You can also disable parallel function calling by setting parallel_tool_calls: false.



Parallel function calling and Structured Outputs
When the model outputs multiple function calls via parallel function calling, model outputs may not match strict schemas supplied in tools.

In order to ensure strict schema adherence, disable parallel function calls by supplying parallel_tool_calls: false. With this setting, the model will generate one function call at a time.

Configuring function calling behavior using the tool_choice parameter
By default, the model is configured to automatically select which functions to call, as determined by the tool_choice: "auto" setting.

We offer three ways to customize the default behavior:

To force the model to always call one or more functions, you can set tool_choice: "required". The model will then always select one or more function(s) to call. This is useful for example if you want the model to pick between multiple actions to perform next.
To force the model to call a specific function, you can set tool_choice: {"type": "function", "function": {"name": "my_function"}}.
To disable function calling and force the model to only generate a user-facing message, you can either provide no tools, or set tool_choice: "none".
Note that if you do either 1 or 2 (i.e. force the model to call a function) then the subsequent finish_reason will be "stop" instead of being "tool_calls".

python

from openai import OpenAI

client = OpenAI()

tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["c", "f"]},
},
"required": ["location", "unit"],
"additionalProperties": False,
},
},
},
{
"type": "function",
"function": {
"name": "get_stock_price",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"symbol": {"type": "string"},
},
"required": ["symbol"],
"additionalProperties": False,
},
},
},
]

messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]
completion = client.chat.completions.create(
model="gpt-4o",
messages=messages,
)

print(completion)
Understanding token usage
Under the hood, functions are injected into the system message in a syntax the model has been trained on. This means functions count against the model's context limit and are billed as input tokens. If you run into token limits, we suggest limiting the number of functions or the length of the descriptions you provide for function parameters.

It is also possible to use fine-tuning to reduce the number of tokens used if you have many functions defined in your tools specification.

Tips and best practices
Turn on Structured Outputs by setting strict: "true"
When Structured Outputs is turned on, the arguments generated by the model for function calls will reliably match the JSON Schema that you provide.

If you’re not using Structured Outputs, validate arguments with a library like Pydantic.

Name functions intuitively with detailed descriptions. If the model doesn’t call the correct functions, update function names and descriptions. Avoid abbreviations and acronyms.

Describe when a tool should be called. For complex functions, describe each argument.

Name function parameters intuitively with detailed descriptions. Specify expected formats for parameters.

Provide clear instructions in your system message to guide the model. For example, “Use check_order_status when the user asks about an order’s status.”

Use enums for function arguments when possible. This can reduce hallucinations.

For example, if your AI assistant helps with ordering a T-shirt, provide fixed sizes in an enum like “s”, “m”, “l” for small, medium, and large.

Example:

{
    "name": "pick_tshirt_size",
    "description": "Call this if the user specifies which size t-shirt they want",
    "parameters": {
        "type": "object",
        "properties": {
            "size": {
                "type": "string",
                "enum": ["s", "m", "l"],
                "description": "The size of the t-shirt that the user would like to order"
            }
        },
        "required": ["size"],
        "additionalProperties": false
    }
}
If you don’t constrain the output, a user may say “large” or “L”, and the model may return either value. Your code may expect a specific structure, so it’s important to limit the number of possible formats the model can choose from.

Keep the number of functions low for higher accuracy
We recommend that you use no more than 20 functions in a single tool call. Developers typically see a reduction in the model’s ability to select the correct tool once they have between 10-20 tools.

If your use case requires the model to be able to pick between a large number of functions, you may want to break out the tools and group them logically to create a multi-agent system.