Trelis

/

Llama-2-7b-chat-hf-function-calling

like 47

This model will be improved as the underlying fine-tuning dataset improves and expands.

The 7B model already does well for simple functions, whereas the 13B model can capture more complexity (i.e. more data structures as function arguments).

Using the "test" split of the Function Calling Extended dataset, I'm scoring the performance as":

• 7B = 5 out of 7 (see below for an example I consider a failure).
• 13B = 6 out of 7.

Note that the "test" split was not used for training.

Sample fLlama-13B prompt (7B does well on this too)

Prompt:
[INST] <<SYS>>
You are a helpful research assistant. The following functions are available for you to fetch further data to answer user questions, if relevant:

{
    "function": "save_chat",
    "description": "This function saves content from current chat to a user's files. Use this function if the user asks to save this chat or content from this chat. If the user doesn't specify a name or description, use the content of the chat to determine a suitable chat name and description.",
    "arguments": [
        {
            "name": "fileName",
            "type": "string",
            "description": "A suitable name for the file where the chat history will be saved."
        },
        {
            "name": "fileDescription",
            "type": "string",
            "description": "A brief description of the chat history"
        },
        {
            "name": "fileContent",
            "type": "string",
            "description": "Content to save. Format the content in markdown. Do not include system messages"
        }
    ]
}

{
    "function": "delete_file",
    "description": "This function provides the ability to delete files in a user's folder. It doesn't delete 'files.json'. It updates 'files.json' to remove entries of deleted files.",
    "arguments": [
        {
            "name": "fileNames",
            "type": "array",
            "items": {
                "type": "string"
            },
            "description": "An array of file names to be deleted"
        }
    ]
}
                

To call a function, respond - immediately and only - with a JSON object of the following format:
{
    "function": "function_name",
    "arguments": {
        "argument1": value1,
        "argument2": value2
    }
}
<</SYS>>

There are one and two string kites. The two string ones are easier to control, although you can get the cords tangled. The one-stringed ones are sometimes used for kite fights, and you lose the kite and have to run after it if the string breaks. 

 Save this discussion of kites as KiteDiscussion. [/INST]

Generated Assistant Response:
 {
    "function": "save_chat",
    "arguments": {
        "fileName": "KiteDiscussion",
        "fileDescription": "A discussion about kites and their different types",
        "fileContent": "--- **KiteDiscussion** Kites are a popular toy for children and adults alike. There are many different types of kites, including one-stringed and two-stringed kites. One-stringed kites are easier to control, but two-stringed kites are more stable and easier to fly. Two-stringed kites are also easier to control, but they can be more difficult to maneuver. Kite fights are a popular activity, but you risk losing your kite if the string breaks. ---"
    }
}</s>

Correct Assistant Response:
{
    "function": "save_chat",
    "arguments": {
        "fileName": "KiteDiscussion",
        "fileDescription": "Notes on one and two stringed kites",
        "fileContent": "--- **Types of Kite** There are one and two string kites. The two string ones are easier to control, although you can get the cords tangled. The one-stringed ones are sometimes used for kite fights, and you lose the kite and have to run after it if the string breaks. ---"
    }
}

fLlama-7B performance on a tricky prompt

Prompt:
[INST] <<SYS>>
You are a helpful research assistant. The following functions are available for you to fetch further data to answer user questions, if relevant:

{
    "function": "delete_file",
    "description": "This function provides the ability to delete files in a user's folder. It doesn't delete 'files.json'. It updates 'files.json' to remove entries of deleted files.",
    "arguments": [
        {
            "name": "fileNames",
            "type": "array",
            "items": {
                "type": "string"
            },
            "description": "An array of file names to be deleted"
        }
    ]
}

{
    "function": "save_chat",
    "description": "This function saves content from current chat to a user's files. Use this function if the user asks to save this chat or content from this chat. If the user doesn't specify a name or description, use the content of the chat to determine a suitable chat name and description.",
    "arguments": [
        {
            "name": "fileName",
            "type": "string",
            "description": "A suitable name for the file where the chat history will be saved."
        },
        {
            "name": "fileDescription",
            "type": "string",
            "description": "A brief description of the chat history"
        },
        {
            "name": "fileContent",
            "type": "string",
            "description": "Content to save. Format the content in markdown. Do not include system messages"
        }
    ]
}
                

To call a function, respond - immediately and only - with a JSON object of the following format:
{
    "function": "function_name",
    "arguments": {
        "argument1": value1,
        "argument2": value2
    }
}
<</SYS>>

Irish stew is typically made using beef, onions and potatoes. Start by frying some onions in some butter in a pan. Add in and brown some cubes of beer - very lightly. Add salt, pepper and some herbs. Then add the sliced potatoes on top. Simmer over a medium heat for 15 minutes and then leave to simmer on low heat for another hour. 

 Save this Irish stew recipe as irish_stew_recipes. [/INST]

Generated Assistant Response:
 {
    "function": "save_chat",
    "arguments": {
        "fileName": "irish_stew_recipes",
        "fileDescription": "Irish stew recipe",
        "fileContent": [
            "Beef, onions and potatoes",
            "Fry onions in butter",
            "Add cubes of beef and brown lightly",
            "Add salt, pepper and herbs",
            "Add sliced potatoes and simmer for 15 minutes",
            "Leave to simmer on low heat for another hour"
        ]
    }
}</s>

Correct Assistant Response:
{
    "function": "save_chat",
    "arguments": {
        "fileName": "irish_stew_recipes",
        "fileDescription": "Some recipes for Irish stew",
        "fileContent": "--- **Irish Stew Recipe** Irish stew is typically made using beef, onions and potatoes. Start by frying some onions in some butter in a pan. Add in and brown some cubes of beer - very lightly. Add salt, pepper and some herbs. Then add the sliced potatoes on top. Simmer over a medium heat for 15 minutes and then leave to simmer on low heat for another hour. \n\n Save this Irish stew recipe as irish_stew_recipes. ---"
    }
}

fLlama-13b performance on that same prompt above (not 100% perfect, but very good):

Prompt:
[INST] <<SYS>>
You are a helpful research assistant. The following functions are available for you to fetch further data to answer user questions, if relevant:

{
    "function": "delete_file",
    "description": "This function provides the ability to delete files in a user's folder. It doesn't delete 'files.json'. It updates 'files.json' to remove entries of deleted files.",
    "arguments": [
        {
            "name": "fileNames",
            "type": "array",
            "items": {
                "type": "string"
            },
            "description": "An array of file names to be deleted"
        }
    ]
}

{
    "function": "save_chat",
    "description": "This function saves content from current chat to a user's files. Use this function if the user asks to save this chat or content from this chat. If the user doesn't specify a name or description, use the content of the chat to determine a suitable chat name and description.",
    "arguments": [
        {
            "name": "fileName",
            "type": "string",
            "description": "A suitable name for the file where the chat history will be saved."
        },
        {
            "name": "fileDescription",
            "type": "string",
            "description": "A brief description of the chat history"
        },
        {
            "name": "fileContent",
            "type": "string",
            "description": "Content to save. Format the content in markdown. Do not include system messages"
        }
    ]
}
                

To call a function, respond - immediately and only - with a JSON object of the following format:
{
    "function": "function_name",
    "arguments": {
        "argument1": value1,
        "argument2": value2
    }
}
<</SYS>>

Irish stew is typically made using beef, onions and potatoes. Start by frying some onions in some butter in a pan. Add in and brown some cubes of beer - very lightly. Add salt, pepper and some herbs. Then add the sliced potatoes on top. Simmer over a medium heat for 15 minutes and then leave to simmer on low heat for another hour. 

 Save this Irish stew recipe as irish_stew_recipes. [/INST]

Generated Assistant Response:
{
    "function": "save_chat",
    "arguments": {
        "fileName": "irish_stew_recipes",
        "fileDescription": "Irish stew recipe",
        "fileContent": "--- Irish stew recipe Irish stew is typically made using beef, onions and potatoes. Start by frying some onions in some butter in a pan. Add in and brown some cubes of beer - very lightly. Add salt, pepper and some herbs. Then add the sliced potatoes on top. Simmer over a medium heat for 15 minutes and then leave to simmer on low heat for another hour. ---"
    }
}</s>

Correct Assistant Response:
{
    "function": "save_chat",
    "arguments": {
        "fileName": "irish_stew_recipes",
        "fileDescription": "Some recipes for Irish stew",
        "fileContent": "--- **Irish Stew Recipe** Irish stew is typically made using beef, onions and potatoes. Start by frying some onions in some butter in a pan. Add in and brown some cubes of beer - very lightly. Add salt, pepper and some herbs. Then add the sliced potatoes on top. Simmer over a medium heat for 15 minutes and then leave to simmer on low heat for another hour. \n\n Save this Irish stew recipe as irish_stew_recipes. ---"
    }
}