Model issue about code generation
Looks like the model can only handel some basic task, is there anyway to improve the performance?
This is the function I have given to the model:
{
"function": "write_file",
"description": "Write a file, creating it if necessary. If the file exists, it is overwritten.",
"arguments": [
{
"name": "filename",
"type": "string",
"description": "name of the file"
},
{
"name": "contents",
"type": "string",
"description": "contents of the file"
}
]
}
For some simple task, it works fine
Prompt:
<FUNCTIONS>{"function": "write_file", "description": "Write a file, creating it if necessary. If the file exists, it is overwritten.", "arguments": [{"name": "filename", "type": "string", "description": "name of the file"}, {"name": "contents", "type": "string", "description": "contents of the file"}]}</FUNCTIONS>
Instruction:
write hello world to out.txt
Response:
Model return:
{
"function": "write_file",
"arguments": {
"filename": "out.txt",
"contents": "Hello World!"
}
}
However, when I tried some complicated question, especially coding question, it didn't return properly
Prompt:
<FUNCTIONS>{"function": "write_file", "description": "Write a file, creating it if necessary. If the file exists, it is overwritten.", "arguments": [{"name": "filename", "type": "string", "description": "name of the file"}, {"name": "contents", "type": "string", "description": "contents of the file"}]}</FUNCTIONS>
Instruction:
Write a python program to out.py to print hello world
Response:
Model return:
The following Python code can be used to write a simple "Hello World!" script in a file named out.py:
with open('out.py', 'w') as f:
f.write("print('Hello World!')\n")
Hi Grant, thanks for having provided a clear example.
I've read through and the responses all seem reasonable.
What kind of response were you expecting to the second question?
Hi Ronan, for the second question, I was expecting the model to return with function format, something like
{
"function": "write_file",
"arguments": {
"filename": "out.py",
"contents": "print('Hello World!')\n"
}
}
Is there anyway to achieve this?
A few thoughts:
"Write a file, creating it if necessary. If the file exists, it is overwritten."
This function description is unclear because a language model cannot "create" a file, and it's confusing what the difference is between "write" and "create". Probably it's best to call it "save_file" and put the description as "Function to call if asked to save a file.". If you further want the language model to know whether files can be overwritten or not, then you need to provide a list of files to the LLM and further explain that in the description."Write a python program to out.py to print hello world"
There isn't a clear way for the LLM to know whether to just write a response (i.e. respond) with the python code OR use the write_file function. Again, using save_file as the function name will probably help because then you could say something like "Write a python program that prints -hello world- and save that program as out.py.".
BTW, chatgpt is very good at helping to refine your function defintions and prompts. You coud paste everything you wrote into chatgpt, tell it you are feeding into an LLM that is fine-tuned for function calling, and ask how to improve the function description and prompt.
Hi Ronan,
Just for comparison, I tried the same prompt provided before with gpt-3.5-turbo-0613 model using tool calls, this is what I get:
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_xz8NjzhqKiZA8UJkDR2KQvpd",
"type": "function",
"function": {
"name": "write_file",
"arguments": "{\n "filename": "out.py",\n "contents": "print('Hello, world!')"\n}"
}
}
]
},
"logprobs": null,
"finish_reason": "tool_calls"
}
]
I should have said this before, but the v3 version of this model is now recommended. I have updated the model card (which I should have before).
I have also done some testing on the v3 model with your functions and prompt. It works well.
Here is a formatted version of the conversation I had:
user: Write a python program to out.py to print hello world
function_call: {
"name": "write_file",
"arguments": {
"filename": "out.py",
"contents": "print('Hello, world')"
}
}
function_response: File 'out.py' written successfully.
And here is the raw prompt I fed in (with functions). Note that this format is for the v3 model (not the v2 model here):
<｜begin▁of▁sentence｜>You have access to the following functions. Use them if required:
[
{
"type": "function",
"function": {
"name": "write_file",
"description": "Write a file, creating it if necessary. If the file exists, it is overwritten.",
"parameters": {
"type": "object",
"properties": {
"filename": {
"type": "string",
"description": "name of the file"
},
"contents": {
"type": "string",
"description": "contents of the file"
}
},
"required": [
"filename",
"contents"
]
}
}
}
]
### Instruction:
Write a python program to out.py to print hello world
### Response:
add a note on chat_template