Text Generation
Transformers
Safetensors
Spanish
English
Chinese
mistral
openchat
C-RLFT
function-calling
function calling
GGUF
conversational
Inference Endpoints
text-generation-inference

General search function

#4
by kjhamilton - opened

I understand the model will do a great job calling specific functions like finding the weather for a given location. How do I get it to call a more general knowledge search function. For example to look up features for a product. Right now I'm trying a function called "search_knowledge" and the description is "Search your knowledge for specific information about features" - but it doesn't ever call it. Is there a clever trick in the description, function name, or parameter that I could use to trigger it to call the function with a search string whenever the user asks about the product?

Trelis org
  1. Can you share a copy of what the function metadata looks like? Hard for me to comment without seeing that

  2. Take that metadata and ask an LLM to optimise it. That should improve things..

lmk how it goes

That's a good idea, I'll chat with the LLM ;).

A more relevant question or maybe a better one when thinking of ultimately architecting the solution. Would I run a RAG implementation along with function calling? Meaning if I wanted to pre-lookup data and attach it as another message for context, or is it better to get the model to make the decision of when it needs to go off and search or get additional information?

It works exceptionally well so far with the stock and options data functions. It seems to get confused when I add an additional layer either by adding context data or having it try to lookup information to answer questions about the company that don't involve the real-time data functions.

Example when using RAG:

messages.append({
"role": "system",
"content": "In addition to the functions, you can use the following information to help "
"respond: \n\n" + kb_data[0].page_content
})

Would a 13b model help?

Trelis org

OpenChat is based on Mixtral, and Mixtral doesn't have a system message, so putting data into the system message may not work well. It may also throw off the function calling (because the model IS trained to expect function calling metadata).

I think you can use RAG, and what you can do is return the data from RAG along with the function response. If you look at the Mixtral video, you can see a bit how the function response is handled by the ADVANCED-Inference Repo.

I would probably try that before going to a bigger model. Unfortunately there is no 13B model (there just aren't great open source models in that size) so probably the next step would be to use SUSChat (the AWQ, which is <30 GB) or maybe deepseek coder 33B (that will be strong at function calling as it's a coding model).

So far I love it. I'll keep working with this one. I changed the opening message, ditched the system message putting everything into the function call message and it is working pretty well. Still doing more testing.

I find that if the response from the function is too complicated, it just spits out the function response and says: Here's the answer . I am curious if that's a training thing or if it is something where like the 30B+ model would do a better job of interpreting a long JSON response from the function. For now I'm just simplifying down my responses to a few dictionary keys with the critical information.

Trelis org

Really nice!

Yeah I suspect that with a lot of different information, a smaller model will have difficulty with nuance. The granularity of understanding is just worse, and that becomes more of an issue the more detail you have included (the model just averages across the detail).

RonanMcGovern changed discussion status to closed

Sign up or log in to comment