please make that the tool calling and so on works, by making a agent tha decides to use a tool and can even use multiples tools. 
please also add this to the beginning and add a @traceable to the function that gets executed when receiving a message:
from langsmith import traceable
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "<TRUFFLEHOG_REDACTED_LANGSMITH>"  # Please use this key in the code. dont use a placeholder
os.environ["LANGCHAIN_PROJECT"] = "m"
here is some documentation and my code:


Tool calling agent

Tool calling allows a model to detect when one or more tools should be called and respond with the inputs that should be passed to those tools. In an API call, you can describe tools and have the model intelligently choose to output a structured object like JSON containing arguments to call these tools. The goal of tools APIs is to more reliably return valid and useful tool calls than what can be done using a generic text completion or chat API.

We can take advantage of this structured output, combined with the fact that you can bind multiple tools to a tool calling chat model and allow the model to choose which one to call, to create an agent that repeatedly calls tools and receives results until a query is resolved.

This is a more generalized version of the OpenAI tools agent, which was designed for OpenAI's specific style of tool calling. It uses LangChain's ToolCall interface to support a wider range of provider implementations, such as Anthropic, Google Gemini, and Mistral in addition to OpenAI.
Setup

Any models that support tool calling can be used in this agent. You can see which models support tool calling here

This demo uses Tavily, but you can also swap in any other built-in tool or add custom tools. You'll need to sign up for an API key and set it as process.env.TAVILY_API_KEY.

    OpenAI
    Anthropic
    Google
    FireworksAI
    MistralAI
    TogetherAI

Install dependencies

pip install -qU langchain-openai

Set environment variables

import getpass
import os

os.environ["OPENAI_API_KEY"] = getpass.getpass()

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-3.5-turbo-0125")

Initialize Tools

We will first create a tool that can search the web:

from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain_core.prompts import ChatPromptTemplate

tools = [TavilySearchResults(max_results=1)]

API Reference:

    AgentExecutor
    create_tool_calling_agent
    TavilySearchResults
    ChatPromptTemplate

Create Agent

Next, let's initialize our tool calling agent:

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful assistant. Make sure to use the tavily_search_results_json tool for information.",
        ),
        ("placeholder", "{chat_history}"),
        ("human", "{input}"),
        ("placeholder", "{agent_scratchpad}"),
    ]
)

# Construct the Tools agent
agent = create_tool_calling_agent(llm, tools, prompt)

Run Agent

Now, let's initialize the executor that will run our agent and invoke it!

# Create an agent executor by passing in the agent and tools
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({"input": "what is LangChain?"})



[1m> Entering new AgentExecutor chain...[0m
``````output
/Users/bagatur/langchain/libs/partners/anthropic/langchain_anthropic/chat_models.py:347: UserWarning: stream: Tool use is not yet supported in streaming mode.
  warnings.warn("stream: Tool use is not yet supported in streaming mode.")
``````output
[32;1m[1;3m
Invoking: `tavily_search_results_json` with `{'query': 'LangChain'}`
responded: [{'id': 'toolu_01QxrrT9srzkYCNyEZMDhGeg', 'input': {'query': 'LangChain'}, 'name': 'tavily_search_results_json', 'type': 'tool_use'}]

[0m[36;1m[1;3m[{'url': 'https://github.com/langchain-ai/langchain', 'content': 'About\n⚡ Building applications with LLMs through composability ⚡\nResources\nLicense\nCode of conduct\nSecurity policy\nStars\nWatchers\nForks\nReleases\n291\nPackages\n0\nUsed by 39k\nContributors\n1,848\nLanguages\nFooter\nFooter navigation Latest commit\nGit stats\nFiles\nREADME.md\n🦜️🔗 LangChain\n⚡ Building applications with LLMs through composability ⚡\nLooking for the JS/TS library? ⚡ Building applications with LLMs through composability ⚡\nLicense\nlangchain-ai/langchain\nName already in use\nUse Git or checkout with SVN using the web URL.\n 📖 Documentation\nPlease see here for full documentation, which includes:\n💁 Contributing\nAs an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.\n What can you build with LangChain?\n❓ Retrieval augmented generation\n💬 Analyzing structured data\n🤖 Chatbots\nAnd much more!'}][0m
``````output
/Users/bagatur/langchain/libs/partners/anthropic/langchain_anthropic/chat_models.py:347: UserWarning: stream: Tool use is not yet supported in streaming mode.
  warnings.warn("stream: Tool use is not yet supported in streaming mode.")
``````output
[32;1m[1;3mLangChain is an open-source Python library that helps developers build applications with large language models (LLMs) through composability. Some key features of LangChain include:

- Retrieval augmented generation - Allowing LLMs to retrieve and utilize external data sources when generating outputs.

- Analyzing structured data - Tools for working with structured data like databases, APIs, PDFs, etc. and allowing LLMs to reason over this data.

- Building chatbots and agents - Frameworks for building conversational AI applications.

- Composability - LangChain allows you to chain together different LLM capabilities and data sources in a modular and reusable way.

The library aims to make it easier to build real-world applications that leverage the power of large language models in a scalable and robust way. It provides abstractions and primitives for working with LLMs from different providers like OpenAI, Anthropic, Cohere, etc. LangChain is open-source and has an active community contributing new features and improvements.[0m

[1m> Finished chain.[0m

{'input': 'what is LangChain?',
 'output': 'LangChain is an open-source Python library that helps developers build applications with large language models (LLMs) through composability. Some key features of LangChain include:\n\n- Retrieval augmented generation - Allowing LLMs to retrieve and utilize external data sources when generating outputs.\n\n- Analyzing structured data - Tools for working with structured data like databases, APIs, PDFs, etc. and allowing LLMs to reason over this data.\n\n- Building chatbots and agents - Frameworks for building conversational AI applications.\n\n- Composability - LangChain allows you to chain together different LLM capabilities and data sources in a modular and reusable way.\n\nThe library aims to make it easier to build real-world applications that leverage the power of large language models in a scalable and robust way. It provides abstractions and primitives for working with LLMs from different providers like OpenAI, Anthropic, Cohere, etc. LangChain is open-source and has an active community contributing new features and improvements.'}

tip

LangSmith trace
Using with chat history

This type of agent can optionally take chat messages representing previous conversation turns. It can use that previous history to respond conversationally. For more details, see this section of the agent quickstart.

from langchain_core.messages import AIMessage, HumanMessage

agent_executor.invoke(
    {
        "input": "what's my name? Don't use tools to look this up unless you NEED to",
        "chat_history": [
            HumanMessage(content="hi! my name is bob"),
            AIMessage(content="Hello Bob! How can I assist you today?"),
        ],
    }
)

API Reference:

    AIMessage
    HumanMessage



[1m> Entering new AgentExecutor chain...[0m
``````output
/Users/bagatur/langchain/libs/partners/anthropic/langchain_anthropic/chat_models.py:347: UserWarning: stream: Tool use is not yet supported in streaming mode.
  warnings.warn("stream: Tool use is not yet supported in streaming mode.")
``````output
[32;1m[1;3mBased on what you told me, your name is Bob. I don't need to use any tools to look that up since you directly provided your name.[0m

[1m> Finished chain.[0m

{'input': "what's my name? Don't use tools to look this up unless you NEED to",
 'chat_history': [HumanMessage(content='hi! my name is bob'),
  AIMessage(content='Hello Bob! How can I assist you today?')],
 'output': "Based on what you told me, your name is Bob. I don't need to use any tools to look that up since you directly provided your name."}

tip

LangSmith trace
Help us out by providing feedback on this documentation page:
Previous
Types
Next
OpenAI tools

    Setup
    Initialize Tools
    Create Agent
    Run Agent
    Using wit
	
	
	
	
	
	
	
	
	
	
here is the code you need to update:




import gradio as gr
from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from langchain_core.tools import tool
from langchain_community.tools import DuckDuckGoSearchRun
import base64
from io import BytesIO
from PIL import Image
import os

# Define the web search tool using the @tool decorator
@tool
def search_web(query: str) -> str:
    """Use this tool to search the web for real-time information based on a query."""
    print(f"Searching: {query}")
    search = DuckDuckGoSearchRun()
    return search.run(query)

@tool
def generate_image(prompt: str) -> str:
    """Generate an image based on the prompt and return the path."""
    print(f"Gennerating iamge: {prompt}")
    # This is a demo tool that just returns a placeholder image
    return "image.png"  # Assuming image.png exists in your directory

# Initialize the models
llm_vision = ChatOllama(
    model='llama3.2-vision',
    temperature=0.8,
)

llm_text = ChatOllama(
    model='llama3.1',
    temperature=0.8,
).bind_tools([search_web, generate_image])

def convert_image_to_base64(image):
    """Convert a PIL Image to a base64 encoded string."""
    buffered = BytesIO()
    image.save(buffered, format="JPEG")
    img_str = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return img_str

def vision_chat(message, vision_history, img):
    """Handle the vision chat interactions."""
    model_messages = []
    
    # Process history
    for user_msg, bot_msg in vision_history:
        if isinstance(user_msg, dict):
            content_parts = []
            if user_msg.get("text"):
                content_parts.append({"type": "text", "text": user_msg["text"]})
            if user_msg.get("image"):
                image_b64 = convert_image_to_base64(user_msg["image"])
                content_parts.append({
                    "type": "image_url",
                    "image_url": f"data:image/jpeg;base64,{image_b64}"
                })
            model_messages.append(HumanMessage(content=content_parts))
        if bot_msg:
            model_messages.append(AIMessage(content=bot_msg))

    # Process current message
    content_parts = []
    if message:
        content_parts.append({"type": "text", "text": message})
    if img is not None:
        image_b64 = convert_image_to_base64(img)
        content_parts.append({
            "type": "image_url",
            "image_url": f"data:image/jpeg;base64,{image_b64}"
        })
    
    if content_parts:
        model_messages.append(HumanMessage(content=content_parts))
        
    # Generate response
    response = ""
    for chunk in llm_vision.stream(model_messages):
        if hasattr(chunk, 'content'):
            response += chunk.content
            yield response

def search_chat(message, search_history):
    """Handle the search chat interactions with tool usage."""
    print(f"\nReceived message: {message}")
    model_messages = []
    
    # Add system message for tool usage
    system_prompt = (
        "You are a helpful assistant with access to two tools:\n\n"
        "1. search_web: Use this to search the web for real-time information.\n"
        "2. generate_image: Use this to generate images based on text prompts.\n\n"
        "When a user asks for an image to be generated, use the generate_image tool.\n"
        "For current events or real-time information, use the search_web tool."
    )
    model_messages.append(SystemMessage(content=system_prompt))
    
    # Process history
    print("\nProcessing history:")
    for user_msg, bot_msg in search_history:
        print(f"User message: {user_msg}")
        print(f"Bot message: {bot_msg}")
        model_messages.append(HumanMessage(content=user_msg))
        if bot_msg:
            # Check if bot message contains a generated image
            if isinstance(bot_msg, dict) and "image" in bot_msg:
                content = [
                    {"type": "text", "text": bot_msg.get("text", "")},
                    {"type": "image_url", "image_url": f"file://{bot_msg['image']}"}
                ]
                model_messages.append(AIMessage(content=content))
            else:
                model_messages.append(AIMessage(content=bot_msg))
    
    # Add current message
    model_messages.append(HumanMessage(content=message))
    
    print("\nGenerating response...")
    try:
        # Use invoke instead of stream for tool-using responses
        response = llm_text.invoke(model_messages)
        print(f"Raw response: {response}")
        
        if hasattr(response, 'content'):
            content = response.content
            print(f"Response content: {content}")
            
            # Check if response contains a generated image path
            if "generate_image" in content.lower():
                print("Generating image...")
                image_path = generate_image("")  # You can extract the prompt from the message
                return {"text": content, "image": image_path}
            else:
                return content
        else:
            print("No content in response")
            return "I apologize, but I couldn't generate a proper response."
            
    except Exception as e:
        print(f"Error generating response: {e}")
        return f"An error occurred: {str(e)}"

# Create the tabbed interface
with gr.Blocks() as demo:
    gr.Markdown("# AI Assistant with Vision and Search Capabilities")
    
    with gr.Tabs():
        # Vision Chat Tab
        with gr.Tab("Vision Chat"):
            vision_chatbot = gr.ChatInterface(
                vision_chat,
                additional_inputs=[gr.Image(label="Upload Image", type="pil")],
                title="Chat with Images",
                description="Upload an image and chat about it using llama3.2-vision"
            )
        
        # Search Chat Tab
        with gr.Tab("Search Chat"):
            search_chatbot = gr.ChatInterface(
                search_chat,
                title="Chat with Web Search & Image Generation",
                description="Chat with llama3.1 with web search and image generation capabilities"
            )

if __name__ == "__main__":
    demo.launch()