Help me build the Following App ::

AI powered Ayurvedic App.

Conducts health assessment, provides diagnosis along with treatment planning and medication recommendation.

1. Overall Architecture:
    - RAG (Retrieval-Augmented Generation) architecture
    - Built with Langchain, Langgraph, and GoogleGenerativeAI (Gemini API)
    - Developed in Google Colab notebook for prototyping
    - Uses Qdrant vector database (pre-populated with Ayurvedic knowledge)
2. Agent System:
    - Two-agent workflow
    
    A. Agent 1: Triage and Interview
    
    - Conducts dynamic, realistic interview
    - Creates detailed report on current scenario
    - Does not provide diagnosis
    - Interview process:
        1. Asks user what brought them to the app
        2. For each user response:
            - Checks vector database for context
            - Considers entire chat history
            - Formulates next question based on retrieved information
        3. Repeats until a comprehensive, relevant report is generated
        4. Sends report to Agent 2
    
    B. Agent 2: Diagnostics and Treatment Planning
    
    - Receives report from Agent 1
    - Provides diagnosis
    - Develops treatment plan
    - Recommends medications
    - Uses vector database for Ayurvedic knowledge retrieval
3. Key Components:
    - GoogleGenerativeAI with Gemini API for natural language processing
    - Qdrant vector database for storing and retrieving Ayurvedic knowledge
    - Langchain for building the language model pipeline
    - Langgraph for managing the agent workflow
4. Workflow:
    1. User initiates interaction
    2. Agent 1 conducts interview and generates report
    3. Report is sent to Agent 2
    4. Agent 2 analyzes report and provides diagnosis, treatment plan, and medication recommendations
    5. Results are presented to the user
5. Development Environment:
    - Google Colab notebook for prototyping and development

To achieve a realistic interview, Agent 1 employs a dynamic and context-aware approach. Here's a detailed explanation of how this process works:

1. Initialization:
    - The agent starts with a broad, open-ended question like "What brings you here today?" or "How can I help you?"
    - This allows the user to express their concerns in their own words.
2. Active Listening and Context Building:
    - As the user responds, the agent processes the input and begins building a context model.
    - It identifies key terms, symptoms, or concerns mentioned by the user.
3. Vector Database Querying:
    - The agent queries the Qdrant vector database using the identified key terms and concepts.
    - This retrieves relevant Ayurvedic knowledge and potential follow-up questions.
4. Natural Language Understanding:
    - Using the Gemini API, the agent analyzes the user's response for sentiment, urgency, and implicit information.
5. Dynamic Question Formulation:
    - Based on the retrieved information and NLU analysis, the agent formulates the next question.
    - Questions are tailored to probe deeper into the user's specific concerns.
    - The agent avoids redundant questions by keeping track of information already provided.
6. Contextual Memory:
    - The agent maintains a running summary of the conversation, updating it with each exchange.
    - This allows for reference to previously mentioned information, making the conversation feel more natural and connected.
7. Empathetic Responses:
    - The agent is programmed to recognize emotional cues and respond with appropriate empathy.
    - It may use phrases like "I understand that must be difficult" or "It's good that you're seeking help for this."
8. Clarification and Expansion:
    - If the user's responses are vague or incomplete, the agent asks for clarification or more details.
    - It might say, "Could you tell me more about when you first noticed this symptom?"
9. Holistic Approach:
    - In line with Ayurvedic principles, the agent asks about various aspects of the user's life, including diet, sleep patterns, stress levels, and daily routines.
10. Adaptive Questioning:
    - The agent adjusts the complexity and depth of questions based on the user's responses and perceived health literacy.
11. Non-Linear Conversation Flow:
    - While the agent has certain key areas it needs to cover, it allows the conversation to flow naturally, following the user's lead when appropriate.
12. Time and Relevance Management:
    - The agent is programmed to keep the interview focused and relevant, gently steering the conversation back on track if it diverges too far.
13. Comprehensive Coverage:
    - The agent ensures all relevant areas are covered before concluding the interview.
    - It may use a checklist approach internally but presents questions in a conversational manner.
14. Conclusion and Handover:
    - Once sufficient information is gathered, the agent summarizes the key points of the interview.
    - It informs the user that it will now compile a report for further analysis.

By combining these elements, Agent 1 can conduct an interview that feels natural, responsive, and thorough, mimicking the approach of a skilled Ayurvedic practitioner. The use of AI and the vector database allows for a vast knowledge base to inform the questioning, while the natural language processing capabilities ensure a human-like interaction.

Let me elaborate on how the agent utilizes the entire chat context and continuously refers to the vector database to guide the interview process:

1. Continuous Context Update:
    - After each user response, the agent updates its understanding of the entire conversation context.
    - This context includes not just the latest response, but the cumulative information from the entire chat history.
2. Vector Database Querying:
    - With each context update, the agent formulates a query based on the entire conversation context.
    - This query is sent to the Qdrant vector database.
3. Relevant Information Retrieval:
    - The vector database returns the most relevant Ayurvedic knowledge, potential questions, and interview strategies based on the current context.
4. Dynamic Interview Path:
    - Using the retrieved information, the agent determines the most appropriate direction for the interview.
    - This could involve exploring a new area of concern, diving deeper into a previously mentioned issue, or shifting focus based on newly understood connections.
5. Adaptive Questioning Strategy:
    - The agent uses the vector database's suggestions to formulate questions that are most relevant to the user's specific situation.
    - This ensures that the interview doesn't follow a pre-determined script, but rather adapts in real-time to the user's unique health profile.
6. Contextual Understanding:
    - As the interview progresses, the agent builds a more comprehensive understanding of the user's health situation.
    - This understanding is continually refined with each database query and user response.
7. Holistic Approach Maintenance:
    - The vector database helps the agent maintain a holistic Ayurvedic approach by suggesting questions about different aspects of the user's life that may not be immediately obvious but are relevant in Ayurvedic practice.
8. Knowledge-Guided Probing:
    - When the user mentions symptoms or conditions, the agent queries the database for related Ayurvedic concepts.
    - This allows the agent to ask informed follow-up questions that a knowledgeable Ayurvedic practitioner would ask.
9. Conversation Flow Management:
    - The agent uses the database's input to manage the flow of the conversation, ensuring all relevant areas are covered while maintaining a natural dialogue.
10. Relevance Checking:
    - With each potential question, the agent cross-references with the vector database to ensure its relevance to the current context of the conversation.
11. Insight Generation:
    - As the interview progresses, the agent uses the database to generate insights about potential connections between different pieces of information provided by the user.
12. Interview Progression Tracking:
    - The agent tracks the progress of the interview against Ayurvedic diagnostic criteria stored in the vector database, ensuring comprehensive coverage.

By constantly referring to the vector database and considering the entire chat context, the agent can conduct a highly personalized, knowledge-driven interview that feels natural and adapts in real-time to the user's responses. This approach allows for a dynamic interview process that can uncover relevant health information even if the user doesn't initially present it, mimicking the intuitive and holistic approach of an experienced Ayurvedic practitioner.

Please use the following documentation as reference ::

Agentic RAG
Retrieval Agents are useful when we want to make decisions about whether to
retrieve from an index.
To implement a retrieval agent, we simply need to give an LLM access to a retriever
tool.
We can incorporate this into LangGraph.
Setup
First, let's download the required packages and set our API keys:
Sign up for LangSmith to quickly spot issues and improve the performance of your
LangGraph projects. LangSmith lets you use trace data to debug, test, and monitor
your LLM apps built with LangGraph ? read more about how to get started here.
%%capture --no-stderr
%pip install -U --quiet langchain-community tiktoken langchain-
openai langchainhub chromadb langchain langgraph langchain-text-
splitters
import getpass
import os
def _set_env(key: str):
if key not in os.environ:
os.environ[key] = getpass.getpass(f"{key}:")
_set_env("OPENAI_API_KEY")
Set up LangSmith for LangGraph development
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 1/12
Retriever
First, we index 3 blog posts.
Then we create a retriever tool.
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import
RecursiveCharacterTextSplitter
urls = [
"https://lilianweng.github.io/posts/2023-06-23-agent/",
"https://lilianweng.github.io/posts/2023-03-15-prompt-
engineering/",
"https://lilianweng.github.io/posts/2023-10-25-adv-attack-
llm/",
]
docs = [WebBaseLoader(url).load() for url in urls]
docs_list = [item for sublist in docs for item in sublist]
text_splitter =
RecursiveCharacterTextSplitter.from_tiktoken_encoder(
chunk_size=100, chunk_overlap=50
)
doc_splits = text_splitter.split_documents(docs_list)
# Add to vectorDB
vectorstore = Chroma.from_documents(
documents=doc_splits,
collection_name="rag-chroma",
embedding=OpenAIEmbeddings(),
)
retriever = vectorstore.as_retriever()
from langchain.tools.retriever import create_retriever_tool
retriever_tool = create_retriever_tool(
retriever,
"retrieve_blog_posts",
"Search and return information about Lilian Weng blog posts
on LLM agents, prompt engineering, and adversarial attacks on
LLMs.",
)
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 2/12
Agent State
We will de?ne a graph.
A state object that it passes around to each node.
Our state will be a list of messages .
Each node in our graph will append to it.
Nodes and Edges
We can lay out an agentic RAG graph like this:
The state is a set of messages
Each node will update (append to) state
Conditional edges decide which node to visit next
tools = [retriever_tool]
from typing import Annotated, Sequence
from typing_extensions import TypedDict
from langchain_core.messages import BaseMessage
from langgraph.graph.message import add_messages
class AgentState(TypedDict):
# The add_messages function defines how an update should be
processed
# Default is to replace. add_messages says "append"
messages: Annotated[Sequence[BaseMessage], add_messages]
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 3/12
This notebook uses Pydantic v2 BaseModel , which requires langchain-core >= 0.3 .
Using langchain-core < 0.3 will result in errors due to mixing of Pydantic v1 and v2
BaseModels .
Using Pydantic with LangChain
from typing import Annotated, Literal, Sequence
from typing_extensions import TypedDict
from langchain import hub
from langchain_core.messages import BaseMessage, HumanMessage
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
from pydantic import BaseModel, Field
from langgraph.prebuilt import tools_condition
### Edges
def grade_documents(state) -> Literal["generate", "rewrite"]:
"""
Determines whether the retrieved documents are relevant to
the question.
Args:
state (messages): The current state
Returns:
str: A decision for whether the documents are relevant
or not
"""
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 4/12
print("---CHECK RELEVANCE---")
# Data model
class grade(BaseModel):
"""Binary score for relevance check."""
binary_score: str = Field(description="Relevance score
'yes' or 'no'")
# LLM
model = ChatOpenAI(temperature=0, model="gpt-4-0125-
preview", streaming=True)
# LLM with tool and validation
llm_with_tool = model.with_structured_output(grade)
# Prompt
prompt = PromptTemplate(
template="""You are a grader assessing relevance of a
retrieved document to a user question. \n
Here is the retrieved document: \n\n {context} \n\n
Here is the user question: {question} \n
If the document contains keyword(s) or semantic meaning
related to the user question, grade it as relevant. \n
Give a binary score 'yes' or 'no' score to indicate
whether the document is relevant to the question.""",
input_variables=["context", "question"],
)
# Chain
chain = prompt | llm_with_tool
messages = state["messages"]
last_message = messages[-1]
question = messages[0].content
docs = last_message.content
scored_result = chain.invoke({"question": question,
"context": docs})
score = scored_result.binary_score
if score == "yes":
print("---DECISION: DOCS RELEVANT---")
return "generate"
else:
print("---DECISION: DOCS NOT RELEVANT---")
print(score)
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 5/12
return "rewrite"
### Nodes
def agent(state):
"""
Invokes the agent model to generate a response based on the
current state. Given
the question, it will decide to retrieve using the retriever
tool, or simply end.
Args:
state (messages): The current state
Returns:
dict: The updated state with the agent response appended
to messages
"""
print("---CALL AGENT---")
messages = state["messages"]
model = ChatOpenAI(temperature=0, streaming=True,
model="gpt-4-turbo")
model = model.bind_tools(tools)
response = model.invoke(messages)
# We return a list, because this will get added to the
existing list
return {"messages": [response]}
def rewrite(state):
"""
Transform the query to produce a better question.
Args:
state (messages): The current state
Returns:
dict: The updated state with re-phrased question
"""
print("---TRANSFORM QUERY---")
messages = state["messages"]
question = messages[0].content
msg = [
HumanMessage(
content=f""" \n
Look at the input and try to reason about the underlying
semantic intent / meaning. \n
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 6/12
Here is the initial question:
\n ------- \n
{question}
\n ------- \n
Formulate an improved question: """,
)
]
# Grader
model = ChatOpenAI(temperature=0, model="gpt-4-0125-
preview", streaming=True)
response = model.invoke(msg)
return {"messages": [response]}
def generate(state):
"""
Generate answer
Args:
state (messages): The current state
Returns:
dict: The updated state with re-phrased question
"""
print("---GENERATE---")
messages = state["messages"]
question = messages[0].content
last_message = messages[-1]
docs = last_message.content
# Prompt
prompt = hub.pull("rlm/rag-prompt")
# LLM
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0,
streaming=True)
# Post-processing
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
# Chain
rag_chain = prompt | llm | StrOutputParser()
# Run
response = rag_chain.invoke({"context": docs, "question":
question})
return {"messages": [response]}
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 7/12
Graph
Start with an agent, call_model
Agent make a decision to call a function
If so, then action to call tool (retriever)
Then call agent with the tool output added to messages ( state )
print("*" * 20 + "Prompt[rlm/rag-prompt]" + "*" * 20)
prompt = hub.pull("rlm/rag-prompt").pretty_print() # Show what
the prompt looks like
********************Prompt[rlm/rag-prompt]********************
================================ Human Message ===================
==============
You are an assistant for question-answering tasks. Use the followi
ng pieces of retrieved context to answer the question. If you do
n't know the answer, just say that you don't know. Use three sente
nces maximum and keep the answer concise.
Question: {question}
Context: {context}
Answer:
from langgraph.graph import END, StateGraph, START
from langgraph.prebuilt import ToolNode
# Define a new graph
workflow = StateGraph(AgentState)
# Define the nodes we will cycle between
workflow.add_node("agent", agent) # agent
retrieve = ToolNode([retriever_tool])
workflow.add_node("retrieve", retrieve) # retrieval
workflow.add_node("rewrite", rewrite) # Re-writing the question
workflow.add_node(
"generate", generate
) # Generating a response after we know the documents are
relevant
# Call agent node to decide to retrieve or not
workflow.add_edge(START, "agent")
# Decide whether to retrieve
workflow.add_conditional_edges(
"agent",
# Assess agent decision
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 8/12
tools_condition,
{
# Translate the condition outputs to nodes in our graph
"tools": "retrieve",
END: END,
},
)
# Edges taken after the `action` node is called.
workflow.add_conditional_edges(
"retrieve",
# Assess agent decision
grade_documents,
)
workflow.add_edge("generate", END)
workflow.add_edge("rewrite", "agent")
# Compile
graph = workflow.compile()
from IPython.display import Image, display
try:
display(Image(graph.get_graph(xray=True).draw_mermaid_png()))
except Exception:
# This requires some extra dependencies and is optional
pass
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 9/12
import pprint
inputs = {
"messages": [
("user", "What does Lilian Weng say about the types of
agent memory?"),
]
}
for output in graph.stream(inputs):
for key, value in output.items():
pprint.pprint(f"Output from node '{key}':")
pprint.pprint("---")
pprint.pprint(value, indent=2, width=80, depth=None)
pprint.pprint("\n---\n")
---CALL AGENT---
"Output from node 'agent':"
'---'
{ 'messages': [ AIMessage(content='', additional_kwargs={'tool_cal
ls': [{'index': 0, 'id': 'call_z36oPZN8l1UC6raxrebqc1bH', 'functio
n': {'arguments': '{"query":"types of agent memory"}', 'name': 're
trieve_blog_posts'}, 'type': 'function'}]}, response_metadata={'fi
nish_reason': 'tool_calls'}, id='run-2bad2518-8187-4d8f-8e23-2b950
1becb6f-0', tool_calls=[{'name': 'retrieve_blog_posts', 'args':
10/3/24, 10:36 AM Agentic RAG
https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/ 10/12
Quick Start
In this comprehensive quick start, we will build a support chatbot in LangGraph that
can:
Answer common questions by searching the web
Maintain conversation state across calls
Route complex queries to a human for review
Use custom state to control its behavior
Rewind and explore alternative conversation paths
We'll start with a basic chatbot and progressively add more sophisticated
capabilities, introducing key LangGraph concepts along the way.
Setup
First, install the required packages:
Next, set your API keys:
%%capture --no-stderr
%pip install -U langgraph langsmith langchain_anthropic
import getpass
import os
def _set_env(var: str):
if not os.environ.get(var):
os.environ[var] = getpass.getpass(f"{var}: ")
_set_env("ANTHROPIC_API_KEY")
ANTHROPIC_API_KEY: ········
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 1/61
Sign up for LangSmith to quickly spot issues and improve the performance of your
LangGraph projects. LangSmith lets you use trace data to debug, test, and monitor
your LLM apps built with LangGraph ? read more about how to get started here.
Part 1: Build a Basic Chatbot
We'll ?rst create a simple chatbot using LangGraph. This chatbot will respond
directly to user messages. Though simple, it will illustrate the core concepts of
building with LangGraph. By the end of this section, you will have a built rudimentary
chatbot.
Start by creating a StateGraph . A StateGraph object de?nes the structure of our
chatbot as a "state machine". We'll add nodes to represent the llm and functions our
chatbot can call and edges to specify how the bot should transition between these
functions.
Set up LangSmith for LangGraph development
from typing import Annotated
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
class State(TypedDict):
# Messages have the type "list". The `add_messages` function
# in the annotation defines how this state key should be
updated
# (in this case, it appends messages to the list, rather than
overwriting them)
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 2/61
The ?rst thing you do when you de?ne a graph is de?ne the State of the graph. The
State consists of the schema of the graph as well as reducer functions which specify
how to apply updates to the state. In our example State is a TypedDict with a single
key: messages . The messages key is annotated with the add_messages reducer
function, which tells LangGraph to append new messages to the existing list, rather
than overwriting it. State keys without an annotation will be overwritten by each
update, storing the most recent value. Check out this conceptual guide to learn more
about state, reducers and other low-level concepts.
So now our graph knows two things:
?? Every node we de?ne will receive the current State as input and return a value
that updates that state.
?? messages will be appended to the current list, rather than directly overwritten.
This is communicated via the prebuilt add_messages function in the Annotated
syntax.
Next, add a " chatbot " node. Nodes represent units of work. They are typically
regular python functions.
Notice how the chatbot node function takes the current State as input and returns
a dictionary containing an updated messages list under the key "messages". This is
the basic pattern for all LangGraph node functions.
The add_messages function in our State will append the llm's response messages to
whatever messages are already in the state.
Note
from langchain_anthropic import ChatAnthropic
llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")
def chatbot(state: State):
return {"messages": [llm.invoke(state["messages"])]}
# The first argument is the unique node name
# The second argument is the function or object that will be
called whenever
# the node is used.
graph_builder.add_node("chatbot", chatbot)
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 3/61
Next, add an entry point. This tells our graph where to start its work each time we
run it.
Similarly, set a finish point. This instructs the graph "any time this node is run, you
can exit."
Finally, we'll want to be able to run our graph. To do so, call " compile() " on the graph
builder. This creates a " CompiledGraph " we can use invoke on our state.
You can visualize the graph using the get_graph method and one of the "draw"
methods, like draw_ascii or draw_png . The draw methods each require additional
dependencies.
Now let's run the chatbot!
Tip: You can exit the chat loop at any time by typing "quit", "exit", or "q".
graph_builder.add_edge(START, "chatbot")
graph_builder.add_edge("chatbot", END)
graph = graph_builder.compile()
from IPython.display import Image, display
try:
display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
# This requires some extra dependencies and is optional
pass
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 4/61
Congratulations! You've built your ?rst chatbot using LangGraph. This bot can
engage in basic conversation by taking user input and generating responses using an
LLM. You can inspect a LangSmith Trace for the call above at the provided link.
However, you may have noticed that the bot's knowledge is limited to what's in its
training data. In the next part, we'll add a web search tool to expand the bot's
knowledge and make it more capable.
Below is the full code for this section for your reference:
def stream_graph_updates(user_input: str):
for event in graph.stream({"messages": [("user",
user_input)]}):
for value in event.values():
print("Assistant:", value["messages"][-1].content)
while True:
try:
user_input = input("User: ")
if user_input.lower() in ["quit", "exit", "q"]:
print("Goodbye!")
break
stream_graph_updates(user_input)
except:
# fallback if input() is not available
user_input = "What do you know about LangGraph?"
print("User: " + user_input)
stream_graph_updates(user_input)
break
Assistant: LangGraph is a library designed to help build stateful
multi-agent applications using language models. It provides tools
for creating workflows and state machines to coordinate multiple A
I agents or language model interactions. LangGraph is built on top
of LangChain, leveraging its components while adding graph-based c
oordination capabilities. It's particularly useful for developing
more complex, stateful AI applications that go beyond simple query
-response interactions.
Goodbye!
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 5/61
Part 2: Enhancing the Chatbot with Tools
To handle queries our chatbot can't answer "from memory", we'll integrate a web
search tool. Our bot can use this tool to ?nd relevant information and provide better
Full Code
from typing import Annotated
from langchain_anthropic import ChatAnthropic
from typing_extensions import TypedDict
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages
class State(TypedDict):
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")
def chatbot(state: State):
return {"messages": [llm.invoke(state["messages"])]}
# The first argument is the unique node name
# The second argument is the function or object that will be
called whenever
# the node is used.
graph_builder.add_node("chatbot", chatbot)
graph_builder.set_entry_point("chatbot")
graph_builder.set_finish_point("chatbot")
graph = graph_builder.compile()
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 6/61
responses.
Requirements
Before we start, make sure you have the necessary packages installed and API keys
set up:
First, install the requirements to use the Tavily Search Engine, and set your
TAVILY_API_KEY.
Next, de?ne the tool:
The results are page summaries our chat bot can use to answer questions.
Next, we'll start de?ning our graph. The following is all the same as in Part 1, except
we have added bind_tools on our LLM. This lets the LLM know the correct JSON
format to use if it wants to use our search engine.
%%capture --no-stderr
%pip install -U tavily-python langchain_community
_set_env("TAVILY_API_KEY")
TAVILY_API_KEY: ········
from langchain_community.tools.tavily_search import
TavilySearchResults
tool = TavilySearchResults(max_results=2)
tools = [tool]
tool.invoke("What's a 'node' in LangGraph?")
[{'url': 'https://medium.com/@cplog/introduction-to-langgraph-a-be
ginners-guide-14f9be027141',
'content': 'Nodes: Nodes are the building blocks of your LangGra
ph. Each node represents a function or a computation step. You def
ine nodes to perform specific tasks, such as processing input, mak
ing ...'},
{'url': 'https://saksheepatil05.medium.com/demystifying-langgraph
-a-beginner-friendly-dive-into-langgraph-concepts-5ffe890ddac0',
'content': 'Nodes (Tasks): Nodes are like the workstations on th
e assembly line. Each node performs a specific task on the produc
t. In LangGraph, nodes are Python functions that take the current
state, do some work, and return an updated state. Next, we define
the nodes, each representing a task in our sandwich-making proces
s.'}]
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 7/61
Next we need to create a function to actually run the tools if they are called. We'll do
this by adding the tools to a new node.
Below, we implement a BasicToolNode that checks the most recent message in the
state and calls tools if the message contains tool_calls . It relies on the LLM's
tool_calling support, which is available in Anthropic, OpenAI, Google Gemini, and a
number of other LLM providers.
We will later replace this with LangGraph's prebuilt ToolNode to speed things up, but
building it ourselves ?rst is instructive.
from typing import Annotated
from langchain_anthropic import ChatAnthropic
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
class State(TypedDict):
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")
# Modification: tell the LLM which tools it can call
llm_with_tools = llm.bind_tools(tools)
def chatbot(state: State):
return {"messages":
[llm_with_tools.invoke(state["messages"])]}
graph_builder.add_node("chatbot", chatbot)
import json
from langchain_core.messages import ToolMessage
class BasicToolNode:
"""A node that runs the tools requested in the last
AIMessage."""
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 8/61
With the tool node added, we can de?ne the conditional_edges .
Recall that edges route the control ?ow from one node to the next. Conditional
edges usually contain "if" statements to route to different nodes depending on the
current graph state. These functions receive the current graph state and return a
string or list of strings indicating which node(s) to call next.
Below, call de?ne a router function called route_tools , that checks for tool_calls in
the chatbot's output. Provide this function to the graph by calling
add_conditional_edges , which tells the graph that whenever the chatbot node
completes to check this function to see where to go next.
The condition will route to tools if tool calls are present and END if not.
Later, we will replace this with the prebuilt tools_condition to be more concise, but
implementing it ourselves ?rst makes things more clear.
def __init__(self, tools: list) -> None:
self.tools_by_name = {tool.name: tool for tool in tools}
def __call__(self, inputs: dict):
if messages := inputs.get("messages", []):
message = messages[-1]
else:
raise ValueError("No message found in input")
outputs = []
for tool_call in message.tool_calls:
tool_result =
self.tools_by_name[tool_call["name"]].invoke(
tool_call["args"]
)
outputs.append(
ToolMessage(
content=json.dumps(tool_result),
name=tool_call["name"],
tool_call_id=tool_call["id"],
)
)
return {"messages": outputs}
tool_node = BasicToolNode(tools=[tool])
graph_builder.add_node("tools", tool_node)
from typing import Literal
def route_tools(
state: State,
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 9/61
Notice that conditional edges start from a single node. This tells the graph "any time
the ' chatbot ' node runs, either go to 'tools' if it calls a tool, or end the loop if it
responds directly.
Like the prebuilt tools_condition , our function returns the END string if no tool
calls are made. When the graph transitions to END , it has no more tasks to complete
and ceases execution. Because the condition can return END , we don't need to
explicitly set a finish_point this time. Our graph already has a way to ?nish!
):
"""
Use in the conditional_edge to route to the ToolNode if the
last message
has tool calls. Otherwise, route to the end.
"""
if isinstance(state, list):
ai_message = state[-1]
elif messages := state.get("messages", []):
ai_message = messages[-1]
else:
raise ValueError(f"No messages found in input state to
tool_edge: {state}")
if hasattr(ai_message, "tool_calls") and
len(ai_message.tool_calls) > 0:
return "tools"
return END
# The `tools_condition` function returns "tools" if the chatbot
asks to use a tool, and "END" if
# it is fine directly responding. This conditional routing
defines the main agent loop.
graph_builder.add_conditional_edges(
"chatbot",
route_tools,
# The following dictionary lets you tell the graph to
interpret the condition's outputs as a specific node
# It defaults to the identity function, but if you
# want to use a node named something else apart from "tools",
# You can update the value of the dictionary to something
else
# e.g., "tools": "my_tools"
{"tools": "tools", END: END},
)
# Any time a tool is called, we return to the chatbot to decide
the next step
graph_builder.add_edge("tools", "chatbot")
graph_builder.add_edge(START, "chatbot")
graph = graph_builder.compile()
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 10/61
Let's visualize the graph we've built. The following function has some additional
dependencies to run that are unimportant for this tutorial.
Now we can ask the bot questions outside its training data.
from IPython.display import Image, display
try:
display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
# This requires some extra dependencies and is optional
pass
while True:
try:
user_input = input("User: ")
if user_input.lower() in ["quit", "exit", "q"]:
print("Goodbye!")
break
stream_graph_updates(user_input)
except:
# fallback if input() is not available
user_input = "What do you know about LangGraph?"
print("User: " + user_input)
stream_graph_updates(user_input)
break
Assistant: [{'text': "To provide you with accurate and up-to-date
information about LangGraph, I'll need to search for the latest de
tails. Let me do that for you.", 'type': 'text'}, {'id': 'toolu_01
Q588CszHaSvvP2MxRq9zRD', 'input': {'query': 'LangGraph AI tool inf
ormation'}, 'name': 'tavily_search_results_json', 'type': 'tool_us
e'}]
Assistant: [{"url": "https://www.langchain.com/langgraph", "conten
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 11/61
t": "LangGraph sets the foundation for how we can build and scale
AI workloads \u2014 from conversational agents, complex task autom
ation, to custom LLM-backed experiences that 'just work'. The next
chapter in building complex production-ready features with LLMs is
agentic, and with LangGraph and LangSmith, LangChain delivers an o
ut-of-the-box solution ..."}, {"url": "https://github.com/langchai
n-ai/langgraph", "content": "Overview. LangGraph is a library for
building stateful, multi-actor applications with LLMs, used to cre
ate agent and multi-agent workflows. Compared to other LLM framewo
rks, it offers these core benefits: cycles, controllability, and p
ersistence. LangGraph allows you to define flows that involve cycl
es, essential for most agentic architectures ..."}]
Assistant: Based on the search results, I can provide you with inf
ormation about LangGraph:
1. Purpose:
LangGraph is a library designed for building stateful, multi-ac
tor applications with Large Language Models (LLMs). It's particula
rly useful for creating agent and multi-agent workflows.
2. Developer:
LangGraph is developed by LangChain, a company known for its to
ols and frameworks in the AI and LLM space.
3. Key Features:
- Cycles: LangGraph allows the definition of flows that involve
cycles, which is essential for most agentic architectures.
- Controllability: It offers enhanced control over the applicat
ion flow.
- Persistence: The library provides ways to maintain state and
persistence in LLM-based applications.
4. Use Cases:
LangGraph can be used for various applications, including:
- Conversational agents
- Complex task automation
- Custom LLM-backed experiences
5. Integration:
LangGraph works in conjunction with LangSmith, another tool by
LangChain, to provide an out-of-the-box solution for building comp
lex, production-ready features with LLMs.
6. Significance:
LangGraph is described as setting the foundation for building a
nd scaling AI workloads. It's positioned as a key tool in the next
chapter of LLM-based application development, particularly in the
realm of agentic AI.
7. Availability:
LangGraph is open-source and available on GitHub, which suggest
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 12/61
Congrats! You've created a conversational agent in langgraph that can use a search
engine to retrieve updated information when needed. Now it can handle a wider
range of user queries. To inspect all the steps your agent just took, check out this
LangSmith trace.
Our chatbot still can't remember past interactions on its own, limiting its ability to
have coherent, multi-turn conversations. In the next part, we'll add memory to
address this.
The full code for the graph we've created in this section is reproduced below,
replacing our BasicToolNode for the prebuilt ToolNode, and our route_tools
condition with the prebuilt tools_condition
s that developers can access and contribute to its codebase.
8. Comparison to Other Frameworks:
LangGraph is noted to offer unique benefits compared to other L
LM frameworks, particularly in its ability to handle cycles, provi
de controllability, and maintain persistence.
LangGraph appears to be a significant tool in the evolving landsca
pe of LLM-based application development, offering developers new w
ays to create more complex, stateful, and interactive AI systems.
Goodbye!
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 13/61
Full Code
from typing import Annotated
from langchain_anthropic import ChatAnthropic
from langchain_community.tools.tavily_search import
TavilySearchResults
from langchain_core.messages import BaseMessage
from typing_extensions import TypedDict
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition
class State(TypedDict):
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
tool = TavilySearchResults(max_results=2)
tools = [tool]
llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")
llm_with_tools = llm.bind_tools(tools)
def chatbot(state: State):
return {"messages":
[llm_with_tools.invoke(state["messages"])]}
graph_builder.add_node("chatbot", chatbot)
tool_node = ToolNode(tools=[tool])
graph_builder.add_node("tools", tool_node)
graph_builder.add_conditional_edges(
"chatbot",
tools_condition,
)
# Any time a tool is called, we return to the chatbot to decide
the next step
graph_builder.add_edge("tools", "chatbot")
graph_builder.set_entry_point("chatbot")
graph = graph_builder.compile()
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 14/61
Part 3: Adding Memory to the Chatbot
Our chatbot can now use tools to answer user questions, but it doesn't remember the
context of previous interactions. This limits its ability to have coherent, multi-turn
conversations.
LangGraph solves this problem through persistent checkpointing. If you provide a
checkpointer when compiling the graph and a thread_id when calling your graph,
LangGraph automatically saves the state after each step. When you invoke the graph
again using the same thread_id , the graph loads its saved state, allowing the
chatbot to pick up where it left off.
We will see later that checkpointing is much more powerful than simple chat
memory - it lets you save and resume complex state at any time for error recovery,
human-in-the-loop work?ows, time travel interactions, and more. But before we get
too ahead of ourselves, let's add checkpointing to enable multi-turn conversations.
To get started, create a MemorySaver checkpointer.
Notice we're using an in-memory checkpointer. This is convenient for our tutorial (it
saves it all in-memory). In a production application, you would likely change this to
use SqliteSaver or PostgresSaver and connect to your own DB.
Next de?ne the graph. Now that you've already built your own BasicToolNode , we'll
replace it with LangGraph's prebuilt ToolNode and tools_condition , since these do
some nice things like parallel API execution. Apart from that, the following is all
copied from Part 2.
from langgraph.checkpoint.memory import MemorySaver
memory = MemorySaver()
from typing import Annotated
from langchain_anthropic import ChatAnthropic
from langchain_community.tools.tavily_search import
TavilySearchResults
from langchain_core.messages import BaseMessage
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 15/61
Finally, compile the graph with the provided checkpointer.
Notice the connectivity of the graph hasn't changed since Part 2. All we are doing is
checkpointing the State as the graph works through each node.
from langgraph.prebuilt import ToolNode, tools_condition
class State(TypedDict):
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
tool = TavilySearchResults(max_results=2)
tools = [tool]
llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")
llm_with_tools = llm.bind_tools(tools)
def chatbot(state: State):
return {"messages":
[llm_with_tools.invoke(state["messages"])]}
graph_builder.add_node("chatbot", chatbot)
tool_node = ToolNode(tools=[tool])
graph_builder.add_node("tools", tool_node)
graph_builder.add_conditional_edges(
"chatbot",
tools_condition,
)
# Any time a tool is called, we return to the chatbot to decide
the next step
graph_builder.add_edge("tools", "chatbot")
graph_builder.add_edge(START, "chatbot")
graph = graph_builder.compile(checkpointer=memory)
from IPython.display import Image, display
try:
display(Image(graph.get_graph().draw_mermaid_png()))
except Exception:
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 16/61
Now you can interact with your bot! First, pick a thread to use as the key for this
conversation.
Next, call your chat bot.
Note: The con?g was provided as the second positional argument when calling our
graph. It importantly is not nested within the graph inputs ( {'messages': []} ).
Let's ask a followup: see if it remembers your name.
# This requires some extra dependencies and is optional
pass
config = {"configurable": {"thread_id": "1"}}
user_input = "Hi there! My name is Will."
# The config is the **second positional argument** to stream() or
invoke()!
events = graph.stream(
{"messages": [("user", user_input)]}, config,
stream_mode="values"
)
for event in events:
event["messages"][-1].pretty_print()
================================ Human Message ===================
==============
Hi there! My name is Will.
================================== Ai Message ====================
==============
Hello Will! It's nice to meet you. How can I assist you today? Is
there anything specific you'd like to know or discuss?
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 17/61
Notice that we aren't using an external list for memory: it's all handled by the
checkpointer! You can inspect the full execution in this LangSmith trace to see
what's going on.
Don't believe me? Try this using a different con?g.
user_input = "Remember my name?"
# The config is the **second positional argument** to stream() or
invoke()!
events = graph.stream(
{"messages": [("user", user_input)]}, config,
stream_mode="values"
)
for event in events:
event["messages"][-1].pretty_print()
================================ Human Message ===================
==============
Remember my name?
================================== Ai Message ====================
==============
Of course, I remember your name, Will. I always try to pay attenti
on to important details that users share with me. Is there anythin
g else you'd like to talk about or any questions you have? I'm her
e to help with a wide range of topics or tasks.
# The only difference is we change the `thread_id` here to "2"
instead of "1"
events = graph.stream(
{"messages": [("user", user_input)]},
{"configurable": {"thread_id": "2"}},
stream_mode="values",
)
for event in events:
event["messages"][-1].pretty_print()
================================ Human Message ===================
==============
Remember my name?
================================== Ai Message ====================
==============
I apologize, but I don't have any previous context or memory of yo
ur name. As an AI assistant, I don't retain information from past
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 18/61
Notice that the only change we've made is to modify the thread_id in the con?g.
See this call's LangSmith trace for comparison.
By now, we have made a few checkpoints across two different threads. But what
goes into a checkpoint? To inspect a graph's state for a given con?g at any time,
call get_state(config) .
conversations. Each interaction starts fresh. Could you please tel
l me your name so I can address you properly in this conversation?
snapshot = graph.get_state(config)
snapshot
StateSnapshot(values={'messages': [HumanMessage(content='Hi there!
My name is Will.', additional_kwargs={}, response_metadata={}, id
='8c1ca919-c553-4ebf-95d4-b59a2d61e078'), AIMessage(content="Hello
Will! It's nice to meet you. How can I assist you today? Is there
anything specific you'd like to know or discuss?", additional_kwar
gs={}, response_metadata={'id': 'msg_01WTQebPhNwmMrmmWojJ9KXJ', 'm
odel': 'claude-3-5-sonnet-20240620', 'stop_reason': 'end_turn', 's
top_sequence': None, 'usage': {'input_tokens': 405, 'output_token
s': 32}}, id='run-58587b77-8c82-41e6-8a90-d62c444a261d-0', usage_m
etadata={'input_tokens': 405, 'output_tokens': 32, 'total_tokens':
437}), HumanMessage(content='Remember my name?', additional_kwargs
={}, response_metadata={}, id='daba7df6-ad75-4d6b-8057-745881cea1c
a'), AIMessage(content="Of course, I remember your name, Will. I a
lways try to pay attention to important details that users share w
ith me. Is there anything else you'd like to talk about or any que
stions you have? I'm here to help with a wide range of topics or t
asks.", additional_kwargs={}, response_metadata={'id': 'msg_01E41K
itY74HpENRgXx94vag', 'model': 'claude-3-5-sonnet-20240620', 'stop_
reason': 'end_turn', 'stop_sequence': None, 'usage': {'input_token
s': 444, 'output_tokens': 58}}, id='run-ffeaae5c-4d2d-4ddb-bd59-5d
5cbf2a5af8-0', usage_metadata={'input_tokens': 444, 'output_token
s': 58, 'total_tokens': 502})]}, next=(), config={'configurable':
{'thread_id': '1', 'checkpoint_ns': '', 'checkpoint_id': '1ef7d06e
-93e0-6acc-8004-f2ac846575d2'}}, metadata={'source': 'loop', 'writ
es': {'chatbot': {'messages': [AIMessage(content="Of course, I rem
ember your name, Will. I always try to pay attention to important
details that users share with me. Is there anything else you'd lik
e to talk about or any questions you have? I'm here to help with a
wide range of topics or tasks.", additional_kwargs={}, response_me
tadata={'id': 'msg_01E41KitY74HpENRgXx94vag', 'model': 'claude-3-5
-sonnet-20240620', 'stop_reason': 'end_turn', 'stop_sequence': Non
e, 'usage': {'input_tokens': 444, 'output_tokens': 58}}, id='run-f
feaae5c-4d2d-4ddb-bd59-5d5cbf2a5af8-0', usage_metadata={'input_tok
ens': 444, 'output_tokens': 58, 'total_tokens': 502})]}}, 'step':
4, 'parents': {}}, created_at='2024-09-27T19:30:10.820758+00:00',
parent_config={'configurable': {'thread_id': '1', 'checkpoint_ns':
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 19/61
The snapshot above contains the current state values, corresponding con?g, and the
next node to process. In our case, the graph has reached an END state, so next is
empty.
Congratulations! Your chatbot can now maintain conversation state across sessions
thanks to LangGraph's checkpointing system. This opens up exciting possibilities for
more natural, contextual interactions. LangGraph's checkpointing even handles
arbitrarily complex graph states, which is much more expressive and powerful than
simple chat memory.
In the next part, we'll introduce human oversight to our bot to handle situations
where it may need guidance or veri?cation before proceeding.
Check out the code snippet below to review our graph from this section.
'', 'checkpoint_id': '1ef7d06e-859f-6206-8003-e1bd3c264b8f'}}, tas
ks=())
snapshot.next # (since the graph ended this turn, `next` is
empty. If you fetch a state from within a graph invocation, next
tells which node will execute next)
()
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 20/61
Full Code
from typing import Annotated
from langchain_anthropic import ChatAnthropic
from langchain_community.tools.tavily_search import
TavilySearchResults
from langchain_core.messages import BaseMessage
from typing_extensions import TypedDict
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode
class State(TypedDict):
messages: Annotated[list, add_messages]
graph_builder = StateGraph(State)
tool = TavilySearchResults(max_results=2)
tools = [tool]
llm = ChatAnthropic(model="claude-3-5-sonnet-20240620")
llm_with_tools = llm.bind_tools(tools)
def chatbot(state: State):
return {"messages":
[llm_with_tools.invoke(state["messages"])]}
graph_builder.add_node("chatbot", chatbot)
tool_node = ToolNode(tools=[tool])
graph_builder.add_node("tools", tool_node)
graph_builder.add_conditional_edges(
"chatbot",
tools_condition,
)
graph_builder.add_edge("tools", "chatbot")
graph_builder.set_entry_point("chatbot")
graph = graph_builder.compile(checkpointer=memory)
10/3/24, 10:39 AM Quick Start
https://langchain-ai.github.io/langgraph/tutorials/introduction/#requirements 21/61
Basic Multi-agent Collaboration
A single agent can usually operate effectively using a handful of tools within a single
domain, but even using powerful models like gpt-4 , it can be less effective at using
many tools.
One way to approach complicated tasks is through a "divide-and-conquer" approach:
create an specialized agent for each task or domain and route tasks to the correct
"expert".
This notebook (inspired by the paper AutoGen: Enabling Next-Gen LLM Applications
via Multi-Agent Conversation, by Wu, et. al.) shows one way to do this using
LangGraph.
The resulting graph will look something like the following diagram:
Before we get started, a quick note: this and other multi-agent notebooks are
designed to show how you can implement certain design patterns in LangGraph. If
the pattern suits your needs, we recommend combining it with some of the other
fundamental patterns described elsewhere in the docs for best performance.
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 1/48
Setup
First, let's install our required packages and set our API keys:
Sign up for LangSmith to quickly spot issues and improve the performance of your
LangGraph projects. LangSmith lets you use trace data to debug, test, and monitor
your LLM apps built with LangGraph ? read more about how to get started here.
Create Agents
The following helper functions will help create agents. These agents will then be
nodes in the graph.
You can skip ahead if you just want to see what the graph looks like.
%%capture --no-stderr
%pip install -U langchain langchain_openai langsmith pandas
langchain_experimental matplotlib langgraph langchain_core
import getpass
import os
def _set_if_undefined(var: str):
if not os.environ.get(var):
os.environ[var] = getpass.getpass(f"Please provide your 
{var}")
_set_if_undefined("OPENAI_API_KEY")
_set_if_undefined("TAVILY_API_KEY")
Set up LangSmith for LangGraph development
from langchain_core.messages import (
BaseMessage,
HumanMessage,
ToolMessage,
)
from langchain_core.prompts import ChatPromptTemplate,
MessagesPlaceholder
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 2/48
De?ne tools
We will also de?ne some tools that our agents will use in the future
from langgraph.graph import END, StateGraph, START
def create_agent(llm, tools, system_message: str):
"""Create an agent."""
prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"You are a helpful AI assistant, collaborating
with other assistants."
" Use the provided tools to progress towards
answering the question."
" If you are unable to fully answer, that's OK,
another assistant with different tools "
" will help where you left off. Execute what you
can to make progress."
" If you or any of the other assistants have the
final answer or deliverable,"
" prefix your response with FINAL ANSWER so the
team knows to stop."
" You have access to the following tools: 
{tool_names}.\n{system_message}",
),
MessagesPlaceholder(variable_name="messages"),
]
)
prompt = prompt.partial(system_message=system_message)
prompt = prompt.partial(tool_names=", ".join([tool.name for
tool in tools]))
return prompt | llm.bind_tools(tools)
from typing import Annotated
from langchain_community.tools.tavily_search import
TavilySearchResults
from langchain_core.tools import tool
from langchain_experimental.utilities import PythonREPL
tavily_tool = TavilySearchResults(max_results=5)
# Warning: This executes code locally, which can be unsafe when
not sandboxed
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 3/48
Create graph
Now that we've de?ned our tools and made some helper functions, will create the
individual agents below and tell them how to talk to each other using LangGraph.
De?ne State
We ?rst de?ne the state of the graph. This will just a list of messages, along with a
key to track the most recent sender
repl = PythonREPL()
@tool
def python_repl(
code: Annotated[str, "The python code to execute to generate
your chart."],
):
"""Use this to execute python code. If you want to see the
output of a value,
you should print it out with `print(...)`. This is visible
to the user."""
try:
result = repl.run(code)
except BaseException as e:
return f"Failed to execute. Error: {repr(e)}"
result_str = f"Successfully
executed:\n```python\n{code}\n```\nStdout: {result}"
return (
result_str + "\n\nIf you have completed all tasks,
respond with FINAL ANSWER."
)
import operator
from typing import Annotated, Sequence
from typing_extensions import TypedDict
from langchain_openai import ChatOpenAI
# This defines the object that is passed between each node
# in the graph. We will create different nodes for each agent and
tool
class AgentState(TypedDict):
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 4/48
De?ne Agent Nodes
We now need to de?ne the nodes. First, let's de?ne the nodes for the agents.
messages: Annotated[Sequence[BaseMessage], operator.add]
sender: str
import functools
from langchain_core.messages import AIMessage
# Helper function to create a node for a given agent
def agent_node(state, agent, name):
result = agent.invoke(state)
# We convert the agent output into a format that is suitable
to append to the global state
if isinstance(result, ToolMessage):
pass
else:
result = AIMessage(**result.dict(exclude={"type",
"name"}), name=name)
return {
"messages": [result],
# Since we have a strict workflow, we can
# track the sender so we know who to pass to next.
"sender": name,
}
llm = ChatOpenAI(model="gpt-4o")
# Research agent and node
research_agent = create_agent(
llm,
[tavily_tool],
system_message="You should provide accurate data for the
chart_generator to use.",
)
research_node = functools.partial(agent_node,
agent=research_agent, name="Researcher")
# chart_generator
chart_agent = create_agent(
llm,
[python_repl],
system_message="Any charts you display will be visible by
the user.",
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 5/48
De?ne Tool Node
We now de?ne a node to run the tools
De?ne Edge Logic
We can de?ne some of the edge logic that is needed to decide what to do based on
results of the agents
De?ne the Graph
We can now put it all together and de?ne the graph!
)
chart_node = functools.partial(agent_node, agent=chart_agent,
name="chart_generator")
from langgraph.prebuilt import ToolNode
tools = [tavily_tool, python_repl]
tool_node = ToolNode(tools)
# Either agent can decide to end
from typing import Literal
def router(state):
# This is the router
messages = state["messages"]
last_message = messages[-1]
if last_message.tool_calls:
# The previous agent is invoking a tool
return "call_tool"
if "FINAL ANSWER" in last_message.content:
# Any agent decided the work is done
return END
return "continue"
workflow = StateGraph(AgentState)
workflow.add_node("Researcher", research_node)
workflow.add_node("chart_generator", chart_node)
workflow.add_node("call_tool", tool_node)
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 6/48
workflow.add_conditional_edges(
"Researcher",
router,
{"continue": "chart_generator", "call_tool": "call_tool",
END: END},
)
workflow.add_conditional_edges(
"chart_generator",
router,
{"continue": "Researcher", "call_tool": "call_tool", END:
END},
)
workflow.add_conditional_edges(
"call_tool",
# Each agent node updates the 'sender' field
# the tool calling node does not, meaning
# this edge will route back to the original agent
# who invoked the tool
lambda x: x["sender"],
{
"Researcher": "Researcher",
"chart_generator": "chart_generator",
},
)
workflow.add_edge(START, "Researcher")
graph = workflow.compile()
from IPython.display import Image, display
try:
display(Image(graph.get_graph(xray=True).draw_mermaid_png()))
except Exception:
# This requires some extra dependencies and is optional
pass
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 7/48
Invoke
With the graph created, you can invoke it! Let's have it chart some stats for us.
events = graph.stream(
{
"messages": [
HumanMessage(
content="Fetch the UK's GDP over the past 5
years,"
" then draw a line graph of it."
" Once you code it up, finish."
)
],
},
# Maximum number of steps to take in the graph
{"recursion_limit": 150},
)
for s in events:
print(s)
print("----")
{'Researcher': {'messages': [AIMessage(content='', additional_kwar
gs={'tool_calls': [{'id': 'call_3zDlnDMUkWEJxnHASo59doCL', 'functi
on': {'arguments': '{"query":"UK GDP 2018 to 2023"}', 'name': 'tav
ily_search_results_json'}, 'type': 'function'}]}, response_metadat
10/3/24, 10:33 AM Collaboration
https://langchain-ai.github.io/langgraph/tutorials/multi_agent/multi-agent-collaboration/ 8/48
Tutorials Conversational RAG
Conversational RAG
PREREQUISITES
This guide assumes familiarity with the following concepts:
Chat history
Chat models
Embeddings
Vector stores
Retrieval-augmented generation
Tools
Agents
In many Q&A applications we want to allow the user to have a back-and-forth conversation,
meaning the application needs some sort of "memory" of past questions and answers, and
some logic for incorporating those into its current thinking.
In this guide we focus on adding logic for incorporating historical messages. Further details
on chat history management is covered here.
We will cover two approaches:
?? Chains, in which we always execute a retrieval step;
?? Agents, in which we give an LLM discretion over whether and how to execute a retrieval
step (or multiple steps).
For the external knowledge source, we will use the same LLM Powered Autonomous Agents
blog post by Lilian Weng from the RAG tutorial.
Setup
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 1/27
Dependencies
We'll use OpenAI embeddings and a simple in-memory vector store in this walkthrough, but
everything shown here works with any Embeddings, and VectorStore or Retriever.
We'll use the following packages:
We need to set environment variable OPENAI_API_KEY , which can be done directly or loaded
from a .env file like so:
LangSmith
Many of the applications you build with LangChain will contain multiple steps with multiple
invocations of LLM calls. As these applications get more and more complex, it becomes
crucial to be able to inspect what exactly is going on inside your chain or agent. The best way
to do this is with LangSmith.
Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after
you sign up at the link above, make sure to set your environment variables to start logging
traces:
Chains
%%capture --no-stderr
%pip install --upgrade --quiet langchain langchain-community beautifulsoup4
import getpass
import os
if not os.environ.get("OPENAI_API_KEY"):
os.environ["OPENAI_API_KEY"] = getpass.getpass()
os.environ["LANGCHAIN_TRACING_V2"] = "true"
if not os.environ.get("LANGCHAIN_API_KEY"):
os.environ["LANGCHAIN_API_KEY"] = getpass.getpass()
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 2/27
Let's first revisit the Q&A app we built over the LLM Powered Autonomous Agents blog post
by Lilian Weng in the RAG tutorial.
OpenAI Anthropic Azure Google Cohere NVIDIA FireworksAI Gr
pip install -qU langchain-openai
import getpass
import os
os.environ["OPENAI_API_KEY"] = getpass.getpass()
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini")
import bs4
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
# 1. Load, chunk and index the contents of the blog to create a retriever.
loader = WebBaseLoader(
web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
bs_kwargs=dict(
parse_only=bs4.SoupStrainer(
class_=("post-content", "post-title", "post-header")
)
),
)
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, 
chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = InMemoryVectorStore.from_documents(
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 3/27
API Reference: create_retrieval_chain | create_stuff_documents_chain | WebBaseLoader |
ChatPromptTemplate | InMemoryVectorStore | OpenAIEmbeddings |
RecursiveCharacterTextSplitter
Note that we have used the built-in chain constructors create_stuff_documents_chain and
create_retrieval_chain , so that the basic ingredients to our solution are:
documents=splits, embedding=OpenAIEmbeddings()
)
retriever = vectorstore.as_retriever()
# 2. Incorporate the retriever into a question-answering chain.
system_prompt = (
"You are an assistant for question-answering tasks. "
"Use the following pieces of retrieved context to answer "
"the question. If you don't know the answer, say that you "
"don't know. Use three sentences maximum and keep the "
"answer concise."
"\n\n"
"{context}"
)
prompt = ChatPromptTemplate.from_messages(
[
("system", system_prompt),
("human", "{input}"),
]
)
question_answer_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)
response = rag_chain.invoke({"input": "What is Task Decomposition?"})
response["answer"]
"Task decomposition is the process of breaking down a complicated task into 
smaller, more manageable steps. Techniques like Chain of Thought (CoT) and 
Tree of Thoughts enhance this process by guiding models to think step by step 
and explore multiple reasoning possibilities. This approach helps in 
simplifying complex tasks and provides insight into the model's reasoning."
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 4/27
?? retriever;
?? prompt;
?? LLM.
This will simplify the process of incorporating chat history.
Adding chat history
The chain we have built uses the input query directly to retrieve relevant context. But in a
conversational setting, the user query might require conversational context to be
understood. For example, consider this exchange:
Human: "What is Task Decomposition?"
AI: "Task decomposition involves breaking down complex tasks into smaller and simpler
steps to make them more manageable for an agent or model."
Human: "What are common ways of doing it?"
In order to answer the second question, our system needs to understand that "it" refers to
"Task Decomposition."
We'll need to update two things about our existing app:
?? Prompt: Update our prompt to support historical messages as an input.
?? Contextualizing questions: Add a sub-chain that takes the latest user question and
reformulates it in the context of the chat history. This can be thought of simply as
building a new "history aware" retriever. Whereas before we had:
query -> retriever
Now we will have:
(query, conversation history) -> LLM -> rephrased query -> retriever
Contextualizing the question
First we'll need to define a sub-chain that takes historical messages and the latest user
question, and reformulates the question if it makes reference to any information in the
historical information.
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 5/27
We'll use a prompt that includes a MessagesPlaceholder variable under the name
"chat_history". This allows us to pass in a list of Messages to the prompt using the
"chat_history" input key, and these messages will be inserted after the system message and
before the human message containing the latest question.
Note that we leverage a helper function create_history_aware_retriever for this step, which
manages the case where chat_history is empty, and otherwise applies prompt | llm |
StrOutputParser() | retriever in sequence.
create_history_aware_retriever constructs a chain that accepts keys input and
chat_history as input, and has the same output schema as a retriever.
API Reference: create_history_aware_retriever | MessagesPlaceholder
This chain prepends a rephrasing of the input query to our retriever, so that the retrieval
incorporates the context of the conversation.
from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import MessagesPlaceholder
contextualize_q_system_prompt = (
"Given a chat history and the latest user question "
"which might reference context in the chat history, "
"formulate a standalone question which can be understood "
"without the chat history. Do NOT answer the question, "
"just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
[
("system", contextualize_q_system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
history_aware_retriever = create_history_aware_retriever(
llm, retriever, contextualize_q_prompt
)
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 6/27
Now we can build our full QA chain. This is as simple as updating the retriever to be our new
history_aware_retriever .
Again, we will use create_stuff_documents_chain to generate a question_answer_chain ,
with input keys context , chat_history , and input -- it accepts the retrieved context
alongside the conversation history and query to generate an answer. A more detailed
explaination is over here
We build our final rag_chain with create_retrieval_chain. This chain applies the
history_aware_retriever and question_answer_chain in sequence, retaining intermediate
outputs such as the retrieved context for convenience. It has input keys input and
chat_history , and includes input , chat_history , context , and answer in its output.
API Reference: create_retrieval_chain | create_stuff_documents_chain
Let's try this. Below we ask a question and a follow-up question that requires
contextualization to return a sensible response. Because our chain includes a
"chat_history" input, the caller needs to manage the chat history. We can achieve this by
appending input and output messages to a list:
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
qa_prompt = ChatPromptTemplate.from_messages(
[
("system", system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)
rag_chain = create_retrieval_chain(history_aware_retriever, 
question_answer_chain)
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 7/27
API Reference: AIMessage | HumanMessage
TIP
Check out the LangSmith trace.
Stateful management of chat history
NOTE
This section of the tutorial previously used the RunnableWithMessageHistory
abstraction. You can access that version of the documentation in the v0.2 docs.
As of the v0.3 release of LangChain, we recommend that LangChain users take
advantage of LangGraph persistence to incorporate memory into new LangChain
from langchain_core.messages import AIMessage, HumanMessage
chat_history = []
question = "What is Task Decomposition?"
ai_msg_1 = rag_chain.invoke({"input": question, "chat_history": 
chat_history})
chat_history.extend(
[
HumanMessage(content=question),
AIMessage(content=ai_msg_1["answer"]),
]
)
second_question = "What are common ways of doing it?"
ai_msg_2 = rag_chain.invoke({"input": second_question, "chat_history": 
chat_history})
print(ai_msg_2["answer"])
Common ways of task decomposition include using simple prompting techniques, 
such as asking for "Steps for XYZ" or "What are the subgoals for achieving 
XYZ?" Additionally, task-specific instructions can be employed, like "Write a 
story outline" for writing tasks, or human inputs can guide the decomposition 
process.
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 8/27
applications.
If your code is already relying on RunnableWithMessageHistory or
BaseChatMessageHistory , you do not need to make any changes. We do not plan on
deprecating this functionality in the near future as it works for simple chat applications
and any code that uses RunnableWithMessageHistory will continue to work as
expected.
Please see How to migrate to LangGraph Memory for more details.
We have added application logic for incorporating chat history, but we are still manually
plumbing it through our application. In production, the Q&A application will usually persist
the chat history into a database, and be able to read and update it appropriately.
LangGraph implements a built-in persistence layer, making it ideal for chat applications that
support multiple conversational turns.
Wrapping our chat model in a minimal LangGraph application allows us to automatically
persist the message history, simplifying the development of multi-turn applications.
LangGraph comes with a simple in-memory checkpointer, which we use below. See its
documentation for more detail, including how to use different persistence backends (e.g.,
SQLite or Postgres).
For a detailed walkthrough of how to manage message history, head to the How to add
message history (memory) guide.
from typing import Sequence
from langchain_core.messages import BaseMessage
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, StateGraph
from langgraph.graph.message import add_messages
from typing_extensions import Annotated, TypedDict
# We define a dict representing the state of the application.
# This state has the same input and output keys as `rag_chain`.
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 9/27
API Reference: BaseMessage | MemorySaver | StateGraph | add_messages
This application out-of-the-box supports multiple conversation threads. We pass in a
configuration dict specifying a unique identifier for a thread to control what thread is run.
This enables the application to support interactions with multiple users.
class State(TypedDict):
input: str
chat_history: Annotated[Sequence[BaseMessage], add_messages]
context: str
answer: str
# We then define a simple node that runs the `rag_chain`.
# The `return` values of the node update the graph state, so here we just
# update the chat history with the input message and response.
def call_model(state: State):
response = rag_chain.invoke(state)
return {
"chat_history": [
HumanMessage(state["input"]),
AIMessage(response["answer"]),
],
"context": response["context"],
"answer": response["answer"],
}
# Our graph consists only of one node:
workflow = StateGraph(state_schema=State)
workflow.add_edge(START, "model")
workflow.add_node("model", call_model)
# Finally, we compile the graph with a checkpointer object.
# This persists the state, in this case in memory.
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)
config = {"configurable": {"thread_id": "abc123"}}
result = app.invoke(
{"input": "What is Task Decomposition?"},
config=config,
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 10/27
The conversation history can be inspected via the state of the application:
)
print(result["answer"])
Task decomposition is the process of breaking down a complicated task into 
smaller, more manageable steps. Techniques like Chain of Thought (CoT) and 
Tree of Thoughts enhance this process by guiding models to think step by step 
and explore multiple reasoning possibilities. This approach helps in 
simplifying complex tasks and provides insight into the model's reasoning.
result = app.invoke(
{"input": "What is one way of doing it?"},
config=config,
)
print(result["answer"])
One way of doing task decomposition is by using simple prompting, such as 
asking the model, "What are the subgoals for achieving XYZ?" This method 
encourages the model to identify and outline the smaller tasks needed to 
accomplish the larger goal.
chat_history = app.get_state(config).values["chat_history"]
for message in chat_history:
message.pretty_print()
================================?[1m Human Message 
?[0m=================================
What is Task Decomposition?
==================================?[1m Ai Message 
?[0m==================================
Task decomposition is the process of breaking down a complicated task into 
smaller, more manageable steps. Techniques like Chain of Thought (CoT) and 
Tree of Thoughts enhance this process by guiding models to think step by step 
and explore multiple reasoning possibilities. This approach helps in 
simplifying complex tasks and provides insight into the model's reasoning.
================================?[1m Human Message 
?[0m=================================
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 11/27
Tying it together
For convenience, we tie together all of the necessary steps in a single code cell:
What is one way of doing it?
==================================?[1m Ai Message 
?[0m==================================
One way of doing task decomposition is by using simple prompting, such as 
asking the model, "What are the subgoals for achieving XYZ?" This method 
encourages the model to identify and outline the smaller tasks needed to 
accomplish the larger goal.
from typing import Sequence
import bs4
from langchain.chains import create_history_aware_retriever, 
create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.messages import AIMessage, BaseMessage, HumanMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, StateGraph
from langgraph.graph.message import add_messages
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 12/27
from typing_extensions import Annotated, TypedDict
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
### Construct retriever ###
loader = WebBaseLoader(
web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
bs_kwargs=dict(
parse_only=bs4.SoupStrainer(
class_=("post-content", "post-title", "post-header")
)
),
)
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, 
chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = InMemoryVectorStore.from_documents(
documents=splits, embedding=OpenAIEmbeddings()
)
retriever = vectorstore.as_retriever()
### Contextualize question ###
contextualize_q_system_prompt = (
"Given a chat history and the latest user question "
"which might reference context in the chat history, "
"formulate a standalone question which can be understood "
"without the chat history. Do NOT answer the question, "
"just reformulate it if needed and otherwise return it as is."
)
contextualize_q_prompt = ChatPromptTemplate.from_messages(
[
("system", contextualize_q_system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
history_aware_retriever = create_history_aware_retriever(
llm, retriever, contextualize_q_prompt
)
### Answer question ###
system_prompt = (
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 13/27
"You are an assistant for question-answering tasks. "
"Use the following pieces of retrieved context to answer "
"the question. If you don't know the answer, say that you "
"don't know. Use three sentences maximum and keep the "
"answer concise."
"\n\n"
"{context}"
)
qa_prompt = ChatPromptTemplate.from_messages(
[
("system", system_prompt),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
]
)
question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)
rag_chain = create_retrieval_chain(history_aware_retriever, 
question_answer_chain)
### Statefully manage chat history ###
class State(TypedDict):
input: str
chat_history: Annotated[Sequence[BaseMessage], add_messages]
context: str
answer: str
def call_model(state: State):
response = rag_chain.invoke(state)
return {
"chat_history": [
HumanMessage(state["input"]),
AIMessage(response["answer"]),
],
"context": response["context"],
"answer": response["answer"],
}
workflow = StateGraph(state_schema=State)
workflow.add_edge(START, "model")
workflow.add_node("model", call_model)
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 14/27
API Reference: create_history_aware_retriever | create_retrieval_chain |
create_stuff_documents_chain | WebBaseLoader | AIMessage | BaseMessage |
HumanMessage | ChatPromptTemplate | MessagesPlaceholder | InMemoryVectorStore |
ChatOpenAI | OpenAIEmbeddings | RecursiveCharacterTextSplitter | MemorySaver |
StateGraph | add_messages
Agents
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)
config = {"configurable": {"thread_id": "abc123"}}
result = app.invoke(
{"input": "What is Task Decomposition?"},
config=config,
)
print(result["answer"])
Task decomposition is the process of breaking down a complicated task into 
smaller, more manageable steps. Techniques like Chain of Thought (CoT) and 
Tree of Thoughts enhance this process by guiding models to think step by step 
and explore multiple reasoning possibilities. This approach helps in 
simplifying complex tasks and improving the model's performance.
result = app.invoke(
{"input": "What is one way of doing it?"},
config=config,
)
print(result["answer"])
One way of doing task decomposition is by using simple prompting, such as 
asking the model, "What are the subgoals for achieving XYZ?" This method 
encourages the model to identify and outline the smaller steps needed to 
complete the larger task.
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 15/27
Agents leverage the reasoning capabilities of LLMs to make decisions during execution.
Using agents allow you to offload some discretion over the retrieval process. Although their
behavior is less predictable than chains, they offer some advantages in this context:
Agents generate the input to the retriever directly, without necessarily needing us to
explicitly build in contextualization, as we did above;
Agents can execute multiple retrieval steps in service of a query, or refrain from
executing a retrieval step altogether (e.g., in response to a generic greeting from a user).
Retrieval tool
Agents can access "tools" and manage their execution. In this case, we will convert our
retriever into a LangChain tool to be wielded by the agent:
API Reference: create_retriever_tool
Tools are LangChain Runnables, and implement the usual interface:
from langchain.tools.retriever import create_retriever_tool
tool = create_retriever_tool(
retriever,
"blog_post_retriever",
"Searches and returns excerpts from the Autonomous Agents blog post.",
)
tools = [tool]
tool.invoke("task decomposition")
'Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple 
reasoning possibilities at each step. It first decomposes the problem into 
multiple thought steps and generates multiple thoughts per step, creating a 
tree structure. The search process can be BFS (breadth-first search) or DFS 
(depth-first search) with each state evaluated by a classifier (via a prompt) 
or majority vote.\nTask decomposition can be done (1) by LLM with simple 
prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving 
XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." 
for writing a novel, or (3) with human inputs.\n\nFig. 1. Overview of a LLM-
powered autonomous agent system.\nComponent One: Planning#\nA complicated task 
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 16/27
Agent constructor
Now that we have defined the tools and the LLM, we can create the agent. We will be using
LangGraph to construct the agent. Currently we are using a high level interface to construct
the agent, but the nice thing about LangGraph is that this high-level interface is backed by a
low-level, highly controllable API in case you want to modify the agent logic.
API Reference: create_react_agent
We can now try it out. Note that so far it is not stateful (we still need to add in memory)
usually involves many steps. An agent needs to know what they are and plan 
ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has 
become a standard prompting technique for enhancing model performance on 
complex tasks. The model is instructed to “think step by step” to utilize more 
test-time computation to decompose hard tasks into smaller and simpler steps. 
CoT transforms big tasks into multiple manageable tasks and shed lights into 
an interpretation of the model’s thinking process.\n\n(3) Task execution: 
Expert models execute on the specific tasks and log 
results.\nInstruction:\n\nWith the input and the inference results, the AI 
assistant needs to describe the process and results. The previous stages can 
be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model 
Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must 
first answer the user\'s request in a straightforward manner. Then describe 
the task process and show your analysis and model inference results to the 
user in the first person. If inference results contain a file path, must tell 
the user the complete file path.\n\nFig. 11. Illustration of how HuggingGPT 
works. (Image source: Shen et al. 2023)\nThe system comprises of 4 
stages:\n(1) Task planning: LLM works as the brain and parses the user 
requests into multiple tasks. There are four attributes associated with each 
task: task type, ID, dependencies, and arguments. They use few-shot examples 
to guide LLM to do task parsing and planning.\nInstruction:'
from langgraph.prebuilt import create_react_agent
agent_executor = create_react_agent(llm, tools)
query = "What is Task Decomposition?"
for event in agent_executor.stream(
{"messages": [HumanMessage(content=query)]},
stream_mode="values",
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 17/27
):
event["messages"][-1].pretty_print()
================================?[1m Human Message 
?[0m=================================
What is Task Decomposition?
==================================?[1m Ai Message 
?[0m==================================
Tool Calls:
blog_post_retriever (call_WKHdiejvg4In982Hr3EympuI)
Call ID: call_WKHdiejvg4In982Hr3EympuI
Args:
query: Task Decomposition
=================================?[1m Tool Message 
?[0m=================================
Name: blog_post_retriever
Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what 
they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting 
technique for enhancing model performance on complex tasks. The model is 
instructed to “think step by step” to utilize more test-time computation to 
decompose hard tasks into smaller and simpler steps. CoT transforms big tasks 
into multiple manageable tasks and shed lights into an interpretation of the 
model’s thinking process.
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning 
possibilities at each step. It first decomposes the problem into multiple 
thought steps and generates multiple thoughts per step, creating a tree 
structure. The search process can be BFS (breadth-first search) or DFS (depth-
first search) with each state evaluated by a classifier (via a prompt) or 
majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps 
for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-
specific instructions; e.g. "Write a story outline." for writing a novel, or 
(3) with human inputs.
(3) Task execution: Expert models execute on the specific tasks and log 
results.
Instruction:
With the input and the inference results, the AI assistant needs to describe 
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 18/27
the process and results. The previous stages can be formed as - User Input: {{ 
User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model 
Assignment }}, Task Execution: {{ Predictions }}. You must first answer the 
user's request in a straightforward manner. Then describe the task process and 
show your analysis and model inference results to the user in the first 
person. If inference results contain a file path, must tell the user the 
complete file path.
Fig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 
2023)
The system comprises of 4 stages:
(1) Task planning: LLM works as the brain and parses the user requests into 
multiple tasks. There are four attributes associated with each task: task 
type, ID, dependencies, and arguments. They use few-shot examples to guide LLM 
to do task parsing and planning.
Instruction:
==================================?[1m Ai Message 
?[0m==================================
Task Decomposition is a process used in complex problem-solving where a larger 
task is broken down into smaller, more manageable sub-tasks. This approach 
enhances the ability of models, particularly large language models (LLMs), to 
handle intricate tasks by allowing them to think step by step.
There are several methods for task decomposition:
1. **Chain of Thought (CoT)**: This technique encourages the model to 
articulate its reasoning process by thinking through the task in a sequential 
manner. It transforms a big task into smaller, manageable steps, which also 
provides insight into the model's thought process.
2. **Tree of Thoughts**: An extension of CoT, this method explores multiple 
reasoning possibilities at each step. It decomposes the problem into various 
thought steps and generates multiple thoughts for each step, creating a tree 
structure. The evaluation of each state can be done using breadth-first search 
(BFS) or depth-first search (DFS).
3. **Prompting Techniques**: Task decomposition can be achieved through simple 
prompts like "Steps for XYZ" or "What are the subgoals for achieving XYZ?" 
Additionally, task-specific instructions can guide the model, such as asking 
it to "Write a story outline" for creative tasks.
4. **Human Inputs**: In some cases, human guidance can be used to assist in 
breaking down tasks.
Overall, task decomposition is a crucial component in planning and executing 
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 19/27
We can again take advantage of LangGraph's built-in persistence to save stateful updates to
memory:
API Reference: MemorySaver
This is all we need to construct a conversational RAG agent.
Let's observe its behavior. Note that if we input a query that does not require a retrieval step,
the agent does not execute one:
Further, if we input a query that does require a retrieval step, the agent generates the input
to the tool:
complex tasks, allowing for better organization and clarity in the problem-
solving process.
from langgraph.checkpoint.memory import MemorySaver
memory = MemorySaver()
agent_executor = create_react_agent(llm, tools, checkpointer=memory)
config = {"configurable": {"thread_id": "abc123"}}
for event in agent_executor.stream(
{"messages": [HumanMessage(content="Hi! I'm bob")]},
config=config,
stream_mode="values",
):
event["messages"][-1].pretty_print()
================================?[1m Human Message 
?[0m=================================
Hi! I'm bob
==================================?[1m Ai Message 
?[0m==================================
Hello Bob! How can I assist you today?
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 20/27
query = "What is Task Decomposition?"
for event in agent_executor.stream(
{"messages": [HumanMessage(content=query)]},
config=config,
stream_mode="values",
):
event["messages"][-1].pretty_print()
================================?[1m Human Message 
?[0m=================================
What is Task Decomposition?
==================================?[1m Ai Message 
?[0m==================================
Tool Calls:
blog_post_retriever (call_0rhrUJiHkoOQxwqCpKTkSkiu)
Call ID: call_0rhrUJiHkoOQxwqCpKTkSkiu
Args:
query: Task Decomposition
=================================?[1m Tool Message 
?[0m=================================
Name: blog_post_retriever
Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what 
they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting 
technique for enhancing model performance on complex tasks. The model is 
instructed to “think step by step” to utilize more test-time computation to 
decompose hard tasks into smaller and simpler steps. CoT transforms big tasks 
into multiple manageable tasks and shed lights into an interpretation of the 
model’s thinking process.
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning 
possibilities at each step. It first decomposes the problem into multiple 
thought steps and generates multiple thoughts per step, creating a tree 
structure. The search process can be BFS (breadth-first search) or DFS (depth-
first search) with each state evaluated by a classifier (via a prompt) or 
majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps 
for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-
specific instructions; e.g. "Write a story outline." for writing a novel, or 
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 21/27
(3) with human inputs.
(3) Task execution: Expert models execute on the specific tasks and log 
results.
Instruction:
With the input and the inference results, the AI assistant needs to describe 
the process and results. The previous stages can be formed as - User Input: {{ 
User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model 
Assignment }}, Task Execution: {{ Predictions }}. You must first answer the 
user's request in a straightforward manner. Then describe the task process and 
show your analysis and model inference results to the user in the first 
person. If inference results contain a file path, must tell the user the 
complete file path.
Fig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 
2023)
The system comprises of 4 stages:
(1) Task planning: LLM works as the brain and parses the user requests into 
multiple tasks. There are four attributes associated with each task: task 
type, ID, dependencies, and arguments. They use few-shot examples to guide LLM 
to do task parsing and planning.
Instruction:
==================================?[1m Ai Message 
?[0m==================================
Task Decomposition is a technique used to break down complex tasks into 
smaller, more manageable steps. This approach is particularly useful in the 
context of autonomous agents and large language models (LLMs). Here are some 
key points about Task Decomposition:
1. **Chain of Thought (CoT)**: This is a prompting technique that encourages 
the model to "think step by step." By doing so, it can utilize more 
computational resources to decompose difficult tasks into simpler ones, making 
them easier to handle.
2. **Tree of Thoughts**: An extension of CoT, this method explores multiple 
reasoning possibilities at each step. It decomposes a problem into various 
thought steps and generates multiple thoughts for each step, creating a tree 
structure. This can be evaluated using search methods like breadth-first 
search (BFS) or depth-first search (DFS).
3. **Methods of Decomposition**: Task decomposition can be achieved through:
- Simple prompting (e.g., asking for steps to achieve a goal).
- Task-specific instructions (e.g., requesting a story outline for 
writing).
- Human inputs to guide the decomposition process.
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 22/27
Above, instead of inserting our query verbatim into the tool, the agent stripped unnecessary
words like "what" and "is".
This same principle allows the agent to use the context of the conversation when necessary:
4. **Execution**: After decomposition, expert models execute the specific 
tasks and log the results, allowing for a structured approach to complex 
problem-solving.
Overall, Task Decomposition enhances the model's ability to tackle intricate 
tasks by breaking them down into simpler, actionable components.
query = "What according to the blog post are common ways of doing it? redo 
the search"
for event in agent_executor.stream(
{"messages": [HumanMessage(content=query)]},
config=config,
stream_mode="values",
):
event["messages"][-1].pretty_print()
================================?[1m Human Message 
?[0m=================================
What according to the blog post are common ways of doing it? redo the search
==================================?[1m Ai Message 
?[0m==================================
Tool Calls:
blog_post_retriever (call_bZRDF6Xr0QdurM9LItM8cN7a)
Call ID: call_bZRDF6Xr0QdurM9LItM8cN7a
Args:
query: common ways of Task Decomposition
=================================?[1m Tool Message 
?[0m=================================
Name: blog_post_retriever
Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning 
possibilities at each step. It first decomposes the problem into multiple 
thought steps and generates multiple thoughts per step, creating a tree 
structure. The search process can be BFS (breadth-first search) or DFS (depth-
first search) with each state evaluated by a classifier (via a prompt) or 
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 23/27
majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps 
for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-
specific instructions; e.g. "Write a story outline." for writing a novel, or 
(3) with human inputs.
Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what 
they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting 
technique for enhancing model performance on complex tasks. The model is 
instructed to “think step by step” to utilize more test-time computation to 
decompose hard tasks into smaller and simpler steps. CoT transforms big tasks 
into multiple manageable tasks and shed lights into an interpretation of the 
model’s thinking process.
Resources:
1. Internet access for searches and information gathering.
2. Long Term memory management.
3. GPT-3.5 powered Agents for delegation of simple tasks.
4. File output.
Performance Evaluation:
1. Continuously review and analyze your actions to ensure you are performing 
to the best of your abilities.
2. Constructively self-criticize your big-picture behavior constantly.
3. Reflect on past decisions and strategies to refine your approach.
4. Every command has a cost, so be smart and efficient. Aim to complete tasks 
in the least number of steps.
(3) Task execution: Expert models execute on the specific tasks and log 
results.
Instruction:
With the input and the inference results, the AI assistant needs to describe 
the process and results. The previous stages can be formed as - User Input: {{ 
User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model 
Assignment }}, Task Execution: {{ Predictions }}. You must first answer the 
user's request in a straightforward manner. Then describe the task process and 
show your analysis and model inference results to the user in the first 
person. If inference results contain a file path, must tell the user the 
complete file path.
==================================?[1m Ai Message 
?[0m==================================
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 24/27
Note that the agent was able to infer that "it" in our query refers to "task decomposition", and
generated a reasonable search query as a result-- in this case, "common ways of task
decomposition".
Tying it together
For convenience, we tie together all of the necessary steps in a single code cell:
According to the blog post, common ways to perform Task Decomposition include:
1. **Simple Prompting**: Using straightforward prompts such as "Steps for 
XYZ.\n1." or "What are the subgoals for achieving XYZ?" to guide the model in 
breaking down the task.
2. **Task-Specific Instructions**: Providing specific instructions tailored to 
the task at hand, such as asking for a "story outline" when writing a novel.
3. **Human Inputs**: Involving human guidance or input to assist in the 
decomposition process, allowing for a more nuanced understanding of the task 
requirements.
These methods help in transforming complex tasks into smaller, manageable 
components, facilitating better planning and execution.
import bs4
from langchain.tools.retriever import create_retriever_tool
from langchain_community.document_loaders import WebBaseLoader
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langgraph.checkpoint.memory import MemorySaver
from langgraph.prebuilt import create_react_agent
memory = MemorySaver()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
### Construct retriever ###
loader = WebBaseLoader(
web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
bs_kwargs=dict(
parse_only=bs4.SoupStrainer(
class_=("post-content", "post-title", "post-header")
)
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 25/27
API Reference: create_retriever_tool | WebBaseLoader | InMemoryVectorStore |
ChatOpenAI | OpenAIEmbeddings | RecursiveCharacterTextSplitter | MemorySaver |
create_react_agent
Next steps
We've covered the steps to build a basic conversational Q&A application:
We used chains to build a predictable application that generates search queries for each
user input;
We used agents to build an application that "decides" when and how to generate search
queries.
To explore different types of retrievers and retrieval strategies, visit the retrievers section of
the how-to guides.
),
)
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, 
chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = InMemoryVectorStore.from_documents(
documents=splits, embedding=OpenAIEmbeddings()
)
retriever = vectorstore.as_retriever()
### Build retriever tool ###
tool = create_retriever_tool(
retriever,
"blog_post_retriever",
"Searches and returns excerpts from the Autonomous Agents blog post.",
)
tools = [tool]
agent_executor = create_react_agent(llm, tools, checkpointer=memory)
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 26/27
For a detailed walkthrough of LangChain's conversation memory abstractions, visit the How
to add message history (memory) guide.
To learn more about agents, head to the Agents Modules.
Edit this page
Was this page helpful?
You can also leave detailed feedback on GitHub.
10/3/24, 10:26 AM Conversational RAG | 🦜 🔗 LangChain
https://python.langchain.com/docs/tutorials/qa_chat_history/#agents 27/27