Language-based AI
Agents andLarge Action Models(LAMs)CVPR 2024 Tutorial “Generalist Agent AI”

Juan Carlos Niebles, Research Director@jcniebles | www.niebles.net
He/HimConversational AI Assistants
Work Agent
Hello I’m Einstein
Copilot your trusted AI
assistant

Can you help route mycase for approval?

I have modiﬁed thiscase status to Escalated
and created an
escalation request withthe subject “Modify
deliver center toPortland, OR”.

Personal Agent

Creative Agent

Let’s start booking mysummer vacation
travel. Find and book
family friendly hotel for
my upcoming stay inFrance..

I’m writing a ﬁction
story about animals in
the jungle. What aresome adventures
these animals might
have? Give me someideas and illustratethem with pictures.

I found this nice hotelnear the venues you
are planning to visit. I
booked 2 rooms for thefull duration of yourstay.

Sure, some ideas:
1. The animals mightorganizing a party.

[View Booking][Edit Case]
2. The animals mayneed to solve a
problem [...]

LLM Parameter countSource: https://www.predli.com/post/llm-deep-dive-phi-2

Language-based Agents

📝 Research
PapersHuang et al, Zero-shot Planners, 2022
Yao et al, ReAct, 2022
Schick et al, Toolformer, 2023
Shen et al, HuggingGPT, 2023
Paul et al, Reﬁner, 2023
Shinn et al, Reﬂection, 2023
Madaan et al, Self-reﬁne, 2023
Park et al, Generative Agents, 2023…

BabyAGI…

High-level Language-based Agent framework
Environment
ActionObservation

Task Instruction
Agent Response

Place an online
order for a newpair of blackrunning shoes
for meYour shoes will be
arriving next week
to your homeaddress.

LLM
Language-based AI Agent

Agent Benchmarking
What tasks are agents solving today?

❓HotPotQA - Knowledge Reasoning
https://hotpotqa.github.io/●●

113k question + answer pairs
Require multiple steps of reasoning and actions(Wikipedia article search, string lookup, ﬁnish)"User_query":"which magazine was started first: Arthur's Magazine or First for Women?","steps":"content": "Thought: I need to Search[Arthur's Magazine] and then
Search[First for Women] to collect their starting information.","next observation": "OK.""content": "Action: Search[Arthur's Magazine]","next observation": "Arthur's Magazine was published in 1844.""content": "Action: Search[First for Women]","next observation": "First for Women is a woman's magazine published in
1989."[Yang et al. “HotPotQA: A Dataset for Diverse, Explainable
Multi-hop Question Answering”. EMNLP 2018]"content": "Action: Finish[Arthur's Magazine]","next observation": "Answer is CORRECT."🛒 WebShop - Decision Making
https://webshop-pnlp.github.io/●●●

Simpliﬁed e-commerce shopping
environment containing 1.18M
real-world products
12k human instructions, groundtruth items
Agent receives single instruction fromuser, then performs multiple actions
to achieve the goal

[Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022]

🛒 WebShop - Decision Making
https://webshop-pnlp.github.io/●●●

Simpliﬁed e-commerce shopping
environment containing 1.18M
real-world products
12k human instructions, groundtruth items
Agent receives single instruction fromuser, then performs multiple actions
to achieve the goal

[Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022]

🛒 WebShop - Decision Making
https://webshop-pnlp.github.io/●●●

Simpliﬁed e-commerce shopping
environment containing 1.18M
real-world products
12k human instructions, groundtruth items
Agent receives single instruction fromuser, then performs multiple actions
to achieve the goal

[Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022]

🛒 WebShop - Decision Making
https://webshop-pnlp.github.io/●●●

Simpliﬁed e-commerce shopping
environment containing 1.18M
real-world products
12k human instructions, groundtruth items
Agent receives single instruction fromuser, then performs multiple actions
to achieve the goal

[Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022]

🛒 WebShop - Decision Making
https://webshop-pnlp.github.io/●●●

Simpliﬁed e-commerce shopping
environment containing 1.18M
real-world products
12k human instructions, groundtruth items
Agent receives single instruction fromuser, then performs multiple actions
to achieve the goal

[Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022]

🛒 WebShop - Decision Making
https://webshop-pnlp.github.io/
●

●
●

●

Simpliﬁed e-commerce shopping
environment containing 1.18M
real-world products
12k human instructions, groundtruth items
Agent receives single instruction fromuser, then performs multiple actions
to achieve the goal
Interaction is purely text-based[Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022]

🛠 ToolBench – Tool-use capability
https://github.com/OpenBMB/ToolBench[Qin et al “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs”, 2023]

🛠 ToolBench – Tool-use capability
https://github.com/OpenBMB/ToolBench
Resources

ToolBench

Number of tools

3,451

Number of APIs

16,461

Avg. Reasoning Traces

4

Number of evaluation queries

1,100

Evaluation metrics- Pass Rate
- Win Rate

[Qin et al “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs”, 2023]

📈 AgentBoard: evaluation framework
https://github.com/hkust-nlp/AgentBoard[Ma et al “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents”, 2023]

High-level Language-based Agent framework
Environment
ActionObservation

Task Instruction
Agent Response

Place an online
order for a newpair of blackrunning shoes
for meYour shoes will be
arriving next week
to your homeaddress.

LLM
Language-based AI Agent

Agent Architectures
How are we building Agents today?

High-level Language-based Agent framework
Environment
ActionObservation

Task Instruction
Agent Response

Place an online
order for a newpair of blackrunning shoes
for meYour shoes will be
arriving next week
to your homeaddress.

LLM
Language-based AI AgentHigh-level Language-based Agent framework
Environment
Action

Task Instruction

ActionParserAgent Response

LLM
PromptLanguage-based AI Agent

MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Architecture Complexity

Language-based AI Agents

Prompt-template
AgentsAgent Capability

Zero-shot Language-based Agent
Environment
Action

Task Instruction❄❄

Zero-Shot❄

LLM

Agent Response
MemoryPlace an online
order for a newpair of blackrunning shoes
for meActionParserObservation

Your shoes will be
arriving next week
to your homeaddress.

PromptLanguage-based AI Agent[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating LLM-augmented autonomous agents”. ICLR 2024 Workshop on LLM Agents]

Zero-shot Agent Example
System Prompt
You are an intelligent agent. You should follow your [Role], [Action_Doc] to take actions. Your generation
should follow the example format. Finish the task as best as you can.
[Role] You can interact with the webshop. [End of Role][Constraint] You generation should be simple and clear. [End of Constraint][Action_Doc] [{'name': 'search', 'description': '<fc_description>', 'parameters': {'product':'<param_description>'}}, {'name': 'click', 'description': '<fc_description>', 'parameters': {'button':'<param_description>'}}, {'name': 'Finish', 'description': '<fc_description>', 'parameters': {'response':'<param_description>'}}] [End of Action_Doc]
Using the following action format example to generate well formatted actions.
[ActionFormatExample]
Action:search[{"product": "the name of the product to search for"}]Action:Finish[{"response": "this is the ﬁnish action response. Respond towards the task instruction."}]Action:click[{"button": "the name of the button to click"}][Zhiwei Liu. “BOLAA: Benchmarking and orchestrating
LLM-augmented autonomous agents”. ICLR 2024
[End of ActionFormatExample]
Workshop on LLM Agents]

Zero-shot Agent Example
Task Instruction: I need a long clip-in hairextension which is natural looking, and pricelower than 40.00 dollars[Execution]
Action: search[{"product": "long clip-in hairextension"}][Environment]
Observation: WEB PAGE: {[Back to Search] Page 1 (Total results: 15)[Next >]
[B09C337K8S]
Hair Clip Extensions for Women Natural Looking Hair
Extensions clip in Hairpieces Black Synthetic Hair Extensions$29.99[B099K9Z9L2]
Clip in Hair Extensions 20" Double Weft 100% Remy
Human Hair Grade 8A Quality 7pcs 16clips 70grams $43.99}

Action: click[{"button": "B09C337K8S"}]

Action:Finish[{"response": "I found a long clip-in hairextension that is natural looking and priced lower
than $40.00. It is the Hair Clip Extensions for
Women Natural Looking Hair Extensions Clip in
Hairpieces Black Synthetic Hair Extensions, priced
at $29.99."}]Observation: WEB PAGE: {[Back to Search][< Prev]
Hair Clip Extensions for Women Natural Looking Hair Extensions Clip in
Hairpieces Black Synthetic Hair Extensions Price: $29.99
Rating: N.A.
[Buy Now]}[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating
LLM-augmented autonomous agents”. ICLR 2024
Workshop on LLM Agents]

Zero-shot Agent WebShop Results: Average Reward
LLM

LengthZero-Shot

fastchat-t5-3b

2k

0.40

vicuna-7b

2k

0.00

vicuna-13b

2k

0.03

vicuna-33b

2k

0.14

llama-2-7b

4k

0.00

llama-2-13b

4k

0.07

llama-2-70b

4k

0.01

mpt-7b-instruct

8k

0.00

mpt-30b-instruct

8k

0.17

xgen-8k-7b-instruct

8k

0.00

longchat-7b-16k

16k

0.02

longchat-13b-16k

16k

0.00

text-davinci-003

4k

0.53

gpt-3.5-turbo

4k

0.51

gpt-3.5-turbo-16k

16k

0.57

●
●
●

Bigger LLMs tend to achieve better agent
performance out of the box
Some LLMs struggle in Agent tasks in thezero-shot setting
Big gap open source <-> GPT models[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating
LLM-augmented autonomous agents”. ICLR 2024
Workshop on LLM Agents]

Zero-shot Agent HotPotQA Results
LLM

LengthZero-Shot

fastchat-t5-3b

2k

0.03

vicuna-7b

2k

0.13

vicuna-13b

2k

0.15

vicuna-33b

2k

0.22

llama-2-7b

4k

0.04

llama-2-13b

4k

0.17

llama-2-70b

4k

0.28

mpt-7b-instruct

8k

0.10

mpt-30b-instruct

8k

0.16

xgen-8k-7b-instruct

8k

0.15

longchat-7b-16k

16k

0.08

longchat-13b-16k

16k

0.11

text-davinci-003

4k

0.34

gpt-3.5-turbo

4k

0.33

gpt-3.5-turbo-16k-0613

16k

0.30

●

GPT models ahead again[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating
LLM-augmented autonomous agents”. ICLR 2024
Workshop on LLM Agents]

Zero-shot Language-based Agent
Environment
Action

Task Instruction❄❄

Zero-Shot❄

LLM

Agent Response
MemoryPlace an online
order for a newpair of blackrunning shoes
for meActionParserObservation

Your shoes will be
arriving next week
to your homeaddress.

PromptLanguage-based AI Agent[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating LLM-augmented autonomous agents”. ICLR 2024 Workshop on LLM Agents]

ReAct Agent
https://github.com/ysymyth/ReAct
Environment
Action

Task Instruction❄❄

Agent Response

Few-shot ❄LLM

MemoryThink

Place an online
order for a newpair of blackrunning shoes
for meActionParserObservation

PromptLanguage-based AI Agent[Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023]

Your shoes will be
arriving next week
to your homeaddress.

ReAct Agent Example
System Prompt
You are an intelligent agent. You should follow your [Role], [Action_Doc] to take actions. Your generation should follow
the example format. Finish the task as best as you can.
[Role] You can interact with the webshop. [End of Role][Constraint] You generation should be simple and clear. [End of Constraint][Action_Doc] [{'name': 'search', 'description': 'search for a product in the webshop', 'parameters': {'product': 'the name ofthe product to search for'}}, {'name': 'Finish', 'description': 'Complete the task with a response.', 'parameters': {'response':'this is the ﬁnish action response. Respond towards the task instruction.'}}, {'name': 'click', 'description': 'click a button in
the web page', 'parameters': {'button': 'the name of the button to click'}}, {'name': 'Think', 'description': 'Conduct thinking
and reasoning process for solving task.', 'parameters': {'response': 'this is your thinking response. Be speciﬁc andcritical.'}}] [End of Action_Doc]
Using the following action format example to generate well formatted actions.
[ActionFormatExample]
Action:search[{"product": "the name of the product to search for"}]Action:Finish[{"response": "this is the ﬁnish action response. Respond towards the task instruction."}]Action:click[{"button": "the name of the button to click"}][End of ActionFormatExample]

ReAct Agent Example[Example]
Task: i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00
dollars
Action: Think[{"response": "I should ﬁrst use Search to search bright citrus deodorant for sensitive skin."}]Observation: OK
Action: search[{"product": "bright citrus deodorant for sensitive skin"}]Observation: WEB PAGE: {[Back to Search][< Prev]
scent [assorted scents][bright citrus][calming lavender]
Action: click[{"button": "bright citrus"}]
Observation: "You have clicked bright citrus.
Action:click[{"button": "3 ounce (pack of 1)"}]
Observation: "You have clicked 3 ounce (pack of 1).
Action: click[{"button": "Buy Now"}]

size [3 ounce (pack of 1)]}

ReAct Agent Example
Task Instruction: I need a long clip-in hairextension which is natural looking, and pricelower than 40.00 dollars

Action: Think[{"response": "I should first use
Search to search for a long clip-in hairextension."}]Observation: WEB PAGE:[Back to Search] Page 1 (Total results: 15) [Next >][B09C337K8S]
Hair Clip Extensions for Women Natural Looking
Hair Extensions Clip in Hairpieces Black Synthetic Hair
Extensions $29.99[B093BKWHFK]MJMOJO Hair Extensions Clip in Human Hair
Extensions 120g 7pcs Straight Natural Black Hair Clip in(16 inches #NST) $63.99

Observation: OK

Action: search[{"product": "long clip-in hairextension"}]Action: Think[{"response": "The first product ismost relevant to the requirements. I shouldclick the first product."}]Observation: OK

ReAct Agent Example (continued)
Action: click[{"button": "B09C337K8S"}]

Observation: WEB PAGE:[Back to Search] [< Prev]
Hair Clip Extensions for Women
Natural Looking Hair Extensions Clip in
Hairpieces Black Synthetic Hair
Extensions Price: $29.99 Rating: N.A.
[Buy Now] }Action:click[{"button": "Buy Now"}]

Observation: shopping is finished.

Action: Finish[{"response": "Task
Finished."}]Observation: Task Finished. Reward:
1.0

ReAct Agent WebShop Results
LLM

Length Zero-Shot

ReAct

fastchat-t5-3b

2k

0.40

0.31

vicuna-7b

2k

0.00

0.10

vicuna-13b

2k

0.03

0.15

vicuna-33b

2k

0.14

0.19

llama-2-7b

4k

0.00

0.12

llama-2-13b

4k

0.07

0.26

llama-2-70b

4k

0.01

0.44

mpt-7b-instruct

8k

0.00

0.06

mpt-30b-instruct

8k

0.17

0.31

xgen-8k-7b-instruct

8k

0.00

0.07

longchat-7b-16k

16k

0.02

0.07

longchat-13b-16k

16k

0.00

0.24

text-davinci-003

4k

0.53

0.55

gpt-3.5-turbo

4k

0.51

0.54

gpt-3.5-turbo-16k

16k

0.57

0.49

●
●

ReAct helps smaller models boost theirperformance.
Big models/GPTs do not beneﬁt much.

ReAct Agent HotPotQA Results
LLM

LengthZero-Shot

ReAct

fastchat-t5-3b

2k

0.03

0.07

vicuna-7b

2k

0.13

0.03

vicuna-13b

2k

0.15

0.26

vicuna-33b

2k

0.22

0.26

llama-2-7b

4k

0.04

0.26

llama-2-13b

4k

0.17

0.25

llama-2-70b

4k

0.28

0.36

mpt-7b-instruct

8k

0.10

0.17

mpt-30b-instruct

8k

0.16

0.33

xgen-8k-7b-instruct

8k

0.15

0.19

longchat-7b-16k

16k

0.08

0.22

longchat-13b-16k

16k

0.11

0.24

text-davinci-003

4k

0.34

0.45

gpt-3.5-turbo

4k

0.33

0.32

gpt-3.5-turbo-16k-0613

16k

0.30

0.19

●
●
●

ReAct helps smaller models boost theirperformance.
Big models/GPTs do not beneﬁt much.
Model size more important than context
lengthReAct Agent
https://github.com/ysymyth/ReAct
Environment
Action

Task Instruction❄❄

Agent Response

Few-shot ❄LLM

MemoryThink

Place an online
order for a newpair of blackrunning shoes
for meActionParserObservation

PromptLanguage-based AI Agent[Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023]

Your shoes will be
arriving next week
to your homeaddress.

REX Agent Architecture
Environment
Action

Task Instruction❄

scoring❄❄
Prompt

LLM

Agent Response
MemoryPlace an online
order for a newpair of blackrunning shoes
for meActionParserObservation

Your shoes will be
arriving next week
to your homeaddress.

Language-based AI Agent[Murthy, et al. “REX: Rapid Exploration and eXploitation for AI agents”. ICLR 2024 LLM Agents Workshop.]REX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoTREX: UCB-CoTREX:
UCB-CoT

REX Results

REX {[Murthy, et al. “REX: Rapid Exploration and eXploitation for AI agents”. ICLR 2024 LLM Agents Workshop.]Architecture Complexity

Language-based AI Agents

Learnable-prompt
Agents
Prompt-template
AgentsAgent Capability

High-level Language-based Agent framework
Environment
Action

Task Instruction

ActionParserAgent Response

LLM
PromptLanguage-based AI Agent

MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Learnable-prompt Agents
Environment
Action

Task Instruction

ActionParser❄🔥

LLM

PromptLanguage-based AI Agent

Agent Response
MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Retroformer
https://github.com/weirayao/Retroformer
Environment
Action

Task Instruction

ActionParser❄
LLM

PromptRetrospective
LM

Agent Response🔥Persistent
MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Language-based AI Agent[Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024]

Retroformer: Store useful reﬂections[Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024]

Retroformer: Policy Gradient Optimization
Use Proximal Policy Optimization (PPO) to fine-tune the retrospective LM[Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024]

Retroformer: HotPotQA Result
Question Answering Task
Task Description● Solve a question answering task bysearching in Wikipedia pages.
● At each step, the agent is asked tochoose from three action APIs:- Search[entity]- Lookup[keyword]- Finish[answer]● Evaluation metric (success rate)measures if the answer and groundtruth matches exactly or not, over alltasks.

[Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024]

72

Retroformer: Alfworld & Webshop Results[Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024]

73

Retroformer takeaways: Reﬁne prompts, decompose
https://github.com/weirayao/Retroformer
Environment
Action

Task Instruction

ActionParser❄
LLM

PromptRetrospective
LM

Agent Response🔥Persistent
MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Language-based AI Agent[Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024]

Architecture Complexity

Language-based AI Agents

Large Action
Models (LAMs)
Learnable-prompt
Agents
Prompt-template
AgentsAgent Capability

Large Action Models (LAMs)
https://github.com/SalesforceAIResearch/xLAM
Environment
Action

Task Instruction

ActionParser🔥

LLM

Agent Response
MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

PromptLanguage-based AI Agent[Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024 Workshop
on LLM Agents]

Agent Trajectory Data Example
HotPotQA{{{"question":"Which magazine was started first Arthur's"question": "Which magazine was started first Arthur's"question":"Whichmagazine was started first Arthur's
Magazineor First
for Women?",Magazine or First for Women?",Magazineor FirstMagazine",for Women?","answer":"Arthur's"answer": "Arthur's Magazine","answer":"Arthur's Magazine","correct":true,"correct": true,"correct":"reward":1.0,true,"reward": 1.0,
"reward":1.0,"halted":false,"halted": false,"halted":false,"error": false,"error": false,"error": false,"prompt":""prompt": ""prompt":" Which magazine was started first Arthur's
Question:
Question: Which magazine was started first Arthur's
Question:Whichmagazine
wasstarted first Arthur's
Magazine or First for
Women?
Follow
Action[param]
Magazine or First for Women? Follow Action[param]
Magazineor FirstIfor
Women?
Follow Action[param]
format.\nThought:needto Search[Arthur's
Magazine] andformat.\n Thought: I need to Search[Arthur's Magazine] andformat.\nThought:
I need totoSearch[Arthur's
Magazine] andthen
Search[First
for Women]collect their starting
then Search[First for Women] to collect their starting
then Search[First
Women]
to collect their starting
information.\nModelfor
Action1: Search[Arthur's
Magazine]
information.\n Model Action 1: Search[Arthur's Magazine]
Model
Action 1:
Search[Arthur's
Magazine]\n information.\n
Env Observation
1: Arthur's
Magazine
was publishedin\n Env Observation 1: Arthur's Magazine was published in\n Env
Observation
Magazine
was published1844.
\n ModelAction 1:
2: Arthur's
Search[First
for Women]\n
Env in1844. \n Model Action 2: Search[First for Women]\n Env
1844. \n Model
Action2: Search[First
for Women]\nObservation
2: First
for Women
is a woman's
magazineEnv
Observation 2: First for Women is a woman's magazine
Observation
2:
First
for
Women
is
a
woman'smagazinepublished in 1989. \n Model Action 3: Finish[Arthur's
published in 1989. \n Model Action 3: Finish[Arthur's
published in
1989.
\n Model Action
3: Finish[Arthur's
Magazine]\n
Env
Observation
3: Answer
is CORRECT"
Magazine]\n Env Observation 3: Answer is CORRECT"} Magazine]\n Env Observation 3: Answer is CORRECT"}}WebShop[[[[[["Begin!\n\nUSER:
Can you send a GET request to link 1 and"Begin!\n\nUSER: Can you send a GET request to link 1 and"Begin!\n\nUSER:
Can
sendanothera GET request
to linkto
1 and
then
retrieve the cookie
byyou
sending
GET request
then retrieve the cookie by sending another GET request to
the cookiexxx.\nby sending
request to
linkthen2\n retrieve
Follow: Thought:
Action:anothername \nGET
Actionlink 2\n Follow: Thought: xxx.\n Action: name \n Action
link 2\n Follow:
Thought: xxx.\n Action: name \n Action
Input:{param}
format.",Input:{param} format.",
Input:{param}
format.",I need to send a request to set the"ASSISTANT
Thought:"ASSISTANT Thought: I need to send a request to set the"ASSISTANT
Thought:
I need to send a request to set thecookie.
\nASSISTANT
Action:cookie. \nASSISTANT Action:cookie.
\nASSISTANT
Action:sendHttpRequest\nASSISTANT Action Input:
sendHttpRequest\nASSISTANT Action Input:
sendHttpRequest\nASSISTANT
Input:{link1}\nASSISTANT
Observation:Action",{link1}\nASSISTANT Observation: ",{link1}\nASSISTANT
Observation:"Status
Code: 200.... Response
1\n", ",
"Status Code: 200.... Response 1\n","Status Code:200.... Response
1\n","ASSISTANT
Thought:
The request
was"ASSISTANT Thought: The request was"ASSISTANT Thought:Action:
The request was
successful.\nASSISTANT
successful.\nASSISTANT Action:
successful.\nASSISTANT Action: Action Input: {linkgetClientRequestData\nASSISTANT
getClientRequestData\nASSISTANT Action Input: {linkgetClientRequestData\nASSISTANT
Action Input: {link2}\nASSISTANT
Observation: ",2}\nASSISTANT Observation: ",2}\nASSISTANT
Observation:","Status Code: 200. ...\nASSISTANT Thought: ","Status Code: 200. ...\nASSISTANT Thought: ","StatusCode:
Thought:","The
cookiedata 200.
was ...\nASSISTANT
retrieved.\nASSISTANT
Response:"The cookie data was retrieved.\nASSISTANT Response:"The
cookiedata
was retrieved.\nASSISTANT Response:The
requestwassuccessful."
The request was successful."], The request was successful."],],[ false,true, false, true, false, true][ false, true, false, true, false, true]]] [ false, true, false, true, false, true]]]]][Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024 Workshop
on LLM Agents]

AgentOhana: Uniﬁed Diverse Training Data
https://github.com/SalesforceAIResearch/xLAM

Environment 1

Environment 2

Input:
Output:Observation:

Input:
Output:Observation:

AggrInput:
Output:Observation:...

Combiner

ShuﬄerAggrAgent Trajectory Standardizer...

CustomFilterEnvironment n

Public LLMs
Interleaver

Agent Rater
Multi-turn Environments

Model APIs

DataLoader[Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024 Workshop
on LLM Agents]

Training xLAM (Large Action Model)
Supervised Fine-tuning (SFT)🔥

Environment 1

Environment 2

AgentOhana
UniﬁedTraining
Trajectories

LLM

...
Environment n

Next token
prediction loss

Multi-turn Environments[Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024 Workshop
on LLM Agents]

xLAM: Results
https://github.com/SalesforceAIResearch/xLAM[Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024 Workshop
on LLM Agents]

xLAM: Results
https://github.com/SalesforceAIResearch/xLAM[Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024 Workshop
on LLM Agents]

Architecture Complexity

Language-based AI Agents

Multi-agent
Orchestration
Large Action
Models (LAMs)
Learnable-prompt
Agents
Prompt-template
AgentsAgent Capability

From Single Agents to Multiple Agents
https://github.com/SalesforceAIResearch/AgentLite

Equip agents with LLMs and Tools
Agent Manager

Deﬁne agent hierarchy: managers
and specialists
Lean implementation, easy-to-follow
tutorials[Liu. AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System. 2024]

BOLAA: Orchestrating multiple agents
https://github.com/salesforce/BOLAA
Environment

ActionObservation
Agent Pool

ActionParser

MemoryTask
Instruction

Agent
Selection
Agent Message
Controller

...

LLM

LLM

LLM

Agent
PromptAgent
PromptAgent
PromptAgent 1

Agent 2Agent n[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating LLM-augmented autonomous agents”. ICLR 2024 Workshop on LLM Agents]

BOLAA Multi-Agent WebShop Results
LLM

Length Zero-Shot

ReAct BOLAA

fastchat-t5-3b

2k

0.40

0.31

0.52

vicuna-7b

2k

0.00

0.10

0.06

vicuna-13b

2k

0.03

0.15

0.54

vicuna-33b

2k

0.14

0.19

0.56

llama-2-7b

4k

0.00

0.12

0.46

llama-2-13b

4k

0.07

0.26

0.37

llama-2-70b

4k

0.01

0.44

0.50

mpt-7b-instruct

8k

0.00

0.06

0.06

mpt-30b-instruct

8k

0.17

0.31

0.44

xgen-8k-7b-instruct

8k

0.00

0.07

0.37

longchat-7b-16k

16k

0.02

0.07

0.20

longchat-13b-16k

16k

0.00

0.24

0.32

text-davinci-003

4k

0.53

0.55

0.63

gpt-3.5-turbo

4k

0.51

0.54

0.66

gpt-3.5-turbo-16k

16k

0.57

0.49

0.65

●

●

The BOLAA multi-agent architecture
signiﬁcantly improves theperformance of smaller models.
GPT models also improve[Zhiwei Liu. “BOLAA: Benchmarking andorchestrating LLM-augmented
autonomous agents”. ICLR 2024
Workshop on LLM Agents]

BOLAA: Task Instruction Complexity on WebShop●

Task Complexity = number of attributes described in query[Zhiwei Liu. “BOLAA: Benchmarking and orchestrating LLM-augmented autonomous agents”. ICLR 2024 Workshop on LLM Agents]

Architecture Complexity

Language-based AI Agents

Multi-agent
Orchestration
Large Action
Models (LAMs)
Learnable-prompt
Agents
Prompt-template
AgentsAgent Capability

What’s next?
Interesting avenues to further explore

High-level Language-based Agent framework
Environment
Action

Task Instruction

ActionParserAgent Response

LLM
PromptLanguage-based AI Agent

MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Multimodality
Environment
Action

Task Instruction

ActionParserAgent Response

LLM
PromptLanguage-based AI Agent

MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

OSWorld
https://github.com/xlang-ai/OSWorld[Xie et al. Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. 2024]

Benchmarking & Environments
Environment
Action

Task Instruction

ActionParserAgent Response

LLM
PromptLanguage-based AI Agent

MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Learning & Generalization
Environment
Action

Task Instruction

ActionParser🔥

LLM
PromptLanguage-based AI Agent

Agent Response
MemoryPlace an online
order for a newpair of blackrunning shoes
for meObservation

Your shoes will be
arriving next week
to your homeaddress.

Resources & References
Selected papers & open source
implementations

Resources to Get Started
Salesforce AI Research Open Source Ecosystem

Architecture Complexity

Multi-agent
Orchestration
Large Action
Models (LAMs)
Learnable-prompt
Agents
Prompt-template
AgentsAgentLite: https://github.com/
SalesforceAIResearch/AgentLite

xLAM: https://github.com/
SalesforceAIResearch/xLAM

Retroformer:
https://github.com/weirayao/retroformer

BOLAA
https://github.com/salesforce/BOLAA

Agent Capability

References
Yang et al. “HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering”. EMNLP 2018
Yao et al “WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents”. NeurIPS 2022
Ma et al. “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents”, LLM Agents ICLR Workshop 2024
Zhiwei Liu. “BOLAA: Benchmarking and orchestrating LLM-augmented autonomous agents”. ICLR 2024 Workshop on
LLM Agents
Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023
Murthy, et al. “REX: Rapid Exploration and eXploitation for AI agents”. ICLR 2024 LLM Agents Workshop.
Yao et al. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. ICLR 2024
Zhang el al. The AgentOhana: Designing Uniﬁed Data and Training Pipeline for Eﬀective Agent Learning. ICLR 2024
Workshop on LLM AgentsLiu. AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System. 2024
Xie et al. “Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”. 2024
Huang et al, “Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents”, 2022
Schick et al, “Toolformer: Language Models Can Teach Themselves to Use Tools”, 2023
Shen et al, “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace”, 2023
Paul et al, “REFINER: Reasoning Feedback on Intermediate Representations”, 2023
Shinn et al, Reﬂexion: Language Agents with Verbal Reinforcement Learning, 2023
Madaan et al, Self-Reﬁne: Iterative Reﬁnement with Self-Feedback, 2023
Park et al, Generative Agents: Interactive Simulacra of Human Behavior, 2023

Thank you