Language agents:a critical evolutionary step ofartificial intelligenceYu Su
The Ohio State University

Language agents• Definition: autonomous agents, usually powered by large language
models, that can follow language instructions to carry out diverse andcomplex tasks in real-world or simulated environments.
• Examples: Auto-GPT, GPT-Engineer, Voyager, RT-2, and many others

Probably the most heated thread in AI
right now with massive public interests,e.g., Auto-GPT has received over 145kstars on GitHub within merely 4months, making it the fastest growing
repository
2But why?
• The concept of agent has been introduced in AI since
its dawn. It’s the first concept we teach in AI 101.
What’s different this time around?
• I argue that the most fundamental change is thecapability of using language.
• Contemporary AI agents use language as a vehicle forboth thought and communication, a trait that wasunique to humans.

Russel & Norvig, 2020

Therefore, these contemporary AI agents capable of using language
for thought and communication should be called “language agents,”
for language being their most salient trait.
https://www.demandsage.com/chatgpt-statistics/

3

Wei et al., 2022

Evolution of biological intelligence: an analogy

4

Language agents: a conceptual framework

5

Memory• Animals store memory through synaptic plasticity; artificial neuralnetworks also store information in their weights (≈ synaptic strength)• In-context learning ≈ working memory• Long-term memory: LLMs’ parametric memory or vector database?
• Why LLMs can understand and generate language? LLMs may haveinternalized a compact “linguistic representation” by compressing thepre-training corpus, similar to how humans learn and memorize• But can an LLM truly “understand” a concept (e.g., “apple”) without
actually “seeing” the physical object?

6

Textual twin world theory7

Tool augmentation• Solely relying on an LLM’s static parameters can only do limited
things. Tool augmentation brings unlimited possibilities.
• Three main purposes of tool augmentation• Complement language agents with specialized capabilities they may not
have, e.g., high-precision calculation and routing on a map• Provide up-to-date information, e.g., retriever and search engine• Enable language agents to take actions in real-world environments• Two types of tools: read-only vs. state-changing (i.e., w/ side effects)• Robustness and flexibility in using tools is key. None of the existing
systems, including ChatGPT Plugins, is sufficiently robust.
• Tools w/ side effects are riskier and require higher robustness.

8

Reasoning and planning• Reasoning is a continuum, not black and white. There has been too muchevidence for LLMs’ various reasoning capabilities. Simply arguing thatLLMs cannot reason at all may not be constructive discussion.
• Traditionally, reasoning and planning are relatively separate topics in AI.
• Reasoning: deductive (e.g., logic-based), inductive (e.g., ML)• Planning: Given start and goal states, an action space, find (the optimal) solutions• But now they are getting blended in language agents, consider, e.g., anembodied language agent acting in a partially-observable environment.
• New reasoning algorithms (e.g., chain/tree-of-thought) have emerged tounleash LLMs’ capability of using language for thought, but the truly
transformative reasoning algorithms for language agents are probably
yet to come9Grounding• Each environment is a unique context that provides possibly different
interpretations of natural language, which brings the challenge ofgrounding, i.e., linking of (natural language) concepts to contexts• Two types of grounding are central to language agents(Chandu et al., 2021)• #1: Ground natural language to an environment, e.g., mapping an utterance tothe right API call, SQL query, or robot plan
• #2: Ground an agent’s generation and decisions to its own context, including
external information generated by tools• Both remain challenging for current language agents• Some good news for #2? Our recent work finds that LLMs are highly receptive toexternal evidence, if we present it in a coherent and convincing way
Xie et al., 2023. Adaptive Chameleon or Stubborn Sloth: Unraveling the Behavior of Large Language
Models in Knowledge Clashes. arxiv preprint.

10

For the rest of the talk
• Don't Generate, Discriminate: A Proposal for Grounding Language Models toReal-World Environments
Yu Gu, Xiang Deng, Yu Su
ACL 2023 (Outstanding Paper Award)• Mind2Web: Towards a Generalist Agent for the Web

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su
Under Review• LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large
Language Models
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su
ICCV 2023

11

Grounded language understanding
Given a natural language utterance𝑢 and a target environment 𝐸𝜋: (𝑢, 𝐸) à 𝑝, s.t. 𝑢 ! = 𝑝 !
Where 𝑝 is a plan/program in a formal language, and " ! is the denotation

12

Grounded language understanding
Given a natural language utterance𝑢 and a target environment 𝐸
𝜋: (𝑢, 𝐸) à 𝑝, s.t. 𝑢 ! = 𝑝 !
Where 𝑝 is a plan/program in a formal language, and " ! is the denotation

𝑢: What is the latest released computer emulator developed in Java?
𝑝: (ARGMAX (AND ComputerEmulator(JOIN LanguagesUsed Java))
LatestReleaseDate)

13

Grounded language understanding
Given a natural language utterance𝑢 and a target environment 𝐸
𝜋: (𝑢, 𝐸) à 𝑝, s.t. 𝑢 ! = 𝑝 !
Where 𝑝 is a plan/program in a formal language, and " ! is the denotation

𝑢: Bring me a cup of coffee
𝑝: [turn left, move forward, pick up cup, turn around, move forward,…, put cup in coffee maker, toggle coffee maker, …]
14

ACL 2023 (Outstanding Paper Award)
Slides credit to Yu Gu

Pangu: A Unified Framework for Grounded
Language Understanding
Yu Gu, Xiang Deng, Yu Su
The Ohio State University

16

Q1: Find the right program over a KB
Question: Who has ever coached an ice hockey teamin Canada?
Program:A. (AND cricket.cricket_coach (JOIN cricket.cricket_team.coach_inv (JOINsports.sports_team.location Canada)))B. (AND ice_hockey.hockey_coach (JOIN ice_hockey.hockey_team.coach_inv (JOINsports.sports_team.location Canada)))C. (AND ice_hockey.hockey_team (JOIN sports.sports_team.location Canada))17

Q2: Write the corresponding KB program
Question: What's the classification of the M10 engine?
Program:(AND automotive.engine_type (JOIN automotive.engine_type.used_in M10))18

Why is Q2 harder?
You need to learn the grammar
You need to know the environment specifics

19

Key message
Generation

Directly generating plans (programs)
may not be the optimal way of using
LMs for grounded language
understanding
Discrimination20

Pangu:A unified framework that models
grounded language understanding as
a discrimination task
21

Our proposal: Pangu
Goals:§ Allow LMs to focus on
discrimination§ Generic for different tasks

A symbolic agent searches the environment to propose valid candidate
plans, while a neural LM scores the plans to guide the search process22

Algorithmic definition
Initialization of search

Propose candidate plansfrom the environment
Rank candidate plans using
a language model
Repeat until the termination
condition is met23

Instantiation for KBQA
Testbed:§ KBQA
45M entities
3B facts

LMs:§ BERT§ T5§ Codex24

New SoTA for KBQA
Prior Art

78.7

Pangu w/ BERT-base

79.9

Pangu w/ T5-base

79.9

Pangu w/ T5-3B

81.7

Prior Art

34.3

Pangu w/ BERT-base

52.0

Pangu w/ T5-base

53.3

Pangu w/ T5-3B

62.2

Prior Art

78.8

Pangu w/ BERT-base

77.9

Pangu w/ T5-base

77.3

Pangu w/ T5-3B79.6

F1 on GrailQA(i.i.d. + non-i.i.d., ~45Ktraining examples)

Findings:
F1 on GraphQuestions(non-i.i.d., ~2K trainingexamples)

Particularly strong performancefor non-i.i.d. generalization
Stable gain from increasedmodel size

F1 on WebQSP(i.i.d., ~3K trainingexamples)25

In-context learning with LLMs
Prior Art

78.7

Codex 10-shot

48.9

Codex 100-shot

53.3

Codex 1000-shot

56.4

Prior Art

34.3

Codex 10-shot

42.8

Codex 100-shot

43.3

Codex 1000-shot

44.3

Prior Art

78.8

Codex 10-shot

45.9

Codex 100-shot

54.5

Codex 1000-shot

68.3

F1 on GrailQA(i.i.d. + non-i.i.d., ~45Ktraining examples)

Findings:
F1 on GraphQuestions(non-i.i.d., ~2K trainingexamples)

SoTA performance on GraphQwith only 10 training examples
Marginal gain from moretraining data for non-i.i.d.

F1 on WebQSP(i.i.d., ~3K trainingexamples)26

https://osu-nlp-group.github.io/Mind2Web/

Mind2Web: Towards a Generalist Agent for the Web
Xiang Deng, Yu Gu, Boyuan Zheng,
Shijie Chen, Samuel Stevens, Boshi Wang,Huan Sun, Yu Su

28

Motivation●The modern web is very powerful. There’s a website for almosteverything.

●However, modern websites have also become very complex,incurring a steep learning curve and decreasing accessibility.

●A language agent can translate users’ mind to actions on the web,hence Mind2Web.

●On the other hand, such a web agent could also turn the entire
web into an unprecedentedly powerful and versatile tool for LLMs.
29

Mind2Web: the first dataset for generalist web agents●Desiderata for a generalist web agent○○○●

It shall work on any website on the Internet
It shall work on real-world websites, which can be dynamic, complex,and noisy
It shall support diverse and sophisticated interactions with websites.

Unique features of Mind2Web for generalist web agents○○○

Diverse coverage of domains, websites, and tasks: 2,000+ open-ended
tasks curated from 137 websites that span 31 different domains
Use of real-world websites: full traces of user interactions, webpage
snapshots, and network traffic are provided.
A broad spectrum of user interaction patterns: users can click, select,
and type in any elements on the website.
30

Data annotation

31MindAct: collaboration b/w small & large LMs

32

Results●●GPT-4 is particularly strong, close to fine-tuned Flan-T5
There’s still a substantial room for improvement towards generalist
web agentsEle. Acc: Element Selection Accuracy, Op. F1: Operation F1, SR: Success Rate (step-wise and end-to-end)

33

ICCV 2023

LLM-Planner: Few-Shot Grounded Planning for
Embodied Agents with Large Language Models
Chan Hee Song, Jiaman Wu, Clayton
Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su

Embodied language agents§ Embodied agents followinstructions to completetasks in physicalenvironments§ Diverse tasks (7) andenvironments (120)§ Long-horizon tasks: avg.
50 steps§ Can LLMs help?
35

Embodied agent planning with LLMs?
Instruction: “make me a cup of coffee”

LLM?

Low-level Plan: [turn left, move forward, pick up cup, turnaround, move forward, …, put cup in coffee maker, …]
36

Hierarchical planning with LLMs
Instruction: “make me a cup of coffee”
LLM-Planner
High-level Plan: [navigation cup, pick up cup, navigation
coffee machine, …]
Low-level planner
Low-level Plan: [turn left, move forward, pick up cup, turnaround, move forward, …, put cup in coffee maker, …]
37

Dynamic grounded planning
Instruction: “make me a cup of coffee”
LLM-Planner
High-level Plan: [navigation cup, pick up cup,navigation coffee machine, …]
Low-level planner
Low-level Plan: [Turn left, move forward, pick up cup,turn around, move forward, …, put cup in coffee
maker, …]
38

Evaluation on ALFRED• LLM-Planner achieves competitive performance with only 100 training examples• Existing methods can barely complete any task under the same low-data setting

SR: Success Rate, GC: Goal Completion Rate, HLP ACC: High-level Planning Accuracy

What’s the journey ahead of us?
• Is NLP dead/solved?
• Absolutely not. It’s the most exciting time for NLP ever!
• However, instead of natural language processing, perhaps weshould focus on natural language programming next41Natural language programming

Language Agent
42

Acknowledgements

43

Thanks &Embodiment• A lot of excitement in this space. LLMs are fully embraced by robotics.
• Generally, there are two threads: real-world vs. simulated• Real-world embodied agents (e.g., RT-2) need to operate with a lot of lowlevel constraints from current hardware, but leading to immediate apps
• Simulated embodied agents (e.g., Voyager) allow for research on moresophisticated learning and reasoning• Multimodal foundation models are key

46

Pangu Improves Sample Efficiency

47

Pangu vs. Constrained Decoding
Autoregressive models tend to overfit seen
structures during training
Highly Skewed48

https://arxiv.org/abs/2305.13300

Adaptive Chameleon or Stubborn Sloth:
Unraveling the Behavior of Large Language Models inKnowledge Clashes
Jian Xie*, Kai Zhang*,
Jiangjie Chen, Renze Lou, Yu Su

Transparency and Explainability: LLMs facing external information●●Parametric memory of an LLM isformed during pre-training. However,such static parametric memory may beinaccurate or become outdated.
External evidence provided by external
tools such as retrievers is a promisingsolution to augment LLMs with up-todate and accurate information (e.g., NewBing). Inevitably, such evidence may
conflict with parametric memory.
Question: How receptive are LLMs toexternal evidence?

Systematically elicit memory and simulate conflicts

Xie et al. https://arxiv.org/pdf/2305.13300.pdf

Supportive and contradictory evidence for control study

LLMs are highly
receptive(or gullible?)Only
this isshown• When only counter-memory is presented as
evidence, LLMs are very happy to change their mind
• However, only when the evidence is presented in a
coherent and convincing way• LLMs can be easily deceived by adversarial tool!

• When conflicting evidence (both supportive andLLMs show strong
confirmation bias

Bothareshownconflicting) is present, LLMs show a strong confirmation
bias and tend to cling to their parametric memory• Challenges for LLMs to unbiasedly orchestrate multiple
pieces of conflicting evidence, a common scenario faced
by generative search engines