Neuro-Symbolic Approaches:Large Language Models + Tool Use
Tutorial on Complex Reasoning over Natural Language

Tao Yuhttps://taoyds.github.io/
University of Hong Kong
07/09/2023

Language models are powerful, but they still suffer from● Lack of interpretability● Inconsistency●Limited scalability● Restricted capabilities● …

About this tutorial

🙋: Can LLMs utilize external tools to not only expand theircapacities but also to make our NLP systems more robust,
scalable, and interpretable?

🎯: Neuro-symbolic approaches in the era of large language
models (LLMs): LLMs + tool use or tool augmented LLMs.

Augmented Language Models: a Survey
Tool Learning with Foundation Models
LLM Powered Autonomous Agents

Paper collection on LLM + tool use
https://github.com/xlang-ai/llm-tool-use

Tutorial agenda●●Motivations
Introduction to LLM + tool use○○●

Recent methods of LLM + tool use○○●

LLM prompting for tool use
LLM finetuning/pretraining for tool use

Other recent related work○○○●

LLM + tool use in the perspective of executable language grounding
LLM + tool use examplesTool making
Planning
Code generation

Challenges and future work

Human + tool use: motivations
As humans, we have limited time and memory,
feel tired, and have emotions.
● Human + tool use○ Enhanced scalability○ Improved consistency○ Greater interpretability○ Higher capacity and productivity●

https://www.tm-robot.com/en/why-you-should-automate-your-assembly-lines/
https://www.pbs.org/newshour/health/a-new-ruling-could-chip-away-at-the-affordable-care-act-most-in-the-u-s-see-health-care-as-a-basic-rightLLMs + tool use: motivations
Just like humans, LLMs suffer from the similar
limitations. But in the same way,
● LLMs + tool use○ Enhanced scalability○ Improved consistency○ Greater interpretability○ Higher capacity and productivity●

https://www.npr.org/2011/12/23/143833929/myth-busting-the-truth-about-animals-and-tools

Tutorial agenda●●Motivations
Introduction to LLM + tool use○○●

Recent methods of LLM + tool use○○●

LLM prompting for tool use
LLM finetuning/pretraining for tool use

Other recent related work○○○●

LLM + tool use in the perspective of executable language grounding
LLM + tool use examples

Tool making
Planning
Code generation

Challenges and future work

LLMs + tool use in perspective of executable language grounding
Ground language models into executable actions●Mapping natural language instructions into code or actions executable within various
environments such as databases, web applications, and robotic physical world.

●LM (planning and reasoning) + actions

Data analysis
https://openai.com/blog/chatgpt-plugins
https://code-as-policies.github.io/Web/Apps

Robotic physical world

LLMs + tool use in perspective of executable language grounding
LLMs + tool use in executable language grounding tasks
Inputs●Language: user question/request●

Toolkit: code, APIs to search engines, self-defined functions, expert models…

●

Environment: databases, IDE, web/apps, visual and robotic physical world…

Outputs●

Grounded reasoning code/action seq that can be executed in the corresponding
environment○

What tools to select, when and how to use the selected tools

Example of LLMs + tool use in executable language grounding
LLMs + tool use in executable language grounding●Language: user question/statement about a database●Toolkit: code, APIs to NLP functionalities (expert models)●Environment: databases, SQL/Python IDEs

Binder:Binding Language Models in Symbolic Languages

Project website: https://lm-code-binder.github.io, ICLR 2023

InputKnowledge:

Question:Binding Language Models in Symbolic Languages

Details● 100% cotton● 220GSM (6.5 oz)● Natural stretch for
comfortable fit…
● …..

Details● 90% cotton,10% polyester
● …
…

…

Shirt

Made_inSalesPrice

PoloU.S.

500

$29.8

Luckyland

U.S.

900

$19.8

Timber Bay

Mexico400$25.9

OfficeTurkey600$31.8Which is the best-selling shirt made in North America and with no chemicals?

LLM + no tool
Input
Details● 100% cotton● 220GSM (6.5 oz)● Natural stretch for
comfortable fit…
● …..

Knowledge:

Question:

Details● 90% cotton,
10% polyester● …
…

Made_in

Sales

Price

Polo

U.S.

500

$29.8

Luckyland

U.S.

900

$19.8

Timber Bay

Mexico400$25.9

OfficeTurkey600$31.8Which is the best-selling shirt made in North America and with no chemicals?

End-to-EndInput

GPT-3
Codex

Luckyland

Binding Language Models in Symbolic Languages

…

Shirt

❌ tool usePros●Cons●●●General❌Interpretable❌Scalable❌RobustLLM + no tool
Input
Details● 100% cotton● 220GSM (6.5 oz)● Natural stretch for
comfortable fit…
● …..

Knowledge:Question:

Details● 90% cotton,
10% polyester● …
…

Chain-of-Thought

Input

Input

GPT-3
Codex

GPT-3
Codex

Reasoning path
Polo, Luckyland, Timber Bay are from
North America. Polo and Office areof no chemicals. Luckyland has themost sales of 900. So the shirt isLuckyland.

✗Luckyland

Uninterpretable & not robust
Binding Language Models in Symbolic Languages

Made_inSales

Price

PoloU.S.

500

$29.8

Luckyland

U.S.

900

$19.8

Timber Bay

Mexico400$25.9

OfficeTurkey600$31.8Which is the best-selling shirt made in North America and with no chemicals?

End-to-End

Luckyland

…

Shirt

❌ tool use❌ tool use

Pros●Cons●

Improved but still…
❌Interpretable
Unreliable answer entailment●●❌Scalable❌RobustLLM + code
InputDetails● 100% cotton● 220GSM (6.5 oz)● Natural stretch for
comfortable fit…
● …..

Knowledge:Question:

Details● 90% cotton,
10% polyester
● …
…

Shirt

Made_inSales

Price

PoloU.S.

500

$29.8

Luckyland

U.S.

900

$19.8

Timber Bay

Mexico400$25.9

OfficeTurkey

600

$31.8

…

Which is the best-selling shirt made in North America and with no chemicals?

End-to-End

Chain-of-Thought

Semantic Parsing/Code Generation

Input

Input

Input

GPT-3
Codex

GPT-3
Codex

GPT-3
Codex

Reasoning path

SQL

Polo, Luckyland, Timber Bay are from
North America. Polo and Office areof no chemicals. Luckyland has themost sales of 900. So the shirt isLuckyland.

SELECT Shirt FROM T WHERE
Shirt NOT LIKE ‘%chemicals%’
AND Made_in = ‘North America’
ORDER BY Sales DESC LIMIT 1SQL/Python
Interpreter

Luckyland

Binding Language Models in Symbolic Languages✗Luckyland✗

Uninterpretable & not robust

Unreliable answer entailment❌ tool use❌ tool use<Empty>tool use: SQL/Python

Pros
●
●
●Cons●Interpretable
ScalableRobust❌CapableLLM + code and NLP expert function APIs
Input
Details● 100% cotton● 220GSM (6.5 oz)● Natural stretch for
comfortable fit…
● …..

Knowledge:Question:Details● 90% cotton,10% polyester
● …
…

Shirt

Made_in

Sales

Price

PoloU.S.

500

$29.8

Luckyland

U.S.

900

$19.8

Timber Bay

Mexico400$25.9

OfficeTurkey

600

$31.8

…

Which is the best-selling shirt made in North America and with no chemicals?

End-to-End

Chain-of-Thought

Semantic Parsing/Code Generation

LLM + tool use: Binder (ours)

Input

Input

Input

Input

GPT-3
Codex

GPT-3
Codex

GPT-3
Codex

GPT-3
Codex

Binder-SQL

Reasoning path

SQL

Polo, Luckyland, Timber Bay are fromNorth America. Polo and Office areof no chemicals. Luckyland has themost sales of 900. So the shirt isLuckyland.

SELECT Shirt FROM T WHERE
Shirt NOT LIKE ‘%chemicals%’
AND Made_in = ‘North America’
ORDER BY Sales DESC LIMIT 1

SQL/Python
Interpreter

Luckyland

Binding Language Models in Symbolic Languages✗Luckyland✗

Uninterpretable & not robust

Unreliable answer entailment❌ tool use❌ tool use<Empty>✗Limited PL grammar capacity

tool use: SQL/Python

SELECT Shirt FROM T WHERE
fCodex(“No chemicals?”;Shirt)=‘yes’ AND
fCodex(“North America?”;Made_in)=‘yes’
ORDER BY Sales DESC LIMIT 1

SQL/Python
InterpreterGPT-3
Codex

Polotool use:
SQL/Python + NLP expert model APIsWhy
LLM + tool use?
Pros●●●●Interpretable
Scalable
RobustCapableLLM + code: PAL, PoT

PAL: Program-aided Language Models
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

LLM + APIs to search/browser for gathering information
Retrieval augmented language models●

Covered in Yuchen and Michi’s sessions●

Another ACL tutorial by Akari Asai, Danqi Chen, Sewon Min, Zexuan Zhong

ChatGPT + browsing
WebGPT: Browser-assisted question-answering with human feedback

LLM + webs/apps or personalized functions

ChatGPT + Plugins: third-party apps/webs, Function calling

ReAct: Synergizing Reasoning and Acting in Language Models
https://openai.com/blog/chatgpt-plugins
Mind2Web: Towards a Generalist Agent for the Web

LLM + APIs to expert models

HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs

LLM + code, robotic arm, expert models: Code as Policies

Code as Policies: Language Model Programs for Embodied Control
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Mind's Eye: Grounded Language Model Reasoning through Simulation

Tutorial agenda●●Motivations
Introduction to LLM + tool use○○●

Recent methods of LLM + tool use○○●

LLM prompting for tool use
LLM finetuning/pretraining for tool use

Other recent related work○○○●

LLM + tool use in the perspective of executable language grounding
LLM + tool use examples

Tool making
Planning
Code generation

Challenges and future work

LLM prompting for tool use

PAL: Program-aided Language Models

LLM prompting for tool use
Knowledge:

Question:Details● 100% cotton● 220GSM (6.5 oz)● Natural stretch for
comfortable fit…
● …..

Details● 90% cotton,10% polyester
● …
…

…

Shirt

Made_in

Sales

Price

Polo

U.S.

500

$29.8

Luckyland

U.S.

900

$19.8

Timber Bay

Mexico400$25.9

OfficeTurkey600$31.8Which is the best-selling shirt made in North America and with no chemicals?

LLM + tool use: Binder (ours)
Input

Task/tool description
GPT-3Codex

Binder-SQL
SELECT Shirt FROM T WHERE
fCodex(“No chemicals?”;Shirt)=‘yes’ AND
fCodex(“North America?”;Made_in)=‘yes’
ORDER BY Sales DESC LIMIT 1

SQL/Python
Interpreter

PoloBinding Language Models in Symbolic Languages

GPT-3
Codex

environment description

User question
Output code bounded with
API callsLLM + tool use for QA and decision making: ReAct

observations

ReAct: Synergizing Reasoning and Acting in Language Models

LLM finetuning/pretraining for tool use: TALM

TALM: Tool Augmented Language Models

TALM: Tool Augmented Language Models
Toolformer: Language Models Can Teach Themselves to Use Tools
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningLLM finetuning/pretraining for tool use: Toolformer

TALM: Tool Augmented Language Models
Toolformer: Language Models Can Teach Themselves to Use Tools
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningLLM finetuning/pretraining for tool use: Toolformer

Toolformer: Language Models Can Teach Themselves to Use Tools
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Gorilla: Large Language Model Connected with Massive APIs

Tutorial agenda●●Motivations
Introduction to LLM + tool use○○●

Recent methods of LLM + tool use○○●

LLM prompting for tool use
LLM finetuning/pretraining for tool use

Other recent related work○○○●

LLM + tool use in the perspective of executable language grounding
LLM + tool use examples

Tool making
Planning
Code generation

Challenges and future work

Other recent related work: LLM as tool maker

LATM: Large Language Models as Tool Makers

Large Language Models as Tool Makers

Other recent related work: planning via classical resolver

LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
PDDL Planning with Pretrained Large Language Models

Other recent related work: code generation

CodeT: Code Generation with Generated Tests
Teaching Large Language Models to Self-Debug
DocPrompting: Generating Code by Retrieving the Docs
Coder Reviewer Reranking for Code Generation

Challenges and future work●

Complexity: more complex domain professional/unseen tools?

●Interactivity: go beyond single turn?

●

Evaluation: multiple possible solutions? Real-time interactive evaluation?

●

Efficiency: smaller models?

●

Reliability: know when to abstain, know its capacity, memorizing and querying tools?

●Others○

Better tool API design/tool making?

○

Personalization?

○……

Thank you!

General tutorial site

https://wenting-zhao.github.io/complex-reasoning-tutorial/

Paper collection on LLM + tool use
https://github.com/xlang-ai/llm-tool-use

ReferencesReferences
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsDo As I Can, Not As I Say: Grounding Language in Robotic Affordances
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
TALM: Tool Augmented Language Models
Inner Monologue: Embodied Reasoning through Planning with Language Models
JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied AgentsProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Code as Policies: Language Model Programs for Embodied Control
Binding Language Models in Symbolic Languages
Synergizing Reasoning and Acting in Language Models
Code4Struct: Code Generation for Few-Shot Event Structure Prediction
Mind's Eye: Grounded Language Model Reasoning through Simulation
PAL: Program-aided Language Models
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
Planning with Large Language Models via Corrective Re-promptingReferences
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]

Augmented Language Models: a Survey
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments
Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning
Toolformer: Language Models Can Teach Themselves to Use Tools
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World
Multi-Task Agents
Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning
Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control
PaLM-E: An Embodied Multimodal Language Model
ViperGPT: Visual Inference via Python Execution for Reasoning
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs
ART: Automatic multi-step reasoning and tool-use for large language models
API-Bank: A Benchmark for Tool-Augmented LLMs

References[31][32]
[33]
[34]
[35]
[36]
[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]
[45]
[46]
[47]

Tool Learning with Foundation Models
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information
LLM as A Robotic Brain: Unifying Egocentric Memory and Control
Voyager: An Open-Ended Embodied Agent with Large Language Models
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents
Large Language Models as Tool Makers
Gorilla: Large Language Model Connected with Massive APIs
On the Tool Manipulation Capability of Open-source Large Language Models
Making Language Models Better Tool Learners with Execution Feedback
Small models are valuable plug-ins for large language models
Hierarchical Prompting Assists Large Language Model on Web Navigation
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models

References
[48]
[49]
[50]
[51]
[52]
[53]
[54]
[55]
[56]
[57]
[58]
[59]
[60]
[61]
[62]

SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning
SQL-PaLM: Improved Large Language ModelAdaptation for Text-to-SQL
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Modular Visual Question Answering via Code Generation
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases
Mind2Web: Towards a Generalist Agent for the Web
LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
Search-in-the-Chain: Towards Accurate, Credible and Traceable Large Language Models for Knowledge-intensiveTasks
OpenAGI: When LLM Meets Domain Experts
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task
Planning
ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models
ToolCoder: Teach Code Generation Models to use API search tools
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Theory of Mind

Other proof-of-concept projects
●●
●
●
●
●
●Auto-GPT
LangChain Agent
ChatGPT plugins
BabyAGIGPT-Engineer
ToolBench, BMTools…