AgentKit: Structured LLM Reasoning with Dynamic Graphs
Authors: Yue Wu
Yewen Fan
So Yeon Min
Shrimai Prabhumoye
Stephen Marcus McAleer
Ruslan Salakhutdinov
Yonatan Bisk
Yuanzhi Li
Tom Mitchell
Venue: COLM (2024)
Keywords: LLM Agents
TL;DR: An intuitive LLM prompting framework for multifunctional agents, by explicitly constructing a complex "thought process" from simple natural language prompts.

Abstract: We propose an intuitive LLM prompting framework (AgentKit) for multifunctional agents. 
AgentKit offers a unified framework for explicitly constructing a complex "thought process" from simple natural language prompts. The basic building block in AgentKit is a **node**, containing a natural language prompt for a specific subtask. The user then puts together chains of nodes, in order to build a "thought process" for any problem, like stacking LEGO pieces. The chains of nodes can be designed to explicitly enforce a naturally **structured** "thought process". For example, for the task of writing a paper, one may start with the thought process of 1) identify a core message, 2) identify prior research gaps, etc.
The nodes in AgentKit can be designed and combined in different ways to implement multiple advanced capabilities including on-the-fly hierarchical planning, reflection, and learning from interactions. 
In addition, due to the modular nature and the intuitive design to simulate explicit human thought process, a basic agent could be implemented as simple as a list of prompts for the subtasks and therefore could be designed and tuned by someone *without any programming experience*. Quantitatively, we show that agents designed through AgentKit achieve SOTA performance on Webshop and Crafter. These advances underscore AgentKit's potential in making LLM agents effective and accessible for a wider range of applications.

Published as a conference paper at COLM 2024
AgentKit: Structured LLM Reasoning with Dynamic Graphs
Yue Wu Yewen Fan', So Yeon Min', Shrimai Prabhumoye Stephen McAleer', Yonatan Bisk', Ruslan Salakhutdinov', Yuanzhi LiL3, Tom Mitchell
CCarnegie Mellon University, ANVIDIA, Microsoft, 4 *ootonn University M ywu5@andrew.cmm cmu.edu
Abstract
We propose an intuitive LLM prompting framework (AgentKit) for mul- tifunctional agents. AgentKit offers a unified framework for explicitly constructing a complex "thought process" from simple natural language prompts. The basic building block in AgentKit is node, containing a nat- ural language prompt for a specific sub-task. The user then puts together chains of nodes, like stacking LEGO pieces. The chains of nodes can be designed to explicitly enforce naturally structured "thought process". For example, for the task of writing paper, one may start with the thought process of 1) identify a core message, 2) identify prior research gaps, etc. The nodes in AgentKit can be designed and combined in different ways to implement multiple advanced capabilities, including on-the-fly hierarchical planning, reflection, and learning from interactions. In addition, due to the modular nature and the intuitive design to simulate an explicit human thought process, a basic agent could be implemented as simple as list of prompts for subtasks and therefore could be designed and tuned by someone without any programming experience. Quantitatively, we show that agents designed with AgentKit achieve SOTA performance in WebShop and Crafter. These advances underscore the potential of AgentKit in creating effective and accessible agents for wider range of applications. Github
1 Introduction
Recently, large language models (LLM) (Chowdhery et al., 2022; OpenAI, 2023; Manyika; Touvron et al., 2023; Jiang et al., 2023; Li et al., 2023; Paammar et al., 2024) have demonstrated remarkable performance in a variety of tasks, including embodied planning and acting (Wu et al., 2024a; Ahn et al., 2022; Du et al., 2023; Wang et al., 2023c; Wu et al., 2023a;b; Wang et al., 2023a), QA or dialogue (Ouyang et al., 2022; Hendrycks et al., 2020; Bubeck et al., 2023; Madaan et al., 2023), and general problem solving (Brown et al., 2020; Yao et al., 2022b; Shinn et al., 2023). However, two challenges remain to be overcome to apply LLMs to general real-world agent (Wooldridge& & Jennings, 1995; Laird et al., 1987; Russell, 2010) tasks.
The first challenge is adhering to procedural requirements. For example, a self-driving car must adhere strictly safety rules while making situational adaptations. The existing agent frameworks Sagnificant-Gravitaa Chase, 2022; Khattab et al., 2023; Shinn et al., 2023) do not follow explicit reasoning procedures. An error in one step can propogate and influence later steps (Yao et al., 2022b). Wu et al. (2024b) observes that simple chain-of-thoughhee et al., 2022) generate good actions in the first few steps of the Tower of Hanoi, but their performance quickly degrades in later steps. Code generation (Wang et al., 2023c;;;) attempts to address the first challenge by converting the procedural requirements into code.
agents (Wei
However, code-based agents present the second challenge: accessibility and ease of use. Code-based agents rely on hand-crafted API platformss (Wang et al., 2023a) specialized for the task and often require many code examples that may be hard to produce. On the other hand, humans can make use of or transfer procedural knowledge effectively by tracing a series of thoughts or instructions in natural language.

---

Published as a conference paper at COLM 2024
Write an Essay
Process Design
Prompt Design
LLM Execution
fron agentkät import Graph, BussMoce
Task --
graph Graph()
sustaski "waat
and LUM Agents rodel bosoudersuataskk subtaskl, grash, LPL FUNCTION agentkic, CENDOSE.
praph.aool_ rods (nodes)
Natural Longuage Prompts
rode2 Basellder suntask2, subitanid, graph, API FUNCTEON, agentket, praph..edt, rode (n00000
Pros Cons
List pros and cons...
essay graph,
"LUM Agents
subtusk3, gruph. rode (nooed)
Make outline
sor depresteccter behwnen nodie
Outline
grash.aed ode (suttassk sulitask)
Writing
Write an essay given ...
proph. wogph..oo_.os/sotooooa isubaskk.. subcaskki
russit graph evelustee) autpute dictisnary proept, arner pairs
UNK
N
V.S.
Exampla
Essay
Thought
IF... ELSE
Leariig
Examples
Traditional Prompting
AgentKit Prompting
Figure 1: A user breaks down task into subtasks (nodes) representing "thought process" and creates prompts for the subtasks (nodes). Subtasks (nodes) in AgentKit can be designed and assembled in different ways to achieve diverse functionalities, similar to LEGO pieces.
To close the gap, we introduce AgentKit, an LLM prompting framework designed for assem- bling simple natural language subtasks into a solution to complicated task. This design intuitively instructs an LLM to follow the predefined "thought process" by accomplishing all the subtasks before arriving at the final solution.
An agent AgentKit consists of standardized units called nodes. Each node is designed to complete a specific subtask by sequentially performing 1) preprocess input (aggregate out- puts from dependency nodes and an external database) and prompt the LLM, 2) postprocess the result for storage and use. For example, to operate self-driving vehicle, an agent may be designed to first explicitly list and predict the intentions of entities such as pedestrians or other vehicles [nodel], before deciding which driving action to take [node3]. Following customized "thought process", AgentKit provides the user with precise modular control over the overall problem solving process, without writing single line of code. This design also comes with the benefit of modular interpretability, allowing one to pin-point the error-causing subtask by examining the natural language outputs of each node.
To expand the capabilities, nodes and dependencies can be dynamically added and removed at inference time through coding, enabling complicated routing like ELSE branching or FOR...LOOPS. For example, when road condition is bad, self-driving agent can add subtask identify slippery roads [node2], before deciding the driving action [node3].
The set of dynamic nodes and dependencies naturally form a dynamic Directed A-cyclic Graph (DAG), with prompts as nodes and dependencies as edges. In our agent setting, we traverse the DAG computing the LLM results for each node in the topological order of the dynamic graph. We design one of the later nodes in the graph to directly output an environment action.
To demonstrate the full potential of AgentKit, we implement an agent for the Crafter (Hafner, 2021) game with several advanced capabilities: planning and dynamic goal prioritization; mistake identification and reflection; learning from experience. The agent consistently achieves SOTA performance and learns good knowledge base from the environment. To show the generalization of AgentKit, we port a cost-effective version of our Crafter prompt to web agent task and surpass SOTA by 5% on WebShop (Yao et al., 2022a).
Our key contribution is a framework for natural language "coding" of end-to-end multi- functional Al agents, with the ability to plan, reflect, and learn in challenging situations.
2

---

Published as a conference paper at COLM 2024
Input: outputs (02) from dependencies:1 12g
Input: outputs (0z, 0g) from dependencies: Rg.,
Inside n44
12
22
Inside n2
Compose
Compose
retrieve
C., concat(o www.Daateeeeeeeeeeeeeeeeeeeeeeee
744
LLM
LLM
Error message
15
store
Database DB
After-ques
71.6
After-query yes'ne from
Formet checking parsing
127
Output: string 04
72.1
Output: string
Temporarily add/remove node[s] based on LLM output
Figure 2: Each node in AgentKit takes outputs from its dependencies and outputs string to complete predefined subtask. The orange components (After-query) are optional and can be further customized with minimal programming through the AgentKit API. Left: The evaluation process inside node consists of compose and after-query. Right: Nodes can be dynamically added / removed during the inference time. For example, the after-query operation of иt adds conditional node /n- based yes/no nno answer from the LLM to the node query. This induces conditional branching.
AgentKit
To use AgentKit, the user first defines a "thought process" as of natural language subtasks (prompts) that depend on each other. Each subtask is presented to AgentKit as node, centered around a natural language prompt pt for the subtask. The nodes are linked together by the dependency specifications in directed acyelic graph (DAG), to implement different different logic and throught processes. Different arrangements of nodes could represent different functionalities, allowing the user to integrate various functionalities to build multifunctional agent (Figure (b)).
All nodes have access to database DB. With central database, the user pass task specifications, instructions, and current game observation to every node in the graph. The database also allows nodes to store and pass on permananet information (e.g. plans or knowledge base) to future steps.
In this section, explain the implementation details behind AgentKit and how it could be used, with a running example of a self-driving car. See the benefits of our codebase C.
2.1 Node Components
For each node v, the user defines subtask by specifying the prompt qu and a list of dependencies D. Inside the node, AgentKit runs built-in flow that preprocesses the input (Compose), queryies the LLM with a preprocessed input and prompt fo, and optionally postprocesses the output of the LLM (After-query). For example, node n4 can be designed to "Identify the intentions of other road users" " (left of Fig. 2).
2.1.1 Compose: Gathering, compiling, and formatting information into the prompt
The compose operation takes the outputs of all dependencies and, optionally, information from the database to make a single prompt for the LLM. AgentKit comes with default compose operation that can be used without programming. With the output from as O,,
C = concat (DB(" "tsdd..EEEYS"") (Oald e D),4,)
(1)
For example, the compose operation for n4 would gather the output of n1 - "List any road-users in your surrounding." (Figure 2).
The compose operation could be further customized by coding "compose function". This enables support for special system prompts and even RAG (Lewis et al., 2020).
3

---

Published as a conference paper at COLM 2024
2.1.2 Query/After-query: Parsing, formatting, modifying based on LLM outputs
The Query operation prompts the LLM with Cp to get (LM,( and an optional after-query operation parses re-formats LLM((2), and edits the database DB. For example, LLM C4), "infentions of other road users", may need to be post-processed into Json format and added to the database, as illustrated in Algorithm 1 and Figure 2.
Algorithm 1 After-query operation example
Ap < (LLM((o) result Parss(AA))
b initialize As with the LLM response
if result does not match format then
raise EnterQueryEroor/// Messsage for LLM") AgentKit catches the error and reties
end if
Update database DB if needed
Og + result
> set output to parsed result
2.2 Dynamic Components
To support advanced capabilities such as branching, AgentKit offers API for user to dynam- ically modify the DAG at inference time (Figure 2 right). All dynamic modifications are temporary and will be reverted at the end of a graph traversal pass. Note that modifications to nodes already evaluated in this pass are forbidden and will be automatically rejected. In the self-driving example, after identifying the intentions of road users, Ity could be the task of "Identify if any road user poses the risk of collision". the LLM output is "Yes", after-query of f1 could add node n that "adjust route in order to avoid the road users".
Temporarily removing edges/nodes. typical use case can be to conditionally skip a set of unnecessary nodes to save computation if the node results can be reused in this pass. For example, in section 3, if the plan does not require update, the planner questions are skipped.
Adding temporary nodes/edges. A typical use case is to create conditional branches. For example, a node could direct the flow by adding conditional edges and nodes based on "yee/no" answer from the LLM.
2.3 Dynamic Graph Traversal
We implement Kahn's Algorithm (Algorithm 2) to traverse the DAG of both static nodes and dynamically added/ removed nodes. Since there may exist more than one topological order for a given graph, the dynamic addition/ removal of components may result in non-deterministic or unexpected behaviors. Safeguards are put in place to catch potential unexpected behaviors (section A).
3 Example Agent Powered by AgentKit
We demonstrate an agent for the Crafter (Hafner, 2021) game with several advanced capa- bilities: hierarchical planning, reflection, and learning from interactions. At each step T, the agent takes the text description (0y) of the visual game observation and an instruction manual I (Appendix D) as input similar Wu et al. (2024b), and outputs an action index (RT) the number of times the action should be repeated (nT). All features are implemented with prompts on a single graph in AgentKit (complete list of prompts in Appendix F.1).
As illustrated in Figure 3 (a), at every step, the agent summarizes and keeps track of: 1) observation (OT TT ); 2) plan and reasoning (O) (O% 3; infer if the previous action succeeded Blyebss
based changes in the observation (OT Nssactionn

---

Published as a conference paper at COLM 2024
(a) Environment
recenis stops deservation summary
(s')
stapa obseeranno
stapo ooscrvation dp-s manual
All Gameplay Hattory
Subguais
under skill
Subgusie
Actin
(b) Planner
(d) Knowledge Base
(c) Reflection (evary
Fly
Bes-add
Visual Disccipttr
Actor Nodes
nreed
n
Rost of planner
SOU Seciiic
Subgesia stop T (Paumpla]:
Feedback ofr (Example)-
Knoww asdga Dhaw
"OOka anc Compaariin rxsett
Hissiki Thes shoer openterlyy thrccts such anaemb
Luxt
Rettreee
ingrraa Collect wold
Used (b) Planner
battrr now eg sathithr
Used Actor
L strpo obsenation Py
Used Planner
Skill 1s y Exploroiorresoo.... cressources
Gameplay Summary at stop
Nawigato rumrou mcllngs find spnoritc resturross
Used by Actor
Figure 3: Node names are abbreviated for space. (a) At every step in the game, three summary nodes (green) Rs-obs- Ms-plan Rssacctoon summarize the observation, plan, and action of the current step. (b) At step T, all planner nodes (blue) take 0r-1,07 and manual I as input, and output subgoals and a skill s". Preflect reflects the summary of the 25 most recent steps, and "ccallleggr fggate determines whether the subgoals for (T- 1) are carried updated. (c) Every 3 steps under skill s",, (Hifed purple) reflects on all gameplay history under s and generates skill specific feedback for the planner (b). (d) Every step
T, akkb-add (gray) examines 0T and to identify new information from Lunk- Nunknown adds Lunk by identifying missing information from I for the current sub-goal.
3.1 Hierarchical Planner with Short-term Reflection (Figure 3 b)
At step T, the LLM first generates summary and reflection (Hteeftect) of the previous 25 steps:
MLLM (concat (Concct(O' " Oneppan Of Oigpaction ){ (T-
25,11}) (T-25,T1
Preflect
(2)
The LLM is then prompted to identify the top challenge of 3 of the most important high-level challenges (node Mchallenge). Then node Agaat determines the sub-goals from the previous step (T 1) need to be updated based on Ortigtaat Ortaullnnge and the "completion criteria".
an update is not required, the planner and knowledge base are skipped directly to the Actor nodes, saving more than 1/3 the number of tokens. Otherwise, node Hsubgoal prompts the LLM to identify 3 subgoals in the order of execution, each paired with "completion criteria".
Using information on the top sub-goal, the agent keeps track of a list of "skills" (Nakils) that prompts the LLM to retrieve and create new skills build skill library S similar Wang et al. (2023a), implemented as Python dictionary containing LLM generated attributes like: skill name, skill description, etc. See Figure 3 (b) and Appendix E.2 for details.
3.2 Long Term Reflection (Figure 3
Every 3 steps under a skill E S, we compute skill-specific feedback by prompting the LLM with all previous steps under skill
T Ffeed
M..LM (concat (onncat(O
" "yntt "eyplam action Iss s 9teed)
O
(3)
The feedback Offeed is provided as context to both the actor and the planner.
3.3 Learning from Interaction with a Knowledge Base (Figure 3 d)
The agent uses two carefully designed prompts (nodes): Hunknown/ Mkb-add, to maintain an unknown information list Lunk and a knowledge base KB.
5

---

Published as a conference paper at COLM 2024
Method
Score
Reward
Costs
Human Experts
50.5 + 6.8%
14.3 2.3
N/A
AgentKit (Model mix")
20.64%+
12.8 - 2.1
$138
SPRING (GPT-4) (Wu et al., 2023b)
27.3 + 1.2%
12.3 - 0.7
$243*
DreamerV3 (Hafner et al., 2023)
14.5 1.6%
11.7 1.9
1M steps
EDE (Jang et al., 2022)
11.7 1.0%
N/A
1M steps
DreamerV2 (Hafner et al., 2020)
10.0 1.2%
9.0
1M steps
SPRING (GPT-4-turbo
official repo)
7.74%
7.4
$85
ELLM (Du et al., 2023)
N/A
6.0
5M steps
Rainbow (Hessel et al., 2018)
4.3 1 0.2%
5.0
1M steps
PPO (Schulman et al., 2017)
4. 0.3% N/A
4.2
1M steps
CoT (Wei et al., 2022) (GPT-4-turbo)
1.9
$20
Plan2Explore (Sekar et al., 2020)
2.1 2.1 0.1%
2.1
1M steps
RND (Burda al., 2018)
2.0 0.1%
0.7
1M steps
Random
1.6 0.0%
2.1
0
Table 1: Table comparing AgentKit and popular RL algorithms in terms of reward, and training steps. The results for AgentKit summarized over 3 independent trials. "Our AgentKit implementation uses a combination of GPT-4-0613, GPT-4-turbo, and GPT-3.5- turbo to balance cost and performance. Score is computed on group of 3, we did not run multiple groups for std. *Cost of GPT-4 estimated from token counts of GPT-4-turbo.
At each step T, add prompts the LLM to identify the information from Lank by reasoning with the last two steps of observation or 0T- Indentified information is stored in the knowledge base KB. Then, flunknown prompts the LLM to list missing information (from the instruction manual and knowledge base) required for the current sub-goal. The list of missing information is then added to Lunk. We show learned KB in Appendix E.3
Experimental Results
We present our experiments as follows. First, we use Crafer (Hafner, 2021) for benchmarking and analysis of different modules, described in section 3. We then describe our experimental results for WebShop (Yao et al., 2022a) to demonstrate the generalization of AgentKit.
4.1 Crafter
Crafter (Hafner, 2021) is open-world survival game featuring procedural generation, designed to benchmark RL algorithmn.. The game includes a tech tree with 22 achievements that span levels. The original observation of the game is top-down view of the shape (7 x 9), with a discrete action space comprising 17 options. Drawing inspiration from Minecraft, the game presents similar challenges including crafting, exploration, survival, and progressing the tech-tree. Following Du et al. (2023); Wu et al. (2023b; 2024b), we provide the text description and instructions from Wu et al. (2023b) as input the agent. The text description includes: 4 blocks to the agent's 's immediate surrounding, the closest objects of each kind within view, the agent vitals, and list of remaining achievements. We show an example description in Appendix F.1. Additionally, we allow the agent to output apeaalnnumber to repeat the chosen action without requerying the agent. This feature reduces the cost by 1/3 average.
Notably, the ыРTEX source code (Hafner, 2021), from which Wu et al. (2023b) extrated the instructions, omitted many crucial details about the actual game, since the research paper was never intended to be a game manual. Omitted details include: specific amount of materials required to craft table (2 woods) and furnance (4 stones), damage/ hamage/attackk attack range of weapons or enemies. Such an omission poses a great challenge to first-time human players, making the game essentially unsolvable without trying and learning from the environment.

---

Published as a conference paper at COLM 2024
4.1.1 Crafter Results
We compare the performance of AgentKit with SPRING (Wu et al., 2023b) and RL baselines'. Note that the original version of SPRING uses GPT-4, which is expensive to Crafter. The GPT-4-turbo is significantly cheaper and features improved reproducibility and instruc- tion following². Although consistency and instruction following is generally considered desirable feature, it increases the risk of getting stuck, for example, in situations when the instruction manual is inaccurate. Also, we observe that GPT-4-turbo is worse at spatial reasoning compared to GPT-4. We hypothesize that the two differences explained above are the cause of the deteriorated performance with GPT-4-turbo (Table row 7).
In contrast to SPRING, AgentKit features active reflection and learning capabilities, along with explicit planning. Therefore, our agent does not rely on randomness to get past unexpected situations (section 4.1 4.1.2). Overall, our agent achieves 80% improvement in terms of reward compared to SPRING (GPT-4-turbo) and attains reward comparable to SPRING (GPT-4) while being 45% cheaper?. The consistency of our agent over SPRING is also reflected in the score metric, which favors low unlock rates for more achievements over consistently high unlock rates for less achievements (Hafner, 2021). Empirically, we observe that SPRING tend to randomly unlock less important achievements i.e. Eat Plant, Collect Sampling, etc. Such behavior improves the score metric but does not affect the reward.
4.1.2 Planning, Reflection, and Learning from Interactions
We show an illustrated example of the first 11 steps taken by our Crafter agent in Figure 4.1.1. The agent started empty handed in the environment (step 0) and decided to approach the challenge nº. of "Resource Collection". Given the challenge, it generated the sub-goal of "Move toward tree". After reaching the tree at step 6, the short-term reflection node nł and the planer node nf detects the completion of the sub-goal, and updated the plan and action to collect wood and place table. Simultaneously, "TableeWodCConsumpiii as missing from the manual and added to the unknown information list (Lunk) by Munknown
was identified
However, due to the omitted information (Hafner, 2021) the amount of wood required for the table, the agent attempts to place the table with 1 wood and fails (step 8). The observation nodes (o ) automatically detect the failure by comparing the current observation with the expected action outcomes. The observation nodes then referred to Lunk and the manual I and identified "Insufficient wood" to be the cause of failure. The plan and actions are updated accordingly to gather more wood before placing the table. Finally, "TonseWWoddConsumpiio"" is discovered as 2 and added to the knowledge base KB (step 11).
Finally, the long-term reflection node eeed identifies failed movement attempts due to obstacles and insufficient resource collection. The planner is suggested to "ensure flat ground before movement" and "prioritize wood collection".
4.2 Webshop
Creating autonomous agents for the Web could bring convenience to our lives. We demon- strate zero-shot agent for Webshop (Yao et al., 2022a), a simulated virtual shopping environment. A shopping task in Webshop involves searching, browsing, and identifying the desired products on a simulated e-Commerce platform. Due to the specific nature of the shopping task, all prior LLM agents rely on few-shot demonstrations of complete human trajectories (Yao et al., 2022b; Shinn et al., 2023; Liu et al., 2023) to guide agent behavior. Since AgentKit can customize agents with nodes, our agent does not require demonstrations.
Inspired by the agent design for Crafter (section 3), we design a simple 6-node agent with short-term reflection and planning similar to the Crafter agent. We include the complete
'The 1M cap for number of interactions is set by the benchmark developers (Hafner, 2021)
2 pplatform.opena..com /occ/ssss/////////////////// docs/ models/ Cost calculated based on official API pricing as of Mar 2024: appema..oom/prcingg
7

---

Published as a conference paper at COLM 2024
Step 0:
Agent Reasoning (snippets):
Features
Distribation actions taken each
Osservatin (t-tep):
grass:
mł Challengee: Glver the chellerge that requiree the moon othention Resource Collection.
K Knowledge Base: amppy
gress (sacing))
Near objects gid):
Subgoat: Move towarde the necrest tree to collect wood Quantificaioo:... next to tres, raady coliact wood.
-rea stepk to aouthweist
Furt Unknown Into List *how musii wead per oction TimeCrrf CergyCoot
"ree to south ecst
Actor: altar moving south and to soprocimate nct path towerds tree
89 21
Hove
Virie 15 Vee
Step 6:
of Short term reflection ... history has been focused on moving (owarts the meorest tree .. Pot of the must recent obse vation, the player is have successfully new gated towarcs the tree... no reported mistakes or oostacies that were not effectivvll addressed
Skill: NavigateTo Targett Navigite towardss spwerffed target in the game weridl Terght: Trew
PhaceSbject
-Oseevation (t st-step::
-greses west
moudh Beezing Near objects grd): stopp....
"t previous subgowl:... Updated Subgoat:
subgool complette....
100,2N
Woud
trew
Quantification heving least wood
Colecflcaurce
Step 7:
Do setion
Actor: The player should use the 'D' oction to
nG previous subgoal:.... the subgool ls complete... Taktte Table" ochievement being unlecked
Knowledge Bese: wooo per 'Do' action
Do
76.2%
Observation (1- st-tppp
Updated Subgool: Quantitication
grass: west
Luxk Unknnwnn
Culiiamn
graas: snuth (tocing) Near-ty objects gridl
Actor: ... lake action "Place Tabke" ...
TaseeWoodCansmmphh
TinuD Enw Coot
ca Tatio sorte
n6-, previous action:
Inventoryy /wooc:
current step does not cate the presence of table... place, Lutile lailed, couueet .. Insullisent wood .n.
Skill: PlaceObject
100,00
Place a speortee object in the game world.
Updated Subgoal:
Crilict Wood
Step |Action Falled:
(1stepp
Quartification hoving icost units wood
Knowledge Base
60 0S
srass:
"5" previous subgiaal: the sulggoel camplleie. ... Updated Subgost: Pacee a Tazka using ... wood.
TassWooocanssmpioon
wood per Do action
De
gross: south (tacing)
Neerr-ip try otjjects (foP giuls
Quantification: "able" schievement being untocked...
Luna Unknown |EmeGrEniegCooo
Haxigetell Safel ocation
inwwnttrry:
Actor: take action "Place Table"
wood::
Oolinc Wer
Place
Step 10
Long Long-term Reflection nfeed
Step Ceservvttio i1 st00)) grasse wwst
100/2
-oservation (1 ststep::
Folled novement atterpt. Ensure the ground attempting
-graso: weat
lentted direcien unrecessary on fellures.
Rnodenthhaccela get
Near objects (7X5 gid):
grass: south (lacngg)
2. Inefffiiint reamuros cxllact on now crllecting woce from trees by too ng them directly and using the 'Dur action to eff isiuntly guther cssemtaall or ariiingg materists
lekket suaalh (lacing) objects 17x8
inventony: wOooes
Finventor: empty
Figure 4: Left three columns: an example trajectory in Crafter. Different nodes on planning reflection, feedback, knolwedge discovery work tögether to complete the first 11 steps and successfully crafting the table. Through environment interactions and error identification// /- correction, the agent discovered two pieces of information regarding "wood per Do action" and "table wood consumption"" originally omitted by the instructions (Hafner, 2021). Right column: the end-of-game distribution of all actions (classified into categories of Move, Do Interact, Craft) taken by the agent, for each skill in the skill library. The action distribution aligns well with human expectations based on skill names.
prompts in Appendix F22 As shown in Table 2, our zero-shot agent achieves a 5% perfor- mance improvement compared to SOTA few-shot agent, ReAct (Yao et al., 2022b).
Method
Act (Yao et 2122))
ReAct
AgentBench
AgentKit GPT44tubo
AgentKit GPT-4-633
(Yao el al,, 2022b)
al., 2023)
PaLM-540B
PaLM-5406
Best of GPT-4/35
Score
62.3%
66.6%
64.1%
69.4%
70.2%
Table 2: Table comparing our agent against baselines on WebShop (Yao et al., 2022a). For the sake of cost, we report the score on the first 100 samples of WebShop. Our agent achieves SOTA performance using both GPT-4 and the lower-cost GPT-4-turbo.
Related Works
5.1
LLM for Planning
Early attempts to use LLM for agent tasks often involve high-level planning. Huang et al. (2022); Ahn et al. (2022) proposed generating abstract high-level plans for embodied tasks,
8

---

Published as a conference paper at COLM 2024
with limited action space and trajectory length. Song et al. (2022); Wu et al. (2023a) improves Ahn et al. (2022), introducing more complexity in the task and the planner. Du et al. (2023); Wang et al. (2023c); Yuan et al. (2023) adopts similar LLM planner framework in the domain of open-world games like Crafter and Minecraft. Prior works along this line all share the same prompt design of providing expert human trajectory examples as context for the LLM. The output of the LLM is used as static high-level plan, independent of the agent's s situation.
5.2 LLM Agents
In more interactive setting, the LLM typically has access to some status information of the agent and can react to environment signals. Yao et al. (2022b) solves simple natural language tasks through LLM interaction with chain-of-thoughe prompts. Shinn et al. (2023)
further adds the ability to read task feedback signals. Wang et al. (2023a) plays Minecraft with an LLM that creates, retrieves, and refines code from a skill library. Wang et al. (2023b)
uses VLM for high-level planning and re-planning in Minecraft. However, the actual generated plans are often less than 10 stpes. In addition, the LLM operates independent from the rest of the agent (i.e., low-level controllers execute the plan and the LLM planner receives success/ fail message) and cannot actively react to the environment signals. This design limits the agent in Minecraft to peace_ peace_mode mode (Wang et al., 2023b) due to the inability to react to survival situations, an essential part of Crafter (Hafner, 2021).
Notably, all above works rely few-shot prompting. Although few-shot examples are useful for enforcing specific output formats, they tend to become long and unhelpful for complex tasks. Even writing high-level householding requires up to 17 full demonstra- tions (Ahn et al., 2022), making it costly for long-horizon tasks such as Crafter (Wu et al., 2023b).
5.3 Structured prompting for LLMs
Wei et al. (2022) first showed the benefit of explicit reasoning through thain-of-thought prompts. Yao et al. (2023); Besta et al. (2023) demonstrated success in guiding traditional search algorithms with LLM scoring functions, but the applications are limited to search- based problems. Khattab et al. (2023) parameterize prompts with trainable NN layers, optimized for specific metrics. Chase (2022); Beurer-Kellner et al. (2023) offers pre-built implementations of different prompting frameworks under a unified code base, but intro- duces lot of coding hasle. Unlike existing works that identify particular structure of structured prompting (Yao et al., 2022b; Shinn et al., 2023; Madaan et al., 2023) (e.g,, the sequence generate, self-evaluate, self-reflect), we present a modular perspective (Andreas et al., 2016) of designing and building agents from simple subtasks, with mostly intuitive natural language specification.
The most similar to AgentKit is Wu et al. (2023b), which uses LLM reasoning by traversing a fixed DAG to achieve SOTA performance in Crafter. Our framework adds dynamic com- ponents, pre-/post-proceee capabilities allow us to implement additional features like hierarchical planning, reflection, and knowledge base.
and database on top of Wu et al. (2023b). The additional
6 Conclusion
In this work, we present a framework (AgentKit) that allows a user to intuitively create specific "thought process" for complex problem by assembling prompts for subtasks. AgentKit evaluates LLM for each prompt in the order defined by the dependencies to achieve consistent and structured reasoning. While the basic features in AgentKit can be used without coding experience (Figure G), custom Python preprocess and postprocess functions can enable advanced features such database, conditional branching, and output formatting / parsing. To demonstrate the power and versatility of AgentKit, we implement WebShop (Yao et al., 2022a) agent and Crafter (Hafner, 2021) agent, both achieving

---

Published as a conference paper at COLM 2024
SOTA performance. We further demonstrate advanced capabilities that include on-the-fly hierarchical planning, reflection, and learning from interactions in the Crafter agent.
Our work demonstrates intuitive way to prompt LLM for challenging tasks by explicitly writing a human thought process into the prompts. Given the strong results, we envision AgentKit early demonstration of "coding" end-to-end multifunctional AI agents to solve challenging tasks, even for humans, with natural language.
7 Limitations
Two main limitations remain for AgentKit: Increased LLM querying cost and manual efforts for process design and prompt creation. Majumder et al. (2023); Nottingham et al. (2024) propose to learn prompts from environment rewards. However, experiments are conducted with limited task length ( (<= steps), and scaling the learning to long-horizon tasks featuring environment noise and concurrent objectives remains an open problem.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Hin Huee Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not as i say: Grounding language in robotic affordances, 2022. URL https::/arriiii https::/arrdv.org//b///2222 01691.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48, 2016.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski,, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXivv2308.0966 096627, 2023.
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: query language for large language models. Proceedings of the ACM on Programming Languages, 7 194.D1)1946-1169, 2023.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint ar arXio:2005.14165 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Farly experiments with gpt-4. arviv preprint ar Xttx:2303.12212 2023.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.1220 2018
Harrison Chase. LangChain, October 2022. URL langchain.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arviv preprint ar N20:2220.02311,
2022.
10

---

Published as a conference paper at COLM 2024
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with lange language models. arXiv preprint 0rX6::200.00662, 2023.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. arviv preprint
(#Xiv:2109.06780, 2021.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint ar arXio:2010.02193, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04004 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ar Xiv preprint arXiv:2009.0333 2020.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference artificial intelligence, 2018
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022. URL https://arxiv..or//bbl 207200
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. Xiv preprint ar drXiv:2310.06825, 2023.
Yiding Jiang, Zico Kolter, and Roberta Raileanu. Uncertainty -rriven exploration for generalization in reinforcement learning. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Arthur Kahn. Topological sorting of large networks. Communicctions of the ACM, 5(11): 558-562, 1962.
Omar Khattab, Arnv Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint ar Xiv:2310.03714, 2023.
John E Laird, Allen Newell, and Paul Rosenbloom. Soar: An architecture for general intelligence. Artificial intelligence, 33(1):1-64, 1987.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for snowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-94777 9474, 2020.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. Xiu preprint arXiv:2309.05463 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. In The Twelfth International Conference Learning Representations, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In
11

---

Published as a conference paper at COLM 2024
A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 46534-46594. Curran Associates, Inc., 2023. URL https:
neurips. cc/pppr Files/paper/2003) pdf.
Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burdh and Peter Clark. Clin: A continually learning lan- guage agent for rapid task adaptation and generalization. arXiv preprint ar Xi1:22110.0,,,, 2023.
James Manyika. An overview of bard: an early experiment with generative ai. https: //ai. //ai.google// google/ static/document/ weg.imllesstttggleeeeeeeoo googgl-abbuutbooeeeeooooeeoo .df.. Accessed: May 27, 2023.
Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, and Roy Fox. Skill set optimization: Reinforcing language model behavior via transferable skills. Xiv preprint arXio:2402.002244 2024.
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subra- manian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu, Patrick LeGresley, Denys Fridman, Jared Casper, Ashwath Aithal, Oleksii Kuchaiev, Mohammad Shoeybi, Jonathan Cohen, and Bryan Catanzaro. Nemotron-4 15b technical report, 2024.
Stuart Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ariiv preprint Xitx:777.006477, 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583-8592. PMLR, 2020.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.
Significant-Graiitas.. Sggificant-rraittss/tttooog An experimental open-source attempt to make gpt-4 fully autonomous. URL https://gi thub. Auto-GPT.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. Xrviv preprint arXiv:2212.04002 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint Xiv::2330 002.2887,.998,, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Xiu preprint ar arXiv:2305.16291, 2023a.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arxio preprint xxxi:2331.00599,, 2023b.
12

---

Published as a conference paper at COLM 2024
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arv preprint ar xxio:2302.001660 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougge prompting elicits reasoning in large language models. Advances in neural information processing syslems, 35:24824-24837, 2022.
Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engincering review, 10(2):115-152, 1995.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and track-language models are good teachers for embodied agents. arXiv preprint arXiv:2305.02412 2023a.
Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Russ R Salakhutdinov, Amos Azaria, Tom M Mitchell, and Yuanzhi Li. Spring: Studying papers and reasoning to play games. In Advances in Neural Information Processing Systems, volume 36, 2023b.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. Ad- vances in Neural Information Processing Systems, 36, 2024a.
Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. Smartplay: A benchmark for Ilms as intelligent agents. In The Twelfth International Conference on Learning Representations, 2024b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-2077 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and
Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxie. org/pdf/2305. 10601, pdf, 2023.
Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303.16563 2023.
Appendix
Table of Contents
A Safeguards for Dynamic Operations
14
B Kahn's Algorithm
14
C Design
14
C.1 Encapsulation
14
C.2 Extensibility
15
Manual
15
D.1 Crafter
15
13

---

Published as a conference paper at COLM 2024
D.2 Webshop
16
E
Examples
17
E.1
Observation Description
17
Skill Library generated by the Crafter agent
................................
17
E.3
Knowledge Base learned by the Crafter agent
................................
17
F
Prompts
18
F1
Crafter Prompts
18
F2
Webshop Prompts...
22
G UI
23
A
Safeguards for Dynamic Operations
For dynamic add/ remove of edge (u,v), AgentKit asserts: have not been evaluated for add and both have not been evaluated for remove.
The assert statements avoid not only loops, but also unpredictable behaviors. For example, graph A - F, C - F has multiple topological orders. A dynamic edge C A could potentially alter the previous perceived topological order. The assert statements raise the attention
of the user. A loop can still be implemented by adding new nodes (chain A B .... that keeps growing) and up to the user to prevent infinite loops.
B Kahn's Algorithm
Algorithm 2 Kahn's Algorithm (Kahn, 1962) for topological transverse
1: + empty list that will contain the sorted elements
2: F set of all nodes with no incoming edges
Initialize the frontier F
3: inDegree + map of node to in-degree
4: while is not empty do
5: remove a node from F
Retrieve the next node from the frontier F
6: add и to the end of and evaluate
b Evaluate the node
7:
for all node with edge from to do inDegree(m m)] < Degeree [mgreemm]
8:
9: if inDegree m] = 0 then
10:
insert m into F
D Add the next node with in-degree to the frontier
11: end if
12: end for
13: end while
14: return L
> Topological traversal successful
C Design
Our implementation in graph.py provides solid foundation for managing complex graph structures, emphasizing support for both permanent and temporary nodes and edges, in addition to dynamic evaluation and modification capabilities. This design is firmly rooted in Object-Oriented Programming (OOP) principles, ensuring that the AgentKit API remains flexible, extendable, and easy to maintain.
C.1 Encapsulation
The Graph class effectively encapsulates the complex logic required for managing graph elements, including nodes and edges (whether permanent temporary), and determining
14

---

Published as a conference paper at COLM 2024
the execution order of these elements. By hiding the internal state and offering carefully selected suite of methods for interacting with the graph, we ensure that the class remains ac- cessible and secure against unintended manipulations. Inheritance and Polymorphism Our design philosophy accommodates extensibility through inheritance, enabling developers to derive specialized graph types from the Graph class.
C.2 Extensibility
The Graph class was conceived with future growth in mind. Methods such add node, add_tem ttt.tttmorryyooot add add_edg,, and welddeedeeeeeepoooeeeee
empower developers to adapt the graph structure dynamically, reflecting real-time changes or experimental scenarios. This distinction between temporary and permanent graph components allows one to keep clear track of the original structure, while maintaining a level flexibility that is critical for advanced logics like branching.
Moreover, the architecture supports enhancements through subclassing, opening avenues for introducing novel node or edge types with unique characteristics or for deeper integra- tion with external frameworks, such as wandb_LLM for advanced logging and traceability.
D Manual
D.1 Crafter
List of desired interactions:
avoid zombies, skeletons, and spiders.
collect saplings.
craft a wood pickaxe.
collect wood.
-raft stone pickaxe.
collect stone.
craft a furnace.
collect coal.
collect iron.
craft an iron pickaxe.
collect diamonds. craft an iron sword. chase cows.
grow fruits.
drink from lake.
sleep in a safe place.
craft a table.
eat food.
drink water.
rest.
build stone tools defend myself against monsters.
build bridges to cross lakes
dig tunnels to hide from monsters.
block arrows with stones.
dig through walls to surprise skeletons.
seek shelter in caves.
build plantations of saplings and defend them against monsters.
eat the growing fruits to ensure steady food supply.
place table.
eat cow.
place a plant.
defeat zombie.
place stone.
eat plant.
defeat a skeleton.
wake up.
place a furnace.
15

---

Published as a conference paper at COLM 2024
List of game achievements and their requirements:
1. Collect Wood: No requirements
2. Place Table: Requires Collect Wood
3. Eat Cow: No requirements
4. Collect Sampling: No requirements
5. Collect Drink: No requirements
6. Make Wood Pickaxe: Requires Place Table
7. Make Wood Sword: Requires Place Table
8. Place Plant: Requires Collect Sampling
9. Defeat Zombie: No requirements
10. Collect Stone: Requires Make Wood Pickaxe
11. Place Stone: Requires Collect Stone
12. Eat Plant: Requires Place Plant
13. Defeat Skeleton: No requirements
14. Make Stone Pickaxe: Requires Collect Stone
15. Make Stone Sword: Requires Collect Stone
16. Wake Up: No requirements
17. Place Furnace: Requires Collect Stone
18. Collect Coal: Requires Make Wood Pickaxe
19. Collect Iron: Requires Make Stone Pickaxe
20. Make Tron Pickaxe: Requires Place Furnace, Collect Coal, and Collect
Tron
21. Make Iron Sword: Requires Place Furnace, Collect Coal, and Collect Iron
22. Collect Diamond: Requires Make Iron Pickaxe
List of all actions and their requirements:
1. Move West: Flat ground to the west of the agent.
2. Move East: Flat ground
the east of the agent. the agent.
3. Move North: Flat ground to the north
4. Move South: Flat ground to the south of the agent.
5. Do: Facing creature or material; have necessary tool.
6. Sleep: Energy level is below maximum.
7. Place Stone: Stone
inventory.
8. Place Table: Wood inventory.
9. Place Furnace: Stone in inventory.
10. Place Plant: Sapling in inventory.
11. Make Wood Pickaxe: Nearby table; wood in inventory.
12. Make Stone Pickaxe: Nearby table; wood, stone in inventory.
13. Make Iron Pickaxe: Nearby table, furnace; wood, coal, iron an
inventory.
14. Make Wood Sword: Nearby table; wood in inventory.
15. Make Stone Sword: Nearby table; wood, stone inventory.
16. Make Iron Sword: Nearby table, furnace; wood, coal, iron in inventory
17. Noop: Always applicable.
Health restores automatically over time, independent from food and hydration.
Notes:
Diagonal actions are not supported, only use the four cardinal directions.
The game world is infinitely large and procedurally generated from a fixed random seed.
If you within close proximity to zombie, it will chase you. You must kill the zombie to survive.
- When sleeping, the player will not be able to take any actions until energy is full and will take triple damage from zombies. Therefore, do not sleep when threats are nearby.
D.2 Webshop
Try make purchase in 25 steps.
16

---

Published as a conference paper at COLM 2024
Puchase product with less than 80% match with the task. No need look for a perfect match. If multiple products match the task. purchase the one with the highest match.
-The search engine uses bag-of-words. Optimize the search terms accordingly and don't put price in search. If you cannot find relevant items, consider going the next page of the search results
The search result page only contain short product descriptions. Click on the product to reveal other customizable Atributes/opptios..
Examples
E.1 Observation Description
Gamestep 205
Observation (1 (1-step)) ):
zombie: west, , IW
zombie: east, , 1E
grass: north, IN (facing)
grass: south, 15
Near-by objects (7x9 grid): tree 2 steps to west, 2W
tree 3 steps to north-west, 2N
table 5 steps to south-east, 2S 3E
zombie 1 steps to west, IW
zombie steps to east, 1E
Further to the north: 25 grass (s), tree (s).
Vitals:
health: 1/9
food: 7/9
drink: 6/9
energy: 9/9
Inventory:
sapling: 1
coal:
iron:
wood pickaxe:
stone pickaee:
Action: sove.south step (s)
E.2 Skill Library generated by the Crafter agent
"NavigateTooLocationn: "Move towards specific location using cardinal directions.", "CollectResource": "Interact with an environment object to collect a resource.", "PlaceObject"; "Place an object in the game world from the player's inventory.",
"CraftItem": "Craft specific item using available resources and tools.".
"UseFurnace": "Use furnace to smelt or craft items.". "Interacttithhntityy"" "Interact
entity in the game world to achieve specific outcome.", "Rest": "Restore energy by sleeping or resting in safe place."
E.3 Knowledge Base learned by the Crafter agent
We note mistake in "Exact Quantity of Stone Required", the correct value should be 1.
17

---

Published as a conference paper at COLM 2024
"Collecting_ Requii "No specific tool needed collect wood.", Linteraction.with... "Direct interaction with trees possible.", "Wood
Corpnntttyyfo__obllltt "2 wood for table",
"1 wood per action",
"Movement does not consume energy.", "Adjacent to table for crafting.",
Requirree"e "3 units of stone for Stone Pickaxe.",
"1 unit of wood for Stone Pickaxe.",
ooroee/eonnoooooooooooooooooooo Collection": "'Do' action for wood collection.",
wwwpenttttto_oacaooolleeeeeeeeee Pert "I coal per action",
"'Do' action collects coal", www.Sttt__iccee_ffoooe Coall "Stone Pickaxe not required for coal",
"Use_of_Stome
Lauls_for_WWteee
M. "Water sources indicated as 'water' in observations.",
Aactiin__t__Collct___0 ''do' action collects drink",
"Increases drink level", Souucce_oo_colllection""" "Adjacent to water for collection", waweetttWWWeeeeee
ptroximity_to_Weee
"+1 drink per action", "No tool required", "No table required for furnace placement.", "Flexible furnace placement.""
"Tool "Tool_Requireee
"Tool_
het.aeurrremeetsffoorrrriiiiiiie Furnace": "No tool required for furnace.", "Envirromental__bbtttccess""" "Trees block movement.",
wtrr__ttttaeeeeo_oeeeeeeeeeeeeee "Cows block movement",
"Proximity_
near tree for 'Do' action.", "Wood_Sworf Swwrd_wrafttingggggccccccc.. Ret.inggeeeuuureeeeeeeeeeeeeeeee rements": "Wood and table needed for wood sword.", _oor__ood____ : "Near table, facing not specified.",
wew.eity_gggureeeessfsrrrrr Dottiooooooooo "Must face
hwtp:a/ooooossssssselslee Wood. Sword": "'Do' action near table with wood.",
"Energy_
http://eoooouumttooooooooooeeeee Moveemeet_ "Movement does not consume energy.",
Siterrgg_eeurreemtts_ffooooooo Moveementt: "Moving does not consume energy.", "Feedback_Mechanissti "Feedback through changes in observation, status, or inventory.", _ "Not all actions impact vitals directly.", Baraaturre_MoveeeetBBokciiimmi "Creatures block movement.",
"Energy_Requii
"Feedback_Mechanis PaattniissssssssPMPaeL "oo_eoootttoettt "Observation changes post-movement.""
"Interaction_wif : "Adjacency required for cow interaction.", "Proximity_too__ Targetts"" " "Adjacency needed for interaction.", "CCww_Mooemeree "Cows can move closer.",
oftprisstteeeeeessoooooeooo ooworeett__oeeeeeeoooooooP.. Player__Status "No immediate effect vitals inventory." "Environment relative to player changes with movement." BeatinggCow_PPrceeee "Direct 'Do" action eats cow.", Ltw._oo__oooeeeeee__oorrooooo "Tool_ htw._Reuuueeee_ffooooo
"Immediate
"Food level increased significantly.",
CotermmmeettorrrEiiinooooooooo "No tool required."
Prompts
E1
Crafter Prompts
Lots of prompts require custom (acilompre///siiiie pre/postprocessing. See github repo for full implementa- tion.
bes_obi":f
"proept -an
player. Describe type, direction, First, describe requirenents interact with object instruction nanual. coordinates, precise and accurate whth the direction, coordinates.
Output Json 15st in the following fornat:
("object": Stype, "direction" Sprecise_eirectt "distance" Sdistances, "Facing" "aanng=-Syyee S[yes/no], "coordinate" "preecise coordinates", by "requreeents"" "requtreennss to interact fros instruction manual, put "NA" not apolicable").
Second, in one sentence, descrila be what the player sees further to the Front.
"dep':
18

---

Published as a conference paper at COLM 2024
obs "es_inv":f prospe DDscrrbe the playen's inventory as ison dictionary, with the itee_nones key and itee_anount as value. Write 'O' expty."".,
dep':[]. ).
obs vit':( p """Desoribe player's vitals as 1son dictionary, format [vital_nane: "value/max"") ")."", dep':L).
obs jes_chh":6 proupt : UUsing 1ist, docunent any changes to the player observation, inventory, and status in the nost recent step. precise about changes to numbers and details about the vitals, inventory, and the distance to objects around the player. 'NA" there change""",
concise only 1ist the 'dep'-E) ).
"obs, las__tas__cctt
pprrppt' srrrirst, consider how the recent action should have changed player's dssrrvation/ssatuu//eeeerrrrrr,, specific
my with the action and the target.
Then, deduce if the last action successed.
yes,
here.
failure, by focusing the player's surronding/dderratiinn. inventory, instruction Be concise with the language, but precise with the details.""
not. icentify potential cause
r. sanual, and nissinn information.
'dep' " "bbb_chg"
"s-obs-obs's of
prorptts II sentence, describe observation and inventory. observed object types inventory iters.""", dep'-E coba_ob"", " "obs_iin'J,
"s-obs-vitala"-C
proept:: 'In one sentence, describe the current vitals.",
dep'tE "jdbiv
"s-action::" :f
pproept """"utpuu Ison diotionary of the Following format: ...
"action": faction, The rost recant action, including the direction if st's noverent action
"repeats"; Srepeats The mumber times the action was repested
"target": Starget
the
rovennt/steep
"auccass: Ssuccess [yes/no] the action succaaded
"causes_of_ Failure', $couses_of_fallur In one sentence, write the cause(s) of the failure. Write 'NK" 1f the action succeeded.
"dep'-" DDobD_Db_aasss ccb__last_act.11
"oba aeba_ccrrent_acti anttioons [
prospe : "oor each action in the mumbered list of all actions, reason with the current observation/uurrmmiigg and identify if the action is allowed the current ginestep according the requireeents.
predict target action interact with.
Forrat the output as Json dictionary of the following format:
"Saction": ("resauresents'' Brequirments of action, "related deservation": Swhat the doservation affect/block the action?, "reasoning": Sprecise very
why action blocked, "allowed": Syes/no, "targes": Sinferred target, "unlock achieverent". syes/no, will the success this action unlock unacoompl ihede achievenet??) not outout white spaces
...
depp' "'sraction',, 'obs_oob'1.. 1,
"refllet':<
prospt : ""Coonsderr how the player reached the current state, concisely summarize the player's high-level trajectory in the and not nake future predictions. top subyoal aubgoals, subgoals") player effectively
ganeplay history. Focus the
player been naking effective progress + addressing all the nistakes/obstacle encountered?
output IW if there is no history. 'obs_chg', "obs, last_
'epp' obobs_ 'obs_inv',
plaaner unexppcceed:- {
prospt "p player
any obstacles unexcected scenarios? Also, did player unexpectedly?? expectancy.
Viwy" answer otherwise, should the player modify the subgools to react?***, ddpp' of s-obs', " 'sevic', 'obs,chg', last
"iammrrnisstake" :
prospt snddd the player make any mistakes approaching the top subgoal Sob, subgoals, subgoal5"? Nrite Y"HA"" and stop the answer here if there's nistake. otherwise, identify potential causes mistake and rost probable cause analyzing current situstion.
19

---

Published as a conference paper at COLM 2024
Then susgest precise and executable modirication the subgoais ristake.""", dep' :E 5-obs'. "s-vic", "ebs_chg', chg', "obs, cesrenttttt currentt_ccioons] }.
"challenge": if
prospt "ddentify three high level challenges.
Based on the current situation, score their urgency out 5 is the highest urgency) and score their convenience out 5 convenient).
being
choose challenge address on urgancy, conventence, overall objective gane. Be concise, one, dep" rreflect', "plamerr_iisake" "cbs.obj'. 'obs_inv', "obs_vit'),
"gste-olon_skete sketch":t {f
"proept (oppc:=*eeee "eassoo with the instruction ranual and knoxledge base. Explainn concisely how the choosen challenge nay approached situation, with actions permitted
player's
Then,
Confirn if the subgoal 'Sdb subgoais, subgoels' is still accuracte for the choosen challenge.
2. Confirm if the subgoal 'sdb subgoais. subgools" is incomolete and up-to-date according the completion criteria aib.suhgalls.cc wwwpletion.eriteeeeeeeee cratttiiannritrria....
If yes to both, end the answer here.
Then, skotch an updated high-level plan for addressing the choesen challange, Do not plan back to sorething 1f you cannot
it's location.
schieving relevant unscconplished achievements order difficulty, dop'cl 'rrfllect,, "plamerruuexpecte"" unexpected", "plawer_riisaaee "obs_ob1", "oos_inv', "obs_yit', "achtevesents", "challenge"), J,
situation allows, plan-sketch
ry without impacting player safaty.
gate': [[
prospt "Bason with the previous conversation fon and context, and output Json dictiionary with answers to the following questions:
"uregpeted__ncouatrr""" ancounters": SANSVER, # [yea/no] Did the player encounter any unexpected scenar or obstacles?
"urespected_
"nistake": SANSVER, [yes/no] Did the player nake any sistakes?
"correction_pleeee
[yes/no]] correction planned
step
player seen confused? "top_suue sugguugg completed" SANSVER, # [yes/no] Is the nost recent top subgoal complete according to the corpletion criteria?
"confused": $4NEEER [yes/no]
"top_sibgoal
Lehrrge changed": SANSWER, a [yes/no] Has the neat recent top subgos! been changed??
"roplan": SANSVER [yes/no] Was re-plan/charge for the sketch?
dep's "D" ['reelect"', ppaspee_unespected "planner.ntisakke' "gate-plan.skette ).
"ubgonls':( :f
pproept
plan sketch. subgoals should quantifiable and aot oonale with in-gane actions. Put subgoala of highest priority first.
subgoala
player according
For each subgoel, specify hox the completion may be precisely quantified in at nost one sencence (include the numbers and details). not include tional infortation other than the aubgoalis and quantification.. **., 'dep' ('reelect', 'obs,obj', "obs,inv", 'obs.vit', "pate-plon_skeedf"
'top-ssn "top-subgoal"s
pposppt "hrite highest priority subgoal coepletion criteria Json dictionary following forrat: ..x
"ubgons": Ssuhgoal, The highest priority subgosl ccmplettoon crateria": scorpletion.ce certterra completion criteria subgoal quude" Sguide A brief high-level guide for completing the subgoal
..* D'reflectt" 'snaction', idb__current_,ctionns "anbgosls's 'J,
prospt "Ch"Ch the instruction nanual for requiremenss to cosplete the top subgoal. Identify list of unknown/sissing inforracion/detaill and do not nake assupptions. Pay close attention the details such numbers and specifications." **", subgoll
dep'' rreflect', wubs.current.actii
skill's(
prospt "'"'First, try find existing Fron library' that could represent current top subgoal.
If none of the existing skills could be applied, create new skill.
The skill, paraneters, guide
general erough reusable the same class.
Forrat the choosen skill as Ison object of the folloxing format:
(Sakill
Ssuppor paraneerrs sskill_puidee)
top-subgoal").
pllamer-sdaptive "
prospt Liitt should promot player think concretely precisely.
questions help guice the subgoal: subb.sug subgoal5". The questions
20

---

Published as a conference paper at COLM 2024
Exarpie questions could ask the player to:
identify goal-relevant details fron the observation, history, and instruction nanual
recall related historical encouncers
petential aostacles and how
then
specify concrete ways to improve efficiency
Output only the 1ist of questions and nothing else. Write V"NA"" ir there's 's no question to ask. "h,, depp :C :C'eflect', "planner, unexpected"; 'planner mrsstake' 'tep-subgeall, "subgoal, anallyiiss D.
"kbaadd': :f
"proept "ooWoritte unknown information dstalls 1ist into Json dictionary.
Give each ites concise but precise name as key, and dictionary of answers to the felloxing inquiries as the value:
"ster_name":
"discovered"; SANSKER, Can uninown inforration precisely uncovered deternined the current ganestep? [yes/no]
"discovery": SANSNER, In concise but precise terns, write what has been uncovered CIr the information la not uncovered, write
by 'NA" -J
"discovery_shorrt short": SWEMER, E Condensed version of "discovery', only containing precisely the discovered info [If the information
uncovered, write "Nk".3
"general"; SANSVER, Confimu
this uncovered information resain uncharged in subsequent stops of this gane. [yes/no]
"urknown": SANSIER, Confimu that this uncovered information is missing froe instruction ranual. [yes/ne]
"concrata_and" sunerats_ind_precsee"" SANSWER, # Is the uncovered information concrete and pracise enough to add to instruction manua1? [yes/no] speculative deduced based assunation. Lyes/no]
"solid" SANSKER, Confirr that information ).
Toclude with one-sentance justification detail. repeat inadrriss. Mrite '()" if there is nothing the Hist of "unknown information and datails'. dep' :l "pppammeerrreeee Lineppcctdd, , pplamer_nistakee "coo-subgoal" "subgeal_ analysis", 'ebs_ob'', "obs_inv" 'obs_vit', "obs_chs", "obs_last__ee" "],
"urknman": :{
"proept: enrMargo the "urknoww/nisssing information" in the previous answer and the 'Previous unknown information and details" into single Json dietionary.
Give each ites conciso but precise name as key, and dictionary of answers to the following inquiriias as the value:
"stes_niem": ( if spol teable, use the same ites nome in the previnus answer knoeledge base
"info": SANSMER, concise but prectse language, "M
exactly is wissing. [IF the information is not wissing,
"krowledge"; SUSAFR, Mhat do you already know about the current requested info? ETF nothing fa knowe, write 'NA'-]
"urknoen": SANSVER, Confims that this requested info is missing Froe the instruction nanual. [yes/no]
"novel": SANSKER, Confirs
the kroxledge base "general"; SANSWER, Ia the requested info expected ressin unchanged in future gase stops? [yes/no]
not already
precise
this requested info. [yes/ho]
"relevant": SANSNER, Is this requested info helpful for succending the gama? [yes/no]
"correct": SANSVER Confirs that this request does not disagree with the instruction narual, [yes/no] followed ore-sentence justifination.
only include the answers, not the inquires. Remove icates and arrange multi-target inquirias into saparate itess. **" 'dep' ('ssbgool.oaalyssa),, ).
scctor-reflect's 'x(
"prospt's : ""rFssst, identify types of information other observation important
(Fron gampplay history) may subgoal Sch, subgoals is. subgoals', and output. then in concise Hist.
Secondly, sumsriize the gareplay history using table. action corresponding to current step, and include observations, action (including number steps + result), and the identified inforstion in the teble.
Use "T3D"
firally, analyze and evaluate each action in 'sob alb..alowwd.dco actionss' Utillize spatial and temporal reasoning with the table determine if action safe, Forrat the analysia as Jaon dictionary of the following format, and write "MA" for not applicable Fields:
to unexplored state.
"Saction::
action
"spatial.relatinn reltiio..ti wwthh.cureett.des. Sanswer, llo does this action spatially relate to objects in the observation? focus on u important objects (1 sentence)
"aligment, with history":
"terporal": Sanswer, How does action relate with the historical trajectory terporally? Explain your reasoning connecting with the history table. (1 sentence)
"spatial": Sansver, How does this action relate to the historical, path spatially? Explain your reasoning by connecting with the history table. (1 sentence)
"risk": Sanswer, list potential risks or hazards associated with the action as concisely as possible "benefät": Sanswer, list potential benefits or advantages assuciated with the action as concisely as possible
dep':C "'oo'oopsuuuooll 'malpssl.. analysis' lobs, coss.currett.aee actiinss
"obs, 'obs_inv' vis.vit'],
sccoor-pann-seee sk
prospt Firssst,
the current dbsrvatton/surreuniing with rocus related to subgoal
"db. www.:uabbb.uugoooooae... subgoais' and answer following questions (no more than sentence per-ansver);
out of the goal-relevant objects, which ones are easily reachable and which ones are harder to reach? Are there direct obstacles in the way?
Are there risks in the Are sodressable?
21

---

Published as a conference paper at COLM 2024
Then, deterrine previous plan "Sob.actii ielllaacttonsunate ceroonnsurrrrrrrrie sumary.pian-skethhs still applies subgoal based criceiras the s expiration condition.
Relevance criteria: 9db, reteere relevance releeeecee--rrirria
Expiration condition: action, sumary, englrationncondiiion
the plan still applies, exanine the current observation for any obstacles, hazorads, on unexpected scenarios and reason spatially how they nay acdressed.
necessary, upeate
'cetaalls' of previous
achieve target '5db aub.octtion.o sunmary. targets" the plan.
If the plan does not asply, explain your reasoning, and arite an new plan-sketch.
Reason spetially and cemporally with the current observation, the ganeplay history, and the action analysis.
Finally output updated plan-skecch. plan-sketch details concretely describe procedure specific direction + follow, and nust be a Json cictionary of the folloring format:
"plan-sketch"" splan-sketch, I-line summary of plaisskathh "detials", Concrete descriation of what procedure or a specific direction to rollow. Do not offer muitiple options or possibilities.
write the of plan "relevanse-crierta"" $relevance_criee criteria, texpiration, gexprration_conditionn criteria
get",
plan relevant the currant situstion The condition that may cause the plan to expire, like specific changes in the observation or the inventory, after cortain number of staps. situation reasoning that neaningtul renember future.
"notes": Snotes Anything about
"dep' pplpppppameernneeea unexperdeepettedd "planner_nie stake"" "Saprauhgoal" "sukgnal..eallsiss 'obs_ob''' 'oba_ine", 'abs_vis', "ssaction, "oos_currant_actiio""" 'actor-raflsct"), ).
actorractions' [[[
prosptts Givenn target: Sob, action, sbbaactionssumarry targets
First, idancify any nazards in the observation/Surroudiing and scentify obstacles that Interfere with achioving the target.
Plan-sketch: action_suneee sonuun detailas
sumourysurrryyooooom commary-plaesdee plaersketchs
details: act
Diacuss how to spatially address evade the hazards and obstacles based on analysis of the doservation, target, and the plan-sketch.
Then, reason with the instruction on sanual and the current observation, and identify sequance of actions to achieve the target.
The sequerce of identified actions should from current only include actions up first steps, without skipping ng any steps.
Think stop by step and explain your reasoning bafore arriving at the answer.
Group
actions togather speed up gase, explicicty stating
repeats each actiion.
e* Note ex:
The player's reach is step the Frant.
object is steps estraight line facing directions, you only stepes object.
"dep'-''baa_inn" I'ob,a_nee", 'obaLvit', "Araction", "dbi_courent_aacioon "aba_ohj"1 "1,
'actor-Final's
prospc 1= "Examine the actor plan and reasoning and identify the first action (only the exact action nare from the list of all actions in the ssnual) in the plan.
number tines action should reseated. Firally, identify if there are any observed/neerbly hozards obstacles, no ratter if they interfere with the plan.
identified action rexplicitly staced repeate,
Foreat the nutput as Json dictionary of the following fornat:
actions the ranusl. "repeacs": Srepeats The number of times this action should be repeated. write if the action is not stated as reseated. "bazard": Shazard [yes/no] Presence of hazards obstacles in the inttenatiin/surruunettn
"action": Saction, Write only the exact actiion name from the
they interfere with the
'dep' [['obs.obj', 'actor-actions" '..
).
's-plann:" :f{
prospt's "In sentence, high-level subgonls and reasoning behind subgoals.",
dep'cE "E reeflect', 'planner unexpected pnastaer.iisakke..... "top-subgoal". "subgool, analysis'. "ecter--eflect').
bbbaree_unexpected
F.2 Webshop Prompts
'abs's
"pronpt"""" """"ccrrbb the products and Interactible elesents the page as Ison dictionarias:
products
22

---

Published as a conference paper at COLM 2024
interoctible..lleents
dept:
ttakk filter'st
products,
Otherwise, exenine the current weopage and the task:
and output Isen dictionary explaining how the product(s) the page natch the task apecifications.heeee Check the guidance for criterias to natch the products.
dictionary should contain
reasoning
esstinated aetch-percentage (equally weight for all features)
esstinated sutch-percentage exithout taking into account of any mismatch in custoxizable Fasturase (equally neight for all, but not including customizable) estinated inportance-adjuste natch-percentage
the features importance) pecch-peecentagee highest above three)
dep ["obs ].
actor sketch":
pronptt sumpirstt identify to types of Information other than wilpage and action (from the browsing history) that be important for the task, then concise
tessiites, tes, actions, and the identified information from the browsing histooyy using table. Use "TED" for the action corresponding to the current stop. chronological
Secandly, sumarize the webs
of step.
Third, describe the current webpage in lass than two sentences, drawing connection to the cask:
Sdb www.wetwwwtt.....mmm. ref
Then, reason with the browsing summary, the current setpago, and the guicance to deternine how the task may be addressed with actiions parnitted the webstte.
Explain your reasoning before arriving answer.
Then, deternine if the plan Froe the rost recent stop should be continued.
Finally, drow updated planosketch starting from the current only consisting actions pemiitted by the website. """
Llabs'. filltri
).
aattonn
prompt': """Using information plan-sketch, write actionable plan approach
5dh wwwromeent www.ettrrefiiii raFined_tasks
Then Identify best, action at current step, output action dictionary following Fornat:
actioon: faction, search/click
"target": Starget, the target the action
ens
deg': ['actor_sketch"" J,
sumsery.nbs'
prompt': """In one sentence, hos the itees on the pape natch the task.
5db www.sewwweenn....mmmmmmmmm www.eeedeeeeddeeeeeeeeo tasks
Only provide details on the iter that ratch the task the nost (when messuresents are present, provide the highest).
dep ['task_filcer"" filcer")
sumery.actoo actor__paann
prompt' "* sentence. cescribe detail, including future
describe the reasoning behind
deg" d'actor_sketchh "astion')
)
G UI
23

---

Published as a conference paper at COLM 2024
DAG Manager Dashboard rd
Current DAG Nodes and their details:
The graph is currently empty.
Options:
[A] Add node
[E] Edit node
[L] Load graph
[S] Save graph
[R]
Evaluate graph
[X]
Exit
Choose action (A, L, 5, R, x):
Figure 5: Screenshot of our command line interface (CLI) to produce a graph without coding.
24


Reviews:
[{"review_id": "uliHpW3Dqa", "reviewer": "Reviewer_DxSd", "summary": "This paper proposes an abstraction for structured reasoning using LLMs based on a Direct Acyclic Graph (DAG) structure where \n\n1. nodes handle a specific subtask correspond to LLM calls with pre and post-processing \n2. directed edges indicate which node outputs become which node inputs\n\nIn addition to this static and local processing flow \n\n3. there is a shared database of values that is read and written by every node that allows global communication\n4. there is the possibility of dynamically adding or removing edges\n\nthe experimental setup features comparison with other agent frameworks on the Crafter and Webshop tasks", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our paper explores a timely topic and our setup/evaluation make sense.\n\n# [A] local vs. global\nPlease note a fully static process has been explored by [1]. We believe that **global components (e.g., memory, planning) are indispensable to AI agents**.\n\nThe primary contribution of AgentKit is an easy-to-use **framework** to build agents with **multiple** global features **working together**, with modular, simple-to-understand subtasks and DB. Our experiments focus on showing generalization to **different** and challenging domains: game and web, in a **zero-shot** manner.\nFor more insights on static vs. multifunctional implementations, consider SPRING [1] vs. AgentKit in S4.1.1. AgentKit quantitatively demonstrates reduced cost and higher consistency and qualitatively demonstrates more advanced capabilities (S4.1.2).\n\nFinally, note that `branching` only is used in a user-controlled setting: skipping planner when there's no plan updates.\n\n## Ablations\nAs shown in appendix E, the `static/local` prompts are designed to achieve global features by using a database. Removing the global functions plan, reflect, KB will result in a completely different design. \n\n# [B] Mixing models\nPlease note that the baseline on Crafter [1] also uses GPT-4 for fair comparison. Insights on mixing GPTs: `GPT-4` dominates `3.5` in performance. We use GPT-3.5 for `s-X` and some `obs-X` (summarization/rewrite). We use `4-turbo` for the rest instead of `0613` except `actor-reflect` because `4-turbo` has trouble reasoning with space (sometimes cannot tell north opposite to south).\n\n## WebShop\nNote that both AgentBench and ReAct are **few-shot** and AgentKit is **zero-shot**, which makes the task harder for AgentKit.\n\n[1] SPRING: Studying Papers and Reasoning to play Games\n\n# Q [S2.3]\nFor dynamic add/remove of edge (u,v), AgentKit asserts: `v` have not been evaluated for `add` and both `u,v` have not been evaluated for `remove`. The assert statements avoid not only loops but also unpredictable behaviors. Consider a graph `A-F, C-F` with multiple topological orders. A dynamic edge `C-A` could potentially alter the previous topological order. The assert statements raise it to the attention of the user. A loop can still be implemented by adding new nodes (chain A-B-… that keeps growing) and it is up to user to prevent infinite loops.\n\n# Changes\nUpdated the repo github.com/anonymousLLM/AgentKit with replacement to Figure 1 to better showcase contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**[A]** The topic is very timely and explores an area of growing importance: Abstractions needed in building agent systems.\n\n**[B]** The paper proposes a framework with a specific set of abstractions and an experimental setup with two commonly used agent benchmarks (Crafter, Webshop) where to test a particular instantiation of these against existing systems. The setups make sense and the comparison is against recent state of the art systems.", "reasons_to_reject": "It is hard to judge the contributions to the paper because: \n\n**[A] It is hard to know which components contribute to what**\n\nThe paper proposes a framework with a specific set of abstractions but it remains unclear how these help because the have conflicting properties i.e.\n\n1.  the system starts with desirable properties that bring simplicity and understandability. These are \n\n    a. locality: provided by imposing a DAG structure. This indeed allows to think of sub-tasks independently and diagnose better\n    b. static behavior: provided by having this DAG being pre-defined before execution\n\n2. but then it adds properties that have the opposite effect, to attain flexibility and generality\n\n    a. globally shared variables through a data-base\n    b. dynamic modification of the DAG via \"an API\"\n\nso its hard to know if the locality/staticity help, what are the tradeoffs with regard to flexibility and generality. The paper just presents aggregate results with no ablations. For all we know the conflicting properties above give enough freedom to write any \"neural computer program\" and would make this paper more of a software package technical report, with not a strong research component. \n\nA possible improvement would be to compare local/static implementations versus the full version to analyze the effect of these design choices.\n\n**[B] The experimental setup for the Crafter setup has important confounding factors**\n\nIt is well known that agent systems are very dependent on the LLM used, e.g. switching from GPT-4 to GTP-3.5 severely affecting performance [(Wang et al 2023)](https://arxiv.org/abs/2305.16291)\n\nBut the experimental setup compares system with multiple different LLMs including some with markedly different performance \n\n>Our AgentKit implementation uses a combination of GPT-4-0613, GPT-4-turbo, and GPT-3.5- turbo to balance cost and performance\n\nThis is not the authors fault. Its a sad state of affairs that closed-source proprietary systems are so far ahead open source alternatives for agent systems, but this does not change the fact that it is hard to judge results (i.e. was it the use of GPT-4 at some task that made the different?, have the authors identified tasks or flows that for which GPT-3.5 is good enough and competitive with GPT-4. All of this would be important for the papers scientific contribution.)\n\nIt should be noted that the experimental setup for Webshop **does** have comparable setup using GPT-4 but also that it only uses the first 100 samples of Webshop, a specific category of Agent Bench, see \n\n[Table 3 in (Liu et al; 2023)](https://arxiv.org/abs/2308.03688)\n\nAgent Bench contains many more frameworks that were left out. So the setup for this second task could also be stronger. Again this is not exactly the authors fault.", "questions_to_authors": "[S2.3] \n>Since there may exist more than one topological order for a given graph, the dynamic addition/removal of components may result in non-deterministic or unexpected behaviors. Safeguards are put in place to catch potential unexpected behaviors.\n\nCould you clarify this statement. If the graph is statically defined and topo-sorted before starting the processing but then dynamic graph changes are allowed, this can cause infinite loops. What are these safeguards?", "extra_scores": {}}, {"review_id": "R0dnteGtLZ", "reviewer": "Reviewer_Lfw9", "summary": "The paper introduces AgentKit, a prompting framework for multifunctional agents. AgentKit uses simple natural language prompts to explicitly build complex \"thought processes.\" The framework's core component is a node, which includes a prompt for a specific subtask. Users can combine these nodes to form a structured \"thought process\". Quantitative results show that agents designed using AgentKit achieve SOTA performance on Webshop and Crafter.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for recognizing that AgentKit is comprehensive, reliable, and detailed, which can serve users in diagnosing and improving LLM agents for a wide range of tasks.\n\n# 1 Novel\nWe will update our final paper to cite Majumder et al., 2023, and Nottingham et al., 2024. These works focus on extracting `knowledge` or `prompts` with a curriculum involving multiple turns with the environment. Key differences in our setting include:\n1. **Contribution** The main contribution of AgentKit is a unified, easy-to-use framework for **multi-functional** agents, whereas existing works focus on one functionality. AgentKit can be combined with Nottingham et al. for learning from multi-turn interactions.\n2. **Use of reward/feedback**: AgentKit never uses environment rewards.\n3. **Rollout length**: Crafter spans 150-300 steps, up to 500, making credit assignment challenging and token-costly.\n4. **Subtask length**: Simple subtasks in Crafter (e.g., `find_wood`) can take over 10 steps; complex ones (e.g., finding a diamond) over 100 steps. Nottingham et al. tested with subtask length <=5, scaling might be an issue.\n5. **Environment noise**: Crafter has conflicting objectives (e.g., survival vs. tech progress) and distractions from procedural generation and random spawning, which present challenge as noted by Nottingham et al.\n\nNote: Both papers are recent, with one accepted by ICML2024 and the other released less than 2 months before the COLM deadline, making it difficult to build AgentKit on top.\n\n# 2.1 Eval.\nAgentKit supports **multifunctional** agents, unlike individual features like planning, reflection, and KB. Our Crafter agent demonstrates three advanced agent functionalities **working together** under AgentKit's unified framework. Our experiments highlight generalization to **distinct** and challenging domains (a game and the web) in a **zero-shot** setting.\n\n# 2.2 Example & Detail\nQualitative analysis and examples of Planning, Reflection, and Learning are in section 4.1.2 and Figure 4.\n- Remove KB: Slower progress past stuck situations (sec 4.1.2), leading to worse performance.\n- Plan summarization: Necessary due to lengthy and noisy observations.\n- Context window: [20,30] range should be fine, depending on the game.\n- Dynamic graph: Reduces cost by 1/3 with no performance impact.\n\n# Changes\n- We will add detailed information in the final version.\n- Updated the repo github.com/anonymousLLM/AgentKit with a replacement for Figure 1 to showcase contributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Comprehensive, reliable and detailed agent framework**: AgentKit is a comprehensive agent framework that incorporates numerous essential functionalities for critical reasoning tasks. The framework offers transparent reasoning steps, enabling users to easily identify and rectify any reasoning errors. This makes it particularly valuable for those who aim to diagnose and improve their reasoning processes.", "reasons_to_reject": "**1. Novelty of the framework**: In my view, this framework resembles an advanced self-reflection agent system, enhanced by specialized knowledge modules. Several unique nodes are employed to summarize previous trajectories and guide planning at the current step. However, this also brings to mind two relevant works: CLIN (Majumder et al., 2023) and SSO (Nottingham et al., 2024). Both incorporate memory modules (akin to a database) that refine agent skills. Thus, while this framework is quite sophisticated, it fundamentally builds on prior research with incremental, finer-grained modifications.\n\n- Majumder et al., 2023. CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization. \n- Nottingham et al., 2024. Skill Set Optimization:Reinforcing Language Model Behavior via Transferable Skills.\n\n**2. Lack of in-depth evaluation**: The authors present only two performance tables for two tasks without a more detailed, fine-grained analysis, leaving several critical questions unanswered. For instance, what impact would the removal of node types n_{kg add} and n_{unk} have on performance? If the context window length were reduced from 25 to 20, would there be a significant performance decline? What role does the plan summarization node play in the overall performance, and how crucial is dynamic editing for the graph structure? Furthermore, given the framework's complexity, providing qualitative examples, particularly for the self-driving task, would be essential for helping readers understand the workflow and better grasp the practical implications.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jcz0MFfGhT", "reviewer": "Reviewer_GbVx", "summary": "This paper proposes a framework (AgentKit) that constructs a graph with LLM prompts for reasoning, where each node contains a natural language prompt for a specific subtask. The paper claims that the proposed framework has modular interpretability and is easy to use (without the need to write explicit code). Experiments on Crafter and Webshop shows the framework works well and can generalize to different environments.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Our core contribution is 1.SOTA agents in Crafter and WebShop from a unified structure of “nodes”, 2. a **unified** framework that enables for developing **multi-functional** agents, with SOTA performance in multiple tasks.\n\n## 1.1 Crafter\nApologies for the confusion. Details on the score vs. reward for SPRING and AgentKit are in Sec. 4.1.1, 4.1.2\n- AgentKit outperforms SPRING with GPT-4-turbo by 80% and costs 45% less than SPRING with GPT-4, while achieving slightly higher average (SOTA) rewards\n- Score favors random behaviors (unlocking more achievements at lower probabilities)\n\nSPRING relies on randomness to get past stuck situations (Section 4.1.2), whereas AgentKit uses *planning and learning from mistakes*. SPRING tends to unlock less important achievements (e.g., Eat Plant, Collect Sampling), **improving the score but not benefiting survival**. Therefore, using score to determine the inferiority of AgentKit is **unfair**, as the score metric favors RL agents with better exploration.\n\n## 1.2 WebShop\nThank you for pointing out the omission. We will update the final version to include the LLMs used by baselines in the WebShop table. Specifically, Act and React use PaLM-540B, and AgentBench reports the best of LLMs (including GPT-3.5/4). Note that AgentBench and ReAct are **few-shot**, whereas AgentKit is **zero-shot**, making the task harder for AgentKit. Please take this into consideration when evaluating the fairness of the comparison.\n\n## 2.1 Components\nThank you for highlighting the confusion. We will update the presentation to better state our core contribution. The primary contribution of AgentKit is an easy-to-use framework for assembling simple language subtasks for multifunctional agents. **Multiple** important agent features work together under a **unified and modular framework**. Our experiments show generalization to different and challenging domains: game and web in a **zero-shot** manner.\n\n## 2.2 Complexity\nWe include qualitative analysis and examples of Planning, Reflection, and Learning in Section 4.1.2 and Figure 4. AgentKit's design simplifies building agents to writing a list of natural language subtasks and dependencies. While we showcase a Crafter agent with sophisticated capabilities, the underlying building blocks are **unified** in **natural language form** (Appendix E) and transfer easily to the web.\n\n## Changes\nUpdated github.com/anonymousLLM/AgentKit with a replacement for Figure 1 to better showcase our contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed framework is general, and intuitively makes sense, I would expect it to work on a diverse set of tasks.\n2. The idea of incorporating dynamic graph into LLMs for reasoning could have high potential.", "reasons_to_reject": "1. Despite the high potential of the graph structure, it's unclear from the experiments whether it actually helps. First, the performance is (significantly) worse than Spring (20.6 vs 27.3) on Crafter. And there's no clear explanation from the paper why it is worse; Second, on Webshop, the improvement is also unclear. Actually, I'm not sure if the reported baseline (ReAct) uses GPT-3 or GPT-4 as base LLM, and the paper does not mention it as well. Therefore, I think the results of the proposed approach are quite weak to validate its effectiveness.\n2. The writing of the paper needs significant improvement. First, it's very hard to get where the key motivation is. The paper proposes multiple components, but it's not explained well why we need this component. e.g., knowledge base (do we actually need an explicit KB, or just list all possible actions?). Second, some of the phrases are not accurate. E.g., in conclusion, it says \"State-of-the-art performance\" which is not accurate, as on Crafter, the performance is significantly worse.\n\nTo summarize, the proposed framework is a bit overcomplicated, while the effectiveness over simpler baselines is not validated; Also the authors need to make the writing clear, especially in introducing the key motivations and the intuitions of different components.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BjqqlO2tpe", "reviewer": "Reviewer_geit", "summary": "This paper presents a novel framework for structuring the reasoning process of large language models to tackle complex tasks. The use of DAG for structuring the thought process is technically sound. The authors provide quantitative results demonstrating the effectiveness of their framework, showing SOTA performance on WebShop and Crafter. It also gives a detailed explanation of the AgentKit framework and its organization aids in understanding the flow of ideas and the progression of the research.The framework has the potential to impact how LLMs are used in a variety of applications. Overall, it presents a high-quality, clear and significant work.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition that our framework is **novel** and has the potential of **impacting** how LLMs are used in **real-world tasks**, and thank you for the recognition that our work is of **high-quality**.\n\n# Evaluation\nWe include comparison with Act, React, and the custom framework by AgentBench in WebShop. Please note that since frameworks like React/Act requires few-shot demonstrations, we are unable to apply them to Crafter due to the trajectory length (can go easily above 200 steps).\n\n# Changes\nUpdated the repo github.com/anonymousLLM/AgentKit with replacement to Figure 1 to better showcase contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a framework for natural language “coding” of end-to-end multifunctional AI agents, with the abilities to plan, reflect, and learn under challenging situations. This approach is innovative in its use of a node-based system to simulate a structured thought process. And it could be useful in many real-world tasks. Also the authors will open source its GitHub repository, which can lead to reproducibility and further community involvement.", "reasons_to_reject": "The paper may lack a thorough comparative analysis with similar frameworks.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a rather complicated graph (DAG)-based framework to learn agent behaviors and shows positive results on benchmarks such as Webshop and Crafter. The majority of reviewers agree that this paper has enough merits to warrant acceptance. There has been good faith discussion during the rebuttal period where the authors have attempted to address reviewers' concerns. However, the paper's writing has certain clarity issues, and some reviewers suggested discussing the strengths of the system and its novelty. We suggest the authors incorporate the reviewers' suggestions in the final version to make the paper more accessible to readers who are not specifically working in this space.

[At least one review was discounted during the decision process due to quality]
Decision: Accept