AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback
Authors: Jian Guan
Wei Wu
zujie wen
Peng Xu
Hongning Wang
Minlie Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Agent
Knowledge
Feedback-Driven Adaptation
Primary area: natural_language_processing
TL;DR: We propose a general framework for building knowledge agents, featuring FSM-based reasoning logic and a process feedback mechanism, which is demonstrated by extensive experiments to surpass previous agents by a large margin.

Abstract: The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present AMOR, an agent framework based on open-source LLMs, which reasons with external knowledge bases and adapts to specific domains through human supervision to the reasoning process. AMOR builds reasoning logic over a finite state machine (FSM)
that solves problems through autonomous executions and transitions over disentangled modules. This allows humans to provide direct feedback to the individual modules, and thus naturally forms process supervision. Based on this reasoning and feedback framework, we develop AMOR through two-stage fine-tuning: warm-up and adaptation. The former fine-tunes the LLM with examples automatically constructed from various public datasets, enabling AMOR to generalize across different knowledge environments, while the latter tailors AMOR to specific domains using process feedback. Extensive experiments across multiple domains demonstrate the advantage of AMOR to strong baselines, thanks to its FSM-based reasoning and process feedback mechanism. The code and data are publicly available at
https://github.com/JianGuanTHU/AMOR.

AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback
Jian Guan 1.2 Wei Wu²*, Zujie Wen", Peng Xu², Hongning Wang', Minlie Huang
The CoAI group, DCST, Institute for Artificial Intelligence,
State Key Lab of Intelligent Technology and Systems,
Beijing National Research Center for Information Science and Technology, 'Tsinghua University, Beijing 100084, China. 2 "Ant Group.
{ (jianguanthu, wuwei19850318 wang hongn @gmain com, (zujie. wzi, peng. @antgroup. com, @ihuang@tsinghhnng edu, cn.
Abstract
The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present AMOR, agent framework based knowledge bases and adapts to specific domains through human supervision to the reasoning process. AMOR builds reasoning logic over finite state machine (FSM) that solves problems through autonomous executions and transitions over disen- tangled modules. This allows humans provide direct feedback to the individual modules, and thus naturally forms process supervision. Based on this reasoning and feedback framework, we develop AMOR through two-stage fine-tuning: warm-up and adaptation. The former fine-tunes the LLM with examples automatically con- structed from various public datasets, enabling AMOR to generalize across different knowledge environments, while the latter tailors AMOR specific domains using process feedback. Extensive experiments across multiple domains demonstrate the advantage of AMOR to strong baselines, thanks to its FSM- reasoning and process feedback mechanism. The code and data are publicly available at httpp:://iitii com wttp://w....ooaaaaaaaaaaaaaa http://aaaaann/HHAAOOO
open-source LLMs, which reasons with external
Introduction
LLMs, with astounding performance over general natural language processing (NLP) problems 42 33), have spurred great interest in building LLM-based agents to solve complex tasks by interacting with external resources such as web knowledge (27). specialized tools [31], etc.
We focus developing agents for knowledge-intensive tasks, where the agent completes users' informattonn-sekking requests by interacting with specific knowledge bases [22]. To address the complexity of such tasks, we posit the desiderata for qualifying agent follows: Firstly, the agent should possess a robust reasoning logic about the task to solve individual problems with precise pathways. Secondly, the agent should maintain adaptive mechanism adjust specific environments, rather than staying static. Thirdly, the reasoning process should be amenable to human interventions, enabling humans to the agent's behavior through direct feedback to the process rather than only to the outcome. This ability can significantly facilitate alignment between agent behavior and human intent [39]
Although extensive studies have been conducted on building language agents, few, if any, can fulfill all the required criteria due their uncontrollable reasoning logic, static model capability,
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table I: Comparison between AMOR and representative methods for building agents. Appendix A.1
provides a more comprehensive discussion in detail.
Reasoning Logle
Method
Adaptive Mechanism
Feedback
Step
Stter-Step Dependency
WebGPT
27
Tool Invoking
Undefined
Imitation Learning from Humans
Outcome
CoTl 43
Reasoning
Undefined
Prompting
Undefined
ToT 50
Reasoning
Undefined
Prompting
Process
ReAct
51
Reasoning& Tool Invoking
Undefined
Prompting
Undefined
Reflexion
35
Reasoningg Tool Invoking
Undefined
Prompting
Process
AgentL.
53
Reasoning& Tool Invoking
Undefined
Imitation Learning from LLMs
Outcome
MetaGPT
Specialized Module
Sequential Pipeline
Prompting
Process
LUMOS
Specialized Module
Sequential Pipeline
Imitation Learning from Humans
Undefined
AMOR
Specialized Module
Finite State Machine
Exploration&&Xxlloiaaiooo
Process
Pig Quenttin Decompositin
Releaanee Judement
Passage Retrlenal
Aneaer Extraction
D.T. cliior
mhy (Q)
Way - (0.ge(0.
ma: www.erteea....m..... K.
Anooreter ( de varra Foery
my SearchPsgge
Bevebaie Thane sener
if wobl miverd
(1i0c) fowr silemaie
Completon
a. NextDoct)
18y (
qusiue cood
nethevd documents
noveed paw.aat DueM
Qureett documane
States Modula LLll
States Medules for imacking Tool
5:44 collectcc enderio pansagas
Reblewadd Passages
Figure 1: AMOR'S 's state transition diagram. Each box represents a state and the corresponding module that is executed when entering the state. There may be multiple categories of execution results distinguished by special branch tokens such as "[NEXT]." Then AMOR determines the next state based on the branch tokens.
sparse/missing feedback signals, as detailed in Table Consequently, is still challenging for users to critique, and thus guide existing agents to follow targeted manners, especially when the agents built upon less powerful LLMs (25]
We introduce AMOR, an Adaptable ModulaR knowledge agent that can reason and adapt, with the reasoning process amenable to human supervision, based on open-source LLMs. AMOR'S reasoning that solves problems via series of executions and transitions over a set of modules (Figure D. This naturally enables the desired process- based supervision mechanism, allowing users give feedback to each LLM-controlled module. AMOR supports flexible forms of feedback, either binary judgments regarding the correctness or refinement of the outputs. The reasoning logic and process feedback mechanism together frame how AMOR thinks, acts, and interacts with users and task environments.
logic is formalized as finite state machine (FSM) 7 21
We build AMOR upon an LLM equipped with distinct parameters for different modules to efficiently handle multiple tasks. The training in AMOR happens in two stages: (1) Warm-up: the modular design enables us to construct training data separately for each disentangled module without requiring complete trajectories for specific tasks. As a result, we create a large dataset of 50k examples covering multiple distinct tasks, simply using public datasets. We fine-tune AMOR this data for generalization over various knowledge-secking scenarios. (2) Adaptation: when deployed, we tailor AMOR to the target domain by letting it autonomously address user tasks (i.e., exploration), collecting process feedback for each LLM output, and evolving through further fine-tuning on the exploration trajectories with feedback (i.e., exploitation). Our contributions summarized follows:
We propose a general framework for building knowledge agents, featuring FSM-based reasoning logic and process feedback mechanism. We focus on text corpora as knowledge bases, but the approach can be flexibly extended to other knowledge types and user tasks by customizing the modules and dependencies within the FSM framework.

---

II. Experiments across multiple domains show the strong advantage of the FSM-based reasoning logic with 30%-40% improvements over baselines when based on off-the-shelf LLMs (e.g. G Switching to fine-tuned L.L.s, the warm-up stage empowers AMOR to generalize to multiple domains and surpass strong baselines. After we adapt MOR to specific domains, subsequent domain-specific adaptations reveal that process feedback is significantly more effective in improving the reasoning process than outcome feedback.
2 Related work
Language agents. Interest is surging in building agents for tasks necessitating multi-step reasoning. Existing work falls into two groups. The first group focuses designing agent architectures, such as CoT's step-by-stey reasoning (44, ReAct's integration of reasoning, action, and observation to allow tool use (51), and CODEPLAN'S two-stage reasoning framework that first generates a code- form plan and then realizes low-level reasoning steps (45). Nevertheless, such free-form reasoning constraints human intervention. In contrast, modular agents follow pipeline execute specialized modules [19 14
I 3 [52). improving the case of intervention. The second group aims to design adaptive mechanisms for adapting agents specific scenarios. ToT [50] and Reflexion [35] use environment feedback for multi-path pruning and iterative single-path refinement, respectively, but suffer from poor inference efficiency and need for real-time feedback. As a fine-tuning approach, recent work equipped open-source LLMs with specific agent abilities by learning from examples synthesized based on human priors [5), or expert trajectories from humans [27] or GPT-4 [ with correctness validation through outcome feedback. In contrast, our modular agent AMOR employs FSM-based reasoning with a stronger capacity for handling complex tasks than simple pipelines and adapts effectively to specific environments via process feedback.
Retrieval- -ugmeette generation (RAG). The RAG paradigm augments the inputs of LLMs with retrieved passages to enhance factuality [ reasoning-retrieval for better information recall than one-step retrieval [38] 16 28). However, retrieval may introduce noise that leads to low-quality answers [3). To tackle this, Self-RAG [2] trained LLMs to selectively perform retrieval and utilize retrieved passages. Unlike RAG approaches, AMOR emphasizes an explainable reasoning process for proactively decomposing questions and seeking evidence for grounded generation, and allows for process feedback from humans. Nevertheless, RAG mainly focuses on integrating parametric factual knowledge in LLMs and retrieved non- conparraettric knowledge, which is less explainable and intervenable.
22 10 Recent studies have developed interleaved
3
AMOR agent
AMOR relies on three key techniques: FSM-based reasoning logic, process feedback mechanism, and two-stage fine-tuning strategy. We detail the definition of the reasoning logic and its specifica- tion assuming the knowledge base is a text corpus in I3.1, the method for fine-tuning open-source LLMs warm-up stage in $3.2, and the adapta- tion stage driven by process feedback in $3.3
Algorithm 1 FSM-based Reasoning Logic
Input: Agent at the state = sg: Q: Question.
Output: A: Final Answer; R: Reasoning Process. ==
2 while # SN-1
3
= =m(s) // Obtain the output y given from the corresponding module m.. Rappend (oopend(("stttt""" ("state": "output": y])
5 A = y
6 return A, R
3.1 Reasoning logic
Algorithm Notllnnes how to deduce the answer for an input question Q with a reasoning process R using FSM-based reasoning logic, which can be defined by quadruple: (S,M. M, E, μ),, where
S (so....; 1 } is set of states with so as the initial state and - as the final state. Each state holds variables to track context information.
set of modules with my triggered when the reasoning flow reaches state $3- The modules are categorized into two types: (a) Tool modules ( (Mroou) for invoking tools, and (b) LLM modules (MLL)) for calling LLMs.
M {mo..... my-11
2 In this work. GP5-3.5/4 refers to the OpenAl's API "gpt-3.5-turbe /ept-44 "rppt--1106-previeee 1106-preview." respectively.
3

---

Knowledge Augnentation
Original Sarple
Qunction (1 On ohat cate did the publischer Ohiic thick floe unell ths new systees?
Seardloworuuryy (y, Cils
Boon")
Passages HiR
the publiaherr Cluck Clück Boom?
(LaUier thukk tausk tean) Alter bitre losk Ule
re:tle; Oiice Chice basnt The cttets wst: scunce througtsar their
for Extter 3387 Gerone deen ioper Eatra Tease are epceccred by Mirteece of furope
Pii rLitler Cluck cluck 5o0d 2001 drawing is gueee at accurasy raturg
Seanchbos -i, tite "iitendo Entertairment Systent")
6: Muu day did Narisnelo snreett the not ayvllmml
L. (tilles Wateeu dheecter Norto is derracee created by the
October 18, 1905
SeardiFigQuery 6, 6ite "Ninoondo Enterzinmenn Systen?) /6
(tallee Ruelerde leientedd
Mintercce sextiol los
Pia (tat.er Nuncerds beter caunner bysites) After developing severat
isested
text pardete etartiing New
berober I6,
Pia huscerto bnter bystes)
the
Firl Armwer Octeber 1985
Pry
Decompose
Judge
Complete
Mix Quotaard
Mtan Qperker
Miar Qponinn 0
Men Men
[14 Daper
Hewsyy (1 Olnn Nl His. Hi
(Ralowed (rstonee (ulloood (takowed Joloned
Oerge (kenvoalic) knmer by Recoas Pooege DD It (t (hranrepht))
Mice Oritarnd
(NeeeNN it.
Sehved Saa-
Ness de II Iu Ilr
Naar Caentec
1. 12 1u
[R II [aalnnel) (kdlowa)) (I [tabowdd @I (odeowe
Diter (Pond)
providing ample information (in green) for constructing training examples for four LLM modules of AMOR (bottom). We augment extra knowledge (in blue) for the Judge and Answer module by invoking the SearchDoc and SearchPsg tools (top right). In each example, we highlight the prompt in purple to format the current state (before "Output:") and output (after "Output:"), and use "Il" to separate different examples for training.
Figure 2: On the top left is a sample question from Musique 37
E is the set of all possible outputs of M.
S:Sx
+ E S is the transition function that determines the next state of the reasoning flow given x W:Sx&
the current state and the execution result of the corresponding module.
When the external knowledge base a text corpus, an instantiation of the reasoning logic can be represented by the state transition diagram in Figure T In this case, Mrroo. perform document and passage retrieval using external retrievers; while MLLM leverage the LLM to analyze and digest the question, documents, and passages to deduce the final answer. To distinguish different types of outputs from a module that requires different subsequent modules, we employ set of special branch tokens such as "[NEXT]" to guide μ to determine the next state. In summary, AMOR answers question Q by (1) iteratively decomposing Q to a sub-query q at state 8p, and finding the answer a to q and the evidence passage through iterative knowledge retrieval, relevance evaluation, retrieval refinement (i.c., "Passage Retrieval"), and answer extraction, until no more knowledge is needed: and (2) deducing the final answer A based on the collected evidence passages at the final state.
Defining reasoning logic as an FSM offers three advantages: (1) Structured Thinking, FSM makes specifications of inter-step dependencies (e.g., prioritization, branch selection) easy, and thus enables narrowing down the exploration space. (2) Skill Disentanglement. By decomposing complex tasks into modular steps, one can independently construct training data for each module, which significantly reduces the difficulty of implementing AMOR with open-source LLMs (cf., $3.2). This feature also allows A MOR to focus on single steps, thereby mitigating the weakness of LLMMs in reasoning over long context formed by task-solving trajectories (24. (3) Intervenable Workflow. The structured reasoning process enables users to casily diagnose the agent's 's mistakes and provide process feedback for improving the reasoning capability of the agent ($3.))
3.2 Warming-up open-source LLMs
Recent studies have improved their reasoning abilities through imitation learning using trajectories from advanced LLMs such as GPT-4 [3
Open-source LLMs are observed to fall short in complex agent tasks [46 25
However, even GPT-4 can struggle with producing high-quality reasoning
trajectories
29
AMOR'S 's modular design enables us to construct training data for each module separately from existing datasets without simulating the whole trajectories, thus greatly alleviating the above issue. Formally, given a sample question Q with annotations of the final answer Å, all sub-queries and answers A ((0. do), (Ộ1, âi), ... and all evidence passages É [êo, é we can directly

---

transform these annotations into a suitable format to serve as training data for Decompose and Complete in Figure Since Judge and Answer require multiple types of retrieved knowledge (e.g.. relevant or not), we employ retrieval tools to augment the input. Figure exxemliieee the construction pipeline, which can be easily extended to other mnowledge-intensin datasets and specific domains. Appendix A.4 shows more details.
When fine-tuning open-source LLMs handle multiple tasks defined by different modules, we are inspired by the Mixture-of-Expee parameters in the final quarter of the Transformer blocks to balance the trade-off between performance and inference efficiency. These module-speciio parameters are initialized using the original model's FFN layers. We call the proposed architecture Module-A Aare Mixture-of of of-Exxerts (MA-MoE) Then, we fine-tune the MA-MoE model with the standard language modeling loss:
approach [3] to learn distinct Feed-Forward Network (FFN)
L1 Mun..))))) Am log Te,.. (i]
(1)
where т refers to the policy model MA-MoE that maps the state 8 to an action d. On denotes the parameter for the module m E MMLM Dm is the corresponding collection of training examples, (8. 9) is a state-output pair from Dm and {Am} are tunable hyper- syper-parameter.
3.3 Adaptation through process feedback
Feedback crucial for adapting language agents to specific environ- ments [40], especially when deal- ing with unseen, long-tail, or ever- changing domain knowledge. Prior agents commonly used outcome feedback for adaptation which as-
Algorithm 2 Adaptation through Process Feedback
Input: {mom J: Initial Policy; T: Exploration Steps between Ex- ploitation: Number of Iterations.
Output: (Ta,..): Adapted Policy.
while <- to do
R [] // Feedback-Refined Refined Reasoning Processes
while - To
sesses the correctness of interme- diate steps based on the success failure of the outcome [53] [4.
// Exploration
Receive an input question Q.
Collect AMORg's 's reasoming process R. // Algorithm // Feedback Collection for Each LLM Module
However, outcome feedback is too sparse improve intermediate rea-
foreach Step rı E R (k 0, 1, 1.2, ........ do
soning (23]. Recent studies also highlighted that L.MMs reasoning steps are likely to contradict the out-
Extract the state 84 and output yı from Ts..
The corresponding module
Muu then
Collect feedback fx for su and yh-
Determine ÿA and og based on fs. R.append ([sk. Uk, ox])
Eq.
come (26). which means that out- " come feedback may inevitably in- troduce noise during training (see examples in Appendix B.8) In
// Exploitation
Optimize (0m) minimize
R. Eq.
13 return {
contrast, AMOR's 's process feedback mechanism can effectively alleviate these issues.
Algorithm 2 describes the adaptation mechanism of AMOR parameterized by 6 specifically as three steps: (1) Exploration. AMOR answers the input question Q by interacting with a knowledge base. (2) Feedback Collection. AMOR'S reasoning process for Q is evaluated with feedback f& for the output yy the LLM at each step during reasoning, which is either "right/wrong" or a refined version of y. We convert into a reedback-refined target output based on the feedback fx and determine the immediate reward Og as follows:
yks 1 fc = "right",
Uk, Ok
yk. if fa "wrong",
(2)
fe, if fx is refinement.
(3) Exploitation. Every T steps of the former exploration and feedback collection, we optimize the initial policy based on the resulting trajectories and corresponding feedback [32]:
C (a Am [o Glog Te... (ÿx (sx)s))/e0 "m (5x (}kls)))]
(3)
3 "Module-Ar Awaree means when AMOR executes a certain module, its module index will be provided to the routers of the model to indicate which expert should be activated.
5

---

Table 2: Automatic annotation strategy for silver process feedback for different LLM modules.
Module
Oulput
Silver Process Feetback
Decompose H)
[NEXT] 4
"right", if the retrieved documents using averlop the documents correpeoding to As "wrong" otherwise. "right", in S (uc, evidence passages collected by AMOR); "wrong") otherwise.
[FINISH]
Judge(Q (Q. M, (,
[ELEVANT]
"[RELENANT]", if one of passages in comes from the same document as d; EVANT] otherwise.
[RRELEYANT]
(nswer(Q.uug P)
[ANSWERABLE] as [UNANSWERABLE]
"righi", E "wrong", otherwise
"righ",, if Pn 8: "wooeg". otherwise
Complete QQ. E))
A
À AfE E; "wrong", otherwise.
where R.m. C R denotes the training examples for module m, Th WPT refers to the initial warm-up policy, Notably, this loss function non-diffcrentiable, necessitating the use of a specialized optimization technique. We use a recently proposed alignment algorithm KTO [9] with an MLE regularization [47 for optimization, which optimizes the policy without requiring paired human preferences. Crucially, when optimizing particular module m, the gradient induced by the feedback signal propagates through the entire MA-MoE model, except for the FFN layers corresponding to other modules. This targeted optimization approach enables AMOR to effectively align its outputs with the desired intermediate results and final answers, leveraging the fine-gra mne-grained process feedback provided by human supervisors.
Experiments
4.1 Experimental setup
Tools modules. We construct retrievers for both SearchDoc and SearchPsg using Contriever-Mr MARCO [15]. SearchDoc retrieves single document snippet per query. while SearchPsg fetches the top three relevant passages from a given document. By invoking NextDoc, at most nine more document snippets are returned. Appendix B. presents more details.
Warm-up datasets. We employ four question-answering (QA) datasets to warm up open-source LLMs, including 2WikiMultHHopQA [33, Musique [37], NaturalQuestions (20 and BoolQ ( They vary in levels of question complexity (single- or multi-hop), answer types (phrase spans yes/no). and types of dependency structures between sub-queries (e.g., serial or parallel), etc. Appendix A.4 shows the statistics in detail.
Adataation & evaluation datasets. We consider three benchmarks, by which we simulate different deployment scenarios: (1) HotpotQA 49 challenging multi-hop QA dataset built Wikipedia articles. We use the Wikipedia dump provided in [15] as the knowledge base. (2) PubMedQA [7]: biomedical QA dataset that requires answering a question by "yes/no" given PubMed abstract. We adapt the data to retrieval-based QA by piling all 274k abstracts provided in the paper as a knowledge base, where each document comprises one abstract passage. (3) QASPER [8] answering questions in free form based on a long NLP paper. For each question, we regard the corresponding paper as knowledge base and each section of the paper as document with several passages. We use the training and validation sets for adaptation fine-tuning and the test sets for evaluation. For evaluation metrics, we use exact match (EM) and F1 scores for HotpotQA and QASPER; and the accuracy (ACC) of ""yes/no" for PubMedQA. More details are in Appendix B.2
Feedback annotation. Considering limited resources, we simulate human behavior and provide silver feedback to AMOR'S 's reasoning processes based on the gold answer Á and gold evidence passages f = [Eg, ê for each target question Q, which are already included in the training and validation data of the three benchmarks. output y. Note that AMOR is applicable for gold feedback from humans in realistic applications. Appendix discusses the accuracy of the silver feedback through human evaluation.
how annotate the feedback for each L.LM
Implementation details. We set Am in Eq. and Eq. 3 1 for all modules, I= in Algorithm 2 and 7' to the size of the training set for each dataset, and fine-tune LLAAMA---BB/13B--CN for two epochs with learning rate of -5 using 8 NVIDIA 80GB A100 GPUs. While applying

---

Table 3: Refining each module output y to y based on the outcome feedback fo adapt AMOR, where y denotes converting the binary output y to its opposite label.
Module m
Target Output Dh. and Immediate Reward oi
Decompose(Q; H)
DI y and
1 if f. A: fi y and on 0, otherwise.
Judge Q, H. q,d)
Dk = y and O& = 1, if fa = A: 8jk
and o& = 1, otherwise.
Answer (Q, q; P)
Dr = y and oR if fa A: jiu y and ok = 0. otherwise.
Complete QQ E)
yh A and
if E S
E: DK y and o 0. otherwise.
AMOR for inference, we use greedy decoding for all generations. Besides, set the maximum number of decomposed sub-queries to the maximum count of gold evidence passages, i.e. 2/1/1 for TappooQAAPPMMedQA///APPPPPP transited to state 86 ("Task Completion") to finalize the answer.
respectively. Once the maximum number is reached, AMOR is
Baselines. We compare AMOR to various baselines with or without fine-tuning: (1) CoT 44
prompts an off-the-shelf LLM to generate the answer through step-by-step reasoning. (2) RAG: One-Step Retrieval (OneR) uses the question as query to retrieve top- document snippets with the SearchDoc module to augment the input. We set K as the maximum number of gold evidence passages in each dataset. Under the fine-tuning setting, we use the gold evidence passages for training. Self-RAG (2] selectively performs retrieval and utilizes retrieved passages while does not explicitly introduce question decomposition. They can be viewed as simplifications of AMOR. (3) ReAct [5 it interleaves thought, action, and observation steps. An action can be either invoking the retrieval tools or finalizing an answer. We also compare AMOR with fine-tuned ReAct-style agents including AgentLM (53) and FIREACT 4. We set the maximum number of action steps to 20. (4) Modular Agents: ReWoo 446] follows a pipeline that plans all sub-goals, generates actions, and then executes, while LUMOS [52) applies this pipeline iteratively, tackling one sub-goal at a time with each interaction. Both agents utilize GPT-3.5 as supplementary QA tool during action generation. Similar to AMOR, they modularize language agents; however, they lack explicit mechanisms for assessing the relevance of retrieved information. Under the setting without fine-tuning, we provide in-context examples for the baselines following their official implementations.
Furthermore, we also conduct ablation studies to investigate the influence of different components, resulting two more baselines: (1) AMORwFT: AMOR with only warm-up fine-tuning, without fur- ther adaptation; and (2) A MMRRoutoome outcome feedback instead of process feedback is utilized in adaptation after AMOR is warmed-up up. Specifically, we determine the target output and corresponding immediate reward for an LLM module as detailed in Table and then adapt AMOR using Eq. For clarity, we denote our final method as AMOR AMORProcess..
4.2 Main results
Table/Areport the evaluation results of AMOR and baselines on three datasets, revealing three key find- ings: (1) The FSM paradigm is clearly advantageous to prior agent frameworks. A FT delivers strong performance by improving 41.9%. 32.1 1 %, and 41.2% over ReAct on average when built on top of aff-the-sheed LLMs, including L-7B, GPT-3.5, and GPT-4, respectively. This indicates that our proposed FSM paradigm is more effective in leveraging LLMs for complex reasoning. (2) Warm-up fine-tuning generally enhances A AMOR in downstream tasks. When based on L-7B, A AMORWFT outperforms A MO FT across all datasets. Furthermore, AMORWPT also surpasses other fine-tuned ReAct-style and modular agents, even including FIREACT that is fine-tuned with in-domain HotpotQA trajectories from GPT-4 This suggests the potential of utilizing existing datasets for developing powerful agents with well-defined reasoning logic. (3) Process feedback is more effective than outcome feedback in facilitating the adaptation of agents. The order that
AMORProcess > AMORQutcome > AMORWFT indicates the impact of feedback in terms of tailoring agent behavior to specific domains, and process feedback is more helpful than outcome feedback for leading to the correct final answers.
presents the results from employing various model architectures (including our proposed MA-MoEE model, the standard MoE model, and the Transformer model) as well as optimiza- tion algorithms (namely KTO and Supervised Fine- Fine-Tuning, i.c., SFT), The MoE model is identical to
Additionally, Table 5
7

---

We Table 4: Results of AMOR and baselines. "L-7/13B" short for "LLAMA-2-7/13BB highlight the best results in bold and underline the second best. Models marked with are fine-tuned on the target datasets. Results marked with are reported in the original paper and those marked with t are reported 4.. N/A means the method does not apply to the datasets. AMORpsocess outperforms baselines under the same setting significantly (p 0.01, sign test).
HotpotQA
PubMedQA ACC
QASPER EM
Method
Base LLM
EM
F1
F1
Without Fine-Tuning
ReAct
L-7B
12.2
16.6
61.8
6.0
19.2
AMORw/o
L-7B
26.0
34.6
62.9
21.3
CoT
GPT-3.5
28.0
N/A
N/A
N/A
OneR
GPT-3.5
33.4
42.1
72.6
6.8
23.3
ReAct
GPT-3.5
30.8
38.8
58.2
5.8
27.0
ReWoo
GPT-3.5
30.41
40.1
AMOR w/00
GPT-3.5
39.6
49.3
68.8
10.0
30.8
CoT
GPT-4
45.0
N/A
N/A
N/A
ReAct
GPT-4
42.0
62.1
7.1
26.2
A AMOR
GPT-4
55.2
65.2
80.0
11.5
37.4
With Fine-TTuiing
OneR*
L-7B
34.8
43.8
75.3
11.0
25.5
Self-RAG
L-7B
22.4
32.9
62.6
2.1
17.9
AgentLM
T-7B
22.0
64.9
4.2
20.2
FIREACT
L-7B
26.2
18.4
LUMOS
L-7B
29.41
70.3
7.1
19.5
AMORProcese
L-7B
45.8
54.9
81.1
19.1
35.3
AMORWFT
L-7B
33.6
41.9
73.4
11.1
23.6
AMORoutcome
L.7B
49.3
77.5
AgentLM
L-13B
29.61
67.9
7.1
24.4
AMORProoss
L-13B
48.6
55.3
82.2
18.1
38.0
AMORWFT
L-13B
36.8
44.1
74.6
15.2
27.3
AMOROuteome
L-13B
42.4
51.6
80.1
26.5
Table 5: Results of AMORProcess based L-7B with different architectures and optimization
algorithms. The architecture setting is also applied on the warm-up fine-tuning stage. i refers to our final method.
HotpotQA
PubMedQA ACC
QASPER
Architecture
Optimization
EM
FT
EM
FT
MA-MoEt
KTO
45.8
54,9
81.1
19.1
35.3
MA-MoE MoE
SFT
43.2
53.1
79.3
18.6
34.2
SFT
41.6
51.1
78.8
17.5
33.5
Transformer
SFT
41.4
50.9
78.2
17.8
33.2
the MA-MoE model except lacks module-specific awareness. SFT refers optimizing the model only for outputs ts that receive an immediate reward o₁ 1, using standard language modeling loss. The results show: (1) The standard MoE architecture struggles effectively differentiate its experts for multitasking scenarios, leading to performance comparable to the Transformer model. In contrast, the MA-MoE's module-specific awareness enables handle diverse tasks within the agent more adeptly. (2) KTO outperforms SFT in aligning the agent's performance with external feedback, owing to its exploitation of negative samples.
4.3 Discussions
The main results have substantiated the benefits different components of AMRR for successfully completing tasks. Nonetheless, we are still curious about four key research questions: (1) RQ1:

---

Table Recall scores under different settings.
Method
Base LLM
HotpotQA
PubMedQA
QASPER
OneR
N/A
31.1
67.6
24.9
AMOR
L-7B
24.1
54.2
24.3
AMORProces
L-7B
53.5
79.8
41.9
AMORM MORRWFT
L-7B
41.1
69.2
27.5
A AMOR, Outcome
L-7B
40.2
70.0
27.5
AMOR Process
L-13B
53.7
80.5
42.4
A AMORWPZ WTT
L-13B
43.0
69.4
27.0
AMOR Rutcome
L-13B
41.5
68.1
27.7
Is process feedback more data-efficient than outcome feedback for adaptation? (3) RQ3: What if using human feedback for adaptation of AMOR? (4) RQ4: To what extent does feedback-driven adaptation enhance the AMOR's reasoning process? Besides, Appendix B.6 and B.7 further demonstrate the efficient token usage of AMOR and the flexibility of AMOR's ' reasoning framework, respectively.
How do the AMOR variants differ in the ability to collect evidence? (2) RQ2:
RQ1: Evidence collection comparison. We recall of gold evidence passages (É) among those
collected by AMOR (E) to assess AMOR's ability to collect evidence, formally as #(EnE)
#(E)
As shown in Table observe: (1) Warm-up fine-tuning consistently enhances evidence collection, with A WFT achieving higher recall than AMOR FT across all datasets. (2) Adaptation through outcome feedback (AMOROuutemme) exerts negligible impact the recall results compared with AMOR suggesting the superiority of AMOROutcome to AMOR in final answers (see Table 4) may stem from the improvement of Complete. (3) Process feedback is crucial improve the evidence collection ability, with AMORProces substantially outperforming the other variants.
RQ2: Data efficiency for adaptation. We aim to compare the data efficiency of differ- ent feedback types for adaptation terms of the number of exploratory instances required.
AMOR
ANDRoa
39
AMOR
To this end, we adjust the exploration steps T in Algorithm?2 2 selecting values intervals of 200, ranging up to 2,000 steps on HotpotQA. Appendix further discusses the cases with in Algorithm where AMOR is optimized multiple rounds.
27
Figure 3 shows the post-adaptation performance of AMOR varying with the number of ex- ploratory instances (i.e., T). Compared to AMORoutcomes AMORProces requires signifi- cantly fewer exploration steps to achieve compa- rable performance. Notably, AMORontcome shows marked decline in performance when exposed to a limited number exploratory instances (< 800), suggesting a reduced adaptability in exploration- limited scenarios. Conversely, AMORP robust performance under such constraints highlights its superior adaptability and efficiency with minimal interaction.
630
1000
1902
Figure 3: EM/F1 HotpotQA varying with the number exploratory instances for adaptation.
RQ3: Adaptation through human feedback Due limited resources, we use automatically annotated silver feedback as proxy for gold human feedback in our main experiments. We would like to emphasize that our experimental framework inherently designed to seamlessly incorporate human feedback in place of its automated counterpart. To illustrate this, we carry out human study demonstrate how AMOR adapted through human feedback on HotpotQA. For this study, we hire an NLP expert to provide human feedback for each module within AMORWFT 2,000 exploratory

---

Table Adaptation results of AMOR through human feedback on HotpotQA based on L-7B.
Agents
Feedback Type
EM
F1
AMOR
Automatic Feedback
45.8
54.9
AMORPMoOCH Process
Human Feedback
50.8
59.2
Table 8: Accuracy of four LLM modules. All AMOR variants are based on L-7B.
Method
Decompose
Judge
Answer
Complete
AMORp Process
73.0
97.2
82.5
50.0
AMORWFT
59.5
95.3
77.2
32.0
AMOR AMORoutcome
61.2
96.0
75.T
44.0
instances following the annotation strategy Appendix using the collected human feedback.
B.3 Table shows the adaptation results
The results distinctly suggest: human feedback more effectively adapts AMOR to specific knowl- edge environments than automatic feedback. Our study lays robust groundwork for the practical deployment and real-world utilization of the AMOR framework.
RQ4: Reasoning process assessment. To measure the accuracy of AMOR'S 's reasoning process, performed human study HotpotQA, which involved: (1) selecting 50 random questions; (2) manually annotating the gold feedback fnuman for each LLM module output the following instructions using the same annotation protocol in Appendix B.3 and (3) calculating the accuracy of each L.M module output based on fhuman (1/0 indicating "right/wrong").
Table 8 presents the accuracy of AMOR variants, affirming RQI's 's findings: process feedback sig- nificantly improves the reasoning process over AMORwFT that lacks adaptation, while outcome feedback has negligible effect. Moreover, AMORP relatively lags in the Decompose and Complete modules, hinting that future enhancements could focus on including more corresponding data during two fine-tuning stages.
4.4 Case study
Appendix B.8 presents several examples further illustrate AMOR's strengths in reasoning logic and intervenability. as well as the limitations of relying on outcome feedback for adaptation, emphasizing the crucial role of process feedback.
Conclusion
In this work, we develop AMOR, an adaptable modular agent designed for knowledge-intesivee tasks, featuring FSM-based reasoning logic and process feedback mechanism. Based on open- source LLMs, AMOR undergoes a two-stage fine-tuning: initial warm-up to generalize across task environments and subsequent domain-specific adaptation through process feedback. Extensive experiments demonstrate AMOR'S 's advantages over strong baselines across multiple domains. Further discussions highlight the effectiveness and efficiency of process feedback adaptation. compared to previous agents. Future work will explore extending our paradigm to more knowledge types (c.g.. structured knowledge bases) and broader agent tasks, ultimately empowering L.LMs autonomously design FSM-based reasoning logic on top of our paradigm.
Acknowledgements
thank anonymous reviewers and area chairs for their valuable feedback and insightful comments that helped improve this work. This work was supported by the NSFC projects(Key project with No. 61936010). This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333.08877. 2023.
[2] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024.
[3] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Proceed- ings of the AAAI Conference Artificial Intelligence, 38(16;:16822176668 Mar. 2024.
[4] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023.
[5] Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. From the least the most: Building a plug-and-play visual reasoner via data synthesis. arXiv preprint arXiv:2406. /9934, 2024.
[6] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2936, 2019.
[7] Edmund M Clarke, Orna Grumberg, and Michael C Browne. Reasoning about networks with many identical finite-state processes. Proceedings of the fifth annual ACM symposium on Principles of distributed computing, pages 240-248, 1986.
[8] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-ssekking questions and answers anchored research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610, 2021.
[9] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning, 2024.
[10] Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. Language models hallucinate, but may excel at fact verification. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Mexico City, Mexico, June 2024. Association for Computational Pupers), pages 1090-1111, Mexico Linguistics.
[11] Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference Learning Representations, 2024.
[12] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 9929-3938. PMLR, 2020.
[13] Xanh Ho, Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6622 2020.
[14] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. The Twelfth International Conference on Learning Representations, 2024.

---

[15] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022.
[16] Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Y Yu, Yiming Yang. Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 7969-7992, Singapore, December 2023, Association for Computational Linguistics.
[17] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 2567-2577, 2019.
[18] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6781,, Online, November 2020. Association for Computational Linguistics.
[19] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson. Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations. 2023.
[20] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[21] David Lee and Mihalis Yannakakis. Principles and methods of testing finite state machines-a survey, Proceedings of the IEEE, 84(8):1090- -000099--11,,,011101111001001110 1996.
[22] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-94477 2020.
[23] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
[24] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 173, 2024.
[25] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlic Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference Learning Representations, 2024.
[26] Ziyi Liu, Isabelle Lee, Yongkang Du, Soumya Sanyal, and Jieyu Zhao. Score: A framework for seff-contradictory reasoning evaluation. arXiv preprint 2tXiv:23/1.09610 2023.
[27] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answeriig with human feedback. arXiv preprint arXiv:2112.09332.. 2021.
[28) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational I. Linguisties: EMNLP 2023, pages 5687-5711, 1, Singapore, December 2023, Association for Computational Linguistics.
12

---

[29] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations, 2024.
[30] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, and lly Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
[31] Timo Schick, Jane Dwivedi-Yus Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
[32] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1 0ts/1707.06347, 2017.
[33] Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely pparsely-gated gated mixture-of- experts layer. In International Conference on Learning Representations, 2017.
[34] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31210-31227. PMLR, 23-29 Jul 2023.
[35] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[36] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini 0tti:230770098888 2023.
[37] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-555 2543,,554,40 2022.
[38] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014-10037, Toronto, Canada, July 2023. Association for Computational Linguistics.
[39] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome- based feedback. arXiv preprint arXiv:2211..12275 2022.
[40] Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents, 2023.
[41] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[42] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Perey Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Machine Learning Research, 2022. Survey Certification.
13

---

[43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- -hain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-248873 2022.
[44] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- -haon-of- prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 15:24824-24883 2022.
[45] Jiaxin Wen, Jian Guan, Hongning Wang, Wei Wu, and Minlie Huang. Codeplan: Unlocking reasoning potential in large langauge models by scaling code-form planning, 2024.
[46] Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models. arXiv preprint arXiv:2305. 18323, 2023.
[47] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the bound- aries of LLM performance in machine translation. In Forty-first International Conference Machine Learning 2024.
[48] Xin Xu, Yue Liu, Panupong Pasupat, Mchran Kazemi, et al. In-context learning with retrieved demonstrations for language models: A survey. arXiv preprint arXiv:2401. 1624, 2024.
[49] Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D) Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, 2018.
[50] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[51] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023.
[52] Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 12380-12403, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[53] Aohan Zeng. Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310. /2823, 2023.
14

---

A Methodology
A.1 Comparing AMOR with related works in detail
As illustrated in Figure g
AMOR addresses the issues of prior reasoning methods in terms of three
aspects:
AMOR is equipped with controllable FSM-based reasoning logic with a stronger capacity for handling complex tasks than simple pipelines employed by Self-RAG, ReWoo, LUMOS. For instance, no relevant passages are retrieved from a document, AMOR can dynamically transit to the next document, while LUMOS would be constrained to generate answers based the irrelevant passages, potentially leading to incorrect or low-quality outputs.
AMOR adapts to new environments through exploration and exploitation. AMOR is designed with an adaptation fine-tuning stage, enabling it to adapt effectively to specific domains based on human feedback. This adaptive mechanism sets AMOR apart from prior modular agents that lack the ability to incorporate expert guidance.
AMOR enables humans to conveniently and effectively intervene and provide feedback at each reasoning step. AMOR introduces a process feedback mechanism that enables humans to provide direct feedback on the individual modules within the FSM-based reasoning process. This approach facilitates more natural and interpretable form of supervision, allowing for targeted improvements and fine-tuning of specific reasoning components.
In summary, AMOR achieves more controllable, adaptable, and human- human-guided reasoning capabilities compared to existing methods.
Deseppatton
lectbeel
Eulgcaant
Plocest POOSS madio
Concibatre
kurpatee
Auciooal
the 16000)
The days
Machie
Rewfice Beseiing
Figure 4: Left: Elaboration regarding the advantages and drawbacks when comparing AMOR with prior agents in terms of three aspects in Table [ related works.
Right: The reasoning processes of AMOR and
A.2 Full algorithm of AMOR
Algorithm in the main paper illustrates general FSM-based reasoning logic, which can be adapted to various agent environments by defining the FSM including the states, modules, etc.
As shown in Algorithm 3 AMOR provides an instantiation of the FSM-based reasoning logic for the knowledge-seeking scenarios following the state transition diagram in Figure in the main paper. We expect extend this work more environments in the future.
A.3 Prompts for LLM modules
Table TO and 13 show the prompts for four LLM modules in AMOR under the "Without Fine-tuning" setting on HotpotQA. They can be converted to the "With Fine-tuning" setting by removing the in-context examples. The prompts for PubMedQA and QASPER are similar.
15

---

Algorithm 3 Answering Question Q Using AMOR
Data: AMOR at the initial state so (Q, H, E); Q: Question: H []: All solved sub-queries and answers;
[] evidence passages collected by AMOR.
Result: A: Final Answer; At: Reasoning Process.
while True do
if
then
y +Decompos(ssQQ 8.H R.append( (("state": s. "output": y)) // Transit to the next state.
if starts with "[NEXT]" then
Extract the next sub-query 9 from
(1 S.H, S.E.
else if starts with "[FINISH]" then sa(s.Q.8.E)
else 81 then 10
11
SearchDoc(s.g)
// Transit to the next state.
D. [
13
8 (2. s.H. s.E, s.q, D,
else if then
y Judge(s.QQ s. 8.q, s.d)
16
s, "output": y]) // Transit to the next state.
Raappend( ""rppend(""ttte""" (""state::
17
if starts with "[IRRELEVANT]" then
$a(s.Q, S. S. s.q, s.D)
19
else y starts "[RELEVANT]" then (s(Q.Q s. s.H, F. 8. s.D, s.d)
20
else 33 then
V NextDoc()
// Transit to the next state.
23
if is NONE then
H s.H + [ss.q, "No Answer")]
E s.E [[]]
26
s, E,
else
D,d = s.D by).
29
(2 H. E. then
30
else
y SearchPsg(**. (s.gg..... s.d)
Transit the next state.
32
y
33
34
else 8o then
35
y = Answer(Q H, g, P)
36
Rappend ( "statee": s, "output": y])
/ Transit to the next state.
if starts with "[UNANSWEAALEE" then
38
8 (s. H, S. 8.q. s.D)
39
else if starts with "[ANSWERABLE]" then
40
Extract the answer a and the evidence from
H s.H. [s.q.a) a]
E =. [e]
43
s = (s H)
else if
then y CC.mplete(s.Q...F)
46
R.append "state": s, "output": y))
47
A = y Reach the final state.
48
break
return A, R
16

---

Table 9: Statistics of the warm-up data.
Module
Branch Token
2WikiMulliHopQA
Musique
NaturalQuestions
BoolQ
[NEXT] [FINISH]
3,500
3,500
500
500
Decompose
500
500
500
500
[RELEVANT]
2,000
2,000
2,000
2,000
Judge
[IRRELEVANT]
2,000
2,000
2,000
2,000
[ANSWERABLE]
500
3,000
1,500
3,000
Answer
[UNANSWERABLE]
500
1,000
1,000
1,000
Complete
3,000
4,000
1,500
4,000
Overall
12,000
16,000
9,000
13,000
A.4
Construction of warm-up examples
In this section, elaborate the pipeline collect training examples for the warm-up stage of AMOR. Given a sample question Q with annotations of the final answer Á, all sub-queries and answers A [( a 11y=0 and all evidence passages É = 1² where is the number of necessitated sub-queries of Q, we construct training examples for four LLM modules of AMOR as follows:
Decompose Q, H): We construct total of J + 1 training examples for this module. For each of the J sub-queries, we create an example with the main question and the preceding sub-queries and answers H- Hej as the input, and the next sub-query Φ coupled with the branch token "[NEXT]" as the output (for j = 0, I......JJ 1). Here, He, denotes the sequence containing the first j pairs of sub-queries and their corresponding answers from H. Additionally, we create one example where the input includes Q and the complete set of sub-queries and answers H = H, with the branch token "[FINISH]" as the output, indicating the end the decomposition. Judge( Q, H, 4, d): For this module, the input consists of the main question Q, the previous sub-queries and answers H Åe. the current sub-query q فj. and document snippet d (for = 0, 1, ... .J 1). The output is a branch token that classifies the snippet as either "[RELEVANT]" or "[IRRELEVANT]" in relation to the current sub-query âj- We consider three scenarios for the document snippet d: (1) When is the gold evidence passage éj. the output is "[RELEVANT]". (2) When d is passage from different document from éj. it is marked as "[IRRELEVANT]". We obtain this type of snippet, denoted as dlj by using O3 as the query in SearchDoo, ensuring originates from distinct document compared to éÿ. (3) When d is a passage from the same document as éj but is not é, itself. it is deemed "[RELEVANT]". We acquire such snippets by invoking SearchPsg with q, to retrieve passages from the same document as é,, excluding é from the results. We refer to this set of passages P- considering each of them relevant to qj- These varied document snippet scenarios are designed train the module to discern the relevance of a query to a document based solely on portions of the document content.
Answer Q H, q, P). Similar to the Judge module, the input for this module comprises the main question Q. the previous sub-queries and answers И = He the current sub-query = qj. and set of passages P from the same document. The output either the branch token [UNANSWERABLE]]] or a combination of the branch token [ANSWERA ALLE]", the corresponding answer å,. and evidence passage éj. We consider two scenarios for P: (1) When P does not include éj. indicating that the sub-query ij cannot be answered, the output is (UNANSWERABLEY)
Here, P is set to the previously mentioned set P- (2) When P includes ěj, suggesting that O answerable, we create P by replacing random passage in p with ēj. For both scenarios, we present the passages to the module random order when constructing training examples.
Complete Q, E). We construct one training example for this module by setting the input to the main question Q and gold evidence É and the output to the final answer Á.
After generating examples from the warm-up datasets using the aforementioned pipeline, we randomly select a specified number of examples. This random sampling aims to ensure balanced representation of the various modules and branch tokens the final dataset. Table/D the detailed statisties of the warm-up data.
17

---

Decompose H)
Please continue decumpose the provided muin question inlo unswerable sub queries following previously already sulved sub queries.
There are two cases as follows:
()) [Next] If the question requires further decomposition: Identify and oulput the logical sub query that addressed order progress towards answering the main question.
(2) [Finish]
question does not require further decompustiio and can be unswered us
HERE ARE SEVERAL EXAMPLES:
Examples Startm==
(1) Main Question: What U.S Highway gives to Zilpo Road, and also known Midland Trail?
Output: [Nex]] How can Zilpo Road acceased?
(2) Main Question: Which magazine was started first Arthur's Magazine or First for Women?
Solved Sub Queries:
Q: When was Arthur's 's Magazine started? A: 1844-1846
Output: [Nex]] When First for Women magazine started?
(3) Main Question: Which magazine started first Arthur's Magazine or First for Women?
Solved Querbeseeis:
Q: When was Arthur's Magazine started? A: 1844-1846
Q: When was First for Women magazine started? A: 1989
Output: [Finish]
= Examples End==
Now Please Complete the Following Task. Please ensure sub-query specific enough understand isolation.
Main Question: (QHH')) [%H is formatted string representing the solved sub-queries and answers constructed from H.%
Ounput:
Table 10: Prompt for the Decompose module for HotporQA.
JudgerQ, d.).d)
Given sub-query derived from the main question and document snippet with its title, please assess whether the document is patentially relevant sub query hased on the title and shown content the document. Assign one the following categories: (1) [Relevant]: Choose this category il the document is relevant to the sub sub-query.
(2) [Ireeeeant]: Choose this calegory if the document is irrelevant the sub- qub-q.
HERE ARE SEVERAL EXAMPLES:
===== Examples Start====
(1) Main Question: Which maguzine started first Arthur's Magazine First for Women?
Next Sub- Query: When was Arthur's 's Magazine started?
Document Snippet: (tile: Arthur's Magazine) Arthur's Magazine Arthur's 's Magazine (1844-1846)
Outpul: [Relevant]
(2) Main Question: Which maguzine was slarted first Arthur's Magazine or First for Women?
Solved Sub-Queries:
Q: When was Arthur's 's Magazine started? 1844-1846
Next Sub-Query: When was First for Women magazine started?
Document Snippet: (title: History of women's mugazines) In 1693 the first issue the first women's 's maguzine in Britain Output: [Irelevant]
(3) Main Question: What U.S Highway gives access to Zilpo Road, and also known Midland Trail?
Next Sub-Query: How can Zilpo Road accessed? Document Snippet: (uile: Zilpo Read) constructed the Licking River by the Army Corps of Engineers.
Ounput: [Relevant]
=xa=Examples End=
Now Please Complete the Following Task:
Main Question: (Q)(')) I [&H formatted string representing solved sub-queries answers constructed from H.%
Next Suh- Query: 19]
Document Snippet:
Output:
Table 11: Prompt for the Judge module for HotpotQA.
B Experiments
B.1
Tool modules
Tool implementation AMOR. We implement both SearchDoc and SearchPsg by adapting Contriever. Given query, SearchDoc first uses Contriver to retrieve number of passages from a specific knowledge base and only retains the most relevant passage from each document serve as the document's ss representative snippet. Then, SearchDoc returns the top one document snippet and NextDoc can return at most nine more snippets from the remaining ones. On the other hand, SearchPsg returns the top three passages within given document retrieved using Contriever.
18

---

Answer(Q. H. P)
Please assess whether the sub-query detived from the main question can be answered using the information from the provided passages.
Your evaluation should categorize the sufficiency of the information in the passages with respect to the sub-query. Assign one of the following three calegories:
(1) [Unanswerable]:: Chonse this category if the given passages do not contain information to answer it directly. (2) lAnswerable]: Use this calegory one of the given passaees contains sufficient information directly unswer the sub query. Provide clear and concise answer to the sub-query, and the ID of the the corresponding passage.
HERE ARE SEVERAL EXAMPLES:
=EEE Examples Start=sss
(1) Main Question: Which magazine was started first Arthur's 's Magazine or First for Women?
Solved Sub Queries:
Q: When was First for Women magazine started? A: 1989
Next Sub Query: When Arthur's Magazine started?
Passages: [1] (title: Arthur's as Magazine) He was also the author of dozens
[2] (litle: Arthur's Magazine) Arthur's 's Magazine Arthur's Maguzine (1844-18466
[31 (title: Arthur's 's Magazine) The articles were widely reprinted and helped fuel... ....
Output: [Answerable] Answer: 1844-1846, Relevant Passage ID: [2]
(2) Main Question: What U.S Highway gives access to Zilpo Road, and also known Midland Trail?
Next Sub-Quii Lub--Qury:: How can Zilpo Road be accessed?
Passages: [1] (idee Zilpo Road) the city which transports people in and out of the city 121 (title: Zilpo Road) Grand Terrace. Access provides public transportation services [3] (title: Zilpo Road) On the other side of the lake is the 700-acre (280 ha) ...
Output: [Unanswerahle]
=m Examples End
Now Please Complete the Following Task:
Main Question: (Q)(H') [%7 formatted string representing solved sub-queries answers constructed from 1.%
Next Sub-Query: [g]
Passages: (P)
Output:
Table 12: Prompt for the Answer module for HotpotQA.
Complete(Q. E)
Answer the question ONLY based the provided passages. Your oulput should be "yev/no" short entily.
HERE ARE SEVERAL EXAMPLES:
Exampex Startsass
(1) Question: Which magazine started first Arthur's 's Magacine or First for Wonenn?
Passages: [1] (title: Arthur's 's Magazine) Arthur's Mugarine Arthur's 's Magazine (1 844 1846) was an
[2] (litle: First for Women) First for Women ... started in 1989
Output: Arthur's Magazine
(2) Question: What U.S Highway gives access Zilpo Road, and is also known as Midland Trril?
Passages: [1] (tille: Zälpo Road) Zilpo Road ... can be accessed by Kentucky Route 211 (KY 2112) [2] (title: Morehead, Kentucky) Morchead is home rule-class city[5] located along US 60 (the historic Midland Trail)
Output: US 60
Examples End=
Question: [Q)
Passages: (E')
[%E is a formatted string representing all evidence passages constructed from E.%
Output:
Table 13: Prompt for the Complete module for HotpotQA.
The operation of these tools mirrors the hierarchical interaction paradigm that humans use with search engines [5T [5]: they first identify a relevant document based on short snippets and then refine the search results by focusing within the document.
Performance comparison across different tool implementations. On PubMedQA and QASPER, we implement the tools of both AMOR and all baselines using Contriever for fair comparison. However, for the HotpotQA dataset, the basclines used various approaches for tool implementation, and Table 4 in the main paper directly shows the performance reported in their original papers. To bridge the gap between AMOR and basclines in tool implementations, we conduct experiments to investigate how AMOR performs when using the same tool implementation as two representative baselines, including LUMOS and AgentLM*) respectively. Please kindly note that LUMOS utilizes GPT-3.5 as tool for answering questions based on retrieved passages. Therefore, utilize CPT- 3.5 to implement the Complete module when comparing AMOR with LUMOS. Furthermore, the
"We do not include FIREACT in our comparison due to its reliance on the Google Search API, which exceeds budget constraints.
19

---

Table 14: Comparison results between AMOR and several representative baselines based on L-7B using the same tools on HotpotQA.
Method
SearchDoc
SearchPsg
Complete
EM
AMORWFR
Contriever
Contriever Contriever
Fine-tuned L-7B
30.4
AMOR Process
Contriever
Fine-tuned L-7B
43.8
LUMOS
Wikipedia API
DPR [18]
GPT-3.5
29.4
AMORWFT
Wikipedia API
DPR [88
GPT-3.5
31.2
MMOCCcs AMOR Process
Wikipedia API
DPR [8
GPT-3.5
41.4
AgentLM
Wikipedia API
Exact Keyword Match
Fine-tuned L-7B
22.3
AMORWFF WFT
Wikipedia API
Exact Keyword Match
Fine-tuned L-7B
32.0
AMOR Process
Wikipedia API
Exact Keyword Match
Fine-tuned L-7B
43.0
Table 15: Datasets for adaptation and evaluation. Avg. Len refers to the average length of passages in the corresponding knowledge base, counted by the GPT tokenizer [30]. Val is the validation set.
Dataset
Knowledge Base
Avg. Len
Train
Val
Test
HotpotQA
Wikipedia Articles
138
2,000
100
500
PubMedQA
PubMed Abstracts
303
445
QASPER
One NLP Paper
102
700
45
382
adaption to tool implementations of LUMOS and AgentLM necessitates several adjustments in AMOR, including (1) When using "Wikipedia API" to implement SearchDoc, the Decompose module should identify entity names for SearchDoc. (2) When using "Exact Keyword Match" to implement SearchPsg, the Decompose module should predict keywords for SearchPsg.
(3) We craft elaborate rules to create warm-up data that follow the above formats for the Decompose module. For example, we use the title of the gold passage as the target entity name for a sub-query and use the longest common string between each sub-query and the corresponding gold passage the target keyword for "Exact Keyword Match."
As demonstrated in Table 14 AMORWFT, without being fine-tuned on HotpotQA, surpasses both LUMOS and AgentL. in performance by employing identical tool implementations. Moreover, AMORProces: upon fine-tuning with process feedback on HotpotQA, exhibits substantial and significant enhancements in performance. These outcomes collectively underscore AMOR's 's superior performance compared to baseline models and its robustness and adaptability across various tool implementations.
B.2
Adaptation & evaluation datasets
We describe how we process the datasets as follows: (1) HotpotQA: Each document is Wikipedia article. Since the original test set is hidden, we randomly sample 500 examples from the original validation set for evaluation and split the training set for adaptation fine-tuning and validation. (2) PubMedQA [17] We follow the official split. And we only remain examples whose answers are "yes" or "no" and discard those labeled "maybe." (3) QASPER [8]: For each question, we regard the corresponding paper as a knowledge base and each section the paper as a document with the section name the title (e.g.. "Experiments::
including several passages. Although many LLMs support context longer than the average paper length of 7k tokens, we focus on testing the ability of language agents to seek and utilize information in this work. We exclude questions that labeled "unanswerable. Since the original test set is also hidden, we use the original validation set for evaluation and redivide the training set for training and validation. Table shows the statistics of the three datasets.
B.3 Reasoning process assessment
To investigate the extent to which the adaptation stage enhances AMOR's reasoning process, we conducted human study with one NI.P expert using the HotpotQA test set, Table TT demmonssrate the protocol for annotating the gold feedback fauman and then we calculate the accuracy of the
20

---

Table 16: Manual annotation strategy for gold process feedback for different LLM modules.
Module
Output y
Gold Feedback faumun
[NEXT] g [FINISH]
"right". q is reasonable sub-query solving Q: "wrong". otherwise.
Decompose H)
"right". if there no more sub-queries required; "wreng". otherwise.
(Q. g.
[ELEVVNT]] [IRRELRVANT]
"[RELEVANT]".
relevant q; "[IRRELEVANT]"
"right". if a is the correct answer to evidenced by e "wrong". otherwise [NSSERRBBLE] ae [NANSWERABLE]
Answer H. P)
"right". g
based "Wrong", otherwise
Complete (Q. E)
A
"right", if E evidence that Q can be unswered by A: A. clse if E evidence that can be answered by A: "wrong", otherwise.
Table 17: Accuracy of the silver feedback for four LLM modules based L-7B.
Method
Decompose
Judge
Answer
Complete
AMOR
A.ORProcess Process
81.3
95.2
84.4
82.0
Table 18: Proportion of cases where the corresponding error exists. All agents are based on L-7B.
N/A means the LUMOS agent does not explicitly execute the relevance judgment step.
Error Type
LUMOS
A
AMORProces
Format Error
5%
0 %
0%
Low Quality Retrieval
28%
18%
12%
Question Decomposition Error
68%
60%
44%
Relevance Judgment Error
N/A
18%
20%
Answer Extraction Error
46%
40%
28 %
Task Completion Error
72%
64%
48% %
automatic silver feedback by comparing it to the gold human feedback. Based on fhuman, we measured the accuracy of each L.M module's output y (denoted as ACC,,. as follows:
if fnuman "right";
if fimma is refinement of and fruman y.
ACCm
(4)
if fnuman "wrong".
if fimma is refinement of y and fruman * y.
The accuracy of the reasoning process ACC,, has been discussed in Table g of the main paper. Furthermore, Table 17 presents the accuracy the silver feedback ACC, for AMORPrecess The silver feedback achieves an ACC, above 80% for all modules, lending credibility to the use of silver feedback in the adaptation experiments.
B.4 Error analysis
Table of the main paper has presented a detailed analysis the accuracy of each module within AMOR on HotpotQA, which suggests the relative weakness of AMOR in question decomposition and task completion. We further conduct a comprehensive error analysis by manually examining the error proportion of different agents. We summarize six error types and show the results of manual annotation in Table [8
Our thorough analysis accentuates the specific strengths and weaknesses of different agents, unequiv- ocally demonstrating AMOR'S 's relative improvements across key aspects of the complex reasoning process.
Furthermore, we would like emphasize a significant advantage of AMOR: the ease of diagnosing errors by examining the outputs of its modular architecture. Unlike other agents, where errors are
21

---

Table 19: Performance of AMOR parameterized by during multi-round adaptation. In the i-th iteration (i = O, 1, 2). AMORs, is used to explore over the same set of questions or different ones and then is updated to AMORØ... based on the exploratory instances.
Same Questions
Different Questions
Metric
01
02
03
01
0g
03
EM
45.8
45.4
45.2
45.8
45.2
45.2
F1
54.9
53.4
54.7
54.9
54.5
53.6
Table 20: Average step/token numbers of different agents. For ReAct and AgentL.M, step refers to a "Thought," "Action," " or "Observation" step. For AMOR, a step means a reasoning step within certain module. And tokens count both input and output tokens.
Method
Base LLM
HotpotQA
PubMedQA
QASPER
ReAct
GPT-4
-/-
13.4/ 19.0k
17.5 / 25.3k
AMORw/o PT
GPT-4
19.3/ 11k
9.3 7.7k
10.9 6.3k
AgentLM
L-7B
-/-
11.5
12.3 8.9k
AMORProcess
L-7B
20.5 4.3k
11.1 2.66
11.4/2.1k
often intertwined and obscure (see ReAct example in Figure 6). AMOR facilitates the provision of precise, module- feedback. We will add the above analysis in our revision.
B.5 Multi-round adaptation
In the main paper, we set - 1 in Algorithm for for all experiments, which means that all exploratory instances in the adaptation stage induced by the warm-up policy AMORWFT- We call this setting "single-round adaptation." We are curious about how multi-round adaptation influences the performance of AMOR by adjusting 1. For the i-th iteration (i = 1,2, ....,.... 1)), we denote the initial parameter of AMOR as 0;-1, which is used to explore over a set of input questions and is updated to 0; after exploitation using these exploratory instances. AMORg, exactly AMORwFT. During different iterations, we can provide either the same or different questions for AMOR to explore over. The case with the same set of questions is used simulate an exploration-limie scenario. Note that this case, the exploratory instances with the same questions are still different due to the ever-changing policy leading to different outputs.
Table 19 shows the performance of AMOR under the multi-round adaptation setting with / - 3. We find that the performance is almost unchanged whether using the same different input questions for each adaptation round. This result suggests that one iteration may be sufficient for the adaptation fine-tuning stage in our study.
B.6 Token efficiency
Language agents interact with environments to solve problems through frequent calls of LLMs. leading to huge costs in terms of token consumption. Building agents with minimal token usage is essential for curbing deployment costs [46).
Table 20 displays the average number of steps and tokens used by AMOR and ReAct-style agents to answer a question. ReAct-style agents, lacking explicit modeling of inter-step dependencies, require the inclusion of all preceding information in the input for each step. This often results in undesired redundancy. contrast, AMOR consumes significantly fewer tokens with each module relying only on essential historical information, which highlights the token efficiency of its architecture. When built upon GPT-4, AMORw/ FT uses fewer steps but more tokens than AMORprecess based on L-7B due to the additional in-context examples inserted into the prompts of GPT-4.
22

---

Table 21: AMOR can be enhanced through targeted fine-tuning and flexibly accommodate additional
tools. All results are based on L-7B.
Method
EM
F1
AMORp Process
45.8
54.9
Targeted Fine-tuning of Complete
46.4
55.9
+ Additional Tool SearchDemo
46.8
56.7
B.7 Flexibility
FSM-based reasoning logic is flexible facilitating targeted enhancements of specific modules and easily accommodating new tools. We conduct two experiments as follows on HotpotQA to demonstrate the flexibility AMOR, with results shown in [T
(1) Targeted fine-tuning, Table 8 reveals that the Complete module of AMOR Process still falls short in performance, achieving only ~ 50% accuracy. We construct 6k examples for the module from the original training set of HotporQA by treating the final answer À as input, and the question Q and evidence passages E as output, and then fine-tune the L-7B model on the data. Table 21 shows the performance gains when substituting the original Complete module in AMORProces with this individually fine-tuned L-7B model.
(2) Accommodating new tools. Numerous studies have demonstrated the benefits of retrieval- based in-context learning, where a retriever selectively curates tailored demonstrations for each specific input [48]. We implement this by inserting before the final state 56 shown in the state transition diagram in Figure [ making AMOR reach s/g when Decompose outputs "[FINISH]] state sg. The new state s/g holds two variables, including the main question Q and all collected evidence E, and employs a tool module SearchDemo to retrieve the top K similar questions to Q from external demonstration memory, along with their answers and evidence, collectively noted as K. = [ux, Ås, F Subsequently, state 86, the Complete module takes K. as the in-context examples, helping generate the final answer A given Q and P.. We the HotpotQA training set our demonstration memory and employ Contriever-MS MARCO [15) to implement the SearchDemo module, setting K to 5. We fine-tune the L-7B model on the training set to act as the Complete module while ensuring that the demonstration does not include the target question. As T indicates, this integration of such an additional tool further improves AMORProcess with targeted fine-tuning.
state $/6. named "Demonstration Retrieval,"
Additionally, AMOR'S 's reasoning logic can be easily expanded from single-pati to multi-pati rea- soning, akin to the approaches used in Self-Consias Self-Consistency (41), ToT [50], and GoT [3]. This can be achieved by generating multiple outputs within specific modules and incorporating modules that synthesize these multi-path results. Consequently, we advocate for the adoption of the FSM paradigm the design of future agents. This framework offers the dual benefits of flexibility and the capacity adapt agents based on process feedback.
B.8
Case study
We annotate the silver outcome feedback fo for the Complete module at the final state as Á if all gold evidence passages are successfully collected (i.e., E C E), and "wrong" otherwise.
We demonstrate the advantages of the FSM-based reasoning logic and process feedback mechanism through the comparison between AMORProcess and ReAct Figure 5land respectively. We observe that ReAct without explicit reasoning logic constraints fails to decompose the question and terminates retrieval prematurely in "Thought/Action 5." Besides, ReAct also mixes right and wrong steps in "Thought 2/4/5," making it challenging for users critique and improve the agent in targeted manner. In contrast, AMOR successfully answers the question with a controllable reasoning logic and allows direct process feedback to drive the evolution.
Additionally, Table 22 shows a case where A AMORWFT correctly answers question with the right evidence, yet employs wrong reasoning process. This underscores the potential unreliability of using
23

---

Belewenne Judgmurs meg Judge (g.
Q- Id Cone for You single the bardd formed in whha Abbetta biy?
by. Quesiion Decompusiiio
Sy Documeet Rea loel
by Relexance Julgmentt Judge(Q. Kiq.4) (ecaovoll
5. Decamen Nadigation
Sy Relevsnce Indgmeet
Decospose 10 it)
Searchboclo
NextBock)
mig Judige (.
de
Wiich ae angk Color Yealy?
lile Hev You Calleoa hard to verive of
Fer Saan lulae pap
Passage Retrreaal
Answer Estratios
by: Pasaan Retrivall
sy Releuans judgnere
by. Answer Extraction my Answarr (0. el.e.. P1
Myy searchPsgle.d)
g.Pl
my S
<
my: Judge (9.)).;.)) Wetto
Ultiic 1St Sort wii
Home
(18b) Natturhie i Nokk soketon Sakkchacc up Gandia rokk
Nabalack
the U. Theaes bavrs baciig ois (1t Uud for hino sscend tasopen thenh Canata Nukaelind at
Rctiival
Question Decomposttion
Sy. Question Decompasitin
caau Nekabaok dh aorid your cttic Nutuehecc)
my Searchbocial
my: Decomposer () HF)
"Tick
Decorposelg.e
Direr aie slettcii on
(kkk End Conetter hout
femolla is Hold flams, Albett...
3x. Answer Extraction
Ky Thak Compleeen
Horss
2 E
myg Ansve ri o.P.
Cosp (emelee E)
Cornect Reasoning Path
Process Feedbock
dg Taok Completion
Incorrect Reasoning Path
Mai Coup (ete F)
Outputs of LLM Modules
ft. Question Dewoneodion Decoepose (g.
Ounputs of Tool Modules
Figure 5: An example demonstrating how AMOR Process answers a complex question from HotpotQA. Users are allowed to provide direct process feedback to drive the evolution of the agent.
Question:
single by
the
Let's
formation
Action chbo infommttiiitttiion
formed the Come U.S. iunnes Store, fueling debut the "Billoooro" Hon 100
Observation Come Yos?
Thought az hi neleont mation i found let's
neat document)
Action NextDocil
Observation at fitter Coming for tou) coming for You "Coming for Your is oong by the Amenn can pank noce band the
X
'Thonhh zu There's is still relevane information found, tet's searching for information about single
Action 3: Search totteaooosss"aaaemeeeee
absut the singi 13 comee for tout
Observation 3i pitee ltd Come for You #14 on the U.S. Tunne Store, fanking 1988 debut the "Mllbaor"" for 100...
Thought 4: No relevon for mation Leds searching BearchDocttaens Observation at lible Come You| Come
singll
the second European single (fouth in Canadal from Miceebaok's
Horse"
Thought ralevart olermation
I'd Come Your back Durk Hore"" and wetebaane formed riberts, when sseette intermert oen aboun We not information
Sand
trik perfectly another quection
Action F'ndshtfficate be hot enough infor natiion arswer question perfostyy)
Figure 6: A failure case of ReAct (built upon GPT-3.5) when answering complex question from HotpotQA by reasoning over retrieved knowledge. The wrong steps are highlighted in red while the right ones are in green.
Table 22: A case where AMORWFT answers question from HotpotQA with the right final answer and evidence but a wrong reasoning process. Each line denotes a reasoning step and shows the output of the corresponding module. The steps highlighted in green are right while those in red wrong.
Question: Were the bands Skin Yard and Ostava from the U.S.?
Decompose: [NEXT] Which band has Skin Yird been
SearchDoc: Sai Yard! Stutes America
with Finn still drumming
Judge: [R
SearchPag: resle: Yard) States America
reunited,
drumming
[3] (tilee Skin Yiud) Yard Skin Yard American grunge hand Scattio, Wishhington, were active from to 1993
[3] (till:: Skin Yird) ... He proluced several albums by the gringe bunds Soundgamten (a hand including original Skin Yird drummer Mant Cameron) ...
Answer: (ANSWERABLE) Answer: Scundgarden; Relevant Passage ID:
Decompose: [NEXT] Which band has Oslava boen part in?
SearchDoc: (niie: Ostsva) Ostova Ostava altemative rock hand fromt Bulgaria...
Judge: [VALE]
SearchPPg: resle: Ostaval Ostava Ostava in alsernative rock band from
[2] Ostava) live album.
supported Ladytnon their second this Sany Besch, Bulgaria
(stava) officially roleased version, they play cccssionally
Answer: [ANSWERABLE] Answe: Ladytoon; Resevunt Passage ID:
Decompose: [FINISH)
Complete: mo
outcome feedback to judge the correctness of the reasoning process and the necessity of employing process feedback for adapting agents to specific environments.
24

---

C Limitations
This study has demonstrated the benefits two components: (1) explicitly defined FSM-based reasoning logic, and (2) the process feedback mechanism. Nonetheless, notable limitation must be acknowledged when extending our approach to other tasks. While we have made initial efforts to outline the general principles for crafting the FSM in $3.1 and show the flexibility of adapting AMOR's FSM Appendix B.7, it still requires a human-driven design process. Looking ahead, our future work aims to enable LLMs to autonomously instantiate FSM-based reasoning logic in Algorithm for diverse user tasks, thereby reducing reliance on human design. Furthermore, we believe that the FSM-based reasoning logic makes it casier for humans supervise LLMs that potentially outperform humans on the task.
D Broader impacts and safeguards
The innovation introduced through AMOR carries significant potential for both positive and negative societal impacts.
On the positive side, the ability of AMOR to adapt to diverse knowledge environments and domains through supervised reasoning could lead to advancements in personalized education, healthcare diagnostics, and customer service. Such applications could democratize access to information and expertise, bridging gaps in knowledge and service availability across different regions and socioeconomic groups.
However, the sophisticated reasoning capability of AMOR also brings about considerations of misuse, such as the generation of disinformation aiding in the automation of social engineering attacks. Furthermore, if not properly balanced, the tailored knowledge adaptation could unintentionally reinforce biases present in the training data or human feedback, leading to unfair outcomes in decision-making processes that might disproportionately affect marginalized groups.
To mitigate these concerns, it's crucial to engage in transparent development and deployment practices, including bias audits and the establishment of ethical guidelines for use. Additionally, mechanisms for detecting and correcting misinformation or biased reasoning paths should be incorporated into the system's design.
Given the potential for misuse inherent in powerful language models like AMOR, it's vital to implement safeguards. AMOR will be made available under a framework that requires users agree to ethical usage guidelines before access. Our intention is to maximize AMOR'S societal benefits while curtailing the potential for negative impacts, thus ensuring responsible deployment and use of this ad advanced agent framework.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose general framework for building knowledge agents, featuring FSM- reasoning logic and a process feedback mechanism. We focus on text corpora knowledge bases, but the approach can be flexibly extended other knowledge types and user tasks by customizing the modules and dependencies within the FSM framework
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please Appendix Section C
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to
make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-souree dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
27

---

Justification:
The authors should provide scripts to reproduce all experimental results for the new reproducible, they Please see the supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips...
public/guide CodeSubmi ssionPolicy for more details. While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https)
//nips cc htte/uublic// wwtpc///wie//eeeeeeeooooeoe http:/udddee//eddueeooee Sttps//ooSSuuuiiaaaa wwwiisssiioooioiooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
28

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:p/s/..aarrrr......opppllbea
Answer: [Yes]
Justification:
We have read and conformed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix Section D
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact
or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is
29

---

being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please see Appendix Section D.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [paperswi www.thhe.d.e...oo com/ dataasttsssssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please see Supplementary File
Guidelines:
The answer NA means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
30

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We hire an NLP expert for manual annotation and pay him $7.25/h. Please see Table for detailed annotation instruction.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: IRB is not applicable to our country. But all potential risks were fully disclosed the participant prior his involvement in our study. We ensured that the participant provided informed consent, being fully aware of what the research entailed and any risks they might face.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "3MkpYyAQvF", "reviewer": "Reviewer_4AJ6", "summary": "This paper proposed an architecture for advanced reasoning in LLMs. The architecture contains several modules dedicated to different tasks in the reasoning flow, each of which can be trained separately using related datasets constructed from public datasets. The proposed method disentangles the reasoning process into sub-steps that are easier to train and suffer less from sparse feedback which is a critical problem for reasoning in traditional LLMs. In the experiments part, the authors conducted sufficient empirical studies, showing the effectiveness of the architecture proposed.", "questions": "See \"Weaknesses\" part. My concerns listed there are also my questions to the authors.", "limitations": "The authors discussed the limitations of their method adequately in the \"Limitations\" part in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. There's a clear analysis of the bottlenecks that limit the reasoning ability of LLMs. The architecture proposed is designed to solve these problems accordingly. The motivation is clear and the method is reasonable.\n2. The structure of the proposed method is well-explained with text and pictures, making it easy for reviewers and readers to understand.\n3. In experiments, the choice of baselines has good coverage. The dataset selection respects each subsection's theme, and the experiment setup is reasonable. The experiments are well explained, making it easy to reproduce them.", "weaknesses": "1. Currently, many works focus on enhancing the reasoning capability of LLMs. Although the authors mentioned previous works about LLM reasoning and RAG in the related works part, an in-depth explanation of why this method surpasses its predecessors is still needed. A good way to do this is to explain clearly, probably with a few pictures or equations, how those methods dealt with the difficulties and their advantages/drawbacks, and then show that AMOR is better as it avoids some of the existing drawbacks in previous methods.\n\n2. The state transition diagram is well displayed. However, when readers look into the implementation of each module, some key information such as the architecture, hyperparameters, and techniques used for better performance is missing, making it more difficult to understand and reproduce the whole model. A table of these details or a later public code repository would be appreciated.\n\n3. In section 3.3, the detail of how the policy is trained is not well-explained. Did the author use parameterized policies such as policy networks and use a policy-based method in RL for optimization? If so, is the loss presented in equation (3) an auxiliary loss to the loss in policy gradient methods or the pure source of gradients?\n\n4. Following the above question. Will the gradient induced by feedback go through the whole model or only certain blocks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Regarding the difference between AMOR and prior reasoning methods\n\nPlease kindly refer to \"Author Rebuttal by Authors\" provided at the very beginning.\n\n> Weakness 2: Regarding the technical details.\n\nWe acknowledge the importance of transparency and reproducibility in scientific research, and therefore we have included the implementation details, including hyper-parameters,  in Lines 204-210 of our manuscript. Additionally, we have submitted the source code for MA-MoE implementation as part of the supplementary material. We intend to release a publicly accessible code repository after the double-blind review period. This repository will facilitate follow-up research to replicate and extend our work.\n\nTo further address the reviewer's comments, we plan to include an additional supplementary section in the final version of our paper, which will provide a comprehensive explanation of the algorithm underlying AMOR, elucidating the reasoning process in detail.\n\nWe believe that these actions will adequately address the reviewer's concerns, thereby enhancing the clarity and reproducibility of our model.\n\n\n> Weakness 3: Regarding the training details\n\nEquation 3 is the weighted sum of the standard RLHF optimization objective over all modules. Each module corresponds to a parameterized policy network $\\pi_{\\theta_m}$. The optimization objective is to maximize the reward with a KL divergence penalty. In this paper, we adopt the KTO algorithm [1] for optimization. Equation 3 illustrates the gradients for KTO.\n\nWe brief the derivation from Equation 3 to a differentiable loss as follows: As introduced by prior works [1, 2], it is straightforward to show that the closed-form expression for the optimal reward function $r^*_m$ for each module $m$ takes the form: $r^*_m=\\beta\\text{log}({\\pi\\_{\\theta_m}}/{\\pi\\_{\\theta_m}^{\\text{old}}})+\\beta\\text{log}Z_m$, where $Z_m$ is the partition function. Applying the expression to the Kahneman-Tversky model [1], we can get the differentiable KTO loss. Please kindly refer to the KTO paper for more details. We will provide detailed descriptions to improve the clarity of my work in the revision.\n\n[1] Ethayarajh et al. KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024.\n\n[2] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.\n\n\n> Weakness 4: Regarding the gradients from the feedback\n\nIn the MA-MoE architecture, different modules correspond to different FFN layers and share the same embedding layers and multi-head self-attention layers. Therefore, regarding a training example from $\\mathcal{R}_m$ for module $m$, the gradient induced by the feedback will go through the whole MA-MoE model, except those FFN layers corresponding to other modules.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ReVSq7E14o", "reviewer": "Reviewer_CkwG", "summary": "This work presents a modular pipeline for QA tasks. The pipeline consists of several modules such as question decomposition, document/passage retrieval, answer extraction, etc. Training data is separately constructed for each module (based on existing datasets) and models are individually fine-tuned for the respective modules. This processes is referred to as the 'warm-up' stage. Models are further updated based on feedback obtained during inference using an alignment algorithm (KTO is used in this work). Experiments on QA datasets shows that the proposed method performs better than various baselines in the literature. Several ablations are also provided to study the impact of each component.", "questions": "* What is the without FT setting in Table 6? \n* Are the comparisons against baselines fair? Does the proposed approach use more data compared to the baselines? \n* What is the major delta of this paper compared to prior modular reasoning works (e.g., Rewoo, Lumos)?", "limitations": "I did not find a discussion of limitations in the papers, I suggest the authors to add some discussion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed modular approach is sensible and performant.\n* Extensive baselines were considered. \n* Several ablations were performed that show the impact of various components. It is interesting that the KTO based approach performed better than a simple SFT approach. The human feedback experiment was also interesting. \n* Strong performance on three QA benchmarks (HotpotQA, PubmedQA, Qasper). \n\nOverall I think this paper presents interesting ideas. Various presentation aspects can be improved (see weaknesses).", "weaknesses": "* The scope for this work was not properly introduced. The paper is motivated from very broad goals only for the reader to later find that the approach and experiments focus on QA tasks. In fact, the paper does not present a formal task description or problem formulation before presenting the methods, leaving the reader to piece together things. \n* Method description: In addition to the task not being formally presented, several notations were not clearly defined. For example, equations 1, 3 start talking about a policy which was not described earlier. Although I do understand what's happening in these equations, clearly defined problem and terminology would have made things much clearer.\n* In general, I felt the ideas in the paper could've been framed/described in a much simpler way. Readability could be improved.\n* The introduction provides motivations but fails to convey any details/intuitions of the proposed method or how it overcomes the challenges mentioned.\n* The 'processed based feedback' was not clear to me. I think the paper tries to squeeze in many ideas but this also distracts the reader from the core ideas. \n* The experimental setup could have been made clearer, including the various settings such as with/without finetuning and process/outcome feedback. \n* Figures and tables are just too small and impossible to parse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Regarding the scope and problem formulation\n\n- **Scope.** AMOR aims to develop a general framework for building adaptable modular LLM agents that can leverage external knowledge sources to tackle complex reasoning tasks. However, we appreciate the reviewer's advice that being more explicit in framing the specific tasks and domains upfront would strengthen our readability. We will clarify the scope early in the next revision.\n- **Problem Formulation.**  We also appreciate the reviewer's advice to provide a clear task formulation upfront. In the revision, we will add a problem statement and formulation section, which will precisely define the QA-style reasoning tasks we use AMOR to solve. \n\n> Weakness 2: Regarding the method description\n\nIn Equations 1 and 3, the policy $\\pi$ refers to the strategy of the MA-MoE model that maps from the state $s$ to an action $y$. \n\nWe will explicitly define it to improve clarity in the revision.\n\n> Weakness 3: Regarding simpler presentation\n\nAMOR uses several novel techniques, including FSM-based reasoning logic, process feedback mechanism, and two-stage fine-tuning strategy. The paper is organized to provide clear definitions, examples, and explanations without oversimplifying the technical details. We will simplify the content in the revision and welcome specific advice from the reviewer on how we could improve the clarity.\n\n> Weakness 4: Regarding the details of AMOR overcoming challenges in the introduction\n\nWe believe we have made every effort to convey the key insights and intuitions behind AMOR and how it addresses the stated challenges.\n\nAs summarized in Tab. 1, current agents face challenges in three main aspects: **(1):** uncontrollable reasoning logic; **(2):** lack of adaptation mechanisms for new environments; and **(3):** difficulty for humans to intervene in the reasoning process.\n\nIn the fourth paragraph (Lines 34-41), we outline the core idea behind AMOR. The FSM-based reasoning logic enables AMOR to solve problems via executions and transitions over a set of modules, allowing for process-based feedback. This design directly addresses **challenges (1)** and **(3)**.\n\nIn the fifth paragraph (Lines 42-50), we provide technical details on AMOR's adaptive mechanism, which tackles **challenge (2)**.\n\nWe welcome any further feedback from the reviewer on how we could improve the presentation of our work in the revision.\n\n\n> Weakness 5: Regarding process-based feedback\n\nAlg. 1 shows that AMOR solves problems through executions and transitions over a set of modules. In each reasoning step, AMOR executes the module $m$ while in a specific state $s$, obtains the output $y$, and transits to the next state. The overall reasoning process $R$ is formed by a series of such steps. Each step can receive human feedback, termed process-based feedback, as opposed to outcome feedback typically provided for the final step. We will elucidate this concept more explicitly in the revision.\n\n> Weakness 6: Regarding the experimental setup\n\nWe provide additional details about the experimental setup below:\n\n- **Without Fine-tuning:** We apply different methods directly to off-the-shelf LLMs without fine-tuning. We provide in-context examples to instruct LLMs to solve given problems. Tab. 12-15 in Appendix A.1 show the prompts for four LLM modules in AMOR under this setting on HotpotQA.\n- **With Fine-tuning:** LLMs are fine-tuned on specific datasets. For AMOR, we employ a two-stage fine-tuning strategy:\n    - **Warm-up Fine-tuning:** The stage fine-tunes an LLM on trajectories from public datasets.\n    - **Adaptation Fine-tuning:** The stage allows AMOR to adapt to specific environments by leveraging different forms of feedback:\n        - **Process Feedback:** AMOR is optimized using feedback provided for both intermediate and final steps of its reasoning process.\n        - **Outcome Feedback:** AMOR is optimized using feedback provided only for the final step of its reasoning process.\n\nWe will include these clarifications in our revision.\n\n> Weakness 7: Regarding the table and figure size\n\nWe will explore ways to enhance their clarity in the revision.\n\n> Question 1: Regarding the fairness between AMOR and baselines\n\nWe have carefully designed our experimental setup to ensure fair comparisons in terms of training data sizes. It involves two groups of comparisons:\n\n- **Comparisons between AMOR$_{\\rm WFT}$ and baselines that are NOT fine-tuned on the target datasets:** The table below lists the training data sources and sizes for AMOR$_{\\rm WFT}$ and baselines.\n    \n    \n    | Methods | Data Source | Size |\n    | --- | --- | --- |\n    | FireAct | GPT-4 Trajectories | 2.5k |\n    | AgentLM | GPT-4/3.5 Trajectories | 1.8k |\n    | Self-RAG | Public Resources | 145k |\n    | LUMOS | Public Resources | 57k |\n    | AMOR$_{\\rm WFT}$ | Public Resources | 50k |\n    - While FireAct and AgentLM use fewer data than AMOR$_{\\rm WFT}$, they rely on the GPT-4/3.5 API for data annotation, which hinders them from scaling up training examples. We believe that AMOR's ability to achieve sota results without any proprietary LLMs is a substantial milestone.\n    - Compared to Self-RAG and LUMOS, which also use public resources, AMOR$_{\\rm WFT}$ uses fewer or a comparable number of training examples.\n    \n    In summary, this group of comparisons is fair, as AMOR$_{\\rm WFT}$ uses fewer or a comparable number of data from public resources without relying on proprietary LLMs.\n    \n- **Comparisons between AMOR$_{\\rm Process}$ and baselines that are fine-tuned on the target datasets:** AMOR$\\_{\\rm Process}$, AMOR$_{\\rm Outcome}$ and OneR use the same training data from the target datasets, leading to a fair comparison. Table 3 shows the data statistics.\n\nWe will provide detailed explanations in the revision.\n\n\n> Question 3: Regarding the difference between AMOR and prior modular agents\n\nPlease kindly refer to \"Author Rebuttal by Authors\" provided at the very beginning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YrgVvsdIEj", "reviewer": "Reviewer_QRnu", "summary": "This work proposes AMOR, a modular approach to building knowledge agents using open-source LLMs. AMOR decomposes tasks into reasoning logic, represented as a finite state machine (FSM) composed of sequentially chained modules. These modules include tools e.g, document retrieval and LLMs, e.g., answer extraction. AMOR retrieves relevant documents for a query and combines information from different sources to produce an answer.\n\nThe development process of AMOR includes three stages:\n1. **Warm-up Phase:** Each module is fine-tuned on datasets containing not only input-output pairs but also the relevant intermediate steps for each module. This phase ensures that AMOR can generalize across various tasks and knowledge environments.\n2. **Adaptation Phase:** The agent is further fine-tuned on specific domains using process feedback. This feedback, which can be human-generated or derived from evaluation benchmarks, is provided at each reasoning step to refine the agent's performance.\n\nEmpirical evaluation demonstrates that AMOR effectively utilizes the rich warm-up and evaluation datasets, outperforming relevant baselines. Ablation studies confirm that each component of AMOR is essential for achieving maximal performance.", "questions": "How are the modules chosen in MA-MoE? Is the module/step ID provided to the routers?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This work is well-motivated and addresses a need for knowledge agents. The modular FSM-based approach is innovative and sensible, allowing for precise reasoning logic. The paper is well-written, and the empirical study is thorough and well-conducted, demonstrating the method's effectiveness through extensive experiments and ablation studies.", "weaknesses": "All experiments do not report any measure of uncertainty. Thus, it is impossible to determine if the conclusions are statistically significant. Including measures of uncertainty would strengthen the validity of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness: Regarding the measure of uncertainty.\n\nWe agree that it is crucial to provide measures of uncertainty to assess the statistical significance and robustness of the results. To address this concern, we show the mean and standard deviation across three independent runs in the table below.\n\n| Method | Base LLM | HotpotQA EM | HotpotQA F1 | PubMedQA Acc | Qasper EM | Qasper F1 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| AMOR$_{\\rm Process}$ | L-7B | $45.8{_{\\pm0.25}}$ | $54.8_{\\pm0.26}$ | $81.6_{\\pm0.33}$ | $19.0_{\\pm0.05}$ | $35.4_{\\pm0.5}$ |\n    \nAs shown in the table, the standard deviations across multiple runs are very small, suggesting that our approach yields consistent and robust results. This strengthens the validity of our conclusions and provide confidence in the reported performance.\n\nWe will incorporate the uncertainty measures in the revised manuscript to enhance the transparency and credibility of our work.\n    \n> Question: Regarding module choosing\n\nEach module corresponds to a distinct expert in the MA-MoE model. When AMOR executes a certain module, its module ID will be provided to the routers of the MA-MoE model to indicate which expert should be activated, thereby enabling our model to be “module-aware.” We will endeavor to provide a more comprehensive explanation of this module-expert mapping mechanism in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NcJykk6t8q", "reviewer": "Reviewer_bbCe", "summary": "This paper proposes AMOR, which is a modular framework for answering questions by reasoning over external knowledge bases. AMOR is designed as a Finite State Machine (FSM), which provides a structured way to break down the question into smaller pieces and solve complex tasks through various steps. Each state in AMOR is a module, which can be an LLM call or a tool call. Each LLM module is fine-tuned separately using data relevant to only that step, thus making data collection and evaluation easier.", "questions": "I don't have any questions", "limitations": "The authors were transparent about possible limitations and negative impacts in the **Broader impacts and safeguards** session and also included suggestions on how to mitigate these problems.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors propose an innovative framework that incorporates FSMs, which is a \"classical\" technique, with state-of-the-art LLM and tool calling that gracefully incorporates human feedback. The text is well-written and clearly explains all parts of the framework. The figures are also helpful in understanding their proposal. I can see AMOR framework being useful in real-world scenarios like question-answering in private datasets. The paper also presents extensive experiments using popular datasets like HotPotQA and PubMedQA.", "weaknesses": "The weaknesses I see are the complexity of this framework: many moving parts, including various LLMs that need to be fine-tuned separately; and the 2 stages fine-tuning approach (warm-up + fine-tuning) makes it even more complex and resource intensive. Dependency on human feedback is another weakness I can see here. However, these are weaknesses common to other agents frameworks so I wouldn't consider this as a disqualifier for this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Regarding the complexity and human feedback dependency.\n\nWe would like to address your concerns as follows:\n\n- **Regarding the “Separately Fine-tuned LLMs”:** We would like to emphasize that this argument might be inaccurate since we use the same MA-MoE model for all modules and activate different experts to execute their corresponding modules. This approach eliminates the need for separately fine-tuning multiple LLMs.\n- **Regarding the Two-Stage Fine-Tuning:** We argue that the two-stage fine-tuning process is analogous to the widely adopted \"pretrain and fine-tune\" paradigm. The warm-up stage, which is performed only once, enables AMOR to generalize across different knowledge environments. Once the warm-up stage is complete, AMOR can be deployed and adapted to various domains through the efficient adaptation stage. We believe that the benefits of improved generalization and domain adaptation justify the additional computational cost.\n- **Regarding the Dependency on Human Feedback:** We underscore the importance of maintaining an adaptive mechanism that leverages human feedback to continuously improve performance. Previous agent frameworks often overlook this crucial aspect. AMOR's process feedback mechanism enables efficient and targeted feedback, reducing the overall burden on human supervisors.\n\nIn summary, while AMOR introduces complexities, its modular design, two-stage fine-tuning, and process feedback mechanism are deliberate choices that enable it to handle complex tasks effectively and adapt to specific domains efficiently. We believe that the benefits of improved performance and domain adaptability outweigh the concerns raised regarding complexity and human feedback dependency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
> Regarding the difference between AMOR and prior reasoning methods

Please kindly refer to the attached pdf file for an illustration of the reasoning processes of AMOR and prior reasoning methods. The table below further elaborates the advantages and drawbacks of prior agents in terms of the following three aspects.

| **Prior Reasoning Methods** | **Reasoning Logic** | **Adaptive Mechanism for New Environments** | **Human Intervention in the Reasoning Process** |
| --- | --- | --- | --- |
| **Retrieval-Augmented Generation** (e.g., Self-RAG) | Sequential Pipeline. **Drawback:** It is difficult to handle complex tasks. | $\times$ (*Undefined*) | $\times$ (*Undefined*) |
| **Agents with Modular Reasoning** (e.g., LUMOS) | *ditto* | Prompting or Imitation Learning from Humans/LLMs. **Drawbacks:** The former often leads to suboptimal results, while the latter suffers from the scarcity of high-quality data. | $\times$ (*Undefined*) |
| **Agents with Free-form Reasoning** (e.g., AgentLM, FireAct) | $\times$ (*Undefined*) | *ditto* | Outcome Feedback. **Drawbacks:** **(1)** Outcome feedback alone is often too sparse and insufficient to improve the intermediate reasoning steps effectively [1]; **(2)** The reasoning steps taken by LLMs can frequently contradict or deviate from the desired outcome [2]. |

In contrast, AMOR addresses the issues of prior reasoning methods in terms of the above three aspects:

**(1) AMOR is equipped with a controllable FSM-based reasoning logic with a stronger capacity for handling complex tasks than simple pipelines employed by Self-RAG, ReWOO, LUMOS.** For instance, if no relevant passages are retrieved from a document, AMOR can dynamically transit to the next document, while LUMOS would be constrained to generate answers based on the irrelevant passages, potentially leading to incorrect or low-quality outputs. 

**(2) AMOR adapts to new environments through exploration and exploitation .** AMOR is designed with an adaptation fine-tuning stage, enabling it to adapt effectively to specific domains based on human feedback. This adaptive mechanism sets AMOR apart from prior modular agents that lack the ability to incorporate expert guidance. 

**(3) AMOR enables humans to conveniently and effectively intervene and provide feedback at each reasoning step.** AMOR introduces a process feedback mechanism that enables humans to provide direct feedback on the individual modules within the FSM-based reasoning process. This approach facilitates a more natural and interpretable form of supervision, allowing for targeted improvements and fine-tuning of specific reasoning components.

In summary, AMOR achieves more controllable, adaptable, and human-guided reasoning capabilities compared to existing methods. We hope this clarifies how AMOR advances the state-of-the-art in building adaptable and modular knowledge agents. We will include the clarification to better illustrate the proposed method in the final version.

[1] Lightman et al. Let’s verify step by step. 2023.

[2] Liu et al. Score: A framework for self-contradictory reasoning evaluation. 2023.

Meta-review: The authors propose a modular pipeline for QA tasks that performs reasoning over external knowledge bases. The system, AMOR, is designed as a Finite State Machine, whch decomposes complex questions into smaller pieces, then each piece can be handled by different modules such as document retrieval or answer extraction. Each module can be fine-tuned individually, using existing datasets (during warm-up phase); the agent can further be fine-tuned on specific domains using process feedback (via KTO) obtained from either human annotators or an evaluation benchmark. 

All reviewers agree that the paper is above the bar. In particular, the reviewers consider this work as well-motivated, addressing important problem; innovative and sensible; experiments are well designed and presented; the method shows strong performance against a good set of baseline systems. 

However, multiple reviewers have concerns about the writing, especially the clarity of the submission. To help the authors further improve their work, the reviewers suggested the following:
- To better connect AMOR with prior reasoning methods (like in their rebuttal message to all reviewers). 
- Make the KTO objective part clearer.
- Improve the clarity of method description, especially the process-based feedback. 
- Make the experiment setup more structured.
- Add error bars.

In my opinion, these suggestions require minor change to the paper, some of them has been done already during the rebuttal discussion. In addition, there is no major concern regarding the method nor the experiments. I believe this work could be found useful to the NeurIPS community so I recommend to accept this paper.
Decision: Accept (poster)