AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback

Authors: jian  guan
wei  wu
zujie  wen
peng  xu
hongning  wang
minlie  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:12:00

Abstract: The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present AMOR, an agent framework based on open-source LLMs, which reasons with external knowledge bases and adapts to specific domains through human supervision to the reasoning process. AMOR builds reasoning logic over a finite state machine (FSM)
that solves problems through autonomous executions and transitions over disentangled modules. This allows humans to provide direct feedback to the individual modules, and thus naturally forms process supervision. Based on this reasoning and feedback framework, we develop AMOR through two-stage fine-tuning: warm-up and adaptation. The former fine-tunes the LLM with examples automatically constructed from various public datasets, enabling AMOR to generalize across different knowledge environments, while the latter tailors AMOR to specific domains using process feedback. Extensive experiments across multiple domains demonstrate the advantage of AMOR to strong baselines, thanks to its FSM-based reasoning and process feedback mechanism. The code and data are publicly available at
https://github.com/JianGuanTHU/AMOR.

Main paper: 
 introduction: 
 LLMs, with astounding performance over general natural language processing (NLP) problems [42, 1, 36] , have spurred great interest in building LLM-based agents to solve complex tasks by interacting with external resources such as web knowledge [27] , specialized tools [31] , etc.We focus on developing agents for knowledge-intensive tasks, where the agent completes users' information-seeking requests by interacting with specific knowledge bases [22] . To address the complexity of such tasks, we posit the desiderata for a qualifying agent as follows: Firstly, the agent should possess a robust reasoning logic about the task to solve individual problems with precise pathways. Secondly, the agent should maintain an adaptive mechanism to adjust to specific environments, rather than staying static. Thirdly, the reasoning process should be amenable to human interventions, enabling humans to steer the agent's behavior through direct feedback to the process rather than only to the outcome. This ability can significantly facilitate alignment between agent behavior and human intent [39] .Although extensive studies have been conducted on building language agents, few, if any, can fulfill all the required criteria due to their uncontrollable reasoning logic, static model capability, or WebGPT [27] Tool Invoking Undefined Imitation Learning from Humans Outcome CoT [43] Reasoning Undefined Prompting Undefined ToT [50] Reasoning Undefined Prompting Process ReAct [51] Reasoning&Tool Invoking Undefined Prompting Undefined Reflexion [35] Reasoning&Tool Invoking Undefined Prompting Process AgentLM [53] Reasoning&Tool Invoking Undefined Imitation Learning from LLMs Outcome MetaGPT [14] Specialized Module Sequential Pipeline Prompting Process LUMOS [52] Specialized Module Sequential Pipeline Imitation Learning from Humans Undefined AMOR Specialized Module Finite State Machine Exploration&Exploitation ProcessFigure 1 : AMOR's state transition diagram. Each box represents a state and the corresponding module that is executed when entering the state. There may be multiple categories of execution results distinguished by special branch tokens such as " [NEXT] ." Then AMOR determines the next state based on the branch tokens.sparse/missing feedback signals, as detailed in Table 1 . Consequently, it is still challenging for users to critique, and thus guide existing agents to follow targeted manners, especially when the agents are built upon less powerful LLMs [25] .We introduce AMOR, an Adaptable MOdulaR knowledge agent that can reason and adapt, with the reasoning process amenable to human supervision, based on open-source LLMs. AMOR's reasoning logic is formalized as a finite state machine (FSM) [7, 21] that solves problems via a series of executions and transitions over a set of modules (Figure 1 ). This naturally enables the desired processbased supervision mechanism, allowing users to give feedback to each LLM-controlled module. AMOR supports flexible forms of feedback, either binary judgments regarding the correctness or refinement of the outputs. The reasoning logic and process feedback mechanism together frame how AMOR thinks, acts, and interacts with users and task environments.We build AMOR upon an LLM equipped with distinct parameters for different modules to efficiently handle multiple tasks. The training in AMOR happens in two stages: (1) Warm-up: the modular design enables us to construct training data separately for each disentangled module without requiring complete trajectories for specific tasks. As a result, we create a large dataset of 50k examples covering multiple distinct tasks, simply using public datasets. We fine-tune AMOR on this data for generalization over various knowledge-seeking scenarios.(2) Adaptation: when deployed, we tailor AMOR to the target domain by letting it autonomously address user tasks (i.e., exploration), collecting process feedback for each LLM output, and evolving through further fine-tuning on the exploration trajectories with feedback (i.e., exploitation). Our contributions are summarized as follows:I. We propose a general framework for building knowledge agents, featuring FSM-based reasoning logic and a process feedback mechanism. We focus on text corpora as knowledge bases, but the approach can be flexibly extended to other knowledge types and user tasks by customizing the modules and dependencies within the FSM framework.II. Experiments across multiple domains show the strong advantage of the FSM-based reasoning logic with 30%-40% improvements over baselines when based on off-the-shelf LLMs (e.g., GPT-4 2 ). Switching to fine-tuned LLMs, the warm-up stage empowers AMOR to generalize to multiple domains and surpass strong baselines. After we adapt AMOR to specific domains, subsequent domain-specific adaptations reveal that process feedback is significantly more effective in improving the reasoning process than outcome feedback.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We propose a general framework for building knowledge agents, featuring FSM-based reasoning logic and a process feedback mechanism. We focus on text corpora as knowledge bases, but the approach can be flexibly extended to other knowledge types and user tasks by customizing the modules and dependencies within the FSM framework Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Language agents. Interest is surging in building agents for tasks necessitating multi-step reasoning.Existing work falls into two groups. The first group focuses on designing agent architectures, such as CoT's step-by-step reasoning [44] , ReAct's integration of reasoning, action, and observation to allow tool use [51] , and CODEPLAN's two-stage reasoning framework that first generates a codeform plan and then realizes low-level reasoning steps [45] . Nevertheless, such free-form reasoning constraints human intervention. In contrast, modular agents follow a pipeline to execute specialized modules [19, 14, 11, 3, 52] , improving the ease of intervention. The second group aims to design adaptive mechanisms for adapting agents to specific scenarios. ToT [50] and Reflexion [35] use environment feedback for multi-path pruning and iterative single-path refinement, respectively, but suffer from poor inference efficiency and need for real-time feedback. As a fine-tuning approach, recent work equipped open-source LLMs with specific agent abilities by learning from examples synthesized based on human priors [5] , or expert trajectories from humans [27] or GPT-4 [53, 4] with correctness validation through outcome feedback. In contrast, our modular agent AMOR employs FSM-based reasoning with a stronger capacity for handling complex tasks than simple pipelines and adapts effectively to specific environments via process feedback.Retrieval-augmented generation (RAG). The RAG paradigm augments the inputs of LLMs with retrieved passages to enhance factuality [12, 22, 10] . Recent studies have developed interleaved reasoning-retrieval for better information recall than one-step retrieval [38, 16, 28] . However, retrieval may introduce noise that leads to low-quality answers [34] . To tackle this, Self-RAG [2] trained LLMs to selectively perform retrieval and utilize retrieved passages. Unlike RAG approaches, AMOR emphasizes an explainable reasoning process for proactively decomposing questions and seeking evidence for grounded generation, and allows for process feedback from humans. Nevertheless, RAG mainly focuses on integrating parametric factual knowledge in LLMs and retrieved non-parametric knowledge, which is less explainable and intervenable.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Please see Appendix Section C Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
amor agent algorithm 1 fsm-based reasoning logic: 
 Input: Agent at the state s = s0; Q: Question. Output: A: Final Answer; R: Reasoning Process. R.append({"state": s, "output": y}) 5 A = y 6 return A, R AMOR relies on three key techniques: FSM-based reasoning logic, a process feedback mechanism, and a two-stage fine-tuning strategy. We detail the definition of the reasoning logic and its specification assuming the knowledge base is a text corpus in §3.1, the method for fine-tuning open-source LLMs as a warm-up stage in §3.2, and the adaptation stage driven by process feedback in §3.3.1 R = [ ] 2 while s ̸ = sN-1 doAlgorithm 1 outlines how to deduce the answer A for an input question Q with a reasoning process R using FSM-based reasoning logic, which can be defined by a quadruple: {S, M, E, µ}, where • S = {s 0 , . . . , s N -1 } is a set of states with s 0 as the initial state and s N -1 as the final state. Each state holds variables to track context information. • M = {m 0 , . . . , m N -1 } is a set of modules with m k triggered when the reasoning flow reaches state s k . The modules are categorized into two types: (a) Tool modules (M TOOL ) for invoking tools, and (b) LLM modules (M LLM ) for calling LLMs. • E is the set of all possible outputs of M.• µ : S × E → S is the transition function that determines the next state of the reasoning flow given the current state and the execution result of the corresponding module.When the external knowledge base is a text corpus, an instantiation of the reasoning logic can be represented by the state transition diagram in Figure 1 . In this case, M TOOL perform document and passage retrieval using external retrievers; while M LLM leverage the LLM to analyze and digest the question, documents, and passages to deduce the final answer. To distinguish different types of outputs from a module that requires different subsequent modules, we employ a set of special branch tokens such as "[NEXT]" to guide µ to determine the next state. In summary, AMOR answers question Q by (1) iteratively decomposing Q to a sub-query q at state s 0 , and finding the answer a to q and the evidence passage e through iterative knowledge retrieval, relevance evaluation, retrieval refinement (i.e., "Passage Retrieval"), and answer extraction, until no more knowledge is needed; and(2) deducing the final answer A based on the collected evidence passages at the final state.Defining reasoning logic as an FSM offers three advantages: (1) Structured Thinking. FSM makes specifications of inter-step dependencies (e.g., prioritization, branch selection) easy, and thus enables narrowing down the exploration space. (2) Skill Disentanglement. By decomposing complex tasks into modular steps, one can independently construct training data for each module, which significantly reduces the difficulty of implementing AMOR with open-source LLMs (cf., §3.2). This feature also allows AMOR to focus on single steps, thereby mitigating the weakness of LLMs in reasoning over long context formed by task-solving trajectories [24] . (3) Intervenable Workflow. The structured reasoning process enables users to easily diagnose the agent's mistakes and provide process feedback for improving the reasoning capability of the agent ( §3.3).Open-source LLMs are observed to fall short in complex agent tasks [46, 25] . Recent studies have improved their reasoning abilities through imitation learning using trajectories from advanced LLMs such as GPT-4 [53, 4] . However, even GPT-4 can struggle with producing high-quality reasoning trajectories [29] .AMOR's modular design enables us to construct training data for each module separately from existing datasets without simulating the whole trajectories, thus greatly alleviating the above issue. Formally, given a sample question Q with annotations of the final answer Â, all sub-queries and answers Ĥ = [(q 0 , â0 ), (q 1 , â1 ), • • • ], and all evidence passages Ê = [ê 0 , ê1 , • • • ], we can directly transform these annotations into a suitable format to serve as training data for Decompose and Complete in Figure 1 . Since Judge and Answer require multiple types of retrieved knowledge (e.g., relevant or not), we employ retrieval tools to augment the input. Figure 2 exemplifies the construction pipeline, which can be easily extended to other knowledge-intensive datasets and specific domains. Appendix A.4 shows more details.When fine-tuning open-source LLMs to handle multiple tasks defined by different modules, we are inspired by the Mixture-of-Experts approach [33] to learn distinct Feed-Forward Network (FFN) parameters in the final quarter of the Transformer blocks to balance the trade-off between performance and inference efficiency. These module-specific parameters are initialized using the original model's FFN layers. We call the proposed architecture Module-Aware Mixture-of-Experts (MA-MoE) 3 . Then, we fine-tune the MA-MoE model with the standard language modeling loss:EQUATIONwhere Feedback is crucial for adapting language agents to specific environments [40] , especially when dealing with unseen, long-tail, or everchanging domain knowledge. Prior agents commonly used outcome feedback for adaptation which assesses the correctness of intermediate steps based on the success or failure of the outcome [53, 4] . However, outcome feedback is too sparse to improve intermediate reasoning [23] . Recent studies also highlighted that LLMs' reasoning steps are likely to contradict the outcome [26] , which means that outcome feedback may inevitably introduce noise during training (see examples in Appendix B.8). In contrast, AMOR's process feedback mechanism can effectively alleviate these issues.πAlgorithm 2 describes the adaptation mechanism of AMOR parameterized by θ, specifically as three steps: (1) Exploration. AMOR answers the input question Q by interacting with a knowledge base.(2) Feedback Collection. AMOR's reasoning process for Q is evaluated with feedback f k for the output y k of the LLM at each step during reasoning, which is either "right/wrong" or a refined version of y. We convert y into a feedback-refined target output ỹ based on the feedback f k and determine the immediate reward o k as follows:ỹk , o k =      y k , 1 if f k = "right", y k , 0 if f k = "wrong", f k , 1 if f k is refinement. (2)(3) Exploitation. Every T steps of the former exploration and feedback collection, we optimize the initial policy based on the resulting trajectories and corresponding feedback [32] : [NEXT] q "right", if the retrieved documents using q overlap the documents corresponding to Ê; "wrong", otherwise.EQUATION"right", if Ê ⊆ E (i.e, evidence passages collected by AMOR); "wrong", otherwise.Judge(Q, H, q, d)[RELEVANT] "[RELEVANT]", if one of passages in Ê comes from the same document as d; "[IRRELEVANT]", otherwise.Answer(Q, H, q, P )[ANSWERABLE] a e "right", if e ∈ Ê; "wrong", otherwise [UNANSWERABLE] "right", if P ∩ Ê = ∅; "wrong", otherwiseComplete(Q, E) A Â, if Ê ⊆ E; "wrong", otherwise.where R m ⊆ R denotes the training examples for module m, π WFT θ refers to the initial warm-up policy. Notably, this loss function is non-differentiable, necessitating the use of a specialized optimization technique. We use a recently proposed alignment algorithm KTO [9] with an MLE regularization [47] for optimization, which optimizes the policy without requiring paired human preferences. Crucially, when optimizing a particular module m, the gradient induced by the feedback signal propagates through the entire MA-MoE model, except for the FFN layers corresponding to other modules. This targeted optimization approach enables AMOR to effectively align its outputs with the desired intermediate results and final answers, leveraging the fine-grained process feedback provided by human supervisors.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 Tools modules. We construct retrievers for both SearchDoc and SearchPsg using Contriever-MS MARCO [15] . SearchDoc retrieves a single document snippet per query, while SearchPsg fetches the top three relevant passages from a given document. By invoking NextDoc, at most nine more document snippets are returned. Appendix B.1 presents more details.Warm-up datasets. We employ four question-answering (QA) datasets to warm up open-source LLMs, including 2WikiMultiHopQA [13] , Musique [37] , NaturalQuestions [20] and BoolQ [6] . They vary in levels of question complexity (single-or multi-hop), answer types (phrase spans or yes/no), and types of dependency structures between sub-queries (e.g., serial or parallel), etc. Appendix A. 4 shows the statistics in detail.Adaptation & evaluation datasets. We consider three benchmarks, by which we simulate different deployment scenarios: (1) HotpotQA [49] : a challenging multi-hop QA dataset built on Wikipedia articles. We use the Wikipedia dump provided in [15] as the knowledge base. (2) PubMedQA [17] : a biomedical QA dataset that requires answering a question by "yes/no" given a PubMed abstract. We adapt the data to retrieval-based QA by piling all 274k abstracts provided in the paper as a knowledge base, where each document comprises one abstract passage. (3) QASPER [8] : answering questions in free form based on a long NLP paper. For each question, we regard the corresponding paper as a knowledge base and each section of the paper as a document with several passages. We use the training and validation sets for adaptation fine-tuning and the test sets for evaluation. For evaluation metrics, we use exact match (EM) and F1 scores for HotpotQA and QASPER; and the accuracy (ACC) of "yes/no" for PubMedQA. More details are in Appendix B.2.Feedback annotation. Considering limited resources, we simulate human behavior and provide silver feedback to AMOR's reasoning processes based on the gold answer Â and gold evidence passages Ê = [ê 0 , ê1 , • • • ] for each target question Q, which are already included in the training and validation data of the three benchmarks. Table 2 shows how we annotate the feedback for each LLM output y. Note that AMOR is applicable for gold feedback from humans in realistic applications. Appendix B.3 discusses the accuracy of the silver feedback through human evaluation.Implementation details. We set λ m in Eq. 1 and Eq. 3 to 1 for all modules, I = 1 in Algorithm 2, and T to the size of the training set for each dataset, and fine-tune LLAMA-2-7B/13B-Chat for two epochs with a learning rate of 2e -5 using 8 NVIDIA 80GB A100 GPUs. While applying Table 3 : Refining each module output y to ỹ based on the outcome feedback f o to adapt AMOR, where ¬ y denotes converting the binary output y to its opposite label. AMOR for inference, we use greedy decoding for all generations. Besides, we set the maximum number of decomposed sub-queries to the maximum count of gold evidence passages, i.e., 2/1/1 for HopotQA/PubMedQA/QASPER, respectively. Once the maximum number is reached, AMOR is transited to state s 6 ("Task Completion") to finalize the answer.Baselines. We compare AMOR to various baselines with or without fine-tuning: (1) CoT [44] : it prompts an off-the-shelf LLM to generate the answer through step-by-step reasoning.(2) RAG: One-Step Retrieval (OneR) uses the question as a query to retrieve top-K document snippets with the SearchDoc module to augment the input. We set K as the maximum number of gold evidence passages in each dataset. Under the fine-tuning setting, we use the gold evidence passages for training. Self-RAG [2] selectively performs retrieval and utilizes retrieved passages while does not explicitly introduce question decomposition. They can be viewed as simplifications of AMOR.(3) ReAct [51] : it interleaves thought, action, and observation steps. An action can be either invoking the retrieval tools or finalizing an answer. We also compare AMOR with fine-tuned ReAct-style agents including AgentLM [53] and FIREACT [4] . We set the maximum number of action steps to 20. (4) Modular Agents: ReWoo [46] follows a pipeline that plans all sub-goals, generates actions, and then executes, while LUMOS [52] applies this pipeline iteratively, tackling one sub-goal at a time with each interaction. Both agents utilize GPT-3.5 as a supplementary QA tool during action generation. Similar to AMOR, they modularize language agents; however, they lack explicit mechanisms for assessing the relevance of retrieved information. Under the setting without fine-tuning, we provide in-context examples for the baselines following their official implementations.Furthermore, we also conduct ablation studies to investigate the influence of different components, resulting in two more baselines: (1) AMOR WFT : AMOR with only warm-up fine-tuning, without further adaptation; and (2) AMOR Outcome : outcome feedback instead of process feedback is utilized in adaptation after AMOR is warmed-up. Specifically, we determine the target output and corresponding immediate reward for an LLM module as detailed in Table 3 , and then adapt AMOR using Eq. 3. For clarity, we denote our final method as AMOR Process .Table 4 reports the evaluation results of AMOR and baselines on three datasets, revealing three key findings: (1) The FSM paradigm is clearly advantageous to prior agent frameworks. AMOR w/o FT delivers strong performance by improving 41.9%, 32.1%, and 41.2% over ReAct on average when built on top of off-the-shelf LLMs, including L-7B, GPT-3.5, and GPT-4, respectively. This indicates that our proposed FSM paradigm is more effective in leveraging LLMs for complex reasoning. (2) Warm-up fine-tuning generally enhances AMOR in downstream tasks. When based on L-7B, AMOR WFT outperforms AMOR w/o FT across all datasets. Furthermore, AMOR WFT also surpasses other fine-tuned ReAct-style and modular agents, even including FIREACT that is fine-tuned with in-domain HotpotQA trajectories from GPT-4. This suggests the potential of utilizing existing datasets for developing powerful agents with well-defined reasoning logic. (3) Process feedback is more effective than outcome feedback in facilitating the adaptation of agents. The order that AMOR Process > AMOR Outcome > AMOR WFT indicates the impact of feedback in terms of tailoring agent behavior to specific domains, and process feedback is more helpful than outcome feedback for leading to the correct final answers.Additionally, the MA-MoE model except it lacks module-specific awareness. SFT refers to optimizing the model only for outputs ỹk that receive an immediate reward o k = 1, using standard language modeling loss. The results show: (1) The standard MoE architecture struggles to effectively differentiate its experts for multitasking scenarios, leading to performance comparable to the Transformer model. In contrast, the MA-MoE's module-specific awareness enables it to handle diverse tasks within the agent more adeptly. (2) KTO outperforms SFT in aligning the agent's performance with external feedback, owing to its exploitation of negative samples.The main results have substantiated the benefits of different components of AMOR for successfully completing tasks. Nonetheless, we are still curious about four key research questions: (1) RQ1: 6 , we observe: (1) Warm-up fine-tuning consistently enhances evidence collection, with AMOR WFT achieving higher recall than AMOR w/o FT across all datasets. (2) Adaptation through outcome feedback (AMOR Outcome ) exerts a negligible impact on the recall results compared with AMOR WFT , suggesting the superiority of AMOR Outcome to AMOR WFT in final answers (see Table 4 ) may stem from the improvement of Complete. (3) Process feedback is crucial to improve the evidence collection ability, with AMOR Process substantially outperforming the other variants. RQ2: Data efficiency for adaptation. We aim to compare the data efficiency of different feedback types for adaptation in terms of the number of exploratory instances required.To this end, we adjust the exploration steps T in Algorithm 2, selecting values at intervals of 200, ranging up to 2,000 steps on HotpotQA. Appendix B.5 further discusses the cases with I > 1 in Algorithm 2 where AMOR is optimized over multiple rounds.Figure 3 shows the post-adaptation performance of AMOR varying with the number of exploratory instances (i.e., T ). Compared to AMOR Outcome , AMOR Process requires significantly fewer exploration steps to achieve comparable performance. Notably, AMOR Outcome shows a marked decline in performance when exposed to a limited number of exploratory instances (< 800), suggesting a reduced adaptability in explorationlimited scenarios. Conversely, AMOR Process 's robust performance under such constraints highlights its superior adaptability and efficiency with minimal interaction.RQ3: Adaptation through human feedback Due to limited resources, we use automatically annotated silver feedback as a proxy for gold human feedback in our main experiments. We would like to emphasize that our experimental framework is inherently designed to seamlessly incorporate human feedback in place of its automated counterpart. To illustrate this, we carry out a human study to demonstrate how AMOR is adapted through human feedback on HotpotQA. For this study, we hire an NLP expert to provide human feedback for each module within AMOR WFT on 2,000 exploratory instances following the annotation strategy in Appendix B.3. Table 7 shows the adaptation results using the collected human feedback.The results distinctly suggest: human feedback more effectively adapts AMOR to specific knowledge environments than automatic feedback. Our study lays a robust groundwork for the practical deployment and real-world utilization of the AMOR framework.RQ4: Reasoning process assessment. To measure the accuracy of AMOR's reasoning process, we performed a human study on HotpotQA, which involved: (1) selecting 50 random questions;(2) manually annotating the gold feedback f human for each LLM module output the following instructions using the same annotation protocol in Appendix B.3; and (3) calculating the accuracy of each LLM module output based on f human (1/0 indicating "right/wrong").Table 8 presents the accuracy of AMOR variants, affirming RQ1's findings: process feedback significantly improves the reasoning process over AMOR WFT that lacks adaptation, while outcome feedback has a negligible effect. Moreover, AMOR Process relatively lags in the Decompose and Complete modules, hinting that future enhancements could focus on including more corresponding data during two fine-tuning stages.Appendix B.8 presents several examples to further illustrate AMOR's strengths in reasoning logic and intervenability, as well as the limitations of relying on outcome feedback for adaptation, emphasizing the crucial role of process feedback.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Please see Section 4.1 
conclusion: 
 In this work, we develop AMOR, an adaptable modular agent designed for knowledge-intensive tasks, featuring FSM-based reasoning logic and a process feedback mechanism. Based on opensource LLMs, AMOR undergoes a two-stage fine-tuning: initial warm-up to generalize across task environments and subsequent domain-specific adaptation through process feedback. Extensive experiments demonstrate AMOR's advantages over strong baselines across multiple domains. Further discussions highlight the effectiveness and efficiency of process feedback in adaptation. compared to previous agents. Future work will explore extending our paradigm to more knowledge types (e.g., structured knowledge bases) and broader agent tasks, ultimately empowering LLMs to autonomously design FSM-based reasoning logic on top of our paradigm. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Please see Section 4.2Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please see Section 4.1Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please see Appendix Section D Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is• The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: We hire an NLP expert for manual annotation and pay him $7.25/h. Please see Table 16 for detailed annotation instruction. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human SubjectsQuestion: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [No] Justification: IRB is not applicable to our country. But all potential risks were fully disclosed to the participant prior to his involvement in our study. We ensured that the participant provided informed consent, being fully aware of what the research entailed and any risks they might face. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As illustrated in Figure 4 , AMOR addresses the issues of prior reasoning methods in terms of three aspects:• AMOR is equipped with a controllable FSM-based reasoning logic with a stronger capacity for handling complex tasks than simple pipelines employed by Self-RAG, ReWoo, LUMOS. For instance, if no relevant passages are retrieved from a document, AMOR can dynamically transit to the next document, while LUMOS would be constrained to generate answers based on the irrelevant passages, potentially leading to incorrect or low-quality outputs. • AMOR adapts to new environments through exploration and exploitation. AMOR is designed with an adaptation fine-tuning stage, enabling it to adapt effectively to specific domains based on human feedback. This adaptive mechanism sets AMOR apart from prior modular agents that lack the ability to incorporate expert guidance. • AMOR enables humans to conveniently and effectively intervene and provide feedback at each reasoning step. AMOR introduces a process feedback mechanism that enables humans to provide direct feedback on the individual modules within the FSM-based reasoning process. This approach facilitates a more natural and interpretable form of supervision, allowing for targeted improvements and fine-tuning of specific reasoning components.In summary, AMOR achieves more controllable, adaptable, and human-guided reasoning capabilities compared to existing methods. Grounding ExecutionStep 1Step N Prompting or Imitation Learning from Humans/LLMs. Drawbacks: The former often leads to suboptimal results, while the latter suffers from the scarcity of high-quality data.Agents with Free-Form Reasoning (e.g., AgentLM, FireAct) UndefinedPrompting or Imitation Learning from Humans/LLMs. Drawbacks: The former often leads to suboptimal results, while the latter suffers from the scarcity of high-quality data.Outcome Feedback.(1) Outcome feedback alone is often too sparse and insufficient to improve the intermediate reasoning steps effectively;(2) The reasoning steps taken by LLMs can frequently contradict or deviate from the desired outcome.Figure 4 : Left: Elaboration regarding the advantages and drawbacks when comparing AMOR with prior agents in terms of three aspects in Table 1 . Right: The reasoning processes of AMOR and related works.Algorithm 2 in the main paper illustrates a general FSM-based reasoning logic, which can be adapted to various agent environments by defining the FSM including the states, modules, etc.As shown in Algorithm 3, AMOR provides an instantiation of the FSM-based reasoning logic for the knowledge-seeking scenarios following the state transition diagram in Figure 1 in the main paper. We expect to extend this work to more environments in the future. In this section, we elaborate the pipeline to collect training examples for the warm-up stage of AMOR. Given a sample question Q with annotations of the final answer Â, all sub-queries and answers Ĥ = [(q j , âj )] J-1 j=0 , and all evidence passages Ê = [ê j ] J-1 j=0 , where J is the number of necessitated sub-queries of Q, we construct training examples for four LLM modules of AMOR as follows:• Decompose(Q, H): We construct a total of J + 1 training examples for this module. For each of the J sub-queries, we create an example with the main question Q and the preceding sub-queries and answers H = Ĥ<j as the input, and the next sub-query qj coupled with the branch token "[NEXT]" as the output (for j = 0, 1, . . . , J -1). Here, Ĥ<j denotes the sequence containing the first j pairs of sub-queries and their corresponding answers from Ĥ. Additionally, we create one example where the input includes Q and the complete set of sub-queries and answers H = Ĥ, with the branch token "[FINISH]" as the output, indicating the end of the decomposition. • Judge(Q, H, q, d): For this module, the input consists of the main question Q, the previous sub-queries and answers H = Ĥ<j , the current sub-query q = qj , and a document snippet d (for j = 0, 1, • • • , J -1). The output is a branch token that classifies the snippet d as either "[RELEVANT]" or "[IRRELEVANT]" in relation to the current sub-query qj . We consider three scenarios for the document snippet d: (1) When d is the gold evidence passage êj , the output is "[RELEVANT]". (2) When d is a passage from a different document from êj , it is marked as "[IRRELEVANT]". We obtain this type of snippet, denoted as d j , by using qj as the query in SearchDoc, ensuring it originates from a distinct document compared to êj . (3) When d is a passage from the same document as êj but is not êj itself, it is deemed "[RELEVANT]". We acquire such snippets by invoking SearchPsg with qj to retrieve passages from the same document as êj , excluding êj from the results. We refer to this set of passages as P -, considering each of them relevant to qj . These varied document snippet scenarios are designed to train the module to discern the relevance of a query to a document based solely on portions of the document content. • Answer(Q, H, q, P ). Similar to the Judge module, the input for this module comprises the main question Q, the previous sub-queries and answers H = Ĥ<j , the current sub-query q = qj , and a set of passages P from the same document. The output is either the branch token [UNANSWERABLE]" or a combination of the branch token [ANSWERABLE]", the corresponding answer âj , and evidence passage êj . We consider two scenarios for P : (1) When P does not include êj , indicating that the sub-query qj cannot be answered, the output is "[UNANSWERABLE]". Here, P is set to the previously mentioned set P -. (2) When P includes êj , suggesting that qj is answerable, we create P by replacing a random passage in P -with êj . For both scenarios, we present the passages to the module in random order when constructing training examples. • Complete(Q, E). We construct one training example for this module by setting the input to the main question Q and gold evidence Ê and the output to the final answer Â.After generating examples from the warm-up datasets using the aforementioned pipeline, we randomly select a specified number of examples. This random sampling aims to ensure a balanced representation of the various modules and branch tokens in the final dataset. Table 9 shows the detailed statistics of the warm-up data. adaption to tool implementations of LUMOS and AgentLM necessitates several adjustments in AMOR, including (1) When using "Wikipedia API" to implement SearchDoc, the Decompose module should identify entity names for SearchDoc. (2) When using "Exact Keyword Match" to implement SearchPsg, the Decompose module should predict keywords for SearchPsg. (3) We craft elaborate rules to create warm-up data that follow the above formats for the Decompose module.For example, we use the title of the gold passage as the target entity name for a sub-query and use the longest common string between each sub-query and the corresponding gold passage as the target keyword for "Exact Keyword Match."As demonstrated in Table 14 , AMOR WFT , without being fine-tuned on HotpotQA, surpasses both LUMOS and AgentLM in performance by employing identical tool implementations. Moreover, AMORProcess, upon fine-tuning with process feedback on HotpotQA, exhibits substantial and significant enhancements in performance. These outcomes collectively underscore AMOR's superior performance compared to baseline models and its robustness and adaptability across various tool implementations.We describe how we process the datasets as follows: (1) HotpotQA: Each document is a Wikipedia article. Since the original test set is hidden, we randomly sample 500 examples from the original validation set for evaluation and split the training set for adaptation fine-tuning and validation. (2) PubMedQA [17] : We follow the official split. And we only remain examples whose answers are "yes" or "no" and discard those labeled "maybe." (3) QASPER [8] : For each question, we regard the corresponding paper as a knowledge base and each section of the paper as a document with the section name as the title (e.g., "Experiments::Datasets") including several passages. Although many LLMs support context longer than the average paper length of 7k tokens, we focus on testing the ability of language agents to seek and utilize information in this work. We exclude questions that are labeled "unanswerable." Since the original test set is also hidden, we use the original validation set for evaluation and redivide the training set for training and validation. Table 15 shows the statistics of the three datasets.To investigate the extent to which the adaptation stage enhances AMOR's reasoning process, we conducted a human study with one NLP expert using the HotpotQA test set, Table 16 demonstrates the protocol for annotating the gold feedback f human and then we calculate the accuracy of the 6 ), AMOR facilitates the provision of precise, module-specific feedback. We will add the above analysis in our revision.In the main paper, we set I = 1 in Algorithm 2 for all experiments, which means that all exploratory instances in the adaptation stage are induced by the warm-up policy AMOR WFT . We call this setting "single-round adaptation." We are curious about how multi-round adaptation influences the performance of AMOR by adjusting I. For the i-th iteration (i = 1, 2, • • • , I), we denote the initial parameter of AMOR as θ i-1 , which is used to explore over a set of input questions and is updated to θ i after exploitation using these exploratory instances. AMOR θ0 is exactly AMOR WFT . During different iterations, we can provide either the same or different questions for AMOR to explore over. The case with the same set of questions is used to simulate an exploration-limited scenario. Note that in this case, the exploratory instances with the same questions are still different due to the ever-changing policy leading to different outputs.Table 19 shows the performance of AMOR under the multi-round adaptation setting with I = 3. We find that the performance is almost unchanged whether using the same or different input questions for each adaptation round. This result suggests that one iteration may be sufficient for the adaptation fine-tuning stage in our study.Language agents interact with environments to solve problems through frequent calls of LLMs, leading to huge costs in terms of token consumption. Building agents with minimal token usage is essential for curbing deployment costs [46] .Table 20 displays the average number of steps and tokens used by AMOR and ReAct-style agents to answer a question. ReAct-style agents, lacking explicit modeling of inter-step dependencies, require the inclusion of all preceding information in the input for each step. This often results in undesired redundancy. In contrast, AMOR consumes significantly fewer tokens with each module relying only on essential historical information, which highlights the token efficiency of its architecture. When built upon GPT-4, AMOR w/o FT uses fewer steps but more tokens than AMOR Process based on L-7B due to the additional in-context examples inserted into the prompts of GPT-4. 21 .(1) Targeted fine-tuning. Table 8 reveals that the Complete module of AMOR Process still falls short in performance, achieving only ∼ 50% accuracy. We construct 6k examples for the module from the original training set of HotpotQA by treating the final answer Â as input, and the question Q and evidence passages Ê as output, and then fine-tune the L-7B model on the data. Table 21 shows the performance gains when substituting the original Complete module in AMOR Process with this individually fine-tuned L-7B model.(2) Accommodating new tools. Numerous studies have demonstrated the benefits of retrievalbased in-context learning, where a retriever selectively curates tailored demonstrations for each specific input [48] . We implement this by inserting a new state s ′ 6 , named "Demonstration Retrieval," before the final state s 6 shown in the state transition diagram in Figure 1 , making AMOR reach s ′ 6 when Decompose outputs "[FINISH]" at state s 0 . The new state s ′ 6 holds two variables, including the main question Q and all collected evidence E, and employs a tool module SearchDemo to retrieve the top K similar questions to Q from an external demonstration memory, along with their answers and evidence, collectively noted as K = [Q k , Âk , Êk ] K k=1 . Subsequently, at state s 6 , the Complete module takes K as the in-context examples, helping generate the final answer A given Q and E. We use the HotpotQA training set as our demonstration memory and employ Contriever-MS MARCO [15] to implement the SearchDemo module, setting K to 5. We fine-tune the L-7B model on the training set to act as the Complete module while ensuring that the demonstration does not include the target question. As Table21 indicates, this integration of such an additional tool further improves AMOR Process with targeted fine-tuning.Additionally, AMOR's reasoning logic can be easily expanded from single-path to multi-path reasoning, akin to the approaches used in Self-Consistency [41] , ToT [50] , and GoT [3] . This can be achieved by generating multiple outputs within specific modules and incorporating modules that synthesize these multi-path results. Consequently, we advocate for the adoption of the FSM paradigm in the design of future agents. This framework offers the dual benefits of flexibility and the capacity to adapt agents based on process feedback.We annotate the silver outcome feedback f o for the Complete module at the final state as Â if all gold evidence passages are successfully collected (i.e., Ê ⊆ E), and "wrong" otherwise.We demonstrate the advantages of the FSM-based reasoning logic and process feedback mechanism through the comparison between AMOR Process and ReAct in Figure 5 and 6 , respectively. We observe that ReAct without explicit reasoning logic constraints fails to decompose the question and terminates retrieval prematurely in "Thought/Action 5." Besides, ReAct also mixes right and wrong steps in "Thought 2/4/5," making it challenging for users to critique and improve the agent in a targeted manner. In contrast, AMOR successfully answers the question with a controllable reasoning logic and allows direct process feedback to drive the evolution.Additionally, Table 22 shows a case where AMOR WFT correctly answers a question with the right evidence, yet employs a wrong reasoning process. This underscores the potential unreliability of using This study has demonstrated the benefits of two components: (1) explicitly defined FSM-based reasoning logic, and (2) the process feedback mechanism. Nonetheless, a notable limitation must be acknowledged when extending our approach to other tasks. While we have made initial efforts to outline the general principles for crafting the FSM in §3.1 and show the flexibility of adapting AMOR's FSM in Appendix B.7, it still requires a human-driven design process. Looking ahead, our future work aims to enable LLMs to autonomously instantiate FSM-based reasoning logic in Algorithm 1 for diverse user tasks, thereby reducing reliance on human design. Furthermore, we believe that the FSM-based reasoning logic makes it easier for humans to supervise LLMs that potentially outperform humans on the task.The innovation introduced through AMOR carries significant potential for both positive and negative societal impacts.On the positive side, the ability of AMOR to adapt to diverse knowledge environments and domains through supervised reasoning could lead to advancements in personalized education, healthcare diagnostics, and customer service. Such applications could democratize access to information and expertise, bridging gaps in knowledge and service availability across different regions and socioeconomic groups.However, the sophisticated reasoning capability of AMOR also brings about considerations of misuse, such as the generation of disinformation or aiding in the automation of social engineering attacks. Furthermore, if not properly balanced, the tailored knowledge adaptation could unintentionally reinforce biases present in the training data or human feedback, leading to unfair outcomes in decision-making processes that might disproportionately affect marginalized groups.To mitigate these concerns, it's crucial to engage in transparent development and deployment practices, including bias audits and the establishment of ethical guidelines for use. Additionally, mechanisms for detecting and correcting misinformation or biased reasoning paths should be incorporated into the system's design.Given the potential for misuse inherent in powerful language models like AMOR, it's vital to implement safeguards. AMOR will be made available under a framework that requires users to agree to ethical guidelines before access. Our intention is to maximize AMOR's societal benefits while curtailing the potential for negative impacts, thus ensuring responsible deployment and use of this advanced agent framework.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.States & Modules for Calling LLMsIn this work, GPT-3.5/4 refers to the OpenAI's API "gpt-3.5-turbo" / "gpt-4-1106-preview," respectively."Module-Aware" means when AMOR executes a certain module, its module index will be provided to the routers of the model to indicate which expert should be activated.We do not include FIREACT in our comparison due to its reliance on the Google Search API, which exceeds our budget constraints. 


Decision: The authors propose a modular pipeline for QA tasks that performs reasoning over external knowledge bases. The system, AMOR, is designed as a Finite State Machine, whch decomposes complex questions into smaller pieces, then each piece can be handled by different modules such as document retrieval or answer extraction. Each module can be fine-tuned individually, using existing datasets (during warm-up phase); the agent can further be fine-tuned on specific domains using process feedback (via KTO) obtained from either human annotators or an evaluation benchmark. 

All reviewers agree that the paper is above the bar. In particular, the reviewers consider this work as well-motivated, addressing important problem; innovative and sensible; experiments are well designed and presented; the method shows strong performance against a good set of baseline systems. 

However, multiple reviewers have concerns about the writing, especially the clarity of the submission. To help the authors further improve their work, the reviewers suggested the following:
- To better connect AMOR with prior reasoning methods (like in their rebuttal message to all reviewers). 
- Make the KTO objective part clearer.
- Improve the clarity of method description, especially the process-based feedback. 
- Make the experiment setup more structured.
- Add error bars.

In my opinion, these suggestions require minor change to the paper, some of them has been done already during the rebuttal discussion. In addition, there is no major concern regarding the method nor the experiments. I believe this work could be found useful to the NeurIPS community so I recommend to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposed an architecture for advanced reasoning in LLMs. The architecture contains several modules dedicated to different tasks in the reasoning flow, each of which can be trained separately using related datasets constructed from public datasets. The proposed method disentangles the reasoning process into sub-steps that are easier to train and suffer less from sparse feedback which is a critical problem for reasoning in traditional LLMs. In the experiments part, the authors conducted sufficient empirical studies, showing the effectiveness of the architecture proposed.
strength_weakness: strength_weakness: 1. Currently, many works focus on enhancing the reasoning capability of LLMs. Although the authors mentioned previous works about LLM reasoning and RAG in the related works part, an in-depth explanation of why this method surpasses its predecessors is still needed. A good way to do this is to explain clearly, probably with a few pictures or equations, how those methods dealt with the difficulties and their advantages/drawbacks, and then show that AMOR is better as it avoids some of the existing drawbacks in previous methods.

2. The state transition diagram is well displayed. However, when readers look into the implementation of each module, some key information such as the architecture, hyperparameters, and techniques used for better performance is missing, making it more difficult to understand and reproduce the whole model. A table of these details or a later public code repository would be appreciated.

3. In section 3.3, the detail of how the policy is trained is not well-explained. Did the author use parameterized policies such as policy networks and use a policy-based method in RL for optimization? If so, is the loss presented in equation (3) an auxiliary loss to the loss in policy gradient methods or the pure source of gradients?

4. Following the above question. Will the gradient induced by feedback go through the whole model or only certain blocks?
questions: questions: See "Weaknesses" part. My concerns listed there are also my questions to the authors.
limitations: limitations: The authors discussed the limitations of their method adequately in the "Limitations" part in the appendix.
paper_summary: paper_summary: This work presents a modular pipeline for QA tasks. The pipeline consists of several modules such as question decomposition, document/passage retrieval, answer extraction, etc. Training data is separately constructed for each module (based on existing datasets) and models are individually fine-tuned for the respective modules. This processes is referred to as the 'warm-up' stage. Models are further updated based on feedback obtained during inference using an alignment algorithm (KTO is used in this work). Experiments on QA datasets shows that the proposed method performs better than various baselines in the literature. Several ablations are also provided to study the impact of each component.
strength_weakness: strength_weakness: * The scope for this work was not properly introduced. The paper is motivated from very broad goals only for the reader to later find that the approach and experiments focus on QA tasks. In fact, the paper does not present a formal task description or problem formulation before presenting the methods, leaving the reader to piece together things. 
* Method description: In addition to the task not being formally presented, several notations were not clearly defined. For example, equations 1, 3 start talking about a policy which was not described earlier. Although I do understand what's happening in these equations, clearly defined problem and terminology would have made things much clearer.
* In general, I felt the ideas in the paper could've been framed/described in a much simpler way. Readability could be improved.
* The introduction provides motivations but fails to convey any details/intuitions of the proposed method or how it overcomes the challenges mentioned.
* The 'processed based feedback' was not clear to me. I think the paper tries to squeeze in many ideas but this also distracts the reader from the core ideas. 
* The experimental setup could have been made clearer, including the various settings such as with/without finetuning and process/outcome feedback. 
* Figures and tables are just too small and impossible to parse.
questions: questions: * What is the without FT setting in Table 6? 
* Are the comparisons against baselines fair? Does the proposed approach use more data compared to the baselines? 
* What is the major delta of this paper compared to prior modular reasoning works (e.g., Rewoo, Lumos)?
limitations: limitations: I did not find a discussion of limitations in the papers, I suggest the authors to add some discussion.
paper_summary: paper_summary: This work proposes AMOR, a modular approach to building knowledge agents using open-source LLMs. AMOR decomposes tasks into reasoning logic, represented as a finite state machine (FSM) composed of sequentially chained modules. These modules include tools e.g, document retrieval and LLMs, e.g., answer extraction. AMOR retrieves relevant documents for a query and combines information from different sources to produce an answer.

The development process of AMOR includes three stages:
1. **Warm-up Phase:** Each module is fine-tuned on datasets containing not only input-output pairs but also the relevant intermediate steps for each module. This phase ensures that AMOR can generalize across various tasks and knowledge environments.
2. **Adaptation Phase:** The agent is further fine-tuned on specific domains using process feedback. This feedback, which can be human-generated or derived from evaluation benchmarks, is provided at each reasoning step to refine the agent's performance.

Empirical evaluation demonstrates that AMOR effectively utilizes the rich warm-up and evaluation datasets, outperforming relevant baselines. Ablation studies confirm that each component of AMOR is essential for achieving maximal performance.
strength_weakness: strength_weakness: All experiments do not report any measure of uncertainty. Thus, it is impossible to determine if the conclusions are statistically significant. Including measures of uncertainty would strengthen the validity of the results.
questions: questions: How are the modules chosen in MA-MoE? Is the module/step ID provided to the routers?
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes AMOR, which is a modular framework for answering questions by reasoning over external knowledge bases. AMOR is designed as a Finite State Machine (FSM), which provides a structured way to break down the question into smaller pieces and solve complex tasks through various steps. Each state in AMOR is a module, which can be an LLM call or a tool call. Each LLM module is fine-tuned separately using data relevant to only that step, thus making data collection and evaluation easier.
strength_weakness: strength_weakness: The weaknesses I see are the complexity of this framework: many moving parts, including various LLMs that need to be fine-tuned separately; and the 2 stages fine-tuning approach (warm-up + fine-tuning) makes it even more complex and resource intensive. Dependency on human feedback is another weakness I can see here. However, these are weaknesses common to other agents frameworks so I wouldn't consider this as a disqualifier for this paper.
questions: questions: I don't have any questions
limitations: limitations: The authors were transparent about possible limitations and negative impacts in the **Broader impacts and safeguards** session and also included suggestions on how to mitigate these problems.

Comments: comment: The authors adequately answered my questions and now I have a better understanding of their work. I've raised my grades accordingly.