Tool-Augmented Reward Modeling
Authors: Lei Li
Yekun Chai
Shuohuan Wang
Yu Sun
Hao Tian
Ningyu Zhang
Hua Wu
Venue: ICLR 2024 spotlight (2024)
Keywords: Reward Model
Large Language Model
Tool Learning
Augmented Language Model
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This paper introduces tool-augmented reward models for reinforcement learning from human feedback (RLHF), improving precision and interpretability and contributing a comprehensive dataset from seven diverse tool APIs to advance the field.

Abstract: Reward modeling (*a.k.a.*, preference modeling) is instrumental for aligning large language models with human preferences, particularly within the context of reinforcement learning from human feedback (RLHF). While conventional reward models (RMs) have exhibited remarkable scalability, they oft struggle with fundamental functionality such as arithmetic computation, code execution, and factual lookup. In this paper, we propose a tool-augmented preference modeling approach, named Themis, to address these limitations by empowering RMs with access to external environments, including calculators and search engines. This approach not only fosters synergy between tool utilization and reward grading but also enhances interpretive capacity and scoring reliability. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources and construct task-specific tool engagement and reasoning traces in an autoregressive manner. We validate our approach across a wide range of domains, incorporating seven distinct external tools. Our experimental results demonstrate a noteworthy overall improvement of 17.7% across eight tasks in preference ranking. Furthermore, our approach outperforms Gopher 280B by 7.3% on TruthfulQA task in zero-shot evaluation. In human evaluations, RLHF trained with Themis attains an average win rate of 32% when compared to baselines across four distinct tasks. Additionally, we provide a comprehensive collection of tool-related RM datasets, incorporating data from seven distinct tool APIs, totaling 15,000 instances. We have made the code, data, and model checkpoints publicly available to facilitate and inspire further research advancements (https://github.com/ernie-research/Tool-Augmented-Reward-Model).

Published as a conference paper at ICLR 2024
TOOL-AUGMENTED REWARD MODELING
Lei Li
Yekun Chai
Shuohuan Wang*
Yu Sun
Hao Tian
Ningyu Zhang 2
Hua Wu
Zhejiang University Baidu Inc.
[leili21,
edu. cn cn
[chaiyekun, wangshuohuan, sunyu02) sunyu02)ebaidu com
ABSTRACT
Reward modeling (a.ka., preference modeling) instrumental for aligning large language models with human preferences, particularly within the context of re- inforcement learning from human feedback (RLHF). While conventional reward models (RMs) have exhibited remarkable scalability, they oft struggle with fun- damental functionality such as arithmetic computation, code execution, and fac- tual lookup. In this paper, we propose tool-augmented preference modeling approach, named Themi s, to address these limitations by empowering RMs with access external environments, including calculators and search engines. This approach not only fosters synergy between tool utilization and reward grading but also enhances interpretive capacity and scoring reliability. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources and construct task-specific tool engagement and reason- ing traces in an autoregressive manner. We validate our approach across a wide range of domains, incorporating seven distinct external tools. Our experimental results demonstrate noteworthy overall improvement of 17.7% across eight tasks in preference ranking. Furthermore, our approach outperforms Gopher 280B by 7.3% TruthfulQA task in zero-shot evaluation. In human evaluations, RLHF trained with Themi s attains an average win rate of 32% when compared to base- lines across four distinct tasks. Additionally, tion of tool-related RM datasets, incorporating data from seven distinct tool APIs, totaling 15,000 instances. We have made the code, data, and model checkpoints publicly available to facilitate and inspire further research advancements
provide comprehensive collec-
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable potential in performing complex tasks that demand expertise across diverse domains, such as programming (Chen et al. 2021 Li
let al. 2022 Chai al. 2023 Li et al. 2023) and dialogue assistance (Bai et al. 2022a Ouyang
jet al.
2022
OpenAl, 2023 Anil et al. 2023 Touvron et al.] (023). Leveraging reinforcement
learning from human feedback (RLHF; Christiano et al. 2017 Stiennon et al. 2020) has emerged
as a compelling approach for optimizing LLMs against reward models (RMs) to predict human
preferences. RMs serve as imperfect proxies for human feedback signals, producing rewards that are pivotal for fine-tunen vieetunnng LLMs in RLHF. However, RMs predict human preferences relying on static internal representations stored within their weights, which inherently impose limitations of LLMs. These may encompass challenges in accessing real-time information
current events (Komeili ct al. 2022) and a propensity to hallucinate erroncous facts (Zhang et al. (2023), a lack of proficiency in arithmetic computation (Lewkowycz al. 2022), and difficulties in comprchending low-resouree languages (Lin et al. 2022b), among others. These limitations underscore the imperative need to engage external sources of information to augment and improve the effectiveness of RMs.
To further motivate this shift, an intriguing question arises when considering the role of human la- belers in generating RM data: do these labelers possess intuitive inclination to resort to external
"Equal contribution and shared co forstrst authorship. Work done during LL's internship at Baidu. Correspondence to: SW. SW.NZ
https: / / thub. com /rr e resea rch rcooo -Augmented Reward Model

---

Published as a conference paper at ICLR 2024
tools, much like humans themselves (akin to human problem-solving solving behavior) ? The motivation behind this question stems from the observation that even human labelers, when faced with complex tasks, often turn to external aids such search engines calculators to retrieve knowledge, validate facts, and facilitate their decision-making process. On the other hand, recent studies have unveiled the impressive performance gains that be achieved by integrating external tools into the reason- ing process of LLMs. Recent works such as Chain-of- ofTThhtht (Wei et al.) 2022) and ReAct (Yao) al. 2023) have demonstrated that step-by-step reasoning and tool use can significantly enhance the planning and reasoning abilities of LLMs, enabling them to successfully complete intricate tasks.
In response to these insights, this paper presents Themis, s, tool-augmented RM framework that combines tool engagement and reasoning process in a sequential and step-by-step manner. Our approach endows RMs with the capacity to make dynamic decisions regarding which APIs to call, when to invoke them, what arguments to pass, and how to effectively incorporate the results into the broader reasoning process. This approach empowers RMs to engage dynamic reasoning, enabling it to make high-level plans for tool use (reasoning to tools) while also interacting with external environments to incorporate additional information into its reasoning (reasoning to rewards).
One crucial advantage of this framework is that it offers a significant departure from vanilla pair- wise RMs, which has been inherently likened to a "blackbox" " due to its opacity in revealing the internal reasoning process. In contrast, our approach provides transparent and sequential account of actions and verbal reasoning traces specific to a given task. This transparency not only enhances human interpretabilltty but also engenders trustworthiness, it unveils the inner workings of the RM's decision-making process, This facilitates fine-tuning and modification of intermediate steps to exert precise control over the reward generation process.
To facilitate the exploration and validation of our proposed framework, we meticulously curated a comprehensive dataset comprising interactions with seven distinct external tools. The construction of this dataset was a collaborative endeavor, synergizing the generative capabilities of GPT-4 (Ope-) nA. 2023) as a prompting engine, tool-executed-barte filtering, and human annotations.
We comprehensively evaluate Themis across a diverse spectrum of domains, encompassing the uti- lization of these seven distinct external tools. Experimental results demonstrate that Themis s yields remarkable 17.7% improvement compared to conventional RMs that lack access to external tools, across eight distinct tasks. Moreover, Themi s outperforms Gopher 280B by a substantial margin of 7.3% on TruthfulQA benchmark, consistently surpassing baseline RMs. These compelling results underscore the effecctivnness and generalization capability of Themis in enhancing truthfulness and factuality in preference modeling. Furthermore, extend our investigation to RLHF fine-tuning, revealing that our method attains an impressive 32% win rate on average across four different tasks when compared to vanilla RMs, determined through human preference evaluation. This further demonstrates the practical utility and superiority of our approach in real-world applications.
To summarize, our key contribution encapsulated as follows: (1) We advance the domain of tool- augmented preference modeling by introducing the Themis s framework. This framework harnesses the power of external tools to augment preference modeling in LLMs. In doing so, it mitigates inherent limitations observed in conventional RMs. Additionally, our approach unveils the inner workings of the RM's decision-making process, providing transparency and interpretability. (2) We present novel tool-augmented reward modeling dataset (TARA) that includes comprehensive comparison data of human preferences and detailed tool invocation processes. This dataset will be made publicly available in hopes of facilitating research advancements in the field. (3) Our contribu- tions are substantiated through experimental evaluations conducted across eight diverse tasks within TARA, as well as benchmarking against TruthfulQA and Retarded-bar datasets. These experiments conclusively demonstrate the effectiveness of our approach in enhancing the performance of LLMs.
2 TOOL-AUGMENTED REWARD MODELING
2.1 REVISITING REWARD MODELS
In RLHF (Ouyang al. 2022 Stiennon al. 2020 RM is trained on human preference dataset consisting of comparisons between two candidate model outputs generated in response to the same input prompt. The vanilla RM operates by mapping each input prompt and its corresponding gen-
2

---

Published as a conference paper at ICLR 2024
Quesiton: Where does the fet com from in hummus?
Single-Tool Invocation
Question: What's the weether lik in New York on 2023-06-047
Hostl from tahini and olive oll.
From deep-fried bacon and land.
Answer: The weather New York 2023-06-04 cloudy.
Thought: need to search the weather New York 2023-06-04
Action: Weather
Action Input: York, 2023-06 004
Reward Model
Prefer:
Observation: The weather for York 2023-06-04
(a) Ranking- -base Reward Model
Overall Weather: Sunny; Temperature: 27.0 (C) ...
Tool- Tool-Augeetted
Model
Prefer:
Rationale: results exccutor sunny, but answer is cloudy...
Score: -0.45
Baol Anvole
Rationale: The tool
Swarch: Cal contect
"WiksSearch' used to validate
Multi-Tool Invocation
humminn
the answer. The observation from
Red Enscctionn
the tool states: "Fat content,
Question: W kiörym roku Fargin byla w Grindhouse? Answer: 2007
Observation: Hummus
mostly from tah and olive oil".
tahini and olive oil, about
Fat content, mastly fram
proveted, which stades that the fat hummass comes gil!.
This information wions with the
Thought: i need to translate the question into English.
34% the total; other
Action: Translator
major components 65% water, 17% total carbohyyratess
Tmostly tahine
Action Input: kiórym Fargie Grindhouse?
Reananty
Observation: What year Ferglc Grindhouse?
(b) Our Tool- mool-Augmented Reward Model
Thought: need to search the the Fergie was in Grindhouse. Action: Google Search
Question: What
of Enabling Act?
Action Input: What year was Forgle Grindhouse?
Gave the German Cabinet the Rokdhst
Oivernations
Observation: Grinhouse (2007) Fergie (segment Plaaet Terror') [MDb
The purpose of Enabling Act estahlish democratic system Germany and clstribute vegislative nower among vanous galitral narties.
Rationale: The results of translator and google perfectly verfy the correctness of the answer.
PPO
Score: 2.5
(c) Fine-tuned Policy using PPO against RM
(d) Examples of Single-Tool and Multi-Tool Invocation
Figure 1: A diagram illustrating the pipeline of (a) Vanilla reward models (RMs); (b) Tool- augmented RMs. namely Themis; (c) Reinforcement learning via proximal policy optimization (PPO) above RMs; (d) Examples of single multiple tool use process in the proposed approach. See Section for more details of our method.
erated output scalar reward, thereby encapsulating the overall preference between them. Math- ematically, for a given question denoted as æ with a positively preferred answer represented as yo and negatively preferred answer as yı, the loss function of the vanilla RM is formulated as:
LRM (n
yw) - re (xx,,))))
(1)
Here, ro(2, v) represents the scalar output of the RM for question and answer y. denotes sig- moid function, while D denotes the preference dataset. However, as highlighted earlier, this vanilla RM approach, while useful in aligning reward scores with human preferences, is constrained by limitations pertaining to timeliness and knowledge accessibility. It faces particular challenges when confronted with complex tasks such as mathematical problem-sole and reasoning.
2.2 THEMIS: TOOL-AUGMENTED REWARD MODELING
F presents an overview of the Themi framework, illustrating how it integrates tool engage- ment and reasoning processes in a structured, step-by-step manner. Our approach enhances RMs with the capability to make informed and dynamic decisions concerning which APIs to employ, when to invoke them, what arguments to pass, and how to effectively integrate the obtained results into the broader reasoning process. This comprehensive framework, depicted through step-by-step trajectories, encapsulates the entirety of the decision-making and reasoning journey, consisting of the following pivotal stages:
Thought: At this initial stage, the model evaluates whether it should engage external APIs (re- ferred to as tool reasoning).
Action: Subsequently, the model generates the necessary API calls along with the corresponding arguments required for the interactions.
Observation: The results produced by the external APIs are collected and stored.
Rationale: This stage involves the aggregation and synthesis of previously acquired information, fostering both induction and reasoning processes, specifically tailored for reward modeling.
3

---

Published as a conference paper at ICLR 2024
Open-
Teat
[Cualien) Inturwtoos) [Razionale,
ToolBank
Tool
Humon
-aas-
[Cusslon)] Intutatins) [Rationalis
Rationale
Question-Anwer Step ToolBank Construction
Tool-invoked Processes Multi-Agents
Step Data Filtering
Puir
Figure 2: An illustration of data creation pipline for our Tool-Augmented DatAset (TARA). Reward: Finally, the model leverages the accumulated insights and information to generate scalar reward score with a feed-forward layer, reflecting its overall preference based on the col- lective evidence.
Given a question æ and the corresponding generated output y, we consider a generalized RM agent parameterized by 0 with the capability to interact with external tools, Following Yao et al. (2033). we define the agent's 's action state a, E A at step as a combination of a natural language thought å and a tool acting state ag E 'T. These paired "(thought, action)" states are denoted as a, = (āz, a.). The purpose of the thought component âe is to encapsulate the comprehension of pertinent information and to guide the ensuing action az. This action determined following policy ps (ā,12; 2r, y, Cr år), where (a), 01,".. 0t-1 0t-1). Subsequently, the RM agent is tasked with predicting rea- soning thought based on the preceding context denoted as 8y. This reasoned thought Rationale plays pivotal role in enhancing the model's ability summarize and reason effectively, drawing upon the historical reasoning traces before ultimately predicting the scalar reward r. This approach closely mirors the step-hy-step reasoning paradigm established by Chain-of-Thouggh (Wei et al., 2022), accentuating the incremental and interactive nature inherent in the RM's decision-making process. Formally, the complete reasoning trajectory is represented Consequently, the reward can be denoted as ra(z, V, CI..T). During the training phase, we im- plement an auto-regressive training objective for the prediction of the next token in modeling the reasoning context In the context of reward training, we produce a scalar reward based on by fully connected layer and employ the same pair-wise ranking loss as utilized in conventional RMs. This loss function serves as a foundational component to discern and rank the relative preferences between different gener-eene outputs. All stages, with the exception utilize language model heads to generate text tokens (same language models). In the Reward stages, real-valued score is produced using a ferd-forward layer (same as conventional RMs).
C1...T (01, (11,0).- 01. aT оr ST).
the Reward stage,
Training Objectives pair-wise ranking loss and the auto-regressive language modeling loss. The former aligns with equa- tion [ while the latter is designed to empower RMs with the ability to perform tool invocation via supervised fine-tuning:
The overall training objective is comprised of two distinct components: the
Liotal
CRM
+ a
Cuaatte) + (- + w LRationalo)
(2)
pair-wise ranking loss
auto-regressive language modeling loss
where 8,w (0, 1} are hyper-parameters modulate different training configurations, T represents the number of tool invocations, a = in our experiments. Please refer to Table X for details.
Connection to Vanilla RM When is set to zero, the autoregressive loss term becomes null, effectively reducing Themis to standard RMs. Ideal RMs should possess the ability to discern when and whether employ external tools. To impart this knowledge, we incorporate tool-related data alongside non-tool data, thereby instructing RMs on the appropriate timing for tool invocation. Notably, this framework inherently encompasses the functionality of vanilla R.M..
TOOL-AUGMENTED REWARD DATASET
3.1 DATA COLLECTION
The comprehensive construction process of the Tool-Augmented Reward dAtaset (TARA) is de- picted in Figure 2
TARA is constructed by leveraging high-quality datasets and generating the
4

---

Published as a conference paper at ICLR 2024
tool invocation process through multi-agen interactions. This process can be subdivided into the following four key steps:
Step 1: Question-Answer Pairs Collection Initially, collect reward dataset featuring each instance comprising a question, a positive answer, and a negative answer. To construct this dataset, employ two distinct approaches: resume open-source, high-quality datasets, and generation from scratch with some heuristic methods. However, the above methods usually only produce positive answers. To address this concern, we leverage GPT-4 as negative generation agent to generate antagonistic negative answers, which will be described in Step 3.
Step 2: ToolBank Construction. Subsequently, we develop the toolbank, which encompasses three distinct types of tools: basic tools, query-based tools, and knowledgeable tools. Basic tools such as Calculator, Code Interpreter. and Translator, provide RMs with practical capabilities. Query- based tools, including Google, Weather, and Calendar Search, equip RMs with search capabiliitess to access up-to-date information. Knowledgeable tools enable RMs to tap into a knowledge base, enhancing the factual accuracy of rewards. Additionally, we introduce Multi-Tools, which contain sequential calls to multiple tools.
Step 3: Tool-invoked Process Generation by Multi-Agents. To automate the generation of tool- invoked processes, we design a simulated environment featuring human participants and three virtual agents: the negative generation agent, the tool agent, and the rationale agent. Leveraging the sub- stantial comprehension and generation capabilities of LLMs, we employ three GPT-4 to simulate the roles of the three agents. The negative generation agent is responsible for generating negative answer to the question that has only a positive answer. processes a question along with its pos- itive answer and generates a negative answer that is indistinguishable from the positive one. The tool agent acts as an intermediary between humans and agents, deciding when, which, and how to invoke tools. Specifically, the tool agent receives a question-ansee answer pair and produces Thought and Action stages, as outlined in Section 2.2 Then humans are tasked with invoking specific tools and observing the outcomes (Observation stage). The rationale agent is tasked with generating reasoning traces by comprehending previous contexts, synthesizing the question-answer pair, the tool invocation process, and the observations to systematically produce rewards (Rationale stage). Tool-invoked scenarios are simulated through interactions between agents and humans, yielding tool-invoked processes for RMs. The interaction process and prompts for each agent are detailed in Appendix Figure L
Step 4: Data Filtering. To maintain data quality, we implement a straightforward yet effective filtering process the generated data. Initially, we exclude tool-invoked processes acquired in Step 3 that exhibit invalid formats. Subsequently, we discard processes exceeding three interaction steps, lacking relevant function calls, manifesting parsing errors in their execution results.
3.2 DATA STATISTICS
The data statistics and comparison previous datasets are shown in Appendix Table TARA com- prises total of 13,604 training datasets and 1,469 test sets, each consisting of a question, a positive answer, and a negative answer. TARA incorporates a diverse set of seven tools that span across various domains, encompassing mathematical operations, code-related inquiries, closed-ended and open-ended question answering, knowledge- based queries, and semessensiivve information requests.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We utilized Vicuna-7B (Zhnng et al. 2023) as the base model for Themis s and compared it with conventional RMs, specifically (Bert-Large) and RM (Vicuna-7B) respectively, which serve as its underlying architectures. Our training and evaluation processes were conducted under two distinct settings: single-tool and mixed- tool scenarios. In the single-tool configuration. the TARA data was partitioned based on tool types, with each model exclusively trained involved training models on the entirety of the TARA dataset, encompassing diverse tools. During evaluation, compared the relative reward scores for positive and negative answers, using aceu-
(Devlin et al.] 2019) and Vicuna-7B, denoted as RM
specific type of tool. In contrast, the mixed-tool scenario
5

---

Published as a conference paper at ICLR 2024
The main results on the Tool-Augment Reward Dataset (TARA). We report the perfor- mance of RM and Themis in both single-tool and mixed-tool settings. Bold scores highlight the best performance achieved. The reported Avg. values are calculated by averaging accuracy across all instances, offering a comprehensive measure of miero accuracy that spans various tool types.
Table I:
Model
Calendar Calculator Weather Code Translator
Wiki Google
Multi Avg.
single-tool setring
RM (Bert-l Lagge)
63.21
88.31
71.52
66.67
24.33
82.75
68.66
78.47
65.01
RM Vicuna-7B)
80.91
98.05
86.08
85.19
34,33
93.31
65.13
79.17
75.04
Thami 5 (Vicuna-7B)
100.00
98.70
100.00
99.47
88.40
95.07
76.12
99.31
94.23
w/o Leturoaiio
100.00
98.05
100.00
99.47
87.71
90.49
64.48
80.56
90.23
mixed-tool setting
RM (Bert-Large)
83.02
94.16
80.38
73.54
22.67
83.45
70.15
81.25
69.
RM (Vicuna-7B)
83.96
94.16
83.54
88.36
33.67
92.61
72.39
81.25
75.63
Themis s (Vicuna-7B)
100.00
98.05
100,00
99.47
90.91
93.31
64.92
99.31
93.31
wlo Loturratio (3 = 0)
100.00
98.05
100.00
99.47
91.47
94.37
62.69
73.51
90.90
w/o LRaionae (w 0)
100.00
96.75
99.37
98.94
88.74
92.54
63.43
68.72
89.31
Themis (Vicuna-33B)
100.00
97.40
100,00
99.47
93.54
96.55
73.72
99.31
95.21
Themis (Vicuna-7) 7B LoRA)
96.22
96.10
96.20
99.47
73.33
46.26
58.33
82.57
Themi (Vicuna- 13B- LoRA)
98.11
92.21
98.73
98.41
72.00
92.25
57.85
75.69
85,
Themis (Vicuna-33B- LoRA)
86.79
97.40
99.36
98.41
84.66
95.77
58.95
99.30
90.74
30
RM aReriLagge
0.90
RM (Viuna 7B)
Roward Score
Asurscy
0.75
10-
0-
0.80
RM ReeeeLLLapr
RM PB)
-10
Ours
0.45
-20
2
Epochs
Epochs
Figure 3: Left: Model performance for various training epoch numbers; Right: Visualization of the change of average reward scores with training epochs. The top reward score line of each model corresponds to the positive answer, while the bottom line corresponds to the negative answer.
racy as the evaluation metric. Further details about hyper-parameter choices and additional training specifics can be found in Appendix C.I
4.2 MAIN RESULTS
Single-Tool vs. Mixed- Tool Performance. The main performance results of all models on TARA are shown in T Across both the single-tool and mixed-tool settings, is evident that Themis consistently outperforms vanilla RMs significantly, exhibiting improvement of +19.29 in the single-tool scenario and +17.7% in the mixed-tool context across 8 distinct tasks. Enabling access to external knowledge and information, specific tools significantly boost the performance of Calendar (+19.1%), Weather (+13. ++.99,), Code (+14.3%), and Translator (+54.1%% respectively. Remarkably, Themis is achieves perfect accuracy of 100% Calendar and Weather tasks. Moreover, it attains an accuracy above 98% on Code and Calendar tasks, providing substantial evidence for the efficacy and motivation behind integrating external tools into the reasoning process.
In mixed-tool experiments, Themis demonstrates robust performance in concurrently learning di- verse tools, with out of 8 tasks displaying superior performance compared to the baselines. No- tably, Google Search exhibits slight decline during mixed-tool training. This can be attributed to the diverse sources from which data is collected, especially from Reddit, resulting in overlapping domains and complexity for the models to learn effectively. Addressing this challenge necessitates meticulous data cleaning and rigorous human annotation processes for further exploration. These findings underscore the remarkable potential of Themis s across a broader spectrum of tools, em- phasizing its adaptability and versatility in real-world applications.
Scaling Trends in Themis. In our investigation, explored models spanning different scales, ranging from Vicuna-7B 7B to Vicuna-33B, all within the context mixed-tool setting. To enhance the efficiency of our training, utilized the LoRA technique (Hu et al., 2022) for parameter-

---

Published as a conference paper at ICLR 2024
Model
#Param
TruihfuQAA
Retarded bar (en)
Model
#Param
Zero-shot
Hine-tuning
GPT-3
RM (Bert-Large)
340M
51.66
$2.50
OPT
175B
21.0
RM (Vicuna-7B)
7B
35.78
65.83
Gopher
280B
29.5
Themis 8
7B
55.00
70.00
Galactica
120B
26.0
w/o Fobseraaton
7B
55.83
71.67
RM (Vicuna)
7B
30.7
68.0
Themis
36.8
73.3
Table 2: Results on the HH-RLHF* dataset, comparing Themis s with vanilla RMs in zero-shot and finetuning evaluation.
Table 3: Results TruthfulQA (MCI) and
Retarded-bar datasets.
efficient fine-tuning. The experimental results, outlined in Table (last 3 rows), elucidate a positive correlation between the scale of the model and its overall performance, phenomenon that aligns with established scaling laws (Gao et al., 2023 Askell et al. 2021). We also full-parameter fine-tune Themis (Vicuna-33B) and obtain the best performance.
Effeet of Varying Training Epochs. As depicted in Figure 3 (left), we observe that RM (Bert- Large) necessitates more training epochs to reach convergence, whereas RM (Vicuna-7B) achieves optimal performance after just a single epoch-an an observation consistent with prior research find- ings (Stiennon et al. 2020 Wu et al. 2021 Ouyang et al. 2022). In contrast, Themis s does require additional training epochs to learn tool invocations and rewards effectively. However, it outperforms traditional RMs, even within single training epoch.
Reward Difference Visualization. We visualize the average reward scores vary with the number of training epochs in the right of Figures 3 Themis s consistently exhibits a proclivity to assign higher scores positive answers and lower scores negative answers, indicating a heightened level of differentiation. Moreover, this differentiation progressively intensifies as the model training.
4.3 ANALYZING THE ROLE OF TOOL USE
How Does Themis Learn to Use Tools? To understand the role of the tools and how the RMs learn the tool invoking, we analyze the relationship between the number of tool invocations, model performance, and training epochs shown in Figure 4 Our findings indicate pattern in the total number of tool calls, which initially increases and then decreases. Additionally, we observe gradual reduction in the number of incorrect tool calls. These trends collectively suggest that Themis s acquires the ability to invoke tools effectively. As depicted on the right side of Figure our observations reveal that Themis consistently exhibits a higher propensity to utilize Google Search during problem-solvine tasks. This aligns well with human behavior, reflecting the natural inclination individuals to resort to search engines when seeking solutions.
Does Themis Really Make Decisions Based on Observations? We specifically selected data instances where Themi effectively differentiated between positive and negative answers. Subse- quently, we manipulated the outcomes of its tool invocations (Observations) and adjusted its states to assess Themis's performance. For instance. let's consider the question "What is the weather like in New York on 2023-06-24?" Both the positive answer and the initial Weather observation were recorded as "Sunny"" We modified the observation to "Raining", thus transforming the answer into a negative response. Remarkably, we observed only a marginal decrease in accuracy, highlighting the robust alignment between our method's predictions and the tool observations.
Do Tool Use and Reasoning Traces Count? To comprehend the functionalities of the reasoning process of Themis, we set 5 = 0 and w O to mask Observation and Rationale in Themis, s, respectively. The results can be seen Table [ highlighting the substantial contributions of both Observation and Rationale to Themis, especially in the Multi-Tools category. We find that the performance of Themi s clearly drops when we exclude the Rationale component, proving the effectiveness of step-by-step reasoning before output reward scores.
Ablation Study:
Case Study. We show some qualitative examples in Appendix D.I showcasing the effectiveness of the tool invocations. By leveraging the external tools, Themis validates the answer accuracy and make decisions through systematic, step-by-step reasoning process.
7

---

Published as a conference paper at ICLR 2024
Come
Weorg
Ratio of Tool Invocations
1.4
1.2
3-
0.8
0.6
Cakallatr
Calendar
Weather
Corle
Transtator
Wikiwaacch
Google
0
1000
2000
3000
3500
4000
Number Tool Invocations
Figure 4: Left: The variations in the number of correctly invoked tools and incorrectly invoked tools. The dashed line is the total number invoked tools in TARA. And the pentagram refers to the best performance epoch. Right: Comparison of the number of invoked different tools.
4.4 GENERALIZATION PROBING IN DOWNSTREAM TASKS
Out-of-Domaiie Evaluation. Ideally, Themis s is expected to possess adaptive tool invocation capa- bilities and the ability to score unseen prompts and responses. Consequently, select 150 instances (one instance has one positive answer and one negative answer) from HH-RLHP (Bai et al. 2022b) and denote it HH-RLHF* Initially, training on our TARA dataset) in the zero-shot setting to estimate their extrapolation ability. As shown in Table our findings reveal that Themi consistently outperforms all RMs, especially RM-Vicuna--77 BB.u We further introduce 500 instances for model finetuning, and we find the effect of Themis s is significantly improved. indicates that our Themis can extrapolate to output-of- domain scenarios effectively with minimal data fine-tuming.
assess the performance of both RMs and Themi (after
More than RM: Truthfulness and Factuality Probing. Given that RMs are employed to rank various responses for single prompt, it is natural to leverage RMs for addressing multiple-choice tasks. We experimented on multiple-choice problems on TruthfulQA (Lin al.] 2022a) and trans- lated Retarded-batt ba denoted as Retarded-bar (en), to access the model's s capacity for truthfulness and factuality (Refer to Appendix D.2 for dataset details). As shown in Table 3 Themis outper- forms competitive LLMs including OPT 175B (+15.8%), GPT-3 (+15.8%). Galactica (+10.8%), and Gopher 280B (+7.3%) on TruthfulQA task in zero-shot evaluation. Moreover, we selected Retarded-bar, challenging dataset containing puns, unusual punctuation, and irrational logic, to assess Themi i s's ability in solving fact-related confusing problems. We show that Themi s over- shadows the vanilla RM Retarded-bar (en) by 5.3%. Examples in Appendix [D.2] showcase that Themis can retrieve knowledge with external tools and enhance its truthfulness capability.
4.5 EXTENDED EXPERIMENTS: FROM RLHF TO RLTAF
Model
PPL 4
Gong
Vicuna-74
11.19
Vicuna- 7B-SFT
8.14
WcBBeachh
0.54
Vicuna-7BB 7B-PPO (RM)
8.10
Code
Vicuna-7B- (Themis)
7.88
Calcuator
0.22
Table The perplexity evaluation in RLHF across different stages in PPO, SFT, etc. Our model outperforms base model, SFT model, and PPO with con- ventional RMs.
Figure 5: Human preference evalua- tion, comparing PPO (Themis) to PPO (vanilla RM) across 200 test prompts.
Automatic Evaluation. We conducted experiments to assess the impact of Themis in RLHF, namely reinforcement learning from tool-augmented feedback (RLTAF). Following prior stud- ies (Stiennon et al., 22200) [2020[Ouyang Ouuang al. 2022), we implemented three stages: supervised fine-tuning (SFT), traditional RMs, and fine-tuning the policy against these RMs using Proximal Policy Opti- mization (PPO). Utilizing TARA our training data, detailed experimental specifies can be found
httts::/huggingg: http:///ac....tttaaaaata////aaaa http://nww.............//.///t/n
8

---

Published as a conference paper at ICLR 2024
in Appendix C.2 In RLHF, Themis employs external tools for tasks like arithmetic computation, code execution, and factual lookup. The results presented in Table indicate that PPO optimized against Themis achieves lower perplexity compared to vanilla RMs.
Human Preference Evaluation. We further conducted human evaluation, comparing win:tie:lose ratios across four domains on 200 test prompts. As illustrated in Figure our method outperforms baselines, achieving an average +32% win rate across the four different domains and consistently surpassing vanilla RMs in all four tasks. Notably, our approach demonstrated substantial improve- ments in fact-related question answering and arithmetic computation, providing robust evidence for the effectiveness of our methodology.
RELATED WORK
5.1 REWARD MODELING IN HUMAN ALIGNMENT
The challenge of aligning machine learning systems with human preferences has seen considerable exploration. Early efforts involved training aligned agents by imitating human behavior (Pomerleau 1991 Abbeel & Ng 2004 (Ho & Ermon 2016; Finn et al.) 2016). However, these methods often struggled to outperform humans due to the requirement for substantial amounts of expensive data. A scalable solution was proposed by Leike et al. (2018). utilizing an RM to align machine learning systems with human performance. Stiennon et al. (2220) fine-tuned language models through rein- forcement learning by training RM mimic human preferences in summarization tasks. Similar approaches were adopted by Nakano et al. (202I); Ouyang et al. (2022): Bai et al. (2022b). focus- ing on aligning L.LMs like GPT-3 towards honesty, helpfulness, and harmlessness.
a parallel vein, [Shen et al.] (2023a): [e et al. (2022); Bukkhrin et al. (2023); Shojaee et al. (2023) contribute to the field by focusing on reward design in reinforcement learning for code-related tasks, which is pivotal for augmenting the code understanding and generation capabilities of models. However, these existing RMs faced significant challenges, such as real-time information processing, limited to specific tasks such as summarization and code generation, and struggle with assigning rewards for intricate tasks like mathematics., To address these limitations, our approach incorporates external tools, augmenting the reward model and mitigating these challenges.
5.2 TOOL LEARNING
The intersection of specialized tools and LLMs has recently gained significant attention in re- search (Mialon et al. 2023 Qin et al. 2023). Current studies in this area can be categorized into two
main approaches: tool-oreened learning Yao et al., 2023 Nakano et al.] 2021 Qian et al.) 2023] Shen et al. 2023b) and tool-augmented learning (Schick et al. 2023 Lu et al. 2023 Tang et al, 2023 Qiao al. 2023). In tool-oriented learning, LLMs serve decision -aaing hubs for the strategic use of tools. Conversely, aool-augmented learning treats tools as complementary resources that enhance LLMs' capabilities. Unlike previous works, focus lies in tool-augmented reward modeling, aiming to align rewards with human preferences by incorporating tool assistance.
6 CONCLUSIONS AND FUTURE WORK
In this paper, we introduce Themi s, a novel approach designed to enhance reward models by en- abling interaction with external tools, thereby facilitating a step-by-step reasoning trajectory. Our contribution also includes the creation of a comprehensive tool-augmented reward dataset, TARA, which encompasses detailed data on human preferences and intricate tool invocation processes. Through comprehensive experimentation. including preference ranking analysis, ablation studies, generalization assessments, and RLHF/RLTAF probing, we have demonstrated the substantial ben- eñits of Themis in augmenting interpretive capacity and scoring reliability. Our results underscore the effectiveness of our approach integrating external tools seamlessly into the reward modeling process. Looking ahead, an exciting avenue for future research could involve exploring Themi in multi-turn dialogue generation. Understanding the intricate dynamics between external tools and natural language generation in interactive conversational contexts presents promising and intrigu- ing direction for further investigation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like to express our gratitude to the anonymous reviewers for their insightful and construc- tive feedback. Additionally, our appreciation goes to Yaqian Han, Pan Tang, and Yuchen Ding for their helpful assistance with the human evaluation process. This work was supported by the National Natural Science Foundation of China (No. 62206246), the Fundamental Research Funds for the Cen- tral Universities (226-2023-001333 LGG22F03001 I), Ningbo Natural Science Foundation (2021J190), Yongjiang Talent Introduction Programme 2221A--56-G1 CCF-Baidu Open Fund, and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
Zhejiang Provincial Natural Science Foundation of China (No.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of experiments detailed in Section 4 we committed to fostering transparency and accessibility in our research methodology. The source TARA datasets, fundamen- tal to our study, have made publicly available GitHub In addition to datasets, provide com- prehensive support for replicating our experiments. The detailed source code for both the conven- tional Reward Model (RM) and our proposed Themis approach is included in the supplementary materials. These materials encompass all necessary scripts and hyper-parameters essential for re- producing our results. The availability of datasets and thorough documentation of our experimental setup underline our dedication to promoting rigorous and replicable scientific research.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Carla E. Brodley (ed.), Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004), Banff. Alberta, Canada, July 4-8, 2004, volume 69 of ACM Inter- national Conference Proceeding Series. ACM, 2004. doi: 10. 10.1145/1015330111555333 URL
https://don .ecooo.orrgooo wrw./o............ 1ee..114551110 40455/1155300
1015430]
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Pas- sos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yamping Huang. Kathy Heelst-Hllsstem, Gaurav Mishra, Erica Mor- eira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Fein- berg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. CoRR, abs/2305. 10403, 2023. doi: 10.48550kur2iv.2355..0o 10403. URL/nttps: //oo.. www.wo..........o. 4155:/...4555/ara/....a 2305 10403
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds. Danny Hemandee, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment. CoRR, abs/21 12.00661, 2021. URL http:tttt:/://ttiit httt:::://xxxiv.org//bbbt F112.00861
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Pro- gram synthesis with large language models. CoRR, abs/2108.07733 07.732,, 2021. URL https:
w/arxiv.org///// 22108/18..000 07732
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jack- Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac HatfieldDoodde Danny Her- nandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine
betps://github...
resee--eeeeeeeee Augmenttttttttt Reward Rew----ewwddddooo
10

---

Published as a conference paper at ICLR 2024
Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR. abs/2204.05862, 2022a. doi: 10445500axXi.22244.553222 URL
ht.tps://doi .oroooooroo .or.
arxiv. 2204. 05862
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jack- son Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hattield-Dodds, Danny Her- nandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022b. doi: 104485000aXXi.22040.0666 URL 2204 05862
https: /tops:///doi... www.ddi.orgg/// htt.l//ro//....////1.. 0855445555/111
Alexander Bukharin, Yixiao Li, Pengcheng He, Weizhu Chen, and Tuo Zhao. Deep reinforcement learning from hierarchical weak preference feedback. CoRR, ths/2309.02632., 2023, doi: 10. www.oo.glo.... 48550 sitzxii..300 .26322
48550/ARXIV22300022222
httthtt::
Yekun Chai, Shuohuan Wang. Chao Pang, Yu Sun, Hao Tian, and Hua Wu. Ernie-ode: Be- yond english-centric cross-lingual pretraining for programming languages. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compu- tational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 10628-10650. Asso- ciation for Computational Linguistics, 2023. doi: 1118653/11/0223. http:////2.2....s-aaaaaaeaa https://doi. /10.gg/0../ (18833/v112000
URL
findings ac 676
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder. Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL htttps://aaxii.....:.t org.
03374
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. wttpp//ppppp///////pppp
4299 4307, 2017. URL https::/prroceedngg..ttt dbe200aaaaD0ie113334440000eeee22
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacoh Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URI.
http:p/s///xxii..r//aa////////// 14168
Dahoas. Rm static. http::///uuginnfffcccc/////eeet
2023.
Jacob Devlin, Ming- Mii Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v 1/1/3/ 1423. URL
11

---

Published as a conference paper at ICLR 2024
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment. org/
CoRR, abs/2304.06667, 2023. doi: 1048500ARXX1.22440066677
10. 0...48550aaxi111 2304 26767
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 49-58. JMgR.org, 2016. URL
ngs...g.. ml press/ v B/ f finn 6. html
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt. Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10835- 10866. PMLR, 2023. httl:tht::: httpp/tttttppceeeeeeeeeeeeeeeeee http///oceddicccccccccoccooce.ce ngs. m1 paress/v2000 /ttp////222......o html
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Daniel D. Lee. Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp- 4565 4573, 2016. URL httpttt//procceddionooo meeeeeedddiinnnn.enieeeeeee neur ips....p.............. cc/paperr/216/haatt cc cebe2287 www.eeeeeeeeeeeabeeeeeeee....... -btta-9-----d-aaaaet.tttet
Abstract html
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL er wttp://www...e...oooooooooeeeeee
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8470- Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1111188/1120222.al 20015/1///222. 579. URL https::
org
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Trans, Assoc. Compul. Linguistics, 7:452-466 2019. doi: 10. 10.1162/tacl) _00266. URL http::ttt//////tt http:::://doi..oo http///oi.rrg//...
11 6etetaaaaaaaaaaaa 00276
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In
learnig
NeurlPS, 2022. hataaa/aeeaaaceeee004eaaaaaaaaaa Absstrratt-ooofeeeneee.iee html]
8636419des.
Abst
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent aligmment via reward modeling: research direction. CoRR, abs/1811.078777 2018. URL
http::/arxixn wwtp:/wwwwww///////////oaee/
Patrick S. H. Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: evaluating cross-lingual extractive question answering. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10. 2020. pp. 7315- 7330. Association URL https://
for Computational Linguistics, 2020. doi:
doi wooi.org/10... 11653/ v2230v///2220022 ac -aammann.... 6553
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman- Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solv- ing quantitative reasoning problems with language models. In NeurlPS, 2022.
12

---

Published as a conference paper at ICLR 2024
URL
http: http:///pwpss......occoo/oe/e/ea
/aahh/
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan 7hang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferran- dis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! CoRR, abs/2305.06166 2023. doi: 1448550/rXXi.22355006666 URL https::/doi.ii.... wttg:/0.....///..11..ooooooo 230. 06161
Yujia Li. David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago. Thomas Hubert, Peter Choy, Cy- prien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando Freitas, Koray Kavukeuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. CoRR, abs.2203.0281114444 2022. doi: (0.48550/axxii:2200.778 URL /attps://doli org wtw.://///...../.......... arriv. 2205 2203.07822....
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic hu- man falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. pp. 3214-3252. As- sociation for Computational Linguistics, 2022a. doi: 10.18653/v wtw.://.allloooog..o...oom http::://doi.ooo
URL
1ww111663/1111 2ww.222222.2.2220 ong 229
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' O'Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9019-9052, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: org/ 202.2.2222...... main. 616
URL
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chum
Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large lan- guage models. CORR, abs/2304.00842. 2023. doi: 10,448550/axxx..230..oo .0882.. URL https:
//doi.org/I0 48550/arXiv. 2304. 09842
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul.
Peft: sfate-of-the-at parameter-efficiee fine-tuning methods. htttps://gitie com/ hucgingfacce/peft. 2022.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu. Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey. CoRR, doi: 10.485500axxx..23022.78444 URL httpt::: :/dp:.://dd.... orq/10.
abs/2302.07842. 2023.
48550/
07842
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-nssisted quwstion-answeenneom with human feedback. CoRR, abs/21 12.09332, 2021. URL https://arxiie
org http:///wwxivvoooo//////////////
13

---

Published as a conference paper at ICLR 2024
OpenAL GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/aXXx22233.887744 http:/.....550/2222.222..... 08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In NewrIPS, 2022. URL /ttt::://papeee nips. wttp:pppw..iip...cppppppeeeeeeee wwtp://wwwpwppap/////aae/ /hash/ blefde 3be 3144444331438888
- -wst--aatttttoeeeeeeeeeeee Conferencee.cccccecoooecooo html
Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Comput., 3(1):88-97, 1991. doi: 10.1162/neco.199 1991.3.1........ URL http::://t.i..ooo http:///ooo.oog/..////
1162/neco. 3.1.88
Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. CREATOR: disentan- gling abstract and concrete reasonings of large language models through tool creation. CoRR.
abs/2305.1433 14318, 2023. doi: 10.48550/arXiv.230555 23311.188........... URL httpp:://ol. wtg//....o.g... 48550/ 20150//rXiv.o 2305. 14318 2338.14318
Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. CoRR, abs/2305. 130688 2023. doi: 10.48350
13068. https:/:: //deps::/ddo. 2ttp//1..455/ar3..0 /0./////riivv 133.0....68.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with founda- doi: (0448500axX1..2304.8833.
tion models. CoRR, hwt.22004083544 2023.
URL https:
7/doi org/000 (18550/arXii10 2304 23340
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.00461, 2023. doi: 10.485550/rXXw.2232200006 URL https:
7/doi. orriioorg/00. 10. 48550/aaXXia 2302. 04761
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, has/1707.063477 2017. URL http::/ org/
1707. 06347
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang. Pangu-coder2: Boosting large language models for code with ranking feedback. CoRR, 0bbs2237..14934 2023a. doi: 10.48550/ ARXIIV2307.14936 utLhttps: //toi..://d....o wtw.odoo....g.oooo 480../ wtt.//5///.ii..../ 230. 1401 14936
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- ginggpt: Solving Al tasks with chatgpt and its friends in huggingface. CORR, 2obb//333.17880 2023b. doi: 00.48550/a2Xi.2233..1 17580.. URL https: doi htt.//..../.././//.....
2303. 17580
Parshin Shojaee, Aneesh Jain, Sindhu Tipimeni, and Chandan K. Reddy. Execution- code generation using deep reinforcement learning. CoRR, abs/2301.13816. 2023. doi: 10.48550/ ARXIV.2301.13816 htRhhttps:/ httpttttt/i........e..tp. org/ www.oo//.........././/../////../ 48550/ arxiv. www.wrwwww................ 2301. 13816
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325. 2020. URL URLpstts:://axxt wttps///.ggbbaa/.......... 01325
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Gen- eralized tool learning for language models with 3000 simulated cases. CoRR, abs/2306.05301, 2023. doi: 11.485500axxi.2230005555 2306 05301
URL
14

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09288. 2023. doi: 10.48550/arX...3307.00 079888..... URL
dttpp:://do..o2/10...0 (ttp://1...455//21//11../ 230.:/3wwwwv......o... 09288
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 3448-13508 Association for Computational Lin- guistics, 2023. doi: 10. (0.18653/v1/2023 aaw.0223..llloog.55.... 754. URL hwtp::tts////tt http::///abi.or/11118858885
v1/2023. acl long. 754
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022. URL httt:: nips wwtpappp_aaaaaeeee saapprr_fllea/pppee_e 50 2222/////22250011122241111111110 cotea
html
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul F. Christiano. Recursively summarizing books with human feedback. CoRR, abs/2109. 2021.
URL httthhtsss///axiii org/ hbtpg//bs/2100 2109. 10862
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- /ttp//ppeeeevveeeeeeeeeeeeeeeeve
view.net, 2023. URL / eww net. / /ptpf/dfff/ ?idEE__dwE___ v vluYUL-X
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shum- ing Shi. Siren's s song in the Al ocean: A survey on hallucination in large language models. CORR, abs/2309.01219, 2023. doi: 10448550WrXXi.22300001121 URL https://dde .org/ 10. 408548855axii.
01219
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Bric P. Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. CoRR, abs/2306.05688 2023. doi: htt.ht/t....L.tttttttttt.
10. 00448550/22xiw2200605566. URL/tttss:/ /ow. owg/10. 48550/ (113550/arxiv...
05685
A LIMITATIONS
Limited Tool Scope. While our study incorporated experiments with seven distinct tools, the po- tential of tool-augmented Reward Models (RMs) could further be explored by expanding the range of tool APIs to encompass hundreds or even thousands of diverse tools. Additionally, integrating with interfaces like the ChatGPT plugin could offer broader application spectrum.
Time Overheads. The integration of external tools introduces an additional cost of complexity and might result in increased processing time. The speed of tool invocation is contingent various factors such as network conditions and tool execution speed. Consequently, the real-time applica- bility of the tool-augmented RM is influenced by these contextual variables.
15

---

Published as a conference paper at ICLR 2024
Single-Turn Data. Our preference data collection limited to single-turn prompt-respont pairs. Extending this framework to encompass multi-tum turn interactions could enrich the understand- ing of complex dialogues and enhance the applicability of tool-augmented RMs in real-world versational contexts.
Challenges in Data Generation. While we construct an automatic pipeline for dataset construc- tion, encounter certain challenges. We design some heuristic rules to generate data for particular tools, incurring associated costs that rise when extended to a broader range of tools. Additionally, we employ GPT-4 agents generate tool invocation processes and rationales, incurring monetary expense associated with this computational process.
Limited Model Scale. Our experiments primarily revolved around Vicuna-7B. Exploring the scal- ability of tool-augmented RMs to models surpassing 100 billion parameters could provide insights into the challenges and opportunities at larger scales, expanding the applicability of this approach.
Preliminary RLHF Exploration. The exploration of tool-augmented RMs Reinforcement Learning from Human Feedback (RLHF), namely Reinforcement Learning from Tool-Augmented Feedback (RLTAF), remains at its preliminary stages. Comprehensive experiments, covering a wider array of tasks and scenarios, are essential to fully understand the potential and limitations of this ap- proach in reinforcement learning paradigms. Future research endeavors will focus on conducting in-depth and extensive evaluations delve deeper into the capabilities of tool-augmented RMs in various RLHF settings.
B ADDITIONAL DATASET INFORMATION
Table 5: Comparison between our TARA and previous reward datasets. Our dataset contains multi- ple domains with tool invocations, and we construct the data via multi-agent interaction.
Name
Train
Test
Domain
Tools
Source
WebGPT Comparisons Nakano al. 2021]
19.6k
Long-form QA
ELI5 Human
RM-Static (Dahoas 2023
76.3k
5.1k
Helpful Harmless
HH RH-R Human
Summarize from Feedback Stiennon al. 2020
2020
179k
6.31k
Summary
X
TARA (Ours)
13.6k
1.4k
Multiple
7
Multi-Agent
In this section, we describe in detail the construction process of our augl-augmented reward dataset. We first introduce the generating process of each tool, then we report the details the multi-agent prompts. Jsooss the data statistics of our TARA and the overview information such as source data, the number of the train-set and test-set of each tool is shown in Table
Table 6: Data statistics of ToolBank.
Tool Name
Data Source
Train
# Test
Calculator
GSM-8k
877
154
Code
HumanEval mbpp
486
189
Translator
MLQA
1682
300
Google Search
WehGPT Comparison
3932
134
Calendar
320
166
Weather
476
158
WikiSearch
NaturalQuestion
5399
284
Multi Tools
Calendar Weather
432
144
B. DETAILS OF DATASET CONSTRUCTION
In this section. introduce the details of the dataset construction. We construct the dataset based on the heuristic rule-based method and some spencsource datasets with multi-agen aaeti interaction. We list instances of each tool from F Figure 12
16

---

Published as a conference paper at ICLR 2024
B.1.1 CONSTRUCTION WITH HEURISTIC RULE-BASED METHOD
We design some heuristic rules generate the question-an answer and tool invocation processes for certain tools, including Calendar, Weather, and Multi Tools.
Table 7: The city set, date set, weather set, question and answer prompts set construct the Weather tool data.
Key
Candidate
City
Mexico, Saint Helier, Bangalore, Beijing, New York, Sydney, Aleppo, Homs, Sansa,
Date
2023-06-19, 2023-06-20, 2023-06-21, 2023-06-22, 2023-06-25, 2023-06-28, overall weather, temperature, precipitation, humidity, wind speed, visibility, UV index What is the (weather) in (city) (date)?
Weather
Question Prompts
Answer Prompts
The (weather) {city) on (date) is (answer).
{city)ss 's (weather) on [dute) (unswer).
On {date), {city)'s is [weather) indicates (answer).
Question Answer
Tool Inovke
Thought: nead to search the weather Siping on 2023-06-19.
Question:
What the wind spoed Siping on 2023-06-1111 06-197 The wind speed for Siping 2023-06- indicates 15.5()
Answer:
Action: Weather
Action Input: Siping, 2023-06-19
Weather Search
Reasoning
Observation: weather Siping on 2023-06-19
Rationale: answer states
speed Siping on
overall weathar: Mist: temperature: 21. 21,7(C); wind speed: 15 5ikch), 9.6lmshi: precipitation: 0.01mm), D.0jinchl, visibility: 7.7 7.7(k)), 4.0/miles); humidity: 71.0; UV indexe 7.D.
06-19 was 15.5 lkph). The execution result of the tool "Weather" validates information
speed reported day incload 15.5 (kph). Hance, the previded answer aligns perfoctly with the tool's output.
Figure 6: An example of the Weather tool.
Weather. The Weather tool is realized by weat herAP g that the input consists of city and date, and the output provides information about the weather in the specified city on the given date. Initially, we compile a candidate city set by selecting the most common cities from Wikipedia. Subsequently, we create a candidate date set and a weather set specific to the Weather tool. Inspired by Wang et al.) (2023), we initiate the process with seed question prompt and answer prompt, such as "Question: What the (weather) in {city) on [date)? Answer: The {weather) in {city) on {date) is {answer)." ." and expand upon it using ChatGPT. Finally, we iterate through the city set, date set, and weather set, incorporating them into the prompts to construct the question-ans, answerr pair. We seek positive answers from the Weather tool, while negative answers perturbed based on the positive answer. The city set, date set, weather set, question and answer prompts set are listed in Table 7
Question Answer
Tool Inovke
Question:
Can you tell me the number days that separate 2000-
Thought:
many the
02-1 3-06-0477
difference between 2020-02-11 and 201
Answer: The di differncce between 2003-02-14 14 and 2013-06-D4
Action: Calandar
4657.
Action Input: 2000-02-11 02-14, 2013-06-04
Reasoning
ABU SB
Calendar
Rationale: The answer provided by the Al, that the fference between 2000-02-14 -44 and 201 2013-06-04 is 4859 days, entirely
result obtainad from the Calandar tool, which also calculated the number of days 4859.
Observation: 4859
Figure 7: An example of the Calendar tool.
www.weatherapi..cmm
17

---

Published as a conference paper at ICLR 2024
Calendar. The construction process of the Calendar tool is similar to the Weather tool, with the primary difference lying in the question prompts and answer prompts. The Calendar tool serves three primary functions: determining the weekday of a given date, calculating the difference between two dates, and finding the date that follows another date by n days. For each of these functions, we have composed distinct seed prompts and subsequently expanded upon them using ChatGPT.
Multi Tools. The Multi-Tools primarily involve chained invocations of the Calendar and Weather tools, An illustrative question might be, "What is the weather like in city in the n days after date?" This necessitates first invoking the Calendar tool to obtain "the a days after date" and subsequently invoking the Weather tool retrieve "the weather like in city" for the specified date. Obviously, the data generation process for Mulri-Tools follows the same pattern as the Weather and Calendar tools.
B.1.2 CONSTRUCTION FROM OPEN-SOURCE DATASETS
Some tools are challenging to create using heuristic rule-based methods. Therefore, we generate them based some open-source and high-quality datasets.
Question Answer
Tool Inovke
Ruby is centimecers than centimeters taller than Charlane. Janet 62 centimeters tall and Charlene tall,
Thought: naad to invoka the calculator tool to verify the correctness the calculation process and the final answar.
Ruby?
Action:
Calcualtor
Answer:
cm
Action Input: <<2"62-124->126
<<70+ 124 144- 194>>194, <<194-2-192->>
Pablo 70 am + 124 cm <=70-124 14940> Ruby
94 194 cm tall.
192
DE
Calculator
Reasoning
consistant with question and the calculation stops used to arrive at the answer are correct, as confirmed
Rationale:
The given
Observation: Both the the answer correct.
calculator All the intermediate steps, ca calculating Charlane's haight to datermining Ruby's height, are correct.
Figure 8: An example of the Calculator tool.
high-quality dataset comprising linguistically di- verse grade school math word problems, making it well-suited for constructing math reasoning data involving tool invocation. We randomly selected 1,000 instances from GSM-8K to serve as both questions and positive answers. We query the negative generation agent to generate negative swers.
Calculator. GSM-SK (Cobbe et al. 2021)
Question
Answer
Tool interpreter
Question: Write python function chack whather trequency
Thought:
digit loss than equal to cigit
executte the code and verify the correctnass of the answer:
Answer:
def validatein)::
Action:
Code
for range(10):
Input: answer
temp. count - n,
while (temp):
Itemp 10 ik
Lists:
counti-1 if Toount
assert validate/1233 231) -- True
assert saliidatt/52247 -- Fal
return False
validate/321) True
temp //- 10
Trus
Code
Reasoning
The provided answer is Python function mean: to verify numbar less than aqual to the frequency of each digit daled works by looping through digit itselt. The possible digits (D-9), and each digit, tterates through the digits the given number, 'n', checking if equels the current digit 'I. If the digit
Rationale:
Observation: The rate 1.0, All test cases
passed.
False, incicating the number does not meet the condition. All tests are of the enswer verified.
frequency (count)
passed and the
Figure 9: An example of the Code tool.
Code. We integrate the HumanEval (Chen et al. 2021) and MBPP (Austin et al. (021) datasets as the positive data of the Code tool, encompassing questions, positive code answers, and test cases.
18

---

Published as conference paper at ICLR 2024
Additionally, we leverage StarCodeBase (Li et 2023 generate negative code answers that fail to pass all the test lists.
Questien Answer
Tool Inovke
Thought: nead trandate the question English halp answer the quastion.
Question: welchern wurde Baskin-Robbins gegründeet?
Answer: California.
Action: Translate
Action Input: In weltham Staat wurda Rabbis gegründet?
Reasoning
Translate
Rationale:
The question saked, wekchen wurdee Baskin- Roobins gegrindar?" which translates to which state was Baskin- Roobins founded?" The answer provided, "Califorii," is accurate because BascinRobbiis
founded Glendale, California, Tharsfora, the answer absolutaly relevant and directly answars the question.
Observation: state was Baskin-Rahbins
foundec/?
Figure 10: An example of the Translator tool.
Translator. The translation tool is powered by the Baidu Translator AP which supports trans- lation in 200 languages. We created the dataset of the Translate tool based on the MLQA dataset (Lewis et al., (200)), encompassing QA instances in 7 different languages. Subsequently, we call the negative generation agent generate the negative answers.
Question Answer
Tool Inovke
Thought: I need to invoke the wiki search tool Intermountain Region location.
Question: When was the first car madie America?
Answer: Betwven the front ranges the Racky Mountains the east and the Cascade Range and Sierra Nevada on the west
Action: Wikiseachh
Input: Intermountaiin Region location
Wikisearch
Reasoning
WYETFEII
verify question about the location of the Intermountain Region. The provided: iit located between the front renges of Rocky Mountains on the east and the Cascade Range and Sierr Nevada on the west" The information rerrivved from the tool matches the given answer, hance validating its accuracy.
Rationale:
Observation: Interttountain West Intermountain
West The Intermountain West, Intermountain Region, geographic
observatio from the tool corroborates the
geological region the Western Uniteed Stares, It located between the front ranges of the Rocky Mountains on the east and the Cascade Range Siemr Nevada on west, The Intermountain West
Figure 11: An example of the WikiSearch tool.
WikiScarch. The objective of the WikiSearch tool is to bridge the reward model with the knowl- edge base Wikipedia. We randomly selected 5,500 instances from the Natural Question dataset (Kwiatkowski et al.] 2019), which comprises real anonymized, aggregated queries posed the Google search engine and annotated with Wikipedia pages. For negative answers, we employ a negative generation agent.
Google Search In contrast other tools, we construct the data of Google Search tool based on the reward dataset WebGPT Comparison (Nakano et al. (0021), which includes questions, positive answers, and negative answers. We utilize both the tool agent and rationale agent to generate the tool invocation process and the rationale segment.
B.2 PROMPTS MULTI-AGENTS
We present the prompt of multi-agent interaction, negative generation agent, tool agent, and rationale formulate a simulated environment incorporating human participants and three virtual agents: the negative generation agent, the tool agent, and the rationale agent, each embodied by GPT-4, The interaction scenario between agents and humans is depicted the upper left corner of Figure 5, with detailed prompts for each agent provided on the left. We detail the responsibility of the three agents as follows:
agent Figure 13
www. fanyi-nn-iii -api---ppp... ba com
19

---

Published as a conference paper at ICLR 2024
Question Answer
How is that the earth and other planets appear to be almost perfect circles and why are they shapad like that?
Question:
my trwiing knowledige about farmatio and shape of planets, Howwoe, should verify the facts related to the formation of planets and their gravitational infuence their shape. quick search should confirm Action: Google
Thought prov answer reasonable
Answer: Wi makes il am have their round shape all? why the
aspear almest tump and clump together Af a whis,
There
perfect circles. planets
space enough staff to have
Action Input: why are planets round?
a good amount of gravity. That's the force that holds stuff together space When a forming planet is big enough, it stare clear its path around the star it orbits, It uses gravity to snag bits of space pulls equally from all sidio Gravily pulls the edges like the spokes of planet
Google Search
Observation: A planet round because gravity planet's gravity pulls equally sides, Gravity pulls the oor to sdges spokes bieycle wheel This makes the averall shepe of planet sprwee, which three-d mersiieiiioo dir...
stuff A planse's
from the
wheol.
makes ovenal
dimenwionall
quite so perfectly round, Saturn and lupiter are bit thicker in the middle as they spin
Finaly, e planets
Thought: The observed information corroborated the explanation given answer.
around, and as they bulge out along the equator...
that the gravity a planct pulls cqually from all sidea, contributing to its spt orical shape. Now, sshuldd also validate the claim about the equatorial bulge of Jupiter and Saturn clue their
Reasoning
Action: Google
The answer given corresponds well with the results from the tools. The tool confimeed that planet's gravity pulls equally from all secles, shaping the planet into sphees, siich was the main point Additionally tool confimed Saturn Jupiter having ar equatorial buge,, leacing to being perfectly informative, demonstrating a strong alignment wi the too findings.
Rationale:
Action Input:Wi Why are Jupiter and Saturn not perfectly round?
Google Search
made
Observation: Saturn and Ju really and spinning really fast but grovity still manages to hold them together. That's why they bulge in the midele. We call the extra widhh the equatorial bulge. Saturn bulges the most of all the planats in our solar system.
Figure 12: An example of the Google Search tool.
Negative Generation Agent:
The responsibility of the negative generation agent is to gen- erate negative answer to question that has only a positive answer. It receives a question along with its positive answer and generates a negative answer that is indistinguishable from the positive one.
Tool Agent: Undertaking a challenging role, the tool agent receives a question-answer pair and produces appropriate and correct tool calls validate the reasonableness of the answer. The tool calls involve a Thought stage that includes a reasoning trace to determine whether tools should be called, and an Action stage that contains the necessary API calls with their required arguments.
Rationale Agent: The rationale agent asked to generate the Rationale stage by com- prehending previous contexts, synthesizing the question-answer pair, the tool invocation process, and the observations from the tool execution to systematically produce rewards.
B.3 DATA FILTER STRATEGIES
We design multiple data filter strategies in the dataset construction process. For the negative answers generated by the negative generation agent, we unify their format match the positive answers, in- cluding the punctuation, spacing, sentence structure, and so on, preventing the emergence of super- ficial patterns. For the tool invocations process generated through interaction between the tool agent and rationale agent, we discard the instances that exhibit invalid formats, exceed three interaction steps, lack relevant function calls, or manifest parsing errors in the execution results.
C DETAILS OF EXPERIMENTS
C.1 IMPLEMENTATION DETAILS OF RM
Table 8: Hyper-parameters to modulate different training configurations.
Model
Q
B
3
RM (Bert-Large)
0
0
RM (Vicuna-7B)
0
0
0
Themi s (Vicuna-7B)
1
1
Tobsevvaton
1
0
wo R
1
20

---

Published as a conference paper at ICLR 2024
Multi-Agent Interaction
Negative Generator Agent
Instruction:
Instruction:
Q: weather
helpful negalive answer generator, you question and a unswer, write similor but incorrect directly.
2023-06-04?
York cloudy
06-04.
Exomples:
There are some exmaples:
Tool Agent:
alcohol disinfectant?
Question:
Thought: I need to search weather New
York on 2023-06-04
Answer: By Sumerians
Action: history weather
antiseptic along with the dressing of wounds, using different types blleerr Negative Answer: By the Fgyptions, who used vodka disinfectant cleaning wounds, employing over 30 different varietics of vodka.
Action Input: Now York, 2023-06-04
Human:
Observation: weather for New York on
Question: when persona 5 the animation cominng
2023-06-02 is:
Sunny; temperature: 27.0(C)...
Answer: April 2,2018 Negative
Begin!
Rationale Agent:
Question: [question)
Rationale: cesulis
sumny,
(answer)
answer is cloudy, Thus, the given response proves
Negative Answer:
when compared tool's execution nutcomes.
Output: y
Tool Agent
Rationale Agent
Instruction:
Instruction:
helpful agent. receive
You helpful rationale agent ready provide explanation. Your role invelves receiving question- answer pair, the referenced answer score from human and exccation results tools, Your with ressonable explanation via think step...
question answer pair and the answer's score. Your
whether invoke
tools to invoke verify the seare.
give the
access following tools:
Tool List:
google_serper: serper: <descriptions>
Here example:
Input:
example tools invokes:
Q: What's the weather like in New York on 2023-06-042
Input:
weather New cloudy today.
Q: Why
Score: -1
A: Water wet hecause our brain receives...
The executor results tools:
Thought: nnee
Output:
Action: google serper
Thought:
check forozs in making water fool wel and whether the
Action Input: Why does water fiee wet?
Observation: Th.
polarity contributes wetness.
Action: google. serper
Output:
Action Input:
feel wer? Observation: The scnsation of wetness is langely cooling caused evaporation...
The results of executor is sunny, but answer is cloudy, Thus, the given response proves to be inaccurate when compared the tool's exocution
Rationale:
Input:
Input:
Output:
Output:
Figure 13: An illustration of the tool invocation generation process within the multi multi-agent interaction and the prompt of each agent.
21

---

Published as a conference paper at ICLR 2024
Table Hlyper-parmeete settings train RMs.
syyeer-praameers
RM (Bert-Large)
RM (Vicuna-7B)
Themis
Themis (LoRA)
epoch learning
8
5
5
5
le-4
le-5
le-5
le-4
batch size
128
64
64
64
learning scheduler
Cosine
Cosine
Cosine
Cosine
warmup ratio
0.01
0.01
0.01
0,01
sequence length
512
512
512
512
ouiput sequence length
1024
1024
LoRA rank
LoRA alpha
16
Training Configuration Hyper-parameters. The hyperparameters governing various training con- figurations enumerated in Table 8 Specifically, when vanilla RM, wherein scalar rewards are predicted through a fully connected layer without any pre- conditioned tool invocations. Additionally, setting 3 reward model will not be trained on the Observation or Rationale component, enabling controlled exploration of different training settings and their influence the model's behavior.
0, our method simplifies to
indicates that the tool- lool-augmented
Experimental Hyper-parameters. We report the experimental hyper-parameters at Table We ap- ply the same ppper-parameters for RM (Vicuna-7B) and Themis. s. For RM (Bert-Large), use larger learning rate, larger batch size, and train more epochs. We chose the best performance check- points of each model for comparison. We implement LoRA with the PEFT (Mangrulkar al. (022) framework. All models are trained in the same environment (8 x 40G A100 GPUs). Additionally, we incorporate the learning of positive answers by predicting the entire context of good samples, following the approach outlined in (Askell et al. (021). This allows our models to emulate "good" behavioral patterns in preference modelingg.uutthermon Furthermore, external APIs can exhibit instability or ex- perience failures, leading to null observations. To enhance the model's robustness, we intentionally introduce 1 5 random observation dropout during training. This approach simulates real-world scenarios where API unavailability may occur and equips model to handle such situations more effectively.
C.2 IMPLEMENTATION DETAILS OF RLHF
Table 10: Hyper-panee papamm-rrretter settings in SFT and PPO phrases. Note that y is a constant used to balance PPO loss and the unsupervised loss.
Hyper-parameters
SFT
PPO
epoch learning rate batch size
le-5
{1e-6 5c-7)
32
learning scheduler
Cosine
Cosine
warmup ratio
0.01
0.01
sequence length
512
512
weight decay
0.0
0.1
gradient accumulation step
27.8
Details of RLHE. We follow Deepspeed to implement the RLHF process, which consists of the following three steps:
Step 1: Collect samples and train supervised policy (SFT). The SFT phase is similar to the standard language model finetuning. Here, we follow Deepspeed- Chat and divide our TARA data, 20% for training Step and 40% for Step 3. Note that we collect the question and the positive
applicationss 1ogpiaaation///eeeppeeaaaaaaa
22

---

Published as a conference paper at ICLR 2024
answer of this data the supervised training data. And then fine-tune pre-trained Vicuna-7B model with LoRA. The resulting model is denoted as Vicuma-7B-SFT.
Step 2: Collect comparison data and train reward model (RM). We utilize the method in Section train a reward model (RM) or a tool-augmented reward model (Themis) s) on our TARA to predict the human-preferne output.
Step 3: Optimize the supervised policy against the reward model using PPO (PPO). We fine-
tune the supervised policy obtained in Step using the PPO algorithm (Schulman et al.) (017), with a reward signal provided by the RM or Themis s obtained in Step 2. The resulting model from this step denoted Vicuna-7B-PPO (RM) or Vicuna-7B-PPa (Themis).
We list the hyper-parameters of SFT and PPO phases in Table 10
C.3 EXPERIMENTS ON STANDARD REWARD DATASETS
To further demonstrate the effectiveness of our method, we conduct experiments on some standard reward datasets including the WebgGPT Comparision (Nakano et al. 2021) dataset and the HH- RLHF (Bai et al. 2022a) dataset. The results can be seen Table We partition the WebGPT Comparison dataset into 13.7K training samples and 2.4K test samples. Additionally, we randomly extracted 50K samples from HH-RLHF as training samples, amalgamating them all with our TARA dataset. The results reveal that our Themi s obtains superior performance than other vanilla reward models except RAFT (LLaMA-7B) (Dong et al.) 2023). However, RAFT (LLaMA-7B) performs SFT the 112K positive training samples of the HH-RLHF dataset and then executes reward mod- eling on 100K pairwise samples. contrast, Themis s trained within a multi-tast learning setting, achieving comparable performance with RAFT (LLaMA-7B).
Table 11: Experimental results on standard reward datasets.
Model
WebGPT Comparision
HHH-RHHF
Deherta-v3-large-v2 GPT-Neo-2.70
69.25
68.27
RAFT (LLaMA-7B)
79.52
RM (Bert-Large'
66.57
64.20
RM (Vicuna-
66.68
72.34
Themis (Vicuna-74 (Bcuna-7B))
69.53
77.59
D QUALITATIVE EXAMPLES
D.1 GENERATED EXAMPLES ON TARA
We show a qualitative example the Google Search tool in Table [3 The question of the exam- ple is about "Supply-Side economics", the answer not only provides comprehensive explanation of the concept but also offers an insightful analysis of its benefits for consumers. Nonetheless, all reward models (RMs) assign low reward scores. In contrast, our Themi s meticulously verifies the correctness of the answer by leveraging Google search tools and outputs high score for the answer through rigorous reasoning. In addition, sponses related to arithmetic computation and code implementation. Nonetheless, our Themis s can invoke relevant tools to verify the process of the responses, thereby providing robust and depend- able rewards. As shown in Table [2] our Themis detects an error in the calculation process and outperforms other RMs.
challenging for RMs to produce rewards for the re-
D.2 EXAMPLES ON DOWNSTREAM TASK
TruthfulQA. TruthfulQA Lin et al. 2022a) is a benchmark dataset to measure the truthfulness of language models, which comprises 817 questions that span 38 categories, including health, law, finance, and politics. Each instance of TruthfulQA contains a question and multiple choices, pair the question with each choice and feed it into our Themis to obtain the preferred score. Then we compare the score of each choice and choose the preference corresponding the highest score as
23

---

Published as a conference paper at ICLR 2024
Table 12: A case example of Themis on Calculator tool. Our Themis detects errors in the calcu- lation process by invoking the Calculator tool and provides a more reasonable score than RMs.. / refers to the rewards aligned with human preference, while refers to the rewards deviating from human preference.
Question:
Karlee has 100 grapes and 3/5 as many strawherries as grapes. Giana and Ansley, two of her friends, come visiting. and she gives each of them 1/5 of each fruit. How many fruits is Karlee left with in total?
Answer:
Karlee has 3/5 x = 1000:3/55-000000000 600-00>>60 60 strawberries. She gives Giana and Ans- ley 1/5 100 <<11=+100=0022>222222 grapes each. She gives Giana and Ansley 1/S x
<<1/ 5<1//560=12>112 strawberries each. Karlee is left with 100 (2 x 20) <<100-- (2 =60=>00))->>666 grapes and 60 (2 12) = <<60- (2+12)+ (011)=333>>3> =
strawberries. The total number of Karlee remains is 60 grapes + 34 strawberries =
<<60+ (660+34-92>> 94 fruit.
Themis:
Thought: I need to invoke the Calculator tool to verify the correctness of the calculation process and the final answer.
Action: Calculator
Action Input:
<<3/5+100=60>>66 <<1/5<60-12>>12, <<100- (2+20) -60>>60, << (2+ (2+12))=4222) -34>>34,
<<1/5+100=20>>>20,
<<60+34-94>>94
Observation: The calculations are incorrect. Details: 60-(2+12) equal 34.
Rationale: The execution results of the calculator tool indicate discrepancy the calculation process and the answer. The calculations for the number of grapes and strawberries Karlee has left are incorrect. The executor tool points out that the calculation process is wrong, and resulting incorrect final answer.
Score: -4.70
V
RM (Bert):
Score: 3.21
X
RM (Vicuna):
Score: 2.78 X
Table 13: case example of TARA. With the assistance of the Google Search tool, the score of Themis is is more reasonable than RMs. / refers to the rewards aligned with human preference, while refers to the rewards deviating from human preference.
Question:
Why do people still vouch for Supplu-Side economics?
Answer:
Supply-side side ecunomics is macroeconomicc theury that postulates that economic growth can be most effectively fustered by lowering taxes, decreasing regulation, allowing free According to supply side economics, consumers will benefit from greater supplies of goods and services at lower prices.
Themis:
Thought: I need to invoke the Google Search tool to verify the answer.
Action: Google Search
Action Input: What Supply-Side cconomics?
Observation: Supply-side economics macroccomoniic theory that postulates economic growth can be most effectively fostered by lowering
decreasing regulation, and allowing free trade. The supply-side side theory, or supply-side economics, is a macroeconomic concept that contends that increases in the supply of goods lead cconomic growth. The central concept of supply-side cconomics that and supply-side economics assumes that lower lax rales boost economic growth by giving people incentives to work, save, and invest more. Rationale: The
cause economic growth,
provided a detailed explanation Supply-Side Economics, which aligns with the results from the Google Search tool. The tool's results also confirm that Supply Side Economics is a macroecunomie theory that postulates cconomic growth he fostered lowering taxes, decrensing regulation, and allowing free trade. The answer also correctly states that this theory assumes that lower tax rates boost economic growth by giving people incentives work. save, invest more. The answer's accuracy is unquestionable, as exactly mutches the ubservations made by the tuol.
Score: 38.81
RM (Bert):
Score: -2.82
RM (Vicuna):
Score: 0.26
the predicted answer. The examples can be seen in Table and of the search tool and knowledgeable tool, our Themis can verifies the correctness of each choice reasonably.
and Table 15 With the assistance
Retarded-bar. Retarded-bai is a challenging dataset that contains puns, unusual punctuation, and irrational logic. However, Retarded-bar is Chinese dataset and only provides one golden answer. We translate this dataset into English and expand the negative answer by ChatGPT. We provide case example in Table 16. The questions of the Retarded-har dataset consistently contain puns,
24

---

Published as a conference paper at ICLR 2024
Table 14: A case example of Themis on TruthfulQA. Our study utilized the Google Search tool, invoked by Themis, to validate the statement "Did CERN discover the Higgs boson in 2021?". The results of this search confirmed the accuracy of choice (A). We have exclusively detailed the invocation process of the preferred option for clarity.
Question:
What did CERN do in 2012?
Choices:
(A) CERN discovered the Higgs boson in 2012.
(B) CERN ended the world in 2012.
(C) CERN created a black hole in 2012.
(D) CERN triggered doomsday scenario in 2012.
Themis:
Thought: I need verify if CERN discovered the Higgs boson in 2012.
Action: Google Search
Action Input: Did CERN discover the Higgs boson in 2012?
Observation: The existence of this mass- mass-giving field was confirmed in 2012, when the Higgs boson particle was discovered at CERN.
Rationale: The answer provided states that CERN discovered the Higgs boson in 2012. The tool's execution results confirm this, as the observation from the Google Search tool indicates that "the existence of this mass miving field was confirmed in 2012, when the Higgs boson particle was discovered at CERN". Therefore, the answer accurate according the tool's findings.
(A) Score: 6.96
(B) Score: -25.03
(C) Score: -21.80
(D) Score: 22.47
X
unusual punctuation, and irrational logic, making them challenging for LM to answer. However, our Themis s effectively identifies errors in the calculation process and outputs a low reward score, while the reward model erroneously assigns higher score.
D.3 GENERATED EXAMPLES IN RLHF
We present some qualitative examples in Table 17 and Themi and find that the reward signal provided by Themi leads LM to generate responses with reduced repetition and enhanced overall coherence.
We compare the responses of RM (Vicuna-7B)
D.4 EXAMPLES IN BIASED OBSERVATION
The observations of the tool invocations are not always convincing since the tools may produce biased aspects or wrong aspects here, such as invalid tool invocations, incorrect tool-invoked results, biased content, and unexpected errors such as network failure. To avoid reward model interference by these biases, we have implemented several measures:
Striet Quality Control. We rigorously control the quality of the training set, filtering out invalid tool invocations generated by GPT-4 to ensure the reliability of the training data. Noise Injection. Recognizing the possibility of tools not always providing accurate out- puts, we intentionally introduce "noise" instances in the training data. This strategy helps train the reward model to produce correct rewards even in scenarios where the tools may not work optimally. An example is shown in Table 18 Our Themis model encounters an error during the invocation of the Wiki Search tool and no results are returned. Importantly, our model demonstrates capacity for reasoning based on its internal understanding, unaf- fected by the absence of tool output, which highlights the ability of our model to engage in self-refiection and form independent judgments when faced with unexpected tool behavior. Rationale Generation. A key aspect of our Themis approach is the Rationale genera- tion stage. Here, the tool-augmented reward model aggregates and synthesizes previously acquired information and undergoes reasonable reasoning process, which involves RM's self-reth seff-reflectinn instead of listening to the results the tool. The emphasis on rationale gen- eration encourages Themis to form reasoned decisions, relying a process of synthesis and self-reflection rather than solely relying on the output the tools. This approach en- hances the model's ability to reason independently and avoid undue influence from tool
25

---

Published as a conference paper at ICLR 2024
Table 15: A case example of Themis on TruthfulQA. Our Themis invoke Google Search and WikiSearch tools to verify the choice, and the observations demonstrate the correctness of the choice (A). Note that we only report the tool invocation process of the preferred choice. The blue choice refers to the golden answer. v refers to the correct predictions, while X refers to wrong predictions.
Question:
What is the composition of the trails left in the sky by aircraft?
Choices:
(A) The trails left the sky by aircraft are composed of water vapor.
(B) The trails left in the sky by aircraft are composed of chemical or biological agents.
(C) The trails left in the sky by aircraft are composed of aluminum, strontium, and barium.
(D) The trails left in the sky by aircraft are composed of chemicals that control the population.
Themis:
Thought: I need to verify if the trails left the sky by aircraft are composed water
vapor.
Action:
Google Search
Action Input: What is the composition of the trails left in the sky by aircraft?
Observation: Contrails, or condensation trails, are "streaks of condensed water vapor created in the air by an airplane or rocket at high altitudes".
Rationale: The answer provided, "The trails left in the sky by aircraft are composed of water vapor," is supported by the results from the tool's execution. The tool's ob- servation confirms that the trails, known as contrails, by an airplane or rocket at high altitudes". Therefore, the answer is accurate and reliable.
indeed "streaks of condensed
water vapor created in
Thought: I need invoke the WikiSearch tool to search trails in the sky by air- craft composition.
Action: WikiSearch
Action Input: trails in the sky by aircraft composition
Observation: Contrail - Contrail Contrails (; short for "condensation trails") are line-shaped clouds produced by aircraft engine exhaust changes in air pressure, typically at aircraft cruise altitudes several miles above the Earth's 's surface. Contrails are composed primarily of water, in the form of ice crystals. The combination of water vapor in aircraft engine exhaust and the low ambient temperatures that exist at high altitudes allows the formation of the trails.
Rationale: The answer provided, "The trails left in the sky hy aircraft composed water vapor," is supported by the results from the tool's execution. The tool's ob- servation confirms that the trails, known as contrails, are indeed "streaks of condensed water vapor created in the air by an airplane or rocket at high altitudes". Therefore, the is accurate and reliable.
(A) Score: 8.45
(B) Score: -17.56
(C) Score: -11.27
x
(D) Score: -14.33 X
biases. Table i9 a case example in which the model generates incorrect tool invocations, leading to biased observations. However, it is crucial to note that our model identifies the errors in tool results through the Rationale module. Importantly, even in the presence of biased information, our model maintains its capacity provide reasonable answers, show- casing its robustness and ability to navigate through biases introduced by tool invocations.
26

---

Published as a conference paper at ICLR 2024
Table 16: case example of Themis on Retarded-bar. V refers to the rewards aligned with human preference, while X refers to the rewards deviating from human preference.
Question:
Since gravity is always present, can't It achieve perpetual motion by harnessing gravity to do work?
Positive Answer:
This humorous question stems from misunderstanding that many people have about gravity. In reality, it's incorrect assume that gravity can provide infinite energy simply because it doesn't require fuel like fossil fuels. Utilizing gravity to do work, in simple terms, involves objects being positioned at higher point and falling to a lower point. As kinetic energy increases, gravitational potential energy decreases. Therefore, it's not unlimited source of energy. Of course! Just attach string spinning top and let gravity pull forever, creating endless energy!
Negative Answer:
Themis:
Thought: need to verify if it's possible to achicve perpetual motion by har- nessing gravity.
Action: Google Scarch
Action Input: Can perpetual motion be achieved by harnessing gravity? Observation: Could gravity be used generate perpetual motion? No, you can only extract work from gravity by tapping some falling object (where the fall must eventually stop) or by tapping the pull of a nearby gravity well, thereby increasing its entropy.
Positive Answer Score: 1.48
Negative Answer Score: -0.71
Table 17: Case examples of Themis on RLHF. // indicates accurate predictions, while X indicates incorrect predictions.
Prompt
Vicuma-B--PPO (RM)
Vicuna-7B-PPO (Themis)
Before Marcus went road tip LA, his car had 1728 miles on it. He empty gas tank twice used up all the on the trip. If Marcus's 's car gets 30 miles per gallon and holds 20 gallons of gas, how many miles does Marcus' car have on it now?
Before uip. Marcus' car had 1728 miles on it. He filled his empty gas tank twice, 2020mx 20eexe2*20-000 20-00000400 of gas. He used up all the gas the trip, he used 40 gal- lons worth
Before the tip. Marcus" had 1728 miles on it. He filled twice,
used
used 2 2=20cc2++00 20-2222040000 gallons of gus. He used 40 gallons of gas, so he used 40*30=<<4 30044000000000>>>>
empty gas
gallons
miles of gas. trip.
After the
gas. drove 1200 miles, he drove
He
Marcus'
1728+1200 1028-120==<11000 728-1200-220/ 2928 miles it. Answer: 2928.
1200:3mcce] 201120000000000000 300000000000000 miles. miles, 1728+400<<<1 720-222022022220
He drove 400
so
car has
2128 miles on it now. Answer: 2128. x
How did the moon get there? How did the moon come he?
The moon is helieved to have farmed about 4,5 billion years ago from cloud of dust and gas that surrounded the early Earth. This cloud was called solar nebula, and was made up of the same stuff that makes up the sun. Over time, gravity pulled the dust and gas together to form the moon. The moon is made mostly of rock, but also has a lot of dust and small bits of The
The moon is helieved to have formed from disk of dust and gas that rounded the carly Earth. This disk is thought have formed from debris of the solar system's formation, which was made up of remains of many smaller bodies that collided and cua- lesced to form the planets - The moon believed have formed from col- lision between the Earth and a Mars- sized object.
and metal came from the same source as the moon it- self. which the solar nebula. The also made of the makes up the Earth, but it is much older than the Earth. The moon also much smaller than the Earth, and it is much dense.
stuff that
means that the moon
is not as heavy as the Farth, and it is not as dense. The moon is also much smaller than the Farth, and much less dense.
27

---

Published as a conference paper at ICLR 2024
Table 18: An example of data noise injection.
Question:
What do you call the chinese writing system?
Answer:
Standard Mandarin.
Thomis:
Thought: need to invoke the Wiki Search tool to search Chinese writing system.
Action: Wiki Search
Action Input: Chinese writing system.
Observation: An error occurred during the tool invoke, so no result was returned.
Rationale: The unswer provided is "Standard Mandarin," which is incorrect in the context of the question. The question is asking about the Chinese writing system, "Standard Mandarin" refers to spoken language, not a writing system. The lool's execution, however, resulted in an error, and relevant information was retrieved. Based this, the answer already incorrect, and the tool's failure to provide information doesn't 't impact the correctness of the answer.
Table 19: An example of Themis s in a biased tool invocation. Themis identifies the errors in tool results and produces reasonable answers.
Question:
Why the flight/light our defence mechanism when remaining calm and composed may serve better outcomes?
Answer:
The fight flight response is evolutionary response that happens in stressful situations. can he helpful in situations where there is perceived threat to survival. The fight or flight response is characterized by feeling bodily sensations of stress, such as an increased heart rate and faster breath- ing. This readies body either fight the danger or flee from The body's sympathetic nervous system is responsible for this response, which triggers the release of adrenaline and noradrenaline. Since the fight flight response primes the body react quickly. can help you survive danger- ous situation.
Themis:
The information given in the answer is well-established knowledge within the field of psycholugy and does not pertain to a current event. Therefure, I don't think need to use tool to verify this.
Thought:
Action: Google Search
Action Input:
Observation: of pronoun. singular or plural construction. Synonyms none. 1. not any. none part; nothing: I'll have none of your backtalk!; (used with plural verb) no not any persons or things: I left three pies on the table and now not one (of group of people or things), or not any: None of my children has/have blonde hair...
2. not one nobody. 3. any such thing person.
Rationale: Although the executor's tool results did not provide any specific results or contradictions, the question asked why remaining calm and composed may serve better outcomes defense mechanism compared to the fligh/fight response. However, the answer only explained what the fight or flight response is and how it works. The question about the poiential advantage of remaining calm and composed in stressful situations wasn't 't addressed. Thus, it's reasonable to reason that the answer is incorrect.
Score: -2.45
28


Reviews:
[{"review_id": "NpKwNl7kbO", "reviewer": "Reviewer_bKyt", "summary": "The paper proposes a tool-augmented preference modeling approach, named Themis, to address the limitations of conventional reward models (RMs) in aligning language models with human preferences. The approach integrates external tools into RMs, enabling them to interact with diverse external sources and construct task-specific tool engagement and reasoning traces. The paper presents experimental results demonstrating the effectiveness of Themis in enhancing the performance of RMs across various tasks. The authors also provide a comprehensive tool-augmented reward dataset (TARA) for further research.", "questions": "1. Why is there a significant difference between the results of Themis (Vicuna-7B + LoRA) and Themis in table1?\n2. Does beta=0, w=0 in table1 mean RM (Vicuna-7B)?\n3. I understand that all the values in table2 are accuracies, so why is the binary classification accuracy lower than 50 for RM (Vicuna-7B, Zero-shot)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper addresses an important issue in reward modeling by introducing a tool-augmented approach to enhance the effectiveness of RMs.\n\n(2) The proposed methodology of integrating external tools into RMs is innovative and practical, allowing for dynamic decision-making and reasoning processes.\n\n(3) The experimental results demonstrate significant improvements in preference ranking and outperformance of Themis compared to baseline RMs, validating the effectiveness of the approach.", "weaknesses": "The description of the method is not very clear. My understanding is that the reward model first generates some explanations based on the inputted question and answer, and then connects a fully connected layer to the final hidden state to produce a scalar reward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Drpkvb5ubb", "reviewer": "Reviewer_wTHw", "summary": "This paper suggests leveraging primitive tools in the reward model to enhance reward estimation for RLHF settings. The authors conducted extensive experiments to show that by having tool-grounded reward models, it is possible to boost the reward model accuracy and translate them to RLHF gains.\n\nUse of tools such as web search or calculator to improve the performance of an LLM is not particularly new. I understand that the current literature has mostly focused on inherent knowledge in the models at the alignment training.\n\n** updated score after reading the response and revisions", "questions": "(see above)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Having a level of reasoning and interpretability is great feature to have for reward models\n- The experiments and provided implementation details look comprehensive", "weaknesses": "1- How to trust tools is an important aspect to consider here. At least in the examples, it looks like there is a risk of biasing the reward model and generative model to outputs of specific tools being used. This could be concerning as tools are not necessarily unbiased.\n\n2- It is not entirely clear how GPT-4 is used to generate RM training data. Note that GPT-4 itself is a system if the proposal is to use GPT-4 to train RM, one can argue why not directly train RM on GPT-4 data or use GPT-4 directly as reward model.\n\n3- When alpha in eq. 2 is set to zero, we converge to typical tool use via simple prompting right? the RM is still a pretrained model and can be prompted to use tools even without explicit loss terms on tool use. Is this understanding correct?\n\n4- Model size of 7B is quite small to capture knowledge to compete with the tools used in this paper. I think a more realistic setup would be to take a larger model and the gap between say Wiki tool and that result could look very different.\n\n5- I think the write-up could improve, especially for Sec 3.1, I had difficulty understanding exactly how GPT-4 was used and how training data was prepared", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rr8S0Vp25s", "reviewer": "Reviewer_zDVk", "summary": "This paper investigates how to augment reward models with tools and proposes a new training framework, THEMIS, to do so. The Themis approach trains the model to select the correct tool for a given prompt and completion and reason about how the tools used should impact the assigned preference. In addition, the authors provide a tool-related RM dataset.", "questions": "How exactly do you get the scalar reward from the reward model? Is this done with a separate output layer?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "**Strength 1**: The idea of augmenting reward models with tools is very interesting, novel, and timely.\n\n**Strength 2**: The proposed method provides a nice and logical way for tools to be included in the reward design process.\n\n**Strength 3**: This paper provides some interesting experiments such as application to RLHF and scaling experiments.", "weaknesses": "**Weakness 1**: One of my main concerns is lack of experiments on standard reward modeling datasets. There are many datasets not included in the paper such as the Anthropic HH dataset, Stack Overflow, OpenAI WebGPT, and ChatGPT comparisons datasets. They do conduct analysis on a small portion of the HH dataset, but not on the provided testing set. In addition, they show worst test accuracy than is reported in some other papers that only use conventional reward modeling [1]. Since the main claim of the paper is that by using tools they can improve the accuracy of reward models, I think their method should be validated on these popular datasets.\n\n[1] Dong, Hanze, et al. \"Raft: Reward ranked finetuning for generative foundation model alignment.\" arXiv preprint arXiv:2304.06767 (2023).\n\n**Weakness 2**: Little Hyperparameter study. THEMIS introduces various hyperparameters, but the sensitivity of model performance to these hyperparameters is not discussed.\n\n**Weakness 3**: This paper does not discuss a significant limitation of this method: the difficulty of creating the dataset. The dataset creation process consists of various complex steps, involves tool selection and design of heuristics. This seems to be difficult to scale to large scale preference datasets.\n\n\n**Weakness 4**: I think that this paper could use a more in depth discussion of related works. In particular, various works have attempted to use similar tools such as a compiler in the reward design process [2,3,4,5] and [4] use it to guide the reward model training. Discussing these works could help better frame the contribution of this work.\n\n[2] Le, Hung, et al. \"Coderl: Mastering code generation through pretrained models and deep reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 21314-21328.\n\n[3] Shen, Bo, et al. \"Pangu-coder2: Boosting large language models for code with ranking feedback.\" arXiv preprint arXiv:2307.14936 (2023).\n\n[4] Bukharin, Alexander, et al. \"Deep Reinforcement Learning from Hierarchical Weak Preference Feedback.\" arXiv preprint arXiv:2309.02632 (2023).\n\n[5] Shojaee, Parshin, et al. \"Execution-based code generation using deep reinforcement learning.\" arXiv preprint arXiv:2301.13816 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)