MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
Authors: Xingyao Wang
Zihan Wang
Jiateng Liu
Yangyi Chen
Lifan Yuan
Hao Peng
Heng Ji
Venue: ICLR 2024 poster (2024)
Keywords: large language model
multi-turn interaction
learning from feedback
reinforcement learning from human feedback
instruction tuning
Primary area: datasets and benchmarks

Abstract: To solve complex tasks, large language models (LLMs) often require multiple rounds of interactions with the user, sometimes assisted by external tools.
However, current evaluation protocols often emphasize benchmark performance with single-turn exchanges, neglecting the nuanced interactions among the user, LLMs, and external tools, while also underestimating the importance of natural language feedback from users. These oversights contribute to discrepancies between research benchmark evaluations and real-world use cases.
We introduce MINT, a benchmark that evaluates LLMs' ability to solve tasks with multi-turn interactions by (1) using tools and (2) leveraging natural language feedback.
To ensure reproducibility, we provide an evaluation framework where LLMs can access tools by executing Python code and receive users' natural language feedback simulated by GPT-4.
We repurpose a diverse set of established evaluation datasets focusing on reasoning, coding, and decision-making and carefully curate them into a compact subset for efficient evaluation.
Our analysis of 20 open- and closed-source LLMs offers intriguing findings.
(a) LLMs generally benefit from tools and language feedback, with performance gains (absolute, same below) of 1--8% for each turn of tool use and 2--17% with natural language feedback.
(b) Better single-turn performance does not guarantee better multi-turn performance.
(c) Surprisingly, on the LLMs evaluated, supervised instruction-finetuning (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities.
We expect MINT can help measure progress and incentivize research in improving LLMs' capabilities in multi-turn interactions, especially for open-source communities where multi-turn human evaluation can be less accessible compared to commercial LLMs with a larger user base.

Published as a conference paper at ICLR 2024
MINT: EVALUATING LLMs IN MULTI-TURN INTER- ACTION WITH TOOLS AND LANGUAGE FEEDBACK
Xingyao Wang', ; Zihan Wang 12- Heng Ji
" .Jiatnng Liu', Yangyi Chen', Lifan Yuan', ' Hao Peng',
University of Illinois Urbana-Channe Champa-C 2 Renmin University of China
{xingyao6, 21 zihanw, jiateng5, yangyic3, haopeng, heng @ nois edu
ABSTRACT
To solve complex tasks, large language models (LLMs) often require multiple rounds of interactions with the user, sometimes assisted by external tools. How- ever, current evaluation protocols often emphasize benchmark performance with single-turn exchanges, neglecting the nuanced interactions among the user, LLMs, and external tools, while also underestimating the importance of natural language feedback from users. These oversights contribute to discrepancies between re- search benchmark evaluations and real-world use cases. We introduce MINT, a benchmark that evaluates LLMs' ability to solve challenging tasks with multi-turn interactions by (1) using tools and (2) leveraging natural language feedback. To ensure reproducibility, we provide an evaluation framework where LLMs can ac- cess tools by executing Python code and receive users' natural language feedback simulated by GPT-4. We repurpose diverse set of established evaluation datasets focusing on reasoning, coding, and decision-making and carefully curate them into compact subset for efficient evaluation. Our analysis of 20 open- and closed- source LLMs offers intriguing findings. (a) LLMs generally benefit from tools and language feedback, with performance gains (absolute, same below) of 1-8% for each turn of tool use and 2-17% with natural language feedback. (b) Better single- turn performance does not guarantee better multi-turn performance. (c) Surpris- ingly, among the evaluated LLMs, supervised instruction- finetuniiiinnnn (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities. We expect MINT can help measure progress and incentivize research in improving LLMs' capabilities in multi-tune turn interactions, especially for open- source communities where multi-turn human evaluation can be less accessible compared to commercial LLMs with a larger user base."
INTRODUCTION
To address complex tasks, Large Language Models (LLMs) often need multiple rounds of inter- action with the users, sometimes aided by external tools (Schick al., 2023b; CharGPT Plugins; Mialon et al., 2023). LLMs' performance during multiple turns of user-LLM exchanges is crucial in real-world applications: roughly 73% of Caman-ChatGPP conversations contain more than one turn based on 94k entries of ShareGPT data (2023). Meanwhile, the ability to adapt to user-provided nat- ural language feedback also pivotal for their practical utility. However, current LLM evaluations predominantly focus on single-turn input-output (Hendrycks et al., 2020; Chen et al., 2021) and of- ten overlook user-provided natural language feedback (Liu et al., 2023d; Deng et al., 2023b; Yang et al., 2023a; Shridhar et al., 2020), creating a discrepancy between real-world use cases and evalua- tion. Measuring how much LLMs can benefit from both tools and natural language feedback during multi-turn interaction is essential to incentivize future research improve LLMs' capabilities in a broader range of real-world scenarios.
To bridge these gaps, we introduce MINT. is benchmark for LLMs that measures their per- formance during multi-turn interaction, focusing on two particular capabilities ($2.1): (1) tool-
"Equal contribution. Work done during internship at UIUC.
Code available our project website: https:/ //xi ngyaoww, gi tit.... io/mi -enchh

---

Published as a conference paper at ICLR 2024
augmented task-solving; (2) leveraging natural language feedback. MINT mirrors the world Beer-LLM-too collaborative cooblem-solving setting. To solve a problem, the LLM can use external tools by generating and executing Python programs (Wang et al., 2024) and/or collecting natural lan- guage feedback to refine its solutions; the feedback is provided by GPT-4 (OpenAI, 2023), aiming to simulate human users in reproducible and scalable way. For comprehensive evaluation, we include cight established datasets spanning reasoning, code generation, and decision-making ($2.2). To facilitate affordable multi-turn evaluation, after collecting 29,307 diverse instances from existing datasets (Tab. 1), we construct a subset of 586 challenging and representative instances that require multi-turn interaction to solve".
We evaluate 4 closed- and 16 open-source LLMs with MINT. We measure LLMs' tool-augmented task-solving capability by analyzing their performance from multi-turn tool use ($3.2). To assess the ability to leverage natural language feedback, we measure their performance upon natural language feedback by GPT-4 ($3.3). Our results show that:
All models benefit from tool interaction and natural language feedback, with absolute performance gains by 1-8% for each additional turn tool use, and 2-17% with natural language feedback. Better single-turn performance does nol necessarily entail better multi-turn performance. For example, while Claude-2 outperforms its predecessor Claude-1 in single-turn evaluation, the latter benefit more from interaction and performs better with turns.
There is a notable gap between open- and closed-source LLMs in multi-turn interaction per- formance. For example, with the help of language feedback, even the best open-source model, 7ommu--0b-chat-v1, lags behind the best closed-source model by 8.7% in absolute success rate. On most LLMs we evaluated, models trained with supervised instruction fine-tuning (SIFT, Wei et al., 2022) and reinforcement learning from human feedback (RLHE, Ouyang et al., 2022a) per- form worse in multi-turn settings regardless of the presence of language feedback. For example, SIFT hurts Codellama- -34''s 's multi-turn performance by 11.1% and 15.4% (w/ feedback), and RLHF negatively affects I.LMMA--70FF by 8.5% and 8.7% respectively. Notable exceptions are Vicuna-7B and Lemur-70b-cha-- where SIFT improves multi-turn interaction.
By fixing the LLM evaluate and changing the feedback L.M,, MINT can measure dif- ferent LLMs' capabilities in providing useful feedback ($3.4); We find that feedback-providing ability could be orthogonal to task-solving ability: despite performing the worst in task-solving, provide feedback to improve the stronger GPT-3.5. Additionally, MINT's challenging evaluation reveals undesired artifacts in ShareGPT data (2023), a widely used dataset for instruction tuingg ($3.5). Furthermore, we show that GPT4-simulated language feedback is as helpful as human-written feedback based on human evaluation and task performance ($3.6).
CodeLLaMA-33B-IIeg
We expect that MINT can help track progress and incentivize future research in improving LLM's multi-turn task-solving and/or feedback-providing capabilities, especially for open-source commu- nities where human evaluation can be less accessible than commercial LLMs with large user base.
2
MINT
In this section, we discuss (1) how to evaluate multi-turn interaction ($2.1) with tool use and lan- guage feedback under different differentsettinggs settings; (2) how we repurpose existing datasets for MINT evaluation ($2.2). We use Fig. 1 as running example.
2.1 INTERACTION FRAMEWORK
MINT aims simulate real-world applications of LLMs, emphasizing user-LLM and LLM-tool interactions. In user-LLM collaborative problem-solving. a human user provides initial instruction and aims to obtain correct solution with little effort to help the LLM. On the other hand, augment- ing LLMs' with tools can effectively improve LLMs' task-solving capabilities (Mialon et al., 2023), suggesting the importance of LLM-tool interaction. We instruct the LLM ($F.4.1) to perform the
We use gpt- 4- 00613 version in this work.
"Evaluating an LLM using MINT costs around 100 USD (z 3M prompt tokens and ~ 100K completion tokens) with feedback from gpt-4-00 0-00-440611 1.3 (S0.03/1K K prompt tokens and $0.06/1 K completion tokens), roughly 7% of the cost compared to hiring real-human annotators ($3.6).
2

---

Published as a conference paper at ICLR 2024
User starts Interaction
User (optional, LLM-simulated)
Interaction Turn #1
This GOOD, [..]] Now you can use this information to find how
User
many insec:s Lily found.
Together Lily, David, and Bodhi collected 43 insects. Lily found more than David. found what Bodhi found. How many insects did Lily find? Solution output format: an integer.
LLM
Thought: Now that we know that Bodhi found 16 insects [..] Lity found than David, Lily found 18 insects.
LLM
Execute: lily, insects solution[e] lily_insects
Thought: To salve this problem, let's start by setting up equations (...
assume that David found insects, Based given information, Lily found than
Interaction Turn #4
Lily found insects [..]
Executo: fron sympy "ipoort synbols, EQ, solve
Python interpreter
X, b symbols('x b()
Observation: CJt] 25.00000003000000
set up the equation
You have steps left and chances propose solution left.
equation Eq(x (x
(1/2) 43)
solve the equation
User (optional, LLM-simulated)
solution solve loquatetniioon, x)
This NOT GOOD. You made mistake in calculating the number of
[o]
insects Lily found. should have calculated
David found first, which
what Bachi found [.]
Interaction Turn #2
................................
Python Interpreter
Thought: (Apologize ...J Badhi found insects [..) David found half of what Bodhi found, so David found 11/2) 18 insects. Lily found more than David, so Lily found (1/2 18 Insects.
Observation: Dut(1]: 0.25*D
You have 4 steps left and 2 chances to propese solution left.
User (optional, LLM-simulated)
Execute: insects (1/2) solution[o]
This NOT 600D. [..] You should have used the information that
lily_ insycts david_inseccts lily_insects
David found half of what Bodhi found [..]. Than, you can express the Bodhi's insects and salve for [..]
total insects
Interaction Turn #5
LLM
Thought: [Apologiza ...] Bachi found insects. David found half of what Bodhi found, so David found (1/2)* *b insects [-]
Python Interpreter
Observation: Oui[4]: 16 18,0000000000000
Execute: fron synpy
synbols, Eq, solve
You have stops left and ? chances to propose solution left.
Lymbals("""))
You should take last step propose salution.
equation Ea((1/2) b (1/2) b b, 43)
set up the equation
User (optional, LLM-simulated)
solve equation
OOOD. You have correctty calculated number insects Lily' found. Now you can propose solution...
solution solve (equattin, b) so solutionie]
LLM
Interaction Turn
Propose Solution: The number insects Lily found is 16.
Python Interpreter
- User
Observation: Out[2]: 18.0030000000000
You have 3 steps left and 2 chances to propose solution left.
Good You have successfully solved task! (stops interaction]
An interaction trajectory produced by evaluating gpt-3. 3e0t-3.5tturree 5--tt--o-060111 0613 with MINT on mathematical reasoning task, The evaluated model's outputs are in the blue boxes, and the feedback by gpt 4 4-0613 in red, dotted ones. Some details are omitted for clarity.
Figure 1:
following steps in each turn: (1) optionally express its reasoning process ("Thought:" in Fig. 1, sim- ilar to Yao et al. 2022); (2) then either interact with tools by generating Python code and executing it through a Python interpreter ("Fxecute:" in Fig. 1), or proposing solution to the user ("Propose Solution:" " in Fig. 1). We adopt code as unified tool interface due to flexibility and performance, as demonstrated by Wang et al. (2024). In our implementation, the model is instructed to wrap their "Execute" and "Propose Solution" actions with pairs of <execute> and <solution ion> tags for ease of parsing. We standardize the prompts and in-context examples for different LLM vari- ants (base vs. chat) and for tosk-solving and feedback providing, aiming for fair and reproducible comparisons (Appendix $F4.11 $F4.2, and $F.5). In what follows, we introduce three settings with increased interaction complexities to measure different aspects of multi-turn interaction.
LLMs interacting with lazy user. We consider the scenario where a user provides initial instruction and makes minimal efforts to guide the LLM toward the final solution. This setting covers real-world problems that are "hard solve directly, but easy verify": the user may not know how to solve the problem or understand the reasoning process, but it is still possible for the user to provide minimal feedback on the final outcome (e.g., "this is not what I want"). This will serve as a baseline for evaluating LLM's ability to solve tool-augmented tasks and leverage natural language feedback. The LLM given two attempts to propose solutions for each problem, with a limit on the number of interaction turns & ($3.1). Upon proposed solution, MINT simulates users that check the solution's 's correctness with ground truths. When the first attempt is wrong, the user responds "Your answer is wrong." The interaction ends either after the LLM has made two attempts to propose a solution, or when a proposed solution is correct (5th turn of Fig. 1), or when the k-th turn of interaction is reached. We consider this the case of Lazy User-LLM Interaction since the simulated user provides as most one additional binary feedback during interaction. We follow standard evaluation practice and use established evaluation metrics for each task in $2.2.
LLMs interacting with a lazy user and tools. Under the lazy LLM interaction setting, we measure the L.M''s ability solve tasks using tools by comparing their task task-solving success rate
3

---

Published as a conference paper at ICLR 2024
Table 1: Dataset statistics of re-purposed data instances from existing datasets into MINT. We filter and down-sample existing datasets to construct compact set of complex tasks that require multi- turn interaction to solve ($2.2).
Task Type
Task Name
Original Size
Reduced Size in MINT
HumanEival (Chen al., 2021) MBPP (Austin al., 2021)
164
45
Code Generation
500
91
Decision Making
ALFWorld (Shridhar et al., 2020)
134
134
GSM8K (Cohbe et al., 2021)
1319
HotpotQA ( (Yang et al., 2018)
7,405
43
Reasoning
MATH (Hendrycks et al., 2021)
5,000
100
MMLU (Hendrycks et al., 2020)
13,985
76
TheoremQA (Chen al., 2023a)
800
49
Total
29,307
586
across different interaction limits k. For each turn, the LLM can choose to interact with tools (gener- code to call equation-solver in Fig. or propose solution (5th turn in Fig. 1). To keep the LLM from getting stuck in an infinite loop of tool-calling without proposing a solution, MINT reminds the LLM: "You have X steps left and chances to propose solution left," and provides an addi- tional instruction at the last turn: "You should take the last step to propose solution." Intuitively, with more interaction with tools, the LLM can get more useful observations through the Python interpreter (e.g., calculation results, error messages). We vary k {1,2, 3, 4, 5) and compare the models' success rate with each k. We consider LLM's performance gain w.r.t. and the absolute their tool-augmented task-solving ability ($3.2).
performance at A
Informative user-LLM interaction with language feedback. Beyond lazy User-LLM interaction, investigate how the LLM performs when the user mirrors patient teacher who provides useful suggestions (e.g., technical users who understand the wooliigg process). However, collecting human language feedback for LLM evaluation presents reproducibility challenges due inconsis- tent standards and can be costly, particularly for open-source communities with relatively fewer resources". To address these issues, prompt GPT-4 ($F4.2) to simulate user language feedback (dotted boxes in Fig. I). We validate the effectiveness of GPT-4 feedback in human evaluation ($3.6). We compare the performance between (1) simulated language feedback and (2) lazy user- LLM interaction, both in the setting of tool-augmented interaction with an interaction limit h = 5. We consider performance (absolute) and improvements from language feedback as LLM's s ability to leverage natural language feedback.
2.2 REPURPOSING EXISTING DATASETS FOR MINT
Evaluating I.MSs in multi-turn interaction can be costly due the need for iterative inference. For instance, HotpotQA (Yang et al., 2018) has 7,405 test examples. Evaluation with five turns requires least 7,405 5 37K LLM inference runs. Previous methods (Yao al., 2022; Shinn al., 2023) choose to evaluate on randomly drawn test examples, increasing the barriers to fair comparisons. We select diverse tasks from established datasets that requires multi-turn interaction to solve while also maintaining the selected subset compact for accessible evaluation. The following paragraph describes our three-step approach to repurposing datasets for MINT. We provide dataset sources and statistics in Tab. 1. For more details, please refer to $D in Appendix.
Data sources of MINT. Our primary goal to create comprehensive evaluation covering tasks that benefit from interaction. We choose three types of tasks:
Reasoning, including math reasoning (GSM8K, MATH, TheoremQA), multi-hop question answering (HotpotQA), and knowledge problem-solving (MMLU). We implicitly filter out knowledge-intensive questions that do not require multi-step reasoning in the next step.
Code generation, including HumanEval and MBPP.
Decision-making tasks ALFWorld, an embodied household simulator with a text-only interface based on WexlWorld (Côté et al., 2018).
'Based on our human evaluation ($3.6, SB), estimate annotators, average, take 96 seconds to provide language feedback for one turn, which translates 90 USD per 100 feedback with hourly wage of US workers.
4

---

Published as a conference paper at ICLR 2024
From eight datasets, create an initial test set of 29,307 instances. All instances are initially designed for single-round evaluation without interaction, except for decision-making (ALFWorld). Similarly to Yao et al. (2022); Gao et al. (2023), we adapt reasoning tasks into multi-tury interaction tasks by augmented LLM with tools for problem-solving ($3...). Through prompting ($F.5.2, we encourage L.MMs to the Python interpreter test their generated code the provided public lest suite for code generation problems before committing to a solution.
Keeping instances that require multi-turn interaction. To better answer our research question "how LLM benefits from multi-tum interaction," we only keep instances that are challenging and require multi-turn interaction. Since we allow LLM to propose solutions more than once, we filter out instances that a random guess baseline can do well, c.g., multiple-choice instances with < 4 options. We then run ept-3. www.:--ttubb----- 0613 (OpenAl API) on the initial dataset and exclude instances finished within two turns (e.g., easy problems that can be solved without multi-turn).
Stratified sub-sampling for efficient evaluation. We use stratified sampling (Neyman, 1992) to create a representative set of 586 examples, ensuring that the ratio of correct to incorrect examples in the resulting set mirrors that the original data to balance the difficulty of the resulting samples.
3 EXPERIMENTS
3.1 SETUP
Evaluated LLMs. To comprehensively measure multi-turn interaction capability and identify the potential gap between open- and closed-source LLMs, we evaluate closed- and 16 open-source LLMs. We cover different sizes and training techniques to better understand how they affect LLMs' multi-turn interaction capability. Training techniques lead to three model variants: pre- trained (base) models, supervised instruction fine-tuned (SIFT, Wei et al., 2022) models, and mod- els trained with reinforcement learning from human feedback (RLHF, Ouyang et al., 2022a). For closed-source models, evaluate popular commercial LLMs, including 3. 3.5-turbo-0061111 from OpenAl API; cl sande-instant-11 chat-bi son-001 from Bard API. For open-source LLMs,, evaluate the LLaMA-2 model family (7B, 13B, 70B) (Touvron et al., 2023), including base and chat (RLHF); Vi wwna-vl--lle.. (7B, 13B) (Zheng et al., 2023), a SIFT model fine-tuned on multi-turn conversations based on LLAMA-2-base: the LaMA model family (7B, 13B, 34B) (Rozière et al., 2023) that pre- train LLaMA-2-base on code, including base and instruct (SIFT); Lemur-v1-77 (Xu et al., 2023) pre-train LLaMA-2 on code intensive data, including base and chat (SIFT).
cl aude-2 from Anthropic Claude APP; Bard
Metric. We consider Success Rate SR as our evaluation metric, which measures the percentage of successful task instances. For interaction limit k. we start from scratch and allow each LLM to interact up to the k-th turn and measure their corresponding S Rg. Unless otherwise noted, we limit A E [1, 5] where k 1 means no interaction and k 5 maximizes interaction turns within most modern LLMs' context window (4,096 tokens).
3.2 MEASURING LLM'S TOOL-AUGMENTED TASK-SOLVING IN MULTI-TURN INTERACTION
We ask LLMS to solve tasks ($2.2) with different interaction limits A E {1, 2, 3, 4,5) without natural language feedback (Fig. without red dotted box), and quantify LLMs' tool -ugmeened tosk-solving capability by (1) absolute performance SR; and (2) improvement per additional interaction turn Aboo estimated as the slope from least-square regression mins,a Σ. (b. 1 + a SRa)² (Tab. 2). Since the underlying SR: vs. A relationship might not be linear, we only use the regression coef- ficient (with R²) as a rough estimate of the improvement rate to complement the absolute success rate SR; for more comprehensive understanding of the models' capabilities.
Overall observations. In Fig. 2, we find all open-source models fall behind best commercial closed- source models in both SR₂ and Awols. with e claude-2 and cl aude-inst wwwn-iinnntttttnt surpassing all open-source LLMs in Aools with high R?, suggesting near- linerr improvement. Notably, despite surpasses claude -2 as k increases to 3, even-
performing badly at A 1, www.cllddisstttee
AAcorrding to https: http:///ooss.aarro...oooo anthropic cuwee/cce/mm/eaee/eeeeeee aedeeaaeeeeeeeeeeeeeee seeerrenneeeeeeeeeeee inleiiii---- aaiiin---a--eeeee a-mode1, we version v1.2 for cl audeeinstantee and v2.0 for cl 2.
5

---

Published as a conference paper at ICLR 2024
Table 2: Tool-augmented task-solving success rate with different interaction limit & (i.e., max num- ber of interaction turns allowed) and improvement rate (estimated with least-square regression coef- ficient, regression It is also included). The slope (i.e., coefficient) indicates the rate of improvement while R² denotes the goodness of fit of the regression model the data.
SR (Micro-averaged across tasks)
Improvement Rate
Models
Size
Type
==
A = 2
k ==
4
k:
Slope
Rt
Open-source LLM
Base"
0.3
4.1
7.2
7.2
4.3
+1.1
0.38
7B
SIFT
0.3
7.8
10.2
9.7
8.7
+1.9
0.53
Base
0.5
13.7
17.9
19.3
18.4
+4.1
0.70
13B
SIFT
1.5
12.6
13.1
15.0
14.5
+2.8
0.64
Base
0.2
16.2
23.0
25.9
28.2
+6.66
0.85
34B
SIFT*
2.6
10.1
14.7
15.4
17.1
+3344
0.86
Base
0.2
5.6
7.3
8.9
9.7
+22.2
0.87
7B
RLHF
1.0
4.3
6.7
6.5
7.3
+1.5
0.83
LLaMA-2
Base
0.2
11.4
15.5
15.2
14.5
+33.2
0.63
13B
RLHF
4.1
12.5
12.5
13.3
11.9
+-1.7
0.47
Base
1.9
19.4
24.6
26.4
26.4
+5.66
0.73
70B
RLHF
4.3
14.3
15.7
16.6
17.9
+3.0
0.73
Base
1.0
17.9
23.6
25.3
26.3
+55.8
0.77
Lemur-vi
SIFT
3.8
27.0
35.7
37.5
37.0
+7.77
0.73
7B
SIFT!
0.0
6.7
12.3
15.4
12.6
+3.4
0.77
Vicuna-v1.5
13B
SIFT
0.0
2.2
4.4
6.7
8.4
+2.1
1.00
Closed-source LLM
chat-bisoo-0011
0.3
15.9
14.2
13.0
14.5
+2.5
0.40
claude-2
26.4
35.5
36.0
39.8
39.9
+33.1
0.81
claude-instant-1
12.1
32.2
39.2
44.4
45.9
+8.00
0.84
gpt-3. gpt-4-0613
2.7
16.9
24.1
31.7
36.2
+88.2
0.96
69.5
Evaluated I.IM failed to produce parsable output instructed in some cases. See $3.5 and Tab. A.7 for details.
We identitied potential undesired artifacts in its training data, which hurt its performance. See $3.5 for details.
tually achieving a higher SRş (45.9% vs. 39.9%), suggesting ability to improve with multi-turn interaction.
superior
Absolute performance and improvemeentper--urr scale with model size. For open-source CodeLLaMA and LLaMA-2 2, we observe a trend on all variants (Base, SIFT, and RLHF) that Atools and SR; increase when scaling up LLMs. As we discuss in $3.5, Vicuna -v.. 5 models are an exception, potentially due to their training artifacts that hurt task performance.
SIFT on multi-turn data can be helpful. Despite the issue above, Vicuna-vl. 5 (7B, SIFT) does show
3pl-1-0611:
Pall, miree-ooroe
60
Cisccarn (466 MH11
stronger performance compared LLaMA-2 (Base and RLHF, 7B) Auols (+3.44 vs. +2.2% +1.5%) and SRG (12.6% vs. 9.7% 7.3%). Lemur-va (70B, SIFT) also shows stronger performance than its Base vari- ant. However, except CodeLLaMA (7B), we do not find similar improvements on LodeLLaMA (SIFT). We hy-
LLaMeu Mast
50
40
30
pothesize that the performance gain on Vicuna-v1.5 and Lemur-vl could be attributed to fine-tuning on ShareGPT's multi-turn human-ChatGPT conversations.
Internacioon Tamnk
With increasing interaction limit k, the success rate (%, micro aver- aged) improves at different ferent LLMs. For clarity, only subset of evaluated LLMs are visualized.
Figure 2:
RLHF could hurt LLM-tool multi-turn interaction. We find that on LLaMA- 2 series, RLHF alignment gen- erally hurts models' performance in both Anots (-0.7% to 2..6%) and SR3 (-2.4% to -8.5%), similar to the prior observation that alignment can degrade task perfor- mance (Ouyang et al., 2022b). However, it's hard to conclude that RLHF in general hurts model performance. We leave it for future work explore the role of RLHF in multi-turn interaction.
for dif-

---

Published as a conference paper at ICLR 2024
3.3 MEASURING LLM's ABILITY TO LEVERAGE NATURAL LANGUAGE FEEDBACK
On top of LLM-tool interaction, we use gpt-4-0613 to simulate user feedback for evaluated LLMs (Fig. with red dotted box). With = interaction limit, we measure the LLM''s ability to leverage natural language feedback using the absolute performance S Refcolback and the performance difference after feedback is given: Acceback = SRReeebhack
SR5. We present results in Tab. 3.
Overall observations. We find no significant difference between open- and closed-source models in terms of feedback. Open-source models obtain +1.7 - +17.2% from feedback, while closed- source models obtain +6.5 +5.22% However, there is still gap between them in absolute success rate S SRf as the best open-source model Lemur-va (70B, SIFT) still lags be- hind the best closed-source model claude inndee-inttant11 by 8.7%. Surprisingly, we find that MAd ---bee can achieve comparable performance to GPT-4 on decision-making tasks with language feedback from it, showing its strong ability to leverage language feedback.
The effect of SIFT and RLHE Similar to $3.2, we find that SIFT and RLHF hurt models' ability to leverage feedback. The results on CodeLLaMA (except 7B) and LLaMA- 2 show that SIFT/RLHF models all have lower feedback and S SRfeedback than their base variants. Another two exceptions are Vicuna-vl 5 (7B) and Lemur-vl (70B). We speculate using multi-turn conversations (ShareGPT) for SIFT contributes to these two exceptions.
3.4 MEASURING THE EFFICACY OF DIFFERENT LLM'S ABILITY TO PROVIDE FREDBACK
we assess seven LLMs' feedback- providing capability through Aeedhuck (Tab. 4). Our main finding is that task-solving ability could be orthogonal to feedback- providdng ability: LLM's higher task-solving performance does not guar- antee better feedback-providing performs well in task-solving (SRs ranked 3rd in Tab. 4), leads to a performance degrada- tion of 10.44% in GPT-3.5; Similarly, GPT-4 with self-feedback in Tab. also experiences de- graded performance. On the other hand, despite performing the worst in solving tasks in Tab. 4, 34--aaM-----I---tt----
Fixing the evaluated model to be 3pt--35-turbbo-6e
capability and vice versa. For example, although GPT-3.5 (16k)
can provide feedback that improves the stronger GPT-3.5.
3.5 MINT CAN HELP DETECT FAIL.URE PATTERNS OF EVALUATED LLMS
Surprisingly, beyond evaluating LLMs' multi-turn interaction ability, we find complex multi-turn tasks in MINT can also test LLM for unexpected behavior. We find two main types of anomalies: (1) inability to follow formatting instructions and (2) unexpected outputs likely due to artifacts.
Inability to follow formatting instructions. We find that some models (e.g., smaller Code and LLaMA, chat-bi wewbbisoo-Oo son-001) 001) have trouble producing parsable format as instructed, hindering task-solving (statistics can be found in Tab. A.7).
Unexpected output likely due to data artifact. We find that Vicuna models (SIFT on ShareGPT data) generate escaped underscore (*(_") instead of underscore ("-") across all tasks, causing syn- tax eтrors when executing code and reducing performance. We examine ShareGPT data (2023) and find least one escaped underscore (")"") artifact on 15% examples, suggesting artifacts in training data could cause this issue. We observe a similar issue with IodeLLaMA-Inst ruct: We find that Code Inst (34B) always ignores user-given instructions on the code generation tasks "wrap your code with <execute> tag" and uses [PYTHON] to wrap the code (happens on 100% of code generation tasks, 0% on other tasks). Touvron et al. (2023) uses [PYTHON] as the tag to generate self-instruct data on code problems for SIFT. We suspect CodeLLaMA Mnn-IInstuuuc models are trained and overfitted to [PYTHON] token, causing them to produce [PYTHON] regard- less of user instruction. We refer $E.1 and $E.2 for examples and quantitative results.
3.6 CAN GPT-4 GENERATE HUMAN-LEVEL NATURAL LANGUAGE FEEDBACK?
We perform a human evaluation quantitatively comparing the feedback generated by GPT-4 and written by humans. Details can be found in Appendix $B. In Tab. 5, human annotators consider 91.2% of GPT-4 generated language feedback to be as helpful as, if not better than, human written feedback. It's also hard for humans to distinguish GPT-4 generated feedback from human feedback

---

Published as a conference paper at ICLR 2024
Table 3: LLM's ability to leverage natural language feedback, measured by A reedback between models' performance with and without feedback produced by 1ost-4008 0 6113. All models are eval- uated with an interaction turn limit of 5. For both open- and closed-source LLMs, the best performance is bolded, and the second-best performance is underlined.
Exalunted LLM
Size
Type
Setup
Reasoning
Decision Making
Code
Micro Averuge
Open-source LLM
no feedhack
0.0
18.7
*0.
4.3
Basc
w/ 3PT-4 feedback
4.8
59.7
0.0
16.2
7B
Backk ger-4
+41.0
no feedback
7.9
17.2
2.2
8.7
SIFT
w/ 3P7-1 feedback
17.1
62.7
10.3
25.9
fredback, get-d
no feedback
8.5
56.0
4.4
18.4
Base
GPT-4 feedbback
15.8
73.0
27.9
31.9
Codel1aNA
138
Seedkack,
+17.9
+23.5
+13.5
feedback
488
50.0
12.
14.5
GPT-4
59.0
22.4
A Soadhack,
5.4
+9.0
+12.5
+7.8
feedback
17.4
63.4
18.4
28.2
GPT-4 feedback
42.7
34B
A Sowckack,
13.0
+20.9
+11.8
+14.5
no feedback
37.3
17.1
SIFT
w/ GPT-4 feedback 3 Sewdback, apt-4
20.2
67.9
3.7
27.3
-5.4
+30.6
+1.5
+10.2
Basc
w/ GPT-4 feedbock
4.1
46.3
8.1
14.7
Arwdback, apt-4
#10.5
no foedback
13.6
*0.0
0.0
7.3
RLHP
w/ CPT-4 feedbback
14.6
2.2
2.9
9.0
Sewdbeck, apt
no foedbuck
3.5
50.0
3.2
14.5
Base
w/ GPT-4 feedbback
10.8
60.5
15.4
21.2
MaaM2-2
Seadback, apt-4
+10.5
+10.3
no feedback
19.6
3.7
2.2
11.9
RLHF
w/ 3FT-4 feedback
21.1
9.7
10.3
17.6
reesback, get-d
no foedback
18.7
59.0
12.5
26.4
Base
w/ 3F feedback
22.5
73.1
27.9
35.3
70B
sustrekk opt-na
18.9
no foedback w/ 3FT feedback
20.2
21.6
8.8
17.9
RLHF
23.1
41.8
19.9
28.6
229
+20.1
+11.0
+8.7
no feethack
16.
61.2
15.4
26.3
SPT feedback
70.2
Lenur-v1
708
Brackack,
+9.0
+12.5
+7.5
no feedhack
31.6
59.7
27.2
37 0
feodback
-0.9
+9.0
+17.6
+6.7
feedhack GPT-4 feedback
100.1
12.2
12.6
7B
SIFT
9.8
61.9
21.7
Vicuna-vl.5
A
get-6
0.3
+35.8
+4.4
+9.0
18.2
12.2
13B SIFT
w/ GPT feedback
16.5
5.2
1.5
10.1
Sredback get-a
554
-3.0
00.7
+2.1
Closed-soures
no feedback feedback
111
29.9
*0.0
14.5
chat-bbison-001
25.8
A Areddaak,,
10.8
+17.2
+6.6
+11.3
feedhack feedback
52.2
14.2
36.8
30.9
claude-2
30.0
A Areddaakk,
2.8
+26.9
+10.3
+10.1
feedback
50.0
47.0
35.3
45 9
53.0
A
4.4
+6.0
+11.8
+6.5
feedback
36.7
41.8
29.4
38.2
gpt cpt-3.5-tuvho-0611
3P7-4
50.3
30.0
A Drwdkack,
+ 13.6
+21.6
+9.6
+15.2
feedback
67.4
84.3
59.6
69.5
cpt-4-0613
GPT-4 feedback
67.
56.6
08.8
A Arwwkack,
0.3
10.7
2.9
0.7
Tab. We identified potential undesired artifacts in its training data, which hurt its performance. See $3.5 for details.
Evaluated produce parsable output inseructed cases (82.1).
8

---

Published as a conference paper at ICLR 2024
Table 4: LLMs' ability to provide feedback, mea- sured by Dreedback with fixed evaluated LLM (GPT-3.5). We also report SRs differences be- tween the feedback-provider and evaluated LLM.
Table 5: Human Evaluation of GPT-4 Gen- erated Feedback against human written feed- back, measuring helpfulness and human-like.
Peodback-provider LLM
SR; Difference
Percentage (%)
gpt-4-0E13
33.3
+13.2
Which feedback is more
Holpful
Human-Like
claude instant
+9.7
+1.5
Both are equally
36.3
69.9
+1.1
-10.4
GPT-4 feedback
54.9
22.1
Codellama 34b (Base)
-8.0
+2.4
ama-2-70a (Base)
-9.7
-0.5
Human foodback
8.8
8.0
Llama 2 70p chat (RLHF)
-18.3
-14.0
CodeT:1 ams 34b- (SIFT)
19.1
+3.22
(human-like) in 92% of the cases. We also compare GPT-4 generated and human-written feedback by asking gpt- 5-turbo- 0613 to continue problem-sol with either turn of (1) human language feedback or (2) GPT-4 feedback. Results show that human feedback and GPT-4 feedback lead to similar model performance SR SRfeebback (32.7% vs. 33.6%).
RELATED WORK
4.1 LLM IN INTERACTION
Interact with users. LLMs have demonstrated extensive potential in seamless interaction with human users and in assimilating real-time human feedback during inference processes (Fernandes et al., 2023). According to recent studies, this collaborative synergy between humans and LLMs has been explored across various domains and upplications, including sentence editing (Reid & Neubig, 2022: Schick et al., 2023e; Chen et al., 2023b), semantic parsing (Yan et al., 2023; Yao et al., 2019), code generation (Nijkamp et al., 2023; Elgohary et al., 2020), itcrative output refinement (Saunders al., 2022), and creative writing (Lee et al., 2022a; Shu et al., 2023; Wang et al., 2023b), generative information-seekinn (Kamalloo et al., 2023), and even theorem proving (Yang et al., 2023b). The partnership between users and LLMs continues redefine possibilities across diverse research areas.
Interact with tools. Engaging with external tools allows LLMs can lead to more accurate and reliable outputs (Peng al., 2023; Gou et al., 2023; Qin et al., 2023a). LLMs can be connected with real-world Application Programming Interfaces (APIs), enabling them to actively engage with diverse external tools (Qin et al., 2023b; Parisi et al., 2022; Schick et al., 2023a: Tang et al., 2023; Patil et al., 2023; Song et al., 2023; Hao et al., 2023; Yuan et al., 2023). For example, L.MSs can connect with (1) the Internet
obtain latest information (Nakano al., 2021; Shuster et al., 2022; Paranjape et al., 2023; Liu et al., 2023b); (2) the program interpreter to run the generated code (Chen et al., 2022; Gao et al., 2023; Drori et al., 2022; Pan et al., 2023; Wang et al., 2023a; 2024); (3) multimodal perceiver to obtain the information beyond the language modality (Huang et al., 2023a; Lu et al., 2023); (4) physical simulator to better understand the physical law (Liu et al., 2023a).
4.2 EVALUATING INTERACTION
Existing work on interaction evaluation mostly focuses on a specific task or dimension, like task completion (Liu et al., 2023c), code generation (Yang et al., 2023a), human-LLM collaborative task solving (Lee al., 2022b; Huang et al., 2023b; Fu et al., 2023), tool manipulation (Tang et al., 2023), and web nevigation (Zhou et al., 2023; Deng et al., 2023a). That is, they solely focus on interacting with either the environment or humans, often on specific task, overlooking the funda- mental importance of both elements in L.M interaction. Different from prior work, MINT covers a range of diverse tasks and designed to measure the multi-turn interaction capabilities of LLMs with both tools and user feedback that are more aligned with real-world applications.
5 CONCLUSION
In this work, we present MINT, an evaluation benchmark designed to evaluate LLM's task-solving ability in multi-turn interaction by using tools and leveraging natural language feedback, which we simulate using GPT-4. We hope MINT can serve helpful resource help track progress and incentivize future research in improving LLM's multi-turn task-solving capabilities. We refer to SA for discussion of limitations and future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their suggestions and comments. This research is based upon work supported by U.S. DARPA ECOLE Program No. HR001 12390060 and ITM Program No. FA8650-23-C-7316 and KAIROS Program No. PA8750-19-2-1000 The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for governmental purposes notwith- standing any copyright annotation therein. HP is in part supported by a gift from Apple.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Lc, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 2021.
Bard API.
Baaaggle-Bad-AAPmmm/55-555 hwwg://wwwwwwwwww/oo/oo/oooooooo googl ec ecllouccommunttyy com ge/ ge/AI-ML/
URL https::/p//....... /www
ChatGPT Plugins. URL URL.https://openat wtt.cccmmc/lc.. http:///.ccccooooooocccccopppppp
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. aiv preprint arXiv:2107.0333 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, abs/2211,12588, 2022. doi: 10.48550WrXX.222111.233 48550/arXiv.20i 2tt.i..221..12228..
URL https://doo http///o..r//g...//
Wenhu Chen, Ming Yin, Max Ku, Elaine Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu. Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305. 125224 2023a.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. arXiv preprint arXiv:2311.1008/1 2023b.
Claude API.
URL https: /ttps::/dooee /ww.. authhppic..cmm/caaaae/eeeeeeeee
seatiingsstartad-witte
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew J. Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworkd:
learning environment for text-based games. In Tristan Cazenave, Abdallah Saffidine, and Nathan R. Sturtevant (eds.), Computer Games 7th Workshop. CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018. Revised Selected Papers, volume 1017 of Communications in Computer and Information Science, pp. 41-75. Springer, 2018. doi: 10. 978-3-030-243377
URL htt.h::tt//tttt //oo: org 0 1007 978 3-030-24337--111----------3333--
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. CoRR, abs/2306.06070, 2023a. doi: 2306. 06070.
10.485 10.448550/rxxiv2206.60000 200000000000000... URL https: /ww... aog.oorg/0..
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang. Huan Sun, and Su. Mind2web: Towards generalist agent for the web. arXiv preprint arXiv:2306.06070 2023b.
10

---

Published as a conference paper at ICLR 2024
Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119(2))=21233331 19, 2022.
Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah. Speak to your parser: In- teractive text-to-SQL. with natural language feedback. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Linguisties. doi: 10. 10.18553/1/2020.1 acc-mman..1m5m 187. URL https: http://aaaaantoooooyooo.o.ooo.at org/ 2020. acl-alll- in. 187.
Online, July 2020. Association for Computational
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André T. Martins. Bridging the gap: A survey on integrating (human) feedback for natural language gen- eration. CoRR, 2023.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from feedback, 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. PAL: program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. 10764-10799. PMLR, 2023. URL https: https::/proceeeeeeeeee /pppceediiggsss. ml..rrr...... Salr.presse /vcesssv2200 w22/2/22///////// /a222/gaa2...../ html.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models can self-correct with tool-interactive critiquing. CoRR, abs/2305.11738 2023. doi: 1048550/arX...235511733 URL https: http://o.oo//////// org/10.
48550/ 2305 11738.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. CoRR, abs/2305. 11554, 2023. doi: 10.48550/
arXiv.2305.1 1554. URL htthh://dooot http:///ooo.ooo/....oo
wtt.l//ooo......///.... 48550/ 2305.11554.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Rongjie Huang, Mingze Li, Dongehao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. Audiogpt: Understanding and generating speech, music, sound, and talking head. CoRR, 2tss/2344.11999 2023a. doi: (0448550/a2xi..234..000 2304.12995
URL https: doi. wttp::///..........oo...... wtt.:///.....///211111..
Shulin Huang, Shirong Ma. Yinghui Li, Mengzuo Huang. Wuhe Zou, Weidong Zhang, and Hai-Tao Zheng. Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles. arXiv preprint arXiv:2308. 10855, 2023b.
Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. HAGRID: A human-llm collaborative dataset for generative information-sseking with attribution. CoRR, abs/2307.16883,, 2023. doi: 10.48850/axXx.2207.6688. URL https://dol org/10. 48550/ark
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing human-ai collaborative writing dataset for exploring language model capabilities. In Simone J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani (eds.),

---

Published as a conference paper at ICLR 2024
CHI '22: CHI Conference Human Factors in Computing Systems, New Orleans, LA, USA, 19. ACM, 2022a. doi: 10.1145/3991002.3000033 URL https: / ht//diioon/ org/ 10. 10..0..115499999
29 April 2022- 5 May 2022, pp. 3491102 3502030.
Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E. Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. Evaluating human-language model interaction. CoRR, abs/2212.09746, 2022b. doi: 10. 10.448550/rXX122212.0999 URL doi. org. 200..48550/araa 201220222222 09746.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M. Dai. Mind's eye: Grounded language model reasoning through simula- tion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL httttps://openreeiieee net/pdf/ 4
Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. Webglm: Towards an efficient web-enhanced question answering system with human preferences. In Ambuj Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye (eds.), Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6- 10, 2023, pp. 4549-4560. ACM, 2023b. doi: 10.1145/3580000 //doi. wrg/o0gg/1000 10. 1145/ 3580305 3599931.
URL https:
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating Ilms as agents. CoRR, abs/2308.03688, 2023c. doi: 10.48550/ arXiv.2308.0 036688.......... URL https: //tp:.///ooooee///oe/eeeeeeeeeea http://too..oo..//////// 48550/ aarviv 2308. 03688.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint
2XXi:2338.00668, 2023d.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-pe sa--anddplly -lay compositional reasoning with large lan- guage models. CoRR. abs/2304.098822, 2023. doi: 10.48550Va2X...2344004844 URL https:
org/ 485.44455/aaia wtt.:i........990. 2984.09842.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint utXiv2230..07842. 2023.
Reichhro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. CoRR, abs/2112.09332, 2021. URL https://arxiss org abs 221...1110000002 09332.
Jerzy Neyman. On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. In Breakthroughs in statistics: Methodology and distribution, pp. 250150. Springer, 1992.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
OpenAl. Gpt-4 technical report, 2023.
OpenAl API. URL www.t:tttt/teecc :ttttpps:///peaaal// httpaaaaccccooacacegaeee/c//
12

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022a.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. fww.efilleeppee wttp:/feeeeeeeeeeee//eeaeeeeeeee pptpp/pppppe////////// In NeurIPS, 2022b. URL http: : /ttppppapppeeeeeeeeeeeeeeeeeeeea nips. cc/ Ceeee--oooeeeeeeee html.
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. Fact-checking complex claims with program-guided reasoning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 6981-7004 Association for Computational Linguistics, 2023. doi: 10.18653/v 1/w.t322220222 act.222..clooon.3356 URL https: : /ttp///oi.ooo// org/ 10. 18653/v1/ a023.acl-1os wonc-lloon... 386.
Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Túlio Ribeiro. ART: automatic multi-step reasoning and tool-use for large language models. CoRR, abs/2303.09014, 2023. doi: 10.485500/rxx..2333..000 URL https://doi.
arrii 2303. 09014.
Aaron Parisi, Yao Zhao, and Noah Fiedel. TALM: tool augmented language models. CoRR.
abs/2205. 2022. doi: 10.48550/rxXi..2255.2222 48550/arXiv http://rxxvv.........
URL https:://do http:://dddo..//1..../ org/10.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. CoRR, abs/2305. 15334, 2023. doi: 10.48550/nXi..2055..533
URL /ttp:///doi.ooo wtw.://..r......5./////
2305. 15334.
Jiaxin Pei, Aparna Ananthasubramaniam,, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. Potato: The portable text annotation tool. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback. CoRR, abs/2302.12813,, 2023. doi: 10.48550WaXivv.332.... 12813. URL https: /tttps://dd....// 405510..4555//ra..... 2302 12813.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong. Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang. Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. In arxiv, 2023a.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023b.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ roal-world apis. arXiv preprint arXiv:2307.16788 2023c.
Machel Reid and Graham Neubig. Learning to model editing processes. In Findings of the Association for Computational. Association for Computational Linguistics, 2022.
13

---

Published as a conference paper at ICLR 2024
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu. Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. CoRR, wbwp/226.0088000 2022. doi: 10.485550urXiv.2206..o 05802. URL https: /doi. org/10. 48550 arxiv 2206. 05802.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761, 2023a. doi: 10.48350annXv..332..0777
URL https:
//doi. orgg 10. 48550 /0.50////xvv.. 2302. 04761.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023b.
Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick S. H. Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sehastian Riedel. PEER: A collab- orative language model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nel, 2023c. URL https:// openreview wttp:/eeeeeppp.edeeee.eeoooo
ShareGPT data, 2023. URL www.:eeT_iiicuuaanniiiiae
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:230.111366 2023.
Mohit Shridhar, Xingdi Yuan, Alex-nlex Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020.
Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. Rewritelm: instruction-tuned large language model for text rewriting. CoRR, abs/2305. 15685, 2023. doi: 10.48550WrXXv.23000 15685. URL ttttps///dee org/10. 48550/arxi 2305
15685.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Na- man Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. Blender- bot 3: a deployed conversational agent that continually learns responsibly engage. CoRR, abs/2208.03188. 2022. doi: 10448500arXi..2200.311866 URL https: htops:///do.. htt//ooi.o///////// 48550/arXiv 2208 03188.
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Rest- gpt: Connecting large language models with real-world applications via restful apis. CoRR, abs/2306.06624, 2023. doi: 21..5500rrXxv2230006622222 URL https: / /ww..//iioo//////////..o/ org/10. 48550/arXiv. 2306 06624.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Gen- eralized tool learning for language models with 3000 simulated cases. CoRR, abs/2306.05301, 2023. doi: 10.48550/arXiv.23066.00 2306.00 05301.
URL https: http::////o..or00...88555/////
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
US Bureau of Labor Statistics. Table b-3, average hourly and weekly carnings of all employees private nonfarm payrolls by industry sector, seasonally adjusted, 2023. URL https: www. bls.gove /ewg///www.......oo.. re lease empsit 19. htm. Accessed: 2023-9-3.
14

---

Published as a conference paper at ICLR 2024
Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, and Heng Ji. Leti: Learning to generate from textual interactions. arXiv preprint arXiv:2305.100 10314, 2023a.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better IIm agents. arXiv preprint mrXiv22402.01030 2024.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Large language models are cognitive synergists: Task solving through multi-persona self-collaboration. In arxiv, 2023b.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL htttp::://pppeeeeesese net wwtp:/nuttefottioeeeeeeee gEZrGCozdqR.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents, 2023.
Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen-tau Yih, and Ziyu Yao. Learning to sim- ulate natural language feedback for interactive semantic parsing. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3149-3170, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18663//11/0233allloo https: / ha////aaanntthoooooeoo wwwpiiyy..rg///00 orgg/o23...o....oo acc.oon...l11o..1...
URL
John Yang. Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. arXiv preprint arXiv:2306.148: 148998 2023a.
14898,
Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. Leandojo: Theorem proving with retrieval-augment language models. CoRR, abs/2306. 15626, 2023b. doi: 10,48550/arXi..336..5555 156266. URL
bttps:://oi.. /t///ii.oggg/0// 0t0//1004855000 arxiv. 2306 15626.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio. William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question 2018.
answering. arXiv preprint 1.Xiv:800900900
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. Model-based interactive semantic parsing: A unified framework and text-to-SQL case study. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (cds.), Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-UJCNIP), pp. 5447-5458, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/1//D1--100 1547. URL https::///// perrps://aclannthlogg org/ 319-1547.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung, Hao Peng, and Heng Ji. Craft: Customizing Ilms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17422 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging 11m-as judjuee with mt-bench and chatbot arena, 2023.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environ- ment for building autonomous agents. CoRR, abs/2307. 13854, 2023. doi: 10.485500/rxiv.23077. 13854, URL https: //oo/./////// .org/org//10 48550/ r 2307. 13854.
15

---

Published as a conference paper at ICLR 2024
A LIMITATIONS AND FUTURE WORK
We simulate the natural language feedback of human users with GPT-4. Despite showing in a human experiment that is similar human-written feedback, however, GPT-4 simulated might not cover all the possible responses from real-human users and may not suitably simulate every aspect of human feedback, particularly in tasks (e.g., policy-making) that involve nuanced judgments of human values. While the focus of our work lies on LLM's in-context multi-turn turn interaction, we have yet to explore the potential of directly leveraging language feedback for model training and improvement similar to Wang et al. (2023a), which we leave for future work. Furthermore, our metries may not fully assess the quality of the interaction process beyond outcomes. For example, models repetitively guessing to get higher scores should be penalized. Despite our best efforts to ensure our benchmark contains challenging and comprchensive tasks, there is still a wide range of tools (Qin et al., 2023c) and real-world use cases (e.g., web-browsing Deng et al. (2023b), operating system Liu et al. (2023d)) that MINT did not cover. Instead of making this benchmark a one-time effort, we hope continuously improve this benchmark by integrating more challenging tasks and tools as LLMs get better.
B DETAILS OF HUMAN EVALUATION
We perform two stages of human annotation using the Potato annotation interface (Pei et al., 2022). In the first stage, we ask two human annotators (A and B) provide language feedback for trajectory. We randomly sample 2 instances of interaction trajectories per task from a subset of 8 evaluated LLMs to maximize diversity (in Tab. 3). We filter out task instances that succeed in the first turn (i.e., no need for feedback), resulting in 1 113 interaction trajectories for annotation. We randomly select a turn for each task trajectory and remove all interactions and GPT-4 generated feedback after that turn. We randomly divide the 113 instances into two subsets and assign each subset to one human annotator. Given previous interaction history, human annotators and are asked to provide turn of natural language feedback as if interacting with ChatGPT. Annotation of each feedback, on average, takes 96 seconds. According to US Bureau of Labor Statistics (2023), U.S. private non-farm worker average about $33.82 hourly wage (Aug 2023), which translate to an annotation cost of $90 per 100 turns of feedback.
In the second stage, ask two different human annotators (C and D) to compare -aiman--nnottted feedback (from the first stage) and GPT-4 generated feedback (from the original trajectory) on two dimensions: helpfulness and human-like. Specifically, helpfulness means whether feedback is help- ful for the LLM to succeed in this task, while human-like focuses on the literal similarity of feedback and human usage. For each dimension, ask them determine which feedback is better (i.e., more helpful or human-like) or both are equally good.
C ABLATION STUDY
C.I HOW DO FEEDBACK VARIATIONS IMPACT FEEDBACK QUALITY AFEEDBACK?
To gain deeper insights into the effects of various feedback settings on enhancing the performance of language models, we perform an ablation study on feedback by controlling feedback informativeness and frequency See $F.4.2 for detailed implementation. We present the results in Tab. A.6.
C.1.1 INFORMATIVENESS
We define informativeness in two dimensions: (1) whether the generated feedback is conditioned on the ground-trum solution (w/ GT) or not (w/o GT, default setting); (2) whether the feedback provided to LLM textual (default setting) or binary (i.e., good vs. bad).
Conditioned on Ground-truth Information In Tab. C.I.I. we find that adding ground-truth in- formation into the feedback generator improves the quality of feedback for reasoning and code gen- eration. However, this trend doesn't 't hold for decision decision-making, where using ground-truth information for feedback leads to a performance drop (-8 8.95%% compared to no feedback. We hypothesize that this discrepancy can be attributed the unique nature of decision-making tasks. Unlike other tasks
16

---

Published as a conference paper at ICLR 2024
Table A.6: Ablation of different factors (informativeness, frequency) that impact feedback quality, using 3 5-tuuro--000111 as the evaluated LLM and 9pt-4-0613 to simulate language feedback.
Sctup
Reasoning Decision Making
Code Generation
Micro Average
w/o feedback
36.25
41.79
29.41
35.93 + +15.09
Teedback, textual, w/o
+13.44
+24.63
+9.56
Informativeness of Feedback
A Soedback, x/ GI
+ +16.87
-8.95
+ +18.38
+11.36
A J07 reedback
+3.43
-33.58
+8.82
-3.73
A deedback, Sinary
+2.19
+5.97
+0.74
+2.71
A texcual
11.25
18.66
-8.82
-12.38
Frequency of Feedback
A Seedback, sparse
+5.31
+4.48
+0.74
+4.07
A feedback frequency
-8.13
-20.15
-8.82
-11.02
with definitive solutions, decision-making tasks involve generating action trajectories as solutions (e.g., $F.6). When the initial actions of the model deviate from the ground-truth trajectory, compar- ing its actions with the ground ground-truth truth actions could confuse the feedback -rrovide LLM, resulting in suboptimal feedback quality.
Provide Binary Feedback We find that providing LLM with binary feedback (i.e., a binary label of good or bad) instead of more informative textual feedback (i.e., superset of binary feedback) inevitably hurts performance on all tasks. However, we observe that binary feedback alone provides performance benefits compared no feedback, especially for poeiiiionmmaingg (+5.97), where early action can profoundly impact final task success. In these cases, providing step-wise binary feedback can help LLM agents terminate bad initial actions and backtrack, leading higher task success rate.
C.1.2 FREQUENCY
We investigate the role of feedback frequency: whether we are providing feedback to the LLM every step (Dense) or only when the LLM agent proposes solution (Sparse, i.e., when the LLM thinks it finishes the task).
In Tab. A.6, as expected, we find changing from dense to sparse feedback hurts performance (-11.02 on average). However, we observe positive performance gain on all tasks, similar to binary feedback ($C.1.1), suggesting that sparse feedback alone is valuable. Note that when evaluating on sparse feedback setting, MINT is equivalent to the setting of Reflexion feedback (Shinn et al., 2023).
D DATASET FILTERING AND DOWN-SAMPLING
The dataset curation can be summarized into three steps:
Collect data from the test set of 8 different datasets shown in Table 1.
For HotpotQA we reserve the first 500 instances. Then, we format dataset prompts with ('Task:' solution ion.tiionnt orngge......... For the solution solution_range variable, in GSM8K is set to be integer, and in TheoremQA it is set corresponding the instance requirement (float, integer, list of integers, option). For other datasets, since they don't have a specific solution range require- ment, set solution range to be empty string. An example from TheeoremQ is follows:
Task: Let be the inverse of the group element (33, 5), (4, 6)
in 2.7. What is M[O] [1]? Output format required: integer.
In this example, task description is: "Let M be the inverse the group element ((3, 5), (4, 6)) in Z 7. What is M[O][1]?" and solution range is: "Output format required: integer."
17

---

Published as a conference paper at ICLR 2024
Table A.7: The average number of interaction turns an LLM failed due following the instructed format (i.e., not producing <execute> or <solut ion> tag as instructed under k = 5 and no feedback setting, $2.1) vs. the average number of total turns. All LLMs that produce more than 20% of such invalid actions w.r. total turns are bolded.
Number of Failed Turn due to Format: Total Turms (Averaged) Reasoning
Evaluated LLM
Size
Type
Decision
Code
Micro Average
Open-source LLM
Base
3.96 / 4.99
0.11 / 4.17
2.38/ 4.338
2.71 / 4.66
7B
SIFT
0.46 / 4.32
0.10/ 4.33
0.10 / 4.65
0,30 / 4.40
CodeLlama
Base
0.50 / 4.53
0.00 3.36
0.00 / 4.93
0.27 / 4436
13B
SIFT
0.16 / 4.66
0.01 /.77
0.04 / 4.77
0.10 4.48
Base
0.19/
0.00/
0.05 / 4.77
0.11 0.11/ 4415
34B
SIFT
0.23 3.68
0.04 3.83
1.09/ 2.
0.39 / 3.62
Base
0.59 4.62
0.00 3.53
0.25 4.96
0.38 4.45
7B
RI.FF
0.75 / 4.03
1.13 4.40
0.72 3.79
0.83 / 4.06
LLaMA-2
Base
0.49 / 4.75
0.01 / 3.40
0.13/ 4.96
0.30 / 4.49
13B
RLHF
0.29 / 3.71
0.00 4.54
0.10/ 3.02
0.18 / 3.74
70B
Base
0.19/ 4.19
/
0.16/ 4.49
0.14 4.06
Base
0.29/ 4.25
/
4.33
0,22 4.05
Lemur-vl
70B
SIFT
0.35 3.88 88
0.01 / 3.34
/ 4.07
0,20 3.80
7B
SIFT
0.26 4.64
0.06 1L06/3..0 3.54
0.02 / 4.78
0,16 4.42
Vicuna-vl.5
13B
SIFT
0.08 / /480
0.49 / 4.66
0.07 / 44.0
0.17 4.79
Closed-source
LLM
chat-bison-001 claude-2
2:27 / 3.84
0.10 4.18
4.62 / 4.87
2.32 / 4.16
0.02 / 1.86
0.01 3.51
0.00 2.244 2.24
0.02 2.32
claude claude-instant--
instant-ttt gpt-3.5-turbo-O66
0.06 / 2.81
/.99
0.02/3.77
0.04 3,28
0.50 / 4.18
0.00 3.87
0.07 4.226
0.29 / /.133
gpt-4-0613
0.04/3.11
2.887
0,00 3.442
0.02 3.13
Keeping instances that requires multi-turn interaction.
We first clean up multiple-choice tasks with less than options. These tasks are primarily from MMLU and TheoremQA datasets.
For MMLU and MATH, since their test sets are large and have various classes of tasks (e.g., for MATH they have algebra, geometry, pre-calculus), we firstly roughly clean those classes that do not need interaction (e.g. for MMLU they have "philosophy" do- main which does not need much interaction but only requires some basic knowledge about philosophy) by picking up N instances from each class, run these instances with 3. 1..55tuuoo-00 0613, and exclude those classes whose average interaction turn across instances less than & turns. For math we set N 100 and k 3.5, for MMLU we set N = 20 and = 2.5. Remaining classes of MATH: Intermediate Algebra, Pre- caleulus, Algebra, Geometry, Number Theory. Remaining classes of MMIJ: world reli- gions test, virology test, college mathematics test, astronomy test, college physics test, high school chemistry test, global facts test, high school mathematics test, formal logic test. we run all remaining data with gpt 3. 53.5-tu Sttubbo---o 0613 with turn budget A 5. no feedback, and exclude those instances with k<2
Stratified sub-sampling for efficient evaluation.
After cleaning data, we want to maintain data difficulty and balance different types of tasks while continuing sub-sampling. We stratify the instances based on the dataset and whether gpt-3. 5-turbo-0611 has completed it (i.e., 8 x 2 16 strata). For each stratum set dif- ferent proportions of instances to be preserved: Palfworld = 1, Pmbpp Phomaneval = 0.5, PasmSk
Photpotqa Ptheoremqa = 0.2, PMMLU = 0.1, PMATH 0.05.
18

---

calculate the percentage of turns that Published as a conference paper at ICLR 2024
Table A.8: Summary Tools by Task Type
Task Type
Tool Signature
[1] put (object: str, receptacle: str) -> str
[2] goto (receptacce: str) -> str
[3] take_from (object: str, receptacle: str) -> str
[4]
str) >> str
[5] togglel(oject...rrrccettaal
str) >> str
Decision-Main Making
[6] close.recepaccle (receppacle: str) -> str
[7] clean(object:: str, receptacle: str) -> str
[8] heat (object: str, receptacle: str) str
[9] cool (object: str, receptacle: str) -> str
[10] use (receptacle: str) -> str
[11] look -> str
Reasoning
[1]
wikipedia_search (query: str) -> str
Code Generation
No tool is provided
E ISSUES
E. VICUNA-V1.5 ESCAPE UNDERSCORE ISSUE
The following is random trajectory (in-context example omitted) from Vicuna-13b-vl model". For some unknown reason, tends to escape any underscore (*) ") that appears in the code, causing it to fail some task instances. Quantitatively, contain an escaped underscore over different LLMs in Tab. A.9, and find that this is particular issue with Vi cuna and SIFT version of Lemur-va We checked 94, 145 instances of ShareGPT data" and found that about 15% (14, 110) of the conversations have the pattern of backslash underscore (")"), We believe these artifacts in the instruction tuning dataset could be the reason that causes Vicuna and Lemur-vl to generate code with these backslash underscore patterns (*)-").
user
You are helpful assistant assigned with the task of problem-solving. To achieve this, you will be using interactive coding environment equipped with variety of tool functions to assist you throughout the process.
At each turn, you should first provide your step-by-step thinking for solving the task. Your thought process should be enclosed using "<thought>" tag, for example: <thought> need to print "Hello World!" </thought>.
After that, you have options:
1) Interact with a Python programming ronment and receive the corresponding output. Your ccde should be enclosed using "<execute>"
tag, for example: <execute> print ("Hello World!") </execute>.
2) Directly provide a solution that adheres to the required format for the given task. Your solution should enclosed using "<solution>" tag, for example: The answer is <solution> A </solution>.
You have 5 chances to interact with the environment propose solution. You can only propose a solution times.
ottrpss//hugiinggfacccco// ctt.ee..oomsss///eeeeee dattasets/ann82311
http://nnnaa..eee//////////caaae cuna- 3b-v1.5
unfiltered
19

---

Published as a conference paper at ICLR 2024
Table A.9: Percentage of task instances that assistant outputs escaped underscore ("\-") under the setting of k = 5 without language feedback, which causes Syntax Error during task-solving. This seems be a particular issue with Vicuna LLMs. Percentages greater than 2% are bolded.
Percentage of assistant interactions that contains escaped underscore ( (%) Decision-Making
Evaluated LLM
Size
Type
Reasoning
Code
Open-source
LLM
Base
0.00
0.00
0.00
7B
SIFT
0.00
0.00
0.00
LLM
Base
0.00
0.00
0.00
13B
SIFT
0.00
0.00
0.00
Base
0.00
0.00
0.00
34B
SIFT
0.00
0.00
0.00
Base
0.00
0.00
0.00
7B
RLHF
0.00
0.00
0.00
LLaMA-2
Base
0.00
0.00
0.00
13B
RLHF
0.00
0.00
0.00
70B
Base
0.00
0.00
0.00
Base
0.00
0.00
0.00
Lemur-vl
70B
SIFT
2.22
0.00
0.00
7B
SIFT
30.38
0.75
11.03
Vicuna-vl.5
13B
SIFT
38.61
86.57
90.44
Closed-source LLM
chat-bison-001
0.00
0.00
0.00
claude-2
0.32
0.00
0.00
claude-instant-1
0.00
0.00
0.00
1pt-3.5-turbo-00
0.00
0.00
0.00
apt-3.5-turbo-11 -600o0 gpt-4-0613
0.00
0.00
0.00
0.00
0.00
0.00
Task:
Write a function to check if a nested list is a subset of another nested list.
assert chock_eubsot__iittt... 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14], [[12, 18, 23, 25, 45], [7, 11, 19, 24, 281, [1, 5, 8, 18, 15, 16l1)==False
- assistant
Assistant:
<thought> check if nested 1ist is a subset another nested 1ist, should iterate through the elements of the first list and check if they are present in the second list, If all elements are present, then the first list is subset of the second list. </thought> <execute>
def Boocllssubstt[_litt (list1, list2): for element in listl: if element not in list2: return False return True res
check), booockkssubbstii l list
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14),
[[12, 18, 23, 25, 45],
[7, 11, 19, 24, 281,
20

---

Published as a conference paper at ICLR 2024
[1, 5, 8, 18, 15, 16]]
assert res = False, "Expected False but got {) "* format (res) </execute>
Observation:
Cell Inlll, line 1
def check) check\_ Lis list2):
Syntaxtrrer: unexpected character after line onntinnation character You have 4 steps left and chances to propose solution left.
Expert feedback:
This is BAD. You have syntax
your function definition. The backslashes are not needed in the function name. Also, your logic is incorrect. You are checking if the elements of the first list are in the second list, but the secend list is a 1ist of lists. You need to check if the clements of the first list are in any of the sublists of the secend list.
assistant Assistant:
<thought> apologize the syntax
should remove the backslashes from the functica nare and check the elements of the first list are in any of the sublists of the second list. </thought> <execute>
def Lheck (listl, list2): for sublist in list2: for element in listl: if element not in sublist: return False
return True check
res Shhcck\_subee _subset
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], [[12, 18, 23, 25, 45], [7, 11, 19, 24, 28], [1, 5, 8, 18, 15, 1611
assert res = False, "Expected False but got (res) </execute>
user Observation:
Cell In[2], line 1 def check'_subeett 1ist (listl, list2):
SyntaxError:
unexpected character after 1ine continuation character You have 3 steps left and chances to propose solution left.
Expert feedback:
This is definition. The backslashes are not needed in the function name.
have repeated the same syntax error your function
Also, your logic is still incorrect. You are returning False as soon as you find an element from listl that is not in sublist of 1ist2. that all elements of listl need be in all sublists of list2 for function return Truc. However, you should be checking if each element of list1 is in any sublist of list2.
This would
--- assistant ===
Assistant:
<thought> apologize for the repeated syntax error. should remove the backslashes from the function nane and check it each element of the first list is in sublist of the second 1ist. </thought> <execute>
21

---

Published as a conference paper at ICLR 2024
def (List,, list2):
(listl,
for element in listl:
if clement not in any (sublist for sublist in list2):
return False
return True
res
( [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
I[12, 18, 23, 25, 451,
17, 11, 19, 24, 281,
11, 5, 8, 18, 15, 1611
assert res False, "Expected False but got format (res) </execute>
Observation:
Cell In[3], line 1
def check)_ _iss (list1, list2):
SyntaxError: unexpected character after line continuation character You have 2 steps left and chances to propose solution left.
Expert feedback:
This is BAD. You have repeated the syntax definition. The backslashes are not needed in the function name. Also, your logic is still incerrect. You are using the 'any' function incorrectly. The 'any' functicn returns True if at least one elenent of iterable is true. You shculd be using it to check if the
in your function
each in the sublist. the element not in any of the sublists, then you should return False. If you have checked all elements and have net returned False, then you should
element is in any of the sublists. You need to iterate the element
sublist and check
return True.
assistant
Assistant:
<thought> apologize for the repeated syntax error. should remove the backslashes from the function nare and use the 'any' function correctly check if each element of the first list any sublist of the second list. </thought> <execute>
def succk__ssbtetLiistttttt
list2):
for element in listl: if not any (element in sublist for sublist in list2):
return False
return True res
__ist
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14),
I112, 18, 23, 25, 451,
17, 11, 19, 24, 281 [1, 5, 8, 18, 15, 16]]
assert res False, "Expected False but got format (res) </execute>
Observation:
Cell In[4], line 1
def check list (list1, list2):
SyntaxError: unexpected character after line continuation character You have 1 steps left and chances to propose solution left. You should take the last step to propose solution.
Expert teedback:
22

---

Published as a conference paper at ICLR 2024
This is BAD. You have repeated the same syntax definition. The backslashes are not needed in the function name. Also, your logic is still incerrect. You are using the 'any' function incorrectly, The 'any' functicn returns True if at least one elenent of an iterable is true. You shculd be using it to check if the element is in any of the sublists. You need iterate the element not in any of the sublists, then you should return False. If you have checked all elements and have not returned False, then you should return True.
in your function
each
sublist and check
the element
in the sublist.
assistant
Assistant:
<solution>
The correct solution is:
"'python
def chock_subset_liss (list1, list2); for element in listl:
if not any (element in sublist for sublist in list2):
return False
return True
Explanation:
We iterate over each element in "isstl".
For each element, we check if it is in any of the sublists of "list2" using the ()ny()" function.
If the element is not in any of the sublists, return "False", If have checked all elements in "listl" and have not returned "False", we return "True".
This soluticn correctly checks if each element of "list1" is in any sublist of "list2",
E.2 CODELLAMA CCDELLAMM-IXSTRUCC PRODUCES [PYTHON] TAG
The following is a random trajectory on coding tasks (in-context example omitted) from model Despite being explicitly instructed to output code within pairs of <execute> or <solutions tag through instruction and in-context example, CodeLLaMA -ustrut always wraps their output solution within a pair of [PYTHON] tag as shown in Tab. A.10, despite not explicitly instructed so.
In Tab. A.10, find that [PYTHON] tag is always associated with code generation problems but not other types of tasks. The appearance of [PYTHON] is most frequent in the largest version (34B), but not appearing at all in the smallest versions (7B).
We suspect the association of [PYTHON] tag and code generation task is due to artifacts intro- duced during the instruction-tuning phase of CodeLLaMA- Ieni-Instruuit model. Quoting Rozière et al. (2023), "Code Llama Instrus ruct variants are further fine-tuned on mix of proprietary instruction data for improved safety and helpfulness, and new machine-generated self-instruct dataset created by prompting Llama 2 for coding problems and Code Llama to generate associ- ated unit tests and solutions." Furthermore, Rozière et al. (2023) mentioned that in the self-instruct process, they "prompt the model to generate the code between [PYTHON] and [/PYTHON] tags to make easy to parse the result." " We suspect BoddeLLMMA-Inntaee tuned [PYTHON] tags during instruction-tuningg causing tags (e.g.,<exi (e.ccteccte>) on coding problems. Furthermore, we hypothesize that the larger the LLM, the will be able memorize data points seen during its instruction-tuning phase, causing to produce [PYTHON] more often as observed in Tab. A.
was explicitly fine- ignore instructions about other
hnttps://hugiinggffcc.e www.e/eeeccoooeeeeeeeeeeeeeeeia
34b- Instruct-hf
23

---

Published as a conference paper at ICLR 2024
Table A.10: Percentage of task instances that assistant outputs [PYTHON] tag at least once, un- der the setting of & without language feedback. This seems to be particular issue with CodeLLaMA LLMs on code-related tasks, especially the model with larger scale.
Percentage assistant interactions that contains [PYTHON] tag (%)
Evaluated LLM
Size
Type
Reasoning Decision-Making
Code
Open-source
LLM
Base
0.00
0.00
0.00
7B
SIFT
0.00
0.00
0.00
CodeLLaMA
Base
0.00
0.00
0.00
13B
SIFT
0.00
0.00
2.21
Base
0.00
0.00
0.00
34B
SIFT
0.00
0.00
100.00
Base
0.00
0.00
0.00
7B
RLHF
0.00
0.00
0.00
LLaMA-2
Base
0.00
0.00
0.00
13B
RLHF
0.00
0.00
0.00
70B
Base
0.00
0.00
0.00
Base
0.00
0.00
0.00
Lemur-v1
70B
SIFT
0.00
0.00
0.00
7B
SIFT
0.00
0.00
0.00
Vicuna-v1.5
13B
SIFT
0.00
0.00
0.00
Closed-source LLM
chat-bison-C01
0.00
0.00
0.00
claude-2
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
gpt-4-0613
0.00
0.00
0.00
user ---
You are helpful assistant assigned with the task of problem-solving To achieve this, you will be using interactive coding environment equipped with variety of tool functions to assist you throughout
the process.
At each turn, you should first provide your step-by-step thinking for solving the task. Your thought process should be enclosed using "<thought>" tag, for example: <thought> need to print "Hello World!" </thought>.
After that, you have options:
1) Interact with a Python programming environment and receive the corresponding output. Your ccde should be enclosed using "<execute>"
tag, for example: <execute> print ("Hello World!") </execute>.
2) Directly provide a solution that adheres to the required format the given task, Your solution should be enclosed using "<solution>" tag, for example: The answer is <solution> A </solution>.
You have 5 chances interact with the environment propose solution. You can only propose a solution times.
Task:
24

---

Published as a conference paper at ICLR 2024
Complete the following code:
def (strsnn
Create function that takes string as input which contains only square brackets.
The function should return True if and only there is valid subsequence of brackets
where at least one pracket in the subsequence is nested.
is_nested (' ([]]')
is_nested ( []]]1]][((((1)" -> False
is_nested (']] -> False
d'_nessed -> False
is ('
True
is nestee ('[])
([]](() -> True
assistant
Assistant:
[PYTHON] def (striing
stack II for char in string:
if char -- '[':
(char)
elif char ')':
if len(stack) = return False
stack.pop()
return (sttak) ==
[/PYTHON]
user
Observation:
don't understand your input.
you want to execute code, please <execute> YOUR_CODE_HAY </execute>,
If you want to give me an answer, please use <solution> YOUR SOLUTION HERE </solutien>.
For example: The answer to the question is <solution> 42 </solution>.
You have 4 steps left and chances to propose solution left.
Expert feedback:
The assistant's approach solving the problem is correct, but the assistant didn't follow the correct format for code execution. The assistant should use the <execute> tag to execute the code. Also, the assistant's solution doesn't seer. to check for nested brackets, it only checks for balanced brackets. The assistant needs to modi fy the solution to check for nested brackets.
=== assistant == Assistant:
[PYTHON]
def [_st stack = [] for char in string: if char == '['
atack (char) elif char == 'J': it (stacc)) = 0: return False
25

---

Published as a conference paper at ICLR 2024
stack.pop
return (stack)
[/FYTHON]
You have 4 steps left and chances to propose solation left.
user
Observation:
I don't understand your input.
If you want to execute code, please <execute> YOUR_CODE HERE </execute>,
If you want to give me an answer, please use <solution> rOUR_SOLUTION_HIIS </solution>.
For example: The answer to the question is <solution> 42 </solution>.
You have 3 steps left and chances to propose solution left.
Export feedback:
The assistant's approach to solving the problem is correct, but the assistant didn't follow the correct format for code execution. The assistant should use the <execute> tag to execute the code. Also, the assistant's solution doesn't seem to check for nested brackets, it only checks for balanced brackets. The assistant needs to modity the solution to check for nested brackets.
assistant Assistant:
apologize for the confusion. will make follow the correct format for code execution and check for nested brackets the
solution.
Here is the modified solution:
<execute>
def is_nested (string) stack - [] for char in string:
if char stack (char)
'I':
elif char == '1':
if (snack)))) = 0:
return False
popck.pop() ()
return (stack) = 0
</execute>
You have 3 steps left and chances to propose solution left.
F PROMPTS AND EXAMPLES
We carefully designed prompts and in-context examples for each task type to elicit interaction be- havior from LLMs. In this section, we include tool descriptions, error messages shown to LLM for formatting issues, example task trajectories, prompt templates, and in-context examples.
F.1 TOoL DESCRIPTIONS FOR EACH TASK TYPE
For Code Generation interpreter ($2.1).
No additional tool is provided for code generation apart from the Python
For Reasoning
26

---

Published as a conference paper at ICLR 2024
Tool function available (already imported in <execute> environment) str The Wikipedia Search tool provides access to vast collection of
[1] eikipedia_search (qurr: str)
articles covering wide range of topics.
Can query specific keywords or tepics to retrieve accurate and comprehensive information.
For Decision-Making (ALFWorld)
Tool function available (already imported in <execute> environment) [1] put (object: str, receptacle: str) -> str Put an object in/on a receptacle.
For example: put ("mug 1", "desk 2")
[2] goto (receptacle: str) -> str Go to a lecation of the receptacle. For example: goto("drawer 1*)
[3] take frrm tremfobject: str, receptacle: str) str Take an object from a receptacle.
For example: take_from ( ("mug 1", "shelf 2")
[4] open receptace (receptacle: str) Open a receptacle. For example: open_rece rean_receptaclee ee ("fridge 1")
str) str Toggle an object or receptacle.
[5] toggle
For example: togggh("light 2")
[6] ce (receptacle: str) >> str Close receptacle.
For example: close receptacle (orleeeemmiiii ("microwave 1")
[7] clean (cbject: str, receptacle: str) >> str Clean object with receptacle.
For example: clean("clote ("cloth 1", "sinkbasin 1")
[8] heat (object: str, receptacle: str) >> str Reat an object with a receptacle. For example: heat ("egg 1", "micrcwave 1")
[9] cool (object: str, receptacle: str) -> str Cool an object with a receptacle. For example: cool ("bottle 1", "fridge 1")
[10] use (receptacle: str) -> str
Use a receptacle.
For example: ("lamp 1")
[11] look str
Look around. It will return what you see in the room.
For example: ()
F.2 FORMATTING ERROR MESSAGE
The following error message provided to the L.LM Agent when we fail detect the instructed format (e.g., fail to find <execute> and/or <solut ion> tag).
don't understand your input. If you want to execute code, please use <execute> YOUR_CODE_HENT HERE </cxecute>.
27

---

Published as a conference paper at ICLR 2024
If you want to give an answer, please <solution> YOUR_BOLUTION_HERS </solution>. For example: The answer to the question is <solution> 42 </solution>.
F.3 EXAMPLE TASK TRAJECTORY
The following is an example trajectory on a reasoning task from MATH (Hendrycks et al., 2021), generated by gpt-3. 5-turbo-0611 visor to provide feedback. We release outputs from all LLMs we evaluated in our code repo
agent and gpt 4-0613 simulated human ad-
Task:
Let $a, 9 $b, $ and $c$ be the roets of $x"3 - 3x 5 = 0. Find $abo. $
Assistant:
<thought> To solve this problem, we can use Vieta's formulas. Vieta's formulas state that for a polynemial $ax"n + (n-1) + (n-2)- \dots k 0$ with roots $r_l, r_2, Adots, r_nš, the following relaticnships hold:
$r_1 + r_2 \dots In = (V
$r_1r_2 +r_rr_3 + \dots + r_(n-1lr_n Afracie)
$r 2r_3 2_1r_2r_4 \dots in -\fracid) $1dots$
$r_1r_2 \dots r_n (- (a)
In this case, have cubic polynemial $x"3 - 3x 5 = 0$, can use the formula $abe (\frac(d) find the product of the roots. </theught> <execute>
= 1
5
abc -d/a abe </execute>
Observation:
Out 111: -5.0
You have 4 steps 1eft and chances to propose solution left.
Expert feedback:
This is GOOD. You correctly applied Vieta's formulas to find the product of the roots the given cubic equation. Now you can submit the solution.
Assistant:
The answer <solution> -5.0 </sclutions,
Observation:
Your answer is wrong.
You have 3 steps left and chances to propose solation left.
Expert feedback:
The assistant made mistake in applying Vieta's formulas. The formula for the product of the roots of a cubic equation is $abc (-1) \frac(k) (a) not $abc -Xfrac (d) (a)$. and $a=15, so the correct formula to use is Sabc (-1) (1)a Please correct this and try again.
this case, $n=3$, Sk=-55,
Assistant:
<thought> apologize for the mistake. Let's correct it. </thought> <execute>
a
k -5
abc x*1) k/a
28

---

Published as a conference paper at ICLR 2024
abc </execute>
observation:
Out [2]: 5.0 You have 2 steps left and chances propose solation left.
Expert feedback:
This is GOOD. You have correctly applied Vieta's fermulas this time and founc the correct product of the roots of the given cubic equation.
Now you can submit the solution.
Assistant:
The answer is <solution> 5.0 </sclution>.
F.4 PROMPT TEMPLATES
To instantiate the prompt for each task type, (in_context_eat _xx_peeeeeeeee will be substituted with examples in $F.5, and (tool_desc) will be substituted with tool descriptions in $F.1.
F.4.1 EVALUATE LLM
For each task, replace prompt) with the corresponding task instruction. and [max.propose_steert _stpss will be substitute with the setting for that experiments. In MINT, we default max total steps and 2 max propose steps discussed in $2.1 and $3.1.
You are a helpful assistant assigned with the task of problem-solving, achieve this, you will be using an interactive coding environment equipped with variety of tool functions to assist you throughout the process.
At each turn, you should first provide your step-by-step thinking for solving the task. Your thought process should be enclosed using "<thought>" tag, for example: <thought> I need to print "Hello World!" </thought>.
After that, you have two optiens:
1) Interact with Python programming environment and receive the correspending output. Your code should be enclosed using "<execute>"
tag, for example: <execute> print ("Hello World!") </execute>.
2) Directly provide a solution that adheres to the required format for the given task, Your solution should be enclosed using "<solution>" tag, for example: The answer is <solution> A </solution>,
You have )tepss chances to interact with the environment or propose a solution. You can enly propose solution (max_proposssslltiinn) times.
(tool_desc)
iin_contcxt_cxxmplee
[task_prompt)
29

---

the following Published as a conference paper at ICLR 2024
F.4.2 SIMULATE LANGUAGE FEEDBACK
To instantiate the template for feedback generation, will replace (trajectory] with LLM agent's 's trajectory (e.g., $F.3). When the ground-truth solution is not provided for feedback gen- eration, {gt solution] will be substituted with "NOT GIVEN": Otherwise, the ground-truth solution for that task will be provided.
You are expert tasked with evaluating and providing feedback on assistant's 's performance.
Here is an example, Please follow the format as the following expert acts.
iin_contxx_exxapll
[tool_desc]
(trajectory)
Correct solution (please NOT disclose the correct solution to the assistant): (gt_solution)
Please provide concise and constructive feedback. Remember, your role is similar to teacher. Rather than giving away the solution details about the answer, guide the assistant toward understanding how
arrive at the correct answer. Your feedback should focus on enhancing the assistant's ability to think critically and reapond accurately. Now provide your feedback.
Expert teedback:
F,5 IN-CONTEXT EXAMPLE
We perform 1-shot learning all tasks in MINT. We have two types of in-context examples for each task: (1) without feedback, and (2) with feedback. We show the in-context examples with the provided feedback below. When feedback is provided the LLM (i.e., without feedback in-context examples), we will remove all occurrences of "Expert feedback: examples.
We curate one in-context example for code generation and reasoning problems, and we follow Yao al. (2022) construct examples for each sub-type of decision-making (i.e., ALFWorld) tasks. For each task, we include one corresponding example for in-context learning. For simplicity, we only include subset of prompts here, All prompts are released our code repo
F.5.1 DECISION- MAKINN
We modify in-context prompts from Yao al. (2022) to construct these in-context examples. This is an example of clean type of tasks in ALFWorld.
Task:
You are in the middle of room. Lecking quickly around you, you see
cabinet 13, a cabinet 12, cabinet 11, a cabinet 10, a cabinet 9, cabinet 8, a cabinet 7, a cabinet 6, cabinet 5, a cabinet 4, a cabinet 3, cabinet 2, a cabinet 1, coffeemachine 1, countertop 1, diningtable 1, a drawer 1, fridge 1, garbagecan 1, microwave 1, shelf 3, shelf 2, shelf 1, sinkbasin stoveburner 4, stoveourner 3, a stoveburner 2, stoveburner i, and a toaster 1.
Your task is to: put a clean lettuce in diningtable.
Assistant:
30

---

Published as a conference paper at ICLR 2024
<thought>
To solve the task, need find lettuce, then clean with sinkbasin, then put it in diningtable.
First I need to find a lettuce. A lettuce is more likely to appear in fridge (1), diningtable (1), sinkbasin (1), stoveburner (1-3), check by one.
cabinet (1-13). </thought>
<execute>
for src_to_checce in l'fridge 1", "diningtable 1",
"sinkbasin "stoveburner 1", "stoveburner 2", "stoveburner 3", [f*cabinet (i)" for i in range (1, 14)1):
res goto (cto(lo__t__chect) if "closed" in res:
The receptacle is closed. I need to open it to what " inside. res open_roceptacle (loc_to_check)
if "lettuce" in res: print (res) print (E"Found lettuce in (loc_to_checkk.") break
</ </execute>
Observation:
On the diningtable 1, you see apple 1, bread 1, a butterknife 2, a cup 2, fork 2, a knife 2, a knife 1, a ladle 1, a lettuce 1, a mug 2, mug 1, pan 2, peppershaker tomato 1.
spatula 3, tomato 2, and
Found a lettuce in diningtable 1.
Assistant:
<thought>
Now find Lettuce Next, need take go to sinkbasin and clean the lettuce 1.
Once it is cleaned, I need to put it in diningtable 1. </thought>
<execute>
print (take from prom("lettuee 1", "diningtable 1"))
print (goto ("sinkbasin 1*))
print ("laan" print (goto ("diningtable 1*))
1", "sinkbasin 1"))
print (put ("lettuce 1", "diningtable )"))
</oxecute>
Observation:
You pick up the lettuce 1 from the diningtable On the sinkbasin 1, you see a apple 2, a ladle 2, a spoon 1, and tonato 3.
You clean the lettuce using the sinkbasin 1. On the diningtable 1, you cup 2, fork 2, a knife 2, a knife 1, a ladle pan 2, peppershaker 1, a spatula 3, a tomato 2, and a tomato You put the lettuce 1 in/on the diningtable 1.
apple 1, bread 1, butterknife
mug 2, mug 1,
Assistant:
<thought> have comploted the task. </thought> <solution> done </solution>
F.5.2 CODE GENERATION
We select one example from MBPP's in-context example subset (Austin et al., 2021) and curate it to fit MINT framework (i.e., adding observations and feedback). We the in-context example
31

---

Published as a conference paper at ICLR 2024
after slight format adjustment for HumanEval (Chen et al., 2021) since there dedicated in-context examples provided for HumanEval.
Task:
Write a function to find the similar elements from the given two tuple lists.
assert similar_ee lar_srleeettssssceee (3, 5, 6), (5, 7, 4, 10)) (4,
Assistant:
<thought> The similar elements here are just elements that are in both lists. should run solutien the test before proposing solution. </thought>
<execute>
def similar_elements (test_tupl, test_tup2);
res = tuple(set (test_tup) set(test_tup2) _ return
res cemilar_elemente ((3, 4, 5, 6), (5, 7, 4, 10))
assert res = (4, 5), "Expected (4, 5) but got 1t" format (res) </execute>
Observation:
AssertionErrer
Traceback (most recent call
last)
Cell In[41, line
return res 4 res similar_elements (3,, 4, 5, 6), (5, 7, 4, 10))
5 assert res == (4, 5), "Expected (4, 5) but got ()..for format (res)
AssertionError: Expected (4, but got (3, 6, 7, 10)
Expert feedback:
This is BAD. You should not use the operator which is the union operator.
Assistant:
<thought> To find the similar elements, I should find the intersection of the two sets. should use the & operator instead of the operator. </thought>
<execute>
def similar_e satilarreelmmettse (test_ (test_tupl, test_tup2):
res = tuple (set (tost_tupl) & set (ctstttupup))) return res
res similar_clemente (33, 4, 5, 6), (5, 7, 4, 13))
assert res (4, 5), "Expected (4, 5) but got {1 format (res) </execute>
Observation:
[Executea Successfully with No output
Expert feedback:
This is GOOD. You are doing great! You can try to submit the solution now.
Assistant:
<thought> There is no more AsserticnError. I can now submit the solution. </thought>
<solution>
def eimilar_elements (test_tupl, test_tup2):
res = tuple (set (test_tupl) & (test_tup22)) return res
</solution>
32

---

Published as a conference paper at ICLR 2024
F.5.3 REASONING TASK
We manually curate the in-context example for the reasoning subset include the characteristic for both multi-hop QA-style reasoning (c.g., HotporQA) and mathematical reasoning (c.g.. MATH, TheoremQA).
Task:
Evaluate $lint\int_C (xty) '2 dx, dyš, where C is the circle defined by $x + y MS, and is the birth year of Ti Bai.
Output format requirement: Round three decimal places, using half up rouncing rule.
Assistant:
<thought> solve this problem, let's start by determining Bai's birth year. </thought>
<execute>
print seikip ('IA Bail))
</execute>
Observation:
Page: Li Bai
Summary: Li Bai (pinyin: LI Bái, 701(762), also proneunced Li Bo, courtesy
Taibai, was Chinese poet, acclaimed from his own time to the present as brilliant and romantic figure who took traditional poetic forms to new heights.
Expert feedback:
This is GOOD. You have successfully retrieved the birth year of Bai from the wiki search result. Next you should try to solve the calculus problem with the information you obtained.
Assistant:
<thought>
From the wiki_search result, we can see that Li Bai was born in 701, thus
= 701.
To evaluate the double integral (xxyy) 72$ over the region $CS, where $C$ is the cirele defined by $x 2 7015, we should use poiar coordinates.
In polar coordinates, $x - r'cos (\theta) $ and Sy rAsin (\theta) 8, and the differential area element $dx, dy$ becomes Sr, dr, d'thetas. flint\int_C (xty) "2 dx, \int_0" (2(pp)) \int (\sqrt(700)) (r\cos (\theta) risin (\theta)) "2r d'thetas Let's use the SymPy package to calculate this. </thought>
<execute>
from sympy import symbols, integrate, cos, sin, pi
r, theta - symbols('r theta')
the integrand
integrana (r*cos (theta) r+sin(theta))
perform the double integration double_integie inteerate (integrate integrand, (r, 0, 701 0.5)), (theta, 0, 2xpi)
print bcouble_interre </execute>
Observation:
Out [2]: 245700.5 5+pi
Expert feedback:
33

---

Published as a conference paper at ICLR 2024
This is GOOD. You successfully converted the double integral into form that can caleulated. Now you can convert answer to meet output format requirement.
Assistant:
<thought> Let's round this result three decimal places using half up rounding rule. </thought>
<execute>
import math
print (round (245700. 5+math.. 3))
</execute>
Observation:
771890.886
Expert feedback:
This is GOOD. You have got the sclution! You can try to submit the soluticn now.
Assistant:
The answer is <solution> 771890.886 </solution>.
F.6 EXAMPLE GROUND-TRUTH FOR DECISION-MAKING TASK
The following is an example ground-truth solution for task from Alfworld Shridhar et al. (2020),
Task:
Interact with household to solve a task.
You are in the middle of a room. Lecking quickly around you, you see a cabinet 6, cabinet 5, a cabinet 4, cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countertop 3, countertop 2, countertep 1, drawer 3, drawer 2, drawer garbagecan 1, microwave 1, shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, stoveburner 3, a stoveburner 2, stoveburner 1, and a toaster 1.
fridge 1,
Your task is put two peppershaker in drawer.
Assistant:
<execute>
("shee 1*)
goto ("shelf 2")
goto ("shelf 3")
goto ("drawer 1")
</execute>
Observation:
On the shelf 1, you see nothing. On the shelf 2, you see a bowl 1, and saltshaker 2. On the shelf 3, you see vase 2. The drawer is closed.
Assistant:
<execute>
open ("árawer 1")
close ("érawer 1")
2")
open ("drawer 2")
close ("drawer 2")
goto ("drawer 3")
open ("Grawer 3")
close ("drawer 3")
("ttoottteroo 1")
goto (ccountertop 2")
take "peppershaker 3", "countertop 2")
34

---

Published as a conference paper at ICLR 2024
goto ("drawer )
open ("drawer 1")
put ("peppprsshaker 3", "drawer 1*)
close("Grawe) (*drawer 1")
("the" 1")
(f 2")
go 3")
("ta("rrawrr 2")
open ("arawer 2")
close ("crawer 2")
goto ("árawer 3")
open (*drawer 3")
close ("drawer 3")
goto ("contccott 1")
goto (ccounteettoo 2")
("ccccnttertop 3")
goto (*cabinnt 1")
take peppprssaaerr 2", "cabinct 1")
goto @"to(""rawie ("drawer 1")
open ("drawer 1")
put ""eepershakerr 2", "drawer 1*)
</execute>
Observation:
You open the drawer 1. The drawer is open. In it, you see cellphone 4 1, and fork 1.
You close the drawer 1.
The drawer 2 is closed.
You open the drawer 2. The drawer is open. it, you nothing.
You close the drawer 2.
The drawer 3 is closed.
You open the drawer 3. The drawer is open. In it, you see spatula 2. You close the drawer 3.
On the countertop 1, you see cup 1, fork 2, mug 3, soapbottic a a soapbettle 2, a spatula 1, and a spoon 1.
On the countertop 2, you see a butterknife 1, a kni fe 1, mug 1,
peppershaker 3, and tomato 1.
You pick up the peppershaker closed.
from the countertop
The drawer
You open the drawer 1. The drawer is open. In it, you see cellphone
1, and fork 1.
You put the peppershaker 3 in/on the drawer 1.
You close the drawer On the shelf 1, you see nothing.
On the shelf 2, you see a bowl 1, and saltshaker 2.
On the shelf 3, you see vase 2.
The drawer is closed.
You open the draver 2. The drawer is open. In it, you see nothing. You close the drawer 2.
The drawer is closed.
You open the drawer 3. The drawer is open. In it, you You close the drawer On the countertop 1, you see a cup 1, fork 2, a mug 3, soapbottle 3, a scapbottle 2, a spatula 1, and a spoon 1. On the countertop 2, tomato 1.
spatula 2.
see butterknife 1, knife 1, mug 1, and
On the countertop 3, you see apple 2, a bread 1, a cellphone 3, a cellphone 2, a creditcard 1, a glassbottle 2, a houseplant 1, a plate 2, pot 2, a spatula 3, a speen 3, a spoon 2, and a statue 1. On the cabinet 1, you see mug 2, and a peppershaker 2. You pick up the peppershaker The drawer is closed.
from the cabinet
You open the drawer 1. The drawer is open. In it, you see a cellphone
1, fork 1, and a peppershaker 3. You put the peppershaker 2 in/on the drawer
35


Reviews:
[{"review_id": "9CEgNnu0D2", "reviewer": "Reviewer_VNEC", "summary": "This paper introduced MINT, a benchmark for evaluating LLMs when they can interact with external tools and natural language feedback. In the benchmark environments, \"tools\" are considered as functions that can be called by executing the LLM-generated Python code, and \"natural language feedback\" is simulated using GPT-4. The benchmark covers tasks such as math reasoning and code generation, where only instances that require multiple turns of interaction are retained. In experiments, a comprehensive set of open/closed-source LLMs are evaluated on MINT.", "questions": "1. Can the authors respond to my comment on the feedback simulation discrepancy?\n2. Can the authors share thoughts on reducing the benchmark cost?\n3. Can MINT be integrated with AgentBench (Liu et al., 2023) or InterCode (Yang et al., 2023), so that a larger task scope could be facilitated with both tool and user interaction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. LLM evaluation in an interactive environment is an important and novel topic.\n2. Experiments on a comprehensive set of LLMs under different settings (base vs. SIFT vs. RLHF, open-source vs. closed-source) were conducted, resulting in several interesting observations (e.g., RLHF may hurt LLM-tool interaction while SIFT helps).\n3. The paper is easy to follow.", "weaknesses": "My major concerns lie in the natural language feedback simulation setup:\n1. The example in Figure 1 seems to assume a \"collaborative work\" interaction between the LLM and the user, as opposed to the more common case where a user seeks assistance from the LLM; in the latter case, it is not practical to assume such detailed feedback from the user because the user does not know how to solve the problem (otherwise they have no need to seek help from the LLM). In some sense, the current LLM-user interaction has made the user a \"teacher\" who instructs the LLM to correct its mistakes.\n\n    In fact, user feedback has been widely studied in code generation/semantic parsing tasks (references below, which are missing in related work discussion). Prior work has particularly focused on collecting or simulating natural language feedback that is close to \"what a prospective, help-seeking user in practice could give\". While MINT's feedback interaction is still valuable (as it assesses how well an LLM can incorporate the feedback, no matter if it is realistic or not), I'd like to note that its kind of feedback may not reflect the realistic case, and this should be clarified in the paper. \n\n    References:\n    - Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5447–5458, Hong Kong, China. Association for Computational Linguistics.\n    - Ahmed Elgohary, Saghar Hosseini, and Ahmed Hassan Awadallah. 2020. Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2065–2077, Online. Association for Computational Linguistics.\n    - Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen-tau Yih, and Ziyu Yao. 2023. Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3149–3170, Toronto, Canada. Association for Computational Linguistics.\n\n2. The human evaluation of GPT-4 generated feedback does not look at the discrepancy mentioned above either.\n\n3. As indicated in footnote 4, evaluating one LLM using MINT costs around 100 USD, due to the use of GPT-4 for feedback simulation. This cost can be an issue for iteratively improving (and then evaluating) an LLM. \n\nOther than the feedback simulation, another comment on the tool interaction:\n\n4. The current tool use has been limited to Python function calls. It offers better reproducibility but also loses the benchmark scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vy2Vm0DOuv", "reviewer": "Reviewer_vnHy", "summary": "The paper investigates the capabilities and limitations of large language models (LLMs) when solving tasks with multi-turn interactions by using tools and leveraging natural language feedback. The main contributions of the paper can be summarized as follows:\n(1) The authors propose a structured evaluation framework where LLMs can leverage tools via executing Python code and receive users' natural language feedback;\n(2) The authors have conducted abundant experiments, analyzing 20 open- and closed-source LLMs over 8 datasets.\n(3) From the experiments, the authors have made some interesting observations, like RLHF may hurt multi-turn interaction performances in problem solving.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors introduce MINT, a novel benchmark and framework, that evaluates LLMs' capabilities to solve tasks with multi-turn interactions with tool usage and natural language feedback.\n- The authors conduct abundant experiments, covering 20 open- and closed-source LLMs and 8 datasets from different tasks.\n- The authors have made some interesting statements, including (1) better single-turn performance does not guarantee multi-term performance; (2) RLHF may hurt the multi-turn capabilities. Corresponding experimental results and analysis are included to support the claims.\n- The paper is well-written and is easy-to-follow. The visualizations are also clear and can help readers better understand the framework and model performance comparisons.", "weaknesses": "Overall, this is a very thorough and comprehensive benchmark paper. I have only several concerns: \n- The authors claim that the SIFT can benefit models' capabilities of tool-augmented task-solving in multi-turn interaction (section 3.2), while the authors also claim that the SIFT can hurt models' ability to leverage feedback (Section 3.3). Why do these two claims seem to contradict each other? \n- I am also very curious about the trade-off between tool-use capabilities and abilities to leverage human feedback. SIFT on general domain multi-turn interaction data (like ShareGPT, in section 3.2) can bring performance gain in problem-solving. I think this might be because of the multi-turn feature that caters to the final application. I also think if the model can SIFT on some task-specific data (if exists), the performance can also improve. I am very curious about which is more important for SIFT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reF1GqgtZB", "reviewer": "Reviewer_hgTu", "summary": "This paper presents MINT, a benchmark that evaluates LLMs’ multi-turn interaction capabilities, in particular when LLMs are using external tools and provided with natural language feedback from users (simulated by GPT-4). The experiments use a subset of existing evaluation datasets on reasoning, coding, and decision-making tasks, resulting in 586 instances, which require multi-turn interaction (judged by GPT-3.5). \nThe authors compare 20 open- and close-sourced LLMs on these instances and present 7 findings (listing them in order to easily refer to them in the other parts of the review):  \n\n1. All models benefit from tool interaction and natural language feedback\n2. Better single-turn performance does not lead to better multi-turn performance\n3. There is a notable gap between open- and closed-source LLMs in multi-turn interaction performance\n4. Models trained with supervised instruction fine-tuning (SIFT) and reinforcement learning from human feedback (RLHF) perform worse in multi-turn settings \n\nFrom the text:\n\n5. LLMs’ feedback-providing capability could be orthogonal to task-solving ability\n6. MINT’s evaluation reveals undesired artifacts in ShareGPT data\n7. GPT-4 simulated language feedback is as helpful as human-written feedback based on human evaluation and task performance.", "questions": "* Most of the questions that can change my opinion are described in the weaknesses. I'm generally excited about the direction of the paper, and with clarification and contextualization of this work’s contributions and findings with respect to prior/concurrent work, I’d be happy to reconsider and adjust my ratings. Here, I list some questions/comments I had while reading the paper.\n* Maybe another bit that’s quite important to convey nuanced findings in the paper is the comparison of GPT-4-generated feedback to human-written feedback. Although the surface-level similarity and helpfulness are similar, my guess is that GPT-4 generated feedback lacks the variability that we’d normally observe in human-written feedback, which can easily affect these models’ performance (e.g., Lee et al., 2023). The limitation section in the appendix mentions coverage of GPT-4 generated feedback, which is relevant but different from this point.\n* Similarly, although I generally agree that this is a step towards including more real-world-like elements compared to evaluating LLMs without tool use and natural language feedback, I wouldn’t say that the current setup is strictly “more aligned with real-world applications” than prior work as there is a clear trade-off due to simulating human-written feedback with LLMs and potential drawbacks with the simulation (e.g., Koo et al. (2023), Rajani et al. (2023)).\n* In finding #3, could the size of a model be the main confounder? Is the finding only applicable to multi-turn settings or also in single-turn settings?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* The work studies a **timely, important topic** of LLMs’ evaluation in interactive settings, striving to bridge the gap between traditional static/single-turn benchmarks in NLP and “real-world use cases” (as noted in abstract). The use of LLMs as a proxy for human feedback reduces cost significantly and increases scalability.\n* The paper is overall **well-written** and the design of the experiments seem **sound**. In addition, I find the **details of the experiments** are properly documented either in the main paper or appendix.\n* Even if the exact numbers in the findings are likely to become less relevant in the future due to the fast-changing landscape of LLMs, the **high-level findings and takeaways are likely to be still relevant**, especially about single-turn vs. multi-turn performance between models (e.g., #2 and #4).", "weaknesses": "* Although the paper presents many interesting findings, it wasn’t clear to me which ones are **new findings vs. confirmation of existing findings**. For instance, #1 has been shown by ToolLLM (Qin et al., 2023), Tool Augmented Language Models (TALM) (Parisi et al., 2022), Chain of Hindsight (Liu et al., 2023), etc.;  and #2, #3, and #4 have been (fully or partially) shown by Human-AI Language-based Interaction Evaluation (HALIE) (Lee et al., 2023); and #7 has been studied by Liu et al. (2023), Chiang & Lee (2023), Gao et al (2023), Shen et al (2023), and Duan et al. (2023). Situating the findings in the existing literature will give this work more validity and help readers understand the contributions of the paper.\n* For that reason, it is a bit unclear to me what the **main contribution of this paper** is at the moment. In comparison to prior work, the authors note that “Different from prior work, MINT covers a range of diverse tasks and is designed to measure the multi-turn interaction capabilities of LLMs with both tools and user feedback that are more aligned with real-world applications.” As some of the prior work covers a range of diverse tasks/measures multi-turn interaction capabilities of LLMs/simulates human feedback with LLMs, the main difference of this work is (presumably) accounting for **the use of tools and user feedback** on top of all of these elements. In this case, I wonder if it'll help to scope the description of the tasks more tightly (e.g., tasks that can benefit from tool use, as opposed to open-ended generation) and provide a bit more justification for why it’s important to look at these elements altogether (as there are prior work looking at the tool use and NL feedback) and what we expect to see (do we expect to see different results when these elements are added? do we actually observe any interesting interplay between these elements or is it merely a simple performance boost across all models?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BRLa6JZwBx", "reviewer": "Reviewer_Nxjw", "summary": "This paper evaluates LLMs in multi-turn interaction with the help of tools and language feedback. The authors introduces the MINT benchmark composing of code generation, decision making, and reasoning tasks, and compared 20 open- and closed-source LLMs. Results show that LLMs benefit from multiple rounds of tool-use and language feedback, and LLMs trained with supervised instruction-finetuning (SIFT) and RLHF may hurt multi-turn performance. Moreover, better single-turn performance does not generalize to better multi-turn performance.", "questions": "1. Where did you compare to the \"Lazy User-LLM Interaction\"? Or is this only for k=1?\n2. The proposed method is very similar to the Tree-of-thought paper (Yao et al., 2023) where the authors used LMs as a feedback/reward provider.\n\nYao et al., 2023. Deliberate Problem Solving with Large Language Models.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes an interesting MINT benchmark to evaluate multi-turn interactions using code execution and language feedbacks. The modification of using a unified subset of examples across a bundle of tasks would be useful for the community in general.\n2. Results showing that supervised instruction-finetuning and RLHF may hurt model performance of multi-turn interactions with tools and language feedback, and worse performance on multi-turn compared to single-turn, may suggest that multi-turn interaction data is required. This provides valuable suggestions to alignment research.", "weaknesses": "1. There is no comparison to simple baselines, such as self-critic (which is essentially a special case of multi-turn interactions defined in this paper). Other baselines include only providing binary feedback, or sample k times and ask a LLM to select the final answer. Without such baselines, it is not clear how much language feedback the model actually incorporates in. Furthermore, I generally like the idea t hat using a LLM to provide feedback can improve a model's performance in general, but can the authors provide any benefit (such as efficiency) comparing using a large LLM as a feedback provider, compared to using a LLM to generate the results directly (either using self-critic or not)?\n2. The proposed MINT dataset is relatively biased towards math problems that specifically require code execution and can benefit from explicit human feedback (e.g., correct or incorrect).  It seems that the only tool used is the python execution tool. Compared to other papers leveraging tools (toolformer, ReAct), the claim of utilizing tools through turns would improve model performance is not very convincing, especially for tasks that may not require code execution (e.g., HotpotQA). Moreover, I agree that selecting a subset of datasets would hinder fair comparisons, but sampling for instances, only 43 examples for HotpotQA (which is arguably the only multihop-qa dataset in MINT) may not be statistically large enough to represent the performance on the dataset. More importantly, MINT is limited to multi-turn interactions using the specific code tools and language feedback used in this paper, not the more natural and systematic tasks that require multi-turn interactions.\n3. Accordingly, results suggesting that for example, \"better single-turn performance does not guarantee better multi-turn performance\" and \"RLHG may hurt multi-turn capabilities\" are limited to the setup that those worse performance would largely be due to \"less viable of incorporating feedback\" rather than necessarily indicating that they are indeed worse on multi-turn interactions with agents in general. Therefore, the claims are not well justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)