Retrospective Learning from Interactions

## Abstract

Abstract Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. This creates an avenue for continually learning from interactions without additional annotations. We introduce ReSpect , a method to learn from such signals in past interactions via retrospection. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct an LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.

## 1 Introduction

Figure: Figure 1: Learning via ReSpect. We deploy an LLM policy $\pi_{\theta_{\rho}}(a|x)$ in rounds $\rho$, to interact with users in multi-turn interactions. Following each round, the LLM reasons retrospectively about each of its actions (highlighted in blue) to decode feedback given the interaction context, including follow up utterances. After each round, the model is retrained using all data aggregated so far $D_{\leq\rho}$. The LLM improves over time without any external annotations. The plot on the right shows the performance curve in our experiments – the LLM improves from 31% to 82% task completion rate over six rounds.
Refer to caption: x1.png

Language models (LMs) often engage in multi-turn interactions with human users.
Similar to human-human interactions, these interactions are naturally rich with implicit learning signals.
If the LM fails to respond appropriately, the user is likely to follow with an expression of frustration, a rephrase of their intent, or maybe even completely pivot what they ask for.
Similarly, if the LM does well, the user may express approval or simply continue to their next objective.
Such responses can inform the LM of its performance, thereby creating an opportunity to learn through retrospection.

We study the efficacy of such signals, and how they can lead to a system that improves over time.
We introduce ReSpect, a simple approach to learn from signals the model itself derives about its own past actions through retrospection of past interactions with human users.
We deploy ReSpect in MultiRef, a new multi-turn grounded interaction scenario, which requires models to display complex abstract reasoning, and humans to gradually instruct models to accomplish sequences of goals to complete their tasks.

The key insight underlying ReSpect is that conversational implicit feedback signals occupy a relatively constrained subspace of natural language.
Such signals can include direct approvals (e.g., great!) or signs of frustration (e.g., not again), and also more subtle cues, such as when the user rephrases their request.
Critically, it is relatively simple to disentangle them from task performance.
A human can easily figure out from such cues if they do well or not, even if they have little understanding about what they are asked for.
It is this constrained nature that makes reasoning about such signals to be within the capacities of large language models (LLMs), even if they fail at the task at hand.

ReSpect utilizes this signal in a process where the model interacts with humans, and after interaction decodes feedback for each of its actions from the interaction context including the follow up utterances.
Figure 1 illustrates this process.
The model interacts with humans to accomplish tasks, retrospectively examines its own past interactions, and then re-trains.
This process progresses in rounds, alternating between interaction and training, with the model improving over time.
Critically, unlike common recipes for training from human feedback, ReSpect does not require any external annotation (Ouyang et al., , RLHF) or even soliciting feedback from the users themselves (Suhr & Artzi, ).

We deploy ReSpect in MultiRef over multiple rounds of grounded interactions with human use and re-training. We use IDEFICS2-8B (Laurençon et al., ) as our LLM, and experiment with multiple learning methods, including supervised learning, REINFORCE-style policy gradient (Williams, ; Kojima et al., ), and KTO (Ethayarajh et al., ).
Across our experiments, we observe that IDEFICS2-8B effectively decodes feedback, even as it initially performs poorly in the same interactions.
In our longest running experiment, we observe model task completion rate improves from 31% to 82%.
Our code, data, and models are at [https://lil-lab.github.io/respect](https://lil-lab.github.io/respect).

## 2 Technical Overview and Notation

We conduct continual learning studies by deploying our approach in MultiRef, a new multi-turn grounded interaction scenario (Section 3).
Overall, the study progresses in rounds, where the LLM policy is first deployed to interact with users and complete tasks, and the interactions are then used to re-train the policy. Our study involves multiple rounds, and our goal is to observe and evaluate the long-term dynamics of the process.
This includes the robustness of our award decoding and training methods to the changing distribution of the data likely to be seen in an adaptive system in the wild.
Section 3 describes our interaction scenario in detail, and Section 4 our learning method.
First, we outline our problem of interest and its notation in abstract terms.

##### Task Notation

The policy’s task is to respond effectively to human utterances given in conversational context.
Formally, let $\pi(a_{t}|x_{t})$ be the policy that controls the listener behavior, with $a_{t}$ an action string that represents the model response and $x_{t}$ being the context on which the policy is conditioned, both at time $t$ in the interaction.
The context includes the instruction history up to and excluding time $t$, including current (i.e., at time $t-1$) and past speaker utterances, as well as any other relevant context in which the interaction takes place.
As our learning progresses in rounds, we denote $\theta_{\rho}$ as the model parameters in round $\rho$, and $\pi_{\theta_{\rho}}$ as the parameterized policy.

##### Learning and Deployment

We study a continual learning setup, where the learning signal is acquired from interactions of the deployed model with human speakers.
Our study progresses in rounds (Figure 1). Each round $\rho$ includes a deployment, followed by training.
During deployment at round $\rho$, the model $\pi_{\theta_{\rho}}$ interacts with users.
For each model action $\hat{a}_{t}\sim\pi_{\theta_{\rho}}(a|x_{t})$, we record a tuple $(x_{t},\hat{a}_{t},p_{t},\bar{f}_{t})$, where $x_{t}$ is the context given to the model at time $t$ to predict action $\hat{a}_{t}$, $p_{t}$ is the probability of $\hat{a}_{t}$ at the time of prediction, and $\bar{f}_{t}$ is the remainder of the interaction following $\hat{a}_{t}$.
Critically, these interaction tuples contain no explicit feedback.
We compute the implicit feedback $\hat{\gamma}_{t}$ using a feedback decoder $\phi(x_{t},\hat{a}_{t},\bar{f}_{t})$, to obtain tuples $(x_{t},\hat{a}_{t},\hat{\gamma}_{t},p_{t})$.
We experiment with multiple learning objectives using this feedback: supervised learning (SFT), policy gradient, and KTO.

##### Evaluation

We measure the quality of the listener model $\pi_{\theta_{\rho}}(a_{t}|x_{t})$ at each round $\rho$ primarily by interaction success rates from live human-bot deployments. The same interactions are used to train the model for the next round.
We track various characteristics of model behavior, such as number of turns per interaction as an efficiency measure.
We also do post-hoc annotation of a subset of the interactions to measure utterance-level policy success rate and feedback decoder accuracy.

## 3 MultiRef : a Multi-turn Grounded Interaction Scenario

Figure: Figure 2: The interaction scenario we use in our experiments. MultiRef is a multi-turn reference game. A speaker and a listener both observe a shared set of tangram shapes, but in different order. The goal of the speaker is to describe a subset of targets for the listener to select. Because the target requires multiple abstract shapes, humans often communicate the targets gradually over multiple turns. As an interaction progresses naturally, the speaker produces implicit feedback signals that validate or reject the listener’s actions.
Refer to caption: x2.png

Key to our study is that tasks are relayed gradually across multiple turns, as commonly happens in human interactions.
We create MultiRef, a conversational interaction scenario where two partners, a *speaker* and a *listener*, coordinate on the selection of a set of items.
In our studies, the speaker is always a human, and the listener is a model.

MultiRef generalizes the commonly studied reference game scenario. Its design and our choice of stimuli are grounded in existing work from both cognitive science and computational language modeling (Rosenberg & Cohen, ; Clark & Wilkes-Gibbs, ; Schober & Clark, ; Goodman & Frank, ).
Figure 2 illustrates the scenario.
Both partners observe a shared set of images, but in different order.
The speaker is given a subset of the images as targets, with the goal of communicating the targets to the listener, so the latter selects the exact subset.
Only the speaker can write messages, and only the listener can select or deselect images.
The interaction concludes successfully once all and only targets are selected, or fails if the partners run out of turns, 20 in our studies.

The interaction progresses in turns $t$, alternating between speaker and listener turns.
At each speaker turn, they provide a single unrestricted natural language utterance.
It may direct the listener to select one or more items, ask to deselect items if the listener previously made a mistake, or include whatever other content they desire.
This utterance as well as the history of the interaction, the set of images, and their selection status compose the context $x_{t}$ for the following model turn at time $t$.
The follower responds with an action $a_{t}$, which includes one or more image selects or deselects according to their understanding of the speaker intention.
The action space consists of all possible legal sequences of the form Deselect E select F or Select D G assuming images are code-named alphabetically.

The motivation behind the design of MultiRef is to create a task-oriented scenario that is both accessible to non-expert humans and encourages constructing a solution in multiple turns, thereby creating multi-turn interactions and eliciting the learning signals we aim to study.
The rules of the interaction are simple: the speaker needs to describe targets to select, and the listener has to select what the speaker is referring to.
This makes MultiRef easily accessible to crowdsourcing workers.
At the same time, the solution the speaker communicates to the listener is relatively complex, because of the enormous solution space:
Consider a conventional reference games, where the goal is to select a single image. The number of possible solutions is the number of images in the context.
In MultiRef, the goal is to select a subset of unknown size, so the combinatorial solution space the listener faces is exponential in the number of images.
At the same time, the solution itself is structured, creating easy opportunities to decompose it to multiple steps.

Key to making MultiRef work well is the choice of images.
We use tangram shapes from the diverse KiloGram dataset (Ji et al., ). Tangrams are abstract shapes that are designed to elicit common concepts in humans.
This abstractness often leads to ambiguous descriptions open to interpretation.
For example, Shape A in Figure 2 can be described as a bat, a lowercase w, or even a star wars star fighter.
We select tangrams because they naturally provide an ambiguous and challenging stimuli for human interaction (Clark & Wilkes-Gibbs, ; Schober & Clark, ; Fox Tree, ; Hawkins et al., ), thereby leading to highly diverse language. They also remain challenging for contemporary multimodal LLMs (MLLMs) to reason about (Ji et al., ), leaving significant room for learning.

The free-form natural language human speakers produce in MultiRef is very diverse, and balances between competing pressures.
First, it often requires complex pragmatic reasoning (Clark & Wilkes-Gibbs, ; Schober & Clark, ; Horton & Gerrig, ), because of the abstractness of tangrams.
This is compounded by how the combinatorial solution space drives humans to balance between relaying as much information as possible, and relaying clear objectives to make gradual progress.
This is a balance between two Gricean maxims: quantity and manner.(^1^11Grice’s maxim of quantity: one tries to be as informative as one possibly can, and gives as much information as is needed, and no more; Grice’s maxim of manner: one tries to be as clear, as brief, and as orderly as one can in what one says, and where one avoids obscurity and ambiguity. (Grice, ))
Speakers may or may not include explicit feedback such as good, or deselect the last one; the speaker may describe more than one target in a single utterance, for example, select two men; speakers may refer to previous selections without directly describing targets, for example, the other one, or try again.
In combination with the abstract stimuli tangrams provide, this creates a challenging reasoning problem for the listener model.

MultiRef is not designed to increase complexity in arbitrary ways, but to provide an environment for humans to naturally expose core aspects of human communication.
At the same time, the scenario is both controlled and scoped, allowing for easy measurement of task completion and progress, as well as making learning feasible with relatively limited data.
This makes MultiRef particularly suitable to research in academia or other low-resource settings.

## 4 ReSpect : Retrospective Learning from Past Interactions

ReSpect has two components: decoding implicit feedback from past interactions (*retrospection*) and learning from the decoded feedback signals (*learning*).
We deploy ReSpect in an iterative continual learning scenario, where each round includes both steps. This deployment allows us to observe the dynamics of ReSpect over time.
However, the method itself is not limited to continual learning, and can be applied a single time as well.

The goal of ReSpect is to re-estimate the parameters of a model given interactions that were collected by the model itself, or previous versions of it.
We assume access to a raw dataset $D^{\text{raw}}=\{(x^{(i)},\hat{a}^{(i)},p^{(i)},\bar{f}^{(i)})\}_{i=1}^{N}$, where $x^{(i)}$ is the policy context, $\hat{a}^{(i)}$ is the predicted action, $p^{(i)}$ is the probability of this action, and $\bar{f}^{(i)}$ is the remainder of the interaction following $\hat{a}^{(i)}$.(^2^22For simplicity of notation, we omit the time step in this section.)
In our continual learning setup, $D^{\textrm{raw}}$ is a union of all data collected from past rounds.

The feedback decoder $\phi$ computes a categorical feedback $\hat{\gamma}^{(i)}\in\{\text{{positive}, {neutral}, {negative}}\}$ for each action $\hat{a}^{(i)}$ holistically based its context $x^{(i)}$, action taken $\hat{a}^{(i)}$, follow up utterances $\bar{f}^{(i)}$.(^3^33We do not compute feedback for the last action in each interaction because there is not followup interaction. For simplicity, $D^{\textrm{raw}}$ does not include them.)
This process transforms $D^{\textrm{raw}}$ to $D=\{(x^{(i)},\hat{a}^{(i)},p^{(i)},\hat{\gamma}^{(i)})\}_{i=1}^{N}$.
We use this dataset for training.

Figure: Figure 3: The prompt used to decode feedback from past interactions. The figure combines the prompts for both binary and ternary feedback decoding. The parts that belong to the binary case only are colored green, while parts that belong the ternary case are colored orange. The verbal feedback generated by the model is in bold. Additional *comments for readability* are in magenta italics.

### 4.1 Decoding Implicit Feedback through Retrospection

We implement the feedback decoder $\phi$ by prompting the model to analyze past interaction tuples $(x,\hat{a},p,\bar{f})$ to compute feedback $\hat{\gamma}=\phi(x,\hat{a},\bar{f})$.
The goal is a process where the model bootstraps from its own interactions.
Our hypothesis is that LLMs have the ability to reason about the relatively constrained space of implicit signals, even if they fail at the task.
We show this empirically in our experiments.
Critically, this process does not rely on a stronger LLM for critique or on past interactions created by other LLMs.
Figure 3 shows the decoder prompt.
We experiment with binary or ternary feedback.
Ternary adds neutral on top of the positive and negative binary options.
The feedback decoder is designed to identify general linguistic cues, and not for the specific task we study.
We assume no access to any auxiliary annotation or privileged information (e.g., not inferring based on whether the policy selects a ground truth target in a turn, or whether an entire interaction ends early), although they are likely to be useful signals as explored in Pang et al. ().

### 4.2 Learning

The feedback decoding process transformers the dataset from $D^{\textrm{raw}}$ to $D=\{(x^{(i)},\hat{a}^{(i)},p^{(i)},\hat{\gamma}^{(i)})\}_{i=1}^{N}$.
We study several learning approaches using this data: supervised learning, offline reinforcement learning (RL), or the KTO-style utility maximization (Ethayarajh et al., ).

##### Supervised learning

We fine-tune on positive data points ($\hat{\gamma}^{(i)}=\texttt{positive}$) and discard data points predicted as neutral or negative. We use cross entropy loss with additional label smoothing to prevent overfitting and encourage exploration.
Our setup is distinct from conventional supervised learning in that the data is coming from the model interactions (i.e., on-policy), and not from a given dataset. Also, we run the learning process iteratively, each time with more data.
We do not design the supervised approach in any special way to fit these changes, but this is a potential avenue for future work, which can further improve performance.

##### Reinforcement Learning

We follow prior work (Kojima et al., ) and use simple REINFORCE-style policy gradient (Williams, ).
The categorical feedback $\gamma^{(i)}$ (i.e., the text generated by the prompted LLM) is mapped to a numerical value with a simple reward function:

$$ $R(\gamma)=\begin{cases}1,&\gamma=\texttt{positive}\\ 0,&\gamma=\texttt{neutral}\\ -0.1,&\gamma=\texttt{negative}\end{cases}\;\;.$ (1) $$

Dropping the $i$-superscripts for simplicity, the gradient estimator for a single example is:

$$ $\Delta=cR(\hat{\gamma})\nabla\text{log}P(\hat{a}|x;\theta_{\rho+1})\quad\quad \quad c=\begin{cases}1,&\text{if }R(\hat{\gamma})\geq 0\\ \frac{P(\hat{a}|x;\theta_{\rho+1})}{p},&\text{if }R(\hat{\gamma})<0\;\;,\end{cases}$ (2) $$

where the coefficient $c$ downweights examples with negative reward by their inverse propensity score (Kojima et al., ). This is critical because $\lim_{P(\cdot)\rightarrow 0}\text{log}P(\cdot)=-\infty$.
In practice, we also discard data points with predicted neutral feedback ($R(\hat{\gamma})=0$).

We choose REINFORCE for its simplicity.
The positive case reduces to be mathematically equivalent to the gradient of supervised fine-tuning (SFT), whose optimization is relatively well understood. As opposed to other methods, such as PPO (Schulman et al., ), REINFORCE does not require a reward model and has relatively few hyperparameters.
This is critical with human-in-the-loop experiments, where broad parameter sweeps are not possible.
Recent work (Ahmadian et al., ) also suggests REINFORCE can produce on-par results in LLMs with PPO despite its simplicity.

##### Utility Maximization

To experiment with utility maximization, we use Kahneman-Tversky Optimization (Ethayarajh et al., ).
KTO was developed to learn from per-example binary human feedback, a scenario that fits ours well. We consider examples with decoded positive feedback as *desired* utterances, those with decoded negative feedback as *undesired*, and discard those with neutral feedback.
We refer readers to Ethayarajh et al. () for the definition of the objective.

## 5 Experimental setup

##### Interaction Instantiation

We use the KiloGram (Ji et al., ) tangram images, following Gul & Artzi ().
KiloGram contains 1,013 images. We randomly split them into a main split (912 tangrams) and a development split (101 tangrams).
We create interaction contexts by randomly sampling 10 tangrams, and randomly select 3–5 as targets.
The development split is exclusively used for seeding the initial listener policy $\pi_{\theta_{0}}$, and all human-bot interactions are conducted on images from the main split, i.e., tangrams that the seed policy $\pi_{\theta_{0}}$ has never seen before.

##### Model and Initialization

We use IDEFICS2-8B (Laurençon et al., ) as our model for both the policy and feedback decoder.
We fine-tune with LoRA (Hu et al., ).
We seed the initial policy $\pi_{\theta_{0}}$ by fine-tuning the pretrained IDEFICS2-8B weights on a small supervised dataset of 90 successful turns from 25 human-human games constructed with the development split tangrams, augmented with 12 synthetically generated deselection turns, because while necessary for human-model interactions, deselections are rare in human-human interactions (Appendix B.2).
$D_{0}$ is reused in continual training via rehearsal.
We validate our design online with 30 main-split human-bot pilot interactions, or offline with a validation set of 344 successful main-split human-human turns (Appendix A).
We use the original IDEFICS2-8B for feedback decoding, because the narrow focus of our data is likely to inhibit some general linguistic knowledge. This means we cannot see improvement in the model feedback decoding capability, likely low-balling the potential of the approach. It remains an important direction for future work to keep the decoder model in sync with the policy. This requires deployments that include high domain diversity.
We observe the original IDEFICS2-8B to provide robust feedback decoding out of the box, confirming our hypothesis, and providing a solid ground for our experiments.

##### System variants

We study six system variants based on two dimensions: (a) feedback decoder configuration (binary vs. ternary); (b) optimization methods (supervised vs. REINFORCE vs. KTO):

- •
b-sup and t-sup binary (b) / ternary (t) that only trains on positive data points with a supervised fine-tuning objective (sup).
- •
b-rl and t-rl trains on both positive and negative data points using REINFORCE.
- •
b-kto and t-kto are like b-rl and t-rl, but using KTO.

For variants involving negative data points (b-rl, t-rl, b-kto, and t-kto), we subsample negative ones to keep the positive:negative ratio close to 5:4  (Ethayarajh et al., ).

##### Deployment

We conduct three rounds of training-deployment for all six systems and three more rounds for b-sup. We select b-sup for another three rounds because it is the most promising variant after three rounds, and we want to observe its progress over a longer period.
The reason for this cascaded design is the high cost of experiments.
We do not distinguish between training and evaluation in the traditional sense. Instead, all listener policies are evaluated live on MTurk on about 330 human-bot interactions each round containing roughly 2400 turns. Then the same data is used to train the next iteration of policies respectively. The policies in the same round are deployed concurrently in a randomized experiment on the same set of games to mitigate human biases and variances due to game difficulty. More details on crowdsourcing are in Appendix A.3.

##### Learning Implementation Details

We use the validation set for model selection throughout continual learning.
Following prior work (Misra et al., ; Müller et al., ; Liu et al., ), we add an entropy term and length normalization to all three objectives to reduce over-fitting given the relatively small amount of data.
Appendix B provides additional reproducibility details.
Unlike with REINFORCE, where we train from scratch each round, when using KTO, we continually fine-tune from a previous model checkpoint $\theta_{\rho}$ to obtain $\theta_{\rho+1}$ with data accumulation. This was shown to outperform training from scratch in pilot studies.

##### Evaluation

We evaluate each system variant at each round by the success rate during the live deployment.
We report both interaction- and utterance-level success rates.
The interaction level success rate is straightforward - whether the game ended with all targets selected by the listener and nothing else.
The utterance level success rate is more nuanced because we do not have access to the ground truth, i.e., the intended action.
We sample 1,000 utterances per round from b-sup to annotate by MTurk workers post hoc.
We report two measures: exact match between the annotation and model action and similarity score, which is based on the computed similarity between the tangrams selected or deselected during the turn by the human annotator and the system.
We also evaluate the quality of the feedback decoder by comparing its predictions with human interpretations collected during the post-hoc annotation.
Because of cost, we cannot do post-hoc annotation for all system variants, so we also report click accuracy, which approximates utterance-level performance well. It measures the ratio of the model actions that lead to selection statuses that do not violate the set of targets (i.e., selections of target tangrams are good, deselection of non-target tangrams are good).
Lastly, we track the number of turns per interaction.
Appendix B.4 provides full definitions of our metrics.

## 6 Results and analysis

We deploy our models for three rounds, with additional three rounds for b-sup, the best-performing variant, to better understand long-term dynamics.
All our results are from concurrent randomized deployment, where the models interact with humans in real time.
We collected a total of 7,230 interactions consisting of 55,004 utterances over 4 weeks, at a cost of $11,180 USD.

Figure: Figure 4: Task performance and efficiency improve as the policy learns from more past interactions. We present deployment results across three rounds for six concurrent systems, and three more rounds for the top system b-sup, together with human-human references (hh) and a redeployment of the initial policy $\pi_{\theta_{0}}$ (control). *Left:* interaction-level success rate ($\uparrow$, higher is better). *Center:* interaction-level efficiency by # turns per interactions ($\downarrow$). *Right:* micro-level performance by click accuracy ($\uparrow$). Shades are 95% confidence intervals by bootstrapping with 10,000 resamples.
Refer to caption: x3.png

Figure 5 shows the deployment statistics for all six system variants, as well as control deployments for the initial policy and human-human games.(^4^44We present results in rounds for simplicity. Appendix C connects rounds to cumulative number of interactions.)
Figure 5 shows utterance-level statistics for b-sup from the post-hoc annotations we collected.
The interaction success rate of all systems improves monotonically in the first three rounds, except for b-kto in round three. We conduct three more rounds with b-sup, the leading system after the first three rounds. b-sup then plateaus, and even shows a temporary decrease in performance, before resuming its improvement.(^5^55The reasons behind the plateau are hard to infer. One hypothesis we considered is that changes in the amount of data over time made some settings sub-optimal. Specifically, we considered our LoRA adapter settings, as they impact the expressiveness of fine-tuning. We conducted a separated deployment, branching out from round three for two rounds (four and five) with b-sup and more expressive adapters. We observed this increase in expressivity allows the model to continue its monotonous improvement. Appendix D provides the details. This mini experiment illustrates the complexities of continual learning with current learning systems.)

Overall, b-sup improves interaction-level success rate by 51% (31%$\rightarrow$82%) and utterance-level exact match by 22% (31%$\rightarrow$53%).
At the last round, following the plateau, b-sup interaction success rate improves by 5% (77%$\rightarrow$82%).
The number of turns follows these trends. As the policy gets better, more games are completed within the allotted number of turns, and even faster.
b-sup starts with 8.9 turns per game, and concludes with 6.7 per game.
The center panel of Figure 5 shows that actions taken by the policy increasingly resemble human actions, even mistakes (actions that receive negative feedback) become more similar to human actions.
All other statistics largely track these, except some of the utterance-level statistics around when b-sup plateaus.
While all show a deviation from the monotonous earlier trend, some show a temporary decrease and not just a stagnation, but delayed by one round.
This illustrates the complex dynamics of continual learning, which we explore in more detail below.

There remains a significant gap between b-sup (our leading system) and hh (human-human interactions), which shows perfect task success rate and almost double efficiency (i.e., tasks are completed with much shorter interactions). Our intuition is that the gap is due to the lack of long-term credit assignment in our learning method. This is especially influential in learning to reasons about later turns.
Later turns shows much stronger dependence on earlier turns, creating a more complex reasoning problem and a harder credit assignment problem with the longer context.
This learning challenge is compounded by the data scarcity: we have significantly less data for later turns, because not all interactions include them.
This can potentially be addressed by not including all past turns in the context (i.e., sliding window approach).

##### User adaptation

A potential confounder is user adaptation: the improvement in interaction success rate could have been attributed to users adapting to the interaction scenario and the system, instead of policy improvement (Hawkins et al., ).
We redeploy the initial policy $\pi_{\theta_{0}}$ concurrently to final b-sup round to test this (control in Figure 5).
The interaction success rate of control remains relatively stable over time (31% $\rightarrow$ 33%), suggesting that speaker familiarity and adaptation do not explain the overall 51% absolute improvement in b-sup interaction success rate.

##### Positive Only vs. All data

The difference between systems that use positive learning signals only (b-sup, t-sup) and these that use all (b-rl, t-rl, b-kto, t-kto) is in learning objectives (supervised vs. RL/KTO)
Overall, the systems based on positive signals only perform better.
It is expected that positive signals will be more informative for learning. Our policy acts in a large action space. Negative rewards suppress a specific action, but without more information about what is a good action, they simply encourage a uniform distribution.
This has been shown to have a helpful regularizing effect in past work (Kojima et al., ).
However, not only negative feedback does not help in a meaningful way, it seems to confuse the learner. The positive-only systems that, in effect, have access to fewer learning signals perform better.
Utilizing negative signals better is an important direction for future work.

Figure: Figure 6: Confusion matrices of the binary (top row) and ternary (bottom row) feedback decoders over rounds. The feedback decoder yields precise positive signals, even in early rounds.
Refer to caption: x5.png

##### Feedback Decoder Evaluation

We evaluate the quality of the feedback decoder through our annotation task.
For each turn, workers annotate if the speaker was satisfied with the answer given their followup utterances.
Figure 6 shows feedback decoding confusion matrices over time.
The feedback decoder performance is relatively stable throughout the rounds, showing robustness to changes in the data distribution.
If we collapse together actual positives and neutrals, we observe above 90% precision consistently.
The ternary feedback decoder is more conservative compared to the binary one and labels more positive turns as neutrals. This is a task-dependent trade-off. The zero feedback of neutrals essentially eliminates the examples, but allows for slightly cleaner data. Here we empirically observe it is beneficial to have slightly noisy data but more of it.

##### Supervised vs. REINFORCE vs. KTO

Overall, the supervised variants (b-sup and t-sup) perform best.
The KTO variants (b-kto and t-kto) trail after the REINFORCE variants (b-rl, t-rl). b-kto even diverges at some point and starts losing performance fast. We suspect this is because the KTO recipe does not work well in the challenging optimization scenario of continual learning, where the model is fine-tuned multiple times.
We observe that b-kto deteriorates in rounds two and three, and starts generating illegal outputs (e.g., Deselect select). Appendix B.3 describes a quick intervention we applied to try to mitigate this issue. Although it eliminated the illegal outputs, the quality remained low.
It is possible that further refinement of how KTO is used or further tuning of its hyperparamters will help. However, this is a complex process in a live deployment.

##### Language analysis

We analyze the human instructions and how they change as the policy learns from more interactions (Figure 7).
We observe a reduction in vocabulary size and utterance length early on. This is expected, and follows known observations in how humans adapt to reduce cognitive costs (e.g., Clark & Wilkes-Gibbs, ; Effenberger et al., ).
However, in later rounds, b-sup witnesses an increase in vocabulary size and utterance length.
This surprising trend reversal is attributed to only three outlier workers, so does not express a significant change in population behavior.
The number of reset signals drops, another reflection of improved collaborated task performance. Such trends are fairly consistent across system variants, except for b-kto, which also shows divergence in performance.
We observe that initially workers tend to use Try again instead of directly describing a target, or request a reset with instructions like Deselect everything (Figure 15 and Figure 16). The occurrences of both decrease in later rounds.
Even though the workers change their language, this does not really help the initial policy $\pi_{\theta_{0}}$, which remains poor (Figure 5).

Figure: Figure 7: Language analysis of human instructions. All systems show a decrease in instruction complexity in the first three rounds, except for b-kto, suggesting adaptation and improved efficiency on the speaker’s side. Keyword-based analysis reveals that the number of reset/frustration signals drops, a reflection of the model learning and collaboration improving.
Refer to caption: x6.png

## 7 Related work

##### Learning from feedback

Learning from feedback for LLMs is being studied extensively.
RL from human feedback (RLHF) is maybe the most common technique (Ouyang et al., ).
It relies on soliciting pair-wise preferences from annotators, which is significantly different than our reliance on signals from the interaction itself.
Learning from feedback on a single system output has also been studied, either in the form of binary feedback (Ethayarajh et al., ; Suhr & Artzi, ; Gao et al., ) or through more expressive editing (Gao et al., ) or commenting and refinement (Li et al., ; Sumers et al., ; Scheurer et al., ).
Hancock et al. () trains a separate supervised model to continually predict satisfaction levels, which is then used to pause interactions and solicit explicit feedback.
Unlike this line of work, we do not solicit feedback, but rely on natural signals that arise from the followup interaction. Some of these utterances include explicit feedback, but many do not.
Most important, our approach relies on the natural course of human interactions, and does not add solicitation of feedback.

##### Learning from naturally occurring signals

Kojima et al. () presents an approach to learn to generate instructions by observing how humans follow them, a complementary mode of learning to our focus on general response.
Pang et al. () maximizes heuristics, such as the chance of long responses from humans, in a chatbot scenario.
Artzi & Zettlemoyer () studied the use of naturally occurring recovery efforts (i.e., when the user switches to simpler language to relay information) to train a symbolic semantic parser from a corpus of dialogue interactions.
In contrast, we opt for a general approach to infer feedback from natural language interactions of the model itself.

Concurrently to our work, Don-Yehiya et al. (), as well as  Petrak et al. (), proposes an approach that uses naturally occurring feedback in conversations to filter a large conversational corpus. The linguistic cues they rely on are similar to ours.
Unlike our study, the model they improve is not the model that generated the interactions, creating a distillation-like setup, where improvement is not coming from the model’s own interaction, but from other models.
We focus on model self-improvement, where it is critical that no stronger model is involved, and the model drives its own improvement.
A second distinguishing factor is our interest in continual deployment with humans, whereas they follow a standard train-test benchmarking recipe.
This allows our work to expose dynamics that are otherwise hidden.
Taken together, our work and Don-Yehiya et al. () complement each other, and strengthen our conclusions. Their work shows the signal can be derived from large-scale diverse data, whereas ours shows how a single-model loop can work over a long period of time, and the dynamics it creates.

##### LLMs that self-improve

A common approach to improve models is via AI feedback, solicited from the model itself or another model (Bai et al., ; Burns et al., ; Madaan et al., ; Kumar et al., ; Qu et al., ; Yuan et al., ; Li et al., ).
As opposed to AI feedback, we elicit *real human* feedback automatically from the interaction the model has while it is deployed.
This signal is more on-the-job, and less likely to be influenced by model biases.
We also use the same model for interaction and inferring feedback, ruling out concerns about distillation.
Our work is somewhat analogous to the generator-verifier line of work (Anil et al., ; Li et al., ; Kirchner et al., ), which relies on automatic verifiers (e.g., unittests).
We retrospect and learn from implicit human feedback that naturally occurs in interactions, as an alternative verification signal.

## 8 Discussion

We introduce ReSpect: retrospective learning from interactions, an annotation-free approach by leveraging signals from naturally occurring feedback in interactions.
We demonstrate its effectiveness in long-term deployments and robustness to system variants.
As opposed to evaluating on a static benchmark, we design MultiRef to study real interactions over a period of time. We make trade-offs between the generality of the task, and the ability to iterate on a prototype fast, and without high costs.
It is important to expand this type of study to other tasks, such as summarization or conversational question answering, where similar signals may be more complex, far apart, or demand long-term credit assignment.
Another interesting direction is expanding the expressivity of the feedback decoder, such that it recovers a more expressive signal (e.g., a natural language explanation).

#### Author Contributions

ZC developed and implemented the methods, trained models, conducted pilot studies, and the continual learning study. MOG contributed code for tangram reference games and designed the human evaluation study.
MOG, AW, YC, and GG designed the MultiRef scenario.
YC and GG implemented the MultiRef interface, recruited MTurk workers, and collected human-human interactions, with MOG and AW’s mentoring. ZC, MOG, and AW contributed to the writing. YA advised and oversaw all parts of the project.

#### Acknowledgments

This research was supported by NSF under grants No. 1750499 and OAC-2311521, NASA under award No. 20-OSTFL20-0053, a gift from Open Philanthropy, a gift from Apple, the National Artificial Intelligence Research Resource (NAIRR) Pilot, the Frontera supercomputer supported by the National Science Foundation (award NSF-OAC 1818253) at the Texas Advanced Computing Center (TACC) at The University of Texas at Austin, and the Delta advanced computing and data resource which is supported by the National Science Foundation (award NSF-OAC 2005572).
GG was supported by an NSF REU supplement for NSF grant No. 175049. YC was supported by Bowers Undergraduate Research Experience program.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, NASA, or the other funders.

## Appendix A The MultiRef Game Design and Data Collection

### A.1 Interaction Design

MultiRef is a multi-target, multi-turn reference game between two players, a *speaker* and a *listener*.
Each game starts with 10 tangrams as the *context*, with 3–5 tangrams designated as *targets*.
The target designations are revealed to the speaker but hidden to the listener. The goal is to select all targets without selecting any non-targets.
The speaker can only communicate with the listener through a sequence of utterances, and only the listener can take selection and deselection actions.
The interaction starts with a speaker turn. Turns alternate between speaker and listener, with a maximum of 20 turns.
In each speaker turn, they type an utterance to send to the listener.
Speaker turns are limited to 25 seconds.
In each listener turn, they have 45 seconds to select or deselect images as instructed to by the speaker.
The game concludes when the listener selects only and all targets, or the when the partners run out of turns.
Appendix A.3 shows screenshots of the interface.

##### Context Construction

We follow Gul & Artzi () and construct game contexts using 1,013 tangram images from KiloGram Ji et al. (). We group tangrams randomly into two splits: development split (101 tangrams) and main split (912 tangrams). The development split is exclusively used for seeding the initial listener policy $\pi_{0}$. All human-bot interactions are constructed from the main split, i.e., tangrams that the seed policy $\pi_{0}$ has never seen before. We construct all games with 3–5 target tangrams. More targets are generally harder, given the same maximum number of turns per interaction.

### A.2 Human Evaluation Design

Automatically evaluating turn-level policy performance is hard, because we have no ground truth (i.e., the selection and deselection actions intended by the speaker in each turn) to compare against. Similarly, we have no ground truth to systematically assess the feedback decoder quality. We conduct human evaluation surveys to address these problems. We annotate a subset of b-sup interactions, roughly 120 interactions or 1,000 turns per system-turn.

We show human annotators a complete interaction turn by turn, without revealing the underlying targets. For each turn, the annotation consists for two phases:

- 1.
Ground-truth: we show context, currently selected tangrams, and instruction given by the speaker. We ask the annotator to annotate the listener action. The annotator action $a^{*}$ is considered as ground truth action for this turn. We use these labels for tune-level evaluation. After the action annotation, we reveal the action $\hat{a}$ actually taken by the listener (i.e., the model) during the interaction.
- 2.
Satisfaction: we present the follow-up utterance. We ask the annotator to rate if the speaker is satisfied with the listener’s action, based on the follow-up utterance. They choose one of the following options:
a.
Yes.
b.
Yes, even though the listener did not perform all required selections/deselections.
c.
Yes, even though the listener made incorrect selections/deselections.
d.
No.
The third option accounts for the listener accidentally selecting a target tangram not intended by the speaker, but the speaker choosing to move on without correction or even validating the selection. We treat these labels as ground truth for evaluating feedback decoders.

We annotate 5% of long-term human-bot interactions annotations by three different annotators, to estimate how reliable the annotations are.
We observe 85% agreement on the correctness (whether $\hat{a}=a^{*}$) on ground truth stage,(^6^66The percentage of cases where all annotators agree that the bot did right or wrong.) and 65% agreement on the ground-truth action $a^{*}$ across workers.(^7^77The percentage of cases where all three annotators provided exactly the same set of actions.)
For satisfaction annotation, we observe 93% agreement rate, illustrating the relative simplicity of extracting the signal that drive our learning process.

### A.3 MTurk Details

##### Worker Recruitment

We follow ’s () worker recruitment recipe.
We require workers to have a minimum 98% approval rate, at least 1,000 approved HITs (Human Intelligence Task), and be located in English-majority locales. All workers must watch a video tutorial and pass a quiz before gaining qualification to work on MultiRef interactions. They must read a thorough guideline and pass another quiz before granted access to human evaluation surveys. We recruit 33 expert workers to interact with LLMs in the main study and annotate by completing surveys after the main study. This study is exempted by Cornell’s Institutional Review Board.

##### Payment

We pay workers $0.81 USD per MultiRef game, and a bonus if the game is successful. Overall the estimated hourly wage is $13.00 USD, and closer to $23.00 USD by the end of the continual study when the LLM is fairly good at the game. On average a human-bot game takes under 2 minutes. We pay workers $0.06 USD per turn for human evaluation surveys, or $0.08 USD if the turn annotation involves error modes. The estimated hourly wage is $16.00 USD for human evaluation surveys. On average it takes under 2.5 minutes to annotate one game. We set the payment scheme through pilot studies and aim at $15.00 USD hourly wage.

##### Interface and serving

We implement MultiRef using Empirica (Almaatouq et al., ) and on top of the code base of Gul & Artzi (). The speaker has 25 seconds to type into a chat box each turn and hit Enter or submit, and the listener has 45 seconds to click on the tangrams to select or to deselect. The game ends if one party idles for one turn, and the party idling is not compensated. We serve on an EC2 instance. We serve LLM policies with the Ray framework (Moritz et al., ). We walk through the first turns of a sample interaction in Figure 8, Figure 9, and Figure 10.

Figure: Figure 8: The MultiRef interface for the speaker in turn 1. Predefined targets are revealed to the speaker in black boxes.
Refer to caption: x7.png

Figure: Figure 9: The MultiRef interface for the listener in turn 2, following the speaker turn in Figure 8. Targets are hidden for the listener, and the context tangrams are in a different order. Here the listener has selected a tangram given the instruction select the butterfly.
Refer to caption: x8.png

Figure: Figure 10: The MultiRef interface for the speaker in turn 3, following the listener turn in Figure 9 The listener selected a non-target tangram, shown in red to the speaker.
Refer to caption: x9.png

## Appendix B Learning Details

### B.1 Interaction Representation

We encode the context $x$ as in Figure 11.
We standardize action representation by ordering actions, for example, always produce Select A C rather than Select C A. We shuffle the context images during training as the order of context tangrams should not have any impact on the interaction logic.

Figure: Figure 11: Policy prompt example with a model predicted action and additional *comments* for readability.

### B.2 Policy Initialization

We seed the initial policy $\pi_{0}$ by fine-tuning the model on a small dataset of 90 turns $D_{0}$, where both the speaker and the listener are humans.
We also experimented with prompting to initialize the policy. We find early that few-shot prompting yields a random policy at best, likely because reasoning with abstract shapes such as tangrams is visually out-of-distribution for the model.

There is a significant distribution shift between human-human interactions, and human-policy interaction, especially early on when the model performs poorly.
In practice, two major differences are the length of interactions and the prevalence of deselection instructions, which are rare in human-human interactions.
We address the deselection issue with data augmentation.
We synthetically generate turns where the speaker asks for deselections, and the listener complies.
We augment the data with these at a ratio of 1:12 to the existing data.
This helps the LLM policy learn to deselect and recover from mistakes. This augmentation is only used for $D_{0}$ and such distribution shift is not present in alter rounds, when learning from actual human-bot interactions.

Figure: Figure 12: An example of deselection augmentation with augmented action and *comments*.

### B.3 Hyperparameters and Other Implementation Details

We use the instruction-tuned IDEFICS2-8B model for all policies. We fine-tune with LoRA adapters (Hu et al., ) ($\alpha$=$r$=8, dropout=0.1) due to compute constraints. Appendix D provides more LoRA details.
We train each model with a single GPU, RTX A6000, NVIDIA A100 40GB or 80GB. The time to train ranges between 2–24 hours, longer in later rounds as more data accumulates. For stopping criteria,
we pick checkpoints by highest accuracy (exact match) among three seeds on a hold-out validation set of 344 turns $D_{\text{val}}^{\textsc{hh}}$. The validation set is curated from 92 human-human games the main split of tangrams. We summarize hyperparameters in Table 1.

**Table 1: Hyperparameter settings.**
| Hyperparameter | Search Space | Supervised | REINFORCE | KTO |
| --- | --- | --- | --- | --- |
| Optimizer |  | AdamW | AdamW | RMSProp |
| Learning rate | {1e-6, 1e-5, 1e-4, 2e-4} | 1e-4 | 1e-4 | 1e-5 |
| Learning rate decay | {no, cosine, linear} | cosine | cosine | no |
| Epochs | {5, 10, 20, 40} | 20 | 20 | 20 |
| Warm-up steps | {0, 10, 50} | 10 | 10 | 10 |
| Weight decay | {0, 0.01, 0.1} | 0.01 | 0.01 | 0.01 |
| Effective batch size | {16, 32, 48, 64, 128} | 64 | 64 | 64 |
| Entropy weight | {0, 0.01, 0.5, 0.1} | 0.01 | 0.01 | 0.1 |
| $\beta_{\text{KTO}}$ | {0.01, 0.1, 0.5} |  |  | 0.5 |
| Temperature |  | 1 | 1 | 1 |

##### Data imbalance

The decoded feedback is imbalanced, with more negative examples than positive examples (3:1 to 2:1), especially at early rounds of continual learning. We address this by weighing the loss by the absolute value of the reward, i.e., $-0.1$ for RL or $\lambda_{d}$ and $\lambda_{u}$ for KTO, and by downsampling negative examples per batch, such that the number of positive examples and negative examples is roughly 5:4.

##### KTO stability

Deviation from the original KTO implementation by higher learning rate, higher $\beta$, more epochs, produce better results empirically on the validation set in pilot and round $\rho=1$. However, in round $\rho=2$, b-kto policy start to degenerate by producing nonsensical actions such as Deselect A select A B or Deselect select select. We attempt to mitigate this issue during training round $\rho=3$ by switching from weighing $\lambda_{d}=4$ and $\lambda_{u}=1$ as recommended in Ethayarajh et al. () to $\lambda_{d}=\lambda_{u}=1$, plus downsampling negative examples. We also introduce regex-based constrained decoding to prevent nonsensical actions for b-kto and t-kto policies in round $\rho=3$. Despite that, the KTO group performs worse in live interactions (Figure 5).
We suspect KTO is more challenging to optimize for iterative continual learning, but we suspect further tuning (with higher computational costs) can reduce or even eliminate these issues.

### B.4 Evaluation Metrics

##### Interaction-level metrics

Interaction performance and statistics are computed automatically from live deployment interactions. They do not require further annotation.

- 1.
Success rate = # successful interactions / # all interactions. An interaction is successful if the listener selects all and only targets before running out of 10 turns. This is the primary metric we use to evaluate the performance of the LLM policy.
- 2.
# Turns per interaction. This is a measure of collaborative efficiency.

##### Turn-level metrics with reference to human annotation

We compute turn-level metrics either with respect to hh games where we consider human listener action as ground truth (e.g., $D_{\text{val}}^{\textsc{hh}}$), or with respect to b-sup games where we consider actions $a^{*}$ annotated in post-hoc surveys as ground truth.
When computed with live interactions, these metrics are biased towards longer or failed interactions because they have more turns than successful interaction.

- 1.
Exact match = # exact match / # all turns. An exact match is when the action taken by the policy matches exactly the action labeled/taken by human listeners ($\hat{a}=a^{*}$).
- 2.
Similarity = $\text{Sim}(\hat{a},a^{*})$ is a composite metric. Let $f(p,q):\mathcal{I}\times\mathcal{I}\rightarrow\mathbb{R}$ be a function that evaluates the similarity of between two images $p,q\in\mathcal{I}$. Let the action taken by policy be $\hat{a}=\{\hat{p}_{1},\hat{p}_{2},...,\hat{p}_{\hat{n}},\hat{q}_{1},\hat{q}_{2
},...,\hat{q}_{\hat{m}}\}$ where $p$ are the selected tangrams and $q$ are the deselected tangrams. Denote the ground truth actions as $a^{*}=\{p_{1}^{*},p_{2}^{*},...,p_{n^{*}}^{*},q_{1}^{*},q_{2}^{*},...,q_{m^{*}
}^{*}\}$. The similarity between two actions is defined as:
$\text{Sim}(\hat{a},a^{*})=\frac{1}{\hat{n}n^{*}+\hat{m}m^{*}}\left(\Sigma_{i=1
}^{\hat{n}}\Sigma_{j=1}^{n^{*}}f(\hat{p}_{i},p_{j}^{*})+\Sigma_{i=1}^{\hat{m}}
\Sigma_{j=1}^{m^{*}}f(\hat{q}_{i},q_{j}^{*})\right)\;\;.$
If only one of $\hat{n}$ and $n^{*}$ is zero, we rewrite $\Sigma_{i=1}^{\hat{n}}\Sigma_{j=1}^{n^{*}}f(\hat{p}_{i},p_{j}^{*})$ with $-\max(\hat{n},n^{*})$, and $\hat{n}n^{*}$ in the denominator with $\max(\hat{n},n^{*})$, intuitively assigning -1 for each missed selection. This edge case is similarly treated for $\hat{m}$, $m^{*}$ and deselection. We compute similarities using embeddings from the tangram fine-tuned CLIP model of Ji et al. ()
.
- 3.
Positive feedback = # turns receiving positive feedback / # all turns. An action receives positive feedback if speaker is satisfied with the listener’s action in the followup interaction. This is labelled in human evaluation survey.

##### Micro-level metric with reference to ground truth targets

We compute click accuracy with respect to the ground truth targets (instead of the targets intended by the speaker). This is cheaper because it does not require human annotation, so we can compute it for all system variants and all interactions. However, this measure produces false positives when an action selects a target not intended by the speaker. In practice, though, we find it correlates well with our human-annotated evaluation.

We compute click accuracy for a turn given its context $x$ and action $\hat{a}$. We denote the set of ground truth targets in this interaction as $\mathcal{T}$, the set of currently selected context tangrams as $\mathcal{S}$, then for each click $c$ in $\hat{a}$ (select or deselect), we compute the click accuracy as:

$$ $\text{Click accuracy}(c,\mathcal{T},\mathcal{S})=\begin{cases}1&\text{if }(c \in\mathcal{T}\land c\notin\mathcal{S})\lor(c\notin\mathcal{T}\land c\in \mathcal{S})\\ 0&\text{otherwise}\end{cases}$ $$

Intuitively, a click is approximately accurate if it selects a target or deselects a non-target. We compute this for all clicks from all interactions in a round for all systems in Figure 5.

##### Corpus-level metrics

We analyze speaker instructions per system-round. The keyword used to generate the analysis in Figure 7 are:

- 1.
# Reset = occurrences of phrases in {reset, restart, from scratch, all over, start over, deselect everything, deselect all, remove everything, remove all, clear everything, clear all, unselect everything, unselect all, drop everything, drop all }
- 2.
# Try again = occurrences of phrases in {try again, try one more time, the other one }

## Appendix C Cumulative Number of Interactions Observed

The main text includes results by round.
We collect roughly 330 interactions per policy per round. Due to the uncertainty of live data collection, we do not always hit this exact number for each variant and round.
Figure 13 shows the cumulative number of human-bot interaction seen by a policy variant by each round.

Figure: Figure 13: Cumulative number of human-bot interactions used to train the policy each round.
Refer to caption: x10.png

## Appendix D Additional Enhanced LoRA Launch

We suspect the plateau of b-sup in Figure 5 is partially due to the limited expressivity of LoRA adapters we used.
We test this hypothesis by deploying round $\rho=4$ and $\rho=5$ again with enhanced LoRA adapters. We use the same hyperparameters as in Section B.3 except additional adapters.
The original adapter placement is on the text model, the modality projector, and the perceiver resampler.
Adapters include the down projection layers, the gate projection layers, the up projection layers, and the key/query/value projection layers.
In comparison, the enhanced launch adds adapters on the vision model, including the out projection, the first and the second fully connected layers, besides the projection layers on text models.
Figure 14 shows the results from this complementary deployment.
The enhanced LoRA adapters yield a small improvement in interaction success rate compared to the original launch, yet the overall slowdown is evident. This suggests LoRA expressivity has some effect, but other effects are also limiting the LLM policy from continuing its earlier improvement trends.

Figure: Figure 14: Success rate of b-sup with additional LoRA adapters in round 4 and 5.
Refer to caption: x11.png

## Appendix E Feedback Decoder Design

The prompt design is minimal, general, and task-agnostic. We validate the prompt with manual inspection prior to continual learning launch and human surveys. Considering only the most recent two action-utterance turns $\langle\hat{a}_{i-1},u_{i},\hat{a}_{i},u_{i+1}\rangle$ is sufficient to produce satisfactory decoding results, and more history seems to distract the decoder.

We also experimented with numerical reward (i.e., decoding a real number), experimenting with a discretized reward space of {.0, .1, .5, .9} . Our experiments show the model is not well calibrated for such decoding.

## Appendix F Interaction Case Studies

Figures 15–18 show case studies that illustrate the diversity of MultiRef interaction scenarios. Black borders indicate targets. Yellow dots indicate actions taken by the listener. Green borders indicate correct selections, while red borders indicate wrong selection.

Figure: Figure 15: The speaker is left with the last target at Turn 4. Failing, they provide an additional description in Turn 5, and eventually resort to “try again” without describing the target in Turn 6. The initial turns illustrate how feedback is implied, rather than specified explicitly. The interaction concludes successfully.
Refer to caption: x12.png

Figure: Figure 16: The speaker asks to deselect everything in Turn 3 to reset, an expression of frustration. The interaction concludes successfully.
Refer to caption: x13.png

Figure: Figure 17: The abstractness and ambiguity of tangrams lend to complex interactions. There are two dogs in the context, and the listener struggles to disambiguate or identify the target. The interaction concludes successfully.
Refer to caption: x14.png

Figure: Figure 18: The speaker asks for two targets in Turn 1, exemplifying Grice’s Maxims of Quantity - one tries to be as informative as one possibly can, and gives as much information as is needed, and no more (Grice, ). The interaction concludes successfully.
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. How do you address overfitting given the extensive reuse of training data at each fine-tuning step?
2. Can you provide comparison of models across different evaluation metrics?
3. Have you tested the framework on other LLMs or tasks to confirm generalizability? How might the framework’s usability vary with stronger LLMs that provides less interaction data?
4. Have you considered other optimization technique like Direct Preference Optimization (DPO) which uses the binary labeled data while fine-tuning the LLM?
soundness: 3
strengths: - The idea of learning from past mistakes is really interesting and the proposed framework, RESPECT, doesn't depend on the optimization strategy. As highlighted in the paper, this framework can be used with various optimization strategy (Supervised Learning, Reinforcement Learning, Utility Maximization).
- The paper also contributed with a new task, MULTIREF, which will be very useful for the future development of this domain.
summary: The author proposes a novel framework, Retrospective learning from past interactions (RESPECT), for improving the LLMs based on signals from past interactions via retrospection. They also contributed with a task, Multi-turn Grounded Interaction Scenario (MULTIREF), a conversational interaction scenario where two partners, a speaker and a listener, coordinate on the selection of a set of items. They further fine-tuned an LLM with different optimization techniques with the data collected from the proposed task and framework.
weaknesses: **Major:**
- **Excessive use of training data:** The proposed method relies heavily on data. The model is fine-tuned at each step with all the interaction data acquired from past steps. Now, although the authors mention that they are taking measures to avoid overfitting (lines 246-248), this much repeated data usage would eventually result in overfitting.
- **Lack of metric evaluation:** Although the authors showcases various observations and results through plots and confusion matrix, they lack the tables for comparing different metics. Having those results will significantly boost the paper quality.
- **Lack of generalizability of proposed method:**
  - **Across different LLMs:** The proposed framework is only tested on one LLM (IDEFICS2-8B). Now, although this framework can be applied over other LLMs, it is unclear whether it will boost their performance or not. One reason authors might have seen such improvement is because the tested LLM is bad at that particular task. If we have a very good LLM then this framework might not help much as we will have less interaction data to fine-tune model on.
  - **Across different tasks:** Another interesting extension of the proposed method can be over different tasks. Currently authors have only tested over a particular task but it would be interesting to see if it can be extended over other tasks like summarization (authors have highlighted this as future work in discussion).
- **Scalability:**
  - **Heavy reliance on human feedback:** The proposed framework relies heavily on human feedback. Although the authors have countered the problem of annotating the responses, the problem of getting good interaction data still remains a crucial problem, making it hard to scale.
  - **Cost:** Another issue with scalability is the cost associated with getting quality interaction data. As mentioned by authors, this small experiment took over a month to collect data and costed $11k USD (line 347). This also makes it harder to use the proposed method in real time.

**Minor:**
- Adding information about MULTIREF in the introduction will help in understanding the contribution of the paper.
- Information on LoRA configuration used for fine-tuning is missing.
- Very similar labels are used for control and HH data in figure 4, making it hard to interpret. Maybe changing it with something else will make it more interpretable.
- **Repeated variable:** Authors have repeated the use of variable t. At line 90 it represents time while at line 141 it represents turn and at line 145 it is again used as time.
- **Typo:** Fullstop (.) missing in line 422. "(supervised vs. RL/KTO) Overall" &#8594; "(supervised vs. RL/KTO). Overall"

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Ln 425-426: According to Section 4.2, RL includes both positive and negative rewards. What might be the reasons that including extra rewards would 'encourage a uniform distribution'?
soundness: 3
strengths: 1. The paper proposed a learning method, RESPECT that utilizes implicit human-in-the-loop feedback for explicit action improvement
2. The paper experimented with 3 learning methods: supervised learning, REINFORCE, and KTO
3. The paper conducted thorough experiments in a multimodal referential game
4. The paper conducted pre-training as well as online testing for iterative model improvement and evaluation
5. The paper is very well structured and well-written. The paper analyzed in detail about learning strategy tradeoffs, feedback label selections, feedback decoder evaluation, and language analysis.
summary: The paper proposed a method to train a model with implicit human-in-the-loop feedback for a referential game. The proposed method first translate implicit human natural language feedback and quantize them into positive, (neutral), and negative labels, and then use the feedback to fine-tune the language model for decision making. 

The experiment is situated in a referential game, where human is serving as a speaker to describe a subset of tangrams, and the model is serving as the listener to pick out the objects the human was describing.

The paper experimented with three learning methods: supervised learning, REINFORCE, and KTO. The models were initialized with pre-trained IDEFICS2-8B weights and fine-tuned with LoRA. Each model setting was fine-tuned with 0/1/2/3 rounds (B-SUP for 6 rounds), before being deployed in the online setting to have human-bot evaluation. 

The paper observed that supervised learning method with binary quantization provided best performance, and that the feedback decoder's performance is relative stable across rounds and is consistent with human evaluation. The paper also observed that the human language is getting simpler with smaller vocabulary size and reduced utterance length across the rounds.
weaknesses: The paper wishes to highlight the contribution on 'continual learning' and model's iterative improvement with human's online feedback, but the actual experiments conducted is slightly misleading. The authors were careful to distinguish the differences between 'round' and 'turn. 

- In the setup, each 'round' includes multiple 'turns' of interactions between a human and the bot. 
- The model is retrained after each 'round', with the history of all previous 'rounds'
- After fine-tuning at the end of each 'round', the model is fixed and deployed for evaluation

The main difference between proposed method versus the classic fine-tuning is the increasing context length during each fine-tuning round. It is unclear what the intended benefit is for the increasing context history?
For example:
- Interaction history could help personalize the message or have a better understanding the counter party's message, if the bot was interacting with the exact same human. It was unclear if the bots were interacting with the same human users across different rounds.
- Interaction history could help the bot understand the task goal, through multiple rounds of probing and try-and-error (like RL), only if the bot was not briefed on what the task goal (referential game) was. According to the experiment setup (Figure 1, 2, 3), the model seems to have prior knowledge of the exact task goal.

Beyond the examples illustrated above, the improved performance demonstrated in the paper might just be a result of fine-tuning with more data, and the expensive online evaluation among different turns showcased model's intermediate checkpoint performance. 

Nevertheless, the paper proposed a new method that turns implicit human feedback into explicit rewards that could help improve model's performance. It is the 'Continual Learning' aspects that lacks sufficient support.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - How well do you expect RESPECT to generalize to other domains or tasks beyond MultiRef? Have you tested it in any other scenarios?
- Have you considered ways to mitigate the impact of feedback misinterpretation on learning?
- Have you considered any potential ethical implications of learning from implicit human feedback, such as privacy concerns?
- The paper mentions that negative feedback signals are generally underutilized due to challenges in integrating them effectively. Would a more nuanced approach to weighting or categorizing negative feedback improve the model’s performance? eg some negative feedback could carry more importance than others. For instance, if the user strongly corrects an action (e.g., "No, that's completely wrong"), this feedback could be weighted more heavily than a milder form of dissatisfaction (e.g., "Not quite right"). Assigning different weights would allow the model to learn more from severe mistakes than minor ones.
- What are the computational costs of implementing RESPECT, especially the retrospective analysis of past interactions?
- In Figure 3, there appears to be a formatting issue or typo. It says "positive or negative positive, neutral, or negative feedback," which seems confusing. Likely, this is unintended and should read either "positive, neutral, or negative feedback" or "positive or negative feedback".
soundness: 3
strengths: - The use of continual learning in the RESPECT framework demonstrates strong potential for developing LLMs that improve continuously from real-world interactions.
- The retrospective aspect of RESPECT is particularly compelling, as it enables models to learn from user corrective feedbacks.
summary: - The paper presents RESPECT, a framework for LLMs to learn from implicit user feedback in multi-turn interactions. Rather than relying on external annotations, RESPECT enables models to retrospectively analyze past interactions and learn from cues like rephrased requests, signs of user approval, or frustration.
- This approach is applied in MULTIREF, a new multi-turn reference game where users instruct the model to select abstract shapes (tangrams), and the model gradually improves its accuracy based on decoded feedback signals.
- The study compares three learning strategies: supervised learning, REINFORCE, and KTO, finding that models using only positive feedback perform best.
weaknesses: - The experiments are confined to the MULTIREF scenario with abstract tangram shapes. This limited scope raises questions about the generalizability of RESPECT to other domains. Applying RESPECT to diverse settings, such as conversational agents could demonstrate its robustness and adaptability across a broader range of applications, particularly those involving complex language or high-stakes interactions.
- There's a risk that the model might overfit to specific patterns of implicit feedback rather than truly improving at the task.
- The paper does not compare RESPECT to other established methods for learning from implicit feedback or continual learning. Without such comparisons, it's difficult to assess the relative merits of this approach For example, methods in RLHF using preference modeling or utility maximization strategies  could serve as useful baselines.
- The feedback decoder relies on the model's ability to interpret implicit signals correctly. However, there's no guarantee that the model's interpretation aligns with the human's intended feedback. The paper would benefit from a more thorough analysis of cases where feedback may be misinterpreted and how this affects learning.
- While the paper shows improvement over six rounds for B-SUP, this may not be sufficient to fully understand long-term learning dynamics. The observed plateau and temporary decrease in performance warrant further investigation. Extended experiments over more rounds could provide insights into whether the approach continues to improve or stabilizes at a certain level.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. It's not clear to me how the authors complete their fine-tuning considering that most of the training regimes are not designed for dialogue data specifically. For instance, if you have a dialogue of 4 turns, do you simply treat this as a single example or do you derive many examples for it? This is an important detail which I don't think is specified in the section that describes the training regime.

2. What kind of REINFORCE implementation did you use? Did you adopt a baseline term? I think it's important to report more detail to aid reproducibility

3. Considering that the action space of the model is very limited, have you considered a form of token masking to improve the performance of your algorithms?
soundness: 3
strengths: 1. Interesting evaluation that tests the system with real users over a period of 4 weeks. This represents a great effort to showcase the strengths and weaknesses of the different training regimes.
2. Very interesting idea to simplify the task of the "critic" to fixed labels that can be used for very specific training regimes
3. The authors test different training regimes that are well known in the community such as Supervised Learning, REINFORCE and KTO
summary: In this paper, the authors propose RESPECT, a new method for refining multimodal language models (in this case vision and language models) from interaction data automatically generated by a model while interacting with another agent when solving a referential task. 

To tackle this problem, the authors propose a new benchmark called MultiREF which requires agents to manipulate tangrams, specific abstract shapes that are well-known in the community for their ability to elicit interesting communicative grounding phenomena due to their intrinsic ambiguity.  

Based on this dataset, the authors focus on a very specific training regime which alternates two phases: 1) retrospection = decoding implicit feedback from past interactions by means of a classifier which derives feedback labels (i.e., positive, neutral, negative); 2) learning = refining the model using the feedback received from the previous stage. Because the authors simplify the prediction task to a classification task, they argue that Step 1) can be simply performed by a carefully prompted model to perform a simple binary/three-way classification task. For the second step, the authors test different learning strategies such as a) supervised learning from positive data only; b) Online reinforcement learning (using REINFORCE) with a hand-crafted reward function which leverages the labels derived by the classifier from Step 1. c) Kahneman-Tversky Optimisation (KTO) as a form of reinforcement learning from feedback (in this case, AI feedback).  

The authors set up a really complex evaluation with real users that interact with the system in real-time. In their evaluation, they start from IDEFICS2-8B model as their initialisation and use a frozen IDEFICS2-8B as the feedback decoder. From their evaluation, seems that there is still a long way to go to develop robust training regimes that can facilitate the type of adaptation required for these interactive tasks. In fact, the supervised learning variant seems to be the most robust which relies purely on positive examples and ignores negative ones.
weaknesses: 1. Although I appreciate the rationale behind using tangrams, I wonder whether the authors could have tested this approach in more realistic reference game that are well-known in the community such as 20Q game [1], GuessWhat?! [2] or Photobook [3]. I feel like this would have given a much broader perspective on the robustness and reliability of the proposed training regimes for more complex language generation tasks

2. It's not clear to me what is the rationale behind using KTO compared to DPO which is more established (e.g., used by Meta for Llama 3.2 tuning). Considering that the authors have access to positive and negative examples, I wonder whether they should try DPO instead considering that has been tested more for VLMs

3. It's not clear to me how the authors complete their fine-tuning considering that most of the training regimes are not designed for dialogue data specifically. See Question 1 as well for details.

4. I think the authors are missing a simpler baseline that is fine-tuned using the final reward (i.e., whether you win or not the game) as a reward as was done in previous work [4]. 

5. The related work cites some interesting work related to using AI-generated feedback for improvement. However, the authors do not provide a baseline where this is explored for this game. For instance, the method proposed by Yuan et al 2024.


## Additional suggestions to improve the manuscript

I would suggest the author refine their manuscript by clarifying some aspects of their methodology by answering my questions below. At the same time, I would suggest them to implement the following changes:

- Extend the analysis of the paper with average dialogue turns over the different rounds; this can shed light on the ability of the models to improve their gameplay strategy over time

- Clarify how you pair different users on MTurk to maximise usage. For example, there are dedicated methods to do so (e.g., [5]) but it's not clear to me how you've arranged this. 

- Provide more details regarding the REINFORCE baseline

- Typo on Line 244 "process transformers"  

- It's confusing to see that you call your models (which are Vision and Language Models) as LLMs

- I would suggest improving Figure 1 to show the type of feedback that the system is leveraging for the improvement.

## References

[1]: Bertolazzi, L., Mazzaccara, D., Merlo, F., & Bernardi, R. (2023, September). Chatgpt’s information seeking strategy: Insights from the 20-questions game. In Proceedings of the 16th International Natural Language Generation Conference (pp. 153-162).

[2]: De Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H., & Courville, A. (2017). Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5503-5512).

[3]: Haber, J., Baumgärtner, T., Takmaz, E., Gelderloos, L., Bruni, E., & Fernández, R. (2019, July). The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1895-1910).

[4]: Strub, F., De Vries, H., Mary, J., Piot, B., Courvile, A., & Pietquin, O. (2017, August). End-to-end optimization of goal-driven and visually grounded dialogue systems. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (pp. 2765-2771).

[5]: Götze, J., Paetzel-Prüsmann, M., Liermann, W., Diekmann, T., & Schlangen, D. (2022, June). The slurk Interaction Server Framework: Better Data for Better Dialog Models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 4069-4078).

Meta-review: The authors present a way to farm supervision signals from LMs in multiturn interactions and use this as training data to improve the model. The novelty of this approach beyond reward-modeling / RLHF / RLAIF is not very clear. While authors present this as continual learning (which could be a way to distinguish from RLAIF i.e. continuing to learn post deployment), it still requires parameter updates and is trained over the whole dataset each time.

I agree there is something novel here in that humans interacting with the system are not expressly asked to rate the interactions, it's more implicit. But this novelty is not motivated clearly or sufficient for a main conference paper.

Decision: Reject