Venue: NAACL 2024

# When Life Gives You Lemons , Make Cherryade :

_Converting Feedback from Bad Responses into Good Labels_

 Weiyan Shi\({}^{\dagger}\)

Stanford University

Emily Dinan

Meta AI

&Kurt Shuster\({}^{\diamondsuit}\)

Character.ai

Jason Weston\({}^{*}\)

Meta AI

&Jing Xu\({}^{*}\)

Meta AI

###### Abstract

Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a _satisfaction classifier_ to label the unlabeled data; and (ii) training a _reply corrector_ to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

+
Footnote †: Equal contribution.


* We show that free-form textual feedback is a very useful signal for improving the performance of both a _satisfaction classifier_ to identify good and bad responses, and a _reply corrector_ to generate better corrections.
* Augmenting training data with _reply-corrector_-generated corrections works better than only training with existing gold corrections.
* Models such as Director(Arora et al., 2022) that utilize both gold/predicted good and bad responses further improves the final dialogue model. Our final best models outperform the baseline BlenderBot 2 model or using Director alone.

## 2 Related Work

Many recent works have studied how to align language models with human feedback Nakano et al. (2021); Ouyang et al. (2022); Scheurer et al. (2022); Saunders et al. (2022); Schick et al. (2022). For instance, InstructGPT Ouyang et al. (2022) was fine-tuned using feedback from labelers who ranked model outputs. Scheurer et al. (2022) fine-tuned GPT-3 and InstructGPT on 100 examples of free-form textual feedback from humans to improve summarization tasks and found that only the larger models such as GPT-3 (175B) Brown et al. (2020) can generate accurate refinements using feedback. Saunders et al. (2022) fine-tuned large language models to generate self-critiques for summarization tasks to assist human annotators, and continued to refine the models on feedback. In this work, we focus on improving dialogue agents given various human feedback signals (binary, free-form natural language and gold corrections) and compare our methods to Scheurer et al. (2022).

Existing works have also studied how to correct language model output. For instance, Elgohary et al. (2021) proposed a model to understand natural language feedback and produce a series of edits to correct a text-to-SQL semantic parser. Tandon et al. (2022) trained a memory-augmented corrector to convert feedback to edits and fix model outputs for a script generation task. Some recent large language model research can also repair generations given human feedback Scheurer et al. (2022); Saunders et al. (2022).

Past research has also explored how to integrate feedback into dialogue agents Li et al. (2016, 2016); Hancock et al. (2019); Shuster et al. (2020); Xu et al. (2022). Li et al. (2016) investigated how to improve the chatbot's question-answering ability with general textual feedback in a reinforcement learning setting. Hancock et al. (2019) developed a self-feeding chatbot that can construct new examples from existing human-bot conversations and ask for feedback when necessary to improve itself. Xu et al. (2022) proposed a dataset with internet-augmented dialogues, where each turn is annotated with human feedback, and they found that continuously retraining the model on binary feedback after deployment is helpful. Our work focuses on converting bad responses into good ones to augment the data and learn from feedback about failures.

## 3 Human Feedback Setting

As illustrated in the dialogue example in Figure 1, we consider a deployed system where one can collect three types of feedback:

1. **binary feedback**, where the human conversationalist explicitly likes ( ) or dislikes ( ) a bot response;
2. **free-form textual feedback**, where the human explains conversationally what was wrong when they dislike a response (e.g., _"That's a quick topic change! Let's continue to talk about fruit, perhaps fruit drinks?"_);
3. **gold correction**, where the human conversationalist suggests an alternative reply the bot should have said, (e.g., _"I like watermelons too! They tastes great in drinks."_).

In a deployment setting, it is unnatural to ask users to always click the thumbs up and down and write gold corrections whenever the bot makes a mistake. Instead, users tend to provide free-form textual feedback on what was wrong in their dialogue response to express dissatisfaction when the bot makes errors See and Manning (2021). Therefore many responses may be missing binary feedbackShuster et al. (2022). In this paper, we consider a sparse thumbs up/down signal and sparse gold correction signal setting, but a dense free-form textual feedback signal (i.e., mistakes are followed by textual feedback). After collecting conversations with these feedback signals, we can consider methods to utilize them to improve the dialogue model 



## 4 The Juicer Method

Figure 1 shows the overview of our framework Juicer to incorporate limited human feedback to improve the deployed dialogue model. The framework consists of training a _satisfaction classifier_, a _reply corrector_, and eventually the final dialogue model itself. We define the notation here. For a given bot reply: (1) denotes un-annotated turns; (2) and \(|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!|\!\!|\!|\!|\!|\!|\!|\!\!|\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!\!|\!|\!|\!|\!|\!\!|\!|\!|\!|\!|\!\!|\!|\!|\!|\!|\!\!|\!|\!|\!|\!|\!\!|\!|\!

augmented training set in later steps. In our experiments, both models are trained with human-labeled data which come from the FITS task (Xu et al., 2022), described further in Section 5.1.1.

**(1a) Satisfaction classifier** The training target of the _satisfaction classifier_ is a binary satisfaction label ( or  ). For the input to the classifier, we experimented with two variants: (1) the context + the bot reply to be labeled, and (2) the context + the bot reply to be labeled + the next human response. As shown in the example in Figure 1, when the first bot reply is given a thumbs-up, the next human response is a natural continuation of the conversation (e.g., _"I also like watermelons"_); when the bot reply is disliked (the second bot reply), the next human response is free-form textual feedback on what went wrong (e.g., _"That's a quick topic change! Let's continue to talk about fruit, perhaps fruit drinks?"_). Hence, the next human response can be indicative of the quality of its preceding bot reply, and we include it in the input. In our experiments, the _satisfaction classifier_ is trained by fine-tuning a 311M-parameter transformer pretrained on pushshift.io Reddit data Baumgartner et al. (2020).

**(1b) Reply corrector** The input to the _reply corrector_ is the context + the bad bot reply to correct + the next human free-form textual feedback on what went wrong. The training target is the correction to the bad reply which can be either (1) gold corrections written by crowdworkers; or (2) the next bot replies from the original FITS data that are classified as good ("self-corrections"). We fine-tuned the _reply corrector_ from a 3B parameter R2C2 transformer model Shuster et al. (2022).

### Step 2: Predict missing labels

In a bot-human dialogue, the binary feedback can be quite sparse, with many replies having no explicit feedback. We thus predict labels for these replies with the _satisfaction classifier_ trained in Step 1a. After this step, every bot reply in the dataset has a binary label either from the original human binary feedback ( or ), or predicted by the _satisfaction classifier_ ( or ).

### Step 3: Convert lemons to cherries

We can now augment the training data. We use the _reply corrector_ trained in step 1b to generate improved replies for any examples labeled as bad ( or ), and then add them to the training set for the final dialogue model.

Selecting correctable casesHowever, we note that not all bad responses are easily correctable given free-form textual feedback. For example, the human feedback "You are talking nonsense!" could help indicate this is a using the _satisfaction classifier_, but is less helpful for knowing what the right response is, compared to more constructive feedback such as "That's a quick topic change! Let's continue to talk about fruit, perhaps fruit drinks?" We thus experiment with detecting cases that are "correctable", and only use these to augment our training data. We first embed the free-form textual feedback and the immediate next bot reply in recorded conversations with SentenceBERT Reimers and Gurevych (2019), and then calculate their cosine similarity score. If the score is high, it means that the human free-form textual feedback is easier for a model to comprehend and thus revise its own response accordingly. We define such examples as correctable and then threshold the similarity score to pick out correctable cases.

Predicting reply correctionsTo obtain the corrections, we adopt a reranking-based learning method widely used in many previous studies Nie et al. (2020); Nakano et al. (2021); Askell et al. (2021) to score and rank the generations. We first use the _reply corrector_ to generate many correction candidates (60 in our experiments). Then we concatenate the original context with the correction candidates and feed them into the _satisfaction classifier_ from Step 1a. Finally, we select the top one with the highest probability output by the classifier as the final correction. If all generated corrections are predicted as bad, we will skip this example.

### Step 4: Collect fruits and re-train

After the previous steps, each bot response is annotated with either a gold or predicted binary label, and those labeled as bad are converted from bad responses to good ones using human feedback. The final step is to augment the training set of the final dialogue model with the new data.

One straightforward method to improve the model is to augment the training data with all the positive replies including the corrections ( and ) and use the standard language modeling objective. However, this standard training does not utilize negative/bad replies ( ) to avoid them. We 

hence also apply the recently proposed Director model Arora et al. (2022) to both reinforce the positive responses and penalize the negative ones. Director is a unified decoder-classifier model jointly trained with a language modeling task and a classification task. During inference, it uses its language modeling head to predict the next token probability, and its classifier head to decide if the tokens belong to positive examples to generate the final output. But it is worth noting that in this step in Juicer, we could use any other approach that utilizes both positive and negative responses to re-train and improve the final dialogue model.

## 5 Experimental Setup

In our experiments, we used the 3B parameter BlenderBot2 (BB2 3B) Komeili et al. (2021); Xu et al. (2021) as the base dialogue model and try to improve it with human feedback from deployment.

### Datasets: FITS and DEMO

We performed experiments on the FITS Xu et al. (2022) dataset. We also tested the zero-shot transferability of both the _satisfaction classifiers_ and the _reply correctors_ on a real deployment dataset DEMO Ju et al. (2022).

#### 5.1.1 Fits

FITS contains internet-augmented human-bot dialogues with annotated feedback for every turn, including a binary label, free-form textual feedback and a gold response, with around 39k bot utterances in total. See Section A.1 for more details. To mimic a deployment setting with limited feedback, we uniformly sampled 20% of the bot responses from the training set of FITS and considered them to have binary feedback and gold labels, while the rest were considered unlabeled. However, we did not remove free-form textual feedback when it is present, as it remains part of the conversation, see Figure 1. Table 4 in the Appendix shows the data statistics after sampling.

We used the original FITS validation, test set and unseen test set (of new conversational topics) for evaluation, and employed the same metrics as Xu et al. (2022) for the final dialogue models: perplexity, F1 overlap with the gold annotation, and human evaluation via conversations with the bot. During conversations, crowdworkers click or per turn and give a final rating (a score out of 5) in the end. We report the average good response rate in percentage. See Appendix C for more details

#### 5.1.2 Demo

The dataset DEMO is from the deployment of BlenderBot 3 Shuster et al. (2022) with responses verified by crowdworkers Ju et al. (2022). In total 923 bot responses across 81 conversations are used as an evaluation set.

### Baselines

We have two categories of baselines: (1) without model-augmented data, and (2) with a prompt-based _reply corrector_. In addition, we also compare with oracle methods using 100% labeled feedback data without sampling.

Baselines without augmentation.The most straightforward baselines are to fine-tune with the limited human-labeled feedback only.

* **Gold corrections from 20%** Gold corrections provide a strong learning signal. Here, we simply fine-tune BB2 3B on the gold corrections from 20% human-annotated set.
* **Free-form textual feedback from 20%** Following Hancock et al. (2019), we fine-tune BB2 3B with the context as the input and the free-form textual feedback (identified as the response following the bad responses) as the target.

Baseline with a prompt-based _reply corrector_.Instead of training a supervised _reply corrector_ with gold corrections, this baseline prompts an off-the-shelf model with free-form textual feedback and instructions like "given the feedback, correct the original response" as a _reply corrector_ to generate corrections, and then fine-tunes the final dialogue model on these corrections.

* **3B-all-corrections**: Scheurer et al. (2022) proposed an approach to improve language models with language feedback, originally applied to summarization tasks, which we adapt here for dialogue. Given a small number (100) human feedback samples, they prompted a language model to condition on the context (input+feedback) to re-generate multiple summarization corrections, picked the correction with the highest similarity score with the feedback, and finally fine-tuned the language model on the corrections to improve it. In our implementation, we use the baseline BlenderBot 2 model (3B) as the _reply 

_corrector_. While Scheurer et al. (2022) used larger language models (175B), our implementation of the baseline is more comparable to our Juicer models since our _reply corrector_ also has 3B parameters. In our experiments, instead of using only 100 examples, we make this a stronger baseline by generating corrections for _all_ the bad replies.

### Juicer models

We also compare several variants of Juicer.

* **Juicer**. We fine-tune BB2 3B by augmenting the 20% human-annotated data with (1) predicted good responses by the _satisfaction classifier_ from the remaining 80% un-annotated turns, and (2) predicted corrections generated by the _reply corrector_, filtered to only include the correctable cases rather than using all the predicted corrections.
* **Juicer + Director**. We fine-tune using Director which uses both the positive and negative replies. Both gold annotations and the filtered corrections generated by the _reply corrector_ are used as positive classification data. Bad responses labeled by humans or the _satisfaction classifier_ are used as negative data for fine-tuning the classifier head.
* **w/o predicted corrections (from Step 3)**. In this ablation, we fine-tune the final dialogue model with only predicted good responses by the _satisfaction classifier_, without the corrections generated by the _reply corrector_.
* **w/o selecting correctable cases**. In this ablation, we only augment with (1) predicted good responses by the _satisfaction classifier_, and (2) all the predicted corrections without selecting the more correctable cases. This tests if selecting correctable cases brings improvements.

## 6 Results

We first evaluate the _satisfaction classifier_ (Table 0(a)), and the _reply corrector_ (Table 0(b)). We then perform both automatic and human evaluations on the final dialogue models (Table 2 and Table 3).

### Satisfaction classifier

Table 0(a) shows the classifiers' performance on the FITS data and also their zero-shot performance on DEMO.

Adding the next human response helps.We find the balanced accuracy of detecting satisfaction using only the dialogue context and the bot response itself is \({\sim}\)75% on FITS. It is significantly improved to \({\sim}\)95% by including the next human message in the input. A similar improvement is found when measuring balanced F1 as well. On the deployment dataset DEMO where organic users are not required to always write free-form textual feedback when seeing a bad reply, adding the human response still improves the balanced F1 from 64.77 to 71.24, despite this being zero-shot performance (without training on this dataset). These results indicate the importance of using the next human message to make satisfaction classification decisions. As using the next human response helps, we default to using this _satisfaction classifier_ variant in our standard Juicer setup.

### Reply corrector

Table 0(b) shows the results of training the _reply corrector_, comparing different input feature choices.

Free-form textual feedback improves the correction.We performed an ablation study where the _reply corrector_ trains on (context + bad reply \(\rightarrow\)_good reply_) without the free-form textual feedback, shown in "w/o free-form textual feedback". As expected, adding free-form textual feedback on what went wrong improves the _reply corrector_'s performance. The best results are relatively close to the oracle performance which uses 100% (rather than 20%) gold data for training (23.39 F1 vs. 21.41 and 2.93 PL vs. 3.07).

Augmenting with self-correction pairs helps.The standard _reply corrector_ trains on "gold-correction" pairs (context + bad reply + free-form textual feedback\(\rightarrow\) gold correction). Besides these human-written gold corrections, we can also train the _reply corrector_ on "self-correction" pairs (context + bad reply + free-form textual feedback\(\rightarrow\)_good bot reply_), where a bad reply is followed by a good bot reply either liked by humans or predicted as good by the _satisfaction classifier_, suggesting that the bot "corrects" itself in the following turn. We found that augmenting with these "self-corrections" improves the F1 from 17.10 to 21.41. We can also multitask with various dialogue tasks to further improve the _reply corrector_'s performance. See Section A.4.1 for more details 

Qualitative results show the corrections make sense.We also include generated correction examples on the FITS data in Appendix Table 8 and on the deployment data in a zero-shot fashion in Appendix Table 9. These examples show that the _reply corrector_ can integrate free-form textual feedback to correct the bad replies, even for zero-shot deployment data.

See section A.4.2 for further details and results on the _reply corrector_ evaluation.

### Final dialogue model evaluations

The final dialogue model results are given in Table 2 (automatic evaluations) and Table 3 (human evaluations). All methods are fine-tuned from the 3B parameter BlenderBot 2 (BB2), making the models comparable.

Using Juicer to augment data improves results.Juicer yields significant gains over the baseline transformer BB2 3B in both automatic evaluations and human evaluations. For example, we see an F1 increase from 15.3 to 18.5 on the unseen test set, and an improvement of good responses from 33.2% to 41.9% in human evaluations.Juicer also performs better than baselines without augmentation (e.g., gold corrections from 20%).

Our supervised _reply corrector_ outperforms a prompt-based one.Compared to the prompt-based _reply corrector_ baseline Scheurer et al. (2022), all the Juicer models perform better in automatic evaluations. When the prompt-based model is used as a _reply corrector_ to produce corrections to augment the final dialogue model training, the final model evaluation (F1=14.2, ppl=8.9) is worse than augmenting with the corrections in Juicer (F1=16.7, ppl=8.5).

Augmenting training with predicted correctionsin Juicer helps.Juicer augments training with predicted corrections, which improves both the F1 and perplexity across the board compared to Juicer without predicted corrections, e.g. 18.5 vs. 17.9 on the test unseen F1. This makes sense because the predicted corrections are generated by the _reply corrector_ given human free-form textual feedback which contains valuable information, and fine-tuning the final dialogue model on these corrections can steer it toward better replies.

Selecting correctable cases can help.Juicer picks only correctable cases to augment the training data, with around 62% of cases selected (threshold chosen based on the validation set). Compared to naively augmenting with all predicted corrections, we see gains on valid and unseen test F1 (18.5 vs. 18.0), although there is no gain on the seen test set.

\begin{table}
\begin{tabular}{p{113.8pt}|c||c||c|c|c|c} \hline \hline
**(1a)**_Satisfaction Classifier_**** & \multicolumn{2}{c|}{Valid} & \multicolumn{2}{c|}{Test} & \multicolumn{2}{c}{Test Unseen} & \multicolumn{2}{c}{DEMO (zero-shot)} \\ \cline{2-7}
**Input** & **Acc\(\uparrow\)** & **F1\(\uparrow\)** & **Acc\(\uparrow\)** & **F1\(\uparrow\)** & **Acc\(\uparrow\)** & **F1\(\uparrow\)** & **Acc\(\uparrow\)** & **F1\(\uparrow\)** \\ \hline \hline context+bott+human & **94.66** & **97.25** & **95.76** & **97.83** & **96.74** & **98.34** & **59.73** & **71.24** \\ context+bot & 75.58 & 86.07 & 74.53 & 85.38 & 71.46 & 83.25 & 56.60 & 64.77 \\ \hline \end{tabular}
\end{table}
Table 1: Performance of the modules in Step 1: (a) _satisfaction classifier_, and (b) _reply corrector_.

 

Director provides further gains.Director utilizes both the (predicted) binary feedback signal and textual feedback signal to penalize negative responses. Applying it improves the results further over standard Juicer (45.5% good responses vs. 41.9% for Juicer_without_ Director, as measured by human evaluations). Because Director uses a classifier head to decide if a token should be included in the final generation, the distribution is altered and perplexity measures are not applicable. However, it gives gains in F1 on valid and test sets, although not on the unseen test set. Juicer and Director together also outperforms Director alone, even when Director uses 100% gold binary labels, see Appendix Table 11. Further variants and experiments with Director are also given in Section A.5.2.

Juicer achieves comparable results to methods with oracle access to gold labels.Compared to methods using 100% gold data which was not given to Juicer, our best Juicer models achieve comparable performance, especially on F1 and human evaluation. For example, test unseen F1=17.6 for the best "oracle" method vs. 17.7 for the best Juicer model, 47.0% vs. 45.5% good responses, and 3.38 vs. 3.34 in human ratings. See Appendix Table 11 for further details. These "100% data" methods can be seen as upper bound results, showing that Juicer does extract most of the signal from the portion of the dialogue data without binary or gold feedback.

See Section A.5 for further experiments and details on final model evaluations.

## 7 Conclusion

Deployed dialogue agents should continuously improve by using human feedback gathered during interactions. Unfortunately, feedback collected in the wild can be limited. In this paper, we proposed Juicer, a framework to efficiently use limited organic feedback signals (binary labels and gold corrections) if free-form textual feedback is provided. Juicer works by correcting bad responses into good ones to augment the training data for the final dialogue model. Experiments show that augmenting with such predictions can integrate human feedback and improve overall performance.

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \hline \multicolumn{1}{l|}{**Final dialogue model**} & \multicolumn{5}{c}{**Automatic evaluation**} \\  & \multicolumn{2}{c|}{**Valid**} & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c}{**Test Unseen**} \\  & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** \\ \hline \hline BB2 3B & 14.4 & 10.6 & 14.7 & 10.3 & 15.3 & 9.3 \\ +gold corrections from 20\% & 16.2 & 9.1 & 15.6 & 8.9 & 17.9 & 8.4 \\ +free-form textual feedback from 20\% & 13.1 & 10.4 & 12.6 & 10.3 & 13.7 & 9.6 \\ \hline
3B-all-corrections (prompt-based) & 14.2 & 8.9 & 14.5 & 8.7 & 15.2 & 8.2 \\ \hline \hline \multicolumn{1}{l|}{**Juicer models**} & & & & & & \\ \hline \hline +Juicer & 16.7 & **8.5** & 16.2 & 8.4 & **18.5** & **8.0** \\ +Juicer +Director & **17.2** & n/a & **16.7** & n/a & 17.7 & n/a \\ \hline \hline \multicolumn{1}{l|}{**Juicer ablations**} & & & & & \\ \hline w/o predicted corrections & 15.7 & 9.0 & 15.8 & 8.8 & 17.9 & 8.2 \\ w/o selecting correctable cases & 16.4 & 8.5 & 16.4 & 8.4 & 18.0 & 8.1 \\ \hline \end{tabular}
\end{table}
Table 2: Final dialogue model automatic evaluation results. All the dialogue models are fine-tuned from BB2 3B. Juicer models with augmentations are better than the baselines without augmentations. Juicer with a supervised _reply corrector_ also performs better than the baseline with a prompted-based _reply corrector_. Director utilizing negative examples is effective. Using predicted corrections and selecting correctable cases are useful.

\begin{table}
\begin{tabular}{l|c c} \hline \hline \multicolumn{1}{l|}{**Final dialogue model**} & \multicolumn{2}{c}{**Human evaluation**} \\  & **Good\% \(\uparrow\)** & **Rating \(\uparrow\)** \\ \hline \hline BB2 3B & 33.2\% & 3.09 \\ +gold corrections from 20\% & **39.4\%** & 2.89 \\ \hline \hline \multicolumn{1}{l|}{**Juicer models**} & & \\ \hline \hline +Juicer & **41.9\%** & 3.06 \\ +Juicer +Director & **45.5\%** & **3.34** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Final dialogue model human evaluation results. We report the % of good responses and the overall rating, as judged by crowdworkers during conversations. We bold statistically significant improvements (independent two-sample \(t\)-test, \(p<0.05\)) of methods over the BB2 3B baseline. Juicer outperforms the baselines. Juicer +Director works the best.

 



## 8 Limitations and Discussions

In our experimental setting, we assume dense free-form textual feedback, i.e., a bad reply is always followed by a free-form message explaining what was wrong. In real deployments, this free-form textual feedback signal may not always be given and without it, the binary _satisfaction classifier_ may not necessarily achieve a high accuracy or F1 (e.g., 90+), which could also impact the later steps. It remains to be seen in real deployments how dense this signal is, and what methods can be used to encourage users to make these signals as dense as possible, so that strong feedback signals are available to train on.

We have also assumed good intent from human conversationalists, but it is possible to have adversarial and bad actors interacting with the bot. In particular, incorrect feedback or opposite feedback (e.g., thumbs up instead of down) could be supplied by the human for incorrect bot behavior. We see this as an important research direction that should be pursued in parallel to work on algorithms like the ones we study here. See e.g. Ju et al. (2022) for recent work addressing bad actors and adversarial feedback.

The training/evaluation loop of Juicer can be long due to its iterative nature. The advantage of using a _reply corrector_ is that we can qualitatively evaluate the quality of the generated corrections. But the drawback is that we need to first train a _reply corrector_, use it to generate corrections, and finally improve the dialogue. We assume that the best _reply corrector_ will lead to the best final dialogue model, but this remains to be studied. Another possible direction is to use a latent _reply corrector_ to integrate the feedback in a more end-to-end fashion instead of a supervised _reply corrector_ that will generate explicit corrections separate from the dialogue model.

Additionally, the proposed Juicer framework improves the dialogue model offline rather than correcting the response on the fly. With the necessary infrastructure support, there is potential for improving the models online. This could be a natural setting for reinforcement learning to get interactive feedback and iteratively update the model policy as the conversation continues. Such a direction does not come without dangers, however, such as the model degrading if it receives poor inputs, e.g. from bad actors as mentioned before.

## References

* Arora et al. (2022) Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. Director: Generator-classifiers for supervise language modeling. _arXiv preprint arXiv:_.
* Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. _arXiv preprint arXiv:2112.00861_.
* Baumgartner et al. (2020) Jason Baumgartner, Savvas Zannettou, Brian Keegan Suire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. _arXiv preprint arXiv:2001.08435_.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Conneau et al. (2019) Alexis Conneau, Kartiky Khandelwal, Naman Goyal, Vishrava Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. _arXiv preprint arXiv:1911.02116_.
* Elgohary et al. (2021) Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. Nl-edit: Correcting semantic parse errors through natural language interaction. _ACL_.
* Hancock et al. (2019) Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! _arXiv preprint arXiv:1901.05415_.
* Ju et al. (2022) Da Ju, Jing Xu, Y-Lan Boureau, and Jason Weston. 2022. Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls. _arXiv preprint arXiv_.
* Romeili et al. (2021) Mojtaba Romeili, Kurt Shuster, and Jason Weston. 2021. Internet-augmented dialogue generation. _arXiv preprint arXiv:2107.07566_.
* Li et al. (2016a) Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2016a. Dialogue learning with human-in-the-loop. _arXiv preprint arXiv:1611.09823_.
* Li et al. (2016b) Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2016b. Learning through dialogue interactions by asking questions. _arXiv preprint arXiv:1612.04936_.
* Nakano et al. (2019) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders,

et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. _arXiv preprint arXiv:2112.09332_.
* Nie et al. (2020) Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2020. I like fish, especially dolphins: Addressing contradictions in dialogue modeling. _arXiv preprint arXiv:2012.13391_.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _arXiv preprint arXiv:2203.02155_.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. _arXiv preprint arXiv:1908.10084_.
* Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. _arXiv preprint arXiv:2206.05802_.
* Scheurer et al. (2022) Jeremy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feedback. _The First Workshop on Learning with Natural Language Supervision at ACL 2022_.
* Schick et al. (2022) Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmapantis, Edouard Grave, and Sebastian Riedel. 2022. Peer: A collaborative language model. _arXiv preprint arXiv:2208.11663_.
* See and Manning (2021) Abigail See and Christopher D Manning. 2021. Understanding and predicting user dissatisfaction in a neural generative chatbot. In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 1-12.
* Shuster et al. (2022a) Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022a. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. _arXiv preprint arXiv:2203.13224_.
* Shuster et al. (2020) Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, and Jason Weston. 2020. Deploying lifelong open-domain dialogue learning. _arXiv preprint arXiv:2008.08076_.
* Shuster et al. (2022b) Kurt Shuster, Jing Xu, Mojtaba Romeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kwshal Arora, Joshua Lane, et al. 2022b. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. _arXiv preprint arXiv:2208.03188_.
* Smith et al. (2020) Eric Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020. Can you put it all together: Evaluating conversational agents' ability to blend skills. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. ACL.
* Tandon et al. (2022) Niket Tandon, Aman Maadan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model output errors after deployment using a dynamic memory of feedback. _NAACL Findings.(to appear)_.
* Xu et al. (2021) Jing Xu, Arthur Szlam, and Jason Weston. 2021. Beyond goldfish memory: Long-term open-domain conversation. _arXiv preprint arXiv:2107.07567_.
* Xu et al. (2022) Jing Xu, Megan Ung, Mojtaba Komeili, Kwshal Arora, Y-Lan Boureau, and Jason Weston. 2022. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. _arXiv preprint arXiv:2208.03270_.

 

Appendix

### Details on FITS

In this section, we describe the existing FITS task (Xu et al., 2022) in more details. In FITS, each bot message is annotated with the following feedback. The setting also ensures that after the human provides the feedback, the conversation can be continued with the feedback integrated.

* A binary satisfaction label.
* If it is a bad reply, the human provides free-form textual feedback on what went wrong in the next human message.
* Multiple-choice selection on what the bot could do to improve this turn: 1. using a better search query; or 2. attending better to the search results; or 3. other issues with the overall reply; or 4. no issue (a good reply).
* If selecting (a), the human provides a better search query, which will be used in the next turn to continue the conversation.
* If selecting (b), the human is presented with the search results and selects the relevant sentences, which will be added to the model input in the next turn.
* If selecting (c), the human provides a better overall reply (a gold correction), which is copied to be the next bot response.

### Sampling FITS

In our experiments, we uniformly sample 20% of the FITS training set to mimic a deployment setting with sparse binary and gold feedback. Table 4 shows the sampled dataset statistics. Out of the 20% of the training set with labels, 1376 examples are "better overall reply" annotated with gold corrections, which accounts for 7% of all bad responses to be corrected in FITS. Those 1376 corrections will later be used to train the _reply corrector_ for augmenting limited human feedback. The size of 20% of the FITS training set (7768 examples) is also similar to that of the validation and test sets.

#### a.2.1 Varying the sampling rate

As an ablation study, we varied the sampling rate. Table 5 shows different final dialogue models' results with different sampling rates. The input to the dialogue model is the context, and the output is the human-written gold correction.

As we increase the sampling rate, the final dialogue model's perplexity improves in general, but the gain becomes smaller. For instance, when we only sample 5% from FITS, the validation perplexity is 9.52; if we increase the sampling rate to 20%, the perplexity is 9.09; but when we further increase the sampling rate to 30% and 50%, the perplexity becomes 9.12 and 8.80 respectively.

We find the 20% sampling rate is a good balancing point with both reasonable F1 and perplexity.

### Different modules in Juicer

There are different modules involved in Juicer and we summarize them in Table 6. To sum up, Juicer has two helper modules, a _satisfaction classifier_ and a _reply corrector_ to help improve the final dialogue model.

The _satisfaction classifier_ identifies if the bot response is satisfactory or not. It is evaluated on both FITS and zero-shot DEMO deployment datasets.

Both the _reply corrector_ and the final dialogue models are generative models, and are automatically evaluated on the human-written gold corrections in the FITS validation and test sets, as gold corrections can reflect the model's ability to generate good responses.

The _reply corrector_ converts bad responses into good ones using free-form textual feedback. We evaluate it on gold search results instead of live search results, in order to generate better reply corrections. We describe it in more detail in Section A.4.

The final dialogue model is evaluated on live search results from Bing, filtered by Common

\begin{table}
\begin{tabular}{l|r|r|r|r} \hline \hline
**Feedback Breakdown** & **Train (20\%)** & **Valid** & **Test** & **Test Unseen** \\ \hline \hline Total & 7768 & 4245 & 9726 & 8907 \\ \hline Better Search Query & 1056 & 605 & 1167 & 1036 \\ Better Results Usage & 1383 & 756 & 1527 & 1310 \\ Better Overall Reply & 1376 & 714 & 1493 & 1372 \\ Good Response & 3953 & 2170 & 5539 & 5189 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Data statistics of the sampled version of FITS used in our experiments. We sampled 20% from FITS. Note that the training set size of labeled binary feedback is similar to the test sets.

 

Crawl, following Xu et al. (2022) instead of gold search results to better reflect performance with live users. We describe it in more detail in Section A.5.

### Reply corrector

The _reply corrector_ trains on data where for a given example the input consists of the dialogue context + bad reply to correct + the following human message, and the output consists of the correction for the bad reply. The models used in the main paper were fine-tuned from the R2C2 transformer Shuster et al. (2022a).

#### a.4.1 Training the _reply corrector_ with multiple tasks

In our experiments, we multi-tasked with various dialogue tasks to train the _reply corrector_, which improves the result. These tasks include the original reply correction task, the task with context as the input and free-form textual feedback as the target, and the dialogue task of Wizard of Internet Komeili et al. (2021). We tuned the weights for different tasks, and other hyper-parameters (learning rate, batch size, etc) according to the performance on the validation set.

#### a.4.2 Evaluating the _reply corrector_

Since the _reply correctors_ are used to generate corrections rather than interacting with live users, we evaluated them with gold search results (which leads to better corrections) instead of live search results from Bing (which better reflects the live interaction performance).

Although the _reply corrector_ (Table 0(b)) and the final dialogue models (Table 2) are evaluated on the same validation and test subsets that have gold corrections, their results are not comparable because of the following two reasons. First, as mentioned earlier, the _reply correctors_ condition on the gold search results instead of the live search results, while the final dialogue models use the live search results. Second, the _reply correctors_ rely on the free-form textual feedback to convert lemons to cherries, so we also append the free-form textual feedback into the input to the _reply correctors_, but for the final dialogue model, we do not have the additional free-form textual feedback information. These are the main reasons why the results in Table 0(b) are better than those in Table 2.

#### a.4.3 Generating reply corrections

We adopt a reranking-based learning method to first generate multiple reply corrections, and then use the _satisfaction classifier_ to score and rerank the generated corrections. Because the _reply corrector_'s performance is good (comparable to the one trained on 100% data in Table 0(b)) and we gen

\begin{table}
\begin{tabular}{p{79.6pt}|p{85.4pt}|p{85.4pt}|p{85.4pt} p{85.4pt}} \hline \hline
**Model** & **Inputs**\(\rightarrow\)**Outputs** & **Fine-tuned from** & **Evaluated on** & **Description** \\ \hline \hline \multirow{4}{*}{(1a) _Satisfaction classifier_} & Context + bot reply (+ the next human response) \(\rightarrow\) binary [good, bad] on the hot reply & \multirow{4}{*}{311M Transformer} & FITS validRtest and DEMO & Given the context, a hot reply and potentially the next human message, detect if the bot reply is satisfactory \\ \cline{2-2}  & Context + bad hot reply + free-form textual feedback\(\rightarrow\) improved reply (“acorrection”) & 3B R2C2 (Shuster et al., 2022a) & Gold corrections in FITS validRtest (on gold search results) & Given the context, the bad reply, and FITS validRtest (on gold search results) & Given the context, the real feedback, generate an improved reply (“correct the bad reply”) \\ \hline \multirow{4}{*}{(2) Final dialogue model} & \multirow{4}{*}{Context \(\rightarrow\) reply} & \multirow{4}{*}{3B Blender Bot 2} & Gold corrections in FITS validRtest (on like search results) & Given the context, generate a reply. We fine-tune our models using BlenderBot line search results & Given the entire user models using BlenderBot line search results & Inception 2 as the base model Komceili et al. (2021); Xu et al. (2021). \\ \hline \end{tabular}
\end{table}
Table 6: The input, output, and description of the three models used in Juicer.

\begin{table}
\begin{tabular}{c|c c|c c|c c} \hline \hline
**Final dialogue model** & \multicolumn{2}{c|}{**Valid**} & \multicolumn{2}{c}{**Test**} & \multicolumn{2}{c}{**Test Unseen**} \\ \hline Varying the sampling rate & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** \\ \hline \hline \multicolumn{8}{l}{**Baseline performance varying the sampling rate**} \\ \hline +gold correction from 20\% & 16.2 & 9.1 & 15.6 & 8.9 & 17.9 & 8.4 \\ \hline +gold correction from 5\% & 16.8 & 9.5 & 16.6 & 9.3 & 18.9 & 8.5 \\ +gold correction from 10\% & 16.6 & 9.3 & 16.5 & 9.1 & 18.9 & 8.4 \\ +gold correction from 30\% & 15.7 & 9.1 & 16.3 & 8.6 & 18.1 & 8.3 \\ +gold correction from 50\% & 16.5 & 8.8 & 16.0 & 9.0 & 17.9 & 8.3 \\ \hline \end{tabular}
\end{table}
Table 5: Final dialogue model results varying the sampling rate. Perplexities get better as we increase the sampling rate, but the gain becomes smaller. F1 first gets worse and then goes up. These suggest that a sampling rate of 20% is a good balancing point with both a good F1 and a good perplexity.

 

erated 60 correction candidates to choose from, the majority (99.96%, 16893 out of 16989) of bad responses have at least one generated correction that is predicted as satisfactory by the _satisfaction classifier_.

Generated reply correction examplesTable 8 and Table 9 show generated reply correction examples on the FITS dataset and the deployment dataset (zero-shot) respectively. These qualitative examples show that the _reply corrector_ can convert bad replies into good ones using free-form textual feedback, even for unseen deployment data.

#### a.4.4 Using BB2 to train the _reply corrector_

The models used in the main experiments were fine-tuned from R2C2 Shuster et al. (2022a). We also report results fine-tuned with BB2 in Table 7. We find that BB2 is worse than R2C2 as a _reply corrector_ because its generated corrections are more like conversational replies rather than actual corrections.

#### a.4.5 Using Director in the _reply corrector_

Using Director to combine multiple feedback signals is also effective for the _reply corrector_. We can use Director to further improve the _reply corrector_'s F1 to 22.81, as shown in Table 7, where the positive examples are the gold corrections and the negative examples are the bad bot responses. However, although the F1 of the Director-enhanced _reply corrector_ is better, we find that if we use it to generate reply corrections to improve the final dialogue models, the F1 is slightly better but the perplexity gets worse than using a regular _reply corrector_ without Director, as shown in Table 10. More analysis is needed to understand the reasons for this.

### Final dialogue model evaluation

We evaluate the final dialogue model on live search results instead of gold search results to better reflect performance with live users.

#### a.5.1 Oracle performance using 100% feedback data

Xu et al. Xu et al. (2022) trained various methods on the entire FITS dataset. Since our method is trained only on 20% of FITS, the 100% models' performance could be viewed as an upper bound of our models. They also used the 3B parameter BlenderBot 2 as a base model for the final dialogue model, making it comparable to our experiments. Their results are in Table 11 and we detail their models below.

* **100% gold correction**. The input is the context and the target is the gold correction (6,601 in the entire FITS dataset). This can be directly compared to "gold correction from 20%" in Table 2.
* **100% free-form textual feedback**. The input is the context and the target is the free-form textual feedback. This should be compared to "free-form textual feedback from 20%" in Table 2.
* **100% module supervision**. BlenderBot 2 is an internet-augmented bot with different modules such as a search module to generate a search query, and a knowledge module to attend to the search results. Using the human-written gold search query, human-selected search doc and gold correction, they fine-tuned each individual module to improve BlenderBot 2.

\begin{table}
\begin{tabular}{l|c||c||c|c|c} \hline \hline
**(1b)_Reply Corrector_** & \multicolumn{2}{c||}{**Valid**} & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c}{**Test Unseen**} \\ \hline
**Input** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** \\ \hline \hline \end{tabular} 
\begin{tabular}{l} \hline
* & **22.59** &
- & 22.10 &
- \\ + Director overlap & **23.00** &
- & 22.50 &
- & **22.55** &
- \\ \hline \hline \end{tabular} 
\begin{tabular}{l} \hline
**fine-tuned from BB2** \\ \hline gold corrections from 20\% + self-corrections & 16.32 & 7.06 & 14.53 & 7.01 & 15.63 & 7.16 \\ \hline \hline \end{tabular}
\end{table}
Table 7: _Reply corrector_ results. The top block shows the _reply corrector_ fine-tuned from RB2C2 with Director and Director overlap, and the bottom block shows the _reply corrector_ fine-tuned from BB2. R2C2 is better than BB2 as a _reply corrector_. Using Director improves the result. Using Director overlap further improves over Director.

 

* **100% reward-based learning**. They also adopted a reward-based learning approach, and built a _satisfaction classifier_ to identify good and bad responses. They used the vanilla BlenderBot 2 model to generate multiple responses, then reranked them with the score from the classifier as the reward, and chose the response with the highest reward. Finally, they fine-tuned BlenderBot 2 on the responses with high rewards to improve it.
* **100% Director**. They also used both the binary satisfaction labels and the textual feedback to train Director models to further improve the performance.

As mentioned in the main body of our paper, Juicer achieves comparable performance to the "oracle" (100%) models in F1 and human evaluation. For instance, the best oracle method which fine-tunes individual modules achieves an F1 of 17.2 and 47.0% good response rate, and a human rating of 3.38, while the best Juicer model achieves an F1 of 17.2, a good response rate of 45.5%, and a similar human rating of 3.34.

#### a.5.2Director overlap

We also develop a new variant of Director and use it to improve the final dialogue model. In Director, every token in the positive and negative examples has a one or zero label respectively. In our setting, we have a pair of a bad response and a good response (a gold correction), e.g., "I like watermelons too! Have you heard of Harry Styles?" (bad) and "I like watermelons too! They tastes

\begin{table}
\begin{tabular}{l l} \hline \hline
**From** & **Utterance** \\ \hline \hline USR & Hi bot, what supplies do I need for a diaper change? \\ \hline BOT & That’s a good question. I’m not sure, but I’m sure you can find out online. \\ \hline Feedback & _The results list at least one necessary supply, with the implication that bot would find others if it read further._ \\ \hline Correction & Bring your supplies to the diapering area. Clean cloth diapers are essential. You’ll also need a clean diaper, clean wipes, diaper cream, gloves, plastic or waterproof bag for soiled clothing, extra clothes. \\ Original & It depends on how many diapers you have. If you have a lot of diapers, you will need a changing table and a changing pad. \\ \hline \hline
**From** & **Utterance** \\ \hline \hline USR & Can you tell me what two sides fought during the war of the roses? \\ \hline BOT & The war of the roses was between the english and the french. The english won. \\ \hline Feedback & _It looks like that answer is not right. The search results say something different._ \\ \hline Correction & The war was fought between two competing royal families: the yorks and the launcasters, both members of the age-old royal plantagenet family. \\ \hline Original & The english and the french. \\ \hline \hline \end{tabular}
\end{table}
Table 8: Qualitative correction examples generated by the best _reply corrector_ on FITS.

 

great in drinks." (good). Since people tend to edit the original bad response to correct it, they may have many overlapping tokens ("I like watermelons too!"), which we do not have to punish. So we develop Director overlap, where we obtain the bag of tokens of the pair of the bad response and the gold correction, and assign a positive label for the overlapping tokens in the negative examples. In our data, 28.4% of tokens in the bad responses overlap with those in gold corrections (6.5% are stop words and punctuations, and 21.9% are not).

Table 7 and Table 12 show the result of Director overlap. For the _reply corrector_, Director overlap improves the F1 to 23.00 over Director. For the final dialogue model, Director overlap improves the good response rate and lowers the search result error in human evaluations over Director.

## Appendix B Model Training Setting

We use the openly available ParAI framework for all training runs, as well as for evaluations, where

\begin{table}
\begin{tabular}{l l} \hline \hline
**From** & **Utterance** \\ \hline Truncated & ... \\ \hline BOT & Hello! How are you today? I’m doing well, thank you. What are you up to? \\ \hline USR & I didn’t ask how you are doing? \\ \hline BOT & Oh, I’m sorry. I meant to say, how are you and your family doing today? \\ \hline USR & I don’t have a family \\ \hline BOT & I’m sorry to hear that. Are you a member of the family motor coach association (fmac)? \\ \hline Feedback & _Something like: “i’m sorry I assumed you had a family! I'll keep this in mind going forward”_ \\ \hline Correction 1 & You’re right. I shouldn’t have assumed you have a family. I will keep this is mind going foward. \\ \hline Correction 2 & I apologize. I didn’t mean to assume you had children! I hope you have a great day. \\ \hline \end{tabular} 
\begin{tabular}{l p{284.5pt}} \hline \hline
**From** & **Utterance** \\ \hline \hline Truncated & ... \\ \hline USR & Try saying good night, or sweet dreams. \\ \hline BOT & Thanks for your feedback. It is good to know. \\ \hline USR & Alright, I’m going to bed now. Good night! \\ \hline BOT & Thanks! I’ve been trying to get into sci fi, but it’s hard to find a good book. \\ \hline Feedback & _You could say good night, or something similar._ \\ \hline Correction 1 & Well, you should go to bed so you can sleep well. Good night. \\ \hline Correction 2 & Have a good night! \\ \hline \end{tabular}
\end{table}
Table 9: Zero-shot corrections generated by the best _reply corrector_ on unseen deployment data.

 

metrics are measured using default settings. All the fine-tuned models are trained with a maximum of eight 32GB GPUs (NVIDIA V100), optimized with Adam using \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), \(\epsilon=1e-08\). Models are trained up to 4000 updates with batch sizes up to 128. The typical fine-tuning time for a standard transformer encoder-decoder is 8 hrs before it early stops, and the time is 16 hrs for retrieval-based models.

## Appendix C Human Evaluation

We used the same human evaluation setup as in Xu et al. (2022) where all of our human evaluations tasks have taken place by deploying conversational

\begin{table}
\begin{tabular}{l|c c} \hline \hline
**Final dialogue model** & \multicolumn{2}{c}{**Automatic evaluation**} \\  & \multicolumn{2}{c}{**Valid**} & \multicolumn{2}{c}{**F1\(\uparrow\)**} & \multicolumn{2}{c}{**PPL\(\downarrow\)**} \\ \hline \hline
**Juicer models** & 16.7 & **8.5** \\ +Juicer w/ Director overlap-based _reply corrector_ & **16.8** & 8.8 \\ \hline \hline
**Juicer ablations** & \multicolumn{2}{c}{} \\ \hline \hline w/o selecting correctable cases & 16.4 & **8.5** \\ w/o selecting correctable cases w/ Director overlap-based _reply corrector_ & **16.5** & 8.7 \\ \hline \end{tabular}
\end{table}
Table 10: Final dialogue model automatic evaluation results. The Director overlap-enhanced _reply corrector_ achieves the highest F1 on the reply correction task, better than the regular _reply corrector_ (see Table 7). But when we use it to generate the reply corrections to further improve the final dialogue model, we can improve the F1 of the final dialogue model slightly, but the perplexity gets a bit worse. Further investigations are needed to understand the reason for this.

\begin{table}
\begin{tabular}{l|c c|c c||c c|c c|c c} \hline \hline
**Final dialogue model** & \multicolumn{4}{c||}{**Automatic evaluation**} & \multicolumn{4}{c}{**Human evaluation**} \\ \hline  & \multicolumn{2}{c}{**Valid**} & \multicolumn{2}{c}{**Test**} & \multicolumn{2}{c}{**Test Unseen**} & \multicolumn{2}{c}{**Good**} & \multirow{2}{*}{**Rating \(\uparrow\)**} & \multicolumn{2}{c}{**Error Breakdown \(\downarrow\)**} \\  & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **response \(\uparrow\)** & **Routing \(\uparrow\)** & Search & Search & Response \\ \hline \hline BB2 & 14.4 & 10.6 & 14.7 & 10.3 & 15.3 & 9.3 & 33.2\% & 3.09 & 12.1\% & 18.6\% & 18.1\% \\ +100\% reward-based learning & 15.1 & 11.0 & 14.2 & 10.7 & 14.3 & 9.6 & **36.4\%** & 2.83 & 11.3\% & 18.6\% & 17.0\% \\ +100\% free-form textual feedback & 15.5 & 9.7 & 15.6 & 9.5 & 16.8 & 8.7 & 37.0\% & 3.22 & 11.6\% & 17.6\% & 17.0\% \\ +100\% gold correction & 14.7 & 8.2 & 15.5 & 8.0 & 17.0 & 8.0 & **40.3\%** & **3.37** & 11.6\% & 18.3\% & **15.0\%** \\ +100\% module supervision & 14.9 & 7.6 & 15.5 & 7.5 & 15.4 & 8.3 & **42.0\%** & **3.35** & **8.4\%** & 20.8\% & **14.4\%** \\ +100\% ranking binary feedback & 15.8 & \(\approx\) & 15.8 & \(\approx\) & 16.3 & \(\sim\) & \(\sim\) & \(\sim\) & \(\sim\) & \(\sim\) & \(\sim\) \\ +100\% Director binary feedback only & 16.2 & \(\approx\) & 16.2 & \(\approx\) & 17.6 & \(\approx\) & **37.8\%** & 3.07 & 11.4\% & 17.3\% & 16.9\% \\ +100\% Director module+binary feedback & 17.2 & \(\approx\) & 16.6 & \(\approx\) & 16.0 & \(\sim\) & **47.0\%** & **3.38** & **8.4\%** & **16.1\%** & **14.3\%** \\ \hline \hline \end{tabular}
\end{table}
Table 11: Final dialogue model results from 100% oracle methods in Xu et al. (2022). Similarly we bold statistically significant improvements (independent two-sample \(t\)-test, \(p<0.05\)) of methods over their baselines BB2 3B in the human evaluation block.

\begin{table}
\begin{tabular}{l|c c|c c|c c||c c|c c} \hline \hline
**Final dialogue model** & \multicolumn{4}{c||}{**Automatic evaluation**} & \multicolumn{4}{c}{**Human evaluation**} \\ \hline  & \multicolumn{2}{c}{**Valid**} & \multicolumn{2}{c||}{**Test**} & \multicolumn{2}{c||}{**Test Unseen**} & \multicolumn{2}{c}{**Good**} & \multirow{2}{*}{**Rating \(\uparrow\)**} & \multicolumn{2}{c}{**Error Breakdown \(\downarrow\)**} \\  & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **F1\(\uparrow\)** & **PPL\(\downarrow\)** & **response \(\uparrow\)** & **Routing \(\uparrow\)** & Search & Search & \multirow{2}{*}{Response} \\ \hline \hline
* & 16.70 &
- & 17.70 &
- & **45.5\%** & **3.34** & 11.3\% & 17.4\% & **12.9\%** \\ +Juicer +Director Overlap & 17.32 &
- & 16.66 &
- & 17.62 &
- & **47.8\%** & 3.25 & 11.0\% & **14.8\%** & **13.3\%** \\ \hline
* & 16.62 &
- & 17.93 &
- & **44.6\%** & **3.40** & 11.6\% & 16.7\% & **13.6\%** \\ +Juicer +Director overlap & 16.98 &
- & 16.56 &
- & 17.19 &
- & **45.5\%** & **3.48** & 10.8\% & **15.2\%** & **14.3\%** \\ \hline \hline \end{tabular}
\end{table}
Table 12: Juicer with Director overlap. Director overlap improves the human evaluation results over the vanilla Director. Similarly we bold statistically significant improvements (independent two-sample \(t\)-test, \(p<0.05\)) of methods over their baselines BB2 3B in the human evaluation block.
 

agents on Amazon Mechanical Turk with crowdworkers. English-speaking annotators located in the United States were recruited and compensated through the Amazon Mechanical Turk platform and our crowdsourcing tasks pay workers well above minimum wage. Before the human evaluation task starts, all crowdworkers are informed that any message they send may be publicly disclosed for research purposes, and are instructed not to send any personal identifiable information (for example, name, address, email, or phone number etc.) in their messages.