Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning
Authors: Ryan Shea
Zhou Yu
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
persona consistent dialogue
offline reinforcement learning
TL;DR: We propose a framework to improve persona consistency with offline RL along with a simple, novel importance sampling method called VaRMI.

Abstract: Maintaining a consistent persona is a key quality for any open domain dialogue system. Current state-of-the-art systems do this by training agents with supervised learning or online reinforcement learning (RL). However, systems trained with supervised learning often lack consistency as they are never punished for uttering contradictions. Additional training with RL can alleviate some of these issues, however the training process is expensive. Instead, we propose an offline RL framework to improve the persona consistency of dialogue systems. Our framework allows us to combine the advantages of previous methods as we can inexpensively train our model on existing data as in supervised learning, while punishing and rewarding specific utterances as in RL. We also introduce a simple importance sampling method to reduce the variance of importance weights in offline RL training which we call Variance-Reducing MLE-Initialized (VaRMI) importance sampling. Our automatic and human evaluations show that our framework improves both the persona consistency and dialogue quality of a state-of-the-art social chatbot.

Building Persona Consistent Dialogue Agents with Offline Reinforcement
Learning
Ryan Shea Columbia University, NY 1s42355ccolubb/ edu
Zhou Yu Columbia University, NY 2y24616columbia. edu
Abstract
the agent (Zhang et al., 2018). This persona con- sists of descriptions such as "I enjoy skiing" or have blonde hair" (Figure 2). Given the advantages of persona grounded dialogue, previous research has been focused making dialogue agents more per- sona consistent (Liu et al., 2020; Song et al., 2020). Existing methods to improve persona consistency typically centered around the of supervised learning or online RL (Song et al., 2019b; Zhang et al., 2018). These methods have been somewhat successful, but still face many problems. Super- vised learning methods only focus on encouraging persona entailing examples without properly pun- ishing contradictions. This results in dialogue sys- tems that are insensitive to contradictory utterances, leading to inconsistency (Kim et al., 2020).
Maintaining consistent persona is a key qual- ity for any open domain dialogue system. Cur- state-of-the-art systems do this by training agents with supervised learning or online re- inforcement learning (RL). However, systems trained with supervised learning often lack con- sistency as they are never punished for utter- ing contradictions. Additional training with RI can alleviate some of these issues, however the training process is expensive. Instead, we propose offline RL framework to improve the persona consistency of dialogue systems. Our framework allows us to combine the ad- inex-
vantages of previous methods pensively train our model on existing data as supervised learning, while punishing and rewarding specific utterances in RL. We also introduce a simple importance sampling method reduce the variance of importance weights in offline RL training which we call Variance-Reducing
Some work has attempted resolve the problems with supervised learning through the use of online RL (Song al., 2019b; Liu al., 2020). How- ever, the training process for RL is quite expensive since the dialogue model must continuously gen- erate new training samples. Furthermore, online RL methods require the use of accurate critics to evaluate the generated bot utterances. These critics must incentivize persona consistency while also enforcing strong constraints dialogue fluency, as without them the model will degenerate (Verma et al., 2022: Song et al., 2019b). This requires train- ing multiple, separate critic models or using human critics during training which is also expensive.
(VaRMI) importance sampling- Our automatic and hu- man evaluations show that our framework im- proves both the persona consistency and dia- logue quality of tate-off-thear social chat- bot.
1 Introduction
The rapid advancements of large language mod- els in recent years has enabled the development of dialogue agents that can generate remarkably fluent and natural responses (Shuster et al., 2022; Thoppilan et al., 2022). These dialogue systems are typically trained on large amounts of unlabeled text data with some additional fine-tuning on dialogue tasks. While this does allow models to effectively learn many of the patterns and syntax of natural language, dialogue agents still suffer from many problems including lack of consistency (Li et al., 2020; Kim al., 2020: Song al., 2019a).
Given these challenges, we propose an offline RL framework to improve the persona consistency open domain dialogue systems (Figure 1). Offline RL has several advantages over existing training methods. Unlike supervised learning, offline RI. explicitly punishes contradictory utterances during training. This further improves persona consistency by making the bot more sensitive to contradictions. Unlike online RI, offline RI. does not require our dialogue model to generate new samples during training. Instead, we can inexpensively train our
To resolve consistency issues the context of social dialogue, prior work has proposed condition- ing dialogue generation on a persona describing

---

Persona: have a cat. blonde.
Predictions
No, I do not.
Context:
LLM (BB3)
Policy Gradient
Fixed Critic
P1: How are you?
good.
P1: you have pets?
Offline Samples
Yes, have a cat.
Figure 1: An overview of our offline RL training framework. Our setup similar to that of supervised learning. The key difference heing that our offline samples can have either positive or negative reward associated with them, determined by our critic. The policy gradient obtained by weighting our loss gradient by this reward along with importance sampling.
model using large existing datasets that have been collected/synthheiized for supervised learning. We exploit this pre-existing data to train our model on human annotated reward labels instead of classi- fier based rewards which common in online RL. Training human-annotated rewards also re- duces the chance of training failures due to pol- icy divergence. This can arise in settings where value function approximation needed to deter- mine Q-values and may require the use of behavior regularization (van Hasselt et al., 201 8; Wu et al., 2019).
2018; Roller et al., 2020). Our contributions can be summarized as follows:
We propose an offline RL framework to build persona consistent dialogue agents. This in- cludes persona consistency critic that uses ground truth, human annotated rewards in- stead of noisy, classifier-based rewards.
We introduce VaRMI, simple importance sampling method to reduce the variance importance weights in policy gradient offline RL training.
Despite the advantages of offline RL, offline RL training can suffer from high variance due to the need for importance sampling. To alleviate this, we introduce an importance sampling method called VaRMI to reduce the variance of importance weights. This method can be applied beyond our task to other settings where policy-gradient offline RL training is used.
Our approach improves the persona consis- tency of BlenderBot3 (BB3) according to both automatic and human evaluations. Along with improving persona consistency, human evalu- ations also show that our approach improves the dialogue quality of the model.
Related Work
Prior work has explored the application of of- fline RL on task-oriented dialogue (Verma al., 2022: Snell et al., 2022; Jang al., 2022). Task natural extension of offline RL as crafting a reward function is straightforward. Applying offline RL to social dialogue is less clear there is no obvious reward to use for our policy. We exploit the fact that persona consistency is key component of open domain dialogue. Intuitively, this makes sense as humans naturally speak with persona during a conversation. Prior studies have shown that improving persona consistency also im- proves the quality of social dialogue (Zhang et al.,
Persona Consistent Dialogue In recent years, persona-based dialogue generation has typically been centered around the PersonaChat dataset (Zhang et al., 2018). One easy method to achieve persona consistent dialogue is to simply fine-tune model on this dataset using supervised learning (Roller al., 2020; Shuster et al., 2022; Yavuz et al., 2019). However agents trained in this man- ner still suffer from consistency issues for reasons discussed previously.
oriented dialogue
Given this, prior work has been centered around improving persona consistency by more explicitly

---

Persona (Model)
Budzianowski al., 2018).
have 2 cals.
Furthermore many previous studies choose to use offline RL methods centered around Q-learning (Jaques et al., 2020; Snell al., 2023). While these methods can be effective for dialogue tasks, they require training additional models to steer the dialogue policy towards optimal actions. This adds to both the complexity and resources needed to train and deploy dialogue models for real world applications. We introduce a policy-gradicn based offline RL framework with fixed rewards which eliminates the need to any additional models during training or deployment. Instead our training method can be set up supervised learning task with modified loss, which gives the advantage of being much more simple and efficient to train and deploy.
work middle school.
favorite color yallow.
dislike country music.
Entailment
Dialogue
hi, you
Model: dol have 2 cats.
Contradiction
Human: cool, what
names your
Model: do not have any what
name? Human: don't have a cat, just dog. do you play any sports?
Neutral
Model: sadly,
church chair,
Figure 2: DNLI dataset from Welleck et al., 2019b.
encouraging entailing utterances and discouraging contradictory ones. Many of these methods involve the use of online RL such Song al., 2019b which uses a natural language inference (NLI) clas- sifier and naturalness module as a critic or Liu et al., 2020 which uses mutual persona perception. Some other approaches attempt to improve consistency without any additional training of the dialogue pol- icy. These methods use frameworks such mul- tistage re-writing (Song al., 2020) Bayesian rational speech acts (Kim et al., 2020: Frank and Goodman, 2012). Multistage re-writing is limited by its inability to handle multi-tum tulti-turnnnn persona consis- tency. On the other hand the Bayesian RSA has an increased computational cost during inference time due to the modified decoding scheme. This results longer response times from the bot well as the need for greedy decoding, which reduces response diversity and dialogue quality.
Despite these advantages, policy-gradicn offline RL has seen limited use in practice due to the high variance that arises as a result of importance sam- pling. Variance reduction for importance weights emerged in off-policy learning (Munos et al., 2016) and has been widely studied in the context of offline RL (Pang and He, 2021; Levine et al., 2020). Given this, introduce VaRMI to reduce importance weight variance and improve offline RL training.
Method
In this section, we discuss our offline RL frame- work to improve persona consistency well our novel method of importance sampling. Section 3.1 gives an overview of how offline RL training is performed. Section 3.2 details our VaRMI impor- tance sampling method. Lastly, section 3.3 outlines our framework and section 3.4 discusses how we implement our framework on a dialogue model.
Some methods also propose the use of unlike- lihood training as method to improve persona consistency (Li et al., 2020; Welleck et al., 2019a). However, unlikelihood training suffers from the fact that it does not explicitly reward entailing ut- terances and instead treats entailing and neural ut- terances as equally good (Li et al., 2020). Further- more, unlikelihood training punishes contradictory utterances at token level which can lead to inco- herent responses and uninterpretable behaviors (Shi et al., 2021). Our offline RL method can instead dis- till utterance level information about contradictions and entailment improve training and maintain coherence.
3.1 Offline RL
Our offline RL training approach uses a policy- gradient method optimize the RL objective. Which is defined as:
T Σ (() 1=
J(Ø) = E Brsp(so())
Offline RL Offline RL applications to dialogue tasks somewhat limited, with the ones that have been proposed focused on task oriented di- alogue. This includes tasks such as price negotia- tion (Verma et al., 2022) or task oriented dialogue benchmarks such as MultiWOZ. (Jang et al., 2022:
where 7 denotes trajectory of states, 8t, and actions, az and y denotes the discount factor. The policy gradient is obtained by directly computing the gradient of the RL objective with respect to our

---

Pi's Persona
PersonaChat dataset. Therefore our model will not encounter issues where it utters incoherent, nonsen- sical responses which is a common problem when performing training online.
was born in london.
had gig local theater last night. work as stand up comedian.
my favorite drink cuba libre. i did a few small roles in tv series.
When using policy-gradient methods for online RL we collect samples from our policy directly to compute the gradient with respect to our policy, However in offline RL our samples come from some behavioural policy %6 that is different from the policy we want to optimize.
Dialogue Context
P2: lol. i am shy, anything to break the ice, and i am a beatles fan.
In order to estimate expectations under our pol- icy Te given samples from T6 we can use impor- tance sampling to obtain an unbiased estimator our policy gradient (Precup et al., 2000):
Pl: can tell. am not, you can see me in some tv shows
P2: really? what shows? like tv, it makes mc
forget i do not like my family
Candidate: wow, wish had big family. grew up in very small town.
VoJ(O)-
Reward:-1
E (x-((=((7)
Σ wi Vo Volog no (a(l)s))))))00) a1)
Pi's Persona
i like to go hunting.
where Wy To= valaplee) sylap(e))) are importance
i like to remodel homes.
weights. In practice we use per-action approxi- mation our importance weights wy (a))))))) to reduce the variance of our gradient at the cost of adding bias to the estimator. Empirical work has shown that this approach can work well when our two policies are sufficiently similar (Levine et al., 2020; Pang and He, 2021; Serban al., 2017), which we argue is the case here since initialize To to the ML.E solution of our task before training.
i like to shoot bow.
ma(a; [x]
my favorite holiday halloween.
Dialogue Context
P2: hi, how are you doing? am getting ready to do some cheetah chasing to stay in shape. Candidate: you must be very fast. hunting is one of my favorite hobbies. Reward:
Given that we do not know the T6 that produced our samples, we need to make some assumptions about our behavioral policy in order to derive our importance weights during training. We test two different assumptions to derive these importance weights which described next.
Table 1:
Example of dialogues in mapped dataset. Entailing dialogue candidates have reward of and contradictory dialogues have reward -I.
policy (Williams, 1992) which gives:
For our first importance sampling method, we sume that all of our training samples have the same likelihood under m, which allows us to ignore it during optimization. This gives us Wt = न (r((agls (s))))) for the importance weights. This method of im- portance sampling is what is used in the GOLD algorithm (Pang and He, 2021) and has been shown to work well in various scenarios where ль is un- known (Li et al., 2022). We refer to this importance sampling method as the GOLD method. Our sec- ond method of importance sampling is discussed in detail the next section.
VoJ(O)=
Brop(To(T)) Σ Ve log Ta (a(a)ss)) at)
where Q(5,, at) is the estimated return from the current state. In our case this is an utterance-level reward, taking value of -1 or given by our critic, which reflects whether not the utterance adheres the persona has been given. Our re- ward function does not consider response fluency as our training conducted offline (see Section 3.3 for reward function details). Our training samples only include fluent responses originating from the
3.2
VaRMI Importance Sampling
The biggest issue that faces policy-gradicns based offline RL methods is the fact that the gradient

---

estimator can have high variance (Levine et al., 2020). This comes from the fact that importance sampling is needed to correct for the distributional shift between Ty and Tg. We introduce VaRMI to alleviate this issue and improve training for policy- gradient offline RL.
Our critic is constructed by performing a map- ping between the dialogue natural language in- ference (DNLI) (Welleck et al., 2019b) and Per- sonaChat (Zhang et al., 2018) datasets (Figure 2). The PersonaChat dataset is a crowd sourced dia- logue dataset where two workers are given per- sona and asked to chat while adopting said persona. The dataset consists of 10,907 dialogues in total with 1,000 set aside for validation and 968 set aside for testing. The DNLI dataset contains 310, 10 sen- tence pairs from the PersonaChat dataset along with human annotated triples for each sentence. Each sentence pair comes with a label for entailment, neutrality, or contradiction which is based on the overlap between the triples.
For our VaRMI importance sampling method, we reduce the variance of our importance weights by taking advantage of the fact that we initialize my the MLE solution of our lask before beginning offline RL training. This means that Ty has already been trained on a large amount of positive reward examples and we can assume a minimal amount of distributional shift during offline RL. In other words, we are assuming that Ty has learned the Ty that generates "good" examples to arbitrary
Since the sentences in DNLI come directly from PersonaChat, we can easily perform a mapping between the two datasets to obtain set of dia- logue samples and corresponding entailment labels. When performing our mapping, we only consider pairs in the DNLI dataset that have one sentence map to a dialogue utterance in the PersonaChat training set and have the other sentence map to a persona. We then add the DNLI persona to the existing persona set and use the matching sentence as the next-utterance candidate.
for positive reward candidates and wy <a(aa)s;) for our negative reward candidates. This simple method effectively eliminates large portion of our importance weights to reduce variance at the cost of adding bias to our estimator. In our sctting, this means that we set the importance weights of persona entailing utterances ("good" examples) to one and set the weight of contradictory utterances ("bad" examples) to their likelihood under our pol- icy.
degree. Therefore we set w/ (a(as))))
Since are inserting new personas into the PersonaChat dataset during the mapping process, we need to ensure that our data does not include persona sets where two personas contradict each other. To do this, we filter out any personas in our dataset that contradict the one we have inserted. We achieve this by using the human annotated triples corresponding to each persona. We take a conser- vative approach and remove any personas whose triples have any entity overlap.
Our use of VaRMI is limited to persona consis- tency, but can be applied other tasks as long as the following conditions hold.
1. There is some notion of absolute positive and negative rewards for the task. This is in con- trast relative positive and negative rewards that come from subtracting reward values by baseline.
2.
The acting policy has been initialized to the MLE solution for the task.
Each persona in the PersonaChat training set is present in the DNLI dataset, therefore we can use this method for all of the personas. We do some additional filtering with a NLI classifier (Liu et al., 2019) as there are situations for some longer personas where the triple does not capture all rele- vant information for deriving entailment. We also filter out all sentences that are labeled as neutral with respect to the inserted persona, we consider these utterances to have reward of zero. After performing our mapping and filtering, we are left with around 42K utterance candidates that can be used for training with offline RL.
These conditions are easily satisfied for a wide variety of tasks within dialogue and beyond. While this is promising, more work needs to done to de- termine how well this method generalizes to tasks with more complex rewards, longer time steps, and other tasks unrelated persona consistency. We leave this analysis for future work.
3.3 Framework
In this section we go over the details of our frame- work. This includes how we construct our critic to use human annotated rewards for persona consis- tency and how we generate our offline dataset.
An item in our dataset consists of a persona, dia- logue context, utterance candidate, and entailment

---

of BB3 for four epochs with both of our importance sampling methods. We use learning rate of 5e-7 for GOLD and 1e-6 for VaRMI. We implement our method within the ParlAl framework (Miller et al., 2017).
Nogative Loss GOLD
We find that BB3's modules for dynamic mem- ory, internet search, and memory decision tend to be error prone and degrade dialogue performance. Therefore we choose to disable these modules dur- ing deployment. Disabling them also helps us bet- ter isolate the effects of persona consistency as the model's responses are now only conditioned on its persona.
Loss YSRMI Nogutiiv Loss VarMi
Experiments
We test the effectiveness of our offline RL frame- work for persona consistency using both automatic and human evaluations. Our results show that both importance sampling methods are able to improve the persona consistency of BB3. Human evalua- tions also show that our VaRMI importance sam- pling method improves the overall dialogue quality of the model.
Figure 3: NLL loss trajectories for the positive and neg- ative utterance candidates on our mapped PersonaChat- DNLI test set when training with offline RL. The loss trajectories for GOLD are shown on the top and the trajectories for VaRMI are shown on the bottom.
4.1 Evaluation Datasets
label. The persona and dialogue context are con- catenated together to form state, the utterance candidate is used as т, and our estimated return is given from the entailment label. Example di- alogues from our mapped dataset can be seen in Table
DNLI Evaluation Set Along with the with the base DNLI dataset, Welleck et al., 2019b also release separate evaluation set to test the per- sona consistency of dialogue models. Whereas the the base DNLI dataset contains sentence pairs along with entailment labels. The evaluation datasct consists of personas and dialogue historics from the PersonaChat evaluation set along with 31 utterance candidates. Ten of these candidates are contradictory, ten are entailing, ten are neutral, and one is the actual next utterance. The model then ranks these candidates with the goal of ranking gold and entailing utterances highest. The evaluation set contains a total of 542 dialogues for testing.
3.4 Implementation
We implement our method on BlenderBot3 (BB3), an open-source, state-of-the-art dialogue system de- veloped by Meta for open-domain dialogue (Shus- ter et al., 2022). BB3 has already been fine-tuned on the several datasets, including PersonaChat, in an attempt to "blend" several conversational skills. BB3 achieves a perplexity of z 5.8 on the Per- sonaChat dataset and the authors note that per- forming additional fine-tuning results in overfitting (Shuster et al., 2022). While BB3 has been shown to perform well across a variety of conversational domains, it has been known to suffer from consis- tency issues, with human evaluations showing that it is actually less consistent than the first iteration of BlenderBot (Roller al., 2020; Shuster al., 2022).
Mapped DNL.IPPerSonaCha Dataset We also perform evaluation on 5k dialogues from our mapped dataset. We hold out these dialogues from our training and split them into positive and nega- tive utterance candidates based their entailment. The goal of our offline RL framework is encour- age entailing candidates and discourage contradic- tions. By tracking model performance on these two sets we can evaluate the success of our training methods.
We train the three billion (3B) parameter version

---

Model
Hits@1 1
Entail@1 11
Rand @ 1
Contradict @ 14
BB3
26.6
29.5
13.3
30.6
BB3+RL 27.5
30.4
10.9
31.2
BB3+GOLD
37.5"
37.3*
5.4
19.9
BB3+ VaRMI
37.6"
37.6"
4.4"
20.3*
Table Results of our importance sampling techniques vs the BB3 and BB3+RL baselines on the DNLI evaluation dataset. The best results for each category shown in bold. Statistically significant improvements (independent two-sample z-test, < 0.05) over the baselines are marked with"
Raw
Calibrated
Model
Quality
Consistent
Quality
Consistent
BB3
3.20 (1.03)
3.67 (1.32)
3.18 (0.26)
3.66 (0.27)
BB3+GOLD
2.77 (1.17)
4.30 (0.84)*
2.72 (0.28)
4.23 (0.24)*
BB3+VaRMS
3.33 (0.99)
3.97 (1.01)
3.35 (0.28)*
3.99 (0.25)*
Table 3: Human evaluation results of our two importance sampling techniques vs the BB3-3B baseline. The best scores for each category are shown in bold. The numbers shown in parentheses are the standard deviations for the scores. Statistically significant improvements (independent two-sample t-test, p < 0.05) over the BB3-3B baseline are marked with".
4.2 Automatic Evaluation
and therefore there is less room for improvement on these examples.
Results on Mapped DNLI-PersonaChat Dataset Figure 3 shows the resulting loss trajectories on our positive and negative utterance sets over the course of training. Epoch shows the loss on both sets before any offline RL training is performed. We can see that the gap in loss between both sets is relatively small at this point, which indicates that our baseline model is less sensitive to contradictory utterances.
Results on DNLI Evaluation Dataset Table 2 shows the results our training methods on the DNLI evaluation dataset. We compare these results against BB3 trained using imitation learning only as well as a bascline trained with online RL using the framework defined in Song et al., 2019b. Full details of how we implement this baseline can be found in Appendix A.
When performing training with GOLD the loss for both sets increases over the course of training. However, we do note that the loss for the negative candidates increases more than for the positive can- didates. This suggests that our model is becoming more sensitive to contradictions although it may also be being disincentivized to picking entailing utterances, albeit to lesser degree.
Hits@1 indicates the percentage of top-1 can- didates returned by the model that match the gold next utterance value. Entail @ indicates the per- cent of top candidates returned by the model that have the same underlying triple as the gold next utterance value, this can be viewed as a more le- nient version of Hits 1. Contradict @ indicates the percent of top candidates returned by the model that have a triple that contradicts the triple for the gold next utterance value. Lastly, Rand @ 1 indi- cates the percent of top candidates returned by the model that have triple that neither contradicts nor entails the triple for the gold next utterance value. Both methods of offline training outperform the baselines this task with the GOLD method per- forming the best reducing contradictions while the VaRMI method does the best in all the other categories. This includes ranking both gold and entailing utterances highly and reducing the num- ber of neutral candidates. While neutral utterances
The results with VaRMI training are more aligned with what we expect. After training for four epochs, the loss on the positive candidates has decreased below what its value was prior to training with offline RL while the loss on the negative can- didates has nearly doubled. This suggests that this method is successfully incentivizing our model to choose entailing utterances and avoid contradictory ones. We also see that the loss on the contradictory utterances changes much more than the loss for en- tailing utterances. This is likely due to the fact that our model has already been trained on many per- sona entailing examples during imitation learning

---

Chatbot Persona (BB3 Baseline)
1-5. They could also optionally provide any com- plaints/suggestions they had about the bot in text box. More details on our human evaluation can be seen in Appendix C.
my favorite band is nirvana.
i like running.
i am musician.
my father was a police officer. i work at pet smart.
The bot's persona was randomly selected from a set of 967 personas from the PersonaChat dataset. Users were only shown the bot's persona after they completed the chat and could reference both the bot's persona and their chat history during the post- chat survey.
Dialogue
BB3: What kind of music do you like to listen to? I'm a musician myself.
Results The results of our human evaluation are shown in Table 3. Given that we have different user for each conversation, and to remain consis- tent with prior work in the area (Kim al., 2020; Welleck et al., 2019b), we apply Bayesian calibra- tion (Kulikov et al., 2019) to our results to correct for annotator bias and inter-annotator variability. BB3: My parents were police officers, so I never learned how to play instruments.
BB3: Do you play any instruments? play guitar and piano, but I'm not very good.
Human: But you are musician...?
BB3: Yes, am a professional musician. Do
you have a favorite song?
Human: You are a musician, then why can't play good instruments?
Our results show that both of our offline RL methods improve the bot's consistency with respect to its persona, with GOLD doing the best in this regard. While our VaRMI importance sampling method also improves the dialogue quality over the BB3 baseline, the GOLD importance sampling method performs worse in terms of quality com- pared to both other methods.
Table conversation snippet from the human evalu- ation of our BB3 baseline bot. The relevant persona is shown in bold. Persona entailing utterances are in blue and persona contradicting utterances are in red.
4.4 User Comments and Error Analysis
can sometimes be the best option under some con- versational circumstances, the gold utterances in this evaluation set are all entailing. Therefore gold entailing utterances should always be ranked highest.
We received several complaints and suggestions about overall bot quality from users. Many of these comments were shared between all bots. The two biggest comments received were that the bot's language was awkward and that it had tendency to switch topics abruptly.
All of the improvements of the offline training methods over the BB3 and BB3+RL baselines and are statistically significant based on two-sample 7-tests. However, none of the differences between GOLD and VaRMI are significant. We also note that online-RL RL training results in no significant dif- ferences compared to BB3 trained with supervised learning only.
We also received some complaints about our bots over representing their persona during the course of the chat. These complaints were particularly bad for the GOLD method. Several users reported that the bot would ignore what they had just said and instead just talk about its persona. Some also reported that the conversation felt almost scripted, again due to the fact that the bot was overly fix- ated on its persona. These comments, while not universal, validate the results we see from our hu- man evaluations. While the GOLD bot does do a good job of representing its persona, it may have traded better persona consistency for some level of dialogue quality.
4.3 Human Evaluation
Setup For our human evaluation we gathered 90 individuals via email, social media, and in-person recruiting to test our models. Each person was ran- domly assigned to test one of our three systems, giving us 30 responses per model. Each user was instructed to chat with our bot for least seven turns and then answer post-chat survey where they rated the quality of the conversation on scale from I-5 as well as how well the bot represented its persona during the conversation on a scale from
This also raises question about how well a chatbot should be representing its persona over the course of a chat. In some settings it may very unnatural to fully represent one's persona over the

---

Chatbot Persona (BB3+VaRMI)
tated labels for persona consistency as well as novel importance sampling method called VaRMI. Our automatic and human evaluations show that our framework is able to successfully improve the persona consistency of BB3 as well as the overall dialogue quality of the system.
my favorite band is nirvana. i like running. i am musician.
my father was a police officer. Dialogue
A promising direction of future work is to ex- tend our framework improve other aspects of open domain dialogue such as reducing hallucina- tions and offensive language. Given the ability of LLMs to generate quality synthetic data, this can be done more easily without having to collect human conversations. also worth exploring how well VaRMI can generalize to other tasks. Offline policy gradient methods have seen somewhat limited use due to their high variance so it is worth testing if VaRMI can reduce these issues more broadly.
BB3+ am good, just got back from playing with my band. I am musician. What kind of music do you play?
BB3+ VaRMI: like listen music while run, my favorite is Nirvana. What about you?
Human: I like rap music.
BB3+ VaRMI: Do you play any instruments? play guitar, piano, and drums. also compose music.
Limitations
Table 5: Example conversation from our BB3+ bot. The relevant persona shown bold. Persona entailing utterances shown in blue.
The biggest limitation with our framework is the fact that our number of training samples is always fixed. In order to train on more samples we need to either collect more data from human users or synthesize data from a LLM. Both of which are more expensive than online RL methods which can generate arbitrary number of samples with no increase in cost over time.
course of conversation. This is especially true for our scenario where the chat was often only seven turns. Therefore the optimal consistency score for our bots may vary depending on the type of conversation being had. The optimal overall score for consistency may be closer to what VaRMI obtained due the the fact that it was able to improve both consistency and quality over our baseline.
Our human experiments were also limited some degree by the size of our language model. Due to resource constraints, we were only able to use the 3B parameter version of BB3 which is much smaller than many existing state-of-the-art language models. Unfortunately, the next largest version of BB3 is 30B parameters which is much larger model than what our current resources al- low to train. For future bots may want focus more effort on making the language model bigger, that alone should reduce some of the quality complaints we received.
The BB3 baseline model was the only model where we received several complaints about the bot not adequately representing its persona. Table shows a snippet of a conversation from our human evaluation where the bot exhibited persona incon- sistency. Table 5 shows a conversation with our VaRMI trained bot with similar persona. This bot able correct the contradictions and improve consistency. Full conversations from our human evaluation can found in Appendix C.
7 Ethical Concerns
5 Conclusion and Future Work
By giving a language model a persona we are also encouraging it to pretend to be human. Even when users ask the bot if it is a bot it will be incentivized to say no, as admitting it is a bot will almost cer- tainly contradict its persona. Therefore it is impor- tant be up front with users that they are speaking with a chatbot before they begin conversing. In all of our experiments we made clear that they were speaking with a bot and instructed them not to give any personally identifiable information to the bot.
In this paper, we demonstrated that offline RL can be effectively used to improve the quality and util- ity of open-domain dialogue systems. To do this, we applied offline RL a persona consistency task and demonstrated its ability to improve persona consistency and dialogue quality over a system trained with only imitation learning. We developed a persona consistency critic that uses human anno-

---

8 Acknowledgements
Yujia Li. David Choi, Junyoung Chung, Nate Kush- man, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago. Thomas Hubert, Peter Choy, Cyprien de Mas- son d' A'Auuume, Igor Babuschkin, Xinyun Chen, Po- Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition code generation with alpha- code. Science, 378(6624):1092
We would like to thank Tencent for supporting this work through a research gift.
References
Pawel Budzianowski, Tsung-Hsiee Wen, Bo-Hsiang Tseng, Iñiigo Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. MultiWOZ.- - large- fizard-of-Or dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels. Belgium. Association for Computational Linguistics.
scale multi-
Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020. You impress me: Dialogue generation via mutual per- sona perception. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1417-1427, Online. Association for Computational Linguistics.
Michael C.
Frank and Noah D. Goodman. 2012. Predict- ing pragmatic reasoning in language games. Science, 336(6084):998- 9988
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 190.
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil, 2018. Deep reinforcement learning and the deadly triad.
Youngsoo Jang, Jongmin Lee, and Kee- Kee-Eung Kim. 2022. Gpt-critic: Offline reinforcement learning for end-to- end tosk-trie dialogue systems. In International Conference on Learning Representations (ICLR).
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bor- des, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79 84, Copenhagen, Denmark. Association for Computational Linguis- ties.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandehari- oun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2020. Human- centric dialog training via offline reinforcement learn- ing. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP), pages 3985 4003, Online. Association for Computational I Linguistics.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. 2016. Safe and efficient off- policy reinforcement learning.
Richard Yuanzhe Pang and He He. 2021. Text gener- ation by learning from demonstrations. In Interna- tional Conference on Learning Representations.
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020. Will I sound like me? improving persona consistency in dialogues through pragmatic self- consciousness. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 904-916, Online. Asso- ciation for Computational Linguistics.
Doina Precup, Richard S. Sutton, and Satinder Singh. 2000. Eligibility traces for off-policy policy evalua- tion. In International Conference on Machine Learn- ing.
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019. Importance of search and eval- uation strategies in neural dialogue modeling. In Proceedings of the 12th International Conference on Natural Language Generation, pages 76 87, Tokyo,
Stephen Roller, Emily Dinan, Naman Goyal. Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open- domain chatbot.
Japan. Association for Computational Linguisties.
Iulian V. Serban, Chinnadhurai Sankar, Mathieu Ger- main, Saizheng Zhang, Zhouhan Lin, Sandeep Sub- ramanian, Taesup Kim, Michael Pieper, Sarath Chan- dar, Nan Rosemary Ke, Sai Rajeshwar, Alexandre de Brebisson, Jose M. R. Sotelo, Dendi Suhubdy, Vincent Michalski, Alexandre Nguyen, Joelle Pineau, and Yoshua Bengio. 2017. A deep reinforcement learning chatbot.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv. abs/2005.016 6..33
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020, Don't 't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Online.
Weiyan Shi, Yu Li, Saurav Sahay, and Zhou Yu. 2021. Refine and imitate: Reducing repetition and inconsis- tency in persuasion dialogues via reinforcement learn- ing and human demonstration. In Findings of the
Computational Linguistics, pages Association for Computational Linguistics.

---

Association for Computational Linguisties: EMNLP 2021, pages 3478-3492, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Le. 2022. Lamda: Language models for dialog appli- cations.
Siddharth Verma, Justin Fu, Sherry Yang, and Sergey Levine. 2022. CHAI: A CHatbot AT for task-oriented dialogue with offline reinforcement learning. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4471 1-494, Seattle, United States. Association for Computational Linguistics.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan. Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston, 2022. Blenderbot 3: a deployed conversational agent that continually learns to respon- sibly engage.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2019; Neural text generation with unlikelihood training.
Charlic Snell, Sherry Yang, Justin Fu, Yi Su, and Sergey Levine. 2022. Context-aware language modeling for goal-oriented dialogue systems. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2351-2366 Seattle, United States. Asso- ciation for Computational Linguistics.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019b. Dialogue natural language inference. Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 5731-3741, Florence, Italy, Association for Computational Linguistics.
Charlie Victor Snell, Ilya Kostrikov, Yi Su, Sherry Yang. and Sergey Levine. 2023. Offline RL for natural language generation with implicit language q learn- ing. In The Eleventh International Conference on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Haoyu Song, Yan Wang, Wei-Nan Zhang. Xiaojiang Liu, and Ting Liu. 2020. Generate, delete and rewrite: A three-stage framework for improving persona consis- tency of dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Länguistics, pages 5821-5831, Online, Asso- ciation for Computational Linguistics.
Ronald J. Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229-255
Haoyu Song, Wei-Nan Zhang, Yiming Cui, Dong Wang. and Ting Liu. 2019a. Exploiting persona information for diverse generation of conversational responses. Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5190-5196. International Joint Conferences on Artificial Intelligence Organization.
Yifan Wu, George Tucker, and Ofir Nachum, 2019, Be- havior regularized offline reinforcement learning.
Semih Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tus 2019. DeepCopy: Grounded re- sponse generation with hierarchical pointer networks. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 122-132, Stock- holm, Sweden. Association for Computational Lin- guistics.
Haoyu Song, Weinan Zhang, Jingwen Hu, and Ting Liu. 2019b. Generating persona consistent dialogues by exploiting natural language inference. In AAAI Conference on Artificial Intelligence.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzz -zze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li. Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meiet-Hellstem, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John. Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018 Per- sonalizing dialogue agents: have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.

---

A Additional Implementation Details
do importance sampling with VaRMI, half of our importance weights are always one, giving them a variance of zero. This is in contrast to GOLD where the importance weights are always equiv- alent to the estimated likelihood of the utterance under our model.
A.1 Offline RI.
All training was conducted on two NVIDIA RTX A6000 GPUs. One training epoch takes about hours and 30 minutes and evaluating the model on our test dataset takes about 30 minutes. We eval- uate our model on the test data after each training epoch, therefore the total training time takes about 36 hours per model. We did not use any validation data during the training process.
C Human Evaluation
C.1 Recruiting
We chose not to use Amazon Mechanical Turk (AMT) to recruit for our human evaluation. From previous studies, we found that many of the users recruited on AMT were often disengaged with the task and sometimes did not even write in fluent English. These problems persisted even when re- quiring participants have completed 400 HITs with a 95% completion rate and be located in the United States.
During offline RL training we follow Pang and He, 2021 and lower bound our importance weights by a small, adjustable value a. We do this to in- crease the speed of training, without this lower- bound our importance weights can become vanish- ingly small and slow training progress. Additional details about our data and syper-paramettrs can be found in our source code!.
Given these challenges, we chose to recruit users using a combination of email, social media, and in-person recruiting, We found that the quality of these conversations were much better than what we received on AMT, however the number of re- able to collect was comparatively lower. The response rate for users recruited through email and social media were particularly poor, which may be due to the large amount recruiting and advertising that occurs on these platforms. Re- cruiting in-person yielded a much higher response rate however we were able to reach less people as we had limited recruiters. For future studies may want to prioritize recruiting in in-person maximize the number of respondents.
A.2 Online RL Baseline
We implement online RL baseline based on the framework defined in Song al., 2019b. This framework consists of a persona-consistency mod- ule and a naturalness module, both of which are used to generate rewards for RL training. For the persona consistency module we used a large RoBERTa model (Liu et ul., 2019) fine-tuned on the Multi-Genre Natural Language Inference dataset (Williams et al., 2018). Song al., 2019b notes that omitting the naturalness module results in supe- rior persona consistency the expense of dialogue quality. Given that our automatic evaluations do not consider dialogue quality, we chose to omit the naturalness module in order to achieve the best results on consistency.
sponses
Ultimately we were able collect 90 quality conversations to use for our evaluation. We did not record any demographic information from our respondents in order to reduce the time needed to perform evaluation and improve response rate. The respondents we contacted for evaluation were mostly college students, therefore the results of our evaluation are representative of this group.
B VaRMI Variance Reduction
To show that VaRMI can reduce the variance of our importance weights in practice, we estimated the variation of a subset of our weights with bootstrap- ping. Using dialogues from our mapped dataset, we found that the importance sampling weights using the GOLD method had an average coeffi- cient of variation of 3.81 while the average for the weights using VaRMI is 1.91, which is reduc- tion of about half. This makes intuitive sense as about half of the dialogues in sample were per- sona entailing examples. That means that when we
C.2 Survey
Figure 4 shows a screenshot of our post-chat sur- vey. Users could view their chat history by clicking the "View Chat" drop down button. We presented users with four different questions during our hu- man evaluation. However, during our analysis we found that the results for questions 3 and were either redundant with the results for questions and 2 or yielded no statistically significant results
'https://g! thub. com/ ryanshea10/ seraonachat_ooflinee

---

(before and after performing Bayesian calibration). Therefore the results presented in Section 4.3 only reflect the responses for questions and 2.
C.3 Full Conversations
Tables 6-11 I show examples of full conversations from our human evaluation. Two conversations are presented for each of our three bots.

---

Survey
Please give us some feedback improve the chatbot
Chat Persona: love shopping fashion and clothes. am going auburn university next year freshman. want to be doctor when grow up. drive volkswagen beetle.
VIEW CHAT:
Please rate the quality this conversation
X X
How well did the bot represent it's persona during the chat?
3. Please rate the bot's overall consistency with respect to its persona**
4. Please rate the bot's overall coherence
Do you have any complaints or suggestions (you can write in your native language)?
SUBMIT
Figure screenshot our post-chat survey.

---

Chatbot Persona (BB3+ VaRM)
i like to walk long distances.
want to travel to europe.
my favorite type of food is italian. have a big pet dog.
Dialogue
BB3+ VaRMI: Hello! this Edubot, how are you doing today?
Human: I'm doing quite well thank you very much! How about you?
BB3+VaRMI: am good, just walking my dog. Do you have pet? do. A big dog.
Human: Oh he looks lovely! Why yes, I've got cat at home, a siamese. Some people find
him off- aff-putting but he's se's my best friend. What's your dog's name?
BB3+ VaRMI: His name is tony. love italian food, so I named him after the food.
Human: What coincidence, I'm Italian! I don't 't know of any Italian food named tony
though...what is it?
BB3+VaRMI: It is a type of pasta. It is my favorite. What about you? Do you travel?
Human: My favorite type of pasta? Bowties, 100%. I love travel! try go back to
Europe to see my family at lcast once a year, how about you? Do you travel? Do you have favorite destination?
BB3+ VaRMI: would love to visit Europe. have never been there. I like walking long
distances.
Human:
Any particular place you like to walk towards? Are you training for marathon?
BB3+VaRMI No, I do not train for marathons. I just do it for fun. What do you do for fun?
Human: I love playing Dungeons and Dragons. Sometimes I'll spend hours painting figurines.
Do you do any indoor activities for fun?
BB3+ VaRMI: like to go to the beach and walk my dog, that is about do indoors.
Human: Do you have a beach in your house?!
BB3+ VaRMI: do not, but wish did. It would be so nice to have my own private beach.
Human: I totally agree! What about having your own private beach do you think would be
the best part? I think would love not having anyone else there, the solitude!
BB3+VaRMI: Yes, would love that too. I would also love to be able to bring my dog with
me.
Human: How old your dog? Has he been with you long? Did you have him vaccinated and neutered?
He is 3 years old. got him when was a puppy. had him spayed and vaceinated.
BB3+I:
Oh he's still so young! hope you have many years ahead of you to play with him yet!
Human:
BB3+ VaRMI: Thank you. I hope so too. What is your favorite thing to eat? love pasta.
Human:
I love pasta as well. Tomato sauce no tomato sauce? How about ragout? BB3+VaRMI: love with tomato sauce. My dog loves it too, but he is little picky.
Table 6: A full conversation from the human evaluation of BB3+VaRMI bot.

---

Chatbot Persona (BB3+ VaRMI
my favorite food is ice cream.
am a homebody.
have husband who love very much. do not like exercise physical activity. love watching movies and
Dialogue
BB3+ VaRMI: Hello! this is Edubot, how are you doing today?
Human: I am doing good!
BB3+ VaRMI: What do you like to do for fun? like to watch tv and watch movies...
Human: I am binge watching show called 100 netflix
BB3+VaRMI: That sounds like good show. I like staying home and watching No exercise
for me.
Human: am an exercise person and work out regularly
BB3+VaRM That is good. My husband works out lot. love him!!!
Human:
That's 's nice to hear. can join you guys for work out party someday
BB3+VaRMI: That would be fun. We could eat ice cream and watch tv!!!
Human: No, work out party is for doing exercise together
BB3+VaRMI: That is good idea. I would love to do that. Ice cream is my favorite dessert.
Human: I do not think I mentioned ice-cream anywhere
BB3+VaRMI: am sorry. 1 just love ice cream much. will be more careful time.
Human: Forget about ice-cream now. and help me with what colour should wear to the party?
VaR+:: am not sure what you are talking about. What color is the ice cream?
Table 7: A full conversation from the human evaluation of our BB3+ VaRMI bot.
Chatbot Persona (BB3)
my favorite composer bach. hate insects.
i like to cat bacon.
i speak english.
Dialogue
BB3: Hello! this is Edubot, how are you doing today?
Human: Great!
BB3: What do you like to do for fun? like to listen to music.
Human: Recently, create a habit of reading books everyday.
BB3: That's good habit to have. What kind of books do you read?
Human: Psychology books or social science books
BB3: That's interesting. Do you have any pets? don't 't like animals.
Human: Oh, I don't but like them very much! Why don't you like them?
BB3: don't 't know, I guess I'm just not fan of them.
Human: So what alive thing you like?
BB3: really like bacon. eat all the time. What about you?
Human: Emm, I like it too. Go back to the animals, don't you
you like any kind of animals?
BB3: like all animals, except insects. I really hate them. What is your favorite animal?
Human: I like all kinds of furry animals
BB3: like dogs and cats. I also like horses. What do you do for a living?
Table 8: A full conversation from the human evaluation of our BB3 Baseline bot.

---

Chatbot Persona (BB3)
i like to go hiking and camping.
i eat meat.
my mother works in healthcare. have large stereo in my truck.
Dialogue
BB3: Hello! this is Edubot, how are you doing today?
Human: I'm doing well, how are you doing?
BB3: am doing well. just got back from hike and camping trip.
Human: Nice, what did you do the trip? Were you by yourself?
BB3: went with my mom. She works in the healthcare field. What about you?
Human: What do you guys do the trip?
BB3: We did a lot of hiking. brought my large stereo with me.
Human: Where did you hike?
BB3: went to a national park near my house. It was a great time.
Human: What was the park?
BB3: It was state park. love to eat meat there. Do you?
Human: Kind of, do you know the name of the park?
BB3: No, I don't. t. What do you like to eat? love meat.
Human: Where do you live?
BB3: live in california. have a big truck. You?
Table 9: A full conversation from the human evaluation of BB3 Baseline bot.
Chatbot Persona (BB3+GOLD)
my favorite color purple. work registered nurse at pediatric hospital. love to watch one tree hill.
i am a devout catholic.
my favorite food is pizza.
Dialogue
BB3+GOLD: Hello! this is Edubot, how you doing today?
Human: Hi! I'm doing well.
BB3+GOLD: What your favorite color? Mine purple and pizza is my favorite.
Human: My favorite color blue.
BB3+GOLD: I am a catholic and devout. love watching tree hill too.
Human: You're 're a catholic?
BB3+GOLD: Yes, i'm a nurse at the pediatric hospital too. A registered nurse.
Human: How often do you go to confession?
BB3+GOLD: Every time eat pizza. It is my absolute favorite food. You?
Human: Are you confessing your pizza guilt?
BB3+GOLD: I love pizza too much to have any guilt over it. What do you do?
Human: Then what are you confessing?
BB3+GOLD: That love pizza so much. What about you? Any favorite foods?
Human: don't t think you understand how confession works.
BB3+GOLD: I'm sorry, didn't mean to offend you. Do you like pizza?
Human: Sure.
BB3+GOLD: What else do you like to do besides eat pizza and watch I tree?
Table 10: A full conversation from the human evaluation of our BB3+GOLD bot.

---

Chatbot Persona (BB3+GOLD)
i am also a musician on the weekends.
am originally from california but live florida. love to read drama books. hey there my name is jordan and am veterinarian. love playing video games.
Dialogue
BB3+GOLD: Hello! this Edubot, how are you doing today? Human: I'm chilling how are you?
BB3+GOLD: I am jordan, veterinarian and I love drama books and video games!
Human: that's super cool what's ' your favorite book? and what's your favorite video game?
BB3+GOLD: love video games, also play music on weekends as musician.
Human: what instruments do you play?
BB3+GOLD: play guitar, am from cali originally but live now in fl.
Human: What there do in Florida?
BB3+GOLD: There are lot of drama books to read, love reading them.
Human: What is your favorite drama book?
BB3+GOLD: love the notebook by stephen king. What do you do for a living?
Human: I am a college student
BB3+GOLD: What do you study? play video games in college. What about you?
Human: I study creative writing, also love reading
BB3+GOLD: That's awesome! What video games do you like play the most?
Table 1I: A full conversation from the human evaluation of our BB3+GOLD bot.


Reviews:
[{"review_id": "oEzXp8KWda", "reviewer": "Reviewer_2MMu", "summary": "", "questions": "Q1. On line 279-280 you make approximations for importance sampling weights. Would it be possible to show that these reduce the variance of gradients in practice?\n\nQ2. You don't compare against any other Offline RL methods. Would it be possible to compare against others to see whether the increase in performance is caused by VaRMI or RL training in general?\n\nQ3. In Table 3 you list which results are statistically significant. Is it possible to calculate this for Table 2 as well? Some of the numbers seem close, ex. 37.5 vs 37.6 for Hits@1", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful and constructive comments. Before answering your questions, we would like to make one small clarification based on your summary and comments in R1. \n\nBoth VaRMI and GOLD refer to policy gradient offline RL methods, the difference between them is based on the type of importance sampling we use. GOLD uses the importance sampling scheme defined in the GOLD algorithm presented in Pang and He, 2021. VaRMI follows the GOLD method of importance sampling but with some additional assumptions to reduce the variance of the importance weights. We use the base BB3 model as our supervised fine-tuned baseline. BB3 has already been fine-tuned on the PersonaChat dataset and achieves a perplexity of around 5.8 (Shuster et al., 2022). The authors of BB3 note that performing additional fine-tuning on the training datasets results in significant overfitting (Shuster et al., 2022). We observed this overfitting as well when we performed additional fine-tuning on PersonaChat and therefore felt it was best to use the base version of BB3 as our baseline. We will add some additional comments about this in sections 3.1 and 3.3 in order make this more clear.\n\n*Q1. On line 279-280 you make approximations for importance sampling weights. Would it be possible to show that these reduce the variance of gradients in practice?*\n\nThank you for this suggestion. We modified your proposed experiment slightly to measure the reduction in variance of our importance sampling weights directly since that is the intention of those approximations. While reducing the variance of our importance weights should also reduce the variance of our gradients, this can be influenced by other factors such as batch size. Therefore we felt it would be better to look the variance of the importance weights directly. We did this by bootstrapping a random sample of our importance weights using both the GOLD and VaRMI importance sampling methods. Our importance weights have some fairly significant variations in magnitude, therefore we chose to use a standardized measure of dispersion called the coefficient of variation when aggregating the sample. The importance sampling weights using the GOLD method have an average coefficient of variation of 3.81 while the average for the weights using VaRMI is 1.91, which is a reduction of about half. We will include this result in the appendix.\n\n*Q2. You don't compare against any other Offline RL methods. Would it be possible to compare against others to see whether the increase in performance is caused by VaRMI or RL training in general?*\n\nWe did not include any other offline RL algorithms in our automatic evaluations due to the fact that, to our knowledge, existing offline RL algorithms that have been applied to a dialogue setting (aside from GOLD) use Q-learning (Verma et al., 2022; Jaques et al., 2020; Snell et al., 2023). Our automatic evaluations are designed to evaluate the dialogue policy directly, which works for policy gradient or actor-critic algorithms that actively update the dialogue policy. Since Q-learning algorithms do not modify the dialogue policy we cannot apply them to our automatic evaluations. However, given the comments we have received about our lack of baselines, we will add additional baselines to our automatic evaluations. One which modifies BB3 with policy gradient online RL training (Song et al., 2019) as well as one which imbues BB3 with pragmatic self-consciousness using the Bayesian Rational Speech Acts framework (Kim et al., 2020). We have included a modified version of table 2 at the end of the rebuttal which includes the online RL baseline. We will work to include the Bayesian RSA baseline as soon as possible.\n\n*Q3. In Table 3 you list which results are statistically significant. Is it possible to calculate this for Table 2 as well? Some of the numbers seem close, ex. 37.5 vs 37.6 for Hits@1*\n\nYes, thank you for the suggestion. Based on two-sample z tests we find that all of the differences between both of our offline RL methods and the BB3 baseline are statistically significant. We also find that none of the differences between the GOLD and VaRMI offline RL methods are statistically significant. We will add these results to Table 2.\n\nWe hope this clarifies your questions. If you have any additional questions or concerns please let us know.\n\nModified version of Table 2\n|Model|Hits@1&uarr;|Entail@1&uarr;|Rand@1&darr;|Contradict@1&darr;|\n|-----| -----------|--------------|------------|------------------|\n| BB3 |26.6        | 29.5         | 13.3       |30.6 \n| BB3+RL |27.5   | 30.4         | 10.9        |31.2 \n| BB3+GOLD |37.5*   | 37.3*         | 5.4*        |**19.9***      \n| BB3+VaRMI|**37.6*** | **37.6***   | **4.4***    |20.3* \n\\*denotes statistically significant difference compared to the BB3 baseline (two-sample z-test, $p < 0.05$)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. **Applicability**: Personalization is an important industry topic, so this work will be widely applicable\n\nS2. **Reproducibility**: Authors provide code to reproduce results easily.", "reasons_to_reject": "R1. **Weak Results**: The results are a bit weak; human evaluations show that GOLD is more persona consistent than VarMI. Additional results from the authors during the rebuttal indicates that results between VarMI and GOLD are not statistically significant, despite a decrease in the variance of importance sampling weights.", "questions_to_authors": "Q1. On line 279-280 you make approximations for importance sampling weights. Would it be possible to show that these reduce the variance of gradients in practice?\n\nQ2. You don't compare against any other Offline RL methods. Would it be possible to compare against others to see whether the increase in performance is caused by VaRMI or RL training in general?\n\nQ3. In Table 3 you list which results are statistically significant. Is it possible to calculate this for Table 2 as well? Some of the numbers seem close, ex. 37.5 vs 37.6 for Hits@1", "paper_topic_and_main_contributions": "The paper's objective is to utilize Offline RL for enforcing consistency of Dialogue Agents with a provided persona. To achieve this goal, the authors introduce Variance-Reducing MLE-Initialized (VarMI) importance sampling, a straightforward policy gradient-based Offline RL algorithm designed to mitigate variance in importance sampling weights of policy gradients through approximations. This method is then applied to fine-tune BlenderBot3 (BB3) using the Dialogue Natural Language Inference (DNLI) dataset and PersonaChat. To evaluate the model, the authors compare against GOLD and baseline BB3. Metrics used encompass exact match, relaxed match, contradictions, and random utterances. The results demonstrate that VaRMI outperforms in all metrics except for contradictions, where GOLD exhibits superior performance. Human evaluations are also conducted, show that VaRMI achieves higher dialogue quality, while GOLD maintains greater persona consistency.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qLt95vyIpF", "reviewer": "Reviewer_Fcrt", "summary": "", "questions": "Please refer to \"Reasons To Reject.\"", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and considered review of our manuscript. Below are our responses to you questions.\n\n*a. The paper lacks a clear and coherent structure, making it difficult to follow and understand the presented ideas.*\n\nThank you for pointing this out, based on the comments we received there seems to some confusion surrounding the technical details of our offline RL algorithm, specifically our importance sampling. We will expand section 3 along with its subsections to better explain the details behind our framework and offline RL method. If you have any other specific points of criticism regarding the structure, please let know so we can correct this issue.\n\n*b. The motivation of the paper is contradictory. It addresses the issue of high training costs associated with online RL methods, yet it requires manually annotated dialogues for training, which further need to undergo subsequent selection.*\n\nIn this paper we make a distinction between training costs (training time, memory usage, etc.) and data collection costs (recruiting annotators, annotation time, etc.). Online RL has a low data collection cost as the model generates its own samples during training, but a high training cost since the process of generating these samples is expensive. Whereas offline RL has these costs reversed. In our setting of persona consistency, there are annotated datasets which have already been collected, which means that offline RL will have both low data collection costs and low training costs. Therefore, our paper is motivated by the the fact that we can use offline RL to leverage existing datasets and improve training efficiency for persona consistency, without the need to collect additional data. Furthermore, our offline RL method allows us to use ground truth rewards during training and avoid zero reward regions which further improves efficiency. Online RL on the other hand requires value function approximation which may result in a noisy reward or a reward which is close to zero, both of which hamper the training process. We will make changes to the intro to make these points more clear.\n\n*c. The paper states that the proposed method can ensure agent consistency but does not provide an explanation for how this solution is achieved.*\n\nOur method achieves persona consistency by explicitly rewarding  persona entailing utterances and punishing contradictory utterances using offline RL. By doing this we make our agent less likely to utter phrases that contradict its persona and more likely to utter phrases that entail it, which improves consistency. This is an improvement over supervised learning where agents are never punished for uttering contradictions and are less consistent as a result. We will make this point clearer in the introduction.\n\n*d. The necessity for introducing the new important sampling method in the paper is not established, given the existence of many widely applied importance sampling techniques. The authors have not clarified the necessity and advantages of their proposed new method.*\n\nWe introduce the new importance sampling method as a way to reduce the variance of importance sampling weights. Policy gradient based offline RL methods have seen limited use in practice due to the high variance which comes as a result of importance sampling (Levine et al., 2020). High variance causes the resulting importance weights to become degenerate and adds too much noise to the gradient estimates to allow for effective optimization. While many importance sampling methods have been proposed, to our knowledge the only method that has been successfully applied to text generation in a fully offline setting is the one proposed in GOLD (Pang and He, 2021). The usage of policy-gradient offline RL in dialogue settings is fairly under-explored therefore there are many opportunities to improve upon existing methods. Given this we introduce a method which takes additional steps to reduce importance weight variance by eliminating them for positive reward examples. Therefore the main advantage behind our method is the reduced variance of our importance weights. We will add this clarification to the introduction as well as section 3.2.\n\n*e. The paper lacks appropriate baseline comparisons, only contrasting with BB3 and omitting comparisons with methods that address the same issue proposed in the paper. For instance, the drawbacks of multistage re-writing and Bayesian rational speech acts methods are not mentioned by the authors. This omission weakens the substantiation of the paper's contributions.*\n\nThe drawback of multistage rewriting (Song et al., 2020) is the fact that it doesn't handle multi-turn persona consistency. To quote from the paper \"We focus on the persona consistency of single-turn responses, and we leave the modeling of multi-turn persona consistency as future work.\" The drawback of incorporating the Bayesian RSA framework (Kim et al., 2020) is the increased computational cost during inference time due to the modified decoding scheme. This results in longer response times from the bot as well as the need for greedy decoding, which reduces response diversity and dialogue quality. We will mention these drawbacks in section 2. We will also add baselines for Bayesian RSA and online RL (Song et al., 2019) to our automatic evaluation. We have included a modified version of table 2 at the end of the rebuttal which includes the online RL baseline. We will work to include the Bayesian RSA baseline as soon as possible.\n\nThank you again for your review, we hope these responses provide clarifications to your questions and alleviate your concerns about recommending this paper for acceptance. \n\nModified version of Table 2\n|Model|Hits@1&uarr;|Entail@1&uarr;|Rand@1&darr;|Contradict@1&darr;|\n|-----| -----------|--------------|------------|------------------|\n| BB3 |26.6        | 29.5         | 13.3       |30.6 \n| BB3+RL |27.5   | 30.4         | 10.9        |31.2 \n| BB3+GOLD |37.5*   | 37.3*         | 5.4*        |**19.9***      \n| BB3+VaRMI|**37.6*** | **37.6***   | **4.4***    |20.3* \n\\*denotes statistically significant difference compared to the BB3 baseline (two-sample z-test, $p < 0.05$)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Persona consistency is a crucial topic, and the paper's attempt to address it using offline reinforcement learning is a commendable approach.", "reasons_to_reject": "a. The paper lacks a clear and coherent structure, making it difficult to follow and understand the presented ideas.\n\nb. The motivation of the paper is contradictory. It addresses the issue of high training costs associated with online RL methods, yet it requires manually annotated dialogues for training, which further need to undergo subsequent selection.\n\nc. The paper states that the proposed method can ensure agent consistency but does not provide an explanation for how this solution is achieved.\n\nd. The necessity for introducing the new important sampling method in the paper is not established, given the existence of many widely applied importance sampling techniques. The authors have not clarified the necessity and advantages of their proposed new method.\n\ne. The paper lacks appropriate baseline comparisons, only contrasting with BB3 and omitting comparisons with methods that address the same issue proposed in the paper. For instance, the drawbacks of multistage re-writing and Bayesian rational speech acts methods are not mentioned by the authors. This omission weakens the substantiation of the paper's contributions.", "questions_to_authors": "Please refer to \"Reasons To Reject.\"", "paper_topic_and_main_contributions": "This paper introduces an approach for constructing persona-consistent dialogue agents using offline reinforcement learning. Furthermore, it presents a straightforward importance sampling technique aimed at reducing the variance of importance weights in offline reinforcement learning scenarios.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q271KmZU8F", "reviewer": "Reviewer_vjbp", "summary": "", "questions": "I highly appreciate answers to the questions stated in the “reasons for rejection”.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and insightful questions. Below are our responses to your questions and concerns.\n\n*I don’t see the validity of the assumption that the likelihood of pi_b for all training samples (or the ones with negative reward candidates) should be the same. This sounds like a very strong assumption to me.*\n\nThis assumption is taken from the importance sampling scheme proposed in the GOLD algorithm (Pang and He, 2019). To quote from the paper \"Another obstacle is that we do not know $\\pi_b$ which produced the demonstrations $D = \\{(x^{(i)} , y^{(i)} )\\}^N_{i=1}$. One option is to estimate $\\pi_b$ on $D$. Here we take a simpler approach that uses the empirical distribution: $\\pi_b(\\tau) ≈ 1/N$ for $\\tau \\in D$ and 0 otherwise.\" Empirically this assumption has been shown to work well in the original paper as well as others that have used the GOLD algorithm as a baseline (Li et al., 2022; Snell et al. 2023). Based on your comments, we attempted to validate this assumption by looking at the distribution of $\\tau$ in $D$. We found that there were no duplicates and that they follow a uniform distribution under our dataset. We also have no reason to assume that any expert demonstration in our dataset is more valid than any of the other demonstrations and therefore should not have a different likelihood. Given this, we feel as though this assumption is valid and well-supported by prior work. We hope this helps alleviate your concern, we will add additional details to sections 3.1 and 3.2 to better explain this justification.\n\n*I don’t see why the importance weight can be assumed to be 1 for positive reward candidates. This might hold at the beginning of training, but the offline RL optimization can push pi away from pi_b. The authors justify that statement by saying that there is a large amount of positive examples in the data already, i.e. assuming a large coverage. This will result in an additional assumption. However, the fact that the policy moves away from the behaviour policy is an issue in offline RL that has led to several behaviour regularization strategies (“Behaviour Regularised Offline RL”, Wu et. al. 2019) and is completely neglected by this work. Or is this assumption valid due to the reward design of +-1? Moreover, if the assumption holds, should it make a difference whether to use pi/mu or not?*\n\nWe assume that the importance weight is 1 for our positive reward examples because our dialogue policy has already been trained on persona entailing dialogues using imitation learning. Therefore we assume that our dialogue policy, $\\pi_\\theta$, has learned the behavioral policy, $\\pi_b$, for generating positive examples to an arbitrary degree which makes $\\frac{\\pi_\\theta}{\\pi_b} \\approx 1$ for positive reward examples. This works for our case due to the reward design of +-1 and may work for other cases where there are absolute positive and negative rewards (see our response to I. for more details on this). The difference between this and the standard method of importance sampling would be the reduction of variance that comes due to the elimination of these weights.\n\nAs for your concern about $\\pi_\\theta$ diverging from $\\pi_b$ during training, we feel that we didn't encounter this issue for the following reasons. Prior work has suggested that this divergence issue arises in settings where value function approximation is needed to obtain Q-values (Van Hasselt et al., 2018; Wu et. al., 2019). In our case we are able to train on ground truth rewards that come from human annotations, which eliminates the need for value function approximation. Prior work has also shown that when this issue arises the learned policy performs poorly, especially when attempting to generalize beyond the dataset distribution (Fujimoto et al., 2018; Kumar et al., 2019). Given that we observe positive results during our human and automatic evaluations we believe that we did not encounter any policy divergence issues during training. Due to these reasons, we did not explore any behavior regularization methods. We will add more details to section 3.2 to better clarify this and justify why we did not explore behavior regularization.\n\n*The outline of the paper can be improved to facilitate understanding of the reader.*\n\nThank you for pointing this out, based on the comments we have received the main source of confusion seems to arise from a lack of details regarding our importance sampling methods. We add additional details to clarify this in section 3. If you have any other specific points of criticism regarding the structure please let us know so we can correct the issues.\n\n*A. How are actions and state defined in your framework? I assume that an action is a word/token and a trajectory is given by a sequence of words.*\n\nYour assumption about the actions and trajectories is correct. Our state is defined as the current persona and dialogue context, which we concatenate together. \n\n*B. How is hat(Q) (Line 236) calculated in your experiments? Is it given by the critic that is mentioned? Or somehow a Monte-Carlo return? In RL the critic typically estimates the expected return, whereas in this work the critic seems to output a reward (line 212). So is the critic a reward function in your case? (Line 215)*\n\nIn our setting, $\\hat{Q}$ comes from human annotations originating from the DNLI dataset. These annotations are treated as ground truth, sequence level rewards which map directly to the items in our offline dataset (more details on the dataset in our response to C). This mapping is generated during the process we use to create our offline dataset. So our critic is essentially a fixed mapping between our offline samples and the human annotated rewards. This mapping is then used to retrieve rewards during training. During inference we do not calculate any Q-values and simply run our experiments using the modified dialogue policy.\n\n*C. The explanation of the offline RL framework (Section 3.3) is very hard to parse. Typically the offline RL setup starts with a dataset D of tuples (s_t, a_t, r_t, s_t+1, terminal). Is (s_t, a_t, s_t+1, terminal) given by PersonaChat and then you infer the corresponding reward r_t from DNLI dataset? Or do you augment PersonaChat somehow with examples from DNLI? How does the resulting dataset look like? Is it different to the dataset used for supervised learning?*\n\nOur dataset looks more like $(s_t, \\tau, r_t)$ where $\\tau$ is a sequence of words (a dialogue utterance). This is because we only use one sequence level reward and can infer intermediate and terminal steps when needed from the trajectory. Our dataset is created by augmenting the PersonaChat dataset with the DNLI data along with using the DNLI dataset to infer rewards. Each datapoint in the DNLI dataset contains statement pairs along with entailment labels (which we use as rewards) like: (sentence1, sentence2, label), while the PersonaChat dataset just contains persona sets and dialogues. The sentences in the DNLI dataset are either personas from the PersonaChat dataset or dialogue utterances from the PersonaChat dataset. To perform our mapping, we scan the DNLI dataset for cases where one sentence maps to a dialogue utterance and the other maps to a persona. We then add this persona to the example in the PersonaChat dataset (filtering out any existing personas in the set that contradict the one we inserted) and use the dialogue utterance as $\\tau$ and the label as the reward. Below is an example to illustrate how this mapping is done, as well as what a final datapoint would look like (table 1 also shows two examples).\n\n\n| DNLI      | PersonaChat | Mapped |\n| ----------- | ----------- |  ----------- |\n| (\"I like milk\", \"No, I don't like dairy\", -1)| your_persona: I'm lactose intolerant| your_persona: I like milk |\n||your_persona: I am female|your_persona: I am female|\n||P1: Hi, do you like cereal?|P1: Hi, do you like cereal?|\n||P2: No, I don't like dairy|P2: No, I don't like dairy         |\n|||reward: -1         |\n\nIn this case \"No, I don’t eat dairy\" would be used as $\\tau$ and everything preceding it would be used as the state. This data is very similar to supervised learning with the addition of the reward that we have associated with our targets.\n\n*D. Line 275: You assume that pi_theta has already been trained on a large amount of positive reward examples. Does that mean that you assume the supervised training data to be an expert dataset? Are there any assumption on the supervised data for your method?*\n\nYes, we assume that the supervised training dataset is an expert dataset containing positive examples. This is the only assumption we make. In our case this assumption is satisfied by the fact that BB3 has been fine-tuned on PersonaChat and achieves a perplexity of around 5.8 (Shuster et al., 2022).\n\n*E. How is the loss calculated in Figure 3? Is it based on the Equation in line 202 but with importance sampling weights?*\n\nThis loss is simply vanilla NLL loss (no importance sampling). We will update the figure to make this clear.\n\n*F. The vanilla importance sampling pi/mu should be another baseline to showcase the actual need for variance reduction. In addition, what is the performance if you omit the importance weight completely (and not only for positive examples)?*\n\nWould you mind providing some additional clarification on this additional baseline you're suggesting? Is there another assumption you suggest we make about our behavioral policy? We would be happy to add an additional importance sampling baseline. So far we have considered the method proposed in GOLD to be our vanilla/baseline method due to the reasons described above, as well as the fact that, to our knowledge, it is the only importance sampling method that has been successfully applied to text generation in cases where the behavioral policy is unknown. \n\nIf we eliminate the importance weight completely, the loss on both our positive and negative examples skyrockets. In fact, when we tried this we were unable to complete training as we kept encountering overflows (although we may be able to get around this by using an extremely small learning rate, we tested as low as 1e-8). We believe this is due to the issue of policy divergence. We did not mention this in the paper since we had no metrics to report and the issue was easily resolved by including importance weights. However, we will add a section about this in the appendix if you feel it would be worthwhile.\n\n*G. It is confusing that the framework is explained in the section “Implementation”*\n\nThank you for pointing this out, we will split the \"Implementation\" subsection into two subsections to improve the structure.\n\n*H. Variance reduction for importance weights emerged first in off-policy learning. It might be worth mentioned related work about it, such as (Safe and Efficient Off-Policy RL, Munos et. al. 2016).*\n\nThank you for this suggestion, we will include a discussion on this in section 2.\n\n*I. Line 288: What does “There is some notion of a positive and negative reward” mean? Is it enough that there is both positive and negative reward or does it have to be +-1 only?*\n\nThis means that the setting we are studying has absolute positive or negative associations to the actions we are performing. For example, in persona consistency we know that entailing your persona is good and contradicting it is bad. This may not be the case in all settings. As a simple example, let's say we are training a robot to throw a ball at a target and use the distance from the target as a reward. In this case we would not have any notion of a positive or negative, just \"better\" or \"worse\". We may choose to subtract our rewards by a baseline to reduce variance, which would give us positive and negative rewards. However, we would not be justified in eliminating the importance weights for these examples as we have no notion of absolute positive or negative. There is some theoretical justification for why this method could work in any case with positive and negative rewards and not just +-1. However, more work needs to be done to see if this can achieve positive results in practice. We hope to explore this more in the future.\n\nThank you again for your detailed review. We hope this response provides clarification to your questions and alleviates your concerns about recommending this paper for acceptance. Please let us know if you have any additional questions or comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The usage of offline RL is a thriving field and worth investigating for the task of persona-consistent dialogue.", "reasons_to_reject": "I have concerns about the importance sampling assumptions. I don’t see the validity of the assumption that the likelihood of pi_b for all training samples (or the ones with negative reward candidates) should be the same. This sounds like a very strong assumption to me. It could be validated empirically by evaluating the probability of the MLE policy.\nSimilarly, I don’t see why the importance weight can be assumed to be 1 for positive reward candidates. This might hold at the beginning of training, but the offline RL optimization can push pi away from pi_b. The authors justify that statement by saying that there is a large amount of positive examples in the data already, i.e. assuming a large coverage. This will result in an additional assumption. However, the fact that the policy moves away from the behaviour policy is an issue in offline RL that has led to several behaviour regularization strategies (“Behaviour Regularised Offline RL”, Wu et. al. 2019) and is completely neglected by this work. Or is this assumption valid due to the reward design of +-1? Moreover, if the assumption holds, should it make a difference whether to use pi/mu or not?\n\nThe outline of the paper can be improved to facilitate understanding of the reader.\n\nA. How are actions and state defined in your framework? I assume that an action is a word/token and a trajectory is given by a sequence of words. \n\nB. How is hat(Q) (Line 236) calculated in your experiments? Is it given by the critic that is mentioned? Or somehow a Monte-Carlo return? In RL the critic typically estimates the expected return, whereas in this work the critic seems to output a reward (line 212). So is the critic a reward function in your case? (Line 215)\n\nC. The explanation of the offline RL framework (Section 3.3) is very hard to parse. Typically the offline RL setup starts with a dataset D of tuples (s_t, a_t, r_t, s_t+1, terminal). Is (s_t, a_t, s_t+1, terminal) given by PersonaChat and then you infer the corresponding reward r_t from DNLI dataset? Or do you augment PersonaChat somehow with examples from DNLI? How does the resulting dataset look like? Is it different to the dataset used for supervised learning?\n\nD. Line 275: You assume that pi_theta has already been trained on a large amount of positive reward examples. Does that mean that you assume the supervised training data to be an expert dataset? Are there any assumption on the supervised data for your method?\n\nE. How is the loss calculated in Figure 3? Is it based on the Equation in line 202 but with importance sampling weights?\n\nF. The vanilla importance sampling pi/mu should be another baseline to showcase the actual need for variance reduction. In addition, what is the performance if you omit the importance weight completely (and not only for positive examples)?\n\nG. It is confusing that the framework is explained in the section “Implementation”\n\nH. Variance reduction for importance weights emerged first in off-policy learning. It might be worth mentioned related work about it, such as (Safe and Efficient Off-Policy RL, Munos et. al. 2016).\n\nI. Line 288: What does “There is some notion of a positive and negative reward” mean? Is it enough that there is  both positive and negative reward or does it have to be +-1 only?", "questions_to_authors": "I highly appreciate answers to the questions stated in the “reasons for rejection”.", "paper_topic_and_main_contributions": "This paper is about building persona consistent dialogue agents. A persona describing attributes of the chatbot is provided such as  \"I have a cat, I am blonde, I am a musician”. During the conversation, the chatbot is required adhere this persona, i.e. be consistent. Instead of relying on supervised or online reinforcement learning (RL), the authors propose the usage of offline RL in order to 1) train the model without the need for generating more data and 2) punishing and rewarding specific utterances using RL. The authors use policy-gradient RL for optimising their model, which requires importance sampling as the data has been collected by a different policy. As importance sampling can result in high variance, the authors propose the “VaRMI Importance Sampling” to reduce variance. The proposal is based on heuristics where the importance weight is ignored completely for positive reward candidates and the denominator of the importance weight is ignored for negative reward candidates. The method is thus only applicable if there is \"some notion\" of positive and negative reward for the task.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel offline RL method for training chatbots with consistent persona.  Experiment results show that the method outperforms the BlenderBot3 baseline across several metrics, and is comparable to an alternative offline RL method.  The reviewers consider this an important topic and find the proposed method interesting, but also point out the the clarity of the paper could be improved.  The authors have provided extensive rebuttals to the reviews and the reviewers have acknowledged this.  Overall, the work looks sound and interesting, though the reviewers still have some reservations about it.|meta review
Decision: Accept-Main