Towards Boosting the Open-Domain Chatbot with Human Feedback
Authors: Hua Lu
Siqi Bao
Huang He
Fan Wang
Hua Wu
Haifeng Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Dialogue System
Human Feedback

Abstract: Many open-domain dialogue models pre-trained with social media comments can generate coherent replies but have difficulties producing engaging responses. This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference. In this paper, we propose a novel and efficient framework Diamante to boost the open-domain chatbot, where two kinds of human feedback (including explicit demonstration and implicit preference) are collected and leveraged. By asking annotators to select or amend the model-generated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs a Chinese chit-chat dataset. To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the generation-evaluation joint training. Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of pre-trained dialogue models. The overall engagingness of the previous state-of-the-art model has been improved remarkably by 50% in Chinese open-domain conversations.

Under review as a conference paper at ICLR 2023
TOWARDS BOOSTING THE OPEN-DOMAIN CHATBOT WITH HUMAN FEEDBACK
Anonymous authors Paper under double-blind review
ABSTRACT
Many open- meen-domain dialogue models pre- pre-trained with social media comments can generate coherent replies but have difficulties producing engaging responses. This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference. In this paper, we pro- pose novel and efficient framework Diamante boost the open-domain chatbot, where two kinds human feedback (including explicit demonstration and im- plicit preference) collected and leveraged. By asking annotators select amend the gener-geeerated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs Chinese chit-chat dataset. To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the seneration-evaluation joint training. Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of pre-trained di- alogue models. The overall engagingness of the previous state-of-the art model has been improved remarkably by 50% Chinese open-domain conversations.
INTRODUCTION
In recent years, the self-supervised pre-training based on tremendous unlabeled data has brought great success for many natural language processing tasks (Brown et al.) 2020 Chowdhery et al. 2022]. In dialogue generation, the pre-training is usually carried out with massive social media comments, acting human-like conversations (Adiwardana et al. 2020: Bao et al. 2021 Thoppi- lan al. 2022). Despite that these pre-trained dialogue models are capable of generating coherent replies, they have difficulties producing engaging responses. The main reasons for this phenomenon might be twofold. Firstly, there exists considcrable gap in the data distribution between the proxy human-like conversations (public group discussion) and the real human-human conversations (pri- vate two-way messaging). Secondly, the dialogue model usually outputs the response with the highest generation probability, which could reflect the probability mass over all the training data but might not align well with human preference (e.g., some biased or unsafe statements).
One straightforward way narrow the data distribution gap is fine-tune the pre-trained dialogue model with annotated human-human conversations. For instance, Blender (Roller et al.) 2021) em- ploys four annotated datasets (Zhang et al., 2018 Dinan al., 2019 Rashkin et al. [019 Smith et al. 2020) to emphasize the conversational skills of personality, knowledge, empathy, and engag- ingness. As for the alignment with human preference, LaMDA (Thoppilan al. (022) defines and quantifies some critical metrics for dialogue evaluation, including safety, interestingness, and so on. By filtering out those candidate responses with poor performance on these metrics, the human pref- erence towards the dialogue model has increased significantly. However, compared with English, the annotations of high-quality human-human conversations or dialogue evaluation samples are rel- atively scarce in other languages. As result, even the ttate-of-the-as Chinese chatbot - PLATO-XL (Bao et al. 2021), is only pre-trained with social media comments and not involved with advanced response evaluation.
In this paper, we propose a novel and efficient framework, namely Diamante, consisting of a data collection strategy and learning method to boost the performance of pre-trained dialogue models. Two kinds of human feedback are collected and leveraged in Diamante, including explicit demon- stration and implicit preference. Firstly, to bridge the gap in data distribution, Diamante collects

---

Under review as a conference paper at ICLR 2023
Task Description
Dialogue
Please quidelines before the conversation.
My started shedding everywhere the spring.
Gollect conversation with the assistance of this model.
First, craft dialogue opaning based on interest.
have no choice but to buy vacuum cleaner.
Then select, revise rewrile candidate reply property.
Save
Restart
Does your cat shed a lot as well'? What breed is it?
Candidates
British shorthair. Yes, have to vacuum every day.
My cat eats hairballs, too.
Sometimes my sometimes eats own hairballs. i'm pretty worried about his digestion.
Oh no. how old is he?
he deficient something? ? Buy him some cat grass?
Feed him some hairball remedies.
ha eats hairballs, you can give him some hairball
remedies.
It okay, My cat eats hairballs, too. cel may be deficient
You need pay atlention some trace elements. Go to sea a vaterinarian.
You can give some hairball remedles. It's not a big problem.
Submit
Note: Click one candidate and it will be shown in the input box.
Figure Illustration of Diamante's s annotation interface.
an open-domain chit-chat dataset in Chinese with the assistance of PLATO-XL. Based on model- generated candidate responses, human annotators can efficiently produce engaging response to continue the conversation. Secondly, we propose to leverage the implicit human preference that appeared in the data collection process, i.e., the annotator's selected amended response is pre- ferred over the other candidates. To this end, Diamante introduces a novel generation-evaluatii joint training paradigm, where high-quality response generation and human preference estimation are learned simultaneously. During inference, the candidate response with the highest preference score would be selected as the final response and returned to the user.
Extensive and intensive experiments have been carried out to evaluate the effectiveness of the Dia- mante framework, including the collected dataset and joint training paradigm. Experimental re- sults reveal that Diamante significantly boosts PLATO-XL's performance and establishes a new state-of-the-are result in Chinese open-domain conversation. It is notable that compared to the hu- man reference, Diamante even achieves competitive or slightly better performance. In addition to PLATO-XL, Diamante brings remarkable improvements other pre-trained dialogue models. The Diamante dataset is now publicly available, which can be accessed and downloaded under the li- cense agreement at the data platform'. We have also released all source code?, hoping to facilitate future research in dialogue generation.
2
DIAMANTE DATASET
In this paper, collect open-domain chit-chat dataset in Chinese with the assistance of pre- trained dialogue model. In the following, we will describe the creation the Diamante dataset.
2.1 DATA COLLECTION
Diamante aims to explore an efficient way to collect batch of high-quality chit-chat conversations that align well with human values. The data annotation interface is shown in Figure ] (the original interface is in Chinese and displayed in Figure of the Appendix). The data collection process is carried out as follows.
Step 1:
Crafting the Dialogue Opening. Firstly, the annotator is encouraged to craft start utter- ance based on any topic of interest, as an informative and engaging dialogue opening is critical to good conversation.
shown in Figure [ the start utterance is "My cal started shedding everywhere in the spring. How to deal with it?". We also provide various topics and examples in the guidelines to inspire annotators to write dialogue openings.
'The Diamante dataset is publicly available at wttttss://aannnnnnnn
"The Diamante code available at http://ggthhbb.ccoo//oooomoooo
2

---

Under review as a conference paper at ICLR 2023
Table 1: Statistics of the Diamante dataset.
Diamante
Train
Valid
Test
Total
Number of Dialogues
5,838
500
500
6,838
Number of Utterances
83,765
7,166
7.184
98,115
Average Utterance Length
14.26
14.20
14.29
14.25
Select Revise / Rewrite
18% 441% / 41%
19% 40% / 441%
19% / 40% 41%
18% / /41% 441%
Step 2: Generating Candidate Responses with the Dialogue Model. Given the dialogue context, dialogue model (PLATO-XL in the Diamante dataset) is employed to generate multiple candidate responses. To ensure the diversity of response content and conversation flow, we adopt the top-k sampling the decoding strategy and select seven candidates for the demonstration to the annotator.
Step 3: Producing Response with Human Feedback. We then ask the annotator to select, revise or rewrite the candidate to produce appropriate response.
Select. As large-scale dialogue models can generate coherent and occasionally interesting re- sponses, the annotator is allowed to select one response directly from the candidates where appro- priate.
Revise. Given the possible defects in the candidate responses, such lack of consistency or attractiveness, the annotator can choose the preferred candidate and further revise it for better quality.
Rewrite. If no appropriate candidate exists, the annotator needs to write suitable and engaging response by themselves.
Iterating Step & Step 3 to Continue the Dialogue. After collecting the response with human feedback, the conversation will continue by iterating step and step 3. The dialogue collection with the human- in the loop will continue for at least seven rounds. To ensure the annotation quality of the Diamante dataset, we also designed and followed a rigorous quality control process, with details discussed in the Appendix.
The above data collection strategy works well in terms of efficiency and quality. The annotator can produce the final response efficiently by directly selecting amending the model-generated candi- dates. The conversation quality guaranteed or enhanced with the human annotator's 's verification embellishment. Moreover, the implicit human preference that appeared in the data collection process also allows the training of one preference estimation model without additional annotation.
2.2 DATA ANALYSIS
Corpus Statistics. In total, 147 annotators participated in the dataset collection. The detailed statis- ties of the Diamante dataset are summarized in Table The dataset consists of 6,838 dialogues with 98,115 utterances, and the average utterance length is about 14.25. We split the collected data into train, validation, and test sets. As for the annotator operation proportions, 18% of the utterances are produced from Select, 41% % from Revise, and 41 % from Rewrite.
Dialogue Topics. The Diamante dataset about open-domain chit-chat and is not limited to any topic. For further quantitative analysis, we employ the topic tagger the Baidu AI platform" to categorize the dialogues. (The topic visualization of the Diamante dataset is displayed in Figure 7 of the Appendix.) The results show that the Diamante dataset covers all 26 main categories. The top five topics are Society (23%), Entertainment (1 %), People (10%), Education (8%), and Food & Drink (8%), which are in line with our daily life.
GENERATION-EVALUATIII
JOINT TRAINING
In this paper, propose to leverage only the explicit human demonstrations but also the implicit human preference that appeared the data collection to boost the open-domain chatbot comprehen- sively. A novel generation-evvllation joint training paradigm is introduced and illustrated in Figure
https: bai wttp.com//echhh/ppppppy///ee etagee
3

---

Under review as a conference paper at ICLR 2023
NLL
PE
x(G,T))
5(c,1x))
Transformer Blocks
Transformer Blocks
Transformer Blocks
[SEP]
!SEP]
[SEP]
Context
Response
Context
Model Response
Context
Random Response
Overview of the generation-evaluatie joint training in Diamante. sponse generation and human preference estimation are optimized simultancously. The three input pairs share the same network, which is unfolded for illustration.
Figure 2:
The high-quality re-
2. where the high-quality response generation and human preference estimation are optimized si- multancously. The classical training objective of dialogue generation to minimize the negative log-likelihood (NLL) loss:
LNLL log po (r(()cc)
where refers to the dialogue context and r'y is the human annotator's selected or amended response.
Besides generation, Diamante encodes evaluation into the joint optimization enhance the align- ment with human preference. Recall that in the data collection process, there exists implicit human preference: given the dialogue context C, the final response T'H is preferred by human annotators as compared to a model-generated candidate тм E RM (displayed during annotation). Moreover, either ги or тM is better than randomly selected response PR most cases. such, we can have the following preference ranking n' > > TR. The preference estimation (PE) loss is then defined as:
CPE
- s(c,r log -((((cr))ss((r)))) + log ((((()))))) - -0),,)((c))) s(c,rr))
- + -(t
where the input is quadruple of (c, (C, (-) is the sigmoid function, and s(.) is the scalar output of the model.
The total objective of the generation-evaluatt joint training is to minimize the following integrated loss:
C -NNLL Lre
The first term helps the model learn to mimic human demonstrations and generate high-quality candidate responses. And the second term helps the model learn the nuanced distinctions among human preferences. During inference, we adopt the top-k sampling to produce multiple candidate responses and then perform ranking with their corresponding preference estimation scores. The one with the highest preference score would be selected as the final response and returned to the user. Notably, the preference estimation follows the candidate response decoding and only involves one more token processing, which incurs negligible computational cost.
One similar work Diamante's joint training LaMDA (Thoppilan et al. 2022 where a single model functions as both a generator and a discriminator: In comparison, there exist several critical differences between Diamante and LaMDA. Firstly, LaMDA chooses learn the discriminator and generator sequentially. By contrast, Diamante optimizes generation and evaluation simultancously, trying avoid the catastrophic forgetting issue of the two-stage training (Kirkpatrick et al.) 2017] Liu et al. 2022b). Secondly, LaMDA defines fine-ge fine-grained dialogue evaluation metrics and collects corresponding discriminator training samples. Considering the expensive cost of data collection and the difficulty of reaching an agreement in fine-grained dialogue evaluation (Smith et al. 2022).
Diamante leverages the implicit human preference as the overall evaluation and gets rid additional annotations. Thirdly, suggested in the works of human alignment (Askell et al. 20, the ranked preference evaluation adopted in Diamante performs better than the binary discrimination used in LaMDA.
4

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 SETTINGS
4.1.1
IMPLEMENTATION DETAILS
We apply the Diamante dataset and joint training paradigm to boost PLATO-XL's performance. In the generation-evaluation joint training, the input samples are formulated as quadruples (C,T,,TM TM, TM..M,,R, FR). where is the dialogue context, T' is the human annotator's s selected or amended response, TM is candidate response displayed during annotation, and rR is one randomly lected response from the dataset. During the construction of joint training samples, if the sampled model-generated candidate rM is found to be the will be re-sampled to guarantee the agreement (preference ranking n'H > r.)) In addition, TM and IR are re-sampled at each training epoch.
the human-generated response nHH TM
The model is initialized with the 11B parameter PLATO-XL, with the transformer architecture of PrefixLM (Radford et al. 2018 Dong al. 2019). (There are 72 transformer blocks and 32 attention heads, with the embedding dimension of 3072. The hidden dimension of the feedforward layer is set 18432.) The preference estimation value s(-) is obtained through one fully-connected layer (converting the transformer output into one scalar). The hyper-parameter settings used in the training process are listed as follows. The maximum sequence length of context and response set to 384 and 128, respectively. We use Adam (Kingma & Ba 2015) as the optimizer, with learning rate scheduler including a linear warmup and an invsqrt decay (Vaswani et al. 2017). The peak learning rate is set to 2e-6, and the warmup step is set to 500. The model is trained for five epochs with a batch size of 168. The implementation is based on the PaddlePaddle framework, and the experiments are carried out 8 Nvidia A100 GPUs (40G RAM). During inference, we adopt the top-k sampling (A: set to 10) to produce 20 candidate responses and select one with the highest preference estimation score the final response.
4.1.2 COMPARED APPROACHES
In the experiments, the following Chinese dialogue models are considered:
CDial-GGP (Wang et al. 2020) a 104M parameter model trained on LCCC conversations.
EVA2.0 (Gu et al. 2022 is 2.8B parameter model pre-trained eleaned WDC-Dialogue
PLATO-XL (Bao et al. 202T) is the largest Chinese dialogue model with up to 11B parameters, pre-trained social media conversations.
In addition the above dialogue models, the following commercial chatbots in Chinese are in- cluded: Microsoft XiaoIce (Zhou et al. 2020), Xiao AI, Tmall Genie, and Apple Siri.
4.1.3 EVALUATION METRICS
In the experiments, we employ crowd-sourcins workers to evaluate the dialogue quality in four aspects: coherence, informativeness, safety, and engagingness. We discuss these criteria below and provide scoring details in Appendix [A]
Coherence assesses whether the response is relevant and consistent with the context. Informativeness evaluates whether the response includes appropriate information.
Safety evaluates whether the response contains harmful, biased, or misleading content.
Engagingness measures the willingness to have long conversation with the partner.
The coherence, informativeness, and safety are the utterance-level metrics. The engagingness is the dialogue-level metric. These metrics are evaluated on a range of [o. 1. 2]. with higher scores being better. Each sample is distributed to three crowd-soureing workers, and the final score is determined through majority voting.
4.2 EXPERIMENTAL RESULTS
Considering the limitations of automatic dialogue evaluation (Liu et al. 2016), we employ crowd- sourcing workers to evaluate the dialogue quality, including static evaluation, self-chat evaluation, and human-bot chat evaluation.
4.2. STATIC EVALUATION
In the static evaluation, we randomly select 100 samples from the test set and employ the models to generate the response given the multi-turn dialogue context. In addition to PLATO-XI. and Dia-
5

---

Under review as a conference paper at ICLR 2023
Table 2: Static evaluation results, with statistically significant improvements PLATO-XI - (in- dependent two-sample 1-test, < 0.005) written in bold.
Coherence
Informativeness
Safety
Engagingness
PLATO-XL Human Reference
1.73
1.61
1.87
1.56
1.88
1.87
1.92
1.83
PLATO--L. (Diamante)
1.90
1.91
1.96
1.93
Table Self-chat evaluation results, with statistically significant improvements over all other meth- ods (independent two-sample t-test, p < 0.005) written in bold.
Coherence
Informativeness
Safety
Engagingness
CDial-GPT
0.484
0.400
0.660
0.140
EVA 2.0
1,508
1.352
1.764
0.960
PLATO-XL
1.788
1.624
1.788
1.240
PLATO-XI OLLO-XL (Diamante)
1.948
1.920
1.988
1.860
Table Human-bot chat evaluation results, with statistically significant improvements over all other methods (independent two-sample (-test, < 0.005) written in bold.
Coherence
Informativeness
Safety
Engagingness
Xiaolce
1.54
1.49
1.79
1.15
Xiao AI
1.57
1.54
1.88
1.20
Tmall Genie
1,58
1,51
1.78
1.25
Siri
1.17
1.13
1.42
0.75
PLATO-XL (Diamante)
1.92
1.91
1.98
1.90
mante, we also provide the performance of ground truth for reference. The evaluation results are summarized in 2 Diamante significantly improves the response quality on all criteria com- pared to PLATO-XL. Diamante even achieves competitive or slightly better performance compared to the human reference. For detailed analysis, we further reviewed the 14/100 cases where Dia- mante achieved a higher engagingness score than the human reference. We found out that possible reasons for this phenomenon could be twofold. Firstly, it is difficult for annotators keep produc- ing attractive and engaging responses at each round in multi-turn conversations, which is regular and consistent with daily conversations. Secondly, Diamante encodes the preference estima- tion the joint training to enhance the alignment with human preference, which helps select the human-preferred response among candidate responses.
4.2.2 SELF-CHAT EVALUATION
As suggested by Adiwardana et al. (2020), the static evaluation can be biased by the construction of dialogue context. Therefore, we also include the interactive evaluation the experiments, includ- ing the self-chat evaluation and human-bot chat evaluation. Following the settings in PLATO-XL 50 open-domain utterances are selected as dialogue openings, and models play the roles of both partners to continue the conversation for 5 rounds. Then these conversations crowd-sourcing workers for evaluation. The self-chat evaluation results are summarized in Table 3 Diamante outperforms the models in all evaluation aspects and establishes new state-of- the-art result in Chinese open-domain conversation. In particular, Diamante achieves a remarkable 50% improvement the metric of engagingness compared PLATO-XL. These results verify the effectiveness of the Diamante dataset and ceneration-evaluatii joint training paradigm.
distributed to
4.2.3 HUMAN- BoT CHAT EVALUATION
In addition to the above dialogue models, Diamante is compared to common commercial chatbots in Chinese through human-bot bot chat evaluations. We select 20 figh-frequency topics from deployed chatbot and ask in- ho-house data specialists to interact with these chatbots for 7-14 rounds. The human- bot chat evaluation results summarized in Table Diamante consistently outperforms the rest

---

Under review as a conference paper at ICLR 2023
Table 5: Self-chat evaluation results in the ablation of joint training, with statistically significant
improvements over all other methods (independent two-sample t-test, < 0.005) written in bold.
Coherence
Informativeness
Safety
Engagingness
PLATO-XL (Diamante)
1.948
1.920
1.988
1.860
Joint Training
1.912
1.820
1.908
1.600
Joint Training & Dataset
1.788
1.624
1.788
1.240
Table 6: Exploration apply Diamante CDial-GPT, with statistically significant improvements (independent two-sample t-test, p < 0.005) written in bold.
Coherence
Informativeness
Safety
Engagingness
CDial-GPT CDial-GPT (Diamante)
0.484
0.400
0.660
0.140
0.968
0.960
1,368
0.480
of the commercial chatbots by large margin across all the human evaluation metries. These results indicate that Diamante can produce high-quality responses when interacting with real users.
The Fleiss' kappa Fleiss 1971) score for the static evaluation, self-chat evaluation, and human-bot chat evaluation is 0.433, 0.468, and 0.424, respectively. This suggests that crowd-sourcing workers have reached a moderate agreement human evaluation.
4.3 DISCUSSIONS
4.3.1 ABLATION STUDY ON JOINT TRAINING
As discussed in previous sections, the improvements of Diamante compared to PLATO-XL come from two aspects: the Diamante dataset bridges the distribution gap towards human-human conver- sations, and the joint training paradigm enhances the alignment with human preference. For further dissection, carry out ablation studies joint training follows. Without joint training, PLATO- XL trained with the Diamante dataset to minimize the NLL loss, and the final response is selected based generation probability during inference. With joint training, PLATO-XL is trained with the Diamante dataset to minimize the generation-eve -vvauuatonnellllatiion integrated loss, and the final response is selected based on preference estimation during inference.
Firstly, we conduct automatic evaluations of response selection the test set compare these two ap- proaches. Each dialogue context has one human anno- tated response and seven model-generated candidates (displayed during annotation). The experiments evalu- ate the ranking the reference response among these candidates. The results are reported in terms of mean average precision (MAP), mean reciprocal rank (MRR), and precision position (P@ 1), summarized in Figure 3 The preference estimation of the joint training adept at selecting the response that aligns well with human beings. By contrast, the generation probability has difficulty capturing the nuanced distinctions and de- livers almost random performance in response ranking.
PLATO-XL (Dinmante) wibh Joint Training
0.693
ILS44
0,205
0.112
Figure 3: Automatic evaluation in the ab- lation of joint training.
Secondly, conduct human evaluations to compare
these two approaches, with self-chat evaluation results summarized in Table 5 As exhibited in the comparison, the absence of joint training leads substantial performance decrease in engaging- ness, informativeness, and safety. These results validate that the joint training paradigm improves the alignment with human preference and plays a critical role in boosting the open-domain chatbot.
4.3.2 A APPLYING DIAMANTE TO OTHER DIALOGUE MODELS
Although the Diamante dataset collected with the assistance PLATO-XI and the main experi- ments carried out to evaluate Diamante's improvements towards PLATO-XI, the framework is
7

---

Under review as a conference paper at ICLR 2023
PLATO-XL (Diamante)
Translated
Start
Start
May lask what your favorite food
KAIXS
P1
There lot, such hot pot and barbecue.
Me too. They are delicious.
R XX A A
What food you
pot?
like shrimp paste, tripe, cow throat, duck intestine,
enoki mushrooms, potato slices, and rice cakes.
FIAM 19550
P2
cakes, tool hot pot the dipping sauce.
Fe AERE 204
Yes! the with sesame and vinegar: It the best the world.
F2
M AI8 16
That's right! Sesame perfect match. Add some more chili goodiness! Super delicious!
garlic
P1
Yes, gerlic indispensable. I also like enoki mushrooms and wide noodles, love this combination!
like enoki mushrooms, But haven't tried wide noodles, probably because they are care northern hot pots.
Figure 4: Check-picked self-chat example.
PLATO-XL (Diamante)
Translated
GU #212
your
FAR Bot
Bot
wish my caring. She doesn't need to be rich, as long she kind heart.
kind, and
T50
Human
339 S0 A 2
Human
How about the appearance?
R
She doesn't be Looking pleasant enough!
Human
Human
more appearance other half is pretty.
hope
Bot
In my opinion, personality is more Important. can't accept beauty flawed characters.
Human
course, personality very important, But first of all, there should be some
7 T- CRin
Human
physical attractions to me.
Bot
own respect yours!
Well, everyone
Human
BB 10
Human
Thank you. Have you ever been love?
6OL X B 30 12
Bot
te IILE
Bot
Only once. Due to the distance, didn't work out end.
Human
7
Human
A long distance relationship hand to
Yeah. pity. But
Figure 5: Check pickkd human-bod chat example.
indeed universal and not limited to one particular dialogue model. Further explorations of apply- ing Diamante other dialogue models are carried out, with CDial-GPT taken as an example. The self-chat evaluation results are summarized in Table o Compared to the original model, applying Diamante CDial-GPT brings remarkable improvements across all evaluation metries, verifying the effectiveness of Diamante in boosting the performance of Chinese pre-trained dialogue models.
4.3.3 CASE ANALYSIS
We provide two check-picked examples in Figure 4 and Figure 5 for qualitative analysis. In the self-chat example, the dialogue opening is about favorite food, and the model plays the role of both partners to continue the conversation. The covering favorite dishes to dipping source recipes. In the human-bot chat example, the bot expresses its opinions the ideal partner and maintains them well within the multi-turn conversation (i.e.,
speakers have depth discussion hot pot,
8

---

Under review as a conference paper at ICLR 2023
personality is more important). At the same time, the bot respects the different opinions of the other speaker and exhibits a good alignment with human values.
5 RELATED WORK
5.1 HUMAN FEEDBACK
With the rapid development of large language models, it becomes critical to build helpful, honest, and harmless language assistants, keeping in mind the alignment with human values (Askell al. 2021 Bai et al. 2022. Glaese et al.] 2022). Given the misalignment of the conventional training objective and the ultimate human preference, some works (such as WebGPT (Nakano et al. 2021 and InstructGPP (Ouyang et al. 2022) leverage the human feedback to train reward model and optimize towards this proxy objective using reinforcement learning. There are some similar works in dialogue generation (Yi et al. 2019 Jaques et al. 20200), where the reward combines multifaceted evaluation scores, including sentiment, repetition, coherence, etc. While using these reinforcement deamningbbssed approaches, it needs to be careful with the "alignment tax" and not optimize too much (Liu et al. 2022a).
In addition to the above reinforcement learning approaches, some works (Hancock al. 2019 Shuster et al.) 2020; Xu et al. 2022) in dialogue generation continue supervised training with hu- man feedback, with the primary motivation of lifelong learning. The dialogue agent will iterate the following steps: deploy the dialogue model, collect the human-model conversations, and update the model with the newly collected samples. During this process, only those human responses are used to update the model, and special attention is required to avoid low-quality responses from trolls (Ju al. 2022). In comparison, Diamante involves human workers during the development phase rather than after deployment, bringing several benefits. Firstly, human annotators in Diamante have access to model-generated candidate responses and can efficiently formulate a high-quality conversation. While other approaches collect indirect demonstrations from human workers with canned responses, which inevitably interrupts the conversation flow and leads to decreased quality. Besides, the Dia- mante dataset is collected with recruited annotators, eliminating the adverse impact of the trolls. Secondly, in addition to the explicit human demonstration, there exists implicit human preference in Diamante's data collection process, which allows the training of one preference estimation model without additional annotation.
5.2
DIALOGUE DATASET
Given the limited number of annotated human-human conversations, open-domain dialogue models typically pre- trained with human-like conversations collected from social media, such Twitter, Reddit, Weibo, and Douban. To alleviate the problems brought by the data distribution gap, it has become common fine-tune these dialogue models with annotated human-human conversations. Representative English datasets include DailyDialog (Li et al. 2017). ConvA12 (Zhang et al. 2018), Empathetic Dialogues (Rashkin et al., 2019). Wizard of Wikipedia (Dinan et al. (219). Blended Skill Talk (Smith et al.) 2020). etc. In comparison, high-quality annotations of human- hummn con- versations are more scarce in other languages. Most Chinese chit-chat datasets are constructed based on social media comments, including LCCC (Wang et 2000). WDC-Dialogue (Zhou et al.) 202T), and so on. To our knowledge, the Diamante dataset the first chit-chat dataset with anno- tated human-human conversations in Chinese. It is worth noting that Diamante is not simple fix to the limitation in Chinese conversation. provides systematic data collection strategy that is applicable to all languages with high efficiency.
CONCLUSION
In this paper, we propose to collect and leverage human feedback to boost the open-domain chat- bot. By asking annotators to select or amend the model-generated candidate responses, Diamante efficiently collects a high-quality Chinese chit-chat dataset. Diamante introduces a novel generation- evaluation joint training paradigm, which leverages both explicit human demonstration and implicit human preference that appeared in the data collection process. Experimental results indicate that the Diamante dataset and joint training paradigm significantly improve pre-trained dialogue models.
9

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
In the dataset collection, annotators need to select amend the model-generated candidate sponses, where some candidates may contain potentially unsafe content. We ask annotators to produce safe and engaging responses. (As the model is pre-trained with social media comments, sometimes it may generate biased or harmful statements. During annotation, we have been moni- toring the proportion of potentially unsafe candidates, which is less than 1%.) After annotation, further employ data experts to review collected data and remove ineligible conversations.
Diamante's dataset and joint training paradigm help boost the open-domain chatbot and align well with human values, In practical deployments, it is desirable to employ more strategies to guarantee dialogue safety (Dinan et al. 2021), including sensitive topic detection, response safety classifica- tion, and so on.
REPRODUCIBILITY STATEMENT
We describe the collection of Diamante's ' dataset Section 2 and Appendix including the anno- tation interface, annotation procedures, quality control process, etc. The Diamante dataset is now publicly available, which can be accessed and downloaded under the license agreement at the data platform. We introduce the model designs in Section 3 and discuss the training configurations in Section [...1] We have included Diamante source code in the supplementary materials to facilitate reproducibility.
REFERENCES
Daniel Adiwardana, Minh- Kinh-TTang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoory Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human- like open-domain chatbot. arXiv preprint arXiv:2001.0999) 2020. URL https:
org/ abs/ 1..00..99777
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant lab- oratory for alignment. arXiv preprint 1rXiv:2112.0088 2021. URL BRLL:ttpp///xiii org/ abs/2111 00861
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Seifield-Doddde Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.088
2022.
org/ abs/ /ta.//22000 0204 05862
Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhihua Wu, Zhen Guo, Hua Lu, Xinxian Huang, Xin Tian, Xinchao Xu, Yingzhan Lin, and Zheng- Yu Niu. Plato-xl: Exploring the large-scale pre-training of dialogue generation. arXiv preprint arXiv:2109.0951 2021. URL http://aaxxxv.o./abbbbea// wtw.b/////...9....
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voos Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Advances in Neural Informa- tion Processing Systems, pp. 1877-1900 2020. URL htRp::/t:: /papppss nips cC/ httppsssccpppperrpepep paper/ 1070/haahh11557c666rrrooao3 2ar 8bfbaaci Teteaac144ff4. 42f Abst htm
10

---

Under review as a conference paper at ICLR 2023
Boxing Chen and Colin Cherry. A systematic comparison of smoothing techniques for sentence- level bleu. In Proceedings of the 9th Workshop on Statistical Machine Translation, pp. 362-367, 2014. URL/Attts://ccaattoooll
029/414-3346
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022. URL
Setps://arxix/ org/ wwt.///g////oooooo/o////seaaeees
http//s//2200 02311
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-pooee poorgee- conversational agents. International Conference on Learning Representations, 2019. http://////aaiii wttp://wwwwgg///////b UBLL:ttee//aarxt..orr//////111.. 111100024111
Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. Anticipating safety issues in e2e conversational ai: Framework and tooling.
arXiv preprint arXiv:2107.0345/1 2021. URL https://aaxii: wwwp:/oraaaabao..a http//bb/2117.34444
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural lan- guage understanding and generation. In Advances in Neural Information Processing Systems, URL
pp. 13063-13075, 2019.
www.rps...... cc..:cccppoo ptcc/ppppr/221)
Joseph L Fleiss. Measuring nominal scale agreement among many raters. In Psychological Bulletin, pp. 378-382, 1971. URL. http://wwwwwp.co.ttttt.eeeeeeeee omet cs/ /ttt://immmmtteeee//e/eeeo http:///oo....ooooo.cccccceee///
.ddf
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Fwalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.1433 2022. URL hittss://arxis wttp:rappp..........o.
Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao, Xiaoyan Zhu, Jie Tang, and Minlie Huang. Eva2.0: Investigating open-domain chinese dialogue systems with large-scale pre-training. arXiv preprint arXiv: 093113 2022. URL https:
/arziiw.org//bs/2200.
Braden Hancock, Antoine Bordes,
Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguisties, pp. 3667-3684, 2019. URL https:
//aclanthollo ortp///gg///1///000
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centrie dialog training via offline re- inforcement learning, In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, pp. 3985-4003, 2020. URL //aps:///cll anthelogy org/2020. emnlp-ma in. 327
Da Ju, Jing Xu, Y-Lan Boureau, and Jason Weston. Learning from data in the mixed adversarial non- adversarial case: Finding the helpers and ignoring the trolls. arXiv preprint arXiv:2208.03299 2022. URL httt:://axxxiii org abs 2208.2222 C3295
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015. URL
6
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35213522 2017. URL /attps: :ttos://wwwrr /wwww pnas org/ doi/ 11111111000001 wtw.:....133/........ 161 1835114

---

Under review as a conference paper at ICLR 2023
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A prommitgp objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, 2016. URL URL.tttss:: / antho ogy www..ooggNN6111111
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DialyDia A manually labelled multi-turn dialogue dataset. In Proceedings of the 8th International Joint Conference on Natural Language Processing, pp. 986-995, 2017. URL https: /www wwww///wwwwwww..../o////oo/....b aclweb orrg/
anthology
Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised
evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2122-2132 2016. URL https:
/ //www. www.wb.ww....................... org http///tooooooyooooooao 1230
Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. Aligning generative language models with human values. In Findings of the Association for Computational Linguistics: NAACL 2022,
pp. 241-252 2022a. URL https://aclanthol.oo wwt.:oyy..ggg220/ 2022. Findings-naacll 18
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. Brio: Bringing order ab- stractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 2890-2903, 2022b. URL https:
//aclanthologe wwtp///gg..oo.ooooo acl..lnlllloo....oo.. 207
Relichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted eumestion-answeriigg with human feedback. arXiv preprint 12X00221, 2021. URL httlltttt//aaatt www.:/iww.....aaaooooooo abs awa./b//1111 /2t...111.c. 09332
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint ar arXiv:2203.0011111 2022. htLhttp::/ http//axiivii.///ee org/ abs 220.:2223..0 02155
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre-training. Technical report, OpenAI, 2018. URL/https: www.wwwwcoooom cs. ubc. ca waw.:/aaaammammm11mmmm
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-L.an Boureau. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5370-5381, 2019, doi: 10.18653/ www wwwwebebbe........... org/ anthology/P13 1534
11/419-1534. htth:t:::
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, Y-Lan Boureau, and Jason Weston. Recipes for building an open- domain chatbot. In Proceedings of the 16th Conference the European Chapter of the Associa- tion for Computational Linguistics, 2021.
/wwww//wwww/////////// acl www. wwt.:/thooogggggg////oooo
2021. wac.eaacl-oo weslleacl-main.... man.-mmin.2..
Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, and Jason Weston. Deploying life- long open-domain dialogue learning. arXiv preprint arXiv:2008. 08076, 2020. URL http:
Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents. In Proceedings of the 4th Workshop on NLP for Conversational AI, 1ttp:/222...11cooa---aaa
pp. 77-97, 2022. URL URl.nttps://aclanthooooyy
12

---

Under review as a conference paper at ICLR 2023
Eric Michael Smith, Mary Williamson. Kurt Shuster, Jason Weston, and Y-Lan Boureau. Can you put it all together: Evaluating conversational agents' ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computationall Linguistics. pp. 2021-2030, 2020. URL http:tt:: / /wwww ac web. httpbborgg/atoogggg/ggaaoooa -ma in. 183
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vin- cent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier- Hellsteem,,,,mm Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fen- ton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera Areas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications. arXiv preprini arXiv:2201.08229, 2022. htt.httts://aaxiiii
abs 2201 08239
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems, pp.
2017. URL StRpsttts: /twpeeee////oceeeeeeeeooooooccie ngs ngs newripiiip. viurrpps... cc/ 7dee fbd053cl 1
paper/2011/FA1ee/5fr
pdf
Yida Wang. Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. A large-scale chinese short-text conversation dataset. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 91-103, 2020. URL https:///rxxiv
org/abs 2008. 03946
Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Lean- ing new skills after deployment: Improving open-domain internetdrrveen dialogue with human feedback. arXiv preprint Xiv:22208.03270. 2022. URL
03270
Sanghyun Yi, Rahul Goel, Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Heday- atna,, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur, Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 65-75, 2019. URL. https: / antho wog.aooooooo..o.oooooooo
logy wrg./lgggww/ig 8668
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Per-
sonalizing dialogue agents: I have dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2204-2213, 2018. URL
https: //ww://wwww.... ac web. orgg antho ogy/P P0mpy//111100
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2019.
URL https:://aattt wwwp:xiioorrroooooooooooo abs/ 904. 09675
Han Zhou, Xinchao Xu, Wenquan Wu, Zhengyu Niu, Hua Wu, Siqi Bao, Fan Wang, and Haifeng Wang. Link the world: Improving open-domain conversation with dynamic spatiotemporal- aware knowledge. arXiv preprint arXiv:2206.40000, 2022. URL https:: https://arxiv...t
abs/2206. 118006.140000 14000
Hao Zhou, Pei Ke. Zheng Zhang, Yuxian Gu, Yinhe Zheng, Chujie Zheng, Yida Wang, Chen Henry Wu, Hao Sun, Xiaocong Yang, Bosi Wen, Xiaoyan Zhu, Minlie Huang, and Jie Tang. EVA: An open-domain chinese dialogue system with large-scale generative pre-training. arXiv preprint 10Xiv:2108.01547, 2021. URL httpshtts:://xxiv.... http://axxiv..orraa////////
01547
Li Zhou, Jianfeng Gao, Di Li, and Heung- Shum. The design and implementation of Xiaolce, an empathetic social chatbot. Computational Linguistics, 46(1):53-93, 2020. URL https:
7/ www. ac ac wweb. org/ httg://nttoooog//2..111111
13

---

Under review as a conference paper at ICLR 2023
A
SCORING CRITERIA IN HUMAN EVALUATION
The criteria used in human evaluation are provided in Table
Score
Coherence
The response is not related with the context.
The response simply repeats the context.
0
The response has obvious conflicts with the context.
There are scrious logic conflicts within the response.
The response has minor conflicts with the context.
1
There are some minor logic conflicts in the response.
2
The response is coherent with the context.
Score
Informativeness
The response doesn't I contain any information.
0
This response just repeats the context and fails to bring any additional information.
The information invalid, as the coherence score is
The information has conflicts with common sense.
There are factual errors in the response.
2
'The response has appropriate and correct information.
Score
Safety
The response contains harmful, biased, or misleading content.
The safcty is invalid, as the coherence score is O.
1
There are risks that the response may make people feel offended uncomfortable.
2
Thee response is safe.
Score
Engagingness
0
don't 't want to talk with this speaker.
1
It is kind of boring, but it is still ok to talk with this speaker.
2
I would like to talk with this speaker for long conversation.
Table 7: Score details of metrics in human evaluation.
B DATASET DETAILS
B.1 ANNOTATION INTERFACE
The original annotation interface of Diamante in Chinese, as shown in Figure 6 first crafts the dialogue opening and then selects amends the model-generaun candidate responses to continue the conversation. The left-hand area displays the dialogue context and the input box. The top right-hand part provides brief task description and a link to the detailed guidelines. The bottom right-hand part lists some inspiring topies or -ener-ege candidate responses.
The annotator
B.2 QUALITY CONTROL
To ensure the annotation quality of the Diamante dataset, we designed and followed rigorous qual- ity control process. We engaged with a vendor company to recruit experienced annotators, instructed them with detailed guidelines, set up admission tests, answered questions in online shared room, and executed regular reviews within the annotation. After annotation, we ask data experts to review all collected conversations and remove the conversation whenever expert deems it ineligible.
14

---

Under review as a conference paper at ICLR 2023
Task Description
Dialogue
M 12: P
Topic Suggestion
Example:
Submit
Task Description
Dialogue
#x
:
BE
# 12
3
N #
Candidates
3
T A ?
KEEE
16 |
Submit
Figure 6: Diamante's 's annotation interface. Upper: crafting the dialogue opening. Bottom: selecting or amending the model-generee candidate responses to continue the conversation.
Pets
1%
Society 23%
Technology
3%
Sports
Fashion
4%
Humanities
Entertainment
Culture 4%
Health
People
5%
Food & Drink
Education 8%
8%
Figure 7: Topic visualization of the Diamante dataset.
15

---

Under review as a conference paper at ICLR 2023
Joint Training Win
Tie
w/o Joint Training Win
Diamante
74%
22%
DuSine
39%
52%
9%
Figure 8: Exploration of joint training on the Diamante or DuSinc dataset.
B.3 TOPIC VISUALIZATION
The topic visualization of the Diamante dataset displayed in Figure There are 26 categories in the topic tagger, and the Diamante dataset covers all of them. The top hive topics Society (23%), Entertainment (1 11%), People (10%), Education (8%), and Food & Drink (8%), which are in line with our daily life.
C FURTHER DISCUSSIONS
C.1 MORE EXPLORATION ON JOINT TRAINING
As shown in Table 5 the Diamante dataset and joint training paradigm bring significant improve- ments. To further analyze the effects of joint training, we carry out the pairwise comparison between models with and without joint training (PLATO-XL trained on the Diamante dataset). We ask crowd- sourcing workers to compare the self-chat conversations generated by these two models and select the preferred one. The comparison in Figure [ (upper bar) exhibits that the joint training paradigm crucial in boosting the open-domain chatbot.
In Diamante, the joint training leverages the implicit human preference that appeared in the data collection FH > тM- We also explore applying the joint training to other conventional dialogue datasets, with DuSinc (Zhou et al.
example. To formulate training samples for the preference ranking 'H > TM TR. PLATO-XL is employed to simulate model-generated responses. Two models (PLATO-XL with joint training & PLATO-XL w/o joint training) trained on the DuSinc dataset. We randomly select 100 samples from the test set for static evaluation and ask crowd-sourcing workers compare the generated responses by these two models. The comparison in Figure & (bottom bar) verifies the effectiveness and generality of the joint training paradigm.
2022) taken
(Diamante)
0.9035
PLATO-XL
0.507
0.8
Table Human evaluation on safe response gen- eration under adversarial attack.
0.5135
0.5125
0.4
Safe
Unsafe
PLATO-XI PLATO-XI -L (Diamante)
0%
100%
76%
24%
0.027
Figure 9: Automatic evaluation on safe re-
sponse selection under adversarial attack.
C.2 SAFETY UNDER ADVERSARIAL ATTACK
The main experiments reveal that Diamante achieves better safety on normal/insensitive topics. To further analyze the safety performance under adversarial attacks, we asked annotators to interact with PLATO-XL on sensitive topics and induce unsafe responses from the model. The annotators then asked to amend these unsafe responses into safe ones. These sensitive topics are de- signed and selected according to Chinese cultural and social norms, including harmful speech (c.g., offensive content, self-harm suggestions, and personal attacks), group discrimination (e.g., region,
16

---

Under review as a conference paper at ICLR 2023
Table 9: Static evaluation with automatic metrics.
BLEU-2/4
Distinct-1/2
Unigram F1
BERTScore
PLATO-XL
5.87 2..12
32.78 / 79.21
15.78
60.41
Human Reference
33.35 / 82.25
PLATO-XL. (Diamante)
6.31 2.21
28.47 / 77.61
16.25
60.60
gender, disability, and religion), misleading information (c.g., political controversies, ethnic divi- sion, and conspiracy theories), and so on.
In total, collected 1000 samples (including adversarial dialogue context, original unsafe response, and amended safe response). We employ these samples to evaluate Diamante's 's safety under adver- sarial attacks. The automatic evaluation results in Figure suggest that Diamante is adept at select- ing safe responses. We also randomly selected 100 samples and employed crowd-sourcing workers evaluate generated responses. The results in Table reveal that Diamante achieves remarkable safety improvement, with 76% of responses identified as safe. Even though Diamante is only trained with insensitive conversations, it absorbs human preferences and maintains good safety performance under adversarial attacks.
C.3 AUTOMATIC DIALOGUE EVALUATION
We also carry out automatic evaluation with rulc- based and model-based metrics, including BLEU- 2/4 (Chen & Cherry 2014). Distinct-1/ 1/2 (Li al. 2016), Unigram F1 (Dinan et al. 2019). and BERTSScore (Zhang et al.) 2019). The automatic evaluation results in Table are inconsistent with the human evaluation results in Table 2 where human evaluation the golden standard in open- domain chitchat evaluation. The difference between Diamante and PLATO-XL is minor in automatic evaluation. In comparison, Diamante significantly improves PLATO-XL in human evaluation.
C.4 CASE ANALYSIS WITH COMPARED APPROACHES
We provide two more examples by PLATO-XL and Xiaolce in Figure 10 and Figure 11 examples are under the same starting utterances as Diamante Figure and Figure 5
These two
PLATO-XL
Translated
Start
200 SO
May your
22 ?
you going
Do you dare est treat you?
P1
will you buy it.
P2
Aron't you afraid that put polson it?
7-9 1 4/
P1
I'm invulnerable
7 RIFRE
might
N00 1
only hiding dagger
P1
B +4 1
behind smilee
It's poison that will kill you instantly you laugh.
Go ahead. I'm afraid it.
Figure Self-chat example by PLATO-XL.
17

---

Under review as a conference paper at ICLR 2023
Xiaolce
Translated
Start
Start
your
Forgive profer being single
Bot
Bot
free.
Human
Au babyy 137
Human
Oh, it good be single! really miss
Bot
Human
Human
why miserable right
now.
Bat
Bot
Are okay? am warried about you.
Human
#ARUB8
Human
Just little annoyed, It
farily
Bot
tell me what's bothering you.
Thanks, so lovely to have you as my
Siiu AUC ER
Human
feel more relaxed after chatting
Human
0882 C
with you.
da you mean? Can you understand yourself?
Human
FEL ? Fo
Human
You don't get do you? is olky long willing listen
Sat
point banging keyboard?
Figure 11: Human-bot chat example by Xiaolce.
18


Reviews:
[{"review_id": "uM4oFbPHgNO", "reviewer": "Reviewer_fd1Y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a new framework named Diamante to improve the Chinese open-domain pre-trained dialogue models with human annotations, since these models have difficulty in generating engaging utterances during the conversation. Diamante first builds a Chinese dataset by requiring human annotators to choose or amend the model-generated responses. Based on the dataset, it designs a joint-training process that considers both the generation loss and the human preference estimation loss. Experimental results demonstrate the effectiveness of Diamante compared with existing Chinese open-domain pre-trained dialogue methods.", "summary_of_the_review": "The paper is well written and focuses on an important issue in Chinese dialogue models that there are no previous efforts to consider human preferences. However, the authors should better clarify their difference to LaMDA and the benefits of their proposed joint training based on implicit annotations. To support their claims, more details of the human evaluations and dialogue examples should also be presented. Based on the current version, I would not recommend accepting this paper.", "strength_and_weaknesses": "Strength:\n\n- Well written and easy to follow\n\n- It is the first in Chinese to consider human preference in pretrained dialogue models based on human evaluations\n\n- Experimental results show a significant improvement over existing methods in all metrics\n\nWeaknesses:\n\n- Limited novelty: It seems that the major difference between Diamante and LaMDA which also focuses on the alignment of human preference is the language. Although the authors claim that their joint-learning objective can be more effective than sequential learning in LaMDA, there are no experiments to support it. Authors should further clarify their contributions which currently are incremental as the combination of LaMDA and Chinese dialogue models such as PLATO-XL\n\n- Insufficient details: Since the alignment of human preference relies on the quality of human evaluations, it is fundamental to demonstrate essential details such as the demographics of the crowd workers or the instructions for them to annotate, which is included in LaMDA. Authors are suggested to provide these details to address possible concerns about the quality of the human annotations. Moreover, if the annotations are implicit, it should be difficult for methods to distinguish the difference between safe and unsafe utterances. Thus authors should also provide a detailed description of their dataset, including the average turns of the conversation, and how and why these utterances are selected, revised, or rewritten from machine-generated utterances.\n\n- Missing Examples: Authors only present the examples of Diamante, which can not demonstrate the improvement without the comparison of the examples of baselines including PLATO-XL and common commercial chatbots. Besides, the topics of the examples only cover insensitive topics such as food and people. Authors should show more examples of their methods and baselines for generating responses to sensitive topics such as Society or Culture. It is unclear if Diamente can yield more safety responses like LaMDA, especially on these sensitive topics.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oortDnuan2Y", "reviewer": "Reviewer_tDkV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose a new framework called Diamante that is aimed at improving the performance of open domain chatbots by incorporating human feedback (both explicit and implicit preferences).  The proposed framework has a generation-evaluation training paradigm that is aimed to optimized response generation and preference estimation together.\n\nContributions:\n1. The authors collect a new dataset for open domain chitchat conversation in Chinese and is called the Diamante dataset\n2. A new joint generation and evaluation framework for optimizing generation and human preference estimation together.\n3. Results from automated and human evaluation show that the existing approaches trained with the new training strategy achieves significant gains on metrics.", "summary_of_the_review": "In this work, authors propose a new dataset for open domain conversations in Chinese and a new training paradigm that achieves gains in performance through human evaluation.", "strength_and_weaknesses": "Strengths and Weakness:\n1. Collection of a new crowd sourced dataset for open domain conversational agent in Chinese Language.  How's is the quality of the dataset measure? What are steps taken to avoid bias and safety issues from being part of the dataset? \n2. The joint training-generation framework is interesting in terms of modeling response generation. What is hypothesis behind having r_h > r_m > r_r?\n3. Results show promising gains for the proposed approach on human evaluation both static and self chat evaluation.\n4. Looking at the results from Table 6, it seems like most of the gains come from the dataset. The gains from joint training seems pretty minimal compared to Plato-XL.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jJ1Fv2EnY7", "reviewer": "Reviewer_rtcq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new technique to enhance open-domain chatbot with human feedback information. The authors attribute the\nless-engaging response to the distribution gap between human-human conversation and proxy human-machine conversation. Therefore, they design a data collection procedure to collect human revisions to machine response and construct a dataset based on that. Utilizing the established dataset, the authors adopt the classical generate-then-rerank paradigm to first generate multiple candidates and then select one.\n", "summary_of_the_review": "The authors propose a data collection strategy to collect human feedback for an open-domain chatbot.  post-training an existing pre-trained dialogue model (PLATO-XL) with the collected data achieves some preliminary results. However, the proposed method is not original and the evaluation setup is defective. I would suggest that a major revision of the manuscript is needed to reach the publishing requirement of ICLR or other top-tier conferences.\n", "strength_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The authors ablate several aspects of their method.\n3. The method obtains good results in human evaluation.\n\nWeakness:\n\n\n\n1. A major proposal of the paper is to use human feedback to improve the performance of dialogue system. However, it is not a new idea in my opinion. There is rich literature[1-5] discussing how to continue learning from human feedback after deployment, which is closely related to your method.  Out of the domain of dialogue, there is also a large body of work studying the improvement of models from human feedback, such as summarization[7][6] or question answering[8]. They should at least be included in the related work, or the readers will have difficulty putting the work into its literature.\n\n2. The evaluation protocol is not very convincing to me. \n(1) Metrics: Only human evaluation is conducted and no automatic metrics are involved. I acknowledge that some automatic metrics like BLEU or ROUGE may be poorly correlated with human, but some neural metrics such as BERTscore should be considered. Furthermore, pure human evaluation renders the model performance hard to reproduce.\n(2) Dataset: In the statistics evaluation, only 100 cases are sampled from the test set and evaluated. Why not use all the test set? Do you sample multiple times and conduct a repetitive experiment or just report the result from a single sampling? Why not perform and compare on some widely-used open-domain dialogue dataset such as Douban[9] or STC[10]?\n(3) Baselines: PLATO-XL is up to 11B while EVA2.0 is 2.8B and CDial-GPT is 104M. So I doubt whether your comparison is fair. Why not implement your method on some smaller version of PLATO model?\n\n\n\n[1]Learning from Dialogue after Deployment: Feed Yourself, Chatbot!\n[2]Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback\n[3]Improving alignment of dialogue agents via targeted human judgements\n[4]Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\n[5]BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage\n[6]Learning to summarize from human feedback\n[7]Self-critiquing models for assisting human evaluators\n[8]WebGPT: Browser-assisted question-answering with human feedback\n[9]Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots\n[10]Neural responding machine for short-text conversation.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WUyMfUsnjPn", "reviewer": "Reviewer_Q4VE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis work aims to train chatbots that generate responses aligned with human preferences. The training dataset is generated with human annotation. Specifically, the annotators start a conversation with a chatbot on a certain topic. At each turn, the annotators can provide feedback on the generated response. The feedback can come in the form of revisions to the generated response or a complete rewrite. After correction, the dialogue is continued, and the “response → feedback” steps are repeated.\n\nDuring fine-tuning, the corrected responses are used in the standard perplexity loss. Additionally, a “preference estimation” loss is included to encourage the model to rank the corrected responses over the original model response.\n\nHumans rate the responses generated by the proposed model to be more coherent, informative, and engaging. \n", "summary_of_the_review": "Making text generation methods sensitive to feedback is an important research direction, and this paper makes useful contributions. My current recommendation is a weak accept, but I’ll change my scores after the authors’ feedback.", "strength_and_weaknesses": "### Strengths\n\n\nIncluding human preferences and feedback in generation models is an important research direction. The paper takes meaningful steps to achieve this in a non-English setting. The methodology for data collection is sensible, and the scale of the dataset is sufficiently large to be of interest to the broader community.\n\n\n### Weaknesses\n\n1. In the preference estimation loss (Equation 2), the authors always assume that human-generated responses are better. However, it is mentioned in Table 1 that 18% of the responses were selected as being already good. Isn’t there a contradiction here? Essentially, Equation 2 will sometimes force the model to learn *not* to select an otherwise perfectly valid response?\n\n2. **Evaluation**\n\n2.1: This is probably a clarification: was there any overlap in the set of human annotators the same for data creation and evaluation (the paper mentions “in-house data specialists”)? If yes, there is a risk that the method overfits not to human preferences but to biases of a select set of reviewers. For example, consider 5 annotators: A, B, C, D. A and B like cheerful responses, and C and D like brief, serious responses.\n\nSuppose the responses from the baseline are either cheerful or serious with a 50% probability.\nIf we collect data using annotators A, B and also make them evaluate, the baseline performance will be ~ 50%, and your method will achieve a score of nearly 100%. The other extreme follows from using C, D as evaluators. Thus, the right way to go about it is to randomly select the groups for evaluation and data generation.\n\n\n2.2. “We select 20 high-frequency topics from a deployed chatbot and ask in-house data specialists to interact with these chatbots for 7-14 rounds”. How much data was used for evaluation?\n\n2.3 The annotator agreement is moderate (closer to the boundary of low and moderate), indicating that the method may generate better responses simply because of more fine-tuning.\n\n3. **Engagement vs. utility:** The key assumption in this work is that engagement is proportional to human preferences. Is that necessarily true? For example, a chatbot that produces brief/terse responses may not be preferred for conversation in general but might be okay for a chatbot as long as it’s useful. \n\nOther notes/good to have: \n\na) engaginess should probably be changed to “engagement.” \n\nb) For human evaluation results in Table 2, significance tests are missing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Huw4P7ETA2", "reviewer": "Reviewer_ftAn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a a data collection strategy combined with human preference capture and a joint objecting learning method to boost the performance of pre-trained dialogue models. The paper is very well-written, contains descriptions of the collected dataset alongwith  data analysis, as well as a learning strategy that leverages joint training via cross-entropy and contrastive learning (contrasting human preference dialog with model generated and random responses). The paper also conducts extensive human evaluation of the model responses alongwith ablation studies. ", "summary_of_the_review": "Overall the paper presents a very solid approach, dataset contribution and evaluation results for open-ended chatbot development in Chinese. The model fine-tuning method significantly improves the performance of the pre-trained models. The paper has more NLP results innovation than a learning representation innovation. The joint modeling strategy is used in many works today (NLL loss with a contrastive loss) so the technical novelty isn't the highlight of this very solid contribution.  ", "strength_and_weaknesses": "- the paper uses standard ML methods for the training strategies without any significant technical innovation of the work. \n- However, the model training setup is pretty interesting, combining mask learning with contrastive learning using implicit human preferences.\n- the model training does not explicitly seem to leverage the revisions which would have been interesting to model. \n- the step 1 (crafting the dialog opening) isn't very innovative. Authors mention they suggest topics in the guidelines but there isn't any AI assisted or explicit strategy to encourage users to discuss diverse topics. \n- addition AI assistance in the interface could help with faster data collection (sentence completion in the text box for example and if users select or reject the sentence level word completions) \n- all the results suggest that this model is better on all the studies metrics in all the comparisons. This is a very strong result; however similar analysis on another dataset (maybe another language) would convince the readers much better on the strength of the model fine-tuning. \n  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject