Fine-grained Conversational Decoding via Isotropic and Proximal Search
Authors: Yuxuan YAO
Han Wu
Qiling Xu
Linqi Song
Venue: EMNLP 2023 Main (2023)
Keywords: text generation; dialogue system; decoding strategy
TL;DR: A fine-grained decoding strategy for dialogue response generation

Abstract: General-purpose text decoding approaches are usually adopted for dialogue response generation. Although the quality of the generated responses can be improved with dialogue-specific encoding methods, conversational decoding methods are still under-explored. Inspired by SimDRC that a good dialogue feature space should follow the rules of locality and isotropy, we present a fine-grained
conversational decoding method, termed isotropic and proximal search (IPS). Our method is designed to generate the semantic-
concentrated response, while still maintaining informativeness and discrimination against the context. Experiments show that our approach
significantly outperforms existing decoding strategies in the dialogue field across both automatic and human evaluation metrics. More in-
depth analyses further confirm the effectiveness of our approach.

Fine-grained Conversational Decoding via Isotropic and Proximal Search
Yuxuan Yao
Han Wu
Qiling Xu
Linqi Song"
City University of Hong Kong
yuxuanyao3-cêmy. ci tyt. edu. hk qilingxu2-cemy. citu..cccc........ edu. hk
hanwu32-c@mp ci cit edu. hk linqi. syugecittu.ee edu.
Abstract
including top-k sampling (Fan et al., 2018) and nucleus search (Holtzman et al., 2018). While alleviating degeneration, these sampling methods introduce critical semantic inconsistency and are not aligned with human- written prefix (Basu et al.. 2021). Specifically, bunch of studies (Fthayarajh, 2019; Su and Collier, 2023) have asserted that the problem of anisotropy, i.e., distribution pat- tern in the latent space with features occupying a narrow cone in the space, leads inconsistency and degradation the generation. Although con- trastive search (Su et al., 2022) has been proposed correspondingly to mitigate the issue, as a general- ized text decoding strategy, it still ignores dialogue- specific features, such as utterance dependencies and conversational structure information. There- fore, research on conversational decoding methods is warmly needed.
General-purpose text decoding approaches are usually adopted for dialogue response genera- tion. Although the quality of the generated re- sponses can be improved with dialogue-specific encoding methods, conversational decoding methods are still under-explored. Inspired by Wu et al. (2023) that good dialogue feature space should follow the rules of locality and isotropy, we present fine- ane-ggrined conversa- tional decoding method, termed isotropic and proximal search (IPS). Our method is designed to generate the conentticccneentrteedd response, while still maintaining informativeness and dis- crimination against the context. Experiments show that our approach outperforms existing decoding strategies in the dialogue field across both automatic and human evaluation metrics. More in in-depth analyses further confirm the ef- fectiveness of approach.
1 Introduction
In this work, we propose fine-grained conver- sational decoding method, namely isotropic and proximal search (IPS). Different from traditional approaches, we consider the previous tokens and contexts scparately from a granular perspective. Acknowledging that locality and isotropy are two important properties for refining the dialogue fea- ture space, we design our IPS following these rules: (i) the generated output should be selected from the most probable candidate set predicted by the dialogue model: (ii) the generated tokens in the same utterance should be proximal each other for expressing a concentrated idea; and (iii) the newly generated utterance should be discrimina- tive enough with respect to the context utterances. In this way, our method encourages informative- ness and discrimination among different utterances as well as maintains concentrated idea within an utterance. We evaluate our approach on commonly-used dialogue datasets. Daily (Li et al., 2017) in English and LCCC (Wang et al., 2020) in Chinese. Both human and automatic eval- uation results, i.e., indicators based on GPT3.5,
Dialogue response generation (Li al., 2017: Wang et al., 2020) aims to generate the utterance that forms a coherent and fluent continuation given a dialogue context. Generic text decoding strategies (Rieser et al., 2014; Ritter et al., 201 I; Chen et al., 2017) are usually adopted to produce grammati- cal and contextual responses. As independent technique, decoding strategy can also enhance the generation quality of large language models.
Existing text decoding methods have been ex- plored in various generic text generation tasks, but lack tailoring for dialogue generation, e.g., captur- ing dialoguc-speciffe features and generating an in- formative and discriminative dialogue response (Su et al., 2021; Wu et al.. 2023). Early maximization- based methods, e.g., greedy search (Li et al., 201 and beam search (Wiseman et al., 2017), may lead dullness and degeneration (Fan et al., 2018; Holtzman et al., 2018). Later sampling-bass based im- provements are proposed to tackle these problems, Corresponding author: iingi.sog@@eityu.edu.u

---

consistently show that IPS can generate more fluent, coherent, and human-like responses than existing decoding methods.
determined during the decoding stage, we calcu- late in an approximate way, i.c., averaging the representations of the already generated tokens, as follows:
2
Methodology
T
R
h
(2)
2.1 Preliminary
where haT is the response representation which will be dynamically updated along with the gen- eration process, and T is the number of already generated tokens.
Dialogue response generation Given dialogue context D [11, u2y,.., uN] composed of N ut- terances, where u, { , Fi,2; .... Tr,l } is se- quence of consecutive words, the task of dialogue response generation is to produce the continuation utterance u, = (1, W2,..., W2,..., [|
Up to now, the problem changes how to gen- erate the first token for starting the isotropic and proximal search since the method is heavily de- pendent on the previous tokens. To address this problem, we attempt to finish the first n-steps gen- eration by traditional decoding methods, such beam search, top-k sampling nucleus sampling, On the other hand, as IPS is essentially determin- istic decoding strategy. this solution also enables produce diverse responses by using different decoding strategies the first n steps. Therefore, in each step after the first sampling stage, we cal- culate the proximal and isotropic values as follows:
(r = N + 1). There are generally two key steps to finish the task, including context encoding and response de- coding. For the first step, we obtain the context representations H from the language model by con- catenating the utterances into sequence.
H (1 [FOU] u2 [EOU] ... uN [EOU]);
where [EOU] is the special token inserted as the last token of each utterance.
For the decoding step, the response is generally produced in an auto-regressive manner follows
p_vue,, =
s sha bwr)
(3)
1-1
II [rrl
p(w1:/))))
(1)
1 value = N 1=1 s shRT hus)
Σ
(4)
Dialogue modeling Wu et al. (2023) has demon- strated that locality and isotropy are two key prop- erties for building a good conversational feature space. Specifically, locality encourages the model to aggregate the representations of tokens within utterance while isotropy pushes away the repre- sentations of distinct utterances.
where is the cosine similarity. ha, are the ut- terance representations obtained from the special token [EOU]. The proximal value measures the average distance between the candidate token and the already generated tokens while the isotropic value stands for the average similarity between the undcrgoing response representation and all utter- ance representations. Next, the selection of the candidate token w/ is formulated as,
2.2 Isotropic and Proximal Search
We present a fine- -ne-ggrined conversational decoding method, i.e., isotropic and proximal search (IPS). Specifically, expect the generated response to satisfy two requirements: 1) representations of the response tokens are nearby to convey concen- trated idea, saying proximity; 2) the response repre- sentation is discriminative to the context utterance representations, saying isotropy.
Wt = argmax x p(w; Wer. D) an (V
model confidence
(5)
+ (1 a) x (p pplue, i value,) )
isotropic and proximal penalty
where (1m) is the set of top-m predictions from the model's probability distribution p(w; | Wet, D) and m, is typically set as 4 ~ 8. In Eq. (5), the first term, model confidence, is the probability of the candidate w, predicted by the model. The second term, isotropic and proximal penalty, aims max- imize the discrimination between the undergoing response and previous utterances and minimize the
During the decoding stage, for proximal search, we try select the candidate token having the shortest average distance the existing generated tokens. For isotropic search, we try to choose the token that enables the response representation most discriminative to representations of context utter- ances. As the response representation cannot be

---

DailyDialog
LCCC
Model
Strategy
Distinet
Distinet
BS
MV 1
GE f
BS
MV
GE f
Dis2 1
Dis4 f
Dis2 f
Dis4 1
greedy
0,1275
0.569
2.17
0.344
0.776
0.0636
0.062
1.88
0.126
0.437
beam
0.1317
0.599
2.29
0.341
0.755
0.0639
0.145
1.91
0.155
0.466
top-k
0,1312
0.623
2.20
0.350
0.780
0.0648
0.154
1.94
0.152
0.487
BART
nucleus
0,1298
0.642
2.34
0.352
0.791
0.0626
0.178
1.91
0.156
0.534
contrastive
0.1147
0.622
2.07
0.396
0.810
0.0538
0.205
1.90
0.190
0.583
IPS 0.1335
0.647
2.43
0.355
0.798
0,0653
0.212
1.98
0.176
0,540
greedy
0,1099
0.447
2.21
0,306
0.709
0.0678
0.088
1.82
0.137
0.470
0.1196
0.556
2.27
0.314
0.713
0.0692
0.206
2.02
0.179
0.539
SimCTG (p 0.5)
top-k
0.1169
0.544
2.06
0.322
0.733
0.0695
0.195
2.11
0.168
0,534
nucleus
0.1169
0,571
2.32
0.327
0,753
0.0680
0.223
2.10
0.169
0.575
contrastive
0.1123
0.608
2.17
0.395
0.807
0.0607
0.278
1.98
0.197
0.618
IPS 0.1293
0.628
2.36
0.359
0.787
0.0704
0.294
2.31
0.196
0.580
greedy 0,1255
0.560
2.06
0,345
0.774
0.0699
0.090
2.21
0.136
0.471
beam
0.1315
0.632
2.18
0.338
0.745
0.0715
0.196
2.11
0.180
0.543
SimDRC
top-k
0.1068
0,648
2.20
0.345
0,773
0,0720
0.203
2.19
0.166
0,540
(6 0.7,
nucleus
0.1284
0.632
2.16
0.353
0.793
0.0697
0.226
1.88
0.166
0.569
a = 0.3)
contrastive
0.1174
0.653
2.16
0.397
0.819
0.0613
0.271
2.21
0.197
0.614
IPS
0.1336
0.665
2.46
0.366
0.800
0.0722
0.272
2.32
0.192
0.569
Table Automatic evaluation results DailyDialog and LCCC, where BS means F1 value of BERTScore (Zhang*
et al., 2020), MV represents MAUVE (Pillutla et al., 2021), and GE represents G-Eval (Liu et al., 2023).
token difference within the response. The hyper- parameter a E [0, 1] regulates the importance of these two components. When a = 1, our method degenerates to the greedy search method.
evaluate the performance of decoding strategies with different models, including vanilla BART, BART with SimCTG (Su et al., 2022), and BART with SimDRC (Wu et al., 2023). We compare IPS to greedy search, beam search, top-k sampling (Fan et al., 2018), nucleus sampling (Holtzman et al., 2018) and contrastive search (Su et al., 2022).
We claim our method is fine-grained because the generic auto-regressive generation predicts the next token by jointly considering the already gen- erated tokens wit and the context D, formulated as p(wy)ee Wes, D) while IPS splits these two fac- tors. Specifically, proximity value only focuses on the effects of the already generated tokens, i.e., value, ~ (x)), and isotropy value pays attention to the context, i.e., i value, p(u/ [D, (w<z)) wherein wit just used to obtain the undergoing response representation hęr-
Settings We fine-tune the models on DailyDia- log and LCCC datasets for 6k steps and 7k steps, respectively. We use batch size of 64 and truncate the training samples to a maximum length of 256. The parameters of the models are initialized from HuggingFace libraries and updated by Adam opti- mizer (Kingma and Ba. 2017) with learning rate of 3e-5. We adopt the margin values of SimCTG and SimDRC suggested in their work, i.e., P 0.5 for SimCTG and 0.7, a 0.3 for SimDRC. We conduct the isotropic and proximal search with the first n = 2 steps adopting top-A sampling (k= 7). The weight is 0.6. We run all experiments with five different seeds and report the average score.
3 Experiments
Dataset We evaluate our method on two commonly-used datasets, DailyDialog (Li et al., 2017) in English and LCCC (Wang et al., 2020) in Chinese. Both of them are open-domain multi-turn dialogue datasets, collected from social media. For LCCC, owing to the academic-level computing re- source, we follow previous work (Su et al., 2022), and sample subset of the dataset, consisting of 100,000 dialogue examples.
Evaluation Metrics Traditional n-gram overlap and text matching metrics such BLEU (Pap- ineni et al., 2002) and ROUGE (Lin, 2004) are not proper to evaluate plausible output diversity for open-domain dialog systems. Therefore, for automatic evaluation, we choose the following met- rics, including BERTScore (Zhang et al., 2020),
Baselines Following Wu et al. (2023), we use BART (Lewis et al., 2020) as our backbone, We

---

25
2,5
2.5
2.4
G-Eval
2.3
2.3
2.3
2.2
21
SimBRC +IPS
2,1
-A-SSMDDC-IPP
SimDRC+IT
-Q- SimCTC- contrastive search
-O- SimCC TG 4contrastive search
2.1
econtraative search
2
6
7
8
0.7
0.8
The value of
The value of
The value of a
(a) Ablation the first steps
(b) Ablation top-k sampling
(c) Ablation on the hyperparameler
Figure I: Ablation study on the Daily Dialoo dataset.
MAUVE (Pillutla et al., 2021), Distinet2/4 (Li et al., 2016a), and G-Eval, an automatic evaluation metric based on GPT3.5 (Liu et al., 2023).
Although IPS can be directly used with different models and achieve good performance, the models trained with SimDRC are the best testbed for IPS, We can see that SimDRC+IPS can mostly achieve the best performance across the board on both au- tomatic and human evaluation. This is reasonable because the training process of SimDRC is greatly consistent with the search criterion of IPS, and they both push away the inter-utteranee features and pull close the intra-utterance features.
We also conduct a human evaluation with the help of recruited proficient English/Chiness speak- We randomly sample 100 dialogue exam- ples from DailyDialog and LCCC test sets. For each dialogue context, we generate responses us- ing the aforementioned backbone models (BART, BART+SinCTG, SiRTRR) with six differ- ent inference strategies. Five annotators are hired independently to measure these samples. Annota- tors are instructed to give score ranging from 1 to 5 over the following aspects, including fluency, in- formativeness, coherence, and semantic coverage!.
A Ablation Study Figure 1 shows the ablation stud- ies on different components of the method, includ- ing the first steps, the sampling strategy for the first n-step decoding, and the weight a. As shown in Figure ((a), our method consistently outper- forms the contrastive search no matter the number of first steps. We find some performance drops with the increase the first-stage sampling steps. We think this is because more generic tokens are selected by traditional search methods, thus weak- ening the proximity and isotropy of the response. For strategies in the first steps, attempt beam search, top-k sampling, and nucleus sampling. We finally select top-k sampling as our first stage's strategy owing its better performance in the com- parisons. Figure 1(b) shows the results of different values adopted in top-k sampling. We can see that our method exceeds the baseline by a large margin when to > 5. The effect of weight is also studied, as shown in Figure 1(c). Our method con- sistently outperforms the baseline with the different weights, suggesting the robustness of our method.
Results and Discussion Table lists the auto- matic evaluation results of the different methods with different decoding strategies. Similar results be also found in human evaluation, shown in Table 2. We can see that the models, collaborating with IPS, can produce more semantically consis- tent(high BERTScores and MAUVE scores) and human-like (high G-Eval scores) responses. Al- though contrastive search can generate more novel and diverse tokens (high Distinct scores), it usually suffers from the problem of prediction deviation, i.e., the predicted token being weakly related to the main idea of the response. This also in line with the worse performance of contrastive search on other metrics, such as BERTScore. and G-Eval, indicating that the diverse responses produced by contrastive search are not accurate and human-like enough. Different from contrastive search, IPS tries to concentrate on the core meaning of the response and express it clearly, thus slightly lower Distinct score is acceptable and expected. Note that IPS still has better distinct scores than other traditional de- coding methods since encourages discrimination and isotropy among utterances.
Hyperparameter Analysis To explore the ef- fects of isotropy and proximity, our experiments, we introduced a hyperparameter 3 to balance the p value and v_value as:
(1 3) x value 3 i value
(6)
We tried the effects ranging from 0.2 to 0.8. We surprisingly found that the balance of proximal
Details of human evaluation are in Appendix A.I.

---

value and isotropy value leads to the best perfor- mance. saying equals 0.5. This finding is a bit different from the observations in SimDRC(Wu et al., 2023) which suggests that larger isotropy loss weight is needed to balance the two proper- ties in the training stage. We think this is because our method is a decoding strategy, rather than the training optimization process. The sparse isotropy values would not cause the model bias in the decod- ing stage. So, the harmonious balance of proximity and isotropy can be simply achieved by giving a moderate value of B.
with friendly instructions to support its correct use. However, we still need to emphasize that text gen- eration is not as controllable as we think. It still would generate some novel or unexpected words occasionally. We may take actions to decrease gen- cration diversity to alleviate this problem.
References
Sourya Basu, Govardana Sachitanandam Ramachan- dran, Nitish Shirish Keskar, and Lav R. Varshney. 2021. Mirostat: A neural text decoding algorithm that directly controls perplexity. In International Conference on Learning Representations.
4 Conclusion
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems. ACM SIGKDD Explorations Newsletter, 19(2):25-35.
In this work, we present a fine-grained conversa- tional decoding strategy, namely isotropic and prox- imal search (IPS) to encourage the generation of isotropic and conversational tokens. Superior to existing decoding methods, IPS decouples the pre- vious tokens and the context. Experiments show that our method achieves impressive performance on both automatic and human evaluation.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing FEMNLP-IJJCIL. pages 55-65, Hong Kong, China, Association for Computational Linguistics.
A Ackonwledgements
This work was supported in part by the ImnoHK ini- tiative, the Government of the HKSAR, Laboratory for Al-Powered Financial Technologies.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Limitations
During the experiments, we found that for a single piece of data in the DailyDialog test set, traditional text decoding methods such as beam search, top-k sampling and beam search take less than 1 second, the contrastive search takes about 5.07s, and the de- coding time required by our proposed IPS is about 2. 16s. Although our approach takes longer than the traditional text decoding method, calculation speed is obviously faster than contrastive search. How to further improve the computing speed is still the direction we need to work on.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learn ing to write with cooperative discriminators. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers). pages 1638-1649, Melbourne, Australia. As- sociation for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 7871-7880, Online. Association for Computa- tional Linguisties.
Ethics Statement
In this work, we use publicly released datasets to auxiliary our dialogue response generation. Gener- ally, these previous works have considered ethical issues when creating the datasets. We have man- ually checked some samples for the datasets we used in this work, and do not find any obvious ethical concerns, such as violent or offensive con- tent. We will also release the source decoding code
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.
ob-

---

pages 110-119, San Diego, California. Association for Computational Linguistics.
Yixuan Su. Yan Wang, Deng Cai, Simon Baker, Anna Korhonen, and Nigel Collier. 2021. Prototype-to to- style: Dialogue generation with style-aware editing on retrieval memory. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2152- 2161.
Jiwei Li, Will Monroe, Alan Ritter. Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep rein- forcement learning for dialogue generation. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing, pages 1192- 1202, Austin, Texas, Association for Computational Linguistics.
Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. A large-scale chinese short-text conversation dataset. In Natural Language Processing and Chinese Com- puting: 9th CCF International Conference, NLPCC 2020. Zhengzhou, China, October 1418, 2020, Pro- ceedings, Part I, page 91 103, Berlin, Heidelberg. Springer-Verlee Verlag..
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference Nat- ural Language Processing (Volume I: Long Papers), pages 186-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253 226-2 Copenhagen, Denmark. Association for Computational Linguistics.
Chin- Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74 81, Barcelona, Spain. Association for Computational Linguisties.
Han Wu, Haochen Tan, Mingjie Zhan, Gangming Zhao, Shaoqing Lu, Ding Liang, and Linqi Song. 2023. Learning locality and isotropy in dialogue model- ing. In The Eleventh International Conference on Learning Representations.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human align- ment.
Tianyi Zhang*, Varsha Kishore Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 111-118, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
A
Appendix
A.1
Human Evaluation Instructions
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the gap he- tween neural text and human text using divergence frontiers. In Advances in Neural Information Pro- cessing Systems.
Please rate the quality of the generated response based on the given dialogue context and the target response over the following aspects: (1) Fluency; (2) Informativeness; (3) Coherence; (4) Semantic Coverage. We provide some instructions for your rating.
Verena Rieser, Oliver Lemon, and Simon Keizer. 2014, Natural language generation as incremental planning under uncertainty: Adaptive information presentation for statistical dialogue systems. IEFE/ACM Trans. Audio, Speech and Lang. Proc., 22(5)9979994.
A.1.1 Fluency
This measures whether the generated text has formatting problems, capitalization errors, or ob- viously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read. The definitions of different scores are:
Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 583- 593, Edinburgh, Scotland, UK. Association for Com- putational I. Linguisties.
5:
The text is fluent, grammatically correct, and has no errors. It is easy to read.
Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. Transac- tions on Machine Learning Research.
The text is grammatically correct but has a few spelling or capitalization errors, which does not affect your understanding.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Ling- peng Kong, and Nigel Collier. 2022. A contrastive framework for neural text generation. Advances in Neural Information Processing Systems.
The text has minor errors in both grammar and spelling. The errors slightly affect your understanding.

---

2: The text has major errors in both grammar and spelling. The errors make the text hard to read.
A.1.4 Semantic Coverage
This measures how many semantic content units from the target response are covered by the gener- ated text. The definitions of different scores are:
1: The text does not make sense and it is unreadable.
5: All semantic content units of the target text can be found in the generated text. They are semantically consistent.
A.1.2 Informativeness
This measures whether the generated text has di- verse, informative, novel, logically related con- tent. The definitions of different scores are:
Most of the content units of the target text can found from the generated text while a few missing units do not affect the overall coverage.
5: The text contains very diverse, informative, and novel content. is enjoyable read the text.
3: Some semantic content units can be found in the generated but also miss some im- portant units.
4: The text contains many informative and novel contents. (Choose this score when you hesitate between and 5.)
2: Most of the semantic content units are not covered. Only few insignificant units can be found in the generated text.
3:
The text contains new information but also contains few repetitions of the context.
2: The text only contains a few informative and new terms. (Choose this score when you hesitate between and 3.)
The text does not have any overlapping semantic content units with the target text.
We recruit five human workers to annotate 3,600 samples. To make sure the workers are fairly paid, pay 0.1 dollars for each sample. Therefore. the total amount spent participant compensation is 360 dollars. The annotators take 24 hours finish the task, suggesting the hourly wage for each worker is dollars.
1: The text is dull, repetitive, and has new information. All contents from the dia- logue context.
A.1.3 Coherence
This measures whether the generated text is seman- tically and factually consistent with the dialogue context. The definitions of different scores are:
A.2 More Details of the Task
5: The text is semantically, factually, and topi- cally consistent with the dialogue context. All contents of the text are related the source text or can be inferred from the source.
A.2.1 Evaluation of G-EVAL Score
The API we used to G-EVAI gpt-3.5-turbo, and the following is the prompt (Liu et al., 2023):
You will be given a conversation between two in- dividuals. You will then be given one potential response for the next turn in the conversation. Your task is to give final score for utterance. Please make you read and understand these instruc- tions carefully.
4: The text is very related to the context but has minor inconsistencies or contradictions that do not affect overall relevance.
3: The text is related to the context but has some obvious inconsistencies and contradic- tions.
The evaluation aspects are:
2: The text is slightly consistent with the con- text. Many inconsistencies and contradictions in the context can found.
1. Engagingness: Is the response dull or interest- ing?
2. Naturalness: This measures whether the gen- erated text has no formatting problems, capi- talization errors, or obviously ungrammatical sentences to read.
1: The is totally inconsistent with the con- text. semantically or factually contradicted the context.

---

3. Informativenes:: This measures whether the generated lext has diverse, informative, novel, or logically related content.
search and IPS, redundancy is minimized, result- ing in more concise generated text compared previous methods. Considering the nature of the conversation, our IPS strategy expects proximity and does not enlarge the token distance the same utterance, thus responses of IPS are slightly longer than that of contrastive search.
4. Coherence: This measures whether the gener- ated text is semantically and factually consis- tent with the dialogue context.
The evaluation steps are:
A.4 Qualitative Analysis
1. Read the conversation, the corresponding la- bel, and the response carefully.
A.4.1 Instances Illustration
Some examples are presented to illustrate the effect of our IPS search.
2. Considering the above evaluation aspects, re- turn comprehensive final score ranging from to 5 for each conversation.
In summation, according to Table and Table 7, some qualitative observations are as follows:
3. Please only return 1 overall score, without any extra text descriptions. The return format should be like Score:l.
Replies generated by IPS are natural and accurate.
Now please read the following conversation, and return the score.
IPS tends to generate relatively concise re- sponses.
A.2.2 More Experimental Results
With more complex previous contexts, ob- served that IPS does not prioritize shortening the length of response. IPS can generate re- sponses that are more in line with the situation based on the characteristics of the conversa- tion.
Table 2 lists the results of human evaluation.
A.3
Surface-level Analysis
A.3.1 Score Distribution According the Length of the Previous Context
Table 3 and Table 4 illustrate the relations between the context length and the human evaluation met- ries while using the IPS (the above one) and beam search (the below onc) decoding strategies. Observ- ing the table, when the context length is particularly short (<10), we speculate that the context may con- sist of simple greetings or introductions, resulting in lower difficulty of generation and thus higher scores. When the context length varies in the range of approximately 10 40, due differences in the complexity of context content and semantics, the scores exhibit fluctuating trend. As the length continues to increase, the information provided by the previous context becomes richer, leading to im- proved effectiveness of both decoding methods. We also note that when faced with exceptionally long contexts, the generation quality of IPS is superior the baselines.
A.5 Cosine Similarity Heatmap
To ensure utterances generated by our IPS are isotropic and proximal, and observe the represen- tations produced by different decoding methods, we showcase the cosine similarity matrix of token representations correspondingly.
The larger color difference between different sentences represents greater isotropy, indicating discrimination among utterances; while the darker the color within the same sentence, the greater the proximity, conveying a more concentrated thought. Choosing SimDRC as the backbone model. co- sine similarity heatmaps of different inference methods are shown as follows. Tokens generated by IPS exhibit brighter colors in the heatmap, indi- cating increased proximity within the same sen- tence, while tokens from IPS showease darker colors for different sentences, signifying greater isotropy. Contrastingly, traditional methods like beam search showed anisotropy(i.e. features oc- cupy a narrow cone in the vector space, thus lead- ing to the problem of degeneration.) in the figures.
A.3.2 Utterance Length Analysis
Table 5 shows that both IPS and contrastive search tend to produce shorter sentences than traditional methods. We explain the main text that by in- corporating isotropy, achieved through contrastive

---

Costne Similaity Contrastive Search
Costne Simiarity Marric Search
66
1h
1.0
Doaine Sirddanly Mubrie
00
Figure An image a cosine similarity heatmap

---

DailyDialos
LCCC
Model
Strategy
Fluency
Info.
Coherence
Fluency
Info.
Coherence
SC
greedy
4.636
3.302
3.362
2.386
4.626
2.810
3.414
1.996
heam
4.584
3.362
3,508
2.390
4,452
2,950
3.278
2.054
top-k 4.634
3.416
3.554
2.432
4.554
2.954
3.384
2.072
BART
nucleus
4.666
3,478
3.578
2.420
4,602
3.042
3.434
3.358
contrastive
4.678
3,406
3.476
2,416
4,560
2.888
3.358
2.118
IPS
4.710
3.562
3.768
2.566
4.718
3.152
3.622
2.184
greedy 4.652
3.288
3.362
2.394
4,622
2.884
3.580
2.026
beam
4.696
3,404
3.446
2.390
4.602
2.918
3.224
2,040
SimCTG
top-k
4.718
3.398
3,406
2.414
4.554
2.970
3,464
2.040
(p = 0.5)
nucleus
4.734
3.372
3.348
2.386
4,582
2.938
3,548
2.068
contrastive
4.712
3,304
3.31
2.332
4,582
2.882
3,456
2.076
IPS
4.758
3.586
3.578
2.584
4.708
3.084
3.688
2.176
greedy
4.774
3,632
3.484
2.684
4,634
2.920
3.390
1.990
beam
4.820
3,580
3.404
2.586
4.620
2.920
3.632
2,034
SimDRC (6 0.7,
top-k
4.854
3.614
3.396
2.618
4.590
2.950
3.572
2.048
0.3)
nucleus
4.864
3.622
3,450
2.600
4,588
2.930
3.628
2,088
contrastive
4.872
3.692
3.798
2.694
4.594
2.890
1,984
IPS
4.892
3.768
3.942
2.826
4.712
3.120
3.734
2.332
Table Results human evaluation on DailyDialog and LCCC datasets, where SC means the semantic coverage, info. means informativeness.
Length
Fluency
Informativeness
Coherence
Semantic Coverage
num
[0,10)
4.94
4.56
4.67
3.06
9
[10,20)
4.93
3.5
3.62
2.77
13
(20,30)
4.73
4
3.09
[30,40)
4.75
3.56
4
2.67
12
[40,50)
4.85
3.67
3.56
2.28
9
(50,75)
4.95
3.52
3.61
2.45
17
(75,100)
4.8
3.52
3.79
2.84
15
over 100
4.93
3.79
4.21
2.96
14
Table 3: Relations between the context length and the human evaluation metrics while using the IPS.
A.6 Examples of Generated Texts
For non-native Chinese speakers, translations of Table 9 are presented in Table 10. The quality of the LCCC dataset still requires optimization, as it contains numerous colloquial and slang expres- sions. We are not professional translators, and in our attempts. we noticed that the translated mean- ings sometimes diverged from the original Chinese. We apologize for the inconvenience.

---

Length
Fluency
Infomrativeness
Coherence
Semantic Coverage
(0,10)
4.89
4.44
4.33
2.78
(10,20)
4.77
3.31
3.15
2.46
13
(20,30)
4.55
3.86
3.41
3
11
(30,40)
4.88
3.29
3.13
2.42
12
[40,50)
4.88
3.11
2.89
1.83
[50,75)
4.82
3.43
3.17
2.43
17
(75,
4.78
3,48
3.45
2.5
15
over 100
4.93
3.5
3.64
2.43
14
Table 4: Relations between the context length and the human evaluation metries while using the beam search.
Sentence_ Len
dd-dre
dd-ctg
dd-bart
lece-dre
lcce-ctg
Icce-bart
Greedy
11.33
11.39
10.98
14.05
13.76
14.60
Beam
12.56
12.65
12.24
11.89
11.69
11.88
Top_k k
13.10
13.21
12.71
12.39
12.12
12.40
Top_p
12.76
13.60
13.17
12.55
12.33
12.56
Simetg
9.02
9.05
8.72
9.50
9.17
8.88
IPS
9.59
10.31
9.13
9.95
9.46
9.14
Table 5: Sentence-level Analysis
A: have cell phone my Now it's probably floor the pussenger side.
Why don't you get for me, and then can call the police?
context
B: Alright
A: It doesn't "tesn't work. It looks like it's broken. need to hospital. You should drive me there.
B: yeah? It's better if make a police report first. Then you go the hospital.
label
DDann I'm injured here. We could wail all day the police."
greedy
"Alright, I'll go it.
hean/hup-p/oope
like making police report. to get hospital."
contrastive
"Aligght. I'll call the police."
IPS
"Oh, don't think I afford wait for police. need the hospital,"
Table 6: Qualitative Instance A
A: Good aftemoon!! you call plumber?
context
B: Yes, yes did. Please come in! I' m so glad you came! This old house is falling apart!
Come on into the bathroom, See, here, there' water leaking everywhere!
greedy
"Well. leaking because of the leaking faucct in the shower."
beam/top-p
"Well, it" leaking because of the leaking faucet in the shower."
top-k
"I' sorry, madam, but I don't think that water is leaking. I'm going to have check out."
contrastive
"Well, leaking all over!
know what the plumber can about it!"
IPS
"Well, it' leaking hecause of the leaking fancet in the shower. guess the water is backed up."
Table Qualitative Instance

---

A: didn't receive an unemployment check this week and need know why.
B: Did you mail us your lust Continued Claim Form?
content
A: am afraid that I forgot to send in.
B: cannot pay you unless you turn that form in every two wecks.
A: it too late to mail the Continued Claim Form in now?
greedy
will mail it in right now.
heam search
It loo late to mail
You will receive your check this week.
top-k sampling
will mail right now you can'
nucleus search
It' too late to mail it in now. You will receive your check this week.
contrastive search
Yes, you can do now.
IPS
Yos,
A: Hello, this Mike, Kara,
B: Mike! Gond to hear from you. How are you?
A: Everything is fine, and how you?
content
B: Things are going well with me.
A: Kara, had fun the other night at the movies and was wondering if you would like to go out again this Friday,
greedy
was thinking of it, but am not foeling too well. have to go the hospital for couple of days. to go out with you
beam search
That sounds great. would
top-k sampling
would love to, hut have to work late on Friday. That sounds great. I would love to go out with you.
nucleus search
contrastive search
Mike, thinking of it.
That sounds great. would love to.
content
A: What do you think I should do, doctor?
greedy
think you should have good rest for three days. think you'd better have a good rest for three days think you should go to see the head of the family. think you'dl 'd better have good rest for three days. Well, it's hard to say. You look little pale. don't don't you?
beam search
top-k sampling
nucleus search
contrustive search
IPS
Well, you should have a good rest for three days. Then you'll foel better.
Table 8: Results of generation texts of DailyDialog.
A: A7
B: म
content
A: A (o BU
A
PARWER
greedy
beam search
top-k sampling
t 12
nucleus search
contrastive search
3 t T
IPS
2 T 1-
A: ME
content
B: 32 A. WHITT
greedy
bear search
RiL E 1E
top-k sampling
nucleus search
2 A
contrastive search
IPS
A
content
A: HRi T.
greedy
beam search
# (
top-k sampling
# T. BE
nucleus search
W FA T
contrastive scarch
+ R T. T
IPS
Table 9: Results of generation texts of LCCC.

---

You are good at beverage
like your comment.
content
A: If made for drink, it would definitely be delicious, exquisite delicacy among delicacies.
home. but right now, I just don't feel dealing with them. also like drink, but won't make it for you tn drink.
There
greedy
beam search
I don't want cook either, enjoy eating tomatoes.
top-k sampling
also enjoy eating delicious
nucleus search
tt want make anymore. I just like to it
contrastive scarch
don't want to cook cither. simply enjoy cating this kind of thing.
IPS
don't want do it cither. but I'd love
Shanghai not cold anymore?
content
B: These two days have been bit abnormal... estimate will get colder in the next two days
greedy
We here the same
beam search
We have
weather here
top-k sampling
We have the same weather here
nucleus search
It's been a bit cold these two days, but it's not particularly cold either
contrastive search
It's been cold these days. It's expected to start cold lomormow
The weather been good these two days, it's bit cold today
content
A: I'm hungry
greedy
I'm also hungry
beam search
want to eat too, but don't ' have time
top-k sampling
I'm hungry, I'm going to cat now Why did you go bed so carly I'm also hungry
nucleus search
contrastive search
I'm also hongry, have you caten?
Table 10: Translation of generation texts of LCCC.


Reviews:
[{"review_id": "8sof3Zrtvg", "reviewer": "Reviewer_ybjD", "summary": "", "questions": "- Question A: Did the authors find any surface-level pattern in the utterances generated by IPS (utterance length, unique tokens, etc.)? \n- Question B: Do the authors have any intuition about why human annotators disagree way more when evaluating the informativeness of generated utterances in the DailyDialog task compared to LCCC (0.56 vs. 0.78)? All the other metrics instead (fluency, coherence, semantic coverage) have similar values in the two settings.   \n- Question C: Did the authors inspect more examples compared to the ones reported in Table 3? Did they get any qualitative insight into the main advantages of IPS-generated utterances?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback.  We appreciate your thoughtful comments and suggestions, which help us to improve the quality of our work.\n\n---\n\n**Concern:** significance tests and statement issue\n\n**Response:** As our experimental settings, we ran all experiments with three different random seeds and reported the average scores. We also conducted the significance tests on the metrics of BertScore, MAUVE, and G-Eval. The results show that our contribution is significant with the sign test p-value < 0.05. We will also carefully consider your comments regarding the issue of 'outperform' and will revise the writing in the revision to avoid overstatement.\n\n---\n\n**Concern:**  surface-level pattern in the utterances\n\n**Response:** We appreciate your suggestions related to statistics and analysis of surface-level indicators. We conducted experiments  and the results are shown in the following table:\n\n| Sentence_Len | dd-drc | dd-ctg | dd-bart | lccc-drc | lccc-ctg | lccc-bart  |\n|:------------:|:------:|:------:|:-------:|:--------:|:--------:|:----------:|\n| Greedy       | 11.33 | 11.39  | 10.98   | 14.05 | 13.76    | 14.60      |\n| Beam         | 12.56  | 12.65| 12.24   | 11.89    | 11.69| 11.88      |\n| Top_k        | 13.10| 13.21  | 12.71   | 12.39| 12.12    | 12.40      |\n| Top_p        | 12.76| 13.60  | 13.17   | 12.55| 12.33    | 12.56      |\n| Simctg          | 9.02   | 9.05| 8.72    | 9.50     | 9.17| 8.88       |\n| IPS          | 9.59   | 10.31| 9.13    | 9.95     | 9.46| 9.14       |\n\nThe results show that both IPS and contrastive search tend to produce shorter sentences than traditional methods. We explain in the main text that by incorporating isotropy, achieved through contrastive search and IPS, redundancy is minimized, resulting in more concise generated text compared to previous methods. Considering the nature of the conversation, our IPS strategy expects proximity and doesn’t enlarge the token distance in the same utterance, thus responses of IPS are slightly longer than that of contrastive search.\n\n---\n\n**Concern:** Qualitative advantages of IPS over other decoding strategies\n\n**Response:** Some qualitative observations are as follows:\n\n1. Replies generated by IPS are more natural and accurate.\n2. IPS tends to generate relatively concise responses.\n3. With more complex previous contexts, we observed that IPS does not prioritize shortening the length of response. IPS can generate responses that are more in line with the situation based on the characteristics of the conversation.\n\nWe will attach corresponding instances and the cosine similarity heatmap of responses generated by different decoding methods to show the effectiveness of IPS in our final version,  offering readers a better understanding of isotropy and proximity.\n\n---\n\n**Concern:** The agreement of informativeness on two datasets\n\n**Response:** We sincerely appreciate your feedback. As indicated in human evaluation instructions, *informativeness measures whether the generated text has diverse, informative, novel, or logically related content*. So, it is essentially a **more subjective** evaluation metric than fluency, coherence, and semantic coverage. Additionally, the annotators we recruited for human evaluation are all native Chinese speakers, thus causing a higher disagreement on the Dailydialog dataset which is in English.\n\n---\n\n**Concern:** Title of the paper\n\n**Response:** Thank you for your suggestion regarding the article title. We will change it to \"Fine-grained Conversational Decoding via Proximal and Isotropic search\" in the revision.\n\n---\n\nThank you again for your valuable suggestions to help us improve the quality of the work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. It presents an original conversational decoding strategy and the authors evaluate its effectiveness against several other decoding strategies. The experimental setup is clear and well-described. The authors also conducted a human evaluation on a subset of the generated utterances and perform interesting ablation studies. Overall, the results seem to suggest that IPS outperforms other decoding strategies in most of the settings and the evaluation metrics employed.", "reasons_to_reject": "Although the paper has the potential to represent an original and significant contribution to the field, a few issues deserve a closer look. The differences reported in Tables 1 and 2 are sometimes minor, and statistical significance tests are needed to validate the claim that IPS outperforms other decoding strategies. In case it turns out that some differences are not significant, IPS still represents a novel and original contribution; however, it is important to know which differences are significant to adjust the overall claim of the paper. \nIn the conclusions, the authors mention: “Experiments show that our method achieves impressive performance on both automatic and human evaluation”, and in the abstract they say: “Experiments show that our approach significantly outperforms existing decoding strategies”. These are clearly too strong claims that have to be adjusted depending on the results of significance tests.\nMore qualitative examples and surface-level statistics (utterance length, vocabulary coverage, etc.) would be helpful to assess the effectiveness of IPS. The examples reported in Table 3 do not show a clear qualitative advantage of IPS over other decoding strategies, while Table 1 and Table 2 illustrate that IPS (almost) always outperforms other methods against both automatic and human-based metrics. The examples reported in Table 3 suggest that it is advisable to have a closer look at possible limitations in the evaluation metrics used and/or biases during the human annotation procedure.", "questions_to_authors": "- Question A: Did the authors find any surface-level pattern in the utterances generated by IPS (utterance length, unique tokens, etc.)? \n- Question B: Do the authors have any intuition about why human annotators disagree way more when evaluating the informativeness of generated utterances in the DailyDialog task compared to LCCC (0.56 vs. 0.78)? All the other metrics instead (fluency, coherence, semantic coverage) have similar values in the two settings.   \n- Question C: Did the authors inspect more examples compared to the ones reported in Table 3? Did they get any qualitative insight into the main advantages of IPS-generated utterances?", "paper_topic_and_main_contributions": "The paper presents a conversational decoding strategy named isotropic and proximal search (IPS). As the name itself suggests, the decoding strategy is based on the concepts of locality and isotropy, two key features that are shown to be essential for generating high-quality dialogue responses. IPS has a parameter alpha that controls for the weight assigned to these two components; when alpha=1, the IPS behaves as greedy decoding. The authors assess the performance of IPS on two datasets (English and Chinese) and different variants of BART and decoding strategies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The title sounds a bit too vague, maybe it is worth mentioning the concepts of isotropic and proximal search there."}, "extra_text": {}}, {"review_id": "J6m2HNshFt", "reviewer": "Reviewer_nWpT", "summary": "", "questions": "- Question A: Do the authors analyze the score distribution according to the length of previous context? In Table 3, I think the generated samples of third content looks quite similar, thus I wonder if the proposed decoding strategy works better on long previous context.\n- Question B: Could the authors write English-translated version together in Table 4?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback. We appreciate your thoughtful comments and suggestions, which help us to improve the quality of our work.\n\n---\n\n**Concern:** The ablation study of isotropy and proximity\n\n**Response:** We sincerely appreciate your suggestions about the ablation study. In our experiments, we actually introduced a hyperparameter to balance the p_value and i_ value as $(1-\\beta) \\times p\\\\_value - \\beta \\times i\\\\_value$. We explored the effects of $\\beta$ ranging from 0.2 to 0.8. We surprisingly found that the balance of proximal value and isotropy value leads to the best performance, saying $\\beta$ equals 0.5. This finding is a bit different from the observations in SimDRC which suggests that larger isotropy loss weight is needed to balance the two properties in the training stage. We think this is because our method is a decoding strategy, rather than the training optimization process. The sparse isotropy values would not cause the model bias in the decoding stage. So, the harmonious balance of proximity and isotropy can be simply achieved by giving a moderate value of $\\beta$. We will add the discussions about how to balance the values of proximity and isotropy in the revision.\n\n---\n\n**Concern:** Score distribution according to the length of the previous context\n\n**Response:** Regarding your concern about score distribution according to the length of the previous context, Our statistics are as follows：\n\n| Length   | Fluency | Informativeness | Coherence | Semantic Coverage | num  |\n|----------|---------|-----------------|-----------|-------------------|------|\n| [0,10)   | 4.94    | 4.56            | 4.67      | 3.06              | 9    |\n| [10,20)  | 4.93    | 3.5             | 3.62      | 2.77              | 13   |\n| [20,30)  | 4.73    | 4               | 4         | 3.09              | 11   |\n| [30,40)  | 4.75    | 3.56            | 4         | 2.67              | 12   |\n| [40,50)  | 4.85    | 3.67            | 3.56      | 2.28              | 9    |\n| [50,75)  | 4.95    | 3.52            | 3.61      | 2.45              | 17   |\n| [75,100) | 4.8     | 3.52            | 3.79      | 2.84              | 15   |\n| over 100 | 4.93    | 3.79            | 4.21      | 2.96              | 14   |\n\n| Length   | Fluency | Infomrativeness | Coherence | Semantic Coverage | num  |\n|----------|---------|-----------------|-----------|-------------------|------|\n| [0,10)   | 4.89    | 4.44            | 4.33      | 2.78              | 9    |\n| [10,20)  | 4.77    | 3.31            | 3.15      | 2.46              | 13   |\n| [20,30)  | 4.55    | 3.86            | 3.41      | 3                 | 11   |\n| [30,40)  | 4.88    | 3.29            | 3.13      | 2.42              | 12   |\n| [40,50)  | 4.88    | 3.11            | 2.89      | 1.83              | 9    |\n| [50,75)  | 4.82    | 3.43            | 3.17      | 2.43              | 17   |\n| [75,100) | 4.78    | 3.48            | 3.45      | 2.5               | 15   |\n| over 100    | 4.93    | 3.5             | 3.64      | 2.43              | 14   |\n\nThese two tables illustrate the relations between the context length and the human evaluation metrics while using the IPS (the above one) and beam search (the below one) decoding strategies. Observing the table, when the context length is particularly short (<10), we speculate that the context may consist of simple greetings or introductions, resulting in lower difficulty of generation and thus higher scores. When the context length varies in the range of approximately 10 to 40, due to differences in the complexity of context content and semantics, the scores exhibit a fluctuating trend. As the length continues to increase, the information provided by the previous context becomes richer, leading to improved effectiveness of both decoding methods. We also note that when faced with exceptionally long contexts, the generation quality of IPS is superior to the baselines.\n\n---\n\n**Concern:** problems in presentation\n\n**Response:** Thank you for pointing out the typo and the missing of English translation in Table 4. We will revise the problems in the revision.\n\n---\n\nThank you again for your valuable suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work nicely borrows the locality and isotropy concept and adopts them into decoding process. It is sufficiently intuitive and detailed experimental settings and ablations are evidential. Furthermore, although the IPS is slower than traditional methods (e.g., beam search, top-k sampling), as the authors mentioned in Limitation section, it is still faster than contrastive search. This work would enlighten future possibilities of conversational modeling.", "reasons_to_reject": "Ablation studies exploring the effect of respective of proximal and isotropic values need to be conducted. Others are reasonable to understand, so please refer to the questions in the next section.", "questions_to_authors": "- Question A: Do the authors analyze the score distribution according to the length of previous context? In Table 3, I think the generated samples of third content looks quite similar, thus I wonder if the proposed decoding strategy works better on long previous context.\n- Question B: Could the authors write English-translated version together in Table 4?", "paper_topic_and_main_contributions": "This paper is motivated from the locality and isotropy for modeling dialogue, and proposes isotropic and proximal search (IPS). Specifically, there are additional terms on decoding process, which are proximal value (avg. distance between candidate token and the already generated tokens) and isotripic value (avg. similarity between undergoing response and all utterances). The authors evaluated on various metric including human evaluation and the proposed decoding method shows prominent performance.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "\"Coherenc\" -> \"Coherence\" in Table 2 (DailyDialog column)"}, "extra_text": {}}, {"review_id": "fopGnA3yjT", "reviewer": "Reviewer_T3Fd", "summary": "", "questions": "Question A: How does/can IPS avoid degradation of the generation, if generating repetitive rephrases seems to be favored by $p_\\text{value}$ and not disencouraged by $i_\\text{value}$?\n\nQuestion B: Can you provide more direct/concrete way of showing how IPS generated responses are more proximal and isotropic? Most experiments shown in this paper are too \"end-to-end\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback. We appreciate your thoughtful comments and suggestions, which help us to improve the quality of our work.\n\n---\n\n**Concern:** Settings in the ablation study\n\n**Response:**  Regarding your concern in the ablation study, we establish G-EVAL as a dependable metric due to its strong correlation with human judgments, attributed to its LLM-based nature. Additionally, our method demonstrates superiority in the G-EVAL metric during the ablation study on the alpha hyperparameter as well. We chose MAUVE here since the performance gap on this metric is larger and more intuitive. Given the restriction on sharing image links, we're currently presenting the results in a tabular format. The baseline G-Eval score is 2.17 as shown in Table 1.\n\n|alpha | G-EVAL |\n|:-----:|:------:|\n|0.4| 2.18   |\n|0.5| 2.19   |\n|0.6| 2.46   |\n|0.7| 2.22   |\n|0.8| 2.18   |\n\nWe conducted ablation experiments of different hyperparameters on MAUVE and other metrics as well. All results consistently indicate that our IPS is the superior one.\n\n---\n\n**Concern:** Why are we comparing *SimDRC+IPS* against  *SIMCTG+constrastive search* ?\n\n**Response:**  We appreciate the suggestion about the baseline selection. Note that the contrastive search is designed for SimCTG and IPS is inspired by SimDRC. In this context, we compared *SimDRC+IPS* against *SimCTG+contrastive search* by the intuition that the decoding strategies should work better with the corresponding models.\n\nFor a comprehensive evaluation, we also conducted the comparisons to *SimDRC+nucleus*, *SimDRC+contrastive search* and *SimCTG+IPS*. All comparison results show that *SimDRC+IPS* is the better one.\n\n---\n\n**Concern:** how IPS improves proximity and isotropy & Question B\n\n**Response:** In our research, the proposed decoding strategy doesn't alter the model's feature distribution; instead, it focuses on identifying tokens within the distribution, meeting specific requirements based on proximity and isotropy properties post-fine-tuning. This distinction underscores why our decoding strategy combined with SimDRC yields the best results. To address your question of whether IPS generates more isotropic and proximal tokens compared to other methods, we conducted a comparison by examining the cosine similarity matrices of tokens generated by different decoding strategies with the same context. As we cannot insert images here, we depict the outcomes: tokens generated by IPS exhibit brighter colors in the image, indicating increased proximity within the same sentence, while tokens from IPS showcase darker colors for different sentences, signifying greater isotropy. Contrastingly, traditional methods like beam search showed anisotropy in the images. You can refer to Figure 4 in \"Learning Locality and Isotropy in Dialogue Modeling (SimDRC)\". \n\nWe appreciate your inquiry, and in the final version, we intend to include cosine similarity heatmaps, offering readers a more intuitive experience of IPS.\n\n---\n\n\n**Concern:** The ablation study of isotropy and proximity & Question A\n\n**Response:** We sincerely appreciate your suggestions about the ablation study. In our experiments, we actually introduced a hyperparameter to balance the p_value and i_ value as $(1-\\beta) \\times p\\\\_value - \\beta \\times i\\\\_value$. We explored the effects of $\\beta$ ranging from 0.2 to 0.8. We surprisingly found that the balance of proximal value and isotropy value leads to the best performance, saying $\\beta$ equals 0.5. This finding is a bit different from the observations in SimDRC which suggests that larger isotropy loss weight is needed to balance the two properties in the training stage. We think this is because our method is a decoding strategy, rather than the training optimization process. The sparse isotropy values would not cause the model bias in the decoding stage. So, the harmonious balance of proximity and isotropy can be simply achieved by giving a moderate value of $\\beta$. We will add the discussions about how to balance the values of proximity and isotropy in the revision.\n\n---\n\n**Concern:** amount of annotators\n\n**Response:** Regarding the issue of the number of annotators: Generally, five annotators are enough. For instance, five annotators are hired in both \"A Contrastive Framework for Neural Text Generation (SimCTG)\" and \"Learning Locality and Isotropy in Dialogue Modeling (SimDRC)\" papers.\n\n---\n\nWe sincerely thank you again for your suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a new decoding method for dialogue responses. This can be used in parallel with other methods (e.g. better modeling, better prompts) and could be helpful in building better dialogue systems in the future. The strength of this paper includes:\n\n1. This paper defines a new decoding algorithm that promotes proximity and isotropy, which are found to be important for conversational responses.\n2. Both automatic evaluations (2 datasets with 5 competitive baselines) and human evaluations show that IPS generated responses are better.", "reasons_to_reject": "1. The setup/purpose of the ablation study is confusing. Why is Figure 1a,b comparing G-EVAL but Figure 1c comparing MAUVE? Why are we even looking at G-EVAL/MAUVE as opposed to other metrics? Why are we comparing SimDRC+IPS against *SIMCTG*+constrastive search, but not SimDRC+IPS against SimDRC+contrastive search?\n2. Lack of direct analysis of how IPS improves proximity and isotropy. The automatic metrics used (in both main experiment and ablation), such as \"BERTScore, MAUVE, G-EVAL\" are all very generic. While IPS does show improvement under these metrics, it is unclear if it is due to the utterances being more proximal and isotropic, or other reasons.\n3. Only five annotators are involved in human evaluation, which may be too few as no statistical significance is measured.", "questions_to_authors": "Question A: How does/can IPS avoid degradation of the generation, if generating repetitive rephrases seems to be favored by $p_\\text{value}$ and not disencouraged by $i_\\text{value}$?\n\nQuestion B: Can you provide more direct/concrete way of showing how IPS generated responses are more proximal and isotropic? Most experiments shown in this paper are too \"end-to-end\".", "paper_topic_and_main_contributions": "This paper introduces a new decoding method that is argued to be more suitable for dialogue response generation. The authors based off their method on the finding that a good dialogue feature space should follow the rules of locality and isotropy. To achieve this, the authors designed a decoding objective that optimizes for locality (by maximizing the average cosine similarity between the representation of newly generated word and the previously generated words) and isotropy (by minimizing the cosine similarity between the representation of words generated sofar and that of the past utterances). The paper then used two datasets to compare their method against other decoding methods (e.g. contrastive search), and show that it outperforms others under both automatic evaluations (BERTScore, MAUVE, G-EVAL) and human evaluation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new LLM decoding method called isotropic and proximal search based on locality and isotropy with a parameter to interpolate them. The reviewers appreciate the novelty and performance of the new method as well as the clarity of presentation. They raise a few concerns regarding evaluation, such as statistical significance and ablations, . The authors address all major concerns, conducting new experiments where needed. For these reasons I believe this paper should be accepted.
Decision: Accept-Main