SlowGuess/NAACL_2022_OCR huggingface dataset:
BERTbase	DSTC2	GSIM	Ave.
10-shot	20-shot	10-shot	20-shot
SimCSE-sup*	84.12 || 36.62	86.15 || 36.99	77.22 || 35.03	84.75 || 38.67	59.94
PairSupCon*	84.42 || 36.52	86.22 || 36.87	74.35 || 33.44	82.26 || 37.62	58.96
DSE-dianli* (ours)	83.99 || 36.20	86.74 || 37.02	69.52 || 31.36	79.97 || 36.62	57.68
BERT◇	81.74 || 34.78	86.98 || 37.28	70.67 || 31.24	77.60 || 35.74	57.00
SimCSE-unsup◇	84.41 || 36.62	87.84 || 37.98	75.78 || 34.47	81.73 || 37.64	59.56
TOD-BERT◇	87.12 || 36.83	88.59 || 37.90	85.63 || 38.53	92.15 || 42.04	63.60
DSE-dropout◇ (ours)	83.23 || 36.18	86.65 || 36.95	72.25 || 32.70	81.91 || 37.33	58.62
DSE◇ (ours)	84.58 || 36.02	88.01 || 38.01	79.26 || 35.89	86.73 || 39.51	61.03
DSE2-1 (ours)	84.47 || 36.09	88.86 || 38.41	83.81 || 37.78	88.03 || 40.29	62.22
DSE3-1 (ours)	88.78 || 38.52	89.59 || 38.58	85.27 || 39.10	88.65 || 40.87	63.67
DSE123-1 (ours)	89.48 || 38.60	90.97 || 39.79	87.90 || 40.05	92.48 || 42.22	65.19
answers in the AmazonQA dataset calculated by DSE, SimCSE, and TOD-BERT. With the DSE embedding, each question is placed close to its real answer while far away from other candidates.
Dialogue Action Prediction Table 5 shows that DSE outperforms all baselines except TOD-BERT, which indicates its capability in capturing dialogue-level semantics. To better understand TOD-BERT's superiority over DSE on this task, we further investigate this task and find its data format is special. Concretely, here each input consists of multiple utterances explicitly concatenated by using two special tokens [SYS] and [USR] to indicate the system and user inputs, respectively. For example, ([SYS] hi [USR] how are you? [SYS] I'm good). It follows the same format as the queries<sup>4</sup> used for training TOD-BERT, while DSE uses a single utterance as the query.
Table 5: Results on 10-shot and 20-shot dialogue action prediction fine-tuning on DSTC2 and GSIM. We use "ll" to separate the Micro F1 score and Macro F1 score.  $\clubsuit$ : Supervised models.  $\diamond$ : Unsupervised models.
BERTbase	IC	OOS	u-RS	d-RS	DA
TOD-BERT	41.83	47.25	41.38	12.79	63.60
DSE	60.09	62.43	69.69	24.54	61.03
DSE2-1	56.56	61.55	59.88	19.36	62.22
DSE3-1	57.26	61.19	61.94	22.04	63.67
DSE123-1	59.60	61.59	63.67	22.63	65.19
Table 6: Performance of TOD-BERT, DSE, and its variants on intent classification (IC), out-of-scope detection(OOS), response selection on utterance-level (u-RS) and dialogue-level (d-RS), dialogue action prediction (DA).
4.4 Trade-off in Query Construction
To understand the impact of using multiple utterances as queries, we train three new variants of
DSE. Specifically, we construct positive pairs as:  $(u_{1}$  [SEP]  $u_{2}, u_{3})$ ,  $(u_{2}$  [SEP]  $u_{3}, u_{4})$ , where  $u_{i}$  represents the  $i$ -th utterance in a dialogue. We use the [SEP] token to concatenate two consecutive utterances as query. We refer DSE trained with this data as  $\mathrm{DSE}_{2-1}$  since it uses 2 utterances as the query and 1 utterance as the response. Similarly, we train another variant  $\mathrm{DSE}_{3-1}$ . Lastly, we also combine the positive pairs constructed for training DSE,  $\mathrm{DSE}_{2-1}$ , and  $\mathrm{DSE}_{3-1}$  together to train another variant named  $\mathrm{DSE}_{123-1}$ .
As shown in Table 5, by simply increasing the number of utterances within each query to three, DSE again outperforms TOD-BERT, and the improvement further expands when trained with the combined set, i.e.,  $\mathrm{DSE}_{123-1}$ . Our results demonstrate that using long queries that consist of 5 to 15 utterances as what TOD-BERT does is not necessary even for dialogue action prediction. We further demonstrate this by evaluating DSE and its variants on all the other four tasks in Table 6, where our model outperforms TOD-BERT by a large margin. As it indicates, by using a single utterance as a query, DSE achieves a good balance among different dialogue tasks. In cases where dialogue action prediction is of great importance, augmenting the original training set of DSE with positive pairs constructed by using query consisting of 2 to 3 utterances is good enough to attain better performance while only incurring a slight performance drop on other tasks.
4.5 Potential Limitation
Considering the effectiveness of using consecutive utterances as positive pairs, a natural yet important question is: what are the potential limitations of
4We use query to refer the first utterance in a positive pair and use response to refer the other one
our proposed approach? When using consecutive utterances as positive pairs for contrastive learning, an assumption is that responses to the same query are semantically similar. Vice versa, queries that prompt the same answer are similar. This assumption holds in many scenarios, yet it fails sometimes.
It may fail when answers have different semantic meanings. Take the pairs 2 and 5 in Figure 2 as an example. Through our data construction, we implicitly consider I want some pizza and Korean food, please to be semantically similar since they are both positively paired with What type of food do you like. Although this may be correct in some coarse-grained classification tasks since these two sentences generally represent the same intent (e.g., order food), using them as positive pairs can introduce some noise when considering more fine-grained semantics. This problem is further elaborated when answers are general and ubiquitous, e.g., Thank you. Since these utterances can be used to respond to countless types of dissimilar queries, e.g., I have booked a ticket for you v.s. Happy birthday, we may implicitly increase the similarities among highly dissimilar utterances when training on these samples, which is undesirable.
We verify this on the NLI datasets, where the task is to identify whether one sentence semantically entails or contradicts the anchor sentence. For each anchor sentence, we calculate its cosine similarities with both the true entailment, contradiction sentences in the representation space. We classify the sentence with higher cosine similarity with the anchor as entailment and the other as the contradiction. Despite DSE achieves better classification accuracy (76.62) than BERT (69.40) and TOD-BERT (70.51), it underperforms SimCSE unsup (80.31). Although using dropout to construct positive pairs is not as effective as ours in many dialogue scenarios, this method better avoids introducing fine-grained semantic noise.
Despite the limitations, using consecutive utterances as positive pairs still leads to better dialogue representation than the elaborately labeled NLI datasets, indicating the great value of the information contained in dialogue utterances.
Positive Pair Construction Popular supervised sentence representation learning often takes advantage of the human-annotated natural language
inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2018) for contrastive learning (Gao et al., 2021; Zhang et al., 2021a; Reimers and Gurevych, 2019; Cer et al., 2018). These sentence pairs either entail or contradict each other, making them the great choice for constructing positive and negative training pairs. Unsupervised sentence representation learning often relies on variant data augmentation strategies. Logeswaran and Lee (2018) and Giorgi et al. (2020) propose using sentences and their surrounding context as positive pairs. Other works resort to popular NLP augmentation methods such as word permutation (Wu et al., 2020b) and back-translation (Fang et al., 2020). Recently, Gao et al. (2021) demonstrates the superiority of using Dropout over other data augmentations that directly operate on the discrete texts.
Contrastive Learning Methods Contrastive learning is key to recent advances in learning sentence embeddings. Many contrastive learning approaches utilize memory-based methods, which draw negative samples from a memory bank of embeddings (Hjelm et al., 2018; Bachman et al., 2019; He et al., 2020). On the other hand, Chen et al. (2020) introduces a memory-free contrastive framework, SimCLR, that takes advantage of negative sampling within large mini-batches. Promising results were also reported in the NLP domain. To name a few, Gao et al. (2021) leverages both within batch negatives and the 'contradiction' annotations in NLI; and Zhang et al. (2021a) propose an unsupervised hard-negative sampling strategy.
Dialogue Language Model Learning dialogue-specific language models has attracted a lot of attention. Along this line, Zhang et al. (2019) adapts the pre-trained GPT-2 model (Radford et al., 2019) on Reddit data to perform open-domain dialogue response generation. Bao et al. (2019) evaluates multiple dialogue generation tasks after training on Twitter and Reddit data (Wolf et al., 2019; Peng et al., 2020). For dialogue understanding, Henderson et al. (2019b) propose a response selection approach using a dual-encoder model. They pretrain the response selection model on Reddit and then fine-tune it for different response selection tasks. Following this, Henderson et al. (2019a) introduces a more efficient conversational model that is pre-trained with a response selection target on the Reddit corpus. However, they did not release code or pre-trained models for comparison. Wu
et al. (2020a) combines nine dialogue datasets to obtain a large and high-quality task-oriented dialogue corpus. They introduce the TOD-BERT model by further pre-training BERT on this corpus with both the masked language modeling loss and the contrastive response selection loss.
In this paper, we introduce a simple contrastive learning method DSE that learns dialogue representations by leveraging consecutive utterances in dialogues as positive pairs. We conduct extensive experiments on five dialogue tasks to show that the proposed method greatly outperforms other state-of-the-art dialogue representation models and universal sentence representation methods. We provide ablation study and analysis on our proposed data construction from different perspectives, investigate the trade-off between different data construction variants, and discuss the potential limitation to motivate further exploration in representation learning on unlabeled dialogues. We believe DSE can serve as a drop-in replacement of the dialogue representation model (e.g., the text encoder) for a wide range of dialogue systems.
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. arXiv preprint arXiv:1704.00057.
Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910.
Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2019. Plato: Pre-trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. arXiv preprint arXiv:1909.05358.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Mihail Eric and Christopher D Manning. 2017. Key-value retrieval networks for task-oriented dialogue. arXiv preprint arXiv:1705.05414.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.
John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. 2020. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Matthew Henderson, Inigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulic. 2019a. Convert: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL), pages 263-272.
Matthew Henderson, Ivan Vulic, Daniela Gerz, Inigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019b. Training neural response selection for task-oriented dialogue systems. arXiv preprint arXiv:1906.01543.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network. arXiv preprint arXiv:2006.05702.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learning to classify intents and slot labels given a handful of examples. arXiv preprint arXiv:2004.10793.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.02027.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. 2020. I-mix: A domain-agnostic strategy for contrastive representation learning. arXiv preprint arXiv:2010.08887.
Sungjin Lee, Hannes Schulz, Adam Atkinson, Jianfeng Gao, Kaheer Suleman, Layla El Asri, Mahmoud Adada, Minlie Huang, Shikhar Sharma, Wendy Tay, and Xiujun Li. 2019. Multi-domain task-completion dialog challenge. In Dialog System Technology Challenges 8.
Xiujun Li, Sarah Panda, JJ (Jingjing) Liu, and Jianfeng Gao. 2018. Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems. In SLT 2018.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural language understanding services for building conversational agents.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893.
Ryan Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training end-to-end dialogue systems with the ubuntu dialogue corpus. Dialogue & Discourse, 8(1):31-65.
Nikola Mrksic, Diarmuid O Seaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2016. Neural belief tracker: Data-driven dialogue state tracking. arXiv preprint arXiv:1606.03777.
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task-oriented dialog. arXiv preprint arXiv:2002.12328.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8689-8696.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Pararth Shah, Dilek Hakkani-Tur, Bing Liu, and Gokhan Tur. 2018. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 41-51.
Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638.
Mengting Wan and Julian McAuley. 2016. Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems. In 2016 IEEE 16th international conference on data mining (ICDM), pages 489-498. IEEE.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2018. Dialogue natural language inference. arXiv preprint arXiv:1811.00671.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
Chien-Sheng Wu, Steven Hoi, Richard Socher, and Caiming Xiong. 2020a. Tod-bert: pre-trained natural language understanding for task-oriented dialogue. arXiv preprint arXiv:2004.06871.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020b. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, and Qi Li. 2021. Few-shot intent classification and slot filling with retrieved examples. arXiv preprint arXiv:2104.05763.
Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O Arnold, and Bing Xiang. 2021a. Pairwise supervised contrastive learning of sentence representations. arXiv preprint arXiv:2109.05424.
Dejiao Zhang, Wei Xiao, Henghui Zhu, Xiaofei Ma, and Andrew O Arnold. 2021b. Virtual augmentation supported contrastive learning of sentence representations. arXiv preprint arXiv:2110.08552.
A Pre-train
In this section, we present the training data, implementation details, and training stability of model pre-training.
We utilize the corpus collected by TOD-BERT (Wu et al., 2020a) to construct positive pairs. This dataset is the combination of 9 publicly available task-oriented datasets: MetaLWOZ (Lee et al., 2019), Schema (Rastogi et al., 2020), Taskmaster (Byrne et al., 2019), MWOZ (Budzianowski et al., 2018), MSR-E2E (Li et al., 2018), SMD (Eric and Manning, 2017), Frames (Asri et al., 2017), WOZ (Mrkšić et al., 2016), CamRest676 (Wen et al., 2017). The combined dataset contains 100707 dialogues with 1388152 utterances over 60 domains. We filter out sentences with less or equal to 3 words and end up with 892835 consecutive utterances (for DSE) and 879185 unique sentences (for DSE-dropout). Note that, the training data of SimCSE unsup consists of 1 million sentences from Wikipedia. That says, on the one hand, we use the same dataset as TOD-BERT but with our proposed data construction. On the other hand, we use a similar number of training samples as SimCSE unsup. We believe such data construction makes the comparisons fair enough.
We add a contrastive head after the Transformer model and use the outputs of the contrastive head to perform contrastive learning. We use a two-layer MLP with size  $(d\times d,d\times 128)$  as the contrastive head. We use Adam (Kingma and Ba, 2014) with a batch size of 1024 and a constant learning rate as the optimizer. We set the learning rate for contrastive head as  $3e - 4$  and the learning rate for the Transformer model as  $3e - 6$ . The temperature hyperparameter  $\tau$  is set as 0.05. We train the model for 15 epochs (see Appendix A.3 for more details) and save the model at the end for evaluation. We use the same hyperparameters across all the experiments for BERTbase, RoBERTa, and DistilBERTbase models. For BERTlarge and RoBERTa large, we change the batch size to 512 to fit it into the GPUs. Pretraining of the DistilBERTbase, BERTbase, and BERTlarge model respectively takes 3, 4, and 13 hours on 8 NVIDIA® V100 GPUs<sup>5</sup>.
$^{5}$ Our codes and model are under the Apache-2.0 License
Figure 3: DSE and DSE-dropout's performance on each task at every epoch. The dashed lines represent the results of DSE-dropout.
A.3 Training Stability
In this section, we analyze the model's stability in terms of training steps when training with different type of positive pairs. We compare two data construction methods: consecutive utterances (DSE) and dropout (DSE-dropout). We first train each model for 15 epochs, save the checkpoint at the end of each epoch and evaluate each checkpoint with similarity-based methods. Figure 3 shows the two model's average performances on intent classification, out-of-scope detection, utterance-level response selection and dialogue-level response selection.
This result further illustrates the effectiveness of using consecutive utterances as positive pairs for learning dialogue representation. As shown in the figure, DSE's performance on all the tasks consistently improves during the training process, while DSE-dropout achieves the best performance at the first epoch and significantly loses performance afterwards. Besides, DSE's performance is less sensitive to the training steps. It achieves stable performance after about 5 epochs. In contrast, DSE-dropout's performance drops dramatically during the training process, yet it never surpasses DSE's performance. Therefore, we report DSE-dropout's performance at the first epoch in all the tables.
B Evaluation Setup