SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhihan Zhou * Dejiao Zhang† Wei Xiao† Nicholas Dingwall† Xiaofei Ma† Andrew O. Arnold† Bing Xiang†  
*Northwestern University †AWS AI Labs
Learning high-quality dialogue representations is essential for solving a variety of dialogue-oriented tasks, especially considering that dialogue systems often suffer from data scarcity. In this paper, we introduce Dialogue Sentence Embedding (DSE), a self-supervised contrastive learning method that learns effective dialogue representations suitable for a wide range of dialogue tasks. DSE learns from dialogues by taking consecutive utterances<sup>1</sup> of the same dialogue as positive pairs for contrastive learning. Despite its simplicity, DSE achieves significantly better representation capability than other dialogue representation and universal sentence representation models. We evaluate DSE on five downstream dialogue tasks that examine dialogue representation at different semantic granularities. Experiments in few-shot and zero-shot settings show that DSE outperforms baselines by a large margin. For example, it achieves  $13\%$  average performance improvement over the strongest unsupervised baseline in 1-shot intent classification on 6 datasets.<sup>2</sup> We also provide analyses on the benefits and limitations of our model.
Due to the variety of domains and the high cost of data annotation, labeled data for task-oriented dialogue systems is often scarce or even unavailable. Therefore, learning universal dialogue representations that effectively capture dialogue semantics at different granularities (Hou et al., 2020; Krone et al., 2020; Yu et al., 2021) provides a good foundation for solving various downstream tasks (Snell et al., 2017; Vinyals et al., 2016).
Contrastive learning (Chen et al., 2020; He et al., 2020) has achieved widespread success in represent-
Figure 1: TSNE visualization of the dialogue representations provides by TOD-BERT, SimCSE, and DSE. Left: each color indicates one intent category, while the black circles represent out-of-scope samples. Right: items with the same color stands for query-response pairs, where triangles represent queries. The black circles represent randomly sampled responses.
tations learning in both the image domain (Hjelm et al., 2018; Lee et al., 2020; Bachman et al., 2019) and the text domain (Gao et al., 2021; Zhang et al., 2021a,b; Wu et al., 2020a). Contrastive learning aims to reduce the distance between semantically similar (positive) pairs and increase the distance between semantically dissimilar (negative) pairs. These positive pairs can be either human-annotated or obtained through various data augmentations, while negative pairs are often collected through negative sampling in the mini-batch.
In the supervised learning regime, Gao et al. (2021); Zhang et al. (2021a) demonstrate the effectiveness of leveraging the Natural Language Inference (NLI) datasets (Bowman et al., 2015; Williams et al., 2018) to support contrastive learning. Inspired by their success, a natural choice of dialogue representation learning is utilizing the
*Work done during an internship at AWS AI Labs.
Throughout this paper, we use utterance to refer to all the sentences that belong to the same dialogue turn.
2The code and pre-trained models are publicly available at https://github.com/amazon-research/dse.
Human Language Technologies, pages 754 - 768
Pair 1: I am looking for restaurants. | What type of food do you like?  
Pair 2: What type of food do you like? | I want some pizza.  
Pair 3: I want some pizza. | Domino's is a good place for pizza.
Pair 4: Find me some restaurants. | What type of food do you like?  
Pair 5: What type of food do you like? | Korean food, please.  
Pair 6: Korean food, please. | There is no Korean restaurant.
Figure 2: Illustration of the positive pair construction from dialogues.
Dialogue-NLI dataset (Welleck et al., 2018) that consists of both semantically entailed and contradicted pairs. However, due to its relatively limited scale and diversity, we found learning from this dataset leads to less satisfying performance, while the high cost of collecting additional human annotations precludes its scalability. On the other extreme, unsupervised representation learning has achieved encouraging results recently, among which SimCSE (Gao et al., 2021) and TOD-BERT (Wu et al., 2020a) set new state-of-the-art results on general texts and dialogues, respectively.
SimCSE uses Dropout (Srivastava et al., 2014) to construct positive pairs from any text by passing a sentence through the encoder twice to generate two different embeddings. Although SimCSE outperforms common data augmentations that directly operate on discrete text, we find it performs poorly in the dialogue domain (see Sec. 4.3). This motivates us to seek better positive pair constructions by leveraging the intrinsic properties of dialogue data. On the other hand, TOD-BERT takes an utterance and the concatenation of all the previous utterances in the dialogue as a positive pair. Despite promising performance on same tasks, we found TOD-BERT struggles on many other dialogue tasks where the semantic granularities or data statistics are different from those evaluated in their paper.
In this paper, inspired by the fact that dialogues consist of consecutive utterances that are often semantically related, we use consecutive utterances within the same dialogue as positive pairs for contrastive learning (See Figure 2). This simple strategy works surprisingly well. We evaluate DSE on a wide range of task-oriented dialogue applications, including intent classification, out-of-scope detection, response selection, and dialogue action prediction. We demonstrate that DSE substantially outperforms TOD-BERT, SimCSE, and some other sentence representation learning models in
most scenarios. We assess the effectiveness of our approach by comparing DSE against its variants trained on other types of positive pairs (e.g., Dropout and Dialogue-NLI). We also discuss the trade-off in learning dialogue representation for tasks focusing on different semantic granularities and provide insights on the benefits and limitations of the proposed method. Additionally, we empirically demonstrate that using consecutive utterances as positive pairs can effectively improve the training stability (Appendix A.3).
2 Why Contrastive Learning on Consecutive Utterances?
When performing contrastive learning on consecutive utterances, we encourage the model to treat an utterance as similar to its adjacent utterances and dissimilar to utterances that are not consecutive to it or that belong to other dialogues.
On the one hand, this training process directly increases an utterance's similarity with its true response and decreases its similarities with other randomly sampled utterances. The ability to identify the appropriate response from many similar utterances is beneficial for dialogue ranking tasks (e.g., response selection). On the other hand, consecutive utterances also contain implicit categorical information, which benefits dialogue classification tasks (e.g., intent classification and out-of-scope detection). Consider pairs 1 and 4 in Figure 2: we implicitly learn similar representations of I am looking for restaurants and Find me some restaurants, since they are both consecutive with What type of food do you like?.
In contrast, SimCSE does not enjoy these benefits by simply using Dropout as data augmentation. Although TOD-BERT also leverages the intrinsic dialogue semantics by combining an utterance with its dialogue context as positive pair, the context is often the concatenation of 5 to 15 utterances.
Due to the large discrepancy in both semantics and data statistics between each utterance and its context, simply optimizing the similarity between them leads to less satisfying representations on many dialogue tasks. As shown in Section 4, TODBERT can even lead to degenerated representations on some downstream tasks when compared to the original BERT model.
Let  $\{(x_i, x_{i+})\}_{i=1}^M$  be a batch of positive pairs, where  $M$  is the batch size. In our setting, each  $(x_i, x_{i+})$  denotes a pair of consecutive utterances sampled from a dialogue. Let  $e_i$  denote the representation of the text instance  $x_i$  that is obtained through an encoder. In this paper, we use mean pooling to obtain representations.
3.2 Training Target
Contrastive learning aims to maximize the similarity between positive samples and minimize the similarity between negative samples. For a contrastive anchor  $x_{i}$ , the contrastive loss aims to increase its similarity with its positive sample  $x_{i+}$  and decrease its similarity with the other  $2M - 2$  negative samples within the same batch.
We adopt the Hard-Negative sampling strategy proposed by Zhang et al. (2021a), which puts higher weights on the samples that are close to the anchor in the representation space. The underlying hypothesis is that hard negatives are more likely to occur among those that are located close to the anchor in the representation space. Specifically, the Hard-Negative sampling based contrastive loss regarding anchor  $x_{i}$  is defined as follows:
$$
\ell^ {i, i ^ {+}} = - \log \frac {\exp \left(\operatorname {s i m} \left(e _ {i} , e _ {i ^ {+}}\right) / \tau\right)}{\sum_ {j \neq i} \exp \left(\alpha_ {i j} \cdot \sin \left(e _ {i} , e _ {j}\right) / \tau\right)}. \tag {1}
$$
As mentioned above, here  $i$  and  $i^+$  represent the indices of the anchor and its positive sample. We use  $\tau$  to denote the temperature hyperparameter and  $\mathrm{sim}(e_i,e_j)$  represent the cosine similarity of  $e_i$  and  $e_j$ . In the above loss,  $\alpha_{ij}$  is defined as follows,
$$
\alpha_ {i j} = \frac {\exp (\sin (e _ {i} , e _ {j}) / \tau)}{\frac {1}{2 M - 2} \sum_ {k \neq i ^ {+}} \exp (\sin (e _ {i} , e _ {k}) / \tau)}. \tag {2}
$$
Noted here, the denominator is averaged over all the other  $2M - 2$  negatives of  $x_{i}$ . Intuitively, samples that are close to the anchor in the representation space are assigned with higher weights. In other words,  $\alpha_{ij}$  denotes the relative importance of instance  $x_{j}$  for optimizing the contrastive loss of the anchor  $x_{i}$  among all  $2M - 2$  negatives. For every positive pair  $(x_{i}, x_{i+})$ , we respectively take  $x_{i}$  and  $x_{i+}$  as the contrastive anchor to calculate the contrastive loss. Thereby, the contrastive loss over the batch is calculated as:
$$
\mathcal {L} = \frac {1}{2 M} \sum_ {i = 1} ^ {M} \left(\ell^ {i, i ^ {+}} + \ell^ {i ^ {+}, i}\right) \tag {3}
$$
Here  $\ell^{i^+,i}$  is defined by exchanging the roles of instances  $i$  and  $i^+$  in Equation (1), respectively.
We run experiments with five different backbones: BERTbase, BERTlarge (Devlin et al., 2018), RoBERTa-base, RoBERTaLarge (Liu et al., 2019b), DistilBERTbase (Sanh et al., 2019). Due to the space limit, we only present the results on BERTbase in the main text. The results of other models are summarized in Appendix D. We use the same training data as TOD-BERT for a fair comparison. We summarize the implementation details and data statistics of both pre-training and evaluation in Appendices A and B, respectively.
We compare DSE against several representation learning models that attain state-of-the-art results on both general text and dialogue languages. We categorize them into the following two categories.
Supervised Learning SimCSE-sup (Gao et al., 2021) is the supervised version of SimCSE, which uses entailment and contradiction pairs in the NLI datasets (Bowman et al., 2015; Williams et al., 2018) to construct positive pair and hard negative pair accordingly. In a similar vein, PairSupCon (Zhang et al., 2021a) leverages the entailment pairs as positive pairs only while proposing an unsupervised hard negative sampling strategy that we summarized in Section 3.2. Following this line, we also evaluate DSE against its variant DSE-dianli trained on the Dialogue Natural Language Inference dataset (Welleck et al., 2018) by taking all the entail pairs as positive pairs.
Task	Dataset	Evaluation Setting	Num.
Intent Classification	Clink150, Snips, Hwu64, Bank77, Appen-A, Appen-H	1-shot & 5-shot fine-tune	10
1-shot & 5-shot similarity	10
Out-of-scope Detection	Clink150	1-shot & 5-shot similarity	10
Utterance-level Response Selection	AmazonQA	0-shot similarity	1
500-shot & 1000-shot fine-tune	5
Dialogue-level Response Selection	DSTC7-Ubuntu	0-shot similarity	1
Dialogue Action Prediction	DSTC2, GSIM	10-shot & 20-shot fine-tune	5
Table 1: Summarization of all the experimental settings. Please see Appendix B.2 for details of each dataset. The last column (Num) indicates the number of independent experiments with different random seeds (we report the averaged results). Since there is no randomness in 0-shot evaluations, we only run them once.
Unsupervised Learning TOD-BERT (Wu et al., 2020a) optimizes a contrastive response selection objective by treating an utterance and its dialogue context as positive pair. DialogGPT (Zhang et al., 2019) is a dialogue generation model that learns from consecutive utterance by optimizing a language modeling target. SimCSE-unsup (Gao et al., 2021) uses Dropout (Srivastava et al., 2014) to construct positive pairs. In the general text domain, SimCSE-unsup has attained impressive performance over several explicit data augmentation strategies that directly operate on the discrete texts. To test its effectiveness in the dialogue domain, we compare DSE against its variant DSE-dropout where augmentations of every single utterance are obtained through Dropout.
The evaluations on DSE-dropout and DSE-dianli allow us to fairly compare our approach against the state-of-the-art approaches in both the supervised learning and the unsupervised learning regimes.
4.2 Evaluation Setting
To accommodate the fact that obtaining a large number of annotations is often time-consuming and expensive for solving the task-oriented dialogue applications, especially considering the variety of domains and certain privacy concerns, we mainly focus on few-shot or zero-shot based evaluations.
4.2.1 Evaluation Methods
Considering that only a few annotations are available in our setting, we mainly focus on the similarity-based evaluations, where predictions
are made based on different similarity metrics applied in the embedding space without requiring updating the model.
We use different random seeds to independently construct multiple (See Table 1) few-shot train and validation sets from the original training data and use the original test data for performance evaluation. To examine whether the performance gap reported in the similarity-based evaluations is consistent with the associated fine-tuning approaches, we also report the fine-tuning results. We perform early stopping according to the validation set and report the testing performance averaged over different data splits.
4.2.2 Tasks and Metrics
We evaluate all models considered in this paper on two types of tasks: utterance-level and dialogue-level. The utterance-level tasks take a single dialogue utterance as input, while the dialogue-level tasks take the dialogue history as input. These two types of tasks assess representation quality on dialogue understanding at different semantic granularities, which are shared across a variety of downstream tasks.
Intent Classification is an utterance-level task that aims to classify user utterances into one of the pre-defined intent categories. We use Prototypical Networks (Snell et al., 2017) to perform the similarity-based evaluation. Specifically, we calculate a prototype embedding for each category by averaging the embedding of all the training samples that belong to this category. A sample is classified into the category whose prototype embedding is the most similar to its own. We report the classification accuracy for this task.
Out-of-scope Detection advances intent classification by detecting whether the sample is out-of
3We use mean pooling of its hidden states as sentence representation, which leads to better performance than using only the last token. We use its Medium version that has twice as many parameters as  $\mathbf{BERT}_{\mathrm{base}}$  since we found its Small version performs dramatically worse under our settings.
BERTbase	Clinc150	Bank77	Snips	Hwu64	Appen-A	Appen-H	Ave.
Shot	SimCSE-sup*	52.30	38.05	65.98	40.79	35.35	44.81	46.21
PairSupCon*	55.34	41.30	65.20	41.43	37.55	47.55	48.06
DSE-dianli* (ours)	45.91	38.33	58.23	34.95	33.87	42.26	42.26
BERT◇	36.98	22.05	62.51	27.74	13.19	18.74	30.20
SimCSE-unsup◇	46.44	37.51	59.58	34.34	27.10	36.00	40.16
DialoGPT◇	42.23	28.08	63.10	30.45	18.90	24.48	34.54
TOD-BERT◇	36.67	27.11	62.52	29.52	20.61	26.68	33.85
DSE-dropout◇ (ours)	46.48	30.02	65.03	33.25	16.94	21.77	35.58
DSE◇ (ours)	62.53	43.12	79.57	44.31	37.97	48.71	52.70
Shot	SimCSE-sup*	71.11	56.38	79.98	56.52	49.71	59.42	62.18
PairSupCon*	73.88	60.07	76.14	55.75	52.71	62.23	63.46
DSE-dianli* (ours)	60.65	49.78	73.80	46.65	46.52	54.39	55.30
BERT◇	59.48	38.73	78.65	43.15	21.39	27.61	44.83
SimCSE-unsup◇	65.37	55.03	77.01	48.79	43.35	51.55	56.85
DialoGPT◇	64.53	46.56	82.15	45.67	33.67	39.61	52.03
TOD-BERT◇	57.74	42.98	79.68	42.32	33.58	42.52	49.80
DSE-dropout◇ (ours)	70.46	49.95	80.10	52.16	30.00	37.48	53.36
DSE◇ (ours)	78.73	61.65	88.62	60.87	52.32	62.68	67.48
Table 2: Results on similarity-based 1-shot and 5-shot Intent Classification. Predictions are made purely based on the embeddings provided by each model without any parameter tuning. All the models use BERT<sub>base</sub> as the backbone model. 🚤: Supervised models. 🙦: Unsupervised models.
scope, i.e., does not belong to any pre-defined categories. We adapt the aforementioned Prototypical Networks to solve it. For a test sample, if its similarity with its most similar category is lower than a threshold, we classify it as out-of-scope. Otherwise, we assign it to its most similar category. For each model, we calculate the mean and std (standard deviation) of the similarity scores between every sample and its most similar prototype embedding, and take mean-standard and mean as the threshold, respectively. The evaluation set contains both in-scope and out-of-scope examples. We evaluate this task with four metrics: 1) Accuracy: accuracy of both in-scope and out-of-scope detection. 2) In-Accuracy: accuracy reported on 150 in-scope intents. 3) OOS-Accuracy: out-of-scope detection accuracy. 4) OOS-Recall: recall of OOS detection.
Utterance-level Response Selection is an utterance-level task that aims to find the most appropriate response from a pool of candidates for the input user query, where both the query and response are single dialogue utterances. We formulate it as a ranking problem and evaluate it with Top-k-100 accuracy (a.k.a., k-to-100 accuracy), a standard metric for this ranking problem (Wu et al., 2020a). For every query, we combine its ground-truth response with 99 randomly sampled responses and rank these 100 responses based on their similarities with the
query in the embedding space. The Top-k-100 accuracy represents the ratio of the ground-truth response being ranked at top-k, where k is an integer between 1 and 100. We report the Top-1, Top-3, and Top-10 accuracy of the models.
Dialogue-Level Response Selection is a dialogue-level task. The only difference with the Utterance-level Response Selection is that query in this task is dialogue history (e.g., concatenation of multiple dialogue utterances from different speakers). We also report the Top-1, Top-3, and Top-10 accuracy for this task.
Dialogue Action Prediction is a dialogue-level task that aims to predict the appropriate system action given the most recent dialogue history. We formulate it as a multi-label text classification problem and evaluate it with model fine-tuning. We report the Macro and Micro F1 scores for this task.
Intent Classification & Out-of-scope Detection Tables 2 and 3 show the results of similarity-based intent classification and out-of-scope detection. The fine-tuning based results are presented in Appendix C. As we can see, DSE substantially outperforms all the baselines. In intent classification, it attains  $13\%$  average accuracy improvement over the strongest unsupervised baseline. More impor
BERTbase	Accuracy	In-Accuracy	OOS-Accuracy	OOS-Recall	Ave.
OE	SimCSE-sup*	44.63	51.50	78.63	13.70	47.12
PairSupCon*	51.87	54.34	82.33	40.75	57.32
DSE-dianli* (ours)	44.73	44.88	80.83	44.07	53.63
BERT◇	33.96	36.01	80.58	24.77	43.83
SimCSE-unsup◇	40.45	45.50	77.83	17.73	45.38
DiaLoGPT◇	36.98	40.70	80.73	20.21	44.66
TOD-BERT◇	34.77	36.28	79.74	27.98	44.69
DSE-dropout◇	42.41	45.19	81.26	29.92	49.70
DSE◇ (ours)	58.74	60.52	84.07	50.72	63.51
mean	SimCSE-sup*	36.90	29.07	47.04	72.12	46.28
PairSupCon*	47.29	37.44	58.72	91.63	58.77
DSE-dianli* (ours)	40.70	30.78	57.67	85.30	53.61
BERT◇	35.64	24.78	53.09	84.47	49.50
SimCSE-unsup◇	37.65	28.99	49.38	76.62	48.16
DiaLoGPT◇	38.04	27.00	52.87	87.75	51.42
TOD-BERT◇	36.31	25.76	53.40	83.75	49.81
DSE-dropout◇ (ours)	41.19	30.93	54.76	87.39	53.57
DSE◇ (ours)	50.88	41.72	60.64	92.11	61.34
Table 3: Results on similarity-based 1-shot out-of-scope detection on Clinc150 dataset. The out-of-scope threshold is respectively set as mean (m) and mean-standard (m-d) of each sample's similarity with its closest category. See Sec. 4.2.2 for details. Supervised models. Unsupervised models.
BERTbase	AmazonQA	DSTC7-Ubuntu
Top-1 Acc.	Top-3 Acc.	Top-10 Acc.	Top-1 Acc.	Top-3 Acc.	Top-10 Acc.
SimCSE-sup*	47.03	62.40	76.80	11.37	19.40	33.53
PairSupCon*	52.22	65.09	76.85	15.00	23.02	35.73
DSE-dianli* (ours)	49.16	63.36	76.66	14.92	22.73	34.72
BERT◇	29.70	43.86	60.36	6.75	12.97	24.20
SimCSE-unsup◇	48.02	62.45	76.00	10.03	17.13	29.37
DialoGPT◇	35.96	49.52	64.44	10.20	17.60	29.82
TOD-BERT◇	27.25	40.26	56.63	5.52	10.55	22.30
DSE-dropout◇ (ours)	37.80	51.64	66.58	9.55	16.97	28.80
DSE◇ (ours)	56.62	70.54	81.90	14.78	23.10	35.73
Table 4: Results on 0-shot response selection on AmazonQA (utterance-level) and DSTC7-Ubuntu (dialogue-level).
tantly, DSE achieves a  $5\% - 10\%$  average accuracy improvement over the supervised baselines that are trained on a large amount of expensively annotated data. The same trend was observed in out-of-scope detection, where DSE achieves  $13\% - 20\%$  average performance improvement over the strongest unsupervised baseline. The comparison between DSE, DSE-dropout, and DSE-dianli further demonstrates the effectiveness of using consecutive utterances as positive pairs in learning dialogue embeddings.
The left panel of Figure 1 visualizes the embeddings on the Clinc150 dataset given by TODBERT, SimCSE, and DSE, which provides more intuitive insights into the performance gap. As shown in the figure, with the DSE embeddings, in-scope samples belonging to the same category are
closely clustered together. Clusters of different categories are clearly separated with a large margin, while the out-of-scope samples are far away from those in-scope clusters.
Response Selection Table 4 shows the results of similarity-based 0-shot response selection on utterance-level (AmazonQA) and dialogue-level (DSTC7-Ubuntu). Results of finetune-based evaluation on AmazonQA show similar trend and we summarize in Table 9 in Appendix. In Table 4, the large improvement attained by DSE over the baselines indicate our model's capability in dialogue response selection, in presence of both single-utterance query or using long dialogue history as query. The right panel of Figure 1 further illustrates this. It visualizes the embeddings of questions and