SlowGuess/NAACL_2022_OCR huggingface dataset:
In this section, we present evaluation details and introduction to the evaluation dataset. Throughout this paper, we use cosine similarity as the similarity metric and mean pooling of token embeddings as the sentence representation. For baseline models,
we report the better results of using its default setting (e.g., last hidden state of the [CLS] token as sentence embedding for SimCSE) and mean pooling.
We use the same hyperparameters for all the models. For similarity-based methods, the only hyperparameter is the max sequence length, we empirically choose a number that can fit at least  $99\%$  of the samples. We respectively set it as 64, 64, 128, and 128 for intent classification, out-of-scope detection, utterance-level response selection and dialogue-level response selection. Hyperparameters for fine-tune evaluations as listed as follows:
Intent Classification We fine-tune all the models for 50 epochs with a batch size of 16 and learning rate of  $3e - 05$ . We evaluate the model on the few-shot validation set after every 10 steps. Early stopping is applied based on the model's validation results. The max sequence length is set as 64 and the dropout at the classification layer is set as 0.1.
Utterance-level Response Selection In this task, we set the max sequence length as 128 and batch size as 100. Other hyperparameters are same as those in Intent Classification. We use the original SimCLR loss (Chen et al., 2020) to optimize the model.
Dialogue Action Prediction In this task, we fine-tune all the models for 100 epochs with a batch size of 32 and learning rate of  $5e - 05$ . We evaluate the model on the few-shot validation set after every 30 steps. Early stopping is also applied. The max sequence length is set as 32 since we find shorter inputs leads to much better performance for all the models. We truncate sentences from the head to keep the most recent dialogue utterances as model input. We set the dropout at the classification layer as 0.2.
B.2 Datasets
Intent Classification We use 4 popular publicly available datasets: CInc150 (Larson et al., 2019) with 150 categories and 4500 test sample, Bank77 (Casanueva et al., 2020) with 77 categories and 3080 test sample, Snips (Coucke et al., 2018) with 7 categories and 1447 test sample, and Hwu64 (Liu et al., 2019a) with 64 categories and 3853 test sample. To apply the CInc150 dataset
in intent classification, we remove all the out-of-scope samples. We also use an internal dataset named Appen, whose texts are transcribed from customer recording. This dataset contains 30 categories and 310 test samples. There are two versions of each sentence. One is transcribed by Automatic Speech Recognition (ASR), which includes some ASR noise (e.g., transcribe errors). The other is transcribed by human annotator. We refer them respectively as Appen-A and Appen-H.
Out-of-scope Detection We use the entire Clinc150 dataset, which contains 150 in-scope intents and one out-of-scope intent. There are 5500 test samples in total (4500 in-scope and 1000 out-of-scope).
Utterance-level Response Selection We use the AmazonQA dataset (Wan and McAuley, 2016), which contains 5334606 question-answer pairs about different products. Following Henderson et al. (2019a), we randomly select 300K pairs for model evaluation.
Dialogue-Level Response Selection We use the DSTC7-Ubuntu dataset (Lowe et al., 2017), which contains conversations about the Ubuntu system. Each query of this dataset comes together with one ground-truth response and 100 candidate responses. We combine the validation and test sets together for evaluation, which results in 6000 evaluation samples.
Dialogue Action Prediction We use the DSTC2 (Henderson et al., 2014) and GSIM (Shah et al., 2018) dataset processed by Wu et al. (2020a). These two datasets respectively contains 13/19 actions and 1117/1039 test samples. The average number of samples in 10-shot and 20-shot training is 79 and 149 for DSTC2; 60 and 120 for GSIM.
C Results of BERTbase
In this section, we present other evaluation results on the  $\mathrm{BERT}_{\mathrm{base}}$  model, including 1-shot and 5-shot fine-tune on intent classification (Table 7), 5-shot similarity-based out-of-scope detection (Table 8), and 500-shot and 1000-shot fine-tune on AmazonQA response selection (Table 9).
D Results of Other Backbone Models
In this section, we present similarity-based evaluation results on other four backbone models: BERTlarge, RoBERTabase, RoBERTalarge, and
DistilBERT<sub>base</sub>. Table 10 shows the results of similarity-based intent classification and Table 11 shows the results of similarity based response selection on both utterance-level and dialogue-level. As shown in the tables, DSE leads to consistent and significant performance boost on all the backbone models.
BERTbase	Clinc150	Bank77	Snips	Hwu64	Appen-A	Appen-H	Ave.
sh	SimCSE-sup*	48.30	35.29	49.77	34.09	30.16	38.42	39.34
PairSupCon*	50.33	37.59	52.53	34.21	33.58	41.65	41.65
DSE-dianli* (ours)	43.07	38.02	46.57	30.98	29.39	36.77	37.47
BERT◇	37.01	24.28	52.05	26.36	17.87	19.71	29.55
SimCSE-unsup◇	42.72	33.56	47.13	30.19	24.00	32.68	35.05
TOD-BERT◇	39.48	26.12	46.13	26.81	13.45	23.26	29.21
DSE-dropout◇ (ours)	41.89	30.10	44.46	28.06	18.68	20.48	30.61
DSE◇ (ours)	55.67	38.10	70.67	37.93	32.68	45.03	46.68
sh	SimCSE-sup*	85.49	70.16	88.90	68.86	61.71	73.29	74.74
PairSupCon*	85.15	71.00	86.01	68.12	63.94	73.68	74.65
DSE-dianli* (ours)	81.87	69.33	82.49	64.40	58.90	68.52	70.92
BERT◇	84.00	68.51	85.72	64.79	55.00	65.52	70.59
SimCSE-unsup◇	83.35	70.08	86.82	65.61	59.68	70.03	72.60
TOD-BERT◇	83.15	65.29	88.49	66.29	56.32	67.13	71.11
DSE-dropout◇ (ours)	84.14	69.74	87.39	66.47	57.74	68.13	72.27
DSE◇ (ours)	86.67	71.52	92.56	70.71	63.71	75.10	76.71
Table 7: Results of fine-tuning all the models for 1-shot and 5-shot Intent Classification for BERT<sub>base</sub> models. ⋆: Supervised models. ⋅: Unsupervised models
BERTbase	Accuracy	In-Accuracy	OOS-Accuracy	OOS-Recall	Ave.
mean	SimCSE-sup*	59.65	69.04	80.35	17.40	56.61
PairSupCon*	68.37	71.03	85.62	56.43	70.36
DSE-dianli* (ours)	56.22	56.92	83.38	53.07	62.40
BERT◇	53.92	57.25	81.94	38.95	58.01
SimCSE-unsup◇	55.68	63.06	79.58	22.47	55.20
DialoGPT◇	58.53	63.25	82.62	37.25	60.41
TOD-BERT◇	53.49	56.12	81.75	41.64	58.25
DSE-dropout◇ (ours)	64.21	67.39	83.59	49.92	66.28
DSE◇ (ours)	72.62	74.77	87.16	62.95	74.38
mean	SimCSE-sup*	42.50	34.95	48.26	76.50	50.55
PairSupCon*	56.19	47.53	62.79	95.17	65.42
DSE-dianli* (ours)	46.87	37.75	59.96	87.92	58.13
BERT◇	47.79	38.40	57.86	90.07	58.53
SimCSE-unsup◇	45.13	36.82	52.01	82.51	54.12
DialoGPT◇	49.64	40.26	57.49	91.81	59.80
TOD-BERT◇	47.88	38.46	58.43	90.31	58.77
DSE-dropout◇ (ours)	54.45	46.00	61.31	92.50	63.56
DSE◇ (ours)	59.20	51.35	64.79	94.53	67.47
Table 8: Results on similarity-based 5-shot out-of-scope detection on Clinc150 dataset. The out-of-scope threshold is respectively set as mean (m) and mean-standard (m-d) of each sample's similarity with its closest category. See Sec. 4.2.2 for details. All the models use BERT<sub>base</sub> as the backbone model. Supervised models. Unsupervised models.
BERTbase	AmazonQA 500-shot	AmazonQA 1000-shot
Top-1 Acc.	Top-3 Acc.	Top-10 Acc.	Top-1 Acc.	Top-3 Acc.	Top-10 Acc.
SimCSE-sup*	59.02	72.90	84.40	60.24	73.91	85.17
PairSupCon*	61.24	74.51	85.19	62.31	75.36	86.01
BERT◇	55.63	70.98	83.79	58.00	72.67	84.81
SimCSE-unsup◇	56.04	70.34	82.56	57.85	71.95	84.01
TOD-BERT◇	43.52	59.29	75.06	46.54	62.16	77.15
DSE-dropout◇ (ours)	57.66	72.02	83.72	58.66	72.86	84.67
DSE◇ (ours)	61.71	75.66	86.49	63.02	76.47	87.55
Table 9: Results on 500-shot and 1000-shot fine-tune evaluation on response selection on AmazonQA (utterance-level). All the models use BERTbase as the backbone model.  $\clubsuit$  : Supervised models.  $\diamondsuit$  : Unsupervised models.
Clinc150	Bank77	Snips	Hwu64	Ave.
1 shot	BERTlarge	31.71	20.47	54.31	25.24	32.93
BERTlarge-DSE	65.57	45.45	78.52	46.37	58.97
RoBERTa_base	34.58	20.58	52.25	24.24	32.91
RoBERTa_base-DSE	66.05	45.01	80.58	43.98	58.90
RoBERTaLarge	35.72	20.84	54.80	23.57	33.73
RoBERTaLarge-DSE	69.23	45.42	73.72	44.29	58.16
DistilBERTbase	39.48	23.96	63.00	30.25	39.17
DistilBERTbase-DSE	60.47	43.52	76.38	44.63	56.25
5 shot	BERTlarge	46.78	33.53	70.89	37.06	47.06
BERTlarge-DSE	80.40	64.49	89.08	63.00	74.24
RoBERTa_base	53.58	32.40	68.90	34.98	47.46
RoBERTa_base-DSE	81.73	64.92	89.67	62.81	74.78
RoBERTaLarge	55.43	33.25	78.01	36.25	50.73
RoBERTaLarge-DSE	82.52	62.93	86.64	61.04	73.28
DistilBERTbase	61.00	39.45	78.90	45.00	56.08
DistilBERTbase-DSE	77.16	60.39	86.48	60.81	71.21
Table 10: Results on similarity-based 1-shot and 5-shot Intent Classification with different model as the backbone. DSE leads to significant and consistent performance boost for all the models.
BERTlarge	AmazonQA	DSTC7-Ubuntu
Top-1 Acc.	Top-3 Acc.	Top-10 Acc.	Top-1 Acc.	Top-3 Acc.	Top-10 Acc.
BERTlarge	27.97	41.30	57.04	6.10	11.08	22.31
BERTlarge-DSE	59.63	73.46	84.12	16.40	24.56	36.51
RoBERTa_base	19.60	29.67	44.70	4.86	9.80	20.70
RoBERTa_base-DSE	55.69	70.01	81.68	15.86	24.25	37.38
RoBERTa_large	26.68	37.73	51.70	7.65	14.50	26.10
RoBERTa_large-DSE	58.13	71.65	82.20	18.66	27.70	40.93
DistilBERTbase	31.73	46.47	63.23	6.65	12.46	24.98
DistilBERTbase-DSE	56.36	70.11	81.51	14.56	22.78	35.63
Table 11: Results on 0-shot response selection on AmazonQA (utterance-level) and DSTC7-Ubuntu (dialogue-level). DSE leads to significant and consistent performance improvements on all the models.