SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 8 provides the statistics of the original SNIPS data set (Coucke et al., 2018). For further details on how the data set has been used in the 5-shot evaluation setup we refer the reader to the work of Hou et al. (2020). The data sets are available at:
github.com/AtmaHou/FewShotTagging
Recently, RESTAURANTS-8K and DSTC8 training and evaluation data have been made available via the integrated DialogoGLUE benchmark (Mehri et al., 2020). For further details regarding the two evaluation sets, we also refer the reader to the original work (Rastogi et al., 2019; Coope et al., 2020).
people	time	date	first name	last name	Total
train	2164 (547)	2164 (547)	1721 (601)	887 (364)	891 (353)	8198
test	983 (244)	853 (276)	802 (300)	413 (177)	426 (174)	3731
Table 6: The number of examples for each slot in the RESTAURANTS-8K data set. Numbers in brackets show how many examples have the slot is_requested.
Sub-domain	Train Size	Test Size	Slots
Buses_1	1133	377	from_location (169/54), leaving_date (165/57), to_location (166/52)
Events_1	1498	521	city_of_event (253/82), date (151/33), subcate-gory (56/26)
Homes_1	2064	587	area (288/86), visit_date (237/62)
RentalCars_1	874	328	dropoff_date (112/42), pickup_city (116/48), pickup_date (120/43), pickup_time (119/43)
Table 7: Statistics of the used data sets extracted from the DSTC8 schema-guided dialog dataset. The number of examples in the train and test sets for each slot are reported in parentheses.
Domain	# of Sentences	Labels
Weather	2,100	10
Music	2,100	10
Playlist	2,042	6
Book	2,056	8
Search	2,059	8
Restaurants	2,073	15
Creative	2,054	3
Table 8: Statistics of the original SNIPS data set.
B Baseline Models in the SNIPS Evaluation
This appendix provides a brief summary of the models from Hou et al. (2020) included in the SNIPS evaluation (Table 4) alongside ConVEx.
TransferBERT is a direct application of BERT (Devlin et al., 2019) to sequence labeling. It is first trained on the source domains. As the sequence labeling layers are domain-specific, these are then removed, and new layers are fine-tuned on the indomain training set (i.e., Hou et al. (2020) refer to it as the support set; this is exactly what we use for fine-tuning ConVEx).
SimBERT predicts sequence labels according to the cosine similarity between the representations from a BERT model of the input tokens with tokens in the support set, selecting the labels of the most similar labeled tokens.
WarmProtoZero (WPZ) (Fritzler et al., 2019) applies Prototypical Networks (Snell et al., 2017) to sequence labeling tasks. It treats sequence-labeling as word-level classification, and can either use randomly initialized word embeddings, or pretrained representations in the case of WPZ+BERT.
TapNet is a few-shot learning paradigm originally applied to image classification (Yoon et al., 2019). This works similarly to Prototypical Networks, but includes a task-adaptive network that projects examples into a space where words of differing labels are well separated.
Collapsed Dependency Transfer (CDT) is a technique for simplifying transition dynamics of a CRF, applied to both TapNet and WPZ. This represents the full transition matrix using shared abstract transitions, e.g. modeling transitions between any Begin tag to the Begin tag of any different slot using a shared probability.
Label Enhanced models, denoted  $L$ -WPZ and  $L$ -TapNet use the semantics of the label names themselves to enrich the word-label similarity modeling.
The exact  $F_{1}$  scores corresponding to the results plotted in Figure 2a and Figure 2b are provided in Table 9 and Table 10, respectively. Additional results with model ensembling are available in Table 11.
Fraction	Span-ConveRT	Span-BERT	ConVEx -no-aux	ConVEx -full
1 (8198)	95.8	93.1	95.6	96.0
1/2 (4099)	94.1	91.4	94.1	94.1
1/4 (2049)	91.2	88.0	92.2	92.6
1/8 (1024)	88.5	85.3	90.0	90.6
1/16 (512)	81.1	76.6	86.2	86.4
1/32 (256)	63.8	53.6	78.4	81.8
1/64 (128)	57.6	42.2	73.4	76.0
1/128 (64)	40.5	30.6	70.9	71.7
Table 9: Average  $F_{1}$  scores across all slots for the evaluation on RESTAURANTS-8K test data with varying training set fractions. Numbers in brackets denote the training set sizes. The peak scores in each training setup (i.e., per row) are in bold.
Fraction	Span-ConveRT	Span-BERT	ConVEx -no-aux	ConVEx
Buses_1
1 (1133)	93.5	93.3	95.1	96.0
½ (566)	88.9	85.3	90.4	92.6
¼ (283)	84.0	77.8	88.6	86.7
½ (141)	69.1	69.6	83.6	84.0
½ (16 (70)	58.3	44.4	74.5	75.2
½ (32 (35)	32.7	25.5	65.4	59.2
Events_1
1 (1498)	92.7	84.3	92.4	91.7
½ (749)	86.9	80.2	88.4	87.3
¼ (374)	82.2	78.6	86.4	87.2
½ (187)	70.0	57.4	72.4	82.2
½ (16 (93)	55.9	43.9	65.7	66.6
½ (32 (47)	39.2	25.6	51.4	54.0
Homes_1
1 (2064)	94.8	96.3	95.5	98.3
½ (1032)	96.1	95.7	95.6	95.6
¼ (516)	95.4	95.1	93.0	94.5
½ (258)	93.4	89.5	92.2	94.8
½ (16 (129)	86.3	76.4	94.0	92.3
½ (32 (65)	77.1	61.2	89.4	92.0
RentalCars_1
1 (874)	94.0	92.8	90.7	92.0
½ (437)	93.1	87.9	89.3	91.7
¼ (218)	83.0	81.4	87.9	87.4
½ (109)	66.4	64.8	78.8	77.6
½ (16 (54)	51.6	49.6	62.3	60.6
½ (32 (27)	44.0	30.1	47.3	50.3
Table 10:  $F_{1}$  scores on the DSTC8 single-domain data sets. Numbers in brackets denote the training set sizes. The peak scores in each training setup are in bold.
Fraction	ConVEx	ConVEx ensemble
1 (8198)	96.0	95.8
1/2 (4099)	94.1	94.2
1/4 (2049)	92.5	92.5
1/8 (1024)	90.6	90.7
1/16 (512)	86.4	88.2
1/32 (256)	81.8	83.9
1/64 (128)	76.0	78.2
1/128 (64)	71.7	73.5
Table 11: Average  $F_{1}$  scores across all slots for RESTAURANTS-8K for ConVEx with and without ensembling. The ConVEx ensemble model fine-tunes 3 decoders per slot, and then averages their output scores.