SlowGuess/NAACL_2022_OCR huggingface dataset:
Train
POS	Toks	Vocab	GPT-2	UD	DM	PSD	PTB-phr	PTB-fxn	PTG	EDS
All	22,596	27,344	45.9 ± .1	32.7 ± 0.2	31.4 ± 0.1	30.7 ± 0.1	29.8 ± 0.2	29.0 ± 0.3	26.8 ± 0.3	24.7 ± 0.3
noun	7,731	18,435	142.5 ± .7	122.0 ± 0.1	119.0 ± 1.0	120.9 ± 0.7	138.0 ± 2.5	139.6 ± 4.5	120.6 ± 2.5	98.0 ± 3.6
verb	2,639	7,100	128.8 ± .8	80.4 ± 1.2	89.4 ± 1.1	90.7 ± 0.7	81.7 ± 0.9	79.3 ± 0.7	86.9 ± 2.5	85.9 ± 1.4
mod	2,235	6,292	228.7 ± .5	158.8 ± 3.8	145.9 ± 3.3	130.5 ± 2.0	123.4 ± 2.1	113.5 ± 3.3	100.2 ± 4.0	98.6 ± 4.0
aux	582	95	17.6 ± < .1	11.1 ± 0.1	6.5 ± 0.1	9.3 ± 0.2	11.0 ± 0.2	10.9 ± 0.1	9.4 ± 0.1	5.9 ± 0.1
adp	1,957	232	10.1 ± < .1	7.3 ± 0.1	5.7 ± 0.1	7.2 ± 0.1	4.3 ± < 0.1	3.6 ± < 0.1	3.5 ± 0.1	5.5 ± 0.1
part	645	27	3.7 ± < .1	2.0 ± < 0.1	2.1 ± 0.1	2.3 ± 0.1	1.7 ± 0.1	1.7 ± < 0.1	1.7 ± < 0.1	1.6 ± < 0.1
sconj	268	96	15.4 ± .1	12.3 ± 0.1	7.7 ± 0.2	11.3 ± 0.6	3.5 ± < 0.1	2.6 ± < 0.1	3.5 ± 0.1	6.8 ± 0.1
cconj	548	35	13.0 ± .1	7.4 ± 0.3	6.3 ± 0.5	2.5 ± 0.1	4.3 ± 0.1	4.3 ± 0.1	2.1 ± < 0.1	1.9 ± < 0.1
det	1,726	91	9.4 ± .1	7.8 ± 0.1	6.7 ± 0.4	4.2 ± 0.1	6.5 ± 0.4	6.6 ± 0.4	5.5 ± 0.6	4.4 ± 0.2
pron	868	149	22.9 ± < .1	17.5 ± 0.2	14.4 ± 0.3	12.1 ± < 0.1	9.9 ± 0.5	9.5 ± 0.1	8.5 ± 0.1	5.4 ± 0.1
num	719	1,059	72.6 ± .3	57.1 ± 0.7	55.2 ± 0.6	53.3 ± 0.5	58.4 ± 0.6	58.6 ± 2.5	48.3 ± 0.4	47.5 ± 1.0
punct	2,527	68	4.9 ± < .1	2.3 ± < 0.1	2.8 ± < 0.1	3.2 ± < 0.1	2.3 ± < 0.1	2.4 ± 0.1	2.6 ± 0.1	2.7 ± < 0.1
misc	151	183	7.0 ± < .1	4.6 ± 0.1	5.1 ± 0.5	5.2 ± 0.3	3.7 ± 0.1	3.5 ± 0.2	5.6 ± 0.2	4.0 ± 0.4
Table 10: PPL breakdown by UPOS classes of individual models (3-seed-averages), and the macro-average ± stdev over all SLR-combined models. We show token counts and observed vocabulary sizes for reference. mod—adjectives and adverbs; aux—auxiliary verbs; adp—adpositions; part—particles; sconj—subordinating conjunctions; cconj—coordinating conjunctions; det—determiners; pron—pronouns; num—numbers; punct—punctuation. Best results (within the variance) in each row are bolded.
Ablation	Applied in	UD	DM	PSD	PTB-phr	PTB-fxn	PTG	EDS
Full	32.7	31.4	30.7	29.8	29.0	26.8	24.7
× Labels	testing	+10.2	+4.7	+5.8	+63.0	+73.9	+18.7	+21.8
× Anchors	testing	+110.8	+34.8	+22.4	+177.0	+223.1	+103.4	+70.4
× Both	testing	+92.4	+33.4	+19.2	+168.9	+207.4	+80.1	+70.1
× Labels	training	+2.0	+1.4	+1.6	+8.4	+9.0	+5.3	+1.9
× Anchors	training	+13.0	+8.5	+6.3	+16.9	+17.5	+18.1	+12.8
× Both	training	+13.0	+7.8	+6.1	+17.3	+18.3	+18.5	+12.8
× Labels	both	+2.0	+1.3	+1.7	+8.4	+9.3	+5.4	+2.0
× Anchors	both	+13.1	+7.9	+6.0	+16.7	+17.5	+19.2	+14.0
× Both	both	+13.3	+7.3	+6.0	+16.8	+18.1	+19.4	+14.1
- SLR	both	+13.2	+14.5	+15.2	+16.1	+16.9	+19.1	+21.2
Table 11: Ablations measured in absolute perplexity difference (ΔPPL). Full and -SLR correspond, respectively, to table 2's rows 4 (DM) / 6 (PTB-phr) and row 2 (GPT-2 +Domain).