SlowGuess/ACL_2020_OCR huggingface dataset:
Semantic probing We evaluate the representations on two semantic probing tasks, the Word in Context (WiC; Pilehvar and Camacho-Collados, 2019) and Stanford Contextual Word Similarity (SCWS; Huang et al., 2012) datasets. WiC is a binary classification task, which requires the model to determine if the occurrences of a word in two contexts refer to the same or different meanings. SCWS requires estimating the semantic similarity of word pairs that occur in context. For WiC, we train a linear classifier on top of the fixed sentence pair representation. For SCWS, we obtain the contextual representations of the target word in each sentence by averaging its constituent word pieces, and calculate their cosine similarity.
Syntactic probing We evaluate the same models in the syntactic probing dataset of Marvin and Linzen (2018) following the same setup as Goldberg (2019). Given minimally different pairs of English sentences, the task is to identify which of them is grammatical. Following Goldberg (2019), we feed each sentence into the model masking the word in which it differs from its pair, and pick the one to which the masked language model assigns the highest probability mass. Similar to Goldberg
11We leave out Thai, Hindi, Swahili, and Urdu as their corpus size is smaller than 3M.
10https://gengo.com
en	es	de	el	ru	tr	ar	vi	th	zh	hi
Paragraph	142.4	160.7	139.5	149.6	133.9	126.5	128.2	191.2	158.7	147.6	232.4
Question	11.5	13.4	11.0	11.7	10.0	9.8	10.7	14.8	11.5	10.5	18.7
Answer	3.1	3.6	3.0	3.3	3.1	3.1	3.1	4.5	4.1	3.5	5.6
Table 5: Average number of tokens for each language in XQuAD. The statistics were obtained using Jieba for Chinese and the Moses tokenizer for the rest of the languages.
Lang	Context paragraph w/ answer spans	Questions
en	The heat required for boiling the water and supplying the steam can be derived from various sources, most commonly from [burning combustible materials]1 with an appropriate supply of air in a closed space (called vari-ously [combustion chamber]2, firebox). In some cases the heat source is a nuclear reactor, geothermal energy, [solar]3 energy or waste heat from an internal combustion engine or industrial process. In the case of model or toy steam engines, the heat source can be an [electric]4 heating element.	1. What is the usual source of heat for boiling water in the steam engine?
2. Aside from firebox, what is another name for the space in which combustible material is burned in the engine?
3. Along with nuclear, geothermal and internal combustion engine waste heat, what sort of energy might supply the heat for a steam engine?
4. What type of heating element is often used in toy steam engines?
es	El calor necessario para hervir el agua y suministrar el vapor pueda derivarse de varias fuentes, generalmente de [la quema de materiales combustibles]1 con un sum-inistro adecuado de aire en un espacio cerrado (llamado de varias manos: [cármara de combustión]2, chime-nea...). En algunos casos la fuente de calor es un reactor nuclear, energia geotérmica, [energia solar]3 o calor residual de un motor de combustión interna o proceso industrial. En el caso de modelos o motores de vapor de juguete, la fuente de calor puede ser un calentador [eléctrico]4.	1. ¿Cuál es la fuente de calor habitual para hacer hervir el agua en la这其中aquina de vape?
2. Aparte de cármara de combustión, ¿qué除外 nombre que se le da al espacio en el que se quema el material combustible en el motor?
3. Junto con el calor residual de la energia nuclear, geotérmica y de los motores de combustión interna, ¿qué tipo de energia podía suministrar el calor para una这其中aquina de vape?
4. ¿Qué tipo de elemento calefactor se usa a bajo en las这其中aquinas de vape de juguete?
zh	让水沸腾以提供蒸汽所需热量有多种来源,最常见的是在封闭空间(别称有[燃烧室]2、火箱)中供应适量空气来[燃烧可燃材料]1。在某些情况下,热源是核反应堆、地热能、[太阳能]3或来自内燃机或工业过程的废气。如果是模型或玩具蒸汽发动机,还可以将[电]4加热元件作为热源。	1. 蒸汽机中让水沸腾的常用热源是什么?
2. 除了火箱之外,发动机内燃烧可燃材料的空间的别名是什么?
3. 除了核能、地热能和内燃机废气以外,还有什么热源可以为蒸汽机供能?
4. 玩具蒸汽机通常使用什么类型的加热元件?
Table 6: An example from XQuAD. The full dataset consists of 240 such parallel instances in 11 languages.
(2019), we discard all sentence pairs from the Marvin and Linzen (2018) dataset that differ in more than one subword token. Table 10 reports the resulting coverage split into different categories, and we show the full results in Table 11.
MLDoc	PAWS-X
en	fr	es	de	ru	zh	avg	en	fr	es	de	zh	avg
CLWE	300d ident	93.1	85.2	74.8	86.5	67.4	72.7	79.9	92.8	83.9	84.7	81.1	72.9	83.1
300d unsup	93.1	85.0	75.0	86.1	68.8	76.0	80.7	92.8	83.9	84.2	81.3	73.5	83.1
768d ident	94.7	87.3	77.0	88.7	67.6	78.3	82.3	92.8	85.2	85.5	81.6	72.5	83.5
768d unsup	94.7	87.5	76.9	88.1	67.6	72.7	81.2	92.8	84.3	85.5	81.8	72.1	83.3
JOINT	32k voc	92.6	81.7	75.8	85.4	71.5	66.6	78.9	91.9	83.8	83.3	82.6	75.8	83.5
64k voc	92.8	80.8	75.9	84.4	67.4	64.8	77.7	93.7	86.9	87.8	85.8	80.1	86.8
MULTI	100k voc	92.2	74.0	77.2	86.1	66.8	63.8	76.7	93.1	85.9	86.5	84.1	76.3	85.2
200k voc	91.9	82.1	80.9	89.3	71.8	66.2	80.4	93.8	87.7	87.5	87.3	78.8	87.0
Table 7: MLDoc and PAWS-X results (accuracy) for all CLWE and JOINTMULTI variants.
en	es	de	el	ru	tr	ar	vi	th	zh	hi	avg
CLWE	300d ident	72.5	39.7	33.6	23.5	29.9	11.8	18.5	16.1	16.5	17.9	10.0	26.4
300d unsup	72.5	39.2	34.5	24.8	30.4	12.2	14.7	6.5	16.0	16.1	10.4	25.2
768d ident	73.1	40.6	32.9	20.1	30.7	10.8	14.2	11.8	12.3	14.0	9.1	24.5
768d unsup	73.1	41.5	31.8	21.0	31.0	12.1	14.1	10.5	10.0	13.2	10.2	24.4
JOINT MULTI	32k voc	68.3	41.3	44.3	31.8	45.0	28.5	36.2	36.9	39.2	40.1	27.5	39.9
64k voc	71.3	48.2	49.9	40.2	50.9	33.7	41.5	45.0	43.7	36.9	36.8	45.3
100k voc	71.5	49.8	51.2	41.1	51.8	33.0	43.7	45.3	44.5	40.8	36.6	46.3
200k voc	72.1	55.3	55.2	48.0	52.7	40.1	46.6	47.6	45.8	38.5	42.3	49.5
JOINT PAIR	Joint voc	71.7	47.8	57.6	38.2	53.4	35.0	47.4	49.7	44.3	47.1	38.8	48.3
Disjoint voc	72.2	52.5	56.5	47.8	55.0	43.7	49.0	49.2	43.9	50.0	39.1	50.8
MONOTRANS	Subword emb + pos emb	72.3	47.4	42.4	43.3	46.4	30.1	42.6	45.1	39.0	39.0	32.4	43.6
+ noisig	69.6	51.2	52.4	50.2	51.0	6.9	43.0	46.3	46.4	48.1	10.7	43.2
+ adapters	69.6	51.4	51.4	50.2	51.4	44.5	48.8	47.7	45.6	49.2	45.1	50.5
Table 8: XQuAD results (exact match).
en	es	de	el	ru	tr	ar	vi	th	zh	hi	avg
CLWE	300d ident	84.1	56.8	51.3	43.4	47.4	25.5	35.5	34.5	28.7	25.3	22.1	41.3
300d unsup	84.1	56.8	51.8	42.7	48.5	24.4	31.5	20.5	29.8	26.6	23.1	40.0
768d ident	84.2	58.0	51.2	41.1	48.3	24.2	32.8	29.7	23.8	19.9	21.7	39.5
768d unsup	84.2	58.9	50.3	41.0	48.5	25.8	31.3	27.3	24.4	20.9	21.6	39.5
JOINT MULTI	32k voc	79.3	59.5	60.3	49.6	59.7	42.9	52.3	53.6	49.3	50.2	42.3	54.5
64k voc	82.3	66.5	67.1	60.9	67.0	50.3	59.4	62.9	55.1	49.2	52.2	61.2
100k voc	82.6	68.9	68.9	61.0	67.8	48.1	62.1	65.6	57.0	52.3	53.5	62.5
200k voc	82.7	74.3	71.3	67.1	70.2	56.6	64.8	67.6	58.6	51.5	58.3	65.7
Table 9: XQuAD results (F1) for all CLWE and JOINTMULTI variants.
4636
coverage
Subject-verb agreement
Simple	80 / 140 (57.1%)
In a sentential complement	960 / 1680 (57.1%)
Short VP coordination	480 / 840 (57.1%)
Long VP coordination	320 / 400 (80.0%)
Across a prepositional phrase	15200 / 22400 (67.9%)
Across a subject relative clause	6400 / 11200 (57.1%)
Across an object relative clause	17600 / 22400 (78.6%)
Across an object relative (no that)	17600 / 22400 (78.6%)
In an object relative clause	5600 / 22400 (25.0%)
In an object relative (no that)	5600 / 22400 (25.0%)
Reflexive anaphora
Simple	280 / 280 (100.0%)
In a sentential complement	3360 / 3360 (100.0%)
Across a relative clause	22400 / 22400 (100.0%)
Table 10: Coverage of our systems for the syntactic probing dataset. We report the number of pairs in the original dataset by Marvin and Linzen (2018), those covered by the vocabulary of our systems and thus used in our experiments, and the corresponding percentage.
mono	xx→en aligned
en	en	fr	es	de	el	bg	ru	tr	ar	vi	zh	avg
Subject-verb agreement
Simple	91.2	76.2	90.0	93.8	56.2	97.5	56.2	78.8	72.5	67.5	81.2	71.2	76.5
In a sentential complement	99.0	65.7	94.0	92.1	62.7	98.3	80.7	74.1	89.7	71.5	78.9	79.6	80.7
Short VP coordination	100.0	64.8	66.9	69.8	64.4	77.9	60.2	88.8	76.7	73.3	62.7	64.4	70.0
Long VP coordination	96.2	58.8	53.4	60.0	67.5	62.5	59.4	92.8	62.8	75.3	62.5	64.4	65.4
Across a prepositional phrase	89.7	56.9	54.6	52.8	53.4	53.4	54.6	79.6	54.3	59.9	57.9	56.5	57.6
Across a subject relative clause	91.6	49.9	51.9	48.3	52.0	53.2	56.2	78.1	48.6	58.9	55.4	52.3	55.0
Across an object relative clause	79.2	52.9	56.2	53.3	52.4	56.6	57.0	63.1	52.3	59.0	54.9	54.5	55.7
Across an object relative (no that)	77.1	54.1	55.9	55.9	53.1	56.2	59.7	63.3	53.1	54.9	55.9	56.8	56.3
In an object relative clause	74.6	50.6	59.9	66.4	59.4	61.1	49.8	60.4	42.6	45.3	56.9	56.3	55.3
In an object relative (no that)	66.6	51.7	57.1	64.9	54.9	59.4	49.9	57.0	43.7	46.6	54.9	55.4	54.1
Macro-average	86.5	58.2	64.0	65.7	57.6	67.6	58.4	73.6	59.6	61.2	62.1	61.1	62.7
Reflexive anaphora
Simple	90.0	69.3	63.6	67.9	55.0	69.3	56.4	89.3	75.0	87.1	58.6	60.7	68.4
In a sentential complement	82.0	56.3	63.9	73.2	52.7	65.7	59.1	70.8	71.7	84.5	59.8	53.9	64.7
Across a relative clause	65.6	55.0	54.5	58.6	52.3	55.8	52.5	66.1	61.4	73.3	56.9	50.9	57.9
Macro-average	79.2	60.2	60.7	66.6	53.3	63.6	56.0	75.4	69.4	81.6	58.4	55.2	63.7
Table 11: Complete syntactic probing results (accuracy) of a monolingual model and monolingual models transferred to English on the syntactic evaluation test set (Marvin and Linzen, 2018).