SlowGuess/NAACL_2022_OCR huggingface dataset:
The CLIR baseline for our experiments is a ColBERT model with an XLM-R PLM, which we first fine-tune with 17.5M NQ examples for one epoch and then 2.9M XOR-TyDi triples for five epochs. Our DR.DecR model is initialized with the parameter weights of the baseline, and is further fine-tuned using the two KD objectives. The monolingual teacher model—also a ColBERT model running on top of the pre-trained XLM-R—is trained with only the 17.5M NQ triples for one epoch.
A.2 Model Selection
All the models were trained with single Nvidia A100 GPU. The longest training time for a single model was less than 200 hours. Following are the final hyperparameter configurations of our different models. They were selected based on the respective validation sets performance.
A.3 Qualitative Analysis
To find out what exact weaknesses of the baseline model the proposed method helps to address, we examine thirty random zero-shot test examples where the baseline fails to retrieve the correct answer in the top 5k tokens, but DR.DECR succeeds within the top 3 passages. We show four examples in Table 6 with human translations of the original non-English questions. The vast majority of our observed cases are related to weak cross-lingual encoding on the baseline model's part, where at least one important non-English word/entity in the question seems to be incorrectly matched with a similar but different English entity in the passage (e.g., the name of a different place). For the Korean, Russian and Arabic queries in the table, we observe the presence of such topically similar entities (e.g., microwave  $\leftrightarrow$  gamma-ray, Germany  $\leftrightarrow$  places in North America). Much more rarely, we see cases similar to the Japanese query where the retrieved passage is completely off-topic.
<sup>4</sup>https://opus.nlpl.eu
Hyperparameter	Value
Standard ColBERT hyperparameters:
batch size	192
gradient accumulation steps	6
linear compression dim	128
query maxlen	32
document maxlen	180
Target domain supervision
Baseline model:
lr (NQ)	1.5e-6
lr (XOR)	6e-6
# Epochs (NQ)	1
# Epochs (XOR)	5
Knowledge distillation:
loss function (XOR)	KLDiv
loss function (Parallel corpus)	MSE
KD temperature (XOR)	2
lr (XOR)	6e-6
lr (Parallel corpus)	4.8e-5
# Epochs (XOR)	5
# Epochs (Parallel corpus)	2
Zero-shot
Baseline model:
lr (NQ)	1.5e-6
lr (MKQA)	6e-6
# Epochs (NQ)	1
# Epochs (MKQA)	1
Knowledge distillation:
loss function (MKQA)	KLDiv
loss function (Parallel corpus)	MSE
KD temperature (MKQA)	1
lr (MKQA)	3e-7
lr (Parallel corpus)	2.4e-5
# Epochs (MKQA)	1
# Epochs (Parallel corpus)	2
Table 5: Hyperparameters used in our test set runs.
Korean Query: 유주마이크로과 전쟁은 흠가 썸을 기재 갑고요?
English Translation: Who first discovered the cosmic microwave background?
Answer: Arno Penzias and Robert Wilson
Baseline Top Passage: History of gamma-ray burst research The history of gamma-ray began with the serendipitous detection of a gamma-ray burst (GRB) on July 2, 1967, by the U.S. Vela satellites. After these satellites detected fifteen other GRBs, Ray Klebesadel of the Los Alamos National Laboratory published the first paper on the subject, "Observations of Gamma-Ray Bursts of Cosmic Origin". As more and more research was done on these mysterious events, hundreds of models were developed in an attempt to explain their origins. Gamma-ray bursts were discovered in the late 1960s by the U.S. Vela nuclear test detection satellites. The Velas were built
DR. DECR Top Passage: project to date, it consists of more than 66 dishes and was built in an international collaboration by Europe, North America, East Asia and Chile. A major recent focus of microwave radio astronomy has been mapping the cosmic microwave background radiation (CMBR) discovered in 1964 by radio astronomers Arno Penzias and Robert Wilson. This faint background radiation, which fills the universe and is almost the same in all directions, is "relic radiation" from the Big Bang, and is one of the few sources of information about conditions in the early universe. Due to the expansion and thus cooling of the
Russian Query: B kaKOM cpaKeHHn yuAcTBoBaJ KpeiCep "Abpopa"?
English Translation: What battle did the cruiser Aurora take part in?
Answer: Battle of Tsushima
Baseline Top Passage: Battle of the Arar The Battle of the Arar was fought between the migrating tribes of the Helvetii, and four Roman legions (Legions VII, VIII, IX "Hispana" and X "Equestris"), under the command of Gaius Julius Caesar, in 58 BC. This was the first major battle of the Gallic Wars. The Helvetii were a tribe that originated from what is now Switzerland. Just prior to the battle with Caesar, they had commenced on a mass migration through Roman Gaul towards the Atlantic coast. At Geneva, the Romans destroyed the wooden bridge across the Rhone and constructed 19 mi of fortifications.
DR.DECR Top Passage: ordered back to Asia as part of the Russian 2nd Pacific Squadron, a collection formed from the Russian Baltic Fleet, under the command of Vice-Admiral Zinovy Rozhestvensky. On the way to the Far East, "Aurora" sustained light damage from confused friendly fire, which killed the ship's chaplain and a sailor, in the Dogger Bank incident. On 27 and 28 May 1905 "Aurora" took part in the Battle of Tsushima, along with the rest of the Russian squadron. During the battle her captain, Captain 1st rank Eugene R. Yegoryev, and 14 crewmen were killed. The executive officer, Captain of 2nd rank
Japanese Query: 抑さた記憶を提唱んだは誰
English Translation: Who proposed repressed memory
Answer: Sigmund Freud
Baseline Top Passage: 14 on the charts on another reissue at the end of 1979, and is now regarded as the Moody Blues signature song by British audiences. In the US, "Nights in White Satin" did not make the "Billboard" Hot 100 in 1968, although it reached No. 2 on re-release in 1972; "Tuesday Afternoon" was more successful on initial release stateside, peaking at No. 24 on the Billboard Hot 100. The 1968 follow-up LP, "In Search of the Lost Chord" included "Legend of a Mind", a song written by Ray Thomas in tribute to LSD guru Timothy Leary which encompassed a flute
DR. DECR Top Passage: literary example of traumatic memory that has been repressed by an otherwise healthy individual, and then recovered." Pope claimed that no entrant had satisfied the criteria. Ross Cheit, a political scientist at Brown University, cited Nina, a 1786 opera by the French composer Nicolas Dalayrac. The concept of repressed memory originated with Sigmund Freud in his 1896 essay "Zur Ätiologie der Hysterie" ("On the etiology of hysteria"). One of the studies published in his essay involved a young woman by the name of Anna O. Among her many ailments, she suffered from stiff paralysis on the right side of her
Arabic Query:
English Translation: What is the largest region of Germany?
Answer: Bavaria
Baseline Top Passage: the original name of Montana was adopted. Montana is one of the nine Mountain States, located in the north of the region known as the Western United States. It borders North Dakota and South Dakota to the east. Wyoming is to the south, Idaho is to the west and southwest, and three Canadian provinces, British Columbia, Alberta, and Saskatchewan, are to the north. With an area of , Montana is slightly larger than Japan. It is the fourth largest state in the United States after Alaska, Texas, and California; it is the largest landlocked U.S. state. The state's topography is
DR.DECR Top Passage: Bavaria (; German and Bavarian: "Bayern"; ), officially the Free State of Bavaria (German and Bavarian: "Freistaat Bayern"), is a landlocked federal state of Germany, occupying its southeastern corner. With an area of 70,550.19 square kilometres (27,200 sq mi), Bavaria is the largest German state by land area. Its territory comprises roughly a fifth of the total land area of Germany. With 13 million inhabitants, it is Germany's second-most-populous state after North Rhine-Westphalia. Bavaria's capital and largest city, Munich, is the third-largest city in Germany. The history of Bavaria stretches from its earliest settlement and formation as
Table 6: Examples of cases where the baseline model fails to retrieve a relevant passage but DR.DECR succeeds within top 3. We only show the top retrieval for each system. Most errors are related to potential word/entity mistranslations, the only exception being the Japanese query where the issue is a weaker understanding of the passage content.