Venue: NAACL 2022

# Learning Cross-Lingual IR from an English Retriever

 Yulong Li\({}^{\dagger}\), Martin Franz\({}^{\ddagger*}\), Md Arafat Sultan\({}^{\ddagger*}\), Bhavani Iyer\({}^{\ddagger}\), Young-Suk Lee\({}^{\ddagger}\)  and Avirup Sil\({}^{\ddagger}\)

\({}^{\dagger}\)IBM Research, \({}^{\ddagger}\)IBM Research AI

{yulongl, franzm, bsiyer, ysuklee, avi}@us.ibm.com

arafat.sultan@ibm.com

 Equal contribution.

###### Abstract

We present Dr.Derc**(Dense Retrieval with Distillation-Enhanced Cross-Lingual Representation), a new cross-lingual information retrieval (clir) system trained using multi-stage knowledge distillation (kd). The teacher of Dr.Decr relies on a highly effective but computationally expensive two-stage inference process consisting of query translation and monolingual ir, while the student, Dr.Decr, executes a single clir step. We teach Dr.Decr powerful multilingual representations as well as clir by optimizing two corresponding kd objectives. Learning useful representations of non-English text from an English-only retriever is accomplished through a cross-lingual token alignment algorithm that relies on the representation capabilities of the underlying multilingual encoders. In both in-domain and zero-shot out-of-domain evaluation, Dr.Decr demonstrates far superior accuracy over direct fine-tuning with labeled clir data. It is also the best single-model retriever on the XOR-TyDi benchmark at the time of this writing.

## 1 Introduction

Multilingual models are critical for the democratization of AI. Cross-lingual information retrieval (clir) Braschler et al. (1999); Shakery and Zhai (2013); Jiang et al. (2020); Asai et al. (2021), for example, can find relevant text in a high-resource language such as English even when the query is posed in a different, possibly low-resource, language. In this work, we develop useful clir models for this constrained, yet important, setting where a retrieval corpus is available only in a single high-resource language (English in our experiments).

A straightforward solution to this problem could use machine translation mt to translate the query into English, and then perform English irAsai et al. (2021). While such a two-stage process can produce reasonably accurate predictions, an alternative end-to-end approach that can tackle the problem purely cross-lingually, _i.e._, without involving mt for inference, would clearly be more efficient and cost-effective. Pre-trained multilingual masked language models (plms) such as multilingual bertDevlin et al. (2019) or xlm-robertaxlm-r) Conneau et al. (2020) can provide the foundation for such a one-step solution, as simply fine-tuning a plm with labeled clir data would yield a cross-lingual retriever Asai et al. (2021).

Here we first run an empirical evaluation of these two approaches on a public clir benchmark Asai et al. (2021), which involves both in-domain and zero-shot out-of-domain tests. We use ColbertKhattab and Zaharia (2020); Khattab et al. (2021)--a state-of-the-art (sota) neural ir model that has been shown to outperform other recent methods such as dpr Karpukhin et al. (2020)--as our ir architecture and xlm-r as the underlying plm for both methods (SS2). Results indicate that the mt-based solution can be vastly more effective than direct en + clir fine-tuning, with observed differences of 22.2-28.6 Recall@5k-tokens (SS3). Crucially, the modular design of the former allows it to leverage additional English-only training data for its ir component, providing significant boosts to its performance.

The above findings lead naturally to the central research question of this paper: Can a high-performance clir model be trained that can operate without having to rely on mt? To answer the question, instead of viewing the mt-based approach as a competing one, we propose to leverage its strength via knowledge distillation (kd) into an end-to-end clir model, which we call Dr.Decr**(Dense Rretrieval with Distillation-Enhanced Cross-lingual Representation). kd Hinton et al. (2014) is a powerful supervision technique typically used to distill the knowledge of a large

_teacher_ model about some task into a smaller _student_ model Mukherjee and Awadallah (2020); Turc et al. (2020). Here we propose to use it in a slightly different context, where the teacher and the student retriever are identical in size, but the former has superior performance simply due to utilizing mt output and consequently operating in a high-resource and low-difficulty monolingual environment.

We run two independent kd operations (SS2.2). One directly optimizes an ir objective by utilizing labeled clir data: parallel questions (English and non-English) and corresponding relevant and non-relevant English passages. The teacher and the student, Dr.Decr, are shown the English and non-English versions of the questions, respectively; the training objective is for Dr.Decr to match the soft query-passage relevance predictions of the teacher. The second kd task is representation learning from parallel text, where Dr.Decr learns to encode a non-English text in a way that matches the teacher's encoding of the aligned English text, _at the token level_. The cross-lingual token alignments needed to create the training data for this task are generated using a greedy alignment process, which exploits the multilingual representation capabilities of the underlying plm encoders.

In our evaluation on the XOR-TyDi benchmark Asai et al. (2021), Dr.Decr outperforms the fine-tuned Colbert baseline by 25.4 (in-domain) and 14.9 (zero-shot) Recall@5k-tokens, recovering much of the performance loss from the mt-based solution. It is also the best single-model ir system on the XOR-TyDi leaderboard1 at the time of this writing. Ablation studies show that each of our two kd processes contribute significantly towards the final performance of Dr.Decr.

Footnote 1: [https://nlp.cs.washington.edu/xorqa/](https://nlp.cs.washington.edu/xorqa/)

Our contributions can be summarized as follows: **(1)** We present an empirical study of the effectiveness of a sota ir method (Colbert) on cross-lingual ir with and without mt, **(2)** We propose a novel end-to-end cross-lingual solution that uses knowledge distillation to learn both improved text representation and retrieval, **(3)** We demonstrate with a new cross-lingual alignment algorithm that distillation using parallel text can strongly augment cross-lingual ir training, and **(4)** We achieve new single-model sota results on XOR-TyDi.

## 2 Method

Here we first describe our base ir architecture (Colbert) and then the proposed kd-based cross-lingual training algorithms.

### The Colbert Model

ColbertKhattab and Zaharia (2020) employs a transformer-based encoder to separately encode the input query and document, followed by a linear compression layer. Each training instance is a <\(q\), \(d^{+}\), \(d^{-}\)> triple, where \(q\) is a query, \(d^{+}\) is a positive (relevant) document and \(d^{-}\) is a negative (non-relevant) document. A relevance score \(S_{q,d}\) for the pair \((q,d)\) is first computed using Eq. 1, where \(d\in\{d^{+},d^{-}\}\) and \(E_{q_{i}}\) and \(E_{d_{j}}\) are the output embeddings of query token \(q_{i}\) and document token \(d_{j}\), respectively. For a given training triple, a cross-entropy loss is minimized for the softmax over \(S_{q,d^{+}}\) and \(S_{q,d^{-}}\).

\[S_{q,d}:=\sum_{i\in|q||}max_{j\in|d||}E_{q_{i}}\cdot E_{d_{j}}^{T} \tag{1}\]

For inference, the embeddings of all documents are calculated _a priori_, while the query embeddings and the relevance score are computed at runtime.

### Knowledge Distillation

Our teacher and Dr.Decr are both Colbert models that fine-tune the same underlying multilingual plm for ir. The teacher is first trained with all-English triples using the procedure of SS2.1. The goal of the subsequent kd training is to teach Dr.Decr to reproduce the behavior of this teacher when it sees non-English translations of the teacher's English questions.

We apply kd at two different stages of the Colbert workflow: (\(a\)) relevance score computation (\(S_{q,d}\) in Eq. 1), and (\(b\)) encoding (\(e.g.\), \(E_{q_{i}}\)). Figure 1 depicts (\(a\)) in detail, where training minimizes the KL divergence between the Dr.Decr's and the teacher's output softmax distributions (with temperature) over \(S_{q,d^{+}}\) and \(S_{q,d^{-}}\).

Labeled training data for Clir are scarce, whereas mt, being a more established area of research, has produced a large amount of parallel text over the years. We seek to exploit existing parallel corpora in our second kd training, where we teach Dr.Decr to compute representations of non-English texts that closely match the teacher's representations of aligned English texts. Importantly, since Colbert computes a single vector for each 

individual input token (_i.e._, a plm vocabulary item) and not for the entire input text, our algorithm must support distillation at the token level.

To achieve this, we design an unsupervised cross-lingual token alignment algorithm. Assuming \((ne_{1},...,ne_{S})\) to be the ordered tuple of tokens in a non-English text and \((e_{1},...,e_{T})\) the corresponding tuple from the parallel English text, each iteration of this algorithm greedily picks the next \((ne_{i},e_{j})\) pair with the highest cosine similarity of their output embeddings. Algorithm 1 implements this idea by repositioning the teacher's tokens so that they are position-wise aligned with the corresponding Dr.Dcer tokens. Note that the design choice of fine-tuning a common multilingual plm for the teacher and the Dr.Decr, even though the former is tasked with only handling English content, is key for this algorithm as it relies on the plms' multilingual representation capabilities. See Appendix A.1 for details on our parallel corpora used for training.

In addition to cross-lingual alignment, we also perform a similar kd procedure in which both the teacher and the Dr.Decr are shown the same English text. This step is useful because Colbert uses a shared encoder for the query and the document, necessitating a student that is able to effectively encode text from both English documents and non-English queries.

Using the alignment information, we train Dr.Decr by minimizing the Euclidean distance between its representation of a token (English or non-English) and the teacher's representation of the corresponding English token. Figure 2 shows the kd process for representation learning.

Figure 1: Relevance score distillation. The teacher is shown all-English triples while the student’s (Dr.Dcer) query input is non-English. Training minimizes the student’s KL divergence from the teacher’s output softmax distribution over \(S_{q,d^{+}}\) and \(S_{q,d^{-}}\) (\(\tau\) is the temperature).

Figure 2: Distillation for representation learning. The student (Dr.Decr) learns to encode both English and non-English tokens in context that matches the teacher’s output embeddings for corresponding English tokens.

 



## 3 Experiments

### Setup

Our primary clir dataset is XOR-TyDi Asai et al. (2021), which contains examples in seven typologically diverse languages: Arabic (Ar), Bengali (Bn), Finnish (Fi), Japanese (Ja), Korean (Ko), Russian (Ru) and Telugu (Te). For standard in-domain experiments, we use a train-dev-test split of this dataset. There are 2,113 questions in the test set. For zero-shot experiments, we use the MKQA Longpre et al. (2020) dataset for training and validation, and the following shared languages in the XOR-TyDi test set for evaluation: Ar, Fi, Ja, Ko and Ru. Both training sets contain English questions and their human translations in the other languages, their short answers and corresponding relevant (positive) and non-relevant (negative) Wikipedia snippets. Additionally, we use training examples from the Natural Questions (NQ) dataset Kwiatkowski et al. (2019) for English pretraining of the baseline model. Further details on data pre-processing and the final training sets are provided in Appendix A.1.

The clir baseline used in our experiments is Colbert with an underlying xlm-r plm, which we iteratively fine-tune first on English and then on cross-lingual ir triples for optimal performance. Our Dr.Decr model is initialized with the parameter weights of this baseline, and is further fine-tuned using the two kd objectives. The kd teacher is a Colbert model fine-tuned with only English triples, as stated before. During evaluation, it is given machine-translated questions that come with the XOR-TyDi dataset. Appendices A.1 and A.2 contain additional details on the supervision of these models and the optimal hyperparameter configurations.

We evaluate using Recall at \(t\) tokens for \(t\in\{2000,5000\}\), _i.e._, R@2kt and R@5kt Asai et al. (2021), which compute the fraction of questions for which the ground truth short answer is contained within the top \(t\) tokens of the retrieved passages.

### Evaluation

Table 1 compares the performance of our different models. First, looking at the R@5kt results, we observe that pre-training the baseline model with English ir triples from the NQ train set (rows 2, 6) substantially boosts its performance in both in-domain and zero-shot settings. However, it still underperforms the mt + English ir pipeline (rows 3, 7) by 28.6 and 22.2 points, respectively. By distilling first with the parallel corpus (for representation learning) and then with the ir triples (for clir), Dr.Decr (row 4) yields an improvement of 25.4 points over the baseline model in in-domain evaluation, which, quite impressively, is within 3.2 points

\begin{table}
\begin{tabular}{l|c c}
**Language** & **Baseline** & **Dr.Decr** \\ \hline \multicolumn{3}{l}{_With target domain supervision:_} \\ \hline Te & 63.0 & 83.2 \\ Bn & 53.3 & 85.9 \\ Fi & 49.4 & 69.4 \\ Ja & 39.4 & 65.1 \\ Ko & 44.9 & 68.8 \\ Ru & 39.2 & 68.8 \\ Ar & 44.3 & 70.2 \\ \hline
**Avg** & **47.7** & **73.1** \\ \hline \multicolumn{3}{l}{_Zero-shot:_} \\ \hline Fi & 55.4 & 66.9 \\ Ja & 44.0 & 58.5 \\ Ko & 48.4 & 62.8 \\ Ru & 41.4 & 57.8 \\ Ar & 45.3 & 61.8 \\ \hline
**Avg** & **46.9** & **61.8** \\ \end{tabular}
\end{table}
Table 2: R@5kt scores for in-domain and zero-shot evaluation on individual languages. Baseline for the target domain experiment: Colberten+cl: _f_f_(Nq) \(\rightarrow\)_f_(xor). Baseline for the zero-shot experiment: Colbertén+cl: _f_f_(Nq) \(\rightarrow\)_f_(mkqa)

of the teacher's score. A sizable gain of 14.9 points is also observed in zero-shot evaluation (row 8). Finally, the R@2kt numbers show a very similar pattern.

Table 2 shows the performance (R@5kt) of Dr.Decr and the baseline on each individual language: the former outperforms the latter both with and without target domain supervision, yielding large gains across all languages. These results demonstrate the robustness of our approach, which stems from combining the individual strengths of mt, English ir and kd in a single model.

### Leaderboard Submission

The Dr.Decr model trained on the XOR-TyDi training set, shown in Table 1 row 4, is the best single-model retriever on the XOR-TyDi leaderboard2 at the time of this writing. Since our parallel corpus extraction process relies on in-house source code that is not publicly available, we submitted to the "Systems using External APIs" category. Crucially, all other submitted systems under the External APIs category rely on mt at decoding time, avoiding which is one of the primary goals of our work. We also created parallel corpora purely from public available sources.3 Our model distilled with these instances also achieved top position on the white-box systems leaderboard of XOR-TyDi.

Footnote 2: [https://nlp.cs.washington.edu/xorqa/](https://nlp.cs.washington.edu/xorqa/)

Footnote 3: [https://opus.nlpl.eu](https://opus.nlpl.eu)

### Ablation Study

We experiment with two more student models, one distilled with only clir examples and the other with only the parallel corpus. As the results in Table 3 show, each has a substantial impact on system performance. Interestingly, although the parallel corpus does not provide any ir signal, it contributes more to the model's accuracy. These results also confirm that our cross-lingual alignment algorithm does indeed produce useful alignments.

## 4 Conclusion

We train highly effective end-to-end cross-lingual ir models by distilling the knowledge of an English retriever. We propose separate processes to teach ir and multilingual text representations, and present for the latter a cross-lingual alignment algorithm that only relies on the underlying masked language model's multilingual representation capabilities. Supervised and zero-shot evaluations show that our model recovers much of the performance lost due to operating in an efficient cross-lingual mode. Our kd-based method also yields new single-model sota results on the XOR-TyDi benchmark. Future work will explore ir on unseen languages and evaluation on additional datasets.

## 5 Ethics

### Limitations

We show the effectiveness of multi-stage knowledge distillation and cross-lingual token alignment in training a cross-lingual information retrieval system. We believe that it can be transferred to more datasets and languages, but here we only show proof of concept for the XOR-TyDi and MKQA datasets and the seven languages mentioned in the paper.

### Risks

The intent of this work is to develop a new method for high-performance cross-lingual information retrieval. It is possible that a malicious user could try to attack the system by providing poor or offensive training data. We do not support it being used in such a manner. The risks of our system are the same as other nlp systems and we do not believe we introduce any additional risk.

\begin{table}
\begin{tabular}{c|c c} \hline _System_ & **R@5kt** & **R@2kt** \\ \hline _With target domain supervision:_ & & \\ \hline Dr.Decr:Colberten+CL \(\rightarrow\) KDDC \(\rightarrow\)KDXOR & 73.1 & 66.0 \\ Colberten+CL \(\rightarrow\) KDPC & 68.6 & 60.6 \\ Colberten+CL \(\rightarrow\) KDXR & 63.6 & 56.6 \\ Colberten+CL & 47.7 & 38.1 \\ \hline _Zero-shot:_ & & \\ \hline Dr.Decr:Colberten+CL \(\rightarrow\) KDDC \(\rightarrow\) KDMKQA & 61.8 & 54.3 \\ Colberten+CL \(\rightarrow\) KDFC & 55.9 & 47.7 \\ Colberten+CL \(\rightarrow\) KDMKAQA & 49.3 & 40.9 \\ Colberten+CL & 46.9 & 38.7 \\ \hline \end{tabular}
\end{table}
Table 3: Results of the ablation study. kd with parallel corpus (KDPC) and ir triples (KDXOR) both play key roles in our Dr.Decr model. Interestingly, the former has a greater impact on the model’s performance.

 



## Acknowledgements

We thank Graeme Blackwood and Christoph Tillmann for providing the in-house parallel corpora. We also thank Akari Asai for her help submitting Dr.Derc to the XOR-TyDi leaderboard.

## References

* A. Asai, J. Kasai, J. Clark, K. Lee, E. Choi, and H. Hajishirzi (2021)XOR QA: cross-lingual open-retrieval question answering. In NAACL, Cited by: SS3.4.
* A. Asai, X. Yu, J. Kasai, and H. Hajishirzi (2021)One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval. In NeurIPS, External Links: Link Cited by: SS5.
* M. Braschler, J. Krause, C. Peters, and P. Schauble (1999)Cross-language information retrieval (CLIR) track overview. In TREC, Cited by: SS6.
* A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020)Unsupervised cross-lingual representation learning at scale. In ACL, Cited by: SS1, SS2.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, Cited by: SS1, SS2.
* G. Hinton, O. Vinyals, and J. Dean (2014)Distilling the knowledge in a neural network. In NeurIPS Deep Learning Workshop, Cited by: SS1, SS2.
* Z. Jiang, A. El-Jaroudi, W. Hartmann, D. Karakos, and L. Zhao (2020)Cross-lingual Information Retrieval with BERT. arXiv preprint arXiv:2004.13005. Cited by: SS1, SS2.
* V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020)Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS2.
* O. Khattab, C. Potts, and M. Zaharia (2021)Relevance-guided supervision for openqa with colbert. Transactions of the ACL. Cited by: SS1, SS2.
* O. Khattab and M. Zaharia (2020)Colbert: efficient and effective passage search via contextualized late interaction over bert. In SIGIR, Cited by: SS1, SS2.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019)Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics. Cited by: SS1, SS2.
* S. Longpre, Y. Lu, and J. Daiber (2020)MKQA: a linguistically diverse benchmark for multilingual open domain question answering. arXiv preprint arXiv:2007.15207. Cited by: SS1, SS2.
* S. Mukherjee and A. Awadallah (2020)XTremeDistil: multi-stage distillation for massive multilingual models. In ACL, Cited by: SS1, SS2.
* A. Shakery and C. Zhai (2013)Leveraging comparable corpora for cross-lingual information retrieval in resource-lean language pairs. Information retrieval16 (1), pp. 1-29. Cited by: SS1, SS2.
* I. Turc, M. Chang, K. Lee, and K. Toutanova (2020)Well-Read students learn better: on the importance of pre-training compact models. In ICLR, Cited by: SS1, SS2.



Appendix

### Data Pre-processing

The official XOR-TyDi training set consists of 15,221 natural language queries, their short answers, and examples of corresponding relevant (positive) and non-relevant (negative) Wikipedia snippets. For most queries, there are one positive and three negative examples. We remove the 1,699 (11%) questions that have no answers in the dataset. A random selection of 90% of the remaining examples is used for training and the rest for validation.

Following the original XOR-TyDi process, we also obtain additional training examples by running BM25-based retrieval against a Wikipedia corpus and using answer string match as the relevance criterion. These examples are added to the original set to obtain three positive and 100 negative examples per query. As the blind test set for final evaluation, we use the 2,113 questions in the official XOR-TyDi dev set.

Our monolingual (English) training data containing about 17.5m triples are derived from the third fine-tuning round (Colbert-QA3) of Colbert relevance-guided supervision Khattab et al. (2021) with NQ examples Kwiatkowski et al. (2019).

The parallel corpus used in our kd experiments (Table 1) for representation learning is constructed from three different sources: (1) an in-house crawl of Korean, (2) LDC releases (Arabic), and (3) OPUS.4 The corpus has a total of 6.9M passage pairs which include .9M pairs in Telugu and 1M pairs in each of the other six languages. The parallel corpus used in our white-box system was created purely from OPUS. The statistics and sources are shown in the table below.

Footnote 4: [https://opus.nlpl.eu](https://opus.nlpl.eu)

For zero-shot experiments, the training examples are derived from MKQA Longpre et al. (2020), which consists of 10k queries selected from NQ, human translated into 25 additional languages, five of which overlap with XOR-TyDI: Ar, Fi, Ja, Ko and Ru. We construct training data (triples) from 2,037 queries translated into these five languages for which there are corresponding positive and negative passages in the NQ dataset. For each of the five languages, there are 519k triples for a total of 2.6M triples. We set aside 200 queries translated into the 5 languages for a total of 1,000 queries as a development set. We remove all MKQA queries from the NQ training data for these experiments.

The clir baseline for our experiments is a Colbert model with an xlm-r plm, which we first fine-tune with 17.5M NQ examples for one epoch and then 2.9M XOR-TyDi triples for five epochs. Our Dr.Decr model is initialized with the parameter weights of the baseline, and is further fine-tuned using the two kd objectives. The monolingual teacher model--also a Colbert model running on top of the pre-trained xlm-r--is trained with only the 17.5M NQ triples for one epoch.

### Model Selection

All the models were trained with single Nvidia A100 GPU. The longest training time for a single model was less than 200 hours. Following are the final hyperparameter configurations of our different models. They were selected based on the respective validation sets performance.

### Qualitative Analysis

To find out what exact weaknesses of the baseline model the proposed method helps to address, we examine thirty random zero-shot test examples where the baseline fails to retrieve the correct answer in the top 5k tokens, but Dr.Decr succeeds within the top 3 passages. We show four examples in Table 6 with human translations of the original non-English questions. The vast majority of our observed cases are related to weak cross-lingual encoding on the baseline model's part, where at least one important non-English word/entity in the question seems to be incorrectly matched with a similar but different English entity in the passage (e.g., the name of a different place). For the Korean, Russian and Arabic queries in the table, we observe the presence of such topically similar entities (_e.g._, _microwave \(\leftrightarrow\) gamma-ray_, _Germany \(\leftrightarrow\)_ places in North America). Much more rarely, we see cases similar to the Japanese query where the retrieved passage is completely off-topic.

\begin{table}
\begin{tabular}{c c c}
**Language** & **Amount (M)** & **Source** \\ \hline Ja & 0.9 & WikiMatrix \\ Ru & 1.7 & WikiMatrix \\ Ar & 1.0 & WikidMatrix \\ Te & 0.7 & WikiMatrix + CCAligned \\ Bn & 1.3 & WikidMatrix + CCMatrix \\ Fi & 1.4 & WikidMatrix + CCMatrix \\ Ko & 1.3 & WikidMatrix + CCMatrix \\ \hline \end{tabular}
\end{table}
Table 4: Statistic of parallel corpus used in the XOR-TyDi white-box system.

 

* [150] M. Cacciari, G. P. Salam, and A. Zisserman. The performance of the \({}^{26}\)Li-based multiprocessor system for large-scale data storage systems. In _Proceedings of the IEEE International Conference on Computer Architecture (ICCV)_, pages 378-394, 2006.
* [151] J. L. Krizhevsky, I. Sutskever, and R. E. Hinton. Learning to recognize high dimensional objects with low computational complexity. _Journal of Machine Learning Research_, 1:1-10, 2006.
* [152] S. Bjornsson, D. W. Johnson, and T. H. Lee. Fast object detection using convolutional neural networks. In _Advances in Neural Information Processing Systems_, volume 1, pages 1-10, 2006.
* [153] Y. Bengio et al., "Deep learning with deep convolutional nets. In _NIPS conference on computer vision_, pages 1338-1340, 2015.
* [154] X. Chen, Q. Li, and F. Zhang. Deep residual learning for image recognition. In _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1013-1018, 2015.
* [155] Y. Deng, J. Dong, and J. Sun. Faster rgb: a unified framework for fast object detection. In _International Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1013-1018, 2015.
* [156] Y. Deng, J. Jia, and J. Sun. Faster rgb: a simple and efficient way to prevent it from overfitting. In _Computer Vision (ICCV)_, pages 1013-1018, 2015.
* [157] Y. Deng, J. Jia, and J. Sun. Faster rgb: a simple and robust approach. In _ECCV_, pages 1090-1093, 2016.

[MISSING_PAGE_POST]

 

_Korean Query:_

_English Translation:_ Who first discovered the cosmic microwave background?

_Answer:_ Arno Penzias and Robert Wilson **Baseline Top Passage:** History of gamma-ray burst research The history of gamma-ray began with the serendipitous detection of a gamma-ray burst (GRB) on July 2, 1967, by the U.S. Vela satellites. After these satellites detected fifteen other GRBs, Ray Klebesadel of the Los Alamos National Laboratory published the first paper on the subject, "Observations of Gamma-Ray Bursts of Cosmic Origin". As more and more research was done on these mysterious events, hundreds of models were developed in an attempt to explain their origins. Gamma-ray bursts were discovered in the late 1960s by the U.S. Vela nuclear test detection satellites. The Velas were built

**Dr.Decr_Top Passage:** project to date, it consists of more than 66 dishes and was built in an international collaboration by Europe, North America, East Asia and Chile. A major recent focus of microwave radio astronomy has been mapping the cosmic microwave background radiation (CMBR) discovered in 1964 by radio astronomers Amo Penzias and Robert Wilson. This faint background radiation, which fills the universe and is almost the same in all directions, is "relic radiation" from the Big Bang, and is one of the few sources of information about conditions in the early universe. Due to the expansion and thus cooling of the

_Russian Query:_ B xaxion grazzhenhui quasetobai krebecp Anopopa\({}^{*}\)?

_English Translation:_ What battle did the cruiser Aurora take part in?

_Answer:_ Battle of Tsushima

_Baseline Top Passage:_ Battle of the Arar The Battle of the Arar was fought between the migrating tribes of the Helvetii, and four Roman legions (Legions VII, VIII, IX "Hispana" and X "Equests"), under the command of Gaus Julius Caesar, in 58 BC. This was the first major battle of the Galile Wars. The Helvetii were a tribe that originated from what is now Switzerland. Just prior to the battle with Caesar, they had commenced on a mass migration through Roman Gaul towards the Atlantic coast. At Geneva, the Romans destroyed the wooden bridge across the Rhone and constructed 19 mi of fortifications.

**Dr.Decr_Top_Passage:** ordered back to Asia as part of the Russian 2nd Pacific Squadron, a collection formed from the Russian Baltic Fleet, under the command of Vice-Admiral Zinovy Rozhetsvensky. On the way to the Far East, "Aurora" sustained light damage from confused friendly fire, which killed the ship's chaplain and a sailor, in the Dogger Bank incident. On 27 and 28 May 1905 "Aurora" took part in the Battle of Tsushima, along with the rest of the Russian squadron. During the battle her captain, Captain 1st rank Eugene R. Yegoryev, and 14 crewmen were killed. The executive officer, Captain of 2nd rank

**Japanese Query:**

_English Translation:_ Who proposed repressed memory

_Answer:_ Sigmund Freud

_Baseline Top Passage:_ 14 on the charts on another reissue at the end of 1979, and is now regarded as the Moody Blues signature song by British audiences. In the US, "Nights in White Satin" did not make the "Billboard" Hot 100 in 1968, although it reached No. 2 on re-release in 1972; "Tuesday Afternoon" was more successful on initial release stateside, peaking at No. 24 on the Billboard Hot 100. The 1968 follow-up LP, "In Search of the Lost Chord" included "Legend of a Mind", a song written by Ray Thomas in tribute to LSD guru Timothy Leary which encompassed a flute

**Dr.Decr_Top_Passage:** literary example of traumatic memory that has been repressed by an otherwise healthy individual, and then recovered." Pope claimed that no entrant had satisfied the criteria. Ross Cheit, a political scientist at Brown University, cited Nina, a 1786 opera by the French composer Nicolas Dalayrac. The concept of repressed memory originated with Sigmund Freud in his 1896 essay "Zur Atiologie der Hysterie" ("On the etiology of hysteria"). One of the studies published in his essay involved a young woman by the name of Anna O. Among her many aliments, she suffered from stiff paralysis on the right side of her

_Arabic Querry:_

_English Translation:_ What is the largest region of Germany?

_Answer:_ Bavaria

_Baseline Top Passage:_ the original name of Montana was adopted. Montana is one of the nine Mountain States, located in the north of the region known as the Western United States. It borders North Dakota and South Dakota to the east. Wyoming is to the south, Idaho is to the west and southwest, and three Canadian provinces, British Columbia, Alberta, and Saskatchewanwan, are to the north. With an area of Montana is slightly larger than Japan. It is the fourth largest state in the United States after Alaska, Texas, and California; it is the largest landlocked U.S. state. The state's topography is

**Dr.Decr_Top_Passage:** Bavaria (; German and Bavarian: "Bayern" ; ), officially the Free State of Bavaria (German and Bavarian: "Freistaat Bayern" ), is a landlocked federal state of Germany, occupying its southeastern corner. With an area of 70,550.19 square kilometres (27,200 sq mj), Bavaria is the largest German state by land area. Its territory comprises roughly a fifth of the total land area of Germany. With 13 million inhabitants, it is Germany’s second-most-populous state after North Rhine-Westphalia. Bavaria's capital and largest city, Munich, is the third-largest city in Germany. The history of Bavaria stretches from its earliest settlement and formation as