Venue: NAACL 2025

# Improving Vietnamese-English Cross-Lingual Retrieval for Legal and General Domains

Toan Ngoc Nguyen1, Nam Le Hai1, Nguyen Doan Hieu1, Dai An Nguyen1, Linh Ngo Van1, Thien Huu Nguyen2, Sang Dinh1\({}^{\dagger}\)

Equally contributed.Corresponding author: sangdv@soict.hust.edu.vn

###### Abstract

Document retrieval plays a crucial role in numerous question-answering systems, yet research has concentrated on the general knowledge domain and resource-rich languages like English. In contrast, it remains largely underexplored in low-resource languages and cross-lingual scenarios within specialized domain knowledge such as legal. We present a novel dataset designed for cross-lingual retrieval between Vietnamese and English, which not only covers the general domain but also extends to the legal field. Additionally, we propose auxiliary loss function and symmetrical training strategy that significantly enhance the performance of state-of-the-art models on these retrieval tasks. Our contributions offer a significant resource and methodology aimed at improving cross-lingual retrieval in both legal and general QA settings, facilitating further advancements in document retrieval research across multiple languages and a broader spectrum of specialized domains. All the resources related to our work can be accessed at huggingface.co/datasets/bkai-foundation-models/crosslingual.

## 1 Introduction

Document retrieval systems play a crucial role in question-answering (QA) frameworks by identifying relevant documents that provide the necessary information to answer a given query. However, the majority of existing document retrieval systems Karpukhin et al. (2020); Khattab and Zaharia (2020); Gao et al. (2021); Sachan et al. (2022); Dong et al. (2023) and datasets Nguyen et al. (2016); Kwiatkowski et al. (2019); Thakur et al. (2021); Qiu et al. (2022); Muennighoff et al. (2023) are designed to operate within a single language, typically targeting resource-rich languages like English or Chinese. This monolingual focus limits the effectiveness of these systems in multilingual contexts, where users may pose queries in one language while the relevant documents are in another.

Some studies have tried to explore cross-lingual information retrieval Liu et al. (2020); Bonab et al. (2020); Huang et al. (2023); Louis et al. (2024), yet these efforts have largely concentrated on high-resource languages and general domain knowledge, leveraging extensive resources and pre-existing knowledge bases. Meanwhile, Vietnamese remains largely underexplored in this context, primarily due to the limited availability of datasets necessary for pretraining and fine-tuning representation models in this language. For example, Vietnamese accounts for less than 1% of the total pretraining data in the BGE M3 model Chen et al. (2024). Additionally, general domain datasets Nguyen et al. (2016); Thakur et al. (2021); Muennighoff et al. (2023) are frequently derived from open-domain sources such as Wikipedia, web documents, or news articles, that typically involve quite short documents. While this is valuable for general QA, it fails to address the complexities of specialized domains, where documents are often lengthy and domain-specific, such as legal documentation. Consequently, there is a need to develop cross-lingual document retrieval systems that can effectively handle low-resource languages and specialized domains, ensuring more comprehensive and context-aware QA solutions.

To address these gaps, we present a novel benchmark aimed at evaluating cross-lingual information retrieval (CLIR) between Vietnamese and English. In addition to general knowledge question answering, our dataset enables the investigation of retrieval systems within the legal domain using the Vietnamese Law Library. From this resource, we develop a retrieval model that demonstrates strong performance across both general knowledge and legal domains.

In summary, our contributions are as follows:

1. **Low-Resource Legal Dataset:** We introduce VNLawQC, a dataset designed to explore information retrieval in the legal domain in Vietnamese, along with VNSynLawQC, a synthetic dataset generated by large language models (LLMs) to further augment the training data.
2. **Cross-Lingual Legal Retrieval Dataset:** To enable cross-lingual retrieval, we construct a Vietnamese-English dataset that supports both general and legal domain knowledge. This dataset is constructed using translation models, followed by careful filtering to ensure the selection of high-quality data.
3. **Novel Methodologies for CLIR:** We propose an Auxiliary loss function and Symmetrical training procedure that demonstrates significant improvement in cross-lingual information retrieval scenarios across general knowledge and legal domains.

## 2 Related Work

Recently, Information Retrieval has attracted considerable attention, with document retrieval emerging as one of the central focuses. Several methods have been proposed to address this task, which can generally be classified into three approaches: dense retrieval Karpukhin et al. (2020); Xiong et al. (2021); Wang et al. (2022), lexical retrieval Dai and Callan (2020); Gao et al. (2021), and multi-vector retrieval Khattab and Zaharia (2020); Chen et al. (2024). Numerous datasets have also been developed to evaluate these systems Nguyen et al. (2016); Thakur et al. (2021); Muennighoff et al. (2023).

However, these methods and datasets primarily focus on monolingual scenarios. Recently, several studies have explored cross-lingual settings, where queries and documents are in different languages Liu et al. (2020); Bonab et al. (2020); Huang et al. (2023); Louis et al. (2024). In contrast, some prior studies have investigated specific domains, such as the legal, extending beyond general knowledge QA, but still focusing on monolingual scenarios Sugathadasa et al. (2019); Louis and Spanakis (2022); Sansone and Sperli (2022); Nguyen et al. (2024); Su et al. (2024).

## 3 Methodology

### Dataset Construction

Data Construction Pipeline:Figure 1 illustrates the complete pipeline for constructing our dataset for cross-lingual information retrieval (CLIR) between Vietnamese and English. Overall, our efforts concentrate on collecting data from the Vietnamese legal domain, where resources are limited while utilizing existing datasets from both general and legal fields to generate cross-lingual data through translation approaches. Moreover, to prevent data leakage, we implement data deduplication across the legal datasets using the MinHash technique Luo et al. (2015); Zhu and Markovtsev (2017).

Legal Retrieval Dataset:We introduce VNLawQC sourced from Vietnamese Law Library1 (VLL). The VLL contains articles that address questions spanning multiple aspects of the legal domain. Each article provides an answer supported by one or more legal documents, with hyperlinks directing to the corresponding documents. To create the VNLawQC dataset, we constructed query-passage pairs based on the structure of these articles. Specifically:

Footnote 1: [https://thuvienphapluat.vn](https://thuvienphapluat.vn)

1. The queries were extracted directly from the questions presented in the articles.
2. For the passages, we followed the hyperlinks in each answer to access the referenced legal documents. The relevant sections from these documents were then extracted to serve as the passages.

After parsing content from HTML tags, we apply basic cleaning techniques, including capitalizing legal terms (e.g. "_Dieu_"--_Article_", "_KhoaN_"-_Clause_"), normalizing Unicode characters, and standardizing tone marks, following prior works on Vietnamese text processing Vu et al. (2018); Nguyen and Nguyen (2020). As a result, the VNLawQC dataset is composed of query-passage pairs, where each query can have multiple associated passages if the answer references multiple

\begin{table}
\begin{tabular}{l|c|r|r|r} \hline  & **Language** & **Train** & **Eval** & **Corpus** \\ \hline MsMARCoy Nguyen et al. (2016) & en & 457,361 & 0 & 8,841,823 \\ SQADY Rajpurkar et al. (2018) & en & 60,942 & 0 & 13,317 \\ ZiLojegkaji 

-dataset captures a realistic mapping of questions to relevant legal information.

Synthetic Legal Retrieval Dataset:We used LLama-3-70B model Dubey et al. (2024) to generate synthetic question-context pairs from 140K distinct passages in VNLawQC, creating VNSynLawQC to augment the training data (detailed in Appendix B). Llama-3-70B was chosen for its strong performance, especially in languages like Vietnamese, and its ability to generate high-quality queries. A key challenge in using LLMs for query generation is balancing diversity and relevance. We experimented with different prompt techniques and found that instructing the model to identify 1-5 aspects in a passage and generate a question for each aspect resulted in the most relevant and diverse queries (see more details in Appendix B). This approach enabled the generation of over 620,000 legal queries from 140,000 passages in the VNLawQC dataset. An example of a generated query and its corresponding passage is shown in Table 3. Finally, we merge VNLawQC and VNSynLawQC with the ZaloLegal2021Zalo AI Team (2021) and employ deduplication to prevent data leakage and improve data quality.

Cross-Lingual Dataset using Translation:To facilitate the CLIR scenario, we leverage translation models to produce Vietnamese and English versions of both queries and documents. We integrate multiple datasets, as presented in Table 1, encompassing general and legal domain knowledge in both languages. For the Vietnamese datasets, we employ the VinAI Translate Nguyen et al. (2022) to generate English versions, while Google Translate is used to translate the English datasets into Vietnamese. Both models have demonstrated state-of-the-art performance on Vietnamese-English translation benchmarks, such as PhoMT Doan et al. (2021), highlighting their suitability and effectiveness for our task.

To ensure translation quality, we use back-translation and evaluate the similarity between the original text and its back-translated version with Jaccard similarity Jaccard (1912); Tanimoto (1958). Translation pairs that have a score below the predetermined threshold of **0.5** are then discarded. Additionally, to further assess the quality, we manually reviewed 100 randomly selected samples and verified that they meet satisfactory standards. This multi-step process guarantees that the translations are of high quality for constructing the cross-lingual dataset.

Hard Negatives Mining:To further enhance the training process, we provide hard negatives (e.g., documents) for each example (e.g., query), which offer more informative negative samples and potentially improve training convergence Xiong et al. (2021). Specifically, we utilize a retrieval model like BGE-M3 Chen et al. (2024) to identify the most similar documents and adopt a threshold score to guarantee the selection of true negative samples. We gathered these samples and added additional random contexts, if necessary, to create five negative candidates for each query.

### Cross-Lingual Retrieval Model

Embedding Backbone:We choose the pre-trained BGE-M3 Chen et al. (2024), which can support three retrieval modes: Dense, Lexical, and Multi-Vector, as the backbone model. It supports a long context window of up to 8,192 tokens and is pre-trained in multiple languages, including Vietnamese, which is beneficial for retrieving lengthy legal documents and handling cross-lingual tasks

Figure 1: An Overview of the Our Data Construction Pipeline.

 

involving Vietnamese. The pre-trained BGE-M3 model is also used as the baseline for evaluating the improvements made by our method.

Auxiliary Loss Function:The original BGE-M3 embedding model employs two primary loss functions: \({\cal L}^{InfoNCE}\), an InfoNCE loss Oord et al. (2018) that controls the alignment between queries and both positive and negative passages, and a self-knowledge distillation loss \({\cal L}^{distill}\), which allows the multiple retrieval modes to be jointly learned and mutually reinforced. In cross-lingual scenarios, queries tend to be short and ambiguous. To address this, we propose a loss function that improves alignment between each query and its translated version.

\[{\cal L}^{aux}=-\text{log}\frac{\text{exp}(s(q,\bar{q})/\tau)}{\sum_{a\in Q} \text{exp}(s(q,\bar{a})/\tau)}\]

where \(q\) is a query, \(\bar{q}\) is its translated version of \(q\) , \(Q\) is the set of queries in a batch and \(\tau\) is the temperature hyperparameter. Consequently, we combine these loss functions to train our model: \({\cal L}={\cal L}^{InfoNCE}+{\cal L}^{distill}+{\cal L}^{aux}\).

Symmetrical Training:Currently, retrieval models are trained to minimize the distance between a query and its corresponding relevant documents. We extend this by introducing Symmetrical Training to learn relationships between similar queries and documents across languages. In this approach, a document or query in one language is treated as relevant to its translated version. Given two versions of a document or query, \(S_{A}\) in language \(A\) and \(S_{B}\) in language \(B\), we then consider \(S_{A}\) and \(S_{B}\) as a valid training pair. The model is finetuned to retrieve the translated version of a query or document with a fixed probability, \(p_{sym}\), alongside the standard query-document retrieval task. Hard negatives for these symmetrical pairs are mined similarly to unsymmetrical ones.

## 4 Experiments and Results

Experiment Setup:We trained the models for 4 epochs using the AdamW optimizer Loshchilov and Hutter (2019) with a base learning rate of \(2\times 10^{-5}\). A cosine learning rate scheduler with the warm-up ratio set to \(5\%\) of the total training steps was applied. The temperature \(\tau\) was set to \(0.05\). Besides, we employed smart batching Ge et al. (2021) to group samples with similar sequence lengths. For symmetrical training, the sampling rate \(p_{sym}\) was set to \(0.3\). The trained models are subsequently evaluated on the evaluation sets from VNLawQC and ZaloLegal2021 for the legal domain, as well as ZaloWikipediaQA for the general knowledge domain. We conduct evaluations using various training datasets, retrieval modes, and loss functions.

Evaluation Metric:Following prior work in document retrieval Karpukhin et al. (2020); Wang et al. (2022); Neelakantan et al. (2022); Dai and Callan (2020); Khattab and Zaharia (2020), we leverage four metrics Recall@k, MRR@k, MAP@k and nDCG@k for evaluation. Specifically, we use \(k=10\) and calculate the average of these four metrics for performance comparison. We observe that the average scores exhibit a strong correlation with individual metrics, making them a suitable representation of overall performance. Detailed results for each specific metric are provided in Tables 6 and 7.

Experimental Results:Table 2 presents the results of the baseline models and our proposed models across two cross-lingual scenarios: VietnameseEnglish, where queries are in English and documents are in Vietnamese, and English-Vientnamese, where the roles are reversed. Additional results on monolingual scenarios and detailed metrics for cross-lingual tasks are provided in Appendix C.1 and C.2, respectively. In summary, our proposed datasets and methods enhance the performance of the multilingual embedding backbone (i.e No training), achieving scores that rank among the highest across all evaluation sets. Besides, the retrieval mode with reranking consistently outperforms dense retrieval alone. This improvement is evident due to the additional ranking stage, which enhances the selection of relevant documents, although it also incurs extra costs.

* **Effectiveness of Cross-lingual Data:** The results show that the BGE-M3 model fine-tuned on cross-lingual data significantly outperforms both the original model and the one fine-tuned on Vietnamese data. Specifically, we observe an improvement of over 10% in legal document retrieval and more than 3% in the general domain. This observation further highlights the quality of our construction pipeline with translation.
* **Effectiveness of Synthetic Data:** The inclusion of VNSynLawQC during training generally enhances the performance of all models across the 

evaluation datasets. In particular, an improvement of nearly 3% is observed in the ZaloLegal2021 dataset for the Vietnamese-English scenario, achieving the highest performance with a score of 80.55%. Similarly, all evaluation sets showed improvements when using synthetic data in the English-Vietnamese scenario.
* **Effectiveness of Auxiliary Loss and Symmetrical Training:** The implementation of auxiliary loss functions and symmetrical training yields varying results depending on the dataset domain. While models with symmetrical training demonstrate significant performance in legal retrieval, models trained with auxiliary loss achieve the highest performance in the general knowledge domain. These results align with our motivation for employing auxiliary loss, as queries in the general domain tend to be short and ambiguous.

## 5 Conclusion

In summary, we introduce a novel dataset for cross-lingual information retrieval (CLIR) between Vietnamese and English, covering both general knowledge and the legal domain. Additionally, we develop a CLIR model by finentuning cross-lingual and synthetic data while proposing an auxiliary loss function and training strategy to enhance performance. Our contributions provide valuable resources and methods for advancing cross-lingual retrieval in specialized fields.

## 6 Limitations

The proposed dataset, reliant on translation techniques, may be prone to translation errors and may not fully reflect real-world data patterns. To mitigate this issue, we have made efforts by implementing quality control measures during the generation process to ensure the quality and naturalness of the translations. However, we recommend that mining real-world data or human intervention is crucial for effectively addressing this issue.

In this study, our experiments utilize a single backbone model, which may raise concerns regarding the versatility and adaptability of the proposed methodologies. The backbone model employed in our study, BGE-M3, has already demonstrated state-of-the-art performance across multiple document retrieval benchmarks. As a result, the enhancements observed in this model can well prove the effectiveness of our methodologies. In future work, we aim to extend our techniques to a broader array of models to gain deeper insights into their robustness and adaptability, thereby advancing cross-lingual information retrieval research.

## Acknowledgements

This work was funded by the NAVER Corporation within the framework of collaboration with the International Research Center for Artificial Intelligence (BKAI), School of Information and Communication Technology, Hanoi University of Science and Technology. Nguyen Doan Heu was

\begin{table}
\begin{tabular}{l|c|c|c c c|c c c} \hline \multicolumn{2}{l|}{Training Approach} & Synthetic & Retrieval & \multicolumn{4}{c|}{Vetnamese-English} & \multicolumn{4}{c}{English-Vietnamese} \\  & Augmentation & Mode & VNLwQC & ZaloLegal2021 & ZaloWikipediaQA & VNLw QC & ZaloLegal2021 & ZaloWikipediaQA \\ No training & ✗ & \(D\) & 42.99 & 51.88 & 64.84 & 39.46 & 48.26 & 62.31 \\  & \multirow{2}{*}{✓} & \(D+R\) & 44.92 & 53.63 & 66.18 & 40.81 & 49.16 & 63.45 \\ \cline{7-10}  & & \multirow{2}{*}{✓} & \(D\) & 54.14 & 62.26 & 65.96 & 47.38 & 55.60 & 61.01 \\  & & \(D+R\) & 56.78 & 65.14 & 71.14 & 50.48 & 59.03 & 67.48 \\ \cline{2-10}  & & \multirow{2}{*}{✓} & \(D\) & 56.18 & 62.42 & 64.60 & 48.90 & 53.62 & 60.05 \\  & & \(D+R\) & 58.32 & 65.72 & 70.25 & 53.38 & 57.26 & 66.57 \\ \hline \multirow{3}{*}{Cross-lingual} & \multirow{2}{*}{✓} & \(D\) & 68.02 & 75.32 & 67.90 & 65.39 & 71.33 & 65.06 \\  & & \(D+R\) & 70.21 & 77.57 & 72.90 & 68.04 & 74.33 & 70.88 \\ \cline{1-1} \cline{2-10}  & ✓ & \(D\) & 69.44 & 78.74 & 68.61 & 66.89 & 75.42 & 66.12 \\  & & \(D+R\) & 71.58 & **80.55** & 73.88 & 68.95 & 78.41 & 71.54 \\ \hline Cross-lingual & \multirow{2}{*}{✓} & \(D\) & 68.60 & 75.49 & 69.56 & 66.18 & 73.18 & 66.39 \\  & & \(D+R\) & 71.46 & 78.62 & **74.18** & **69.53** & 76.18 & **71.78** \\ \hline Cross-lingual & \multirow{2}{*}{✓} & \(D\) & 69.75 & 76.33 & 65.64 & 67.66 & 75.97 & 62.15 \\  & & \(D+R\) & **71.71** & **79.53** & 68.75 & **69.91** & **79.75** & 66.73 \\ \hline \end{tabular}
\end{table}
Table 2: Performance of BGE-M3 in CLIR scenario using different training methods, datasets, and retrieval mode across three evaluation sets in legal and general knowledge domains. In the training approach, _Cross-lingual_ refers to the use of datasets in both language versions, while _aux_loss_function_ and _sym_training_ indicate the loss function and Symmetrical Training described in Section 3.2. _Synthetic Augmentation_ refers to the use of VNSynLawQC to augment the training data during the training process. In retrieval modes, \(D\) represents dense retrieval results, while \(D+R\) represents the results when a reranking stage is incorporated for the retrieved documents. **Green scores** indicate the highest score, while Gray scores represent the second highest.

 

funded by the Master, PhD Scholarship Program of Vinyour Innovation Foundation (VINIF), code VINIF.2022.ThS.BK.09.

## References

* H. Bonab, S. M. Sarwar, and J. Allan (2020)Training effective neural clir by bridging the translation gap. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 9-18. Cited by: SS5.
* J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu (2024)Bge m3-embedding: multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216. Cited by: SS5.
* Z. Dai and J. Callan (2020)Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in information Retrieval, pp. 1533-1536. Cited by: SS5.
* L. Doan, E. The Nguyen, N. Tran, T. Hoang, and D. Quco Nguyen (2021)Phomt: a high-quality and large-scale benchmark dataset for Vietnamese-english machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pp. 4495-4503. External Links: Link, Document Cited by: SS5.
* Q. Dong, Y. Liu, Q. Ai, H. Li, S. Wang, Y. Liu, D. Yin, and S. Ma (2023)I3 retriever: incorporating implicit interaction in pre-trained language models for passage retrieval. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 441-451. Cited by: SS5.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024)The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS5.
* L. Gao, Z. Dai, and J. Callan (2021)COIL: revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pp. 3030-3042. Cited by: SS5.
* Z. Ge, L. Kaushik, M. Omote, and S. Kumar (2021)Speed up training with variable length inputs by efficient batching strategies. In Interspeech, pp. 156-160. Cited by: SS5.
* Z. Huang, P. Yu, and J. Allan (2023)Improving cross-lingual information retrieval on low-resource languages via optimal transport distillation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 1048-1056. Cited by: SS5.
* P. Jaccard (1912)The distribution of the flora in the alpine zone. 1. New phytologist11 (2), pp. 37-50. Cited by: SS5.
* V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020)Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781. Cited by: SS5.
* O. Khattab and M. Zaharia (2020)Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39-48. Cited by: SS5.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. (2019)Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7, pp. 453-466. Cited by: SS5.
* J. Liu, X. Zhang, D. Goldwasser, and X. Wang (2020)Cross-lingual document retrieval with smooth learning. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 3616-3629. Cited by: SS5.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In Proceedings of ICLR 2019, Cited by: SS5.
* Antoine Louis, V. Kamar Saxena, G. van Dijck, and G. Spanakis (2024)Colbert-xm: a modular multi-vector representation model for zero-shot multilingual information retrieval. ArXivabs/2402.15059. Cited by: SS5.
* A. Louis and G. Spanakis (2022)A statutory article retrieval dataset in french. In Proceedings of ACL 2022, pp. 6789-6803. Cited by: SS5.
* S. Luo, G. Zhang, C. Wu, S. U Khan, and K. Li (2015)BoaffIt: distributed deduplication for big data storage in the cloud. IEEE transactions on cloud computing8 (4), pp. 1199-1211. Cited by: SS5.
* N. Muennighoff, N. Tazi, L. Magne, and N. Reimers (2023)Mteb: massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037. Cited by: SS5.
* A. Neelakantan, T. Xu, R. Puri, A. Radford, J. Michael Han, J. Tworek, Q. Yuan, N. Tezak, J. Wood Kim, C. Hallacy, et al. (2022)Text and code embeddings by contrastive pretraining. arXiv preprint arXiv:2201.10005 

* Nguyen and Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. Phobert: Pre-trained language models for vietnamese. In _Findings of EMNLP 2020_, volume EMNLP 2020, pages 1037-1042.
* Nguyen et al. (2024) Ha-Thanh Nguyen, Manh-Kien Phi, Xuan-Bach Ngo, Vu Tran, Le-Minh Nguyen, and Minh-Phuong Tu. 2024. Attentive deep neural networks for legal document retrieval. _Artificial Intelligence and Law_, 32(1):57-86.
* Nguyen et al. (2022) Thien Hai Nguyen, Tuan Duy H Nguyen, Duy Phung, Duy Tran Cong Nguyen, Hieu Minh Tran, Manh Luong, Tin Duy Vo, Hung Hai Bui, Dinh Phung, and Dat Quoc Nguyen. 2022. A vietnamese-english neural machine translation system. In _Annual Conference of the International Speech Communication Association (was Eurospeech) 2022_, pages 5543-5544. International Speech Communication Association (ISCA).
* Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In _Proceedings of Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016_, volume 1773 of _CEUR Workshop Proceedings_. CEUR-WS.org.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _arXiv preprint arXiv:1807.03748_.
* Qiu et al. (2022) Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, Qiaoqiao She, Jing Liu, Hua Wu, and Haifeng Wang. 2022. Dureader-retrieval: A large-scale chinese benchmark for passage retrieval from web search engine. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5326-5338.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In _Proceedings of ACL 2018_, pages 784-789.
* Sachan et al. (2022) Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3781-3797.
* Sansone and Sperli (2022) Carlo Sansone and Giancarlo Sperli. 2022. Legal information retrieval systems: State-of-the-art and open issues. _Information Systems_, 106:101967.
* Su et al. (2024) Weihang Su, Yiran Hu, Anzhe Xie, Qingyao Ai, Quezi Bing, Ning Zheng, Yun Liu, Weixing Shen, and Yiqun Liu. 2024. Stard: A chinese statute retrieval dataset derived from real-life queries by nonprofessionals. In _Findings of the Association for Computational Linguistics: EMNLP 2024_, pages 10658-10671.
* Sugathadasa et al. (2019) Keet Sugathadasa, Buddhiyesa, hisansa de Silva, Amal Shehan Perera, Vinudula Jayawardana, Dimuthuk Lakmal, and Madhavi Perera. 2019. Legal document retrieval using document vector embeddings and deep learning. In _Intelligent Computing: Proceedings of the 2018 Computing Conference, Volume 2_, pages 160-175. Springer.
* Tanimoto (1958) Taffee T Tanimoto. 1958. Elementary mathematical theory of classification and prediction.
* Thakur et al. (2021) Namdan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. _arXiv preprint arXiv:2104.08663_.
* Vu et al. (2018) Thanh Vu, Dat Quoc Nguyen, Dai Quoc Nguyen, Mark Dras, and Mark Johnson. 2018. VNorenlp: A uistemamese natural language processing toolkit. In _Proceedings of NAACL-HLT 2018_, pages 56-60.
* Wang et al. (2022) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Davin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. _arXiv preprint arXiv:2212.03533_.
* Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
* Team (2019) Zalo AI Team. 2019. Zalo challenge dataset. Zalo AI Challenge 2019, [https://challenge.zaio.ai](https://challenge.zaio.ai). Accessed: 2025-02-08.
* Team (2021) Zalo AI Team. 2021. Zalo challenge dataset. Zalo AI Challenge 2021, [https://challenge.zaio.ai](https://challenge.zaio.ai). Accessed: 2025-02-08.
* Zhu and Markovtsev (2017) Eric Zhu and Vadim Markovtsev. 2017. ekzhu/datasketch: First stable release. Accessed: 2025-02-08.

 



## Appendix A Additional Details on Data Construction

### B Synthetic Data Generation

[frame=lines,width=0.5cm] (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24) (25) (26) (27) (28) (29) (30) (31) (32) (33) (34) (35) (36) (37) (38) (39) (40) (41) (42) (43) (44) (45) (46) (47) (48) (49) (50) (51) (52) (53) (54) (55) (56) (57) (58) (59) (60) (61) (62) (63) (64) (65) (66) (67) (68) (69) (70) (71) (72) (73) (74) (75) (76) (77) (78) (79) (80) (81) (82) (83) (84) (85) (86) (87) (88) (89) (90) (91) (92) (93) (94) (95) (96) (97) (98) (99) (99) (91) (93) (95) (96) (97) (99) (98) (99) (99) (91) (92) (93) (94) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (92) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99) (99) (91) (93) (95) (96) (97) (99



### Generate synthetic queries

For generating synthetic queries, we utilized the open-source large language model Meta Llama 3 (Dubey et al., 2024) to generate queries based on aspects identified within legal text passages. This process involved extracting key aspects from the texts and formulating corresponding queries. We selected Llama-3-70B for its strong capabilities and performance. Additionally, Llama 3 is believed to include a portion of synthetic data in its training corpus. Upon release, it outperformed many other models with a similar parameter count, demonstrating notable proficiency across multiple languages, including Vietnamese, aligning well with our requirements.

A significant challenge in using LLMs for query generation is maintaining both the diversity and relevance of their outputs. We experimented with different prompt techniques to achieve this balance. One approach instructed the model to generate questions directly from the passage without first identifying different aspects. This method often resulted in less diverse and sometimes irrelevant queries, as the model tended to focus on the most prominent information in the passage, neglecting other potential aspects.

Through various prompt designs, we discovered that instructing the model to identify 1-5 different aspects covered in the passage and then generate a question for each aspect yielded the most relevant and diverse queries. The prompt template used for generating these synthetic queries is illustrated in prompt

\begin{table}
\begin{tabular}{l l} \hline \hline  & **Vietnamese** & **English** \\ \hline
**Header** & Muc 1. CH��UN BI THANH TRA, Chutong II, TRINH TU, THU TUC TIEN HANH CUOC THANH TRA THEO KE HOACH THANH TRA, Thông tu 36/2016/TT-NHNN uq dinh vě trinh tuf, thüt üc thanth tra chuyen ngnah Ngan hân â doThög dóc Ngán hang Nha nuoc Viet Nam ban hanh. \\ \hline
**Content** & 5. Trufø doing dánnh tra to chück hop Déin thannh tra de phö bien kão hoach têtn hainh thahn tra duroc duyét và phén cón nihém vu cho cic Tó htanh tra, Nböm hannh tra, cite dänhv rein caiba Doanin hathn tra; thio lujan, quyelt fhin ve Phûb rübing päjch, chat hätc éd cinh t�h nauh tra, ‘tien hahnh thanz tru; yñi phö bijta eac tahani vién Doan thanth tra, cãc oquan, don vic � lo� lena utrono que ruith trénk kahat hánnh tra. Trong tr� hbp cm thihet, ngvito ri avjet dhit thanni tra hdoe gnuït ducp at rauyet dih finhanh tra uygenh dûrb n� à quënt triènc dich, yeçau, noî dung thann tra va nhiem vu cua Doan thanh tra. Viée phänn Grö m hième vu cho cac TO thank tra, Niön manh tra, cãc thanh víeni Doan thanh tra publi ê hei bnè bang vin bair. \\ \multicolumn{2}{r}{6. To früng thanh tra, Nhom rtüng thanh tra, thinh jeaders, and members of the Inspection team shall heldeep plans to carry out their assigned tasks and report to the Head of the Inspection team before conducting the inspection at the credit institution.} \\ \hline
**Aspecot 1** & Trich nhiém cútu Trüng doan thanh tra trong vice tö chtc và phén cón nhiém vu \\ \hline
**Query 1** & Nğ hâng Nha nutôc guy dihn Thrüng doan thanh tra phai lam gi déc Huanh bi che cuočte thanh tra? \\ \hline
**Aspecot 2** & Quy trinh xaj dtng và bação cio khoach htuc hién nhiém vu cica CIC � tnanh tra, Nhom thanh tra \\ \hline
**Query 2** & Khi duck phán còng nhiém vu, cic Tó thanh tra, Nhom thanh tra phai lam gi déc duün bi cho cuõc insptation? \\ \hline \hline \end{tabular}
\end{table}
Table 3: Example of a generated query-passage pair for the domain ”Tién tê-Nğ hâng” (Currency-Banking)

B. Applying this method, we generated over 620,000 legal queries from 140,000 passages in VNLawQC dataset. An example of a generated query and its corresponding passage is shown in Table 3.

### Filter low-quality queries

After generating the synthetic data, we removed low-quality queries that explicitly referred to the input passage or were only shallowly relevant to the passage content. In particular, we employed the BGE-M3 dense retriever (Chen et al., 2024), which demonstrated strong zero-shot performance in our testing, to filter out queries whose corresponding passages did not appear in the top 40 relevant results. Additionally, we excluded queries that directly referred to the passage using terms like "_quy dihn nay_" (_this regulation_) or "_thong tu nay_" (_this circular_). This process resulted in the final VNSYNLAWQC dataset, which contains over 500,000 high-quality queries.

## Appendix C Additional Experimental Results

In this section, we present additional results for both mono-lingual (Section C.1) and cross-lingual (Section C.2) settings. Additionally, we explore different reranking modes, as discussed in Section 4. For reranking, we employ the multi-vector mode, which incurs minimal overhead since it is trained concurrently with dense retrieval. Only the top 100 passages from dense retrieval are remarked to reduce computational cost. Reranking times were measured on Kaggle's T4 and an RTX3090: cross-encoder reranking (BGE-reranker-v2-m3) took 7.15s/query (T4) and 1.33s/query (RTX3090), while our multi-vector mode took 6.41s/query (T4) and 1.05s/query (RTX3090).

### Mono-lingual Retrieval Results

Table 4 presents the performance of our cross-lingual models in the English-English retrieval setting. All cross-lingual models significantly outperform the baseline on both legal datasets. Notably, the cross-lingual model with symmetrical training achieves the highest R@10 score of 83.10% on the VNLawQC dataset, while the base cross-lingual model attains the highest R@10 score of 89.43% on the ZaloLegal2021 dataset. In contrast, for the ZaloWikipediaQA dataset, although there is a slight decline in dense retrieval performance, incorporating reranking and the auxiliary loss function boosts the cross-lingual model to an optimal R@10 of 88.32%.

However, we noticed that on the two legal datasets, despite having higher performance compared to the baseline model, the performance is lower than in the Vietnamese-English setting. We hypothesize that this issue arises from errors propagated during the translation process. While the documents typically contain multiple sentences and are sufficiently lengthy to provide contextual information, the queries are short and consist of only a single sentence, which may lead to translation inaccuracies due to the lack of contextual cues.

We further evaluated our cross-lingual models in the Vietnamese-only retrieval setting. As presented in Table 5, despite being trained on cross-lingual data, these models perform comparably to the Vietnamese model, which was trained exclusively on Vietnamese data. On both legal datasets, the cross-lingual models surpass the baseline and achieve competitive results. For the VNLawQC dataset, the cross-lingual model augmented with the auxiliary loss function attains an R@10 of 86.5%, which is only marginally

\begin{table}
\begin{tabular}{l|c c c c c c|c c c c c c c} \hline Training Approach & Synthetic & \multirow{2}{*}{
\begin{tabular}{c} Structured \\ Augmentation \\ \end{tabular} } & \multicolumn{2}{c|}{**Kernel**} & \multicolumn{2}{c|}{**VNLawQC**} & \multicolumn{2}{c|}{**ZaloLegal201**} & \multicolumn{2}{c}{**ZaloLegal201A**} \\  & & **R@10** & **MRR@10** & **MMP@10** & **ADCG+10** & **R@10** & **MRR@10** & **MMP@10** & **BDCC@10** & **R@10** & **MRR@10** & **MRR@10** & **MAP@10** & **BDCC@10** \\ \cline{3-13}  & ✗ & \(D\) & 65.09 & 43.73 & 42.66 & 48.07 & 73.07 & 49.67 & 49.39 & 55.13 & 85.25 & 65.61 & 63.00 & 69.31 \\ No mining & ✗ & \(D\) & 65.88 & 45.22 & 43.44 & 49.33 & 75.86 & 52.96 & 53.31 & 86.95 & 67.35 & 64.99 & 71.17 \\ \cline{2-13}  & ✗ & \(D\) & 73.31 & 51.42 & 42.60 & 53.73 & 81.95 & 56.04 & 57.68 & 63.64 & 62.45 & 64.45 & 61.54 & 62.76 \\ \cline{2-13}  & ✗ & \(D\) & 76.14 & 55.06 & 53.01 & 59.18 & 54.69 & 63.22 & 63.97 & 63.29 & 87.14 & 69.25 & 66.68 & 69.76 \\ \cline{2-13}  & ✓ & \(D\) & 76.87 & 55.11 & 50.07 & 56.33 & 52.89 & 69.09 & 69.16 & 65.74 & 82.10 & 64.08 & 66.13 & 61.71 \\ \hline \multirow{3}{*}{Coss-lingual} & ✗ & \(D\) & 76.38 & 55.53 & 54.22 & 59.57 & 83.63 & 69.32 & 63.85 & 86.90 & 69.63 & 66.67 & 77.23 \\ \cline{2-13}  & ✗ & \(D\) & 89.03 & 64.44 & 58.64 & 64.43 & 62.67 & 67.63 & 47.20 & 83.02 & 64.42 & 64.54 & 67.22 \\ \cline{1-1} \cline{2-13}  & ✗ & \(D\) & 82.24 & 63.39 & 64.22 & 67.06 & 88.72 & 70.95 & 70.62 & 70.88 & 87.33 & 69.42 & 66.44 & 69.72 \\ \cline{1-1} \cline{2-13}  & ✓ & \(D\) & 81.95 & 61.19 & 59.88 & 53.14 & 89.53 & 69.58 & 70.96 & 71.18 & 63.44 & 64.20 & 68.12 \\ \hline Cross-lingual & ✗ & \(D\) & 81.05 & 60.67 & 64.01 & 67.36 & **89.43** & **72.38** & **75.88** & **78.33** & 87.64 & 70.06 & 67.46 & 71.25 \\ \cline{1-1} \cline{2-13}  & ✗ & \(D\) & 81.95 & 60.60 & 58.47 & 64.58 & 52.62 & 65.82 & 65.84 & 70.41 & 84.50 & 65.96 & 63.13 & 69.24 \\ \cline{1-1} \cline{2-13}  & ✗ & \(D\) & _84.03 & 63.98 & 61.85 & 65.06 & 88.54 & 69.14 & 68.83 & 73.74 & **88.32** & **79.85** & **68.41** & **73.87** \\ \hline Cross-lingual & ✗ & \(D\) & 81.74 & 62.03 & 59.96 & 65.84 & 36.85 & 65.97 & 73.91 & 79.69 & 64.30 & 57.29 & 63.67 \\ \cline{1-1} \cline{2-13}  & ✓ & \(D\) & \(R\) & **83.10** & **64.32** & **62.14** & **57.84** & 87.01 & 69.16 & 68.95 & 73.41 & 62.68 & 63.30 & 66.64 & 66.50 \\ \hline \end{tabular}
\end{table}
Table 4: English-English retrieval results on different datasets. Both the queries and the documents are in English.

 

lower than the Vietnamese model's score of 87.33% under the same dense + re-ranking pipeline with synthetic augmentation. Similarly, on the ZaloLegal2021 dataset, it also achieves an R@10 of 92.68%, closely aligning with the Vietnamese model's top score of 94.06%. Although performance declines on the ZaloWikipediaQA dataset, the use of reranking and auxiliary loss still helps the cross-lingual model achieve an R@10 of 94.63%, outperforming other configurations. The use of synthetic augmentation generally leads to performance improvements across all training approaches, except for the Vietnamese model on the ZaloWikipediaQA dataset, where the gains are less pronounced.

### Cross-lingual Retrieval

We finally evaluated our models on English-Vietnamese and Vietnamese-English cross-lingual retrieval tasks, as presented in Tables 6 and 7. The results indicate that for both retrieval directions, our cross-lingual models consistently outperform the baseline and the Vietnamese version, achieving the highest performance across all metrics and datasets, including the ZaloWikipediaQA dataset. This superior

\begin{table}
\begin{tabular}{l|c c|c c c c|c c c c|c c c} \hline Training & Synthetic & Retiect & \multicolumn{5}{c|}{**VNL-HQ**} & \multicolumn{5}{c|}{**ZaloLegal2021**} \\ Approach & Agreements & Model & **R@10** & **MRER@10** & **MAPER’10** & **R@CE@F@10** & **R@10** & **MRER@10** & **MMPER’10** & **BDCG@10** & **R@10** & **MRER@10** & **MAPER’10** & **BDCG@10** \\ \hline No training & ✗ & \(D\) & 84.65 & 34.51 & 33.06 & 35.61 & 66.77 & 41.35 & 41.55 & 47.58 & 72.84 & 57.54 & 54.99 & 58.47 \\  & ✓ & \(D\) & 86.50 & 35.63 & 35.63 & 34.20 & 36.91 & 66.77 & 42.52 & 42.27 & 45.68 & 79.05 & 58.64 & 56.17 & 59.92 \\  & ✗ & \(D\) & 84.42 & 65.31 & 35.24 & 60.95 & 72.98 & 88.91 & 48.82 & 53.18 & 58.35 & 58.54 & 58.36 & 57.20 \\ \multirow{3}{*}{Vietnamese} & ✗ & \(D\) & 84.61 & 45.09 & 43.38 & 48.52 & 76.31 & 52.31 & 52.06 & 58.41 & 82.34 & 63.00 & 60.08 & 64.09 \\  & ✓ & \(D\) & 45.58 & 46.09 & 44.56 & 44.56 & 46.95 & 47.20 & 44.98 & 50.49 & 73.54 & 57.22 & 59.44 & 56.22 \\  & ✓ & \(D\) & 89.34 & 48.25 & 46.42 & 49.52 & 73.54 & 51.04 & 58.33 & 55.64 & 81.29 & 62.13 & 59.61 & 63.24 \\ \hline Cows-tagged & ✗ & \(D\) & 81.55 & 60.34 & 55.31 & 61.74 & 84.77 & 65.50 & 65.54 & 65.91 & 80.09 & 60.61 & 57.90 & 61.65 \\  & ✓ & \(D\) & 82.84 & 64.32 & 61.33 & 64.77 & 87.94 & 65.99 & 65.73 & 71.64 & 83.66 & 64.40 & 61.07 & 62.46 \\  & ✓ & \(D\) & 81.90 & 62.10 & 59.99 & 63.88 & 86.22 & 70.24 & 69.76 & 72.96 & 89.33 & 61.82 & 59.06 & 62.45 \\ \hline Cows-tagged & ✗ & \(D\) & 81.79 & 61.18 & 59.10 & 62.64 & 57.63 & 67.32 & 67.07 & 73.00 & 89.06 & 62.14 & 59.40 & 63.66 \\ text\_box\_function & ✓ & \(D\) & 84.77 & 65.11 & 52.61 & 66.33 & 70.13 & 76.66 & 73.44 & 73.59 & 83.66 & 62.18 & 59.47 & 65.41 \\ \hline Cows-tagged & ✗ & \(D\) & 84.96 & 63.11 & 60.84 & 64.49 & 88.88 & 70.90 & 70.33 & 77.93 & 57.30 & 54.76 & 58.41 \\ caption\_loss & ✓ & \(D\) & 83.51 & **65.44** & **63.56** & **64.95** & **91.77** & **75.27** & **74.98** & **77.99** & 81.36 & 62.28 & 59.76 & 63.52 \\ \hline \end{tabular}
\end{table}
Table 6: English-Vietnamese retrieval results on different datasets. The queries are in Vietnamese and the documents are in English.

\begin{table}
\begin{tabular}{l|c|c c c c c|c c c c c} \hline Training & Synthetic & Retiect & \multicolumn{5}{c|}{**VNL-HQ**} & \multicolumn{5}{c|}{**ZaloLegal2021**} \\ Approach & Agreements & Model & **R@10** & **MRER@10** & **MRER@10** & **MRER@10** & **MRER@10** & **MRER@10** & **MRER@10** & **MRER@10** & **MRER@10** & **MRER@10** \\ \hline No training & ✗ & \(D\) & 85.25 & 36.91 & 35.47 & 41.33 & 69.79 & 43.95 & 43.72 & 50.05 & 80.00 & 59.43 & 56.68 & 65.26 \\  & ✓ & \(D\) & 80.44 & 38.76 & 35.23 & 43.23 & 70.73 & 46.11 & 45.83 & 51.86 & 81.39 & 60.66 & 58.05 & 64.62 \\ \cline{2-13}  & ✗ & \(D\) & 70.52 & 47.91 & 45.99 & 52.37 & 79.30 & 54.69 & 54.51 & 63.53 & 88.44 & 61.07 & 57.95 & 64.83 \\ \multirow{3}{*}{Vietnamese} & ✗ & \(D\) & 72.55 & 50.88 & 48.59 & 55.01 & 81.30 & 57.96 & 57.73 & 63.53 & 85.25 & 66.30 & 63.45 & 69.58 \\  & ✓ & \(D\) & 71.42 & 50.37 & 46.42 & 54.51 & 51.72 & 53.52 & 53.64 & 64.48 & 79.60 & 59.33 & 56.46 & 62.59 \\ \cline{2-13}  & ✓ & \(D\) & 73.36 & 25.61 & 50.62 & 56.67 & 52.66 & 58.17 & 57.99 & 64.05 & 84.34 & 63.51 & 62.59 & 68.77 \\ \hline \multirow{3}{*}{Cows-Tingual} & ✗ & \(D\) & 82.00 & 62.99 & 63.08 & 64.62 & 65.95 & 69.68 & 74.21 & 65.94 & 69.68 & 74.12 & 62.25 & 60.07 & 64.37 \\  & ✗ & \(D\) & 84.14 & 65.13 & 62.89 & 68.96 & 86.27 & 72.00 & 76.41 & 58.48 & 68.15 & 65.88 & 71.48 \\ \cline{1-1}  & ✓ & \(D\) & 80.65 & 63.97 & 63.68 & 67.85 & 50.55 & 75.39 & 72.33 & 77.37 & 83.61 & 63.61 & 60.63 & 67.65 \\ \hline Cows-tagged & ✗ & \(D\) & 85.27 & 66.55 & 63.78 & 70.08 & **92.34** & **75.89** & **79.04** & 87.32 & 62.93 & 66.56 & 72.46 \\ \hline Cows-tagged & ✓ & \(D\) & 85.48 & 62.97 & 66.92 & 66.97 & 58.48 & 69.93 & 72.49 & 71.99 & 85.47 & 64.67 & 65.48 & 66.07 & 72.26 \\ \hline \multirow{3}{*}{Cows-Tingual} & ✗ & \(D\) & 85.

 

performance suggests a robust understanding of the semantic relationships between Vietnamese and English content.

For the English-Vietnamese retrieval task, the cross-lingual model with an auxiliary loss function achieves an R@10 of 83.77% on the VNLawQC dataset, which is 27% higher than the baseline and 14% higher than the Vietnamese model. Similarly, in the ZaloLegal2021 dataset, the cross-lingual model with symmetrical training achieves an R@10 of 91.17%, which is 24% higher than the baseline and 15% higher than the Vietnamese model. On the ZaloWikipediaQA dataset, the cross-lingual model records an R@10 of 86.03%, outperforming the baseline by 7% and the Vietnamese model by 4%.

For the Vietnamese-English retrieval task, the cross-lingual models achieve even higher results. On the VNLawQC dataset, the best cross-lingual model attains an R@10 of 85.32%, which is 25% higher than the baseline and 12% higher than the Vietnamese model. In the ZaloLegal2021 dataset, the cross-lingual model achieves an R@10 of 92.34%, reflecting a 22% increase over the baseline and a 10% improvement over the Vietnamese model. For the ZaloWikipediaQA dataset, the cross-lingual model reaches an R@10 of 87.63%, surpassing the baseline by 6% and the Vietnamese model by 3%. These findings underscore the effectiveness of our cross-lingual models, particularly when combined with the Auxiliary Loss Function and Symmetrical Training strategies.

Furthermore, synthetic augmentation results in an average performance improvement of 1% across all datasets. Notably, an improvement of 3% is observed in the ZaloLegal2021 dataset for both the English-Vietnamese and the Vietnamese-English scenario, highlighting its positive impact on retrieval effectiveness.