POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation (2024)

Abstract: Low-resource languages (LRLs) face challenges in supervised neural machine translation (NMT) due to limited parallel data, prompting research in unsupervised NMT (UNMT). UNMT, without requiring ground truth, provides solutions for LRL translations using synthetic pseudo-parallel data and parallel data from auxiliary language pairs. However, they usually encounter translation errors, including errors from synthetic data and from auxiliary language pairs with linguistic biases. We argue that large language models (LLMs) mitigate UNMT’s translation errors by dynamically organizing auxiliary languages in prompts to improve LRL translations. In this paper, we propose PrObability-driven Meta-graph Prompter (POMP), an approach employing a dynamic graph to organize multiple auxiliary languages, to prompt LLMs in LRL translations. POMP proposes a language-specific meta-graph that dynamically samples multiple translation paths to organize auxiliary languages in constructing prompts. Following the path, POMP prompts LLMs to translate with a mixture of auxiliary languages. We achieve the meta-graph’s evolution by back-propagating evaluation scores to update probabilities on the graph. Our experimental improvements show POMP’s effectiveness on LRLs’ translation.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9976–9992 August 11-16, 2024 ©2024 Association for Computational Linguistics
Low-resource languages (LRLs) face challenges in supervised neural machine translation (NMT) due to limited parallel data, prompting research in unsupervised NMT (UNMT). UNMT, without requiring ground truth, provides solutions for LRL translations using synthetic pseudo-parallel data and parallel data from auxiliary language pairs. However, they usually encounter translation errors, including errors from synthetic data and from auxiliary language pairs with linguistic biases. We argue that large language models (LLMs) mitigate UNMT’s translation errors by dynamically organizing auxiliary languages in prompts to improve LRL translations. In this paper, we propose PrObability-driven Meta-graph Prompter (POMP), an approach employing a dynamic graph to organize multiple auxiliary languages, to prompt LLMs in LRL translations. POMP proposes a language-specific meta-graph that dynamically samples multiple translation paths to organize auxiliary languages in constructing prompts. Following the path, POMP prompts LLMs to translate with a mixture of auxiliary languages. We achieve the meta-graph’s evolution by back-propagating evaluation scores to update probabilities on the graph. Our experimental improvements show POMP’s effectiveness on LRLs’ translation.

heading: 1 Introduction
text: Training supervised NMT models requires extensive parallel data (Xue et al., 2021; Fan et al., 2021), which struggles in low-resource languages (LRLs) that have limited parallel data. Researchers extract (Schwenk et al., 2021a,b) and annotate (Goyal et al., 2022) parallel samples of LRLs, which requires a substantial human effort. Therefore, the research community explores unsupervised methods to achieve NMT without parallel data.
*Corresponding Author
Unsupervised low-resource translation methods are categorized into three types: (1) Backtranslation (Sennrich et al., 2016; Lample et al., 2018b) employs existing target-source NMT models to translate target monolingual data to the source language, generating synthetic parallel data for training desired source-target NMT models. This approach is constrained by the translation errors in the synthetic data (Chauhan et al., 2022). (2) Transfer learning-based NMT (Li et al., 2022) train on high-resource auxiliary languages and transfer their ability on LRLs’ inference. The linguistic biases between training (auxiliary) and testing languages lead to poor translation accuracy (Dabre et al., 2017). (3) Pivot-based translation (Kim et al., 2019) first translates from the source to a pivot (auxiliary) language and then translates from the pivot to the target. The multi-hop translation process introduces potential translation errors among different languages (Liu et al., 2019). Overall, those methods suffer from translation errors caused by translating among different languages (including synthetic data), where the languages have linguistic biases with each other and are hard to translate, called linguistically biased translation errors.
LLMs greatly improved NMT (Peng et al., 2023) but hardly performed well in LRL since LLMs’ pretraining corpora mainly derive from high-resource languages. Some researchers conduct prompt engineering (Diao et al., 2023; Shum et al., 2023; Wen et al., 2024) for LLMs in LRL but achieve limited performance because LLMs learn little for LRLs in zero-shot prompting (Hendy et al., 2023). Researchers apply in-context learning (ICL) that feeds few-shot examples with input-output pairs (sourcetarget parallel pairs) to LLMs and asks LLMs to follow the source-target pairs to translate target sentences (Dong et al., 2023). Alves et al. (2023) enhanced supervised translation by fine-tuning LLMs with parallel sentences. Hendy et al. (2023) found that LLMs still require sufficient parallel data and
9976
hardly do well in unsupervised LRLs. As LLMs have a strong ability to understand high-resource languages, we argue that LLMs with high-resource auxiliary languages should be a good way to assist unsupervised LRL translations, where high-resource languages serve to construct prompts for LLMs. Considering multiple auxiliary languages via LLMs can mitigate linguistically biased translation errors in UNMT.
In this paper, we propose PrObability-driven Meta-graph Prompter (POMP),1 which employs a sampling-based dynamic graph organizing auxiliary pseudo-parallel sentences to prompt LLMs to alleviate linguistically biased translation errors in UNMT. Specifically, we design a language-specific meta-graph to generate multiple translation paths to prompt LLMs, where the paths bridge the source and the target, through various auxiliary languages. Multiple paths in the graph prompt LLMs in various ways, which mitigate translation errors caused by an individual language. Using the paths from the meta-graph, we propose two operations to prompt LLMs: (1) Generate prompts LLMs with a single auxiliary language along the translation paths to make full use of the selected auxiliary languages; (2) Aggregate prompts LLMs with all auxiliary in the path and the best Generate’s output to obtain translations considering cross-linguistic information from multiple auxiliary languages. To optimize the meta-graph, we design a probabilistic backward graph evolution strategy that backpropagates the evaluation score for the translation path into each auxiliary language according to the contribution of each language.
Our contributions are as follows: (1) We propose a UNMT that mitigates linguistically biased translation errors using auxiliary languages in translations to achieve SOTA performance on four LRL translations. (2) We propose a prompting graph for LLMs to organize multiple auxiliary languages. (3) We design a probabilistic backward graph evolution algorithm that iteratively updates probabilities of auxiliary languages to construct better prompts.

heading: 2 Related Work

heading: 2.1 Unsupervised NMT on Low-resource Languages
text: UNMT methods aim to learn an NMT model without parallel data, offering more practical alternatives for LRLs. UNMT methods in recent years
1Our code is at github.com/slpanir/POMP.
are mainly divided into three categories: backtranslation, transfer learning, and pivot-based translation.
Back-translation (Sennrich et al., 2016) uses monolingual data to train NMT models by generating synthetic source sentences. Edunov et al. (2018) confirmed its improved performance in both large-scale and low-resource contexts. Lample et al. (2018a); Artetxe et al. (2018) pioneered UNMT using iterative back-translation, mitigating errors of initial NMT models. This method has been effectively applied to LRLs (Chen et al., 2020; Sánchez-Martínez et al., 2020). However, Edman et al. (2020) noted that poor initial quality in word embeddings and cross-lingual alignments might reduce translation performance in LRLs.
Transfer learning employs a model, trained on high-resource languages, to infer LRLs’ translations (Zoph et al., 2016). Chronopoulou et al. (2021) presented a meta-learning algorithm to improve UNMT models in low-resource domains by leveraging knowledge learned from high-resource domains, using slight training data to quickly adapt to new domains. Moreover, Li et al. (2022) continuously transferred knowledge from a high-resource parent model to a low-resource child model during training, ensuring prediction consistency between them.
Pivot-based translation (Leng et al., 2019) bridges from the source to intermediary languages and then onto the target language, facilitating translation when parallel datasets are insufficient. Kim et al. (2019) applied this concept in transfer learning, using pivot languages and parallel corpora for better translations. Currey and Heafield (2019) leveraged monolingual pivot language data to create pseudo-parallel corpora, augmenting data for training. Improper pivot languages chosen in these methods hinder translation results (Liu et al., 2019). Our approach incorporates a dynamic prompting graph to organize auxiliary languages for efficiently prompting LLMs.

heading: 2.2 Neural Machine Translation with LLMs
text: The community has explored various LLMs (Touvron et al., 2023a,b; OpenAI, 2022, 2023), which show promised performance in NMT (Jiao et al., 2023; Hendy et al., 2023; Zhu et al., 2023). Research exploring the translation capabilities of LLMs often involves ICL and fine-tuning methods.
As for ICL methods, Brown et al. (2020) explored the capabilities of LLMs to learn target
tasks with the prompt made up of in-context exemplars and templates. Garcia et al. (2023) showed comparable performances of ICL to those large, supervised models. Vilar et al. (2023); Agrawal et al. (2023) evaluated various strategies for selecting translation examples for ICL, emphasizing the importance of example quality.
As for fine-tuning methods, (Li et al., 2023) explored enhancing translation by fine-tuning XGLM7B (Lin et al., 2022), with translation instructions, especially for low-resource languages. Research (Alves et al., 2023) has shown that fine-tuning methods perform better than ICL in LRL translation but at a cost of high computational resources. In contrast, our work integrates two approaches by (1) training a sampling-based graph efficiently compared to billions of parameters, and (2) employing this graph to strategically organize auxiliary languages and construct prompts in ICL.

heading: 3 Methods

heading: 3.1 Overview
text: The proposed model, POMP (Fig. 1), consists of four modules: (1) UNMT-based Pseudo-parallel Generator (§3.2) generates pseudo-parallel sentences for both source-target and source-auxiliary pairs, where the auxiliary languages assist translation in LLMs’ prompts. (2) Language-specific Meta-graph (§3.3) carries relations among source, target, and auxiliary languages, which generates multiple translation paths to prompt LLMs. (3) Graph-Prompting LLM-based Translator (§3.4) generates multiple translation paths from the metagraph by sampling to prompt LLMs for translation. (4) Probabilistic Backward Graph Evolution (§3.5) updates the probabilities for paths in the meta-graph.
Pseudo-parallel sentences generated in §3.2 are used to calculate probability weights in §3.3, incorporated as prompts in §3.4. The meta-graph in §3.3 samples translation paths, which involve prompting LLMs in LRL translations in §3.4. The evaluation scores of the translations are back-propagated in §3.5 to update probabilities in the meta-graph.

heading: 3.2 UNMT-based Pseudo-parallel Generator
text: Following (Chen et al., 2022), we build a UNMT model to generate pseudo-parallel sentences for both source-target and source-auxiliary language pairs, where the auxiliary languages are extra languages assisting in prompting LLMs. We initialize
the transformer-based NMT model with weights of a multilingual pre-trained XLM-R model (Conneau et al., 2020) and then train with a two-stage training method on six auxiliary language datasets.
In the first stage, to preserve the cross-lingual transferability of the encoder, we train the decoder with the auxiliary-English pairs. In the second stage, to further improve the model learning from the training data, we jointly optimize all parameters of the encoder and decoder. Our empirical observation shows this processing transfers its translation ability learned from auxiliary languages in training to LRLs in testing.
We utilize the UNMT model to generate the translations of the LRL source as inputs in §3.4. Then we generate pseudo-parallel sentences of source-auxiliary pairs to construct the meta-graph in §3.3 and prompts in §3.4.

heading: 3.3 Language-specific Meta-graph
text: We design a language-specific meta-graph to explore diverse translation paths with high-resource auxiliary languages to prompt LLMs for LRL translations. We define the meta-graph as G = (V,E,W ), in which vertices (v ∈ V ) represent languages within the translation process, while an edge (e ∈ E) signifies a conditional transition from the current vertex to the next vertex. A path starts from the source, passes through multiple auxiliary languages, and ends with the target. A weight (w ∈ W ) assigned to each edge represents the conditional probability of transitioning from the current vertex (i.e. language) to the next vertex, given the all previous vertices in the path.
We construct the meta-graph with five steps as the left side of Fig. 1: (1) Vertex Initiation. Create two vertices to represent the source language and target language respectively; (2) Edge Establishing. Connect the source vertex to m different auxiliary vertices. Each auxiliary vertex represents a unique auxiliary language from a set of m available options and is assigned a unique probability computed in the next paragraph. An edge shows a directed connection from the existing vertex to the new-connected vertex; (3) Path Growth. Further extend directed connections of each auxiliary vertex above to the target vertex or all of the other auxiliary vertices, which are not previously connected in the preceding path. (4) Path Completion. A path contains contiguous connections between vertices and is complete if it reaches the target vertex, otherwise continue extending by step (3); (5)
Edge Weighting. Assign a weight for each edge with the geometric average of all probabilities of previous vertices in the path.
In the meta-graph constructed above, we obtain the unique probability of an auxiliary language considering the language similarity between it and the source, since the probability is designed to guide the organization of auxiliary language in constructing prompts for LLMs. The operations are as follows: (1) We first encode n pseudo-parallel sentences of source-auxiliary pairs (si, ai), (i ∈ n) generated in §3.2 to get vector pairs vsi ,vai = Encoder(si, ai); (2) To measure a language similarity between the source and an auxiliary language, we average a cosine similarity (cos) of all vector pairs; (3) Finally, we scale the similarity to the probability of the auxiliary language as p = exp (−1 + cos).
Above all, we construct a language-specific meta-graph that carries relations among the source, target, and auxiliary languages which sample to generate multiple translation paths to prompt LLMs in §3.4.

heading: 3.4 Graph-prompting LLM-based Translator
text: We explore diverse improved translations by sampling multiple translation paths from the metagraph in §3.3 to prompt an LLM as a translator
for LRLs. Specifically, we employ two operations: Generate and Aggregate. First, we independently sample multiple paths from the language-specific meta-graph. In Generate, we construct prompts with the individual auxiliary language at a vertex of the path. In Aggregate, we combine all the auxiliary languages in a path to construct prompts. We prompt LLMs to generate multiple improved LRL translations respectively. See Fig. 4 as a prompt example.

heading: 3.4.1 Sample
text: To explore diverse improved LRL translations, we sample multiple translation paths from the metagraph in §3.3 according to the probabilities of the paths. We obtain the probabilities of the paths by calculating the geometric average of the probabilities pj of m auxiliary language in the path as p[1,m] = ( m∏ j=1 pj) 1 m . These sampled paths provide diverse auxiliary languages and their combination to construct prompts in operation Generate and Aggregate.

heading: 3.4.2 Generate
text: To make full use of auxiliary language and explore LLMs’ ability, we execute a Generate operation for each vertex in a sampled path. As shown in Fig. 4, following ICL methods, we construct the
prompt with 4-shot examples and 1 query. Each example consists of 4 parts: a source sentence, its pseudo-parallel auxiliary sentence at a vertex, its translation output from §3.2, and its pseudo reference from Google Translate as a refined translation. The query contains an input source sentence and its other parts except for the refined translation as the example. The LLM is prompted to generate a refined translation for the input.
We perform the Generate operation once for each of the m auxiliary language vertices on an input source sentence, yielding m translations. For an unsupervised evaluation without ground truth, we need to obtain pseudo ground truth: sequentially select each of the m translations as a pseudo ground truth, and then we evaluate the remaining translations considering the pseudo ground truth to get m − 1 scores. The average of the m − 1 scores stands for an unsupervised evaluation result (gen-score, ei) of the selected (i-th) translation. In training and testing, we evaluate each output of Generate and feed the best output to Aggregate (§3.4.3) as its input.

heading: 3.4.3 Aggregate
text: To utilize multiple auxiliary languages and merge the useful information to feed into LLMs, we execute a Aggregate operation, in which we aggregate all auxiliary languages of the entire sampled path to prompt LLMs. Similar to Generate, the Aggregate operation applies the ICL paradigm with a 4-shot example and 1 query format as the illustrated example in Fig. 4. However, Aggregate differs from Generate as (1) it aggregates pseudo-parallel sentences of all auxiliary languages along the path into prompts, whereas Generate uses sentences of a single auxiliary language at each vertex; (2) the query in Aggregate includes the improved translation from the Generate instead of the original translation.
While Generate operates totally m times to produce m translations, Aggregate is conducted once, aggregating all auxiliary languages in the path to prompt LLMs, yielding one translation. For an unsupervised evaluation without ground truth, we regard the Aggregate’s translation as the pseudoground truth. We calculate the average score of evaluating the Generate’s m translations considering this pseudo ground truth as the unsupervised evaluation score (agg-score, E). In training, the evaluation score of Aggregate’s output carries backpropagated rewards to update probabilities of in-
volved auxiliary languages in §3.5. In testing, we select the highest scoring one among the m Generate’s outputs and one Aggregate output as the final output.

heading: 3.5 Probabilistic Backward Graph Evolution
text: To make the meta-graph better serve for sampling translation paths to prompt LLMs (§3.4), we backpropagate evaluation scores in Aggregate operations to update probabilities of auxiliary languages in the meta-graph (§3.3), which consists of three steps: (1) quantify the contributions (di) of all individual auxiliary languages made for the translation, (2) normalize the contributions (di) to obtain rewards (ri), (3) back-propagate the rewards (ri) to update the probabilities (pnewi ) of auxiliary languages in the meta-graph. Details are as follows.
Quantify contributions. As all auxiliary languages in a sampled translation path contribute to an Aggregate operation’s output, we quantify each auxiliary language i’s contribution di to the translation output. First, we assume all m auxiliary languages used in the translation path equally contribute to the output translated via this translation path. As the translated output is measured by agg-score E in §3.4.3, we equally assign E to contributions as E = ∑m i di. Second, to reflect each language’s contribution, we use gen-score ei in §3.4.2 for auxiliary language i. Introducing ei, we obtain the translation score without considering each language E − ei, and the corresponding contribution without the language i is ∑m−1 i di. Third, we apply the above calculations to all languages and obtain Eq. 1.
   E − e1 = d2 + d3 + · · ·+ dm E − e2 = d1 + d3 + · · ·+ dm
... E − em = d1 + d2 + · · ·+ dm−1
(1)
Further, we obtain contribution di for each language i by combining m equations in Eq. 1 (See deductions in Appendix B).
di = E − 1
m− 1
  m∑
j=1 j ̸=i
ej   (2)
Normalize contributions as rewards. We normalize the contribution di to obtain reward ri,
where ri falls into a range of [0, 1] so that the updated probability is constrained in a range of [0, 1]. We employ a central-symmetry Swish (cs-Swish) function (Ramachandran et al., 2017) to normalize di to ri.
cs-Swish(x) = { −x · Sigmoid(−x), if x ≤ 0; x · Sigmoid(x), if x > 0. (3)
We get ri by combining Eq. 2 and Eq. 3 as:
ri = cs-Swish(di)
= { −di · 11+edi , if di ≤ 0; di · 11+e−di , if di > 0.
(4)
Update probabilities. For one training sample and its sampled translation path, we now obtain m respective rewards ri(i = 0, 1, · · · ,m) of the m auxiliary languages in the path. We back-propagate the ri to update the probability (pnewi ) of the i-th auxiliary language with a learning rate (lr) and its previous probability (poldi ) as
pnewi = (1 + lr · ri) · poldi . (5)
As a result, one training iteration updates the probabilities of its involved auxiliary languages in the meta-graph, which continues to serve for the next iteration. When training converges, we use the translation paths sampled in the last iteration for testing.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Languages. We select four LRL pairs, including Gujarati (Gu)→English (En), Kazakh (Kk)→En, Nepail (Ne)→En, and Sinhala (Si)→En to train their respective prompting graph with our approach and then to test. Following Chen et al. (2022), we utilize German (De), Spanish (Es), Finnish (Fi), Hindi (Hi), Russian (Ru), and Chinese (Zh) as the auxiliary languages, which are high-resource languages from different language families. Datasets. To construct pseudo-parallel datasets for training, we collect datasets from the OPUS2 (Tiedemann, 2012, 2016). Specifically, the datasets are from WMT (Gu, Kk) and CCAligned (Ne, Si). Then we randomly sample 1000 sentences for each dataset and translate the source side with the UNMT model in §3.2. The testing data is from
2opus.nlpl.eu
newstest3 (Gu, Kk) and Flores-200 Testset4 (Team et al., 2022) (Ne, Si). Evaluation Metrics. Recent work has shown that n-gram metrics like BLEU (Papineni et al., 2002) are sub-optimal for evaluating high-quality translations (Kocmi et al., 2021; Freitag et al., 2021). As recommended in Freitag et al. (2022), neural network-based metrics demonstrate a high correlation with human evaluation. Therefore, we adopt COMET5 (Rei et al., 2020), xCOMET5 (Guerreiro et al., 2023), and BLEURT6 (Sellam et al., 2020) as our evaluation metric. Specifically, we use BLEURT-207 model in Pytorch implementation8 for BLEURT metric. As for COMET and xCOMET, we use the wmt22-comet-da model9 and XCOMET-XL model.10 Baselines. We compare with three non-LLM baselines: (1) CRISS (Tran et al., 2020), initialized with mBART and fine-tuned on 180 kinds of translation pairs from CCMatrix dataset; (2) m2m-100 (Fan et al., 2020), a supervised multilingual NMT model trained with 7.5B parallel sentences from CCMatrix and CCAligned datasets; (3) SixT+ (Chen et al., 2022), initialized with XLM-R-large (Goyal et al., 2021), learns on high-resource auxiliary language and inference on LRLs in an unsupervised way. We compare with three LLM baselines: (1) ChatGPT-QS (Hendy et al., 2023), investigating prompt learning on ChatGPT (OpenAI, 2022) on LRL setting; (2) ChatGPT-ICL (Zhu et al., 2023), applying in-context learning to achieve the translation of ChatGPT; (3) ChatGPT-trans, employing ICL in the prompt with 4-shot examples consisting of 4 source sentences and their pseudo-parallel pairs to ask ChatGPT to translate the testing set (See its prompt text in Fig. 5).
See the implementation details in Appendix A.

heading: 4.2 Overall performance
text: Tab. 1 shows the results of all comparing methods. POMP outperforms all the baselines on all metrics. In LRL translations of non-LLM baselines,
3data.statmt.org/wmt19/translation-task 4https://tinyurl.com/flores200dataset 5https://github.com/Unbabel/COMET/ 6https://github.com/google-research/bleurt 7https://github.com/google-research/bleurt/
blob/master/checkpoints.md 8https://github.com/lucadiliello/ bleurt-pytorch 9https://huggingface.co/Unbabel/ wmt22-comet-da 10https://huggingface.co/Unbabel/XCOMET-XL
POMP outperforms unsupervised CRISS, supervised m2m-100, and state-of-the-art SixT+, which indicates the effectiveness of our approach. In LRL translations of LLM baselines, we compare the POMP with the other three LLM-based baselines: ChatGPT-QS, ChatGPT-ICL, and ChatGPT-trans. They all fall behind POMP, which verifies the effectiveness of our strategy of constructing prompts. Notably, for Ne→En translations, all LLM-based baselines underperform non-LLM baselines, indicating that LLMs hardly do well in LRL translations. POMP improves Ne→En translations for LLMs to outperform non-LLM baselines, demonstrating our approach’s effectiveness. Additionally, we conduct experiments on the GPT-4 and show results at Tab. 5 in Appendix C, which are also better than the initial UNMT system. POMP has similar performance on different LLMs, indicating its generality.

heading: 4.3 Ablation Study
text: Tab. 2 presents ablation studies on our model. Our full model outperforms in all metrics across four LRLs, except for the variant w/o updating on the xCOMET metric of Ne. The variant w/o auxiliary, removing all auxiliary languages in prompts, underperforms POMP, which shows helpful assistance of auxiliary languages. W/o Generate excludes Generate operations from POMP, focusing solely on Aggregate to combine all auxiliary languages of a translation path into prompts. Conversely, w/o Aggregate excludes Aggregate from POMP, using
a single auxiliary language in a prompt. Both of them fall behind POMP, indicating that the two operations play vital roles in POMP. In w/o updating, we fix the probabilities of selecting auxiliary languages as their initial ones calculated by language similarities in §3.3, of which the results show the validity of the graph evolution strategy in §3.5. While the Ne→En translation in w/o updating excels POMP, indicating that Ne’s language similarities with auxiliary languages provide solid probabilities for organizing them in prompts. W/o scoring represents the strategy in which we randomly choose the final translation output among translations of Generate and Aggregate. Its poor performances verify the success of our unsupervised evaluation methods in §3.4. W/o meta-graph remove the constructed meta-graph (§3.3) and randomly organize the auxiliary languages in prompts for LLMs. Results show that the translation paths sampling from the meta-graph offer practical improvements to prompt LLMs in LRL translations.

heading: 4.4 Analysis of the Linguistically Biased Translation Errors
text: We attempt to quantify the linguistically biased translation errors of POMP versus some straightforward usages. Specifically, we capture the linguistic characteristics (i.e. output token distributions) on (1) the LLM’s results using a single auxiliary language (1-auxiliary), (2) our full model’s results, and (3) the ground truth. Then, we measure POMP’s linguistically biased errors by evaluating
the distribution gap between the ground truth and POMP’s results; we measure 1-auxiliary’s linguistically biased errors by evaluating the distribution gap between the ground truth and 1-auxiliary’s results. We use Jensen-Shannon divergence (JSD) to quantify the distribution gap. In Tab. 3, 1- auxiliary’s JSD is higher, showing that a single auxiliary language suffers from biases from this language and is likely to cause translation errors in LLM’s output. Conversely, POMP achieves much smaller distribution gaps across 4 LRLs, indicating fewer translation errors in POMP’s results.
We also visualize the gaps in Fig. 2 and Fig. 3 Appendix D. We apply kernel density estimation (KDE) to estimate the discrete token frequencies as continuous curves, which sensitively reflect slight variations in the shapes of distributions. We observe that POMP’s distribution matches well with the reference across 4 LRLs, while 1-auxiliary’s distribution falls far behind.

heading: 4.5 Analysis of the Prompting Graph
text: To explore the fitness of POMP’s prompting graphs, we analyze prompting graphs’ structures by measuring degrees of vertices and lengths of translation paths. As a vertex represents a language, we measure the degrees of 8 kinds of vertices (source, target, 6 auxiliary languages). The degrees mean the connections of vertices, which measure the usages
of auxiliary languages. The length of a translation path represents the number of edges in the path. The average degree of total vertices is 3.27, indicating that vertices in prompting graphs connect well with each other. The lengths of translation paths range from [2, 7], corresponding to the involvement of one to six auxiliary languages. This range suggests the effective organization of auxiliary languages for LRL translations, facilitating efficient translation pathways. The average length of total translation paths is 3.42, indicating that POMP utilizes 2 or 3 auxiliary languages to prompt LLMs. The results in Tab. 4 show that graphs of each testing language in POMP fit well for LLMs’ prompts.

heading: 4.6 Case Study
text: We show the cases of translation outputs of POMP versus baselines in Fig. 9 and Fig. 10. POMP generates fluent and accurate translations in all cases. Non-LLM baselines like CRISS and m2m-100 are likely to mistranslate words and repeat meaningless words, while SixT+ achieves better but less accuracy than POMP. LLMs-based baselines tend to generate facts unrelated to the given source. As a comparison, cases show that POMP’s outputs with more precise words and fewer repetitions are consistent with references.

heading: 5 Conclusion
text: In summary, we propose POMP, an unsupervised method that mitigates linguistically biased translation errors in UNMT on LRLs. This approach involves constructing a language-specific metagraph, from which we sample multiple translation paths with organized auxiliary languages to prompt LLMs as LRL translators. We promote the evolution of the meta-graph by back-propagating evaluation scores to update probabilities of involved auxiliary languages in the graph. We use three metrics for evaluations in testing. Our experiment results demonstrate the effectiveness of our approach, which achieves SOTA performances on four LRLs.

heading: 6 Limitations
text: In our work, there are several limitations. (1) We use GPT-3.5 in our approach rather than GPT-4, and GPT-3.5 is not the most advanced GPT API currently available, which seems to limit the performance of our model. The main reason is that LLM-based baselines use GPT-3.5, we follow them
for a fair comparison. Meanwhile, the cost of utilizing GPT-4 is significantly higher than that of GPT-3.5. According to the official website, the fee for GPT 3.5 is 0.002$/1k Token, and the fee for GPT 4 is 0.06$/1k Token. In our experiments, the API fee of GPT-3.5 costs about 130$ in total. If the GPT 4 is used, the same number of tokens will cost about 3900$ in total. Hence, employing GPT-3.5 not only results in a substantial reduction in costs but also yields a largely comparable generation effect. Currently, we are utilizing GPT-3.5 for the validation of our methods, with plans to employ GPT-4 for effect validation in subsequent phases. (2) POMP is limited for translation tasks that require precise words (evaluated by BLEU-like score). Since LLMs are pre-trained and fine-tuned with large amounts of data in various domains, they prefer diverse generations. We encourage future works with a constrained dictionary and precise instructions in prompts to achieve accurate translations.

heading: 7 Ethical Considerations
text: We take ethical considerations very seriously, and strictly adhere to the ACL Ethics Policy. (1) LLM training predominantly relies on data from highresource languages, potentially leading to biases in low-resource languages (LRL) translation, exacerbating linguistic and cultural biases. (2) LLMs may lack exposure to specific customs, beliefs, or values prevalent in LRL, resulting in translation inaccuracies, misunderstandings, and potential offense. To mitigate these ethical challenges, we plan to implement restrictions within the prompt templates to discourage the use of discriminatory or biased language. Moreover, we plan to compile a
sensitivity lexicon to identify sensitive words, enabling us to either avoid translating them or to find alternative translations. Thus, we believe that the ethical issues raised in this research can be handled with some carefully designed strategies and thus the usage of our model would not cause serious ethical problems.

heading: Acknowledgements
text: This work is supported by the following foundations: the National Natural Science Foundation of China under Grant No.62025208, the Young Elite Scientist Sponsorship Program by CAST (2023QNRC001) under Grant No.YESS20230367, the National Natural Science Foundation of China under Grant No.62306330, No.62106275, the Xiangjiang Laboratory Foundation under Grant No.22XJ01012, and the Natural Science Foundation of Hunan Province under Grant No.WDZC20235250103.

heading: B Details for Solving Eq. 1
text: To directly derive a formula for di from the equations, we follow a mathematical process:
Given the system of equations:
   E − e1 = d2 + d3 + · · ·+ dm E − e2 = d1 + d3 + · · ·+ dm
... E − em = d1 + d2 + · · ·+ dm−1
we want to calculate the expression for di. 1. Sum all equations to get a total that combines each E − ei term:
mE−(e1+e2+· · ·+em) = (m−1)(d1+d2+· · ·+dm)
2. Solve for the total sum of di’s:
m∑
i=1
di = mE −∑mi=1 ei
m− 1

heading: 3. Subtract the sum of all ei’s except ei from
text: mE to isolate the contributions relevant to di:
Total without ei = mE − m∑
j=1 j ̸=i
ej
4. Derive di by recognizing it’s the missing piece in each equation’s total sum, balanced by m− 1:
di = E − 1
m− 1
  m∑
j=1 j ̸=i
ej  

heading: C Performance on the GPT-4
text: We conduct experiments on the GPT-4 (POMPgpt4) and show results at Tab. 5. The additional results on GPT-4 are also better than the initial UNMT system. POMP has similar performance on different LLMs, indicating its generality.
D Visualizations of Distribution Gaps
We apply kernel density estimation (KDE) for the discrete token frequencies to visualize the distributions in continuous curves. The horizontal axis represents the token IDs that have appeared in involved sentences, and the vertical axis represents the kernel density. While KDE distributions of POMP perform well with the references, 1-auxiliary’s distributions fall further behind than JDS values in 3 do. The reasons are as follows: (1) KDE can sensitively reflect slight variations in the shapes of distributions, which might not be as apparent in JSD calculations. (2) Distribution graphs reveal differences in the shape between distributions, such as the location and width of peaks. These shape differences might contribute less to JSD than to visual impressions. (3) If the distributions are multimodal (i.e., they have multiple peaks), distribution graphs might show significant visual differences even if JSD is relatively small. Overall, Fig. 3 shows the visualized significant effectiveness of our approach in mitigating linguistically biased translation errors.

heading: Future Work
text: We encourage future works with a constrained dictionary and precise instructions in prompts to achieve accurate translations