Translation and Fusion Improves Zero-shot Cross-lingual Information Extraction

## Abstract

Abstract Large language models (LLMs) combined with instruction tuning have shown significant progress in information extraction (IE) tasks, exhibiting strong generalization capabilities to unseen datasets by following annotation guidelines.
However, their applicability to low-resource languages remains limited due to lack of both labeled data for fine-tuning, and unlabeled text for pre-training.
In this paper, we propose TransFusion, a framework in which models are fine-tuned to use English translations of low-resource language data, enabling more precise predictions through annotation fusion.
Based on TransFusion, we introduce GoLLIE-TF, a cross-lingual instruction-tuned LLM for IE tasks, designed to close the performance gap between high and low-resource languages.
Our experiments across twelve multilingual IE datasets spanning 50 languages demonstrate that GoLLIE-TF achieves better zero-shot cross-lingual transfer over the base model.
In addition, we show that TransFusion significantly improves low-resource language named entity recognition when applied to proprietary models such as GPT-4 (+5 F1) with a prompting approach, or fine-tuning different language models including decoder-only (+14 F1) and encoder-only (+13 F1) architectures. 1 1 1 Code and data is available at: https://github.com/edchengg/gollie-transfusion

## 1 Introduction

The task of information extraction (IE) is challenging due to fine-grained annotation guidelines for span-level annotations.
Fortunately, recent advances in instruction-following large language models (LLM) (Ouyang et al., ; Team et al., ) such as GoLLIE (Sainz et al., ) have demonstrated the ability to perform zero-shot IE without labels using annotation guidelines.
However, these models are often pre-trained on English-centric data (Touvron et al., ; Roziere et al., ). Even state-of-the-art proprietary models such as GPT-4 exhibit significant performance degradation from 80 English F1 to 55 F1 on low-resource African languages, as shown in Figure 1 (right).

To improve NLP on low-resource languages, the research community has turned to machine translation to translate fine-tuning datasets (translate-train) and translate test data into high-resource languages for easier processing (translate-test) (Hu et al., ). Recent studies (Shi et al., ; Huang et al., ) on prompting LLMs with translated data have shown improvements on diverse tasks such as math reasoning and summarization.
Prior work has explored the use of machine translation to improve multilingual instruction-following on traditional NLP benchmarks, such as natural language inference, and sentiment analysis, however, the use of MT to improve instruction-following IE models is less explored, as there is not a trivial alignment between labels in the native language and translated texts (Ahuja et al., ).
With recent efforts to develop machine translation (MT) models such as M2M (Fan et al., ) and NLLB-200 (Costa-jussà et al., ) that better support low-resource languages, we study how to teach LLMs to leverage an external MT system in a resource-efficient manner to improve low-resource IE.

In this paper, we propose a Translation and Fusion (TransFusion) framework, which aims to teach models to use an external translation model to make better predictions on low-resource languages.
The framework includes three steps: (1) translating low-resource data into English at inference time, to be annotated by a high-resource model. Next, (2) these span-annotated English translations are combined with low-resource language text in a fusion model that is trained to make predictions conditioned on both types of data. Finally (3), the language model generates a TransFusion reasoning chain (annotate and fuse) in a single autoregressive decoding pass.
To train TransFusion models, we construct cross-lingual instruction fine-tuning data by translating and projecting labels from English IE datasets to low-resource languages using EasyProject (Chen et al., ), a simple, yet effective method that has been shown to scale across many NLP tasks and languages.

Our zero-shot cross-lingual IE evaluation reveals that the TransFusion fine-tuned model, GoLLIE-TF, outperforms the base GoLLIE model across 50 languages, spanning high, mid, and low-resource categories, on both seen and unseen label schemas. Notably, in our evaluation on African language named entity recognition (NER) using the MasakhaNER2 dataset (Adelani et al., ), GoLLIE-TF achieves significant improvements in F1 scores compared to the base model and shows an average improvement of +6.6 F1 on unseen label schema datasets.
Furthermore, we demonstrate that the TransFusion framework enhances GPT-4’s performance on MasakhaNER2, yielding an average +5.7 F1 score improvement, and substantially boosts the encoder-only African language model, AfroXLM-R (Alabi et al., ), by +13.3 F1. Our analysis underscores the effectiveness of the TransFusion framework for low-resource language tasks.

Figure: Figure 1: Our TransFusion framework aims to bridge the performance gap between high and low-resource languages on information extraction tasks. (left) TransFusion reasoning includes three steps: translate, annotate, and fuse. (right) GoLLIE-TF shows superior zero-shot cross-lingual evaluation on a range of IE datasets over the base model.
Refer to caption: x1.png

## 2 Background: Annotation Guideline Following LLMs for IE

In this paper, we employ the GoLLIE model (Sainz et al., ), which has been instruction-tuned on English Information Extraction (IE) tasks using label schema guidelines, to achieve state-of-the-art zero-shot IE on unseen datasets. GoLLIE utilizes a Python code representation for both inputs and outputs, providing a clear and human-readable structure that unifies various IE annotation tasks. Each label schema is encapsulated as a Python class object, with the annotation guidelines embedded as strings within these objects (an example of a GoLLIE prompt is provided in the the Appendix in Figure 6.

Limitation of Cross-lingual
Transferbilitiy: Despite GoLLIE’s impressive performance, it is designed for use on English, as it is primarily fine-tuned on English data. This limitation is shown in Figure 1 (right), where we see a significant drop in performance on low-resource African languages, from 95 to 48, compared to English.
In this study, we experiment with zero-shot cross-lingual transfer, where human-labeled data in the target languages are assumed to be unavailable. Collecting such data is costly and time-inefficient, as it requires well-trained native language speakers.
Recent efforts, such as NLLB-200 (Costa-jussà et al., ), have focused on gathering low-resource translation data to train multilingual MT models capable of translating across 200 languages. Building on this progress, we explore whether an instruction-tuned information extraction model can learn to use an external translation model (Schick et al., ) to enhance performance on low-resource language IE tasks. This offers an efficient and effective alternative to computationally intensive pre-training based methods for adapting to new languages (Scao et al., ; Xue et al., ; Alabi et al., ; Üstün et al., ).

## 3 Using Low-Resource Machine Translation to Improve Multilingual IE

As multilingual machine translation (MT) systems, such as M2M-100 (Fan et al., ) and NLLB-200 (Costa-jussà et al., ), gain increasing support for low-resource languages, an opportunity emerges to re-evaluate the utilization of MT systems for enhancing cross-lingual IE.
We propose a Translation-and-fusion approach that benefits from the advancements of MT systems to make robust cross-lingual transfer predictions at inference time.
In this section, we outline the Translation-and-fusion framework and introduce language models trained to utilize translation data at inference time for low-resource language IE tasks.

### 3.1 Translation-and-Fusion (TransFusion)

#### Cross-lingual Transfer.

The conventional cross-lingual transfer method involves fine-tuning a pre-trained language model, on high-resource language annotated data ($src$) and evaluating its performance on test data in other languages $(tgt)$.

In accordance with the low-resource assumption, we assume access to an annotated dataset in the high-resource language (usually English), $\mathcal{D}_{src}={(x_{src}^{i},y_{src}^{i})}_{i=1}^{N}$.
The task-specific fine-tuning loss is formulated as:

$$ $\mathcal{L}(\theta,\mathcal{D}_{src})=\sum_{(x_{src},y_{src})\in\mathcal{D}_{ src}}\mathcal{L}(P(y|x_{src};\theta),y_{src})$ $$

However, previous studies have highlighted the limited performance of fine-tuned models on languages that were unseen during pre-training or are under-represented in the pre-training data (Adelani et al., ; Ebrahimi et al., ). As an additional approach to adapt to low-resource languages (Wang et al., ), we describe the translation-and-fusion framework, which leverages annotations on (translated) high-resource language text to steer predictions on a low-resource language at inference time. The framework encompasses three key steps:

- •
Translate: Use an MT system to translate low-resource language test data into a high-resource language, $\text{MT}(x_{tgt})\mapsto x^{\text{\scriptsize{trans}}}_{src}$.
- •
Annotate: Making predictions to the (high-resource) translated text using a high-resource language supervised fine-tuned model $P(;\theta_{src})$:
$\mathrm{argmax}_{y}\{P(y|x^{\text{\scriptsize{trans}}}_{src};\theta_{src})\}
\mapsto\tilde{y}^{\text{\scriptsize{trans}}}_{src}$.
- •
Fuse:
Given the annotations on the translated data from the previous step ($\tilde{y}^{\text{\scriptsize{trans}}}_{src}$), a fusion model combines the high-resource predictions together with the target language text to make final predictions.

Based on the framework outlined above, we present TransFusion, a fusion model that is trained to makes predictions on the test data conditioned on annotations from the corresponding translated data ($\tilde{y}^{\text{\scriptsize{trans}}}_{src}$):

$$ $\mathrm{argmax}_{y}\{P(y|x_{tgt},x^{\text{\scriptsize{trans}}}_{src},\tilde{y} ^{\text{\scriptsize{trans}}}_{src};\theta_{\text{fusion}})\}\mapsto y_{tgt}^{\prime}$ $$

Below, we describe the training procedure of TransFusion, starting with the approach to create training data.

#### Training Dataset.

To learn a TransFusion model, parallel sentences with IE task annotations on both high-resource and low-resource languages are essential.
To fulfill this requirement, we translate high-resource annotated training data into a list of target languages, while projecting span-level annotations, using a simple mark-then-translate approach - EasyProject (Chen et al., ): MT$(x_{src},y_{src})\to(x^{\text{\scriptsize{trans}}}_{tgt},y^{\text{\scriptsize{
trans}}}_{tgt})$.
We then pair the translation outputs with the original high-resource language data to create a training data set with a mixture of both parallel sentences: $\mathcal{D}_{mix}=\{x_{src},y_{src},x^{\text{\scriptsize{trans}}}_{tgt},y^{
\text{\scriptsize{trans}}}_{tgt}\}^{N}_{i=1}$.

#### Learning.

We train the fusion model $P(;\theta_{\text{fusion}})$ on $\mathcal{D}_{mix}$ using cross-entropy loss:

$$ $\mathcal{L}_{\text{fusion}}(\theta,\mathcal{D}_{mix})=\sum_{\begin{subarray}{c }(x_{src},y_{src},x^{\text{\scriptsize{trans}}}_{tgt},y^{\text{\scriptsize{ trans}}}_{tgt})\in\mathcal{D}_{mix}\end{subarray}}\mathcal{L}(P(y|x^{\text{ \scriptsize{\text{\scriptsize{trans}}}}}_{tgt},x_{src},y_{src};\theta_{\text{ fusion}}),y^{\text{\scriptsize{trans}}}_{tgt})$ $$

The model architecture can vary, encompassing both decoder-only language models (e.g., LLaMA (Touvron et al., )) and encoder-only language models (e.g., mBERT (Devlin et al., )). In this work, we primarily utilize decoder-only language models to integrate the annotate and fuse steps in an autoregressive manner during inference. Additionally, we assess the performance of encoder-only models in Section 5.3 to demonstrate the robustness of our framework across different architectures.

#### Training a Decoder-only LM (GoLLIE-TF).

To implement our TransFusion framework within the instruction-following GoLLIE model, we represent the framework as natural language instructions, providing the model with supplementary English translation text of the original target language sentence, which is illustrated in Figure 1 (left). The TransFusion instruction specifies the output format, guiding the model to first generate annotations for the English translation and subsequently for the target language data, using the English annotations as context (an example can be found in Appendix Figure 6
). This autoregressive approach enables the model to perform the annotate and fuse steps concurrently during inference.
During training, we fine-tune the GoLLIE model to adhere to these instructions, ensuring it generates annotations for both the English and target language data sequentially. We apply the next token prediction loss to the tokens following the TransFusion instruction, as detailed below:

$$ $[\texttt{GoLLIE Guidelines},x,x^{trans},\texttt{TransFusion Instruction}] \stackrel{{\scriptstyle\text{LLM}}}{{\xrightarrow{\hskip 28.45274pt}}}[y^{ trans},y]$ $$

#### Training and Inference with Encoder-only LMs.

Given that encoder-only models are not inherently designed for text generation, we employ a two-step pipeline approach for inference in TransFusion: annotation and fusion.
First, we utilize an English fine-tuned model to annotate the English translation of the target language text. These annotations are marked using XML tags around the relevant spans (e.g., <PER> … </PER>).
Next, we construct the input for the fusion model by embedding these annotations into the English translation. We concatenate the annotated English translation ($x^{trans}$) with the original target language text ($x$), using a marker ($||$) to separate the two segments. The input to the encoder is formatted as follows:

$$ $[x^{trans}_{1},x^{trans}_{2},\texttt{<PER>},x^{trans}_{3},x^{trans}_{4}, \texttt{</PER>},x^{trans}_{5},||,x_{1},x_{2},x_{3},...]$ $$

At training time, we add a linear classification layer to classify each token and only apply the cross-entropy loss to the target language tokens (right of the separation token $||$).

## 4 Experimental Setting

**Table 1: Datasets used in the experiment. The table shows the task, domain, whether it was used in the training and evaluation including the number of languages in the evaluation set.**
| Training Dataset | Domain | Tasks | Language |
| --- | --- | --- | --- |
| CoNLL 03(Tjong Kim Sang and De Meulder, ) | News | NER | English |
| BC5CDR (Li et al., ) | Biomedical | NER | English |
| NCBIDisease (Dogan et al., ) | Biomedical | NER | English |
| OntoNotes 5 (Pradhan et al., ) | News | NER | English |
| WNUT 2017 (Derczynski et al., ) | News | NER | English |
| RAMS (Ebner et al., ) | News | Arg. Extraction | English |
| TACRED (Zhang et al., ) | News | Slot Filling | English |
| CoNLL 04 (Roth and Yih, ) | News | Relation Extraction | English |
| ACE (Walker et al., ) | News | EE, EAE, NER, RE | English |

We use a collection of English Information Extraction (IE) datasets for supervised fine-tuning and multilingual IE datasets for evaluation (see Table 1).
Assessing cross-lingual transfer capabilities requires IE datasets annotated in a diverse set of languages. To this end, we gather multilingual Named Entity Recognition (NER) datasets from MasakhaNER2.0 (Adelani et al., ) (20 African languages) and UNER (Mayhew et al., ) (13 languages) to conduct low-resource language evaluation on label schemas that are seen during fine-tuning. In addition, we evaluate on unseen label schemas using the non-English subset of ACE2005 (Tjong Kim Sang and De Meulder, ) (Chinese and Arabic), which includes several tasks: NER, RE, Event Extraction (EE), and Event Argument Extraction (EAE).
For evaluation on labels that were unseen during fine-tuning, we use MultiNERD (Tedeschi and Navigli, ) (10 high-resource languages), MultiCoNER2 (12 high-resource languages) (Fetahu et al., ), in addition to Slot Intent Detection data from MultiTO (Schuster et al., ), xSID (10 high-resource languages) (van der Goot et al., ),
a subset of Massive (15 low-resource languages were determined based on the NLLB categorization (Costa-jussà et al., )) (FitzGerald et al., ) and Relation Extraction (RE) data from RED-FM (7 high-resource languages) (Cabot et al., ).
We adopt the data pre-processing and task formulation methodologies used by GoLLIE and use publicly available English training data from GoLLIE to train the model. Due to the high cost of inference with GPT-4, we use 200 examples per language, per task, for evaluation.

Multilingual Translation Data.
The TransFusion framework relies on a machine translation system as a core component. In this paper, we utilize the state-of-the-art open-source multilingual translation model - NLLB-200 (Costa-jussà et al., ), which has 3.3 billion parameters and supports translation between 200 languages. The NLLB-200-3.3B model translates target language test data into English at test time. For TransFusion training data, a marker-based translation approach named EasyProject (Chen et al., ), powered by the NLLB-200 model, translates English training data into a collection of 36 target language candidates.
From this translated data, 8 examples per language and each task are randomly sampled, resulting in around 20-40 examples per language.
In total, 20,000 examples are used, combining English (19,109) and translated data (891), to train the TransFusion model (See per task and per language distribution in Appendix Figure 8).
This small portion of translation data ensures that the GoLLIE model generalizes to unseen labels while maintaining English performance.

### 4.1 Language Models and Baselines

#### Models:

We adopt GoLLIE-7B as our primary starting checkpoint. GoLLIE is an instruction fine-tuned version of CodeLLaMA (Roziere et al., ) that is trained on approximately 500,000 English demonstrations. Although the model was not explicitly pre-trained on multilingual data, its pre-training corpus includes a substantial amount of high-resource language content, such as Wikipedia, covering a diverse linguistic range (Touvron et al., ). This makes GoLLIE-7B an appropriate testbed for examining the adaptation of English-centric LLMs to low-resource languages that may be underrepresented in pre-training.
In addition to this decoder-only LLM, we explore encoder-only models specifically pre-trained on African languages, such as AfroXLM-R (Alabi et al., ) in Section 5.3.

#### Training Setup:

Initilized from GoLLIE-7B, we continue fine-tuning the model on a dataset of 20,000 TransFusion training examples using QLoRA (Dettmers et al., ). QLoRA has been shown to better maintain the base model’s performance (Biderman et al., ) and offers faster training times compared to full fine-tuning.
To implement this, we freeze the transformer model weights and apply LoRA (Hu et al., ) to all linear layers within all the transformer blocks. We set the LoRA rank to 128 and the alpha parameter to 16 based on preliminary experiments as we found smaller alpha leads to more stable training and higher rank for fast convergence.
We use the AdamW optimizer (Kingma and Ba, ) with a batch size of 16 and a learning rate of 1e-4, managed by a cosine scheduler. The training process was conducted on a setup of 2 NVIDIA A40 GPUs, each equipped with 48GB of memory. The entire experiment session spanned approximately 6 hours. We use greedy decoding at inference time.

#### Baselines:

We compare to both the base GoLLIE model, in addition to GPT-4, which represents a state-of-the-art proprietary model pre-trained on multilingual corpora (Achiam et al., ).
We report few-shot prompting results using GPT-4 (gpt4-02-14) with a GoLLIE style prompt.
Additionally, we explore the application of the TransFusion framework to GPT-4 in Section 5.2.
Furthermore, we use Translate-train (Trans-train) (Hu et al., ) as another baseline, which shows strong improvements over English fine-tuned (English FT) models (Chen et al., ). We use the same translated training data used by TransFusion and fine-tune GoLLIE-7B on a total of 20,000 examples (English + translated data).

**Table 2: Zero-shot cross-lingual transfer performance (F1 score). The table compiles all the seen label schema and unseen label schema evaluation results. Blue numbers highlight the performance improvements over GoLLIE-7B ($\Delta$). Full results for each language can be found in Appendix. (2024/6/13: fixed data processing bugs when reporting results for ACE/REDFM.)**
| Task | Benchmark | GPT-4 | GoLLIE${}_{\text{7B}}$ | Trans-Train | GoLLIE-TF |
| --- | --- | --- | --- | --- | --- |
| Seen Label Schema |  |  |  |  |  |
| NER | MasakhaNER2 (20 languages) |  |  |  |  |
|  | Bambara | 42.2 | 38.9 | 40.1 | 54.8 (+15.9) |
|  | Ghomala | 58.2 | 43.7 | 49.2 | 50.2 (+6.5) |
|  | Ewe | 72.2 | 74.0 | 73.1 | 73.2 (-0.8) |
|  | Fon | 39.4 | 49.7 | 55.7 | 57.9 (+8.2) |
|  | Hausa | 65.9 | 57.1 | 55.6 | 67.1 (+10.0) |
|  | Igbo | 42.2 | 51.1 | 42.4 | 56.6 (+5.5) |
|  | Kinyarwanda | 47.5 | 45.0 | 47.7 | 58.5 (+13.6) |
|  | Luganda | 62.5 | 61.8 | 66.8 | 75.5 (+13.7) |
|  | Luo | 47.2 | 36.5 | 42.8 | 51.7 (+15.3) |
|  | Mossi | 43.2 | 45.1 | 46.1 | 48.8 (+3.7) |
|  | Chichewa | 71.1 | 39.1 | 59.8 | 78.2 (+39.1) |
|  | Naija | 78.9 | 75.9 | 74.9 | 81.1 (+5.2) |
|  | Shona | 39.5 | 39.7 | 50.4 | 57.4 (+17.6) |
|  | Swahili | 79.2 | 66.9 | 68.3 | 73.5 (+6.5) |
|  | Tswana | 56.3 | 52.1 | 58.9 | 71.0 (+18.9) |
|  | Twi | 44.2 | 41.7 | 50.6 | 74.2 (+32.5) |
|  | Wolof | 52.6 | 49.1 | 55.5 | 61.9 (+12.8) |
|  | Xhosa | 49.8 | 29.2 | 47.6 | 49.9 (+20.7) |
|  | Yoruba | 54.7 | 35.7 | 39.3 | 54.4 (+18.7) |
|  | Zulu | 36.9 | 25.6 | 31.7 | 52.8 (+27.2) |
|  | Average | 54.2 | 47.9 | 52.8 | 62.4 (+14.5) |
| NER | UNER (13 languages) | 69.0 | 73.6 | 73.6 | 77.8 (+4.2) |
| NER | ACE05 (English, Arabic, Chinese) | 41.6 | 58.7 | 61.2 | 61.5 (+2.8) |
| Arg. Extraction | ACE05 (English, Arabic, Chinese) | 11.7 | 92.7 | 92.9 | 86.0 (-6.7) |
| Event Detection | ACE05 (English, Arabic, Chinese) | 21.3 | 42.6 | 40.0 | 44.0 (+1.4) |
| Rel. Extraction | ACE05 (English, Arabic, Chinese) | 4.6 | 37.3 | 39.4 | 39.1 (+1.8) |
| Unseen Label Schema |  |  |  |  |  |
| NER | MultiNERD (10 languages) | 71.9 | 62.2 | 63.9 | 63.0 (+0.8) |
| NER | MultiCoNER2 (12 languages) | 46.1 | 22.2 | 28.4 | 34.5 (+12.2) |
| Slot Detection | xSID (10 languages) | 47.0 | 6.0 | 27.1 | 26.4 (+20.4) |
| Slot Detection | MultiTO (English, Spanish, Thai) | 19.9 | 17.7 | 20.3 | 18.1 (+0.4) |
| Slot Detection | Massive (15 low-resource languages) | 33.3 | 5.8 | 12.1 | 19.0 (+13.1) |
| Rel. Extraction | REDFM (7 languages) | 19.1 | 15.5 | 16.8 | 16.2 (+0.7) |
| Average | Seen | 33.7 | 58.8 | 60.0 | 61.8 (+3.0) |
| Unseen | 39.5 | 21.6 | 28.1 | 29.5 (+8.0) |  |
| English-only | 55.2 | 58.6 | 60.3 | 59.3 (+0.7) |  |
| All | 36.6 | 40.2 | 44.1 | 45.7 (+5.5) |  |

Figure: Figure 2: TransFusion leads to larger NER F1 improvements for low resource languages in MasakhaNER2 (right) compared to high resource languages in UNER (left).
Refer to caption: x2.png

## 5 Results

We present zero-shot cross-lingual transfer results for IE tasks in Table 2, evaluating both seen and unseen label schemas across 36 languages. Our proposed GoLLIE-TF model consistently outperforms the original GoLLIE, achieving an average F1 score improvement of +4.6 across 11 datasets. Notably, GoLLIE-TF demonstrates significant performance gains in low-resource language NER while mainting English performance on average. For instance, on the MasakhaNER2 dataset, TransFusion boosts F1 from 47.9 to 62.4, surpassing both GPT-4 and the translate-train baseline. Furthermore, GoLLIE-TF supports generalization to unseen label schemas. In particular, TransFusion significantly improves performance on MultiCoNER2 (+12.2), xSID (+20.4), and on low-resource language dataset Massive (+13.1) over GoLLIE, showcasing its adaptability to unseen tasks.
While GPT-4 still demonstrates superior performance on unseen label schemas, we would like to highlight that our experiments are conducted in a controlled setting. In contrast, for proprietary models, we are unaware of the dataset used, leading to potential dataset contamination.

#### TransFusion performance on High vs. Low-resource languages.

Figure 2 reveals a noteworthy trend: GoLLIE-TF exhibits substantial performance enhancements particularly in low-resource language settings. This underscores the significance of leveraging external Machine Translation systems to enrich input data for such languages. We followed the categorization of high and low-resource languages from Costa-jussà et al. (), which categorizes a language as low-resource if there are fewer than 1M publicly available deduplicated bitext samples.
While the performance disparity between GoLLIE-TF and other models remains modest in high-resource language scenarios, a notable performance gap emerges in the low-resource language domain. Furthermore, results on the unseen-label low-resource language dataset, Massive, also show that GoLLIE-TF signficiantly outperforms Translate-Train, as shown in in Table 2.

**Table 3: Ablation study.**
| Model | MasakhaNER2 F1 |
| --- | --- |
| GoLLIE-TF | 62.4 |
| - w/o annotate | 55.7 (-10.5) |

### 5.1 Ablation Study

#### Analyzing Performance Improvements

Table 3 shows a critical insight into the performance gains observed in the TransFusion framework, particularly in the annotate step on the English translation, which plays a crucial role in enhancing the performance of MasakhaNER2.
We conduct an ablation study wherein we trained a variant of GoLLIE-TF, termed GoLLIE-TF (w/o annotate), directly generating predictions on target language text from the unlabelled English text. We observe a notable performance drop from 62.4 to 55.7 F1 score. This observation underscores the significance of TransFusion’s ability to leverage English annotations during test time, resulting in more precise predictions.

#### Robustness to translation quality.

TransFusion offers a distinct advantage by leveraging an external multilingual MT system to augment its dataset with English translations.
However, the efficacy of this approach hinges on the translation quality provided by the external MT system.

Figure: Figure 3: TransFusion robustness to different translation systems.
Refer to caption: x3.png

In Figure 3, we explore this aspect by evaluating GoLLIE-TF’s performance with three different MT systems (NLLB-200-600m, 1.3b, 3.3b) and use Flores-200 translation benchmark (X to English) (Costa-jussà et al., ) to measure translation quality (spBLUE) of languages covered by MasakhaNER2. Our experiments reveal that GoLLIE-TF exhibits robustness across various MT systems, as we observe that the F1 score on MasakhaNER2 does not exhibit a significant drop, however performance does improve with a stronger translation system.

### 5.2 Enhancing GPT-4 with TransFusion

Despite GPT-4’s pre-training on multilingual corpora, a notable performance gap persists between its English NER capabilities on CoNLL03 (80 F1) and its performance on low-resource languages (54.2 F1).
In Figure 4, we employ the TransFusion instruction, asking GPT-4 for predictions on the English translation and to then use these labels to predict on the target lagnauge sentence. We show TransFusion prompting yields a substantial enhancement in GPT-4’s NER performance across MasakhaNER2 and three additional low-resource languages from the UNER dataset (Cebuano, Tagalog (Philippines), and Uganda), improving the average F1 score from 53.4 to 62. This shows the GPT-4 can follow TransFusion prompting framework to leverage its English predictions to make more accurate predictions on low-resource languages.

Figure: Figure 4: GPT-4 + TransFusion framework improves NER on low-resource language from MasakhaNER2 and UNER subsets. On average, GPT-4 + TransFusion improves average F1 from 53.4 to 62.
Refer to caption: x4.png

### 5.3 TransFusion with Encoder-only Models

**Table 4: F1 of encoder-only multilingual LM on MasakhaNER2, average of 3 random seeds.**
| Model | F1 |
| --- | --- |
| EasyProject (Chen et al., ) | 64.9 |
| Codec (Le et al., ) | 70.1 |
| AfroXLM-R (550M) | 58.8 |
| + Trans-train | 65.8 |
| + TransFusion (ours) | 72.1 |

We have demonstrated that TransFusion can be applied to GPT-4 to improve low-resource language NER performance and also with the decoder-only LLM GoLLIE, which has the benefit of generalizing to unseen label schemas. In this section, we experiment with encoder-only multilingual LMs (Devlin, ) as the encoder architecture is one of the standard approaches for NER tasks used in practice.

As encoder-only models generally assume the same label schema between fine-tuning and evaluation, we focus on the seen label schema experiment setting, where we use CoNLL03 English as training data and test on the full test set of MasakhaNER2. We use AfroXLM-R (Alabi et al., ), an African language pre-trained language model as MasakhaNER is an African language dataset.
For each language, we fine-tuned the model on a combination (50/50%) of English and translation (Trans-train) or TransFusion data for 5 epochs with a learning rate of 2e-5. The specific TransFusion implementation is introduced in Section 3.1.

In Table 4, we show the effectiveness of the TransFusion framework which boosts the F1 from 58.8 to 72.1 F1 on MasakhaNER2 with AfroXLM-R. In addition, it outperforms the Trans-train baseline significantly with a +6.3 F1 improvement and achieves state-of-the-art performance on MasakhaNER2, surpassing the previous state-of-the-art Codec (Le et al., ).
Codec uses constrained decoding within a translation model to generate precise label projections from English to the target language for Translate-test. In contrast, TransFusion introduces a model that learns to fuse annotations, showing robustness to errors in English annotation predictions.
Overall, this shows the generalization of the TransFusion to the encoder-only multilingual language model.

### 5.4 Error Analysis

To understand the reasons why GoLLIE-TF makes mistakes, we conducted a manual error analysis on the MasakhaNER2 (Akan) subset and annotated 31 errors from the model. In Figure 5, we show examples of two common error types made by GoLLIE-TF: (1) English prediction errors, where the predictions on English translation are incorrect, and (2) Fusion errors, where the error arises from the fusion stage.
We identified 22 out of 31 cases where the model made errors in predicting NER for the English translation text, and thus these errors propagated to the final predictions. On the other hand, we found 12 out of 31 cases where the model made incorrect fusion processes, leading to hallucinations in the final predictions or predictions in the English text.

Figure: Figure 5: Error analysis of GoLLIE-TF’s 31 incorrect predictions on MasakhaNER2 (Akan). Two common errors are categorized as English prediction error (22/31) and fusion error (12/31).
Refer to caption: x5.png

## 6 Related Work

#### Multilingual language models.

Multilingual language models (Devlin, ; Conneau and Lample, ; Conneau et al., ; Xue et al., ; Scao et al., ; Asai et al., ), have facilitated cross-lingual transfer by leveraging pre-training on large-scale multilingual corpora. Recent models such as Gemini (Team et al., ) show emergent capabilities such as ultra low-resource language translation with a book and wordlist in context.
However, their performance tends to be subpar on languages that were not seen during pre-training or are underrepresented in the training data (Adelani et al., ; Ebrahimi et al., ). To address this limitation, several approaches have been explored, including bilingual models (Lan et al., ; Wang et al., ), language-specific extensions (Ogueji et al., ; Alabi et al., ; Yoon et al., ), continued training (Wang et al., ; Pfeiffer et al., ; Wang et al., ; Imani et al., ). Recently, multilingual instruction-tuning datasets such as Aya (Singh et al., ; Üstün et al., ) focusing on text generation and IEPile (Gui et al., ) (English and Chinese) have been proposed to facilitate this direction of research.

#### Translation for cross-lingual transfer.

To enhance LLM on multilingual NLP tasks, translating train or test data into English has proven as an effective approach (Hu et al., ; Xue et al., ). Recent studies on prompting LLMs with translation demonstrate improvements on multilingual math reasoning (Shi et al., ), text generation (Huang et al., ; Intrator et al., ; Liu et al., ) and sentence classification (Etxaniz et al., ). In contrast, our work focuses on challenging IE tasks that require extracting span annotations on the target language directly, instead of generating text. It is even more challenging to construct translated data for translate-train as span annotations are missing after translation.
To solve this, word alignment models (Och and Ney, ; Dyer et al., ; Lan et al., ; Dou and Neubig, ; Parekh et al., ; Le et al., ) and a simple mark-then-translate approach  (Lee et al., ; Lewis et al., ; Hu et al., ; Bornea et al., ; Chen et al., ) have been utilized to project labels across different languages. In contrast, we train a model to fuse annotations from English and directly make predictions on target language.

## 7 Conclusion

We introduce TransFusion, a framework that bridges the performance gap between high and low-resource languages in information extraction by leveraging machine translation. Our experiments demonstrate that TransFusion significantly improves the zero-shot cross-lingual transfer capabilities of instruction-tuned LLMs, surpassing both proprietary models and encoder-only architectures on low-resource languages NER. This work demonstrates the potential of translation-based techniques to unlock the power of LLMs for a wider range of low-resource languages, paving the way for more inclusive and equitable IE capabilities across diverse linguistic communities.

## Limitation and broader impact

The NER experiments conducted on GPT-4 have yielded promising results for low-resource languages.
However, concerns remain regarding potential data contamination resulting from the possibility that GPT-4 was pre-trained or fine-tuned on the test data.(^2^22https://hitz-zentroa.github.io/lm-contamination/blog/)
The Translation-and-fusion framework, while effective in enhancing cross-lingual transfer, does introduce additional inference costs during test time inference. These additional steps include translation using an external MT system and annotation processes, which can contribute to an increased number of token generation. This is similar to chain-of-thought prompting or retrieval augmented generation, which uses additional computational cost at inference for better quality generation.
Thus, practitioners should consider the trade-off between performance and efficiency when deciding to adopt the Translation-and-fusion approach.

The proposed method carries minimal risk, given that it addresses a traditional IE task. Its primary objective is to enhance IE cross-lingual transfer performance for low-resource languages lacking annotated training data. Consequently, our work aims to have a broader impact by facilitating research for global communities with diverse languages.

## Acknowledgements

This material is based upon work supported by the NSF (IIS-2052498) and IARPA via the BETTER and HIATUS programs (2019-19051600004, 2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## Appendix A Appendix

**Table 5: Evaluation datasets used and the language code for each dataset.**
| Dataset | Language Code |
| --- | --- |
| MasakhaNER2.0 [Adelani et al., ] | Bambara (bam), Ghomala (bbj), Ewe (ewe), Fon (fon), Hausa (hau), |
| afl-3.0 License | Igbo (ibo), Kinyarwanda (kin), Luganda (lug), Luo (luo), Mossi (mos), |
| [masakhane/masakhaner2](masakhane/masakhaner2) | Nyanja (nya), Naija (pcm), Shona (sna), Swahili (swh), Tswana (tsn) |
|  | Twi (twi), Wolof (wol), Xhosa (xho), Yoruba (yor), Zulu (zul) |
| UNER [Mayhew et al., ] | Cebuano (ceb_gja), Danish (da_ddt), German (de_pud), |
| [universalner.org/](universalner.org/) | English (en_ewt), English (en_pud), Croatian (hr_set), |
| (Unknown License) | Portuguese (pt_bosque), Portuguese (pt_pud),<br>Russian (ru_pud), |
|  | Slovak (sk_snk), Serbian (sr_set), |
|  | Swedish (sv_pud),<br>Swedish (sv_talbanken), |
|  | Tagalog (tl_trg), Tagalog (tl_ugnayan), Chinese (zh_gsd), |
|  | Chinese (zh_gsdsimp),<br>Chinese (zh_pud) |
| ACE05 [Walker et al., ] | English (en), Arabic (ar), Chinese (zh) |
| LDC license: LDC2006T06 |  |
| MultiNERD [Tedeschi and Navigli, ] | German (de), Spanish (es), French (fr), Italian (it), Dutch (nl), |
| CC BY-NC-SA 4.0 | Polish (pl), Portuguese (pt), Russian (ru), Chinese (zh) |
| [Babelscape/multinerd](Babelscape/multinerd) |  |
| MultiCoNER2 [Fetahu et al., ] | Bengali (bn), German (de), Spanish (es), Persian (fa), French (fr), |
| CC BY 4.0 | Hindi (hi), Italian (it), Portuguese (pt), Swedish (sv), |
| [MultiCoNER/multiconer_v2](MultiCoNER/multiconer_v2) | Ukrainian (uk), Chinese (zh), English (en) |
| xSID [van der Goot et al., ] | Arabic (ar), Danish (da), German (de),<br>English (en), Indonesian (id), |
| CC BY-SA 4.0 | Italian (it), Japanese (ja), Kazakh (kk), Dutch (nl), Serbian (sr), |
|  | Turkish (tr), Chinese (zh) |
| MultiTO [Schuster et al., ] | English (en), Spanish (es), Thai (th) |
| CC-BY-SA |  |
| RED-FM [Cabot et al., ] | Arabic (ar), German (de), English (en), Spanish (es), French (fr), |
| CC BY-SA 4.0 | Italian (it), Chinese (zh) |
| [Babelscape/REDFM](Babelscape/REDFM) |  |
| MASSIVE [FitzGerald et al., ] | Afrikaans (af-za),<br>Amharic (am-et),<br>Azeri (az-za),<br>Bengali (bn-bd), |
| CC BY 4.0 | Armenian (hy-am),<br>Georgian (ka-ge),Khmer (km-kh),<br>Mongolian (mn-mn), |
| [AmazonScience/massive](AmazonScience/massive) | Burmese (my-mm),<br>Kannada (kn-in),<br>Malayalam (ml-in), |
|  | Tamil (ta-in),<br>Telugu (te-in),<br>Tagalog (tl-ph),<br>Welsh (cy-gb) |

Figure: Figure 6: Example of input and output representation. (left) An example of a named entity recognition prompt and output annotations. (right) The same example but with translation text appended in the input prompt with instructions to guide the model to generate annotations on English translation text first, followed by annotations on the target language.
Refer to caption: x6.png

Figure: Figure 7: Examples of GoLLIE-TF model generation out (colored in gray).
Refer to caption: x7.png

Figure: Figure 8: TransFusion training dataset mixture for a total of 20,000.
Refer to caption: x8.png

**Table 6: We report GoLLIE-TF on MasakhaNER2 and Massive for 3 different seeds.**
| Dataset | Seed 0 | Seed 1 | Seed 2 | Mean | Std dev |
| --- | --- | --- | --- | --- | --- |
| masakhaner.bam.ner | 54.8 | 53.7 | 56.1 | 54.9 | 1.2 |
| masakhaner.bbj.ner | 50.2 | 46.2 | 50.9 | 49.1 | 2.6 |
| masakhaner.ewe.ner | 73.2 | 72.7 | 73.1 | 73.0 | 0.3 |
| masakhaner.fon.ner | 57.9 | 54.3 | 55.7 | 56.0 | 1.8 |
| masakhaner.hau.ner | 67.1 | 65.6 | 66.2 | 66.3 | 0.8 |
| masakhaner.ibo.ner | 56.6 | 54.2 | 55.7 | 55.5 | 1.3 |
| masakhaner.kin.ner | 58.5 | 59.5 | 59.6 | 59.2 | 0.6 |
| masakhaner.lug.ner | 75.5 | 74.5 | 75.1 | 75.0 | 0.5 |
| masakhaner.luo.ner | 51.7 | 51.6 | 51.5 | 51.6 | 0.1 |
| masakhaner.mos.ner | 48.8 | 43.8 | 44.4 | 45.7 | 2.7 |
| masakhaner.nya.ner | 78.2 | 78.7 | 78.9 | 78.6 | 0.3 |
| masakhaner.pcm.ner | 81.1 | 80.8 | 80.6 | 80.8 | 0.2 |
| masakhaner.sna.ner | 57.4 | 59.2 | 56.7 | 57.7 | 1.3 |
| masakhaner.swh.ner | 73.5 | 72.6 | 72.9 | 73.0 | 0.5 |
| masakhaner.tsn.ner | 71.0 | 70.3 | 71.1 | 70.8 | 0.5 |
| masakhaner.twi.ner | 74.2 | 68.6 | 76.6 | 73.1 | 4.1 |
| masakhaner.wol.ner | 61.9 | 55.6 | 60.2 | 59.2 | 3.2 |
| masakhaner.xho.ner | 49.9 | 54.4 | 51.3 | 51.9 | 2.3 |
| masakhaner.yor.ner | 54.4 | 52.4 | 53.4 | 53.4 | 1.0 |
| masakhaner.zul.ner | 52.8 | 53.3 | 51.4 | 52.5 | 1.0 |
| Average | 62.4 | 61.1 | 62.1 | 61.9 | 0.7 |
| massive.en-us.ner | 53.6 | 51.6 | 51.6 | 52.3 | 1.1 |
| massive.af-za.ner | 24.2 | 21.2 | 24.2 | 23.2 | 1.7 |
| massive.am-et.ner | 6.5 | 5.4 | 7.2 | 6.4 | 0.9 |
| massive.az-az.ner | 1.2 | 1.3 | 1.3 | 1.2 | 0.1 |
| massive.bn-bd.ner | 18.1 | 18.8 | 19.4 | 18.8 | 0.6 |
| massive.hy-am.ner | 19.4 | 16.2 | 21.1 | 18.9 | 2.5 |
| massive.ka-ge.ner | 18.4 | 16.0 | 19.6 | 18.0 | 1.9 |
| massive.km-kh.ner | 20.4 | 21.1 | 23.2 | 21.5 | 1.5 |
| massive.mn-mn.ner | 5.8 | 5.4 | 5.2 | 5.5 | 0.3 |
| massive.my-mm.ner | 31.7 | 32.4 | 33.2 | 32.4 | 0.8 |
| massive.kn-in.ner | 17.2 | 14.2 | 20.7 | 17.3 | 3.2 |
| massive.ml-in.ner | 11.0 | 10.6 | 10.3 | 10.7 | 0.4 |
| massive.ta-in.ner | 17.0 | 11.6 | 17.3 | 15.3 | 3.2 |
| massive.te-in.ner | 18.8 | 17.6 | 23.5 | 20.0 | 3.1 |
| massive.tl-ph.ner | 32.0 | 32.0 | 34.7 | 32.9 | 1.5 |
| massive.cy-gb.ner | 8.3 | 5.8 | 7.0 | 7.0 | 1.2 |
| Average | 19.0 | 17.6 | 20.0 | 18.8 | 1.2 |

**Table 7: Full experimental results (1) for each dataset and language. Format: [task name].[language code].[task].**
|  | GPT-4 | GoLLIE | Trans-train | GoLLIE-TF (ours) |
| --- | --- | --- | --- | --- |
| uner.ceb_gja.ner | 44.4 | 49.6 | 52.9 | 87.5 |
| uner.da_ddt.ner | 77.2 | 76.7 | 79.4 | 84.8 |
| uner.de_pud.ner | 80.3 | 80.1 | 82.3 | 83.8 |
| uner.en_ewt.ner | 59.9 | 84.7 | 67.6 | 66.4 |
| uner.en_pud.ner | 75.4 | 82.4 | 85.5 | 84.9 |
| uner.hr_set.ner | 82.1 | 83.0 | 87.7 | 89.6 |
| uner.pt_bosque.ner | 82.7 | 84.5 | 84.2 | 81.3 |
| uner.pt_pud.ner | 80.5 | 87.2 | 89.6 | 90.3 |
| uner.ru_pud.ner | 69.8 | 68.3 | 71.6 | 73.3 |
| uner.sk_snk.ner | 70.9 | 71.2 | 81.4 | 85.5 |
| uner.sr_set.ner | 85.9 | 86.2 | 88.5 | 88.9 |
| uner.sv_pud.ner | 73.7 | 81.5 | 79.6 | 85.7 |
| uner.sv_talbanken.ner | 68.7 | 69.4 | 64.6 | 75.7 |
| uner.tl_trg.ner | 55.7 | 58.8 | 60.3 | 54.2 |
| uner.tl_ugnayan.ner | 44.8 | 61.0 | 57.1 | 74.2 |
| uner.zh_gsd.ner | 60.6 | 62.5 | 58.8 | 67.6 |
| uner.zh_gsdsimp.ner | 57.9 | 62.4 | 61.4 | 68.8 |
| uner.zh_pud.ner | 72.0 | 74.8 | 72.6 | 77.7 |
| average | 69.0 | 73.6 | 73.6 | 78.9 |
| ace.en.eae | 24.5 | 97.3 | 97.9 | 98.3 |
| multiace.ar.eae | 1.6 | 84.3 | 83.8 | 81.8 |
| multiace.zh.eae | 9.6 | 96.6 | 97.1 | 77.9 |
| average | 11.7 | 92.7 | 92.9 | 86.0 |
| ace.en.ee | 27.8 | 67.5 | 64.0 | 60.4 |
| multiace.ar.ee | 24.4 | 16.1 | 12.8 | 25.0 |
| multiace.zh.ee | 11.6 | 44.2 | 43.3 | 46.7 |
| average | 21.3 | 42.6 | 40.0 | 44.0 |
| ace.en.ner | 58.0 | 78.3 | 87.3 | 86.5 |
| multiace.ar.ner | 32.3 | 29.5 | 30.3 | 37.5 |
| multiace.zh.ner | 34.6 | 68.2 | 66.0 | 60.6 |
| average | 41.6 | 58.7 | 61.2 | 61.5 |
| ace.en.re | 5.40 | 58.2 | 59.8 | 58.1 |
| multiace.ar.re | 3.2 | 14.1 | 13.5 | 15.8 |
| multiace.zh.re | 5.1 | 39.5 | 44.8 | 43.3 |
| average | 4.6 | 37.3 | 39.4 | 39.1 |
| multinerd.de.ner | 75.8 | 69.3 | 73.2 | 74.4 |
| multinerd.es.ner | 69.4 | 72.0 | 68.1 | 69.5 |
| multinerd.fr.ner | 71.8 | 71.9 | 74.4 | 72.5 |
| multinerd.it.ner | 76.2 | 69.8 | 74.2 | 70.5 |
| multinerd.nl.ner | 76.9 | 67.8 | 73.0 | 72.5 |
| multinerd.pl.ner | 72.1 | 62.0 | 64.0 | 61.5 |
| multinerd.pt.ner | 67.7 | 67.7 | 66.3 | 64.9 |
| multinerd.ru.ner | 65.3 | 57.9 | 55.7 | 58.7 |
| multinerd.zh.ner | 7.8 | 7.1 | 13.9 | 8.8 |
| multinerd.ner | 71.5 | 76.2 | 75.6 | 76.2 |
| average | 71.9 | 62.2 | 63.9 | 63.0 |

**Table 8: Full experimental results (2) for each dataset and language. Format: [task name].[language code].[task].**
|  | GPT-4 | GoLLIE | Trans-train | GoLLIE-TF (ours) |
| --- | --- | --- | --- | --- |
| multiconer2.bn.ner | 43.9 | 2.7 | 7.9 | 27.6 |
| multiconer2.de.ner | 54.4 | 27.3 | 30.8 | 33.1 |
| multiconer2.es.ner | 44.8 | 18.1 | 23.9 | 26.1 |
| multiconer2.fa.ner | 40.1 | 15.6 | 34.9 | 41.4 |
| multiconer2.fr.ner | 54.2 | 29.2 | 32.1 | 34.2 |
| multiconer2.hi.ner | 46.9 | 5.0 | 14.8 | 33.5 |
| multiconer2.it.ner | 51.1 | 41.4 | 46.0 | 46.5 |
| multiconer2.pt.ner | 49.7 | 23.6 | 31.5 | 34.7 |
| multiconer2.sv.ner | 52.5 | 14.8 | 16.1 | 19.6 |
| multiconer2.uk.ner | 55.9 | 41.1 | 47.7 | 51.7 |
| multiconer2.zh.ner | 5.1 | 14.0 | 20.9 | 28.3 |
| multiconer2.en.ner | 54.6 | 34.1 | 34.7 | 36.7 |
| average | 46.1 | 22.2 | 28.4 | 34.5 |
| xsid.ar.ner | 53.2 | 0.0 | 29.7 | 28.7 |
| xsid.da.ner | 48.1 | 2.7 | 15.5 | 16.0 |
| xsid.de.ner | 48.9 | 9.8 | 36.0 | 35.5 |
| xsid.en.ner | 63.1 | 28.8 | 38.4 | 37.5 |
| xsid.id.ner | 49.4 | 0.7 | 25.6 | 23.2 |
| xsid.it.ner | 52.1 | 3.4 | 30.2 | 32.8 |
| xsid.ja.ner | 28.1 | 10.1 | 32.8 | 26.5 |
| xsid.kk.ner | 34.9 | 0.0 | 0.0 | 2.5 |
| xsid.nl.ner | 48.9 | 4.9 | 33.8 | 31.4 |
| xsid.sr.ner | 48.7 | 0.0 | 19.4 | 16.8 |
| xsid.tr.ner | 40.8 | 0.8 | 20.9 | 22.2 |
| xsid.zh.ner | 47.3 | 10.7 | 43.5 | 43.7 |
| average | 47.0 | 6.0 | 27.1 | 26.4 |
| multito.en.ner | 51.1 | 35.3 | 39.0 | 40.3 |
| multito.es.ner | 1.4 | 2.5 | 3.0 | 2.3 |
| multito.th.ner | 7.3 | 15.4 | 18.9 | 11.8 |
| average | 19.9 | 17.7 | 20.3 | 18.1 |
| redfm.ar.re | 18.3 | 11.6 | 9.0 | 13.9 |
| redfm.de.re | 31.0 | 22.3 | 24.8 | 13.1 |
| redfm.en.re | 19.9 | 14.8 | 18.6 | 15.7 |
| redfm.es.re | 17.4 | 13.8 | 18.6 | 14.4 |
| redfm.fr.re | 17.1 | 15.2 | 19.2 | 17.6 |
| redfm.it.re | 17.2 | 20.0 | 17.1 | 29.1 |
| redfm.zh.re | 12.9 | 10.4 | 10.5 | 9.7 |
| average | 19.1 | 15.5 | 16.8 | 16.2 |

**Table 9: Full experimental results (3) for each dataset and language. Format: [task name].[language code].[task].**
|  | GPT-4 | GoLLIE | Trans-train | GoLLIE-TF (ours) |
| --- | --- | --- | --- | --- |
| massive.en-us.ner | 55.2 | 45.9 | 54.7 | 53.6 |
| massive.af-za.ner | 52.6 | 8.2 | 23.4 | 24.2 |
| massive.am-et.ner | 17.0 | 0.0 | 0.8 | 6.5 |
| massive.az-az.ner | 25.7 | 4.0 | 11.0 | 1.2 |
| massive.bn-bd.ner | 33.1 | 5.7 | 13.0 | 18.1 |
| massive.hy-am.ner | 33.6 | 1.2 | 11.9 | 19.4 |
| massive.ka-ge.ner | 32.1 | 10.4 | 12.2 | 18.4 |
| massive.km-kh.ner | 33.9 | 0.0 | 11.3 | 20.4 |
| massive.mn-mn.ner | 19.5 | 0.0 | 5.3 | 5.8 |
| massive.my-mm.ner | 27.9 | 4.8 | 15.2 | 31.7 |
| massive.kn-in.ner | 33.1 | 0.0 | 2.6 | 17.2 |
| massive.ml-in.ner | 25.1 | 0.0 | 4.5 | 11.0 |
| massive.ta-in.ner | 30.7 | 1.2 | 5.0 | 17.0 |
| massive.te-in.ner | 28.7 | 0.0 | 0.0 | 18.8 |
| massive.tl-ph.ner | 50.3 | 12.3 | 20.2 | 32.0 |
| massive.cy-gb.ner | 33.6 | 0.0 | 3.1 | 8.3 |
| average | 33.3 | 5.9 | 12.1 | 19.0 |

**Table 10: Comparison of GPT-4 and GPT-4+Transfusion.**
| Language | GPT-4 | GPT-4+Transfusion |
| --- | --- | --- |
| MasakhaNER2 |  |  |
| bam | 42.2 | 60.2 |
| bbj | 58.2 | 52.9 |
| ewe | 72.2 | 72.4 |
| fon | 39.4 | 53.6 |
| hau | 65.9 | 71.6 |
| ibo | 42.2 | 37.9 |
| kin | 47.5 | 56.4 |
| lug | 62.5 | 68.2 |
| luo | 47.2 | 58.7 |
| mos | 43.2 | 44.8 |
| nya | 71.1 | 76.4 |
| pcm | 78.9 | 75.7 |
| sna | 39.5 | 51.0 |
| swh | 79.2 | 73.2 |
| tsn | 56.3 | 71.2 |
| twi | 44.2 | 65.3 |
| wol | 52.6 | 59.1 |
| xho | 49.8 | 62.7 |
| yor | 54.7 | 52.1 |
| zul | 36.9 | 43.6 |
| MasakhaNER2 average | 54.2 | 59.9 |
| UNER |  |  |
| ceb_gja | 44.4 | 83.5 |
| tl_trg | 55.7 | 67.7 |
| tl_ugnayan | 44.8 | 61.2 |
| All average | 53.4 | 62.0 |

Figure: Figure 9: Annotation guidelines for MasakhaNER and UNER.

Figure: Figure 10: Annotation guidelines for MultiCoNER2.

Figure: Figure 11: Annotation guideline for MultiNERD.

Figure: Figure 12: Annotation guidelines for Massive.

Figure: Figure 13: Annotation guidelines for xSID.

Figure: Figure 14: Annotation guidelines for REDFM.

Figure: Figure 15: Annotation guidelines for MultiTO.

Figure: Figure 16: Annotation guidelines for ACE EAE.

Figure: Figure 17: Annotation guidelines for ACE EE.

Figure: Figure 18: Annotation guidelines for ACE RE.

Figure: Figure 19: Annotation guidelines for ACE NER.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: - The main novelty of the work seems to be this idea of generating annotations of the example translated to English before annotating the original sentence (as also discussed under "Weaknesses"). Can the authors comment on the increased cost of this approach and if there is a way to cut the cost of additional generation?

- It is unclear to me what happens if a language is unseen by NLLB. Will the proposed method still work or not (and to what extent)? There were some approaches (e.g., see https://aclanthology.org/2023.emnlp-main.242) that proposed easy adaptations of NLLB to unseen languages - it probably makes sense to try out such approaches as well. 

- Given that the evaluation datasets used in this work are quite standard, I would recommend providing also additional reference points (i.e., current state-of-the-art scores) from the literature. This would help the reader to grasp not only the gains over the baseline GOLLIE model (and GPT-4) but also situate the results in a wider context of work on those datasets.

- The authors empirically verify that varying the MT model does not have a profound impact on performance. Can the authors discuss why this observation might hold - is it the inherent limitation of the MT system (e.g., it cannot cover some low-resource languages well regardless of its size) or the inherent advantage of the MT system? Is it also related to simplicity/complexity of the task chosen for this experiment?

- How do language properties (beyond its 'resourceness') affect performance? Can we see some patterns based on language proximity/distance to English?
soundness: 3
strengths: - A simple idea which is well motivated and well explained the paper, with a good coverage of IE datasets and languages, and good side analyses (e.g., ablation studies, error analyses).
- This idea can be easily combined even with API-gated models without fine-tuning for some quick wins on different IE tasks. Given its simplicity, the entry point to try out the model is quite low (which is a plus).
- The experimental work can be easily reproduced by other researchers.
summary: This work proposes a simple yet effective approach for improving cross-lingual transfer with a focus on Information Extraction (IE) tasks such as NER, relation extraction, and slot filling. The idea is to leverage an external Machine Translation (MT) system such as NLLB (or any other similar MT system) to translate data from low-resource languages to English, and then leveraging the translation data as additional signal/context for making predictions. The second key ingredient of the approach is the idea of bypassing the alignment step for IE task, where the 'trick' is to first do annotations on the example translated to English before predicting annotations in the target language, where the former seems to be crucial to enable good performance, as verified by an ablation study. In general, this simple idea does provide benefits across a range of IE tasks and languages. Moreover, the authors show the usefulness of the approach on IE tasks with unseen annotation sets, and they show how a similar approach can be applied both to decoder-only models (which are in the main focus of the paper), but also to encoder-style models as well.
weaknesses: - I have concerns about the actual novelty of the work (as advertised in the paper). The use of external MT systems to create some 'silver data' as well as to help with cross-lingual transfer is definitely not a novel idea, and has been tried many times before. We do have many examples of work that explore translation-based cross-lingual transfer for low-resource languages, such as:
-- https://aclanthology.org/2023.emnlp-main.242
-- https://aclanthology.org/2024.naacl-long.298/
-- https://aclanthology.org/2023.emnlp-main.399
- In fact, some of the key baselines are missing from the paper (e.g., improved 'translate-test' baselines as in the aforementioned work)
- Also, it is not new to use external MT systems to enable multilingual instruction-tuning, here are just a small selection of papers:
-- https://aclanthology.org/2024.findings-eacl.90.pdf
-- https://arxiv.org/abs/2407.09879
-- https://aclanthology.org/2024.findings-acl.136.pdf
I would like to see a much broader discussion on how exactly the proposed approach is different and novel here.

- Only one model (GOLLIE-7B) is used as the main baseline model. It would be beneficial to try out the same approach with additional open-weights model (e.g., LLama-3) and also vary the size of the model to verify the impact of model size as well and whether some gains might diminish with larger models. Is it possible to also run some experiments with, say, 70B models from some standard families such as Llama? What about Gemma 2 9B?

- The focus of the work is on IE task simply because of the idea on how to improve the results via generating annotations for the example translated to English first - this seems to me as another hint that this is the only real novelty of the work, as in other 'non-IE' tasks the work would not bring much novelty. I would like to see an extended discussion here. Is the same method applicable to non-IE tasks that require sequence labels such as the 'niche' NLP tasks of POS tagging and dependency parsing? Would the method bring any performance gains to NLU tasks that are typically used to evaluate cross-lingual transfer in some previous research such as NLI on AmericasNLI and XNLI or QA on TydiQA?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Can you clarify the notations on pages 3 and 4? Specifically, on page 3, src refers to english and tgt/trans refers to the low resource language. On page 4, L173, trans refers to the english examples. Is this understanding correct? I believe this is because Page 3 is talking about generating the training data which translates **from** English, while Page 4 talks about the inference time workflows (as used in finetuning) where the input example is translated **to** English. 
2. L238-L241 are very confusing. From here, it seems somehow that en data has 19k examples and translated data has another 891. Figure 8 in Appendix also shows the same. However, so far in Section 3.1, L145-151 clearly state that all finetuning/training TF data has to be 4-way parallel. So, what do we mean by just 891 translated examples? Don't the 19k in English also need to have translations and spans mapped as labels?
3. The difference between Trans-Train and GoLLIE-TF is that while both use the same 4-way parallel data, Trans-Train only finetunes in the original GoLLIE style using the translated data pairs (x, y) while GoLLIE-TF uses the 4-way data to finetune the task of translate/annotate/fuse. Is this accurate? If so, can you make this explicit inline?
4. Since you were cost-limited per L229, how did that impact the standard test set sizes that you need to run tests on for a fair comparison with literature?
5. Why do you not compare the fusion step alongside standard rule based aligners?
soundness: 3
strengths: ## Quality:
1. Overall the proposal is a simple prompting and Chain-of-thought (CoT) trick. It is clearly defined, motivated and explained from data generation to training fully as seen in Sec 3.1. 
2. Clear ablation study
3. Good stronger baseline addition with TransTrain
4. Very extensive experiments, appendix visualizations and results. This also includes manual error analysis.
5. Also shows improvements when built with GPT-4. 
6. Good error analysis to the last section of the paper.  

## Significance:
This method establishes a clear path for performing information extraction (IE) with generative or encoder-only models, showing a clear tradeoff and presenting easily usable recipes for many low-resource African languages. This would be very impactful for various applications. 


## Coherence and Clarity
1. The formulations on page 3 help set expectations on the model before jumping into the LLM prompting. This is very useful to fully understand the proposal.  
2. Related work is clearly organized by themes and presented with a comparison to the proposed work.
summary: ### Summary:
This paper proposes a solution called TransFusion (TF) for Information Extraction (IE) tasks in low-resource languages such as African languages with access to only high-resource (en) IE annotated data. It has two key parts 1/ a simple update to the incumbent model GoLLIE's prompt: At inference time, the low resource text input is translated to English and then the model is instructed to first annotate this english text and then "fuse" the final annotations on the low-resource language input. 2/ Fine-tuning GoLLIE for the above inference. For this, high resource (en) data is used and annotations are projected on the translated text. This gives 4-way tuples of the text as well as annotations in both languages. This is used to finetune GoLLIE to do translate, annotate, fuse.

The paper also discusses training data preparation, the modeling formulation and how an LLM's autoregressive decoding is used to execute on this modeling objective. Finally, results are comprehensively shown on a wide variety of languages and datasets where GoLLIE with TF has distinctly better results. Additionally, the paper also shows a stronger baseline than GoLLIE which first translates the en data to target language(s) and uses that to finetune GoLLIE. 

### Overall Recommendation:
My overall recommendation is a soft accept currently. This is mainly because one of the key parts of the finetuning data is currently unclear (as asked in the questions). If that can be simply addressed and/or explained clearly, this paper does have the potential as it solves a wide swath of IE tasks cross lingually.
weaknesses: 1. The novelty of the cross lingual transfer and alignment usage is stated with just 2020+ references. This is overstating the novelty. For example: Structured Prediction as Translation between Augmented Natural Languages (Paolini et al): would be very apt to relate to as it also does IE with generative models. 
2. The clarity around the finetuning data formation and some of the formalisms as pointed out in Q1 below is lacking. See questions below for details for where the paper can be improved

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * Figure 1 could have shown more clearly the fact that the initial portion of the text is translated into English, ran on an english-based model and then with its output and the portion in the original language, the output is predicted. None of these aspects (nor the fact the TransFusion module is trained separately) is shown in this image. 

* Please fix the reference "Team et all, 2023", which took the "Gemini Team" as the name of the first author, and "Team" as its last name

* Although it's not an issue, a better presentation would have completely filled the 10 pages in the submission.
 
* line: 419  "lagnauge"

* line 423 fix double parenthesis. Authors might be referring about two varieties of Tagalog, from Philippines and from Uganda

* Authors might like to stress the fact that the different configurations of TransFusion are, indeed, different usages of the same approach.
soundness: 3
strengths: The approach is effective and can be applied in a variety of configurations, i.e in decoder only models, on a large pretrained model like GPT-4 to be used zero shot on even on an encoder only model.
The work makes a good effort in covering several low resource languages from available datasets.
The approach improves over GPT-4 and for translate-traine baselines in most of the languages.
summary: This work introduces a new framework to deal with information extraction in the context of zero shot cross-lingual approaches on low resource regimes. The framework, called TransFusion, consists in two parts i) rely on a high resource language model that that is fed with the translation of the source part in the original language and ii) to fuse the response in the high resource language and the portion on the low resource one training a model to predict an answer based on both inputs.
The framework is applied to the existing English-based model, GoLLIE, to produce GoLLIE-TF, a model to deal with cross-lingual information extraction for low resource languages.
The resulting model is run on a variety of datasets, covering tasks such as NER, Slot detection and relation extraction on up to 50 different languages.
weaknesses: Although the approach is relatively simple and straightforward it is not clearly explained on the paper and requires a thorough reading to understand it (see below).
One of the salient points of this paper could also be considered a drawback. TransFusion has been shown to be flexible enough to be used in three different scenarios, each one of them having some minor differences in implementation. One could argue these three "flavours"of TransFusion are, in fact, different models (see below)

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Did you consider skipping the translation step during inference for the fine-tuned Gollie-TF? It would be interesting whether training on translation + fusion already results in performance improvements without the need to translate and annotate the test data.
- In lines 48 you mention that your TransFusion method trains language models on tools usage (MT model), however your method does not integrate an interactive use of the MT model. Can you elaborate this description?
soundness: 3
strengths: - The evaluation results suggest performance gains for low-resource languages in comparison to GPT-4, regular Gollie and translate-train baselines.
- The performance benchmarks includes various information extraction tasks across 12 datasets and low-resource languages. Specifically the performance benchmark on low-resource languages shows promising performance benefits for prompting decoder-only and fine-tuning encoder-only models.
- The authors provide ablation studies evaluating the contribution of the annotation in English and the impact of different sizes of machine translation models.
summary: - The study proposes a translation and fusion (TransFusion) method as an extension to Gollie for zero-shot cross-lingual information extraction.
- The method involves appending translated target language instances to the input prompt. The model is prompted to perform information extraction in English first, which serves as a reference label to improve information extraction performance in the target language.
- For encoder-only language models this entails a two-step approach, where a model fine-tuned in English is used to provide the reference labels for the translated text first. The translated text with the reference labels is then concatenated with the inputs in the target languages and used for the final prediction outcome of the target language data. For decoder-only models, the two steps are combined in one prompt by providing instructions to first generate reference labels in English and then perform the task in the target language.
weaknesses: - The benchmarks lack the translate-test baseline. The annotations of the translated English text are not formally included in the evaluation. This evaluation would provide insights in the contribution of the “fusion” step of the framework and should be included for both GPT-4 and Gollie.
- The proposed method lacks innovation. Providing annotated translations in the input prompt to enhance multilingual performance has been explored in related work [1,2]. Furthermore, the engineering approach to cross-lingual information extraction lacks detailed evaluation of each individual component.
- The discussion of the additional computational complexity introduced through machine translation and annotation process is not included in the main body of the paper (included in Appendix Table 7).

References

- [1] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, et al.. 2022. Few-shot Learning with Multilingual Generative Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- [2] Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, and Mirella Lapata. 2023. QAmeleon: Multilingual QA with Only 5 Examples. Transactions of the Association for Computational Linguistics, 11:1754–1771.

Meta-review: This paper proposes TransFusion, a framework for improving multilingual performance in LLMs by fine-tuning models on English translations of low-resource language data and using "annotation fusion" (i.e., showing a fusion model data in the source language and English and annotating both texts). The experiments use the TransFusion approach to finetune an LLM (GOLLIE-7B) with this setup and evaluate the approach on multiple IE tasks. The results show that this approach improves performance across several IE tasks and languages. The paper also provides many experiments demonstrating the approach's effectiveness beyond the primary task evaluations (ablation, error analysis, and task transfer to unseen label sets).

Strengths:
- The reviewers found the paper to be generally clear and easy to understand (TcxU, qvp3), and the approach is well-motivated (TcxU).
- The paper provides extensive experiments examining how the TransFusion framework works in different settings (TcxU, qvp3, T2WB), and the results and trends are clear and consistent (qvp3, T2WB). The experiments also provide good coverage for many low-resource languages across different datasets (ow8F, T2WB).
- The proposed method is broadly applicable (TcxU, ow8F).

Weaknesses:
The primary weakness of this work is a lack of novelty compared to the proposed methods and findings in prior works. Specifically, the components combined into this framework have already been proposed for improving multilingual prompting in prior work (as raised by TcxU, ow8F, and T2WB). The paper does not clearly differentiate why this approach is significantly different from these prior works and misses some related work that previously used these methods (TcxU). Notably, a very related baseline (translate-test) is not included for the full set of results (TcxU, T2WB) - only for a subsequent results table (Table 5) on a subset of tasks.

Some other minor weaknesses were raised by the reviewers, such as that it is unclear how effective the method will be beyond IE tasks (TcxU) and that some points within the paper were not described clearly (qvp3, ow8F). The authors fully addressed other weaknesses and points of confusion during the discussion period.

Decision: Reject