Improving Word Alignment Using Semi-Supervised Learning
Authors: Zhongtao Miao, Qiyu Wu, Masaaki Nagata, Yoshimasa Tsuruoka
Accepted at: acl25

Improving Word Alignment Using Semi-Supervised Learning

Zhongtao Miao1, Qiyu Wu1*, Masaaki Nagata2, Yoshimasa Tsuruoka1
1The University of Tokyo, Tokyo, Japan
2NTT Communication Science Laboratories, NTT Corporation, Kyoto, Japan
1{miao, qiyuw, tsuruoka}@logos.t.u-tokyo.ac.jp
2masaaki.nagata@ntt.com

Abstract
Word alignment plays a crucial role in various natural language processing tasks, such as serving as cross-lingual signals for sentence embedding, reducing hallucination and omission in machine translation, and facilitating the construction of training data for simultaneous speech translation. Current state-of-the-art approaches usually rely on: (1) supervised data and large-scale weakly supervised data constructed from Wikipedia and (2) multilingual Transformer encoder-based models. However, we find that the current state-of-the-art encoder-based method, BinaryAlign, suffers from the issue of insufficient labeled data, and we further improve it with self-training with a small amount of parallel data. In addition, considering the impressive performance of multilingual large language models on many natural language processing tasks, we also explore the possibility of using these decoder-based large language models as word aligners. We observe that although fine-tuning large language models with labeled data produces acceptable results, augmenting the training with pseudo-labeled data further enhances model performance. Based on the findings, we propose a semi-supervised framework to improve the large language model-based word aligners. Experimental results demonstrate that the proposed method with a small amount of parallel data outperforms the current state-of-the-art method on various word alignment datasets.

1 Introduction
Word alignment aims to identify correspondences between source and target words in a translation sentence pair, as shown in Figure 1. Although word alignment was initially proposed to enhance statistical machine translation (Brown et al., 1993), advancements in both word alignment and deep learning techniques have broadened its application to a wide range of tasks in natural language processing (NLP). For example, Chi et al. (2021) use word alignment in cross-lingual language model pretraining. Word alignment is utilized by Miao et al. (2024a) to improve cross-lingual sentence embedding. Wu et al. (2024) leverage word alignment preference data to mitigate hallucination and omission in machine translation models. Word alignment can also be utilized in the simultaneous speech translation domain for data curation (Ouyang et al., 2025b; Fu et al., 2025). Other possible applications include alleviating the over-translation and under-translation problems in machine translation (Tu et al., 2016), XML-structured parallel text segment extraction (Hashimoto et al., 2019) and constrained neural machine translation (Song et al., 2019; Chen et al., 2021).

Representative studies for improving word alignment with supervised methods include SpanAlign (Nagata et al., 2020), WSPAAlign (Wu et al., 2023) and BinaryAlign (Latouche et al., 2024). SpanAlign (Nagata et al., 2020) reformulates the word alignment task as a SQuAD-style span prediction question answering task. WSPAAlign (Wu et al., 2023) relaxes the requirement of

![A Japanese-English word alignment example. Note that word alignment not only takes care of the corresponding word(s) within the source and target sentences but also considers the positional information. The English words [hino] in green and [hino] in blue are different for the Japanese word [日野], which appears in the second position (zero-indexed) of the Japanese sentence. The English word [wife] in red does not have the corresponding Japanese word (null word alignment).](page_1012_370_388_246.png)

Figure 1: A Japanese-English word alignment example. Note that word alignment not only takes care of the corresponding word(s) within the source and target sentences but also considers the positional information. The English words [hino] in green and [hino] in blue are different for the Japanese word [日野], which appears in the second position (zero-indexed) of the Japanese sentence. The English word [wife] in red does not have the corresponding Japanese word (null word alignment).
Figure 2: Trend of F1 scores as the amount of unlabeled parallel sentences in the training data increases, using our semi-supervised framework on the test set of the Japanese-English word alignment dataset, KFTT (Neubig, 2011). Note that one unit (653) is the size of the labeled data. The yellow dotted line denotes the F1 score of the current state-of-the-art method, BinaryAlign (Latouche et al., 2024).

manually labeled data in SpanAlign with a large-scale weak supervision pretraining dataset (2 million noisy sentence pairs) based on entity links from Wikipedia and contextual word embeddings from multilingual encoder-based language models. BinaryAlign (Latouche et al., 2024) further reformulates the word alignment task as a binary sequence labeling task, achieving the new state-of-the-art performance.

These supervised methods for word alignment are highly dependent on (1) manually labeled datasets (Nagata et al., 2020; Latouche et al., 2024) or large-scale weak word alignment signals extracted from Wikipedia entity links and contextual word embeddings of encoder-based models to relax the strict requirement of labeled data (Wu et al., 2023) and (2) multilingual encoder models (Nagata et al., 2020; Wu et al., 2023; Latouche et al., 2024). This raises the following questions: (1) can the potential of multilingual encoder-based models be fully leveraged by only using the labeled datasets for the current state-of-the-art method, BinaryAlign? If not, is there a way of using a small amount of data to improve it instead of utilizing a large-scale weakly supervised pre-training dataset like WSPAAlign? (2) given the success of LLMs across a wide range of NLP tasks, how effective are multilingual LLMs for the word alignment task?

We find that the issue of insufficient labeled data in word alignment prevents both multilingual encoder-based and decoder-based large language models from fully utilizing their advantages, leading to suboptimal performance as shown in Figure 2. Figure 2 presents the trend of F1 scores on the Japanese word alignment dataset KFTT, as we progressively increase the number of parallel sentences used for training our multilingual encoder (SemiAlign-E) and decoder (SemiAlign-D) models.

Based on the aforementioned findings, we propose a semi-supervised framework named SemiAlign, which is short for leveraging Semi-supervised learning to improve word Alignment, designed to alleviate the challenge of limited labeled data in word alignment by utilizing a small amount of unlabeled parallel text. We demonstrate the effectiveness of our method for both multilingual encoder-based and decoder-based language models. The contributions of this paper can be summarized as follows:

• We find that the current state-of-the-art method, BinaryAlign (Latouche et al., 2024), which is based on a multilingual encoder model, suffers from the issue of limited labeled data, leading to suboptimal performance.

• We also observe the issue of limited labeled data when fine-tuning multilingual decoder-based language models as word aligners. We investigate the important factors that affect the word alignment performance of multilingual decoder language models.

• We propose a semi-supervised framework to mitigate the issue of limited labeled data in both scenarios, and the proposed method achieves new state-of-the-art performance on all language pairs that we evaluate with a small amount of unlabeled data.

2 Methodology

Background. Given a source sentence \( \mathbf{x} = [x_1, \ldots, x_n] \) and its target sentence \( \mathbf{y} = [y_1, \ldots, y_m] \), where \( x_i \) denotes the i-th word in the source sentence (\( 1 \leq i \leq n \)) and \( y_j \) represents the j-th word in the target sentence (\( 1 \leq j \leq m \)), word alignment aims to find a set of source and target word pairs in sentences \( \mathbf{x} \) and \( \mathbf{y} \):

\[
\mathcal{A} = \{ (x_i, y_j) : x_i \in \mathbf{x}, y_j \in \mathbf{y} \},
\]

The output space is a set of two-element ordered tuples. \( x_i \) and \( y_j \) in each pair of \( \mathcal{A} \) are the words that
Figure 3: The proposed semi-supervised framework, SemiAlign, illustrated with the Japanese-English training process as an example. The left sub-figure illustrates the process of training multilingual encoder models with self-training. The right sub-figure presents the process of training multilingual decoder models using LoRA with the combination of the pseudo-labeled data predicted by the encoder models and the labeled data. Note that for decoder models, we append positional information to each word to resolve positional ambiguities arising from repeated occurrences of the same word in a sentence. The detailed prompt example for the right sub-figure can be found in Figure 5 (Appendix A.6).

are semantically equivalent to each other within the context of sentences \( x \) and \( y \). The \( x_i \) and \( y_i \) are typically represented as the positions of words in the source and target sentences because word alignment considers positional information. It is worth noting that in some word alignment datasets, each alignment pair in the set \( \mathcal{A} \) is annotated with one of two types of labels: possible alignment (\( \mathcal{P} \)) and sure alignment (\( \mathcal{S} \)).

2.1 Unlabeled Data Collection and Preparation

To improve word alignment with semi-supervised learning, we first need unlabeled data. Unlike WS-PAlign, which uses entity hyperlinks in Wikipedia for Wikipedia entity words and contextual word embeddings for common words as weak word alignment signals, we utilize parallel corpora that do not contain any weak signals. The parallel corpora are denoted as \( \mathcal{D}_u \).

Preprocessing of Unlabeled Data. Given the raw parallel sentences, we perform some preprocessing steps for further training. The first step is tokenization. For English sentences, we use the Moses tokenizer (Koehn et al., 2007), while for other languages, we employ corresponding language-specific tokenizers. The second step involves lowercasing the parallel sentences if the labeled data is in lowercase. This step is omitted for Chinese and Japanese, as these languages do not distinguish between uppercase and lowercase characters. Beyond the aforementioned steps, we also remove any empty lines in the parallel corpora. As the parallel corpora are sourced from different origins, some may have been partially preprocessed. In such cases, we omit the corresponding preprocessing steps that have been applied.

2.2 Multilingual Encoders: Self-Training

For encoder models, we utilize the idea of self-training (Amini et al., 2025) to fully leverage the power of encoder-based language models and mitigate the issue of limited labeled data instead of using weak supervision signals from Wikipedia entity links (Wu et al., 2023). After the preprocessing of unlabeled data, we utilize the current state-of-the-art model, BinaryAlign, to generate the pseudo-labels for unlabeled data \( \mathcal{D}_u \). The pseudo-labeled data is denoted as \( \mathcal{D}_p \). Then we combine the pseudo-labeled data \( \mathcal{D}_p \) and the labeled data \( \mathcal{D}_l \) to re-train an encoder model via the binary sequence labeling task of BinaryAlign, as shown in the left sub-figure of Figure 3. After training, we can obtain SemiAlign-E.
2.3 Multilingual LLMs: Exploring the Possibility as Word Aligners

In this section, we first evaluate the performance of LLMs that are fine-tuned on labeled data to gain insights into the effectiveness of LLM-based word aligners using standard supervised fine-tuning. Subsequently, we investigate whether incorporating semi-supervised learning can further improve the performance of LLM-based word aligners. Given that generating labels with LLM-based word aligners is relatively computationally intensive, we primarily leverage the pseudo-labeling capabilities of encoder-based methods. Specifically, we employ the trained encoder model, SemiAlign-E, from the previous section to generate pseudo-labels for the unlabeled data.

Controlling Generation Granularity. Word alignment can be reformulated as different tasks (Nagata et al., 2020; Latouche et al., 2024). Previous studies (Nagata et al., 2020; Wu et al., 2023) define the task as a SQuAD question answering task via span prediction or a binary sequence labeling task (Latouche et al., 2024). For decoder-based LLMs, it is necessary to reformulate the word alignment task as a generative task while incorporating positional information. Additionally, the generation granularity should be carefully considered when feeding translation pairs into these LLMs. However, the optimal generation granularity for LLM-based word aligners remains underexplored. We investigate two levels of generation granularity: (1) “full mode”, where the model outputs all aligned word pairs with their positions given the source and target sentence in a single pass; and (2) “marker mode”, where a specific word in the source sentence is marked with special tokens, and the model outputs its corresponding aligned words with positions in the target sentence. The special tokens are represented by the orange symbols in the right panel of Figure 3. The prompt formats for “full mode” and “marker mode” can be found in Appendix A.6.

We use the cross-entropy loss for next token prediction to train our decoder models. The training data is the combination of pseudo-labeled data and labeled data. We mask the instruction prompt part when computing the loss during the training following self-instruct (Wang et al., 2023). To augment the training data, we incorporate both source-to-target and target-to-source alignment directions.

Symmetric Alignment for LLMs. For the marker mode generation, we mark one word in the source sentence and let the model predict its corresponding words and positions in the target sentence. This prediction can also be performed in the other direction. Following SpanAlign (Nagata et al., 2020) and WSPAAlign (Wu et al., 2023), we perform the symmetric alignment during evaluation or prediction. Specifically, we obtain predictions from both alignment directions and combine them via union operation, preserving all predictions from both directions.

3 Experiments

3.1 Datasets

We mainly evaluate the performance of the proposed method on four language pairs: Japanese-English (ja-en), German-English (de-en), Romanian-English (ro-en) and Chinese-English (zh-en). We use ISO 639-1 language codes1 to denote languages in the following tables. Specifically, “ja” denotes Japanese. “zh” denotes Chinese. “ro” denotes Romanian. “de” represents German. “en” corresponds to English. We also test our method on some low-resource languages including Swedish (sv), Finnish (fi), and Hebrew (he).

Labeled Data. For Japanese-English (ja-en) word alignment, we use The Kyoto Free Translation Task (KFTT) word alignment dataset (Neubig, 2011). In this dataset, we adopt the same data splits as those used in SpanAlign (Nagata et al., 2020), WSPAAlign (Wu et al., 2023) and BinaryAlign (Latouche et al., 2024) with the script available at the repository of SpanAlign2. The Romanian-English word alignment dataset comes from Mihalcea and Pedersen (2003). The German-English word alignment dataset is provided by Vilar et al. (2006). The Chinese-English word alignment dataset is obtained from the TsinghuaAligner website3. We use the v1 version of the dataset following BinaryAlign. Note that we use the same splitting approach as WSPAAlign following BinaryAlign in the above datasets. Moreover, we utilize the Finnish-Greek (fi-el) and the Finnish-Hebrew (fi-he) word alignment datasets from Imani et al. (2021) and the

1https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes
2https://github.com/nttcslab-nlp/word_align
3https://nlp.csai.tsinghua.edu.cn/~ly/systems/TsinghuaAligner/TsinghuaAligner.html
<table>
  <tr>
    <th>Lang</th>
    <th>Method</th>
    <th>Precision (%)</th>
    <th>Recall (%)</th>
    <th>F1 (%)</th>
    <th>AER (%)</th>
  </tr>
  <tr>
    <td rowspan="6">ja-en</td>
    <td>SpanAlign (Nagata et al., 2020)</td>
    <td>77.3</td>
    <td>78.0</td>
    <td>77.6</td>
    <td>22.4</td>
  </tr>
  <tr>
    <td>WSPAIngn (Wu et al., 2023)</td>
    <td>81.6</td>
    <td>85.9</td>
    <td>83.7</td>
    <td>16.3</td>
  </tr>
  <tr>
    <td>BinaryAlign* (Latouche et al., 2024)</td>
    <td>87.74</td>
    <td>84.46</td>
    <td>86.07</td>
    <td>13.93</td>
  </tr>
  <tr>
    <td>Llama3.1 Inst + SFT</td>
    <td>78.65</td>
    <td>89.76</td>
    <td>83.84</td>
    <td>16.16</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ours)</td>
    <td>88.34</td>
    <td>86.91</td>
    <td>87.62</td>
    <td>12.38</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ours)</td>
    <td>88.09</td>
    <td>88.63</td>
    <td><b>88.36</b></td>
    <td><b>11.64</b></td>
  </tr>
  <tr>
    <td rowspan="6">de-en</td>
    <td>SpanAlign (Nagata et al., 2020)</td>
    <td>89.9</td>
    <td>81.7</td>
    <td>85.6</td>
    <td>14.4</td>
  </tr>
  <tr>
    <td>WSPAIngn (Wu et al., 2023)</td>
    <td>90.7</td>
    <td>87.1</td>
    <td>88.9</td>
    <td>11.1</td>
  </tr>
  <tr>
    <td>BinaryAlign* (Latouche et al., 2024)</td>
    <td>93.79</td>
    <td>90.73</td>
    <td>92.23</td>
    <td>7.74</td>
  </tr>
  <tr>
    <td>Llama3.1 Inst + SFT</td>
    <td>85.46</td>
    <td>90.98</td>
    <td>88.14</td>
    <td>11.93</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ours)</td>
    <td>94.55</td>
    <td>90.93</td>
    <td><b>92.70</b></td>
    <td><b>7.27</b></td>
  </tr>
  <tr>
    <td>SemiAlign-D (ours)</td>
    <td>93.97</td>
    <td>90.69</td>
    <td>92.31</td>
    <td>7.66</td>
  </tr>
  <tr>
    <td rowspan="6">ro-en</td>
    <td>SpanAlign (Nagata et al., 2020)</td>
    <td>90.4</td>
    <td>85.3</td>
    <td>86.7</td>
    <td>12.2</td>
  </tr>
  <tr>
    <td>WSPAIngn (Wu et al., 2023)</td>
    <td>92.0</td>
    <td>90.9</td>
    <td>91.4</td>
    <td>8.6</td>
  </tr>
  <tr>
    <td>BinaryAlign* (Latouche et al., 2024)</td>
    <td>92.67</td>
    <td>92.51</td>
    <td>92.59</td>
    <td>7.41</td>
  </tr>
  <tr>
    <td>Aya Expanse + SFT</td>
    <td>80.37</td>
    <td>87.72</td>
    <td>83.89</td>
    <td>16.11</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ours)</td>
    <td>93.55</td>
    <td>92.68</td>
    <td><b>93.11</b></td>
    <td><b>6.89</b></td>
  </tr>
  <tr>
    <td>SemiAlign-D (ours)</td>
    <td>94.65</td>
    <td>89.52</td>
    <td>92.02</td>
    <td>7.98</td>
  </tr>
  <tr>
    <td rowspan="6">zh-en</td>
    <td>SpanAlign (Nagata et al., 2020)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>8.9</td>
  </tr>
  <tr>
    <td>WSPAIngn (Wu et al., 2023)</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>7.6</td>
  </tr>
  <tr>
    <td>BinaryAlign* (Latouche et al., 2024)</td>
    <td>95.63</td>
    <td>94.13</td>
    <td>94.87</td>
    <td>5.12</td>
  </tr>
  <tr>
    <td>Llama3.1 Inst + SFT</td>
    <td>92.60</td>
    <td>96.00</td>
    <td>94.27</td>
    <td>5.73</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ours)</td>
    <td>95.57</td>
    <td>95.14</td>
    <td>95.35</td>
    <td>4.65</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ours)</td>
    <td>96.18</td>
    <td>95.13</td>
    <td><b>95.65</b></td>
    <td><b>4.35</b></td>
  </tr>
</table>

Table 1: Results of different methods on word alignment datasets. SemiAlign-E denotes the multilingual encoder models that are trained with the proposed method. SemiAlign-D indicates the multilingual large language models that are trained with our framework. “SFT” indicates the results of supervised fine-tuning using labeled word alignment datasets. “BinaryAlign*” denotes the best reproduced results of BinaryAlign with precision, recall, F1 and AER shown to give a detailed comparison. We highlight in bold the best performance of F1 and AER scores because higher F1 and lower AER scores indicate better performance.

English-Swedish (en-sv) word alignment dataset from Holmqvist and Ahrenberg (2011) to evaluate the performance of various models on low-resource languages. The number of training, validation and test examples in the labeled data can be found in Table 10.

Unlabeled Data. For Japanese-English, we use a subset of the KFTT translation data (Neubig, 2011). For Romanian-English, we use the training parallel corpus4 from Vilar et al. (2006). For the German-English pair, we use a subset of News Commentary v18.1 (Kocmi et al., 2023) as the unlabeled data. For Chinese-English pair, we use the News Commentary v16. For English-Swedish, Finnish-Greek and Finnish-Hebrew pairs, we utilize subsets of CCMatrix v1.0 (Schwenk et al., 2021; Fan et al., 2021; Tiedemann, 2012). The max number of unlabeled data that we use is shown in Table 11. More details about the unlabeled data can be found in Appendix A.2

3.2 Evaluation Metrics

We evaluate the word alignment quality using Precision (P), Recall (R), F1 and Alignment Error Rate (AER) (Och and Ney, 2003) for all experiments following previous works (Nagata et al., 2020; Wu et al., 2023). The details of evaluation metrics can be found in Appendix A.5.

Note that BinaryAlign (Latouche et al., 2024) only presents the AER scores for all language pairs. We reimplement BinaryAlign using BinaryAlign repository5 and present the best reproduced precision, recall, F1 and AER scores for a fair and detailed comparison.

3.3 Models

To evaluate the effectiveness of our method, we consider two types of multilingual language models: encoder-based and decoder-based language models.

Multilingual Encoders. We utilize mDeBERTa-v3-base6 (He et al., 2023) as our base model following BinaryAlign. For the reproduced BinaryAlign

4http://web.eecs.umich.edu/~mihalcea/wpt.data.protected/Romanian–English.training.tar.gz
5https://github.com/ubisoft/ubisoft-laforge-BinaryAlignWordAlignmentasBinarySequenceLabeling/
6https://huggingface.co/microsoft/mdeberta-v3-base
experiments, we also use mDeBERTa-v3-base as the base model for a fair comparison. All hyper-parameters in the reproduced BinaryAlign experiments are the same as BinaryAlign. We present the best reproduced results on all language pairs. For the experiments that use pseudo-labeled data, the total batch size is 32. The number of epoch is 1. For other hyperparameters, we follow the setting in BinaryAlign which can be found in its GitHub repository.

Multilingual LLMs. For decoder-based LLMs, we mainly use Llama-3.1-8B-Instruct (Dubey et al., 2024) as our base model. The only exception is the Romanian English pair because we find that the best result of the Romanian-English pair is based on the aya-expanse-8b (Dang et al., 2024). The total batch size is 32. The initial learning rate is 2e-5. The scheduler type is “cosine”. We utilized LoRA for parameter-efficient training. For the experiments that leverage pseudo-label data, the number of epochs is 1. We use bf16 mixed precision training. The optimizer is “adamw_torch”. The weight decay is set to 0. Additional training details and LoRA settings can be found in Appendix A.3 and A.4.

3.4 Baselines

We mainly compare our method with three approaches: SpanAlign, WSPAlign and BinaryAlign. SpanAlign is a supervised word alignment training method that reformulates the word alignment task as a question answering task (Rajpurkar et al., 2018) via span prediction. WSPAlign focuses on utilizing large-scale weak signals from entity hyperlinks in Wikipedia articles for word alignment pre-training to improve SpanAlign. BinaryAlign reformulates the word alignment task as a binary sequence labeling task achieving a good balance between outputting all word alignments at once (all-at-once) and predicting one target word for each source word (one-by-one way) in one pass. For multilingual decoder-based LLMs, we present the results of the models that are supervised fine-tuned on labeled data for reference (SFT row) in Table 1.

3.5 Low-Resource Settings

Zero-Shot Setting. In the zero-shot setting, the training sets of the word alignment datasets for our target language pairs are not available. We assess the performance of our models that are trained on a different language pair on the test set of the target word alignment dataset directly. We use the models that are trained on the word alignment datasets of high-resource language pairs (ja-en, ro-en, de-en and zh-en) and evaluate their performance on three low-resource languages, that is, English-Swedish (en-sv), Finnish-Greek (fi-el), and Finnish-Hebrew (fi-he).

Few-Shot Setting. In this setting, we train our models with few-shot examples for a language pair and evaluate the models in the same language pair. Specifically, we use 32 examples from the training sets of word alignment datasets to serve as supervision signals. For SemiAlign-D, we use the same number of pseudo-labeled examples as the labeled examples (32), considering the training cost. For SemiAlign-E, we evaluate two different settings for pseudo-labeled data: one using 32 pseudo-labeled examples and the other using the pseudo-labeled examples that yield the best results in the fully supervised setting. We present the superior results from these two configurations. For the pseudo-labelers, we choose the BinaryAlign models that are trained with 32 examples for each language pair. These models also serve as our baseline methods for comparison. We evaluate different approaches in this setting with seven language pairs (ja-en, ro-en, de-en, zh-en, en-sv, fi-el and fi-he).

4 Results and Analysis

The main results are presented in Table 1, demonstrating that the proposed method achieves new state-of-the-art performance. The Japanese-English word alignment dataset shows the most significant improvement. For the encoder-based model, the AER score of SemiAlign-E improves from 13.93 to 12.38 compared to BinaryAlign. For decoder-based LLMs, the AER score of SemiAlign-D improves from 16.16 to 11.64 compared with its SFT baseline. The average F1 and AER scores of SemiAlign-E and SemiAlign-D across the four datasets differ by about 0.1 and better than BinaryAlign. On average, the improvement in AER scores across the four datasets shows that SemiAlign-D achieves a greater performance gain over the SFT baseline compared to SemiAlign-E’s improvement over BinaryAlign. This indicates that the size of the training data has a relatively larger impact on multilingual large language models.

A Small Amount of Unlabeled Data Boosts the Performance of BinaryAlign. We re-train Bi-
<table>
  <tr>
    <th>Lang</th>
    <th>Method</th>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>AER</th>
  </tr>
  <tr>
    <td rowspan="3">en-sv</td>
    <td>BinaryAlign (ja-en)</td>
    <td>94.54</td>
    <td>97.28</td>
    <td>95.89</td>
    <td>4.21</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ja-en)</td>
    <td>91.42</td>
    <td>92.47</td>
    <td>92.17</td>
    <td>7.19</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ja-en)</td>
    <td>91.77</td>
    <td>97.63</td>
    <td>94.18</td>
    <td>3.92</td>
  </tr>
  <tr>
    <td rowspan="3">en-sv</td>
    <td>BinaryAlign (ro-en)</td>
    <td>91.55</td>
    <td>97.25</td>
    <td>94.31</td>
    <td>5.95</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ro-en)</td>
    <td>91.04</td>
    <td>94.29</td>
    <td>91.76</td>
    <td>8.30</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ro-en)</td>
    <td>92.38</td>
    <td>97.43</td>
    <td>94.84</td>
    <td>5.40</td>
  </tr>
  <tr>
    <td rowspan="3">en-sv</td>
    <td>BinaryAlign (de-en)</td>
    <td>95.73</td>
    <td>96.86</td>
    <td>96.29</td>
    <td>3.75</td>
  </tr>
  <tr>
    <td>SemiAlign-D (de-en)</td>
    <td>95.37</td>
    <td>94.61</td>
    <td>95.00</td>
    <td>4.98</td>
  </tr>
  <tr>
    <td>SemiAlign-E (de-en)</td>
    <td>96.60</td>
    <td>96.86</td>
    <td>96.73</td>
    <td>3.28</td>
  </tr>
  <tr>
    <td rowspan="3">fi-el</td>
    <td>BinaryAlign (zh-en)</td>
    <td>98.24</td>
    <td>93.53</td>
    <td>95.81</td>
    <td>4.11</td>
  </tr>
  <tr>
    <td>SemiAlign-D (zh-en)</td>
    <td>97.44</td>
    <td>93.92</td>
    <td>95.65</td>
    <td>4.24</td>
  </tr>
  <tr>
    <td>SemiAlign-E (zh-en)</td>
    <td>98.30</td>
    <td>93.95</td>
    <td>96.08</td>
    <td>3.85</td>
  </tr>
  <tr>
    <td rowspan="3">fi-el</td>
    <td>BinaryAlign (ja-en)</td>
    <td>83.09</td>
    <td>76.69</td>
    <td>79.76</td>
    <td>20.24</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ja-en)</td>
    <td>82.07</td>
    <td>76.18</td>
    <td>79.02</td>
    <td>20.98</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ja-en)</td>
    <td>83.03</td>
    <td>78.53</td>
    <td>80.72</td>
    <td>19.28</td>
  </tr>
  <tr>
    <td rowspan="3">fi-el</td>
    <td>BinaryAlign (ro-en)</td>
    <td>80.87</td>
    <td>74.53</td>
    <td>82.30</td>
    <td>17.30</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ro-en)</td>
    <td>81.73</td>
    <td>73.25</td>
    <td>82.92</td>
    <td>19.91</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ro-en)</td>
    <td>83.76</td>
    <td>83.63</td>
    <td>86.99</td>
    <td>16.31</td>
  </tr>
  <tr>
    <td rowspan="3">fi-el</td>
    <td>BinaryAlign (de-en)</td>
    <td>86.66</td>
    <td>70.72</td>
    <td>77.64</td>
    <td>22.36</td>
  </tr>
  <tr>
    <td>SemiAlign-D (de-en)</td>
    <td>87.63</td>
    <td>77.27</td>
    <td>82.13</td>
    <td>17.87</td>
  </tr>
  <tr>
    <td>SemiAlign-E (de-en)</td>
    <td>87.09</td>
    <td>71.89</td>
    <td>78.76</td>
    <td>21.24</td>
  </tr>
  <tr>
    <td rowspan="3">fi-el</td>
    <td>BinaryAlign (zh-en)</td>
    <td>88.87</td>
    <td>68.47</td>
    <td>77.35</td>
    <td>22.65</td>
  </tr>
  <tr>
    <td>SemiAlign-D (zh-en)</td>
    <td>86.15</td>
    <td>77.49</td>
    <td>81.59</td>
    <td>18.41</td>
  </tr>
  <tr>
    <td>SemiAlign-E (zh-en)</td>
    <td>87.84</td>
    <td>70.03</td>
    <td>77.93</td>
    <td>22.07</td>
  </tr>
  <tr>
    <td rowspan="3">fi-he</td>
    <td>BinaryAlign (ja-en)</td>
    <td>70.54</td>
    <td>63.49</td>
    <td>66.83</td>
    <td>33.17</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ja-en)</td>
    <td>71.06</td>
    <td>62.03</td>
    <td>66.24</td>
    <td>33.76</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ja-en)</td>
    <td>72.96</td>
    <td>62.86</td>
    <td>67.54</td>
    <td>32.46</td>
  </tr>
  <tr>
    <td rowspan="3">fi-he</td>
    <td>BinaryAlign (ro-en)</td>
    <td>70.83</td>
    <td>57.08</td>
    <td>63.22</td>
    <td>36.78</td>
  </tr>
  <tr>
    <td>SemiAlign-D (ro-en)</td>
    <td>77.90</td>
    <td>58.96</td>
    <td>67.12</td>
    <td>32.88</td>
  </tr>
  <tr>
    <td>SemiAlign-E (ro-en)</td>
    <td>79.39</td>
    <td>55.35</td>
    <td>65.23</td>
    <td>34.77</td>
  </tr>
  <tr>
    <td rowspan="3">fi-he</td>
    <td>BinaryAlign (de-en)</td>
    <td>82.80</td>
    <td>57.86</td>
    <td>65.35</td>
    <td>39.04</td>
  </tr>
  <tr>
    <td>SemiAlign-D (de-en)</td>
    <td>82.52</td>
    <td>58.86</td>
    <td>67.39</td>
    <td>30.61</td>
  </tr>
  <tr>
    <td>SemiAlign-E (de-en)</td>
    <td>86.70</td>
    <td>42.48</td>
    <td>57.02</td>
    <td>42.98</td>
  </tr>
  <tr>
    <td rowspan="3">fi-he</td>
    <td>BinaryAlign (zh-en)</td>
    <td>84.87</td>
    <td>54.04</td>
    <td>66.04</td>
    <td>33.96</td>
  </tr>
  <tr>
    <td>SemiAlign-D (zh-en)</td>
    <td>62.16</td>
    <td>40.20</td>
    <td>48.82</td>
    <td>51.18</td>
  </tr>
  <tr>
    <td>SemiAlign-E (zh-en)</td>
    <td>81.34</td>
    <td>55.43</td>
    <td>65.93</td>
    <td>34.07</td>
  </tr>
</table>

Table 2: Zero-shot evaluation results. The first column denotes the test language pair. The language pairs in parentheses in the Method column indicate the training language pair.

BinaryAlign using self-training with one iteration. All hyperparameter settings are the same as BinaryAlign except that the number of epochs is set to 1 and the batch size is changed from 8 to 32. We can observe that the self-training with the small-scale corpora boost the performance of BinaryAlign on all language pairs in Table 1. The most obvious one is Japanese-English pair and the self-training improves the AER score by 1.6.

Utilizing Labeled Data Is Not Enough for Multilingual LLM-Based Word Aligners. For a better comparison and investigation of factors that affect the word alignment performance of decoder-based language models, we mainly fine-tune the Llama-3.1-8B-Instruct for all languages with the labeled data as the baseline. For Romanian-English, we use the aya-expanse-8b model because we find that its performance on this language pair is better. For the SFT version of these large language models, the number of epochs is 5. The other hyperparameters are the same as the main experiments. In Table 1, we find that the performance of using the labeled data is not competitive compared with BinaryAlign. We also observe that the proposed framework with a small amount of parallel data improves the decoder-based word aligners by 1.4 on Chinese-English, 4.5 on Japanese-English, 4.3 on German-English, 6.68 on Romanian-English respectively. This indicates that the multilingual large language model-based word aligners are leveraging the pseudo-labeled data efficiently.

Zero-Shot Setting. The results of zero-shot setting are shown in Table 2. For the models that are trained on Japanese-English data and evaluated on other language pairs, SemiAlign-E achieves the best performance. For the models that are trained on Romanian-English data, SemiAlign-E is a good choice in most cases. Overall, SemiAlign maintains its generalization performance on low-resource languages and, in most cases, SemiAlign-E even outperforms BinaryAlign. The reason may be that the base model of SemiAlign-E has been pre-trained on data from a large number of languages. In contrast, the base models of SemiAlign-D, such as Llama-3.1-8B-Instruct and aya-expanse-8b, are typically pretrained on a relatively smaller number of languages during the pre-training phase.

<table>
  <tr>
    <th>Lang</th>
    <th>Method</th>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>AER</th>
  </tr>
  <tr>
    <td rowspan="3">ja-en</td>
    <td>BinaryAlign</td>
    <td>81.86</td>
    <td>72.33</td>
    <td>76.8</td>
    <td>23.2</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>68.44</td>
    <td>78.04</td>
    <td>72.93</td>
    <td>27.07</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>84.10</td>
    <td>73.39</td>
    <td>78.38</td>
    <td>21.62</td>
  </tr>
  <tr>
    <td rowspan="3">ro-en</td>
    <td>BinaryAlign</td>
    <td>90.65</td>
    <td>90.92</td>
    <td>90.78</td>
    <td>9.22</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>76.75</td>
    <td>85.86</td>
    <td>81.05</td>
    <td>18.95</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>91.80</td>
    <td>91.18</td>
    <td>91.49</td>
    <td>8.51</td>
  </tr>
  <tr>
    <td rowspan="3">de-en</td>
    <td>BinaryAlign</td>
    <td>93.07</td>
    <td>90.49</td>
    <td>91.76</td>
    <td>8.21</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>83.10</td>
    <td>87.45</td>
    <td>85.22</td>
    <td>14.83</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>93.34</td>
    <td>90.90</td>
    <td>92.10</td>
    <td>7.78</td>
  </tr>
  <tr>
    <td rowspan="3">zh-en</td>
    <td>BinaryAlign</td>
    <td>94.73</td>
    <td>91.40</td>
    <td>93.04</td>
    <td>6.96</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>89.67</td>
    <td>90.24</td>
    <td>89.96</td>
    <td>10.04</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>95.07</td>
    <td>91.39</td>
    <td>93.19</td>
    <td>6.81</td>
  </tr>
  <tr>
    <td rowspan="3">en-sv</td>
    <td>BinaryAlign</td>
    <td>95.96</td>
    <td>97.46</td>
    <td>96.70</td>
    <td>3.34</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>87.17</td>
    <td>92.99</td>
    <td>89.99</td>
    <td>10.18</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>96.73</td>
    <td>97.22</td>
    <td>96.97</td>
    <td>3.04</td>
  </tr>
  <tr>
    <td rowspan="3">fi-el</td>
    <td>BinaryAlign</td>
    <td>91.06</td>
    <td>86.22</td>
    <td>88.58</td>
    <td>11.42</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>82.79</td>
    <td>87.69</td>
    <td>85.17</td>
    <td>14.83</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>91.48</td>
    <td>87.79</td>
    <td>89.60</td>
    <td>10.40</td>
  </tr>
  <tr>
    <td rowspan="3">fi-he</td>
    <td>BinaryAlign</td>
    <td>86.03</td>
    <td>76.21</td>
    <td>80.82</td>
    <td>19.18</td>
  </tr>
  <tr>
    <td>SemiAlign-D</td>
    <td>78.00</td>
    <td>88.54</td>
    <td>82.94</td>
    <td>17.06</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td>86.85</td>
    <td>79.17</td>
    <td>82.83</td>
    <td>17.17</td>
  </tr>
</table>

Table 3: Few-shot evaluation results. The first column denotes the training and test language pair. The number of available labeled word alignment examples is 32.

Few-Shot Setting. We present the results of different methods in the few-shot setting in Table 3. In this setting, we only use 32 labeled examples to train our base models, and generate pseudo-labeled examples for unlabeled data using the
<table>
  <tr>
    <th>Lang</th>
    <th>Method</th>
    <th># Parameters</th>
    <th># Examples</th>
    <th>Inference Time</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>BinaryAlign/SemiAlign-E</td>
    <td>276M</td>
    <td>357</td>
    <td>0h:00m:4.5s</td>
  </tr>
  <tr>
    <td></td>
    <td>SemiAlign-D*</td>
    <td>8B</td>
    <td></td>
    <td>2h:03m:15s</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>BinaryAlign/SemiAlign-E</td>
    <td>276M</td>
    <td>98</td>
    <td>0h:00m:2.5s</td>
  </tr>
  <tr>
    <td></td>
    <td>SemiAlign-D*</td>
    <td>8B</td>
    <td></td>
    <td>1h:00m:12s</td>
  </tr>
  <tr>
    <td>de-en</td>
    <td>BinaryAlign/SemiAlign-E</td>
    <td>276M</td>
    <td>208</td>
    <td>0h:00m:1.3s</td>
  </tr>
  <tr>
    <td></td>
    <td>SemiAlign-D*</td>
    <td>8B</td>
    <td></td>
    <td>1h:04m:11s</td>
  </tr>
  <tr>
    <td>zh-en</td>
    <td>BinaryAlign/SemiAlign-E</td>
    <td>276M</td>
    <td>450</td>
    <td>0h:01m:02s</td>
  </tr>
  <tr>
    <td></td>
    <td>SemiAlign-D*</td>
    <td>8B</td>
    <td></td>
    <td>3h:12m:48s</td>
  </tr>
</table>

Table 4: Actual inference time on the test sets of word alignment datasets. SemiAlign-D* in the ro-en row indicates that we use aya-expanse-8b as the base model rather than llama-3.1-8b-instruct model.

model trained with 32 labeled examples. After that, we re-train our base models with our semi-supervised framework. We find that SemiAlign-E usually achieves better performance in this setting compared with BinaryAlign and SemiAlign-D except the fi-he pair. We observe that, in fi-he pair, SemiAlign-D that is trained with 32 pseudo-labeled examples achieves the best performance.

Error Analysis. We conduct an error analysis to explore scenarios in which the proposed method fails. Specifically, we collect all incorrect predictions in the test sets of the Japanese-English, German-English, Romanian-English, and Chinese-English word alignment datasets at the word level. We categorize the incorrect predictions into three types: (1) refusal errors, where the model produces no target word predictions despite the existence of valid alignments; (2) single misalignment error, where one incorrect target word is predicted; and (3) multiple misalignment errors, where several incorrect target words are generated for a single source word. We also present an analysis of incorrect predictions obtained by examining each source words that have gold alignments.

The statistics of incorrect predictions of SemiAlign-D and SemiAlign-E by iterating over each source word and the source words that have gold alignments are shown in Table 5, 6, 7, and 8 correspondingly.

<table>
  <tr>
    <th>Lang</th>
    <th># refusal (%)</th>
    <th># single misalignment (%)</th>
    <th># multiple misalignments (%)</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>20.74</td>
    <td>46.20</td>
    <td>33.40</td>
  </tr>
  <tr>
    <td>de-en (w. p.)</td>
    <td>45.40</td>
    <td>58.15</td>
    <td>18.44</td>
  </tr>
  <tr>
    <td>de-en (w/o. p.)</td>
    <td>27.50</td>
    <td>52.92</td>
    <td>19.58</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>42.77</td>
    <td>41.33</td>
    <td>15.90</td>
  </tr>
  <tr>
    <td>zh-en (w. p.)</td>
    <td>39.61</td>
    <td>40.02</td>
    <td>20.36</td>
  </tr>
  <tr>
    <td>zh-en (w/o. p.)</td>
    <td>27.48</td>
    <td>51.40</td>
    <td>21.12</td>
  </tr>
</table>

Table 5: Distribution of prediction error types for SemiAlign-D, obtained by iterating over each source word of the test sets of different languages. “(w. p)” indicates that we consider possible alignments while “(w/o. p)” denotes that we do not consider them during the analysis.

<table>
  <tr>
    <th>Lang</th>
    <th># refusal (%)</th>
    <th># single misalignment (%)</th>
    <th># multiple misalignments (%)</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>22.24</td>
    <td>53.24</td>
    <td>25.52</td>
  </tr>
  <tr>
    <td>de-en (w. p.)</td>
    <td>44.84</td>
    <td>40.11</td>
    <td>15.04</td>
  </tr>
  <tr>
    <td>de-en (w/o. p.)</td>
    <td>25.95</td>
    <td>57.79</td>
    <td>16.26</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>18.84</td>
    <td>51.09</td>
    <td>30.07</td>
  </tr>
  <tr>
    <td>zh-en (w. p.)</td>
    <td>33.88</td>
    <td>44.98</td>
    <td>21.14</td>
  </tr>
  <tr>
    <td>zh-en (w/o. p.)</td>
    <td>22.86</td>
    <td>55.58</td>
    <td>21.57</td>
  </tr>
</table>

Table 6: Distribution of prediction error types for SemiAlign-D by iterating over the source words that have gold alignments of the test sets. “(w. p)” indicates that we consider possible alignments while “(w/o. p)” denotes that we do not consider them during the analysis.

<table>
  <tr>
    <th>Lang</th>
    <th># refusal (%)</th>
    <th># single misalignment (%)</th>
    <th># multiple misalignments (%)</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>20.74</td>
    <td>46.20</td>
    <td>33.40</td>
  </tr>
  <tr>
    <td>de-en (w. p.)</td>
    <td>45.40</td>
    <td>58.15</td>
    <td>18.44</td>
  </tr>
  <tr>
    <td>de-en (w/o. p.)</td>
    <td>27.50</td>
    <td>52.92</td>
    <td>19.58</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>42.77</td>
    <td>41.33</td>
    <td>15.90</td>
  </tr>
  <tr>
    <td>zh-en (w. p.)</td>
    <td>39.61</td>
    <td>40.02</td>
    <td>20.36</td>
  </tr>
  <tr>
    <td>zh-en (w/o. p.)</td>
    <td>27.48</td>
    <td>51.40</td>
    <td>21.12</td>
  </tr>
</table>

Table 7: Distribution of prediction error types for SemiAlign-E by iterating over each source word of the test sets. “(w. p)” indicates that we consider possible alignments while “(w/o. p)” denotes that we do not consider them during the analysis.

ment datasets, model sizes, and number of examples. SemiAlign-E has the same efficiency as BinaryAlign because they use the same base encoder models and their inference processes are identical.

For SemiAlign-D, which uses LLMs as the base model, although it achieves the highest scores on ja-en and zh-en as shown in Table 1, SemiAlign-D has the following disadvantages: slower inference speed and larger model size. Therefore, for pipelines or tasks that are sensitive to latency, it is preferable to use a more lightweight encoder model, SemiAlign-E.

Effect of the generation granularity. Decoder-based large language models treat the word alignment task as a generative task. Before performing the prediction, we must determine the generation granularity first.

To determine the optimal level of generation granularity for the decoder-based language models, we evaluate the performance of the models across different granularity levels using the labeled data. The results of using different levels
<table>
  <tr>
    <th>Lang</th>
    <th># refusal (%)</th>
    <th># single involvement (%)</th>
    <th># multiple involvements (%)</th>
  </tr>
  <tr>
    <td>en</td>
    <td>39.79</td>
    <td>41.06</td>
    <td>30.43</td>
  </tr>
  <tr>
    <td>de-en (w. p.)</td>
    <td>50.40</td>
    <td>33.33</td>
    <td>16.26</td>
  </tr>
  <tr>
    <td>de-en (w/o. p.)</td>
    <td>33.56</td>
    <td>48.71</td>
    <td>17.73</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>22.91</td>
    <td>44.05</td>
    <td>33.04</td>
  </tr>
  <tr>
    <td>zh-en (w. p.)</td>
    <td>47.95</td>
    <td>23.22</td>
    <td>28.83</td>
  </tr>
  <tr>
    <td>zh-en (w/o. p.)</td>
    <td>33.03</td>
    <td>37.84</td>
    <td>29.13</td>
  </tr>
</table>

Table 8: Distribution of prediction error types for SemiAlign-E by iterating over the source words that have gold alignments. “(w. p)” indicates that we consider possible alignments while “(w/o. p)” denotes that we do not consider them during the analysis.

of generation granularity are shown in Table 9. As shown in Table 9, we find that the performance of “marker mode” demonstrates significantly superior performance. This indicates that LLM-based word aligners perform well at the word level, outputting the corresponding words given one marked word. The word alignment performance deteriorates when LLM-based word aligners generate all aligned words for a sentence in a single pass.

<table>
  <tr>
    <th>Generation Mode</th>
    <th>Ja-En</th>
    <th>De-En</th>
    <th>Ro-En</th>
    <th>Zh-En</th>
  </tr>
  <tr>
    <td>Full Mode</td>
    <td>26.00</td>
    <td>17.33</td>
    <td>25.73</td>
    <td>9.95</td>
  </tr>
  <tr>
    <td>Marker Mode</td>
    <td><b>16.16</b></td>
    <td><b>11.93</b></td>
    <td><b>14.66</b></td>
    <td><b>5.73</b></td>
  </tr>
</table>

Table 9: AER scores of different generation granularities using labeled data.

More ablation studies and experimental results are available in Appendix B.

5 Related Work

Word alignment. The research community has proposed various methods to improve word alignment based on the multilingual Transformer encoder models (Devlin et al., 2019; Conneau et al., 2020) . There are two categories of research that focus on improving word alignment performance on multilingual encoder-based language models (Latouche et al., 2024). One category consists of methods based on contextual word embeddings extracted from encoder models (Jalili Sabet et al., 2020; Dou and Neubig, 2021; Wang et al., 2022a). For example, SimAlign (Jalili Sabet et al., 2020) proposes three new alignment methods based on contextual word embeddings. The other category involves supervised learning for word alignment. SpanAlign (Nagata et al., 2020) redefines the word alignment task as a span prediction-based question answering task. WSPAIngn (Wu et al., 2023) proposes large-scale weakly pre-training for word alignment, using entity links and contextual word embeddings for entity and common words, due to the different representation level of them (Schick and Schütze, 2020; Wu et al., 2021). BinaryAlign (Latouche et al., 2024) reformulates the word alignment task as a binary sequence labeling task, achieving the new state-of-the-art performance. Both of the above categories are based on the multilingual encoder language models. However, considering that autoregressive decoder-based LLMs have achieved impressive performance on many NLP tasks, such as coding, reasoning and machine translation (Shojaee et al., 2023; Wu et al., 2024; Dubey et al., 2024; Miao et al., 2024b; Dang et al., 2024), we also explore the possibility of using multilingual LLMs as word aligners, an avenue that remains relatively unexplored by the research community.

Semi-Supervised Learning. Semi-supervised learning aims to improve various machine learning systems with labeled data and unlabeled data (Yang et al., 2023). Recently, researchers have successfully applied semi-supervised learning to various emergent domains, such as sentence semantics (Wu et al., 2022; Zhao et al., 2024), multimodal domains (Wang et al., 2022b; Deng et al., 2024; Ouyang et al., 2025a; Hoyer et al., 2025), aspect sentiment quad prediction (Zhang et al., 2024), improving reasoning ability of large language models (Wang et al., 2024) and reward modeling (Li et al., 2024; He et al., 2024). In this work, we investigate the application of semi-supervised learning on word alignment through self-training and pseudo-labeling.

6 Conclusion

In this paper, we find that the current state-of-the-art method, BinaryAlign, can be further improved with semi-supervised learning with a small amount of unlabeled data. We also explore the possibility of utilizing multilingual LLMs as word aligners. We observe that fine-tuning multilingual LLMs using the labeled data is insufficient, and not comparable to the current state-of-the-art BinaryAlign. Therefore, we also utilize the pseudo-labeled data from the encoder-based word alignment models to improve multilingual LLM-based word aligners in our proposed framework. Both multilingual encoder models and LLMs that are trained with the proposed method achieve competitive performance, outperforming the current state-of-the-art method on average. The extensive experimental results demonstrate the effectiveness of the proposed method.
Limitations

We explore the possibility of using multilingual LLMs as word aligners and propose SemiAlign-D. However, compared to SemiAlign-E, the primary limitation of SemiAlign-D lies in their high computational requirements, while their performance in word alignment is not significantly superior to that of SemiAlign-E. In order to fully leverage the potential of LLMs, we need to explore better ways to do word alignment using LLMs in the future. Another interesting future direction is how to mitigate confirmation bias. Confirmation bias is a common dilemma for semi-supervised learning. All pseudo-labelers cannot avoid introducing confirmation bias. Figure 2 in the paper shows the performance trend of our method when the number of pseudo-labeled data increases. From the figure, we find that the impact of confirmation bias is relatively small on the performance of our method. The reason for the minimal impact of confirmation bias in our method might be that our models are trained using a single iteration within the semi-supervised framework. This single iteration minimizes the accumulation of errors in the predicted pseudo-labels during training.

Ethics Statement

This paper aims to improve the word alignment performance of multilingual pre-trained language models. All datasets that are used in this paper are publicly available without copyright issues. We use mDeBERTa-v3-base, Llama-3.1-8B-Instruct and aya-expanse-8b models in our experiments adhering to the respective model licenses. Note that certain models built upon large language models may generate content irrelevant to word alignment if their prompts are not properly configured.

References

Massih-Reza Amini, Vasilii Feofanov, Loïc Pauletto, Liès Hadjadj, Émilie Devijver, and Yury Maximov. 2025. Self-training: A survey. Neurocomputing, 616:128904.

Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311.

Guanhua Chen, Yun Chen, and Victor O.K. Li. 2021. Lexically constrained neural machine translation with explicit alignment guidance. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12630–12638.

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2021. Improving pretrained cross-lingual language models via self-labeled word alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3418–3430, Online. Association for Computational Linguistics.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261.

Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Zou, Kai-Wei Chang, and Wei Wang. 2024. Enhancing large vision language models with self-training on image comprehension. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112–2128, Online. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave,
Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1).

Biao Fu, Donglei Yu, Minpeng Liao, Chengxi Li, Yidong Chen, Kai Fan, and Xiaodong Shi. 2025. Efficient and adaptive simultaneous speech translation with fully unidirectional architecture. arXiv preprint arXiv:2504.11809.

Kazuma Hashimoto, Raffaella Buschiazzo, James Bradbury, Teresa Marshall, Richard Socher, and Caiming Xiong. 2019. A high-quality multilingual dataset for structured documentation translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 116–127, Florence, Italy. Association for Computational Linguistics.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DebertaV3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations.

Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, and Han Zhao. 2024. Semi-supervised reward modeling via iterative self-training. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7365–7377, Miami, Florida, USA. Association for Computational Linguistics.

Maria Holmqvist and Lars Ahrenberg. 2011. A gold standard for English-Swedish word alignment. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pages 106–113, Riga, Latvia. Northern European Association for Language Technology (NEALT).

Lukas Hoyer, David Joseph Tan, Muhammad Ferjad Naeem, Luc Van Gool, and Federico Tombari. 2025. Semivil: Semi-supervised semantic segmentation with vision-language guidance. In Computer Vision – ECCV 2024, pages 257–275, Cham. Springer Nature Switzerland.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Ayyoob Imani, Masoud Jalili Sabet, Lutfi Kerem Senel, Philipp Dufter, François Yvon, and Hinrich Schütze. 2021. Graph algorithms for multiparallel word alignment. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8457–8469, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643, Online. Association for Computational Linguistics.

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pages 1–42, Singapore. Association for Computational Linguistics.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattentention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA. Association for Computing Machinery.

Gaetan Latouche, Marc-André Carbonneau, and Benjamin Swanson. 2024. BinaryAlign: Word alignment as binary sequence labeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10277–10288, Bangkok, Thailand. Association for Computational Linguistics.

Siyuan Li, Weiyang Jin, Zedong Wang, Fang Wu, Zicheng Liu, Cheng Tan, and Stan Z. Li. 2024. Semireward: A general reward model for semi-supervised learning. In The Twelfth International Conference on Learning Representations.

Zhongtao Miao, Qiyu Wu, Kaiyan Zhao, Zilong Wu, and Yoshimasa Tsuruoka. 2024a. Enhancing cross-lingual sentence embedding for low-resource languages with word alignment. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3225–3236, Mexico City, Mexico. Association for Computational Linguistics.

Zhongtao Miao, Kaiyan Zhao, and Yoshimasa Tsuruoka. 2024b. Improving arithmetic reasoning ability of large language models through relation tuples, verification and dynamic feedback. arXiv preprint arXiv:2406.17873.
Rada Mihalcea and Ted Pedersen. 2003. An evaluation exercise for word alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pages 1–10.

Masaaki Nagata, Katsuki Chousa, and Masaaki Nishino. 2020. A supervised word alignment method based on cross-language span prediction using multilingual BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 555–565, Online. Association for Computational Linguistics.

Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt.

Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51.

Liangyang Ouyang, Ruicong Liu, Yifei Huang, Ryosuke Furuta, and Yoichi Sato. 2025a. Actionvos: Actions as prompts for video object segmentation. In Computer Vision – ECCV 2024, pages 216–235, Cham. Springer Nature Switzerland.

Siqi Ouyang, Xi Xu, and Lei Li. 2025b. Infinites: Simultaneous translation of unbounded speech with large language model. arXiv preprint arXiv:2503.02969.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics.

Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8766–8774.

Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6490–6500, Online. Association for Computational Linguistics.

Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. 2023. Execution-based code generation using deep reinforcement learning. Transactions on Machine Learning Research.

Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019. Code-switching for enhancing NMT with pre-specified translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 449–459, Minneapolis, Minnesota. Association for Computational Linguistics.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).

Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76–85, Berlin, Germany. Association for Computational Linguistics.

David Vilar, Maja Popovic, and Hermann Ney. 2006. AER: do we need to “improve” our alignments? In Proceedings of the Third International Workshop on Spoken Language Translation: Papers, Kyoto, Japan.

Tianduo Wang, Shichen Li, and Wei Lu. 2024. Self-training with direct preference optimization improves chain-of-thought reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11917–11928, Bangkok, Thailand. Association for Computational Linguistics.

Weikang Wang, Guanhua Chen, Hanqing Wang, Yue Han, and Yun Chen. 2022a. Multilingual sentence transformer as a multilingual word aligner. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2952–2963, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022b. SimVLM: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Qiyu Wu, Masaaki Nagata, Zhongtao Miao, and Yoshimasa Tsuruoka. 2024. Word alignment as preference for machine translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3223–3239, Miami, Florida, USA. Association for Computational Linguistics.

Qiyu Wu, Masaaki Nagata, and Yoshimasa Tsuruoka. 2023. WSPAAlign: Word alignment pre-training via large-scale weakly supervised span prediction. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11084–11099, Toronto, Canada. Association for Computational Linguistics.

Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, and Daxin Jiang. 2022. PCL: Peer-contrastive learning with diverse augmentations for unsupervised sentence embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12052–12066, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, and Tie-Yan Liu. 2021. Taking notes on the fly helps language pre-training. In International Conference on Learning Representations.

Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. 2023. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering, 35(9):8934–8954.

Yice Zhang, Jie Zeng, Weiming Hu, Ziyi Wang, Shiwei Chen, and Ruifeng Xu. 2024. Self-training with pseudo-label scorer for aspect sentiment quad prediction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11862–11875, Bangkok, Thailand. Association for Computational Linguistics.

Kaiyan Zhao, Qiyu Wu, Xin-Qiang Cai, and Yoshimasa Tsuruoka. 2024. Leveraging multi-lingual positive instances in contrastive learning to improve sentence embedding. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 976–991, St. Julian’s, Malta. Association for Computational Linguistics.
A Additional Experimental Details

A.1 Word Alignment Dataset Statistics

Table 10 shows the number of training, validation and test sets of word alignment datasets that we use covering different levels of supervision.

<table>
  <tr>
    <th>Lang</th>
    <th># Train</th>
    <th># Val</th>
    <th># Test</th>
  </tr>
  <tr>
    <th colspan="4">fully supervised</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>653</td>
    <td>225</td>
    <td>357</td>
  </tr>
  <tr>
    <td>de-en</td>
    <td>300</td>
    <td>-</td>
    <td>208</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>150</td>
    <td>-</td>
    <td>98</td>
  </tr>
  <tr>
    <td>zh-en</td>
    <td>450</td>
    <td>-</td>
    <td>450</td>
  </tr>
  <tr>
    <th colspan="4">few shot</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>32</td>
    <td>225</td>
    <td>357</td>
  </tr>
  <tr>
    <td>de-en</td>
    <td>32</td>
    <td>-</td>
    <td>208</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>32</td>
    <td>-</td>
    <td>98</td>
  </tr>
  <tr>
    <td>zh-en</td>
    <td>32</td>
    <td>-</td>
    <td>450</td>
  </tr>
  <tr>
    <td>en-sv</td>
    <td>32</td>
    <td>-</td>
    <td>192</td>
  </tr>
  <tr>
    <td>fi-el</td>
    <td>32</td>
    <td>-</td>
    <td>791</td>
  </tr>
  <tr>
    <td>fi-he</td>
    <td>32</td>
    <td>-</td>
    <td>200</td>
  </tr>
  <tr>
    <th colspan="4">zero shot</th>
  </tr>
  <tr>
    <td>en-sv</td>
    <td>-</td>
    <td>-</td>
    <td>192</td>
  </tr>
  <tr>
    <td>fi-el</td>
    <td>-</td>
    <td>-</td>
    <td>791</td>
  </tr>
  <tr>
    <td>fi-he</td>
    <td>-</td>
    <td>-</td>
    <td>200</td>
  </tr>
</table>

Table 10: Number of training, validation and test examples in different supervision settings.

A.2 Details of Parallel Data

For Japanese-English, we use a subset of the KFTT translation data (Neubig, 2011). Specifically, we use a subset of the cleaned and tokenized version of the training set of the translation data. The original training set contains about 330K sentences. For the tokenized English sentences in the training set, we further preprocessed the text by lowercasing all English tokens.

For Romanian-English, we use the training parallel corpus7 from Vilar et al. (2006). The parallel corpus comes from three sources, Orwell’s 1984, the Romanian Constitution and parallel texts collected from the Web. We remove empty lines and use a subset of the parallel sentences.

For the German-English pair, we use a subset of News Commentary v18.1 (Kocmi et al., 2023) as the unlabeled data. The only preprocessing step before using these parallel sentences is the tokenization with Moses tokenizer 8 (Koehn et al., 2007).

For Chinese-English pair, we use the News Commentary v16. For preprocessing, we use Moses tokenizer to tokenize the English sentences and utilize the Jieba tool9 (accurate mode) to tokenize the Chinese sentences in the parallel sentences and remove empty lines.

For English-Swedish, Finnish-Greek and Finnish-Hebrew pairs, we utilize subsets of CCMatrix v1.0 (Schwenk et al., 2021; Fan et al., 2021; Tiedemann, 2012). We use spacy to do the word segmentation for these language pairs.

The max number of unlabeled data that we use is shown in Table 11.

<table>
  <tr>
    <th>Lang</th>
    <th>Max Num</th>
  </tr>
  <tr>
    <td>ja-en</td>
    <td>52,240</td>
  </tr>
  <tr>
    <td>de-en</td>
    <td>30,000</td>
  </tr>
  <tr>
    <td>ro-en</td>
    <td>30,000</td>
  </tr>
  <tr>
    <td>zh-en</td>
    <td>36,000</td>
  </tr>
  <tr>
    <td>en-sv</td>
    <td>30,000</td>
  </tr>
  <tr>
    <td>fi-el</td>
    <td>30,000</td>
  </tr>
  <tr>
    <td>fi-he</td>
    <td>30,000</td>
  </tr>
</table>

Table 11: Size of parallel data.

A.3 Training Details

For all supervised fine-tuning experiments, we used the PyTorch(2.4.0) (Paszke et al., 2019), transformers (4.45.2) (Wolf et al., 2020) and PEFT (0.12.0) libraries for loading model weights and training. We utilized vLLM10 (version 0.6.2) (Kwon et al., 2023) to speed up inference and evaluation. Most experiments are conducted on a server with four NVIDIA RTX 6000 Ada GPUs. Our largest experiment takes approximately 60 hours of training time on a single NVIDIA RTX 6000 Ada GPU. For LLM-based word aligners, we mainly utilized LoRA (Hu et al., 2022) for parameter-efficient fine-tuning.

A.4 LoRA training details

We use LoRA for decoder-based language models. The rank parameter is set to 64. The alpha parameter for LoRA scaling is set to 256. The dropout probability for LoRA layers is set to 0.05. The target modules for LoRA layers contains “gate_proj”, “down_proj”, “down_proj”, “up_proj”, “q_proj”, “k_proj”, “v_proj”.

7http://web.eecs.umich.edu/~mihalcea/wpt/data.protected/Romanian–English.training.tar.gz
8https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl
9https://github.com/fxsjy/jieba
10https://github.com/vllm-project/vllm
“v_proj”, “k_proj” and “o_proj” for Llama and Aya series models.

A.5 Evaluation Metric Details

Given a set of predicted alignment (\( \mathcal{A} \)), a set of sure alignment (\( S \)) and a set of possible alignment (\( P \)). Precision, Recall, F1 and AER are computed as follows:

\[
Precision(\mathcal{A}, P) = \frac{|\mathcal{A} \cap P|}{|\mathcal{A}|}
\] (2)

\[
Recall(\mathcal{A}, S) = \frac{|\mathcal{A} \cap S|}{|S|}
\] (3)

\[
F_1 = \frac{2 \cdot Precision \cdot Recall}{Precision + Recall}
\] (4)

\[
AER(\mathcal{A}, S, P) = 1 - \frac{|\mathcal{A} \cap S| + |\mathcal{A} \cap P|}{|\mathcal{A}| + |S|}
\] (5)

A.6 Prompt Format and Example

Figure 6 is the prompt format of “marker mode”. The model response contains a JSON output. The above panel shows the situation where the model outputs a JSON output when the marked source word has valid alignments. The below panel presents how the model deals with “no alignment” situation. Figure 4 shows a prompt example of “full mode”. Figure 5 shows a prompt example of “marker mode”.

B Ablation Studies

Multi-language Training. We discuss whether combining the training data of all language pairs could further improve the word alignment performance of SemiAlign-D.

We mainly discuss two settings:

• Whether the combination of the labeled data from the four language pairs improves the performance of LLM-based word aligners or not.

• Whether combining labeled and unlabeled data from all language pairs based on the size of labeled data per each language pair improves the performance or not.

Table 12 shows the AER scores of single language pair (single-lang) and all language pair (multi-lang) training. We find that the results of using all training data are better if we only use the labeled data. However, if we add pseudo-labeled data to our training data, the results of separately training SemiAlign-D are better in most cases except the Chinese-English pair.

<table>
  <tr>
    <th>SemiAlign-D</th>
    <th>Ja-En</th>
    <th>De-En</th>
    <th>Ro-En</th>
    <th>Zh-En</th>
  </tr>
  <tr>
    <th colspan="5">labeled only</th>
  </tr>
  <tr>
    <td>single-lang</td>
    <td>16.16</td>
    <td>11.93</td>
    <td>14.66</td>
    <td>5.73</td>
  </tr>
  <tr>
    <td>multi-lang</td>
    <td><b>14.47</b></td>
    <td><b>9.21</b></td>
    <td><b>10.61</b></td>
    <td><b>4.54</b></td>
  </tr>
  <tr>
    <th colspan="5">labeled + unlabeled</th>
  </tr>
  <tr>
    <td>single-lang</td>
    <td><b>11.64</b></td>
    <td><b>7.27</b></td>
    <td><b>7.98</b></td>
    <td>4.36</td>
  </tr>
  <tr>
    <td>multi-lang</td>
    <td>11.92</td>
    <td>7.98</td>
    <td>8.72</td>
    <td><b>4.35</b></td>
  </tr>
</table>

Table 12: AER results of single language training and multi-language training. Lower AER scores indicate better performance.

Number of Iterations. In this section, we discuss the effect of increasing iterations for self-training of SemiAlign-E. Figure 7 shows the AER scores of the first iteration and second iteration on the Japanese-English word alignment dataset. We observe that the second iteration of training provides limited improvement in word alignment performance, possibly due to the presence of noisy pseudo-labeled data.

Effect of Using BinaryAlign-predicted Pseudo-labeled Data and Using SemiAlign-E-predicted labels. Table 13 shows the results of SemiAlign-D on the Japanese-English dataset when using BinaryAlign-predicted labels and SemiAlign-E-predicted labels. We present the best results in their settings. We find that using SemiAlign-E-predicted labels is slightly better than using BinaryAlign-predicted labels.

<table>
  <tr>
    <th></th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>AER</th>
  </tr>
  <tr>
    <td>BinaryAlign</td>
    <td>88.53</td>
    <td><b>87.99</b></td>
    <td>88.26</td>
    <td>11.74</td>
  </tr>
  <tr>
    <td>SemiAlign-E</td>
    <td><b>88.09</b></td>
    <td>88.63</td>
    <td><b>88.36</b></td>
    <td><b>11.64</b></td>
  </tr>
</table>

Table 13: Results of SemiAlign-D using different pseudo-labeled data. BinaryAlign means the results of using BinaryAlign-predicted labels. SemiAlign-E means the results of using SemiAlign-E-predicted labels

Zero-shot Cross-lingual Transfer Between High-resource Languages. In this section, we evaluate the zero-shot cross-lingual transfer performance of multilingual large language models as word aligners between high-resource languages. We use the SemiAlign-D model that is trained on one language pair, for example, Japanese-English or Chinese-English pair. Then we evaluate the model on the Romanian-English word alignment dataset directly.
Prompt Example for LLM-based Word Alignment (Type: Full Mode)

Role: system
Content: You are a helpful AI assistant for word alignment.

Role: user
Content: Please give me the word alignment information between the following two sentences. The parallel sentences:
足利_0 義満_1（2 あしかが_3 よしみつ_4）_5（は_6 室町_7 幕府_8 の_9 第_10 3_11 代_12 征夷_13 大_14 将軍_15（16 在位_17 1368_18 年_19 -20 1394_21 年_22）_23で_24 あ_25 る_26。_27

yoshimitsu_0 ashikaga_1 was_2 the_3 3rd_4 sei_5 taishogun_6 of_7 the_8 muromachi_9 shogunate_10 and_11 reigned_12 from_13 1368_14 to1394_15_16

Role: assistant
Content: Here is the word alignment information:
```json
[{"足利_0": ["ashikaga_1"], {"義満_1": ["yoshimitsu_0"]}, {"あしかが_3": ["ashikaga_1"]}, {"よしみつ_4": ["yoshimitsu_0"]}, {"室町_7": ["muromachi_9"]}, {"幕府_8": ["shogunate_10"]}, {"の_9": ["of_7"]}, {"第_10": ["the_3"]}, {"3_11": ["3rd_4"]}, {"代_12": ["3rd_4"]}, {"征夷_13": ["sei_5"]}, {"大_14": ["taishogun_6"]}, {"将軍_15": ["taishogun_6"]}, {"在位_17": ["reigned_12"]}, {"1368_18": ["1368_14"]}, {"年_19": ["1368_14"]}, {"1394_21": ["to1394_15"]}, {"年_22": ["to1394_15"]}, {"で_23": ["was_2"]}, {"あ_25": ["was_2"]}, {"る_26": ["was_2"]}, {。_27: ["_16"]}]
```

Figure 4: Prompt example of the “full mode”.

Through this setting, we want to know the potential zero-shot cross-lingual transfer performance if our LLM-based word aligner SemiAlign-D is used on an unseen language pair.

Specifically, we trained the SemiAlign-D on the combination of the German-English labeled and pseudo-labeled data. Then we test its performance on Romanian-English word alignment test set.

The results are shown in Table 14. These results demonstrate that the pseudo-labeled data helps zero-shot cross-lingual transfer. We notice that the F1 and AER scores when we train models on De-En and test on Ro-En are larger than the ones of training models on Ja-En and testing on Ro-En. The reason might be that German is more similar to Romanian compared to Japanese in some linguistic aspects.

<table>
  <tr>
    <th>SemiAlign-D</th>
    <th>Ro-En (F1)</th>
    <th>Ro-En (AER)</th>
  </tr>
  <tr>
    <td>De-En (only labeled)</td>
    <td>82.45</td>
    <td>17.55</td>
  </tr>
  <tr>
    <td>De-En (labeled + unlabeled)</td>
    <td><b>84.35</b></td>
    <td><b>15.65</b></td>
  </tr>
  <tr>
    <td>Ja-En (only labeled)</td>
    <td>79.24</td>
    <td>20.76</td>
  </tr>
  <tr>
    <td>Ja-En (labeled + unlabeled)</td>
    <td><b>80.20</b></td>
    <td><b>19.80</b></td>
  </tr>
</table>

Table 14: Result of zero-shot cross-lingual transfer on SemiAlign-D. The row name denotes the training language pair. The column name shows the test language pair. Higher F1 and lower AER scores indicate better performance.

Effect of Possible Alignment. Some word alignment datasets have possible alignments. For example, German-English word alignment dataset has possible alignments. In this section, we investigate the effect of training with possible alignment and without it using multilingual large language models.

Table 15 shows the results on the German-English dataset when using the labeled data with possible alignments and the labeled data without possible alignments. We find that the results without using possible alignment are better in F1 and AER scores.

<table>
  <tr>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>AER</th>
  </tr>
  <tr>
    <td colspan="4">w/ possible alignment</td>
  </tr>
  <tr>
    <td>93.63</td>
    <td>90.75</td>
    <td>92.17</td>
    <td>7.80</td>
  </tr>
  <tr>
    <td colspan="4">w/o possible alignment</td>
  </tr>
  <tr>
    <td>93.97</td>
    <td>90.70</td>
    <td>92.31</td>
    <td>7.66</td>
  </tr>
</table>

Table 15: Results of SemiAlign-D trained with the labeled data and pseudo-labeled data on German-English language pair. "w/ possible alignment" means the results of using possible alignments in the labeled data. "w/o possible alignment" means the results of not using possible alignment in the labeled data.

Effect of Using LoRA, QLoRA and Full Parameter Fine-tuning. Table 16 shows the results of using LoRA, QLoRA and Full parameter fine-tuning on the test set of Japanese-English word
Prompt Example for LLM-based Word Alignment (Type: Marker Mode)

Role: system
Content: You are a helpful AI assistant for word alignment.

Role: user
Content: Please give me the word alignment information between the following two sentences given the marked word with ¶. The parallel sentences:
¶足利_0 ¶義満_1（2 あしかが_3 よしみつ_4）_5は_6 室町_7 幕府_8の_9 第_10 3_11 代_12 征夷_13 大_14 将軍_15（16 在位_17 1368_18 年_19 -_20 1394_21 年_22）_23で_24 あ_25 る_26。_27

yoshimitsu_0 ashikaga_1 was_2 the_3 3rd_4 seii_5 taishogun_6 of_7 the_8 muromachi_9 shogunate_10 and_11 reigned_12 from_13 1368_14 to1394_15_16

Role: assistant
Content: Here is the word alignment information:

```json
{"足利_0": ["ashikaga_1"]}
```

Figure 5: Prompt example of the “marker mode” for supervised fine-tuning of LLM-based word alignment models. The input is the marked source sentence and target sentence.

alignment dataset, KFTT. We find that the LoRA verison achieves the best performance.
Prompt Format for LLM-based Word Alignment (Type: Marker Mode)

Role: system
Content: You are a helpful AI assistant for word alignment.

Role: user
Content: Please give me the word alignment information between the following two sentences given the marked word with {context_sep}. The parallel sentences: {marked_sent_src}\n\n{sent_tgt_with_id}

Role: assistant
Content: Here is the word alignment information:\n\n```json\n{label_str}\n```.

Prompt Format for LLM-based Word Alignment (Type: Marker Mode)

Role: system
Content: You are a helpful AI assistant for word alignment.

Role: user
Content: Please give me the word alignment information between the following two sentences given the marked word with {context_sep}. The parallel sentences: {marked_sent_src}\n\n{sent_tgt_with_id}

Role: assistant
Content: There is no word alignment information for the marked word.

Figure 6: Prompt format of “marker mode” for supervised fine-tuning of LLM-based word alignment models. The input is the marked source sentence and target sentence.

<table>
  <tr>
    <th>Method</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
    <th>AER</th>
  </tr>
  <tr>
    <td>SemiAlign-D (LoRA)</td>
    <td>88.09</td>
    <td>88.63</td>
    <td>88.36</td>
    <td>11.64</td>
  </tr>
  <tr>
    <td>SemiAlign-D (QLoRA)</td>
    <td>87.82</td>
    <td>88.48</td>
    <td>88.15</td>
    <td>11.85</td>
  </tr>
  <tr>
    <td>SemiAlign-D (Full)</td>
    <td>89.12</td>
    <td>87.18</td>
    <td>88.14</td>
    <td>11.86</td>
  </tr>
</table>

Table 16: Results of using LoRA, QLoRA and “full parameter fine-tuning” on the test set of Japanese-English word alignment dataset, KFTT.

![Line graph comparing F1 scores of SemiAlign-E after first and second iteration training, and BinaryAlign](page_324_1042_377_246.png)

Figure 7: Comparison between the F1 scores of the SemiAlign-E after the first iteration and second iteration training on the Japanese-English dataset. Higher F1 scores mean better performance.

Reviewer paper summaries: This paper introduces a novel semi-supervised approach to enhancing word alignment models. The study explores the limitations of the current state-of-the-art model, BinaryAlign, and proposes SemiAlign, a semi-supervised framework that leverages both labeled and pseudo-labeled data. Concretely, the study investigates the effectiveness of semi-supervised learning for multilingual word alignment as well as the use of Large Language Models (LLMs) as word aligners.

Through experiments on multiple language pairs (e.g., Japanese-English, German-English, Romanian-English, and Chinese-English), the study demonstrates that SemiAlign-E (encoder-based) and SemiAlign-D (decoder-based LLMs) outperform existing models in various settings. The study also analyzes computational efficiency and discusses the impact of confirmation bias in semi-supervised learning.
Summary of strengths: The proposed semi-supervised learning approach consistently improves word alignment performance over BinaryAlign, especially in low-resource settings.

Extensive experimentation on multiple language pairs, including zero-shot and few-shot settings, strengthens the credibility of the findings. Compared to the last submission, this revision add the experiments on out-of-domain target languages, further improve the effectiveness of the proposed method.
Summary of weaknesses: 1. The SemiAlign-D model, which uses large language models (LLMs) as word aligners, requires significantly more computational resources than its encoder-based counterpart, SemiAlign-E. The paper’s efficiency analysis shows that SemiAlign-D takes several hours to process test sets, whereas SemiAlign-E completes the same tasks in seconds. This makes SemiAlign-D impractical for real-time or large-scale applications where low latency is essential. Additionally, the paper does not explore optimization techniques to mitigate the high inference time, such as model compression, distillation, or more efficient decoding strategies.

2. Although the paper evaluates its approach on some low-resource language pairs, these languages are primarily from European language families and do not include truly low-resource or typologically diverse languages. Many underrepresented languages, such as African or Indigenous languages, lack large-scale parallel corpora, making word alignment even more challenging. The absence of evaluation on languages with highly different morphological and syntactic structures, such as agglutinative or polysynthetic languages, limits the generalizability of the proposed method beyond Indo-European languages.

3. The effectiveness of the proposed approach heavily relies on multilingual pre-trained models like mDeBERTa-v3 and LLaMA 3.1, which do not cover all languages equally well. Since these models are trained on unbalanced datasets, their performance varies across languages, with high-resource languages benefiting the most. The paper does not investigate how its approach would perform on languages that lack strong pre-trained embeddings or whether alternative strategies, such as language-specific fine-tuning, could help mitigate this issue. This reliance on existing multilingual models may reduce the method’s applicability to truly low-resource languages that lack sufficient pre-training data.

4. The study introduces pseudo-labeling to improve word alignment but does not thoroughly analyze its impact or potential drawbacks. It assumes that adding pseudo-labeled data enhances performance without investigating factors such as error propagation, confidence thresholds, or the trade-offs between different weak-supervision techniques. Additionally, the method only applies a single iteration of self-training, leaving open the question of whether multiple iterations would improve or degrade performance. A more detailed exploration of pseudo-labeling effectiveness would strengthen the paper’s findings and provide clearer guidelines for its application.
Comments, suggestions and typos: BinaryAlgin (zh-en)" in Table 2, ‘en-sv’ 4th line: "BinaryAlgin" → "BinaryAlign"
Overall assessment scores: 3
Metareview score: 3
Metareview: This paper proposes a semi-supervised approach SemiAlign to improve word alignment based on the current existing state-of-the-art encoder-based method BinaryAlign. The variants of SemiAlign, SemiAlign-E (encoder-based) achieve state-of-the-art results, and SemiAlign-D (decoder-based LLMs) investigates the potential of LLMs in word alignment and obtain comparable results to SemiAlign-E.
Summary of reasons to publish: The proposed SemiAlign framework delivers new state-of-the-art results on benchmark datasets with extensive experiments.