BinaryAlign: Word Alignment as Binary Sequence Labeling (2024)

Abstract: Real world deployments of word alignment are almost certain to cover both high and low resource languages. However, the state-ofthe-art for this task recommends a different model class depending on the availability of gold alignment training data for a particular language pair. We propose BinaryAlign, a novel word alignment technique based on binary sequence labeling that outperforms existing approaches in both scenarios, offering a unifying approach to the task. Additionally, we vary the specific choice of multilingual foundation model, perform stratified error analysis over alignment error type, and explore the performance of BinaryAlign on non-English language pairs. We make our source code publicly available.1

heading: 1 Introduction
text: Word alignment refers to the task of uncovering word correspondences between translated text pairs. The automatic prediction of word alignments dates back to the earliest work in machine translation with the IBM models (Brown et al., 1993) where they were used as hidden variables that permit the use of direct token to token translation probabilities. While state of the art machine translation techniques have largely abandoned the use of word alignment as an explicit task (Li, 2022) other use cases for alignments have emerged including lexical constraint incorporation (Chen et al., 2021b), analysing and evaluating translation models (Bau et al., 2018; Neubig et al., 2019), and cross-lingual language pre-training (Chi et al., 2021b).
In many real-world applications word alignment must be performed across several languages, often including languages with manually annotated word alignment data and others lacking such annotations. We refer to those languages as high
1https://github.com/ubisoft/ubisoft-laforgeBinaryAlignWordAlignementasBinarySequenceLabeling
and low-resource languages respectively. While word alignment for high-resource languages can be learned in a few-shot or fully supervised setting depending on the amount of data, for low-resource languages zero-shot learning strategies must be employed due to data scarcity.
State-of-the-art supervised techniques formalize the task of word alignment as a collection of SQuAD-style span prediction problems (Nagata et al., 2020; Wu et al., 2023) while in zero-shot settings the best performing methods induce word alignment from the contextualized word embeddings of mulitingual pre-trained language models (mPLMs) (Jalili Sabet et al., 2020; Dou and Neubig, 2021; Wang et al., 2022). From a practical perspective, this discrepancy in the preferred method adds complexity to the deployment of word alignment models in real-world applications where both high and low-resource languages must be supported.
We observe a deeper issue that both span prediction and contextualized word embeddings are sub-optimal as each induces a bias in word alignment models that limits their accuracy. Span prediction methods cannot robustly deal with discontinuous word alignments without relying on complex post-processing and hyper-parameter tuning. Contextualized word embeddings method cannot deal effectively with untranslated words and one-to-
10277
multiple alignments because they rely on a softmax function that normalizes predictions at a sentencelevel while in word alignment; one token being aligned to token T does not mean that another token is less likely to be aligned to T . This poses word alignment as a single-label classification problem, while in reality it is better viewed as a series of binary classifications applied to each possible pair of words. Figure 1 shows some cases of oneto-multiple alignments, non-contiguous spans and untranslated words.
In this paper, we present BinaryAlign, a novel word alignment solution that outperforms the state-of-the-art in zero-shot, few-shot and fullysupervised settings. In particular, we reformulate word alignment as a set of binary classification tasks in which an individual alignment prediction is made for each possible pair of words. This reformulation of the task outperforms all previous approaches over five different language pairs with varying levels of supervision.

heading: 2 Related Work
text: Recently, mPLM based approaches have significantly outperformed bilingual statistical methods (Och and Ney, 2003; Dyer et al., 2013; Östling and Tiedemann, 2016) and bilingual neural methods (Garg et al., 2019; Zenkel et al., 2020; Chen et al., 2020, 2021a; Zhang and van Genabith, 2021). Among those approaches, we distinguish methods that achieve good performance without relying
on manually annotated word alignment datasets (Jalili Sabet et al., 2020; Dou and Neubig, 2021; Wang et al., 2022) from supervised methods that leverage existing word alignment datasets to train high performing word aligners (Nagata et al., 2020; Wu et al., 2023).
The first type of method relies on the approach of SimAlign (Jalili Sabet et al., 2020) which proposes to induce word alignment from the contextualized word embeddings of mPLMs pre-trained on nonparallel data. AwesomeAlign (Dou and Neubig, 2021) builds on top of this approach and proposes to fine-tune mPLMs on parallel text with different objectives to improve the quality of the contextualized word embeddings. More recently, AccAlign (Wang et al., 2022) showed that models trained to learn language-agnostic sentence-level embeddings also learn strong language-agnostic word-level embeddings and set the state of the art in the zeroshot setting. Also, Wang et al. (2022) show that fine-tuning on existing word alignment datasets improves performance of AccAlign on language pairs unseen during word alignment fine-tuning. Our method is different from this body of work because our training and inference objective differ. We formalize word alignment as a binary sequence labeling task while we can see those methods as framing word alignment as a token retrieval task.
In terms of approaches trained and evaluated in a supervised setting SpanAlign (Nagata et al., 2020) formalizes word alignment as a collection of
SQuAD-style span prediction problems which differs from our binary classification objective. However, this method falls short in zero-shot and fewshot when word alignment training data is scarce. To remedy this problem, WSPAlign (Wu et al., 2023) introduces a pre-training method based on weak supervision that significantly improves performance for all amounts of training data.

heading: 3 Method
text: Given a sentence X with n words and a translation into another language Y with m words, the task of word alignment is to produce an n by m adjacency matrix for the bipartite graph with the words of X on one side and the words of Y on the other (refer to Figure 1 for an illustration). As our model will employ commonly used subword tokenization preprocessing we assume access to an invertible tokenizer, often implemented in practice with a list of subword units, a greedy algorithm for subword chunking, and leading symbols to denote word continuation in the vocab file (Sennrich et al., 2016; Wu et al., 2016).
We present BinaryAlign, a novel word alignment approach using a binary sequence labeling model, shown in Figure 2. The inputs to this model are a subword tokenized source sentence X = x1, x2, ..., x|X|, a subword tokenized target sentence Y = y1, y2, ..., y|Y |, and a reference word wX = x[i : j] which is a subspan of X. We model the distribution of a binary alignment vector A of size |Y | in which each entry ak indicates if the word in Y that contains yk is aligned to wX .
We first preprocess X by surrounding wX with unique separator tokens and then cross-encode the source and target sentences with an mPLM. For each token yk in the target sentence we pass its final encoded representation through a linear layer to produce a single logit zk. We model ak with a logistic function using zk as its parameter:
p(ak = 1|wX) = 1
1 + e−zk (1)
A supervised signal for token level alignments ak is easily divined from word alignment data and so this form is sufficient to estimate the parameters of the model. However, our true inference-time goal of word to word alignment requires the use of additional heuristics. To motivate these heuristics we formalize W as the inverse of the surjective mapping between the token indices in Y and its
corresponding words; for any subspan wY of Y , W(wY ) returns the token indices in Y that compose wY .
Given an aggregation function agg, we define the probability of the event a′ that there exists an alignment between word wX in X and word wY in Y as
p(a′) = agg ∀k∈W(wY ) p(ak = 1|wX) (2)
Preliminary experiments suggest that the maximum aggregation strategy yields slightly superior performance compared to the mean and minimum aggregation strategies; hence, we adopt it for all subsequent experiments.
Word alignment in its general form is a symmetric problem in that we would expect the same answer if the source and target were swapped. However, like most leading word alignment methods, our method is asymmetric; the source and target sentences are handled differently. This deficiency is empirically detrimental to performance with the common remedy being to perform alignment in both directions and then to merge the two predictions in some manner.
We use the following symmetrization technique: letting pX→Y denote the use of X as the source and Y as the target sentence, we average pX→Y (a′) and pY→X(a′) and apply a threshold decision rule to make our final inference prediction. While outside the scope of this study, we note that various other options exist and have been explored in previous work such as bidirectional average (Nagata et al., 2020) or intersection, union and grow-diag-final: the default symmetrization heuristics supported in Moses (Koehn et al., 2007).

heading: 4 Experiments

heading: 4.1 Datasets
text: We use seven datasets of manually annotated word alignment data for our main experiments: French-English (fr-en), Chinese-English (zh-en), Romanian-English (ro-en), Japanese-English (jaen), German-English (de-en), Swedish-English (sven) and ALIGN6. The ja-en data comes from the KFTT word alignment data (Neubig, 2011), while the ro-en and fr-en data are taken from Mihalcea and Pedersen (2003) and the de-en data is provided by Vilar et al. (2006). Also, the zh-en data is
obtained from the TsinghuaAligner website2 and the sv-en dataset3 from Holmqvist and Ahrenberg (2011). Finally, ALIGN6 (Wang et al., 2022) is the combination of six different word alignment datasets featuring Dutch-English (Macken, 2010), Czech-English (Mareček, 2011), Hindi-English (Aswani and Gaizauskas, 2005), Turkish-English (Cakmak et al., 2012), Spanish-English (Graca et al., 2008) and Portuguese-English (Graca et al., 2008).
In addition, we use the Finnish-Greek (fi-el) and the Finnish-Hebrew (fi-he) word alignment test dataset from Imani et al. (2021) to experiment on non-English language pairs. Note that all datasets are the same ones used in Wang et al. (2022).

heading: 4.2 Experimental setup
text: Unseen alignment experiments: In our unseen alignment experiments, models are not fine-tuned on manual word alignment data of the tested language pair. This replicates a common real-world situation in which alignment data set is not available for a language pair and models must leverage knowledge gleaned from other language pairs and pre-training. This setting is usually referred as zeroshot cross-lingual transfer (Conneau et al., 2020; Chi et al., 2021a). We follow Wang et al. (2022) and fine-tune our model on ALIGN6 and use sv-en as our validation set. We evaluate our method following the experimental protocol of previous work
2http://nlp.csai.tsinghua.edu.cn/~ly/ systems/TsinghuaAligner/TsinghuaAligner. html
3https://www.ida.liu.se/divisions/hcs/ nlplab/resources/ges/
(Dou and Neubig, 2021; Wang et al., 2022) and use de-en, ro-en, fr-en, zh-en and ja-en for testing. Note that those language pairs are not included in ALIGN6. Few-shot and fully supervised experiments: We follow the protocol of Wu et al. (2023) for our fewshot and fully supervised experiments on de-en, roen, fr-en and ja-en. For ja-en, we train on all eight dev set files, we use four test set files for testing, and the remaining three test files for validation. We separate the de-en, ro-en and fr-en data into training and test sets. We fine-tune using 300 sentences for fr-en and de-en, while we use 150 sentences for ro-en. All remaining sentences are used for testing. Note that we made the same splits4 as Wu et al. (2023). For the zh-en data, we leverage the datasets provided in v1 of the TsinghuaAligner website. We use their dev set and test set as our training and test set respectively. Both contain 450 sentences.
In our non-English experiments, we evaluate on Finnish to Greek (fi-el) and Finnish to Hebrew (fihe) data. For fi-el we use 400 samples for training and test on the 391 remaining samples, and for fi-he, we have 1780 samples for training and 450 for test. We use 32 training samples for all our few-shot experiments.
A detailed account of the number of sentence pairs for each dataset and settings used in our experiments is available in appendix.

heading: 4.3 Baseline methods
text: Unseen alignment experiments: In this setting, we compare BinaryAlign to the three main bodies
4https://huggingface.co/datasets/qiyuw/wspalign_ft_data
of research that evaluate on unseen word alignment language pairs. The first one corresponds to the historical bilingual statistical methods. We report GIZA++ (Och and Ney, 2003), eflomal (Östling and Tiedemann, 2016) and fast-align (Dyer et al., 2013) which are the best-known statistical methods.
Bilingual neural methods represent the second body of work. For this, we report MASK-ALIGN (Chen et al., 2021a) and BTBA-FCBO-SST (Zhang and van Genabith, 2021), the two best performing bilingual neural methods.
Finally, we compare to three methods relying on contextualized word embeddings of mPLMs: SimAlign (Jalili Sabet et al., 2020), AwesomeAlign (Dou and Neubig, 2021) and AccAlign (Wang et al., 2022). Note that AccAlign leverages ALIGN6 and is the state-of-the-art method for unseen alignment.
We reimplemented AccAlign using their adapter method on ALIGN6 as done in there paper. For other baselines, we quote the results from Dou and Neubig (2021) for bilingual statistical methods, from Wang et al. (2022) for AwesomeAlign, SimAlign, and the bilingual neural methods. We also report the zero-shot performance of WSPAlign (Wu et al., 2023) that we computed using the checkpoints provided by the authors. Also, we train SpanAlign on ALIGN6, the same dataset that is used to train AccAlign and BinaryAlign in this setup. Comparing to this baseline allows us to determine if our formalization of word alignment can better leverage existing word alignment datasets than SQuAD-style span prediction techniques.
Few-shot and fully supervised experiments: We
compare to SpanAlign (Nagata et al., 2020) and WSPAlign (Wu et al., 2023), the two state of the art supervised word alignment techniques. Also, we further fine-tune AccAlign on our language specific training sets. Comparing to this baseline is important as it allows us to determine if our proposed method performs better than state of the art contextualized word embedding extraction techniques when we have access to manual word alignments of the same language pair. Also, SpanAlign and AccAlign have not been compared in previous studies.
We reimplemented WSPAlign and SpanAlign for all our few-shot experiments using the source code provided by WSPAlign authors5. We quote the results from Wu et al. (2023) for supervised fine-tuning of SpanAlign and WSPAlign on de-en, ro-en, ja-en, fr-en and reimplement them for zh-en as we use a different train and test dataset than the original papers.

heading: 4.4 Fine-tuning setups
text: We did not perform extensive hyper-parameter tuning of our methods. We arbitrarily use a learning rate of 2e−5, a batch size of 8 for fine-tuning and pre-training, and a threshold of 0.5 for inference. We train all our models for 5 epochs, except for few-shot learning without pre-training on ALIGN6 where we train for 25 epochs. Results would improve with hyper-parameter tuning on a large validation set. We use mdeberta-v3-base (He et al., 2021) as our mPLM. We discuss the choice of mPLM in section 4.6.2. In supervised set-
5https://github.com/qiyuw/WSPAlign
tings we report both BinaryAlign and BinaryAlignnoPre, our method with and without pre-training on ALIGN6. Note that pre-training our model on ALIGN6 is the default setting in BinaryAlign.

heading: 4.5 Word Alignment Evaluation Metric
text: Following previous works (Wang et al., 2022; Dou and Neubig, 2021), we evaluate performance using Alignment Error Rate (AER). Given a set of sure alignments (S), possible alignments (P) and predicted alignments (H) we can calculate AER as follows:
AER(S, P,H) = 1− |H ∩ S|+ |H ∩ P ||H|+ |S| .
Following the protocol in Wu et al. (2023) we use only sure alignments for training but we evaluate on both sure and possible alignments when the distinction is available.

heading: 4.6 Results and Discussion
text: We performed several experiments to validate BinaryAlign. First we compare our method to other state-of-the-art methods in three different levels of supervision: full available supervision, fewshot (32 samples), and unseen languages (zeroshot cross-lingual transfer). We also evaluate on non-English language pairs. Next, we evaluate the impact of choices for mPLM foundation and symmetrization. Finally, we discuss how our problem formulation compares to span prediction and contextualized word embedding based approaches in different situations.

heading: 4.6.1 Comparison to State-of-the-Art
text: Unseen alignments: As a first experiment, we apply all methods to new language pairs without performing word alignment fine-tuning on the tested
language pair as explained in 4.2. Table 1 reports the AER of all methods. BinaryAlign is the new state-of-the-art on all language pairs. In particular, it outperforms AccAlign by 3.0 points of AER on average. Since AccAlign and BinaryAlign share the same pre-training dataset (ALIGN6), this indicates that our word alignment problem formulation performs better than inducing word alignment from contextualized word embeddings. This is also true when comparing with SpanAlign pre-trained on the same data (ALIGN6). This indicates that our formalization of word alignment promotes learning more language-agnostic signals from word alignment datasets when compared to existing methods.
Full and few shot supervision: We compare BinaryAlign to the other baseline methods after finetuning on alignment data with few samples and the whole training data set. Table 2 shows the results for both supervision levels.
Our method achieves new state-of-the-art on all tested language pairs and with both levels of supervision. On average it outperforms WSPAlign, the previous state-of-the-art, by 2.1 points of AER with full supervision and 3.1 with few-shot supervision.
Even without pre-training on ALIGN6 our method outperforms all methods. Given that WSPAlign was pre-trained on 2 millions samples, it indicates that BinaryAlign promotes sample efficiency.
Finally, we highlight that by using only 32 samples for few-shot supervision BinaryAlign outperforms SpanAlign regardless of pre-training. This reinforces the performance improvement of formalizing word alignment as a binary token classification objective over span prediction.
Impact of pre-training on other languages: Table 2 reports the results of our method with and without pre-training on ALIGN6. We conclude that pre-training improves AER with few-shot as well as full supervision. However, we observe a smaller improvement with full supervision which suggests that the benefit of pre-training on other languages is inversely correlated with the amount of in-domain word alignment data. While pre-training encourages sample efficiency, we did not find any indication that it could hinder performance.
Non-English language pairs: Because usually mPLMs perform better in English it is important to investigate how our method performs on nonEnglish language pairs. Table 3 reports results on
language pairs excluding English. We used the checkpoint6 provided by the authors of WSPAlign since paragraph pairs in Finnish-Greek and Hebrew are difficult to obtain for training. Our results show that BinaryAlign outperforms WSPAlign and AccAlign for all degree of supervision. Also, the AER in non-English language pairs seems to be similar to the AER of our main experiments on English-centric language pairs which shows that our method does not depend on English and is robust to variations in language family.

heading: 4.6.2 Design Choices
text: mPLM Architecture: Our proposed reformulation of the word alignment problem does not depend on a particular mPLM architecture. In this experiment,
6https://huggingface.co/qiyuw/WSPAlign-xlm-base
we investigate the impact of using different mPLMs. We explore five different mPLMs: XLM-RoBERTa (base and large)(Conneau et al., 2020), LaBSE7 (Feng et al., 2022), mDeBERTa-v3-base8 (He et al., 2021) and mBERT9 (Devlin et al., 2019).
Table 4 reports AER of BinaryAlign using different mPLMs in few-shot and fully supervised settings. All these versions of BinaryAlign reach or surpass the previous state-of-the-art in terms of average AER on the five tested language pairs. This highlights that the improvement of our method over previous state-of-the-art is not explained by its reliance on a specific mPLM.
While most mPLMs yield similar results, mBERT performs slightly worse than the others. This could be due to a poor parametrization given that we used the same hyper-parameter configuration for all mPLMs. This could also be explained by the training objective of the mPLMs or their capacity. For example we observe that scaling the size of XLM-RoBERTa has an effect on alignment performance. The base model has an approximately similar capacity as the other mPLMs and when we increase this capacity using the large model we obtain the best result over all mPLMs. We suspect that this effect could generalize to other mPLM architectures.
Symmetrization: Here we investigate the impact of different symmetrization heuristics on our results. As stated in Section 3 symmetrization consists in fusing the alignment obtained going from one language to another with the alignment obtained going in the inverse direction. We com-
7https://huggingface.co/sentence-transformers/LaBSE 8https://huggingface.co/microsoft/mdeberta-v3-base 9https://huggingface.co/bert-base-multilingual-cased
pare several symmetrization techniques: intersection, union, average (avg) and bidirectional average (bidi-avg) (Nagata et al., 2020).
In Table 5 we compare results obtained from aligning in a single direction to the results obtained using the best symmetrization heuristics (full details available in the appendix). We report the F1 score (see A.3) as done in Nagata et al. (2020).
Our results indicate that for BinaryAlign, unidirectional alignment does not perform significantly worse (average of 0.3 points of F1 score) than symmetrized alignment. This is not the case for SpanAlign which gains 2.4 points of F1 score on average by applying symmetrization. Performing alignment in only one direction is interesting since it halves the inference time.

heading: 4.6.3 Post-analysis of errors
text: In this section we analyze how the proposed problem formulation of BinaryAlign improves accuracy in complex word alignment situations. We inspect results in three situations: 1) words that are untranslated, also referred as null words (Jalili Sabet et al., 2020) (2) words that are aligned to multiple words (3) words that are aligned to multiple non contiguous words. For each situation, we report the percentage of correctly aligned words in Table 6. Details on how we computed our metric can be found in A.3. Results indicate that our method handles these situations better than both competing methods. This is especially true when aligning
multiple non contiguous words which was the main motivation for our reformulation. The prevalence of these situations in a given language pair modulates the performance gain of our method over the others.

heading: 5 Conclusion
text: We presented BinaryAlign, a novel word alignment training and inference procedure. In particular, we proposed to reformulate the word alignment problem as a binary token classification task. We showed that because of this reformulation BinaryAlign outperforms existing methods regardless of the degree of supervision. In addition we showed that it overcomes the inherent limitations of previous methods relying on span prediction and softmax. As a result, we made the word alignment task easier to tackle by using a single model for both high and low-resource languages.
In the future we plan to explore the use of larger decoder-only or encoder-decoder models such as mT5 (Xue et al., 2021) to see how much alignment performance will increase. We also plan on investigating knowledge distillation techniques to improve the inference time of our method.

heading: 6 Limitations
text: The inference cost is the main limitation of our method. When using symmetrization, it has to perform a forward pass for each word of both sentences, which can be slow with long sequences.
However, this is a drawback that we share with previous state-of-the-art supervised approaches (Nagata et al., 2020; Wu et al., 2023).
In addition, we did not experiment on extremely low-resource languages that the mPLM has not seen during pre-training (Ebrahimi et al., 2023). While the benefits of our new formulation would likely apply to any language, it is unclear how our method will rapidly adapt the mPLM to new languages (Garcia et al., 2021).
In real-world applications, translations are often partial and noisy. Unfortunately, we could not evaluate the robustness of our method to different translation pair quality because this type of word alignment dataset does not exist.

heading: 7 Ethics Statement
text: The ethical and societal implications of word alignment and neural machine translation are generally positive. They facilitate cross-cultural communication and break down language barriers. Nonetheless, as for any machine learning field, potential biases embedded in training data can inadvertently influence translations and perpetuate stereotypes, especially when translating to and from gendered languages. Finally the impact of neural machine translation on employment for human translators raises questions about job displacement and economic inequalities. It is crucial for developers and stakeholders to prioritize fairness, transparency, and accountability in the design and implementation of such systems. We must balance technological advancement and ethical responsibility to ensure societal well-being, inclusive communication and minimize unintended consequences.

heading: A Appendix
text: A.1 Experimental Environment For all our experiments we use one NVIDIA Quadro RTX 6000. Fine-tuning on ALIGN6 took 4 hours and 30 minutes while our fully supervised experiments took on average 20 minutes per dataset.
A.2 Dataset statistics Table 8 shows the number of samples that our training, validation and test set contains for all level of supervision. All dataset are the same as in Wang et al. (2022). The de-en, ro-en, fr-en and ja-en traintest splits are the same as the one used in Wu et al. (2023). We could not get the zh-en data used in Wu et al. (2023) because the dataset is not publicly available.
A.3 Metric details A.3.1 F1 score Given a set of sure alignments (S), possible alignments (P) and predicted alignments (H), we can compute the Recall, Precision and F1 score as follows:
Recall(H,S) = |H ∩ S|
|S|
Precision(H,P ) = |H ∩ P | |H|
F1(H,S, P ) = 2 ∗ Precision ∗Recall Precision+Recall
When S == P , we have:
AER(H,S, P ) = 1− F1(H,S, P )
A.3.2 Post-analysis of errors Untranslated words: We report the number of untranslated words correctly aligned by the models over the total number of untranslated words. We consider a word to be correctly aligned if the model has not aligned it to any words in the corresponding translated sentence. One-to-multiple contiguous and non contiguous words: In this case, we report the number of contiguous/non contiguous words correctly aligned by the model over the total number of contiguous/non contiguous words. We consider a word to be correctly aligned if the model has aligned it to the exact same set of ground truth aligned words.

heading: Future Work
text: In the future we plan to explore the use of larger decoder-only or encoder-decoder models such as mT5 (Xue et al. , 2021) to see how much alignment performance will increase. We also plan on investigating knowledge distillation techniques to improve the inference time of our method