PMI-Align: Word Alignment With Point-Wise Mutual Information Without Requiring Parallel Training Data
Year: 2023

Abstract:
Word alignment has many applications including cross-lingual annotation projection, bilingual lexicon extraction, and the evaluation or analysis of translation outputs. Recent studies show that using contextualized embeddings from pre-trained multilingual language models could give us high quality word alignments without the need of parallel training data. In this work, we propose PMI-Align which computes and uses the point-wise mutual information between source and target tokens to extract word alignments, instead of the cosine similarity or dot product which is mostly used in recent approaches. Our experiments show that our proposed PMI-Align approach could outperform the rival methods on five out of six language pairs. Although our approach requires no parallel training data, we show that this method could also benefit the approaches using parallel data to fine-tune pre-trained language models on word alignments. Our code and data are publicly available1.

heading: 1 Introduction
text: Word alignment, as the task of finding the corresponding source and target tokens in a parallel sentence, was well-known as an essential component of statistical machine translation (SMT) systems. Despite the dominance of neural machine translation (NMT) in recent years, word alignment is still a notable area of research due to its usage in a wide variety of NLP applications, such as annotation projection (Yarowsky et al., 2001; Padó and Lapata, 2009; Huck et al., 2019; Nicolai and Yarowsky, 2019), bilingual lexicon extraction (Ammar et al., 2016; Shi et al., 2021; Artetxe et al., 2019), typological analysis (Lewis and Xia, 2008; Östling, 2015), guided alignment training of NMT (Liu et al., 2016; Chen et al., 2016; Alkhouli et al., 2018), and evaluation and analysis of translation
1https://github.com/fatemeh-azadi/PMI-Align
outputs (Anthony et al., 2019; Neubig et al., 2019; Wang et al., 2020).
For many years statistical methods such as IBM models (Brown et al., 1993) and tools implemented based on them, namely GIZA++ (Och and Ney, 2003) or fast-align (Dyer et al., 2013), were among the most popular solutions to the word alignment task. Following the rise of deep neural models, several attempts have been made to extract word alignments from NMT models and their attention matrices (Peter et al., 2017; Ghader and Monz, 2017; Zenkel et al., 2020; Zhang and van Genabith, 2021). However, most of these methods, as well as the statistical aligners, require a sufficient amount of parallel training data to produce high quality word alignments. Recently, Jalili Sabet et al. (2020) have shown that high quality word alignments could be achieved using pre-trained multilingual language models (LMs), like MBERT Devlin et al. (2019) and XLMR Conneau et al. (2020). Their proposed method called SimAlign, extracts word alignments from similarity matrices induced from multilingual contextualized word embeddings with no need for parallel training data, which is very useful for lowresource language pairs. Afterwards, Dou and Neubig (2021) and Chi et al. (2021) proposed methods
12366
called probability thresholding and optimal transport to extract alignments using the similarity matrices derived from pre-trained LMs. They have also proposed some word alignment objectives to fine-tune the pre-trained models over parallel corpora.
In this paper, we follow the work done by Jalili Sabet et al. (2020) to extract alignments from pre-trained LMs without requiring any parallel training data and propose PMI-Align. Our main contribution is proposing to compute the point-wise mutual information (PMI) between source and target tokens and using the PMI matrices instead of similarity matrices made of cosine similarities between the representation vectors of each source and target tokens, to align words. We argue that our proposed PMI-based method could align better as it considers the total alignment probability of each source or target token, as well as the joint alignment probabilities (equivalent to cosine similarities). This could alleviate the so-called hubness problem (Radovanovic et al., 2010) in high dimensional spaces, where some token’s representation is close to many others (see _went in Figure 1). We perform experiments on six different language pairs and show that our method could surpass other alignment methods on five of them. We also conduct our experiments on different pre-trained LMs to show that PMI-Align could be advantageous regardless of the pre-trained model used.

heading: 2 Proposed Method
text: In this section, we first discuss how we define and compute the PMI matrix for each sentence pair and then we describe our alignment extraction method using the PMI matrix.

heading: 2.1 Point-Wise Mutual Information
text: Point-wise mutual information (PMI) is a wellknown measure of association in information theory and NLP and it shows the probability of two events x and y occurring together, compared to what this probability would be if they were independent (Fano, 1961). It is computed as follows:
PMI(x,y) := log p(x,y)
p(x)p(y) (1)
In the context of word alignments, we define the PMI for a source and target token in a sentence pair as how more probable two tokens are to be aligned than if they are aligned randomly. Given a sentence
x =< x1, ...,xn > in the source language and its corresponding target sentence y =< y1, ...,ym >, the joint alignment probability of two tokens, xi and y j, could be computed as:
p(xi,yi) = esim(hxi ,hy j )
∑ i′, j′
e sim(hxi′ ,hy j′ )
, (2)
where hxi is the contextualized embedding vector of xi extracted from a pre-trained multilingual language model and sim(.) is the cosine similarity measure. The total alignment probability of xi and y j, i.e., p(xi) and p(y j), could also be computed according to the total probability rule as follows:
p(xi) = ∑ 1≤ j≤m p(xi,y j) (3)
By calculating the PMI for each source and target token in a parallel sentence, we obtain the PMI matrix for that sentence pair, that could be used to extract alignments instead of similarity matrix in SimAlign (Jalili Sabet et al., 2020). The advantage of using PMI to align words is that it also considers the total alignment probability of each source and target token in addition to their joint alignment probability, which is equivalent to the similarity measure. This leads to reduce the probability to align the token pairs that one of them has high similarities to many other tokens, and thus could alleviate the so-called hubness problem in high dimensional spaces where some data points called hubs are the nearest neighbors of many others.

heading: 2.2 Extracting Alignments
text: To extract word alignments, we follow the simple Argmax method proposed in Jalili Sabet et al. (2020). Thus, we first obtain the source to target and target to source alignment matrices using the argmax over each row and each column of the PMI matrix, respectively. Next, we intersect these two matrices to get the final word alignment matrix. In other words, the final alignment matrix Ai j = 1 iff i = argmaxk(PMIk j) and j = argmaxk(PMIik).
Since the above method would extract alignments on the subword level, we follow the heuristic used in previous work to obtain the word-level alignments by considering two words to be aligned if any of their subwords are aligned (Jalili Sabet et al., 2020; Zenkel et al., 2020; Dou and Neubig, 2021).

heading: 3 Experiments and Results

heading: 3.1 Datasets
text: We perform our experiments on six public datasets, as in (Jalili Sabet et al., 2020), consists of EnglishCzech (En-Cs), German-English (De-En), EnglishPersian (En-Fa), English-French (En-Fr), EnglishHindi (En-Hi) and Romanian-English (Ro-En) language pairs. The statistics and URLs of these datasets are available in Table 2 in Appendix A.

heading: 3.2 Models and baselines
text: We compare our method with the following three state-of-the-art methods proposed to extract alignments from pre-trained multilingual LMs without using parallel training data. For all these methods default parameters were used in our experiments.
SimAlign2 (Jalili Sabet et al., 2020): They propose three methods to extract alignments from similarity matrices, called Argmax, Itermax and Match. Although Itermax and Match methods could not make significant improvements over Argmax and the Argmax method had better AER results for most of language pairs while using the XLMR-base model, they have argued that the Itermax method, which tries to apply Argmax iteratively, could be beneficial for more distant language pairs. Thus, we report both Argmax and Itermax results in our experiments to compare with our method.
Probability Thresholding3 (Dou and Neubig, 2021): In this method they apply a normalization function, i.e., softmax, to convert the similarity matrix of tokens into source to target and target to source alignment probability matrices. Afterwards, they extract the aligned words as the words that their alignment probabilities in both matrices exceed a particular threshold.
Optimal Transport4 (Chi et al., 2021): This method was proposed in both Dou and Neubig (2021) and Chi et al. (2021), and tried to model the word alignment task as the known optimal transport problem (Cuturi, 2013). Using the similarity matrix, this method attempted to find the alignment probability matrix that maximizes the sentence pair similar-
2https://github.com/cisnlp/simalign 3https://github.com/neulab/awesome-align 4https://github.com/CZWin32768/XLM-Align
ity. In our experiments, we use the method proposed by Chi et al. (2021) that utilizes the regularized variant of the optimal transport problem (Peyré et al., 2019), as it reported better results.
There are also many attempts made to improve the pre-trained LMs by fine-tuning on some parallel corpora to better align words. However, as our approach is irrelevant to the pre-trained model and our focus is on the alignment extraction instead of the model, we do not include those methods in our experiments. To demonstrate the effectiveness of our PMI-based alignment regardless of the utilized pre-trained multilingual LM, we conduct our experiments on M-BERT (Devlin et al., 2019), XLMR-Base (Conneau et al., 2020) and XLM-Align (Chi et al., 2021) which is fine-tuned on a word-alignment task, to show that our method could also be advantageous on more cross-lingually aligned models. All these models are publicly available in the Hugging Face platform (Wolf et al., 2020).

heading: 3.3 Results
text: Table 1 shows the results of our alignment technique compared to previous methods while using different pre-trained LMs. Following the previous work (Jalili Sabet et al., 2020; Dou and Neubig, 2021; Chi et al., 2021), we use the 8th layer’s representations of each pre-trained model to compute the similarity or PMI matrices. We also use the alignment error rate (AER) (Och and Ney, 2003) as the evaluation metric.
As Table 1 shows, our PMI-Align method could consistently outperform the other methods in all language pairs except En-Fr, regardless of the pretrained model used. Compared to Argmax, our method performs better for about 1% or more in AER, while using the XLMR-Base model (except for En-Fr), which exclusively shows the benefits of using the PMI matrix instead of the similarity matrix. We also see that the PMI-Align could surpass the Itermax method for more distant language pairs such as En-Fa and En-Hi, where it was claimed to have the most advantage. Results show that our method could also be beneficial while using a model pre-trained on a word alignment task, i.e., XLM-align, which is expected to have more crosslingually aligned representations, and less hubness problem.
The only language pair that our method could
not outperform prior methods is En-Fr. This could be due to the closeness of these two languages, as they have many shared subwords and similar word orderings. As a result, pre-trained models for this language pair are better trained and could strongly produce similar representations for aligned words, which reduces the hubness problem to a great extent. Thus, using PMI instead of the similarity matrix could not help. However, our method’s performance while using the M-BERT model is comparable to the best results, with about 0.1% difference in AER. Several samples are shown in Appendix B, to better intuitively compare PMIAlign and Argmax, which could better show the benefits of using the PMI matrix instead of the cosine similarities.

heading: 4 Related Work
text: Statistical aligners based on IBM models (Brown et al., 1993), such as Giza++ (Och and Ney, 2003) and fast align (Dyer et al., 2013) were the most dominant tools for word alignment until the late 2010s. With the rise of neural machine translation models, several attempts made to extract alignments from them (Ghader and Monz, 2017; Garg et al., 2019; Li et al., 2019; Zenkel et al., 2020; Chen et al., 2021; Zhang and van Genabith, 2021). However, all these models need parallel training
data and could not utilize pre-trained contextualized embeddings. Recently, Jalili Sabet et al. (2020) have proposed methods to extract alignments from similarity matrices induced from multilingual LMs without the need for training on parallel data. Following this work, we propose a PMI measure to score and align words in each sentence pair, instead of cosine similarity. Some other alignment extraction methods using multilingual LMs were also provided by Dou and Neubig (2021) and Chi et al. (2021). They both also proposed several training objectives related to word alignments to fine-tune multilingual LMs on parallel data, as in some other recent works (Cao et al., 2020; Wu and Dredze, 2020; Lai et al., 2022).

heading: 5 Conclusions
text: This paper presents a word alignment extraction method based on the PMI matrices derived from cross-lingual contextualized embeddings, instead of just the similarity matrices. We proposed a way to compute the PMI matrix for each sentence pair and argued that using this PMI measure would be beneficial since for each source-target word pair, it considers not only their similarity to each other but also their similarity values to the other tokens of the sentence, that could mitigate the hubness problem.
Experimental results show that our PMI-Align method could outperform the previous alignment extraction methods in five out of six language pairs, regardless of the base pre-trained language model used to derive word embeddings. Although our method does not require any parallel training data, our experiments show that it could also benefit the approaches using such data to fine-tune the pretrained models for better word alignments. In future work, the proposed PMI matrix could be investigated in other cross-lingual or even monolingual applications, like the translation quality estimation or the evaluation of text generation tasks, instead of the similarity matrix.
Limitations
Although our proposed aligner has surpassed the existing LM-based alignment extraction methods in most of the datasets, it could not make any improvement for the En-Fr language pair, as shown in Table 1. This suggests that our proposed method might be only beneficial for more distant languages. On the other hand, for similar languages, it not only cannot add any information to the similarity matrix, but also its estimation for the alignment probabilities might add noise to the alignment extraction method. Thus, investigating ways to more effectively estimate the alignment probabilities of source and target tokens might be helpful in future work.
Another limitation of our method, as well as other LM-based aligners, is that they first extract subword-level alignments, and then heuristically map them to word-level. By observing the aligner outputs, we realize that many errors occur when the pre-trained LM can not efficiently split words into meaningful subwords. This happens more often for low-resource languages or far languages from English (like Persian or Hindi). Thus, achieving better subword tokenization in pre-trained LMs or applicable methods to convert subword-level representations into word-level could help improve the quality of LM-based aligners.

heading: A Data Statistics
text: Table 2 shows the number of sentences and the download links of the test datasets we used in our experiments.

heading: B Alignment Examples
text: Figures 2 and 3 illustrate some sentence pair examples comparing our PMI-Align method to SimAlign. They clearly show the advantages of using the PMI matrix over the similarity matrix. Both matrices are normalized with min-max normalization to be comparable.

heading: C Number of Parameters and Runtimes
text: We use 3 pre-trained models in our experiments:
MBERT (Devlin et al., 2019), which is pre-trained with masked language modeling (MLM) and next sentence prediction on Wikipedia of 104 languages.
XLMR-base (Conneau et al., 2020), pre-trained with MLM on large-scale CommonCrawl data for 100 languages.
XLM-align (Chi et al., 2021), pre-trained with translation language modeling (TLM) and denoising word alignment (DWA) for 14 English-centric language pairs, along with MLM for 94 languages.
Our method has no parameters itself. However, considering the parameters of the used pre-trained LM, MBERT has about 170 million parameters, while XLMR-base and XLM-align both have about 270 million parameters.
Since our word aligner is simple and efficient, we did all our experiments on an Intel(R) Core(TM) i7-6700 CPU with 32GB memory and it just took about 0.1 seconds on average to align each parallel sentence in our whole dataset, while using XLMRbase model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations section
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3

heading: 3 B1. Did you cite the creators of artifacts you used?

heading: 3, Appendix A
text: 7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data we used was a standard publicly available data used for the intended task in many prior works.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A, C
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.

heading: Appendix A
text: C 3 Did you run computational experiments?

heading: Appendix C

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Not applicable. Our method doesn’t have any hyperparameters

heading: C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Our results don’t vary in different runs.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? No response.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? No response.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

heading: No response.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: No response.

heading: Future Work
text: In future work, the proposed PMI matrix could be investigated in other cross-lingual or even monolingual applications, like the translation quality estimation or the evaluation of text generation tasks, instead of the similarity matrix. Limitations
Although our proposed aligner has surpassed the existing LM-based alignment extraction methods in most of the datasets, it could not make any improvement for the En-Fr language pair, as shown in Table 1. This suggests that our proposed method might be only beneficial for more distant languages. On the other hand, for similar languages, it not only cannot add any information to the similarity matrix, but also its estimation for the alignment probabilities might add noise to the alignment extraction method. Thus, investigating ways to more effectively estimate the alignment probabilities of source and target tokens might be helpful in future work. Another limitation of our method, as well as other LM-based aligners, is that they first extract subword-level alignments, and then heuristically map them to word-level. By observing the aligner outputs, we realize that many errors occur when the pre-trained LM can not efficiently split words into meaningful subwords. This happens more often for low-resource languages or far languages from English (like Persian or Hindi). Thus, achieving better subword tokenization in pre-trained LMs or applicable methods to convert subword-level representations into word-level could help improve the quality of LM-based aligners