Towards Accurate Translation via Semantically Appropriate Application of Lexical Constraints
Year: 2023

Abstract:
Lexically-constrained NMT (LNMT) aims to incorporate user-provided terminology into translations. Despite its practical advantages, existing work has not evaluated LNMT models under challenging real-world conditions. In this paper, we focus on two important but understudied issues that lie in the current evaluation process of LNMT studies. The model needs to cope with challenging lexical constraints that are “homographs” or “unseen” during training. To this end, we first design a homograph disambiguation module to differentiate the meanings of homographs. Moreover, we propose PLUMCOT, which integrates contextually rich information about unseen lexical constraints from pre-trained language models and strengthens a copy mechanism of the pointer network via direct supervision of a copying score. We also release HOLLY, an evaluation benchmark for assessing the ability of a model to cope with “homographic” and “unseen” lexical constraints. Experiments on HOLLY and the previous test setup show the effectiveness of our method. The effects of PLUMCOT are shown to be remarkable in “unseen” constraints. Our dataset is available at https://github. com/papago-lab/HOLLY-benchmark.

heading: 1 Introduction
text: Lexically-constrained neural machine translation (LNMT) is a task that aims to incorporate pre-specified words or phrases into translations (Hokamp and Liu, 2017; Dinu et al., 2019; Song et al., 2019; Susanto et al., 2020; Xu and Carpuat, 2021a; Chen et al., 2021a,b; Wang et al., 2022b, inter alia). It plays a crucial role in a variety of real-world applications where it is required to translate pre-defined source terms into accurate target terms, such as domain adaptation leveraging domain-specific or user-provided terminology. For example, as shown in Case A of Table 1, an LNMT
∗Equal Contribution
model successfully translates the source term (“코 로나”) into its corresponding target term (“Covid19”) by adhering to a given lexical constraint (“코 로나” → “Covid-19”).
Despite its practicality, previous studies on LNMT have not evaluated their performances under challenging real-world conditions. In this paper, we focus on two important but understudied issues that lie in the current evaluation process of the previous LNMT studies.
Semantics of lexical constraints must be considered. In previous work, at test time, lexical constraints are automatically identified from the source sentences by going through an automatic string-matching process (Dinu et al., 2019; Ailem et al., 2021; Chen et al., 2021b). For example, in Case B of Table 1, a source term (“코로나”) in the bilingual terminology is present as a substring in the source sentence. Accordingly, its corresponding target term (“Covid-19”) is automatically bound together as a lexical constraint (“코 로나”→ “Covid-19”) without considering the semantics of the matched source term,1 which can lead to a serious mistranslation. This automatic string-matching cannot differentiate textually identical yet semantically different source terms. Thus, the more accurately the LNMT reflects the lexical constraint, the more pronounced the severity of the homographic issue is. To address this homograph issue, LNMT systems must be equipped to understand the semantics of identified lexical constraints and determine whether or not these constraints should be imposed.
Unseen lexical constraints need to be examined. One desideratum of LNMT systems is their robustness to handle “unseen” lexical constraints, thereby responding to random, potentially neologistic, or technical terms that users might bring up. However,
1Here,코로나 in Case B indicates Corona, a brand of beer produced by a Mexican brewery.
4839
in previous studies, a significant portion of the lexical constraints is exposed during training. Wang et al. (2022b) demonstrated the overlapped ratio of lexical constraints between the training and evaluation data (35.6% on average). Meanwhile, Zeng et al. (2022) also raises the issue of the high frequency of lexical constraints for test sets appearing in the training data.
When lexical constraints are included in the training examples, we find that a well-optimized vanilla Transformer (Vaswani et al., 2017) already satisfies lexical constraints by merely learning the alignment between the source and target terms cooccurring in the parallel training sentences.2 This presents difficulties in identifying whether the presence of target terms in the output is attributed to the learned alignment, or the proposed components in previous studies. Therefore, it is important to control lexical constraints not exposed during training to examine the model’s ability to cope with “unseen” lexical constraints.
As a response, we present a test benchmark for evaluating the LNMT models under these two critical issues. Our benchmark is specifically crafted not only to evaluate the performance of LNMT models but also to assess its ability to discern whether given lexical constraints are semantically appropriate or not. To the best of our knowledge, we are the first to release a hand-curated highquality test benchmark for LNMT. Concurrently, we suggest a pipeline that allows researchers in LNMT communities to simulate realistic test conditions that consider the homograph issue and assign “unseen” lexical constraints.
To this end, we propose a two-stage framework to deal with these issues. We first develop a homograph disambiguation module that determines whether LNMT models should apply a given lexi-
2We observe that the vanilla Transformer achieves a 66.67% copy success rate.
cal constraint by evaluating its semantic appropriateness. Further, we propose an LNMT model that integrates provided lexical constraints more effectively by learning when and how to apply these lexical constraints. Our contributions are summarized as follows:
• We formulate the task of semantically appropriate application of lexical constraints and release a high-quality test benchmark to encourage LNMT researchers to consider realworld test conditions.
• We propose a novel homograph disambiguation module to detect semantically inappropriate lexical constraints.
• We present an LNMT model which shows the best translation quality and copy success rate in unseen lexical constraints.
2 HOLLY Benchmark
Here, we introduce HOLLY (homograph disambiguation evaluation for lexicallly constrained NMT), a novel benchmark for evaluating LNMT systems in two circumstances; either the assigned lexical constraints are semantically appropriate or not, as illustrated in Table 2. The entire test data includes 600 test examples on 150 Korean→ English lexical constraints.

heading: 2.1 Test Examples
text: Each test example consists of three main elements, as presented in Table 2: (1) a lexical constraint (양 수 → amniotic fluid), (2) a source sentence containing the source term (양수 ) of the lexical constraint, and (3) its reference translation.3
3We outsourced the translation process to a professional translation company, and each translation was manually reviewed.
While the source term is a homograph with multiple meanings, one of them is chosen to serve as its lexical constraint.4 Then, based on the meaning of the source term in the source sentence, each test example is classified into one of two groups:
• Positive Example where the source term in its source sentence is semantically aligned to the lexical constraint (See test examples (a) and (b) in Table 2). For positive test examples, we expect lexical constraints should always be applied.
• Negative Example where the given lexical constraint is semantically improper to impose (See test examples (c) and (d) in Table 2). Negative test examples allow us to evaluate how LNMT models respond to inappropriate lexical constraints.

heading: 2.2 Positive References
text: As seen in Table 3, we provide two auxiliary sourceside example sentences demonstrating the specific use of the source term of its lexical constraint, assuming that the meaning can be differentiated by the context used in the sentences rather than the terminology itself. Hereafter, we name these example sentences as positive references.
4Out of multiple different meanings of a homograph, we select the least frequent one as its lexical constraint. We describe the data construction details in Appendix A.

heading: 3 Methodology
text: Our methodology for semantically appropriate application of lexical constraints consists of two stages. Initially, we propose a homograph disambiguation module that can differentiate the semantics of lexical constraints. This module determines whether LNMT models should incorporate a lexical constraint or not. Subsequently, LNMT models, PLUMCOT in our case, perform the translation, either with or without the given lexical constraints.

heading: 3.1 Homograph Disambiguation
text: Given a few example sentences demonstrating how to specifically use a word, humans can infer the proper meaning. Likewise, our conjecture is that we can fulfill the homograph disambiguation task by leveraging these inter-sentential relationships.

heading: 3.1.1 Task Specification
text: Given n example sentences illustrating one specific meaning of a homograph, our homograph disambiguation module aims to determine whether the same word in a newly given sentence, denoted as ‘New Sentence’ in Fig. 1, carries the same meaning (label: 1) or not (label: 0). We conducted experiments with two example sentences (i.e., n = 2),5 and the corresponding model architecture is described in Section 3.1.2.
5We experiment with n = 1,2, and 3. The effect of varying the number of example sentences is analyzed in Appendix C.2.
Prediction

heading: 3.1.2 Model Architecture
text: Input Representations As illustrated in Fig. 1, sentence embeddings of example sentences and the new sentence are individually obtained from the PLM and fed into the classifier. Embedding vectors for all the sentences are extracted from the averaged hidden representations of the last K layers of frozen PLM.6 Here, the embedding vector is obtained by the average of hidden representations for the tokens that make up a homograph within the sentence. We denote this averaging operation as Pooling in Fig. 1.
Binary Classifier Similar to Sentence-BERT (Reimers and Gurevych, 2019), we use the concatenation (z ∈ R6m+3) of the following as an input to the classifier:
• Contextualized representation of a homograph (u, v,w ∈ Rm),
• element-wise difference for each pair (∣u − v∣ ,∣v −w∣ ,∣u −w∣ ∈ Rm,)
• pair-wise cosine similarity scores (sim(u, v), sim(v,w), sim(u,w) ∈ R),
where m is the dimension of the embeddings and sim(⋅, ⋅) denotes the cosine similarity function. Our prediction o ∈ [0,1] for a "New Sentence" is calculated as o = σ(max(0, zWr + br)W + b), (1) where Wr ∈ R(6m+3)×m and br are the weight matrix and bias vector of an intermediate layer, respectively. W ∈ Rm×1 and b are the weight matrix and bias vector for the final prediction layer followed by σ(⋅), which represents the sigmoid function.
6We utilize the last 16 layers of klue/roberta-large, a RoBERTa-based PLM trained on Korean corpus. See https: //huggingface.co/klue/roberta-large for details.
3.2 PLUMCOT In this subsection, we introduce our LNMT model, PLUMCOT, which stands for leveraging pre-trained language model with direct supervision on a copying score for LNMT, and its detailed implementation. To better incorporate target terms into the translations, PLUMCOT combines LeCA (Chen et al., 2021b) with PLM and strengthens a pointer network with supervised learning of the copying score.

heading: 3.2.1 Problem Statement
text: Lexically-constrained NMT Suppose X =(x1, x2,⋯, x∣X ∣) as a source sentence and Y =(y1, y2,⋯, y∣Y ∣) as a target sentence. Given the constraints C = (C1,C2,⋯,Cn) where each constraint Ci = (Ci,S ,Ci,T ) consists of the source term Ci,S and corresponding target term Ci,T , LNMT aims to incorporate C1∶n,T into its generation. The conditional probability of LNMT can be defined as
p(Y ∣X,C; θ) = ∣Y ∣∏ t=1 p(yt∣y<t,X,C; θ). (2)
Input Data As in Chen et al. (2021b), we modify X as X̂ = (X,<sep>,C1,T ,⋯,<sep>,Cn,T ) by appending <sep> tokens followed by target terms, as illustrated in Table 4.7 If there are no lexical constraints, a source sentence remains the same, i.e., X̂ = X .8 Combining a source sentence with target terms leads to the modification of Eq. (2) as the following:
p(Y ∣X,C; θ) = ∣Y ∣∏ t=1 p(yt∣y<t, X̂; θ). (3)

heading: 3.2.2 Integration of PLM
text: As PLM such as BERT (Devlin et al., 2019) is trained on large amounts of unlabeled data, leveraging PLM for LNMT can provide rich contextualized information of X , even in controlled unseen lexical constraint scenarios.
We first feed the source sentence X to a frozen PLM to obtain a representation B of a source sentence, where B is the output of the last layer of the PLM. Conversely, our NMT model based on Vaswani et al. (2017) receives a modified source sentence X̂ as input.
7In our training, we randomly sample target terms from the target sentence. Please refer to Appendix E for details.
8At test time, we append target terms only when lexical constraints are determined to be used by the homograph disambiguation module (as indicated in Table 4).
Let L denote the number of encoder and decoder layers of NMT, H l be the output of the encoder of NMT at the l-th layer, and hlt denote the t-th element of H l. For each layer l ∈ [1, L], we employ multi-head attention with the output of PLM as in Zhu et al. (2019), denoted as MHAB. This maps the output of the NMT encoder at l − 1th layer into queries and output of PLM, B, into keys and values.9 The output of the t-th element of the NMT encoder at the l-th layer is given by
h̃lt = 12(MHA(hl−1t ,H l−1,H l−1)+MHAB(hl−1t ,B,B)) + hl−1t , hlt = LN(FFN(LN(h̃lt)) + h̃lt),
(4)
where LN(⋅) denotes Layer normalization in Ba et al. (2016) and MHA and FFN(⋅) are the multihead attention and feed-forward network, respectively.
Similar to the encoder, multi-head attention with PLM is introduced for each decoder layer.10 Combined with Section 3.2.3, a highly contextualized representation is given to the pointer network.

heading: 3.2.3 Supervision on a Copying Score
text: Pointer Network To copy target terms from X̂ , we introduce a pointer network (Gu et al., 2016) as in Song et al. (2019); Chen et al. (2021b). For each time step, a pointer network takes in the output of the encoder and outputs a copying score g copy t ∈ [0,1], which controls how much to copy. The output probability of the target word yt can be calculated as
p(yt∣y<t, X̂; θ) = (1−gcopyt )×pwordt +gcopyt ×pcopyt , (5)
9Please refer to Appendix D for more details. 10Please refer to Appendix F for more details.
where pcopyt is a probability of copying, and p word t is a probability of the target word yt in the vocabulary.11
Copying Score As implied by Eq. (5), inaccurately predicted gcopyt results in the failure of copying target terms. However, in previous research on LNMT, the importance of a copying score was relatively understated. Despite the high probability of copying pcopyt , an incorrect copying score can even lower the output probability of the target terms. Therefore, we propose a novel supervised learning of the copying score gcopyt to obtain a more accurate value.
Our supervision of the copying score strengthens the copy mechanism of the pointer network by allowing the model to learn exactly when to copy. Since target terms are in the source sentence, we can determine which words should be copied from the source sentence. For example, when translating a source sentence in Table 4, the appended target term, Covid-19, must be copied. Thus, the copying score gcopyt of the target term Covid-19 should be higher, and gcopyt should be lower for the remaining words in the target sentence. Our training objective can be defined as
L(θ) = − ∣Y ∣∑ t=1 log p(yt∣y<t, X̂; θ) − λJ(θ), J(θ) = α ∑ t∉C1∶n,T (1 − gt) × log (1 − g copy t )
+ β ∑ t∈C1∶n,T gt × log g copy t ,
(6)
where a gold copying score gt is set to zero for t ∈ {t∣yt ∉ C1∶n,T }; otherwise, gt is set to one for t ∈ {t∣yt ∈ C1∶n,T }. To mitigate the length imbalance between the target terms and remaining words in the target sentence, we set α and β to the
11Please refer to Appendix G for more details.
value obtained by dividing their respective lengths from the total length.
4 Experiments on the HOLLY benchmark
In this section, we report the performance of our methodology when tested on the HOLLY benchmark. In Section 4.1, we evaluate the performance of our homograph disambiguation module in determining the semantic appropriateness of a lexical constraint. In Section 4.2, we assess the performance of LNMT models using positive examples from the HOLLY benchmark under conventional settings. Subsequently, we investigate the potential advantages that the homograph disambiguation module might bring when applied to the negative examples from the HOLLY benchmark.

heading: 4.1 Homograph Disambiguation

heading: 4.1.1 Data
text: Here, we present our dataset for training the homograph disambiguation module. Our training data was collected from the Korean dictionary12 and we manually inspected the quality of each sentence. In line with Fig. 1, each example consists of a triplet of example sentences containing a common homograph. Depending on the inter-sentential relationships between each input sentence, the homograph disambiguation module outputs a binary label: “1” is assigned if the homograph carries the same meaning in all sentences, and “0” if used differently in one example sentence. The brief data statistics of the training data are reported in Table 5. Note that any homographs are not allowed to be overlapped across train, validation, and test datasets.
At test time, we evaluated our model on the HOLLY benchmark. Specifically, for each lexical constraint, two positive references (refer to Table 3) and one of the four test example sentences ((a), (b), (c), or (d) in Table 2) is given as a triplet.

heading: 4.1.2 Results
text: We conducted experiments with two well-known variants of PLM trained on Korean corpora:
12To release data, we collected the examples that are controlled by the appropriate license. CC BY-SA 2.0 KR.
klue/roberta-base, and klue/roberta-large. Our homograph disambiguation module achieved a test accuracy of 88.7%, and 92.3% when using klue/roberta-base, and klue/roberta-large, respectively. In spite of the imbalanced data distribution shown in Table 5, the values of precision and recall are balanced in both classes, as shown in Table 6.

heading: 4.2 Lexically-constrained NMT

heading: 4.2.1 Training Data
text: We used 1.83M sentence pairs from two publicly available Korean-English datasets as training corpora: IWSLT 17 training data and AI Hub parallel data.13 We pre-tokenized the Korean corpora with Mecab and built a joint vocabulary for both languages by learning a Byte Pair Encoding (Sennrich et al., 2016) model in sentencepiece (Kudo and Richardson, 2018) with 32K merge operations.
To simulate the unseen lexical constraints, we filtered out about 160K training sentence pairs with test lexical constraints on both sides when experimenting with the HOLLY benchmark. This filtering process is crucial for examining how the models cope with any lexical constraints that users might introduce.

heading: 4.2.2 Evaluation
text: Metrics We evaluated the performance of our model in terms of BLEU14 and copy success rate (CSR). CSR is a metric for investigating the ratio of imposed lexical constraints met in translations. For a statistical significance test, we use compare-mt (Neubig et al., 2019) with p = 0.05 and 1,000 bootstraps.
Test Scenarios There were two important test cases, as shown in Table 7. Given a source sentence, we can consider the Soft Matching test case,
13The AI HUB data can be found here: https: //www.aihub.or.kr/aihubdata/data/view. do?currMenu=115&topMenu=100&aihubDataSe= realm&dataSetSn=126.
14We measure the BLEU scores using sacreBLEU (Post, 2018) with the signature nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.
which allows some morphological variations, as introduced in Dinu et al. (2019). As illustrated in Table 7, since the Korean word 소화 can be used in multiple different forms via inflection, any one of the expected candidates (digest, digestion, and digestive) presented in the translation is considered to be correct in terms of CSR.
We also have the Hard Matching15 test case where the exact target term (e.g., digestion) presented in its reference has to be incorporated in the translation. Note that, this cannot be tested on negative examples since the target terms in lexical constraints do not appear in their references.
Baselines
• Code-Switching (CS) (Song et al., 2019) replaces source terms with aligned target terms and learns to copy them via pointer network.
• LeCA (Chen et al., 2021b) modifies the source sentence as described in Table 4, and utilizes pointer network during training.
• Cdalign (Chen et al., 2021a) proposes constrained decoding based on alignment.16

heading: 4.2.3 Main Results
text: Simulating Unseen Lexical Constraints Table 8 shows the importance of simulating unseen lexical constraints. When lexical constraints are exposed during training, the vanilla Transformer already achieves 66.67% of CSR by mere memorization. We observe that eliminating around 160K overlapping training examples results in a significant reduction of CSR (66.67% → 11.97%), indicating that we manage to simulate the conditions where lexical constraints are nothing short of unseen.
Results on Positive Examples The performances of LNMT models on positive examples17
15This test case is suggested by previous work (Chen et al., 2021b; Song et al., 2019; Chen et al., 2021a).
16We compare our model to the ATT-INPUT approach, which suffers from high time complexity but guarantees a high CSR.
17Recall that positive examples are bound together with semantically appropriate lexical constraints. Refer to (a) and (b) in Table 2 for details.
are compared in Table 9. It is shown that PLUMCOT outperforms all the baselines in both metrics by a large margin. Since we simulate the unseen lexical constraints, the external information from the PLM contributes to the increase in BLEU. Combined with the supervision on a copying score, PLUMCOT achieves the highest CSR.18 The overall BLEU scores of Hard Matching are shown to be greater than Soft Matching as the expected target terms drawn from reference translations are given to the models.
Benefits of Homograph Disambiguation Here, we analyze beneficial effects of homograph disambiguation on LNMT. Since lexical constraints in negative examples19 are semantically improper, the homograph disambiguation module determines whether they should be imposed or not. Corrections are made with its decisions; the corresponding lexical constraints are removed. The effects of the correction are shown in Fig. 2.
We observed significant drops in CSR across all of the models, which is desirable since the lex-
18Instead of using the HOLLY benchmark, we also tested the performance of PLUMCOT in a test benchmark used in previous studies (Chen et al., 2021a,b) to compare its effectiveness, as shown in Appendix C.3.
19Refer to examples (c) and (d) in Table 2.
ical constraints are irrelevant to the context. By removing inappropriate constraints, all the models achieve a consistent and statistically significant improvement in translation quality by a large margin.

heading: 4.2.4 Ablation Study
text: We study the effect of each component of PLUMCOT, and the results are provided in Table 10. Compared to the PLUMCOT without supervision, the supervision on a copying score significantly improves CSR (93.85% vs. 98.06%). We find that leveraging rich contextual representation of PLM can further improve the translation quality (18.51→ 20.91). The BLEU score of a model that combines only PLM without supervision on a copying score is lower than that of the PLUMCOT model. This may simply be due to a higher BLEU score from better reflecting the target terms in the positive examples (an increase in CSR from 93.85% to 98.06%). Combining the two components yields the best performance in both metrics. More ablations can be found in Appendix C.

heading: 4.3 Qualitative Analysis
text: Table 11 provides translated examples. Given a lexical constraint, PLUMCOT incorporates the target term correctly. In a negative example, the meaning of 세제 is properly translated into detergent
by PLUMCOT with correction.20 We provide more examples in Table 15.

heading: 5 Related Work

heading: 5.1 Lexically-constrained NMT
text: Recent work on LNMT broadly falls into two categories: decoding algorithms and inline annotation. During beam search, decoding algorithms enforce target terms to appear in the output (Hokamp and Liu, 2017; Anderson et al., 2017; Chatterjee et al., 2017; Hasler et al., 2018). This approach ensures a high CSR, but the decoding speed is significantly degraded. To alleviate this issue, Post and Vilar (2018) suggests a decoding algorithm with a complexity of O(1) in the number of constraints. Another variation on decoding algorithms utilizes word alignments between source and target terms (Song et al., 2020; Chen et al., 2021a).
In inline annotation studies, the model is trained to copy target terms via modification of training data. Either a source term is replaced with the corresponding target term, or the target term is appended to the source sentence (Song et al., 2019; Dinu et al., 2019; Chen et al., 2021b). Concurrently, Bergmanis and Pinnis (2021); Niehues (2021); Xu and Carpuat (2021b) consider the morphological inflection of lexical constraints during the integration of target terms. While these methods incur a slight computational cost and provide better translation quality, target terms are not guaranteed to appear (Chen et al., 2021a; Wang et al., 2022a). To better copy target terms in a source sentence, a pointer network (Vinyals et al., 2015; Gulçehre et al., 2016) that uses attention weights to copy elements from a source sentence is intro-
20Note that the correction is made by homograph disambiguation.
duced (Gū et al., 2019; Song et al., 2019; Chen et al., 2021b). In this work, we further enhance the copying mechanism of a pointer network via supervised learning of a copying score that achieves better performance in terms of BLEU and CSR.

heading: 5.2 Homograph Issue in LNMT
text: Michon et al. (2020) points out the homograph issue in LNMT in an in-depth error analysis of their model. To the best of our knowledge, the homograph issue was explicitly addressed first in Öz and Sukhareva (2021). In their work, given a source homographic term, the most frequent alignment is selected as its correct lexical constraint, while the other alignments are treated as negative terms that should be avoided in the translation. However, low-frequency meanings are important for LNMT since it is not guaranteed that users always bring up generic terminology.
Different from their method, our homograph disambiguation module infers the meaning of lexical constraints and makes decisions to impose them or not. Furthermore, we confirm that our method works equally well on “unseen” homographs.

heading: 5.3 Integration of PLM with NMT
text: Followed by the success of PLM, researchers attempted to distill the knowledge of PLM into NMT (Zhu et al., 2019; Weng et al., 2020; Xu et al., 2021). BERT-fused (Zhu et al., 2019) is one such method; it plugs the output of BERT into the encoder and decoder via multi-head attention. We borrowed the idea from BERT-fused, and for the first time, combined LNMT and PLM, which works well even in “unseen” lexical constraints by leveraging the rich contextual information of PLM.

heading: 6 Conclusions
text: In this paper, we investigate two unexplored issues in LNMT and propose a new benchmark named HOLLY. To address the homograph issue of the source terms, we built a homograph disambiguation module to infer the exact meaning of the source terms. We confirm that our homograph disambiguation module alleviates mistranslation led by semantically inappropriate lexical constraints. PLUMCOT is also proposed to improve LNMT by using the rich information of PLM and ameliorating its copy mechanism via direct supervision of a copying score. Experiments on our HOLLY benchmark show that PLUMCOT significantly outperforms existing baselines in terms of BLEU and CSR.

heading: 7 Limitations
text: Our study includes some limitations that must be addressed. Some test examples might have wrong predictions made by the homograph disambiguation module. Specifically, in positive examples where lexical constraints should be imposed, its errors result in wrong corrections (i.e., the elimination of necessary lexical constraints). Table 12 shows how these erroneous corrections affect the results.
We can observe an overall decline in CSR; however, it does not hurt the translation quality. We
verify that the differences in BLEU resulting from wrong corrections are not statistically significant for all the methods. Considering the gain achieved in negative examples, as seen in Fig. 2, our proposed homograph disambiguation might serve as a useful starting point to address homographs in LNMT; however, there is still room for improvement. Our current homograph disambiguation module is designed as a stand-alone system outside the LNMT. However, building an end-to-end system can be beneficial, which can be addressed in future work.

heading: Acknowledgments
text: Authors would like to thank all Papago team members for the insightful discussions. Also, we sincerely appreciate the fruitful feedback from Won Ik Cho and CheolSu Kim. We thank the anonymous reviewers for their valuable suggestions for enhancing this work. This work was supported by Papago, NAVER Corp, the Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1A2B5B02001913).

heading: C Ablation Studies
text: C.1 Weights of the supervised learning of a copying score
The results in Table 13 were reported according to the different weights λ of the supervised learning of the copying score in Eq. (6). Based on experimental results, we were able to find a compromise where λ is 0.2.
C.2 Number of example sentences We experimented with a varying number of example sentences. As we use more example sentences, the information from the inter-sentential relationships becomes richer, eventually improving homograph disambiguation performance. Experiments with n = 1,2, and 3 show an accuracy of 91.33%, 92.33%, and 92.67%, respectively. Although an experiment with n = 3 provides the best accuracy, collecting positive references can sometimes be burdensome to users. Therefore, we conclude that n should be decided by considering its trade-off.
C.3 Randomly Sampled Test Constraints Different from our HOLLY benchmark, at test time, lexical constraints were randomly sampled from the alignments in each sentence pair in previous studies (Dinu et al., 2019; Song et al., 2019; Chen et al., 2021b,a; Wang et al., 2022b). Ten different
21Please refer to 16 for more details.
test sets were built based on ten randomly sampled sets of lexical constraints, as described in Chen et al. (2021a,b). Test statistics are reported in Table 14. It is shown that PLUMCOT achieves the highest BLEU. The CSR is slightly lower than Cdalign, indicating that the gain for “seen” constraints is insignificant.22

heading: D Equation Details
text: Let Q, K, and V be the query, key, and value in (Vaswani et al., 2017), respectively. Then MHA in Eq. (4) and Eq. (8) can be calculated as
MHA(Q,K,V ) = CatHh=1[head1,⋯,headn]Wo, headi = Attn(QWQi ,KWKi , V W Vi ),
Attn(q, k, v) = softmax( qkT√ dk )v,
(7) where the projection matrices are parameters WO ∈ RHdv× dmodel , WQi , WKi , W Vi ∈ Rdmodel×dk for MHA. In this paper, we employ dmodel = 768, H = 12, dk = dv = dmodel/H = 64. Note that all baselines and our model PLUMCOT use the same number of heads and the same projection matrices size. We use additional multi-head attention, MHAB , which only differs in projection matrices size where WQi , W K i , W V i ∈ RdPLM×dk for MHAB .23
E Input data augmentation
As illustrated in Table 4, we modify a source sentence X as X̂ by appending <sep> tokens followed by target terms. Since lexical constraints are domain-specific or user-provided terminology, we exclude the top 1,500 frequent words from a 32K joint dictionary. In our training, we randomly
22Note that the models are trained with full data as we cannot remove training examples that overlap with random test lexical constraints in advance.
23Here, we utilize klue/roberta-large, a RoBERTa-based PLM trained on Korean corpus. The size of dPLM is 1024.
sample at most 3 target terms from the target sentence. For each sentence, from 0 to 3 target terms are sampled following the distribution [0.3, 0.2, 0.25, 0.25].
F Integration of PLM in Decoder
Here, we follow the same notations in Section 3.2.2. Let Sl denote the decoder output at lth layer. slt is the t-th element of Sl, and Sl1∶t denotes t number of elements from sl1 to s l t, masking elements from
t + 1 to the end. The output of each layer of the decoder can be calculated as
ŝlt = LN(MHA(sl−1t , Sl−11∶t , Sl−11∶t )) + sl−1t , s̃lt = 12(MHA(ŝlt, HL, HL) +MHAB(ŝlt, B, B)) + ŝlt,
slt = LN(FFN(LN(s̃lt)) + s̃lt). (8)

heading: G Pointer Network
text: We use the same notations in Section 3.2 and Appendix D and F. Let ∣X̂ ∣ denotes the length of the modified source sentence X̂ , and(αt,1, αt,2,⋯, αt,∣X̂ ∣) denotes the averaged attention weight of MHA(ŝLt , HL, HL) over the multiheads in Eq. (8). Then our copying score gcopyt can be calculated as
g copy t = σ(Wg[ct; sLt ] + bg), ct = ∣X̂ ∣∑
i=1αt,i × hLi , (9)
where ct and sLt are concatenated, Wg and bg are the weight matrix and bias vector, and σ(⋅) is the sigmoid function.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? We summarized the paper’s main claims both in the abstract and introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We created a test benchmark. Refer to Section 2 for more details.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 2
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 2, Appendix
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2, Section 4
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4, Appendix
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We reported the results of just a single run.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? As we outsourced the translation process to a professional company, only qualified professional translators were participated. We guarantee that our institution paid sufficient amount of cost by signing a contract.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix A
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We manually review the data. Our data has nothing to do with demographic or geographic issues.

heading: Future Work
text: However, building an end-to-end system can be beneficial, which can be addressed in future work