Bring More Attention to Syntactic Symmetry for Automatic Postediting of High-Quality Machine Translations
Year: 2023

Abstract:
Automatic postediting (APE) is an automated process to refine a given machine translation (MT). Recent findings present that existing APE systems are not good at handling highquality MTs even for a language pair with abundant data resources, English–German: the better the given MT is, the harder it is to decide what parts to edit and how to fix these errors. One possible solution to this problem is to instill deeper knowledge about the target language into the model. Thus, we propose a linguistically motivated method of regularization that is expected to enhance APE models’ understanding of the target language: a loss function that encourages symmetric self-attention on the given MT. Our analysis of experimental results demonstrates that the proposed method helps improving the state-of-the-art architecture’s APE quality for high-quality MTs.

heading: 1 Introduction
text: Automatic postediting (APE) is an automated process to transform a given machine translation (MT) into a higher-quality text (Knight and Chander, 1994). Since 2015, Conference on Machine Translation (WMT) has been hosting an annual shared task for APE, and most of the recently developed APE systems are within the common framework of representation learning using artificial neural networks to learn postediting patterns from the training data (Chatterjee et al., 2018, 2019, 2020; Akhbardeh et al., 2021).
Since 2018, all participants in the shared task have used Transformer-based models (Vaswani et al., 2017), but recent findings of the shared task (Chatterjee et al., 2018, 2019, 2020; Akhbardeh et al., 2021) cast doubt on whether Transformer-based APE models learn good generalizations because such models’ APE quality appears to be significantly affected by external factors such as the source–target language pair, the qualitative
Doppelbaum
characteristics of the provided data, and the quality of the given MT.
Especially, the good quality of the given MTs has brought great difficulty in performing APE on the WMT 2019 test data set: the better the given MT is, the harder it is to decide what parts to edit and how to correct these errors (Chatterjee et al., 2018, 2019). The thing to notice is that this outcome is not a question of data scarcity because the language pair of this test data set, English–German, is a language pair provided with abundant training, validation, and test data. Also, it is not a question of data heterogeneity, either: the domain of this test data set, IT, shows a high degree of lexical repetition, which indicates that data sets in this domain use the same small set of lexical items (Chatterjee et al., 2018, 2019; Akhbardeh et al., 2021). Thus, it would be a question of modeling, and one possible solution is to implant deeper knowledge about the target language into the model.
To this end, we propose a new method of regularization that is expected to enhance Transformerbased APE models’ understanding of German translations. Specifically, the proposed method is based on Feldermodell (§2), an established linguistic model, which implies the need for proper treatment
1433
of the underlying symmetry of German sentence structures. To instill the idea of syntactic symmetry into Transformer-based APE models, we introduce a loss function that encourages symmetric self-attention on the given MT. Based on experimental results, we conduct a careful analysis and conclude that the proposed method has a positive effect on improving the state-of-the-art architecture’s APE quality for high-quality MTs.

heading: 2 Linguistic Theory
text: In German linguistics, das topologische Satzmodell (‘the topological sentence model’) or das Feldermodell (‘the field model’) (Reis, 1980; Wöllstein, 2018; Höhle, 2019) describes how constituents of a sentence are closely related even if they are far apart from each other. Usually, Feldermodell divides a clause into das Vorfeld (‘the prefield’; VF), die linke Satzklammer (‘the left bracket’; LSK), das Mittelfeld (‘the middlefield’; MF), die rechte Satzklammer (‘the right bracket’; RSK), and das Nachfeld (‘the postfield’; NF).
(1) [Heute VF] [ habe LSK] [ ich MF] [ gesehen RSK] [ zufällig NF], (2) [ [ dass LSK] [ du eine Tasse Kaffee MF] [ getrunken hast RSK] NF].
These parts are all interrelated; LSK and RSK are a typical example: while the former holds a finite verb or a complementizer, the latter holds a past participle, an infinitive, and a particle. In (1), VF holds “Heute” (‘today’); LSK holds “habe” (‘have’); MF holds “ich” (‘I’); RSK holds “gesehen” (‘seen’); and NF holds “zufällig” (‘by chance’). (2) is an additional NF of (1) and includes its own LSK holding “dass” (‘that’); MF holding “du eine Tasse Kaffee” (‘you a cup of coffee’); and RSK holding “getrunken hast” (‘drank’).
For such analyses, special tree structures such as Doppelbaum (Wöllstein, 2018) (‘double tree’) can be used, which is a bimodal tree (Fig. 1), where two CP, C, IP, I, and VP subtrees are ‘symmetric’ with respect to V. We assume that this structural symmetry is parameterized from the perspective, not only of generative linguistics (Wöllstein, 2018; Höhle, 2019), but also of a parametric model P ={Pθ ∣ θ ∈ Θ}, where Pθ and Θ are a probability distribution and the parameter space, respectively.
Especially, if we look at APE in terms of sequence-to-sequence learning (Sutskever et al.,
2014), the probability distribution of the output sequence (y1,⋯, yLy) is obtained in the following manner:
Pθ(y1,⋯, yLy ∣ x1,⋯, xLx , z1,⋯, zLz) = Ly∏
t=1Pθ(yt ∣ u, v, y1,⋯, yt−1), where u and v are the representations of a source text (x1,⋯, xLx) and its MT (z1,⋯, zLz), respectively. In this process, we presume that the syntactic symmetry of the target language affects the resulting distribution Pθ; in other words, this syntactic symmetry would be an inductive bias (Mitchell, 1980) that should be handled properly.

heading: 3 Methodology
text: We implement a multi-encoder Transformer model consisting of the “Joint-Final” encoder and the “Parallel” decoder, which is a state-of-the-art architecture for APE (Shin et al., 2021), and conduct a controlled experiment without concern for usage of performance-centered tuning techniques. Specifically, the Joint-Final encoder consists of a sourcetext encoder and an MT encoder, which process the given source text and MT, respectively. Based on this baseline architecture, we propose a method to encourage the MT encoder to perform symmetric self-attention by minimizing the skewness of each self-attention layer’s categorical distribution pself.
The used measure of skewness is
(µ̈3)i = ⎛⎜⎝ ⌊Lz 2 ⌋∑ j=1 pself[i, j] − Lz∑
j=⌈Lz 2 ⌉+1
pself[i, j]⎞⎟⎠ 2 ,
for each token zi in the given MT (z1,⋯, zLz). Accordingly, the basic cross-entropy loss LCE
is regularized by (µ̈3)i, resulting in a new loss function
LDOPPELBAUM = LCE +E[α]E[µ̈3] + (1 − α), where
E[α] = ∑Bb=1∑Lzi=1 αb,i B ×Lz
is the expected value of coefficients αb,i = σ(W Tvb,i + β) in the given minibatch, and
E[µ̈3] = ∑Bb=1∑Nn=1∑Hh=1∑Lzi=1(µ̈3)b,n,h,i B ×N ×H ×Lz
is the expected value of (µ̈3)b,n,h,i. In addition,(1 − α) is an initial inducement to utilizing µ̈3. In the equations above, σ is the sigmoid function, v is the output of the final layer of the MT encoder, W ∈ Rdmodel and β ∈ R are learned parameters, B is the number of data examples, N is the number of layers, and H is the number of heads.

heading: 4 Experiment
text: In the conducted experiment, all hyperparameters are the same as those of Shin et al. (2021) except the learning rate (Appendix A); we basically reproduce their experimental design.
Both the baseline model and the proposed model are trained by using the training data sets and the validation data set listed in Table 1; we first train the models by using eSCAPE-NMT mixed with the WMT 2019 training data in the ratio of 27 ∶ 1, and then tune them by using the WMT 2019 training data solely.

heading: 5 Results and Analysis
text: The result of automatic evaluation (Table 2) indicates that the proposed model improves on the baseline model in terms of BLEU (75.47) but does not in terms of TER (16.54), which is unusual. Although those measures have a strong correlation overall (Fig. 2), the proposed model has more outliers, δBLEU (the value obtained by subtracting a given MT’s BLEU from the postedited result’s BLEU) of which is over 20, compared to the baseline model; they must be the ones that bring the improvement in BLEU.
Thus, we present an additional evaluation result to further investigate this mismatch between TER improvements and BLEU improvements: a relative frequency distribution of successes and failures in APE with regard to the TER difference
between a given MT and each model’s output (Table 3). Then, the mentioned outliers correspond to PERF, which is the set of the cases where an APE system succeeds in perfectly correcting the given MT with one or more errors, considering that the proposed model’s PERF has a µδBLEU (the average of sentence-level BLEU improvements) of 27.21. We see that the proposed model has substantially more PERF cases (5.87%) than the baseline model (4.30%) and that because most of those ‘new’ (1.57pp) cases are results of nontrivial postediting (Table 4), this increase in the proportion of perfect postediting is valid evidence of the proposed method’s effect on enhancing the baseline model’s APE quality for high-quality MTs.
In addition, in an actual example where only the proposed model corrects the given MT perfectly (Table 5), we observe that the proposed model successfully captures the close relation between the verb “enthält” (‘contains’) and its object so that the correct form “Variablen” (‘variables’) is used. Considering that the adverb phrase “zum Beispiel”
(‘for example’) in the given MT makes some distance between the verb and its object, it appears that the proposed model integrates information from a wider range of constituents than the baseline model; hence the conclusion that the proposed method instills Feldermodell’s idea of syntactic symmetry into Transformer-based APE models and enhances their understanding of German translations.
Another example (Table 6) suggests that the increase in the proportion of ACCE (0.3pp), which is the set of the cases where an APE system adopts the given, already perfect MT, should be cautiously interpreted. Although professional translators tend to perform “only the necessary and sufficient corrections" (Bojar et al., 2015), the validity of test data created by professional translators, including the WMT 2019 test data set, can also be disputable because other native speakers might argue that they can perform better postediting. For example, some people may consider hyphenated compound “Zoom-Werkzeug” (‘Zoom tool’) more natural than closed compound “Zoomwerkzeug” (Table 6).
However, considering the big differences in the proportion of NEGL (2.35pp), which is the set of the cases where an APE system neglects to postedit the given MT, and the F1 score (Table 3), it appears that such a risk need not be considered in this analysis. Moreover, the proposed model has fewer RUIN cases (1.56%), where it injects errors to the given, already perfect MT, than the baseline model (1.86%). Although the proposed model has more DEGR cases (7.33%), where it degrades the given MT, than the baseline
CASE 1: PERF TER↓ BLEU↑
Source Text For example , the following function contains variables that are defined invarious block scopes .
(6.65%), the proposed model’s quality degradation µδBLEU = −11.72 is less severe than that of the baseline (µδBLEU = −13.51). Therefore, we conclude that the proposed method results in small but certain improvements.

heading: 6 Conclusion
text: To improve the APE quality for high-quality MTs, we propose a linguistically motivated method of regularization that enhances Transformer-based APE models’ understanding of the target language: a loss function that encourages APE models to perform symmetric self-attention on a given MT. Experimental results suggest that the proposed method helps improving the state-of-the-art architecture’s APE quality for high-quality MTs; we also present a relative frequency distribution of successes and failures in APE and see increases in the
proportion of perfect postediting and the F1 score. This evaluation method could be useful for assessing the APE quality for high-quality MTs in general. Actual cases support that the proposed method successfully instills the idea of syntactic symmetry into APE models. Future research should consider different language pairs and different sets of hyperparameters.

heading: 7 Acknowledgements
text: This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (Ministry of Science and ICT) (No. 2019- 0-01906, Artificial Intelligence Graduate School Program (POSTECH)). We thank Richard Albrecht for assistance in the manual categorization of cases.

heading: 8 Limitations
text: First, neither Feldermodell (Reis, 1980; Wöllstein, 2018; Höhle, 2019) nor Doppelbaum (Wöllstein, 2018) has obtained complete concurrence among linguists. Also, we limit our scope to the English– German language pair and the IT domain using the WMT 2019 training, validation, and test data sets. A broader scope would not provide confidence in the validity of conducted experiments because there are hardly any standard setups for experimental research (Chatterjee et al., 2018, 2019; Akhbardeh et al., 2021).
In addition, the conducted experiment should take into consideration the effect of randomness that is attended in the process of training artificial neural networks; different techniques, different hyperparameters, and multiple runs of optimizers (Clark et al., 2011) may present different results. However, as previous studies (Chatterjee et al., 2018, 2019, 2020; Akhbardeh et al., 2021), including the study on the baseline model (Shin et al., 2021), do not consider the effect of randomness, we also do not investigate the effect of randomness further, considering that training multiple models (Appendix A) to obtain good estimators (TER and BLEU) will cost a lot.

heading: A Experimental Details
text: We use the following hyperparameters: the number of layers N = 6, the number of heads H = 8, the dimension of key vectors dk = 64, the dimension of value vectors dv = 64, the vector dimension for multi-head attention layers dmodel = 512, the vector dimension for the inner layer of position-wise feedforward networks dff = 2,048, the dropout (Srivastava et al., 2014) probability Pdrop = 0.1, the label smoothing value ϵLS = 0.1, minibatches of 25,000 tokens, a learning rate of 2.0, warmup for 18,000 training steps, and a shared vocabulary consisting of 32,000 subword units (Sennrich et al., 2016)1. We also use weight tying (Pappas et al., 2018) and the Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.998, and ϵ = 10−8. Decoding options are beam search with a beam size b = 5, a length penalty multiplied by a strength coefficient α = 0.6, and beam search stopping (Yang et al., 2018) with the length ratio lr = 1.3.
We use OpenNMT-py 3.0 (Klein et al., 2017)2 with the random seed 1128. We first train the models for 100,000 steps, about 36 hours on one NVIDIA GeForce RTX™ 3090, and then tune them around 1,000 steps.
1We used SentencePiece (Apache License 2.0) 2The MIT License.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8.
7 A2. Did you discuss any potential risks of your work? With the standard setup, studies in the field of automatic postediting, including this work, do not involve potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Section 5.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 and Section 5.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4, Section 5, and Appendix A.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The used artifacts are not considered to have any extraordinary usages.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The used artifacts are not considered to contain any information that names or uniquely identifies individual people or offensive content.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 1 and Section 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 and Section 5.
C 3 Did you run computational experiments? Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A. The number of parameters are reported in the study on the baseline system.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, Section 5, and Appendix A.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5. However, the person who helped us only double-checked our analysis; he did not annotate any data.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future research should consider different language pairs and different sets of hyperparameters