Easy Guided Decoding in Providing Suggestions for Interactive Machine Translation
Year: 2023

Abstract:
Machine translation technology has made great progress in recent years, but it cannot guarantee error-free results. Human translators perform post-editing on machine translations to correct errors in the scene of computer-aided translation. In favor of expediting the post-editing process, many works have investigated machine translation in interactive modes, in which machines can automatically refine the rest of translations constrained by human’s edits. Translation Suggestion (TS), as an interactive mode to assist human translators, requires machines to generate alternatives for specific incorrect words or phrases selected by human translators. In this paper, we utilize the parameterized objective function of neural machine translation (NMT) and propose a novel constrained decoding algorithm, namely Prefix-Suffix Guided Decoding (PSGD), to deal with the TS problem without additional training. Compared to the state-of-the-art lexically constrained decoding method, PSGD improves translation quality by an average of 10.87 BLEU and 8.62 BLEU on the WeTS1 and the WMT 2022 Translation Suggestion datasets2, respectively, and reduces decoding time overhead by an average of 63.4% tested on the WMT translation datasets. Furthermore, on both of the TS benchmark datasets, it is superior to other supervised learning systems trained with TS annotated data.

heading: 1 Introduction and Related Work
text: The emergence of machine translation technology (Lopez, 2008; Koehn, 2009) assists human translation to improve translation efficiency (Green et al., 2014, 2015; Herbig et al., 2020). Even though there is a quality gap between the outputs of machine
∗indicates equal contribution. † indicates the corresponding author.
1https://github.com/ZhenYangIACAS/WeTS 2https://www.statmt.org/wmt22/
translation-suggestion-task.html
translation (MT) systems and manual translations by professional translators, MT can still practically reduce time in comparison with translating from scratch (Läubli et al., 2013). Later, the advances at the sequence-to-sequence model (Sutskever et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017) further made a breakthrough in translation technology, inspiring the industry to transform human translation into computer-aided translation (CAT) (Knowles and Koehn, 2016; Santy et al., 2019) to a great extent. CAT usually relies on an MT engine and a platform with a user-friendly interface (Bowker, 2002; Lengyel et al., 2004; Bowker and Fisher, 2010; Bowker, 2014; Pal et al., 2016; Chatterjee, 2019), with which humans perform postediting (PE) on machine translations to achieve final results with quality standards.
In the past, the post-editing process was typically static and machines would no longer respond to humans’ modifications once humans started postediting. Recent works (Domingo et al., 2016; González-Rubio et al., 2016; Peris et al., 2017) investigate interactive protocols and algorithms so that humans and machines can collaborate and machines automatically refine the translations according to the human’s edits. One promising mode is Translation Suggestion (TS) proposed by Yang et al. (2021a) as a pioneer, which requires the machine to provide alternatives for specific spans of incorrect words or phrases selected by humans, namely making suggestions given prefix and suffix constraints. In practical applications as shown in Figure 1, it usually happens when human translators would like to edit part of the MT output. It can be easily implemented with a user interface if machines correctly provide suggestions for the selected incorrect spans. Yang et al. (2021a) has proven the significance of TS in post-editing in terms of resolving two pitfalls of earlier works. The importance has also been recognized by the Conference of Machine Translation (WMT), and they released the Naive Translation
7840
Suggestion shared task (Yang et al., 2022) in WMT 20223.
One of the solutions to TS can be training a supervised model with TS annotated data. Yang et al. (2021a) trained such an end-to-end Transformerlike model as a benchmark system. Ge et al. (2022) applied model fine-tuning with TS data augmentation on pre-trained NMT models. However, supervised learning, which relies on a large amount of labeled data, is too heavy to be easily adjusted to other domains. In addition, due to the complicated post-editing process, it is expensive to obtain such limited annotated data.
Our idea is to investigate inference algorithms given prefix and suffix constraints. We tested the state-of-the-art lexically constrained decoding algorithm (Post and Vilar, 2018; Hu et al., 2019) on the WeTS dataset, and found that omissions frequently occur in suggestion generations. There are two reasons behind: (1) the division of beams in the dynamic beam allocation narrows the search space so that there would not be enough candidates that match constraints to be picked. This is more likely to happen when constraints are much longer than the average length of the span selected, such as in TS applications; (2) the beam search stops when the probability of eos, the special token for the end of a sentence, appears largest in the softmax distribution, but the probability of the entire sentence generation has not been considered. In terms of efficiency, this decoding algorithm contains unnecessary calculations since it always generates
3https://statmt.org/wmt22/ translation-suggestion-task.html
suggestions step by step from the beginning of the translation to the end of the sentence, including prefix and suffix constraints.
In this paper, we propose a neat prefix-suffix guided decoding (PSGD) algorithm for the TS task. There are three main contributions: (1) PSGD emphasizes the probability of the entire generation including prefix and suffix constraints, but only decodes for the incorrect span rather than the whole translation sentence, which improves both suggestion quality and time efficiency. (2) PSGD theoretically avoids dividing beams as in Hu et al. (2019) so that the original beam search space can be used to improve the quality of generated translation suggestions. (3) PSGD does not require any additional training/fine-tuning on the original NMT model, which means it can be applied to any autoregressive machine translation system with flexibility.
Our experimental observations show that PSGD significantly outperforms the state-of-the-art lexically constrained decoding method (Post and Vilar, 2018; Hu et al., 2019) by an average increase of 10.87 BLEU and 8.62 BLEU on the benchmark datasets WeTS and WMT 2022 Naive Translation Suggestion datasets (WMT22-TS), respectively. Experimental results also demonstrate PSGD’s superiority in overall time efficiency by a 63.4% time reduction. In addition, on both the WeTS and WMT22TS datasets, PSGD is superior over other supervised learning systems trained with TS annotated data.

heading: 2 Preliminary
text: Before introducing the PSGD, we first elaborate on the problem in the scene of TS with the mathematical notations in Table 1. The sequence-to-sequence machine translation model is generally a conditional auto-regressive language model that follows the factorized probability distribution.
PMT(y|x; Θ) = ∏
i
PMT(yi|y<i,x; Θ) (1)
Given training pairs of (x,y), maximizing the probability in Formula 1 will return an estimate of the model parameter, Θ̂.
During inference, we are supposed to maximize
the probability of the generated sequence:
ŷ = argmax y
PMT(y|x; Θ̂) (2)
However, the solution space {y} is infinite and impossible to search. Therefore, an auto-regressive decoding process is applied as an approximation. The decoder of the model decodes the translation step by step greedily4 as following until eos is met:
ŷi = argmax y
PMT(y|ŷ<i,x; Θ̂) (3)
In the scene of TS, a span r is masked for suggestion generation, and thus we need to guarantee that y starts with the given prefix p and ends with the given suffix s. Then, the problem becomes:
r̂ = argmax r
PMT(⟨p, r, s⟩|x; Θ̂) (4)

heading: 3 Methodology
text: As mentioned previously, earlier works of constrained decoding require step-by-step generations for the whole sentence even with prefix and suffix constraints. It causes efficiency issues especially when the constraints are long. What’s worse, the division of beams might bring about low-quality suggestions. Accordingly, PSGD generates translation suggestions given prefix and suffix constraints through maximizing the probability of the whole generated sequence. Meanwhile, PSGD performs a step-by-step beam search generation only for the span without dividing beams. A brief example of the PSGD algorithm for TS can be found in Figure 2. With this picture in mind, we will give details of the PSGD algorithm in this Section. The pseudo codes of this algorithm are presented in Algorithm 1. And the fairseq based implementation is available on github5.

heading: 3.1 Decoding Process for TS
text: Similar to the normal translation process, we apply the auto-regressive decoding process to overcome the issue of infinite solution space {r}. We can factorize the entire probability in Formula 4 into three parts:
4We do not involve beam-search here to simplify the equations, but in practice, all our experiments are conducted with beam-search.
5https://github.com/wangke1996/ translation-suggestion-psgd
Algorithm 1 Algorithm of PSGD Input: x, p, s Parameter: pt Output: r̂
1: Let r̂ = [], n = 0, maxPr = −∞, bestStep = 0. 2: while n− bestStep < pt do 3: ŷ = ⟨p, r̂, s⟩ 4: {f(pi)}, {f(r̂j)}, {f(sk)} = Output prob-
ability distribution of every tokens in ŷ = ⟨p, r̂, s⟩
5: calculate P (n)r with Formula 9 6: if P (n)r > maxPr then 7: maxPr = P (n) r 8: bestStep = n 9: end if
10: calculate r̂n+1 with Formula 10 11: append r̂n+1 to r̂ 12: n = n+ 1 13: end while 14: return the first bestStep elements of r̂
PMT(⟨p, r, s⟩|x; Θ) = ∏
i
PMT(pi|p<i,x; Θ)
· ∏
j
PMT(rj |⟨p, r<j⟩,x; Θ)
· ∏
k
PMT(sk|⟨p, r, s<k⟩,x; Θ)
(5)
Notice that in Formula 5, as all pi and sk are
given, only the second product requires a step-bystep auto-regressive generation. The first product, referring to the probability of the prefix sequence p, can be calculated at the first step when r1 is decoded. And when all of the decoding steps of r are completed, we can easily obtain the third product, which refers to the probability of the suffix sequence s. Hence, the decoding process at each step is actually:
r̂j = argmax r
PMT(r|⟨p, r̂<j⟩,x; Θ̂) (6)
Theoretically, we only need to go forward through the model for tr times, which is smaller than that of a normal constraint decoding (Hu et al., 2019), i.e. tp + tr + ts. In particular, for the TS task, human translators might select a short span with incorrect words, and then tp + tr + ts could be much larger than tr.

heading: 3.2 Condition for Stopping Decoding
text: The decoding process aforementioned is straightforward. Now, it comes to a problem: when should the inference stop? In Post and Vilar (2018) and Hu et al. (2019), the model generates the full sequence and the generation stops when the eos is met. However, in PSGD, only the span r will be generated and r does not end with eos. We cannot utilize the generation of eos as a signal to stop decoding.
The key to solving this problem can be aligned with the objective function of NMT training as described in Formula 1. We assume that we have finished decoding in Formula 6 for N steps, where N is large enough to obtain a hypothesis sequence
r∗. The solution space {r} becomes {r∗<n}Nn=1. The most optimal sequence r̂ = r∗<n̂ is obtained by calculating n̂ as:
n̂ = argmax n
PMT (⟨p, r∗<n, s⟩|x; Θ̂) (7)
The normal decoding process stops when eos is reached, and it does not necessarily mean that the probability of the entire sequence generation is maximized. In other words, there might be a better hypothesis if the model generates fewer or more steps in the normal decoding process. Our condition for stop decoding in PSGD approaches the training objective function in Formula 1 as much as possible. This is the reason why PSGD achieves better performance in suggestions.

heading: 3.2.1 Early Stopping
text: Although the above workaround provides a proper condition for stopping decoding, it might cause efficiency issues since the number of decoding steps would become N rather than tr. Obviously, N should be equal to or greater than tr to achieve the best performance. However, we have no idea how large tr is. We have to set N to a large enough value to cover as many cases as possible, which is unacceptable in practice.
Instead of setting a large N , we try to apply an early stopping mechanism to save unnecessary decoding steps. In Formula 6, after every decoding step, PSGD calculates and records the averaged probability of entire sequence generation with both prefix and suffix constraints concatenated, denoted as P (n)r as follows:
P (n)r = PMT (⟨p, r̂<n, s⟩|x, Θ̂)
tp + n+ ts (8)
PSGD stops decoding when P (n)r does not increase any more within pt consecutive steps, where pt represents patience for early stopping.

heading: 3.2.2 Parallel Acceleration
text: Because the early stopping mechanism requires extra calculations for P (n)r at each step, it seems that we would need to go forward through the model for 2 ∗ (tr + pt) times. In fact, by utilizing the future mask in Transformer’s decoder, we only run tr+pt times since P (n)r and r̂n+1 can be obtained at the same time. Details are explained as follows.
Figure 3 shows the forward path for calculating P (n) r , where f(ŷt) is the softmax probability distribution at the t-th decoding step. f(y|yt) indicates the probability of token y at (t+ 1)-th step, which is short for PMT (y|ŷ≤t,x; Θ̂). We have:
P (n)r = 1
tp + n+ ts · f(p1|bos) · f(p2|p1) · . . . ·f(ptp |ptp−1) · f(r̂1|ptp) · f(r̂2|r̂1) · . . . ·f(r̂n|r̂n−1) · f(s1|r̂n) · f(s2|s1) · . . . ·f(sts |sts−1) · f(eos|sts)
(9)
and
r̂n+1 =argmax r
PMT(r|⟨p, r̂≤n⟩,x)
= argmax r
f(p1|bos) · f(p2|p1) · . . .
·f(ptp |ptp−1) · f(r̂1|ptp) · f(r̂2|r̂1)· · · · · f(r̂n|r̂n−1) · f(r|r̂n)
= argmax r
f(r|r̂n)
(10)
At the n-th step, we can simultaneously get the softmax distributions {f(pi)}, {f(r̂j)}, and {f(sk)}. Then, according to Formula 9 and 10, P (n)r (the probability of the entire generation for early stopping) and r̂n+1 (the next token generation) can be obtained together. Therefore, the total step of the decoding process in PSGD is tr + pt.

heading: 3.3 Summary of Methodology
text: In this section, we have provided full details of PSGD, and the pseudo code is presented in Algorithm 1. By utilizing the parameterized objective function, PSGD maximizes the probability of the entire generation for better suggestion quality, but only generates the suggestion for the incorrect span instead of the whole sequence. Besides, it designs a proper early-stopping mechanism for efficiency improvement. Only tr + pt decoding steps are required, which is smaller than tp+tr+ts in Hu et al. (2019), since tr is much smaller than tp + tr + ts
on average in the scene of TS, and the value of the hyper-parameter pt is proven to be quite small in the subsequent experimental observations.

heading: 4 Experiments

heading: 4.1 Generation Performance on Translation Suggestion
text: Translation suggestion is a significant CAT task where human translators select spans with incorrect words or phrases in the MT results. It requires models to automatically provide alternatives for the spans for efficiency improvement in translation. WeTS (Yang et al., 2021a) and WMT22-TS (Yang et al., 2022) are the benchmark datasets for the TS task. Each sample consists of a source sentence, a masked MT sentence, and a golden reference corresponding to the masked span. The masked spans with potential translation errors are selected and annotated by humans. The annotations mainly focus on three types of translation errors: under (over)-translations, grammatical or syntactic errors, and semantic errors.
We compare the PSGD with three state-ofthe-art systems on WeTS and WMT22-TS: SATransformer (Yang et al., 2021a), TSMind (Ge et al., 2022) (without TS data augmentation) and VDBA (Hu et al., 2019). SA-Transformer is an end-to-end transformer-like model, which initializes the encoder with weights from the pre-trained XLM-Roberta (Conneau et al., 2020) and finetunes it with TS labeled data. TSMind utilizes the pre-trained NMT model and fine-tunes it with TS triplets as well, and it wins in three out of four language pairs of the Naive Translation Suggestion task in WMT 2022 Yang et al. (2022). VDBA is basically an improved version of the Dynamic Beam Allocation algorithm proposed in Post and Vilar (2018), that is remarkably capable in lexically constrained decoding.
We should notice that both of SA-Transformer and TSMind perform supervised learning with human-annotated data for translation suggestion, while VDBA and our PSGD generate suggestions solely based on the original NMT model (released by Yang et al. (2021a)) without any additional training/fine-tuning on any TS-labeled data. Comparisons between SA-Transformer and TSMind have been reported in Yang et al. (2021a) and Ge et al. (2022), respectively. VDBA and PSGD are both implemented with the Fairseq toolkit (Ott et al., 2019) to generate suggestions. In evaluation,
we calculate SacreBLEU metric (Post, 2018) between the golden references for the masked spans and the alternatives generated by each system to measure their performances.
Results on the test set of WeTS are presented in Table 2. Notice that results of TSMind are not listed in Table 2. This is because TSMind is a system submitted to the WMT22-TS shared task. Its results on the WeTS dataset are those of systems trained with a large amount of augmented data. To demonstrate the performance of our designed unsupervised method, comparisons with models without data augmentation are more reasonable and much fairer. Therefore, we compared our model performances with the results of TSMind without data augmentation on the development set of WMT22-TS in Table 3. It can be observed that our PSGD outperforms VDBA sig-
nificantly by an average of 10.87 BLEU and an average of 8.62 BLEU separately on the two datasets for German-English, English-German, ChineseEnglish, and English-Chinese. Moreover, without any extra training, PSGD is even better than the supervised SA-Transformer and TSMind systems that are trained with TS-labeled data.

heading: 4.2 Patience for Early Stopping in Decoding
text: In PSGD, we involve a hyper-parameter pt, the number of extra decoding steps we take for early stopping. Larger pt can usually avoid omissions since when a local optimum is reached, the model should be encouraged to generate in more steps to ensure that it is globally optimal. However, a larger pt would slow down the decoding process.
To determine a proper value for pt, we investigate the model performances in SacreBLEU and the decoding time cost on the German-English development set of WeTS. The results are shown in Figure 4, which indicates that when pt increases from a relatively small value, the performance of PSGD will significantly increase as well. However, there is no more much gain when pt is over 5. Taking the trade-off between time efficiency and the suggestion quality into consideration, we empirically set pt = 5. For other language pairs, we found pt = 5 is also relatively optimal.

heading: 4.3 Experiments with Different Mask Ratios
text: Samples in the WeTS dataset have a short length of masked spans on average, whose masked spans
contain less than 10% words of the whole translation sentence as displayed in the histograms in Figure 5.
To fairly verify the performance of PSGD on longer masked spans, we consider applying various mask ratios on the translation datasets. We mask a span in the golden references with a random length ratio from 20% to 90% and obtain predictions for the spans by PSGD and VDBA separately. Then, we calculate SacreBLEU scores for comparison.
In practice, we use the WMT 2014 EnglishFrench test set (Bojar et al., 2014) and WMT 2021 English-German and German-English test sets (Akhbardeh et al., 2021) in the news translation for evaluation. PSGD and VDBA decode based on the publicly released NMT models, Ng et al. (2019) for English-German and German-English and Ott et al. (2018) for English-French.
Experimental results are shown in Figure 6. It shows that the proposed PSGD outperforms VDBA in all language pairs for each mask ratio. The performance of VBDA reaches the lowest point at the mask ratio of 40%, and then dramatically increases as the mask ratio increases. By contrast, the performance of PSGD appears more stable.

heading: 4.4 Computation Efficiency
text: In translation suggestion, compared with the decoding steps tp + tr + ts of Hu et al. (2019), PSGD is more efficient with tr + pt decoding steps when pt is smaller than tp + ts. To thoroughly evaluate the speed of PSGD’s decoding process, we
conduct a running-time cost analysis of translation suggestions with different mask ratios, relying on the WMT news translation test sets mentioned in Section 4.3. All experiments are run on a single Tesla P-100 GPU with CUDA V10.2. The target sequences are generated with a batch size of 10 and a beam size of 5. We obtain the time cost by averaging the running time of each sample in the test sets. Notice that times for pre-processings, such as tokenization and subwords segmentation, are not counted because these parts are identical for both methods. Hyper-parameter pt is empirically set to 5, the same value in the experiments shown in Figure 6.
The time costs of PSGD and VDBA are displayed in Figure 7. It is clear that PSGD runs faster than VDBA in all cases. As discussed in Section 4.3, in the scene of TS, the incorrect spans are usually short, which contain less than 10% of words in the whole translation sentences on average. In such a case, referred in Figure 7, PSGD can speed up at least 2x times compared with VDBA.

heading: 4.5 Robustness Test
text: In the above experiments, the given prefix and suffix constraints come from the golden translation references. While in the applications of CAT, prefix and suffix constraints could include erroneous words even if the incorrect span has been selected.
Therefore, it is necessary to carry out experiments where the prefix and suffix constraints contain potential errors. We assume that the performances of a robust decoding algorithm will not decrease too much in such a circumstance.
Again, we use the machine translation datasets mentioned in Section 4.3 and 4.4. The difference is that rather than masking golden references, we try to mask a span with random length in the MT result to get the prefix and suffix constraints. This setup is much closer to that of WeTS’s. In evaluation, the Sacre-BLEU score can only be calculated between the entire golden reference and the suggestions for the masked span alone with the prefix and suffix constraints, since we don’t have the parallel references for the masked spans.
Results of PSGD and VDBA are plotted in Figure 8. It illustrates that PSGD significantly outperforms VDBA in all the cases of mask ratios. When the masked span is extremely short, both algorithms will naturally not impact much on the whole sequence evaluation. On the other hand, when the masked span is extremely long, the decoding process of PSGD or VDBA is almost equivalent to that of the original NMT model. Therefore, in both cases, it is unsurprising to see that the gap between the performances of PSGD and VDBA is small. Except for these two situations, the performance of PSGD remains at a higher level than that
of VDBA. And VDBA reaches the lowest point when nearly half of MT words are masked.

heading: 4.6 Summary of Experiments
text: In this section, we conduct exhaustive experiments to evaluate the suggestion quality, time efficiency, and robustness of PSGD. PSGD significantly outperforms VDBA by an average increase of 10.87 BLEU and 8.62 BLEU on the benchmark datasets, WeTS and WMT22-TS, respectively. Experiments in time efficiency show its superiority by an overall 63.4% deduction on the WMT translation datasets. In robustness tests, PSGD remains at a higher level than VDBA all the time. Finally, on the TS benchmark datasets, PSGD is superior over two supervised TS systems, SA-Transformer and TSMind, which are trained with human-annotated data.

heading: 5 Conclusion
text: In this paper, we propose a neat prefix-suffix guided decoding (PSGD) algorithm for the translation suggestion task in computer-aided translation. It emphasizes the probability of the entire generation including prefix and suffix constraints and decodes only for incorrect spans selected by human translators with an early stopping mechanism. Given a pre-trained auto-regressive NMT model, PSGD can be easily applied during inference without any additional training/fine-tuning. Comprehensive experimental results demonstrate that PSGD significantly outperforms the state-of-the-art constrained decoding algorithm, VDBA, in all of the translation quality, time efficiency, and robustness. Meanwhile, it is also superior to supervised TS systems trained with human-annotated data.
Limitations
PSGD is a straightforward constrained-decoding algorithm for the translation suggestion task. How-
ever, the early-stopping mechanism involves extra time costs. Though PSGD is more efficient than VDBA in the scene of TS, where only two constraints (prefix and suffix) appear, it could be slower than VDBA if there were more short constraints. Besides, even if we take both prefix and suffix constraints into consideration for emphasis on the whole translation generation, the decoding process is still auto-regressive from left to right. The algorithm could be improved if we made better use of the information of suffix constraints. For example, how to apply PSGD on non-autoregressive models, such as (Gu et al., 2019) and (Yang et al., 2021b) will be our future work.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: , 2021b) will be our future work