Refer to the Reference: Reference-focused Synthetic Automatic Post-Editing Data Generation
Authors: Sourabh Dattatray Deoghare, Diptesh Kanojia, Pushpak Bhattacharyya


Reviewer paper summaries: To train an automatic post-editing (APE) system for machine
translation (MT), large scale synthetic training data have been
generated from a bilingual parallel corpus and used for pre-training.
However, two translations, i.e., an MT output for a source sentence
and a reference translation for the same source in the parallel
corpus, are inherently independent and thus would lead to only
suboptimal impact.  This paper proposes to generate an MT output from
the reference translation of the given parallel corpus to better
approximate the phenomena between actual MT output and its post-edited
version.  The method first paraphrases given reference translation,
and then performs round-trip translation.  An experiment shows that an
APE system trained on the synthetic data generated by the proposed
method performs better, according to translation edit rate (TER),
compared to a system trained on synthetic data generated by an
existing method.

This paper introduces a novel and simple yet effective reference-focused synthetic APE data generation technique. APE requires triplet element data (src, mt, pe) to train an APE model, it requires the causal relationship between mt and pe sentences. However, due to its sparse nature, the conventional APE model employs a data augmentation strategy to generate the surrogate data. The data construction strategy leverages the NMT translation results as the surrogate mt, and the target sentence as pe, which naturally breaks the causal relationship between mt and pe sentences. To address such an issue, this paper proposes a novel reference-centered synthetic data generation strategy. It leverages the paraphrase model and round-trip translation to obtain mt. Consequently, the causal relationship between mt and pe is preserved. Experimental results demonstrate its effectiveness.
Summary of strengths: The proposed method, depending on the translation direction and configuration, leads to better translation quality measured by TER.

- The motivation is reasonable and the corresponding proposed strategy is sound and effective in principle.
- Experimental results show strong performance across baselines.
Summary of weaknesses: - The paper misses to compare the proposed method with an existing
  method using a masked language model
  (https://aclanthology.org/2021.eacl-main.50/).
The above paper proposed a method to generate MT-like texts from reference translations in a parallel corpus. The resulting data set has the same form as in this paper, i.e., (src, mt, ref), where ref is regarded as pe, although their purpose for the data set is not the same. Therefore, to confirm the effectiveness of revisiting round-trip translation in the proposed method, the authors should have included this method in the comparison.

- The core part of the proposed method, formulation of paraphrasing
  has flaws.
(a) In l.225, $S$ is defined as a sequence of $n$ words; on the other hand, in l.227, $S$ is referred to as a set with a $\notin$ operation.
We may see this way of notation in many other papers.
However, this does not mean we can do it;  without a precise definition of the operation, readers interpret it arbitrary.
(b) In l.227 introduces another sequence $S'$ consisting of $m$ words as an output of a paraphraser.
Here, only the last word is conditioned, i.e., $\exists s'_{m} \notin S$.
Therefore, for a given $S = (a, b, c)$, another sequence, such as $(a, e, b)$, is not regarded eligible for $S'$.
If the authors intend so, while it is theoretically not problematic, it does not make sense as the output of a paraphraser.
On the other hand, if the authors does not intend so, this is a flaw.

- As confirmed by an ablation experiment (Table 2), the paraphrasing
  module is an indispensable piece of the proposed method.  However,
  Section 4 (ll.217-232) does not explain requirements for the
  paraphraser and Section 6 does not analyze to what extent the final
  results depend on the quality of the paraphraser.

- Presentation has a large room for improvement.  For example, Figures
  1 and 3 should not be shown before they are mentioned.  Motivation
  of several system configurations are first/verbosely explained in
  the result section (Section 6) but they should be clearly explained
  in Section 5.  Reference for Dale (2021) is broken.  While showing
  TER gains in Abstract and Section 1, they are derived from three
  different models, where model selection is carried out on test set.

- Although the proposed strategy preserves the causal relationship between me and pe, it breaks the causal relationship between src and mt as well. I do see the novelty of the strategy, however, authors should at least demonstrate or discuss such a trade-off relationship between preserving src-mt or mt-pe. Such a discussion is valuable for real-world applications of the APE system.
- The proposed strategy seems to have a quite large computation overhead compared to traditional strategies. Is it worth applying the proposed strategy given the a certain computation budget?
Comments, suggestions and typos: - Figure 2 is strange: src and pe must be passed from "Existing
  Synthetic APE Data" to "New Synthetic APE Data."

- There are several massively multilingual pre-trained models.  They
  can be theoretically usable as a paraphraser, and this would be the
  motivation for the authors to claim the applicability of the
  proposed method.  However, do they also improve the APE quality as
  much as in this paper?  If it is not guaranteed, what are the keys
  for a success?

- A comparison between "Existing Synthetic x2" and "Existing +
  Proposed Synthetic" suggests that diversity improves the
  performance.  As demonstrated in the literature of MT, using
  multiple decoded results (https://aclanthology.org/W18-2707/) would
  be helpful to APE.

See weakness
Overall assessment scores: 2

3.5