TeCS: A Dataset and Benchmark for Tense Consistency of Machine Translation
Year: 2023

Abstract:
Tense inconsistency frequently occurs in machine translation. However, there are few criteria to assess the model’s mastery of tense prediction from a linguistic perspective. In this paper, we present a parallel tense test set, containing French-English 552 utterances1. We also introduce a corresponding benchmark, tense prediction accuracy. With the tense test set and the benchmark, researchers are able to measure the tense consistency performance of machine translation systems for the first time.

heading: 1 Introduction
text: Translation tools are often found in a variety of social situations to enable cross-linguistic communication. Tenses are used to express time relative to the moment of speaking. Human translators frequently pay close attention to tense correspondence (Gagne and Wilton-Godberfforde, 2020). Similarly, machine translation (MT) systems are supposed to maintain temporal consistency between the original text and the predicted text to avoid misunderstandings by users. However, accurately keeping the tense consistency is undoubtedly difficult. Taking French-English (one of the most classic language pairs for MT) as an example in Table 1, the original text is in plus-que-parfait de l’indicatif of French, corresponding to the past perfect tense in English, while the English prediction provided by Google Translator is in the past simple tense.
In fact, this is not an isolated case. You can also find several examples in Appendix B. Besides. the translation mechanics may not the only reason leading to tense inconsistency. The corpora matter as well. For example, we have extracted 20,000 pairs English-French parellel sentences from the widely used dataset Europarl (Koehn, 2005), and
∗Corresponding author 1The following updates will be shown at: https://github.com/rutilel/
TeCS-A-Dataset-and-Benchmark-for-Tense-Consistency.
we have observed all groups of parallel utterances where the original French texts are in the plus-queparfait de l’indicatif tense, examining the tenses of their English counterparts. As a sentence may include several tenses, there are 195 occurences of plus-que-parfait tense in total. Among them, only 35.28% English sentences are in the correct past perfect tense, as shown in Table 2. Although, compared to other tense correspondences, the pair of plus-que-parfait and past-perfect is prone to error in datasets and there are only 0.94% of sentences in Europarl are in plus-que-parfait, we cannot easily ignore this issue. Like Europarl, tense correspondences are generally credible but unreasonable for certain tenses in several common datasets.

heading: Tense of Counterpart Proportion
text: In addition to the train set, the difficulty of remaining tense consistency also stems from the lack of metrics on measuring the model’s mastery of tense information. The research of Marie et al.
1930

heading: French Tenses English Tense Format Example
text: (Where the modal verb contains can, may, shall, must, could, might, should and would.)
(2021) shows that 98.8% of *ACL papers2 in the field of MT from 2010 to 2020 used BLEU (Papineni et al., 2002) scores to evaluate their models. However, the reliability of BLEU has been questioned in the era of neural machine translation (NMT) as its variants only assess surface linguistic features (Shterionov et al., 2018), and many studies have shown that BLEU has difficulty in portraying the degree of semantic information mastered by the model, i.e. its score does not necessarily improve when more semantic information is mastered (Mathur et al., 2020; He et al., 2023), not to mention specific tense information. We have also applied BLEU to measure various baselines on our tense test set in Section 4, and the results explicitly support the above statement. In addition, reviewing the evaluation criteria related to MT tasks over the past ten years, we are surprised to find that there are no criteria to assess the model’s mastery of tense prediction from a linguistic perspective.
Therefore, our paper is devoted to the study of NMT based on semantic understanding in terms of tense. We construct a tense parallel corpus test set consisting of 552 pairs of tense-rich, error-prone parallel utterances for NMT systems, and then propose a new task for evaluating the effectiveness of model translations from the perspective of tense consistency. This paper makes three contributions: (1) the presentation of the construction of the tense test set, including its tense labels; (2) the proposal of a feasible and reproducible benchmark for measuring the tense consistency performance of NMT systems; and (3) the various experiments for different baselines with the above test set and corresponding benchmark.
2The papers only includes *ACL main conferences, namely ACL, NAACL, EACL, EMNLP, CoNLL, and AACL.

heading: 2 Annotation Rules and Tools
text: As the first work of the MT tense study, we choose English-French, one of the most classic language pairs of MT, to construct the dataset3.
TENSE, the dominant topic of our research, is a combination of tense and aspect. In the modern grammar system of English, “a tense system is a system associated with the verb where the basic contrasts in meaning have to do with the location in time of the situation, or the part of it under consideration” (Huddleston et al., 2021). The modern grammatical system divides tense into present and preterit based on the inflections added to the end of verbs, and the aspect into perfective and progressive on the state where an action is (Kamp, 1991). While this tense classification system is too crude for daily life, we therefore apply the following classification methods. On the one hand, we classify the tenses according to the macro-temporal interval of the action into three major time intervals, namely present, past and future tenses; on the other hand, we classify the tenses according to the state of the action into general, progressive and perfect aspects. Hence, 9 kinds of tenses are born through combining the three tenses and the three aspects.
French and English belong to the same IndoEuropean language family and share many similarities in various respects. The main difference is that in French there is another grammatical point called mode, part of which is like the aspect in English. In terms of tenses, we will generally discuss the tenses in the indicative mode of French and will describe the others later in this section. In the following, if there is no mode qualifier before a tense, it is by default in the indicative mode. Careful identification and comparison of the subdivided tenses in the three main tense intervals, English and French, reveals a very similar usage of the tenses, as sum-
3Please refer to the Limitations for more details.
marised in Table 3. As there is no progressive tense in French, we do not distinguish the progressive tense in English, but rather merge the progressive tense into its corresponding base tense, e.g. the present perfect progressive tense into the category of the present perfect tense.
When discussing tenses from a semantic point of view, the modes also need to be taken into account. The grammatical correlations between French and English modes are quite complicated. Considering the corresponding grammatical expressions of 2 modes strongly related to tense, conditionnel and subjonctif, in French rely on the usage of modal verbs, we introduce modal verbs to simplify the distinguishment of the modes.
Based on these grammatical rules, we merge the nine common tenses in English into seven categories that correspond reasonably and rigorously to French, namely the 6 tense categories of past/present/future + simple/perfect and statements containing modal verbs that correspond to the French subjonctif and conditionnel tenses. We construct an automatic annotation method based on the spaCy package (Honnibal et al., 2020). First, we label the grammatical components of each word in the sentence based on the spaCy package, and then we define and compare the grammatical structures of the verb phrases with the structures of each tense classification to derive the sentence tense labels. During this process, to simplify the annotation process and better correspond with French futur proche tense, we classify the expression ‘be going to do’, grammatically in Future tense, into the Present tense, just like expressions ‘be about to do’ and ‘be + verb progressive’, whose stucture are in Present tense but the real meaning is about the close future. Also, a sentence may have several tense structures, in this case, the tense label consists several tenses. For example, the label of the sentence ‘So it is in that spirit that we have made this change.’ is ‘Present+PrePerfect’.

heading: 3 Corpus Design and Characteristics

heading: 3.1 Corpus Design
text: We choose the tense-rich Europarl, namely EuroparlPV, processed by Loáiciga et al. (2014) as the source corpus, for it contains all the sentences with predicate verb structures in the original Europarl dataset (Koehn, 2005). First, we cleaned the source corpus, including deleting sentences without counterparts, English sentences in the French

heading: Classfication Times Proportion
text: part and vice versa. After this, we obtain 201,374 tense-rich parallel French-English sentence pairs, namely EuroparlTR. We randomly divided them into a training set, a validation set and a test set in the ratio of 8:1:1, and trained a transformer baseline based on this using fairseq (Ott et al., 2019) with a BLEU value of 33.41. Then we compared a total of 20,000 parallel sentences’ triples (original Europarl French text, original Europarl English text, transformer English prediction).
In the construction process, with the code mentioned in Section 2, we first automatically annotated the original English text and English prediction in the 20,000 pairs of parallel utterances, given the corresponding tense labels. Then, we filtered 6,779 parallel French-English sentence triples with different tense labels for English originals and predictions. On the basis of the automatic selection, we manually screened out the representative parallel French-English sentence pairs with a certain degree of translation difficulty and a complex grammatical structure. We also corrected the reference translations that did not justify the tense or semantics. It is worth noting that the author has a level of English and French that meets the C1 standard of The Common European Framework of Reference for Languages (CEFR), representing the ability to express herself effectively and flexibly in English and French in social, academic and work situations. A total of 570 parallel pairs of statements were selected at this stage.
Following this, two other reviewers at CEFR C1 level, reviewed the tense test set for semantic and tense correspondence, and the tense labels marked by the automatic annotation code. The tense test set was further refined. The final test set contains 552 parallel French-English sentence pairs. You can see more details in Appendix D.

heading: 3.2 Corpus Characteristics
text: In the following paragraphs, we describe the statistical features of our corpus and the elimination of gender coordination influence.
Tense distribution. The corpus consists of 780 tense structures in 552 sentences, and the distribution of tense classifications is shown in Table 4. In the test set, sentences in present tense are the most, corresponding the situation of the reality: we use present tense most frequently and future perfect sense least frequently.
Elimination of gender effect. Unlike English, gender coordination exists in French. For example, the French sentences ‘Nous nous sommes donc abstenus.’ and ‘Nous nous sommes donc abstenues.’ both correspond to the English ‘We therefore abstained.’. That is, the MT system’s ability to learn gender coordination affects its ability to recognize tense structures, which in consequence affects the maintenance of tense consistency between original French text and predicted English sentence. Therefore, to better measure the tense-predicting capability of different MT systems, rather than their ability to recognize pronominal gender, we controlled for the gender variable by defaulting all pronouns, which do not indicate explicitly their genders, as masculine. These pronouns consists of 167 je (I), 114 nous (we, us) and 28 vous (you).

heading: 4 Experimental Results
text: To measure the tense consistency performance of different systems, we introduce a benchmark called tense (prediction) accuracy, as shown in Eq. (1).
Accuracy = Nc Nt , (1)
where Nc is the number of predicted utterances with the same tense as its reference and Nt is the total number of utterances in the tense set.
To verify the validity of our tense corpus, the following approach was adopted: To begin with, 100, 000 parallel utterance pairs from the EuroparlTR (containing 201, 374 pairs) mentioned in Section 3.1 were extracted as the tense-rich train set, and 100, 000 parallel utterance pairs from the Europarl corpus (Koehn, 2005) were extracted as the tense-poor train set. There were no overlapping utterances between the latter and the former. We performed the same preprocessing procedure, including data cleaning, tokenization and BPE coding. We then trained four pairs of French-English NMT systems with different architectures based on fairseq (Ott et al., 2019), where two systems in each pair differed only in the train set. After this, we summarized the scores evaluated by SacreBLEU (Post, 2018) and COMET (Rei et al., 2020) and tense prediction accuracies of the eight systems on different test sets. We have applied three types of test sets: our tense set, the Europarl test set and the WMT15 test set. The Europarl test set contains 3,000 parallel utterance pairs drawn from the Europarl corpus, the exact same field of train set, while the WMT15 is a test set for the WMT15 (Bojar et al., 2015), deriving from data in the different field of train set. Besides, we also apply our approach to mesure the tense consistency performance of several business translators, includ-
ing Bing Translator, DeepL Translator and Google Translator. The results are listed in Table 5:
1) The BLEU and COMET scores based on the Europarl set and the WMT15 set are quite similar for each system pair, which indicates that the translation capabilities of the two systems are similar in the general evaluation dimension. This suggests that by relying solely on the difference in BLEU scores on traditional test sets, we are unable to measure the tense prediction ability of the systems.
2) However, there are large differences in our tense set. The tense consistency performance of systems trained on the tense-rich train set was significantly better than that of systems trained on the tense-poor train set. This indicates that our tense set can capture the tense consistency performance.
3) Further investigation of the BLEU or COMET) scores and tense prediction accuracy for each system reveals their positive correlation for the same architecture, but not across architectures. To measure the tense consistency performance across different architectures, we should focus more on tense accuracy rather than BLEU scores only.

heading: 5 Conclusion
text: We presented the French-English parallel tense test set and introduced the corresponding benchmark tense prediction accuracy, providing a brand-new approach to measure the tense consistency performance of machine translation systems. This test set firstly focuses on the tense prediction ability, posing a new dimension to improve the MT quality.
In the future, we will endeavour to generalize the test set to other languages. Considering there are statements like "the use of tense A in language X is equivalent or similar to the use of tense B in English" in grammar books of other languages(Durrell et al., 2015), even across language families(Gadalla, 2017) and human translators also apply such rules(Santos, 2016), we are confident in taking this forward.

heading: Limitations
text: In this work, we focus on creating the EnglishFrench tense corpus. These two languages are among the most frequently and widely used languages in the world. In addition, they have several similarities in tenses, which are pretty helpful for research on tense consistency through machine translation. Thanks to the distinctive tense struc-
tures, the study of these two languages makes it possible to examine many common tense issues, but there are also some tense issues in other languages that are not covered by this study. For example, the implicit tense expressions in Chinese are difficult to correspond to the explicit tense expressions in English (Jun, 2020). Hence, our next step will be to extend the tense test set to other language families and even cross-language families to further study tense consistency. Also, as for future work, we will optimize both the tense annotation method and the tense prediction accuracy calculation. Besides, we did not propose a new method to improve the tense prediction accuracy. To be further, we will endeavour to improve the existing machine translation systems according to tense consistency.

heading: Acknowledgements
text: Yiming, Zhiwei, and Rui are with MT-Lab, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, and also with the MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200204, China. Rui is supported by the General Program of National Natural Science Foundation of China (6217020129), Shanghai Pujiang Program (21PJ1406800), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), Beijing Academy of Artificial Intelligence (BAAI) (No. 4), CCF-Baidu Open Fund (No. CCF-BAIDU OF2022018, and the Alibaba-AIR Program (22088682). We also thank the computational resource from the SJTU student innovation center.

heading: Ethics Statement
text: Our tense test set is based on the widely used public corpus Europarl in the field of machine translation. In creating this test set, we only corrected tense and description errors of some English references and did not change the original semantics, so there are no ethical issues arising.

heading: A Online Translation
text: • Google Translator: https://translate. google.com/ as of December of 2022.
• Bing Translator: https://www.bing.com/ translator as of December of 2022.
• DeepL Translator: https://www.deepl. com/translator as of December of 2022.

heading: B Examples of Translators’ Errors
text: Table 6 shows several translating errors of common business translators. The display form is a group of five sentences: original French sentence, corresponding English reference, Bing translation, DeepL translation, and Google translation.

heading: C Examples of Baseline Prediction Errors and Corresponding Annotations
text: Table 7 shows several examples of predictions and corresponding annotations of baselines in Section 4. Each group consists ten sentences, which are original French sentence, corresponding English reference, Transformer(tense-rich) prediction, Transformer(tense-poor) prediction, LSTM(tenserich) prediction, LSTM(tense-poor) prediction, CNN(tense-rich) prediction, CNN(tense-poor) prediction, Bi-Transformer(tense-rich) prediction and Bi-Transformer(tense-poor) prediction.

heading: D Additional Notes On Human Review

heading: D.1 Recruitment of Human Reviewers
text: We recruited reviewers from students majoring in French. Taking Diplôme Approfondi de Langue Française(DALF) C1 French exam results, International English Language Testing System(IELTS) exam results, and their GPA in French courses into account, we recruited 2 reviewers in the same country of the authors’ at last.
D.2 Instructions Given to Reviewers
We offer the annotation rules in Section 2, and require the reviewers to accomplish the following tasks：
• Determine whether the tense of the English translation is accurate and reasonable. If not, give an English translation that you consider reasonable.
• Determine whether the meaning of the English translation is correct. If not, give an English translation that you consider reasonable.
• Determine whether the corresponding tense label of the English translation is correct according to the natural language understanding.

heading: E Experimental Setup

heading: E.1 Model
text: Table 8 provides the number of parameters, training budget, and hyperparameters of each model. All experiments were performed on a single V100 GPU and the hyperparameters are by default. We report the result of a single run for each experiment.

heading: E.2 Data
text: Table 9 shows the data statistics we used in this paper.

heading: E.3 Packages
text: Table 10 shows the packages we used for preprocessing, model training, evaluation and tense labeling.

heading: Sentence Tense Origin: J ’avais considéré que Mme Lulling était une Luxembourgeoise.
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations
3 A2. Did you discuss any potential risks of your work? Section Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1,2,3,4
3 B1. Did you cite the creators of artifacts you used? Section 1,2,3,4 and Appendix E
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 2,3,4 and Appendix E
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section Ethics Statement
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section Ethics Statement
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3 and Appendix D, E
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3,4 and Appendix E
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix E
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Appendix E
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2,4 and Appendix E
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.1
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix D
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix D
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix D and Section Ethics statement
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section Ethics statement
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Appendix D

heading: Future Work
text: Also, as for future work, we will optimize both the tense annotation method and the tense prediction accuracy calculation. Besides, we did not propose a new method to improve the tense prediction accuracy. To be further, we will endeavour to improve the existing machine translation systems according to tense consistency