SlowGuess/NAACL_2022_OCR huggingface dataset:
where, for simplicity,  $F(\hat{\mathbf{y}}_{1:t})$  and  $SLSS(\hat{\mathbf{y}}_{1:t},\mathbf{X})$  denote the normalised Sentence Fluency and SLSS scores respectively. The terms  $w_{F}$  and  $w_{S}$  are fixed weights controlling the relative importance of Sentence Fluency and SLSS, respectively. It is important to emphasise that the exponential function is applied to each score since linearly adding each score leads to high variance and lower correlation with human scores (Song et al., 2021).
Using the Pay-off Function to determine the effectiveness of generating a token  $(\hat{y}_t)$  at timestep  $t$ , the final reward function is defined using reward shaping as the difference between the current Payoff and the Pay-off of the previous timestep.
$$
R \left(\hat {y} _ {t}\right) = P O \left(\left[ \hat {\mathbf {y}} _ {1: t - 1}, \hat {y} _ {t} \right], \mathbf {X}\right) - P O \left(\hat {\mathbf {y}} _ {1: t - 1}, \mathbf {X}\right) \tag {23}
$$
In our experiments, we used the German-English OpenSubtitles corpus (Lison and Tiedemann, 2016). There are approximately 14 million sequence pairs in this dataset extracted from subtitles of movies and TV shows, making it very diverse. The dataset was then divided into training, validation and test sets. The training set has approximately 13 million sentence pairs while each of the validation and test sets has roughly 5,000 sentence pairs.
The trained models were also tested on translating the IWSLT 2014 German-English test dataset, a popular dataset to benchmark RL-based methods. This dataset contains a parallel corpora with one reference per source sequence, obtained from TED talks (Cettolo et al., 2015). The test dataset contains approximately 6,000 pairs, with each sequence containing a few sentences of text. See Appendix A.2 for justification for treating the two datasets as coming from different domains.
5.2 Training
Following (Bahdanau et al., 2016), to ensure good initialisation of the model, the actor network is first pre-trained using XENT and the teacher forcing method. After that, the critic is pre-trained using TD Learning with the fixed pre-trained actor weights. At the last step, we train the actor and the critic jointly.
Generally, the actor architecture follows the OpenNMT Transformer architecture (Klein et al., 2017), with a few enhancements to enable stepwise decoding during training (i.e., action selection based on the model's previous outputs) and diversity in each agent's exploration strategy (see Appendix A.3 for more details). During training the actor selects a token at each timestep using the Top-K sampling method (Fan et al., 2018), in which a token is sampled from K tokens with the highest probabilities.
The critic architecture follows the Transformer architecture (Vaswani et al., 2017), with two major differences (see Appendix A.3.2 for detailed explanation). There are also two critic types, namely Q-critic and V-critic. The first critic type, Q-critic, is used in model variants with Q-Learning while the second type, V-critic, is used in other variants utilising the A2C architecture.
Multi-GPU Training When training synchronously in a multi-GPU environment, the model is deployed on a one-model-per-device basis to reduce training time. Each model has its own optimiser (we used Adam (Kingma and Ba, 2014)). During every update, the gradients are reduced and re-scaled across all devices to ensure that they are consistent across the models.
However, when training asynchronously, the global agent resides on one GPU device while three parallel agents are deployed on the remaining GPU devices. Instead of using one optimiser per agent, only a global optimiser is used. On every asynchronous update, the global optimiser updates the global model by using the gradients transferred from the parallel agent which invoked the update. The global optimiser used (SharedAdam) is a standard Adam optimiser modified to support multi-GPU communication.
Our formulation of the unsupervised reward uses the pre-trained OpenAI GPT Language Model from Hugging Face (Wolf et al., 2020). It also uses the Sentence Transformers tool (Reimers and Gurevych, 2020) with the XLM RoBERTa
model (Conneau et al., 2020) to generate the sentence embeddings. We do not expect the performance to change drastically by using other models.
During the joint actor-critic training, it took approximately one day to train each of the BLEU variants while the training time for SURF variants ranges from 3 to 5 days (see Appendix A.5 for the computational resource used). The increase in training time is mainly due to the usage of the large pre-trained models by SURF. We expect this time to reduce if a smaller language model is used. We leave this investigation to future work.
More details on the implementation and hyperparameters are given in Appendix A.3 and A.4.
Model configurations. We experimented using the nine configurations listed below:
- Transformer baseline (MLE, no RL) - state-of-the-art (SOTA) model;
- Synchronous ACQ RL architecture with the standard BLEU reward (ACQ-BLEU) and with our SURF (ACQ);
- Asynchronous ACQ RL architecture with the standard BLEU reward (Async-ACQ-BLEU) and with our SURF (Async-ACQ);
- Synchronous A2C RL architecture with the standard BLEU reward (A2C-BLEU) and with our SURF (A2C);
- Asynchronous A2C RL architecture with the standard BLEU reward (A3Q-BLEU) and with our SURF (A3C).
Each model was trained on the OpenSubtitles dataset and tested on both OpenSubtitles (in-domain) and IWSLT (out-of-domain) test sets.
By choosing this selection of models we are able to do the following: (a) generate the benchmark result using the Transformer baseline; (b) exhibit the advantage of SURF over the BLEU reward; and finally (c) explore the performance of SURF within the family of the multi-agent models (ACQ, Async-ACQ, A2C and A3C).
We used the standard set of MT evaluation metrics: BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014). We performed significance testing via bootstrap resampling using the Multeval tool (Clark et al., 2011).
6.1 Performance on Automatic Evaluation Metrics
Results for the OpenSubtitles test set are in Table 1. They show that all of our model variants outperform the Transformer benchmark (+1.5 BLEU on average) with the A3C model performing the best (+2.5 BLEU vs. Transformer) while the Async-ACQ model is the second best performing variant being (+1.2 BLEU vs. Transformer). Mostly, our unsupervised reward contributes around +0.5 BLEU to the performance of each model when trained with the BLEU reward. Especially high improvement of +7.5 BLEU is observed for A3C. We attribute this result to the BLEU reward sparsity and hence impossibility to properly learn the relative improvement over the average for the actions as modelled by the advantage function. Note that Q-learning learns the absolute reward which is easier to model in this sparsity condition. This sparsity is particularly detrimental in the asynchronous setting, where the updates of parallel agents seem to exhibit too much variance to be useful. Those observations in the Advantage learning setting emphasise the important potential of our dense reward to elicit the benefits of different RL architectures.
Model	B	M	B	M
Transformer	33.3	29.9
BLEU	SURF
ACQ	34.6*	30.7*	35.2*†	31.0*†
Async-ACQ	35.0*	30.9*	35.5*†	31.1*†
A2C	32.9*	29.9*	33.4*†	29.9*
A3C (Async-A2C)	28.3*	27.9*	35.8*†	31.2*†
Table 1: Results for the OpenSubtitles German-English test set. We report BLEU (B) and METEOR (M) scores. The symbol  $\star$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the Transformer model while  $\dagger$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the BLEU variant of the same RL-based architecture. The best result is highlighted in bold.
To probe the generalisation capacity of our models in the out-of-domain scenario, we have applied our models to the IWSLT test set. As shown in Table 2, the performance drop for the Transformer model is much higher than for our RL models (-6 BLEU vs. -2.4 BLEU on average). The AsyncACQ is the best-performing model on both metrics with the ACQ model being the second best (+6.3 BLEU and +5.8 BLEU vs. Transformer, re
Model	B	M	B	M
MIXER‡ (Ranzato et al., 2015)	20.7	-	-	-
AC‡ (Bahdanau et al., 2016)	28.5	-	-	-
ERAC‡ (Dai et al., 2018)	29.0	30.6	-	-
SAC-BLEU‡ (Ive et al., 2021)	29.6	31.0	-	-
SAC-Unsuper‡ (Ive et al., 2021)	29.8	31.2	-	-
Transformer	27.3	29.5	-	-
BLEU	SURF
ACQ	32.4*	32.7*	33.1*†	33.1*†
Async-ACQ	32.8*	32.7*	33.6*†	33.3*†
A2C	29.4*	31.0*	30.2*†	31.0*
A3C (Async-A2C)	22.2*	27.6*	32.8*†	32.8*†
Table 2: Results for the IWSLT 2014 German-English test set. We report BLEU (B) and METEOR (M) scores. The symbol  $\star$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the Transformer model while  $\dagger$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the BLEU variant of the same RL-based architecture. The best result is highlighted in bold.
spectively). Mostly, our unsupervised reward contributes around  $+0.8$  BLEU to the performance of each model. Especially high improvement of  $+10$  BLEU is again observed for A3C showing the potential of SURF.
By way of offering a guideline of our model performance, we also report the scores of the previous SOTA on the IWSLT test set. Though those results are not directly comparable to our results as the pre-processing conditions are different: previous models have mainly applied a cut-off vocabulary implying the presence of unknown words in the training data while we are using the subword units that dispense us of the unknown words.
Note that, on both test sets, the asynchronous variants (Async-ACQ and A3C) performed better than their corresponding synchronous counterparts (ACQ and A2C respectively). We emphasise the potential of asynchronicity with our dense reward to positively influence performance.
Additionally, regarding the usage of Q- or V-critics: both Q-version Async-ACQ and V-version A3C have shown comparable performance on the OpenSubtitles dataset. However, the Q-version Async-ACQ has achieved better performance on the IWSLT test set. We hypothesise that this may be due to the fact that the Q-critic network in the ACQ architecture outputs the state-action values of the entire vocabulary at each timestep rather than a single state value (as in the V-critic network). Hence it performs a more fine-grained policy evaluation with lower variance in the critic outputs, leading to a more stable model overall. A more thorough investigation would lead to better insights.
6.2 Performance on Semantic-level Evaluation Metrics
As with the automatic evaluation results, similar conclusions could be drawn from the results of the assessment with the three semantic metrics used in the reward formulation: Sentence Fluency, Token-level Semantic Similarity (TLSS) and Sentence-level Semantic Similarity (SLSS) scores (See Appendix A.6 for the description of the TLSS score).
For the IWSLT test set, as shown on Table 4, there is a noticeable increase in the Fluency score across our models (in comparison to the Transformer). ACQ and Async-ACQ are also able to achieve distinctly better TLSS ans SLSS scores than the Transformer model. We observe that variants of ACQ and Async-ACQ models achieve similar performance. When comparing the BLEU and SURF variants, the BLEU variants of A2C and A3C models obtain higher Fluency scores but score less on TLSS and SLSS. This can be explained by fact that BLEU RL sentences are prone to be more verbose, repeating the same meaning in different words. The results for OpenSubtitles show similar tendencies (see Appendix A.7).
6.3 Human Evaluation
Finally, to gain deeper insights, we performed human evaluation on the translations of the OpenSubtitles and IWSLT 2014 test sets by the Transformer, Async-ACQ-BLEU and the best performing SURF variants (A3C for OpenSubtitles and Async-ACQ for IWSLT).
For this human analysis, we randomly selected 50 test samples from each test set. A rank of quality is assigned by the human evaluator (second author,
Source	all dies wurzelt in der mythologischen vergangenheit . das eigenartige an diesen große häusern , in denen aufgrund der mischehen sechs oder sieben sprachen gesprochen werden , ist jedoch , dass man niemals{jemanden hört, der eine sprache lert.
Target	and this is all rooted in the mythological past , yet the curious thing is in these long houses , where there are six or seven languages spoken because of intermarriage , you never hear anyone practicing a language .
Transformer	all this mythology in the mythological context , which is curious about these great houses , judging by the patter of six or seven languages , however , you never hear anyone learning a language .
Async-ACQ-BLEU	the strange thing about these big houses where they speak six or seven languages , is that you never hear anyone who learns a language .
Async-ACQ	all this rambling in the mythological past , the curious thing about these big houses where they speak six or seven languages based on the basic language is that you never hear anyone who learns a language.
Table 3: Examples of translation generated by Transformer, ACQ-BLEU and Async-ACQ. We also report the original source sequence (SOURCE) and its reference (TARGET). The best translation is highlighted in italics.
Model	Fluency	TLSS	SLSS	Fluency	TLSS	SLSS
Transformer	1.024	2.454	2.339	-	-	-
BLEU	SURF
ACQ	1.029*	2.456*	2.350*	1.029*†	2.457*†	2.357*†
Async-ACQ	1.032*	2.455*	2.350*	1.029*†	2.456*†	2.350*†
A2C	1.035*	2.451*	2.300*	1.027*†	2.454*†	2.339*†
A3C (Async-A2C)	1.047*	2.422*	2.267*	1.027*†	2.454*†	2.342*†
Table 4: Results for the IWSLT 2014 German-English test set. We report Sentence Fluency, Token-level Semantic Similarity and Sentence-level Semantic Similarity scores. Also, the symbol  $\star$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the scores of Transformer model while  $\dagger$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the BLEU variant of the same RL-based architecture. The best result is highlighted in bold. Note that some of the improvements are beyond the displayed precision of 3 decimal points.
Test Set	Transformer	BLEU	SURF
OS	0.10	0.08	0.20
IW	0.06	0.48	0.60
Table 5: Human ranking results comparing the OpenSubtitles (OS) test outputs for Async-ACQ-BLEU and A3C and the IWSLT 2014 (IW) test outputs for Async-ACQ-BLEU and Async-ACQ. The best result is highlighted in bold.
fluent speaker of both English and German) from 1 to 3, allowing ties. Following the common practice in MT, each system was then assigned a score which reflects how often on average it was judged to be better or equal to other systems (Bojar et al., 2017). Table 5 shows that most of our variants have higher evaluation scores than the Transformer model. In particular, on the IWSLT test set, both Async-ACQ variants outperform the Transformer by a large margin. As compared to the best BLEU model, the A3C and Async-ACQ models perform significantly better on both the OpenSubtitles and the IWSLT test sets (+0.12 point). Table 3 shows translations generated by the three models on the IWSLT test set. Note that Async-ACQ demonstrates the best fluency and adequacy.
We have presented SURF, a new unsupervised semantic-level reward function, efficiently addressing the reward sparsity issue and mimicking human evaluation by considering both sentence fluency and semantic similarity. We have explored this reward for a new family of Actor-Critic Transformer-based Architectures with synchronous and asynchronous variants that promote the exploration of the search space. We demonstrate that SURF shows strong potential to elicit the benefits of various RL architectures. Our results show that it outperforms the traditional sparse BLEU reward for the same architectures in the automatic, semantic-level and human evaluation. Our code is available at https://github.com/AtomAnu/SURF.
There are several directions to take our work further: we can investigate the utility of our reward for other architectures and we can also explore different sampling strategies for each of the agents of our multi-agent models. Finally, we have investigated only two datasets to ensure comparability to the existing benchmarks. A more thorough investigation with more datasets and language pairs is needed to fully assess the scope of our contribution.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. Cite arxiv:1409.0473Comment: Accepted at ICLR 2015 as oral presentation.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? . FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169-214, Copenhagen, Denmark. Association for Computational Linguistics.
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and Marcello Federico. 2015. Report on the 11 th iwslt evaluation campaign, iwslt 2014.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations.
Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176-181, Portland, Oregon, USA. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Zihang Dai, Qizhe Xie, and Eduard Hovy. 2018. From credit assignment to entropy regularization:
Two new algorithms for neural sequence prediction. arXiv preprint arXiv:1804.10974.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. pages 889-898.
Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1347-1354, Online. Association for Computational Linguistics.
Sonal Garg, Sumanth Prabhu, Hemant Misra, and G. Srinivasaraghavan. 2021. Unsupervised contextual paraphrase generation using lexical control and reinforcement learning.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehension. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4099-4106. International Joint Conferences on Artificial Intelligence Organization.
Julia Ive, Zixu Wang, Marina Fomicheva, and Lucia Specia. 2021. Exploring supervised and unsupervised rewards in machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1908-1920, Online. Association for Computational Linguistics.
Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1673-1681, Online. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
Vijay Konda and John Tsitsiklis. 2001. Actor-critic algorithms. Society for Industrial and Applied Mathematics, 42.
Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portož, Slovenia. European Language Resources Association (ELRA).
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799-2808, Florence, Italy. Association for Computational Linguistics.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA. PMLR.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179-1195.