SlowGuess/NAACL_2022_OCR huggingface dataset:
Yurun Song, Junchen Zhao, and Lucia Specia. 2021. SentSim: Crosslingual semantic evaluation of machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3143-3156, Online. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
R. S. Sutton and A. G. Barto. 2014. Reinforcement Learning: An Introduction, second edition edition. Cambridge, Massachusetts: The MIT Press.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 2-6. Curran Associates, Inc.
Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544-3552, Online. Association for Computational Linguistics.
Shuo Wang, Zhaopeng Tu, Zhixing Tan, Shuming Shi, Maosong Sun, and Yang Liu. 2021. On the language coverage bias for neural machine translation. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 4778-4790, Online. Association for Computational Linguistics.
Xiangpeng Wei, Yue Hu, Luxi Xing, and Li Gao. 2019. Unsupervised neural machine translation with future rewarding. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 281-290, Hong Kong, China. Association for Computational Linguistics.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
A.1 Ethics Considerations
We are aware of the discussions around the risks related to unintended harmful effects and uses, environmental consequences, fairness and privacy considerations of large language models in general (Bender et al., 2021), and machine translation models specifically (Wang et al., 2021). We note here that our models constitute a primarily theoretical contribution and were trained and tested on standard datasets. Before deployment in a production setting our methodology is subject to retraining with data pre-processed in the appropriate way (as our model is not equipped with relevant security, privacy and fairness mechanisms), systematic debugging, extensive simulation, testing and validation under the supervision of experts.
A.2 Domain Distance
To justify that the IWSLT test set is indeed considered out-of-domain, we have trained a German language model using the source sentences (in German) from the OpenSubtitles training set. For this we used the fairseq toolkit (Ott et al., 2019). The resulting difference in language model perplexity values for the OpenSubtitles and IWSLT test sets (45.52 and 555.71, respectively) is important enough to justify that IWSLT is considered out-of-domain.
A.3.1 The Actor
OpenNMT Transformer Implementation In the OpenNMT framework (Klein et al., 2017), the Transformer architecture is implemented slightly differently from the original architecture in
(Vaswani et al., 2017). Its implementation follows the up-to-date implementation of the tensor2tensor framework (Vaswani et al., 2018), created by the authors of (Vaswani et al., 2017). The main difference lies in the normalisation technique used in the Transformer. That is, pre-normalisation is applied in each sub-layer of the encoder and the decoder instead of post-normalisation. The output of each sub-layer with pre-normalisation can be expressed as follows:
$$
x + \text {S u b l a y e r} (\text {L a y e r N o r m} (x)) \tag {24}
$$
In the original architecture where postnormalisation is used, layer normalisation (LayerNorm) is applied after the summation  $(x + Sublayer(x))$ , as shown below:
$$
\operatorname {L a y e r N o r m} (x + \text {S u b l a y e r} (x)) \tag {25}
$$
Step-wise Decoding and Exploration Strategies During the joint AC training, instead of just computing the policy distributions as the output, the actor would perform step-wise decoding by selecting a token to generate at each timestep, given the encoded source sequence and the previously generated tokens. As mentioned before, this is done to ensure that there would be no exposure bias during inference as the model is trained to condition the generation process using its own outputs.
To allow each agent to be diverse in their exploration strategies, the actor can operate in two possible main modes of token selection. In the first mode, the actor selects a token at each timestep using the Top-K sampling method (Fan et al., 2018), in which a token is sampled from K tokens with the highest probabilities. In the second mode of operation, the actor performs token selection based on Nucleus sampling (Holtzman et al., 2020). In the Nucleus sampling method, a token is chosen from the smallest possible set of tokens that has an accumulated probability equal or higher than a pre-defined probability value  $(p)$ . For instance, if  $p$  is set to 1, the actor would perform token selection from the entire vocabulary. Similarly, if the value of  $K$  for Top-K sampling is set to the vocabulary size, the actor would sample from the entire vocabulary as well.
The actor also incorporates the notion of Temperature to further increase the diversity of exploration strategies. A pre-defined value of Temperature (temp) is used to increase the probability of
probable tokens while reducing others that are not probable.
$$
p \left(w _ {i} \mid \hat {\mathbf {y}} _ {1: t - 1}\right) = \frac {\exp \left(l _ {i} / \text {t e m p}\right)}{\sum_ {j} \exp \left(l _ {j} / \text {t e m p}\right)} \tag {26}
$$
where  $l_{i}$  is the logit of the token  $w_{i}$ . One can observe that, as temp decreases, the probability of probable tokens would increase. If temp is set to 1, the above expression would simplify to the normal softmax operation.
We have empirically found that the sampling Temperature should not be applied in conjunction with Top-K or Nucleus sampling as it leads to highly greedy policies, especially when K or  $p$  is already low. It should be applied on when the agent samples from the entire vocabulary. After experimenting with different configurations of exploration strategies, we found that Top-K sampling was the most effective.
A.3.2 The Critic
The critic architecture follows the Transformer architecture, with two major differences.
Reference Encoding The first difference between the actor and the critic is that the encoder of the critic encodes the reference sequences instead of the source sequences. The reason is to allow to critic to evaluate each generated sequence by comparing with its reference sequence.
Output Layer The second difference is the output layer used in the critic. In the Q-critic model, its output layer is a one-layer feed-forward network with the output dimension equal to the vocabulary size of the target language. This is because the Q-critic model outputs the state-action value (i.e., Q-value) for every word in the vocabulary.
For the V-critic model, its output layer also contains a one-layer feed-forward network with the output size of 1. Given a generated sequence, the critic outputs the state value (i.e., V-value) for each token in that sequence.
A.4 Hyper-parameters
A.4.1 Actor Pre-training
Table 6 lists all the hyper-parameters used during actor pre-training.
A.4.2 Critic Pre-training
Table 7 lists all the hyper-parameters used during critic pre-training.
Hyper-parameter	Value
Optimizer	Adam
Adam Beta 1	0.9
Adam Beta 2	0.998
Learning Rate	2
LR Decay Method	noam
Warmup Steps	6000
Batch Size	4096
Gradient Accumulation Steps	3
Source Vocabulary Size	100000
Target Vocabulary Size	100000
Word Embedding Size	512
Hidden Layers Size	512
Encoder Layers	6
Decoder Layers	6
Attention Heads	8
Table 6: List of the hyper-parameters used during the actor pre-training stage.
Hyper-parameter	Value
Optimizer	Adam
Adam Beta 1	0.9
Adam Beta 2	0.998
Learning Rate	0.001
LR Decay Rate	0.9
LR Decay Steps	1000
Batch Size	4096
Gradient Accumulation Steps	3
Source Vocabulary Size	100000
Target Vocabulary Size	100000
Word Embedding Size	512
Hidden Layers Size	512
Encoder Layers	6
Decoder Layers	6
Attention Heads	8
γ (Discount Factor)	0.99
λvar (Q-critic)	0.25
Multi-step Return (V-critic)	5
wF (Sentence Fluency Weight)	1
wS (SLSS Weight)	1
Table 7: List of the hyper-parameters used during the critic pre-training stage.
A.4.3 Joint Actor-Critic Training
Table 8 lists all the hyper-parameters used during synchronous and asynchronous joint Actor-Critic training.
A.5 Computational Resource
Each of our models was trained on a GPU-accelerated instance with four NVIDIA V-100 SXM2 GPUs.
A.6 Token-level Semantic Similarity
The Token-level Semantic Similarity (TLSS) score is used as one of the semantic-level evaluation metrics. It can be used as an additional score metric in the reward function as well. TLSS measures the semantic similarity between tokens in the gener-
Hyper-parameter	Value
Training Mode	Sync	Async
Optimizer	Adam	SharedAdam
Adam Beta 1	0.9	0.9
Adam Beta 2	0.998	0.998
Actor LR	0.000001	0.000001
Critic LR	0.00001	0.00001
LR Decay Rate	0.9	0.9
LR Decay Steps	1000	1000
Batch Size	2000	2000
Grad. Accum.	1	1
γ	0.99	0.99
λxent	0.01	0.01
λvar	0.25	0.25
Multi-step Return	5	5
wF	1	1
wS	1	1
K	300	[100,200,500]
ε	0.2	-
Table 8: List of listing the hyper-parameters used during synchronous and asynchronous joint Actor-Critic training stage.
ated sequence and its source sequence. Following the methodology adopted in the SentSim evaluation metric (Song et al., 2021), each token in the source and generated sequences is passed to a cross-lingual language model to obtain its cross-lingual embedding. Then, each token in the source sequence  $(x_{i})$  is matched to a token in the generated sequence  $(\hat{y}_j)$  with the highest cosine similarity value to compute the recall. Similarly, the precision value is computed by matching each token in the generated sequence to a token in the source sequence based on cosine similarity. As a results, the recall and precision of a generated sequence can be expressed as follows:
$$
R (\hat {\mathbf {y}} _ {1: t}, \mathbf {X}) = \frac {1}{| \mathbf {X} |} \sum_ {x _ {i} \in \mathbf {X}} \max _ {\hat {y} _ {j} \in \hat {\mathbf {y}} _ {1: t}} \mathbf {x} _ {i} ^ {n x e} \cdot \hat {\mathbf {y}} _ {j} ^ {n x e} (2 7)
$$
$$
P \left(\hat {\mathbf {y}} _ {1: t}, \mathbf {X}\right) = \frac {1}{\left| \hat {\mathbf {y}} _ {1 : t} \right|} \sum_ {\hat {y} _ {j} \in \hat {\mathbf {y}} _ {1: t}} \max  _ {x _ {i} \in \mathbf {X}} \mathbf {x} _ {i} ^ {n x e} \cdot \hat {\mathbf {y}} _ {j} ^ {n x e} \tag {28}
$$
where  $\mathbf{x}_i^{nxe}$  and  $\hat{\mathbf{y}}_j^{nxe}$  denote the pre-normalised (i.e.,  $\mathbf{x}_i^{nxe} = \frac{\mathbf{x}_i^{x e}}{||\mathbf{x}_i^{x e}||})$  cross-lingual embeddings of  $x_{i}$  and  $\hat{y}_j$  , respectively. Using the precision and recall, the final TLSS score is defined as the F1 measure.
$$
\begin{array}{l} T L S S \left(\hat {\mathbf {y}} _ {1: t}, \mathbf {X}\right) = F \left(\hat {\mathbf {y}} _ {1: t}, \mathbf {X}\right) (29) \\ = 2 \frac {P (\hat {\mathbf {y}} _ {1 : t} , \mathbf {X}) \cdot R (\hat {\mathbf {y}} _ {1 : t} , \mathbf {X})}{P (\hat {\mathbf {y}} _ {1 : t} , \mathbf {X}) + R (\hat {\mathbf {y}} _ {1 : t} , \mathbf {X})} (30) \\ \end{array}
$$
The TLSS scores are computed using the BERTScore tool (Zhang et al., 2020).
A.7 Semantic-level Evaluation on the OpenSubtitles Test Set
For the OpenSubtitles test set, as shown in Table 9, there is a slight increase in the Fluency scores for all our model as compared to the Transformer. There are more apparent increases in the TLSS and SLSS scores. Among all the variants, the ACQ model achieves the highest on all three scores. The Async-ACQ model is the second best with its scores being very close to the scores of the ACQ model. When comparing the reward functions, the BLEU and SURF variants of the ACQ and Async-ACQ models achieve similar performance. However, for the A2C and A3C models, the BLEU variants achieve higher Fluency scores than the SURF variants but their SLSS scores are noticeably lower than that of SURF. This can be explained by fact that BLEU RL sentences are prone to be more verbose, repeating the same content. This was observed during human evaluation (See Subsection 6.3).
Model	Fluency	TLSS	SLSS	Fluency	TLSS	SLSS
Transformer	1.036	2.450	2.360	-	-	-
BLEU	SURF
ACQ	1.037*	2.455*	2.384*	1.037*†	2.456*†	2.386*†
Async-ACQ	1.037*	2.455*	2.387*	1.037*†	2.455*†	2.385*†
A2C	1.039*	2.455*	2.377*	1.037*†	2.454*†	2.379*†
A3C (Async-A2C)	1.043*	2.443*	2.347*	1.037*†	2.454*†	2.383*†
Table 9: Results for the OpenSubtitles German-English test set. We report Sentence Fluency, Token-level Semantic Similarity and Sentence-level Semantic Similarity scores. Also, the symbol  $\star$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the scores of Transformer model while  $\dagger$  indicates statically significant changes (p-value  $\leq 0.05$ ) as compared to the BLEU variant of the same RL-based architecture. The best result is highlighted in bold. Note that some of the improvements are beyond the displayed precision of 3 decimal points.