SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} L _ {r} = \frac {1}{| \mathcal {S} |} \sum_ {(T, \tau) \in \mathcal {S}} E _ {\hat {\boldsymbol {y}} \sim p (\hat {\boldsymbol {y}} | T _ {x, \tau})} [ R (\hat {\boldsymbol {y}}, \boldsymbol {y}) ] \\ (x _ {x, \tau}; y) \in \mathcal {S} \tag {15} \\ = \frac {1}{| \mathcal {S} |} \sum_ {(T _ {x, \tau}, \boldsymbol {y}) \in \mathcal {S}} \sum_ {\hat {\boldsymbol {y}} \in Y} p (\hat {\boldsymbol {y}} | T _ {x, \tau}) R (\hat {\boldsymbol {y}}, \boldsymbol {y}), \\ \end{array}
$$
Here,  $Y$  is the set of the candidate comment sequences. The reward  $R(\hat{(\pmb{y})},\pmb{y})$  is the non-differentiable evaluation metric, i.e., BLEU and ROUGE (details are in Section 7). The expectation in Eq. (15) is approximated via sampling  $\hat{\pmb{y}}$  from the distribution  $p(\hat{\pmb{y}} |T_{x,\tau})$ . The procedure of sampling  $\hat{\pmb{y}}$  from  $p(\hat{\pmb{y}} |T_{x,\tau})$  is composed of the sub-procedures of sampling  $\hat{y}_m$  from  $p(\hat{y}_m|\hat{y}_{< m};T_{x,\tau})$  in each decoding step  $m$ .
As mentioned above, the predicted sequence  $\hat{\pmb{y}}$  comes from the two branches of Word Selection Stage, depending on the Operation Selection Stage.  $a$  is defined as the action of the Operation selection stage. After involving the action  $a_{m}$  in time step  $m$ , Eq. (15) can be constructed by the joint distribution of the two stages:
$$
\begin{array}{l} \frac {1}{| \mathcal {S} |} \sum_ {(T _ {x, \tau}, \boldsymbol {y}) \in \mathcal {S}} \sum_ {\hat {\boldsymbol {y}} \in Y} p (\hat {\boldsymbol {y}} | T _ {x, \tau}) R (\hat {\boldsymbol {y}}, \boldsymbol {y}) \\ = \frac {1}{| \mathcal {S} |} \sum_ {\dots} \sum_ {\hat {\boldsymbol {y}} \in Y} \left(\prod_ {m = 1} ^ {M} \sum_ {\hat {a} _ {m}} \underbrace {p \left(\hat {y} _ {m} , \hat {a} _ {m} \mid \hat {y} _ {<   m} ; T _ {x , \tau}\right)} _ {\text {T w o - s t a g e J o u n t D i s t r i b u t i o n}}\right) R (\hat {\boldsymbol {y}}, \boldsymbol {y}) \\ = \dots \underbrace {p \left(\hat {y} _ {m} \mid \hat {a} _ {m} ; \hat {y} _ {<   m} ; T _ {x , \tau}\right)} _ {\text {W o r d D i s t r i b u t i o n}} \underbrace {p \left(\hat {a} _ {m} \mid \hat {y} _ {<   m} ; T _ {x , \tau}\right)} _ {\text {O p e r a t i o n D i s t r i b u t i o n}} \dots \tag {16} \\ \end{array}
$$
As shown in Eq. (16), the model finally selects the word  $\hat{y}_m$  in time step  $m$  from the word distribution conditioned on  $\hat{y}_{< m}, T_{x,\tau}$  and the operation  $\hat{a}_m$  which is determined in the operation selection stage. In other words, there is a hierarchical dependency between the word selection stage and the operation selection stage.
As mentioned above,  $Y$  represents the space for all candidate comments, which is too large to practically maximize  $L_{r}$ . Since decoding is constructed via sampling from  $p(\hat{y}_m|\hat{a}_m,\hat{y}_{< m};T_{x,\tau})$  and  $p(\hat{a}_m|\hat{y}_{< m};T_{x,\tau})$ , We adopt the Gumbel-Max solution (Gumbel, 1954) for the following sampling procedure:
$$
\hat {a} _ {m} \sim p \left(\hat {a} _ {m} \mid \hat {y} _ {<   m}; T _ {x, \tau}\right), \tag {17}
$$
$$
\hat {y} _ {m} \sim p (\hat {y} _ {m} | \hat {a} _ {m}, \hat {y} _ {<   m}; T _ {x, \tau}).
$$
Through the maximum sampling step M, Eq. (16) could be further approximated as the following equation:
$$
\hat {L} _ {r} = \frac {1}{| \mathcal {S} |} \sum_ {\boldsymbol {y} \in \mathcal {S}} R (\hat {\boldsymbol {y}}, \boldsymbol {y}) \tag {18}
$$
The objective in Eq. (18) remains another challenge: for the entire sequence  $\hat{y}$ , there is only a final reward  $R(\hat{y},\mathbf{y})$  available for model training, which is a sparse reward and leads to inefficient training of the model. So we introduce reward shaping (Ng et al., 1999) strategy to provide intermediate rewards to proceed towards the training goal, which adopts the accumulation of the intermediate rewards to update the model.
To further stabilize the HRL training process, we combine our HRL objective with the maximum-likelihood estimation(MLE) function according to
Wu et al. (2018a, 2016); Li et al. (2017); Wu et al. (2018b):
$$
\begin{array}{l} L _ {e} = \frac {1}{| \mathcal {S} |} \sum_ {\left(T _ {x, \tau}, \boldsymbol {y}\right) \in \mathcal {S}} \sum_ {\hat {\boldsymbol {y}} \in Y} \log p (\boldsymbol {y} \mid T _ {x, \tau}) \tag {19} \\ L = \mu L _ {e} + (1 - \mu) \hat {L} _ {r}, \\ \end{array}
$$
where  $\mu$  is a variational controlling factor that controls the trade-off between maximum-likelihood estimation function and our HRL objective. In the current training step  $tr$ ,  $\mu$  varies according to the training step  $tt$  as follows:
$$
\mu = 1 - \frac {t r}{t t} \tag {20}
$$
7 Evaluation and Analysis
7.1.1 Datasets
We evaluate our TAG framework on three widely used benchmark data sets, which are WikiSQL (Zhong et al., 2017), ATIS (Dong and Lapata, 2016) and CoNaLa (Yin et al., 2018). WikiSQL is a dataset of 80654 hand-annotated examples of SQL query and natural language comment pairs distributed across 24241 tables from Wikipedia. These SQL queries are further split into training (56355 examples), development (8421 examples) and test (15878 examples) sets. ATIS is in the form of lambda-calculus, which is a set of 5410 inquiries for flight information containing 4434 training examples, 491 development examples and 448 test examples. CoNaLa is a python related dataset. Its original version is used which includes 2879 snippet/intent pairs crawled from Stack Overflow, split into 2379 training and 500 test examples. We extract 200 random examples from its training set as the development set.
We transfer the SQL queries of WikiSQL into ASTs with 6 types according to the Abstract Syntax Description Language (ASDL) grammar, where the ASDL grammar for SQL queries is proposed in Yin and Neubig (2017). We transfer the lambda calculus logical forms of ATIS to tree structure with 7 types according to the method proposed in Dong and Lapata (2016). The python snippets of CoNaLa are transformed into ASTs with 20 types, following the official ASDL grammar of  $\mathsf{p}\mathsf{o}\mathsf{n}\mathsf{h}\mathsf{l}^{\mathsf{1}}$ . The data of the ASTs of these datasets is shown in Table 1, where the maximum depth of ASTs (Max-Tree-Depth), the maximum number of child
nodes in ASTs (Max-Child-Count) and the average number of tree nodes in ASTs (Avg-Tree-Node-Count) are shown.
Dataset	WikiSQL	ATIS	CoNaLa
Max Tree Depth	5	18	28
Max Child Num	4	15	10
Avg Tree Node Count	11.11	33.54	28.37
Table 1: Statistics of ASTs on the datasets.
7.1.2 Baselines Frameworks
We choose the representative designs for code comment generation as our baselines for comparison. Code-NN (Iyer et al., 2016) is chosen because of it is the first model to transform the source code into sentences. Pointer Generator (See et al., 2017) (P-G) is a seq2seq based model with a standard copying mechanism. In addition, we choose the attention based Tree-to-Sequence (Tree2Seq) model proposed by Eriguchi et al. (2016). Moreover, we also add the copying mechanism into Tree2Seq model as another baseline  $(\mathrm{T}2\mathrm{S} + \mathrm{CP})$ . We choose Graph-to-Sequence (Graph2Seq) (Xu et al., 2018b) as a graph-based baseline for comparison. Since the authors have not released the code for data-preprocessing, we convert the tree-structured representation for the source code of SQL data into directed graphs for our replication.
7.1.3 Hyperparameters
Code-NN uses embedding size and hidden size both as 400, and applies random uniform initializer with 0.35 initialized weight, and adopts stochastic gradient descent algorithm to train the model with a learning rate at 0.5. P-G uses 128 embedding size, 256 hidden size and applies random uniform initializer with 0.02 initialized weights for initialization and Adam optimizer to train the model with 0.001 learning rate. Graph2Seq uses 100 embedding size, 200 hidden size and applies the truncated normal initializer for initialization. Adam optimizer is used to train the model with a 0.001 learning rate.
We use the Xavier initializer (Glorot and Bengio, 2010) to initialize the parameters of our proposed TAG framework. The size of embeddings is equivalent to the dimensions of LSTM states and hidden layers, which is 64 for ATIS and CoNaLa and 128 for WikiSQL. TAG is trained using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.001. In order to reduce the size of the vocabulary, low-frequency words are not kept in both the
$^{1}$  https://docs.python.org/3.5/library/ast.html
Model	WikiSQL (SQL)	ATIS (lambda-calculus)	CoNaLa (Python)
BLEU-4	ROUGE-2	ROUGE-L	BLEU-4	ROUGE-2	ROUGE-L	BLEU-4	ROUGE-2	ROUGE-L
Code-NN	6.7	9.7	30.9	37.1	43.28	59.4	8.1	12.2	26.1
P-G	25.7	29.2	50.1	41.9	47.3	60.5	10.0	13.8	28.0
Tree2Seq	22.0	22.0	43.4	40.1	47.2	60.9	6.6	9.2	25.2
Graph2Seq	17.6	24.3	45.7	34.6	41.8	58.3	10.4	14.1	28.2
T2S+CP	31.0	36.8	54.5	39.0	43.7	58.4	13.3	18.5	31.5
TAG(B)	35.8	41.0	57.8	42.4	47.4	61.2	14.1	19.4	31.8
TAG(R)	35.2	41.1	58.1	40.6	47.1	61.5	12.6	19.7	32.2
vocabulary for the source codes and the vocabulary for target comments. Specifically, the minimum threshold frequency for WikiSQL and ATIS is set as 4 while for CoNaLa it is set as 2. The hyperparameters of Tree2Seq and T2S+CP is equivalent to ours. The minibatch size of all the baseline models and ours are set to 32.
7.1.4 Evaluation Metric
We illustrate the n-gram based BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) evaluations to evaluate the quality of our generated comments and also use them to set the reward in the HRL based training. Specifically, BLEU-4, ROUGE-2 and ROUGE-L are used to evaluate the performance of our model since they are the most representative evaluation metric for context-based text generation.
7.2 Results and Analysis
7.2.1 Comparison with the Baselines
Table 2 presents the evaluation results of the baseline frameworks and our proposed ones. Since our HRL could be switched to different reward functions, we evaluate both the BLEU oriented and ROUGE oriented training of our framework, denoted as TAG(B) and TAG(R). The results of TAG(B) and TAG(R) varies slightly compared to each other. However, both of them are significantly higher than all the selected counterparts, which demonstrates the state-of-the-art generation quality of our framework on all the datasets with different programming languages.
Specifically, TAG improves over  $15\%$  of BLEU-4, over  $10\%$  of ROUGE-2 and  $6\%$  of ROUGE-L on WikiSQL when compared to T2S+CP, which is the best one among all the baseline target for all the evaluations. For the lambda-calculus related corpus, TAG improves  $1.0\%$  of BLEU,  $0.2\%$  ROUGE-2 and  $0.5\%$  ROUGE-L on ATIS. The performance is more difficult to be improved on ATIS
Table 2: Comparisons with baseline models on different test sets.
Model	BLEU-4	ROUGE-2	ROUGE-L
TAG-TA	34.8(-1.4)	41.0(-1.3)	57.8(-1.6)
TAG-MV	35.2(-1.0)	41.1(-1.2)	58.1(-1.3)
TAG-CD	33.5(-2.7)	40.0(-2.3)	57.1(-2.3)
TAG-RL	34.6(-1.6)	41.4(-0.9)	58.7(-0.7)
TAG(B)	36.2	42.0	58.8
TAG(R)	35.6	42.3	59.4
Table 3: Ablation study of TAG framework.
than the other two corpora due to the great dissimilarity of sub-trees of the lambda-calculus logical forms in it. In terms of the python related corpus, TAG improves  $6\%$  of BLEU,  $6.4\%$  of ROUGE-2 and  $2.2\%$  of ROUGE-L on CoNaLa when compared to the best one in our baselines. The low evaluation score and improvement of CoNaLa are due to the complex grammatical structures and lack of sufficient training samples, i.e., 20 types across only 2174 training samples, which result in an inadequately use of the advantage of our approach. However, our TAG framework still outperforms all the counterparts on these two datasets.
7.2.2 Ablation Study
To investigate the performance of each component in our model, we conduct ablation studies on the development sets. Since all the trends are the same, we omit the results on the other data sets and only present the ones of WikiSQL. The variants of our model are as follows:
- TAG-TA: remove Type-associated Encoder, use Tree-LSTM instead.
- TAG-MV: remove the mask vector  $\mathbf{d}_m$ .
- TAG-CD: remove Copying Decay Strategy.
- TAG-RL replace HRL with MLE, marginalize the actions of the operation selection.
The results of the ablation study are given in Table 3. Overall, all the components are necessary to TAG framework and providing important contributions to the final output. When compared to TAG-TA, the high performance of standard TAG
Code	Comment
SQL: SELECT MAX(Capacity) FROM table WHERE Stadium = “Otkrytie Arena”	Ground-Truth: What is the maximum capacity of the Otkrytie Arena Stadium ?
Code-NN: What is the highest attendance for ?
P-G: Who is the % that ’s position at 51 ?
Tree2Seq: What is the highest capacity at <unk> at arena ?
Graph2Seq: What is the highest capacity for arena arena ?
T2S+CP: What is the highest capacity for the stadium ?
TAG: What is the highest capacity for the stadium of Otkrytie Arena ?
Python: i: d [i] for i in d if i != 'c'	Ground-Truth: remove key ‘c’ from dictionary ‘d’
Code-NN: remove all keys from a dictionary ‘d’
P-G: select a string ‘c’ in have end of a list ‘d’
Tree2Seq: get a key ‘key’ one ‘,’ one ‘,<unk>
Graph2Seq: filter a dictionary of dictionaries from a dictionary ‘d’ where a dictionary of dictionaries ‘d’
T2S+CP: find all the values in dictionary ‘d’ from a dictionary ‘d’
TAG: remove the key ‘c’ if a dictionary ‘d’
Table 4: Case study comparisons.
benefits from the Type-associated Encoder which adaptively processes the nodes with different types and extracts a better summarization of the source code. The downgraded performance of TAG-MV and TAG-CD indicates the advantages of the type-restricted masking vector and Copying Decay Strategy. These together ensure the accurate execution of the copy and word selection. The comparison of TAG and TAG-RL shows the necessity of the HRL for the training of our framework.
7.2.3 Case Study
In order to show the effectiveness of our framework in a more obvious way, some cases generated by TAG are shown in Table 4. SQL and Python are taken as the targeted programming languages. The comments generated by TAG show great improvements when compared to the baselines. Specifically, for the case in SQL, the keyword "Otkrytie Area" is missing in all the baselines but accurately generated by our framework. For the case in Python, the comment generated by TAG is more readable than the others. These cases demonstrate the high quality of the comments generated by our TAG framework.
In this paper, we present a Type Auxiliary Guiding encoder-decoder framework for the code comment generation task. Our proposed framework takes full advantage of the type information associated with the code through the well designed Type-associated Encoder and Type-restricted Decoder. In addition, a hierarchical reinforcement learning method is provided for the training of our framework. The ex
experimental results demonstrate significant improvements over state-of-the-art approaches and strong applicable potential in software development. Our proposed framework also verifies the necessity of the type information in the code translation related tasks with a practical framework and good results. As future work, we will extend our framework to more complex contexts by devising efficient learning algorithms.
This research was supported in part by Natural Science Foundation of China (61876043, 61976052), Natural Science Foundation of Guangdong (2014A030306004, 2014A030308008), Science and Technology Planning Project of Guangzhou (201902010058). Besides, this project is also partly supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. This research was also made possible by NPRP grant NPRP10-0208-170408 from the Qatar National Research Fund (a member of Qatar Foundation). The findings herein reflect the work, and are solely the responsibility of the authors.
Krishan K Aggarwal, Yogesh Singh, and Jitender Kumar Chhabra. 2002. An integrated measure of software maintainability. In Annual Reliability and Maintainability Symposium. 2002 Proceedings (Cat. No. 02CH37318), pages 235-241. IEEE.
Miltiadis Allamanis, Hao Peng, and Charles Sutton.
2016. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning, pages 2091-2100.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400.
Ruichu Cai, Boyan Xu, Zhenjie Zhang, Xiaoyan Yang, Zijian Li, and Zhihao Liang. 2018. An encoder-decoder framework translating natural language to database queries. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 3977-3983. AAAI Press.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33-43, Berlin, Germany. Association for Computational Linguistics.
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 823-833, Berlin, Germany. Association for Computational Linguistics.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured neural summarization. arXiv preprint arXiv:1811.01824.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1631-1640, Berlin, Germany. Association for Computational Linguistics.
Emil Julius Gumbel. 1954. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office.
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, pages 200-210. ACM.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073-2083, Berlin, Germany. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K Reddy. 2019. Deep reinforcement learning for sequence-to-sequence models. IEEE Transactions on Neural Networks and Learning Systems.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2157-2169, Copenhagen, Denmark. Association for Computational Linguistics.
Yuding Liang and Kenny Qili Zhu. 2018. Automatic generation of text descriptive comments for code blocks. In Thirty-Second AAAI Conference on Artificial Intelligence.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kocisky, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 599-609, Berlin, Germany. Association for Computational Linguistics.
Dana Movshovitz-Attias and William W. Cohen. 2013. Natural language models for predicting programming comments. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 35-40, Sofia, Bulgaria. Association for Computational Linguistics.
Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278-287.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation
and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139-1149, Vancouver, Canada. Association for Computational Linguistics.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556–1566, Beijing, China. Association for Computational Linguistics.
Ted Tenny. 1988. Program readability: Procedures versus comments. IEEE Transactions on Software Engineering, 14(9):1271-1279.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700.
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. 2018. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 397-407. ACM.
Yuk Wah Wong and Raymond Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 172-179, Rochester, New York. Association for Computational Linguistics.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and TieYan Liu. 2018a. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3612-3621, Brussels, Belgium. Association for Computational Linguistics.
Lijun Wu, Yingce Xia, Fei Tian, Li Zhao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018b. Adversarial neural machine translation. In *Asian Conference on Machine Learning*, pages 534-549.