SlowGuess/ACL_2020_OCR huggingface dataset:
As Table 4 and 5 demonstrate, in unsupervised settings: (1) two-stage semantic parser is superior to one-stage, which bridges the vast discrepancy between natural language utterances and logical forms by utilizing canonical utterances. Even in supervised experiments, this pipeline is still competitive (76.4% compared to 76.0%, 71.6% to 71.9%). (2) Not surprisingly, model performance is sensitive to the word embedding initialization. On OVERNIGHT, directly using raw Glove6B word vectors, the performance is the worst among all baselines (19.7%). Benefiting from pre-trained embeddings ELMo or Bert, the accuracy is dramatically improved (26.2% and 32.7%). (3) When we share the encoder module in a one-stage parser for multi-tasking (MULTITASKDAE), the performance is not remarkably improved, even slightly lower than EMBED+BERT (31.9% compared to 32.7%, 38.1% to 40.7%). We hypothesize that a semantic parser utilizes the input utterance in a way different from that of a denoising auto-encoder,
$^{2}$ https://github.com/rhythmcao/unsup-two-stage-semantic-parsing $^{3}$ https://github.com/percyliang/sempre
Method	Bas	Blo	Cal	Hou	Pub	Rec	Res	Soc	Avg
Supervised
Previous	SPO (Wang et al., 2015)	46.3	41.9	74.4	54.0	59.0	70.8	75.9	48.2	58.8
DSP-C (Xiao et al., 2016)	80.5	55.6	75.0	61.9	75.8	-	80.1	80.0	72.7
NORECOMB* (Jia and Liang, 2016)	85.2	58.1	78.0	71.4	76.4	79.6	76.2	81.4	75.8
CROSSDOMAIN* (Su and Yan, 2017)	86.2	60.2	79.8	71.4	78.9	84.7	81.6	82.9	78.2
SEQ2ACTION (Chen et al., 2018)	88.2	61.4	81.5	74.1	80.7	82.9	80.7	82.1	79.0
DUAL* (Cao et al., 2019)	87.5	63.7	79.8	73.0	81.4	81.5	81.6	83.0	78.9
Ours	One-stage	85.2	61.9	73.2	72.0	76.4	80.1	78.6	80.8	76.0
Two-stage	84.9	61.2	78.6	67.2	78.3	80.6	78.9	81.3	76.4
Unsupervised
One-stage	EMBED + GLOVE6B	22.3	23.6	9.5	26.5	18.0	24.5	24.7	8.4	19.7
+ELMO	36.8	21.1	20.2	21.2	23.6	36.1	37.7	12.8	26.2
+BERT	40.4	31.6	23.2	35.5	37.9	30.1	44.0	19.2	32.7
WMDSAMPLES	34.5	33.8	29.2	37.6	36.7	41.7	56.6	37.0	38.4
MULTITASKDAE	44.0	25.8	16.1	34.4	29.2	46.3	43.7	15.5	31.9
Two-stage	WMDSAMPLES	31.9	29.0	36.1	47.9	34.2	41.0	53.8	35.8	38.7
COMPLETEMODEL	64.7	53.4	58.3	59.3	60.3	68.1	73.2	48.4	60.7
- CYCLELEARNING	32.5	43.1	36.9	48.2	53.4	49.1	58.7	36.9	44.9
- SHAREDENCODER	63.4	46.4	58.9	61.9	56.5	65.3	64.8	42.9	57.5
Semi-supervised
DUAL (Cao et al., 2019) + 50% labeled data	83.6	62.2	72.6	61.9	71.4	75.0	76.5	80.4	73.0
COMPLETEMODEL + 5% labeled data	83.6	57.4	66.1	63.0	60.3	68.1	75.3	73.1	68.4
+ 15% labeled data	84.4	59.4	79.2	57.1	65.2	79.2	77.4	76.9	72.4
+ 30% labeled data	85.4	64.9	77.4	69.3	67.1	78.2	79.2	78.3	75.0
+ 50% labeled data	85.9	64.4	81.5	66.1	74.5	82.4	79.8	81.6	77.0
+ 100% labeled data	87.2	65.7	80.4	75.7	80.1	86.1	82.8	82.7	80.1
Table 4: Denotation level accuracy of logical forms on dataset OVERNIGHT. Previous supervised methods with superscript * means cross-domain or extra data sources are not taken into account.
Method	GEOGRANNO
Supervised
Previous	COPYNET+ELMO (Herzig and Berant, 2019)	72.0
Ours	One-stage	71.9
Two-stage	71.6
Unsupervised
One-stage	EMBED + GLOVE6B	36.7
+ ELMO	38.9
+ BERT	40.7
WMDSAMPLES	32.0
MULTITASKDAE	38.1
Two-stage	WMDSAMPLES	35.3
COMPLETEMODEL	63.7
- CYCLELEARNING	44.6
- SHAREDENCODER	59.0
Semi-supervised
COMPLETEMODEL + 5% labeled data	69.4
+ 30% labeled data	71.6
+ 100% labeled data	74.5
Table 5: Denotation level accuracy of logical forms on dataset GEOGRANNO.
thus focusing on different zones in representation space. However, in a paraphrasing model, since the input and output utterances are exactly symmetric, sharing the encoder is more suitable to attain an excellent performance (from  $57.5\%$  to  $60.7\%$  on OVERNIGHT,  $59.0\%$  to  $63.7\%$  on GEOGRANNO). Furthermore, the effectiveness of the DAE pre-training task  $(44.9\%)$  and  $44.6\%$  accu
racy on target task) can be explained in part by the proximity of natural language and canonical utterances. (4) WMDSAMPLES method is easy to implement but has poor generalization and obvious upper bound. While our system can self-train through cycle learning and promote performance from initial  $44.9\%$  to  $60.7\%$  on OVERNIGHT, outperforming traditional supervised method (Wang et al., 2015) by 1.9 points.
As for semi-supervised results: (1) when only  $5\%$  labeled data is added, the performance is dramatically improved from  $60.7\%$  to  $68.4\%$  on OVERNIGHT and  $63.7\%$  to  $69.4\%$  on GEOGRANNO. (2) With  $30\%$  annotation, our system is competitive  $(75.0\% / 71.6\%)$  to the neural network model using all data with supervised training. (3) Compared with the previous result reported in Cao et al. (2019) on dataset OVERNIGHT with  $50\%$  parallel data, our system surpasses it by a large margin  $(4\%)$  and achieves the new state-of-the-art performance on both datasets when using all labeled data  $(80.1\% / 74.5\%)$ , not considering results using additional data sources or cross-domain benefits.
From the experimental results and Figure 4, we can safely summarize that (1) our proposed method
6812
resolves the daunting problem of cold start when we train a semantic parser without any parallel data. (2) It is also compatible with traditional supervised training and can easily scale up to handle more labeled data.
Figure 4: Semi-supervised results of different ratios of labeled data on OVERNIGHT. Baselines are one-stage and two-stage models with merely supervised training.
In this section, we analyze the influence of each noise type in the DAE task and different combinations of schemes in the cycle learning phase on dataset OVERNIGHT.
4.3.1 Noisy channels in the pre-training DAE
# Types	Drop	Addition	Shuffling	Acc
none	26.9
one	✓	33.7
✓	32.1
✓	31.6
two	✓	✓	43.0
✓	✓	38.0
✓	✓	36.0
all	✓	✓	✓	44.9
According to results in Table 6, (1) it is interesting that even without any noise, in which case the denoising auto-encoder degenerates into a simple copying model, the paraphrase model still succeeds to make some useful predictions  $(26.9\%)$ . This observation may be attributed to the shared encoder for different utterances. (2) When we gradually complicate the DAE task by increasing the number of noise types, the generalization capability continues to improve. (3) Generally speaking, importance-aware drop and mixed-source addition are more useful than bigram shuffling in this task.
Table 6: Ablation study of different noisy channels.
DAE	BT	DRL	Acc
✓	44.9
✓	51.9
✓	55.9
✓	✓	53.2
✓	✓	53.7
✓	✓	60.7
✓	✓	✓	59.2
Table 7: Ablation study of schemes in cycle learning
Input: who has gotten 3 or more steals
Baseline: player whose number of steals ( over a season ) is at most 3
Ours: player whose number of steals ( over a season ) is at least 3
(a) domain: BASKETBALL
Input: show me all attendees of the weekly standup meeting
Baseline: meeting whose attendee is attendee of weekly standup
Ours: person that is attendee of weekly standup and that is attendee of weekly standup
(b) domain: CALENDAR
Input: what is the largest state bordering _state_
Baseline: state that has the largest area
Ours: state that borders _state_ and that has the largest area
Input: which state has the highest population density ?
Baseline: population of state that has the largest density
Ours: state that has the largest density
(c) domain: GEOGRANNO
Table 8: Case study. The input is natural language utterance, and the intermediate output is canonical utterance. Entities in dataset GEOGRANNO are replaced with its types, e.g. "state".
4.3.2 Strategies in the cycle learning
The most striking observation arising from Table 7 is that the performance decreases by 1.5 percent when we add the DAE task into the cycle learning phase (BT+DRL). A possible explanation for this phenomenon may be that the model has reached its bottleneck of the DAE task after pre-training, thereby making no contribution to the cycle learning process. Another likely factor may stem from the contradictory goals of different tasks. If we continue to add this DAE regularization term, it may hinder exploratory trials of the DRL task. By decoupling the three types of rewards in DRL, we discover that style and relevance rewards are more informative than the fluency reward.
4.4 Case study
In Table 8, we compare intermediate canonical utterances generated by our unsupervised paraphrase model with that created by the baseline WMDSAMPLES. In domain BASKETBALL, our system succeeds in paraphrasing the constraint into "at least
6813
$3^{\prime \prime}$ , which is an alias of "3 or more". This finding consolidates the assumption that our model can learn these fine-grained semantics, such as phrase alignments. In domain GEOGRANNO, our model rectifies the errors in baseline system where constraint "borders_state" is missing and subject "state" is stealthily replaced with "population". As for domain CALENDAR, the baseline system fails to identify the query object and requires "meeting" instead of "person". Although our model correctly understands the purpose, it is somewhat stupid to do unnecessary work. The requirement "attendee of weekly standup" is repeated. This may be caused by the uncontrolled process during cycle learning in that we encourage the model to take risky steps for better solutions.
Annotation for Semantic Parsing Semantic parsing is always data-hungry. However, the annotation for semantic parsing is not user-friendly. Many researchers have attempted to relieve the burden of human annotation, such as training from weak supervision (Krishnamurthy and Mitchell, 2012; Berant et al., 2013; Liang et al., 2017; Goldman et al., 2018), semi-supervised learning (Yin et al., 2018; Guo et al., 2018; Cao et al., 2019; Zhu et al., 2014), on-line learning (Iyer et al., 2017; Lawrence and Riezler, 2018) and relying on multi-lingual (Zou and Lu, 2018) or cross-domain datasets (Herzig and Berant, 2017; Zhao et al., 2019). In this work, we try to avoid the heavy work in annotation by utilizing canonical utterances as intermediate results and construct an unsupervised model for paraphrasing.
Unsupervised Learning for Seq2Seq Models Seq2Seq (Sutskever et al., 2014; Zhu and Yu, 2017) models have been successfully applied in unsupervised tasks such as neural machine translation (NMT) (Lample et al., 2017; Artetxe et al., 2017), text simplification (Zhao et al., 2020), spoken language understanding (Zhu et al., 2018) and text style transfer (Luo et al., 2019). Unsupervised NMT relies heavily on pre-trained cross-lingual word embeddings for initialization, as Lample et al. (2018) pointed out. Moreover, it mainly focuses on learning phrase alignments or word mappings. While in this work, we dive into sentence-level semantics and adopt the dual structure of an unsupervised paraphrase model to improve semantic parsing.
In this work, aiming to reduce annotation, we propose a two-stage semantic parsing framework. The first stage utilizes the dual structure of an unsupervised paraphrase model to rewrite the input natural language utterance into canonical utterance. Three self-supervised tasks, namely denoising auto-encoder, back-translation and dual reinforcement learning, are introduced to iteratively improve our model through pre-training and cycle learning phases. Experimental results show that our framework is effective, and compatible with supervised training.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415-1425, Baltimore, Maryland. Association for Computational Linguistics.
Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. 2019. Semantic parsing with dual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 51-64, Florence, Italy. Association for Computational Linguistics.
Bo Chen, Le Sun, and Xianpei Han. 2018. Sequence-to-action: End-to-end semantic graph generation for semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 766-777.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep
6814
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. 2018. Weakly supervised semantic parsing with abstract examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1809-1819.
Daya Guo, Yibo Sun, Duyu Tang, Nan Duan, Jian Yin, Hong Chi, James Cao, Peng Chen, and Ming Zhou. 2018. Question generation from sql queries improves neural semantic parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1597-1607.
Jonathan Herzig and Jonathan Berant. 2017. Neural semantic parsing over multiple knowledge-bases. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 623-628.
Jonathan Herzig and Jonathan Berant. 2019. Don't paraphrase, detect! rapid and effective data collection for semantic parsing. arXiv preprint arXiv:1908.09940.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963-973.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jayant Krishnamurthy and Tom M Mitchell. 2012. Weakly supervised training of semantic parsers. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 754-765.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957-966.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049, Brussels, Belgium. Association for Computational Linguistics.
Carolin Lawrence and Stefan Riezler. 2018. Improving a neural semantic parser by counterfactual learning from human bandit feedback. arXiv preprint arXiv:1805.01252.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D. Forbus, and Ni Lao. 2017. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23-33.
Wei Lu, Hwee Tou Ng, Wee Sun Lee, and Luke S Zettlemoyer. 2008. A generative model for parsing natural language to meaning representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 783-792.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. A dual reinforcement learning framework for unsupervised text style transfer. arXiv preprint arXiv:1905.10060.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
6815
Yu Su and Xifeng Yan. 2017. Cross-domain semantic parsing via paraphrasing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1235-1246.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057-1063.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103. ACM.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Yuk Wah Wong and Raymond Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 960-967.
Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1341-1350, Berlin, Germany. Association for Computational Linguistics.
Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. 2018. StructVAE: Tree-structured latent variable models for semi-supervised semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 754–765, Melbourne, Australia. Association for Computational Linguistics.
John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed ccg grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language
Processing and Computational Natural Language Learning (EMNLP-CoNLL).
Yanbin Zhao, Lu Chen, Zhi Chen, and Kai Yu. 2020. Semi-supervised text simplification with back-translation and asymmetric denoising autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence.
Zijian Zhao, Su Zhu, and Kai Yu. 2019. Data augmentation with atomic templates for spoken language understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3637-3643, Hong Kong, China. Association for Computational Linguistics.
Su Zhu, Lu Chen, Kai Sun, Da Zheng, and Kai Yu. 2014. Semantic parser enhancement for dialogue domain extension with little data. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 336-341. IEEE.
Su Zhu, Ouyu Lan, and Kai Yu. 2018. Robust spoken language understanding with unsupervised asrerror adaptation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6179-6183. IEEE.
Su Zhu and Kai Yu. 2017. Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5675-5679. IEEE.
Yanyan Zou and Wei Lu. 2018. Learning cross-lingual distributed logical representations for semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 673-679, Melbourne, Australia. Association for Computational Linguistics.
6816
A.1 Model Implementations
In this section, we give a full version discussion about all models used in our two-stage semantic parsing framework.
Unsupervised paraphrase model We use traditional attention (Luong et al., 2015) based Seq2Seq model. Different from previous work, we remove the transition function of hidden states between encoder and decoder. The initial hidden states of decoders are initialized to 0-vectors. Take  $D_z \circ E$  paraphrase model as an example:
(1) a shared encoder encodes the input utterance  $x$  into a sequence of contextual representations  $\mathbf{h}$  through a bi-directional single-layer LSTM (Hochreiter and Schmidhuber, 1997) network  $(\psi$  is the embedding function)
$$
\begin{array}{l} \overrightarrow {\mathbf {h}} _ {i} = \mathrm {f} _ {\text {L S T M}} \left(\psi \left(x _ {i}\right), \overrightarrow {\mathbf {h}} _ {i - 1}\right), i = 1, \dots , | x | \\ \overleftarrow {\mathbf {h}} _ {i} = \mathrm {f} _ {\text {L S T M}} (\psi (x _ {i}), \overleftarrow {\mathbf {h}} _ {i + 1}), i = | x |, \dots , 1 \\ \mathbf {h} _ {i} = \left[ \overrightarrow {\mathbf {h}} _ {i}; \overleftarrow {\mathbf {h}} _ {i} \right] \\ \end{array}
$$
(2) on the decoder side, a traditional LSTM language model at the bottom is used to model dependencies in target utterance  $z$  ( $\phi$  is the embedding function on target side)
$$
\begin{array}{l} \mathbf {s} _ {t} = \mathrm {f} _ {\text {L S T M}} \left(\phi \left(z _ {t - 1}\right), \mathbf {s} _ {t - 1}\right) \\ \mathbf {s} _ {0} = 0 - v e c t o r \\ \end{array}
$$
(3) output state  $s_t$  at each time-step  $t$  is then fused with encoded contexts  $\mathbf{h}$  to obtain the features for final softmax classifier  $(\mathbf{v}, \mathbf{W}_*$  and  $\mathbf{b}_*$  are model parameters)
$$
u _ {i} ^ {t} = \mathbf {v} ^ {T} \tanh (\mathbf {W} _ {h} \mathbf {h} _ {i} + \mathbf {W} _ {s} \mathbf {s} _ {t} + \mathbf {b} _ {a})
$$
$$
a _ {i} ^ {t} = \frac {\exp (u _ {i} ^ {t})}{\sum_ {j = 1} ^ {| x |} \exp (u _ {j} ^ {t})}
$$
$$
\mathbf {c} _ {t} = \sum_ {i = 1} ^ {| x |} a _ {i} ^ {t} \mathbf {h} _ {i}
$$
$$
P (z _ {t} | z _ {<   t}, x) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {o} [ \mathbf {s} _ {t}; \mathbf {c} _ {t} ] + \mathbf {b} _ {o}\right)
$$
In both pre-training and cycle learning phases, the unsupervised paraphrase model is trained for 50 epochs, respectively. To select the best model during unsupervised training, inspired by Lample et al. (2017), we use a surrogate criterion since we have no access to labeled data  $(x,z)$  even during validation time. For one natural language utterance