SlowGuess/ACL_2020_OCR huggingface dataset:
Ruisheng Cao
Chen Liu
sheng Cao
Rao Ma
Su Zhu
Yanbin Zhao
Chenyu Yang
Kai Yu*
MoE Key Lab of Artificial Intelligence
SpeechLab, Department of Computer Science and Engineering
AI Institute, Shanghai Jiao Tong University, China
{211314,paul2204,yangcy,chris-chen,rm1031,zhaoyb}@sjtu.edu.cn
{chenlusz,kai.yu}@sjtu.edu.cn
One daunting problem for semantic parsing is the scarcity of annotation. Aiming to reduce nontrivial human labor, we propose a two-stage semantic parsing framework, where the first stage utilizes an unsupervised paraphrase model to convert an unlabeled natural language utterance into the canonical utterance. The downstream naive semantic parser accepts the intermediate output and returns the target logical form. Furthermore, the entire training process is split into two phases: pre-training and cycle learning. Three tailored self-supervised tasks are introduced throughout training to activate the unsupervised paraphrase model. Experimental results on benchmarks OVERNIGHT and GEOGRANNO demonstrate that our framework is effective and compatible with supervised training.
Semantic parsing is the task of converting natural language utterances into structured meaning representations, typically logical forms (Zelle and Mooney, 1996; Wong and Mooney, 2007; Zettlemoyer and Collins, 2007; Lu et al., 2008). One prominent approach to build a semantic parser from scratch follows this procedure (Wang et al., 2015):
a). (canonical utterance, logical form) pairs are automatically generated according to a domain-general grammar and a domain-specific lexicon.
b). Researchers use crowdsourcing to paraphrase those canonical utterances into natural language utterances (the upper part of Figure 1).
c). A semantic parser is built upon collected (natural language utterance, logical form) pairs.
Figure 1: Two-stage semantic parsing framework, which is composed of an unsupervised paraphrase model and a naive neural semantic parser.
Canonical utterances are pseudo-language utterances automatically generated from grammar rules, which can be understandable to people, but do not sound natural. Though effective, the paraphrasing paradigm suffers from two drawbacks: (1) dependency on nontrivial human labor and (2) low utilization of canonical utterances.
Annotators may struggle to understand the exact meanings of canonical utterances. Some canonical utterances even incur ambiguity, which enhances the difficulty of annotation. Furthermore, Wang et al. (2015) and Herzig and Berant (2019) only exploit them during data collection. Once the semantic parsing dataset is constructed, canonical utterances are thrown away, which leads to insufficient utilization. While Berant and Liang (2014) and Su and Yan (2017) have reported the effectiveness of leveraging them as intermediate outputs, they experiment in a completely supervised way, where the human annotation is indispensable.
In this work, inspired by unsupervised neural machine translation (Lample et al., 2017; Artetxe et al., 2017), we propose a two-stage semantic parsing framework. The first stage uses a paraphrase model to convert natural language utterances into corre
*The corresponding author is Kai Yu.
6806
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6806-6817
sponding canonical utterances. The paraphrase model is trained in an unsupervised way. Then a naive neural semantic parser is built upon autogenerated (canonical utterance, logical form) pairs using traditional supervised training. These two models are concatenated into a pipeline (Figure 1).
Paraphrasing aims to perform semantic normalization and reduce the diversity of expression, trying to bridge the gap between natural language and logical forms. The naive neural semantic parser learns inner mappings between canonical utterances and logical forms, as well as the structural constraints.
The unsupervised paraphrase model consists of one shared encoder and two separate decoders for natural language and canonical utterances. In the pre-training phase, we design three types of noise (Section 3.1) tailored for sentence-level denoising autoencoder (Vincent et al., 2008) task to warm up the paraphrase model without any parallel data. This task aims to reconstruct the raw input utterance from its corrupted version. After obtaining a good initialization point, we further incorporate back-translation (Sennrich et al., 2015) and dual reinforcement learning (Section 2.2.2) tasks during the cycle learning phase. In this phase, one encoder-decoder model acts as the environment to provide pseudo-samples and reward signals for another.
We conduct extensive experiments on benchmarks OVERNIGHT and GEOGRANNO, both in unsupervised and semi-supervised settings. The results show that our method obtains significant improvements over various baselines in unsupervised settings. With full labeled data, we achieve new state-of-the-art performances (80.1% on OVERNIGHT and 74.5% on GEOGRANNO), not considering additional data sources.
- A two-stage semantic parser framework is proposed, which casts parsing into paraphrasing. No supervision is provided in the first stage between input natural language utterances and intermediate output canonical utterances.
- In unsupervised settings, experimental results on datasets OVERNIGHT and GEOGRANNO demonstrate the superiority of our model
over various baselines, including the supervised method in Wang et al. (2015) on OVERNIGHT  $(60.7\%)$  compared to  $58.8\%$ .
- The framework is also compatible with traditional supervised training and achieves the new state-of-the-art performances on datasets OVERNIGHT (80.1%) and GEOGRANNO (74.5%) with full labeled data.
For the rest of our discussion, we use  $x$  to denote natural language utterance,  $z$  for canonical utterance, and  $y$  for logical form.  $\mathcal{X}, \mathcal{Z}$  and  $\mathcal{Y}$  represent the set of all possible natural language utterances, canonical utterances, and logical forms respectively. The underlying mapping function  $f: \mathcal{Z} \longrightarrow \mathcal{Y}$  is dominated by grammar rules.
We can train a naive neural semantic parser  $P_{nsp}$  using attention (Luong et al., 2015) based Seq2Seq model (Sutskever et al., 2014). The labeled samples  $\{(z,y), z \in \mathcal{Z}, y \in \mathcal{Y}\}$  can be automatically generated by recursively applying grammar rules.  $P_{nsp}$  can be pre-trained and saved for later usage.
As for the paraphrase model (see Figure 1), it consists of one shared encoder  $E$  and two independent decoders:  $D_{x}$  for natural language utterances and  $D_{z}$  for canonical utterances. The symbol  $\circ$  denotes module composition. Detailed model implementations are omitted here since they are not the main focus (Appendix A.1 for reference).
Given an input utterance  $x \in \mathcal{X}$ , the paraphrase model  $D_z \circ E$  converts it into possible canonical utterance  $\hat{z} = D_z \circ E(x)$ ; then  $\hat{z}$  is passed into the pre-trained naive parser  $P_{nsp}$  to obtain predicted logical form  $\hat{y} = P_{nsp} \circ D_z \circ E(x)$ . Another paraphrase model,  $D_x \circ E$ , is only used as an auxiliary tool during training.
2.2 Unsupervised training procedures
To train an unsupervised paraphrase model with no parallel data between  $\mathcal{X}$  and  $\mathcal{Z}$ , we split the entire training procedure into two phases: pre-training and cycle learning.  $D_{x} \circ E$  and  $D_{z} \circ E$  are first pretrained as denoising auto-encoders (DAE). This initialization phase plays a significant part in accelerating convergence due to the ill-posed nature of paraphrasing tasks. Next, in the cycle learning phase, we employ both back-translation (BT) and dual reinforcement learning (DRL) strategies for self-training and exploration.
<sup>1</sup>We use word "naive" just to differentiate from traditional semantic parser, where our module expects to accept canonical utterances instead of natural language utterances.
6807
2.2.1 Pre-training phase
In this phase, we initialize the paraphrase model via the denoising auto-encoder task. All auxiliary models involved in calculating rewards (see Section 3.2) are also pre-trained.
Figure 2: Denoising auto-encoders for natural language utterance  $x$  and canonical utterance  $z$ .
Denoising auto-encoder Given a natural language utterance  $x$ , we forward it through a noisy channel  $\mathcal{N}_x(\cdot)$  (see Section 3.1) and obtain its corrupted version  $\tilde{x}$ . Then, model  $D_x \circ E$  tries to reconstruct the original input  $x$  from its corrupted version  $\tilde{x}$ , see Figure 2. Symmetrically, model  $D_z \circ E$  tries to reconstruct the original canonical utterance  $z$  from its corrupted input  $\mathcal{N}_z(z)$ . The training objective can be formulated as
$$
\begin{array}{l} \mathcal {L} _ {D A E} = - \sum_ {x \sim \mathcal {X}} \log P (x | \mathcal {N} _ {x} (x); \Theta_ {D _ {x} \circ E}) \\ - \sum_ {z \sim \mathcal {Z}} \log P (z | \mathcal {N} _ {z} (z); \Theta_ {D _ {z} \circ E}) \tag {1} \\ \end{array}
$$
where  $\Theta_{D_x\circ E}$  and  $\Theta_{D_z\circ E}$  are parameters for the system.
2.2.2 Cycle learning phase
The training framework till now is just a noisycopying model. To improve upon it, we adopt two schemes in the cycle learning phase, backtranslation (BT) and dual reinforcement learning (DRL), see Figure 3.
Back-translation In this task, the shared encoder  $E$  aims to map the input utterance of different types into the same latent space, and the decoders need to decompose this representation into the utterance of another type. More concretely, given a natural language utterance  $x$ , we use paraphrase model  $D_z \circ E$  in evaluation mode with greedy decoding to convert  $x$  into canonical utterance  $\hat{z}$ . We will obtain pseudo training sample  $(\hat{z}, x)$  for paraphrase model  $D_x \circ E$ . Similarly,  $(\hat{x}, z)$  pair can be synthesized from model  $D_x \circ E$  given canonical
utterance  $z$ . Next, we train the paraphrase model from these pseudo-parallel samples and update parameters by minimizing
$$
\begin{array}{l} \mathcal {L} _ {B T} = - \sum_ {x \sim \mathcal {X}} P (x | D _ {z} \circ E (x); \Theta_ {D _ {x} \circ E}) \\ - \sum_ {z \sim \mathcal {Z}} P (z | D _ {x} \circ E (z); \Theta_ {D _ {z} \circ E}) \quad (2) \\ \end{array}
$$
The updated model will generate better paraphrases during the iterative process.
Dual reinforcement learning Back-translation pays more attention to utilize what has been learned by the dual model, which may lead to a local optimum. To encourage more trials during cycle learning, we introduce the dual reinforcement learning strategy and optimize the system through policy gradient (Sutton et al., 2000).
Starting from a natural language utterance  $x$ , we sample one canonical utterance  $\tilde{z}$  through  $D_z \circ E$ . Then, we evaluate the quality of  $\tilde{z}$  from different aspects (see Section 3.2) and obtain reward  $R_x(\tilde{z})$ . Similarly, we calculate reward  $R_z(\tilde{x})$  for sampled natural language utterance  $\tilde{x}$ . To cope with high variance in reward signals, we increase sample size to  $K$  and re-define reward signals via a baseline  $b(\cdot)$  to stabilize learning: (take  $\tilde{z}^k$  for an example)
$$
R _ {x} (\tilde {z} ^ {k}) \doteq R _ {x} (\tilde {z} ^ {k}) - b (\tilde {z}), \quad k = 1, \dots , K
$$
We investigate different baseline choices (such as running mean, cumulative mean of history, and reward of the greedy decoding prediction), and it performs best when we use the average of rewards within samples of per input, especially with larger sample size. The training objective is the negative sum of expected reward:
$$
\begin{array}{l} \mathcal {L} _ {D R L} = - \sum_ {x \sim \mathcal {X}} \sum_ {k} P (\tilde {z} ^ {k} | x; \Theta_ {D _ {z} \circ E}) \cdot R _ {x} (\tilde {z} ^ {k}) \\ - \sum_ {z \sim \mathcal {Z}} \sum_ {k} P \left(\tilde {x} ^ {k} \mid z; \Theta_ {D _ {x} \circ E}\right) \cdot R _ {z} \left(\tilde {x} ^ {k}\right) \tag {3} \\ \end{array}
$$
The gradient is calculated with REINFORCE (Williams, 1992) algorithm:
$$
\begin{array}{l} \nabla \mathcal {L} \approx - \sum_ {x \sim \mathcal {X}} \frac {R _ {x} (\tilde {z} ^ {k})}{k} \nabla \log P (\tilde {z} ^ {k} | x; \Theta_ {D _ {z} \circ E}) \\ - \sum_ {z \sim \mathcal {Z}} \frac {R _ {z} (\tilde {x} ^ {k})}{k} \nabla \log P (\tilde {x} ^ {k} | z; \Theta_ {D _ {x} \circ E}) \\ \end{array}
$$
The complete loss function in the cycle learning phase is the sum of cross entropy loss and policy gradient loss:  $\mathcal{L}_{Cycle} = \mathcal{L}_{BT} + \mathcal{L}_{DRL}$ . The entire training procedure is summarized in Algorithm 1.
6808
Figure 3: Cycle learning tasks: back-translation and dual reinforcement learning.
Algorithm 1 Training procedure
Input: Unlabeled dataset  $\mathcal{X},\mathcal{Z}$  ; Labeled  $(z,y)$  pairs synthesized from grammar; Iterations  $M$
Output: Paraphrase model  $D_z^{(M)}\circ E^{(M)}$ $\triangleright$  Pre-training phase
1: Pre-train all auxiliary models: language models  $LM_{x}$  and  $LM_{z}$ , naive neural semantic parser  $P_{nsp}$  and utterance discriminator  $P_{dis}$
2: Pre-train paraphrase models  $D_x^{(0)} \circ E^{(0)}$  and  $D_z^{(0)} \circ E^{(0)}$  via objective  $\mathcal{L}_{DAE}$  based on Eq.1  $\triangleright$  Cycle learning phase
3: for  $i = 0$  to  $M - 1$  do  
4: Sample natural language utterance  $x \sim \mathcal{X}$
5: Sample canonical utterance  $z \sim \mathcal{Z}$  Back-translation
6: Generate  $\hat{z}$  via model  $D_{z}^{(i)}\circ E^{(i)}(x)$ ;
7: Generate  $\hat{x}$  via model  $D_x^{(i)}\circ E^{(i)}(z)$
8: Use  $(\hat{z},x)$  and  $(\hat{x},z)$  as pseudo samples, calculate loss  $\mathcal{L}_{BT}$  based on Eq.2; Dual Reinforcement Learning
9: Sample  $\tilde{z}$  via model  $D_z^{(i)}\circ E^{(i)}(x)$
10: Compute total reward  $R_{x}(\tilde{z})$  via models  $LM_{z}, P_{dis}, P_{nsp}$  and  $D_{x}^{(i)} \circ E^{(i)}$  based on Eq.4
11: Sample  $\tilde{x}$  via model  $D_x^{(i)}\circ E^{(i)}(z)$
12: Compute total reward  $R_{z}(\tilde{x})$  via models  $LM_{x}, P_{dis}$  and  $D_{z}^{(i)} \circ E^{(i)}$  based on Eq.5
13: Given  $R_{x}(\tilde{z})$  and  $R_{z}(\tilde{x})$ , calculate loss  $\mathcal{L}_{DRL}$  based on Eq.3 Update model parameters
14: Calculate total loss  $\mathcal{L}_{Cycle} = \mathcal{L}_{BT} + \mathcal{L}_{DRL}$
15: Update model parameters, get new models  $D_{x}^{(i + 1)}\circ E^{(i + 1)}$  and  $D_z^{(i + 1)}\circ E^{(i + 1)}$
16: end for  
17: return  $D_{z}^{(M)} \circ E^{(M)}$
3 Training details
In this section, we elaborate on different types of noise used in our experiment and the reward design in dual reinforcement learning.
3.1 Noisy channel
We introduce three types of noise to deliberately corrupt the input utterance in the DAE task.
Importance-aware word dropping Traditional word dropping (Lample et al., 2017) discards each word in the input utterance with equal probability  $p_{wd}$ . During reconstruction, the decoder needs to recover those words based on the context. We further inject a bias towards dropping more frequent words (such as function words) in the corpus instead of less frequent words (such as content words), see Table 1 for illustration.
Input x	what team does kobe bryant play for
Ordinary drop	what does kobe bryant for
Our drop	team kobe bryant play for
Table 1: Importance-aware word dropping example.
Each word  $x_{i}$  in the natural language utterance  $x = (x_{1},x_{2},\dots ,x_{|x|})$  is independently dropped with probability
$$
p _ {w d} (x _ {i}) = \min  \{p _ {m a x}, w (x _ {i}) / \sum_ {j = 1} ^ {| x |} w (x _ {j}) \}
$$
where  $w(x_{i})$  is the word count of  $x_{i}$  in  $\mathcal{X}$ , and  $p_{max}$  is the maximum dropout rate ( $p_{max} = 0.2$  in our experiment). As for canonical utterances, we apply this word dropping similarly.
6809
Mixed-source addition For any given raw input, it is either a natural language utterance or a canonical utterance. This observation discourages the shared encoder  $E$  to learn a common representation space. Thus, we propose to insert extra words from another source into the input utterance. As for noisy channel  $\mathcal{N}_x(\cdot)$ , which corrupts a natural language utterance, we first select one candidate canonical utterance  $z$ ; next,  $10\% -20\%$  words are randomly sampled from  $z$  and inserted into arbitrary position in  $x$ , see Table 2 for example.
To pick candidate  $z$  with higher relevance, we use a heuristic method:  $C$  canonical utterances are randomly sampled as candidates  $(C = 50)$ ; we choose  $z$  that has the minimum amount of Word Mover's Distance concerning  $x$  (WMD, Kusner et al., 2015). The additive operation is exactly symmetric for noisy channel  $\mathcal{N}_z$ .
Input x	how many players are there
Selected z	number of team
Output x̂	how many number players are there
Bigram shuffling We also use word shuffling (Lample et al., 2017) in noisy channels. It has been proven useful in preventing the encoder from relying too much on the word order. Instead of shuffling words, we split the input utterance into n-grams first and shuffle at n-gram level (bigram in our experiment). Considering the inserted words from another source, we shuffle the entire utterance after the addition operation (see Table 3 for example).
Table 2: Mixed-source addition example.
Input x	what is kobe bryants team
1-gram shuffling	what is kobe team bryants
2-gram shuffling	what is team kobe bryants
Table 3: Bigram shuffling example
3.2 Reward design
In order to provide more informative reward signals and promote the performance in the DRL task, we introduce various rewards from different aspects.
Fluency The fluency of an utterance is evaluated by a length-normalized language model. We use individual language models  $(LM_{x}$  and  $LM_{z})$  for each type of utterances. As for a sampled natural
language utterance  $\tilde{x}$ , the fluency reward is
$$
R _ {z} ^ {f l u} (\tilde {x}) = \frac {1}{| \tilde {x} |} \log L M _ {x} (\tilde {x})
$$
As for canonical utterances, we also include an additional  $0/1$  reward from downstream naive semantic parser to indicate whether the sampled canonical utterance  $\tilde{z}$  is well-formed as input for  $P_{nsp}$ .
$$
\hat {y} = \underset {y} {\operatorname {a r g}} \max  _ {P _ {n s p}} (y | \tilde {z}), \text {g r e e d y d e c o d i n g}
$$
$$
\begin{array}{l} R _ {x} ^ {f l u} (\tilde {z}) = \frac {1}{| \tilde {z} |} \log L M _ {z} (\tilde {z}) \\ + \mathbb {I} \cdot \{\text {n o e r r o r w h i l e e x e c u t i n g} \hat {y} \} \\ \end{array}
$$
Style Natural language utterances are diverse, casual, and flexible, whereas canonical utterances are generally rigid, regular, and restricted to some specific form induced by grammar rules. To distinguish their characteristics, we incorporate another reward signal that determine the style of the sampled utterance. This is implemented by a CNN discriminator (Kim, 2014):
$$
R _ {z} ^ {s t y} (\tilde {x}) = 1 - P _ {d i s} (\tilde {x}); R _ {x} ^ {s t y} (\tilde {z}) = P _ {d i s} (\tilde {z})
$$
where  $P_{dis}(\cdot)$  is a pre-trained sentence classifier that evaluates the probability of the input utterance being a canonical utterance.
Relevance Relevance reward is included to measure how much content is preserved after paraphrasing. We follow the common practice to take the loglikelihood from the dual model.
$$
R _ {x} ^ {r e l} (x, \tilde {z}) = \log P (x | \tilde {z}; \Theta_ {D _ {x} \circ E})
$$
$$
R _ {z} ^ {r e l} (z, \tilde {x}) = \log P (z | \tilde {x}; \Theta_ {D _ {z} \circ E})
$$
Some other methods include computing the cosine similarity of sentence vectors or BLEU score (Papineni et al., 2002) between the raw input and the reconstructed utterance. Nevertheless, we find log-likelihood to perform better in our experiments.
The total reward for the sampled canonical utterance  $\tilde{z}$  and natural language utterance  $\tilde{x}$  can be formulated as
$$
R _ {x} (\tilde {z}) = R _ {x} ^ {f l u} (\tilde {z}) + R _ {x} ^ {s t y} (\tilde {z}) + R _ {x} ^ {r e l} (x, \tilde {z}) \tag {4}
$$
$$
R _ {z} (\tilde {x}) = R _ {z} ^ {f l u} (\tilde {x}) + R _ {z} ^ {s t y} (\tilde {x}) + R _ {z} ^ {r e l} (z, \tilde {x}) \tag {5}
$$
In this section, we evaluate our system on benchmarks OVERNIGHT and GEOGRANNO in both un
supervised and semi-supervised settings. Our implementations are public available<sup>2</sup>.
OVERNIGHT It contains natural language paraphrases paired with logical forms over 8 domains. We follow the traditional  $80\% /20\%$  train-valid to choose the best model during training. Canonical utterances are generated with tool  $\mathrm{SEMPRE}^3$  paired with target logical forms (Wang et al., 2015). Due to the limited number of grammar rules and its coarse-grained nature, there is only one canonical utterance for each logical form, whereas 8 natural language paraphrases for each canonical utterance on average. For example, to describe the concept of "larger", in natural language utterances, many synonyms, such as "more than", "higher", "at least", are used, while in canonical utterances, the expression is restricted by grammar.
GEOGRANNO Due to the language mismatch problem (Herzig and Berant, 2019), annotators are prone to reuse the same phrase or word while paraphrasing. GEOGRANNO is created via detection instead of paraphrasing. Natural language utterances are firstly collected from query logs. Crowd workers are required to select the correct canonical utterance from candidate list (provided by an incrementally trained score function) per input. We follow exactly the same split (train-valid/test 487/59/278) in original paper Herzig and Berant (2019).
Throughout the experiments, unless otherwise specified, word vectors are initialized with Glove6B (Pennington et al., 2014) with  $93.3\%$  coverage on average and allowed to fine-tune. Out-of-vocabulary words are replaced with  $\langle unk\rangle$ . Batch size is fixed to 16 and sample size  $K$  in the DRL task is 6. During evaluation, the size of beam search is 5. We use optimizer Adam (Kingma and Ba, 2014) with learning rate 0.001 for all experiments. All auxiliary models are pre-trained and fixed for later usage. We report the denotation-level accuracy of logical forms in different settings.
Supervised settings This is the traditional scenario, where labeled  $(x,y)$  pairs are used to train a one-stage parser directly,  $(x,z)$  and  $(z,y)$  pairs are respectively used to train different parts of a two-stage parser.
Unsupervised settings We split all methods into two categories: one-stage and two-stage. In the one-stage parser, EMBED semantic parser is merely trained on  $(z,y)$  pairs but evaluated on natural language utterances. Contextual embeddings ELMo (Peters et al., 2018) and Bert-base-uncased (Devlin et al., 2018) are also used to replace the original embedding layer; WMDSAMPLES method labels each input  $x$  with the most similar logical form (one-stage) or canonical utterance (two-stage) based on WMD (Kusner et al., 2015) and deals with these faked samples in a supervised way; MULTITASKDAE utilizes another decoder for natural language utterances in one-stage parser to perform the same DAE task discussed before. The two-stage COMPLETEMODEL can share the encoder or not (-SHAREENCODER), and include tasks in the cycle learning phase or not (-CYCLELEARNING). The downstream parser  $P_{nsp}$  for the two-stage system is EMBED + GLOVE6B and fixed after pre-training.
Semi-supervised settings To further validate our framework, based on the complete model in unsupervised settings, we also conduct semi-supervised experiments by gradually adding part of labeled paraphrases with supervised training into the training process (both pre-training and cycle learning phase).
4.2 Results and analysis