SlowGuess/ACL_2020_OCR huggingface dataset:
Daniel C. Wang, Andrew W. Appel, Jeffrey L. Korn, and Christopher S. Serra. 1997. The Zephyr abstract syntax description language. In Proceedings of DSL.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of ACL.
Xiaojun Xu, Chang Liu, and Dawn Song. 2017. SQL-Net: Generating structured queries from natural language without reinforcement learning. arXiv, abs/1711.04436.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of NIPS.
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of ACL.
Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of EMNLP Demonstration Track.
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir R. Radev. 2018a. TypeSQL: Knowledge-based type-aware neural text-to-sql generation. In Proceedings of NAACL-HLT.
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir R. Radev. 2018b. Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task. In Proceedings of EMNLP.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018c. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of EMNLP.
John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of AAAI.
Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2019a. Editing-based sql query generation for cross-domain context-dependent questions. ArXiv, abs/1909.00786.
Yuchen Zhang, Panupong Pasupat, and Percy Liang. 2017. Macro grammars and holistic triggering for efficient semantic parsing. In Proceedings of EMNLP.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019b. Ernie: Enhanced language representation with informative entities. In Proceedings of ACL.
Zhuosheng Zhang, Yu-Wei Wu, Hai Zhao, Zuchao Li, Shuaijiang Zhang, Xi Zhou, and Xiaodong Zhou. 2019c. Semantics-aware bert for language understanding. ArXiv, abs/1909.02209.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv, abs/1709.00103.
8423
A Pretraining Details
We collect parallel examples of tables and their surrounding NL sentences from two sources:
Wikipedia Tables We extract all the tables on English Wikipedia<sup>9</sup>. For each table, we use the preceding three paragraphs as the NL context, as we observe that most Wiki tables are located after where they are described in the body text.
WDC WebTable Corpus (Lehmberg et al., 2016) is a large collection of Web tables extracted from the Common Crawl Web scrape<sup>10</sup>. We use its 2015 English-language relational subset, which consists of 50.8 million relational tables and their surrounding NL contexts.
Preprocessing Our dataset is collected from arbitrary Web tables, which are extremely noisy. We develop a set of heuristics to clean the data by: (1) removing columns whose names have more than 10 tokens; (2) filtering cells with more than two non-ASCII characters or 20 tokens; (3) removing empty or repetitive rows and columns; (4) filtering tables with less than three rows and four columns, and (5) running spaCy to identify the data type of columns (text or real value) by majority voting over the NER labels of column tokens, (6) rotating vertically oriented tables. We sub-tokenize the corpus using the Wordpiece tokenizer in Devlin et al. (2019). The pre-processing results in 1.3 million tables from Wikipedia and 25.3 million tables from the WDC corpus.
A.2 Pretraining Setup
As discussed in § 5, we create training instances of NL sentences (as synthetic utterances) and content snapshots from tables by sampling from the parallel corpus of NL contexts and tables. Each epoch contains  $37.6M$  training instances. We train TABERT for 10 epochs. Tab. 6 lists the hyper-parameters used in training. Learning rates are validated on the development set of WIKITABLEQUESTIONS. We use a batch size of 512 for large models to reduce training time. The training objective is sum of the three pretraining objectives in § 3.2: the masked
language modeling (MLM) objective for utterance tokens, the masked column prediction (MCP) objective for columns, and the column value recovery (CVR) objective for their cell values. An exception is pretraining the  $\mathrm{TABERT}(\mathrm{K} = 1)$  models. Since there are no additional vertical attention layers, we do not use the CVR objective, and the MCP objective reduces to the vanilla MLM objective over encodings from the base Transformer model. Our largest model  $\mathrm{TABERTLarge}(\mathrm{K} = 3)$  takes six days to train for 10 epochs on 128 Tesla V100 GPUs using mixed precision training.
B Semantic Parsing
B.1 Supervised Parsing on SPIDER
Model We develop our text-to-SQL parser based on TranX (Yin and Neubig, 2018), which translates an NL utterance into a tree-structured abstract meaning representation following user-specified grammar, before deterministically convert the generated abstract MR into an SQL query. TranX models the construction process of an abstract MR (tree-structured representation of an SQL query) using a transition-based system, which decomposes its generation story into a sequence of actions following the user defined grammar.
Formally, given an input NL utterance  $\pmb{u}$  and a database with a set of tables  $\mathcal{T} = \{T_i\}$ , the probability of generating of an SQL query (i.e., its semantically equivalent MR)  $\pmb{z}$  is decomposed as the production of action probabilities:
$$
p (\boldsymbol {z} | \boldsymbol {u}, \mathcal {T}) = \prod p (a _ {t} | a _ {<   t}, \boldsymbol {u}, \mathcal {T}) \tag {2}
$$
where  $a_{t}$  is the action applied to the hypothesis at time stamp  $t$ .  $a_{<t}$  denote the previous action history. We refer readers to Yin and Neubig (2018) for details of the transition system and how individual action probabilities are computed. In our adaptation of TranX to text-to-SQL parsing on SPIDER, we follow Guo et al. (2019) and use SemQL as the underlying grammar, which is a simplification of the SQL language. Fig. 2 lists the SemSQL grammar specified using the abstract syntax description language (Wang et al., 1997). Intuitively, the generation starts from a tree-structured derivation with the root production rule selectStmtt→SelectStatement, which lays out overall the structure of an SQL query. At each time step, the decoder algorithm locates the current opening node on the derivation tree, following a depth-first, left-to-right order. If the opening node
<sup>9</sup>We do not use infoboxes (tables on the top-right of a Wiki page that describe properties of the main topic), as they are not relational tables.
10http://webdatacommons.org/webtables
8424
Parameter	TABERTBase(K = 1)	TABERTLarge(K = 1)	TABERTBase(K = 3)	TABERTLarge(K = 3)
Batch Size	256	512	512	512
Learning Rate	2 × 10-5	2 × 10-5	4 × 10-5	4 × 10-5
Max Epoch	10
Weight Decay	0.01
Gradient Norm Clipping	1.0
Table 6: Hyper-parameters using in pretraining
is not a leaf node, the decoder invokes an action  $a_{t}$  which expands the opening node using a production rule with appropriate type. If the current opening node is a leaf node (e.g., a node denoting string literal), the decoder fills in the leaf node using actions that emit terminal values.
To use such a transition system to generate SQL queries, we extend its action space with two new types of actions, SELECTTABLE  $(T_{i})$  for node of type table_ref in Fig. 2, which selects a table  $T_{i}$  (e.g., for predicting target tables for a FROM clause), and SELECTCOLUMN  $(T_{i}, c_{j})$  for node of type column_ref, which selects the column  $c_{j}$  from table  $T_{i}$  (e.g., for predicting a result column used in the SELECT clause).
As described in § 4.1, TABERT produces a list of entries, with one entry  $\langle \mathbf{T}_i, \mathbf{X}_i, \mathbf{C}_i \rangle$  for each table  $T_i$ :
$$
\begin{array}{l} \mathbb {M} = \left\{\langle \mathbf {T} _ {i}, \mathbf {X} _ {i} = \{\mathbf {x} _ {1}, \mathbf {x} _ {2}, \ldots \}, \right. \\ \mathbf {C} _ {i} = \left\{\mathbf {c} _ {1}, \mathbf {c} _ {2}, \dots , \right\} \rangle_ {i} \bigg \} _ {i = 1} ^ {| \mathcal {T} |} \tag {3} \\ \end{array}
$$
where each entry  $\langle \mathbf{T}_i, \mathbf{X}_i, \mathbf{C}_i \rangle$  in  $\mathbb{M}$  consists of  $\mathbf{T}_i$ , the representation of table  $T_i$  given by the output vector of the prefixed [CLS] symbol, the table-specific representations of utterance tokens  $\mathbf{X}_i = \{\mathbf{x}_1, \mathbf{x}_2, \ldots\}$ , and representations of columns in  $T_i$ ,  $\mathbf{C}_i = \{\mathbf{c}_1, \mathbf{c}_2, \ldots\}$ . At each time step  $t$ , the decoder in TranX performs hierarchical attention over representations in  $\mathbb{M}$  to compute a context vector. First, a table-wise attention score is computed using the LSTM's previous state,  $\mathbf{state}_{t-1}$  with the set of table representations.
$$
\begin{array}{l} \operatorname {s c o r e} \left(T _ {i}\right) = \operatorname {S o f t m a x} \Big ( \\ \mathrm {D o t P r o d u c t} (\mathbf {s t a t e} _ {t - 1}, \mathrm {k e y} (\mathbf {T} _ {i})) \Big), (4) \\ \end{array}
$$
where the linear projection  $\mathrm{key}(\cdot)\in \mathbb{R}^{256}$  projects the table representations to key space. Next, for each table  $T_{i}\in \mathcal{T}$ , a table-wise context vector  $\mathbf{ctx}(T_i)$  is generated by attending over the union
of vectors in utterance token representations  $\mathbf{X}_i$  and column representations  $\mathbf{C}_i$ :
$$
\mathbf {c t x} (T _ {i}) = \text {D o t P r o d u c t A t t e n t i o n}
$$
$$
\left. \mathbf {s t a t e} _ {t - 1}, \operatorname {k e y} \left(\mathbf {X} _ {i} \cup \mathbf {C} _ {i}\right), \operatorname {v a l u e} \left(\mathbf {X} _ {i} \cup \mathbf {C} _ {i}\right)\right), \tag {5}
$$
with the LSTM state as the query,  $\mathrm{key}(\cdot)$  as the key, and another linear transformation value  $(\cdot)\in \mathbb{R}^{256}$  to project the representations to value vectors. The final context vector is then given by the weighted sum of these table-wise context vectors  $\mathbf{ctx}(T_i)$ $(i\in \{1,\ldots ,|\mathcal{T}|\})$  weighted by the attention scores score  $(T_{i})$ . The generated context vector is then used to update the state of the decoder LSTM to state  $t$ .
The updated decoder state is then used to compute the probability of carrying out the action defined at time step  $t$ ,  $a_{t}$ . For a SELECTTABLE( $T_{i}$ ) action, its probability of is defined similarly as Eq. (4). For a SELECTCOLUMN( $T_{i}$ ,  $c_{j}$ ) action, it is factorized as the probability of selecting the table  $T_{i}$  (given by Eq. (4)), times the probability of selecting the column  $c_{j}$ . The latter is defined as
$$
\operatorname {s c o r e} \left(c _ {j}\right) = \operatorname {S o f t m a x} \left(\operatorname {D o t P r o d u c t} \left(\mathbf {s t a t e} _ {t}, \mathbf {c} _ {j}\right)\right). \tag {6}
$$
We also add simple entity linking features to the representations in  $\mathbb{M}$ , defined by the following heuristics: (1) If an utterance token  $x \in \mathbf{u}$  matches with the name of a table  $T$ , we concatenate a trainable embedding vector (table_MATCH  $\in \mathbb{R}^{16}$ ) to the representations of  $x$  and  $T$ . (2) Similarly, we concatenate an embedding vector (column_MATCH  $\in \mathbb{R}^{16}$ ) to the representations of an utterance token and a column if their names match. (3) Finally, we concatenate a zero-vector  $(\mathbf{0} \in \mathbb{R}^{16})$  to representations of all unmatched elements.
Configuration We use the default configuration of TranX. For TABERT parameters, we use an Adam optimizer with a learning rate of  $3e - 5$  and linearly decayed learning rate schedule, and
8425
Figure 2: ASDL Grammar of SemQL used in TranX
another Adam optimizer with a constant learning rate of  $1e - 3$  for all remaining parameters. During training, we update model parameters for 25000 iterations, and freeze the TABERT parameters at the first 1000 update steps. We use a batch size of 30 and beam size of 3. We use gradient accumulation for large models to fit a batch into GPU memory.
B.2 Weakly-supervised Parsing on WIKITABLEQUESTIONS
Model We use MAPO (Liang et al., 2018), a strong weakly-supervised semantic parser. The original MAPO models comes with an LSTM encoder, which generates utterance and column representations used by the decoder to predict table queries. We directly substitute the encoder with TABERT, and project the utterance and table representations from TABERT to the original embedding space using a linear transformation. MAPO uses
a domain-specific query language tailored to answer compositional questions on a single table. For instance, the example question in Fig. 1 could be answered using the following query:
MAPO is written in Tensorflow. In our experiments we use an optimized re-implementation in PyTorch, which yields  $4 \times$  training speedup.
Configuration We use the same optimizer and learning rate schedule as in § B.1. We use a batch size of 10, and train the model for 20000 steps, with the TABERT parameters frozen at the first 5000 steps. Other hyper-parameters are kept the same as the original MAPO system.
8426