SlowGuess/ACL_2020_OCR huggingface dataset:
WIKITABLEQUESTIONS (Pasupat and Liang, 2015) is a popular dataset for weakly supervised semantic parsing, which has 22,033 utterances and 2,108 semi-structured Web tables from Wikipedia. Compared to SPIDER, examples in this dataset do not involve joining multiple tables, but typically require compositional, multi-hop reasoning over a series of entries in the given table (e.g., to answer the example in Fig. 1 the parser needs to reason over the row set  $\{R_2,R_3,R_5\}$ , locating the Venue field with the largest value of Year).
Base Semantic Parser MAPO (Liang et al., 2018) is a strong system for weakly supervised semantic parsing. It improves the sample efficiency of the REINFORCE algorithm by biasing the exploration of queries towards the high-rewarding ones already discovered by the model. MAPO uses a domain-specific query language tailored to answering compositional questions on single tables, and its utterances and column representations are derived from an LSTM encoder, which we replaced with our TABERT model. See Appendix § B.2 for details of MAPO and our adaptation.
In this section we evaluate TABERT on downstream tasks of semantic parsing to DB tables.
Pretraining Configuration We train two variants of the model,  $\mathrm{TABERT_{Base}}$  and  $\mathrm{TABERT_{Large}}$ , with the underlying Transformer model initialized with the uncased versions of  $\mathrm{BERT_{Base}}$  and  $\mathrm{BERT_{Large}}$ , respectively. During pretraining, for each table and its associated NL context in the corpus, we create a series of training instances of paired NL sentences (as synthetically generated utterances) and tables (as content snapshots) by (1) sliding a (non-overlapping) context window of sentences with a maximum length of 128 tokens, and (2) using the NL tokens in the window as the utterance, and pairing it with randomly sampled rows from the table as content snapshots. TABERT is implemented in PyTorch using distributed training. Refer to Appendix § A.2 for details of pretraining.
Comparing Models We mainly present results for two variants of TABERT by varying the size of content snapshots  $K$ . TABERT( $\mathbf{K} = 3$ ) uses three rows from input tables as content snapshots and three vertical self-attention layers. TABERT( $\mathbf{K} = 1$ ) uses one synthetically generated row as the content snapshot as described in § 3.1. Since this model does not have multi-row input, we do not use additional vertical attention layers (and the cell value recovery learning objective). Its column representation  $\mathbf{c}_j$  is defined by mean-pooling over the Transformer's output encodings that correspond to the column name (e.g., the representation for the Year column in Fig. 1 is derived from the vector of the Year token in Eq. (1)). We find this strategy gives better results compared with using the cell representation  $\mathbf{s}_j$  as  $\mathbf{c}_j$ . We also compare with BERT using the same row linearization and content snapshot approach as TABERT( $\mathbf{K} = 1$ ), which reduces to a TABERT( $\mathbf{K} = 1$ ) model without pretraining on tabular corpora.
Evaluation Metrics As standard, we report execution accuracy on WIKITABLEQUESTIONS and exact-match accuracy of DB queries on SPIDER.
Tab. 1 and Tab. 2 summarize the end-to-end evaluation results on WIKITABLEQUESTIONS and SPIDER, respectively. First, comparing with existing strong semantic parsing systems, we found our
<sup>8</sup>We also attempted to train TABERT on our collected corpus from scratch without initialization from BERT, but with inferior results, potentially due to the average lower quality of web-scraped tables compared to purely textual corpora. We leave improving the quality of training data as future work.
While some of the 421 testing Wikipedia tables might be included in our pretraining corpora, they only account for a very tiny fraction. In our pilot study, we also found pretraining only on Wikipedia tables resulted in worse performance.
8418
Previous Systems on WikiTableQuestions
Model	DEV	TEST
Pasupat and Liang (2015)	37.0	37.1
Neelakantan et al. (2016)	34.1	34.2
Ensemble 15 Models	37.5	37.7
Zhang et al. (2017)	40.6	43.7
Dasigi et al. (2019)	43.1	44.3
Agarwal et al. (2019)	43.2	44.1
Ensemble 10 Models	-	46.9
Wang et al. (2019b)	43.7	44.5
Our System based on MAPO (Liang et al., 2018)
DEV	Best	TEST	Best
Base Parse†	42.3 ±0.3	42.7	43.1 ±0.5	43.8
w/ BERTBase (K = 1)	49.6 ±0.5	50.4	49.4 ±0.5	49.2
- content snapshot	49.1 ±0.6	50.0	48.8 ±0.9	50.2
w/ TABERTBase (K = 1)	51.2 ±0.5	51.6	50.4 ±0.5	51.2
- content snapshot	49.9 ±0.4	50.3	49.4 ±0.4	50.0
w/ TABERTBase (K = 3)	51.6 ±0.5	52.4	51.4 ±0.3	51.3
w/ BERTLarge (K = 1)	50.3 ±0.4	50.8	49.6 ±0.5	50.1
w/ TABERTLarge (K = 1)	51.6 ±1.1	52.7	51.2 ±0.9	51.5
w/ TABERTLarge (K = 3)	52.2 ±0.7	53.0	51.8 ±0.6	52.3
parsers with TABERT as the utterance and table encoder perform competitively. On the test set of WIKITABLEQUESTIONS, MAPO augmented with a  $\mathrm{TABERTLarge}$  model with three-row content snapshots,  $\mathrm{TABERTLarge}(\mathrm{K} = 3)$ , registers a single-model exact-match accuracy of  $52.3\%$ , surpassing the previously best ensemble system  $(46.9\%)$  from Agarwal et al. (2019) by  $5.4\%$  absolute.
On SPIDER, our semantic parser based on TranX and SesQL (§ 4.1) is conceptually similar to the base version of IRNet as both systems use the SesQL grammar, while our system has a simpler decoder. Interestingly, we observe that its performance with BERTBase (61.8%) matches the full BERT-augmented IRNet model with a stronger decoder using augmented memory and coarse-to-fine decoding (61.9%). This confirms that our base parser is an effective baseline. Augmented with representations produced by TABERTLarge(K = 3), our parser achieves up to 65.2% exact-match accuracy, a 2.8% increase over the base model using BERTBase. Note that while other competitive systems on the leaderboard use BERT with more sophisticated semantic parsing models, our best DEV. result is already close to the score registered by the best submission (RyanSQL+BERT). This suggests that if they instead used TABERT as the representation layer, they would see further gains.
Comparing semantic parsers augmented with
Table 1: Execution accuracies on WIKITABLEQUESTIONS. Results from Liang et al. (2018). (TA)BERT models are evaluated with 10 random runs. We report mean, standard deviation and the best results. TEST  $\mapsto$  BEST refers to the result from the run with the best performance on DEV. set.
Top-ranked Systems on Spider Leaderboard
Model	DEV. ACC.
Global-GNN (Bogin et al., 2019a)	52.7
EditSQL + BERT (Zhang et al., 2019a)	57.6
RatSQL (Wang et al., 2019a)	60.9
IRNet + BERT (Guo et al., 2019)	60.3
+ Memory + Coarse-to-Fine	61.9
IRNet V2 + BERT	63.9
RyanSQL + BERT (Choi et al., 2020)	66.6
Our System based on TranX (Yin and Neubig, 2018)
Mean	Best
w/ BERTBase (K = 1)	61.8 ±0.8	62.4
- content snapshot	59.6 ±0.7	60.3
w/ TABERTBase (K = 1)	63.3 ±0.6	64.2
- content snapshot	60.4 ±1.3	61.8
w/ TABERTBase (K = 3)	63.3 ±0.7	64.1
w/ BERTLarge (K = 1)	61.3 ±1.2	62.9
w/ TABERTLarge (K = 1)	64.0 ±0.4	64.4
w/ TABERTLarge (K = 3)	64.5 ±0.6	65.2
Table 2: Exact match accuracies on the public development set of SPIDER. Models are evaluated with 5 random runs.
TABERT and BERT, we found TABERT is more effective across the board. We hypothesize that the performance improvements would be attributed by two factors. First, pre-training on large parallel textual and tabular corpora helps TABERT learn to encode structure-rich tabular inputs in their linearized form (Eq. (1)), whose format is different from the ordinary natural language data that BERT is trained on. Second, pre-training on parallel data could also helps the model produce representations that better capture the alignment between an utterance and the relevant information presented in the structured schema, which is important for semantic parsing.
Overall, the results on the two benchmarks demonstrate that pretraining on aligned textual and tabular data is necessary for joint understanding of NL utterances and tables, and TABERT works well with both structured (SPIDER) and semi-structured (WIKITABLEQUESTIONS) DBs, and agnostic of the task-specific structures of semantic parsers.
Effect of Content Snapshots In this paper we propose using content snapshots to capture the information in input DB tables that is most relevant to answering the NL utterance. We therefore study the effectiveness of including content snapshots when generating schema representations. We include in Tab. 1 and Tab. 2 results of models without using content in row linearization ("content snapshot"). Under this setting a column is rep
8419
u: How many years before was the film Bacchae out before the Watermelon?
Input to TABERTLarge (K = 3) ▷ Content Snapshot with Three Rows
Film	Year	Function	Notes
The Bacchae	2002	Producer	Screen adaptation of...
The Trojan Women	2004	Producer/Actress	Documutary film...
The Watermelon	2008	Producer	Oddball romantic comedy...
Input to TABERTLarge (K = 1) ▷ Content Snapshot with One Synthetic Row
Film	Year	Function	Notes
The Watermelon	2013	Producer	Screen adaptation of...
represented as "Column Name | Type" without cell values (c.f., Eq. (1)). We find that content snapshots are helpful for both BERT and TABERT, especially for TABERT. As discussed in § 3.1, encoding sampled values from columns in learning their representations helps the model infer alignments between entity and relational phrases in the utterance and the corresponding column. This is particularly helpful for identifying relevant columns from a DB table that is mentioned in the input utterance. As an example, empirically we observe that on SPIDER our semantic parser with TABERTBase using just one row of content snapshots  $(\mathrm{K} = 1)$  registers a higher accuracy of selecting the correct columns when generating SQL queries (e.g., columns in SELECT and WHERE clauses), compared to the TABERTBase model without encoding content information (87.4% v.s. 86.4%).
Additionally, comparing TABERT using one synthetic row  $(\mathrm{K} = 1)$  and three rows from input tables  $(\mathrm{K} = 3)$  as content snapshots, the latter generally performs better. Intuitively, encoding more table contents relevant to the input utterance could potentially help answer questions that involve reasoning over information across multiple rows in the table. Tab. 3 shows such an example, and to answer this question a parser need to subtract the values of Year in the rows for "The Watermelon" and "The Bacchae".  $\mathrm{TABERTLarge}$ $(\mathrm{K} = 3)$  is able to capture the two target rows in its content snapshot and generates the correct DB query, while the  $\mathrm{TABERTLarge}$ $(\mathrm{K} = 1)$  model with only one row as content snapshot fails to answer this example.
Effect of Row Linearization TABERT uses row linearization to represent a table row as sequential input to Transformer. Tab. 4 (upper half) presents results using various linearization methods. We find adding type information and content snapshots improves performance, as they provide more hints about the meaning of a column.
Table 3: Content snapshots generated by two models for a WIkITABLEQUESTIONS DEV. example. Matched tokens between the question and content snapshots are underlined.
Cell Linearization Template	WIKIQ.	SPIDER
Pretrained TABERTBase Models (K = 1)
Column Name	49.6 ±0.4	60.0 ±1.1
Column Name | Type† (−content snap.)	49.9 ±0.4	60.4 ±1.3
Column Name | Type | Cell Value†	51.2 ±0.5	63.3 ±0.6
BERTBase Models
Column Name (Hwang et al., 2019)	49.0 ±0.4	58.6 ±0.3
Column Name is Cell Value (Chen19)	50.2 ±0.4	63.1 ±0.7
Table 4: Performance of pretrained TABERTBase models and BERTBase on the DEV. sets with different linearization methods. Slot names are underlined. Results copied from Tab. 1 and Tab. 2.
Learning Objective	WIKIQ.	SPIDER
MCP only	51.6 ±0.7	62.6 ±0.7
MCP + CVR	51.6 ±0.5	63.3 ±0.7
Table 5: Performance of pretrained  $\mathrm{TABERT_{Base}}(\mathrm{K} = 3)$  on DEV. sets with different pretraining objectives.
We also compare with existing linearization methods in literature using a TABERT<sub>Base</sub> model, with results shown in Tab. 4 (lower half). Hwang et al. (2019) uses BERT to encode concatenated column names to learn column representations. In line with our previous discussion on the effectiveness content snapshots, this simple strategy without encoding cell contents underperforms (although with TABERT<sub>Base</sub> pretrained on our tabular corpus the results become slightly better). Additionally, we remark that linearizing table contents has also been applied to other BERT-based tabular reasoning tasks. For instance, Chen et al. (2019) propose a "natural" linearization approach for checking if an NL statement entails the factual information listed in a table using a binary classifier with representations from BERT, where a table is linearized by concatenating the semicolon-separated cell linearization for all rows. Each cell is represented by a phrase "column name is cell value". For completeness, we also tested this cell linearization approach, and find  $\mathrm{BERT}_{\mathrm{Base}}$  achieved improved results. We leave pretraining TABERT with this linearization strategy as promising future work.
Impact of Pretraining Objectives TABERT uses two objectives (§ 3.2), a masked column prediction (MCP) and a cell value recovery (CVR) objective, to learn column representations that could capture both the general information of the column (via MCP) and its representative cell values related to the utterance (via CVR). Tab. 5 shows ablation results of pretraining TABERT with different objectives. We find TABERT trained with both MCP and the auxiliary CVR objectives gets a slight advantage, suggesting CVR could potentially lead to
more representative column representations with additional cell information.
Semantic Parsing over Tables Tables are important media of world knowledge. Semantic parsers have been adapted to operate over structured DB tables (Wang et al., 2015; Xu et al., 2017; Dong and Lapata, 2018; Yu et al., 2018b; Shi et al., 2018; Wang et al., 2018), and open-domain, semistructured Web tables (Pasupat and Liang, 2015; Sun et al., 2016; Neelakantan et al., 2016). To improve representations of utterances and tables for neural semantic parsing, existing systems have applied pretrained word embeddings (e.g., GloVe, as in Zhong et al. (2017); Yu et al. (2018a); Sun et al. (2018); Liang et al. (2018)), and BERT-family models for learning joint contextual representations of utterances and tables, but with domain-specific approaches to encode the structured information in tables (Hwang et al., 2019; He et al., 2019; Guo et al., 2019; Zhang et al., 2019a). TABERT advances this line of research by presenting a general-purpose, pretrained encoder over parallel corpora of Web tables and NL context. Another relevant direction is to augment representations of columns from an individual table with global information of its linked tables defined by the DB schema (Bogin et al., 2019a; Wang et al., 2019a). TABERT could also potentially improve performance of these systems with improved table-level representations.
Knowledge-enhanced Pretraining Recent pretraining models have incorporated structured information from knowledge bases (KBs) or other structured semantic annotations into training contextual word representations, either by fusing vector representations of entities and relations on KBs into word representations of LMs (Peters et al., 2019; Zhang et al., 2019b,c), or by encouraging the LM to recover KB entities and relations from text (Sun et al., 2019; Liu et al., 2019a). TABERT is broadly relevant to this line in that it also exposes an LM with structured data (i.e., tables), while aiming to learn joint representations for both textual and structured tabular data.
We present TABERT, a pretrained encoder for joint understanding of textual and tabular data. We show that semantic parsers using TABERT as a general-purpose feature representation layer
achieved strong results on two benchmarks. This work also opens up several avenues for future work. First, we plan to evaluate TABERT on other related tasks involving joint reasoning over textual and tabular data (e.g., table retrieval and table-to-text generation). Second, following the discussions in § 5, we will explore other table linearization strategies with Transformers, improving the quality of pretraining corpora, as well as novel unsupervised objectives. Finally, to extend TABERT to cross-lingual settings with utterances in foreign languages and structured schemas defined in English, we plan to apply more advanced semantic similarity metrics for creating content snapshots.
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. 2019. Learning to generalize from sparse and underspecified rewards. In ICML.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of EMNLP.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of ACL.
Ben Boin, Matt Gardner, and Jonathan Berant. 2019a. Global reasoning over database structures for text-tosql parsing. ArXiv, abs/1908.11214.
Ben Bogin, Matthew Gardner, and Jonathan Berant. 2019b. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of ACL.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. ArXiv, abs/1909.02164.
Donghyun Choi, Myeong Cheol Shin, EungGyun Kim, and Dong Ryeol Shin. 2020. *Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases.* ArXiv, abs/2004.03125.
Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke S. Zettlemoyer, and Eduard H. Hovy. 2019. Iterative search for weakly supervised semantic parsing. In Proceedings of NAACL-HLT.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
Li Dong and Mirella Lapata. 2018. coarse-to-fine decoding for neural semantic parsing. In Proceedings of ACL.
8421
Yoav Goldberg. 2019. Assessing bert's syntactic abilities. ArXiv, abs/1901.05287.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of ACL.
Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-sql: reinforce schema representation with context. ArXiv, abs/1908.08113.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). ArXiv, abs/1606.08415.
Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. ArXiv, abs/1902.01069.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. In Proceedings of EMNLP.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of EMNLP.
Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer. 2016. A large public corpus of web tables containing time and context metadata. In Proceedings of WWW.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of NIPS.
Jindrich Libovicky and Jindrich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of ACL.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2019a. K-bert: Enabling language representation with knowledge graph. ArXiv, abs/1909.07606.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proceedings of NIPS.
Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of CoNLL.
Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. 2016. Neural programmer: Inducing latent programs with gradient descent. In Proceedings of ICLR.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of ACL.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL.
Matthew E. Peters, Mark Neumann, IV RobertLLogan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of EMNLP.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of EMNLP.
Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. Incsql: Training incremental text-to-sql parsers with non-deterministic oracles. ArXiv, abs/1809.05054.
Huan Sun, Hao Ma, Xiaodong He, Wen tau Yih, Yu Su, and Xifeng Yan. 2016. Table cell search for question answering. In Proceedings of WWW.
Yibo Sun, Duyu Tang, Nan Duan, Jianshu Ji, Guihong Cao, Xiaocheng Feng, Bing Qin, Ting Liu, and Ming Zhou. 2018. Semantic parsing with syntax- and table-aware SQL generation. In Proceedings of EMNLP.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. ArXiv, abs/1904.09223.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Margot Richardson. 2019a. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. ArXiv, abs/1911.04942.
Bailin Wang, Ivan Titov, and Mirella Lapata. 2019b. Learning semantic parsers from denotations with latent structured alignments and abstract programs. In EMNLP/IJCNLP.
Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Xin Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-sql generation with execution-guided decoding. ArXiv, abs/1807.03100.
8422