SlowGuess/ACL_2020_OCR huggingface dataset:
TABERT: Pretraining for Joint Understanding of Textual and Tabular Data
Pengcheng Yin* Graham Neubig
{pcyin,gneubig}@cs.cmu.edu
Wen-tau Yih Sebastian Riedel
{scottyih,sriedel}@fb.com
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TABERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TABERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TABERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WIKITABLEQUESTIONS, while performing competitively on the text-to-SQL dataset SPIDER.
Recent years have witnessed a rapid advance in the ability to understand and answer questions about free-form natural language (NL) text (Rajpurkar et al., 2016), largely due to large-scale, pretrained language models (LMs) like BERT (Devlin et al., 2019). These models allow us to capture the syntax and semantics of text via representations learned in an unsupervised manner, before fine-tuning the model to downstream tasks (Melamud et al., 2016; McCann et al., 2017; Peters et al., 2018; Liu et al., 2019b; Yang et al., 2019; Goldberg, 2019). It is also relatively easy to apply such pretrained LMs to comprehension tasks that are modeled as text span selection problems, where the boundary of an answer span can be predicted using a simple classifier on top of the LM (Joshi et al., 2019).
However, it is less clear how one could pretrain and fine-tune such models for other QA tasks that involve joint reasoning over both free-form NL text and structured data. One example task is semantic parsing for access to databases (DBs) (Zelle and Mooney, 1996; Berant et al., 2013; Yih et al., 2015), the task of transducing an NL utterance (e.g., "Which country has the largest GDP?") into a structured query over DB tables (e.g., SQL querying a database of economics). A key challenge in this scenario is understanding the structured schema of DB tables (e.g., the name, data type, and stored values of columns), and more importantly, the alignment between the input text and the schema (e.g., the token "GDP" refers to the Gross Domestic Product column), which is essential for inferring the correct DB query (Berant and Liang, 2014).
Neural semantic parsers tailored to this task therefore attempt to learn joint representations of NL utterances and the (semi-)structured schema of DB tables (e.g., representations of its columns or cell values, as in Krishnamurthy et al. (2017); Boin et al. (2019b); Wang et al. (2019a), inter alia). However, this unique setting poses several challenges in applying pretrained LMs. First, information stored in DB tables exhibit strong underlying structure, while existing LMs (e.g., BERT) are solely trained for encoding free-form text. Second, a DB table could potentially have a large number of rows, and naively encoding all of them using a resource-heavy LM is computationally intractable. Finally, unlike most text-based QA tasks (e.g., SQuAD, Rajpurkar et al. (2016)) which could be formulated as a generic answer span selection problem and solved by a pretrained model with additional classification layers, semantic parsing is highly domain-specific, and the architecture of a neural parser is strongly coupled with the structure of its underlying DB (e.g., systems for SQL-based and other types of DBs use different encoder mod
*Work done while at Facebook AI Research.
Available at github.com/facebookresearch/TaBERT
8413
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413-8426 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
els). In fact, existing systems have attempted to leverage BERT, but each with their own domain-specific, in-house strategies to encode the structured information in the DB (Guo et al., 2019; Zhang et al., 2019a; Hwang et al., 2019), and importantly, without pretraining representations on structured data. These challenges call for development of general-purpose pretraining approaches tailored to learning representations for both NL utterances and structured DB tables.
In this paper we present TABERT, a pretraining approach for joint understanding of NL text and (semi-)structured tabular data (§ 3). TABERT is built on top of BERT, and jointly learns contextual representations for utterances and the structured schema of DB tables (e.g., a vector for each utterance token and table column). Specifically, TABERT linearizes the structure of tables to be compatible with a Transformer-based BERT model. To cope with large tables, we propose content snapshots, a method to encode a subset of table content most relevant to the input utterance. This strategy is further combined with a vertical attention mechanism to share information among cell representations in different rows (§ 3.1). To capture the association between tabular data and related NL text, TABERT is pretrained on a parallel corpus of 26 million tables and English paragraphs (§ 3.2).
TABERT can be plugged into a neural semantic parser as a general-purpose encoder to compute representations for utterances and tables. Our key insight is that although semantic parsers are highly domain-specific, most systems rely on representations of input utterances and the table schemas to facilitate subsequent generation of DB queries, and these representations can be provided by TABERT, regardless of the domain of the parsing task.
We apply TABERT to two different semantic parsing paradigms: (1) a classical supervised learning setting on the SPIDER text-to-SQL dataset (Yu et al., 2018c), where TABERT is fine-tuned together with a task-specific parser using parallel NL utterances and labeled DB queries (§ 4.1); and (2) a challenging weakly-supervised learning benchmark WIKITABLEQUESTIONS (Pasupat and Liang, 2015), where a system has to infer latent DB queries from its execution results (§ 4.2). We demonstrate TABERT is effective in both scenarios, showing that it is a drop-in replacement of a parser's original encoder for computing contextual representations of NL utterances and DB tables.
Specifically, systems augmented with TABERT outperforms their counterparts using BERT, registering state-of-the-art performance on WIKITABLE-QUESTIONS, while performing competitively on SPIDER (§ 5).
Semantic Parsing over Tables Semantic parsing tackles the task of translating an NL utterance  $\pmb{u}$  into a formal meaning representation (MR)  $\pmb{z}$ . Specifically, we focus on parsing utterances to access database tables, where  $\pmb{z}$  is a structured query (e.g., an SQL query) executable on a set of relational DB tables  $\mathcal{T} = \{T_t\}$ . A relational table  $T$  is a listing of  $N$  rows  $\{R_i\}_{i=1}^N$  of data, with each row  $R_i$  consisting of  $M$  cells  $\{s_{\langle i,j\rangle}\}_{j=1}^M$ , one for each column  $c_j$ . Each cell  $s_{\langle i,j\rangle}$  contains a list of tokens.
Depending on the underlying data representation schema used by the DB, a table could either be fully structured with strongly-typed and normalized contents (e.g., a table column named distance has a unit of kilometers, with all of its cell values, like 200, bearing the same unit), as is commonly the case for SQL-based DBs (§ 4.1). Alternatively, it could be semi-structured with unnormalized, textual cell values (e.g., 200 km, § 4.2). The query language could also take a variety of forms, from general-purpose DB access languages like SQL to domain-specific ones tailored to a particular task.
Given an utterance and its associated tables, a neural semantic parser generates a DB query from the vector representations of the utterance tokens and the structured schema of tables. In this paper we refer schema as the set of columns in a table, and its representation as the list of vectors that represent its columns<sup>2</sup>. We will introduce how TABERT computes these representations in § 3.1.
Masked Language Models Given a sequence of NL tokens  $\pmb{x} = x_{1},x_{2},\dots ,x_{n}$ , a masked language model (e.g., BERT) is an LM trained using the masked language modeling objective, which aims to recover the original tokens in  $\pmb{x}$  from a "corrupted" context created by randomly masking out certain tokens in  $\pmb{x}$ . Specifically, let  $\pmb{x}_{m} = \{x_{i_{1}},\ldots ,x_{i_{m}}\}$  be the subset of tokens in  $\pmb{x}$  selected to be masked out, and  $\widetilde{\pmb{x}}$  denote the masked sequence with tokens in  $\pmb{x}_{m}$  replaced by a [MASK] symbol. A masked LM defines a distribu
2Column representations for more complex schemas, e.g., those capturing inter-table dependency via primary and foreign keys, could be derived from these table-wise representations.
8414
In which city did Piotr's last 1st place finish occur?
Year	Venue	Position	Event
2003	Tampere	3rd	EU Junior Championship
2005	Erfurt	1st	EU U23 Championship
2005	Izmir	1st	Universiade
2006	Moscow	2nd	World Indoor Championship
2007	Bangkok	1st	Universiade
Selected Rows as Content Snapshot:  $\{R_2,R_3,R_5\}$
(A) Content Snapshot from Input Table
(B) Per-row Encoding (for each row in content snapshot, using  $R_{2}$  as an example)
Figure 1: Overview of TABERT for learning representations of utterances and table schemas with an example from WIKITABLE-QUESTIONS<sup>3</sup>. (A) A content snapshot of the table is created based on the input NL utterance. (B) Each row in the snapshot is encoded by a Transformer (only  $R_{2}$  is shown), producing row-wise encodings for utterance tokens and cells. (C) All row-wise encodings are aligned and processed by  $V$  vertical self-attention layers, generating utterance and column representations.
R [CLS] In which city did Piotr's ... [SEP] Year | real | 2005 [SEP] Venue | text | Erfurt [SEP] Position | text | 1st [SEP] ...
tion  $p_{\theta}(\pmb{x}_m|\widetilde{\pmb{x}})$  over the target tokens  $\pmb{x}_m$  given the masked context  $\widetilde{\pmb{x}}$ .
BERT parameterizes  $p_{\theta}(\pmb{x}_m|\widetilde{\pmb{x}})$  using a Transformer model. During the pretraining phase, BERT maximizes  $p_{\theta}(\pmb{x}_m|\widetilde{\pmb{x}})$  on large-scale textual corpora. In the fine-tuning phase, the pretrained model is used as an encoder to compute representations of input NL tokens, and its parameters are jointly tuned with other task-specific neural components.
3 TABERT: Learning Joint Representations over Textual and Tabular Data
We first present how TABERT computes representations for NL utterances and table schemas (§ 3.1), and then describe the pretraining procedure (§ 3.2).
3.1 Computing Representations for NL Utterances and Table Schemas
Fig. 1 presents a schematic overview of TABERT. Given an utterance  $\mathbf{u}$  and a table  $T$ , TABERT first creates a content snapshot of  $T$ . This snapshot consists of sampled rows that summarize the information in  $T$  most relevant to the input utterance. The model then linearizes each row in the snapshot, concatenates each linearized row with the utterance, and uses the concatenated string as input to a Transformer (e.g., BERT) model, which outputs row-wise encoding vectors of utterance tokens and cells. The encodings for all the rows in
the snapshot are fed into a series of vertical self-attention layers, where a cell representation (or an utterance token representation) is computed by attending to vertically-aligned vectors of the same column (or the same NL token). Finally, representations for each utterance token and column are generated from a pooling layer.
Content Snapshot One major feature of TABERT is its use of the table contents, as opposed to just using the column names, in encoding the table schema. This is motivated by the fact that contents provide more detail about the semantics of a column than just the column's name, which might be ambiguous. For instance, the Venue column in Fig. 1 which is used to answer the example question actually refers to host cities, and encoding the sampled cell values while creating its representation may help match the term "city" in the input utterance to this column.
However, a DB table could potentially have a large number of rows, with only few of them actually relevant to answering the input utterance. Encoding all of the contents using a resource-heavy Transformer is both computationally intractable and likely not necessary. Thus, we instead use a content snapshot consisting of only a few rows that are most relevant to the input utterance, providing an efficient approach to calculate content-sensitive column representations from cell values.
We use a simple strategy to create content snap
Example adapted from stanford.io/38iZ8Pf
8415
shots of  $K$  rows based on the relevance between the utterance and a row. For  $K > 1$ , we select the top- $K$  rows in the input table that have the highest  $n$ -gram overlap ratio with the utterance. $^4$  For  $K = 1$ , to include in the snapshot as much information relevant to the utterance as possible, we create a synthetic row by selecting the cell values from each column that have the highest  $n$ -gram overlap with the utterance. Using synthetic rows in this restricted setting is motivated by the fact that cell values most relevant to answer the utterance could come from multiple rows. As an example, consider the utterance "How many more participants were there in 2008 than in the London Olympics?", and an associating table with columns Year, Host City and Number of Participants, the most relevant cells to the utterance, 2008 (from Year) and London (from Host City), are from different rows, which could be included in a single synthetic row. In the initial experiments we found synthetic rows also help stabilize learning.
Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for  $R_{2}$ , which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type of the column, together with its actual value, separated by a vertical bar. As an example, the cell  $s_{\langle 2,1\rangle}$  valued 2005 in  $R_{2}$  in Fig. 1 is encoded as
Year real 2005 Column Name Column Type Cell Value
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.
Existing works have applied different linearization strategies to encode tables with Transformers (Hwang et al., 2019; Chen et al., 2019), while our row approach is specifically designed for encoding content snapshots. We present in §5 results with different linearization choices.
Vertical Self-Attention Mechanism The base Transformer model in TABERT outputs vector encodings of utterance and cell tokens for each row. These row-level vectors are computed separately and therefore independent of each other. To allow for information flow across cell representations of different rows, we propose vertical self-attention, a self-attention mechanism that operates over vertically aligned vectors from different rows.
As in Fig. 1(C), TABERT has  $V$  stacked vertical-level self-attention layers. To generate aligned inputs for vertical attention, we first compute a fixed-length initial vector for each cell at position  $\langle i,j\rangle$ , which is given by mean-pooling over the sequence of the Transformer's output vectors that correspond to its variable-length linearization as in Eq. (1). Next, the sequence of word vectors for the NL utterance (from the base Transformer model) are concatenated with the cell vectors as initial inputs to the vertical attention layer.
Each vertical attention layer has the same parameterization as the Transformer layer in (Vaswani et al., 2017), but operates on vertically aligned elements, i.e., utterance and cell vectors that correspond to the same question token and column, respectively. This vertical self-attention mechanism enables the model to aggregate information from different rows in the content snapshot, allowing TABERT to capture cross-row dependencies on cell values.
Utterance and Column Representations A representation  $\mathbf{c}_j$  is computed for each column  $c_j$  by mean-pooling over its vertically aligned cell vectors,  $\{\mathbf{s}_{\langle i,j\rangle}:R_i$  in content snapshot}, from the last vertical layer. A representation for each utterance token,  $\mathbf{x}_j$ , is computed similarly over the vertically aligned token vectors. These representations will be used by downstream neural semantic parsers. TABERT also outputs an optional fixed-length table representation  $\mathbf{T}$  using the representation of the prefixed [CLS] symbol, which is useful for parsers that operate on multiple DB tables.
3.2 Pretraining Procedure
Training Data Since there is no large-scale, high-quality parallel corpus of NL text and structured tables, we instead use semi-structured tables that commonly exist on the Web as a surrogate data source. As a first step in this line, we focus on collecting parallel data in English, while extending to multilingual scenarios would be an
<sup>4</sup>We use  $n \leq 3$  in our experiments. Empirically this simple matching heuristic is able to correctly identify the best-matched rows for 40 out of 50 sampled examples on WIKITABLEQUESTIONS.
5We use two data types, text, and real for numbers, predicted by majority voting over the NER labels of cell tokens.
8416
interesting avenue for future work. Specifically, we collect tables and their surrounding NL text from English Wikipedia and the WDC WebTable Corpus (Lehmberg et al., 2016), a large-scale table collection from CommonCrawl. The raw data is extremely noisy, and we apply aggressive cleaning heuristics to filter out invalid examples (e.g., examples with HTML snippets or in foreign languages, and non-relational tables without headers). See Appendix § A.1 for details of data pre-processing. The pre-processed corpus contains 26.6 million parallel examples of tables and NL sentences. We perform sub-tokenization using the Wordpiece tokenizer shipped with BERT.
Unsupervised Learning Objectives We apply different objectives for learning representations of the NL context and structured tables. For NL contexts, we use the standard Masked Language Modeling (MLM) objective (Devlin et al., 2019), with a masking rate of  $15\%$  sub-tokens in an NL context.
For learning column representations, we design two objectives motivated by the intuition that a column representation should contain both the general information of the column (e.g., its name and data type), and representative cell values relevant to the NL context. First, a Masked Column Prediction (MCP) objective encourages the model to recover the names and data types of masked columns. Specifically, we randomly select  $20\%$  of the columns in an input table, masking their names and data types in each row linearization (e.g., if the column Year in Fig. 1 is selected, the tokens Year and real in Eq. (1) will be masked). Given the column representation  $\mathbf{c}_j$ , TABERT is trained to predict the bag of masked (name and type) tokens from  $\mathbf{c}_j$  using a multi-label classification objective. Intuitively, MCP encourages the model to recover column information from its contexts.
Next, we use an auxiliary Cell Value Recovery (CVR) objective to ensure information of representative cell values in content snapshots is retained after additional layers of vertical self-attention. Specifically, for each masked column  $c_{j}$  in the above MCP objective, CVR predicts the original tokens of each cell  $s_{\langle i,j\rangle}$  (of  $c_{j}$ ) in the content snapshot conditioned on its cell vector  $\mathbf{s}_{\langle i,j\rangle}$ . For instance, for the example cell  $s_{\langle 2,1\rangle}$  in Eq. (1), we predict its value 2005 from  $\mathbf{s}_{\langle 2,1\rangle}$ . Since a cell
could have multiple value tokens, we apply the span-based prediction objective (Joshi et al., 2019). Specifically, to predict a cell token  $s_{\langle i,j\rangle_k} \in s_{\langle i,j\rangle}$ , its positional embedding  $\mathbf{e}_k$  and the cell representations  $\mathbf{s}_{\langle i,j\rangle}$  are fed into a two-layer network  $f(\cdot)$  with GeLU activations (Hendrycks and Gimpel, 2016). The output of  $f(\cdot)$  is then used to predict the original value token  $s_{\langle i,j\rangle_k}$  from a softmax layer.
4 Example Application: Semantic Parsing over Tables
We apply TABERT for representation learning on two semantic parsing paradigms, a classical supervised text-to-SQL task over structured DBs (§ 4.1), and a weakly supervised parsing problem on semistructured Web tables (§ 4.2).
4.1 Supervised Semantic Parsing
Benchmark Dataset Supervised learning is the typical scenario of learning a parser using parallel data of utterances and queries. We use SPIDER (Yu et al., 2018c), a text-to-SQL dataset with 10,181 examples across 200 DBs. Each example consists of an utterance (e.g., "What is the total number of languages used in Aruba?"), a DB with one or more tables, and an annotated SQL query, which typically involves joining multiple tables to get the answer (e.g., SELECT COUNT(*) FROM Country JOIN Lang ON Country.Code = Lang.CountryCode WHERE Name = 'Aruba').
Base Semantic Parser We aim to show TABERT could help improve upon an already strong parser. Unfortunately, at the time of writing, none of the top systems on SPIDER were publicly available. To establish a reasonable testbed, we developed our in-house system based on TranX (Yin and Neubig, 2018), an open-source general-purpose semantic parser. TranX translates an NL utterance into an intermediate meaning representation guided by a user-defined grammar. The generated intermediate MR could then be deterministically converted to domain-specific query languages (e.g., SQL).
We use TABERT as encoder of utterances and table schemas. Specifically, for a given utterance  $\mathbf{u}$  and a DB with a set of tables  $\mathcal{T} = \{T_t\}$ , we first pair  $\mathbf{u}$  with each table  $T_{t}$  in  $\mathcal{T}$  as inputs to TABERT, which generates  $|\mathcal{T}|$  sets of table-specific representations of utterances and columns. At each time step, an LSTM decoder performs hierarchical attention (Libovický and Helcl, 2017) over the list of table-specific representations, constructing an
The cell value tokens are not masked in the input sequence, since predicting masked cell values is challenging even with the presence of its surrounding context.
MR based on the predefined grammar. Following the IRNet model (Guo et al., 2019) which achieved the best performance on SPIDER as the time of writing, we use SemQL, a simplified version of the SQL, as the underlying grammar. We refer interested readers to Appendix § B.1 for details of our system.
4.2 Weakly Supervised Semantic Parsing
Benchmark Dataset Weakly supervised semantic parsing considers the reinforcement learning task of inferring the correct query from its execution results (i.e., whether the answer is correct). Compared to supervised learning, weakly supervised parsing is significantly more challenging, as the parser does not have access to the labeled query, and has to explore the exponentially large search space of possible queries guided by the noisy binary reward signal of execution results.