SlowGuess/NAACL_2022_OCR huggingface dataset:
Fei Wang, Zhewei Xu, Pedro Szekely and Muhao Chen
Department of Computer Science & Information Sciences Institute
{fwang598,zheweixu,szekely,muhaoche}@usc.edu
Controlled table-to-text generation seeks to generate natural language descriptions for highlighted subparts of a table. Previous SOTA systems still employ a sequence-to-sequence generation method, which merely captures the table as a linear structure and is brittle when table layouts change. We seek to go beyond this paradigm by (1) effectively expressing the relations of content pieces in the table, and (2) making our model robust to content-invariant structural transformations. Accordingly, we propose an equivariance learning framework, LATTICE (3), which encodes tables with a structure-aware self-attention mechanism. This prunes the full self-attention structure into an order-invariant graph attention that captures the connected graph structure of cells belonging to the same row or column, and it differentiates between relevant cells and irrelevant cells from the structural perspective. Our framework also modifies the positional encoding mechanism to preserve the relative position of tokens in the same cell but enforce position invariance among different cells. Our technology is free to be plugged into existing table-to-text generation models, and has improved T5-based models to offer better performance on ToTTo and HiTab. Moreover, on a harder version of ToTTo, we preserve promising performance, while previous SOTA systems, even with transformation-based data augmentation, have seen significant performance drops.
Table-to-text generation seeks to generate natural language descriptions for content and entailed conclusions in tables. It is an important task that not only makes ubiquitous tabular data more discoverable and accessible, but also supports downstream tasks of tabular semantic retrieval (Wang et al., 2021a), reasoning (Gupta et al., 2020), fact
1Our code is available at https://github.com/ luka-group/Lattice.
Figure 1: Description generation on content-equivalent tables with different layouts by T5 and LATTICE. Correct film-role pairs in generations are in orange. We report also the BLEU-4 score of each generation. T5 is brittle to layout changes, while LATTICE returns consistent results.
checking (Chen et al., 2019; Wang et al., 2021b) and table-assisted question answering (Chen et al., 2020c). While rich and diverse facts can be presented in a table, the controlled table-to-text generation task, which generates focused textual descriptions for highlighted subparts of a table, has garnered much attention recently (Parikh et al., 2020; Kale and Rastogi, 2020; Cheng et al., 2022).
Prior studies on controlled table-to-text generation often employ a sequence-to-sequence generation method, which merely captures the table as a linear structure (Parikh et al., 2020; Kale and Rastogi, 2020; Su et al., 2021). However, table
2This example is from the ToTTo dataset. Original film names and role names are too long. For presentation, we replace the actor name, film names and role names.
Human Language Technologies, pages 5037 - 5048
Layout, though overlooked by prior studies, are key to the generation from two perspectives. First, table layouts indicate the relations among cells that collectively present a fact, which are however not simply captured by a linearized table. For example, if we linearize the first table row-wise in Fig. 1, Wai Siu-bo will be next to both Royal Tramp and King of Beggers, so that it is not clear this role belongs to which film. Second, the same content can be equivalently expressed in tables with different layouts. While linearization simplifies the layout representation, it causes brittle generation when table layouts change. Fig. 1 shows two tables with the same content but different layouts, for which the generations by T5 are largely inconsistent.
In this paper, we focus on improving controlled table-to-text generation systems by incorporating two properties: structure-awareness and transformation-invariance. Structure-awareness, which seeks to understand cell relations indicated by the table structure, is essential for capturing contextualized cell information. Transformation-invariance, which seeks to make the model insensitive to content-invariant structural transformations (including transpose, row shuffle and column shuffle), is essential for model robustness. However, incorporating structure-awareness and transformation-invariance into existing generative neural networks is nontrivial, especially when preserving the generation ability of pretrained models as much as possible.
We enforce the awareness of table layouts and robustness to content-invariant structural transformations on pretrained generative models with an equivariance learning framework, namely Layout Aware and Transformation Invariant Controlled Table-to-Text GGeneration (LATTICE). LATTICE encodes tables with a transformation-invariant graph masking technology. This prunes the full self-attention structure into an order-invariant graph-based attention that captures the connected graph of cells belonging to the same row or column, and differentiates between relevant cells and irrelevant cells from the structural perspective. LATTICE also modifies the positional encoding mechanism to preserve the relative position of tokens within the same cell but enforces position invariance among different cells. Our technology is free to be plugged into existing table-to-text generation models, and has improved T5-based models (Raffel et al., 2020) on ToTTo (Parikh et al., 2020)
and HiTab (Cheng et al., 2022). Moreover, on a harder version of ToTTo, we preserve promising performance, while previous SOTA systems, even with transformation-based data augmentation, have seen significant performance drops.
Our contributions are three-fold. First, we propose two essential properties of a precise and robust controlled table-to-text generation system, i.e. structure-awareness and transformation-invariance. Second, we demonstrate how our transformation-invariant graph masking technology can enforce these two properties, and effectively enhance a representative group of Transformer-based generative models, i.e. T5-based models, for more generalizable and accurate generation. Third, in addition to experiments on ToTTo and HiTab benchmarks, we evaluate our model on a harder version of ToTTo with a special focus on robustness to content-invariant structural transformations.
In this section, we first describe the preliminaries of content-invariant table transformations, base models and the input format for controlled table-to-text generation (§2.1). Then we introduce the technical details about how the transformation-invariant graph masking technology in LATTICE enforces the model to be structure-aware and transformation-invariant (§2.2). Finally, we present two alternative techniques for strengthening the transformation-invariance to be compared with LATTICE (§2.3).
Content-Invariant Table Transformations. Tables organize and present information by rows and columns. A piece of information is presented in a cell (with headers), which is the basic unit of a table. Rows and columns are high-level units indicating relations among cells, and are combined to express more comprehensive information. We discuss two categories of transformations that may be made on a table, as shown in Fig. 2. First, content-invariant transformations modify or exchange a part of cells in different rows or columns, therefore changing the semantics of the table. In such cases, new tabular content are created to express information being inconsistent with the original table. Second, content-invariant transformations consist of operations that do not influence content within (combinations of) the same row or column, resulting in semantically equal (sub-)tables. Specifically,
Original Table
Year	Title	Role
1992	Justice, My Foot!	Sung Sai Kit
Royal Tramp	Wai Siu-bo
King of Beggars	So Chan
Content-Variant Table Transformations
Year	Title	Role
1992	Justice, My Foot!	Sung Sai Kit
Titanic	Wai Siu-bo
King of Beggars	So Chan
Year	Title	Role
1992	Justice, My Foot!	Sung Sai Kit
King of Beggars	Wai Siu-bo
Royal Tramp	So Chan
Year	1992
Title	Justice, My Foot!	Royal Tramp	King of Beggars
Role	Sung Sai Kit	Wai Siu-bo	So Chan
Content-Invariant Table Transformations
Year	Role	Title
1992	Sung Sai Kit	Justice, My Foot!
Wai Siu-bo	Royal Tramp
So Chan	King of Beggars
Year	Title	Role
1992	Justice, My Foot!	Sung Sai Kit
King of Beggars	So Chan
Royal Tramp	Wai Siu-bo
Figure 2: Examples of different types of table transformations. Arrows indicate how specific operations change the positions of tables components. Modifications causing semantic changes are in red.
such operations include transpose, row shuffle and column shuffle. By performing any or a combination of such operations, we can present the same information in different table layouts.
Base Models. Pretrained Transformer-based generative models achieve SOTA performance on various text generation tasks (Raffel et al., 2020; Lewis et al., 2020). In order to adapt this kind of models to table-to-text generation, prior works propose to linearize the table into a textual sequence (Kale and Rastogi, 2020; Chen et al., 2020b; Su et al., 2021). Our method LATTICE is model-agnostic and can be incorporated into any such models. Following Kale and Rastogi (2020), we choose a family of the best performing models, T5 (Raffel et al., 2020), as our base models. Models of this family are jointly pretrained on a series of supervised and self-supervised text-to-text tasks. Models can switch between different tasks by prepending a task-specific prefix to the input. Our experiments (\$3.3 and  $\S 3.4$ ) point out that base models are brittle to content-invariant table transformations and can only capture limited layout information.
Input Format. Prior works (Kale and Rastogi, 2020; Chen et al., 2020b; Su et al., 2021) linearize (highlighted) table cells based on row and column indexes. The input sequence often starts with the metadata of a table, such as page title and section title. Then, it traverses the table row-wise from the top-left cell to the bottom-right cell. Headers of each cell can be either treated as individual cells or appended to the cell content. Each metadata/cell/header field is separated with special tokens. This linearization process suits the input to text-to-text generation models, yet discards much of the structural information of a table (e.g.,
two cells in the same column can be separated by irrelevant cells in the sequence, while the last cell and first cell in adjacent rows can be adjacent although they are irrelevant), and is sensitive to content-invariant table transformations.
2.2 Transformation-Invariant Graph Masking
LATTICE realizes equivariance learning by modifying the Transformer encoder architecture. It also improves the base model's ability of capturing structures of highlighted tabular content. Specifically, we incorporate a structure-aware self-attention mechanism and a transformation invariant positional encoding mechanism in the base model. The workflow is shown in Fig. 3.
Structure-Aware Self-Attention. Transformer (Vaswani et al., 2017) adopts self-attention to aggregate information from all the tokens in the input sequence. The attention flows form a complete graph connecting each token. This mechanism works well for modeling sequences but falls short of capturing tabular structures. The non-linear layout structure reflects semantic relations among cells, hence should be captured by self-attention.
We incorporate structural information by pruning the attention flows. According to the nature of information arrangement in a table, two cells in neither the same row nor the same column are not semantically related, or at least the combination of them do not directly express information this table seeks to convey. Intuitively, representations of these cells should not directly pass information to each other. In LATTICE, attention flows among tokens of structurally unrelated cells are removed from the attention graph, while those within the metadata, within each cell, and between metadata and each cell are preserved. In this way, we also
Figure 3: Attention flows of the base model and LATTICE. In this example, we adopt the input format which appends headers to each cell, so headers can be seen as part of the cell content. We omit the attention flows among tokens within a cell, as they are in the same type of the flows between headers and corresponding cells.  $P_{ij}$  represents the relative position between tokens at both ends of the attention flow, where  $i$  and  $j$  are absolute positions of tokens in the linearized table and  $P_{max}$  is the max relative position allowed. The base model has a complete attention graph among all cells with relative positions based on linear distance. LATTICE prunes the attention flow based on the table layout and assigns transformation-invariant relative positions between cells.
ensure the transformation-invariance property of the self-attention mechanism, since related cells in the same row or same column are all linked in an unordered way in the attention graph. It is easy to show that for any individual cell, the links in the attention graph will remain the same after any content-invariant operations (§2.1) are applied.
Transformation-Invariant Positional Encoding. When calculating the attention scores between each pair of tokens, the base model captures their relative position in the sequence of linearized table as an influential feature. Specifically, the attention flow from the  $i$ -th token to the  $j$ -th token is paired with a relative position  $P_{ij} = |i - j|$ . This easily causes positional biases among distinct cells, since the relative positions in the sequence do not fully reflect relations among cells in the table. Moreover, the relative position between the same token pair will change as the table layout change, which is the source of inconsistent generation shown in Fig. 1.
As discussed in §2.1, for a given cell, its relations with other cells in the same row or column should be equally considered. It is natural to assign the same relative positions among (tokens of) cells in the same row or column, no matter how far their distance is in the linear sequence. Meanwhile, we preserve the relative positions of tokens inside the same cell (or the metadata). Specifically, the relative position between the  $i$ -th token and the  $j$ -th token in the input sequence is
$$
P _ {i j} = P _ {j i} = \left\{ \begin{array}{l l} | i - j |, & \text {i f i n t h e s a m e f i e l d ;} \\ P _ {m a x}, & \text {o t h e r w i s e ;} \end{array} \right.
$$
where "same field" means the two tokens are from the same cell or both of them are from the metadata, and  $P_{max}$  is the max relative position allowed. As a
result, LATTICE represents cells (and the metadata) in a way that is invariant to their relative positions in the sequence. As content-invariant table transformations do not change the relations among cells in the table (i.e. whether two cells are from the same row or column), this positional encoding mechanism is transformation-invariant.
Training and Inference. After obtaining the structure-aware and transformation-invariant table representation, LATTICE conducts similar training and inference as the base model. Given the linearized table  $T_{i}$ , its layout structure  $S_{i}$ , and target sentence  $Y_{i} = \{y_{1}^{i}, y_{2}^{i}, \dots, y_{n_{i}}^{i}\}$ , training minimizes the negative log-likelihood. For a dataset (or batch) with  $N$  samples, the loss function is
$$
L = - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n _ {i}} \log P (y _ {j} ^ {i} | y _ {<   j} ^ {i}, T _ {i}, S _ {i}).
$$
During inference, the model generates a sentence token by token, where each time it outputs a distribution over a vocabulary.
2.3 Alternative Techniques
In addition to the equivariance learning realized by transformation-invariant graph masking, we present and compare with two alternative techniques.
Layout-Agnostic Input. The first technique is to adjust input sequences to be invariant to content-invariant table transformations. A simple way is to reorder headers and cells by an arbitrary order not based on table layouts (e.g., lexicographic order) to form a sequence. Special tokens to separate cells and headers should also include no layout information<sup>3</sup>. As a result, this input format
3For example, using <header> instead of <row_header> and <column_header>.
loses all information about table layouts to ensure transformation-invariance.
Data Augmentation. The second technique is data augmentation by content-invariant table transformation. This technique augmt tables with different layouts to training data, seeking to enhance the robustness of the base model by exposing it to more diverse training instances.
Our experiments systematically compares these two techniques with transformation-invariant graph masking in §3.3, revealing how directly performing equivariance learning from the perspective of neural network structure leads to better performance and robustness than using layout-agnostic input or data augmentation.
In this section, we conduct experiments on two benchmark datasets. First, we introduce the details of datasets, baselines, evaluation metrics and our implementation (§3.1). Then, we show the overall performance of LATTICE (§3.2). After that, we analyze the model robustness on a harder version of the ToTTo dataset where content-invariant perturbations are introduced (§3.3). Finally, we provide ablation study on components of transformation-invariant graph masking (§3.4).
Datasets. We evaluate our model on ToTTo (Parikh et al., 2020) and HiTab (Cheng et al., 2022) benchmarks. Details of them are described as follows:
- ToTTo: An English dataset released under the Apache License v2.0. The dataset is dedicated to controlled table-to-text generation. It consists of 83,141 Wikipedia tables, 120,761/7,700/7,700 sentences (i.e. descriptions of tabular data) for train/dev/test. Target sentences in test set are not publicly available. Each sentence is paired with a set of highlighted cells in a table, and each table has metadata including its page title and section title. The dev and test sets can be further split into 2 subsets, i.e. overlap and non-overlap, according to whether the table exists in the training set.
- HiTab: An English dataset released under Microsoft's Computational Use of Data Agreement (C-UDA). It is intended for both controlled table-to-text generation and table-based question answering with a special focus on hierarchical tables. It contains 3,597 tables, including tables
from statistical reports and Wikipedia, forming 10,686 samples distributed across train (70%), dev (15%), and test (15%). Each sample consists of a target sentence and a table with highlighted cells and hierarchical headers.
Evaluation Metrics. We adopt three widely used evaluation metrics for text generation. BLEU (Papineni et al., 2002) is one of the most common metric for text generation based on  $n$ -gram co-occurrence. We use the commonly used BLEU-4 following prior works (Parikh et al., 2020; Cheng et al., 2022). PARENT (Dhingra et al., 2019) is a metric for data-to-text evaluation taking both references and tables into account. BLEURT (Sellam et al., 2020) is a learned evaluation metric for text generation based on BERT (Devlin et al., 2019). Following prior studies (Parikh et al., 2020; Cheng et al., 2022), we report all three metrics on ToTTo and the first two metrics on HiTab using the evaluation tool released by Parikh et al. (2020).
Baselines. We present baseline results of the following representative methods:
- Pointer-Generator (Gehrmann et al., 2018): An LSTM-based encoder-decoder model with attention and copy mechanism, first proposed by See et al. (2017) for text summarization.
- BERT-to-BERT (Rothe et al., 2020): A Transformer-based encoder-decoder model, where the encoder and decoder are initialized with BERT (Devlin et al., 2019).
- T5 (Kale and Rastogi, 2020): A pretrained generation model first proposed by Raffel et al. (2020). The model is Transformer-based, pretrained on text-to-text tasks, and finetuned on linearized tables to offer the previous SOTA performance.
All the baseline results on ToTTo can be found in the official leaderboard<sup>4</sup>, except for T5-small and T5-base, for which we reproduce the results on dev set reported by Kale and Rastogi (2020) and submit the predictions on hidden test set to the leaderboard. For HiTab, we run T5 and LATTICE using our replication of the linearization process introduced by Cheng et al. (2022)<sup>5</sup>. Results of other baselines are from Cheng et al. (2022).
4https://github.com/ google-research-datasets/ToTTo
<sup>5</sup>According to the authors, their linearization process needs unreleased raw excel files. We reproduce it with released tables which results in less precise and informative inputs.
ToTTo:  
<page_title> Stephen Chow  
</page_title> <section_title>  
Filmography </section_title>  
<table> <cell> Royal Tramp  
<header> Title </header> </cell>  
<cell> Wai Siu-bo <header> Role  
</header> </cell> </table>
HiTab:  
Title [SEP] Royal Tramp [SEP] Role [SEP] Wai Siu-bo
Figure 4: Illustration of the input format for ToTTo and HiTab. Highlighted cells are marked in yellow (i.e. Royal Tramp and Wai Siu-bo).
Implementation Details. We adopt the pretrained model weights released by Raffel et al. (2020). Specifically, we use T5-small and T5-base<sup>6</sup>. For finetuning, we use a batch size of 8 and a constant learning rate of  $2e^{-4}$ . Following Kale and Rastogi (2020) and Cheng et al. (2022), all input sequences are truncated to a length of 512 to accommodate the limit of the pretrained models. LATTICE does not add any parameters to the base model, so LATTICE (T5-small) has 60 million parameters and LATTICE (T5-base) has 220 million parameters, same as the base models. For ToTTo, we use a beam size of 4 to generate sentences with at most 128 tokens. For HiTab, we use a beam size of 5 to generate sentences with at most 60 tokens following Cheng et al. (2022). Our implementation is based on Pytorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020). We run experiments on a commodity server with a GeForce RTX 2080 GPU. It takes about 0.5 hour to train LATTICE (T5-small) for 10,000 steps and about 1 hour to train LATTICE (T5-base) for 10,000 steps. Considering different sizes of two datasets, we train models for 150,000 steps on ToTTo, and for 20,000 steps on HiTab. Results of LATTICE on ToTTo dev set and HiTab are average of multiple runs. For ToTTo test set, we report the results on official leaderboard.
As shown in Fig. 4, we use different input formats for ToTTo and Hitab following prior works (Kale and Rastogi, 2020; Cheng et al., 2022), since the tables and annotations in these two datasets are different. For ToTTo, we follow the linearization procedure by Kale and Rastogi (2020). Specifically, the textual sequence consists of the page title,
section title, table headers and cells. Each cell may be associated with multiple row and column headers. Special markers are used to denote the begin and end of each field. Different from Kale and Rastogi (2020), we use the same markers for row headers and column headers. For HiTab, we follow the linearization procedure of Cheng et al. (2022). Specifically, the textual sequence consists of highlighted cells and headers, headers of highlighted cells, and cells belong to highlighted headers. A universal separator token [SEP] is used. While our model can achieve consistently the same performance with any ordering of inputs, we adopt the same lexicographic as the layout-agnostic input format ( $\S 2.3$ ) to avoid uncertainty due to truncation and special markers.