SlowGuess/NAACL_2022_OCR huggingface dataset:
Tab. 1 shows model performance on ToTTo test set. Among the baselines, methods based on pretrained Transformer models (i.e. BERT-to-BERT and T5) outperform the others and T5 models perform the best. Our method LATTICE can be plugged into such models. We compare our method with pure T5 models of different sizes, and LATTICE consistently performs better. Overall, LATTICE (T5-small) achieves improvements of 2.1 BLEU points and 0.8 PARENT points in comparison with T5-small, and LATTICE (T5-base) achieves improvements of 1.0 BLEU points and 1.7 PARENT points in comparison with T5-base. These results indicate the importance of structure information, which is almost totally abandoned by baselines. Further, the performance gain on tables both seen and unseen during training are significant. Specifically, on the overlap subset, LATTICE (T5-small) achieves improvements of 2.9 BLEU points and 1.3 PARENT points, and LATTICE (T5-base) achieves improvements of 0.9 BLEU points and 1.3 PARENT points, indicating better intrinsic performance. On the non-overlap subset, LATTICE (T5-small) achieves improvements of 1.3 BLEU points and 1.0 PARENT points, and LATTICE (T5-base) achieves improvements of 1.3 BLEU points and 2.2 PARENT points, indicating LATTICE is more generalizable to unseen tables. We also observe that the improvement on BLEURT is not as much as the other two metrics. It is reasonable as BLEURT is trained with machine translation annotations and synthetic data by mask filling, backtranslation and word drop. These training data ensures its robustness to sur
Although a previous study (Kale and Rastogi, 2020) has obtained better results using the much larger T5-3B, we were not able to run that model on our equipment even with a batch size of 1 due to the overly excessive GPU memory usage.
Model	Overall	Overlap	Non-Overlap
BLEU	PARENT	BLEURT	BLEU	PARENT	BLEURT	BLEU	PARENT	BLEURT
Pointer-Generator	41.6	51.6	0.076	50.6	58.0	0.244	32.2	45.2	-0.092
BERT-to-BERT	44.0	52.6	0.121	52.7	58.4	0.259	35.1	46.8	-0.017
T5-small	45.3	57.0	0.187	52.7	61.0	0.316	37.8	53.0	0.057
LATTICE (T5-small)	47.4	57.8	0.207	55.6	62.3	0.337	39.1	53.3	0.077
T5-base	47.4	56.4	0.221	55.5	61.1	0.344	39.1	51.7	0.098
LATTICE (T5-base)	48.4	58.1	0.222	56.1	62.4	0.345	40.4	53.9	0.099
Table 1: Results on the ToTTo test set. Best scores are in bold.
Model	BLEU	PARENT
Pointer-Generator	5.8	8.8
BERT-to-BERT	11.4	16.7
T5-small	14.2	22.0
LATTICE (T5-small)	15.7	23.8
T5-base	14.7	21.9
LATTICE (T5-base)	16.3	22.7
Table 2: Results on the HiTab test set.
face generation but not reasoning-based generation. Although the effectiveness of BLEURT is verified on an RDF-to-text dataset, tabular data holds different properties with RDF data<sup>7</sup>.
Results on HiTab in Tab. 2 further verify the effectiveness and generalizability of LATTICE. For different model sizes, LATTICE consistently performs better than T5 models. We also observe that on this dataset the model with highest BLEU score is not the model with highest PARENT score. It is partially because of the annotations. Many numbers appear in both tables and target sentences are of different precision. Copying such numbers from tables to generated sentences may increase PARENT score but reduce BLEU score.
3.3 Robustness Evaluation
To further evaluate model robustness against content-invariant perturbations on tables, we create a harder version of the ToTTo dev set, where each table is perturbed with a combination of row-wise shuffling, column-wise shuffling and table transpose. Especially, models can no longer benefit from memorizing the layout of tables appearing in both the training set and the dev set. We compare four methods based on T5, including the basic version proposed by Kale and Rastogi (2020), enhanced T5 with the layout-agnostic input or data augmentation ( $\S 2.3$ ), and T5 incorporated in LATTICE.
According to the results shown in Tab. 3,
vanilla T5 models face a severe performance drop when content-invariant perturbations are introduced. Overall, BLEU scores drop by 3.4 for T5-small, and 4.5 for T5-base. We also observe that the performance drop on overlap subset is larger than on non-overlap subset. This indicates that the performance gain of T5 models is somehow due to their memory of some tables existing in the training set, which is however brittle and not generalizable. Applying layout-agnostic input format, which linearizes tables by lexicographic order instead of cell index order, ensures models to return stable predictions, but results in worse overall performance due to the loss of structural information. Not surprisingly, layout-agnostic input causes performance drops by 1.5 BLEU points and 1.2 BLEU points to T5-small and T5-base on original dev set.
Another common way to improve model robustness is to increase the diversity of training instances with data augmentation. We augment the original training set by 8-fold using the three content-invariant transformation operations and their combinations. Training with augmented data reduces the gap between model performance on original tables and transformed tables. However, data augmentation is never exhaustive enough to guarantee true equivariance. Also, this introduces different variants of the same table into the training set, so there is a gap between the same table in training set and dev set. As a result, the performance on overlap subset is slightly worse than without data augmentation, but the performance on non-overlap subset is not negatively influenced. LATTICE guarantees consistent predictions towards content-invariant table transformations while achieving the best performance. In comparison with using layout-agnostic input format which also guarantees equivariance, LATTICE (T5-small) provides additional 3.3 BLEU points, and LATTICE (T5-base) provides additional 2.4 BLEU points on original dev set.
For example, in the ToTTo dataset,  $21\%$  samples require reasoning while  $13\%$  samples require comparison.
Model	Overall	Overlap	Non-Overlap
Origin	Transform	Δ	Origin	Transform	Δ	Origin	Transform	Δ
T5-small	45.7	42.3	-3.4	53.7	49.3	-4.4	37.7	35.4	-2.3
+ layout-agnostic input	44.2	44.2	0	51.6	51.6	0	37.0	37.0	0
+ data augmentation	45.3	44.4	-0.9	52.8	52.0	-0.8	37.9	37.0	-0.9
LATTICE (T5-small)	47.5	47.5	0	55.5	55.5	0	39.5	39.5	0
T5-base	47.4	42.9	-4.5	55.8	50.7	-5.1	39.2	35.4	-3.8
+ layout-agnostic input	46.2	46.2	0	54.3	54.3	0	38.3	38.3	0
+ data augmentation	47.2	46.9	-0.3	55.3	54.8	-0.5	39.2	38.9	-0.3
LATTICE (T5-base)	48.6	48.6	0	56.6	56.6	0	40.8	40.8	0
Table 3: Robustness evaluation on ToTTo dev set. Origin is the BLEU score on original tables, while Transform is the BLEU score on transformed tables. All transformed tables are transposed, row shuffled and column shuffled.  $\Delta$  is the difference between the two scores. Best scores in each group are in bold.
Att	Pos	Overall	Overlap	Non-Overlap
-	-	45.7	53.7	37.7
✓	-	47.0	54.4	39.6
✓	✓	47.5	55.5	39.5
Table 4: Ablation study on ToTTo dev set. Scores are BLEU. Att and Pos denote structure-aware self-attention and transformation-invariant positional encoding.
To help understand the effect of two key mechanisms in transformation-invariant graph masking, we hereby present ablation study results in Tab. 4.
Structure-Aware Self-Attention. We examine the effectiveness of structure-aware self-attention. In comparison with original (fully-connected) self-attention, incorporating structural information by pruning attention flows can improve the overall performance by 1.3 BLEU points. Detailed scores on two subsets show that both tables seen and unseen during training can benefit from structural information. The consistent improvements on two subsets indicate that structure-aware self-attention improves model ability of capturing cell relations rather than memorizing tables.
Transformation-Invariant Positional Encoding.
We further test the effectiveness of transformation-invariant positional encoding. We observe that although this technique is mainly designed for ensuring model robustness towards layout changes, it can bring an additional improvement of 0.5 BLEU points to overall performance. Interestingly, the improvement is mainly on the overlap subset. We attribute it to the fact that the same table in training and dev sets may have different highlighted cells, so that memorizing the layout information in the training set hinders in-domain generalization.
We review two relevant research topics. Since both topics have a large body of work, we provide a selected summary.
Table-to-text Generation. Table-to-text generation seeks to generate textual descriptions for tabular data. In comparison to text-to-text generation, the input of table-to-text generation is semistructured data. Early studies adapt the encoder-decoder framework to data-to-text generation with encoders aggregating cell information (Lebret et al., 2016; Wiseman et al., 2017; Bao et al., 2018). Followed by the success of massively pre-trained sequence-to-sequence Transformer models (Raffel et al., 2020; Lewis et al., 2020), recent SOTA systems apply these models to table-to-text generation (Kale and Rastogi, 2020; Su et al., 2021), where the input table is linearized to a textual sequence.
A table can include ample information and it is not always able to be summarized in one sentence. A line of work learns to generate selective descriptions by paying attention to key information in the table (Perez-Beltrachini and Lapata, 2018; Ma et al., 2019). However, multiple statements can be entailed from a table when different parts of the table are focused on. To bridge this gap, Parikh et al. (2020) proposes controlled table-to-text generation, allowing the generation process to react differently according to distinct highlighted cells. As highlighted cells can be at any positions and of arbitrary numbers, simple linearization, which breaks the layout structure, hinders relations among cells from being captured, therefore causing unreliable or hallucinated descriptions to be generated.
A few prior studies introduce structural information to improve model performance on table-to-text generation, either by incorporating token
position (Liu et al., 2018), or by aggregating row and column level information (Bao et al., 2018; Nema et al., 2018; Jain et al., 2018). However, none of existing methods can be directly applied to pretrained Transformer-based generative models, especially when we want to ensure model robustness to content-invariant table transformations. Our method enforces both structure-awareness and transformation-invariance to such models.
Equivariant Representation Learning. Equivariance is a type of prior knowledge existing broadly in real-world tasks. Earlier studies show that incorporating equivariance learning can improve visual perception model robustness against turbulence caused by geometric transformations, such as realizing translation, rotation, and scale equivariance of images (Lenc and Vedaldi, 2015; Worrall et al., 2017; Ravanbakhsh et al., 2017; Sosnovik et al., 2019; Yang et al., 2020). The input to those tasks presents unstructured information, and several geometrically invariable operations are incorporated in neural networks to realize the aforementioned equivariance properties. For example, Convolutional Neural Networks (CNNs) are equivariant to translations in nature (Lenc and Vedaldi, 2015). Harmonic Networks and Spherical CNNs extend the equivariance of CNNs to rotations (Worrall et al., 2017; Esteves et al., 2018). Group Equivariant Convolutional Networks are equivariant to more spatial transformations including translations, rotations and reflections (Cohen and Welling, 2016). Nonetheless, none of these geometrically invariable techniques can be directly applied to Transformer-based generative models to ensure equivariance on (a part of) structured tabular data, which is exactly the focus of this work. Our method realizes equivariant intermediate representations against content-invariant table transformations in table-to-text generation.
Some other works, while not explicitly using equivariant model structures, seek to realize equivariant representations by augmenting more diverse changes into training data (Chen et al., 2020a; Wu et al., 2020). Although the model can benefit from seeing more diverse inputs involving content-invariant transformations (Wu et al., 2020), this strategy has two drawbacks. Specifically, the augmented data, while introducing much computational overhead to training, are never exhaustive enough to guarantee true equivariance. By contrast, our method guarantees equivariance through the
neural network design and do not introduce any training overhead.
We propose LATTICE, a structure-aware equivariance learning framework for controlled table-to-text generation. Our experimental results verify the importance of structure-awareness and transformation-invariance, two key properties enforced in LATTICE, towards precise and robust description generation for tabular content. The proposed properties and equivariance learning framework aligns well with the nature of information organized in tables. Future research can consider extending the structure-aware equivariance learning framework to other data-to-text generation tasks (Koncel-Kedziorski et al., 2019; Nan et al., 2021), tabular reasoning or retrieval tasks (Gupta et al., 2020; Wang et al., 2021a,b; Eisenschlos et al., 2021), and pretraining representation on textual and tabular data (Yin et al., 2020; Herzig et al., 2020; Iida et al., 2021).
We appreciate the reviewers for their insightful comments and suggestions. This work is partly supported by the National Science Foundation of United States Grant IIS 2105329, and partly by the Air Force Research Laboratory under agreement number FA8750-20-2-10002.
This work seeks to develop a structure-aware equivariance learning framework for table-to-text generation. Since the proposed method focuses on improving prior generation systems by better utilization of structural information, it does not introduce bias towards specific content. The distinction between beneficial use and harmful use depends mainly on the data. Proper use of the technology requires that input corpora are legally and ethically obtained. We conduct experiments on two open benchmark in the way they intended to. Although we create a harder version of ToTTo dev set, the table transformation operations we use are content-invariant, whereas the ground-truth generation remains the same as it is in the original dataset, ensuring no further social bias is introduced.
Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, and Tiejun Zhao. 2018. Table-to-text: Describing table region with natural language. In Thirty-Second AAAI Conference on Artificial Intelligence.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. 2020a. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21(245):1-71.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020b. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929-7942.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1026-1036.
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022. Hitab: A hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
Taco Cohen and Max Welling. 2016. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884-4895.
Julian Eisenschlos, Maharshi Gor, Thomas Mueller, and William Cohen. 2021. Mate: Multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7606-7619.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. 2018. Learning so(3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-68.
Sebastian Gehrmann, Falcon Dai, Henry Elder, and Alexander M Rush. 2018. End-to-end content and plan selection for data-to-text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 46-56.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. Infotabs: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisenschlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320-4333.
Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. Tabbie: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446-3456.
Parag Jain, Anirban Laha, Karthik Sankaranarayanan, Preksha Nema, Mitesh M Khapra, and Shreyas Shetty. 2018. A mixed hierarchical attention based encoder-decoder approach for standard table summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 622-627.
Mihir Kale and Abhinav Rastogi. 2020. Text-to-text pretraining for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97-102.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2284-2293.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213.
Karel Lenc and Andrea Vedaldi. 2015. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Shuming Ma, Pengcheng Yang, Tianyu Liu, Peng Li, Jie Zhou, and Xu Sun. 2019. Key fact as pivot: A two-stage model for low resource table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2047-2057.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2021. Dart: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 432-447.
Preksha Nema, Shreyas Shetty, Parag Jain, Anirban Laha, Karthik Sankaranarayanan, and Mitesh M Khapra. 2018. Generating descriptions from structured data using a bifocal attention mechanism and gated orthogonalization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1539-1550.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of EMNLP.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037.
Laura Perez-Beltrachini and Mirella Lapata. 2018. Bootstrapping generators from noisy data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1516-1527.