SlowGuess/NAACL_2021_OCR huggingface dataset:
Due to the large number of labels, we resort to negative sampling during training instead of the full softmax to cut down on fine-tuning time. Negative samples are formed by uniform random sampling on the label space.
<sup>9</sup>Again, we ensure that none of the test tables in this dataset occur in TABBIE's pretraining data.
10https://github.com/megagonlabs/sato
Method	n=1000	n=10000	n=all
Sherlock	-	-	86.7
SATO	-	-	90.8
TaBERT	84.7	93.5	97.2
TABBIE (FREQ)	84.7	94.2	96.9
TABBIE (MIX)	84.1	93.8	96.7
ods: Sherlock (Hulsebos et al., 2019), which uses a multi-input neural network with hand-crafted features extracted from each column, and the aforementioned SATO (Zhang et al., 2019), which improves Sherlock by incorporating table context, topic model outputs, and label co-occurrence information. Table 4 shows the support-weighted F1-score for each method. Similar to the previous two tasks, TABBIE and TaBERT significantly outperform the prior state-of-the-art (SATO). Here, there are no clear differences between the two models, but both reach higher F1 scores than the other baselines even when given only 1,000 training examples, which demonstrates the power of table-based pretraining.
The results in the previous section show that TABBIE is a powerful table representation method, outperforming TaBERT in many downstream task configurations and remaining competitive in the rest. In this section, we dig deeper into TABBIE's representations by comparing them to TaBERT across a variety of quantitative and qualitative analysis tasks, including our own pretraining task of corrupt cell classification, as well as embedding clustering and nearest neighbors. Taken as a whole, the analysis suggests that TABBIE is able to better capture fine-grained table semantics.
4.1 Corrupt Cell Detection
We first examine how TaBERT performs on TABBIE's pretraining task of corrupt cell detection, which again is practically useful as a post-processing step after table structure decomposition (Tensmeyer et al., 2019; Raja et al., 2020) because mistakes in predicting row/column/cell boundaries (sometimes compounded by OCR errors) can lead to inaccurate extraction. We fine-tune TaBERT on 100K tables using the MIX corruption strategy for
Table 4: Support-weighted F1-score of different models on column type prediction. TaBERT and TABBIE perform similarly in low resource settings  $(n = 1000)$  and when the full training data is used  $(n = \mathrm{all})$
Corruption	Method	Prec.	Rec.	F1
Intra-row swap	TaBERT	85.5	83.0	84.2
TABBIE (FREQ)	99.0	81.4	89.4
TABBIE (MIX)	99.6	95.8	97.7
Intra-column swap	TaBERT	31.2	19.0	23.7
TABBIE (FREQ)	90.9	22.3	35.8
TABBIE (MIX)	91.5	55.0	68.8
Intra-table swap	TaBERT	81.2	69.5	74.9
TABBIE (FREQ)	98.2	73.3	84.0
TABBIE (MIX)	98.4	86.2	91.9
Random FREQ cell	TaBERT	86.7	87.0	86.8
TABBIE (FREQ)	99.3	98.2	98.8
TABBIE (MIX)	99.1	98.1	98.6
All	TaBERT	75.6	65.2	70.0
TABBIE (FREQ)	98.2	69.5	81.4
TABBIE (MIX)	97.8	84.1	90.5
Table 5: A fine-grained comparison of different models on corrupt cell detection, with different types of corruption. TaBERT struggles on this task, especially in the challenging setting of intra-column swaps. Unlike our downstream tasks, the MIX strategy is far superior to FREQ here.
ten epochs, and construct a test set of 10K tables that are unseen by both TaBERT and TABBIE during pretraining. While TABBIE of course sees an order of magnitude more tables for this task during pretraining, this is still a useful experiment to determine if TaBERT's pretraining objective enables it to easily detect corrupted cells.
As shown in Table 5, TaBERT performs significantly worse than TABBIE on all types of corrupt cells (both random corruption and intra-table swaps). Additionally, intra-column swaps are the most difficult for both models, as TABBIE achieves a 68.8 F1 on this subset compared to just 23.7 F1 by TaBERT. Interestingly, while the MIX strategy consistently performs worse than FREQ for the TABBIE models evaluated on the three downstream tasks in the previous section, it is substantially better at detecting more challenging corruptions, and is almost equivalent to detecting random cells sampled by FREQ. This result indicates that perhaps more complex table-based tasks are required to take advantage of representations derived using MIX corruption.
4.2 Nearest neighbors
We now turn to a qualitative analysis of the representations learned by TABBIE. In Figure 6 (top), we display the two nearest neighbor columns from our validation set to the date column marked by the red box. TABBIE is able to model the similarity of feb.
(a) Input table
#	name	year
15	allysia	junior
18	maria	senior
17	emily	sophomore
16	hydn	sophomore
19	hayley	sophomore
20	michelle	graduate
(b) TABBIE (MIX)
#	name	year
0.0%	0.1%	0.0%
100%	0.0%	0.0%
0.0%	0.0%	0.0%
99.9%	0.0%	0.0%
0.0%	0.0%	0.0%
0.0%	0.0%	0.0%
(c) TaBERT
#	name	year
2.6%	1.6%	8.9%
3.2%	2.6%	1.9%
4.3%	7.6%	5.2%
2.2%	0.3%	0.5%
3.3%	3.3%	1.5%
4.0%	6.6%	2.9%
Figure 5: In this figure, (b) and (c) contain the predicted corruption probability of each cell in (a). Only TABBIE MIX is able to reliably identify violations of numerical trends in columns.
16 and saturday. february 5th despite the formatting difference, while TaBERT's neighbors more closely resemble the original column. Figure 6 (bottom) shows that TABBIE's nearest neighbors are less reliant on matching headers than TaBERT, as the neighbors all have different headers (nom, nombre, name).
4.3 Clustering
Are the embeddings produced by TABBIE useful for clustering and data discovery? To find out, we perform clustering experiments on the FinTabNet dataset from Zheng et al. (2021). This dataset contains  $\sim 110\mathrm{K}$  tables from financial reports of corporations in the S&P-500. We use the [CLS] embedding at the  $(0,0)$  position (i.e., the top left-most cell in the table), extracted from a TABBIE model trained with the FREQ strategy, as a representative embedding for each table in the dataset. Next, we perform  $k$ -means clustering on these embeddings using the FAISS library (Johnson et al., 2017), with  $k = 1024$  centroids. While the FinTabNet dataset is restricted to the homogenous domain of financial tables, these tables cluster into sub-types such as consolidated financial tables, jurisdiction tables, insurance tables, etc. We then examine the contents of these clusters (Figure 7) and observe that TABBIE embeddings can not only be clustered into these sub-types, but also that tables from reports of the same company, but from different financial years, are placed into the same cluster.
4.4 Identifying numeric trends
Next, we analyze how well TABBIE understands trends in numerical columns by looking at specific examples of our corrupt cell detection task. The first column of the table in Figure 5 contains jersey numbers sorted in ascending order. We swap two cells in this column, 16 and 18, which violates
date	opponent
feb. 16	northern colorado
feb. 17	colorado mesa (ncaa div. ii)
feb. 22	utah state*
feb. 23	westminster
mar. 2	vs. uc-santa barbara
mar. 3	@ unlv
mar. 7	loyola marymount
mar. 9	simon fraser
date	date
saturday, february 5th	11.20
wednesday, february 9th	11.20
11.21
saturday, february 12th	11.21
TaBERT's top-2 columns:
date	date
feb. 18	sept. 7
feb. 26	sept. 14
sept. 21
nom	artiste
run cold	holly golightly
indeed you do	holly golightly
i let my daddy do that	holly golightly
for all this	holly golightly
painted on	holly golightly
a length of pipe	holly golightly
TABBIE's top-3 columns:
nom	nome	name
nothing	brividi d'am	amor express
once	primo appur	cahuates, pistaches
hero in ft	una volta ne	la suaeta
bad bloc	un cuore ma	a la luz de las estrellas
TaBERT's top-3 columns:
nom	nom	nom
nothing ch	the crac	trois piä'ces bra'ves: alle
once	the crac	trois piä'ces bra'ves: and
hero in flâ	the crac	trois piä'ces bra'ves: ass
bad blood	the crac	sechs bagatellen: allegro
Figure 6: Nearest neighbors of the date and nom columns from the tables on the left, from both TABBIE and TaBERT. TABBIE's nearest neighbors exhibit more diverse formatting and less reliance on the header, which is an example of its semantic representation capability.
the increasing trend. Both TaBERT (fine-tuned for corrupt cell detection) and TABBIE FREQ struggle to identify this swap, while TABBIE MIX is almost certain that the two cells have been corrupted. This qualitative result is further evidence that the MIX model has potential for more complex table-based reasoning tasks.
The staggering amount of structured relational data in the form of tables on the Internet has attracted considerable attention from researchers over the past two decades (Cafarella et al., 2008; Limaye et al., 2010; Venetis et al., 2011; Suchanek et al., 2007; Embley et al., 2006), with applications including retrieval (Das Sarma et al., 2012), schemamatching (Madhavan et al., 2001, 2005), and entity linking (Zhang et al., 2020).
Similar to popular large-scale language models pretrained on tasks involving unstructured natural language(Peters et al., 2018; Devlin et al., 2018; Liu et al., 2019), our work is part of a recent trend of self-supervised models trained on structured tabular data. TaBERT (Yin et al., 2020) and TaPaS (Herzig et al., 2020) jointly model tables
Table of contents	Tax Year Ended (Millionen)	Tax Years Ended (Millionen)
2015	2016	2017	2018	2015	2016	2017	2018
Fixed maturities	$ 433,097	$ 462,757	$ 473,483	Total Investments	$ 482,757	$ 473,483	$ 490,951	$ 433,097
Equity securities	45,817	41,183	45,387	Equity securities	47,193	45,387	59,165	42,707
Short-term investments and cash	1,577	1,635	1,295	Short-term investments and cash	1,635	1,295	1,592	1,769
Other invested assets	-	-	-	Other invested assets	-	-	-	-
Limited partnerships	-	-	-	Limited partnerships	-	-	-	-
Other	1,804	3,619	7,329	Other	3,619	7,329	7,329	1,804
Gross investment income before adjustments	496,526	556,072	574,425	Gross investment income before adjustments	556,072	574,425	628,968	496,526
Funds held interest income (expense)	10,767	8,471	10,663	Funds held interest income (expense)	8,471	10,663	10,663	1,804
Future policy benefit income (expense)	(1,807)	(1,886)	(1,886)	Future policy benefit income (expense)	(1,886)	(1,770)	(1,902)	(1,807)
Gross investment income	505,388	563,857	582,288	Gross investment income	563,857	562,288	628,968	505,388
Investment expenses	(31,561)	(32,387)	(33,798)	Investment expenses	(33,587)	(33,798)	(34,452)	Investment expenses
Net investment income	$ 473,825	$ 530,570	$ 548,509	Net investment income	$ 530,570	$ 548,509	$ 600,202	Net investment income
Figure 7: Sample tables from clusters obtained by running  $k$ -means on TABBIE's [CLS] embeddings on the FinTab-Net dataset. TABBIE not only clusters embeddings into reasonable semantic types, such as Table of Contents (first row), but it also places tables of the same type from the same company into the same cluster (second and third rows). We provide the source images of the corresponding tables in this figure.
with text (typically captions or questions), and are thus more suited for tasks like question answering (Pasupat and Liang, 2015). For pretraining, TaBERT attempts to recover the name and datatype of masked column headers (masked column prediction), in addition to contents of a particular cell (cell value recovery). The pretraining objectives of TaPaS, on the other hand, encourage tabular textual entailment. In a concurrent work, the TUTA model (Wang et al., 2020) uses masked language modeling, cell-level cloze prediction, and table-context retrieval as pretraining objectives. Further, in addition to traditional position embeddings, this work accounts for the hierarchical nature of tabular data using tree-based positional embeddings. Similarly, in Deng et al. (2020), the authors perform a variant of MLM called masked entity recovery. In contrast, TABBIE is pretrained strictly on tabular data and intended for more general-purpose table-based tasks, and uses corrupt-cell classification as its pretraining task.
In this paper, we proposed TABBIE, a self-supervised pretraining method for tables without associated text. To reduce the computational cost of training our model, we repurpose the ELECTRA objective for corrupt cell detection, and we use two
separate Transformers for rows and columns to minimize complexity associated with sequence length. On three downstream table-based tasks, TABBIE achieves competitive or better performance to existing methods such as TaBERT, and an analysis reveals that its representations include a deep semantic understanding of cells, rows, and columns. We publicly release our TABBIE pretrained models and code to facilitate future research on tabular representation learning.
As with any research work that involves training large language models, we acknowledge that our work has a negative carbon impact on the environment. A cumulative of 1344 GPU-hours of computation was performed on Tesla V100 GPUs. Total emissions are estimated to be  $149.19\mathrm{kg}$  of  $\mathrm{CO}_{2}$  per run of our model (in total, there were two runs). While this is a significant amount (equivalent to  $\approx 17$  gallons of fuel consumed by an average motor vehicle), it is lower than TaBERT's cost per run by more than a factor of 10 assuming a similar computing platform was used. Estimations were conducted using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
Semantic type
Centroid No.
<sup>11</sup>https://www.epa.gov/greenvehicles/
We thank the anonymous reviewers for their useful comments. We thank Christopher Tensmeyer for helpful comments and pointing us to relevant datasets for some of our experiments. We also thank the UMass NLP group for feedback during the paper writing process. This work was made possible by research awards from Sony Corp. and Adobe Inc. MI is also partially supported by award IIS-1955567 from the National Science Foundation (NSF).
Michael J. Cafarella, Alon Halevy, Daisy Zhe Wang, Eugene Wu, and Yang Zhang. 2008. Webtables: Exploring the power of tables on the web. Proc. VLDB Endow., 1(1):538-549.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pretraining text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Anish Das Sarma, Lujun Fang, Nitin Gupta, Alon Halevy, Hongrae Lee, Fei Wu, Reynold Xin, and Cong Yu. 2012. Finding related tables. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, page 817-828, New York, NY, USA. Association for Computing Machinery.
Li Deng, Shuo Zhang, and Krisztian Balog. 2019. Table2vec: Neural word and entity embeddings for table population and retrieval. In Proceedings of SIGIR 2019.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. Turl: Table understanding through representation learning. Proc. VLDB Endow., 14(3):307-319.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
D. Embley, Matthew Hurst, D. Lopresti, and G. Nagy. 2006. Table-processing paradigms: a research survey. International Journal of Document Analysis and Recognition (IJDAR), 8:66-86.
Jonathan Herzig, P. Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. Tapas: Weakly supervised table parsing via pretraining. In ACL.
Kevin Zeng Hu, Snehalkumar (Neil) S. Gaikwad, Madelon Hulsebos, Michiel A. Bakker, Emanuel Zgraggen, Cesar A. Hidalgo, Tim Kraska, Guoliang
Li, Arvind Satyanarayan, and Cagatay Demiralp. 2019. Viznet: Towards A large-scale visualization learning and benchmarking repository. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, 2019.
M. Hulsebos, K. Hu, M. Bakker, Emanuel Zgraggen, Arvind Satyanarayan, T. Kraska, cCaugatay Demiralp, and C'esar A. Hidalgo. 2019. Sherlock: A deep learning approach to semantic data type detection. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
Girija Limaye, Sunita Sarawagi, and Soumen Chakrabarti. 2010. Annotating and searching web tables using entities, types and relationships. Proc. VLDB Endow., 3(1):1338-1347.
Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Jayant Madhavan, Philip A. Bernstein, AnHai Doan, and Alon Halevy. 2005. Corpus-based schema matching. In Proceedings of the 21st International Conference on Data Engineering, ICDE '05, page 57-68, USA. IEEE Computer Society.
Jayant Madhavan, Philip A. Bernstein, and Erhard Rahm. 2001. Generic schema matching with cupid. In Proceedings of the 27th International Conference on Very Large Data Bases, VLDB '01, page 49-58, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
D. Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Y. Li, Ashwin Bharambe, and L. V. D. Maaten. 2018. Exploring the limits of weakly supervised pretraining. In ECCV.
Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Higashinaka, and Yoshihiro Matsuo. 2017. Understanding the semantic structures of tables with a hybrid deep neural network architecture. In Thirty-First AAAI Conference on Artificial Intelligence.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Erhard Rahm and Philip A. Bernstein. 2001. A survey of approaches to automatic schema matching. VLDB J., 10(4):334-350.
Sachin Raja, Ajoy Mondal, and C. V. Jawahar. 2020. Table structure recognition using top-down and bottom-up cues. In Computer Vision - ECCV 2020, pages 70-86, Cham. Springer International Publishing.
Fabian M. Suchanek, Gjergji Kasmeci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web.
C. Tensmeyer, V. I. Morariu, B. Price, S. Cohen, and T. Martinez. 2019. Deep splitting and merging for table structure decomposition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 114-121.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ¥L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Petros Venetis, Alon Halevy, Jayant Madhavan, Marius Paşa, Warren Shen, Fei Wu, Gengxin Miao, and Chung Wu. 2011. Recovering semantics of tables on the web. Proc. VLDB Endow., 4(9):528-538.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Empirical Methods in Natural Language Processing.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2020. Structure-aware pre-training for table understanding with tree-based transformers. ArXiv, abs/2010.12537.
Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Annual Conference of the Association for Computational Linguistics (ACL).
Dan Zhang, Yoshihiko Suhara, Jinfeng Li, Madelon Hulsebos, Cagatay Demiralp, and Wang-Chiew Tan. 2019. Sato: Contextual semantic type detection in tables.
Shuo Zhang and Krisztian Balog. 2017. Entitables: Smart assistance for entity-focused tables. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Trans. Intell. Syst. Technol.
Shuo Zhang, Edgar Meij, Krisztian Balog, and Ridho Reinanda. 2020. Novel entity discovery from web tables. In Proceedings of The Web Conference 2020, WWW '20, page 1298-1308, New York, NY, USA. Association for Computing Machinery.
Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. 2021. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 697-706.