SlowGuess/NAACL_2022_OCR huggingface dataset:
two different experimental settings, i.e.,  $a$ ) Using Entity Mentions Only, and  $b$ ) Using Extra metadata. Table 2 illustrates a comparison of different data artifacts used by both TURL and our approach, in either of the experimental settings.
The dataset statistics are available in Table 15 of the Appendix. Also, we will release the zs-WT-TURL-CT dataset once the anonymity period ends.
Table Artifact	Only Entity Mentions	Extra metadata
TURL	Ours	TURL	Ours
Page Title	✕	✕	✓	✕
Section Title	✕	✕	✓	✕
Table Caption	✕	✕	✓	✕
Column Header	✕	✕	✓	✓
Linked Entity Info	✕	✕	✓	✓
5.1 Results: Column Type Annotation
Table 3 shows the results on the Column Type annotation task for the WT-TURL-CT dataset. The first entry illustrates the performance of Sherlock (Hulsebos et al., 2019), a model that uses 1588 features describing statistical properties, character distributions, word embeddings and paragraph vectors of the cell mentions in a column. TURL (Deng et al., 2020) uses a structure-aware Transformer encoder model, and a pretraining/finetuning strategy to attain the current state of the art for this task.
Table 2: A comparison of Table artifacts used by TURL and our approach on the three table interpretation tasks. See Table 1 for an example.
Approach	Macro F1	Micro F1
Sherlock (Only Entity Mentions)	-	0.785
TURL (Only Entity Mentions)	0.628	0.889
Ours (Only Entity Mentions)	0.716	0.903
TURL (Extra metadata)	0.805	0.948
Ours (Extra metadata)	0.832	0.948
Table 3: Results for the Column Type Annotation (CT) task on the WT-TURL-CT dataset. Both TURL and our approach run over a pretrained TinyBERT model.
The third row and the fifth row in Table 3 illustrates the performance of our approach on this task, which is an original contribution of this work. Our approach substantially outperforms TURL on the Macro F1, under both the settings, i.e., "Only Entity Mentions" and "Extra metadata". In contrast, for Micro F1, our approach generates a statistically significant gain of  $1.4\%$  (p-value  $< 1e-3$  using McNemar's test) under the former setting, and
$^{3}$ https://github.com/ alan-turing-institute/SemAIDA  $^{4}$ https://github.com/sunlab-osu/TURL
achieves similar performance on the latter setting.
Next, we compare our approach to the TCN model as introduced by Wang et al. (2021). This model learns unique representation for tables by aggregating information from both a single table and across different tables. Moreover, it employs a supervised multi-task training objective for jointly learning column types and relations between column pairs. In comparison, our approach does not use a multi-task training objective, and is column-centric, i.e., encodes the query column only for learning the column types.
TCN is evaluated on the intersection of the WT-TURL-CT and the WT-TURL-RE datasets, i.e., columns present in both these datasets are only considered. The intersection yields a smaller dataset having 55,318 tables with 201 type labels and 121 relation labels. Table 4 illustrates the results.
Approach	F1-weighted
TaBERT (Yin et al., 2020)	0.895
TURL (Deng et al., 2020)	0.906
TCN (Wang et al., 2021)	0.933
BERT-Base+PI	0.944
Table 5 illustrates a comparison of Accuracy on T2D and Efthymiou datasets. Following Chen et al. (2019b), we train a single model using  $70\%$  of the T2D dataset and then score separately on both T2D and Efthymiou datasets. We observe that our approach achieves the new state-of-the-art on T2D, whereas on Efthymiou, it is almost similar in performance (lags behind by  $0.7\%$ ) to TURL. On the other hand, when evaluated on entity mentions only, our approach greatly improves over TURL.
Table 4: Comparison of weighted F1 values on an intersection of the WT-TURL-CT and the WT-TURL-RE dataset. The results from the first two rows are taken from Wang et al. (2021).
Approach	T2D	Efthymiou
HNN+P2Vec (Chen et al., 2019b)	0.966	0.650
TURL + table metadata	0.962	0.746
Ours (entity mentions + col headers)	0.985	0.739
TURL (entity mentions only)	0.940	0.516
Ours (entity mentions only)	0.962	0.584
5.2 Results: Zero shot Column Type Annotation
Table 6 shows the results on Zero shot Column Type Annotation task when the choice of both the Transformer encoder models is either  $a$  ) TinyBERT (Jiao et al., 2020), or  $b$  ) BERT-Base (Devlin et al., 2019). In either case, we observe that using our PI Strategy yields a substantially higher performance, compared to without it.
Table 5: Accuracy results on T2D and Efthymiou, following the setting in Chen et al. (2019b). Both the approaches run over a pretrained TinyBERT model.
TinyBERT	BERT-Base
With PI	Without PI	With PI	Without PI
MRR	0.604	0.506	0.674	0.535
Hits@1	0.402	0.310	0.487	0.346
5.3 Results: Relation Extraction
Table 7 shows the results on the Relation Extraction task for the WT-TURL-RE dataset. The first block of two rows compares the performance of TURL with our approach when only entity mentions from a given column pair are used. In comparison, the second block illustrates relative performance when extra metadata information is also used.
Table 6: Results on the Zero-shot Column Type Annotation (zs-CT) task. All approaches here use only the entity mentions and column headers.
Approach	Macro F1	Micro F1
TURL (Only Entity Mentions)	0.813	0.905
Ours (Only Entity Mentions)	0.845	0.914
TURL (Extra metadata)	0.914	0.949
Ours (Extra metadata)	0.906	0.941
Table 7: Results on the supervised Relation extraction (RE) task. Both the approaches run over a pretrained TinyBERT model.
We observe that when using entity mentions only, our approach improves over TURL by  $3.2\%$  on the Macro F1, whereas using additional table metadata results in an almost similar Macro F1 by both methods. The Micro F1 remains practically identical in either case, i.e.,  $\pm 1\%$  between both approaches.
Following Table 2, we notice that, unlike TURL, our approach does not use Page Title, Section Tile, nor Table Caption while operating under the "Extra metadata" setting, yet achieves comparable performances to TURL. How do we incorporate this additional contextual information within our approach is something that we leave as future work.
5.4 Results: Entity Linking
Table 8 shows the results on the Entity Linking task for the WT-TURL-EL and the T2D dataset, wherein the final row illustrates the performance of our approach.
Method	WT-TURL-EL	T2D
MRR	Hits@1	MRR	Hits@1
Wikidata (Lookup)	0.842	0.785	0.931	0.894
TURL	0.901	0.852	0.899	0.848
Ours	0.949	0.918	0.949	0.917
Note that the Wikidata lookup baseline achieves a higher score than TURL on the T2D dataset, which is in line with the findings reported by Deng et al. (2020). Nevertheless, our approach substantially outperforms over both Wikidata lookup as well as TURL, on both the datasets.
We observe that running this experiment without our Permutation Invariant (PI) strategy yields similar performance compared to using the PI strategy. We encode the cell values across a row, unlike the CT and RE tasks. Each row within the WT-TURL-EL dataset has roughly three textual cell values on average, as opposed to hundreds for the CT and RE tasks. We believe that due to this small number of values to encode, we cannot maximize the efficacy of the PI strategy for this task compared to CT and RE tasks. Thus we conclude that the PI Strategy does not provide gains over the default approach when the number of permutation invariant cells being encoded is small.
6 Analysis: Encoding the entire Table
In this section, we compare our column-centric approach against DODUO (Suhara et al., 2021) on the CT task. The DODUO model encodes the entire table and implements a multi-task learning objective for jointly learning column types and relations between column pairs. Table 9 illustrates the comparison results.
The DOSOLO model encodes the entire table, whereas the  $\mathrm{DOSOLO}^{\dagger}$  model encodes the query column only. In comparison, our approach is column-centric, i.e., it does not encode the entire table. Moreover, unlike DODUO, the last three rows in Table 9 are finetuned only on the CT Task.
Table 8: Results on the Entity Linking (EL) task. Since, we use different scoring metrics, we re-evaluate TURL based on the above benchmark datasets.
Approach	Encode full Table	Multi-task Learning	Micro F1
DODOU	✓	✓	0.925
DOSOLO	✓	×	0.914
DOSOLO†	×	×	0.825
Our Approach	×	×	0.920
Table 9: Comparison with existing approaches that encode the entire Table (on the CT Task). All the experiments use entity mentions only as illustrated in Table 2, and fine-tune over a BERT-Base model.
We observe that our approach substantially outperforms DOSOLO†, wherein both these approaches are evaluated under similar experimental settings. Moreover, our method performs slightly below DODOU, even though the latter encodes the entire table as additional context information and performs multi-task learning. These results demonstrate the effectiveness of our approach and open up the possibility of employing a multi-task learning strategy within our method, which we leave as future work.
7 Analysis: Ablation Tests
Table 10 illustrates the ablation experiments performed on the supervised CT and RE tasks respectively. The first row in the top block corresponds to using the PI Strategy for the Position IDs. Replacing PI Strategy with the default approach (Section 3) yields the second row. In the third row, we donot use any Position IDs akin to the SetTransformer model (Lee et al., 2019), whereas the fourth row uses a constant Position ID of zero throughout.
The results indicate that removing the Position ID parameter altogether performs the same as the default approach. We expected it to be the case, since there is no notion of an ordering between cell-values across a single column. Nevertheless, using our proposed permutation invariant strategy results in the best overall performance.
We also ran additional experiments wherein we created five random permutations of the test set corresponding to the supervised CT task. Within each newly created test set, the cell values within each column were randomly permuted and then scored against both trained default TinyBERT and TinyBERT+PI models. Compared to a constant macro f1 score of 0.832 for each of the five tries by the TinyBERT+PI model, the default TinyBERT model had a macro f1 score of 0.81 with a standard devi
Approach (using Extra metadata)	Macro F1
TinyBERT+PI strategy for Position IDs	0.832
TinyBERT (Default approach)	0.812
TinyBERT+No Position ID	0.811
TinyBERT+Constant Position IDs	0.816
TinyBERT (Default Approach)	0.807
TinyBERT+PI strategy for Position IDs	0.845
Table 10: Ablation tests on the CT task (top block) and RE task (bottom block) when evaluated on the WT-TURL-CT and WT-TURL-RE datasets respectively.
ation of 8e-4. These results further reinforce our hypothesis that the performance gains are indeed due to PI.
For the default approach to mirror the PI results, one needs to augment the training dataset with additional shuffled training instances. Such a strategy is not optimal as it increases the runtime complexity for training. In comparison, the PI approach learns a consistent model regardless of the ordering of the cells within a column, thereby demonstrating its efficacy.
The experimental settings, hyper-parameters for our experiments, error analysis, and additional ablation experiments are described in the Appendix.
This paper introduces a Permutation Invariant position encoding strategy for transformer-based language models that can encode a set of textual instances independent of its ordering. We argued that such a strategy generates meaningful representations for the rows/columns in a relational table. We show that our proposed method is flexible enough to apply to many downstream tasks. Unlike existing state-of-the-art approaches on table interpretation tasks, our method yields models that do not require additional pretraining yet achieve a new state-of-the-art on almost all the benchmarks.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Jiaoyan Chen, Ernesto Jiménez-Ruiz, Ian Horrocks, and Charles Sutton. 2019a. Colnet: Embedding the semantics of web tables for column type prediction. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 29-36. AAAI Press.
Jiaoyan Chen, Ernesto Jiménez-Ruiz, Ian Horrocks, and Charles Sutton. 2019b. Learning semantic annotations for tabular data. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 2088-2094. ijcai.org.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. TURL: table understanding through representation learning. Proc. VLDB Endow., 14(3):307-319.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Harrison Edwards and Amos J. Storkey. 2017. Towards a neural statistician. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Vasilis Efthymiou, Oktie Hassanzadeh, Mariano Rodriguez-Muro, and Vassilis Christophides. 2017. Matching web tables with knowledge base entities: From entity lookups to entity embeddings. In The Semantic Web - ISWC 2017 - 16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, Part I, volume 10587 of Lecture Notes in Computer Science, pages 260-277. Springer.
Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William W. Cohen. 2021. MATE: multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7606-7619. Association for Computational Linguistics.
Michael R. Glass, Mustafa Canim, Alfio Gliozzo, Saneem A. Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics in transformer based question answering over tables. In Proceedings of the 2021
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1212-1224. Association for Computational Linguistics.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4320-4333. Association for Computational Linguistics.
Madelon Hulsebos, Kevin Zeng Hu, Michiel A. Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Cagatay Demiralp, and Cesar A. Hidalgo. 2019. Sherlock: A deep learning approach to semantic data type detection. CoRR, abs/1905.10688.
Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3446-3456. Association for Computational Linguistics.
Maximilian Ilse, Jakub M. Tomczak, and Max Welling, 2018. Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2132-2141. PMLR.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4163-4174. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3744-3753. PMLR.
Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer. 2016. A large public corpus
of web tables containing time and context metadata. In Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11-15, 2016, Companion Volume, pages 75-76. ACM.
Varish Mulwad, Tim Finin, Zareen Syed, and Anupam Joshi. 2010. Using linked data to interpret tables. In Proceedings of the First International Workshop on Consuming Linked Data, Shanghai, China, November 8, 2010, volume 665 of CEUR Workshop Proceedings. CEUR-WS.org.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Dominique Ritze, Oliver Lehmberg, and Christian Bizer. 2015. Matching HTML tables to dbpedia. In Proceedings of the 5th International Conference on Web Intelligence, Mining and Semantics, WIMS 2015, Larnaca, Cyprus, July 13-15, 2015, pages 10:1-10:6. ACM.
Rita Sallam and Carlie Idoine. 2019. Augmented analytics is the future of analytics. Gartner.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Tim Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4967-4976.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2895-2905. Association for Computational Linguistics.
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Cagatay Demiralp, Chen Chen, and Wang-Chiew Tan. 2021. Annotating columns with pre-trained language models. arXiv preprint arXiv:2104.01785.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Daheng Wang, Prashant Shiralkar, Colin Lockard, Binxuan Huang, Xin Luna Dong, and Meng Jiang. 2021. TCN: table convolutional network for web table interpretation. In WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 4020-4032. ACM / IW3C2.
Bo Yang, Sen Wang, Andrew Markham, and Niki Trigoni. 2020. Robust attentional aggregation of deep feature sets for multi-view 3d reconstruction. Int. J. Comput. Vis., 128(1):53-73.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8413-8426. Association for Computational Linguistics.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3391-3401.
Shuo Zhang and Krisztian Balog. 2018. Ad hoc table retrieval using semantic similarity. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 1553-1562. ACM.
Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Trans. Intell. Syst. Technol., 11(2):13:1-13:35.
Ziqi Zhang. 2017. Effective and efficient semantic table interpretation using tableminer+. Semantic Web, 8(6):921-957.
This section enumerates the experimental details and the hyperparameters used in all of our experiments. For all the methods, we do a grid search over the hyper-parameter space using the validation set, and report the test set results corresponding to the best performing settings. For choosing a threshold value to compute the Macro/Micro F1 metrics, we employ a search space of [0.05, 1.0] with a step size of 0.05. For the learning rate, we performed a grid search over the following values {2e-5, 3e-5, 5e-5}.
For the WT-TURL dataset, the supervised CTA task uses a learning rate of 2e-5. In contrast, the supervised RE task uses a value of 5e-5. These tasks use a single CPU and a GPU, a batch size of 32, are evaluated for a max of 40 epochs, and have early stopping criteria of 5 epochs. The CTA task on T2D and Efthymiou datasets follow the same experimental configuration as the WT-TURL dataset, except that it runs for a max of 100 epochs and has early stopping criteria of 20 epochs.
Similarly, the zero-shot CTA (zs-CTA) and the Entity linking tasks also follow the same experimental configuration as the CTA task, except for the following. The transformer models for this task use a max tokenizer length of 128. The zs-CTA task on TinyBERT/BERT-Base uses eight/sixteen x86 CPUs for concurrent data loading, whereas the Entity linking task uses eight x86 CPUs for the same purpose. Moreover, the zs-CTA/Entity linking task on TinyBERT uses a single GPU for training (with a batch size of 32), whereas on BERT-Base, it uses four GPUs with a combined batch size of 24 for faster training.
We used PyTorch v1.8.1+cu102 for running our experiments, using Intel x86 CPU and Nvidia v100 GPU machines. All of our experiments used a random seed of 73.
A.2 Comparison of Model sizes
Table 11 illustrates a comparison of model sizes in terms of the number of trainable parameters of TURL against our proposed approach.
Approach	Additional Pretraining	Finetuning
CT	RE	EL
TURL	303 M	14.6 M	14.6 M	14.9 M
Ours	N/A	14.4 M	14.4 M	28.7 M
Table 11: Comparing the model sizes in terms of number of trainable parameters (M stands for millions).
The TURL method uses a pretrained TinyBERT model. It performs a pretraining step, to begin with, wherein it learns a total of 303 million parameters<sup>5</sup>. Out of these, 289 million are kept fixed during finetuning. In comparison, we do not perform additional pretraining for our approach. Moreover, our approach requires roughly 200K less trainable parameters for fine-tuning on the CT and RE tasks. For the EL task, we employ a siamese network architecture, therefore our approach uses roughly 2x trainable parameters compared to TURL. However, the overall number of model parameters, involving both pre-training and fine-tuning, required by our approach is one order of magnitude less.
A.3 Using Larger Transformer Encoder Models