SlowGuess/NAACL_2022_OCR huggingface dataset:
Our PI Strategy also yields a performance improvement when evaluated against BERT-Base and
${}^{5}$  Used PyTorch's numel method on trained models available at https://github.com/sunlab-osu/TURL
BERT-Large models (Table 12).
Approach	TinyBERT	BERT-Base	BERT-Large
Default	0.812	0.823	0.832
With PI	0.832	0.836	0.833
Params	14M	109M	335M
Time/EPOCH	2.5K	18K	69K
Table 12: Ablation tests on the supervised CT task showing Macro F1 when evaluated on the WT-TURL-CT dataset. Training using BERT-Large is done for three epochs only. All experiments use Extra metadata as illustrated in Table 2 and are done using one v100 GPU. Furthermore, the Time/Epoch row is rounded off to the nearest thousands and is measured in seconds.
As we move from TinyBERT to BERT-Base to BERT-Large, we observe that the number of trainable parameters increases by an overall factor of 24x (comparing TinyBERT to BERT-Large). This increase in the model size is why we believe that the performance improvement due to the PI strategy decreases from  $2\%$  (for TinyBERT) to  $0.1\%$  (for BERT-Large). Or, in other words, BERT-Large has too many parameters that the learner can finetune, which effectively offsets any gain in performance obtained due to the PI strategy. On the other hand, training BERT-Large is quite cumbersome, i.e., it takes at least one order of magnitude longer to train compared to TinyBERT and is computationally (and environmentally) expensive.
Since TinyBERT is smaller in size, faster in inference, and competitive in performance compared to BERT-Base, a higher performance improvement due to the usage of PI Strategy is highly significant in the context of building and deploying practical systems on real-world use cases.
In this section, we perform error analysis for the Column Type Annotation and the Relation Extraction tasks. The goal is to identify the type labels that are most difficult to predict using our approach. Note that, for this analysis, we use PI Strategy over a TinyBERT model. In addition, we also use column headers and canonical labels corresponding to the linked entity associated with the table cell mention. We selected the top five type labels sorted by the sum of the false positives and false negatives on the test set. Table 13 illustrates these results.
The fine grained types such as sports.pro ATHlete and government.politician in Table 13 have a
Type label	Test counts	False neg	False pos
sports.pro ATHLETE	1688	129	319
organization organization	2122	121	156
location.citytown	562	89	138
location.location	3245	80	50
government.politician	495	73	52
Table 13: False negative/positive type predictions on the WT-TURL-CT dataset.
classification error rate of  $26.5\%$  and  $25.3\%$  respectively, whereas coarse-grained types such as people.person have a  $2.5\%$  error rate. Furthermore, out of 138 false positives for the label location.citytown, 135 of them have location.location as a true label. On the other hand, all 89 test instances that get marked as False negatives (for location.citytown) have location.location as one of the ground truth labels, and we can identify 74 of them correctly. Table 14 illustrates the result of a similar analysis on the Relation Extraction task.
If we generalize these observations, we can conclude that our approach  $a)$  Makes lot more errors on fine-grained type labels as opposed to coarse-grained labels, and  $b)$  Confuses often between labels that are semantically related. This behavior is expected since it is difficult for any model to resolve these issues based only on tabular data, i.e., without additional table contextual information. As mentioned before, we leave the strategy of inculcating contextual information into our learning approach for future work.
Relation Label	False Neg	False Pos
award.award_nominated_work.award_nominations-award.award_nomination.award_nominee	37	29
award.award-winning_work.awards_won-award.award_honor.award_winor	23	19
film.film.written_by	16	10
Table 14: False negative/positive relation predictions on the WT-TURL-RE dataset. Out of 37 counts of false negatives for the first relation label, our approach predicts film.film.directed_by (23 counts), film.film.starring-film-performanceactor (7 counts), and film.film.written_by (7 counts). Similarly, out of 23 counts of false negatives for the second relation label, our approach predicts award.award_nominated_work.award_nominations-award.award_nomination.award_nominee (12 counts) and film.film.written_by (5 counts). For all of these relation labels, the corresponding test instances contained two columns, one containing names of people, and the other containing names of movies, songs, albums, etc. Since, our approach does not use table context information, resolving these label confusions seems difficult.
Task Name	Dataset	Train	Dev	Test	Target Labels
CT Task	WT-TURL-CT	628,254	13,391	13,025	255
T2D	250	-	133	37
Efthymiou	250	-	620	37
zs-CT Task	zs-WT-TURL-CT	409,125 (190)	106,756 (24)	29,259 (25)	-
RE Task	WT-TURL-RE	62,954	2,175	2,072	121
EL Task	WT-TURL-EL	1,264,217	76,720	225,777	-
T2D	-	-	21,270	-
Table 15: Dataset statistics for the Column Type Annotation (CT), Zero-Shot Column Type Annotation (zs-CT), Relation Extraction (RE), and Entity Linking (EL) Tasks. For the zs-CT task, the number within the brackets (for the train/dev/test folds) indicates the number of class labels available for that fold. For the EL task, we have a total of roughly 5 million candidate entities due to Wikidata lookup across all the instances.