SlowGuess/ACL_2020_OCR huggingface dataset:
Task Group	Task	Encoder
BERTBASE	BERTLARGE	GPT	GPT2
last layer	best layer	top-12 heads	last layer	best layer	top-16 heads	last layer	best layer	top-12 heads	last layer	best layer	top-12 heads
Surface	Length	58.0	87.8 (51.5)	95.0 (64.0)	54.8	94.4 (72.2)	95.2 (73.6)	52.2	96.4 (84.7)	96.2 (84.3)	57.8	88.9 (53.9)	92.8 (60.6)
WordContent	25.2	25.2 (0.0)	73.1 (190.3)	12.2	32.2 (165.2)	79.8 (556.4)	35.3	35.3 (0.0)	71.3 (102.0)	37.5	37.5 (0.0)	71.0 (89.5)
Syntactic	Depth	29.8	31.7 (6.5)	38.3 (28.7)	27.8	33.3 (19.6)	39.5 (41.9)	27.2	30.6 (12.2)	38.9 (42.8)	28.0	31.0 (10.6)	40.0 (42.7)
TopConstituents	69.8	74.7 (7.0)	84.2 (20.6)	62.8	78.2 (24.6)	85.6 (36.4)	53.0	65.1 (22.8)	79.5 (50.0)	57.3	63.1 (10.2)	82.8 (44.5)
BigramShift	78.1	78.1 (0.0)	88.3 (13.1)	77.2	81.1 (5.1)	90.9 (17.8)	69.3	69.3 (0.0)	80.7 (16.5)	68.8	70.5 (2.5)	78.8 (14.6)
Semantic	Tense	86.0	87.0 (1.2)	89.0 (3.6)	85.6	86.7 (1.2)	88.9 (3.8)	88.6	88.6 (0.0)	89.0 (0.5)	88.2	88.6 (0.5)	89.3 (1.2)
SubjNumber	82.0	84.7 (3.4)	88.2 (7.6)	80.0	87.9 (9.9)	90.5 (13.0)	78.8	78.8 (0.0)	84.2 (6.8)	83.5	83.5 (0.0)	87.7 (5.1)
ObjNumber	75.4	75.4 (0.0)	83.4 (10.6)	64.2	78.0 (21.5)	84.4 (31.4)	71.2	71.9 (0.9)	80.5 (13.0)	70.2	74.1 (5.5)	82.5 (17.5)
OddManOut	59.6	59.6 (0.0)	65.1 (9.3)	55.5	59.2 (6.6)	69.0 (24.2)	55.0	58.2 (5.9)	63.0 (14.6)	54.6	54.9 (0.6)	5.5 (1.6)
CoordInversion	65.5	65.9 (0.5)	74.6 (13.8)	64.9	70.9 (9.2)	78.5 (20.9)	57.7	60.7 (5.1)	70.2 (21.6)	58.3	60.0 (2.9)	67.6 (15.9)
Table 5: A summary of the probing tasks for three different embedding methods used in the five pre-trained architectures. The numbers in the parenthesis denote the percent increment of accuracy compared to those of the last layer.
Figure 6: Heatmaps of the attention head-wise evaluation on the ten sentence probing tasks with BERT<sub>BASE</sub>, BERT<sub>LARGE</sub>, GPT, GPT2, ELECTRAS<sub>SMALL</sub>, ELECTRABASE, and ELECTRALarge. 70 heat maps correspond to the ten tasks (Length, WordContent, Depth, TopConstituents, SubjNum, ObjNum, BigramShift, CoordInversion, OddManOut, and Tense from the left). In each heat map, x-axis and y-axis show the index values of the attention heads and the layer numbers (the lower, the closer to the initial input), respectively. The brighter the color, the higher the accuracy for the attention head and hence more important for the task. Note that the attention heads on the same layer are ordered by their classification accuracy values (i.e. an attention head with the highest accuracy on a layer is at the left-most location).
Figure 7: Unsorted heatmaps of the attention head-wise evaluation on the ten sentence probing tasks with BERT<sub>BASE</sub>, BERT<sub>LARGE</sub>, GPT, GPT2, ELECTRA<sub>SALL</sub>, ELECTRA<sub>BASE</sub>, and ELECTRA<sub>LARGE</sub>. 70 heat maps correspond to the ten tasks (Length, WordContent, Depth, TopConstituents, SubjNum, ObjNum, BigramShift, CoordInversion, OddManOut, and Tense from the left). In each heat map, x-axis and y-axis show the index values of the attention heads and the layer numbers (the lower, the closer to the initial input), respectively. The brighter the color, the higher the accuracy for the attention head and hence more important for the task. Note that the attention heads on the same layer are not ordered like Figure 6.