SlowGuess/NAACL_2022_OCR huggingface dataset:
Domain	% Test entities (e) seen during training
Train	Test	Varying levels of training set
0.01%	0.1%	1%	5%	25%	50%	100%
Wiki BLINK	NED	4.21	13.24	37.51	64.33	84.30	89.85	92.78
CoNLL	13.42	35.70	63.14	76.41	85.15	88.21	91.55
Wiki FGET	NED	1.04	7.60	26.26	51.88	75.24	82.48	87.09
CoNLL	8.74	31.62	53.94	69.45	82.23	86.20	89.16
Table 6: Percentage of test examples whose gold entities (e) were seen during training time.
Domain	% Test mentions (m) seen during training
Train	Test	Varying levels of training set
0.01%	0.1%	1%	5%	25%	50%	100%
Wiki BLINK	NED	5.27	15.52	44.85	72.74	90.81	94.35	96.40
CoNLL	8.23	29.92	51.48	62.34	70.97	74.49	77.35
Wiki FGET	NED	0.06	9.47	34.11	62.84	85.48	91.09	94.26
CoNLL	6.62	26.71	46.93	58.55	68.52	72.00	75.03
Table 7: Percentage of test examples whose mentions  $(m)$  were seen during training time.
Domain	% Test (entity, mention) pairs seen during training
Train	Test	Varying levels of training set
0.01%	0.1%	1%	5%	25%	50%	100%
Wiki BLINK	NED	3.24	10.40	26.75	48.92	68.93	77.06	82.47
CoNLL	6.44	23.79	39.42	51.17	61.38	65.71	69.41
Wiki FGET	NED	0.00	4.33	17.07	36.69	60.32	68.49	75.18
CoNLL	4.93	19.62	35.01	46.27	57.44	61.56	65.28
Table 8: Percentage of test examples whose gold entity and mention pairs  $(e,m)$  were seen during training time.
Parameter	Symbol	Tuning Range
Parameter for type-based inference strategy	βzel	{0,1}
Parameter for type-based inference strategy	βtype	{0,1}
Parameter for type-based inference strategy	βboth	{0,1}
Parameter for type-based inference strategy	top-k	{2,3,5,10,100}
Pruning level of type hierarchy during inference	Tree Level	{1,2,3,4,5,6,7}
Table 9: List of hyperparameters and tuning range. For optimal value of the hyperparameters, please refer Tables 16, 17, and 18.
Domain	ZEL Accuracy
Train	Test	Varying levels of training set
0.01%	0.1%	1%	5%	100%
Wiki BLINK	NED	56.62	65.38	67.66	68.87	71.72
CoNLL	61.18	71.61	72.27	73.50	71.50
Wiki FGET	NED	47.62	62.00	67.34	68.59	67.74
CoNLL	52.77	68.40	73.08	72.99	74.36
Table 10: BLINK's bi-encoder accuracy with varying level of training set size.
Mention (m)	in Marin County, California, Lucasfilm Ltd. is one of the most ... The Seattle Mariners are an American professional baseball team based in Seattle, Washington. Enfranchised in 1977 , the Mariners are a member of the Western Division of Major League Baseball ’s American League . Safeco Field has
Gold entity (e*)	Washington (state)
NeSLET’s top-3 predictions with scores Szel(e | mtest)	[Seattle (0.54), Washington, D.C. (0.23), Washington (state) (0.22)]
Type set for Seattle	[City, Settlement, Place, PopulatedPlace]
Type set for Washington (state)	[Settlement, AdministrativeRegion, PopulatedPlace, Place, Region]
Our predictions p(tj | mtest)	[City (0.99), Settlement (0.99), Place (0.99), PopulatedPlace (0.99), AdministrativeRegion (0.99), Region (0.99)]
Type Score Stype(e | mtest)	[Seattle (0.21), Washington, D.C. (0.21), Washington (state) (0.57)]
Type-based inference top-3 predic- tions with scores Stotal(e | mtest)	[Washington (state) (0.79), Seattle (0.75), Washington, D.C. (0.44)]
Table 11: An anecdotal example where the NeSLET model predicts the gold entity at Rank-2 position but type-based inference promotes it to Rank-1.
Mention (m)	Pink Floyd song ‘Keep Talking’. [PERSON1]’s early life and the onset of his illness was the subject of the 2004 BBC4 TV film [MOVIE1] in which he was portrayed by [PERSON2]. In 2008, [PERSON1] was the subject of and featured in the documentary series [DOCUMENTARY1]
Gold entity (e*)	[MOVIE1]
NeSLET's top-3 predictions with scores Szel(e | mtest)	[ [PERSON1] (0.51), [MOVIE1] (0.48), [MOVIE2] (0.0042)]
Type set for [PERSON1]	[Scientist, Person, Agent]
Type set for [MOVIE1] (2004 film)	[TelevisionShow, Film, Work]
Our predictions p(tj | mtest)	[Scientist (6.6 × 10-8), Person (0.061), Agent (0.061), TelevisionShow (0.87), Film (0.99), Work (0.99)]
Type Score ŷtype(e | mtest)	[ [PERSON1] (0.097), [MOVIE1] (0.63), [MOVIE2] (0.26)]
Type-based inference top-3 predictions with scores Stotal(e | mtest)	[ [MOVIE1] (1.11), [PERSON1] (0.60), [MOVIE2] (0.26)]
Table 12: One more anecdotal example where the NeSLET model predicts the gold entity at Rank-2 position but type-based inference promotes it to Rank-1. Personally identifiable information has been masked for privacy reasons.
Mention (m)	way she does. All this makes me feel this character is made up of a strongly ironic stance on [PERSON1]'s part. More ironic than in Cat's Eye. Yet a pattern to identify with is emerging also. Joan is growing up. She is beginning to break away from Arthur to question him.
Gold entity (e*)	Cat's Eye (novel)
NeSLET's top-3 predictions with scores Szel(e | mtest)	[Cat's Eye (novel) (0.52), Eye of the Cat (0.30), Eye of Cat (0.16)]
Type set for Cat's Eye (novel)	[Book, Work, WrittenWork]
Type set for Eye of the Cat	[Film, Work]
Our predictions p(tj | mtest)	[Book (0.07), Work (0.89), WrittenWork (0.07), Film (0.89)]
Type Score ŷtype(e | mtest)	[Cat's Eye (novel) (0.23), Eye of the Cat (0.53), Eye of Cat (0.23)]
Type-based inference top-3 predic- tions with scores Stotal(e | mtest)	[Eye of the Cat (0.83), Cat's Eye (novel) (0.75), Eye of Cat (0.39)]
Table 13: An anecdotal example where the NeSLET model predicts the gold entity at Rank-1 position but type-based inference pushes it down to Rank-2. Personally identifiable information has been masked for privacy reasons.
Mention (m)	for National Harbor to take a stroll and do some window shopping. The kids had fun on ‘the man in the sand’ better known as The Awakening. We also walked along the trail that connects to the Woodrow Wilson Bridge and leads into Alexandria - lots of bikers and walkers
Gold entity (e*)	The Awakening (sculpture)
NeSLETs top-3 predictions with scores Szel(e | mtest)	[The Awakening (sculpture) (0.35), Dalai Lama Awakening (0.33), The Great Awakening (0.31)]
Type set for The Awakening (sculpture)	[Artwork, Work ]
Type set for Dalai Lama Awakening	[Film, Work]
Our predictions p(tj | mtest)	[Artwork (5.2 × 10-7), Work (0.99), Film (0.99), Work (0.99)]
Type Score Štype(e | mtest)	[The Awakening (sculpture) (0.19), Dalai Lama Awakening (0.53), The Great Awakening (0.26)]
Type-based inference top-3 predictions with scores Stotal(e | mtest)	[Dalai Lama Awakening (0.86), The Great Awakening (0.57), The Awakening (sculpture) (0.54)]
Table 14: One more anecdotal example where the NeSLET model predicts the gold entity at Rank-1 position but type-based inference pushes it down to Rank-2.
Training Data Split	Number of Update Steps
0.01%	1k
0.1%	4k
1%	10k
Table 15: Number of update steps used for fine-tuning GENRE for different data splits.
Domain	ETP Loss	Tree Level top-k	β	BLINK Acc.	NeSLET Acc. when inferred	ETP F1
Val	Test	w/o types	w/ types
Train	Test	Val	Test	Val	Test	Val	Test
Wiki BLINK	NED	Gödel	1	10 (1,1,0)	56.55	56.62	57.49	56.99	57.63	57.07	22.50	22.03
Łukasiewicz	3	2 (1,1,0)	56.55	56.62	58.41	56.98	58.46	56.95	25.15	24.98
Flat	3	2 (1,1,0)	56.55	56.62	56.25	57.69	56.66	57.72	54.35	52.66
CoNLL	Gödel	1	3 (1,1,0)	61.73	61.18	62.90	62.57	63.26	63.07	14.19	13.27
Łukasiewicz	3	3 (1,1,0)	61.73	61.18	63.41	64.28	63.72	64.46	27.83	25.32
Flat	2	2 (1,1,0)	61.73	61.18	64.67	63.47	65.11	63.95	62.74	62.16
Wiki FGET	NED	Gödel	1	5 (1,1,0)	45.44	47.62	52.18	52.82	52.52	52.79	22.42	21.74
Łukasiewicz	1	10 (1,1,0)	45.44	47.62	52.33	52.11	52.54	52.17	31.29	31.31
Flat	3	3 (1,0,1)	45.44	47.62	55.71	55.33	55.90	55.30	47.36	47.23
CoNLL	Gödel	1	5 (1,1,0)	55.79	52.77	62.00	60.10	62.23	60.14	12.79	11.61
Łukasiewicz	1	3 (1,1,0)	55.79	52.77	62.06	60.08	62.27	60.03	12.79	11.63
Flat	3	3 (1,1,0)	55.79	52.77	61.46	58.39	62.06	58.73	64.87	61.64
Table 16: Performance of proposed NeSLET vis-à-vis BLINK at  $0.01\%$  training dataset.
Domain	ETP Loss	Tree Level top-k	β	BLINK Acc.	NeSLET Acc. when inferred	ETP F1
Val	Test	w/o types	w/ types
Train	Test	Val	Test	Val	Test	Val	Test
Wiki BLINK	NED	Gödel	1	3	(1,1,0)	66.08	65.38	64.94	64.57	65.22	64.72	57.64	57.50
Łukasiewicz	1	2	(1,1,0)	66.08	65.38	66.31	65.58	66.57	65.58	46.73	46.47
Flat	1	3	(1,1,0)	66.08	65.38	65.70	65.35	66.00	65.25	65.31	64.19
CoNLL	Gödel	3	2	(1,1,0)	75.74	71.61	73.92	70.47	74.26	70.74	67.45	62.06
Łukasiewicz	3	2	(1,1,0)	75.74	71.61	76.00	73.01	76.44	73.26	54.47	52.97
Flat	3	3	(1,1,0)	75.74	71.61	74.28	70.58	74.76	71.14	77.28	74.94
Wiki FGET	NED	Gödel	1	3	(1,1,0)	61.14	62.00	62.88	63.72	63.11	63.88	55.92	54.33
Łukasiewicz	1	100	(1,1,0)	61.14	62.00	63.18	63.61	63.20	63.59	26.56	25.92
Flat	3	3	(1,1,0)	61.14	62.00	62.78	62.98	63.30	62.81	61.55	60.56
CoNLL	Gödel	1	2	(1,1,0)	71.93	68.40	73.23	70.49	73.60	70.60	70.05	65.98
Łukasiewicz	2	2	(1,1,0)	71.93	68.40	73.21	69.84	73.81	70.06	62.08	56.61
Flat	1	2	(1,1,0)	71.93	68.40	73.44	69.59	73.81	69.64	75.72	70.77
Table 17: Performance of proposed NeSLET approach vis-à-vis BLINK at  $0.1\%$  training dataset.
Domain	ETP Loss	Tree Level top-k	β	BLINK Acc.	NeSLET Acc. when inferred	ETP F1
Val	Test	w/o types	w/ types
Train	Test	Val	Test	Val	Test	Val	Test
Wiki BLINK	NED	Gödel	3	2	(1,1,0)	69.37	67.66	70.91	70.02	71.32	69.69	64.53	62.76
Łukasiewicz	5	3	(1,1,0)	69.37	67.66	70.68	69.65	70.93	69.64	46.81	47.28
Flat	1	2	(1,1,0)	69.37	67.66	70.85	69.91	71.10	70.09	69.38	68.60
CoNLL	Gödel	4	2	(1,1,0)	76.12	72.27	78.70	74.74	79.14	75.06	73.80	71.02
Łukasiewicz	1	3	(1,1,0)	76.12	72.27	78.51	75.24	78.72	75.37	56.47	53.81
Flat	3	2	(1,1,0)	76.12	72.27	77.89	74.38	78.43	74.92	81.31	78.73
Wiki FGET	NED	Gödel	1	5	(1,1,0)	67.89	67.34	69.40	68.08	69.54	68.16	62.34	59.68
Łukasiewicz	1	2	(1,1,0)	67.89	67.34	68.73	67.58	68.86	67.67	41.94	41.04
Flat	1	2	(1,1,0)	67.89	67.34	70.50	69.21	70.58	69.05	66.75	66.48
CoNLL	Gödel	3	3	(1,1,0)	76.88	73.08	78.18	75.03	78.47	74.90	73.45	69.78
Łukasiewicz	2	5	(1,1,0)	76.88	73.08	78.41	75.06	78.72	75.39	56.27	54.22
Flat	3	2	(1,0,1)	76.88	73.08	78.37	73.77	78.66	74.13	79.01	75.52
Table 18: Performance of proposed NeSLET approach vis-à-vis BLINK at  $1\%$  training dataset.
B.6 Entity Path Probability  $p(\text{path}_j \mid e)$
The entity's path probability  $p(\text{path}_j \mid e)$  is computed as follows.
1.  $p(path_j \mid e) \gets 0$  if leaf node of path  $j$  is not reachable from any valid type of  $e$ . For example, in Figure 2), we would be having  $p(path_j \mid e_i) = 0$ ,  $\forall j = 1 \rightarrow 6$ .
2.  $p(\text{path}_j \mid e) \gets 0$  if leaf node of path  $j$  is reachable from some valid type (say  $t_k$ ) of  $e$  but a direct child of  $t_k$  not lying on the path  $j$  is also a valid type for  $e$ . For e.g., in Figure 2, imagine node  $t_4$  having one more child called  $t_{14}$  and the corresponding path being path  $8$ . Then, we would be having  $p(\text{path}_8 \mid e_i) = 0$ .
3. For each of the remaining path, let  $t_k$  be the deepest node which is also a valid type of  $e$ . Traverse the lower portion of path starting from node  $t_k$  all the way up to leaf node and multiply conditional probabilities of the edges on the way to get  $p(\text{path}_j \mid e)$  as follows:  $p(\text{path}_j \mid e) = \prod_{(b,a) \in \text{path}_j, b \text{ is descendant of } t_k} p(t_b \mid t_a)$ . We approximate the conditional probabilities  $p(t_b \mid t_a)$  with prior probabilities from the given dataset about entities and the corresponding type set.