SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2: Datasets summary. For train (test) domain datasets, we have specified a dash  $(-)$  in its test (train) column as that set is never used. For the last row, the splits are as per Onoe and Durrett (2020).
Implementation Details: We used the BLINK
source code from (Wu, 2020) as the base model and implemented NeSLET on top of it. The hyperparameters that we used closely follow BLINK. For both the mention and entity BERT models, we use the bert-base-uncased model. We use a batch size of 128, maximum context length of 64 (32 tokens on each side of the mention), and the maximum entity description length is set to 128. The learning rates are  $\eta_{zel} = 10^{-5}$  and  $\eta_{type} = 10^{-3}$ . We use ADAM (Kingma and Ba, 2015) to optimize the objective. For BLINK and NeSLET models, we train each of them in two stages - first with in-batch negative entities, followed by hard-negative (Gillick et al., 2019) entities. We obtained the hard-negatives similar to Wu et al. (2020) by finding the top-10 predicted entities for each training example. These hard negatives are combined with the random in-batch negatives during training. We do not perform hard negative mining for the types because we use all the negatives types while computing  $\ell_{type}$ . In both these stages, the number of training epochs is 30 for  $0.01\%$  and  $0.1\%$  data splits, and 4 for the  $1\%$  data splits. The best model is selected based on the source domain validation set accuracy which is computed after each epoch. We initialize the NeSLET model with the weights obtained by training BLINK on the corresponding training data splits ( $0.01\%$ ,  $0.1\%$  and  $1\%$ ). We use BERT to compute the initial embedding for types based on their names. The type loss weight,  $\alpha$ , for each iteration is obtained using:  $\alpha = \frac{2}{1 + e^{-\gamma p}} - 1$  where,  $\gamma$  is set to 10 and  $p \in [0,1]$  is the training progress. In the NeSLET model, we use a weighted sum (learnt) of the CLS vectors from layer #5 to #11 of the mention BERT as the mention representation for the purpose of type prediction.
Computing Infrastructure: We trained our models on a single machine having  $2 \times 20$  core POWER9 processors,  $6 \times$  Nvidia Volta V100 GPUs with 32GB memory, and 512GB system RAM. A single training epoch for the NeSLET model on a train set of 900 samples takes 2 to 3 minutes.
Results: Table 3 summarizes our experimental results, where we have compared performance of NeSLET with two baselines - BLINK and GENRE (Cao et al., 2021). For training and inference of GENRE, we use the hyperparameters reported in (Cao et al., 2021) (more details given in Section B.4 of the appendix). The performance
Domain	Training Data %	Method
Train	Test	GENRE	BLINK	NeSLET-G	NeSLET-L	NeSLET-F	Gain	Hierarchy Gain
Wiki BLINK	NED	0.01	33.1	56.6	57.0	57.0	57.7	1.9	-1.2
0.1	44.3	65.4	64.6	65.6	65.4	0.3	0.3
1	55.0	67.7	70.0	69.7	69.9	3.4	0.1
CoNLL	0.01	49.4	61.2	62.6	64.3	63.5	5.1	1.3
0.1	60.2	71.6	70.5	73.0	70.6	2.0	3.4
1	68.5	72.3	74.7	75.2	74.4	4.0	1.1
Wiki FGET	NED	0.01	22.3	47.6	52.8	52.1	55.3	16.2	-4.5
0.1	36.8	62.0	63.7	63.6	63.0	2.7	1.1
1	50.7	67.3	68.1	67.6	69.2	2.8	-1.6
CoNLL	0.01	37.6	52.8	60.1	60.1	58.4	13.8	2.9
0.1	53.2	68.4	70.5	69.8	69.6	3.1	1.3
1	62.5	73.1	75.0	75.1	73.8	2.7	1.8
numbers for NeSLET in Table 3 were reported using the models (hyperparameter configurations) that resulted in the best performance on the source domain's validation set. In all of our experiments, BLINK performed better than GENRE in low training data regimes. The detailed performance numbers for validation and test sets across tuning ranges of various hyperparameters are captured in Tables 16, 17, and 18 of the appendix. These tables also capture the performance on the auxiliary task. Table 10 of the appendix shows the performance saturation trend for BLINK beyond  $1\%$  training data. We have captured the statistics related to leakage of entities and mentions in Tables 6, 7, and 8 of the appendix.
Table 3: Performance of NeSLET compared to GENRE and BLINK on target domain's test set. NeSLET-G, NeSLET-L, and NeSLET-F correspond to the variants of NeSLET that use Gödel (G), Lukasiewicz (L) norms to exploit the type hierarchy, or, assume that the types are Flat (F) i.e., no hierarchy. The Gain column denotes the performance gain (in %) obtained by the best of NeSLET-G and NeSLET-L and NeSLET-F relative to the best of GENRE and BLINK. The Hierarchy Gain column denotes the performance gain (in %) obtained using the hierarchical structure of entity types and is calculated as the best of NeSLET-G and NeSLET-L relative to NeSLET-F.
Domain	Method
Train	Test	BLINK (100%)	NeSLET (1%)
Wiki BLINK	NED	71.7	70.0
CoNLL	71.5	75.2
Wiki FGET	NED	67.7	69.2
CoNLL	74.4	75.1
Table 4: Comparing the accuracy of BLINK trained on  $100\%$  data and NeSLET trained on  $1\%$  data.
Insights: In Table 3, we see that NeSLET outperforms the baselines in all twelve training and
test domain combinations (2 training datasets x 2 test sets x 3 training data percentages). These results validate our claim that learning entity linking and entity type prediction in a multi-task learning fashion leads to improved performance on entity linking in low training data regimes. This characteristic is beneficial for real-world applications, where acquiring training data is quite expensive. From Table 3, we can see that the variants of NeSLET that exploit the type hierarchy using Gödel and Lukasiewicz norms (NeSLET-G and NeSLET-L) outperform NeSLET-F (which ignores the type hierarchy) in nine out of the twelve experiments. It shows that exploiting type hierarchy boosts the entity linking accuracy most times. The experiments in Table 3 also suggest that one fuzzy logic operator need not perform the best on all domains. The choice of these operators can be considered as a hyperparameter. In Table 4, we compare the accuracy of BLINK trained on  $100\%$  data with NeSLET trained on  $1\%$  data. For the purpose of this study, we consider the version of NeSLET (Gödel, Lukasiewicz or Flat) that achieves the highest accuracy on the given combination of train and test datasets. NeSLET trained on  $1\%$  data outperforms BLINK trained on  $100\%$  data in three out of the four experiments. NeSLET not only manages to recover the accuracy that BLINK loses due to
the reduction in training data size, but also goes beyond. The positive transfer of knowledge from the auxiliary task of entity type prediction to the primary task of entity linking enables NeSLET to outperform the baselines in low data regimes.
6 What if ZEL Model is a Black-Box?
Note that NeSLET model exploits entity type information only at the time of training a ZEL model (BLINK in this case). However, what if the ZEL model is available only as a black-box? For such a scenario, which is quite plausible in practice, it is not possible to train NeSLET like models that exploit entity types. To address this, we propose a neuro-symbolic inference Algorithm 1 that leverages entity type information only at inference time and thereby improves the performance of the given black-box ZEL model. Algorithm 1 assumes access to two black-box models - one for ZEL task and other for entity types prediction. Both of these models may be trained on separate datasets and made available as pre-trained black-box models. Algorithm 1 uses the types model to re-rank entities outputted by the ZEL model.
Algorithm 1: Type-based Inference
1 For a given  $m_{test}$ , pick a set  $S$  of top- $k$  entities from  $\mathcal{E}$  using the zel score
$$
S _ {z e l} (e \mid m _ {t e s t}) = p (e \mid m _ {t e s t}; \theta_ {m} ^ {z e l *}, \theta_ {e} ^ {*});
$$
2 For each entity  $e \in S$ , compute type based ranking score by  $S_{type}(e \mid m_{test}) =$
$$
\sum_ {t _ {j} \in \mathcal {T}} \mathbb {1} _ {e} (t _ {j}) \cdot p (t _ {j} \mid m _ {t e s t}; \theta_ {m} ^ {t y p e ^ {*}}, \theta_ {t} ^ {*});
$$
3 For each entity  $e \in S$ , compute a total ranking score given by
$$
\begin{array}{l} S _ {t o t a l} = \left(\beta_ {z e l} \cdot S _ {z e l}\right) + \left(\beta_ {t y p e} \cdot S _ {t y p e}\right) + \\ \left(\beta_ {\text {b o t h}} \cdot S _ {z e l} \cdot S _ {\text {t y p e}}\right) \text {w h e r e}, \\ 0 \leq \beta_ {z e l}, \beta_ {t y p e}, \beta_ {b o t h} \text {a r e w e i g h t s o f} \\ \end{array}
$$
$$
\text {d i f f e r e n t}
$$
4 Output the entity with highest joint score.
When  $\beta_{zel} = 1$ ,  $\beta_{type} = \beta_{both} = 0$ , Algorithm 1 gets simplified to the inference strategy of the ZEL model. The inference strategy of Algorithm 1 runs into a problem if types are arranged in a hierarchy. Consider two entities  $e_1$  and  $e_2$ . After type augmentation, suppose valid type sets of  $e_1$  and  $e_2$  are given by the paths  $root \to t_2 \to t_6$  and  $root \to t_2 \to t_6 \to t_{10}$ , respectively, in type hierarchy of Figure 2. In this case, the type-based score  $S_{type}(\cdot)$  will always score  $e_2$  higher than  $e_1$ . Thus,
it has a bias in terms of preferring the entities having deeper penetration in the hierarchy. If we modify the formula  $S_{type}(\cdot)$  and divide it by the path length then also it will be problematic as it will now favor shallow paths. To mitigate this bias, we suggest following revision in this formula:  $\tilde{S}_{type}(e \mid m_{test}) = \sum_{path_j} p(path_j \mid m_{test}) \cdot p(path_j \mid e)$  where, sum is taken over all the paths from root till leaves in the type hierarchy (see Figure 2). For each such path,  $p(path_j \mid m_{test})$  denotes predicted probability for mention's type being the leaf node on that path. The entity's path probability  $p(path_j \mid e)$  depends on whether  $path_j$  contains any valid type of  $e$ . Details of computing  $p(path_j \mid e)$  are given in Section B.6 of the appendix.
Hierarchy Pruning: Computation of  $p(\text{path}_j \mid e)$  involves estimating prior probability  $p(t_b|t_a)$  of an entity having finer type  $t_b$  given it has a coarse type  $t_a$  (parent of  $t_b$ ). If  $t_b$  is too fine-grained, we do not have enough data to reliably estimate  $p(t_b|t_a)$ . To mitigate such data sparsity issues, we consider only top- $k$  levels of the hierarchy from the root and restrict all the calculations on these types (see Figure 2). Our experiments show strong evidence that pruning improves the performance.
Results: Table 5 shows the performance gain obtained by using type-based inference on top of the NeSLET-L model (treated as black-box). There are two hyperparameters in Algorithm 1 - top- $k$  entities return by ZEL model, and pruning level of the type hierarchy. The values and tuning ranges of these hyperparameters are given in Table 9 of the appendix. The detailed performance numbers are given in Tables 16, 17, and 18 of the appendix. Anecdotal examples given in Tables 11 and 12 show how our inference scheme pushes the gold entity to Rank-1 which otherwise lies at Rank-2 in ZEL model's prediction. Also, in few cases, our type-based inference pushes down the correct prediction from Rank-1 position. Two such examples are given in Tables 13 and 14 of appendix.
Domain	Method
Train	Test	NeSLET-L	NeSLET-LT
Wiki BLINK	NED	69.65	69.64
CoNLL	75.24	75.37
Wiki FGET	NED	67.58	67.67
CoNLL	75.06	75.39
Table 5: Accuracy boost for ZEL black-box model (NeSLET-L trained on  $1\%$  data) when applied type inference (NeSLET-LT).
We have developed a multi-task approach called NeSLET wherein one can leverage the auxiliary domain knowledge about entity types so as to improve the performance on zero-shot entity linking task, beyond what SOTA methods such as BLINK and GENRE offer, in an extremely low training data regime. We believe, this research opens up an avenue for such deep learning based methods to be tried in real applications where training examples are very less. The future directions include exploring other architectures and learning schemes to train our hard parameter sharing model. Design of newer auxiliary tasks based on self-supervision is another potential direction.
Any ZEL technique, like ours, that advances the SOTA with less training data is a boon for applications such as KB question answering, document understanding, dialogue systems, etc. Our solution, however, comes with its own limitations and risks as follows. 1) Our assumption of entity types coming from the same hierarchy for both train and test domains is not always true in practice. Moreover, if there is no entity type information available in the first place, then our approach can not even be used. 2) The sensitivity of our predictions with respect to mild perturbation in input text could be a risk factor while deploying it in real-life applications. It warrants a rigorous study that we leave as future work.
We use publicly available datasets for training and testing our models. The Wiki BLINK and CoNLL-YAGO datasets are available under the MIT and CC-BY-3.0 licenses respectively. For the other datasets mentioned in Table 2, we obtained permissions from the authors (Onoe and Durrett, 2020) as the license was not explicitly mentioned on their Github. All of these datasets were constructed using publicly available sources such as Wikipedia for the purpose of developing entity linking systems and they are being used as intended. We believe these datasets do not contain any information that is offensive or uniquely identifies any individual.
We would like to thank the entire team from the IBM Research AI Hardware Center and the Center
for Computational Innovation at Rensselaer Polytechnic Institute for computational resources on the AiMOS Supercomputer. We conducted most of our experiments on the AiMOS computing platform. We would also like to acknowledge IBM Cognitive Computing Cluster (CCC) for providing resources to carry out various experiments.
2020. DBpedia Ontology. http://akswnc7.informatik.uni-leipzig.de/dstreitmatter/archivo/dbpedia.org/ontology--DEV/2020.10.15-031000/ontology--DEV_type=sparsed.owl.
Debayan Banerjee, Debanjan Chaudhuri, Mohnish Dubey, and Jens Lehmann. 2020. PNEL: Pointer network based end-to-end entity linking over knowledge graphs. In ISWC, pages 21-38.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In ICLR.
Rich Caruana. 1995. Learning many related tasks at the same time with backpropagation. In NeurIPS, pages 657-664.
Shuang Chen, Jinpeng Wang, Feng Jiang, and Chin-Yew Lin. 2020. Improving entity linking by modeling latent entity type information. In AAAI, pages 7529-7537.
Andrew Chisholm and Ben Hachey. 2015. Entity disambiguation with web links. Transactions of the Association for Computational Linguistics, 3:145-156.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-fine entity typing. In ACL, pages 87-96.
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2009. Introduction to Algorithms. MIT press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186.
Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In EMNLP, pages 2619-2629.
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context-dependent fine-grained entity type tagging. arXiv preprint arXiv:1412.1820.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In CoNLL, pages 528-537.
E Giunchiglia and T Lukasiewicz. 2020. Coherent hierarchical multi-label classification networks. In NeurIPS, pages 9662-9673.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In EMNLP, pages 2681-2690.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In EMNLP, pages 782-792.
Hang Jiang, Sairam Gurajada, Qiuhao Lu, Sumit Neelam, Lucian Popa, Prithviraj Sen, Yunyao Li, and Alexander Gray. 2021. LNN-EL: A neuro-symbolic approach to short-text entity linking. In ACL, pages 775-787.
Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2019. Transferable contrastive network for generalized zero-shot learning. In ICCV, pages 9764-9773.
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander Gray, Ramón Fernández Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue-Nkoutche, et al. 2021. Leveraging abstract meaning representation for knowledge base question answering. In Findings of ACL-IJCNLP 2021.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Erich-Peter Klement, Radko Mesiar, and Endre Pap. 2000. Triangular Norms, volume 8 of Trends in Logic. Springer.
George Klir and Bo Yuan. 1995. Fuzzy sets and fuzzy logic, volume 4. Prentice hall New Jersey.
Phong Le and Ivan Titov. 2018. Improving entity linking by modeling latent relations between mentions. In ACL, pages 1595-1604.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Soren Auer, and Christian Bizer. 2015. DBpedia - A large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, pages 7871-7880.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. 2020. Efficient one-pass end-to-end entity linking for questions. In EMNLP, pages 6433-6441.
Xiao Ling and Daniel S Weld. 2012. Fine-Grained entity recognition. In AAAI, pages 94-100.
Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In ACL, pages 3449-3460.
Andreas Maurer. 2006. Bounds for linear multi-task learning. Journal of Machine Learning Research, 7(1):117-139.
Max Planck Institute for Informatics. 2013. AIDA CoNLL-YAGO Dataset. https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads.
Mohamed Morsey, Jens Lehmann, Soren Auer, Claus Stadler, and Sebastian Hellmann. 2012. Dbpedia and the live extraction of structured data from wikipedia. Program: Electronic Library and Information Systems, 46(2):157-181.
Yasumasa Onoe. 2020. Et4el. https://github.com/yasumasaonoe/ET4EL.
Yasumasa Onoe and Greg Durrett. 2020. Fine-grained entity typing for domain independent entity linking. In AAAI, pages 8576-8583.
Laurel Orr, Megan Leszczyński, Simran Arora, Sen Wu, Neel Guha, Xiao Ling, and Christopher Re. 2021. Bootleg: Chasing the tail with self-supervised named entity disambiguation. In CIDR.
Jonathan Raiman and Olivier Raiman. 2018. Deeype: Multilingual entity linking by neural type system evolution. In AAAI, pages 5406-5413.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
Santosh K. Srivastava, Dinesh Khandelwal, Dhiraj Madan, Dinesh Garg, Hima Karanam, and L. Venkata Subramaniam. 2020. Inductive quantum embedding. In NeurIPS.
Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Learning sparse sharing architectures for multiple tasks. In AAAI, pages 8936-8943.
Hongyin Tang, Xingwu Sun, Beihong Jin, and Fuzheng Zhang. 2021. A bidirectional multi-paragraph reading model for zero-shot entity linking. In AAAI, pages 13889-13897.
Yogarshi Vyas and Miguel Ballesteros. 2021. Linking entities to unseen knowledge bases with arbitrary schemas. In *NAACL-HLT*, pages 834-844.
Ralph Weischedel and Ada Brunstein. 2005. Bbn pronoun coreference and entity type corpus. Linguistic Data Consortium, Philadelphia, 112.
Ledell Wu. 2020. BLINK. https://github.com/facebookresearch/BLINK.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, pages 6397-6407.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. In SIGNLL, pages 250-259.
A Leakage Analysis for Entity and Mention
Tables 6, 7, and 8 show the percentage of test examples in our benchmark datasets whose gold entities  $(e)$ , mentions  $(m)$ , entity and mention pairs  $(e,m)$  were seen during training, respectively.
B.1 Type-based Inference Hyperparameters
Table 9 shows hyperparameters and their tuning ranges related to our proposed type-based inference Algorithm 1. Initially, we tuned these hyperparameters on the source domain's validation set, but that did not result in any performance gain. The reason could be a significant difference in the source and target domain's data distributions. To explore further, we tuned these hyperparameters on the target domain's validation set and obtained a boost in entity linking performance. Improving the type-based inference algorithm to obtain gains by tuning on the source domain alone is a direction of future work.
B.2 BLINK Accuracy
Table 10 shows BLINK's bi-encoder accuracy with varying level of training set size. This serves as a key baseline for our results.
B.3 Anecdotal Examples
Tables 11 and 12 show anecdotal examples where, NeSLET model predicts the gold entity at Rank-2 but, our proposed type-based inference corrects the prediction of NeSLET by promoting it to Rank-1 during inference time. Tables 13 and 14, on the other hand, show anecdotal examples where type-based inference makes the prediction of the ZEL model incorrect. However, at an aggregate level, our type-based inference overall improves ZEL performance. Here the type model comes from NeSLETwith the Lukasiewicz norm.
B.4 GENRE Experiment Details
GENRE (Cao et al., 2021) is a SOTA entity linking system based on BART (Lewis et al., 2020). Given an input mention  $m$  to GENRE, it autoregressively generates an entity name  $e$ . We initialized BART weights with pre-trained BART from (Lewis et al., 2020) and fine-tuned it on Wiki BLINK (Wu et al., 2020) and Wiki FGET (Onoe and Durrett, 2020) datasets for different training data splits (0.01%, 0.1% and 1%). We used the hyperparameters reported in (Cao et al., 2021) and the training script available at https://github.com/facebookresearch/GENRE for training GENRE. For different training data splits, we used different numbers of training update steps as reported in Table 15. We selected the model using test domain's validation set. For inference, we used the same hyperparameters as used by (Cao et al., 2021) for the entity disambiguation task. We have constructed a trie (Cormen et al., 2009) using BLINK's 5.9M entities set to perform constrained decoding at the inference time.
B.5 Comparison of NeSLET vs. BLINK
Tables 16, 17, and 18 show detailed comparison of NeSLET with BLINK when both are trained with  $0.01\%$ ,  $0.1\%$ , and  $1\%$  of training data splits. The ETP  $F_{1}$  is the performance of the auxiliary ETP task. Different rows capture variations in train/test domain datasets as well as key hyperparameters for NeSLET. 'Flat' loss means treating the type hierarchy as flat. Column  $\beta$  corresponds to the values for  $(\beta_{zel}, \beta_{type}, \beta_{both})$ . The NeSLET accuracy when inferred w/o types corresponds to the case  $\beta_{zel} = 1$ ,  $\beta_{type} = \beta_{both} = 0$ . For each row, the NeSLET performance w/ types is taken for the combination of 'Tree Level', 'top- $k$ ', and  $\beta$  that resulted in the best NeSLET performance on test domain's validation set.