Entity Disambiguation on a Tight Labeling Budget
Authors: Audi Primadhanty
Ariadna Quattoni
Venue: EMNLP 2023 Findings (2023)
Keywords: entity linking
learning under a budget
tensor bilinear model
TL;DR: Effective strategy to learn an entity disambiguation model with small annotation budget, using feature diversity and low rank correction.

Abstract: Many real-world NLP applications face the challenge of training an entity disambiguation model for a specific domain with a small labeling budget. In this setting there is often access to a large unlabeled pool of documents. It is then natural to ask the question: which samples
should be selected for annotation? In this paper we propose a solution that combines feature diversity with low rank correction. Our sampling strategy is formulated in the context of bilinear tensor models. Our experiments show that the proposed approach can significantly reduce the amount of labeled data necessary to achieve a given performance.

Entity Disambiguation on a Tight Labeling Budget
Audi Primadhanty and Ariadna Quattoni Universitat Politècnica de Catalunya Barcelona, Spain audi, www.amaatttuupp..c....mm edu, aquattoniécs. upc. edu
Abstract
follow this line of work and develop method that combines a model correction step designed to im- prove the generalization performance of models learnt under tight annotation budgets with a diver- sity sampling strategy.
Many real-world NLP applications face the challenge of training an entity disambiguation model for specific domain with small label- ing budget. In this setting there is often access to a large unlabeled pool of documents. is then natural to ask the question: which samples should be selected for annotation? In this paper we propose solution that combines feature di- versity with low rank correction. Our sampling strategy is formulated in the context of bilinear tensor models. Our experiments show the proposed approach can significantly reduce the amount labeled data necessary to achieve a given performance.
Our work is framed in the context of bilinear ten- models for entity disambiguation. These types of models score mention-entity pairs by exploiting rich representation of the mention context and entity descriptions the target KB. Furthermore, our model has a bilinear lexicalized attention mech- anism that assigns score to every pair in the Carte- sian product space of context and entity description words. Our first observation is that the lexicalized bilinear attention matrix will be typically low-rank. Following this insight develop model correc- tion technique based matrix completion that improve the generalization performance of bilinear tensor models trained on tight annotation budgets. We combine this with feature diversity sampling. and it is by combining these two ideas that we obtain optimal results.
1 Introduction
Many real-world NLP applications face the chal- lenge of training an entity disambiguation model for specific domain with small labeling budget. For example, defense research analyst might be interested in mapping mentions of military equip- ment appearing in military research articles target knowledge base describing emergent defense technologies. The time of an expert might be costly and we would like to minimize the amount of an- notations required from them.
The two main contributions of this work are: 1) We develop novel strategy to improve the gen- eralization performance of bilinear tensor models trained under tight annotation budgets. Our strategy combines low-rank matrix completion of lexical attention matrix and feature diversity sampling. 2) Our experiments on entity disambiguation show that the proposed approach can significantly reduce the amount of labeled data necessary to achieve given entity disambiguation performance.
When training an entity disambiguation model has often access to large unlabeled pool of documents, We assume that we can recognize men- tions of the target class and that we can generate set of candidate Knowledge Base (KB) entities for each of them. Solving the entity disambiguation task involves training a model that takes a target mention and set of candidate KB entities as inputs and outputs the correct grounding (i.e. the correct entity in the KB). We assume that have small annotation budget and that our goal is to train the best model possible under the budget constraints.
Related Work
The sampling problem in learning under budget is related to active lcarning (Zhang et al., 2022). A few works addressed active learning for entity link- ing (Lo and Lim, 2020; Oberhauser al., 2020). Different to active learning the low budget set- ting, we don't assume the existence of prior model and thus our setting resembles the cold start prob-
One natural approach to tackle this problem is to design sampling strategies that attempt to max- imize the diversity of the selected samples. We

---

lem (Yuan et al., 2020; Jin al., 2022). To the best of our knowledge we are the first ones ad- dress cold start for entity disambiguation. As in active learning, the sampling approach can exploit either the informativeness or the representative- ness of data points (Zhang et al., 2022). Hacohen et al. (2022) showed that, for low-budget scenar- ios, representativeness is a better query strategy. We propose two representative-basee sampling ap- proaches and study their effectiveness for the case of entity disambiguation. Our method combines diversity sampling with low-rank matrix comple- tion, which has been shown to help generalization in low-budget scenarios (Primadhanty et al., 2015; Quattoni et al., 2014).
s(e) return the set of their unique words
In a learning under a budget setting we assume that we are given an unlabeled dataset of tuples (m, me, C(m)) E U and a labeling budget n. We can select n samples from 2/ to be labeled to create the training set T. The goal is to train the most accurate model under the given budget constraint.
3.2 Tensor Entity Disambiguation Model We will consider tensor models of the form:
expleemm,.Cm))
P(el(m, me, C(m)) = Σ mco
e'EC(m)
The scoring function O(e, m, me) computes the compatibility between an entity and a mention in context. This function will be parameterized by a dm x xd, attention matrix A, where dmn is the number of unique words appearing in the context of any mention and d, is the number of unique words appearing in the description of any entity, The attention matrix is indexed by context mention words and entity description words and the (i, matrix cell expected to capture the compatibility between a mention context word and an entity de- scription word. The scoring function is also param- eterized by two db x xd, matrices W and Z, where db is the dimensionality of the BERT embeddings. This two matrices are expected to capture the com- patibility between mention and entity description embeddings and between mention context and en- tity description embeddings, respectively. The scor- ing function is the sum of a contextual and mention term:
3 Tensor Model with Bilinear Attention
3.1 Learning Setting
Our goal is to train a local entity disambiguation model that takes as input an entity mention in con- text and a set of candidate KB entities and predicts the gold entity in the KB to which the mention should be linked. More formally, we assume that we are given: 1) A set of target entities KB = e2..e....... each with an associated description. 2) A training set tuples (m, mc, C(m)) gmm) C T, where m is an entity mention, me is the mention context, C(m) is set of candidate entities (i.e. a subset of KB) generated by some candidate gen- eration algorithm, and g(m) is the correct entity for the given mention Our goal is use T and KB to train a model that takes as input a tuple (m, me, ((m) and predicts an entity²
KB.
In order to train our model, we also assume that we can compute set of features of the mention, its context and all entities in the KB. More precisely, for each mention, we compute (contextual) pre- trained BERT embedding vectors of the mention text bm and its context Dmc- as well as a sparse fea- ture vector of the context smcc). On the KB side, we represent each entity by a pre-trained BERT embedding be and sparse feature function (e) of its description. The embeddings bmc and b. are computed by taking the average of the BERT embeddings of all words in and the entity description, respectively. The functions sme) and
(ee, m, mc) - App +b Wb. + -b Zb.
pes(m)),gEE(e)
To train a model we perform standard max- likelihood estimation and find the parameter matri- ces A, W and Z that minimize the negative log- likehood of the examples in T.
Diversity Sampling and Low-rank Matrix Completion Correction
In this section we present our approach for learning an entity disambiguation under budget setting. Our approach combines a low-rank model correc- tion step with diversity sampling.
'For training we assume that s(m) E C(m). otherwise we manually add it to the set returned by ((m).
't test time we cannot ussume that the correct entity for a given mention will be in the candidate set, in practice however we work with candidate generation a algorithms of high recall.
3 After removing stop words.

---

4.1 Matrix Completion Correction
budget constraint we want to select a set of n samples (m, me, ((m) U that gives us the most information to produce a good completion of A. To achieve this we propose to select a sample that maximizes the coverage of the features in the Carte- sian product space. We call this sampling strategy: cross- cross-produt product diversity.
The intuition behind our model correction strategy is quite simple. Think of the attention matrix A as an instance of the classical collaborative filter- ing problem. In this problem one assumes that some users (rows of the matrix) have rated a set of movies (columns of the matrix) and the goal is to predict the ratings of users on movies that they have not rated. The underlying assumption behind the matrix completion solution to this problem is that, to predict unknown ratings for given user, we can interpolate the predictions of similar users (i.e. users that made similar ratings). The low-rank observation stems from the realization that there might be few users (i.e. rows) and a few movies (i.c. columns) that can serve as a basis to make all predictions, i.e. unknown predictions can be expressed as linear interpolations of these basis vectors. In the case of the model's 's attention matrix A, rows correspond to mention contextual words and columns to entity description words. The low- rank basis assumption states that there is a subset of context and entity words from which one can guess the compatibility of all other word pairs by performing low-rank matrix completion.
The cross-product diversity sampling problem can be reduced to an instance of a combinato- rial N-set-cover problem. In the classical N-set cover problem, one is given a set of elements P = (pI,P2....} (called the universe), collec- tion S of k subsets of P and a budget N on the num- ber of subsets that can be used for the cover. The problem consists in identifying sub-collection of N subsets of S whose union maximizes the number of elements of P covered. While this problem is known to be NP-hard, there exists several approx- imation algorithms, we use a simple greedy algo- rithm. In our case the universe will be the set of fea- tures in the Cartesian-produue space and the subsets are the samples in the unlabeled pool. Each sample (m, mc, C(m)) can be thought as set of features obtained by taking the union of the cross features with each candidate. More formally, the subset cor- responding to sample (m, me, C(m)) l will be defined as: ULEC(m) s(m, me) x s(c).
More precisely, the model correction algorithm that we propose as follows: Given a set R of I rank value parameters for the matrix com- pletion, we compute the mar(R))-ank singu- lar value decomposition of A. Then, for each E R. we build the rank-r; low-rank reconstruc- tion, keep only the positive values and convert it into binary matrix Ar,, and compute their mean Am- Next, for each entry (i,j) we com- pute its reconstruction uncertainty: A) Var([A,, (i,j) ... A,, (t,)))) Finally, we create the complete matrix A Am- - Au and add bias; given some threshold (h, if A (i, < th then A (i. = th.
We observed that considering all the features in the cuttesian-produe might not be optimal and that it is beneficial to filter out very infrequent fea- tures. Therefore we include feature frequency threshold so that only features that appear more than the allowed threshold are considered. The ex- periments section provides more details about how this parameter is validated.
The cross-product diversity strategy attempts to explicitly maximize diversity in the sparse Carte- sian product space. An alternative is to design sampling strategy that exploits the dense represen- tations, we refer to this strategy as: dense diversity. For this strategy we first compute the BERT repre- sentation b. of all entities appearing in 24 as well as the contextual BERT representation byc of all mentions in 14. Then for each tuple (m, me, C(m)) we generate vector representation by summing together the b and the average b. vectors of all candidate entities in C(m). To select n sam- ples from U we perform k-means clustering and obtain n clusters, then for each cluster we select the sample closest to the cluster centroid. Compared to the cross-product diversity strategy, the dense
4.2
Diversity Sampling for Optimal Attention Completion
The model correction step described in the previ- ous section exploits the information in the observed entries of A in order to predict the unseen entries. To guaranter the success of the completion step we must ensure that the observed entries in A provide sufficient information produce a good estimate (Ruchansky et al., 2015). The weights in A cor- respond to features in the Cartesian product space of mention and entity sparse word features. Under

---

In addition to the Cross-Product Diversity (CPD) and Dense Diversity (DD) methods, we experiment with two other baseline methods. Random (R) sam- pling and Candidate Diversity (CD) which tries to cover as many unique candidate entities as possible (see section 4.2 for the approach to the N-set-cover problem.).
0.9
Accuracy testa*
0.85
CD
0.8
DD+
CPD+
All data
0.75
0.82
R+
CD
CD
DD
DD+
CPD
Accuracy
0
500
1,000
1,500
2,000
0.8
Mentions
0.78
0.85
0.76
300
100
Accuracy testb
0.8
# Mentions
Figure Performance on AIDA testb*
0.75
Figure 1 shows performance on testa* and testh* partition. The methods marked with "+" use low- rank matrix completion correction. We can that CPD and DD sampling combined with low-rank matrix completion help increases model accuracy compared to random and CD sampling, especially with lower budgets. To give some concrete exam- ples, on testa* CPD+ model correction achieves 81 % accuracy with 50 labeled mentions, to get the same accuracy with random sampling you need at least 150 mentions, that is 3 times the number of samples. With 200 mentions CPD+ model correc- tion attains 84.4% accuracy, get that accuracy with random sampling we need more than 500 an- notations. In both instances with our approach the amount of supervision required to attain a given performance is reduced by at least half.
0.7
500
1,000
1,500
2,000
Mentions
Figure 1: Performance on AIDA testa* and testb*
diversity strategy is significantly more costly since it requires running BERT over all mentions and contexts in the unlabeled pool.
5 Experiments: Learning with tight budget
We run our experiments on AIDA C (Hoffart et al., 2011) annotated with entities from W We follow van Hulst et al. (2020) to se- lect the candidate entities and ignore samples anno- tated with out-of-KB entities. We produce Iearn- ing curve by training models with increasing anno- tation budgets. Each point uses five random seeds, and report their average performance. We val- idate all papee--rrseetrss (both for the sampling strategy and model optimization) on randomly se- lected 50 samples from each testa and testb par- titions. We will use * and testh* to refer to the testa and testb set that excludes the 50 samples used for validation.
Figure 5, shows two arbitrary points from the learning curve and decouple the sampling strate- gies from the low-rank correction. Both CPD and DD sampling helps the model's performance even without low-rank matrix completion. Moreover, low-rank matrix complction increase models' accu- racies regardless of the sampling strategies.
Conclusion
We have shown that by combining low-rank matrix completion of lexicalized attention matrix with diversity sampling we can significantly improve the generalization performance of a bilinear entity dis- ambiguation model under tight annotation budget constraints.
"htpss//resourcess. http:///rssouurresmpiiiiiiii mpw.inf.iiiiiiii inw. mpg. www..gddeeaooongaaaaaa thapss/wwww.wwiddaaa..rgg

---

Limitations
Pei-Chi Lo and Ee-Peng Lim. 2020. Interactive entity linking using entity- word representations. In Pro- ceedings of the 43rd International ACM SIGIR Con- ference on Research and Development in Information Retrieval, pages 1801-1804.
As mentioned in section 3.1, we are focusing on the task of local entity disambiguation, that is, our model is trained to disambiguate entities in isola- tion. One could further improve the local disam- biguation model by imposing some global consis- tency constraints that capture the interaction be- tween entities but this is outside the scope of this work.
Tom Oberhauser, Tim Bischoff, Karl Brendel, Maluna Menke, Tobias Klatt, Amy Siu, Felix Alexander Gers, and Alexander Löser. 2020. TrainX named entity linking with active sampling and bi-encoders. In Proceedings of the 28th International Conference on Computational Linguisties: System Demonstra- rions, pages 64-69, Barcelona, Spain (Online). In- ternational Committer on Computational Linguistics (ICCL).
Moreover, in entity disambiguation, we are as- suming that the mentions have been correctly de- tected in the text. It will be interesting to consider how to adapt the methods to the task of end-to-end Entity Linking, where we also need to identify the mentions.
Nathanael Perraudin, Vassilis Kalofolias, David Shu- man, and Pierre Vandergheynst. 2016. Unlocbox: A matlab convex optimization toolbox for proximal- splitting methods.
In section 2, we mention how learning under budgets can be related to active learning. We do not try to use our methods as a cold-start technique in active learning in this study, but this can be con- sidered for future works.
Audi Primadhanty, Xavier Carreras, and Ariadna Quat- toni. 2015. Low-rank regularization for sparse con- junctive feature spaces: An application to named en- tity classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 126-135, Beijing, China. Association for Computational Linguistics.
Acknowledgements
This project has received funding from the Euro- pean Research Council (ERC) under the European Union's Horizon 2020 research and innovation pro- gramme under grant agreement No 853459. The authors gratefully acknowledge the computer re- sources at ARTEMISA, funded by the European Union ERDF and Comunitat Valenciana as well as the technical support provided by the Instituto de Física Corpuscular, IFIC (CSIC-UV). This research is supported by a recognition 2021SGR-CAS Cat (01266 LQMC) from AGAUR (Generalitat de Catalunya).
Ariadna Quattoni, Borja Balle, Xavier Carreras, and Amir Globerson. 2014. Spectral regularization for max-mnagin sequence tagging. In Proceedings of the 31st International Conference on Machine Learn- ing, volume 32 of Proceedings of Machine Learning Research, pages 1710-1718, Bejing, China. PMLR.
Natali Ruchansky, Mark Crovella, and Evimaria Terzi. 2015. Matrix completion with queries. In Proceed- ings of the 21th ACM SIGKDD international confer- ence on knowledge discovery and data mining, pages 1025-1034,
References
Johannes M. van Hulst, Faegheh Hasibi, Koen Derek- sen, Krisztian Balog, and Arjen P. de Vries. 2020. Rel: An entity linker standing on the shoulders of gi- ants. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20 ACM.
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. 2022. Active learning on a budget: Opposite strate- gies suit high and low budgets. In International Conference on Machine Learning, pages 8175-8195. PMLR.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self- supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3948-7948. Online. Association for Computational Linguistics.
Johannes Hoffart, Mohamed Amir Yosef, alrii Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 conference on empir- ical methods in natural language processing, pages 782-792.
Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. A survey of active learning for natural language pro- cessing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1666-199,, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Qiuye Jin, Mingzhi Yuan, Shiman Li, Haoran Wang, Manning Wang, and Zhijian Song. 2022. Cold-start active learning for image classification. Information Sciences, 316:11 1636.

---

A Additional Details
A.1 Model Description
In figure 3, we show an example of an entity disam- biguation problem and the source of features and representations that use in model:
((me) is obtained by averaging the BERT embeddings of the context of the mention,
b(m) is the BERT embedding of the mention itself,
Algorithm 1: LR Attention Completion
Input:
(EE) is the average of BERT embeddings of the entity's description
A Attention matrix.
In addition to that, also use s(me) and s(E) which can be considered as hag-of-words features of the mention context and entity description.
R. set of rank value parameters for the matrix completion.
th, bias threshold.
We also show an example of the learned matrix entry (i. that is expected to capture the com- patibility between a mention context word and an entity description word.
Output: A
Compute the max (a)-())raakk singular value decomposition of A USV
A.2 Detailed Model Correction Algorithm
For each R build the rank-r low-rank reconstruction:
Algorithm describes in more details the low rank (LR) attention completion algorithm we briefly de- scribed in section 4.1.
A = (t. r)S(1: : r, r) : rr;:)
1 if (f,,)) 0,
(1,
B Additional Results
0 otherwise.
Figure 4 shows the same evaluation as Figure done on testa* partition. These results show the same observation as seen in evaluation on testh# partition; the Cross-Product Diversity and Dense Diversity sampling, well low rank completion correction can help in hight-budger scenarios.
Compute the mean reconstruction:
Am ΣiER Ar
[
For each entry Ai, compute its reconstruction uncertainty:
Au(i,) Var([A,, (1,5)... (i,))
Figure 5 and 6 shows the breakdown seen in Figure and for all points in the learning curve, and as we can see, the same observation still holds throughout the curve.
Create the completed matrix: A' Am
Au
A'(i,) if Af th,
A A'(i,) th otherwise.
=
Dataset
Return:A
We run our experiments on AIDA YAGO (Hoffart al., 2011 1). The dataset is partitioned into three sets: train, testa and testb. We follow van Hulst al. (2020) to select the candidate entities and ignore samples annotated with out-of-KB enti- ties. Using this strategies, we obtain 17174, 4422, and 4226 number of annotated mentions for the train, testa and testb partition, respectively.
Originally, testa is a validation partition, and testb is a test partition. But in scenario, it

---

KB Candidates
National Bank Act NBA Development League
Mention
NBA TV
context
Isaiah
National Basketball
gold entity
Thomas
Association
mayor
the
b(mc)
professional
best
Appaaerrleague]
basketball
b(E)
playe1
league
of
north
b(m)
NBA
america
BERT
BERT
Embeddings
Embeddings
Figure 3: This figure displays a concrete disambiguation training example, where the mention is NBA, and the gold KB entity is National Basketball Association.
0.87
0.85
CD
CDi
DV
DD+
CPD
CPD
0.86
0.8
Accuracy
Accuracy
0.85
0.75
0.84
CD
CD+
DD
DD+
0.7
CFD
CPD
0.83
All dam
0
500
1,000
1,500 2,000
0.82
# Mentions
300
400
Figure Performance on AIDA testb*
# Mentions
Figure 4: Performance on AIDA testa*.
is unrealistic to assume that we have access to a large validation partition since we only have small budget for annotation. We create small validation set from both testa and testb instead, selecting 50 random examples from each. It has also been well documented in previous studies that due some difference in the distribution, testa seems be an "casier" partition (models trained on the standard train set will perform better testa) than testb. So, we test on both to see if the strategies will behave differently depending on how "hard" the data is.
0.9
Accuracy
0.8
DD
DD+
CPO
CPD1
All
500
1,000
1,500
2,000
# Mentions
D Qualitative Analysis
Figure 5: Performance on AIDA testa*.
To understand better how the matrix completion correction might help improve performance, we performed a qualitative analysis of our results. We use the model trained on 300 random training sam-

---

ples and compare the correctly identified test sam- ples before and after applying matrix completion correction (i.e. Rand R+). focus on correctly identified entities not in the training set and choose one, in this case, "Juventus F.C."; two mentions of "Juventus F.C." are identified after applying ma- trix completion correction. We then look at the feature cross products of one mention and the en- tity and compare their weights in Rand R+, which shows that R+ can propagate weight to 35 unseen feature products, one being "defender" "asso- ciation ciation_football club". In 7a, we select a subset of the cross space to illustrate how the weight "propagation" works similarly to collab- orative filtering mentioned in section 4.1. If we treat the row as users and columns as movies, then let's look at the user "defender" and movie "asso- ciation football_clube
boat
0.50
fist
0.25
0.00
york
-0.25
tuesday
national
reland
00.5
defender
-1.00
We can see that the user "defender" has quite similar patterns to many other users such as "national", "tuesday", "cup", and "new". But we don't 't know this user will like the movie "association seo__foobblllllhe _cluu"," so we can use the clue from user "Tuesday", "cup", and "new" to guess that this user will like it. We can see in Fig- ure 7b that the matrix completion correction does that and "propagates" the weight to the cell.
human
econciation fouthall Borthan
person
amerccan
club
player
politciin
Craaaiie di
united statess
association
(a) Before matrix completion correction.
2,00
us.
0.75
beat-
0.50
new
We can also see from the patterns of seen cross products that loneiciationooooaalllllco is quite similar to "association foottaafoooaallaaoo ployyrrllpllacrr and "footballer", fender" would also be compatible with "associa- tion football_clute This illustrates how the matrix completion correction step is implicitly learning latent classes of features. In this example, "associ- cito__football_oom club",, and "footballer" seem to belong the same class of features.
cup
tueday
seems natural that "de-
-0.50
reland-
-0.75
defender
-1.00
unteed states of aameicc
human
musccation costtal club
scncietion fouthall player
emerican
politccin
eppubiic of Inelarnd
actor
gealiingi meggrati
(b) After matrix completion correction.
Figure 7: Heatmap of subset of the attention matrix before and after matrix completion correction. The model was trained using 300 random mentions. The rows are features of the mentions, while the columns are features of the entities.
E Hyperparameters
Here specify the different hyperparameters we to obtain the results shown in the paper. We run each experiments with 5 random seeds (1.2.3.4.5) and report the average performance. The learn- ing curve is created using the following number of samples: {10, 30, 50, 70, 90, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200, 1300, 1400. 1500, 1600, 1700, 1800, 1900, 2000). We use UN- LocBoX (Perraudin et al., 2016) for optimization and uses LIP = 10, TAU 0.01, and maximum iter- ation of 30. We use th 0.1 and R (1...20).


Reviews:
[{"review_id": "tkOMQPI7yu", "reviewer": "Reviewer_U5Tk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the suggestions to improve the paper and for highlighting that our study contributes valuable insights for addressing a relevant and practical problem of training a model in resource-constrained settings.\n\n- Due to space limitations and given the fact that AIDA is a well known dataset for entity linking, we omitted a more detailed description. But we understand that some readers might not be familiar with the dataset and we will add all the details in the appendix.\n\n- We agree that qualitative and interpretative analysis would be interesting for this study. In fact, we have done some qualitative analysis of the tensor-features that are completed by the low-rank correction. Unfortunately,  we decided to omit that section at the end due to the space limit. Because the contribution is quite technical we gave priority to explaining the technical details that are necessary to understand the work. Now we realize that the qualitative analysis could be added to the appendix and we will do so in the final version. It is indeed quite interesting because it shows that the matrix completion step is implicitly learning latent classes of features.\n\n- Unfortunately, this paper lacks comparisons with previous studies because to the best of our knowledge, there have been no other studies on sampling strategies for the cold start scenario (we don’t assume the existence of a prior model) for entity disambiguation. Due to this reason, we tried to be rigorous and create strong baselines, taking ideas from other related tasks and domains. \n\n- We realize now that the implications of our contribution might not be obvious. We will highlight here the main implications and try to fit them in the main content in the final version.\n\n   - The main practical implication of our work is that we can use the proposed sampling strategies to train an entity linker more efficiently when we have a new or domain-specific knowledge base, and we need to start from zero, which is a realistic and very frequent scenario in the industry. The sampling strategies can be used to obtain a better model with fewer annotations, this is important because there are many scenarios in which obtaining annotations can be very expensive. For example, as we mentioned in the introduction, a defense research analyst might be interested in mapping mentions of military equipment appearing in military research articles to a target knowledge base describing emergent defense technologies. In this case, we must focus the annotation effort on the most informative examples in the corpus.\n\n   - Furthermore, we believe that the technical implications of our contribution go beyond entity linking. The sampling ideas and model correction strategies that we propose can be useful for any task that can benefit from a bilinear component.  This includes a wide range of tasks. Essentially, any prediction task that needs to compute a compatibility score between an input and an output and that it does so by first projecting each of them to some feature space.\n\n   - More excitingly, in light of the current popularity of transformer models, we can think of ways to apply these strategies to the attention matrix. Although it is not trivial because the matrix is positional, we can think of a more general model that includes a lexicalized attention (i.e. a matrix with parameters associated with words from a dictionary rather than positions in the sequence). It is not hard to see how a lexicalized attention matrix can be thought of as weights of a bilinear scoring function. Therefore, it should be possible to extend our sampling and model correction ideas to work on this setting. While this is outside the scope of this study, we believe its publication might open the door for contributions that exploit our proposed ideas in the context of other models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n- The paper addresses a relevant and practical problem in NLP: training entity disambiguation models with limited labeling budgets.\n- The paper provides a clear and detailed explanation of the proposed method, including the formulation, algorithms, and reasoning behind the approach.\n- Experimental results are presented with a clear comparison of different sampling strategies and the impact of low-rank correction on model performance.", "reasons_to_reject": "\n- The paper could benefit from providing more details about the specific datasets used for experiments, including their characteristics and properties.\n- While the paper mentions the benefits of the proposed approach, a more qualitative or interpretative analysis of the results could enhance the understanding of why certain methods work better.\n- The paper could include a more in-depth comparison with other related works, discussing the strengths and weaknesses of existing approaches and how the proposed method compares.\n- Consider discussing potential practical implications of the findings, such as how the proposed method could be applied to real-world applications or scenarios beyond the scope of the current study.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThe paper introduces a novel approach for training entity disambiguation models with limited labeled data by combining feature diversity sampling and low-rank matrix completion within a tensor model framework. The method aims to improve generalization performance under tight annotation budgets. Experiments on the AIDA dataset demonstrate that the proposed approach, particularly when integrating low-rank correction, significantly reduces the required labeled data while achieving competitive accuracy. The study contributes valuable insights for effectively addressing the challenge of entity disambiguation in resource-constrained settings.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lbox88aZO1", "reviewer": "Reviewer_d5Yd", "summary": "", "questions": " \nSection 4.1 is somewhat confusing. It is not easy to follow the connection between collaborative filtering and the proposed approach. Could the authors clarify it a bit?\n\nWhat is the difference between \"testa\" and \"testb\"?\n\nCould the authors provide more clues about the implications of cross-product versus dense diversity strategies?\n\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, suggestions, and questions that will help us improve the paper. \n\n- The state of the art is somewhat short because we tried to compress it due to the space limit. Since this paper is quite technical, we focus on writing the technical aspects. Moreover, to the best of our knowledge, there have been no other studies on designing better sampling strategies on a cold start scenario (we don’t assume the existence of a prior model) for entity disambiguation. We have added state-of-the-art related tasks, including those published in conferences outside NLP. Following your concern, we will describe the related studies in more detail, and move some technical explanations to the appendix.\n\n- We agree that some design decisions might benefit from more explanations. Although the experiments might not look like they cover too many aspects, we tried to be selective and attempted to study the most relevant ones. This being said, we see how the paper will benefit from a more qualitative analysis of the results. We have indeed performed a qualitative analysis of the tensor features that are completed in the low-rank correction step but we omitted the section at the end because of space limitations.  This qualitative analysis can shed light on what the model is doing and provide better intuitions for some of our modeling decisions. We will include such analyses in the appendix.\n\n- In section 4.1, we use the analogy of collaborative filtering because we assume it is a more familiar example for the readers. The bilinear model represents two objects in a feature space and computes their compatibility, just as collaborative filtering (e.g. computing compatibility between user vs. movie). Related to the previous point, some qualitative analysis and examples in the appendix might help illustrate how the bilinear model learns sub-classes of words - analogous to sub-classes of users and movies in the collaborative filtering example above. For instance, in AIDA, there are some entities of sports teams with many training examples while other sports team entities have very few examples. In the analogy of user vs. movie collaborative filtering, we can think of these sports team entities as users, and the low-rank approach will “group” users that behave similarly and propagate features to those with fewer examples. We will include these intuitive examples and analysis from the experiment in the appendix to better illustrate what is happening.\n\n- “Testa” and “Testb” are standard partitions in the AIDA data, which is a standard benchmark for entity disambiguation, and thus we omitted the details. However, we agree that we can add these details to the appendix for clarity. Originally, testa is a validation partition, and testb is a test partition. But in our scenario, it is unrealistic to assume that we have access to a large validation partition since we only have a small budget for annotation. We create a small validation set from both testa and testb instead. It has also been well documented in previous studies that due to some difference in the distribution, testa seems to be an “easier” partition (models trained on the standard train set will perform better in testa) than testb. So, we test on both to see if the strategies will behave differently depending on how “hard” the data is.\n\n- Cross-product and dense diversity are both sampling strategies that try to maximize the diversity of the annotated data. The cross-product approach does this by covering as many features as possible, while the dense diversity strategy is trying to cover as many clusters in the data as possible. The implication is that if the feature space is homogenous, there is no natural clustering, and the result of the two strategies will be similar. Otherwise, if the features are highly clustered, we might gain by doing the clustering strategy because we will focus on dense areas of the feature space. On the other hand, it is also possible that the cluster space does not reflect the underlying structure of the feature space and will not help the sampling. We can see that the clustering diversity strategy requires stronger assumptions to work, but this is an obvious strategy that is quite popular in similar scenarios, so we considered both options in our study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n- The goal is well defined.\n- The description of the proposed approach is easy to follow.\n- The experiments are not very exhaustive but sufficient to demonstrate the effectiveness of the proposal.", "reasons_to_reject": "- The state of the art is somewhat short.\n- There are many design decisions not entirely justified.\n- The experiments do not cover too many aspects.\n\n", "questions_to_authors": " \nSection 4.1 is somewhat confusing. It is not easy to follow the connection between collaborative filtering and the proposed approach. Could the authors clarify it a bit?\n\nWhat is the difference between \"testa\" and \"testb\"?\n\nCould the authors provide more clues about the implications of cross-product versus dense diversity strategies?\n\n", "paper_topic_and_main_contributions": "A strategy to improve the generalization performance of bilinear tensor models trained under tight annotation budgets. They combine low-rank matrix completion of a lexical attention matrix and feature diversity sampling. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hlANht5svN", "reviewer": "Reviewer_XGHN", "summary": "", "questions": "A. Would be good to see -- certainly it would strengthen the work-- how the discussed data sampling approaches perform across multiple models and datasets. Did you do any exploration to this end?\nB. For dense diversity you used k-means clustering to end up selecting for each cluster the sample closest to the centroids. Why not use k-medoids for this task? As opposed to k-means the cluster centres are actual data points from the data, and you can specify any dissimilarity measure, i.e, you are not limited to the Euclidian distance\nC. Why did you settle on that particular entity disambiguation model? Would be good to see some references if that is the standard approach in the field.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review, questions and suggestions, especially the technical suggestions that are very interesting to explore.\n\n- To begin with, let us clarify our perspective on entity disambiguation; we see it as a component, instead of a simplified version, of entity linking. To perform entity linking, one must first recognize the entities and generate candidates before proceeding to disambiguation. While it is possible to learn these tasks jointly, we can also leverage the latest state-of-the-art NER models and combine it with the disambiguation component to create a complete entity linker. It is worth noting that in entity linking studies, the model is usually expected to abstain when the correct entity is not in the candidate list, while in entity disambiguation studies, it is assumed that the correct entity is already in the candidate list. Nevertheless, one can always add logic to the model to allow it to abstain, for example by having a minimum confidence threshold.\n\n- This paper is meant to introduce a new way of looking into a bilinear model and how we can exploit it for sampling. The experiments were done to illustrate that. We agree that it would be interesting to add more experiments on more datasets to strengthen the support of the observations, especially considering that the strategies can be used for other tasks as well. This is a good direction for future work, where we can show how to adopt these strategies to other tasks and datasets. This being said, we believe that the technical insights that we provide make a useful contribution on its own. To the best of our knowledge, we are the first ones to propose a solution for the cold-start setting in the context of bilinear models.\n\n- More excitingly, in light of the current popularity of transformer models, we can think of ways to apply these strategies to the attention matrix. Although it is not trivial because the matrix is positional, we can think of a more general model that includes a lexicalized attention (i.e. a matrix with parameters associated with words from a dictionary rather than positions in the sequence). It is not hard to see how a lexicalized attention matrix can be thought of as weights of a bilinear scoring function. Therefore, it should be possible to extend our sampling and model correction ideas to work on this setting. While this is outside the scope of this study, we believe its publication might open the door for contributions that exploit our proposed ideas in the context of other models. \n \n- We use k-means as it has been used in previous active learning studies. However, we see no reason not to use the clustering method that the reviewer suggested. While exploring different clustering methods is out of the scope of our study, this can make an interesting expansion for our future study.\n\n- With regards to the question of why we settled on a bilinear entity disambiguation model. The truth is that we first developed the idea for cold-start sampling in bilinear models and then we picked a task for which a bilinear model is a natural choice. Bilinear models are natural for entity disambiguation because both the mention context and the KB entity can be embedded in a feature space. Furthermore, while this might seem less obvious, bilinear models for entity disambiguation have also the advantage that they can naturally handle the zero-shot prediction (i.e. making predictions at test time that involve target entities that were not seen in the training data). This is because as long as we have the KB entry for an entity, we can generate its feature representation and compute its compatibility score with a given mention context. \n  \n- To the best of our knowledge, there has been no prior work on sampling strategies on a cold start scenario (we don’t assume the existence of a prior model) for entity disambiguation. There have been other studies on entity disambiguation that do not focus on sampling, and in the case of a strictly local model (where there is no additional global document information), results from a bilinear model are competitive.\n\n- The goal of our paper is to introduce a new way of looking into a bilinear model and how we can exploit it for sampling. We think it’s important and timely to introduce this idea now. With the popularity of transformer models with their attention matrices, it is not hard to see that at some level these models also exploit bilinear components.\n\n- A very good point regarding the hyperparameters, we agree that it would improve the reproducibility of the study to include these, we will add all the details in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors demonstrate that selecting data which maximizes feature diversity combined with low-rank matrix completion consistently outperforms under different data budgets simple baselines such as random data selection or candidate selection.", "reasons_to_reject": "The targeted scope is too narrow, even for a short paper: entity disambiguation, a simplified entity linking task, covering a single model applied on a single dataset.", "questions_to_authors": "A. Would be good to see -- certainly it would strengthen the work-- how the discussed data sampling approaches perform across multiple models and datasets. Did you do any exploration to this end?\nB. For dense diversity you used k-means clustering to end up selecting for each cluster the sample closest to the centroids. Why not use k-medoids for this task? As opposed to k-means the cluster centres are actual data points from the data, and you can specify any dissimilarity measure, i.e, you are not limited to the Euclidian distance\nC. Why did you settle on that particular entity disambiguation model? Would be good to see some references if that is the standard approach in the field.", "paper_topic_and_main_contributions": "The paper presents a study of different data sampling strategies for an entity disambiguation model. The approached task is a simplified entity linking task where the mentions are assumed to have been previously identified. The authors additionally experiment with a low-rank matrix completion method applied to a mention context to entity descriptor attention matrix part of the objective function of the model.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "For reproducibility you should state the hyperparameters you considered and also ended up selecting."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper innovatively combines feature diversity sampling, low-rank matrix completion in tensor models, and improving entity disambiguation with limited labeled data.

The proposed method for entity linking is clearly explained, and the experimental results support the paper's claims. However, the experimental section primarily focuses on a specific dataset, which somewhat limits its ability to fully demonstrate the method's applicability in different scenarios.
Decision: Accept-Findings