Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy
Authors: Na Li
Zied Bouraoui
Steven Schockaert
Venue: EMNLP 2023 Findings (2023)
Keywords: ultra-fine entity typing
word embeddings
conceptual neighbourhood

Abstract: Ultra-fine entity typing (UFET) is the task of inferring the semantic types from a large set of fine-grained candidates that apply to a given entity mention. This task is especially challenging because we only have a small number of training examples for many types, even with distant supervision strategies. State-of-the-art models, therefore, have to rely on prior knowledge about the type labels in some way. In this paper, we show that the performance of existing methods can be improved using a simple technique: we use pre-trained label embeddings to cluster the labels into semantic domains and then treat these domains as additional types. We show that this strategy consistently leads to improved results as long as high-quality label embeddings are used. Furthermore, we use the label clusters as part of a simple post-processing technique, which results in further performance gains. Both strategies treat the UFET model as a black box and can thus straightforwardly be used to improve a wide range of existing models.

Ultra-Fine Entity Typing with Prior Knowledge about Labels: A Simple Clustering Based Strategy
Na Li USST, Shanghai China
Zied Bouraoui
Steven Schockaert Cardiff University
CRIL CNRS & Univ. Artois
France
UK
1i li_na@usst.edu..
bouraoui@cril.ff fr
schockaerts1@carddff
Abstract
fine-grained entity typing. However, the 122 entity types that were considered in their work are still too coarse for many applications. For instance, the most specific type from their taxonomy that applies to the example above is company. A further refine- ment of entity types was proposed by Choi et al. (2018), who considered total of 10, 331 seman- tie types and introduced the term ultra-fine entiry typing (UFET). For instance, the ultra-fine type pharmaceutical is available for the above example.
Ultra-fima fine entity typing (UFET) is the task of inferring the semantic types, from large of fine-grained candidates, that apply to a given entity mention. This task is especially chal- lenging because only have small num- ber of training examples for many of the types, even with distant supervision strategies. State- of-the-art models, therefore, have to rely on prior knowledge about the type labels in some way. In this paper, show that the perfor- mance of existing methods can be improved using simple technique: we use pre-trained label embeddings to cluster the labels into mantic domains and then treat these domains additional types. We show that this strat- egy consistently leads to improved results, as long as nigh-quality label embeddings are used. We furthermore a simple post-pros pest-processing technique, which re- sults in further performance gains. Both strate- gies treat the UFET model as black box and can thus straightforwardly be used to improve a wide range of existing models.!
UFET is essentially a multi-label text classifi- cation problem. The main challenge stems from the fact that we only have a few labelled train- ing examples for most of the semantic types, and sometimes even none at all. The solution proposed by Choi et al. (2018) is based on two distant su- pervision strategies. Their first strategy is to use mentions that link to Wikipedia page and use the definition on that page infer semantic types, Their second strategy relies on nominal expressions which include their semantic type as a head word (e.g. president Joe Biden). More recent work has improved on these strategies using denoising tech- niques (Onoe and Durrett, 2019; Pan et al., 2022).
the label clusters part of
1 Introduction
Entity typing is the task of inferring the seman- tic type(s) of an entity that is mentioned in some sentence. While only a few coarse types were tra- ditionally considered, such as person and organi- sation, the focus has shifted to increasingly finer- grained types. Consider the following example:2 "The company said its production upgrades would also have a "short-term impact" on the delivery of vaccines to the UK". From this sentence, we cannot only infer that "the company" refers to an organ- isation but also that it refers to a pharmaceutical company. To allow for more informative predic- tions, Ling and Weld (2012) proposed the task of
Even with distant supervision, the number of training examples remains prohibitively small for many types. State-of-the-art methods, therefore rely on prior knowledge about the labels them- selves. For instance, Pan et al. (2022) use the rep- resentation of the labels in the decoder of BERT initialise the weights of the label classifiers, while Huang et al. (2022) use a fine-tuned BERT (Devlin et al., 2019) model to map type labels onto proto- types. However, the token embeddings from BERT are known to be sub-optimal as pre-trained rep- resentations of word meaning (Bommasani et al., 2020), whereas fine-tuning BERT encoder may lead overfitting, this paper, we therefore pur- sue a different strategy to inject prior knowledge about the labels into UFET models. We use pre- trained embeddings to cluster the labels into groups,
'Our code and evaluation scripts are available at https: //gi thub. Ctt.ub.commlina--D11 www.inallllcuffee /ute/ffettee with__omainss..... 'https://wwar bbc. co. uk/news/
world world-europe-S56655 55666399

---

which we call domains. Rather than changing the UFET model itself, we simply treat these domains as additional semantic types. In other words, given a training example with labels 41,-.,, I, we now also add the synthetic labels c1...., Cn. where o; is the cluster to which 4 belongs. This strategy has the advantage that arbitrary label embeddings can be used and that the UFET model itself does not need to be modified. Despite its simplicity, find this strategy to perform remarkably well, pro- vided that high-quality label embeddings are used. The best results are obtained with recent BERT- based strategies for learning word vectors (Li et al., 2023). In contrast, the representations from classi- cal word embedding models (Mikolov et al., 2013; Pennington et al., 2014) are too noisy to have pos- itive impact, with ConceptNet Numberbatch (Speer et al., 2017) being a notable exception.
the domains to post-process its predictions (Section 5). In this way, we can infer missing labels or remove conficting labels.
Related Work
Ultra-Fine Entity Typing Methods for ultra- fine entity typing typically rely on training sets that are automatically constructed from Wikipedia links (Ling and Weld, 2012: Choi et al., 2018). As a different strategy, Dai et al. (2021) extract weakly labelled examples from a masked language model, using Hearst-like patterns (Hearst, 1992) to rephrase the original sentence. For instance, the sentence "In late 2015, Leonardo DiCaprio starred in The Revenant" is transformed into "In late 2015, [MASK] such as Leonardo DiCaprio starred in The Revenant". 'he predictions of the language model for the [MASK] token are then used as weak labels for the target entity. Yet an- other possibility, considered by Li et al. (2022), relies on transfer learning. Specifically, they use pre-trained Natural Language Inference (NLI) model, where the original sentence is used the premise, and the hypothesis is sentence express- ing that the target entity has given type. While their approach achieves strong results, they have to run the NLI model for every possible type, which is prohibitively expensive for ultra-fine entity typing.
Apart from using pre-trained label embeddings, another line of work has focused on modelling la- bel correlations (Jiang et al., 2022). Essentially, the idea is to train a probabilistic model on top of the predictions of a base UFET classifier, While mean- ingful improvements are possible in this way, the extent to which label correlations can be learned is inherently limited due to the sparsity of the training data. As an alternative, we focus on two simple strategies to improve the predictions of the base UFET model. Both strategies directly rely on prior knowledge about the labels and are thus well-suited for labels that are rare in the training set. The first strategy is aimed at inferring missing labels: if the domain label is predicted but none of the labels that belong to that domain, add the most likely label from G₂ to the predicted set. The second strat- egy is aimed at removing conflicting labels: if an entity predicted to have multiple labels from the same cluster, then we use a pre- pre-trained classifier to check whether these labels are mutually incon- sistent, and if so, we remove the labels with the lowest confidence degree.
Label Dependencies In fine-grained entity typ- ing, there are usually various dependencies be- tween the labels. For instance, in the case of FIGER (Ling and Weld, 2012) and OntoNotes (Gillick et al., 2014), the labels are organised into a tree, which models can exploit (Ren et al., 2016; Shi- maoka et al., 2017; Xu and Barbosa, 2018; Murty et al., 2018; Chen et al., 2020). In the case of UFET, however, the label set is essentially flat (Choi et al.. 2018), which means that label dependencies have to be learned from the training data and/or external sources. Xiong et al. (2019) use Graph Neural Network (GNN) (Kipf and Welling, 2017) to take label correlations into account. The edges of the graph, which indicate potential label interactions, are selected based on (i) label co-occurrence in the training data and (ii) the similarity of the GloVe embeddings of the label names. The model thus relies on the assumption that labels with similar embeddings are likely apply to the same entities, which is too strong: the embeddings of mutually exclusive labels, such as teacher and student, can also be similar. Liu et al. (2021b) rely on two strate-
The contributions of this paper can be sum- marised as follows. We use pre-trained label em- beddings to group the given set of type labels into semantic domains. These domains are then used to improve a base UFET model in different ways:
By adding the domains as synthetic labels to the training examples (Section 4), the UFET model is exposed to the knowledge that these labels have something in common.
After the UFET model is trained, we can use

---

gies for modelling label correlation. One strategy is based on directly observed label correlations. For the other strategy, they replace the entity by the [MASK] token and obtain predictions from BERT for this token. Then they model correlations be- tween these predictions and the actual labels. Li et al. (2021a) also use GNN to model label inter- actions. The nodes of their graphs correspond to both entity types and keywords from the given sen- tences. Jiang et al. (2022) model label correlations using a probabilistic graphical model.
'hhe representation of each label typically derived from the sparse features of the documents that label, and the focus is on learning balanced clusters.
Problem Setting
In ultra-fine entity typing, the input consists of a sentence in which particular entity mention is highlighted. The aim is to assign all the seman- tic types that apply to that entity, given large C of around 10,000 possible types (Choi et al., 2018). This task can be treated as a standard multi- label classification problem. In particular, let & be an encoder which maps an input sentence s with entity mention m to a corresponding embedding ((8, m) IR. Together with the encoder Φ, we also learn scoring function fi R" + [0, 1] for each label I L. The probability that label ap- plies the entity mentioned in the input is then estimated as f((o(s, m))) As a representative re- cent approach, we consider the DenoiseFET model from Pan et al. (2022), which uses a BERT encoder with the following input:
Different from the aforementioned works, we exploit prior knowledge about label dependencies in way which is decoupled from the entity typ- ing model, by adding synthetic labels to training examples and by post-processing the predictions. This makes our method more transparent (i.e. we know which dependencies are used and why) and modular (i.e. our method can directly be applied to existing entity typing models). While the lack of integration with the entity typing model may seem overly simplistic, note that for most entity types we only have few training examples. It is difficult learn meaningful label dependencies from such data without also learning spurious correlations.
s [Pi) m [P][[3][MASK]
Here s is the input sentence, m is the span corre- sponding to the entity mention, and [Pi], [Pa], [Ps] are trainable tokens. These trainable tokens intu- itively allow the model to learn soft prompt, and were found to improve performance by Pan et al. (2022). The encoding o(s, m) is then taken to be the embedding of [MASK] in the final layer of the BERT model. The scoring functions fi take the form fi(x) = o (x Yt + by), where a is the sig- moid function, yi E IR" corresponds to a prototype of label and by C TR is a bias term. The parameters yi and by are initialised according to the correspond- ing weights in the decoder of the masked language model (where the average across all tokens is used if I consists of several tokens).
Label Embeddings Recent UFET models typi- cally rely on some kind of prior knowledge about the labels. As already mentioned, Xiong et al. (2019) rely on GloVe embeddings to construct label graph. Pan et al. (2022) initialise the scoring function for each label based on that label's repre- sentation in the decoder of the language model (see Section 3). Rather than using pre-trained embed- dings, Huang et al. (2022) encode the labels using a language model, which is fine-tuned together with the entity mention encoder. A similar approach was used by Ma et al. (2022) in the context of few-shot entity typing. In these approaches, the label embed- dings are directly used as label prototypes for the classification model, which could make it harder to distinguish between similar but mutually exclusive labels. In a broader context, label embeddings have also been used for few-shot text (Hou et al., 2020; Halder et al., 2020; Luo et al., 2021) and image (Xing et al., 2019; Yan et al., 2021, 2022) classi- fication. Clusters of label embeddings have been used in the context of extreme multi-label text clas- sification (Prabhu et al., 2018; Chang et al., 2020; Jiang et al., 2021). However, in this setting, clus- ters are used to make the problem tractable, rather than being a strategy for injecting prior knowledge.
Adding Domain Labels
We now describe our main strategy for improving UFET models based on pre-trained label embed- dings. Broadly speaking, the label embeddings provide us with prior knowledge about which la- bels are similar. Crucially, however, the fact that two labels have similar embeddings does not nec- essarily mean that the occurrence of these labels is positively correlated. Labels can be similar be- cause they denote categorics with similar meanings (e.g. student and learner), but similar labels can

---

also denote mutually exclusive categories from the same domain (c.g. student and teacher, or ambu- lance and fire truck). For this reason, we do not require that the scoring functions (or prototypes) for similar labels should always be similar. Instead, we propose a strategy based on clusters of labels.
Ls. If oj is predicted for given input (s, m), then seems reasonable assume that we should also predict at least one label from Li. Indeed, the mean- ing of o was precisely that one of the labels from Lj applies. If oj is predicted but none of the labels from L,, we therefore add the label from L; with the highest confidence score conf (1; s, m) to the set of predicted labels. For instance, consider again the cluster L, from (1). If e is predicted but none of the labels in L then this intuitively means that the model believes the entity is an emergency vehi- cle. but it has insufficient confidence in any of the individual types in L. However, its confidence in ambulance may still be higher than its confidence (1)
We use an off-the-shed clustering method to cluster labels based on their pre-trained embedding. Each cluster intuitively represents a domain. We associate a synthetic label with each of the domains and add these synthetic labels to the training exam- plcs, For instance, consider the following cluster:
L = (fire ffre engine, air ambulance, ambulance, police car}
in the other options, in which case we add ambu- lance to the set of predicted labels. Note that this is clearly recall-oriented strategy.
Whenever training example has any of these la- bels, we add "cluster i" as an additional label. This tells the model that examples about different types of emergency vehicles all have something in com- mon.3 If linear scoring function is used, as in DET, examples involving emergency vehi- cles are thus encouraged to be linearly separated from other examples. This helps the model to un- cover features that are common to emergeney vehi- cles, while not requiring that prototypes of different types of emergency vehicles be similar.
5.2 Identifying Conceptual Neighbourhood
As we already discussed, the labels in given clus- ter often correspond to mutually exclusive cate- gories from the same domain. We will refer to such labels as conceptual neighbours. For instance, the same entity cannot be both an ambulance and a police car, even though these entity types are simi- lar. Now suppose we have prior knowledge about which labels are conceptual neighbours. We can then implement a straightforward post-processing method: if the model predicts a label set which con- tains two conceptual neighbours, we should only keep one of them. In such a case, we simply discard the label in which the model was least confident.
5 Post-processing Predictions
Domains can also be used for improving the pre- dictions of an UFET model post hoc. We focus on two simple strategies that exclusively rely on prior knowledge about the label dependencies. In particular, we do not attempt to learn about label de- pendencies from the training data. We assume that dependencies between frequent labels are already modelled by the base UFET model, whereas for rare labels, relying on the training data may lead to overfitting. This also has the advantage that we can continue to treat the base UFET model as a black box. We only require that the base UFET model can provide us with confidence score (I; s, m) for any label E L and input (s, m)..
We now discuss how knowledge about concep- tual neighbourhood can be learned. Bouraoui et al. (2020) proposed method for predicting whether two BabeiNet (Navigli and Ponzetto, 2010) con- cepts are conceptual neighbours. Their method can only be used for concepts whose instances are named entities (for which we have a pre-trained embedding). Hence we cannot use their strategy directly. However, using their method, we obtained a list of 5521 positive examples (i.e. concept pairs from BabelNet which are believed to be concep- tual neighbours) and 2318 negative examples. We then trained a classifier to detect conceptual neigh- bourhood. We started from Natural Language Inference (NLI) model that was pre-trained on the WANLI dataset (Liu et al., 2022), initialised from
5.1 Inferring Missing Labels
The use of domain labels allows us improve the predictions of the model through a simple post- processing heuristic. Let C,,.., Lg be the different domains, and let us write 6; for the synthetic do- main label that was introduced to describe cluster
FFrrmally, conceptual neighbours are concepts that are represented by adjacent regions in a conceptual space (Gär- denfors, 2000). In this paper, however, we treat the notion of conceptual neighhourhond in a more informal fashion.
'Note that the label "emergency vehicle" is not included in the vocabulary of the UFET datasct from (Choi et al., 201 8).

---

RoBERTa-large (Liu et al., 2019), and fine- tuned this model using the training examples from BabelNet. Specifically, for concept pair (21 we use "The category is l1" as the premise and "The category is 122 as the hypothesis. We train the model to predict contradiction for positive exam- ples and neutral for negative examples (as l and l2 are always co oo-hyponymm in the training examples).
Method
LM
R
F1
Box4Types"
52.8
38.8
44.8
LRN"
Bb
54.5
38.9
45,4
MLMET
Bb
53.6
45.3
49.1
DenoiseFET
Bh
55.6
44.7
49.5
UNIST"
Rb
49.2
49.4
49.3
UNIST"
50.2
49.6
49.9
NPCRF
52.1
47.5
49.7
NPCRF
55.3
46.7
50.6
LITE"
RIMNLI
48.9
To test the effectiveness of the conceptual neigh- bourhood classifier, we carried out an experiment where 20% of the BabelNet training data was held out. After training the Conceptual Neighbourhood classifier on the remaining 80% of the training data, we achieved an accuracy of 97.2% on the held-out fragment, suggesting that the considered strategy is indeed highly effective.
DenoiseFET
Bb
52.6
46.2
49.2
DenoiseFET
52.6
47.5
49.8
DenoiseFET
Rb
52.3
46.0
49.0
DenoiseFET
52.9
47.4
50.0
DenoiseFET + PKL
Bb
52.7
49.2
50.9
DenoiscFET PK1.
B1
53.8
50.2
51.9
DenoiseFET + PKL
Rb
53.1
48.5
50.7
DenoiscFET PKI.
RI
53.7
50.1
51.8
Table Results for Ultra-Fine Entity Typing (UFET), in terms of macro-averaged precision, recall and FI. Results with are taken from the original papers. The LM encoders BERT-base (Bb), BERT-large (BI), RoBERTa- (Rb), RoBERTa-large (RI) and RoBERTA-larg fine-tuned on MNLI (RIMNLI).
Once the NLI model is fine-tuned, use it to identify conceptual neighbourhood among the la- bels in C. To this end, test every pair of labels from the same domain. The label pairs that are classified as positive examples are then used for im- plementing the considered post-processing strategy. Note that this is precision-oriented strategy.
(2022), which we will refer to ComBiEnc. The former a contrastively fine-tuned BERT-base en- coder, which maps any given word onto a static vector. The latter is BERT-base encoder that was fine-tuned to predict commonsense properties. Fi- nally, we include two approaches which learn word embeddings by averaging the contextualised rep- resentations of different mentions: the AvgMASK method from (Li et al., 2021b) and the ConCN method from (Li et al., 2023). The former obtains the representation of word w by finding up 500 mentions of that word in Wikipedia. These mentions are masked, and the contextualised repre- sentations of the MASK token are obtained using RoBERTa-lagge. The resulting vectors are finally averaged to obtain the embedding of w. ConCN fol- lows similar strategy, but instead of a pre-trained RoBERTa model, they a RoBERTa- model that was contrastively fine-tuned using distant su- pervision from ConceptNet. We use this latter model as our default choice for the experiments in Section 6.2. A comparison with the other em- beddings presented in Section 6.3.
6 Experimental Results
We evaluate approach the ultra-fine entity typing (UFET) benchmark from Choi al. (2018). Although we expect our strategies be most effec- tive for ultra-fine entity types, we also carry out an evaluation on two standard fine-grained entity typ- ing benchmarks: OntoNotes (Gillick et al., 2014) and FIGER (Ling and Weld, 2012).
6.1 Experimental Set-up
Pre-trained Label Embeddings We consider several pre-trained label embeddings. First, we in- clude number of standard word embedding mod- els: Skip-gram² (Mikolov al., 2013), GloVe (Pen- nington al., 2014),, SynGCN (Vashishth al., 2019). Word2Sense (Panigrahi et al., 2019), and ConceptNet Numberbatch (Speer et al., 2017). Be- yond traditional word embeddings, we also include two models that distil static word embeddings from BERT-based language models: MirrorBERT (Liu et al., 2021a) and the model from Gajbhiye al.
$http://nlp.co washington. wawhington..edu/eentity iodu/enniiy_tyype htta/ultraffne_acl11 aclis.tar...... tar.
"https://www.co cs. jhu. edu/-s. lafaan/ddtt//igeee wiki.zip
Training Details For the conceptual neighbour- hood classifier, we tune the confidence level above which we consider two labels to be conceptual neighbours using the development set. We use Affinity Propagation (AP) as the clustering algo-
We used model that was trained on Google News (https: htt::://ooeegooeoooe
wttpaahiiie//////eveee/e/// "Wo used the model that was trained on Common Crawl (https: //nlp. stanford ordd

---

Method
LM
OntoNotes
FIGER
Method
F1
DenoiseFET + DL missing CN
52.7
49.2
50.9
DenoiseFET + DL missing
$2.9
48.5
50.6
Box4Types
B1
77.3
70.9
79.4
75.0
DenoiseFET + DL
52.5
48.4
50.4
LRN"
Bb
84.5
79.3
DenoiseFFT I.I.R
53.0
46.1
49.4
MLMET
Bb
85.4
80.4
DenoiseFET
52.6
46.2
49.2
DenoiseFET
Bb
87.1
81.5
NPCRF
Bb
85.2
80.0
NPCRF"
RI
86.0
81.9
Table 3: Ablation analysis on UFET for different vari-
DSAM
LSTM
83.1
78.2
83.3
81.5
ants of our model, using BERT-base for the entity coder.
SEPREM
RI
86.1
82.1
RIB"
ELMo
84.5
79.2
87.7
84.4
LITE
RIMALI
86.4
80.9
86.7
83.3
Embedding
R
Fl
DET
Bb
87.2
81.4
86.2
82.8
DenoiseFET
BI
87.5
81.6
86.5
82.9
Skip-gram
53.9
44.8
49.0
DenoiseFET
Rb
87.4
81.5
86.4
82.9
Glo
53.5
45,9
49.4
DE
RI
87.6
81.8
86.7
83.0
Numberbatch
53.5
46,9
50.0
Word2Sense
53.7
45.5
49.3
DenoiseFET + PKL
Bb
87.7
81.9
86.8
82.9
SynGCN
54.6
44.3
48,9
DenoiseFET + PKL
BI
87.9
82.1
87.0
83.1
MirrorBERT
51.7
46.9
49.2
DenoiseFET + PKL Rb
87.8
82.2
86.9
83.0
ComBiEnc
53.2
47.9
50.4
DenoiseFET + PKL RI
87.9
82.3
87.1
83.1
AvgMASK
53.7
46.9
50.1
ConCN 52.7
49.2
50.9
Table 2:
Results for fine- Ainegrained entity typing, in terms of macro-FI and micro-F1. Results with are taken from the original papers. The LM encoders are BERT-base (Bb), BERT-large (BI), RoBERTa-baa (Rb), RoBERTa-large (RI) and RoBERTA-large fine-tuned on MNLI (RIMNLI).
Table 4: Performance of different pre-trained label em- beddings UFET, using DenoiseFET with BERT-base as the base UFET model.
model used for initialising the entity encoder. As can be seen, our approach outperforms all base- lines. This is even the case when using BERT-base or RoBERTa-base for the entity encoder, although the best results are obtained with BERT-large.
rithm. Affinity Propagation does not require us to specify the number of clusters, but instead re- quires so-called preference value to be speci- fied. Rather than tuning this preference value, obtain clusters for each of the following values: (0.5, 0.6, 0.7, 0.9, We use these clusters together. This means that our method essentially uses domains at multiple levels of granularity. We use DeT (Pan et al., 2022) as the base entity typing model. For UFET and OntoNotes, we also rely their denoised training set?, For FIGER, use the standard training set.
The results for fine-grained entity typing are pre- sented in Table 2. Following the tradition for these benchmarks, we report both macro-averaged and micro-averaged F1. In addition to the baselines from Table (for which results for OntoNotes FIGER are available), we also include results for the following fine-grained entity typing models: DSAM (Hu et al., 2021), RIB (Li et al., 2021a) and SEPREM (Xu et al., 2021). For both OntoNotes and FIGER, our model consistently outperforms the base DenoiscFET model. While the improve- ments smaller than for UFET, this is nonethe- less remarkable, given that our strategies are tar- geted ultra-fine entity types. For OntoNotes, our model outperforms baselines. In the case of FIGER, its performance is comparable to LITE (i.e. slightly better macro-F1 but worse micro-FI) and outperformed by RIB. The latter model uses a GNN based strategy to learn correlations between labels and keywords from the training data.
6.2 Results
We refer to our proposed strategy PKL (Prior Knowledge about Labels). The results for UFET are presented Table 1. Following the original pa- per, we report macro-averaged precision, recall and F1. We compare our model with Box4Types (Onoe et al., 2021), I), LRN (Liu et al., 2021b), MLMET (Dai et al., 2021), DenoiseFET (Pan et al., 2022). UNIST (Huang et al., 2022), NPCRF (Jiang et al., 2022) and LITE (Li et al., 2022). The baseline results were obtained from the original papers. For each method, we also mention which pre-trained
6.3 Analysis
We now present some additional analysis, to better understand which components of the model are
AAailabbe from https://githin https://gituub.commccIII cttpcccIcc/ccc////////cccccc DenoiseFET.

---

Input
Predictions
Jazz coach Jerry Sloan, ungry the officiating, had two technicals called against him and ejected from the game in the closing minutes.
person, coach, trainer, sthlete, player
Queen Mary, the Queen Mother, came see and after the performance asked Cotes where found "those wonderful actors."
play; performance, show, event, opera
Eurostat said output fell euro nations where had data, with Finland booking sharpest drop at 23.2 per cent
organization, institution, corporation agency, administration, compaаy,, business, frm
Table 5: Examples of differences between the predictions of the base model and those of the full model, using BERT-base. Labels in green are only predicted by the full model: labels red are only predicted by the base model.
Input
Predictions
Gold standard
Yes although regular coffee drinking isn't harmful for most people, that might not hold true for pregnant
person, adult, female, patient, woman
person, adult, female,
They, the other children, are listed miss- ing.
person, child, son
person, child
During the American Revolutionary War he served as secretary to Admiral Molyneux Shuld- ham. in Boston in 1776 and again at Plymouth (1777-78).
event, conflict, battle, fight, war
time, event, conflict, bautl, era. struggle, war
Matlock almost immediately formed his own band, Rich Kids, with Midge Ure, Steve New, and Rusty Egan.
group, organization, musician, band
group, musician, band
As the War of the Fourth Coalition broke out in September 1806, Emperor Napoleon took his Grande Armee into the heart of Germany in a memorable campuign against Prussia.
event, conflict, battle, dispute, fight, struggle, war
event, conflict, battle, dispute, fight, struggle, war, competitive- ness, group-action
During the Second World War, Ailsa built ves- sels for the Navy, including several Bangor class minesweepers.
object, ship, hoat, craft, vessel
object, ship, hoat, thing, vessel
"If you get your pitch, and take a good swing, anything can happen," he later remarked.
person, athlete, ceach, player
person, coach, player, trainer
Table 6: Error analysis, showing cases where the full model incorrectly added removed label (compared to the base model), using BERT-baas.. Labels in green are only predicted by the full model; labels in red are only predicted by the base model.
most responsible for its outperformance.
we use LLE is provided in the appendix. While adding LLE to the base UFET model has a small positive effect, we can see that the performance is much worse than that of our proposed clustering based strategy (DL). The post-processing tech- niques (missing and CN) both have positive effect on the overall performance, although they have a smaller impact than the DL strategy.
Ablation Study In Table 3, we report the results that obtained when only using some of the com- ponents of our strategy. We use DL refer to the use of domain labels (Section 4), missing to refer to post-processing based on domain labels (Sec- tion 5. and CN for the post-processing based on conceptual neighbourhood (Section 5.2). We also include a variant that uses Local Linear Embed- dings (LLE) as an alternative strategy for taking into account the pre-trained label embeddings. LLE acts a regularisation term in the loss function, which essentially imposes the condition that proto- types with similar labels should themselves also be similar. As we have argued in Section 4, this is not always appropriate. A detailed description of how
Label Embedding Comparison In Table 4, we compare the performance of different pre-trained label embeddings. In all cases, use our full model with BERT-base for the entity encoder. The results show that the choice of word embedding model has a substantial impact on the result, with ConCN clearly outperforming the traditional word embedding models. Overall, we might expect that

---

Label
Conceptual Neighbours
Domains
hip
knee, thigh, ankle, shoulder-joint intermediate-schooo middle-school, junior-school, junior-high stomach-cancer, train-eancer, lung-cancer
(grape, red wine, rice wine, white wine, wine, cham-
high-school
pagne, ice wine, port wine]
liver-cancor
GloVe
[bag, luggage, sack, plastic bag. purse, shoulder bag)
maple pink
cherrywood, oak, cedar purple, blue, red, yellow monthly, yearly, weekly silver-fox, duck, sheep throat, nose, ear ringgit, yen, yuan bua,bear, bear-cat, cave-bear monster-truck, wagion, estate-car white-wine, tahle-wine, ice-wine third-baseman, first-baseman, left-fielder, right-fielder
(reception, desk, help desk, lobby, reception desk) )
quarterly
(hnn, hanning, moratorium, prohibition) )
rahbit
[knee, leg, ankle. elbow. hamstring, knee cap. thigh] [outlook, perspective, view, viewing, vista) [image, photo, photograph, picture, portrait)
mouth
baht
native-bear pickup-truck
Numberbatch
red-wine
{digit, finger, hand, thumb, toe, index finger, ring finger)
second-buseman
(approval, licence, license, permission, permit, autho- rization, consent, planning permission)
summer-camp
camp. work-camp, labour-camp
bar-chart dinner-jacket
pie-chart, box-plot, heat-map
evening dress, morning dress, dress
|artificial intelligence, computer vision, expert sys- tem, machine lcarning, speech recognition, voice recognition]
Table 7: Examples of conceptual neighbours for some labels from the UFET benchmark.
{permit, residence, residence permit, work permit)
[century, decade, era, period, generations, millen- nium)
MirrorBERT
word embeddings distilled from language models capture word meaning in a richer way. From this perspective, the strong performance of Number- batch also stands out. Interestingly, ConCN, Com- BilEne and Numberbatch all rely on ConceptNet, which suggests that having label embeddings that capture commonsense properties might be impor- tant for achieving the best results.
Janalysis, assessment, evaluation, measurement, ap- praisal|
(career criminal, hate crime, serial killer, war crime, war criminal, capital crime
ddoctor, medical, putient, physician, psychiatrist, sur- geon]
(social, social media, social network, social network- ing, twitter, facebook, gongle, social life, video chat, yahoo, youtube
Qualitative Analysis Table shows some exam-
ConCN
[change, transformation, transition, sea change, sex change, step change, transform, alteration
ples where the post-processing strategies improve the predictions of the base model (DenoiseFET)) On the one hand, our method results in a num-
[acre, inch, square. square foot, square inch, square meter, square mile)
(client, eustomer, customer base, clientele)
ber of additional labels being predicted. On the other hand, the conceptual neighbourhood strat- egy removes some incompatible labels. In the first example, athlete and player are removed because they are conceptual neighbours of coach; in the second example, opera is removed because it is a conceptual neighbour of play; in the third example, company, business and firm are removed because they conceptual neighbours of corporation.
[beef, flesh, meat, pork, ground beef, lamb, mystery meat, red meat)
Table 8: Examples of the domains that were found with different word embeddings.
was (incorrectly) predicted to be conceptual neighbour of conftict. In the following example, vessel was incorrectly assumed to be a conceptual neighbour of ship. In the final example, coach was removed as a conceptual neighbour of player. Un- like the previous two cases, it is less clear whether this is actually a mistake. Indeed, it is highly un- likely that the person who is referred to is both a coach and player. However, the sentence leaves ambiguous which of these labels applies. In gen- eral, for DenoiseFET with BERT-large on UFET, the missing strategy from Section 5.1 was applied to 744 out of 1998 test instances. A total of 879
Table shows a number of examples where the full model has introduced a mistake, compared to the base model. In the top four examples, the full model predicted an additional label that was not included in the ground truth. As can be seen, the added labels are somewhat semantically close, es- pecially in the last two cases, where it could be argued that the labels are actually valid. In the bot- tom three examples, the conceptual neighbourhood strategy incorrectly removed a label. In particular, in the first of these, dispute was removed because

---

labels were added, and 513 of these were correct. The conceptual neighbourhood strategy affected 504 instances. A total of 983 labels were removed, and 709 of these deletions were correct.
Limitations
Using conceptual neighbourhood for post- processing the predictions of an entity typing model is straightforward and effective, but there are other important types of label dependencies which we are currently not considering. Entailment relations, for instance, could be learned in similar way how learn conceptual neighbourhood. In some cases, it could also be useful to combine the strategies we have proposed in this paper with models that Iearn label correlations from the training data itself, especially for fine-gu fine-grained (rather than ultra-fine grained) entity types, However, as we have argued in the paper, this would come with the risk of overfitting the training data. The label domains that currently use are essentially taxonomic categories, i.c. sets of labels that have similar meanings (e.g. student and teacher). We may expect that incorporating thematic groupings of labels could further improve the results (e.g. where student would be grouped with other university related terms such as lecture hall).
Table 7 lists some examples of conceptual neigh- bours that were predicted by our classifier (for the UFET label set). As can be seen, in most cases, the conceptual neighbours that were found indeed also refer to disjoint concepts. However, summer see small number of false positives (e.g. camp and holiday camp).
Finally, Table 8 shows examples of domains that were found. As can be seen, most of the domains are meaningful. However, the case of Mirror- BERT, we also find clusters containing terms with rather different meanings, such as permit and res- idence in the first example. Often such terms are thematically related but taxonomically distinct. For instance, in the last cluster, we find two sub-types of person, namely career criminal and serial killer, together with sub-types of crime. Treating these la- bels as being part of the same domain may confuse the model, as they apply to very different types of entities. For GloVe, we can similarly see words which are thematically related but taxonomically different. This is illustrated in the first example, where grape is included in cluster about wines.
This work was supported by EPSRC grant EP/V025961/1, ANR-22-CE23-0002 ERIANA and by HPC resources from GENCI- IDRIS (Grant 2023-[AD01 1010333113311 Na Li is supported by Shanghai Big Data Management System Engineering Research Center Open Fund- ing.
Acknowledgments
7 Conclusions
We have analysed the effectiveness of two types of external knowledge for ultra-fine entity typing: pre- trained label embeddings and a conceptual neigh- bourhood classifier. The strategies that we consid- ered are straightforward to implement and can be added to most existing entity typing models. One of the most effective strategies simply requires adding synthetic labels, corresponding to clusters of se- mantic types, to the training examples. We also discussed two post-processing strategies which can further improve the results. Despite their simplic- ity, the proposed strategies were found to consis- tently improve the performance of the base UFET model. The nature of the pre-trained label embed- dings played an important role in the performance of the overall method. An interesting avenue for future work would be to design strategies for identi- fying label domains which are specifically designed for the UFET setting.
References
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representa- tions via Reductions to Static Embeddings. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4758- 4781, Online. Association for Computational Lin- guistics.
Zied Bouraoui, José Camacho-Collados, Luis Espinosa Anke, and Steven Schockaert. 2020. Modelling se- mantic categories using conceptual neighhorhood. In Proceedings AAAI, pages 7448-7455.
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yim- ing Yang, and Inderjit S. Dhillon. 2020. Taming pretrained transformers for extreme multi-label text classification. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, Virtual Event, CA. USA, August 23-27, 2020. pages 3163-3171. ACM.

---

Tongfei Chen, Yummo Chen, and Benjamin Van Durme, 2020. Hierarchical entity typing via multi-level leamn- ing to rank. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages -465-8475, Online. Association for Computa- tional Linguistics.
Marti A. Hearst. 1992. Automatic acquisition of hy- ponyms from large text corpora. In COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics.
Yutai Hou, Wanxiang Che. Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task task-adaptive projection network. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1381- 1393, Online. Association for Computational Linguis- tics.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 87-96, Melbourne, Australia, Association for Computational Linguistics.
Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021. Ultra-fine entity typing with weak supervision from a masked language model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Itth International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1999, Online. Association for Computational I Linguistics.
Yanfeng Hu, Xue Qiao, Luo Xing, and Chen Peng. 2021. Diversified semantic attention model for fine-grained entity typing. IEEE Access, 9.
James Y. Huang, Bangzheng Li. Jiashu Xu, and Muhao Chen. 2022. Unified semantic typing with mean- ingful lahel inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642-2654, Seattle, United States. Association for Computational Lin- guistics.
Jacob Devlin, Ving-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chengyue Jiang, Yong Jiang, Weiqi Wu, Pengjun Xie, and Kewei Tu. 2022. Modeling label correlations for ultra-fine entity typing with neural pairwise condi- tional random field. CoRR, abs/2212.0155
Ting Jiang, Deging Wang, Leilei Sun, Huayi Yang. Zhengyang Zhao, and Fuzhen Zhuang. 2021. Lightxml: Transformer with dynamic negative sam- pling for high-pen ser--perforaece extreme multi-label text classification. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Con- ference on Innovative Applications of Artificial In- telligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, FAAI 2021, Virtual Event, February 2-9, 2021, pages 7987- 7994. AAAI Press.
Amit Gajbhiye, Luis Espinosa-Anke and Steven Schockaert. 2022. Modelling commonsense proper- ties using pre-trained bi-encoders. Proceedings of the 29th International Conference on Computational Linguistics. pages 3971-3983, Gyeongju, Republic of Korea, International Committce on Computational Linguistics.
Peter Gärdenfors. 2000. Conceptual spaces- the geom- etry of thought. MIT Press.
Thomas N. Kipf and Max Welling, 2017. Semi- supervised classification with graph convolutional networks. In 5th International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Dan Gillick, Nevena Lazic. Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context- dependent fine-grate grained entity type tagging. CoRR, abs/1412. 1820.
Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020. Task-aware representation sen- tences for generic text classification. In Procedd- ings of the 28th International Conference Com- putational Linguisties, pages 3202-3213, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Bangzheng Li, Wempeng Yin, and Muhao Chen. 2022. Ultra-fine entity typing with indirect supervision from natural language inference. Transactions of the Association for Computational Linguistics, 10:607- 622.
Jinging Li. Xiaojun Chen, Dakui Wang, and Yuwei Li. 2021a. Enhancing label representations with rela- tional inductive bias constraint for fine-graine en- tity typing. In Proceedings of the Thirtieth Inter- national Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal. Canada, 19-27 August 2021, pages 3843-3849. ijcai.org.
Souleiman Hasan and Edward Curry. 2017. Word re- embedding via manifold dimensionality retention. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 321-326. Copenhagen, Denmark. Association for Computational Linguistics.

---

Na Li, Zied Bouraoui, José Camacho-Collados Luis Es- pinosa Anke, Qing Gu, and Steven Schockaert. 10. Modelling general properties of nouns by selectively averaging contextualised embeddings. In Proceedings of the Thirtieth International Joint Con- ference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3850-3855 ijeai.org.
word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Shikhar Murty, Patrick Verga, Luke Vilnis, Irena Radovanovic, and Andrew McCallum. 2018. Hi- erarchical losses and new resources for fine-grained entity typing and linking. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 97-109, Melbourne, Australia, Association for Computational Linguistics.
Na Li. Hanane Kteich, Zied Bouraoui, and Steven Schockaert. 2023. Distilling semantic concept em- beddings from contrastively fine-tuned language models.
Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In Proceedings of the Twenty- Sixth AAAI Conference on Artificial Intelligence July 22-26, 2012, Toronto, Ontario, Canada. AAAI Press.
Roberto Navigli and Simone Paolo Ponzetto. 2010. Ba- belNet: Building a very large multilingual semantic network. In Proceedings of the 48th Annual Meet- ing of the Association for Computational Linguistics, pages 216-225, Uppsala, Sweden. Association for Computational Linguistics.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: worker and Al collabora- tion for natural language inference dataset creation. CoRR, A5s52201.05955.
Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. 2021. Modeling fine-grained entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Pupers), pages 2051-2064 Online. As- sociation for Computational Linguistics.
Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier: 20 Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442-1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise listantly-labelle data for entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2407-2417, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. 2021b. Fine-grained entity typing via label reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4611-4622, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Weiran Pan, Wei Wei, and Feida Zhu. 2022. Automatic noisy label correction for fine-grained entity typing. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4317-4323. ijeai.org.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692.
Abhishek Panigrahi, Harsha Vardhan Simhadri, and Chi- ranjib Bhattacharyya. 2019. Word2Sense: Sparse interpretable word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages
Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, and Wei Zhang. 2021. Don't 't miss the labels: Label-semantic augmented meta-learnen for few-shot text classifica- tion. In Findings of the Association for Computa- tional Linguistics: ACI-IJCNLP 2021, pages 2773- 2782, Online. Association for Computational Lin- guistics.
Florence. Italy. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Jie Ma. Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022. Label semantics for few shot named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1956- 1971, Dublin, Ireland. Association for Computational Linguistics.
Yashoteja Prabbu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Par- titioned label trees for extreme classification with application to dynamic search advertising. In Pro- ceedings of the 2018 World Wide Web Conference on
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space

---

World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018. pages 993-1002. ACM.
Zenan Xu, Daya Guo, Duyu Tang, Qinliang Su, Linjun Shou, Ming Gong, Wanjun Zhong. Xiaojun Quan, Daxin Jiang. and Nan Duan. 2021. Syntax-enhanced pre-trained model. In Proceedings the 59th Anmual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 5412-5422, Online. Association for Computational Linguistics.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016. AFET: Automatic fine- grained entity typing by hierarchical partial-label em- bedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 1369-1378, Austin, Texas. Association for Computational Linguistics.
Kun Yan, Zied Bouraoui, Ping Wang, Shoaib Jameel, and Steven Schockaert. 2021. Aligning visual proto- types with BERT embeddings for few-shot learning, In ICMR '21: International Conference on Multime- dia Retrieval, Taipei, Taiwan, August 21-24, 2021, pages 367-375. ACM.
Sam T. Roweis and Lawrence K. Saul. 2000. Nonlinear dimensionality reduction by locally linear embedding. Science, 29065500):333-23226
Sonse Shimaoka, Pontus Stenetorp. Kentaro Inui, and Sebastian Riedel. 2017. Neural architectures for fine- grained entity type classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume
Kum Yan. Chenbin Zhang, Jun Hou, Ping Wang, Zied Bouraoui, Shoaib Jameel, and Steven Schockaert. 2022. Inferring prototypes for multi-label few-shot image classification with word vector guided atten- tion. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022. The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Vir- rual Event, February 22 March 2022, pages 2991- 2999. AAAI Press.
1, Long Papers, pages 127 1-180, Valencia, Spain. Association for Computational Linguisties.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444 4451. AAAI Press.
A Locally Linear Embedding
Shikhar Vashishth, Manik Bhandari. Prateck Yadav, Piyush Rai, Chiranjib Bhattacharyya, and Partha Talukdar. 2019. Incorporating syntactic and semantic information in word embeddings using graph convo- lutional networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 3308-3318. Florence, Italy, Associa- tion for Computational Linguistics.
Locally Linear Embedding (LLE) (Roweis and Saul, 2000) is well-known dimensionality re- duction technique, which aims to preserve linear structure in the neighbourhood of each data point. Rather than using LLE for dimensionality reduc- tion, we use it for regularising the space of the label prototypes. This is somewhat reminiscent of how LLE has been used in NLP for transforming word embeddings (Hasan and Curry, 2017).
Chen Xing, Negar Rostamzadeh, Boris N. Oreshkin, and Pedro O. Pinheiro. 2019. Adaptive cross-modal few-shot learning. In Proceedings of the Annual Con- ference on Neural Information Processing Systems, pages 4858-
Let us write N₁ for the k nearest labels to I C L, in terms of their cosine similarity in the pre- trained embedding space. We can approximate the embedding 1 as linear combination of the embeddings of the labels in Nr, by solving the following optimisation problem:
Wenhan Xiong, Jiawei WU, Deren Lei, Mo Yu, Shiyu Chang. Xiaoxiao Guo, and William Yang Wang. 2019. Imposing lebel-nelational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Numan Language Technologies, Volume 1 (Long and Short Papers), pages 773-784, Minneapolis, Min- nesota. Association for Computational Linguistics.
Minimise
(1-
WiPD
2
s.t. Σ uip =
PEN
PEN,
Then we add the following regularisation term to the loss function of the entity typing model:
Peng Xu and Denilson Barbosa. 2018. Neural fine- grained entity type classification with hicrarchy- aware loss. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 16-25, New Orleans, Louisiana. Association for Computational Linguistics.
LLE =
Σ(yr yi Σ wwyN)) ICL
PEN,
In other words, we encourage the model to preserve the linear dependence that holds between / and

---

its neighbours in the pre-trained embedding space. The overall loss then becomes:
L - Centity + ALLLE
where Lentity is the loss function from the entity typing model and À 0 is syper-parameter to control the strength of the regularisation term.


Reviews:
[{"review_id": "raQRoRTDnY", "reviewer": "Reviewer_QZ94", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review.\n\nFor ultra-fine entity typing, the proposed approach improves the corresponding base models by around 2 percentage points (e.g. from 49.8 to 51.9 in the case of BERT-large). While this may appear like a small improvement, it represents a more substantial gain than what has been reported in other recent work in this area. We believe this is an important finding, especially given how easy it is to apply our proposed methods.\n\nMoreover, when we focus on the finest-grained entity types (i.e. the ones designated as ultra-fine in the UFET dataset), the improvement is larger. If we only consider such types, the F1 score improves from 34.6 for the base model to 38.5 for our model (when using BERT-base as the encoder). We will add an analysis focusing only on ultra-fine types to the paper to make this clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed suite of method effectively injects prior knowledge regarding fine-grained types into the entity-typing process. Because of the pseudo-labels, low-occurring types can be better covered during both training and post-hoc inference. ", "reasons_to_reject": "1) The improvement is marginal and incremental, and the paper does not touch or discuss the core problem that leads to the low-performances of the UFET dataset. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a fine-grained entity typing system that adds pseudo-labels representing a cluster of actual labels to the training and inference process. This way, the recall can be improved, especially on the low-occurring types. The authors propose two post-hoc processing constraints associated with the pseudo-labels: 1) if a pseudo-label is predicted, at least one actual type from of the cluster represented by the pseudo-label should be predicted; 2) if an actual type is predicted, then none of its mutually-exclusive types should be predicted. Such mutually exclusive types are called conceptual neighbors and are learned with a contrastive NLI model. \n\nExperiments show that the proposed suite of methods that aim to inject prior knowledge regarding each type into the process collectively improve the baseline model's performances on the UFET dataset and marginally on two other fine-grained entity typing benchmarks. Ablation studies also support the claim that each technique is helpful in the overall performance. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VDSqsdCULN", "reviewer": "Reviewer_AV7U", "summary": "", "questions": "1. What is the intuition behind special tokens P1, P2 and P3? (236–237)\n2. What's a prototype of a label? y_l in lines 241, 242\n3. It is not clear how the clustering method ensures each cluster consists of mutually exclusive categories (322--324). How accurate is the clustering method?\n4. While predicting conceptual neighborhoods using the NLI model, how are conflicts resolved in pairwise decisions? For example, transitivity might not hold between the three labels, fire truck, fire engine, and police car.\n5. How accurate is the NLI model used for conceptual neighbor prediction? both on the BabelNet test set and the domains from the UFET dataset.\n6. Add a short explanation for the local linear neighborhood (474--476).\n7. How to compute embeddings for cluster labels?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review.\n\nAs suggested by the reviewer, we will add an analysis that focuses on ultra-fine entity types in particular. For instance, when using BERT-base as the encoder, if we only consider the ultra-fine types in the UFET benchmark, the performance increases from 34.6 for the base model to 38.5 for our model. \n\nThe reviewer suggests comparing against prior work on modeling label dependencies. In fact, such a comparison is already made in the paper. For instance, NPCRF uses neural conditional random fields to model label dependencies, while LRN and RIB use graph neural networks for this purpose.\n\nQ1: The special tokens are supposed to represent a learnable soft prompt. Including such tokens was found to improve performance in previous work, and is now a standard approach in work on UFET.\n\nQ2: We use the term “prototype” to refer to the vector y_l, since the label l is predicted for an embedding x based on the similarity between x and y_l.\n\nQ3: We do not guarantee that the labels in a cluster are mutually exclusive. We simply make the empirical observation that this is often the case. Note that we use the conceptual neighbourhood classifier to verify when this is actually the case (i.e. if the clustering algorithm were to guarantee mutual exclusivity then we would not actually need the conceptual neighbourhood classifier).\n\nQ4: We start by removing the label with the lowest confidence (which is involved in at least one conflict). This process is repeated until all conflicts are resolved.\n\nQ5: We have now carried out an experiment where 20% of the BabelNet training data was held out. After training the Conceptual Neighbourhood classifier on the remaining 80% of the training data, we achieved an accuracy of 97.2% on the held-out fragment. This confirms the insights from our qualitative analysis, which also suggested that the conceptual neighbourhood classifier is very accurate. We will add this analysis to the paper.\n\nQ6: Note that LLE is explained in Appendix A, while the intuition behind LLE is explained on lines 476-480.\n\nQ7: Note that we only need embeddings of cluster labels for the LLE baseline. In that case, we simply represent a cluster by averaging the embeddings of the labels that belong to that cluster.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-motivated. They make progress toward capturing semantic relatedness among the labels from the large label space of UFET task. This shows a resemblance to hierarchical typing and can provide useful insights for future work on this task.\n2. The ablation studies highlight the effectivness of each proposed module, clustering labels, post-processing to detect missing and conflicting labels from conceptual neighborhoods.", "reasons_to_reject": "1. In its current form, it's hard to follow the methods described in the paper. The paper omits a number of details that are necessary to understand the paper (see questions below).\n2. The paper suggests the effectiveness of proposed methods to the long-tail of entity types. However, it doesn't include any empirical results to showcase this abilities. As I understand it, the baselines already work very well for high-resource types, so it would be important to see how the use of cluster labels benefits the long-tail.\n3. The paper can also benefit from a comparison against prior work on modeling label dependencies. The related work section briefly highlights this, however there is not empirical comparison on the label dependencies captured in prior work and current work.", "questions_to_authors": "1. What is the intuition behind special tokens P1, P2 and P3? (236–237)\n2. What's a prototype of a label? y_l in lines 241, 242\n3. It is not clear how the clustering method ensures each cluster consists of mutually exclusive categories (322--324). How accurate is the clustering method?\n4. While predicting conceptual neighborhoods using the NLI model, how are conflicts resolved in pairwise decisions? For example, transitivity might not hold between the three labels, fire truck, fire engine, and police car.\n5. How accurate is the NLI model used for conceptual neighbor prediction? both on the BabelNet test set and the domains from the UFET dataset.\n6. Add a short explanation for the local linear neighborhood (474--476).\n7. How to compute embeddings for cluster labels?\n", "paper_topic_and_main_contributions": "This paper studies the task of ultra-fine entity typing (UFET). They propose a methodology that takes advantage of the similarities of labels through conceptual neighborhoods. The propose to cluster the labels into domains that constitute of dissimilar but related labels, eg., fire truck, fire engine, air ambulance, ambulance and police car. They include the cluster labels (aka domains) to the existing label space and present a post-hoc filter of labels from a black-box UFET model.\n\nThe paper outperforms the state-of-the-art methods on the UFET dataset from Choi et al., 2018. It also includes a qualitative analysis of the utility of proposed cluster labels.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8kpHExfhrO", "reviewer": "Reviewer_h56F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review.\n\nPlease note our response to the first reviewer (QZ94) regarding the point about the improvement being modest.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method contributes to performance improvement (Tables 1 and 2).\n- The ablation study shows to what extent each element of the proposed method contributes to performance improvement.\n", "reasons_to_reject": "- While the proposed method improves the performance, the improvement is modest (about 1.0 F1 score).", "questions_to_authors": "", "paper_topic_and_main_contributions": "What this paper about? \n- This paper addresses Ultra-Fine Entity Typing (UFET). \n\nWhat problem or question did this paper address?\n- The problems is that the number of training examples remains prohibitively small for many of the types (Line 62-64).\n\nWhat are the main contributions that this paper makes towards a solution or answer?\n- The authors use pre-trained label embeddings to group the given set of type labels into semantic domains. (Line 121-123).\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a fine-grained entity typing system using pseudo-labels to enhance recall for less frequent types. Post-hoc constraints are introduced for these labels to ensure prediction of at least one actual type from the cluster and avoid prediction of mutually-exclusive types. These exclusive types, termed conceptual neighbors, are learned using a contrastive NLI model. Enhancements in the paper's presentation are warranted. The experimental results should explicitly demonstrate the system's superior performance, particularly emphasizing its effectiveness with respect to long-tail types.
Decision: Accept-Findings