Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings (2024)

Abstract: Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e. sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g. the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.

heading: 1 Introduction
text: Many knowledge engineering tasks require knowledge about the meaning of concepts. As a motivating example, let us consider the problem of ontology expansion, which consists in uncovering properties of, and relationships between concepts, given the names of these concepts and an initial knowledge base. Despite the popularity of Large Language Models (LLMs), the use of concept embeddings remains attractive in such settings (Vedula et al., 2018; Li et al., 2019; Malandri et al., 2021; Shi et al., 2023). Indeed, using LLMs directly is often impractical and highly inefficient, as ontologies can involve tens of thousands of concepts. Concept embeddings can also be integrated with structural features more easily (Li et al., 2019), for instance by using them to initialise the node embeddings
of a Graph Neural Network (GNN). Concept embeddings similarly play an important role in many multi-label classification tasks, especially in the zero-shot or few-shot regime (Xing et al., 2019; Yan et al., 2021; Luo et al., 2021; Huang et al., 2022; Ma et al., 2022). As a representative example of such a task, we will consider the problem of ultra-fine entity typing (Choi et al., 2018), which consists in assigning semantic types to mentions of entities, where a total number of around 10K candidate types are considered. In such tasks, the role of pre-trained concept embeddings is to inject prior knowledge about the meaning of the type labels (Xiong et al., 2019; Li et al., 2023a). Note that we cannot straightforwardly accomplish this with LLMs, as they have been found to struggle with information extraction tasks (Han et al., 2023). Moreover, scalability is often an important concern for information extraction systems, which further complicates the use of LLMs.
We take the view that concept embeddings, in the aforementioned applications, are primarily needed to capture commonalities among the concepts involved. For ontology expansion, this is true by definition, since the task explicitly involves identifying sets of concepts that have some property in common. For ultra-fine entity typing, Li et al. (2023a) reported that directly using pre-trained label embeddings was challenging. Instead, they proposed to cluster the set of labels based on pre-trained concept embeddings, and to use the resulting clusters to structure the label space. The idea of using embeddings to structure the label space also lies at the heart of many traditional approaches for zero-shot and few-shot classification.
A key limitation of traditional concept embeddings comes from the fact that they primarily reflect basic taxonomic categories. For instance, the embedding of banana is typically similar to that of other fruits, but dissimilar from the embeddings of other yellow things. Some authors have pro-
1467
posed to learn multi-facet embeddings as a way of alleviating these concerns (Rothe and Schütze, 2016; Jain et al., 2018; Alshaikh et al., 2019, 2020). Essentially, rather than learning a single vector representation of each concept (or entity), they learn a fixed number of different vectors, each focusing on a different facet. However, learning such representations is challenging for two main reasons. First, learning multi-facet representations requires some kind of supervision signal about the facets of interest (Locatello et al., 2019), which is not readily available for many domains. Second, existing approaches consider a fixed set of facets, which makes them unsuitable for open-domain settings. Indeed, the facets of interest strongly depend on the nature of the concepts involved. When modelling food, we may be interested in embeddings that capture their nutritional content. When modelling household appliances, we may want a representation that captures where in the house they are typically found. Rather than using the same set of facets for all concepts, we thus need a more dynamic representation framework.
In this paper, we propose a novel method for learning multi-facet concept embeddings based on two key ideas. First, we rely on ChatGPT1 to collect a diverse set of (property, facet) pairs, such as (yellow, colour), (found in the kitchen, location) or (sweet, taste), allowing us to treat the problem of learning multi-facet embeddings as a supervised learning problem. Second, rather than learning several independent vector representations, we only learn a single embedding for each concept, treating facets instead as masks on the set of coordinates. This approach offers several modelling advantages, including the fact that facets can have a hierarchical structure (e.g. colour is a sub-facet of appearance) and the fact that we do not have to tune the number and dimension of the facets a priori. Specifically, we train three BERT (Devlin et al., 2019) encoders: one encoder to map concepts onto their embedding, one encoder to map properties onto their embedding, and one encoder to map properties onto the embedding of the corresponding facet. We show that these encoders can be effectively trained using only training data obtained from ChatGPT, although the best results are obtained by augmenting this training data with examples from ConceptNet2.
1https://openai.com 2https://conceptnet.io

heading: 2 Related Work
text: Concept Embeddings The idea that language models of the BERT family can be used for learning concept embeddings has been studied extensively. Some approaches simply use the name of the concept as input to the BERT encoded, possibly together with a short prompt (Bommasani et al., 2020; Vulić et al., 2021; Liu et al., 2021a; Gajbhiye et al., 2022). Other approaches instead use contextualised representations from sentences mentioning the concept, selected from some corpus (Ethayarajh, 2019; Bommasani et al., 2020; Vulić et al., 2020; Liu et al., 2021b; Li et al., 2023b). These approaches have been developed with different motivations in mind. One common motivation is to learn something about the language model itself by inspecting the resulting concept embeddings, such as biases (Bommasani et al., 2020) or the model’s grasp of lexical semantics (Ethayarajh, 2019; Vulić et al., 2020). Other authors have rather focused on the use of embeddings for predicting semantic properties of concepts (Gajbhiye et al., 2022; Li et al., 2023b; Rosenfeld and Erk, 2023). Our paper can be seen as a continuation of this latter research line, where we aim to improve the range of properties that can be captured by concept embeddings through the use of facet embeddings.
Commonalities Gajbhiye et al. (2023) recently argued that the main purpose of concept embeddings, when it comes to downstream applications, is usually to identify what different concepts have in common. Specifically, given a set of concepts, they first used the corresponding concept embeddings to predict a set of properties for each concept. The resulting predictions were then filtered using a Natural Language Inference (NLI) model. Finally, properties that were found for at least two concepts where identified as shared properties. They showed, on the task of ultra-fine entity typing, that by augmenting the training data with these shared properties, models were able to generalise better. This idea also relates to the notion of conceptualisation (He et al., 2022; Wang et al., 2023b). Essentially, the latter works have suggested to augment commonsense knowledge graphs by generalising the concepts involved. This often involves replacing a specific concept (e.g. a football game) by a description referring to some salient property (e.g. a relaxing event). Wang et al. (2023a) showed that the resulting generalisations of commonsense knowledge graphs were useful for zero-shot com-
monsense question answering. The aforementioned methods all rely on the availability of a set of properties (or hypernyms) that can be used to generalise a given set of concepts. In practice, however, it is hard to obtain comprehensive property sets, which means that many commonalities may not be discovered. Moreover, certain commonalities are hard to describe, even though they intuitively make sense.3 To avoid such limitations, we identify commonalities by clustering concept embeddings.
Multi-Facet Embeddings The idea of capturing different facets of meaning has been studied in the context of disentangled representation learning, especially in computer vision (Chen et al., 2016; Higgins et al., 2017; Kim and Mnih, 2018; Chen et al., 2018). When it comes to learning disentangled representation of text, He et al. (2017) proposed a method for learning aspect embeddings in the context of sentiment analysis, whereas several authors have proposed multi-facet document embeddings (Jain et al., 2018; Risch et al., 2021; Kohlmeyer et al., 2021). Rothe and Schütze (2016) suggested that word embeddings could be decomposed into meaningful subspaces, which essentially correspond to facets. Most similar to our work, Alshaikh et al. (2019) proposed a method for decomposing a domain-specific concept embedding space into subspaces capturing different facets. To find this decomposition, they relied on the assumption that properties belonging to the same facet tend to have similar word embeddings. Finally, Alshaikh et al. (2020) proposed a mixture-of-experts model to learn multi-facet concept embeddings directly, using a variant of GloVe (Pennington et al., 2014).

heading: 3 Proposed Approach
text: We propose a bi-encoder based concept embedding model which is capable of representing concepts w.r.t. different facets. The key stumbling block in previous work on learning multi-facet embeddings has been the difficulty in acquiring a meaningful supervision signal about which properties belong to the same facet (e.g. that large and small both refer to size). As explained in Section 3.1, we can now overcome this difficulty by collecting propertyfacet pairs from LLMs. Our proposed model itself is described in Section 3.2. Finally, we explain how facet-specific embeddings can be extracted once
3As a simple toy example, among the set {cat, dog, goldfish, rabbit} the concepts cat and dog stand out as similar, even though they are not the only pets nor the only mammals.
the model has been trained (Section 3.3).

heading: 3.1 Obtaining Training Data
text: We need two types of training examples for our model: concept-property judgements (e.g. banana has the property rich in potassium) and propertyfacet judgments (e.g. rich in potassium refers to nutritional content). We use two sources for obtaining these examples: ChatGPT and ConceptNet.
ChatGPT We use the dataset of 109K conceptproperty judgments that were obtained from ChatGPT by Chatterjee et al. (2023).4 To obtain property-facet pairs, we proceed in a similar way, although obtaining suitable information about facets turned out to be more challenging. We obtained the best results with the following prompt, which does not ask about facets explicitly. Instead, we ask about concept-property pairs, but use a format which requires the model to specify the facet of each property that is generated:
I am interested in knowing common properties that are satisfied by different concepts. 1. Sound: loud - thunder, jet engine, siren 2. Temperature: cold - ice, refrigerator, Antarctica 3. Colour: orange - mandarin, basketball, clownfish 4. Shape: round - sun, orange, ball 5. Purpose: used for cleaning - broom, lemon, soap 6. Location: located in the ocean - sand, whale, corals. Please provide me with a list of 30 such examples.
We used this request with the same prompt 10 times. After this, we change the examples that are given (shown in bold above) and repeat. We manually processed the responses to standardise facet spellings and removed duplicates. For instance, facets were sometimes generated in plural (e.g. colors rather than color), or the same facet was generated with different spellings (e.g. color and colour). Even when changing the examples in the prompt after every 10 requests, the number of unique facet-property pairs that were generated saturated relatively quickly. In total, we obtained 828 unique facet-property pairs, covering 127 unique facets.
ConceptNet Starting from a ConceptNet 5 dump5, we first selected the English language
4Dataset available from https://github.com/ ExperimentsLLM/EMNLP2023_PotentialOfLLM_ LearningConceptualSpace.
5https://github.com/commonsense/conceptnet5/ wiki/Downloads
triples. Given a triple such as (boat, at location, sea) we create a corresponding concept-property pair (boat, at location sea) and a property-facet pair (at location sea, at location). In other words, the ConceptNet relations are treated as facets, and properties are obtained by combining a relation with a tail concept. Not all ConceptNet relations are suitable for this purpose. We specifically used: RelatedTo, FormOf, IsA, UsedFor, AtLocation, CapableOf, HasProperty, HasA, InstanceOf and MadeOf. Furthermore, when creating properties, we only consider tail concepts that appear at least 10 times. We thus end up with 884 distinct properties, 10 facets, 18505 concepts, 884 property-facet pairs and 36955 concept-property pairs.

heading: 3.2 Model Formulation
text: Let us write Dcp for the set of (concept, property) pairs that are available for training. Similarly, we write Dpf for the set of available (property, facet) pairs. We build on the following bi-encoder loss from (Gajbhiye et al., 2022):
L=− ∑
(c,p)∈Dcp log σ
( Con(c) · Prop(p) )
− ∑
(c,p)∈Ncp log
( 1− σ ( Con(c) · Prop(p) ))
where Ncp is a set of negative examples. Specifically, for each positive example (c, p), five negative examples (c, p′) are obtained by replacing p by another property p′. The concept embedding Con(c) and property embedding Prop(p) are obtained by two separate BERT encoders. The concept encoder Con uses a prompt of the form <Concept> means [MASK]. The property encoder Prop uses the same prompt. In both cases, the embeddings are obtained from the final-layer embedding of the [MASK] token. However, the concept embedding Con(c) is normalised (w.r.t. the Euclidean norm) whereas the property embedding Prop(p) is not.
Previous work on multi-facet embeddings has relied on learning multiple concept embeddings, where each concept embedding only captures a subset of all properties (Alshaikh et al., 2020). This approach has a number of drawbacks, however. For instance, it relies on the idea that each facet is represented using the same number of dimensions, implicitly assumes that the overall number of facets is relatively small, and that the facets are independent from each other. This is particularly problematic in open-domain settings, where a wide range of
facets may need to be considered, certain facets only make sense for some concepts (e.g. nutritional value only makes sense for food) and facets often have a hierarchical structure (e.g. colour is a subfacet of appearance). Therefore, instead of learning multiple concept embeddings, we instead interpret facets as masks on concept embeddings.
Specifically, we train a third BERT encoder, Facet, which also takes the property p as input and again uses the same prompt. The idea is that Facet(p) indicates which coordinates of the concept embeddings are most relevant when modelling the property p. We define the masked embedding of concept c w.r.t. some property p as follows:
MC(c, p) = Con(c)⊙ Facet(p)
∥Con(c)⊙ Facet(p)∥
where we write ⊙ for the component-wise product. We essentially keep the same bi-encoder model but instead rely on these masked concept embeddings: L1=− ∑
(c,p)∈Dcp log σ
( MC(c, p) · Prop(p) )
− ∑
(c,p)∈Ncp log
( 1− σ ( MC(c, p) · Prop(p) ))
Without further supervision, the facet encoder does not learn meaningful facets. Therefore, we use the (property, facet) examples from Dpf to ensure that properties which belong to the same facet have a similar facet embedding. For a given facet f , we write Pf for the set of properties that we know to belong to this facet, i.e. Pf = {p | (p, f) ∈ Dpf}. We use the InfoNCE loss:
L2=− ∑
f
∑
p,q∈Pf log
exp ( cos(F(p),F(q))
τ
)
∑ exp
( cos(F(p),F(r))
τ
)
where we abbreviated Facet(p) as F(p), the sum in the denominator ranges over r ∈ {q}∪{p | (p, f) /∈ Dpf}, and the temperature τ > 0 is a hyperparameter. The InfoNCE loss encourages properties which belong to the same facet to have facet embeddings that are more similar to each other than to the facet embeddings of properties which do not. The overall model is trained by optimising the loss L1+L2.

heading: 3.3 Extracting Facet-Specific Representations
text: The model from Section 3.2 can be used in several ways. First, we can simply use the concept embeddings Con(c) to represent each concept c.
In this case, the purpose of having facets is to ensure that the concept embeddings capture a broader range of properties, but we only consider these facets during training. We will refer to this approach as ConEmb-F. The concept embeddings from the standard bi-encoder, without facet embeddings, will be referred to as ConEmb.
In some applications, concept embeddings are used for clustering concepts. The purpose of multifacet embeddings is to ensure that different kinds of clusters can be found. In such settings, we extract different facet-specific concept embeddings from the model. Specifically, let P = {p1, ..., pm} be the set of properties of interest. For each property pi we have a corresponding facet vector fi = Facet(pi). We use the K-means algorithm to cluster these facet vectors into clusters X1, ...Xk and treat each of these clusters as facet. We associate each concept c with k facet-specific representations c1, ..., ck, defined as:
cj = Con(c)⊙
(∑ pi∈Xj fi ) ∥∥Con(c)⊙
(∑ pi∈Xj fi )∥∥ (1)
The representations obtained by this approach depend on how the set of properties P is chosen. For our experiments, we simply set P to be the set of all properties that appear in our training set.

heading: 4 Experiments
text: We analyse the effectiveness of the proposed multifacet concept embedding model. We intrinsically evaluate the embeddings on predicting commonsense properties (Section 4.1) and outlier detection (Section 4.2). We furthermore consider two downstream applications: ontology completion (Section 4.3) and ultra-fine entity typing (Section 4.4).6

heading: 4.1 Predicting Commonsense Properties
text: The use of facets should lead to concept embeddings that capture a wider range of properties. To test this hypothesis, we consider the task of commonsense property prediction, which we treat as a binary classification problem: given a concept and a commonsense property, decide whether the property is satisfied by the concept or not. The difficulty of this task depends on how the training-test split is constructed. One strategy, called concept split, ensures that the concepts appearing in the
6Our datasets, pre-trained models and implementation are available at https://github.com/hananekth/facets_ concepts_embeddings
training and test data are different, but the properties are not. Gajbhiye et al. (2022) found that simple nearest neighbour strategies can do well on this task, meaning that this variant does not adequately assess whether the concept embeddings capture commonsense knowledge. For this reason, they proposed a property split, where the properties appearing in training and test are different, but the concepts are the same. Finally, they also considered a C+P (concept+property) split, where both the concepts and properties are different in training and test. In all cases, we first pre-train the encoders Con, Prop and Facet on the ChatGPT and/or ConceptNet training data (see Section 3.1), before fine-tuning on the training splits of the property prediction benchmarks.7
Table 1 summarises the results for all three settings, using two standard benchmarks for commonsense property prediction: the extension of the McRae property norms dataset (McRae et al., 2005) that was introduced by Forbes et al. (2019) and the augmented version of CSLB8 introduced by Misra et al. (2022). Our main baseline is the biencoder model from Gajbhiye et al. (2022), shown as BiEnc, which also forms the basis for our facetbased model. We report the results from Gajbhiye et al. (2022), which are for a model that was pretrained on Microsoft Concept Graph (Ji et al., 2019) and GenericsKB (Bhakthavatsalam et al., 2020), as well as results for models that we trained on the ChatGPT and ConceptNet training sets (see Section 3.1). Finally, we show the result of our full model (i.e. with the facet encoder), shown as BiEnc-F. We compare four different encoders: BERT-base, BERT-large, RoBERTa-base and RoBERTa-large.
The results clearly show the benefit of using facets, as the BiEnc-F models consistently and substantially outperform the BiEnc baselines. Comparing the different training sets, the ChatGPT training examples are more effective than the ConceptNet examples, but the best results are obtained when both sources are combined. Among the different LMs, BERT-large achieves the best results.
Based on the results from this experiment, for the remaining experiments, we will focus on the model based on BERT-large, which is trained on both the ChatGPT and ConceptNet training examples.
7Detailed training details can be found in the appendix. 8https://cslb.psychol.cam.ac.uk/propnorms

heading: 4.2 Outlier Detection
text: To evaluate whether our facet-based concept embeddings can help us to identify commonalities, we consider the task of outlier detection (CamachoCollados and Navigli, 2016; Blair et al., 2017; Brink Andersen et al., 2020). In each instance of this task, we are given a set of concepts (or entities). Some of these concepts have a particular property in common, and the task consists in identifying these concepts (without being given any information about the shared property itself). This task has traditionally been used as an intrinsic benchmark for evaluating word embeddings.
Dataset Existing benchmarks mostly focus on broad taxonomic categories, whereas we are specifically interested in identifying shared commonsense properties. We therefore constructed a new out-
lier detection benchmark based on the extended McRae dataset from Forbes et al. (2019). To create an outlier detection problem instance, we first select a property from this dataset (e.g. dangerous) as well as 3 concepts which have this property and 7 outlier concepts which do not. Many of the properties in the McRae dataset are not suitable for our benchmark, either because they correspond to taxonomic categories (e.g. an animal or edible) or because they are too ambiguous or noisy (e.g. accordion, car and escalator are described as having the property fun, but airplane is not). Therefore, we manually selected 10 properties which do not suffer from these limitations. For each property, we manually clustered the concepts with this property into broad taxonomic groups.9 When selecting the 3 positive concepts, for a given instance, we ensure that all three examples come from a different group. When selecting the 7 outliers, we check that they do not share any properties. Specifically, we ensure that any two of the outliers do not have any properties in common in the extended McRae dataset, in ConceptNet or in Ascent++10. For each property, we sample 100 problem instances, following this process. We report the results in terms of exact match, i.e. the prediction for a given instance is labelled as correct if the three positive examples were correctly identified. We report the percentage
9The resulting clusters can be found in the appendix. 10https://ascentpp.mpi-inf.mpg.de
of correctly labelled instances, for each property.
Methods We compare three strategies for detecting outliers. For the method denoted ConEmb, we use the ConEmb embeddings as follows. For each concept c, we find the second and third nearest neighbour. Let us denote these as c2 and c3. If c is a positive example, cos(Con(c),Con(c2)) should be high and cos(Con(c),Con(c3) should be low. We thus score each concept as score(c) = cos(Con(c),Con(c2))−cos(Con(c),Con(c3)). As positive examples, we then select the concept with the highest score along with its two nearest neighbours. The method denoted ConEmb-F uses the same strategy, but instead uses the ConEmb-F embeddings. Finally, when using the method denoted MultiConEmb, we first obtain 10 facet-specific embeddings of each concept, using (1). We then first apply the same method as before to each of the 10 facet-specific embedding spaces. Finally, we select the prediction for the facet where the score of the highest-scoring concept was maximal.
Results The results are summarised in Table 2. MultiConEMb, which exploits facet-specific representations, substantially outperforms the baselines. The performance of ConEmb and ConEmb-F is comparable, which is as expected: even though ComEmb-F was trained using facets, this method represents concepts as single vectors, and the similarities between these concept vectors still mostly reflect taxonomic relatedness.

heading: 4.3 Ontology Completion
text: Ontologies use rules to encode how the concepts of a given domain are related. They generalise tax-
onomies by allowing the use of logical connectives to encode these relationships. Li et al. (2019) introduced a framework for predicting missing rules in ontologies using a Graph Neural Network. The nodes of the considered graph correspond to the concepts from the ontology, and the input representations are pre-trained concept embeddings. Recent work has used this model to evaluate concept embeddings, as its overall performance is sensitive to the quality of the input representations (Li et al., 2023b). The intuition underpinning the model is closely aligned with the idea of modelling concept commonalities. Essentially, if the ontology contains the rules11 X1 ⊑ Y, ...,Xk ⊑ Y and we know from the pre-trained embeddings that Xk+1 is similar to X1, ..., Xk then it is plausible that the rule Xk+1 ⊑ Y is valid as well.
We test the effectiveness of our model in two ways. First, we use the ConEmb-F concept embeddings as input features, which allows for a direct comparison with the effectiveness of other concept embedding models. In this case, the use of facets only affects how the concept embeddings are learned. As a second strategy, referred to as MClu, we first obtain 10 facet-specific embeddings of all the concepts, using (1). We then cluster the concepts in each of the facet-specific concept embeddings separately. For this step, we have relied on affinity propagation. This results in 10 different clusterings of the concepts. For each cluster C in each of these clustering we add a fresh concept YC to the ontology, and for every concept X in C, we add the rule X ⊑ YC . We then apply the standard GNN model from Li et al. (2019) to the resulting extended ontology. As a baseline, we also apply the clustering strategy to the ConEmb concept embeddings, which we refer to as Clu. Note that in this case there is only one clustering. Note that when MClu or Clu are used, we still need to use concept embeddings to use as input features. We show results with the ConCN embeddings from Li et al. (2023b) and for our ConEmb-F embeddings.
The results in Table 3 show that the ConEmbF input embeddings consistently outperform the ConEmb vectors. Moreover, they achieve a performance which is similar to that of the ConCN embeddings, which is the current state-of-the-art. Moreover, the proposed clustering strategies, which have not previously been considered for ontology
11In description logic syntax, X ⊑ Y means that every instance of the concept X is also an instance of the concept Y , i.e. it represents the rule “if X then Y ”.
completion, are highly effective. The MClu variant outperforms Clu in all but one case, which shows the benefit of explicitly considering facet-specific embeddings. We can also see that the ConEmb-F embeddings as input features perform better than the ConCN embeddings, when used in combination with the clustering strategies.

heading: 4.4 Ultra-Fine Entity Typing
text: We consider the task of ultra-fine entity typing (Choi et al., 2018), which was also used by Gajbhiye et al. (2023) to demonstrate the usefulness of modelling concept commonalities. Given a sentence in which an entity mention is highlighted, the task consists in assigning labels that describe the semantic type of the entity. The task is formulated as a multi-label classification problem with around 10K candidate labels. Many of the candidate labels only have a small number of occurrences in the training data. This makes it paramount to rely on some kind of pre-trained knowledge about the meaning of the labels. Li et al. (2023a) proposed a simple but surprisingly effective strategy: use pretrained concept embeddings to cluster the labels and augment the training set with labels that refer to these clusters. For instance, if a training example is labelled with label l and this label belongs to cluster c then they add the synthetic label “cluster c” to this training example. This intuitively teaches the model which labels are semantically related, as the training objective encourages instances which are labelled with “cluster c” to be linearly separated from other instances. Gajbhiye et al. (2023) improved on this strategy by instead identifying commonsense properties that were satisfied by the different concepts/labels, and by augmenting the training examples with these properties, instead of
synthetic cluster labels. This use of shared properties has the advantage that a broader range of commonalities can be identified, whereas clustering standard concept embeddings leads to clusters that only reflect standard taxonomic categories. Our hypothesis is that we can achieve the same benefits by clustering our multi-facet representations, and that the use of clusters can potentially lead us to capture finer-grained commonalities.
Table 4 summarises the results. All results were obtained using the DenoiseFET model from Pan et al. (2022). The base model in Table 4 shows the results if we use this model without augmenting the training labels. Properties refers to the strategy from Gajbhiye et al. (2023), which adds labels corresponding to shared properties, while Clu is the strategy from Li et al. (2023a), which adds labels corresponding to clusters. We show results for this clustering strategy with three different concept embeddings: the ConCN embeddings from Li et al. (2023b) as well as ConEmb and ConEmb-F. The approach where we use clusterings from different facet-specific embeddings is shown as MClu. We can see that MClu achieves the best results, which confirms the usefulness of facet-specific representations for this task. When using the Clu strategy, ConEmb-F also slighly outperforms ConEmb.

heading: 5 Conclusions
text: Many applications rely on background knowledge about the meaning of concepts. What is needed often boils down to knowledge about the commonalities between different concepts, as this forms the basis for inductive generalisation. Clustering pre-trained concept embeddings has been proposed in previous work as a viable strategy for modelling such commonalities. However, the resulting clusters primarily capture taxonomic categories, while commonalities that depend on various commonsense properties are essentially ignored. In this paper, we proposed a simple strategy for obtaining more diverse representations, by taking into account different facets of meaning when training the concept encoder. We found that the resulting concept representations lead to consistently better results, across all the considered tasks.
Acknowledgments This work was supported by EPSRC grants EP/V025961/1 and EP/W003309/1, ANR-22-CE23-0002 ERIANA and HPC resources from GENCI-IDRIS (Grant 2023-[AD011013338R1]).
Limitations
Our approach relies on encoders from the BERT family, which are much smaller than recent language models. We did some initial experiments with Llama 2, but were not successful in obtaining better-performing concept embeddings with this model. While it seems likely that future work will reveal more effective strategies for using larger models, our use of BERT still has the advantage that we can efficiently encode a large number of labels, which remains important for applications such as extreme multi-label text classification.
We have only looked at modelling commonsense properties of concepts. Modelling facets of meaning is intuitively also important for modelling named entities (e.g. for entity linking) and for sentence/document embeddings (e.g. for retrieval). An analysis of facet-based models for such applications is left as a topic for future work.

heading: A Training Details
text: Predicting Commonsense Properties We use the AdamW optimizer with learning rate 2e-5. We use early stopping with a patience of 20. We use the same settings as Gajbhiye et al. (2022) for both fine-tuning and pre-training. During pre-training, we randomly select 10000 pairs for tuning. For the concept split we use the fixed training-test split from Gajbhiye et al. (2022). For property split, we use 5-fold cross-validation, while for C+P, we used the 3 × 3 fold cross-validation strategy. During fine-tuning, for all splits, we randomly select 20% from the training set as validation data for model selection. We use a batch size of 32 and set the maximal sequence length for the concept and property prompts to 32.
Outlier Detection For these experiments, we use the same bi-encoder models as for predicting commonsense properties. Specifically, we have used the model initialised form BERT-large-uncased, which was trained using ConceptNet and ChatGPT. However, in this case, the model is used without further fine-tuning.
Ontology Completion To get concept clusters from the affinity propagation algorithm, we tune the preference values from {0.5, 0.6, 0.7, 0.8, 0.9}. We set the learning rate to 1e-2, the maximum number of epochs to 200 and the dropout rate to 0.5. We use AdamW as optimizer and tune the number of hidden dimensions from {8, 16, 32, 64} and the number of GNN layers from {3, 4, 5}. We use a weight decay of 5e-2.
Ultra-Fine Entity Typing To get concept clusters, we again use affinity propagation and select the preference values from {0.5, 0.6, 0.7, 0.8, 0.9}. We use a learning rate of 2e-5 with AdamW as op-
timizer, a batch size of 16 and a maximum number of 500 epochs.

heading: B Outlier Detection Dataset
text: Table 5 shows the properties that were selected from the McRae dataset for constructing the outlier detection benchmark. For each property, we selected concepts that are asserted to have this property in the dataset, and we organised them into broad taxonomic categories, which are also shown in Table 5.

heading: C Qualitative Analysis
text: Table 6 shows examples of the nearest neighbours of frisbee and bureau in some of the facet-specific embedding spaces that are used by the MClu strategy. For this analysis, we use the set of concepts from the McRae dataset. These examples illustrate how different facets emphasise different aspects.
For the case of frisbee, the first facet links this concept to other sports related terms. In the second facet, it is instead related to round things. In the third facet, the top neighbours are related to kids. Due to the way in which the facets are learned (e.g. by considering a fixed number of facet embedding clusters), there are also some facets that reflect a mixture of different aspects. For instance, the last facet in Table 6 combines elements of the first three facets, i.e. the nearest neighbours cover sports related concepts, kids related concepts, and round things. For bureau, in the first facet we find office related terms. In the second facet, the top neighbours are different types of furniture. In facets 3 and 4 we see a mixture of different kinds of terms.
Table 7 similarly shows examples of nearest neighbours in different facets for the label space from the UFET dataset.

heading: Future Work
text: While it seems likely that future work will reveal more effective strategies for using larger models, our use of BERT still has the advantage that we can efficiently encode a large number of labels, which remains important for applications such as extreme multi-label text classification. We have only looked at modelling commonsense properties of concepts. Modelling facets of meaning is intuitively also important for modelling named entities (e. g. for entity linking) and for sentence/document embeddings (e. g. for retrieval). An analysis of facet-based models for such applications is left as a topic for future work