What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies
Authors: Amit Gajbhiye
Zied Bouraoui
Na Li
Usashi Chatterjee
Luis Espinosa-Anke
Steven Schockaert
Venue: EMNLP 2023 Main (2023)
Keywords: lexical semantics
commonality detection
ultra-fine entity typing
ontology learning

Abstract: Concepts play a central role in many applications. This includes settings where concepts have to be modelled in the absence of sentence context. Previous work has therefore focused on distilling decontextualised concept embeddings from language models. But concepts can be modelled from different perspectives, whereas concept embeddings typically mostly capture taxonomic structure. To address this issue, we propose a strategy for identifying what different concepts, from a potentially large concept vocabulary, have in common with others. We then represent concepts in terms of the properties they share with the other concepts. To demonstrate the practical usefulness of this way of modelling concepts, we consider the task of ultra-fine entity typing, which is a challenging multi-label classification problem. We show that by augmenting the label set with shared properties, we can improve the performance of the state-of-the-art models for this task.

Various applications rely on knowledge about the meaning of concepts, which is typically encoded in the form of embeddings. For instance, pre- trained concept embeddings are used to provide prior knowledge about the labels few-shot and zero-shot learning tasks (Socher et al., 2013; Ma et al., 2016; Xing et al., 2019; Yan et al., 2022; Xiong et al., 2019; Hou et al., 2020; Li et al., 2020; Yan et al., 2021). Concept embeddings also play central role in the of knowledge manage- ment, for instance for taxonomy learning (Vedula et al., 2018: 8; Malandri et al., 2021), knowledge graph alignment (Trisedya et al., 2019), ontology alignment (Kolyvakis et al., 2018) and ontology completion (Li et al., 2019). In such applications, Language Models (LMs) such as BERT (Devlin et al., 2019) cannot be used directly, since we need What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies
Amit Gajbhiye', Zied Bouraoui', Na Li", Usashi Chatterjee', Luis Espinosa Ankel Steven Schockaert'
CardifNNLP, Cardiff University, UK CRIL CNRS & University of Artois, France
University of Shanghai for Science and Technology, China AMPLYFI, UK (gajbhiyea, chatterjee, espinosa- sthhortssa scrrk jecardiff.ae zied. bouraoui@cril.f li_na@usst. edu. cn
schockaerts
Abstract
concept embeddings which do not depend sen- tence context. For this reason, several authors have proposed strategies for obtaining decontextualised (or static) concept embeddings from LMs (Etha- yarajh, 2019; Bommasani et al., 2020; Vulić et al.. 2020; Li et al., 2021; Vulié et al., 2021; Liu et al., 2021; Gajbhiye et al., 2022).
Concepts play central role in many applica- tions. This includes settings where concepts have to be modelled in the absence of sentence context. Previous work has therefore focused distilling decontextualised concept embed- dings from language models. But concepts can be modelled from different perspectives, whereas concept embeddings typically mostly capture taxonomic structure. To address this is- sue, propose strategy for identifying what different concepts. from a potentially large con- cept vocabulary, have common with others. We then represent concepts in terms of the prop- erties they share with the other concepts. To demonstrate the practical usefulness of this way of modelling concepts, consider the task of ultra-fine entity typing. which is a challenging multi-label classification problem. We show that by augmenting the label set with shared properties, we can improve the performance of the state-of-the-ae models for this task.! !
Decontextualised concept embeddings tend to primarily reflect basic taxonomic structure, e.g. capturing the fact that televisions are electronic devices and that salmon fish. However, applica- tions often rely on different facets meaning. computer vision applications, we want concept em- beddings to reflect visual features; e.g. they should capture the fact that chess boards, televisions and books have something in common (i.e. being rect- angular). When modelling scene graphs (Qi et al., 2019), may rather want embeddings that capture which objects are often found together in the same visual scene. For instance, concept embeddings should then reflect the fact that televisions, sofas and rugs have something in common (i.e. that they are typically found in the living room). When mod- elling food ontologies, concept embeddings should perhaps capture the fact that salmon and walnuts share the property of being rich in Omega-3. In principle, if sufficient training data is available, we could learn application-specifie concept embed- dings by fine-tuning language model. However, the role of concept embeddings is often preciscly to address the scarcity of application speciffc train- ing data. We, therefore focus on strategies that use pre-trained models in an unsupervised way.
1 Introduction
Concept embeddings can be viewed compact representations of similarity metrics. However, what matters in the aforementioned applications is often less about modelling similarity than about modelling what different concepts have in com- mon. This an important distinction because we can list the properties that a concept satisfies in an periication-idependdee way, which is not possi-
'Our datascts and evaluation scripts are available at https: t/github.... com/ ami ygajbbiiy/coocepp. cooeeppt_cooomatttt

---

Property
Concepts
used for staying connected
telephone number, google, area code, user name, facebook, land line, email, call center, server, link farm, name-server
used for beach activities a round shape
deck chair, beach volleyball, beach ball, sun hat, sun dog
bubble, small ball, ball bearing. sphere, disc, round top, centre circle. wheel, tin can, cese, eylinder, eval, tibe parking lot, high street, suburbs, supermarket, mall, food court, piazza, city hall, cinema, general-sterg
located in cities
used for communication
address book, twitter, voice message, instant message, voice mail letter bomb, listening pest; pest,-mait mail hemb, ring tone, stteet--adreese
operaator
accommodation options for camping
beer tent, man cave, rock shelter, holiday camp, camp, mobile home, guest room, inn, rosert, rosort,trading post, cabin huntinu lodgem
accounting
expense account, transaction, register, business intelligence, finance, business record, income stalement, backup, ceusts, consumer ereditt tenance, Harket reserrch
listing, main-
IV series genres
detective, soap opera, reality show, crime, mystery, cartoon blockbuster, hero, radio drama, seuson finale, series finele
aches
fever, strain, head cold, battle fatigue, sore, pain, hurt, fatigue bow shock. cold, dry-eye, hunger, ohesity, sting, streteh
adulthood
middle age, thirty, age group, womanhood, drinking age, aged, carcers, elder, generation, maturity, autumn, hachelor, birth, chiiioplcccccccidooodd.ddd docade, era 'slogan,, "brand image', 'publicity,, 'advertising', , 'product placement', 'banner ad', advertiser', 'advertisement', 'advance", "propagadaa" 'ambush marketing". dde dlod- Het", "ain-mail" "Fan-service"
childhood,
advertisements
baby carriers
basket, baby seat, basket case, car seat, carrier, boek, baby bottle, body-e,
Table I:
Examples of properties found using the bi-encoder model. In red and crossed out are properties initially predicted, but discarded by the DeBERTa-based filtering.
ble for modelling similarity. Based on this view, we propose the following strategy: we represent concepts by explicitly capturing the properties they satisfy. To identify the properties that might be rel- evant in a given application, we look for those that describe what different concepts from the given vocabulary have in common.
demonstrate the practical usefulness of uncover- ing shared properties on the downstream task of ultra-fine entity typing.
2 Related Work
To implement this strategy, we need an efficient mechanism for identifying these commonalities. The vocabulary is often too large to directly use Large Language Models (LLMs) to solve this task. Instead, we rely on a bi-encoder architecture to efficiently retrieve the properties that are satisfied by each concept (Gajbhiye et al., 2022). Once the initial set of properties has been retrieved, we use fine-tuned DeBERTa model (He et al., 2021) to filter these properties. Table shows examples of properties that were thus identified, along with the corresponding concepts. Note how some of these commonalities are unlikely to be captured by stan- dard concept embeddings (c.g. linking telephone number and facebook in the first example).
The task of predicting commonsense properties has been studied by several authors (Rubinstein et al., 2015; Forbes et al., 2019; Gajbhiye et al., 2022; Apidianaki and Garí Soler, 2021; Bosselut et al., 2019). Modelling the commonalities between con- cepts, or conversely, identifying outliers, has also been previously considered. For instance, outlier detection has been used for intrinsic evaluation of word embeddings (Camacho-Collados and Nav- igli, 2016; Blair al., 2017; Brink Andersen al., 2020). However such benchmarks are focused on taxonomic categories. Moreover, we focus on find- ing commonalities in large vocabularies, containing perhaps tens of thousands of concepts. Another re- lated task is entity set expansion (Pantel et al., 2009; Zhang et al., 2020; Shen et al., 2020). Given ini- tial set of entities, e.g. (France, Germany, Italy). this task requires selecting other entities that have the same properties (e.g. being European countries). However, the focus is usually on named entities, whereas we focus on concepts.
After summarising the related work in Section 2, describe our strategy for uncovering shared properties in Section 3. Our subsequent evaluation Section focuses on two tasks. First, we carry out an intrinsic evaluation to show the effective- ness of the proposed filtering strategy. Second, we

---

3 Identifying Shared Properties
Model
Pre-training
F1
BERT-large bi-ene
36.6
Let V be the considered vocabulary of concepts. Our aim is to associate each concept E V with set of properties. To find suitable properties, first retrieve
BaRT-large hi-ene
(Gajhhiye et al., 2022)
49.3
BERT-large large bi-ene
ConceptNet
54.0
BERT-large bi-enc
ChatGPT
50.1
BERT-large bi-enc
ConceptNet+ChaGGT
55.4
of candidate properties for each concept in V using an efficient bi-encoder model. Subsequently, we verify the retrieved properties with joint encoder.
BERT-large joint
51.8
RoBERTa-large joint
58.8
DeBERTa-larg joint
65.9
BERT-large joint
ConceptNet
55.4
RoBERTa-large joint
ConceptNet
60.3
DeBERTa-large joint
ConceptNet
65.7
Retrieving Candidate Properties For each con- cept с E V, we want to find a set of properties that are likely to be satisfied by that concept. The vocabulary V may contain tens of thousands of con- cepts, which makes it expensive to use LLM for this purpose. We instead rely the bi-encoder model from Gajbhiye al. (2022). The idea is fine-tune two BERT models: one for learn- ing concept embeddings (deon) and one for learn- ing property embeddings (oprop). The probability that concept e. has property p is then estimated as (e)oop))))pppp(p)) ). with a the sigmoid function. To train this model, need examples of concepts and the properties they satisfy. The only large- scale knowledge base that contains such training examples is ConceptNet, which is unfortunately rather noisy and imbalanced. Gajbhiye et al. (2022) therefore trained their model on a large set of (hy- ponym, (onym.hypernym) pairs from Microsoft Concept Graph (Ji et al., 2019), together with examples from GenericsKB (Bhakthavatsalam et al., 2020).
RoBERTa-large NLI RoBERTa-large NLI
57.7
WANLI
57.2
RoBERTa-large NLI
WANLI + ConceptNct
57.6
Table 2: Results the McRae property split dataset.
P contain every property that appears least twice in the training data for the bi-encoder. Then for a given concept c, we use maximum inner- onner-product search (MIPS) to efficiently find the 50 properties p from P for which the dot product ócon (c) @prop (p) is maximal. To make hard selection of which properties are satisfied and which ones are not, we rely on joint encoder. Such models are typically more accurate than bi-encoder but cannot be used for retrieval. Specifically, use a masked lan- guage model with the following prompt:
Can be described as p? [MASK].
(1)
We train a linear classifier to predict whether e has the property p from the final layer embedding of the [MASK] token. As the masked language model, we use DeBERTa-v3-large model. To train the classifier and fine-tune the language model, we used the extended McRae dataset from Forbes et al. (2019) and the augmented version of CSLB intro- duced by Misra et al. (2022). Both datasets have true negatives and are focused on commonsense properties, which we found important initial ex- periments. Finally, after obtaining the properties for every concept in C, we remove those properties that were only found for single concept.
Since the performance of the bi-encoder heavily depends on the training data, and existing train- ing sets are sub-optimal, we created a training set of 109K (concept.propertt) pairs using 2hatGPT2 Simply asking ChatGPT to enumerate the prop- erties of some concept tends to result in verbose explanations describing overly specific properties, which are less helpful for identifying commonal- ities. Therefore, we used prompt which specifi- cally asked ChatGPT to identify properties that are shared by several concepts. We also experimented with a training set we derived from ConceptNet 5.5. Specifically, we converted instances of the relations IsA, PartOf. LocatedAr, UsedFor and Proprryyyt into a set of 63,872 (concept.property) pairs. The full details can be found in the appendix.
4 Experiments
Predicting Commonsense Properties As an in- trinsic evaluation of our model for predicting prop- ertics, we use the property split of the McRac dataset that was introduced by Gajbhiye et al. (2022). This benchmark involves classifying (con- cept.property) pairs as valid or not. It is particularly
Selecting Properties To associate properties with concepts, first need to determine set P of properties of interest. For our experiments, let
hhttts://chat.or
Phttps://cslb. wwt.bbppsshol...mo can. ac. proppormsss

---

Representation
Representation
F1
Base model (Li et al., 2023a)
49.2
Base model (Li et al., 2023a)
49.8
ConCN clusters (l. et al., 2023a)
50.4
ConCN clusters (1.i et al., 2023a)
51.9
Properties (ChatGPT)
50.4
Bi-ene clusters properties (ChatGPT + CN)
52.2
Properties (ConceptNet)
50.9
Properties (ChatGPT + CN)
50.9
Table 4: Results for ultra-fine entity typing, using
Bi-ene clusters (ChatGPT CN)
50.6
BaRT-large entity encoder and after applying the post- processing strategies from et al. (2023a).
ConCN clusters properties (ChatGPT CN)
50.9
Bi-ene clusters properties (ChatGPT+ CN)
51.1
Table 3: Results for ultra- fine entity typing. using BERT-base entity encoder with augmented label sets.
lated as a multi-label classification problem, where there are typically several labels that apply to a given entity. The task challenging due to the fact that a large set of more than 10,000 candidate labels is used. Moreover, most of the training data comes from distant supervision signals, and many labels are not covered by the training data at all. This makes it important to incorporate prior knowledge about the meaning of the labels. Li al. (2023a) recently achieved state-of-the-a art results with sim- ple clustering based strategy. They first cluster all the labels based on a pre-trained embedding at dif- ferent levels of granularity. Each cluster is then treated as an additional label. For instance, if the label appears in clusters cI,..., Ct then whenever the label I appears in training example, the labels C1y-..; Ct are added as well. This simple label aug- mentation strategy was found to lead to substantial performance gains, as long as high-quality concept embeddings were used. Their best results were achieved using the ConCN concept embeddings from Li et al. (2023b).
challenging because there is no overlap between the properties in the training and test splits.
The results for the bi-encoder top part of Table 2. As can be seen, the training sets obtained from ChatGPT and from ConceptNet both outperform the dataset that was used by Gajbhiye et al. (2022). The best results were obtained by combining the ChatGPT and ConceptNet training sets. In the middle part of Table 2, we evaluate different variants of the joint encoder. We com- pare three masked language models: BERT-large, RoBERTa-large (Liu al., 2019) and DeBERTa- v3-large (He al., 2021). each case, we con- sider one variant where the models are directly fine- tuned on the McRae training set and one where the models are first pre-trained on ConceptNet (i.e. the dataset we used for training the bi-encoder). As
shown in the
can be seen, the best results are obtained using De- BERTa without pre-training on ConceptNet. One disadvantage of the ConceptNet dataset is that it does not contain true negatives, i.c. the negative training examples are obtained by randomly cor- rupting positive examples. We also experimented with an NLI formulation. To this end, we used a premise of the form "the concept is c" and a hy- pothesis of the form "the concept can be described p". The results are shown in the bottom part of Table 2 for three variants with RoBERTa-large: without pre-training, one where we pre-train on WANLI (Liu al., 2022), and one where first pre-train on WANLI and then continue training on ConcepiNet. As can seen, this NLI based formulation was less successful.
In this experiment, we use the shared properties that we uncovered as additional labels, in the same way that Li et al. (2023b) used clusters. For in- stance, if the property found in the wild was found for elephant, then whenever training example is labelled with elephant we add the label found in the wild. To identify the shared properties, we use the same bi-encoders that we used for the exper- iment in Table 2. We filter the properties using the DeBERTa-large model that was fine-tuned on McRae and CSLB.
The results are summarised in Table 3, for the case where a BERT-base entity encoder is used. The base model, without any additional labels, is the DenoiscFET model from Pan et al. (2022). When adding the clusters of ConCN embeddings. Li al. (2023a) were able increase the Fl score from 49.2 50.4. The configurations where we instead use the shared properties are reported as Properties (X), with X the training set that was
Ultra-Fine Entity Typing To evaluate the use- fulness of identifying shared properties in a down- stream task, we consider the task of ultra-fine en- tity typing (Choi et al., 2018). Given a sentence in which an entity is highlighted, this task consists in assigning semantic types to that entity. It is formu-

---

used for the bi-encoder. As can be seen, with the bi-encoder trained on ConceptNet, we achieve an Fl score of 50.9, which clearly shows the useful- ness of the shared properties. The model that was trained on both ConceptNet and the ChatGPT ex- amples achieves the same result. Next, we tested whether the shared properties and the clusters used by Li et al. (2023b) might have complementary strengths (ConCN properties). We also con- sidered variant where we instead obtained clus- ters from the concept encoder of the bi-encoder model (Bi-enc + properties). For both variants, we used the bi-encoder that was trained on Chat- GPT and ConceptNet. Adding the clusters from the bi-encoder leads to a further improvement to 51.1. With the ConCN clusters, the result remains unchanged. When only the clusters from the bi- encoder are used, we achieve an F1 score of 50.6. Finally, Table shows the result of our best con- figuration when using a BERT-large entity encoder, and when applying the post-processing techniques proposed by Li et al. (2023a). As can be seen, our model surpasses their saate-of-the-ase result.
may be needed. For instance, it may be possible compactly encode the shared properties as dense vectors, e.g. by using partially disentangled repre- sentations to separate different aspects of meaning.
This work was supported by EPSRC grant EP/V025961/1, ANR/://,,ANR-- ERIANA and by HPC resources from GENCI- IDRIS (Grant 2023-[AD011011333RR]
ckknowledgments
References
Marianna Apidianaki and Aina Garí Soler. 2021. ALL dolphins are intelligent and SOME are friendly: Prob- ing BERT for nouns' semantic properties and their prototypicality. In Proceedings of the Fourth Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 79-94, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020. GenericsKB: A knowledge base of generic statements. CoRK, abs/2005.00660.
Philip Blair, Yuval Merhav, and Joel Barry. 2017, Au- tomated generation of multilingual clusters for the evaluation of distributed representations. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Work- shop Track Proceedings. OpenReview.net.
5 Conclusions
Concept embeddings are often used to provide prior knowledge in applications such as (multi-label) few-shot learning. We have proposed an alterna- tive to the use of embeddings for such applications, where each concept is instead represented in terms of the properties it satisfies. Our motivation comes from the observation that concept embeddings tend to primarily capture taxonomic relatedness, which is not always sufficient. We first use bi-encoder to efficiently retrieve candidate properties for each concept, and then use joint encoder to decide which properties are satisfied. The resulting prop- erty assignments have allowed us to improve the state-of-the-art in ultra-fine entity typing.
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting Pretrained Contextualized Representa- tions via Reductions to Static Embeddings. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4758- 4781, Online. Association for Computational Lin- guistics.
Antoine Bosselut, Hannah Rashkin, Maarten Sap. Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Limitations
Jesper Brink Andersen, Mikkel Bak Bertelsen, Mikkel Hørby Schou, Manuel R. Ciosici, and Ira Assent. 2020. One of these words is not like the other: reproduction of outlier identification using non- contextual word representations. the First Workshop on Evaluation and Comparison of NLP Systems, pages 120-130, Online. Association for Computational Linguistics.
A key advantage of concept embeddings is that they can straightforwardly be used as input features to neural network models. Our representations based on shared properties may be less convenient to work with in cases where concept representations have to be manipulated by neural network. In our experiments on ultra-fine entity typing, this was not necessary as the shared properties were merely used to augment the training labels. To adapt our approach to different kinds of tasks, further work
Proceedings of
José Camacho-Collee
and Roberto Navigli. 2016. Find the word that does not belong: A framework for intrinsic evaluation of word vector representations. In Proceedings of the Ist Workshop on Evaluating Vector-Space Representations for NLP, pages 43-50,

---

Berlin, Germany, Association for Computational Lin- guistics.
Prodromos Kolyvakis, Alexandros Kalousis, and Dim- itris Kiritsis. 2018. DeepAlignment Unsupervised ontology matching with refined word vectors. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Humun Language Technologies, Volume (Long Papers), pages 787-798, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computctional Linguistics (Volume 1: Long Papers), pages 87-96, Melbourne, Australia, Association for Computational Linguistics.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
Aoxue Li. Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. 2020. Boosting few-shot learn- ing with adaptive margin loss. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1257--
Na Li, Zied Bouraoui, José Camacho- Conncdoo,, Luis Es- pinosa Anke, Qing Gu, and Steven Schockaert. 2021. Modelling general properties of nouns by selectively averaging contextualised embeddings. In Proceed- ings of the Thirtieth International Joint Conference on Artificial Intelligence. IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3850- 3856. ijeai.org.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNE) pages 55-65, Hong Kong, China, Association for Computational Linguisties.
Na Li. Zied Bouraoui, and Steven Schockaert. 2019. Ontology completion using graph convolutional net- works. In The Semantic Web ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part volume 11778 of Lecture Notes in Computer Science, pages 435-452. Springer.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? In Proceedings of the 41th Annual Meeting of the Cognitive Science Society, CogSei 2019: Creativity + Cognition + Computation, Mon- treal, Canada, July 24-27, 2019, pages 1753-1759. cognifieecienccsscceele..gg
Na Li, Zied Bouraoui, and Steven Schockaert. 2023a. Ultra-fine entity typing with prior knowledge about labels: A simple clustering based strategy. CoRR, abs/2305. 12800..
Na Li, Hanane Kteich, Zied Bouraoui, and Steven Schockaert. 2023b. Distilling semantic concept embeddings from contrastively fine-tuned language models. CORR, abs/2305.09785..
Amit Gajbhiye, Luis Espinosa- Anke, and Steven Schockaert. 2022. Modelling commonsense proper- ties using pre-trained bi-encoders. In Proceedings of the 29th International Conference on Computational Linguistics, pages 9971-3983, Gyeongju, Republic of Korea. International Commitice on Computational Linguistics.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using clectra-style pre- training with muddient-dientaanlhe embedding shar- ing. CoRR, abs/2 1.09543.
Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442-1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task- aas--adativee projection network. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1381- 1393, Online. Association for Computational Linguis- tics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907.1t
Lei Ji, Yujing Wang, Botian Shi, Dawei Zhang, Zhongyuan Wang, and Jun Yan. 2019. Microsoft concept graph: Mining semantic concepts for short text understanding. Data Intell., 13):2388-27.
16692 1692.

---

Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In ICLR.
Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-shot learning through cross-modal transfer. In 1st International Conference on Learn- ing Representations. ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Yukun Ma. Erik Cambria, and Sa Gao. 2016. Label embedding for zero-shot fine-grained named entity typing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 171-180, Osaka, Japan. The COLING 2016 Organizing Committee.
Bayu Distiawan Trisedya. Jianzhong Qi. and Rui Zhang. 2019. Entity alignment between knowledge graphs using attribute embeddings. In The Thirty-Third AAAI Conference Artificial Intelligence, AAAI 2019. The Thirty-First Innovative Applications of Artificial Intelligence Conference, TAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence. EAAI 2019, Honolulu, Hawaii, USA, January 27 -Februrry 1, 2019, pages 297-304. AAAI Press.
Lorenzo Malandri. Fabio Mercorio, Mario Mezzanzan- Taxoref: Embed- dings evaluation for ai-driven taxonomy refinement. In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, TCMLL PKDD 2021, Bilhuo, Spain, September 13-17, 2021, Proceedings, Part III, volume 12977 of Lecture Notes in Computer Science, pages 612-627. Springer.
ica, and Navid Nobani. 2021.
Nikhita Vedula, Patrick K. Nicholson, Deepak Ajwani, Sourav Dutta, Alessandra Sala, and Srinivasan Parthasarathy. 2018. Enriching taxonomies with functional domain knowledge. In The 41st Inter- national ACM SIGIR Conference on Research & De- velopment in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 745-754. ACM.
Kanishka Misra, Julia Taylor Rayz, and Allyson Et- tinger. 2022. A property induction framework for neural language models. CoRR, abs/2205 .069100.
Weiran Pan. Wei Wei, and Feida Zhu. 2022. Automatic noisy label correction for fine-grained entity typing. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4317-4323.
Ivan Vulić, Edoardo Maria Ponti, Anna Korbonen, and Goran Glavaš. 2021. LexFit: Lexical fine-tuning of pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Vol- ume 1: Long Papers), pages 5283, Online. As- sociation for Computational Linguistics.
ijcai.org.
Patrick Pantel, Eric Crestam, Arkady Borkovsky, Ana- Maria Popescu, and Vishnu Vyas. 2009. Web-scale distributional similarity and entity set expansion. In Empirical Methods in Natural Language Processing, pages 938- 947, Singapore. Association for Computational Lin- guistics.
Proceedings of the 2009 Conference
Ivan Vulié, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7222-7240, Online. Association for Computa- tional Linguistics.
Mengshi Qi. Weijian Li. Zhengyuan Yang. Yunhong Wang, and Jiebo Luo. 2019. Attentive relational net- works for mapping images to scene graphs. In IEEF Conference on Computer Vision and Pottern Recogni- tion. CVPR 2019, Long Beach, CA, USA, June 16-20. 2019, pages 3957-3966. Computer Vision Founda- tion/ / IEEE.
Chen Xing, Negar Rostamzadeh, Boris N. Oreshkin, and Pedro O. Pinhciro. 2019. Adaptive cross modal few-shot learning. In Proceedings of the Annual Con- ference on Neural Information Processing Systems. pages 4848-4858.
Dana Rubinstein, Effi Levi, Roy Schwartz, and Ari Rap- poport. 2015. How well do distributional models capture different types of semantic knowledge? In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 726- 730, Beijing, China, Association for Computational Linguistics.
Wenhan Xiong. Jiawei WU, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Imposing label-relational inductive bias for extremely fine- fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 773-784, Minneapolis, Min- nesota. Association for Computational Linguistics.
Jiaming Shen, Wenda Qiu, Jingbo Shang, Michelle Vanni, Xiang Ren, and Jiawei Han. 2020. SynSetEx- pan: An iterative framework for joint entity set expan- sion and synonym discovery. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2292-8307, Online. Association for Computational Linguistics.
Kun Yan, Zied Bouraoui, Ping Wang, Shoaib Jameel, and Steven Schockaert. 2021. Aligning visual proto- types with BERT embeddings for few-shot learning. In Proceedings of the International Conference on Multimedia Retrieval, pages 367-375.

---

Kun Yan, Chenbin Zhang, Jun Hou, Ping Wang, Zied Bouraoui, Shoaib Jameel, and Steven Schockaert. 2022. Inferring prototypes for multi-label few-shot image classification with word vector guided atten- tion. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourly Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Vir- tual Event, February 22- March 1, 2022, pages 2991- 2999. AAAI Press.
too many duplicates. This process allowed us to ob- tain a dataset with 109K unique (concepl.property) pairs.
B Collecting Concept-Property Pairs from ConceptNet
We compiled a training set of concept-property pairs for training the bi-encoder model from Con- ceptNet 5.5. ConceptNet contains broad range of relations, many of which are not relevant for our purposes (e.g. relations about events). Following Bhakthavatsalam et al. (2020), we considered the ConceptNet relations IsA, PartOf, LocatedAr and UsedFor. In addition, we also use the HasProp- erty relation, which is clearly relevant. To con- vert ConceptNet triples into concept-propers pairs, we need to choose verbalisation of the relation- ships. We explain this process by providing example for each relation type: the ConceptNet triple (apartments, PartOf, apartment buildings) is mapped to the coneept-propens pair (apartments, part of apartment buildings); the triple (seaplane, IsA, airplane) is mapped to (seaplane, airplane); the triple (airplane, Usedfor, travelling) is mapped to (airplane, used for travelling); the triple (air- plane, AtLocation, the sky) is mapped to (airplane, located in the sky).
Yunyi Zhang, Jiaming Shen, Jingbo Shang, and Jiawei Han. 2020. Empower entity set expansion via lan- guage model probing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8151-8160, Online. Association for Computational Linguistics.
A Collecting Concept-Property Pairs using ChatGPT
To obtain training data for the bi-encoder model using ChatGPT, we used the following prompt:
I am interested in knowing which prop- erties are satisfied by different concepts. I am specifically interested in properties, such as being green, being round, be- ing located in the kitchen or being used for preparing food, rather than in hyper- nyms. For instance, some examples of what I'm looking for are: 1. Sunflower, daffodil, banana are yellow 2. Guitar, banjo, mandolin are played by strum- ming or plucking strings 3. Pillow, blan- ket, comforter are soft and provide com- fort 4. Car, scooter, train have wheels 5. Tree, log, paper are made of wood 6. Study, bathroom, kitchen are located in house. Please provide me with a list of 50 such examples.
C Additional Details
The bi-encoders are trained using binary cross-entropy. We train the model for 100 epochs, using early stopping with a patience of 10 (using a 10% held-out portion of the training data for validation). We optimise the model param- eters using the AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate of 2e 6. We employ batch size of 8 and L2 weight decay of 0.1.
Bi-encoder Model
Joint encoder The joint encoders are trained us- ing binary cross-entropy. We train the model a maximum of 100 epochs with an early stopping patience of 5. A learning rate of le 5 and batch size of 32 is used to train the model. Further, we use the L2 weight decay of 0.01.
Providing number of examples as part of the prompt helped to ensure that all answers followed the same format, specifying one property and three concepts that satisfy it. One difficulty we faced is that by repeating the same prompt, after a while the model started mostly generating duplicates of concept- propeert pairs that we already collected. To address this, regularly changed the examples in the prompt. Specifically, each question asked for 50 answers, and we repeated a question with the same prompt around 10 times, after which we changed the examples, to prevent the model from generating
At the time of verifying the properties using the joint encoder, avoid introducing too much noise, property p is only assigned concept if the confidence of our DeBERTa classifier is least X, for a hyperparameter X. We consider values of X E {0.5, 0.75, 0.9}. Based on the validation split, we found that = 0.75 gave the best results.

---

Ultra-fine entity typing To train the entity en- coders for the experiments on ultra-fine entily typ- ing, we follow the methodology of Li et al. (2023a), particular, to obtain clusters of concept embed- dings, we use affinity propagation, where we select the preference value from {0.5, 0.6, 0.7, 0.8, 0.9) based on the validation split. The entity encoder is trained using the soft prompt based approach from Pan et al. (2022).
D Qualitative Analysis
Table 5 shows some of the properties found for the vocabulary from the McRae dataset. For this exper- iment, we used version of the DeBERTa model that was only fine-tuned CSLB. On each row, we compare predicted property with the most similar property from the ground truth (where simi- larity was measured in terms the Jaccard overlap the corresponding concept sets). One observa- tion is that the properties we uncover are often more specific than those in the ground truth. For instance, commonly played in orchestra is clearly more specific than used for music. What is also evident from these examples is that strategy is precision-oriented. For given property, the set of concepts that are predicted to have that property are mostly correct, but sometimes some clearly relevant concepts are missing. For instance, for used for cooking food, concepts such as blender, strainer and grater, among others, are clearly also relevant. Finally, while many of the properties that are identified are taxonomic, there are also several non-taxonomic properties, such as used for cooking food and part of firearms.
Our motivation was based on the idea that con- cept embeddings primarily capture taxonomic prop- erties. To test this idea, Table 6 shows the nearest neighbours of the concept embeddings produced by our bi-encoder. The table in particular shows the neighbours of concepts that also appear in Table 6. Compared to the properties that are found in that table, we can that the neighbours in Table 6 indeed mostly reflect taxonomic similarity. For instance, the top neighbours of dining table are different types tables.

---

Predicted property
McRae property
weupons of war: missile, crossbow, pistol, bullet, sword, shotgun, bazooka. spear. revolver, grenade, rifle, cannon, bayonet, bomb, gun, dagger
bayonet, revolver, harpoon, crossbow, pistol, bullet, grenade, shotgun, sword, bazooka, rattlesnake, rifle. machete, bomb, catapult, missile, spear, cannon, gun, dagger
used for killing:
used for cooking food: oven, pan, spatula, skillet, stove, waster, pot, microwave
lemon, cherry, fork, stove, toaster, blender, strainer, mixer, oven, spatula, ladle, clock, pot, microwave, bowl, grater. pan, colander, knife, spoon, tongs, kettle, olive, spinach, skillet, apron
used for cooking:
commonly played in orchestras: trombone, cello, violin, saxophone, trumpet, clarinet, tuba, flute
used for music: piano, banjo, trombone, cello, cell phone, harmonica, bagpipe, radio, clarinet, keyboard, tuba, whistle, accordion, rattle, laptop, stereo, trumpet. guitar, harp, drum, violin, saxophone, harpsi- chord, flute
part of firearms: muzzle, shotgun, bullet, pistol, bazooka, rifle, revolver
used for killing: bayonet, revolver, harpoon, crossbow, pistol, bullet, grenade, shotgun, sword, bazooka, rattlesnake, rifle, machete, bomb, catapult, missile, spear, cannon, gun, dagger
has wheels: cart, wagon, limousine, taxi, tricycle, bicycle, car, truck, buggy, scooter, trolley, van, mo- torcycle
sailboat, wagon, ambulance, buggy. heli- copter, helmet, scooter. motorcycle, yacht, horse, bicycle, train, ship, trailer, canoe, trolley, bus, jet, sled, rocket, pony. limousine, sleigh, truck, surfboard, submarine, saddle, shoes, unicycle, airplane, slip- pers. bridge, boat, jeep. cart, tricycle, escalator. taxi, car, camel, subway, skatchoard, elevator, bike, wheel, tractor, raft
used for transportation:
mammals commonly found in forests: rabbit, hare, carihou, coyote, squirrel, chipmunk, bear, bison, groundhog, skunk, moose, deer, fox, raccoon, cougar, beaver, elk
an animal: flea, flamingo, python, toad, penguin, rattlesnake, cock- roach, iguana, fawn, zebra, ox, crow, pig. grasshopper, gorilla, dove, shrimp, dolphin, woodpecker, buffalo, rat, bison, deer, cat, swan, ostrich, chipmunk, platypus, tortoise, beaver, lion, carihou, hutterfly, salmon, moose, clam, seagull, moth, snail, squirrel, housefly, cheetah, walrus, octopus, dog, hornet, mouse, coyote, spider, cagle, turtle, porcupine, giraffe, alligator, donkey, horse, pony, seal, groundhog, caterpillar, raccoon, cougar, elk, hare, tuna, otter, wasp, panther, bear. camel, owl, falcon, fox. frog, whale, hyena, calf, goat, duck, lobster, rabbit, elephant, hamster, goose, pigeon, squid, leopard, chicken, salamander, tiger, crab, hawk, peacock, turkey, sheep, chimp, gopher, bird, eel, crocodile, trout, rouster, cow, skunk, bull. lamb
Table 5: Comparison of shared properties which uncovered by our method, for the vocabulary of the McRae dataset, with the properties included in the ground truth of that dataset.
Concept
Nearest Neighbors
telephone number
telephone number, phone number, address, web address, house number, street address deck chair, beach chair, swing. lounge, tree house, camp bed, pleasure boat dining table, dinner table, table, coffee table, dining room, counter, desk bubble, air bubble, bubble sort, balloon, tuee, speech halloon, small ball parking lot, car park, parking garage, parking space, bus lane, alley. bicycle lane
deck chair dining table
bubble
parking lot
Table 6: Examples of nearest neighbours, in terms of cosine similarity between the embeddings obtained by the concept encoder of the model trained using ConceptNet and ChatGPT. The considered vocabulary is that of the ultra-fine entity typing dataset.


Reviews:
[{"review_id": "knCC1iHIYv", "reviewer": "Reviewer_vHKK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading of our paper. We are puzzled, however, about the discrepancy between the low scores and the minor nature of the stated “reasons to reject”.\n\nNote that we do not use ChatGPT directly. We have experimented with the use of ChatGPT for generating training data, but this step is not essential for our approach. Indeed, the model that was trained on ConceptNet performs comparably to the model that uses both ConceptNet and ChatGPT data (and better than the model that only uses the ChatGPT generated training data). Moreover, an inspection of the ChatGPT generated data reveals that it is highly accurate. In particular, we manually inspected the first 500 (concept,property) pairs and only found 6 errors, i.e. an error rate of 1.2%.\n\nThe task of ultra-fine entity typing is explained in Section 3. We did not describe it in the introduction to avoid overlap (since this is a short paper) and because we wanted to focus the introduction on the key contribution of this paper, namely the idea of identifying shared properties as an alternative to learning embeddings. \n\nWe will add a short paragraph to the introduction to make the research question and contributions more explicit, and to make the link with ultra-fine entity typing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is exciting and in time with the recent advancements in the Natural Language Processing (NLP) field.", "reasons_to_reject": "Employing ChatGPT may identify a weakness in generating concept properties (it suffers from the hallucination problem).\n\nThe ultra-fine entity tagging task was never introduced before the dedicated paragraph.\n\nThe task for which the model is tested does not reflect the paper's primary focus (shared propertied of diverse concepts). Such a discrepancy is probably due to missing research questions and main contributions in the introduction.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work introduces a strategy for identifying what different concepts have in common by analysing their shared properties. Specifically, two BERT models are fine-tuned to learn concepts and properties embedding separately. The authors employ ChatGPT to enumerate concept properties, and then a selection of data is led by finding the 50 properties using a linear classifier.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d7fsZF14QD", "reviewer": "Reviewer_k24T", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading of our paper.\n\nRegarding the choice of pre-training datasets, for the evaluation of the bi-encoder (Table 2) and the UFET model (Table 3), we have presented a full comparison of the three considered pre-training datasets: ConceptNet, ChatGPT, and the combination of both. For the best-performing joint encoder (i.e. the DeBERTa model), the main message is that pre-training does not help. This is true for all pre-training datasets, but we did not include the full analysis as this was essentially a negative result.\n\nThe joint encoder cannot be used to generate embeddings. We use the bi-encoder to generate embeddings. The embeddings are used to select candidate properties. These candidate properties are finally filtered using the joint encoder.\n\nThe bi-encoder and joint encoders that were used for the ultra-fine entity typing experiments are indeed the pre-trained models. The bi-encoders are identical to the ones that were used in the first part; the joint encoder for the UFET experiments was fine-tuned both on McRae and CSLB. \n\nWe will clarify these points and address the two suggestions from the section on “presentation improvements”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles an interesting and a relevant problem and considers concepts and properties from a different angle (not new but quite uncommon). The authors compute concept embeddings based on properties and perform both intrinsic and extrinsic evaluation. ", "reasons_to_reject": "I am leaning towards accepting the paper, here are some things how the paper could be improved for better understanding of the task and better presentation of the research to the Reader.\n\n First, it is not clear how the choice on pertaining datasets was made. Why for bi-encoders the authors experiment with both CN and ChatGPT data, then for joint encoders the authors use ConceptNet only but for the ET experiments they use ChatGPT data again.\n\nIt is also not clear, whether concept embeddings can be computed for any model (bi-enc or joint) and whether bi-encoder from the first part is the same for the Entity Typing experiments. Otherwise, which embeddings exactly are used for the ET experiments?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is devoted to a strategy for identifying what different concepts, from a potentially large concept vocabulary, have in common with others. They want to associate each concept with a set of properties and use a bi-encoder model. Then they verify the retrieved properties with a joint encoder.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I would suggest writing a list of contribution in the Intro section.\n\nI would also recommend to compute stdev for the Entity Typing experiments as numbers are very similar and it is not clear how different 50.9 and 51.1 are and how much we could benefit from your approach.\n"}}, {"review_id": "T8t3GoUGlg", "reviewer": "Reviewer_zfkC", "summary": "", "questions": "Question A: Would it make sense to refer to Analogy Solving as a related task?\n\nQuestion B: To what extent would Knowledge Graph embeddings be useful for this task?\n\nQuestion C: Why not training in the same datasets as Gajbhiye et al. (2022)?\n\nQuestion D: Would (Open)Cyc be a less noisy alternative to ConceptNet?\n\nQuestion E: Did you check whether the training data, especially that generated by ChatGPT, did not contain examples in the test?\n\nQuestion F: ChatGPT was used for generating data, but, out of curiosity, did the authors test whether it could perform the task?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading of our paper.\n\nWe will add a table to the appendix, with statistics and examples for each of the considered datasets. It is common to provide such information in the appendix, rather than the main paper, especially since the datasets involved are relatively well-known. Note that the claim that the appendix is not referred to in the text is incorrect (see footnote 2 on page 3).\n\nQuestion A: While some analogies can be solved by identifying shared properties, the task of analogy solving is mostly about measuring degrees of relational similarity. Moreover, our framework is specifically designed for cases where we have to identify commonalities among a large set of concepts (meaning that the set of concepts cannot be explicitly listed as part of a language model prompt). Analogy solving would thus not be the most natural application.\n\nQuestion B: It is not clear to us how knowledge graph embeddings could be used for our setting.\n\nQuestion C: A comparison with the pre-training dataset from Gajbhiye et al. (2022) is made in Table 2, where we can see that our pre-training datasets perform better.\n\nQuestion D: We have used ConceptNet because it is a well-known and often used dataset in NLP.  Our understanding of OpenCYC is that it is mostly focused on taxonomic relations, and would therefore be less suitable.\n\nQuestion E: Note that this point only applies to the analysis in Table 2. We found that there is indeed a small amount of overlap between the pre-training datasets and the McRae test set: around 4% of the test set is present in the largest pre-training dataset (i.e. ConceptNet + ChatGPT). However, we do not consider this overlap to be problematic, since the pre-training data was collected independent of the McRae dataset (i.e. the fact that assertions from the McRae test set appear in the pre-training dataset is simply a reflection of the fact that it contains a comprehensive set of commonsense properties). Nonetheless, for completeness, we have now also carried out an evaluation of the bi-encoder models after removing the items from the McRae test set from the pre-training datasets. After removing these items, the resulting performance stays very close to what we report in Table 2:\n\nBi-encoder with ConceptNet: 53.78,\nBi-encoder with ChatGPT: 50.37,\nBi-encoder with ConceptNet + ChatGPT: 54.97\n\nQuestion F: Using ChatGPT or any other LLM for this task would not be straightforward given the huge space of concepts we deal with in this paper. We would need a model with a context window that is large enough to enumerate the full vocabulary (i.e. around 10000 concepts in the case of the UFET dataset). Moreover, while models with large context windows now exist, based on our experience, GPT-4 already struggles with manipulating much smaller lists of concepts. For instance, asking GPT-4 to rank a list of around 100 concepts according to some criteria already leads to various issues (e.g. missing and duplicated concepts).\n\nWe will address the suggestions that were provided in the section on “presentation improvements”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well-motivated.\n\nSome interesting experiments.", "reasons_to_reject": "The paper is not self-contained. Should perhaps be a long paper?\nImportant information is either not present (e.g., description of datasets like MCG, McRae, WANLI, including examples) or only in the Appendix (e.g., creation of the training set, ChatGPT prompt), which is only referred in a footnote, not mentioning the number. The latter make it harder to follow the paper.", "questions_to_authors": "Question A: Would it make sense to refer to Analogy Solving as a related task?\n\nQuestion B: To what extent would Knowledge Graph embeddings be useful for this task?\n\nQuestion C: Why not training in the same datasets as Gajbhiye et al. (2022)?\n\nQuestion D: Would (Open)Cyc be a less noisy alternative to ConceptNet?\n\nQuestion E: Did you check whether the training data, especially that generated by ChatGPT, did not contain examples in the test?\n\nQuestion F: ChatGPT was used for generating data, but, out of curiosity, did the authors test whether it could perform the task?", "paper_topic_and_main_contributions": "The paper describes an approach for identifying properties shared by concepts in a large vocabulary.\nIt is based on a bi-encoder for retrieving properties satisfied by each concept and a joint encoder (fine-tuned DeBERTa) for deciding which properties are satisfied.\nPerformance of different configurations is computed in the \"McRae\" dataset; and when used in the downstream task of ultra-fine entity typing.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper never mentions the language it is on / applies to (English).\n\nThe abstract is too high-level: goal, experiments and contributions could be more explicit.\n\nThe Introduction does not describe the structure of the paper.\n\nTable 2: not clear where / how the models with the pretraining column empty were pretrained.\n\nline 234: consists in -> consists of"}}]

Author rebuttal:

Meta-review: The short paper deals with analysis of concepts collected in a large knowledge base. The method leverages LLMs to find common properties  for a given two concepts, which is opposed to a common practice of distilling concepts from texts. The paper presents a rare approach to concept representation, which is supported by solid experiments. 
All three reviewers are generally agree in their comments and scores. However, some crucial points are left out from the main article body and presented either in appendixes or in rebuttal responses. The authors should think carefully how to use an extra page in the final version.
Decision: Accept-Main