Licon: A Diverse, Controllable and Challenging Linguistic Concept Learning Benchmark
Authors: Shenglong Yu
Ying Zhang
wenya guo
Zhengkun Zhang
Ru Zhou
Xiaojie Yuan
Venue: EMNLP 2023 Findings (2023)
Keywords: Concept Learning
Zero-shot Learning
Linguistic Description

Abstract: Concept Learning requires learning the definition of a general category from given training examples. Most of the existing methods focus on learning concepts from images. However, the visual information cannot present abstract concepts exactly, which struggles the introduction of novel concepts related to known concepts (e.g., ‘Plant’→‘Asteroids’). In this paper, inspired by the fact that humans learn most concepts through linguistic description, we introduce Linguistic Concept Learning benchmark (Licon), where concepts in diverse forms (e.g., plain attributes, images, and text) are defined by linguistic descriptions. The difficulty to learn novel concepts can be controlled by the number of attributes or the hierarchical relationships between concepts. The diverse and controllable concepts are used to support challenging evaluation tasks, including concept classification, attribute prediction, and concept relationship recognition. In addition, we design an entailment-based concept learning method (EnC) to model the relationship among concepts. Extensive experiments demonstrate the effectiveness of EnC. The benchmark will be released to the public soon.

Licon: A Diverse, Controllable and Challenging Linguistic Concept Learning Benchmark
Shenglong Yuu Ying Zhang' Wenya Guo' Zhengkun Zhang' Ru Zhou' Xiaojie Yuan'
College of Computer Science, VCIP, TMCC, TBI Center, Nankai University, China
( nankai edu. cn (yingzhhagg.uuanxjJennann edu. cn [ nankai edu. cn
guowenya)
Abstract
Plonetr
(4)
the Astrowweical
objects in midile alse that
Concept Learning requires learning the defi- nition of general category from given train- ing examples. Most of the existing methods focus on learning concepts from images. How- ever, the visual information cannot present ab- stract concepts exactly, which struggles the in- troduction of novel concepts related to known concepts (e.g., 'Planet' "Asteroids'), In this paper, inspired by the fact that humans learn most concepts through linguistic descrip- tion, we introduce Linguistic Concept Learning benchmark (Licon), where concepts in diverse forms (e.g.. plain attributes, images and text) are defined by linguistic descriptions. The diffi- culty learn novel concepts can be controlled by the number of attributes or the hierarchical relationships between concepts. The diverse and controllable concepts used to support challenging evaluation tasks, including concept classification, attribute prediction, and concept relationship recognition. In addition, we design an entailment- based concept learning method (EnC) to model the relationship among con- cepts. Extensive experiments demonstrate the effectiveness of EnC. The benchmark will be released to the public
Novel:
otar Aiteronda?
Linguistic Description: Asterouds are similar Plancis but in
Astronomicaa object
Queries
[C (Classification)
Asteroid
Plant
(b)
Q: Putfix the
(Paller' deffnitton)) fr Astovidd? A:yus
[Atrribute]
Q How's the size of Asteronds? A: mull size Small
Middle
[Relationship)
size
stze
Q: What is relatinnsup between between..sterrde Asteroids and Planet? A: stbling concept
Concept
Figure 1: An example concept learning. Based on the known concept 'Planets', it is difficult recognize the novel concept, 'Asteroids', according to several images. However, a simple linguistic description helps to associate 'Asteroids' with known 'Planets'. 'hen concept learner can answer questions about Asteroids.
shapes, while abstract concepts cannot exactly pre- sented in the visual contents. As shown in Figure (a), due to the similar appearance, it is difficult recognize the relationship between 'Planets' and 'Asteroids" according to the images separately.
By contrast, humans can learn various concepts through language (Lake et al., 2015), especially in academic areas such as physics and psychology. Compared to images, language can precisely de- scribe the complex relationship between abstract concepts. As shown in Figure 1 (b), the attributes of 'Asteroids' can be clearly described as 'As- teroids are similar to Planets but in small size'. Inspired by this fact, propose to associate novel concepts with known ones through linguistic de- scriptions, called Linguistic Concept Learning. As shown Figure 2 (b), novel concepts are defined by linguistic descriptions rather than images. The linguistic descriptions formulate various relation- ships between novel concepts and known concepts for hinglemodda (Figure 2 (c)) or multi-modal (Fig- ure (b)), leading more challenging task.
1 Introduction
Concept Learning aims learn the definition of a general category from given information (Han et al., 2019; Ye et al., 2020a). This task formulates the innate human ability of generalization in terms of concepts, i..., the ability to quickly learn novel concepts from few information based on known concepts. Prior work mainly focuses on Visual Concept Learning (Mao al., 2019; Han al., 2019; Li et al., 2020; Mei al., 2022), discovering concepts from the visual information (e.g., images). However, the images can only display concrete in- formation via limited examples, such as colors and
*Corresponding author. https://gihubb.oo/yuuhhngggg i/Lionn
In this paper, we introduce linguistic concept

---

Concept learning
Tinguietic Cuncept hurning
to capture the relationships (i.e., entailment and contradiction) between concepts.
red
This the red object.
Step
polygon right ingles and four edges.
cubl
This is the cube ebject
onncept
We conduct extensive experiments to verify the effectiveness EnC. Additionally, we also explore the challenges of linguistic concept learning by designing various experiments.
learning
Red cobe the object
Square is special cane of ressnggl with all equil edges.
Stup
Novel
object rafe ebject
cumoage learning
Q: is thara rod nafe?? A:
Qr thar red cabe? jor
Q Now many eiger doocs aquure have?
Step 3. Exalustion
Related work
(N)
Concept learning. Concept learning focus on the ability of learning concept through diverse set of inputs: by looking at images (Mao et al., 2019; Han al., 2019:Li et al., 2020; Mei al., 2022). reading sentences describing the properties of ob- jects (Dan et al., : et al., 2019; et al.. 2020a); or about the relationships between con- vious concept learning datasets (Lin et al., 2014;
Figure Illustration of visual concept learning and linguistic concept learning. Concept learning process typically consists of three steps: Known concept learn- ing, Novel concept learning, and Evaluation.
learning benchmark (Licon). Compared to pre-
cepts (Han et al., 2019). However, most of the works focus on visual concept learning, i.e., the process of learning novel concepts relies on vi- sual information. We differ from these works by transforming the learning process rely only on linguistic descriptions, which allows us to repre- sent novel concepts with wider distribution in a more flexible way.
Johnson et al., 2017; Hudson and Manning, 2019). Licon has the following properties: 1) Diverse: Known concepts are presented in diverse forms, including plain attributes, concrete visual images and abstract textual definitions, named Licon-A, Licon-I, and Licon-T, respectively. The construc- tion methods are also diverse, which are synthetic about concrete concepts (Licon-A and Licon-I), or artificially designed about real-world abstract con- cepts (Licon-T). 2) Controllable: The synthetic datasets, Licon-A and Licon-l, allow us to control the difficulty of synthetic concepts by changing the number of attributes relationships among them, which can support testing the effectiveness of con- cept learners in various scenarios. 3) Challeng- ing: Licon requires concept learners to recognize comprehensive information about novel concepts, including categories, attribute values and relation- ships with other concepts. The diverse concept forms, especially the real-word datasets (Licon- T), and complete evaluation tasks pose great chal- lenges for linguistic concept learning.
Zero-shot learning, Recently, due to the develop- ment of Large pre-trained language models (LMs) (Devlin et al., 201 2018:Liu et al., 2019:Raffel et al., 2020), many works (Zhang al., 2019;Yin et al., 2019;Ye al., 2020b;Sanh al., 2021:Mishra et al., 2021) focus on zero-shot learning, which aims to train model that can classify objects of unseen classes via transferring knowledge obtained from other seen classes with the help of seman- tic information (Pourpanah et al., 2022). While the experiment setup in our task meets the defini- tions of zero-shot learning, our work is significantly different from these works 1) Our benchmark comprises large set of concepts for learning from linguistic description as opposed to working on a limited set tasks in prior work. 2) Our bench- mark focuses on concept learning, and provides a comprehensive evaluation, including categories, attribute values and relationship about concepts, rather than just classification.
In addition, we propose an entailment-basee lin- guistic concept learning method (EnC), which uses box embeddings and GNN to explicitly model the relationships between concepts. Further, EnC lever- ages concept-level entailment loss optimize the model. EnC achieves the best performance on both synthetic and real-world datasets, severing as a strong baseline for facilitating future work.
Dataset Construction
In this section, describe the collection details of our proposed benchmark, Licon. We divided Licon into three datasets, namely Licon-A (syn- thetic, plain attribute), Licon-I (synthetic, image), and Licon-T T freal-world, text) according to the dif- ferent construction methods and data sources.
In summary, our contributions are:
We are the first to propose the challenging task, Linguistic Concept Learning, and collect a diverse, controllable and challenging benchmark, Licon.
We develop EnC, an entailment-basse model

---

remaining concepts as well their descriptions are split proportionally into the training set and the test set. In sum, we generate sub-datasets with a different learning difficulties and description complexity as Licon-A. For more details, please refer to Appendix A.1.
Langilatic Dvsrription
buckkient Demustii buight
beckbous equals to THIEN It with the comettic equal no
Ia
EX
C. Cia the wish the beight las than
Lieen-A
Ceneept Defiaition
Längubtic Description
LP the coier espaes rad THEN
D. sise cunda to lazs THEN D
nol
opas THIN
Tange
lanas cabs
Licon=1
Alape
coniinattis
3.2 Licon-I
Figure 3: The hierarchical structures, definitions and descriptions of concepts in the synthetic datasets of the proposed Licon benchmark, i.e., Licon-A and Licon-I.
To further explore the linguistic concept learning under multi-modal scenarios, we collect Licon-I. I. In this dataset, the concepts are the kinds of object present in the image, and the example is shown in Figure 11. As used in Johnson et al. (2017), we choose color, size, shape and material as the ba- sic attributes with their own values. As shown in Figure 3, in contrast to Licon-A, we use combina- tion rather than attachment to generate concepts in Licon-I. Then we also generate two types of ques- tions for each concept and split the concepts in the same way as Licon-A.
3.1 Licon-A
To begin with, we collect Licon-A, a single-modal synthetic dataset with controllable difficulty. In Licon-A, the concepts are the fabricated species of the bird, which are represented by plain at- tributes and have strict hierarchical relationships. We mainly focus on the process of learning novel concepts based on known concepts, and the dataset has universality the field of concept learning and is not specific to the narrow domain of birds. The example of Licon-A is shown in Figure 10.
In Licon-I, we generate 2 sub-datasets with dif- ferent concept hierarchies, including 2-layers and 3-layers, which means a concept can be combined by up two or three base concepts, as shown in Figure 3. More details in Appendix A.2.
We generate Licon-A by first selecting attributes from the pre-defined schema used (Menon et al., 2022). As shown in Figure 3, concepts are hierar- chically generated based on the selected attributes. Specifically, we refer to the concepts that do not depend other concepts as base concepts, such concept A in Licon-A and concepts A, D, E in Licon-l of Figure 3. Additional concepts are gener- ated by attaching new constraint the definitions of existing concepts. We use the number of at- tributes (including 4 and 8) to control the difficulty of learning a concept.
3.3 Licon-T
We construct Licon-T by collecting linguistic con- cept learning questions from the National Civil Scr- vice Examination of China, which give candidates linguistic description of the abstract concept and ask them to answer questions about that concept. We collect raw data at official websites? and re- move questions containing images or charts. We then remove questions that require external knowl- edge answer. Finally, we remove duplicated questions and the resulting dataset contains 1008 pairs. Since the original dataset written in Chi- nese, we hired professional English speakers to translate the dataset manually. And we will also re- lease the Chinese version of that dataset. For more details, please refer Appendix A.3.
Afterwards, the descriptions are generated based on the definitions of the concepts. As mentioned in Menon et al. (2022), we use the following types of descriptions to control the linguistic complexity (v; denotes 7th statement that describes the value of an attribute, denotes the name of concept): 1) Simple: IF THEN c; 2) Conjunctive: IF AND 02 THEN c; 3) Disjunetive: IF 11 OR U/2 THEN c. Next, we generate two types of questions for each concept, one focusing on using the learned con- cepts to identify an object, and the other focusing on the concept itself, including the attribute and the relationship between the concepts.
Dataset analysis
In this section, we describe the statistics and data distributions of Licon.
Dataset Statistics: Table 1 shows the statistics of Licon. Compared Licon-A and Licon-I, Licon-T has the most concepts and the fewest questions for each concept to learn, which indicates its difficulty.
Finally, we split the concepts into a training set and a test set. All the base concepts are included into the training set as known concepts, while the
Pht / /wwww chinagwy.

---

Licon-A
Licon-I
Licon-T
where a low score implies complex sentences. The result is shown in Figure 5 (a). The Licon-A is the easiest to read terms of complexity, with scores ranging from 60 (easy to understand for 13-year-old's to 100 (very easy to read for 11- ycar-old's) s).. Licon-I is similar to Licon-A, but a bit harder. Licon-T is the most complex, its score is concentrated in about 20 (very difficult read), including negative values (extremely difficult to read).
Train concepts
90
280
681
Test concepts
18
56
Avg.# Q/Concept
36.05
39.68
Classification
28.05
30.22
Attribute
2
4.34
Relationship
6
7.34
Vocabulary
207
713
9448
Avg.8 lokens/Q
11.79
9,75
49.83
Avg.l tokens/Des
22.24
13.15
13.1
Table I: Statisties of datasets our proposed Licon benchmark. 'Q' and 'Des" denotes question and de- scription, respectively. 'Classification", 'Attribute' and 'Relationship' are the types of the corresponding ques- tions.
We analyze the topic distribu- tion (manually labeled) of Licon-T. As shown Figure 5 (b), the concepts in our dataset are widely distributed across multiple domains, which demon- strates the practicality of our real-world dataset. The top three fields psychology, sociology, and economics and the concepts are the terminology used in these fields.
Topic distribution:
Besides, in real-world datasets, the reasoning steps are complex and difficult to categorize, combining the use of concepts for classification, attributes and relationships of concepts. So the synthetic tasks, Licon-A and Licon-I are created to evaluate the linguistic concept learning task in controlled set- tings with straightforward descriptions and targeted questions for evaluation. This approach allows us to evaluate the learning ability of the model de- tail under different settings.
Analysis of difficulty: The difficulty of the datasets, namely Licon-A, Licon-l, and Licon-T, varies from easy to hard. Licon-A is the easiest with clear attribute set. In contrast, Licon-T is the hardest as it contains the most concepts and the least amount of data to learn each concept. The description of Licon-T is particularly challenging to understand, and it also requires strong reasoning abilities to grasp the given options effectively.
Characteristics of Licon-A: Figure 4 (a) shows the distribution of Licon-A. Licon-A is divided into six sub-datasets based on the number of at- tributes and the type of description. The description types include simple, conjunctive and disjunctive descriptions, which are used to control the linguis- tic complexity of descriptions. And the number of attributes is used to control the difficulty of learn- ing concepts through objects. In addition, to diver- sify descriptions and make Licon-A more similar real-world datasets, proportionally append negation (20%) and numerical judgments (40%) as linguistic variations of the descriptions. The examples are shown in Figure 4 (a).
5 Experiment Setup and Models
5.1 Formal Definition
The dataset D for linguistic concept learn- ing is structured
(c1, (2,42,22)
(c1, Cor: 0mg) ... (cn, Imn am,) )). where (cs, 4s as) is an instance form D with e denot- ing the linguistic description of the concept, 9s and Oy denoting the concept related question and its answer. It is worth noting that each concept de- scription C. is related to multiple questions.
Characteristics of Licon-l: Figure 4 (b) shows the distribution of Licon-I. Licon-I is divided into two sub-datasets based on different concept hierarchies. In Licon-I-1, we generate novel concept by com- bining the features of two base concepts, e.g., red object cube = red cube. And in Licon-I-2, we can combine three base concepts, e.g., red cube in middle size. Objects in Licon-1-2 require more at- tributes (average 2.53) to be judged when classified compared to Licon-I-1 (average 1.81).
Our goal is to train model capable of learn- ing novel concepts from linguistic descriptions that explain novel concepts in light of known con- cepts. We train our model over set of seen con- cepts (ci, C2,..., Ck and evaluate generalization to the novel concepts (Ck+1,..., C..) At evaluation, the model is evaluated by the accuracy of answer- ing multiple choice questions related to the novel concepts. Since the model never sees the novel concepts during training, our experiments per- formed with zero-shot settings.
Reading complexity: We analyze the reading com- plexity of Licon by applying Flesch reading ease
Besides, for synthetic datasets at evaluation, we divide the results into two categories 77 and 72,

---

Description Types
Variations Description
Simple
Conjunctive
Disjunctive
Des:
Hierarchy of concepts
the equal
Namber of Attributes
CFC
IFC1 AND
IF CI OR C2
THEN Concept. A
THENL.
THENL.
THEN I.
Des with Negation:
backbone not equal
Dataset Name
Licon-1-1
Licon-1-2
Licon-A-1
Licon-A-3
Licon-A-5
THEN Concept A
Number of concepts
250
Des with Number: length Concept. A
Questions type
77%8%19%%
67%/13%%/0%
Licon-A-2
Licon-A-4
Licon-A-6
Avg. #Attribute
1.81
2,53
(b)
Figure 4: Distribution of sub-datasets in Licon-A and Licon-I. In (b), the 'questions type' is the percentage of questions about classffcatio/aatrruuccclllloooss And AAttrbute denotes the number of attributes used classify object.
P1ychllog:14%
Logles%
Concept Embedding
<(CLS)
A-cruser
Auafoor
Recoaeer
Beaiffar
ebont
statiilat)
0.
0.1
Sckencne7%
Natural
B
-50
Fleach reading 0500
50
100
Marketing
Methodi4%
Other395%
Á
Mannggmment35%
Embedding
GNNTlayers
Figure 5: Statistics Distribution of Licon. (a) the reading complexity distribution Licon; (b) the topic distribu- tion of Licon-T.
Concept Embedding Layer
Input
[CLS]
Concepu
Cinceyeli
witti
arriibuie
Figure 6: Architecture of proposed method. We transform the embeddings of concepts in descriptions to box embeddings further optimize GNN layers.
which are designed for evaluating the ability of models to identify category (the former) and other information, including attributes and relationships, of concepts (the latter).
format of inputs by 'Questions-as- -ext' 'QaT'.
5.2 Baseline models
We use RoBERTa in two ways, the former we concatenate the concept description and QaT as the input of RoBERTa, noted as RoBERTa.C. And the latter, we separately take the concept descrip- tion and QaT as input, and compute the final proba- bility by dotting the outputs, noted RoBERTa. D. More details can be found in Appendix B.
For our baselines, pre-trained RoBERTa (Liu et al., 2019) model is utilized to better understand the linguistic description. The input of RoBERTa is text, so we need to encode the different modal- ities as text. For Licon-A, we follow the Menon et al. (2022), encoding the plain attributes as text sequence. We linearize it sequence of attribute- value pairs, separated by [SEP] tokens. For Licon-I, we use the pre-trained models, such as ResNet-10 101 (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009) or CLIP (Radford et al., 2021), to en- code the images as embeddings. Then, the embed- dings are appended as the prefix to the questions in the same way as Li and Liang (2021). For Licon-T, the data is all textual, we don't need additional ma- nipulation. After transforming all features into text, we combine these features with questions and can- didates as the inputs of RoBERTa. Then, RoBERTa is able to answer multiple-chotl choice questions, ignor- ing different modalities. And We refer to the textual
5.3 EnC
To better associate the knowledge of known con- cepts with novel concepts, we develop EnC, which uses box embeddings and graph neural networks (GNN) (Hamilton et al., 2017) to model the re- lationship between concepts and use the concept- level entailment loss to optimize the model.
Figure 6 shows the overview of our proposed entailment-based concept learner, EnC. The funda- mental idea behind EnC is generating box embed- dings as concept embeddings to maintain hierar- chical relationships between concepts. The GNN in EnC is used to realize knowledge interaction

---

1-T
2-T
3-T
4-T
5-T
6-T
Avg.#T
Avg.#T1
Avg.#T2
RoBERTa.C
61.84
61.84
65.99
62.82
65.45
67.11
64.18
63.19
67.94
ROBERTa.D
60.53
60.53
72.11
55.13
68.18
61.84
63.05
61.83
67.99
RoBERTa-MNLI
67.11
67.11
78.23
64.10
70.50
67.76
69.14
68.98
70.02
EnC
78.74
77.63
85.71
75.64
77.70
77.63
78.84
80.86
72.91
Table 2: Experimental results Licon-A. 'T' the accuracy on all questions, and 'TI', 'T2' is the accuracy
corresponding questions. "k-T" means the accuracy of all questions on A-k dataset. And 'Avg.tX" is the average accuracy of "X" type questions on all sub-datasets.
1-11
1-12
2-1
2-11
2-12
Avg.#1
Avg.#11
Avg.#12 #12
CNN-LSTM
45.83
48.37
32,77
46.52
50.47
34.31
46.18
49.42
33,54
CI.P-ROBERTa.
53.08
47.22
83.19
58.99
49.72
87.68
56.04
48.47
85.44
ResNet-RoBERTE RoBERTa.D
53.49
48.04
81.51
60.21
51.04
88,56
56.85
49.54
74.39
CLIP-RoBERIaC
50.89
45.42
78,99
56.99
47.06
87.68
53.94
46.24
83.33
ResNet-RoBERTa.C
51.03
45.26
80.67
54.48
44,51
85.34
52.76
44.89
83.01
ResNet-Ene
63.61
59.31
85.71
64.31
56.17
89.44
63.96
57.74
87.58
Table 3: Experimental results on Licon-I: "T" is the accuracy on all questions, and 'T1', 'T2' is the accuracy on corresponding questions. '1-X', '2-X' means the data on I-1 and I-2.
EN
ZH
GNN(Offeo, G). In other words, the center of known concept affects the center of novel concept and so does the offset.
RoBERTa.D
28.71
29.21
RoBERTA.C
32.67
30.19
ROBERTA(TE).C
33.17
30.19
We also propose concept-level entailment as loss to better optimize the embeddings of concepts. For novel concept, we take it as hypothesis, and for its hypernym (parent concept in G. i.e, expand from that known concept) and cohypernym (sibling con- cept in G, i.e, expand from a same known concept), we treat them as different premises. For a premise, we decide whether the premise contains the hypoth- esis (entailment) or contradicts it (contradiction). For hypernym The novel concept with box em- beddings cc. should entail its hypernym с₁ with box embeddings en- And then utilize the feature of box embedding to calculate the overlap values of two boxes e, and cj, which we note le- For cohypernym The novel concept e with box embeddings should contradict its cohypernym Co with box embeddings co. in order to distinguish between the two. We also compute the overlap values of boxes e, and eo. which we note as lo.
EnC
34.15
34.15
Table 4: Experimental results on Licon-T
between concepts. The concept-level entailment is used to train box embedding to represent hierarchi- cal relationships.
Firstly, we extend box embedding for knowl- edge bases (Vilnis et al., 2018) to linguistic con- cept learning task. Each concept is represented as a box in high-dimensional space, i.e., the em- bedding for each concept is tuple of two vectors Co (Cene: fc)- Cen, is the center of the box and Of fe is the offset in box. Both are in Rd. where d the embedding dimension. By this method, each concept is mapped to box in the high-dimensional space, whose center is Cen, with the edge length of * Offe-
In experiments, we build our model based on RoBERTa. We combine the concept description and QaT as the input and transform the embeddings of concepts in description to box embeddings using the method described above. Finally, we combine I and with cross-ent cross-entropy loss to form the final loss L:
Then, to model the hierarchy between concepts, we apply GNN on the box embeddings of concepts. We use a designed semantic parser to parse the lin- guistic description and construct a relational graph G for the known concepts. And for novel concept C, we append it as node in G, and initialize with embeddings e.0 (Ceng, Offe), which is the average embedding of known nodes. Then, we input it to the GNN to get e.1 = (Cen,I, Off.e)..
L= - (le 1o).
And we train EnC using the final loss. More details will be described in Appendix B.
where Cena GNN(CCeno) G) and Offa =

---

6
Results
level reasoning abilities. We discuss more details in Appendix C.
6.1 Results Licon-A
6.4 Ablation experiments
We evaluate the performance of models on Licon- A and the results are summarized in Table 2. For Licon-A, we also use RoBERTa pre-trained on MNLI corpus (Williams et al., 2017) as baseline. general, EnC outperforms all baselines with an accuracy improvement of about 10% under diverse experimental settings, demonstrating the effective- ness of our concept learning approach. Besides, the EnC improves the performance both on 7/ and 72 questions. In addition, by comparing the results of sub-datasets under the same description type, i.e. A-I A-2, A-3 A-4 and A-5 to A-6, we find that fewer attributes can lead higher accuracy.
We perform ablation experiments on EnC, includ- ing the following variants: EnC w/o Box only in- cludes GNN to affect the normal embedding of the concept. EnC w/o GNN transforms the normal embeddings in RoBERTa to box embeddings with- out GNN. EnC w/o Loss has the same structure as EnC, but it excludes the external loss to opti- mize. The results are shown in Table 5, compared to Enc, there is a significant drop in accuracy for other variants, which indicates the effectiveness of our model. In addition, we find that using box em- bedding brings the greatest improvement for EnC.
6.2 Results on Licon-I
Discussion
Our main results on Licon-I are summarized in Table 3. We can draw the following conclusions according to the results. 1) EnC with ResNet as the encoder outperforms the baselines on all types of problems, with an average improvement of 10%. 2) Compared to Licon-A, model on the Licon-I is significantly less accurate, especially for T/ ques- tions, which indicates the difficulty of multi-modal concept learning and application. 3) Different dis- tribution of question types (shown in Figure 4) af- fects the focus of model learning, for example, in the pre-trained models, 72 questions perform better in Licon-1-2 than in Licon-1-1, while the opposite is true for T7. 4) The complexity of concepts hi- erarchy (compared Licon-I-2 to Licon-I-1), which mainly affects the difficulty of object classification (TI questions), may be caused by 'Avg. Ivg.#Attrr #Attribute' shown in Figure
7.1 The Challenge of Negation
Negative sentences are a serious challenge in lan- guage understanding. With negation, sentence can have exact opposite meaning. To identify the challenge of negation in linguistic description, we perform experiments explore the influence of negation. For each subset in Licon-A, we generate two versions, the former consisting of non nog-negative sentences and the latter consisting of all negative sentences. We evaluate EnC on two datasets and show the relative gain of the non-negative version compared to the negative version in Figure 7.
Our results indicate that the negation reduces the performance in all conditions. Besides, there exists similar distribution among the same at- tribute numbers, which indicates the influence of negation is related to the difficulty of learning con- cepts through items. And when an object has fewer attributes (A-1, A-3, A-5), the drop is mainly com- posed in T/ questions, and mainly in 72 questions when an object has more attributes (A-2, A-4, A-6),
6.3 Results Licon-T
Finally, we evaluate the proposed method on Licon- T, which is more complex than synthetic datasets. We also test the performance of the model pre- trained on textual entailment corpus. For English, we use MNLI (Williams al., 2017). And for Chinese, we use CCL2018 4, Our main results are shown in Table 4. Overall, EnC' achieves the high- est accuracy compared to the baselines. However, the performance is still unsatisfactory. The primary reason is the difficulty of understanding complex textual descriptions and the requirement for high-
7.2 Visualization of box embedding
We visualize the box embeddings of concepts in Figure Since the different dimensions in a box embedding are independent, we cannot use t-SNE to reduce the dimension, which will lead to changes in the structure of the box embeddings. Therefore, randomly choose two dimensions of the box embeddings, then draw the edges.
As shown Figure 8, our proposed loss helps to construct the hicrarchical relationship between concepts. In (a), the embedding of the concept A
hhtp://wwwwcippt http://wwww....ll..og//sti/C/CCC 8/acc/ 8/call evaluation.
htmlitask3 cvaluation.hmlltttkk333

---

1-T
2-T
3-T
4-T
5-T
6-T
Avg.#T
Avg.#T1
Avg.#12
EnC
78.74
77.63
85.71
75.64
77.70
77.63
78.84
80.86
72.91
w/o Box
71.05
71.05
83.67
70.51
71.94
74.34
73.76
74.24
72.45
w/o GNN
76.32
73.68
82.99
69.23
76.26
74.34
75.47
76.69
71.47
w/o Loss
77.63
76.32
84.35
73.08
76.36
76.32
77.34
78.72
72.39
Table 5: Results ablation study on Licon-A.
Simple Des
Simple Des More Attributes
Conjunctive Des Loss Attributes
Conjunctive Des More Attributes
Disjunctive Des Less Attributes
Disjunctive Des More Attributes
60
50
Gain (%)
20
10
A-1
A-3
A-5
Figure 7: Analysis of the effect of negation. The bars indicate the relative performance gain the non-negatn non-negative version the negative version.
Type:
12
90
Ovarlac
A-
A-CC
CSA
Mutuelly Exclusve [6e+C]
Mutually Exclusive
0
711
[8+G]]
(5)
LicunA
[b] Lionn
80
Figure 8: Visualization of box embeddings Licon- A and Licon-I. The rectangle is the embedding of the concept corresponding color.
60
2
10
k-shot
k-shol
(a] A-1
(b) 1-1
Figure 9: Few-shot Learning Performance on Licon-A-I and Licon-I-1 datasets.
simultaneously covers the embeddings of children. While the embeddings of the cohypernym concepts (B and C) are mutually exclusive. In (b), the con- cept A is combined by two other concepts and this relationship is also shown in the box embeddings.
especially when the learning process is difficult (the results 1-1 vs A-1). 2) In general, one-shot learn- ing brings the greatest improvement in all types of problems for both datasets. 3) Compared to base- lines, EnC achieves one-shot performance on A-1 and better than five-shot performance on I-1, which indicates the effective of EnC.
7.3 Effective of few-shot
Our experiments are performed with zero-shot set- tings. To further explore the few-shot settings of linguistic concept learning, evaluate the RoBerta.D and ResNet plus RoBerta. model on A-1 and datasets, respectively. For each novel concept, we provide external questions in the training set as k-shot conditions.
Conclusion
In this paper, we introduce Licon, diverse, con- trollable and challenging benchmark for linguistic concept learning, involving plain attributes, im- ages and text. Besides, we develop the EnC, an entailment-basee model to capture the hierarchical relationships between concepts. Extensive experi- ments demonstrate the effectiveness EnC.
The results are shown in Figure And the hor- izontal line indicates the performance of EnC on that dataset. We draw the following conclusions. I) Few-shot learning, even one-shot learning, sig- nificantly improves the performance of the model,

---

Limitations
archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece.
The results on Ticon-T are unsatisfactory. One rea- son is the complexity of linguistic descriptions in the real world. As shown in Figure (a), some descriptions are extremely difficult to read, which indicates that we need to further improve the un- derstanding ability of the pre-trained model in the future.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Representation learning graphs: Methods and applications. arXiv preprint arXiv: 770.005844
We plan introduce curriculum learning en- hance the understanding ability of complex con- cepts in the future. As mentioned in (Li et al., 2020), the process of human learning involves start- ing small and gradually building knowledge. Simi- lar to human learning, we'll 'l train the model to learn from easy concepts to complex ones. The planned curriculum includes three stages: (1) simple lin- guistic concepts, (2) complex options for simple concepts, and (3) more complex concepts. With this training method, we expect better performance on Licon-T. However, limited by the cost of manu- ally annotating the difficulty degree of the concepts, it would require extra significant resources to im- plement this approach. So we plan explore this method in the future.
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. 2019. Visual concept-metaconeep) learning. Advances in Neural Information Processing Systems, 32.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7700-6709,
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei. C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910.
Ethics Statement
Since the original concept concepts are collected from the National Civil Service Examination of China, there is private information (e.g., phone, email, postcode and location) in data collection of the real-world dataset, Licon-T. Besides, EnC is trained to recognize the characteristics of the defined concepts, and the trained model will not produce any sensitive information.
Thomas N Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. arXiv preprint 1rXiv:1609.02900 029077.
Brenden M Lake, Ruslan Salakhutdinov. and Joshua B Tenenbaum. 2015. Human-level concept learning through prohabilistic program induction. Science, (1M6626)::332-1333
Acknowledgments
The research is supported by the National Natural Science Foundation of China (No. 62272250), the Natural Science Foundation of Tianjin, China (No. 22JCJQJC00150, 22JCQNJC01580), the Funda- mental Research Funds for the Central Universities (No. 63231149).
Qing Li, Siyuan Huang, Yining Hong. and Song-Chun Zhu. 2020. A competence-awee aware curriculum for vi- sual concepts learning via question answering. In Computer Vision-F.CC 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part 11 16, pages 141-157 Springer.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001
References
Soham Dan, Osbert Bastani, and Dan Roth. 2021. Few- shot novel concept learning for semantic parsing. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2064-2075.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects context. In European confer- ence on computer vision, pages 740-755. Springer.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier-

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 7ettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1r0Xiv1907.11 1669..
Ziqi Wang. Yujia Qin, Wenxuan Zhou, Jun Yan, Qinyuan Ye, Leonardo Neves, Zhiyuan Liu, and Xi- ang Ren. 2019. Learning from explanations with neu- ral execution tree. arXiv preprint arXiv: /9l1:991... 1.00352.
Adina Williams, Nikita Nangia, and Samuel Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.0544
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. 2019. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint urXiv: 111994422528 12584.
Qinyuan Ye, Xiao Huang, Elizabeth Boschee, and Xi- ang Ren. 2020a. Teaching machine comprehension with compositional explanations. arXiv preprint arXiv:2005.0060 00800.
Lingjie Mei, Jiayuan Mao, Ziqi Wang. Chuang Gan, and Joshua Tenenbaum. 2022. Falcom: Fast visual concept learning by integrating images, linguistic descriptions, and conceptual relations. arXiv preprint arXiv:2203. 166339
Zhiquan Ye, Yuxia Geng, Jiaoyan Chen, Jingmin Chen, Xiaoxiao Xu, SuHang Zheng, Feng Wang, Jun Zhang, and Huajun Chen. 2020b. Zero-shot text classifica- tion reinforced self-training. the 58th Annual Meeting of the Association for Com- putational Linguistics, pages
Rakesh R Menon, Sayan Ghosh, and Shashank Srivas- tava. 2022. Clues: henchmark learning clas- sifiers using natural language explanations. arXiv preprint 22Xhv:2004.77142.
Proceedings of
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. arXiv preprint arXiv:1909.0016.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Cross-task generaliza- tion via natural language crowdsourcing instructions. arXiv preprint 104.073104
Jingging Zhang, Piyawat Lertvittayakumoon, and Yike Guo. 2019. Integrating semantic knowledge tackle sero-shot text classification. arXiv preprint arXiv: 12626.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, perforrrfom deep learning library. Advances neural information processing systems, 32.
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. 2022. review of generalized zero-shot learning methods. IEEE transactions on pattern analysis and machine intelligence.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training ables zero-shot task generalization. arXiv preprint 10.08277
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew Me- Callum. 2018. Probabilistic embedding of knowl- edge graphs with box lattice measures. arXiv preprint arxiv: 1805.000000000 006.06627.

---

A
Additional details of creating Licon
A.1.3 Creating linguistic deseriptions
In this section, we will discuss the details of creat- ing Licon.
We use a lemplate-based approach to convert the definition into linguistic description.
We convert the operator in definition as: "equal to'
A.1 Licon-A
Licon-A is dataset consisting of text data. And we generate Licon-A by the following steps: I) Select the table schema and group the attributes into different groups. 2) Generate definitions and linguistic descriptions of the concepts. 3) Generate questions for the concepts and divide the concepts into training and test sets. And the example is shown in Figure 10.
greater than'
= greater than or equal to"
'lesser than'
<= + 'lesser than or equal to"
! "not equal to'
'not greater than'
'not lesser than'
And for the two types of concepts, the linguistic description is different as shown in Figure 3. One is IF THEN and the other is is the 02 with v. denotes conditional statement that describes the value of an attribute, c denotes the name of concept)
A.1.1 Table schema
For the table schema, which we used to select at- tributes, follow the settings in Menon et al. (2022). For all the attributes in schema, they define a fixed domain from which values for that attribute can be sampled. And we use the Species of bird as our table schema, which shown in Figure 13, and the target to classify a bird into a particular species based on attributes. We con- sider the species of birds as concepts, which are judged by the values of attributes. In table schema, the attributes be either of types categorical or numeral.
A.1.4 Generating questions for concept
We generate types of questions for each con- cept, one about the classification and the other re- lated to the concept itself, including attribute and relationship.
For the classification questions, we randomly generate objects based on attributes and assign la- bels using definitions. And in order to avoid too simple questions, at most one of the attributes of an object does not meet the definition.
is worth noting that our task focuses on explor- ing the ability of the model to learn novel concepts through linguistic descriptions, so unlike Menon et al. (2022), we use only the first table and divide the attributes into groups. Then, we use attributes within groups, which are non-overlapping, to de- fine concepts.
For questions about attributes, we generate ques- tions for each attribute. The options for questions are selected from the fixed domain in the table schema. And if there is AND or OR in the def- inition, we will fix one attribute and require the other.
A.1.2 Creating the definition of concept
To generate the linguistic descriptions and ques- tions for concepts, we first generate the definition concept, as shown in Figure For concept, we classify an object by making a judgment on each attribute.
For questions about the relationship between concepts, we randomly choose other concepts, in- cluding hypernym and cohypernym concepts, and determine their relationships based on their defini- tions.
Then, we generate two types of concepts for Licon-A, one for concepts that do not depend on other concepts, called base concepts and the other for concepts that are generated by attaching exter- nal judgments on attributes to existing concepts. In addition, the definition also varies on the fol- lowing axes:
Then, we split these concepts into training set and a test set and form a sub-dataset.
A.1.5 Different sub-datasets
Finally, create 6 sub-datasets by varying along the following axes:
Hierarchical relationship: H
{2,3}
Structure of description: $ ('simple', 'con- junction', 'disjunction')
Negation: n'no negation', with negation') We apply (0.8.0.2) as the ratio the two axes.
Number of attributes: N= = (4,8)

---

Licon-A
Concepi Examples
the backlone THEN Coceept.A
Step Known concept learning
[Clnssification]
[Attribute]
[Relationshipl The Cuncept Conage Conagpi
backbone
no
demosiic
you
socds
ams
yes
has backbons
meess definition Concept.
A with the domostic equal to
This icm satisfy the definition of A
Concept exlends Concept
Linguistic description
Step Novel concapt learning
Concept B the Concegs A with the domestic equal
Test
[Type-1]
[Type-2]
Tackroone
dessciii
looth
amm
Q: What the valuz of the item moxts the definition Concapt
Ressoning
satisdy Concept
Q: What the relationship between B and
Figure 10: Examples from Licon-A, We first train the model to understand concept A, then the model learns concept B through a linguistic description and at evaluation, answer questions about the concept B.
Licon-1
Train
Des Ar IF object color equal to THEE Concept
[Clasiifiaation]
[Attribute]
[Relationshiip
object the Concopt A
This object hax the Concept
Concept shape equal
Stop
cube THEN Concept B
Known concapt learning
This object doesn't Concegt
The ooler of this object diffeemet from Cuncupt
Concept cohypemmm Concege
(b) längaistic description
Novel concept
Des of C: Concept C is the combisstion of Concept and Concept A
leamiigg
[Tpe-1]
[Tpe=2] Q: that object has the same color Concego C?) option:
Q: What relationsdip
ls that kind Comoceii C? sptioo: I...f
between Conapet A and Concegt fiiiion anowe: hypemyne
Step Reasocing
onvore: po
Figure 11: Examples from Licon-I. We first train the model to understand concept A and concept B, then the model learns concept C.
(8) Linguistic description
Step Novel concept leamming
Climate Mignation; refers the phenomenon mass migration of people who can no longer survive their original place of residence because the deterioration climate and natural civironment.
(b) Test cases
Q Which one of following options belong to climate migration?
option: A. In order to restore the grassland ecosystem and prosect the natural pastures, the forner seattered herders were relocated to sottooments buift wader
B. Every winter, order escape the cold. group of eiderly people ran to group to spond the winter to wane climate
Step 2.
There prolonged drought and year after Local residents are leaving their homes
another and looking
Reasoning,
for new places
D. With global warming, sea levels gradually rising and many islands are being flonded by searater, secnnatos,forcing forcing islanders
places.
awner: D
Figure 12: Examples from Licon-T, since the known concepts stored in the pre re-trriied model commonsense, we only train the model to learn novel concepts through linguistic descriptions.

---

"description": "This dataset is used to predict the type of birds based on the given attributes. Each row provides the relevant attributes bird.",
"colurnn numes":;
"size" ["categorical", ["large". "medium", "small"]]. "size (number)" : ["number", [10, 100]]. "color": ["eategorical", ["rod", "blue", "green", "brown". "pink", orange", "black", "wh ""rnggh"""""teegooiii" ("categorical", ["tall,, "medium", "shor"]],.
"head" ["categorical", ["yes", "bo"]], "length"
"longth (aumber)" : ["number", (ruumer".[10010000 [10,100]].
"tail" [""aatgorriaa", """"""."'o""',.
"number faces" ['number', [1,3]]
"arms" ["eategorical", 1"yes", 'no"]].
"logs" "eattggorical". [2 8]].
"hair" ['callgoorccal" ['yes", "]o
"wings" : Iccateorrical"" ["yes", "no"]],
"feathers" : "'catogorical", ["yes", "no"]].
"airborne" |"categorical", ['yes", "no"]].
"loothed", | "'categorical"" ["yes", "].
"backbone": l"eategorical", ["yes", "]."]].
"venomous" ["eategorical", ["yes", "no"]].
"domestic"; "categorical", ["yes", "no"]]
"region": ("eategorical", ("asia", "europe", "americas", "africas", "atricca""""""ia""""" antartica",
"oceanin"]]
Figure 13: Table schema of Licon-A: Species of Birds.
"description":
"This dataset is used to describe synthetic object ",
A.2.3 Creating linguistic descriptions
"column names":
The template-based approach also used in Licon- The linguistic description for the two types is different. One is IF v THEN c and the other is cy is the combination of c1 and c2, which means that ey must satisfy both the definition of and c2.
l'large', 'middle', 'smalf]]. "color" ("categorical", ('bluc', 'brown', 'cyan', 'gray', 'green, purple' red, yelloww]].
"size": ["eategorical".
"shape" I"categorical", ('cubet, 'eylinder, sphere']], "material catogorical "[ 'motall 'rabbe]]].
A.2.4 Generating questions for concept
Figure Table schema of Licon-I: Species of synthetic objects.
We also generate two types of questions for each concept in Licon-L. For the classification questions, we generate images and answers following Johnson et al. (2017). And to avoid too simple questions, at most one of the attributes of an object doesn't 't meet the definition. For questions about attributes, generate images with the specific attributes shown in Figure 11. And for questions about the relationship, we perform the same operation as Licon-A.
A.2 Licon-I
Licon-I is dataset consisting multi-modal data. We follow the same steps as Licon-A to generate Licon-l. And the example is shown in Figure 11.
A.2.1 Table schema
For the table schema, we use four attributes to form the table schema, which are color, size, shape and material. And the full table schema is shown in Figure 14.
A.2.5 Different sub-data
Finally, create 2 sub-datasets by changing the hierarchical relationship: HI (2.3), which means the number of levels of concepts. Under the 2 setting, a concept can be combined with up to two base concepts. And under the 3 setting, a concept can be combined with three base concepts.
A.2.2 Creating the definition of concept
We also generate the definition of concepts in Licon-I shown in Figure And different from Licon-A, the operators in Licon-I only contains equal.
A.3 Licon-T
To create Licon-T, collected 3142 questions from the Chinese National Civil Service Exami- nation over the past two years. We then hired six annotators to filter out questions involving charts or figures, and questions that can't 't be answered
We generate two types of concepts, one for con- cepts that do not depend on other concepts, base concepts, and the other for concepts that are gener- ated by combining two existing concepts.

---

solely from the concept descriptions. And the final Licon-T contains 1008 questions. We will release the English and Chinese versions of Licon-T for concept learning study.
Conrape
Concept Descipptos
Outpu
C Concepe xily Qucdiom-se-llza +
Concept Discripiizs
Ourpor
As shown in Fig 12, the concepts in Licon-T are defined as "concept name: the corresponding complex description". There is no explicit set of attributes in Licon-T, T, so it is difficult to classify the descriptions into different types, such as Licon-A and Licon-l, so we take Licon-T as a whole in our experiments.
Concept K: xi Utceitm-ar-
Concept Desciipizz
Ouapu.
Kauenn Diaggii leaming [[rris]]
Novel Concepe lextting [Tee]]
Noval Coacept
Concept Dessijpias
Ouipu.
Figure 15: Experimental setup: the model is trained over known concepts and evaluated generalization to novel concepts.
B Training details
For Licon-A A and Licon-I, the max length for RoBERTa-basee models is 64. We increase the limit to 256 tokens in Licon-T. We use the AdamW (Paszke al., 2019) the optimizer for all models. And we choose le 5 as the learning rate. The batch sizes are kept as 16.
In this section, we describe the details of the train- ing, including the parameters of the model and the process of the training model.
B.1 Details of concept split
For Licon-A, we proportionally split the concepts into training sets, validation sets, test sets, account- ing for 60%, 10%, and 30%, respectively. And for Licon-I, because it has more concepts, the ratios are 60%, 20%, and 20%. during training, base concepts function as known knowledge, so such concepts must be the train- ing set. For Licon-T. we also split the concepts according to 60%, 20%, and 20%.
B.4 Experimental setup
We train our model on seen concepts C1 C2 Ck and evaluate on novel concepts Ck+1. On as shown in Figure 6. The model is performed with a zero-shot setting. And during training, the model never sees novel concepts and generalizes to novel concepts at evaluation.
worth noting that
B.5 Training process
B.2 Model parameters
Training Licon-A and Licon-I:
The RoBERTa.C RoBERTa.D, RoBERTa-MNLI in our paper is the same as the pre-trained RoBERTa- based model available And the CLIP is available through Radford al. (2021). The ResNet is used in Johnson al. (2017). We implement EnC based on RoBERTa, with two-layer GNN model the hierarchy of concepts.
Due to the template-basse description generation method, descriptions can be easily transformed into the relational graph G of known concepts by de- signed semantic parser. Besides, for the known concepts in the training set, we randomly initialize the box embeddings. And optimize the box embed- dings of known concepts and parameters of GNN during training. At evaluation, we append novel concepts node on the graph based their lationships and initialize the box embeddings with the average box embedding of all concepts in the training set. Then we get the final box embedding of novel concepts through the GNN based on and transform the normal word embeddings to box embeddings as the input of RoBERTa.
HuggingFacce library.
B.3 Hyper-parameter settings
All hyper--aramettrs of the wranformer-basee models we use are kept default to the settings the HuggingFace library. For RoBERTa, we use 'roberta-base' as our checkpoint. And for RoBERTa-MNLI, we use texttataack/robert-bbaas MNLI' as the checkpoint. For Chinese dataset in Licon-T, we use wewpchinese-rbbrtaawwwweeee the checkpoint for RoBERTa and CCL2018 cor- pus for RoBERTa(TE), For EnC, finetune the checkpoint of RoBERTa pre-trained on textual en- tailment corpus. For CLIP, we use the 'ViT-B/32" as checkpoint. And for ResNet, we follow the set- tings Johnson et al. (2017).
as
Furthermore, during training, for an epoch, we first train the model to learn the base concept and then force the model to learn how to understand lin- guistic descriptions through the remaining concepts in training sets. Finally, we test the model on novel concepts, which give only linguistic descriptions based on known concepts. As an evaluation, we use

---

novel concepts answer questions and accuracy is the result.
Training on Licon-T:
For Licon-T, to generate the relational graph G of concepts, we utilize the topic distribution of concepts, which is labeled manually by experts. The topic of a concept is considered to be its own hypernym concept. And for the loss in Licon-T, we only consider the entailment relationship of concepts, i.e, the relationship between the concept and its topie. The final loss is computed by:
Figure 16: Clustering results of concept embeddings generated by EnC and RoBERTa.
Then we define the lower-bound Min(ee) and upper-bound Maz(e,) for a box embedding of concept
L le-
Moreover, for Licon-T, the known concepts used in the concept learning process are stored in the pre- trained model as commonsense. So we only train the model to learn novel concepts through linguistic descriptions. Besides, to maintain the knowledge stored in the pre-trained model, we initialize the box embeddings of concepts in Licon-T with the value of normal embeddings of concepts the centers and only randomly initialize the offsets. The training process the same as for Licon-A and Licon-I.
Min(ce) = Cen. - Offe
(3)
Max(ee) = Cen, + Offe
(4)
And the overlap value is calculated through:
Hel, 02) - II (()) 0)
(5)
Where M e = Mar(ea) V Max((e2) and m(eet. 222 Min(e.)) A Min(e.2). and are element-wise min and max operators. The final loss is then calculated.
B.6 The details in Graph Neural Network
For Graph Neural Network, we use two-layer graph convolutional network (Kipf and Welling, 2016). For the node i and j in graph G, we assign values to Dij according following formula.
C Performance of EnC on Licon-T
As shown in Table 4, though our proposed method improves the performance of Licon, the results on Licon-T are still unsatisfactory. Because of the following challenges: First, compared syn- thetic datasets, relationships in Licon-T are more complex and difficult to categorize into specific classes, such as hypernym or co-hypernym. There- fore, we only label the topic of concepts as hyper- nym. Second, the linguistic descriptions of novel concepts frequently involve domain-specific ter- minology that can be hard to understand. Finally, Licon-T demands comprehensive reasoning abil- ity, including deductive, abductive, and analogical reasoning skills.
i= Dig 1 else if wi, wj has realtionships, otherwise.
(1)
Then, for the i-th node the 1-th layer, its hidden state representation denoted as h updated by the following equation:
A - Σ Di b1),
(2)
where Wi,by are learned parameters and is the ReLU activation function. In practice, the centers and offsets are calculated independently through GCN.
Due to the aforementioned challenges of Licon- T, we only model the hypernym relationships in- dicated by the annotated topics, without consid- ering co-hypernym relationships. As a result, the improvement of relationship modeling is not significant on synthetic datasets.
B.7
The calculation of concept-level entailment
The concept-level entailment, both entailment and contradiction, is calculated thought the overlap val- ues of two concepts ea1 = {Cenal, fef and e,2 = (Cen,,, Of f(e)-
Besides, Licon-T requires strong reasoning abil- ities to determine whether the options belong the concepts. However, our paper focuses on en- hancing the understanding of concepts through box

---

Licon-A
equal ne (the length equal the legs equal 4) THEN Concept A
1. 2. short cohypemym
RoBERTa.C
Concept the Concept with domestic equal
ycs( 2. short:o.55) 3.none(0.73)
What the value of length the definition Concept B?
RoBERTa.D
longth
legs
Short 4
3. What is the relationship between and C?
yes(o.57) longr0.52) 3.none(0.99
salisly definition Concept
(Concopi Cia the Concept A with the equat to
EnC
1. no(0.75) 2. short(o7)) 3. cohypernymxo.84)
Licon-1
Correct answer
IF the object color equal red THEN Concept A object shape equal cube material equal Concept is the combination of Concept and Concept
no 2. no 3. cohypemmy
Concept
RoBRRTa.C.
ycs(0.51) 2. no(0.59) 3.none(0.63)
Is the value of shope Concept the same that Concept A?
no(0.59) noro.57) 3.none(0.61)
3. what is relationship between and concept D)?
EnC.
(Concept D on another childron concept of cuncept
1. Does this isem satisfy the definition of Concept C?
1. n0(083) no(0.66) 3. cohypemymo..77
Figure 17: Illustration of typical cases. For novel concept and its question, we provide the answers and the corresponding probability distribution generated by RoBERTa.C, RoBERTa.D, and EnC.
embeddings, and the emphasis is not on reasoning ability.
ability of the answer is slightly higher than 0.5, which indicates that these methods do not actually have strong ability to learn concepts through the linguistic description. And the answers generated do not have high degree of certainty. However, EnC is able generate answers with high certainty. 3) Compared to the baselines, EnC performs particularly well on questions about relationships between concepts. Since the box embeddings main- tain the structure of the concepts, EnC is able to answer this type of question. However, for RoBERTa.C and RoBERTa.D, they tend answer "none" when asked about two concepts that do not occur simultancously in a sentence.
Although the performance of EnC on Licon-T is currently unsatisfactory, conduct cluster experi- ments using the embeddings generated by EnC and RoBERTa on Licon-T. The results are illustrated Figure 16. And the results show that the embed- dings of concepts with shared topics have shorter distances than those of other concepts. Addition- ally, the clustering of the embeddings generated by EnC is clearer when compared to those generated by RoBERTa, indicating the effectiveness of EnC. The results show that in the process of learning concepts, EnC able to link the concepts to their topies through embeddings and therefore leverages the knowledge of known concepts.
These findings illustrate the effectiveness of EnC in learning novel concepts based on known con- cepts through box embeddings.
D
Case Study
To further explore the effectiveness of EnC, we present interesting cases in Figure 17. The figure shows the linguistic description of known concepts and novel concepts, the questions about novel con- cepts, and answers generated by different models. We observe that 1) Compared to RoBERTa.C and RoBERTa.D, EnC achieves better performance for both Licon-A and Licon-I. Not only terms of the accuracy of the answers, but also in terms of the probability distribution of the answers.
2) For RoBERTa.C and RoBERTa.D, the prob-


Reviews:
[{"review_id": "64qyxYiRlK", "reviewer": "Reviewer_1ug8", "summary": "", "questions": "As the Licon-T dataset is used for evaluating human performance. I wonder what is the GPT-3.5 / GPT-4 performance on these datasets? With large language models, such as GPT, Llama, Vicuna, etc., I wonder whether the proposed approach EnC would still be useful and achieves superior performance than those large language models? If yes, what is the reason? If no, then would we still need EnC?\n\nTo clarify the excitement section, it is based on the Licon-T and the potential answer of the above question.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your advice.\n\nQ1: How to split the training and the evaluation for Licon-T?\nA1: For Licon-T, the concept descriptions and questions are randomly divided into training, validation and test sets by 60%:20%:20%. \nWe consider the concepts in the training set as known concepts and the concepts in the validation/testing set as novel concepts. In order to establish the relationship between known concepts and novel concepts, we annotate the topic of concepts as the hypernym of concepts, which are shared between known concepts and novel concepts. Then concepts sharing the same topic are considered as  co-hypernyms concepts.\nTherefore, the training and testing process of Licon-T is the same as other datasets.\nFor more details, please refer to Appendix B.5 and Appendix C for the detailed training process and the performance of EnC on Licon-T.\n\nQ2: For Licon-I dataset, I feel it might be little bit redundant, as the main theme of this paper is about learning concepts from linguistic clues. I wonder if this Licon-I fulfill / fits this theme?\nA2: We propose to link novel concepts to known concepts through linguistic description, called \"linguistic concept learning\". As shown in Fig.2, the core of linguistic concept learning is using linguistic descriptions in  to describe relationships between novel concepts and known concepts (i.e., Step 2). As shown in Fig.2(b), for Licon-I, although known concepts are learned through multimodal information (images and their class names), the process of learning novel concepts relies exclusively on linguistic descriptions and maintains themes. Therefore, the Licon-I dataset is an extension of linguistic concept learning in the specific scenario where the concepts contain multimodal information.\n\nQ3: The Licon-T dataset description might be too limited to conclude whether this dataset is useful or not.\nA3: Licon-T is a real-world dataset derived from civil service exams. The descriptions are used to assess the comprehension skills of the average person, and as such, these descriptions \n(1) describe the widely distributed concepts as shown in Figure 5(b). \n(2) have the ability to check the learning ability of the model in the same situation as humans.\nThus, Licon-T has important applications in testing the learning ability of models in real-world scenarios, and we believe it is useful for training artificial general intelligence.\nAnd due to space limitations, more discussion of Licon-T has been placed in the Appendix.\n\nQ4: I wonder what is the GPT-3.5 / GPT-4 performance on these datasets?I wonder whether the proposed approach EnC would still be useful and achieves superior performance than those large language models? If yes, what is the reason? If no, then would we still need EnC.  \nA4: We tested the performance of Licon-T on GPT-3.5. However, since GPT-3.5 is frequently updated, we did not report this result in the manuscript.\nThe GPT-3.5 version is not available as an open source project. This means that we could not directly integrate EnC into GPT-3.5. Therefore, we evaluated the impact of incorporating EnC methods into GPT-3.5 by testing with samples before and after applying the rationale behind EnC (i.e., relating concepts to their basic concepts).\nWe randomly selected 50 concepts and their questions to ask GPT-3.5, and the accuracy was 46% (23/50), which is better than the baseline performance in Table-4 but still below 50%.\nThen we tested EnC differently. When we ask a question, we clearly tell GPT-3.5 which topic the concept belongs to, obtaining accuracy rate of 56% (28/50), which suggests that the EnC method, or specifically, the method of relating novel concepts to their basic concepts, is useful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The Licon dataset is interesting and important, if this is the first dataset proposed for concept learning.\n2. The motivation that learning concept can from both visual world and linguistic world is well justified.\n3. The Licon-T dataset is quite interesting. Unfortunately, the description of the dataset is quite limited. ", "reasons_to_reject": "The reasons to reject are all for the Licon dataset.\n\n1. After reading the paper, I don't know how to use the Licon-T dataset. How to split the training and the evaluation? As the Licon-T dataset seems not having the separation of basic concept and the novel concept.\n2. For Licon-I dataset, I feel it might be little bit redundant, as the main theme of this paper is about learning concepts from linguistic clues. I wonder if this Licon-I fulfills / fits this theme?\n3. The Licon-T dataset description might be too limited to conclude whether this dataset is useful or not.", "questions_to_authors": "As the Licon-T dataset is used for evaluating human performance. I wonder what is the GPT-3.5 / GPT-4 performance on these datasets? With large language models, such as GPT, Llama, Vicuna, etc., I wonder whether the proposed approach EnC would still be useful and achieves superior performance than those large language models? If yes, what is the reason? If no, then would we still need EnC?\n\nTo clarify the excitement section, it is based on the Licon-T and the potential answer of the above question.", "paper_topic_and_main_contributions": "This paper proposed a dataset called Licon to study the concept learning capability. Concretely, the Licon contains three spilts: Licon-A which contains basic concepts that are for concept learning, and novel concepts for evaluation. Licon-I is similar to Licon-A but with image. And Licon-T is collected from real-world questions for concept learning and reasoning. With all the three datasets, the author proposed an approach to leverage the box embedding and graph neural network to learn concept. The proposed approach achieves higher performance than the baseline approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "Zt7rQAfOEB", "reviewer": "Reviewer_y4Hw", "summary": "", "questions": "1. Does the dataset support annotation of relationship between objects for calculating accuracy and precision?\n2. If there are not abstract concepts to learn, what is the issue in visual concept learning?\n3. It is not clear from Figure-3, why C is a hierarchy of A,B,E. We can see that A,B,E are subset of C, however they can also exist independently without being in a hierarchy.\n4. Licon-I clearly focus on combination of concepts for concept learning as opposed to Licon-A which treats that as hierarchy.\n5. How does Licon-T compares itself to other datasets which address reading comprehension task?\n6. What is the evaluation metric used in Table-4 results?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your advice.\nQ1: The Licon dataset is very focused on specific types of concept learning and does not present some scalable options for expansion. \nA1: Our proposal of \"linguistic concept learning\" is based on the fact that humans are able to learn novel concepts quickly from linguistic descriptions.  In Linguistic Concept Learning, the process of learning concepts relies on textual descriptions. However, known concepts and new concepts themselves can be presented by various types of information, including single-modality and multi-modality.\nFor the Licon dataset, we created three sub-datasets in which concepts are presented in different forms, including simple attributes, concrete visual images, and abstract textual definitions. These datasets are controllable and we can test the models with different, scalable hierarchies.\nIn the future, we can also extend the concepts to include more types of information, such as audio. Besides, the EnC approach, which focuses on modeling relationships between concepts, can also be easily extended for various types of concepts.\n\nQ2: Clarification about concept hierarchy\nA2: In concept learning, there are relationships of \"including\" and \"being included\" among concepts. To make this relationship clearer, following existing methods [1], we use a hierarchical structure to represent the relationship between concepts. In hierarchy, child concepts are generated based on more refined definitions of hypernym concepts and child concepts are the subset of hypernym concepts.\nAs shown in Figure 3, for Licon-A, concept B is defined by adding new attribute to A. Therefore, B is the subset of A and A is the hypernym concept of B.\nIn Licon-I, concept B is defined by combining the attributes of D and A. Concept B is the subset of both A and D, and concept A and D are both the hypernym concepts of B. Similarly, C is simultaneously the subset of B, E, and further the subset of A, D.\nTo sum up, the construction of Licon-A focuses on adding new attributes to existing concepts, while Licon-I concentrates on combining attributes of existing concepts. And the concepts in both datasets maintain a hierarchical structure.\n\nQ3: The difficulty part of the dataset is not comprehensible as none of the experiments showcase as it claimed. Future experiments were planned with curriculum learning.\nA3: The challenges of Licon are mainly from the Licon-T dataset. \nAs shown in Figure 5 (a), the reading complexity in Licon-T is quite high and the descriptions are hard to understand. As shown in Figure 5 (b), the distribution of concept topics are wide, increasing the challenge of learning these concepts. Furthermore, we can also observe from Table 4 that the performance of the baselines is significantly below satisfactory, suggesting that Licon-T is challenging.\nIn future, we plan to introduce curriculum learning to enhance the understanding ability for complex concepts. Similar to human learning, we'll train the model to learn from easy concepts to complex ones. The planned curriculum includes three stages: (1) simple linguistic concepts, (2) complex options for simple concepts, (3) more complex concepts. With this training method, we expect better performance on Licon-T.\n\nQ4: Does the dataset support annotation of relationship between objects for calculating accuracy and precision?&& What is the evaluation metric used in Table-4 results?\nA4: In Licon's synthetic datasets (Licon-A and Licon-I), we create novel concepts according to the pre-defined relationships between known concepts. Therefore, there is no need to annotate relationships between objects. And for Licon-T, we have annotated topics as hypernyms of concepts for training.\nDuring evaluation, we utilize questions about the given novel concept (as shown in Figures 10, 11, and 12) to measure the model's understanding of that novel concept. Therefore, we report the accuracy in Tables 2,3 and 4 as the results of the evaluation.\n\nQ5: If there are not abstract concepts to learn, what is the issue in visual concept learning?\nA5: In visual concept learning tasks, concepts are mainly concrete. As shown in Fig. 2 (a), concepts in the visual domain may be related to color, shape, and location, which can be directly represented by images. And the process of learning novel concepts relies on images. As mentioned in [2], \"Visual reasoning aims to reason about object properties and their relationships in given images\", \"such as left, yellow, and cube\".\n[2] \"FALCON: fast visual concept learning by integrating images, linguistic descriptions, and conceptual relations\", ICLR, 2022\n\nQ6: How does Licon-T compares itself to other datasets which address reading comprehension task?\nA6: Reading comprehension mainly focuses on answering questions based on the content of the text, while Linguistic Concept Learning focuses on understanding the concepts and answering the question based on all learned concepts.\nBesides, in reading comprehension, answers are typically derived directly from the provided text. In contrast, Linguistic Concept Learning concentrates on modeling the associations between different concepts, and the answers are not directly sourced from the context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Language can describe complex relationship between abstract concepts in contrast to visual only representations.\n2. Dataset created contain properties such as diverse and controllability.\n3. Concept learning method proposed capture relationships quantified by entailment and contradiction.\n4. Evaluation is mostly zero-shot by answering multiple choice questions related to the novel concepts which are not part of training sets.", "reasons_to_reject": "1. Licon dataset is very focused on specific types of concept learning and not present some scalable options for expansion.\n2. Specific part of dataset like Licon-A about hierarchical concepts are not particularly seen as hierarchical. They can be treated as subset of each other.\n3. The difficulty part of the dataset is not comprehensible as none of the experiments showcase as it claimed. Future experiments were planned with curriculum learning.", "questions_to_authors": "1. Does the dataset support annotation of relationship between objects for calculating accuracy and precision?\n2. If there are not abstract concepts to learn, what is the issue in visual concept learning?\n3. It is not clear from Figure-3, why C is a hierarchy of A,B,E. We can see that A,B,E are subset of C, however they can also exist independently without being in a hierarchy.\n4. Licon-I clearly focus on combination of concepts for concept learning as opposed to Licon-A which treats that as hierarchy.\n5. How does Licon-T compares itself to other datasets which address reading comprehension task?\n6. What is the evaluation metric used in Table-4 results?", "paper_topic_and_main_contributions": "In this paper, authors propose a benchmark dataset known as Licon, where concepts in diverse forms or modalities of data are defined by linguistic descriptions. The goal of the benchmark dataset is to perform concept classification, attribute prediction and concept relationship recognition. Authors also propose an approach named EnC which find the relationship among concepts.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure-1, Abstract, please correct the spelling mistake \"plant\" -> \"planet\"."}, "extra_text": {}}, {"review_id": "g9s99n8tC9", "reviewer": "Reviewer_rgen", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions.\nWe will redraw the chart to ensure that important information is included in the main part of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper releases a new dataset of concept learning. Especially, from my point of view, the task defines in this paper could be used as the instruction to fine-tune or evaluate the LLMs, which may make the LLMs imitate the process of human learning and may improve the generalization of LLMs. Additionally, the dataset itself is diverse, controllable and challenging. ", "reasons_to_reject": "None.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about to release a new dataset called Linguistic Concept Learning benchmark (Licon). Different from those datasets where visual information is provided for models to learn concepts, this dataset provides diverse forms of information including plain attributes, images and text, which aims to encourage the learning of abstract concepts which cannot be present by visual information. Furthermore, the Licon dataset has been proved to have the properties of diverse, controllable and challenging with sufficient evaluation and analysis. In addition, this paper also proposes an entailment-based concept learning method (EnC) to model the relationship between concepts, utilizing box embeddings and graph neural networks. This method shows effectiveness on Licon dataset.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Abstract line 008, I think it should be \"Planets\". And I think some of your figures could be better illustrated, please make sure you have included all information of the figures in the texts. I guess it is because the limitation of number of pages, some of the import information like the examples of your dataset, should be moved to the main of this paper."}}]

Author rebuttal:

Meta-review: **Summary:**
This paper introduces a new dataset called the Linguistic Concept Learning benchmark (abbreviated as Licon) designed to investigate concept learning capabilities. The Licon dataset comprises three components: Licon-A, which includes basic concepts for concept learning and novel concepts for evaluation; Licon-I, similar to Licon-A but with images; and Licon-T, collected from real-world questions to assess concept learning and reasoning. The diverse information types are intended to facilitate the learning of abstract concepts that cannot be solely represented through visual information. The primary objectives of this benchmark dataset are concept classification, attribute prediction, and concept relationship recognition. Additionally, the paper proposes an entailment-based concept learning method (EnC) that leverages box embeddings and graph neural networks to model relationships between concepts, demonstrating its effectiveness on the Licon dataset.

**Strengths:**
The reviewers unanimously acknowledge the strengths of this work. This paper introduces the Licon dataset, a novel concept learning dataset that stands out for its diversity, controllability, and challenge. The motivation behind incorporating both the visual and linguistic worlds into concept learning is well-founded. Language, unlike visual-only representations, can describe complex relationships between abstract concepts. The proposed concept learning method captures relationships through entailment and contradiction quantification. Notably, the evaluation largely consists of zero-shot tasks, involving multiple-choice questions related to novel concepts not included in the training set. Finally, the tasks defined in this paper have the potential to serve as instructions for fine-tuning or evaluating Large Language Models (LLMs), enabling LLMs to mimic human learning processes and potentially enhancing their generalization capabilities.

**Weaknesses:**
The Licon dataset is overly focused on specific types of concept learning and lacks scalable expansion options. Additionally, specific sections of the dataset, such as Licon-A, which pertains to hierarchical concepts, do not distinctly exhibit hierarchical characteristics, and the description of the Licon-T dataset is somewhat limited. One of the reviewers expressed confusion regarding the usage of the Licon-T dataset, as it does not appear to differentiate between basic and novel concepts. Regarding the Licon-I dataset, it may be considered somewhat redundant, as the primary theme of this paper revolves around learning concepts from linguistic cues. Lastly, the claimed difficulty level of the dataset is not readily discernible, as none of the experiments effectively demonstrate it.

**Author-Reviewer discussion and acknowledgment:**
Reviewers raised various questions and concerns, to which the authors provided responses during the rebuttal and discussion phase. They outlined improvements and engaged in further discussions, particularly with one of the reviewers. Two of the reviewers have responded and acknowledged the authors' arguments.

**Conclusion:**
The paper is well-written; reviewers suggest that the authors rectify just a few typos.
Decision: Accept-Findings