VER: Unifying Verbalizing Entities and Relations
Authors: Jie Huang
Kevin Chang
Venue: EMNLP 2023 Findings (2023)
Keywords: definition modeling
relation modeling
entity relationships

Abstract: Entities and relationships between entities are vital in the real world. Essentially, we understand the world by understanding entities and relations. For instance, to understand a field, e.g., computer science, we need to understand the relevant concepts, e.g., machine learning, and the relationships between concepts, e.g., machine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she is related to others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning a new scientific term, people usually start by reading its definition in dictionaries or encyclopedias. To know the relationship between two entities, humans tend to create a sentence to connect them. In this paper, we propose VER: a unified model for Verbalizing Entities and Relations. Specifically, we attempt to build a system that takes any entity or entity set as input and generates a sentence to represent entities and relations. Extensive experiments demonstrate that our model can generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, including definition modeling, relation modeling, and generative commonsense reasoning.

VER :
Unifying Verbalizing Entities and Relations
Jie Huang Kevin Chen-Chuan Chang
Department of Computer Science, University of Illinois at Urbana-Champaige [jeffhj, scchang)@illinoii
Abstract
Basically, we understand entities and relations by "verbalizing" them. Verbalizing entities and rela- tions also tests our knowledge about entities and relations. Literally, by verbalizing entities and rela- tions, we understand the world.
Entities and relationships between entities are vital in the real world. Essentially, we under- stand the world by understanding entities and relations. For instance, to understand a field, e.g., computer science, need to understand the relevant concepts, c.g., machine learning. and the relationships between concepts, e.g.. machine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she related others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning new scientific term, people usually start by reading its definition dictionaries or encyclopedias. To know relationship between two entities, humans tend to create a sentence to connect them. In this paper, we propose VER : a unified model for Verbalizing Entities and Relations. Specif- ically, we attempt to build a system that takes any entity or entity as input and generates sentence to represent entities and relations. Ex- our model generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, includ- ing definition modeling, relation modeling, and generative commonsense reasoning."
Similarly, do machines have the ability to ver- balize entities and relations? Can machines learn about entities and relations from verbalizing them? The answer is "Yes". Recent studies show that by giving the surface name of an entity (and its con- text), models (after training) can generate coherent sentences to represent i.e., definition modeling (Noraset al., 2017: Gadetsky et al., 2018: Bevilac- qua al., 2020; August et al., 2022; Huang et al., 2022b; Gardner al., 2022), and by giving the surface names of a pair of entities, machines can generate coherent sentences describing their rela- tionships, i.c., (open) relation modeling (Huang et al.. 2022a.c). However, verbalizing entities re- quires understanding relationships between enti- ties, and verbalizing entity relationships requires understanding entities themselves, while existing works deal with entity and relation verbalization separately, ignoring the connections between them. Besides, recent works (Devlin al., 2019; Lewis et al., 2020; Radford et al., 2019; Brown et al., 2020) have shown that language models pre-trained with self self-supervised objectives can equip the model with a significant amount of knowledge (Petroni et al., 2019; Roberts et al., 2020) and achieve sub- stantial gains after fine-tuning Can we continually pre-train the models with pre- training objectives entities and relations to en- hance their ability on verbalizing entities and re- lations? In this way, the model can be easier and better adapted to specific tasks on entities and rela- tions and even be used without additional training.
lensive experiments demonstrate
1 Introduction
What is X? What is the relationship between X and Y? We come up with these questions almost ev- ery day. When we come across a new term, e.g., twin prime, we usually refer to its definition to un- derstand it, i.e., "A twin prime is a prime number that is either 2 less or 2 more than another prime mumber". To express the understanding about re- lationship between entities (c.g.. carbon dioxide and water), create sentence represent their relationship: "Carbon dioxide is soluble in water".
a specific task.
Therefore, we aim solve entity and relation verbalization in unified form and pre-train a model for entity and relation understanding. Es- sentially, definition modeling and relation model-
'oode, data, and the trained models are available at https:
//github... wttp:/cccceeeeeeeeee jeffhj/VER,

---

Definition Modeling
Carbon diaxide a chemical compound made up of molecules that each have carhon atom covalently double banded to oxygen atoms, found in the gas room temperalure.
carbon dioxide
Relation Modeling
carbon dioxide; water
[RE]VER
Carbon diaxide soluble wustrr
Plype-Reeltion Modeling
Carbon dioxide reacts with water produce arhouic acid.
carbon dioxide; water; arbonic acid
Figure I: A diagram of [RE]VER. We feed the model with entity(s) and train it to reconstruct sentences containing all the entities. This allows us to use single model to better "verbalize" entities and complex entity relationships.
ing can be unified as an "entity(s) - sentence" task, i.e., given set of entities, generating a sen- tence describing the entities and their relationships. When the size of the set is 1, it is equivalent to definition modeling, and when the size of the set is 2. it is equivalent to relation modeling. By defining the task in this form, we can even model more complex relationships among entities since entity relationships can go beyond pairwise (Bretto, 2013), named hyper-relation modeling, e.g.. (car- bon dioxide, water, arbonic acid) + "Carbon dioxide reacts with water to produce arbonic acid". Based on this, we propose VER (pronunciation: /v3//): a unified model for Verbalizing Entities and Relations (Figure I). Specifically, we pre-train models by forming self-supervised text recon- struction task: given entity or set of entities, reconstruct the original sentences (e.g., defini- tion or a relation description) containing them in the training corpus. In this way, the models ac- quire knowledge about entities and relations and learn to connect entities to a meaningful coherent sentence. Since implicit knowledge stored in the parameters of the models may not be sufficient for generating meaningful sentences for the tar- get entities, we also study VER in the retrieval- enhanced setting (Guu et al., 2020; Izacard al., 2022: Huang et al., 2023b), named REVER, i.e., Retrival-Ennhe Enhanced VER, by pre-training models augmented with sentences contained the target en- tities in the pre-training corpus. Throughout the remainder of this paper, we use "[RE]VER" to denote both VER and REVER.
than baselines definition modeling, relation modeling, and generative commonsense reason- ing. addition, the performance of [RE]VER without additional training is impressive, making itself a potential knowledge source of entities and relations, which may benefit tasks on entities and relations such as entity typing (Ren et al., 2016), relation extraction (Bach and Badaskar, 2007), and knowledge graph completion (Lin et al., 2015).
Background and Formulations
Definition Modeling. Definition modeling aims to generate definitions of entities, which can be formulated as conditioned sequence generation task. For instance, given twin prime, the expected output is the definition of twin prime: "A twin prime is a prime number that is either 2 less more than another prime number". We follow the standard sequence-to-sequence formulation Noraset et al. (2017); Huang et al. (2022b): given entity æ, the probability of the generated definition [ 8m] is computed auto-regressively:
P(s|z) = II P(s.lso. S1-1,
(1)
where m is the length of s, 8; is the ith token of s. and so is special start token.
(Open) Relation Modeling. Relation modeling attempts to generate coherent and meaningful sen- tences describing relationships between entities, where types of relations not pre-specified, i.c., in an "open" setting (Huang al., 2022a). For example. for carbon dioxide and water, their re- lationship can be described as "Carbon dioxide is soluble in water." For machine learning and al- gorithm, the expected output could be "Machine
Experiments on six datasets demonstrate the superiority of our model in verbalizing entities and relations. Especially in low-resource settings, [RE|VER can achieve significantly better results

---

learning explores the study and construction of al- gorithms that can learn from and make predictions on data." Formally, given entity pair (x. y). the probability of the generated relation description
1e6
4126967
4.0
3.0
2962250
2603028
== [$1,.... 8m] is calculated as:
2.5-
P(s|a, II ((s .1...., 1, 1, (2)
15-
1=1
1.0
590189 .40912 2
Hyper-Relation Modeling (Unified Form). Pre- vious works mainly focus on verbalizing single entities or entity pairs. However, the real world, relationships between entities can be more com- plex beyond pairwise, named "hyper" relation- ships (Bretto, 2013; Tu et al., 2018; Huang al., 2019, 2020). For example, "carbon dioxide reacts with water produce carbonic acid". Here, there are tuplewise relationships among carbon dioxide, water, and carbonic acid. Verbalization hyper relationships was initially investigated in Common- Gen (Lin et al., 2020) but was limited to common- sense concepts, and their outputs are simple short sentences describing everyday scenarios containing the given concepts. We attempt model and ver- balize more general complex "hyper" relationships among entities and find unified framework com- bine single entities (I entity), pairwise relationships (2 entities), and "hyper" relationships (23 entities). Combining with definition modeling and relation modeling, we adopt the following unified form:
0.0
Figure 2: Statistics of the pre-training data.
relation modeling, and generative commonsense reasoning, to achieve better performance on spe- cific tasks.
3.1 WIV Data
We prepare the pre-training data with Wikipedia. Wikipedia a large encyclopedia containing a huge number of entities. Wikipedia is well maintained and the content is generally of high quality. We extract entity sets and sentences from Wikipedia. Specifically, we use the 2022-08-01 dump² of En- glish Wikipedia. For each page, extract the plain text by WikiExtractor'.. To find expressions that refer the same entity, we use the neuralcoref (Clark and Manning. 2016) coreference resolution tool in spaCy to preprocess the documents. Since we would like the model to capture the main char- acteristics of entities and relations, we take the first 5 sentences from each page (those sentences are usually definitional sentences sentences express- ing entity relationships). To identify entities, we utilize the hyperlinks in each Wikipedia page build a local mention-entity mapping. And then, we process each sentence and extract the corre- sponding entity set based on the mention-entity mapping. In this way, we build mapping & s, e.g., "(Data mining, data sets, machine learning, statistics, database systems) - Data mining is the process of extracting and discovering patterns in large data sets involving methods the intersec- tion of machine learning, statistics, and database systems." Since for a single entity, we prefer the model to generate a definition-lice sentence rather than a random sentence including it, we collect the first sentence on each page and collect the input- output pair as "(pagg title]) -> 1st sentence". e.g..
m
P(s(E) = II P(si(so
81. 81. ...,, 1....,,.8111.)) 1.
i=1
where E is the entity set and |8| 2
3 [RE]VER Verbalizing Entities and Relations
To verbalize an entity, we are likely to connect it to other entities, which requires knowledge about entity relationships. To understand entity relationships, we need know about entities first. Based on this, we attempt to verbalize en- tities and relations in a unified form and propose [RE]VER : unified model for Verbalizing Entities and Relations. We first create large dataset with the formulation in Eq. (3), and pre- train model this dataset, which equips the model with signifieant amount of knowledge about entities and relations and enables the model to generate coherent and meaningful sentences con- necting the entities. The model can be further fine- tuned on specific datasets, e.g., definition modeling,
htttps: htttps://dumss..c wik.eedikiiaiieeee..ee.. arr.imdddaa.rg//mml//////zo00 Phttps://github.com http://hw...omtttataaaiitttttttr "https://spacy.io/

---

"(deep learning] - Deep learning is part of broader family of machine learning methods based on artificial neural networks with representation learning." We filter out input-output pairs where [E] =1 and + 1st sentence. We keep out pages appearing in the validation and test sets of Huang et al. (2022b.a) and filter out entity sets appearing in the datasets of Huang et al. (2022b); August al. (2022); Huang et al. (2022a); August al. (2022); Lin et al. (2020). We call this dataset WiV (Wikipedia VER). The number of training exam- ples with different sizes of entity sets is summa- rized in Figure
.... l|gl [SEP] 5 [SEP] 2/ [SEP] ... [SEP] s/2"' Similar to VER, we continually pre-train BART- base and BART-large on the input-output pairs and get two versions of the model: REVER-ban and REVER-large.
3.3 Training Process
We pre-train (RE]VER-large and [RE]VER- with the fairseq library². We use Adam with 31 = 0.9, 32 = 0.999, and 10 8 and set the clip threshold of gradients as 0.1. All models use weight decay of 0.001 and dropout of 0.1. We set the learning rate as 5 x 10 and use batch size of 1, 024 tokens, updating every 16 iterations. We set the number of warmup steps as 000. We use small validation set examine whether the training converges. All models were trained on NVIDIA A A100 GPUs or A40 GPUs, and the training con- verged 60 epochs.
3.2 Model
At high level, we pre-train model by training it to reconstruct target sentences conditioned on the entity set with the pre-training data. Specifically, for VER, we continually pre-train BART (Lewis et al., 2020) on the data constructed in Section 3.1. BART adopts a transformer- based encoder- decoder architecture with input text fed to the en- coder and output text produced by the decoder. For our continual pre-training, we encode entity set & = (e1, e to sequence "e1; E2; ...; , e.g., {carbon dioxide, water, carbonic acid) "carbon dioxide; water; carbonic acid". Here we keep the order of entities as the order they appear in the sentence. We choose this design be- cause different orders may correspond to different natural language descriptions (e.g., the descriptions are different when an entity is used as subject vs. object). We would like to mention that although we keep the order here, the model can deal with inputs with random entity orders after fine-tuning (e.g.. CommonGen (Lin et al., 2020) as shown in Section 4.4). We train two versions of the model: VER- base with layers in the encoder and decoder, and VER-large with 12 layers in each, corresponding to BART-based and BART-large respectively.
Experiments
In this section, we evaluate [REJVER on definition modeling, relation modeling, and hyper-relation modeling in three settings: 1) fine-tune the model on the full task-specific training data; 2) fine-tune the model in low-resource settings; 3) directly use the model without fine-tuning, The main goal of the experiments is verify whether the contin- ual pre-training step can enhance models' ability on verbalizing entities and relations with/without external knowledge.
4.1 Experimental Setup
Datasets. For definition modeling, we use the datasets of UJ-CS/Math/Phy (Huang et al., 2022b) and Sci & Med (August et al., 2022). For relation modeling, we use the dataset built in Huang et al. (2022a) (ORM), we take the filtered test set for evaluation since the quality is higher. For hyper- relation modeling, there is no existing dataset. We find that CommonGen (Generative Commonsense Reasoning) (Lin et al., 2020) can serve our pur- pose for evaluation since the task formulation is similar: given set of common concepts, i.e., E (3 < |8| < 5), generating coherent sentence describing an everyday scenario using these con- cepts. By testing on CommonGen, we can also measure the ability of our model for domain adap- tation. Since the reference sentences of the official test set of CommonGen are not released, for the full
For REVER, given an input with entity set & {e1, .e. C|g }. we sample sentences containing the target entities from WiV. Specifically, we search the dataset to find sentences whose entities overlap most with the given entity set repeatedly (the tar- get sentence is excluded). In order for the model be able to handle retrieved knowledge of differ- ent lengths, for each input, we set the maximum number of retrieved sentences as a random num- ber from to 10. With the retrieved sentences s4, 8/y " s/.- we encode the input as "e;; e2;
/ http:///thh....ommececeerrrrrtts

---

UJ-CS
UJ-Math
UJ-Phy
&
Med
100%
BL
R-L
MT
BS
BL
R-L
R-L
MT
BS
BL
R-L
MT
BS
BART
8.31
28.02
12.83
77.97
6.89
28.50
10.97
76.45
5.28
25.75
10.57
76.88
13.13
31.75
13.30
79.31
VER
8.43
30.11
13.06
79.57
7.09
31.94
11.86
78.07
7.09
30.63
12.71
79.18
13,95
33.57
14.84
80.49
SOTA
22.66
38.12
20.30
82.00
23.22
39.39
19.61
80.30
20.84
37.66
19.26
81.18
20.55
37.70
19.24
81.98
REVER
23.04
38.85
20.52
82.79
23.25
41.95
20.49
81.61
21.92
38.01
19.76
81.94
21.29
38.14
19.95
82.55
10%
BL.
R-I
MT
BS
BL.
R-1,
MT
BS
BL
R-L.
MT
BS
BL
R-L.
MT
BS
BART
3,50
22.98
8.68
75.55
4.32
25,42
8.94
75.21
3.27
24.19
8.43
75.72
5,56
23.97
9.47
77.13
VER
6.43
28.24
12.36
78.77
7.24
31,18
11.79
77.82
6.43
30.57
12.42
78.92
7.59
28.25
12.09
78.70
SOTA
17.48
32.32
17.39
80.60
19.74
36.81
18.02
79.85
16.82
32.83
16.96
79.86
12.99
30.82
14.88
80.42
REVER
17.71
34.38
18.14
81.63
21.00
38.83
19.07
80.85
20.32
36.82
19.16
81.59
15,46
33.34
16.83
80.86
0%
BL
R-L
MT
BS
BL
R-L
MT
BS
BL
R-L
MT
BS
BL
R-L
MT
BS
VER
4.81
26.24
11.62
77.55
6.00
30.57
11.41
77.35
5.70
28.62
12.12
78.06
5.98
22.84
11.01
75.32
VER
5.05
26.55
11.96
77.84
6.33
30,36
11.57
76.88
5.95
28.79
12.35
78.13
6.06
23.49
11.22
75.49
REVER
10.38
30.36
14.85
79.98
11.29
35.09
14.60
79.66
12.68
34.49
16.10
80.83
12.13
23.47
13.63
76.53
Table I: Results of definition modeling.
data setting, we submit the results generated by the model to the leaderboard to get the performance. For the low-resource settings, we use the in-house split presented in Wang et al. (2022) to facilitate comparison between our model and the baseline.
CIDEr and SPICE focus on evaluating the concept association instead of n-gram overlap. We also sample 100 examples for test sets and ask three human annotators (graduate students in computer science) to evaluate the generated outputs with 1-5 rating scale used in Ishiwatari al. (2019) (for definition modeling) and Huang et al. (2022c) (for relation modeling).
Baselines. Since [REJVER is trained based on BART (Lewis et al., 2020), we include BART as a baseline for all the tasks. We also compare with SOTA of each task. For definition modeling, the SOTA is CDM-S5,C5 proposed by Huang et al. (2022b), which leverages definitional sentences re- trieved by two definition extractors in its genera- tion. For relation modeling, we compare with the best model in Huang et (2022a), which incor- porates reasoning paths in knowledge graphs to generate entity relation descriptions. For Common- Gen, we compare with DKMR (He et al., 2022) (SOTA), RACo (Yu et al., 2022) (runner-up), and RE-T5 (Wang et al., 2021) in the leaderboard.
Implementation details. For each task, make the results comparable and reproducible, we adopt the same hyperparameters as the implementation of the authors to fine-tune BART. We also use the same hyperparameters as BART to fine-tune [REIVER on specific tasks. For definition model- ing, since Huang al. (2022b) use BART-base, we fine-tune [RE]VER- base for fair comparison. For relation modeling and generative commonsense reasoning, we use [RE]VER-large. To acquire sentences as the external knowledge for REVER, we use the same knowledge leveraged by CDM- S5,C5 (Huang et al., 2022b) for definition model- ing. For relation modeling, we retrieve sentences from WiV as described in Section 3.2. For Com- monGen, we leverage the knowledge retrieved by Matching Retriever of RE-T5 (Wang et al., 2021). For the low-resource settings, we randomly sample the corresponding number of training samples from the train sets. For all the models and settings, we train the models with enough epochs to ensure the training converges and select the checkpoint with the best validation performance.
Metrics. For definition modeling and relation modeling, we follow Huang et al. (2022b,a) to use BLEU (BL) (Papineni et al., 2002)6, ROUGE- L (R-L) (Lin, 2004), METEOR (MT) (Banerjee and Lavie, 2005), and BERTScore (BS) (Zhang et 2019) for automatic evaluation. Among them, BLEU, ROUGE-I, and METEOR focus mea- suring surface similarities by n-gram overlap, and BERTScore is based on the similarities of contex- tual token embeddings. For the evaluation on gener- ative commonsense reasoning, we follow Lin et al. (2020) to use BLEU-4, CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016), where
4.2 Definition Modeling
In Table 1, we report the results of definition mod- eling on the four datasets. For the full-data set-
"The version implemented on mjpost/sacrebleu.
com/

---

ting (100%). VER outperforms BART on all the datasets. In the low-resource setting (10%, i.c.. fine-tune the model with only 10% of the data), we observe that VER achieves a more significant im- provement. Besides, by leveraging the same exter- nal knowledge of the SOTA (Huang et al., 2022b), REVER can outperform the SOTA. The results demonstrate that after continually pre-training the model with the entity(s)-to- to-seeeteee reconstruction task, the model acquires more knowledge about entities and has a better ability to verbalize entities.
4.3 (Open) Relation Modeling
Figure 3 summarizes the results of open relation modeling. We observe that [REJVER consistently outperforms SOTA the four metries, and the performance improvement more significant when the model is fine-tuned on less training data. We also find that the performance of the model without any additional fine-tuning (# Training Examples 0) is quite impressive. On two metrics (R-L and MT), the performance is even better than BART trained with 50,000 examples.
Since [RE]VER
The results indicate that rich entity and relational knowledge are learned by [RE]VER through con- tinual pre-training. Besides, the text reconstruction task enables the model to produce natural language decriptions of relations by connecting entities in a coherent sentence.
Score (1-5)
generate sentence
(possibly a definitional sentence) by taking any entity as input without fine-tuning, we also report the results,
BART
1.13
VER
2.25
SOTA
3.98
REVER 4.51
Table 2: Averaged human annotated scores on UJ- CS (10% training data).
Table 3 showcases the human evalua- tion results for open relation modeling. Remarkably, both VER and REVER signifi- cantly outperform the state-of-the-art in low-resource setting
Score (1-5)
where no training data on the specific tasks are used to fine-tune the model. We find that VER (0%) can achieve better performance than BART (10%) and REVER can even outperform BART trained with full data, which indicates the strong performance [RE]VER on definition modeling without additional training.
BART
2.05
VER
2.79
SOTA
2.23
REVER
3.67
Table 3: Averaged hu- man scores ORM (500 training examples).
(trained with only 500 examples). However, we do note that [RE]VER still grapples with hallucination. leading to inaccuracies in the generated relation descriptions. For instance, some outputs wrongly state a location to be in a different city, albeit in the correct state and country. Nonetheless, considering the model is trained with just 500 examples, the results are still quite impressive.
To validate whether the joint training of relation modeling will benefit or harm the performance of definition modeling, we train version of VER only with data examples where []| = (VERT). From the results in Table 1, we observe that the performance of VER is slightly worse than VER, which means relation understanding by relation modeling and hyper-relation modeling can benefit (at least does not harm) definition modeling.
4.4 Hyper-Relation Modeling (Generative Commonsense Reasoning)
Table 2 presents the human evaluation results.
We observe that when trained with only 10% of the training data (1173 examples), BART struggles generate meaningful definitions, with most of its attempts receiving score of 1. VER, while able to produce some meaningful definitions, still falls short of the desired quality, as many of the definitions it generates contain errors. On the other hand, REVER performs remarkably well, achiev- ing an average score of 4.51. This is significant leap over SOTA results. It demonstrates that, even in the low-resource setting, REVER can generate definitions of exceptional quality. This underscores the importance of both pre-training and retrieved knowledge for generating definitions of entities.
Table 4 reports the CommonGen leaderboard re- sults of [RE]VER and baselines. We find that although the style of sentences used to pre-train [RE]VER quite different from that in Common- Gen, e.g., "A dog leaps to cutch a thrown frisbee"" the continual pre-training step still benefits the generative commonsense reasoning ability of the model. Besides, we observe that REVER outper- forms RACo on two metrics, despite the external knowledge base used in REVER (same RE-T5) is much smaller than RACo.
From the results of low low-resouree experiments in Figure 4, we observe that the improvement of

---

27.5
54
24
52
22.5
50-
70.0
70
V2R
REVER
50TA
BART
15.0
50
500
5000
50000
50
500
5000
50000
5000
50500
50
500
5000
5050n
#Trwning Pramples
drriinng Paamples
#Training Fxsmpies
#Trathing Posmpies
Figure 3: Results of open relation modeling (ORM) with different numbers training examples.
BLEU-4
CIDEr
SPICE
5
Related Work
BART
31.83
13.98
28.00
Definition modeling aims to generate definition for given entity/term. This problem was first studied in Noraset et al. (2017) in form of generating definitions of words with word embeddings. Later works focus on gener- ating definitions with contexts or external knowl- edge (Gadetsky et al., 2018; Ishiwatari et al., 2019; Washio et al., 2019; Mickus et al., 2019; Li et al.. 2020; Reid et al., 2020; Bevilacqua et al., 2020; Huang et al., 2021, 2022b; August et al., 2022). For instance, Bevilacqua et al. (2020) fine-tune BART (Lewis al., 2020) on the word/phrase-definitoon pairs with given contexts. August al. (2022) aim to control the complexity of the definition while generating the defimition for a given term. Huang et al. (2022b) propose to combine definition ex- traction and definition generation to improve the performance of definition modeling.
Definition Modeling.
VER
34.22
16.28
28.28
RE-T5
40.86
17.66
31.08
RACo (runner-up)
43.12
19.14
34.03
DKMR" (SOTA)
44.33
19.54
34.59
REVER
43.55
19.19
33.70
Table 4: Results on CommonGen (leaderboard vl.1).
The best results bold and second best ones underlined.
[RE]VER in the low-resource settings is very sig- nificant, despite the style of sentences in the pre- training and fine-tuning being quite different. Al- though the zero-shot performance of [RE]VER on CommonGen poor, with 50 training examples, [RE]VER can achieve better results than BART trained with 5,000 training examples on CIDEr and SPICE (according to Lin et al. (2020), SPICE and CIDEr correlate the human evaluation the most). This indicates the domain adaptation ability of [RE]VER is high.
(Open) Relation Modeling. Open Relation Mod- eling (Huang et al., 2022a) aims generate sen- tence to describe the relationship within given entity pair. The authors propose to fine-tune BART and incorporate reasoning paths in knowledge graphs as auxiliary knowledge to solve this task. As follow-up work, Huang et al. (2022c); Zhu et al. (2023) construct Descriptive Knowledge Graph by extracting and generating sentences explaining tity relationships with the analysis of dependency patterns and transformer-bart based relation descrip- tion synthesizing model.
In Table 5, we present some sample outputs of the models. Here
catch, throw) is an example in the test set of CommonGen that used as the demonstration in their paper. From the re- sults, we find that despite all the baselines failing in this example. VER and REVER both produce plausible sentence describing correct everyday scenario using all the concepts. We also find that the performance VER low-resource settings is very impressive. For instance, BART trained with 5,000 training examples cannot even generate a sentence containing all the concepts, and the gen- erated sentence describes a weird scenario, while VER trained with only 50 examples can generate a coherent sentence containing all the concepts. Be- sides, without fine-tuning, BART cannot generate anything meaningful, while [RE]VER can still gen- erate a reasonable sentence using all the concepts, although the style of the sentence is different from the ground truth.
Generative Commonsense Reasoning, Genera- tive Commonsense Reasoning (Lin et al., 2020; Liu et al., 2023) is a constrained text generation task that tests machines' ability to generate coherent sentence describing everyday scenarios contain- ing the given concepts. Later works mainly focus on improving performance by retrieving external knowledge to help the generation. For instance, KG-BART (Liu et al., 2021) designs a knowledge

---

34
16
14
30
25
C1DE
28
12
24-
22
REVER
VER
BART
aTraining Examples
Traiiinng Examples
Trairing Examples
Figure 4: Results of the low-resoure experiments CommonGen (in-house) with different numbers of training examples. Since the code the SOTA, ie., DKMR², not released, we do not report its performance here. For comparison in the full data setting, please refer to Table 4.
(dog, frisbee,
throw))
Human I
A dog leaps to catch a shrown frishee.
Human
dog catcher friabee when the
Human
A man throws away his dog favorito frisher expecting him catcl it in the
GPT-2
dog throws frisher focthall player.
UnILM
Two dogs are throwing frishees at each other.
BART
dog throws frishee and dog catches
dog frishee
dog
VER
throwing frisbee dog, who cutcher
BART
VER
a dog that is trained throw and reirieve a frishee by its handler given the task of making a catch and ihrow of the disc.
BART
(50)
boy playing, frishee with his triends
(50)
dog catches frishee
person.
BART
(500)
A dog cutches a frisbee during lootball gume..
VER
frishee throws it.
BART
(5000)
man throwing a frishee to woman who catching it.
VER
(5000)
Two dogs playing frishee and of them catching shrowing
REVER
The hagins shrow fröshee, and the jumps into the eatck
REVER
man shrowing a frizher and his dog catching it.
Table 5: Sentences produced by commonly-used pre-trained models and [REJVER. VER (50) refers to VER fine-tuned with 50 training examples. Here we take the example the demonstration Lin al. (2020).
model that incorporates the em- beddings of relations of concepts from ConceptNet (Speer et al., 2017) as auxiliary inputs of BART. EKI-BART (Fan et al., 2020), Re-T5 (Wang et al., 2021), KFCNet (Li et al., 2021), DKMR² (He et al., 2022), and RACo (Yu et al., 2022) retrieve proto- type sentences from external corpora as auxiliary input language models such BART and T5 (Raffel et al., 2020). In this work, show that continual training on verbalizing entities and re- lations can improve models' generative common- sense reasoning ability, either with or without ex- ternal knowledge.
There various applications of [REIVER. First, [RE]VER itself can be used as tool for humans explore entities and relations by pro- viding interpretable text descriptions, which can help humans better understand entities and rela- tions. This is particularly useful in the scientific domain, where researchers come across new terms every day and want to understand previously un- known concepts and relationships between relevant concepts (Zhu et al., 2023), and in the e-commerce domain, where users want to understand the func- tion of specific products and the relationship be- tween the recommended product and the product he/she already bought (e.g., tripod and camera) (Huang et al., 2023a). Second, as shown in our experiments, [RE]VER can be applied to improve the performance entity and relation verbaliza- tion tasks such definition modeling, relation modeling, and generative commonsense reason- ing. Third, [RE]VER can serve as knowledge source to provide knowledge on entities and re- lations to enhance models designed for entity & relation-related tasks (Ren et al., 2016; Bach and Badaskar, 2007; Lin et al., 2015).
6 Conclusion
In this paper, we propose [RE]VER : unified model for Verbalizing Entities and Relations. We combine definition modeling, relation modeling, and hyper-relation modcling in unified form and large training data by forming the "entity(s) + sentence" reconstruction task. Extensive experiments on three tasks and six datasets demonstrate the superiority of our model, especially in low-resource settings.
pre-train [RE]VER

---

Limitations
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
There are two main limitations of this work. First, we do not address ambiguity explicitly in the model training process. During the data preprocessing process, entities are represented by their unique identifiers in Wikidata, eliminating ambiguity and ensuring consistency. However, the input to the models does not include these identifiers (i.e., only the surface name is used). We have chosen this de- sign to increase the system's 's flexibility, as users are not required to provide identifiers, and the model can handle unseen entities (e.g., those without an identifier). The model may be adapted to deal with ambiguity by including identifiers as part of the input during training.
Michele Bevilacqua, Marco Maru, and Roberto Navigli, 2020. Generationary hhow we went beyond word sense inventories and learned to gloss". In Proceed- ings of the 2020 Conference on Empiricul Methods in Natural Language Processing (EMNLP), pages 7207-7221.
Alain Bretto. 2013. Hypergraph theory. An introduction. Mathematical Engineering. Cham: Springer.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901.
Second, although continually pretraining enables [RE]VER to generate definitions or relation defi- nitions for entities in zero-shot setting, the per- formance still leaves much to be desired. How to further improve the zero-shot performance is an interesting and important future research direction.
Kevin Clark and Christopher D Manning. 2016. Deep reinforcement learning for mention-ranking corefer- ence models. In EMNLP.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.
Acknowledgements
We thank the reviewers for their constructive feed- back. This material is based upon work supported by the National Science Foundation IIS 16- 19302 and IIS 16- 16-33755, Zhejiang University ZJU Re- scarch 083650, IBM-Illinois Center for Cognitive Computing Systems Research (C3SR) and IBM- Illinois Discovery Accelerator Institute (IIDA). grants from eBay and Microsoft Azure, UIUC OVCR CCIL Planning Grant 434S34, UIUC CSBS Small Grant 434C8U, and UTUC New Frontiers Initiative. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the funding agencies.
Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang. Yameng Huang. Jian Jiao, Xuan-Jing Huang. Nan Duan, and Ruofei Zhang. 2020. An enhanced knowl- edge injection model for commonsense generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2014-2025.
Artyom Gadetsky, llya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics ( (Volume 2: Short Popers), pages 266-271.
Noah Gardner, Hafiz Khan, and Chih-Cheng Hung. 2022. Definition modeling: literature review and dataset analysis. Applied Computing and Intelli- gence, 2(1):83-98.
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929 3929-3938. PMLR.
Tal August, Katharina Reinecke, and Noah A Smith. 2022. Generating scientific definitions with control- lable complexity. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 8298-8317.
Xingwei He,, Yeyun Gong, A Jin, Weizhen Qi. Hang Zhang, Jian Jiao, Bartuer Zhou, Biao Cheng. Siu Ming Yiu, Nan Duan, et al. 2022. Metric-guided distillation: Distilling knowledge from the metric to ranker and retriever for generative commonsense reasoning. arXiv preprint arXiv:2210.117000
Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. Literature review for Language and Statistics 11, 2:1 1:1-15.

---

Han Huang, Tomoyuki Kajiwara, and Yuki Arase. 2021. Definition modelling for appropriate specificity. In Proceedings of the 2021 Conference Empirical Methods Natural Language Processing, pages 2499-2509.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising squuence-to-sqqenne pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880
Jie Huang. Kevin Chang, Jinjun Xiong, and Wen-Mei Hwu. 2022a. Open relation modeling: Learning to define relations between entities. In Findings of the Association for Computational Linguisties: ACL 2022, pages 297-308.
Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang. Tim- othy Baldwin, and Nan Duan. 2021. Kfenet: Knowl- edge filtering and contrastive learning network for generative commonsense reasoning. arXiv preprint arXiv:2109.06704
Jie Huang, Chuan Chen, Fanghua Ye, Weibo Hu, and Zibin Zheng. 2020. Nonuniform hyper-network em- bedding with dual mechanism. ACM Transactions on Information Systems (TOIS), 38(3):1-18.
Jiahuan Li. Yu Bao, Shujian Huang, Xinyu Dai, and CHEN Jiajun. 2020. Explicit semantic decomposi- tion for definition generation. In Proceedings of the 58th Annual Meeting of the Association for Compu- national Linguistics, pages 708-717.
Jie Huang, Yifan Gao, Zheng Li, Jingfeng Yang. Yanggiu Song, Chao Zhang. Zining Zhu, Haom- ing Jiang, Kevin Chen-Chuan Chang, and Bing Yin. 2023a. CCGen: Explainable complementary concept generation in e-commerce. arXiv preprint arXiv:2305. 11480.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1823-1840.
Jie Huang, Xin Liu, and Yangqiu Song. 2019, Hyper-path path-based representation learning for hyper- networks. In Proceedingy of the 28th ACM Interna- tional Conference on Information and Knowledge Management, pages 449 458.
Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. 2023b. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint 23X1v22308.07922.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 81-
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embed- dings for knowledge graph completion. Twenty- ninth AAAI conference on artificial intelligence.
Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2022b. Under- standing jargon: Combining extraction and genera- tion for definition modeling. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP).
Chenzhengyi Liu, Jie Huang. Kerui Zhu, and Kevin Chen-Chus Chang. 2023. Dimongen: Diversified generative commonsense reasoning for explaining concept relationships. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.
Jie Huang, Kerui Zhu, Kevin Chen-Chuaa Chang, Jinjun Xiong. and Wen-mei Hwu, 2022c. Deer: Descriptive knowledge graph for explaining entity relationships. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Ye Liu, Yao Wan, Lifang He, Hao Peng, and S Yu Philip. 2021. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6418-6425.
Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshi- naga, Graham Neubig, Shoctsu Sato, Masashi Toy- oda. and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global con- texts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 3467-3470
Timothee Mickus, D Paperno, and Mathieu Constant. 2019. Mark my word: A sequence- coeticcett--qqueee to-seueence ap- proach to definition modeling. In Proceedings of The First NLPL Workshop on Deep Learning for Natural Language Processing, page 1. Linköping University Electronic Press.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sehastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with retrieval augmented language models, arXiv preprint arXiv, 2208.
Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey, 2017. Definition modeling: Learning to define word embeddings in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- lional Linguistics, pages 311-318.
Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. 2021. Retrieval enhanced model for commonsense gener- ation. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 2021, pages 3056- 3062.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473.
Peifeng Wang. Jonathan 7amora, Junfeng Liu, Filip llievski, Muhao Chen, and Xiang Ren. 2022. Con- textualized scene imagination for generative com- monsense reasoning. In International Conference Learning Representations.
Koki Washio. Satoshi Sekine, and Tsuneaki Kato. 2019. Bridging the defined and the defining: Exploiting implicit lexical semantic relations in definition mod- eling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- -JCNLPP, pages
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Πly Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
ral Language Processing 3521-35277
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-tex, text trans- former. Journal of Machine Learning Research, 21:1- 67.
Wenhao Yu, Chenguang Zhu, Zhihan Zhang, Shuohang Wang, Zhuosheng Zhang, Yuwei Fang. and Meng Jiang. 2022. Retrieval augmentation for common- sense reasoning: A unified approach. arXiv preprint arXiv:2210.12887.
Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. 2020, Vedm: Leveraging variational bi- encoding and deep contextualized word representa- tions for improved definition modeling. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6331-6344.
Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi 2019. Bertscore: Evaluating text generation with bert. In International Confer- ence Learning Representations.
Kerui Zhu, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Descriptive knowledge graph in biomedical domain. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
Xiang Ren, Wenqi He, Meng Qu, Clare Voss, Heng Ji, and Jiawei Han. 2016. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), ), pages 5418-5426,
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Thirty-first AAAI conference artificial intelligence.
Ke Tu, Peng Cui, Xiao Wang. Fei Wang, and Wenwu Zhu. 2018. Structural deep embedding for hyper- networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Ramakrishua Vedantam, C Lawrence Zimick. and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4575.


Reviews:
[{"review_id": "guYMkS3far", "reviewer": "Reviewer_ZaBz", "summary": "", "questions": "How to detect and mitigate model hallucination?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thorough review and the insights provided. We understand the concerns raised and would like to address them in the following manner:\n\n**Ambiguity:** \n\nAs mentioned in our limitation section (Lines 617-625), the model can be adapted to deal with ambiguity by including identifiers as part of the input during training. We opted not to include identifiers to enhance the system's flexibility, as users are not required to provide identifiers, and the model can handle unseen entities (e.g., those without an identifier).\n\nIt’s also important to underscore that the expansive dataset our model was trained on equips it to effectively manage a significant majority of potential ambiguities. For instance, in relation modeling for (Apple, Google), the context provided by the word \"Google\" indicates the intended meaning of \"Apple.\" For definition modeling, users can refine the input to circumvent ambiguity, such as specifying \"Apple Inc.\" or \"Apple (company).\"\n\n**Zero-shot Performance:** \n\nOur model's performance in low-resource settings serves as evidence of its potential. For zero-shot scenarios, while there's room for improvement, the model still demonstrates considerable proficiency. We have observed that the model, due to its prior extensive training, has a foundational capability that aids in many zero-shot scenarios. As an illustration, for definition modeling, REVER’s zero-shot performance surpasses that of BART trained on the full training data.\n\n**Diversity in Datasets:**\n\nWe conducted evaluations across a wide spectrum, spanning six datasets and various configurations including zero-shot, few-shot, and full-data settings. This comprehensive assessment should aptly capture the diversity and intricacy inherent to real-world entities and relationships.\n\n**Dependence on Wikipedia:**\n\nAlthough we primarily used Wikipedia, it was selected for its comprehensive representation of general knowledge. As showcased in our experiments on CommonGen (Section 4.4), the model isn't restricted to Wikipedia-specific domains. For instance, even though the style of sentences used to pre-train [RE]VER differ from those in CommonGen – consider the example, \"A dog leaps to catch a thrown frisbee\" – the continual pre-training step still benefits generative commonsense reasoning.\n\n**Regarding Model Hallucination:**\n\nThis concern is at the forefront of many generative models, including the most advanced large language models like ChatGPT. While we haven't explicitly detailed a mechanism in our current paper, our REVER training approach significantly curtails hallucination. For instance, the model is less prone to hallucination when working with higher quality retrieved sentences. We firmly believe that enhancing the quality of retrieved sentences is pivotal in reducing hallucinations. We plan to explore this more thoroughly in our future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A dataset was ready for use by the NLP community to follow up this research on entity and relation verbalization tasks, such as definition modeling, relation modeling, and generative commonsense reasoning.", "reasons_to_reject": "A few limitations need to be addressed as follows.\n\n1. Ambiguity: The authors acknowledge that their model does not explicitly address ambiguity in the training process. This could potentially lead to incorrect or misleading verbalizations, especially when dealing with homonyms or polysemous words. \n\n2. Zero-shot performance: While the model performs well in low-resource settings, its zero-shot performance is not as impressive. This could limit the model's applicability in real-world scenarios where training data may not always be available.\n\n3. Lack of diversity in datasets: The model is trained and evaluated on a limited number of datasets, which may not fully represent the diversity and complexity of real-world entities and relations. \n\n4. Dependence on Wikipedia: The model heavily relies on Wikipedia for pre-training data. This could potentially limit the model's ability to generalize to other domains or types of text.\n", "questions_to_authors": "How to detect and mitigate model hallucination?\n", "paper_topic_and_main_contributions": "This paper presents a system that can generate sentences that represent entities and their relationships. It proposes a unified model for verbalizing entities and relations, i.e., generating high-quality sentences describing entities and their relationships. A large dataset of entity sets and sentences is used to pre-train the model in a self-supervised text reconstruction task. The authors also conduct extensive experiments to evaluate the effectiveness of the model, such as definition modeling, relation modeling, and generative commonsense reasoning. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "P3SW4bIP4i", "reviewer": "Reviewer_W9C6", "summary": "", "questions": "- How did you pre-trained the baseline BART on low-resource settings without using the proposed WiV dataset? \n- In line 280, the representation of the input text is unclear. If you encode the input using the set of entities and the augmented sentences, then what is the output text produced by the decoder? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comprehensive review of our paper and your acknowledgment of its clarity, extensive experiments, and impressive results.\n\n**Response to W1 “novelty”**\n\nThe beauty and novelty of our approach lies in the idea of unifying and its simplicity paired with its effectiveness, as substantiated by the impressive results highlighted by the reviewer.\n\n**Response to W2 “motivation for unifying tasks”**\n\nFirst, training and deploying a single model for multiple tasks conserves resources.\n\nSecond, as evidenced by the results of VER- and VER in Table 1, integrating the two tasks can also offer slight enhancements to definition modeling.\n\nThird, a unified approach is crucial for applications that require seamless integration of entity definitions and entity relationships. A unified approach ensures consistent representation and understanding of entities and their relationships.\n\nWe will emphasize the motivation more explicitly in the final version.\n\n**Response to W3 “performance on hyper-relation modeling”**\n\nThank you for your comments! We agree that a more informative and structured corpus could enhance performance. However, even without this, our model still achieves performance comparable to SOTA on CommonGen. This is noteworthy, especially considering that the SOTA is specifically tailored for the CommonGen benchmark.\n\n**Response to Q1 “Pre-training of Baseline BART on Low-resource Settings”**\n\nWe didn't pre-train the BART baseline from scratch in the low-resource setting. Instead, as indicated in Line 368-371, BART (with the official pretrained weights) was fine-tuned on the low-resource portion of each task-specific dataset, benefiting its pre-existing knowledge gained from large-scale pre-training. We will make this point more clear in the final version.\n\n**Response to Q2 “Representation of Input Text (Line 280)”**\n\nOur encoder takes both the set of entities and the augmented sentences as input. Subsequently, the decoder generates the target sentence containing all entities from the set. As clarified in Line 274, to prevent the target sentence from mirroring the augmented sentence, we exclude the target sentence during retrieval.\n\n**Clarification on Table 4**\n\nThe bolded results indicate the best performing methods across the row, while underlined results signify the second-best performance. We will provide a clear description in the caption of the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-written and easy-to-follow paper. \n- Extensive experiments using three tasks (definition modelling, pairwise relation modelling and hyper-relation modelling for relations between more than two entities) and different settings (no fine-tuning, low-resource fine-tuning and full fine-tuning) and multiple evaluation metrics including human evaluations. \n- The obtained quantitative and qualitative results are impressive despite the simplicity of the proposed idea. ", "reasons_to_reject": "- The novelty of the work is limited. \n- The importance of unifying the two tasks of defining entities and defining relations between entities is not well-motivated. \n- On the challenging setting with more than two entities (hyper-relation modelling), the proposed model shows lower performance than the state-of-the-art model. This indicates the need of using a more informative and structured corpus for relationship definitions between concepts than the proposed dataset. ", "questions_to_authors": "- How did you pre-trained the baseline BART on low-resource settings without using the proposed WiV dataset? \n- In line 280, the representation of the input text is unclear. If you encode the input using the set of entities and the augmented sentences, then what is the output text produced by the decoder? ", "paper_topic_and_main_contributions": "This work proposes a unifying model to generate definitions for a set of entities by pre-training encoder-decoder transformers (BART) on a proposed dataset WiD from Wikipedia that has input-output pairs of entity_set-sentences. WiD uses the first sentences of Wikipedia pages for entity definitions and entity-annotated sentences to represent relations between a set of annotated entities. Considering three settings (no fine-tuning, low-resource fine-tuning and full fine-tuning) for three tasks (definition modelling, pairwise relation modelling and hyper-relation modelling for relations between more than two entities), the proposed model generates better definitions than the baselines. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Clarify the difference between bolded and underlined results in the caption of Table 4. "}, "extra_text": {}}, {"review_id": "E3xrVvM5xw", "reviewer": "Reviewer_hSHA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and positive feedback on our paper. We are glad that the reviewer recognized the benefits of our unified approach for different tasks and valued the comprehensive evaluation.\n\nThank you for your suggestion to include examples in the appendix. In the current version, we displayed some generation examples from CommonGen in Table 5. We will add more examples in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- (+) Unified approach to different tasks (definition, relation and hyper-relation modeling).\n- (+) Thorough evaluation and state-of-the art results.\n", "reasons_to_reject": "- (-) None, really. As a comment, I think the paper would improve by including even a small overview of examples of good and bad generations in the appendix. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a model for verbalising entity and relations, namely generating descriptions of entities and entity relations using pre-trained models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a unified method to address a number of tasks involving entities and relations. The main contribution is the elegance of the model, paired with very positive results in some scenarios. However, the generalizability of the method is not fully proved as demonstrated by mixed results in some scenarios.
Decision: Accept-Findings