VER: Learning Natural Language Representations for Verbalizing Entities and Relations
Authors: Jie Huang
Kevin Chang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Entities and relationships between entities are vital in the real world. Essentially, we understand the world by understanding entities and relations. For instance, to understand a field, e.g., computer science, we need to understand the relevant concepts, e.g., machine learning, and the relationships between concepts, e.g., machine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she is related to others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning a new scientific term, people usually start by reading its definition in dictionaries or encyclopedias. To know the relationship between two entities, humans tend to create a sentence to connect them. In this paper, we propose VER: A Unified Model for Verbalizing Entities and Relations. Specifically, we attempt to build a system that takes any entity or entity set as input and generates a sentence to represent entities and relations, named ``natural language representation''. Extensive experiments demonstrate that our model can generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, including definition modeling, relation modeling, and generative commonsense reasoning.

Under review as a conference paper at ICLR 2023
VER : LEARNING NATURAL LANGUAGE REPRE- SENTATIONS FOR VERBALIZING ENTITIES AND RELA- TIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Entities and relationships between entities are vital in the real world. Essentially, we understand the world by understanding entities and relations. For instance, to understand field, e.g. computer science, we need understand the relevant concepts, e.g., machine learning, and the relationships between concepts, e.g., ma- chine learning and artificial intelligence. To understand a person, we should first know who he/she is and how he/she is related to others. To understand entities and relations, humans may refer to natural language descriptions. For instance, when learning new scientifie term, people usually start by reading its definition in die- tionaries or encyclopedias. To know the relationship between two entities, humans tend to create sentence connect them. In this paper, propose VER A Unified Model for Verbalizing Entities and Relations. Specifically, we attempt to build a system that takes any entity or entity set as input and generates sentence represent entities and relations, named "natural language representation" Exten- sive experiments demonstrate that our model can generate high-quality sentences describing entities and entity relationships and facilitate various tasks on entities and relations, including definition modeling, relation modeling, and generative commonsense reasoning
INTRODUCTION
What is X? What is the relationship between X and Y? We come with these questions almost every day. When we come across a new term, e.g. twin prime, we usually refer its definition to understand it, i.e., "A twin prime is prime number that is either 2 less or 2 more than another prime number". To express the understanding about relationship between entities (e.g., carbon dioxide and water), we create sentence to represent their relationship: "Carbon dioxide is soluble water"" Basically, we understand entities and relations by "verbulizing" them. Verbalizing entities and relations also tests knowledge about entities and relations. Literally, by verbalizing entities and relations, we understand the world.
Similarly, do machines have the ability to verbalize entities and relations? Can machines learn about entities and relations from verbalizing them? The answer is "Yes". Recent studies show that by giving the surface name of an entity (and its context), models (after training)
generate coherent
sentences to represent it, i.c., definition modeling (Noraset et al.) 2017 Gadetsky et al.) 2018 Bevilaaqua al., 2020; August
2022: Huang et al.) 2021 Gardner al. al.] 2022), and by giving the surface names of pair of entities, machines can generate coherent sentences describing their relationships, i.c.. (open) relation modeling (Huang al. 2022a b). However, verbalizing entities requires understanding relationships between entities, and verbalizing entity relationships requires understanding entities themselves, while existing works deal with entity and relation verbalization separately, ignoring the connections between them.
Besides, recent works (Devlin et al. 2019 Lewis al.) 2020 Radford et al. 2019 Brown et al.) 2020 have shown that large language models pre-trained with self-supervised objectives can equip
We release the TER-base model on this anonymous repo otont /vwww _o.loy_ 163d9cdd2254 laRelarge and the pre-training data will be released as open- after the review process (since the anonymous repo has space limitation).

---

Under review as a conference paper at ICLR 2023
Definition Modeling
Carbon dioxide a cherical compound made mulecules that each have carbon atom covalently double bonded two esygen atoms, found in the statle at lemperature.
carbon dioxide
Relation Modeling
carbon dioxide;
VER
Carbon diaxide soluble
Hyper-Reee Relation Modeling
carbon dioxidee; arbonic acid
Carbon disciil reacts with water produce arbonic acid.
Figure I: A diagram of VER. We feed the model with entity(s) and train it to reconstruct sentences containing all the entities. This allows us to use a single model to better "verbalize" entities and complex entity relationships.
the model with a significant amount of knowledge (Petroni et al,, 2019; Roberts et al. 2020) and achieve substantial gains after fine-tuning specific task. Can we continually pre- traitrain the models with pre-training objectives on entities and relations to enhance their ability on verbalizing entities and relations? In this way, the model be easier and better adapted specific tasks entities and relations and even be used without additional training.
Therefore, aim solve entity and relation verbalization in unified form and pre-train model for entity and relation understanding. Essentially, definition modeling and relation modeling can "entity(s) - sentence" task, i.e., given describing the entities and their relationships. When the size of the set is 1, it is equivalent to definition modeling, and when the size of the set is 2, it is equivalent to relation modeling. By defining the task in this form, we can even model more complex relationships among entities since entity relationships can go beyond pairwise (Bretto 2013), named hyper-relation modeling, e.g., {carbon dioxide, water, arbonic acid) - "Carbon dioxide reacts with water to produce arbonic acid". Based on this, we propose VER : A Unified Model for Verbalizing Entities and Relations (Figure []. Specifically, we pre-train models by forming self-supervised text reconstruction task: given an entity or a set of entities, reconstruct the original sentences (e.g., a definition or a relation description) containing them in the training corpus. In this way, the models acquire knowledge about entities and relations and learn to connect entities to a meaningful coherent sentence.
be unified
of entities, generating sentence
From the perspective of representation learning, we propose learn the "natural language repre- sentations" of entities and relations. Compared to latent vector representations, natural language representations are more interpretable since humans understand the representations by reading the texts, while hidden representations are difficult to interpret. Compared to structural represen- tations with pre-specified rules, e.g., a sub-knowledge graph, natural language representations more open owing to the flexibility of free texts.
Experiments on six datasets demonstrate the superiority of our model in verbalizing entities and relations. Especially in low-resource settings, our model can achieve significantly better results than BART LLwis al. 2020) on definition modeling, relation modeling, and hyper-relation modeling (generative commonsense reasoning (Lin et al., (2020). In addition, the performance of VER without additional training is impressive, making itself a potential knowledge source of entities and relations, which may benefit tasks on entities and relations such as entity typing Ren et al. (2016), relation extraction (Bach & Badaskar 2007). and knowledge graph completion (Lin et al. 2015).
The main contributions of our work are summarized as follows:
We connect definition modeling, relation modeling, and hyper-relation modeling a unified form; We pre-train VER large training data by forming the "entity(s) - sentence" reconstruction task, which makes VER a useful tool for learning natural language representations entities and relations;
Extensive experiments demonstrate our model can achieve better results in verbalizing entities and relations, especially in low-resource settings.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND AND FORMULATIONS
Definition Modeling. Definition modeling aims to generate definitions of entities, which can be formulated conditioned sequence generation task. For instance, given twin prime, the expected output is the definition of twin prime: "A twin prime is prime number that is either 2 less or more than another prime number". We follow the standard sgueenceto--eqqenne formulation in (Noraset et al. 2017: Huang et al. 2021b): given entity 2, the probability of the generated definition $1..... Sm) is computed auto regressiellys
P(s(x) = II 81, 1,-1, :=1
where m is the length of s, 8, is the ith token of s, and su is special start token.
(Open) Relation Modeling, Relation modeling attempts to generate coherent and meaningful sen- tences describing relationships between entities, where types of relations in an "open" setting (Huang et al. 2022a)). For example, for carbon dioxide and water, their relation- ship can be described as "Carbon dioxide soluble water." For machine learning and algorithm, the expected output could be "Machine learning explores the study and construction of algorithms that can learn from and make predictions on data." " Formally, given entity pair (r. y), the probability of the generated relation description [si...., 8m] is calculated as:
pre-specified, i.e.,
P(s|2, v) II P(s (S, 81, ................................ 81-1, 22, y)-
Elyper-Relation Modeling (Unified Form). Previous works mainly focus verbalizing single en- tities or entity pairs. However, in the real world, relationships between entities can be more complex beyond pairwise, named "hyper" relationships (Bretto 2013 Tu et al. 2018 Huang al. 2019). For example, "carbon dioxide reacts with water to produce carbonic acid Here, there are tuplewise relationships among carbon dioxide, water, and carbonic acid. Verbalization of hyper relationships was initially investigated in (Lin et al.] (2020) but was limited to commonsense concepts, and their outputs are simple short sentences describing everyday scenarios containing the given concepts. We attempt to model and verbalize more general complex "hyper" relationships among entities and find unified framework
combine single entities (1 entity). pairwise relationships (2 entities), and "hyper" relationships (23 entities). Combining with definition modeling and relation modeling, we adopt the following unified form:
m
P(s = P(s S1, 91..,,,,.........1.1111 (1,,,,,)))..,,,111,))
(3)
where & is the entity set and [8] 1.
Natural Language Representation. Natural language representation aims to represent entities and relations with text descriptions. Natural language representation is interpretable since texts are readable. Natural language representation is open since most things can be described in natural language. Based on the unified form above, we define natural language representation of entities and relations as r(E)= s. Here should notice that the surface name of entities does not provide much semantic knowledge; therefore, producing natural language representations requires the model to generate coherent sentences describing entities and relations based on knowledge stored in the parameters of the model the reasoning ability of the model.
3 VER ee VERBALIZING ENTITIES AND RELATIONS
To verbalize an entity, we are likely to connect it to other entities, which requires knowledge about entity relationships. To understand entity relationships, on this, we attempt to verbalize entities and relations in a unified form and propose VER : A Unified Model for Verbalizing Entities and Relations. We first create large dataset with the for- mulation in Eq. (3). and pre-train& a model on this dataset, which equips the model with signifi- cant amount of knowledge about entities and relations and enables the model to generate coherent
need to know about entities first. Based
3

---

Under review as a conference paper at ICLR 2023
and meaningful sentences connecting the entities. The model be further fine-tuned on specific datasets, e.g., definition modeling, relation modeling, and generative commonsense reasoning, to achieve better performance specific tasks.
3.1 DATA
We prepare the pre-training data with Wikipedia. Wikipedia is a large encyclo- pedia containing a huge number of entities. Wikipedia is well maintained and the content generally of high quality. We extract entity sets and sentences from Wikipedia, Specif- ically, we use the 2022-08-01 dump@ of En- glish Wikipedia. For each page. we extract the plain text by WikiExtractorf use the neuralcoref (Clark & Manning 2016) coreference resolution tool in spacy to pre- process the documents. Since we would like the model to capture the main characteris- ties of entities and relations, first 5 sentences from each page (those sen- tences are usually definitional sentences or sentences expressing entity relationships). To identify entities, we utilize the hyperlinks in each Wikipedia page to build local mention-entee entity mapping. And then, we process each sentence and extract the corresponding entity set based on the mention-entity mapping. In this way, we build map- ping & + 8, e.g.. " "(Data mining, data sets, machine learning, statistics, database systems} - Data mining is the process of extracting and discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems." Since for a single entity, we prefer the model to generate definition-like sentence rather than a random sentence including it, we collect the first sentence on each page and collect the input-output pair as "[[page title/} - Ist sentence", e.g., "[deep learning) - Deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning." We filter out input-output and s # 1st sentence. We keep pages appearing in the validation and test sets of (Huang et al., 2021b 2022a) and filter out entity sets appearing in the datasets of (Huang al. (2021]] August et al.) 2022; Huang et al. 2022a August al.) 2022 Lin al. 2020). The number of training examples with different sizes of entity sets is shown in Figure
4126967
3.0
2603028
2.0-
And we
1.5
1310000
590169
0.5
440912
1
P5
[4
take the
Figure 2: Statistics of the pre-training data.
pairs where [8]
3.2
MODEL
At a high level, we pre-train model by training it to reconstruct target sentences conditioned on the entity set with the pre-training data. Specifically, we continually pre-train BART (Lewis et al. (020) on the data constructed in Section 3.J. BART adopts transformer- ture with imput text fed to the encoder and output text produced by the decoder. For our continual pre-training, encode entity set È {e1, ....... } to sequence "e1; 62 ... Cjg]", e.g., (carbon dioxide, water, curbonic acid) to "carbon dioxide; water; carbonic acid". Here we keep the order of entities as the order they appear in the sentence. We choose this design because different orders may correspond to different natural language representations (c.g., the rep- resentations are different when an entity is used as subject vs. object). We would like to mention that although we keep the order here, the model can deal with inputs with random entity orders after fine-tuning (e.g., CommonGen (Lin et al.] 2020) as shown in Section 4.4). We train two versions of the model: VER-base with 6 layers in the encoder and decoder, and VER-large with 12 layers in each, corresponding to BART-based and BART-large respectively.
encoder-decoder architec-
httea///Athubb.coo////rrrrr////t https://spacy.io.

---

Under review as a conference paper at ICLR 2023
3.3 TRAINING PROCESS
We pre-train VER-large and VER-base with the fairseq library We use Adam with 31 = 0.9. 82 = 0.999, and 10-8 and set the clip threshold of gradients 0.1. Both models use weight decay of 0.001 and dropout of 0.1. We set the learning rate as 5 x 10-8 and use batch size of 1,024 tokens, updating every 16 iterations. We set the number of warmup steps as 1, 000. We use a small validation set to examine whether the training converges. Both models were trained on NVIDIA A100 GPUs, and the training converged in 60 epochs.
5
EXPERIMENTS
In this section, evaluate VER on definition modeling, relation modeling, and hyper-relation modeling in three settings: 1) fine-tune the model on the full task-specific training data; 2) fine-tune the model in low-resource settings; 3) directly use the model without fine-tuning. The main goal of the experiments is to verify whether the continual pre-traue pre-training step can enhance models' ability on verbalizing entities and relations without relying on external knowledge.
4.1 EXPERIMENTAL SETUP
Datasets. For definition modeling, we use the datasets presented in (Huang al., 202 (CS, Math, Phy) and (August et al. 2022) (DC). For relation modeling. we use the dataset built in (Huang et al. 2022a) (ORM), take the filtered test set for evaluation since the quality is higher. For hyper-relation modeling, there is no existing dataset. We find that CommonGen (Generative Commonsense Reasoning) (Lin al. 2020)
our purpose for evaluation since the task formulation is similar: given a set of common concepts, i.e., & (3 < |3| I 5), generating a coherent sentence describing an everyday scenario using these concepts. By testing
CommonGen, can also measure the ability of our model for domain adaptation. Since the reference sentences of the official test of CommonGen are not released, for the full data setting, we submit the results generated by the model to the leaderboard to get the performance. For the low-resource settings, we use the in-house split presented in (Wang al., 2022) to facilitate comparison between our model and the baseline.
Baselines. Since VER is trained based on BART (Lewis et al. 2020). we compare VER with BART. It is not our purpose advocate our model can achieve state-of thr performance on the specific tasks (they are usually achieved by leveraging external knowledge such as a knowledge graph. More details are in Section 5). What we want to verify is whether the continual pre-training step can enhance the ability of the model on verbalizing entities and relations. We may change the base model to other models such as T5 (Raffel et al., (020) and continually train the model in a similar manner to achieve different performance. Furthermore, other methods may replace their base model (e.g.. BART, if used) with VER to achieve better performance (e.g., VER + A Neurologic Lu et al. )022
Metrics. For definition modeling and relation modeling, we follow Huang et al. 2b22bb 2022a to use BL.UU (BI.) (Papineni al. 2002 6l ROUGE-L (R-L.) (Lin) 2004), METHOR (M') (Banerjee & Lavie 2005), and BERTScore (BS) (hang et al. 2019) for automatic evaluation. Among them, BLEU, ROUGE-L, and METEOR focus on measuring surface similarities by n-gram overlap, and BERTScore is based the similarities of contextual token embeddings. For the evaluation generative commonsense reasoning. we follow Lin et al. (2020) to use BLEU-4, CIDEr (Vedantam) et al. 2015). and SPICE (Anderson et al. (016)), where CIDEr and SPICE focus evaluating the concept association instead of n-gram overlap.
Implementation details. For each task, to make the results comparable and reproducible, we adopt the same hyperparameters the implementation of the authors to fine-tune BART. We also the same hyperparameters as BART to fine-tune VER on specific tasks. For definition modeling, since Huang al. (2021b) use BART-base, we fine-tune VER-base for fair comparison. For relation modeling and generative commonsense reasoning, we use VER-large. For the low-resource settings,
httpps: / httpg//hh..ccoocfcoooorreeeeeeee hwt.://oookosseercccaaaaiisseeee
The version implemented
5

---

Under review as a conference paper at ICLR 2023
Table 1: Results of definition modeling.
CS
Math
Phy
DC
100%
BL
R-L
MT
BS
BL
R-L
MT
BS
BL
R-L
MT
BS
BL
R-L
MT
BS
BART
8.31
28.02
12.83
77.97 6.89
28.50
10.97
76.45
5.28
25.75
10.57
76.88
13.13
31.75
13.30
79.31
VER
8.43
30.11
13.06
79.57
7.09
31.94
11.86
78.07
7.09
30.63
12.71
79.18
13.95
33.57
14.84
80.49
10%
R-I.
MT
R-1
MT
R-I.
MT
RS
B1
R-L.
MT
BART
3,50
22.98
8.68
75.55
4.32
25.42
8.94
75.21
3.27
24.19
8.43
75.72
5.56
23.97
9.47
77.13
VER
6.43
28.24
12.36
78.77
31.18
11.79
77.82
6,43
30.57
12.42
78.92
7.59
28.25
12.09
78.70
0%
BL
R-L
MT
BS
BL
R-L
MT
BS
BL
R-L
MT
BS
BL
R-L
MT
BS
VER
5.05
26.55
11.96
77.84
6.33
30.36
11.57
76.88
5.95
28.79
12.35
78.13
6.06
23.49
11.22
75.49
VER
4.81
26.24
11.62
77.55
6.00
30.57
11.41
77.35
5.70
28.62
12.12
78.06
5.98
22.84
11.01
75.32
88-
87-
25
74
85
04
DART
50 500 5000 500050000 50000 full
50 500 5000 50000 ful
5000 50000 Trairing Examples
500
50
500
5000 50000 ful
# Training Exemples
Training Exampies
Iraining Examplis
Figure 3: Results of open relation modeling (ORM) with different numbers of training examples.
we randomly sample the corresponding number of training samples from the train sets. For all the models and settings, we train the models with enough epochs to ensure the training converges and select the checkpoint with the best validation performance.
4.2 DEFINITION MODELING
In Table 1 report the results of definition modeling on the four datasets. For the full-data set- ting (100%). VER outperforms BART on all the datasets. In the low-resource setting (10%, i.c., fine-tune the model with only 10% of the data), observe that VER achieves more signifi- cant improvement. The results demonstrate that after continually pre-training the model with the -ntitt(s)-to-sseeencce reconstruction task, the model acquires more knowledge about entities and has a better ability to verbalize entities.
Since VER can generate sentence (possibly definitional sentence) by taking any entity input without fine-tuning, we also report the "0%" results, where no training data on the specific tasks are used to fine-tune the model. We find that VER (0%) achieve better performance than BART (10%), which indicates the strong performance of VER on definition modeling without additional training.
To validate whether the joint training of relation modeling will benefit or harm the performance of definition modeling, we train version of VER only with data examples where |3] 1 (VER From the results in Table we observe that the performance of VER is slightly worse than VER, which means relation understanding by relation modeling and hyper-relation modeling can benefit (at least does not harm) definition modeling.
4.3 (OPEN) RELATION MODELING
Figure summarizes the results of open relation modeling. We observe that VER consistently outperforms BART on the four metrics, and the performance improvement is more significant when the model is fine-tuned on less training data. We also find that the performance of the model without any additional tuning (# Training Examples 0) is quite impressive. On two metrics (R-L and MT), the performance is better than BART trained with 50,000 examples.

---

Under review as a conference paper at ICLR 2023
Table 2: Sentences produced by BART and VER fine-tuned ORM. VER (500) refers VER fine-tuned with 500 training examples.
Entities
(evaluation, computer science)
BART
(0)
uation:,
VER
The term "evaluation" is used in computer science to refer to the process of eval- uation a system.
BART
(500)
Computer science is the study of computer science.
VER
(500)
evaluation is term used in computer science describe the process of evaluating a program or system.
BART
(full)
In computer science, evaluation is the process of evaluating computer program determine whether it is suitable for the task at hand.
VER
(full)
mathematics and computer science, evaluation the process of determining whether statement is true or false.
Entities
{machine learning, algorithm)
BART
MachineMuchine Learning
VER
The learning algorithm is an algorithm used in the development artificial neural networks.
BART
(500)
Deep learning is type machine learning algorithm.
VER
(500)
In machine learning, a learning algorithm is a class of machine learning algo- rithms that be used to learn from data.
BART
(full)
Machine learning (ML) is a subfield of machine learning concerned with the design of algorithm s that can learn and learn beller over time. Machine learning (ML) is subfield of artificial intelligence that studies algo- rithm that learn how to make predictions based on observed data.
VER
(full)
The results indicate that rich entity and relational knowledge are learned by VER through continual pre-training. Besides, the text reconstruction task enables the model to produce natural language representations of relations by connecting entities in a coherent sentence. The robustness of VER in low-resource settings also justifies the benefits of conducting the continual pre-training step.
Table provides some generation examples of the models. For {evaluation, computer science), VER without any fine-tuning can generate a reasonable sentence describing what evaluation means in computer science, while BART pre-trained 500 training examples cannot produce mean- ingful sentence. For (machine learning, algorithm).. the sentence produced by VER (0) is not very meaningful; however, after fine-tuning the task-specific data (even with only 500 examples), the model can generate coherent sentence describing the relationship between machine learning and algorithm, while the sentence generated by BART trained with full data still does not make sense.
4.4
MODELING (GENERATIVE COMMONSENSE REASONING)
Table 3 reports the CommonGen leaderboard results of VER compared to BART. We find that al- though the style of sentences used to pre-train VER is quite different from that in CommonGen, c.g., "A dog leaps catch a thrown frisbee", the continual pre-training step still benefits the generative commonsense reasoning ability of the model.
From the results of low-resource experiments in Figure 4 we observe that the improvement of VER in the fow-resource settings is very significant, despite the style of sentences in the pre-training and fine-tuning being quite different. Although the zero-shot performance of VER on CommonGen is poor, with 50 training examples, VER can achieve better results than BART trained with 5,000 training examples on CIDEr and SPICE (according to (Lin et al., 2020). SPICE and CIDEr correlate the human evaluation the most). This indicates the domain adaptation ability of VER is high.
In Table we present some sample outputs of the models. Here {dog. frisbee, catch, throw) is an example in the test set of CommonGen that is used the demonstration in their paper. From the results, we find that despite all the baselines failing in this example, VER produces a plausible sentence describing correct everyday scenario using all the concepts. We also find that the perfor- mance of VER in fow-resource settings is very impressive. For instance, BART trained with 5,000 training examples cannot generate sentence containing all the concepts, and the generated

---

Under review as a conference paper at ICLR 2023
Table 3: Results CommonGen (leaderboard v1.1).
BLEU-
CIDEr
SPICE
BART
31.83
13.98
28.00
VER
34.22
16.28
28.28
32
30
25
20
15
20
500
5000
500
5000
50
500
5000
# Training Examples
# Training Examples
# Training Examples
Figure 4: Results of the low-resource experiments CommonGen (in-house) with different num- bers of training examples.
Table 4: Sentences produced by commonly-used pre-trained models and VER fine-tuned Com- monGen. VER (50) refers to VER fine-tuned with 50 training examples. Here we take the example in the demonstration of (Lin et al. 2020).
Concepts
(dog, catch, throw]
Human
A dog Icaps to catch thrown frisbee.
Human
The dog catches the frisbee when the boy throws it.
Human
man throws away his dog favorite favorite expecting him to catch it in the air. football player.
GPT-2
A dog throws frisbee
UniLM
Two dogs are throwing frisbees each other.
BART
dog throws a frisbee and dog catches it.
TS
dog catches a frishee and throws it dog
VER
man throwing a frishee to his dog, who catches it.
BART
(0)
::
VER
(0)
dog that is trained to throw and retrieve afrisbee by its handler is given the task of making catch and throw of the disc.
BART
(50)
boy is playing frishee with his friends
VER
(50)
a dog catches a frisbee and throws it to a person.
BART
(500)
A dog catches a frishee during football game.
VER
(500)
dog catches a f.frisbee and throws it.
BART
(5000)
A man is throwing frishee to woman who is catching it.
VER
(5000)
Two dogs are playing frisbee and one of them catching and throwing it.
sentence describes a weird scenario, while VER trained with only 50 examples can generate co- herent sentence containing all the concepts. Besides, without fine-tuning, BART cannot generate anything meaningful, while VER can still generate a reasonable sentence using the concepts, although the style of the sentence is different from the ground truth.
RELATED WORK
Definition Modeling. Definition modeling aims to generate a definition for a given entity/term.
This problem was first studied in Noraset et al. (2017) in form of generating definitions of words
with word embeddings. Later works include Gadetsky et al. (2018): 8) Ishiwatari et al. (2019): Washio
et al. (2019;; Mickus al. [2019: Li et al. (2020a); Reid et al. (2020); Bevilacqua al. (2020);
Huang et al.
2021a b); August et al. (2022). For instance, Beevilaquua et al. (2020) tune BART
Lewis al. 2020 the word/phrase-definite pairs with given contexts.
August al. (002))
8

---

Under review as a conference paper at ICLR 2023
aims to control the complexity of the definition while generating the definition for given term. Huang et al. (2021b) propose to combine definition extraction and definition generation to improve the performance of definition modeling.
(Open) Relation Modeling, Open Relation Modeling (Huang et al., 2022a) aims to generate sen- tence describe the relationship within given entity pair. The authors propose to fine-tune BART and incorporate reasoning paths in knowledge graphs as auxiliary knowledge to solve this task. As follow-up work, Huuan et al. (202b)) construct Descriptive Knowledge Graph by extracting and generating sentences explaining entity relationships with the analysis of dependency patterns and transformer-based relation description synthesizing model.
Generative Commonsense Reasoning. Generative Commonsense Reasoning (Lin et al. 2020) is
a constrained text generation task that tests machines' ability to generate a coherent sentence de- scribing everyday scenarios containing the given concepts. Later works mainly focus improving performance by retrieving external knowledge to help the generation. For instance, KG-BART (Liu al., 2021) designs knowledge graph-augmented model that incorporates the embeddings rela- tions of concepts from ConceptNet (Speer et al. 2017) as auxiliary inputs of BART. EKI-BART (Fan al., 2020), Re-T5 (Wang al. (021). and KFCNet eLi 202T) retrieve prototype sentences from external corpora as auxiliary input to language models such as BART and T5 (Raffel et al.) 2020 Although these methods enable the model to achieve better performance by utilizing those auxiliary information, they do not really improve the model's ability of generative commonsense reasoning (Note: the original purpose of CommonGen benchmark is "to explicitly test machines for the ability of generative commonsense reasoning" (Lin et al. 2020)). In this work, show that con- tinual training on verbalizing entities and relations, rather than incorporating auxiliary information in the input, can truly improve models' generative commonsense reasoning ability.
Natural Language Representation. In this paper, we define natural language representation as text description entities and relations. The concept can be further extended to other tasks. For stance, image captioning (Stefanini et al. 2021 You et al. 2016 Cornia et al. 2020) Li et al. (200bb) can be considered as learning natural language representations of images, document summarization (Allahyari et al., 2017) can be regarded as learning shorter natural language representations of doc- uments. Since the surface name of entities does not provide much semantic knowledge, different from previous works where the representation can be translated extracted from the input, natural language representations of entities and relations challenge the model's ability to generate coherent sentences deseribing entities and relations based its reasoning ability and knowledge stored in the parameters of the model.
6 CONCLUSION
In this paper, we propose VER A Unified Model for Verbalizing Entities and Relations. We combine definition modeling, relation modeling, and hyper-relation modeling in a unified form and pre-train VER a large training data by forming the "entity(s) sentence" reconstruction task. Extensive experiments on three tasks and six datasets demonstrate the superiority of our model, especially in low-resource settings.
There are various applications of VER. First, VER itself can be used as a tool for humans to explore entities and relations by providing interpretable text descriptions, which can help humans better un- derstand entities and relations. This is particularly useful in the scientific domain, where researchers come across new terms every day and want to understand previously unknown concepts and relation- ships between relevant concepts, and in the e-commerce domain, where users want to understand the function of specific products and the relationship between the recommended product and the product he/she already bought (e.g.. tripod and camera). Second, as shown in our experiments, VER can be applied to improve the performance on entity and relation verbalization tasks such as definition modeling, relation modeling, and generative commonsense reasoning. Third, VER can serve knowledge source to provide knowledge on entities and relations to enhance models designed for entity eretatio-roio tasks such as entity typing Ren et al. (2016), relation extraction (Bach & Badaskar 2007). and knowledge graph completion (Lin et al. 2013), which we leave as future work for and the whole research community.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. Text summarization techniques: brief survey. arXiv preprint arXiv:1707.02268, 2017.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In European conference on computer vision, pp. 382-398. Springer, 2016.
Tal August, Katharina Reinecke, and Noah A Smith. Generating scientific definitions with control- lable complexity. In ACL, 2022.
Nguyen Bach and Sameer Badaskar. A review of relation extraction. Literature review for Language and Statistics 11, 2:1-15, 2007.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Michele Bevilacqua, Marco Maru, and Roberto Navigli. Generationary hoo" we went beyond word sense inventories and learned to gloss". In Proceedings the the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7207-7221, 2020.
Alain Bretto. Hypergraph theory. An introduction. Mathematical Engineering. Cham: Springer, 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Kevin Clark and Christopher D Manning. Deep reinforcement learning for mention-ranking coref- erence models. EMNLP, 2016.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory trans- former for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10578-10587, 2020.
Jacob Devlin, Ming-Wee Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang, Yameng Huang, Jian Jiao, Xuan-Jing Huang, Nan Duan, and Ruofei Zhang. An enhanced knowledge injection model for commonsense gen- eration. In Proceedings of the 28th International Conference on Computational Linguistics, pp- 2014 2025, 2020.
Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. Conditional generators of words defini- tions. In Proceedings of the 56th Annual Meeting the Association for Computational Linguisties (Volume 2: Short Papers), pp. 366-271,20 2018.
Noah Gardner, Hafiz Khan, and Chih-Cheng Hung. Definition modeling: literature review and dataset analysis. Applied Computing and Intelligence, 2(1):83-98, 2022.
Han Huang, Tomoyuki Kajiwara, and Yuki Arase. Definition modelling for appropriate specificity. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2499-2509, 2021a.
Jie Huang, Xin Liu, and Yangqiu Song. Hyper-path. bath-based representation learning for hyper- networks. In Proceedings of the 28th ACM International Conference on Information and Knowl- edge Management, pp. 458, 2019.
10

---

Under review as a conference paper at ICLR 2023
Jie Huang, Hanyin Shao, Kevin Chen-Chuar Chang, Jinjun Xiong, and Wen-mei Hwu. Under- standing jargon: Combining extraction and generation for definition modeling. arXiv preprint arXiv:211..0766, 2021b.
Jie Huang, Kevin Chang, Jinjun Xiong, and Wen-Mei Hwu. Open relation modeling: Learning to define relations between entities. In Findings of the Association for Computational Linguistics:
ACL 2022, pp. 297-308, 2022a.
Jie Huang, Kerui Zhu, Kevin Chen-Chual Chang, Jinjun Xiong, and Wen-mei Hwu. Dkg:
A descriptive knowledge graph for explaining relationships between entities. arXiv preprint 1004: 2022b.
Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. Learning to describe unknown phrases with local and global contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3467-3476, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 787 1-780, 2020.
Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang. Timothy Baldwin, and Nan Duan. Kfenet: Knowl- edge filtering and contrastive learning network for generative commonsense reasoning. arXiv preprint ar arXiv:2109.06700 2021.
Jiahuan Li, Yu Bao, Shujian Huang, Xinyu Dai, and CHEN Jiajun. Explicit semantic decomposi- tion for definition generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 708-717, 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020b.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Findings, pp. 1823-1840) 1840, 2020.
Processing:
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence, 2015.
Ye Liu, Yao Wan, Lifang He, Hao Peng, and S Yu Philip. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6418-6425, 2021.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, et al. Neurologic a* esque decoding: Constrained text generation with lookahead heuristies. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, 2022.
Timothee Mickus, D Paperno, and Mathieu Constant. Mark my word: A sequence-to-sequeee approach to definition modeling. In Proceedings The First NLPI Workshop on Deep Learning for Natural Language Processing, pp. 1. Linköping University Electronic Press, 2019.
Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. Definition modeling: Learn ing to define word embeddings in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
11

---

Under review as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. Vedm: Leveraging variational bi- encoding and deep contextualized word representations for improved definition modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6331-6344, 2020.
Xiang Ren, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, and Jiawei Han. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1825-1834, 2016.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418 0-264422662200 2020.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference artificial intelligence, 2017.
Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From show to tell: A survey on image captioning. arXiv preprint arXiv:2107.069/ 2021.
Ke Tu, Peng Cui, Xiao Wang. Fei Wang, and Wenwu Zhu. Structural deep embedding for hyper- networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern 2015.
recognition, pp.
Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, and Michael Zeng. Retrieval enhanced model for commonsense generation. In Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pp. 3056-3066 2021.
Peifeng Wang. Jonathan Zamora, Junfeng Liu, Filip Ilievski, Muhao Chen, and Xiang Ren. Contex- tualized scene imagination for generative commonsense reasoning. In International Conference on Learning Representations, 2022.
Koki Washio, Satoshi Sekine, and Tsuneaki Kato. Bridging the defined and the defining: Exploiting implicit lexical semantic relations in definition modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 3521-3527, 2019.
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jicbo Luo. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluat- ing text generation with bert. In International Conference Learning Representations, 2019.
12


Reviews:
[{"review_id": "pDOym-h4L-", "reviewer": "Reviewer_WEUx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper connects definition modeling, relation modeling, and hyper-relation modeling in a unified form. The paper pre-trains VER (A\nUnified Model for Verbalizing Entities and Relations) on a large training data by forming the “entity(s) → sentence” reconstruction task, which makes VER a useful tool for learning natural language representations of entities and relations.", "summary_of_the_review": "The paper proposes a unified way of learning natural language representations of entities and relations. Extensive experiments on three tasks and six datasets demonstrate the superiority of the proposed method, especially in low-resource settings. There are no comparisons between the proposed method and SOTA results in each task. It is understandable that the main contribution of the paper is about learning natural language representations of entities and relations in a unified way. However, adding comparisons between the proposed method and SOTA in an aligned way (e.g., without using external knowledge) can help the readers understand how the proposed method advances the SOTA performance in these tasks (or what is the performance gap between the proposed method and SOTA). Besides, it is interesting (and has more impact) if the paper can demonstrates improvements in more than one based models (e.g., T5).", "strength_and_weaknesses": "Strength:\n1) The paper proposes a fundamental and unified way to pre-train the models for learning natural language representations of entities and relations.\n2) Extensive experiments on three tasks and six datasets demonstrate the superiority of the proposed method, especially in low-resource settings.\n\nWeakness:\n1) There are no comparisons between the proposed method and SOTA results in each task. It is understandable that the main contribution of the paper is about learning natural language representations of entities and relations in a unified way. However, adding comparisons between the proposed method and SOTA in an aligned way (e.g., without using external knowledge) can help the readers understand how the proposed method advances the SOTA performance in these tasks (or what is the performance gap between the proposed method and SOTA).\n2) Since the contribution of the paper is to propose a unified way of learning natural language representations of entities and relations, it is interesting (and has more impact) if the paper can demonstrates improvements in more than one based models (e.g., T5).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "msJiHAFiuK4", "reviewer": "Reviewer_Fgdc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a model to turn one or more entities into a natural language sentence that describes their relationship (or define the entity when there's just one). The model is a modified BART model trained with entity->sentence samples from Wikipedia. By doing a single seq2seq training, the model is able to perform definition and relation description tasks with one set of parameters.\n\nEvaluation was done on open datasets of definition (DC), relation modeling (ORM) and hyper-relation modeling (CommonGen). The results were compared with vanilla BART model on an array of commonly used metrics. It showed that the proposed model (VER) outperforms BART in low-resource settings. When the training data is sufficient, their performance is on-par.\n\nThe paper argued that the model can be used as a tool for human users to explore new concept, as well as a base model for fine-tuning towards related tasks. The generated NL descriptions may serve as an interpretable  knowledge base. ", "summary_of_the_review": "The Reviewer thinks the paper is a well written one, very easy to follow. However, the result was not thoroughly tested (weak baseline) and discussed (hallucination risk). The methodology itself is straightforward and training setup is quite routine, with less discussion on design challenges. \n\nThe Reviewer would suggest adding stronger, STOA baselines and check the model's generalization abilities beyond Wikipedia, as well as assess the hallucination risk.\n", "strength_and_weaknesses": "Strength:\n\n1) This paper is really well written. It's very easy to follow the author's thoughts, and all details (e.g training set up) have been explained clearly. The Reviewer didn't spot any obvious errors in writing.\n\nWeakness:\n\n1) The baseline is weak (BART). Given that VER is a model fine-tuned on BART for this task, it's almost guaranteed to be better performing. The Reviewer would like to see comparisons with STOA algorithms, even if VER is not surpassing them.\n2) Unclear about generalization ability. The data for training (Wikipedia) is already a pretty comprehensive set of descriptive relationships of entities. It's not clear how the model could generalize, vs just remembering what appeared in Wikipedia text. It's not clear to the Reviewer about how much overlap do Wikipedia and the evaluation data have.\n3) No discussion on hallucination. In order to make the results useful, e.g. helping human users or serve as a knowledge base, we need to understand how correct is VER's output. If for new (or even existing) entities, the model simply hallucinates to produce seemingly correct sentences, it would do more harm than help.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6f3EcJW5wQn", "reviewer": "Reviewer_yzmq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for combining definition modeling, relation modeling and hyper-relation modeling towards the purpose of generating natural language utterances for verbalizing entities and relations. The contributions of the paper are in combining multiple techniques of entity and relation modeling and devising a new pre-training task for “entity(s)-->sentence” reconstruction. ", "summary_of_the_review": "I found the paper to be an interesting read, but was left with many unanswered questions as listed in the weaknesses section.", "strength_and_weaknesses": "Strengths: \n- The results are impressively better than the baseline (BART)\n- The dataset generation is interesting and quite interesting and relevant to the community – would this dataset be released to the public? \n\nWeaknesses: \n- I was not convinced with the motivation behind verbalizing entities and relations. The paper focuses the story in the beginning on how humans learn entities and relations, and then goes on to saying they have developed a method that can verbalize entities and relations the way humans do. However, humans do this to create a better model of the world for themselves. How is this verbalization useful to machines, if we continue with the analogy to humans? Furthermore, real applications to this method are not mentioned until the end of the introduction section. I would expect these to be mentioned much sooner in the paper. \n- Results compare different generation metrics. However, we know how contentious these are in the literature right now. Could there be an explanation provided on how significant the performance improvements over the baseline BART are? \n- I was expecting a comparison to Lin et al 2020 for the hyper relation modeling since that was mentioned at the end of the introduction section. However, the results once again only compare BART as a baseline for the hyper relation modeling (Table 3). You may correct me if I am looking at the wrong table for those results. \n- For the dataset evaluation, can there be a discussion provided on the train vs test splits –  mainly to answer the question that if you are evaluating the model on generation for entities and relations, do you provide different sets of them for train vs test set ? Does the model run into memorization issues, similar to how many generation models do, and if so, how much? Does the model generate different utterances semantically for the same entities and relations, across different runs? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1MMM48Bw3PQ", "reviewer": "Reviewer_m8ZD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work is about verbalizing entities and the relationships between them. According to the authors, existing \nworks deal with entity and relation verbalization separately. This work aims to solve entity and relation verbalization \nin a unified form by adaptive generative language models to the task of entity and relation understanding.\n\nSpecifically, definition modeling and relation modeling are formulated uniformly as follows:\ngiven a set of entities (or a single entity) in string form, a sentence is generated that describes the entities and their relationships. \nWhen a single entity is given, it is equivalent to definition modeling. When the size of the set is 2, it is equivalent to relation modeling.\n\nThe proposed approach is called VER (a Unified Model for Verbalizing Entities and Relations):\n- first, a large dataset is constrcuted from Wikipedia. The first (5) sentences from each page are included in the dataset, as those sentences \nare often definitions of the entity or its relationships. Entity mentions are identified using Wikipedia hyperlinks. \nThe models are pre-trained using `self-supervised text reconstruction': given an entity or a set of entities, they are matched with the relevant sentences\nin the training corpus. \n\nThe model of choice is BART (a transformer-based encoder-decoder architecture), using both BART-base and BART-large.\nThe following settings are evaluated: using full training data, \nEvaluate three settings: fine-tuning on full training data, partial data (10%; a low-resource setup) and with no fine-tuning.\n\nThe experiments mainly consider datasets in scientific domains (CS, math, physics).\nAlso, the models are tested on CommonGen, where given multiple entities, the goal is to generate a coherent\nsentence describing an everyday scenario using these concepts. The reference sentences of\nthe official test set of CommonGen are not released, and results are computed via a leaderboard. \nThe authors acknowledge however that state-of-the-art performance is achieved by leveraging external knowledge sources. \n\nThe results demonstrate that after continually pre-training the model with the\nentity(s)-to-sentence reconstruction task, the model acquires a better ability to verbalize entities.\nIn the low-resource setup, VER achieves a more significant improvement compared with BART. \n\nTo validate whether the joint training of relation modeling will benefit or harm the performance of\ndefinition modeling, a version of VER is trained only with data examples of a single entity. The results show that relation modeling \nand hyper-relation modeling do not harm definition modeling.", "summary_of_the_review": "The paper is not clear technically, and is not described in enough detail to be reproducible. ", "strength_and_weaknesses": "Pro:\n- An interesting task\n- A creative solution\n\nCons:\n- The experimental setup is not clear\n- The experiments are not comprehensive", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject