Knowledge Graph Compression Enhances Diverse Commonsense Generation
Authors: EunJeong Hwang
Veronika Thost
Vered Shwartz
Tengfei Ma
Venue: EMNLP 2023 Main (2023)
Keywords: commonsense generation
knowledge graph compression

Abstract: Generating commonsense explanations requires reasoning about commonsense knowledge beyond what is explicitly mentioned in the context. Existing models use commonsense knowledge graphs such as ConceptNet to extract a subgraph of relevant knowledge pertaining to concepts in the input. However, due to the large coverage and, consequently, vast scale of ConceptNet, the extracted subgraphs may contain loosely related, redundant and irrelevant information, which can introduce noise into the model. We propose to address this by applying a differentiable graph compression algorithm that focuses on the relevant knowledge for the task. The compressed subgraphs yield considerably more diverse outputs when incorporated into models for the tasks of generating commonsense and abductive explanations. Moreover, our model achieves better quality-diversity tradeoff than a large language model with 100 times the number of parameters. Our generic approach can be applied to additional NLP tasks that can benefit from incorporating external knowledge.

Knowledge Graph Compression
Enhances Diverse Commonsense Generation
EunJeong Hwang!?? Veronika Thost Vered Shwartz! Tengfei Ma
University of British Columbia 2 Vector Institute for Al
MIT-IBM Watson AI Lab, IBM Research
Stony Brook University
(ejhwang, wshwartz)@cs. ubc.
veronika. thost@ibm. com tengfei wwwstonybrooo..ceom edu
Abstract
Input: shark interviews fish.
tank
ocean
Generating commonsense explanations quires reasoning about commonsense knowl- edge beyond what is explicitly mentioned in the context. Existing models use common- sense knowledge graphs such as ConceptNet extract subgraph of relevant knowledge pertaining to concepts in the input. However, due to the large coverage and, consequentl, vast scale of ConceptNet, the extracted sub- graphs may contain loosely related, redundant and irrelevant information, which can introduce noise into the model. We propose to address this by applying a differentiable graph com- pression algorithm that focuses more salient and relevant knowledge for the task. The com- pressed subgraphs yield considerably more di- outputs when incorporated into models for the tasks of generating commonsense and abductive explanations. Moreover, our model achieves better diivrstyy tradeoff than large language model with 100 times the num- ber of parameters. Our generic approach can be applied to additional NLP tasks that can benefit from incorporating external knowledge."
fish
[1]
[$]
swim
water
[4]
interview
[4]
[2]
shark
[4]
talk
business
[1] Antonya:
[2] CapableOf
AtLocation:
14 RelstedTo:
References:
Sharks fish do not talk. A shark cannot talk.
Fish cannot talk.
Figure 1: An example from ComVE (Wang al., 2020). The subgraph obtained for the input sentence includes unimportant information (in red) that can lead noisy outputs.
et al., 2023). Moreover, the retrieved subgraphs around task's s concepts potentially include infor- mation that is not relevant to the context. For ex- ample, in Figure 1, the goal is to generate a rea- son why the input sentence ("A shark interviews fish"" defies commonsense. The concepts tank and business are semantically irrelevant to either the input the reference output sentences. Includ- ing imrlevant information introduces noise that deteriorate the model's performance. Recent work has addressed this by pruning noisy paths based on low edge confidence scores knowledge base embeddings (Lin et al., 2019) or by using language models (LMs) (Yasunaga et al., 2021). Yet, the relevance of paths is not determined in relation to the given task.
1 Introduction
Commonsense knowledge graphs (CSKGs) have been used improve the performance of down- stream applications such as question answering (Yasunaga et al., 2021) and dialogue (Tu et al., 2022), as well as for enhancing neural models for commonsense reasoning tasks (Lin et al., 2019; Yu et al., 2022). Typically, these methods extract keywords from the input and construct subgraph around them using the KG knowledge, which is then incorporated into the model.
Recent popular CSKGs such as ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019) represent nodes in natural language, which allows flexibility but also adds redundancy and noise (Wu
In this paper, propose to differentiable graph compression that enables the model to learn how select the crucial concepts that are actually related to the task. Our method contains two main components: using self-attention scores to select relevant concept nodes in the retrieved subgraph,
Codee is availahle at: sttps://githhe com/eu Saeaanng/KCCCCpreeset

---

and employing optimal transport loss to ensure the chosen concepts preserve the most crucial informa- tion of the original graph. In this way, the irrelevant or redundant concepts can be automatically elimi- nated in the subgraph.
sentence, such as "A shark interviews fish", does not make sense. a-NLG (Bhagavatula al., 2020) presents models with a past observation, such as "Mike spends a lot of his time on the internet" and a future observation such as "Now other people love the internet because of Mike's website". The goal is to generate a plausible explanation for what might have happened in-between, such as "Mike created website that helps people search". In related line of work, researchers collected or gener- ated commonsense explanations for existing tasks (c.g., Camburu et al., 2018; Rajani et al., 2019; Brahman et al., 2021).
We demonstrate the usefulness of our method on two commonsense generation tasks: common- sense explanation generation and abductive com- monsense reasoning. Our method outperforms range of baselines that use KGs in terms of both diversity and quality of the generations. We further conduct a comprehensive analysis, exploring a dif- ferent setup, such as the scenario of incorporating new knowledge into the subgraph. Different from the baselines, our method enables the model to maintain performance, even in the presence of po- tentially increased noisy data. Finally, show that our approach demonstrates better quality-diversie tradcoff than the large language model vicuna-13b, which has 100 times more parameters.
Diverse Sentence Generation. One of the de- sired aspects of generating commonsense explana- tions is the diversity of the outputs. Popular LM decoding methods such as top-k (Fan et al., 2018), top-p (Holtzman et al., 2020), and truncated sam- pling (Hewitt et al., 2022) generate diverse outputs by pruning the probability distribution over the vo- cabulary for the next token and then sampling a token from the pruned distribution. An alternative approach is use a mixture of experts (MoE) produce diverse outputs (Shen et al., 2019; Cho et al., 2019). Our approach extends MoKGE Yu et al. (2022), a model for commonsense explanation generation. MoKGE uses a combination of KGs to diversify the outputs of a MoE model. How- ever, the knowledge that MoKGE retrieves from the KG is not filtered, hence may contain loosely re- lated, redundant and irrelevant information, which can negatively impact the model's performance in generating high-quality diverse outputs. In our ap- proach, we employ knowledge graph compression to prioritize more important information.
2 Background
KG-Enhanced Neural Methods. KGs have been used enhance models for question answer- ing (Lin et ul., 2019; Feng et al., 2020; Yasunaga et al., 2021), relation classification (Wang et al., 2021), textual entailment (Kapanipathi et al., 2020), and more. Typically, such methods extract sub- graph of knowledge related to keywords in the in- put, which is then either embedded or represented in natural language before being incorporated into
the model. For example, both Wang et al. (2023) and Wang, Fang, et al. (2023) used CSKGs en- hance commonsense inference and a QA model by including the abstraction of concepts in the in- put (e.g. vacation > relaxing event). However, some knowledge may be irrelevant in the context of the particular question.
Method
Our goal is to generate diverse sentences, {y1, 1/. yk3 that explain a given instance x (see See 2 for the specific task descriptions). The ob- jective is to maximize the probability of generating each y P()x(z)), as well as to diversify them. Pre- vious KG-enhanced approaches usually add an ex- ternal graph G2 to make the generation also condi- tioned on the graph: P(ys) x. G.)). However, as we discussed in See 1, G.. often contains redundancy or noise. For example, given a target concept A, there is a semantically similar concept (e.g. a syn- onym) A' and a noisy concept B in the graph 9z). Obviously, A' will negatively impact the diversity of generations because the model may select both
To reduce such noise, prior methods have pro- posed to score and prune the paths. Lin et al. (2019) used TransE (Wang al., 2014) to score each edge in the path, while Yasunaga et al. (2021) scores nodes based on the likelihood of pre-trained LM to generate after the input. In both methods, the scores are not trained to represent a node's 's impor- tance in relation to the task.
Generating Commonsense Explanations. This paper focuses on the task of generating common- sense explanations, in particular focusing on the following datasets. In ComVE (Wang et al., 2020) the goal is to generate explanations for why a given

---

Output:
minimize graph transporiation distance
1. Sharks do not interview fish, they swim in the ocean. shark does not talk fish
Input: A shark interviews fish.
Subgraph construction
Transformer
decoder
lank
ocean
Concept selection with MoE
fish
Graph encodiing
fish
ocean
swim
swim
water
Graph compression
water
talk
interview
shark
shark
interview
talk
business
+
A shark interviews fish.
Figure 2: Overview of our approach. We retrieve a subgraph from ConceptNet for the given input sentence, compress it, and use MoE to generate diverse sentences for containing concepts from the compressed graph.
A and A' for generation and the semantics of the generations are similar; concept B will hurt the generation quality since it is irrelevant to the con- text. So, natural idea to solve the problem is to eliminate these concepts by compressing the graph. Our method extends MoKGE (Yu et al., 2022) by compressing the retrieved external knowledge graph. The framework is illustrated in Figure 2 and described in detail subsequently. In nutshell, it aims to identify the concepts within the KG that provide the most relevant knowledge for partic- ular instance. We first extract subgraph from the KG based on the given input sentence, and en- code it into vector representation (Sec 3.1). Then, we learn a compressed graph that maintains only the most relevant concepts for the given instance (Sec 3.2). We train the model with the correspond- ing losses (Sec 3.3) and finally apply MoE to gen- erate diverse outputs (Sec 3.4).
swim - fish).
Graph Encoding. To obtain embeddings for the concept nodes, we apply an off-the-shelf graph en- coder over the extracted subgraph (Wu et al., 2021). In our implementation, we follow Yu et al. (2022) and use the relational graph convolutional network (R- GCNG Schlichtkrull et al., 2018). R-GCN com- putes node representations by iteratively aggregat- ing neighboring node representations and thereby taking the relation types into account. In this way, the final embeddings capture the structural patterns of the subgraph.
3.2 Differentiable Graph Compression
As we discussed before, the extracted subgraphs often contain redundancy and noise, and we aim to compress the graph and remove the irrelevant infor- mation. This introduces to make the graph compression differentiable so that can be trained in the context of downstream tasks; and (2) how to maintain the most important and relevant information in the compressed graph.
challenges: (1) how
3.1 KG Subgraph Extraction and Encoding The subgraph extraction and encoding follows MoKGE (Yu et al., 2022).
Subgraph Extraction. We first associate each input sentence with the set of concepts from the KG that match its tokens. For example, given the sentence q ="A shark interviews fish" (the "query"), we extract the concepts C2 (fish, shark, interview) from ConceptNet. Second, we fix a radius h and extract a subgraph Gg with node set V, 2 Cg from the KG such that it contains all KG nodes and edges that are up to h = 2 hops around the concepts in C (e.g. shark
Self-Attention for Concept Scoring. Since we want to select concepts for the generation step (Sec 3.4), we can't apply differentiable pooling methods (Ying et al., 2018; Ma and Chen, 2020) and instead choose to construet a semantically meaningful subgraph containing the relevant nodes and edges. To do so, we apply self-attention and hence essentially use the features computed in the previous step as main criterion determine the concepts' importance. Specifically, we compute self-attention scores E RCx1 as proposed by Lee et al. (2019) using graph convolution (Kipf
'In what follows, our notation refers to KG concepts and their correspanding KG nodes interchangeably.

---

and Welling, 2017):
from 24 2j,, and meets the requirements that Tl, = a, and T 1m b.
Z 2= AAD--xe
Once the optimal transport distance mini- mized, the compressed graph is expected to keep as much information of the original graph. Thus redundant concepts will be largely removed, since involving them in the compressed graph will lead to less information kept. Take simple example, given an original graph with nodes (A, A', C), the subgraph with node {A, C} should be more in- formative than the one with (A, A'}, and its opti- mal transport distance between the original graph should be smaller.
where is the non-linear activation function tank; C :==11 Vig is the number of concept nodes in the subgraph: Á E RRC' is the adjacency matrix ex- tended by self-connections; Î is the degree matrix of A, which used for normalization; X E RCxF is the matrix of concept embeddings obtained in the previous step, with embedding dimension F; and aut E IR Fx1 is the parameter matrix for the self-attention scores. Given the concept scores Z, we consider a pre-set assignment ratio E (0, 1]. and form the compressed graph, g', by selecting 5% of concept nodes. We denote S as the number of concept nodes selected. In the example in Fig- ure 2. the compressed (third) graph contains 80% the nodes in the original subgraph.
Since solving an OT problem is computationally expensive, we add an entropy regularization term E(T) -is (log Ti 1), to allow for solving it upproximately using Sinkhorn's algorithm (Cu- turi, 2013) in practice, following prior work. With hyperparameter y 0, the entropy- reguuay-regularized loss becomes:
Optimal Transport for Regularization. The self-attention based concept selection make the graph compressed in an differentiable way, how- ever the attention parameters can only be trained from downstream generation tasks which cannot gurantee the compression quality as well as gener- alizability. Consider the case with concept A and its synonym A' in the retrieved graph gg. if AA is selected by the attention scores, it is highly possi- ble A' also has high score to be selected, so the redundancy cannot be removed.
W7(,,)) min < <T, ETE(T)
(2)
3.3 Loss Functions for Training
Following Yu et al. (2022), we train BART-base (Lewis et al., 2020) in seq2seq architecture on the commonsense explanation generation task, with a generation loss, and apply KG concept loss in addition. We also include optimal transport loss.
For this reason, we additionally apply optimal transport (OT; Peyré and Cuturi, 2019), method commonly used for measuring the distance be- tween two probability measures. Here, we regard a graph as discrete distribution, similarly to Ma and Chen (2020), and minimize the OT distance between the original graph and its compressed ver- sion. To this end, we define optimal transport loss between graphs. Given m-node graph and a n-node graph, we assume they have discrete dis- tributions j Σm a,02; and y Σラ= byBry, where z; and žj indicate the nodes, σ is a delta function, a = (a)...., am) and b (b,,.., ........ bn) are weights of nodes (generally uniform). If we define a cost matrix .M whose element My indicates the transport cost from node a; to node zj. then the optimal transport distance is:
Generation Loss. For sentence generation, we maximize the conditional probability of the target sequence y given the input sequence x concate- nated with the selected KG concepts Cı, cs.. We utilize the standard auto-regressive cross- entropy loss as follows:
M
Lg = Σ log Plulæ, cr, C2, CS, V<t) 1=1
where the timestep of the actual output. In the generation step, the model auto-regressively generates the output y with input æ and selected concepts.
KG Concept Loss. The effectiveness of the con- cept selection can be measured in terms of which of the chosen concepts appear in the output sentence (the reference answer). More specifically, we consider a regular binary cross entropy loss with targets ye = I(c E V,nCC) for each É Vg. Here, I(.) represents the indicator function. and C, is
W(P, 1) min T
(1)
T E m is called a transportation plan, whose element Ti denotes the transportation probability

---

the set of concepts that are present in the output. To obtain a probability for each of the S concepts the compressed graph, we apply an MLP. The resulting loss as follows:
4 Experimental Setup
4.1
Datasets
ComVE (Wang et al., 2020) was part of the Se- mīval 2020 commonsense validation task. Given a nonsensical sentence, the task is to generate expla- nations for why it doesn't make sense. The dataset contains 10k training examples and roughly 1000 examples each for test and validation. Each exam- ple comes with reference output sentences. The other dataset, a-NLG (Bhagavatula et al.. 2020), addresses the abductive commonsense reasoning task. Given past observation and future observa- tion, the goal is to generate plausible explanations for what might have happened in-between. The dataset consists of 50k training examples, 1.779 validation and 3,560 test examples. Each example in the dataset includes up to 5 reference outputs.
C. (Evergnc, de P(e) Eerg-o 1e) log! p())
Optimal Transport Loss. To make the optimal transport distance differentiable, we solve Eq.
using the Sinkhorn's 's algorithm (Cuturi, 2013): Starting with any positive vector vo, itera- tively update u and v as follows:
u² a =
be
(3)
where Ø is the wlsment-wise division and K is an intermediate variable derived from the cost matrix
M: K (xp(-M/7).
After & steps, we arrive at the k-step result ph ( K diag(v²) as an approximated optimal transportation plan, hence the optimal transport loss is approximated by
4.2 Baselines
MoE-based Methods. MoE-embed (Cho et al., 2019) and MoE-prompt (Shen et al., 2019) pro- duce diverse sentences by sampling different mix- ture components. While MoE-embed employs in- dependent latent variables when generating diverse outputs, MoE-prompt shares the latent variable between the experts. MoKGE (Yu et al., 2022) is the approach that we extend by adding graph com- pression. It generates outputs by incorporating KG concepts on top of MoE--based methods.
4. W (G, Go) =<
=< - ->E(ph
Altogether, our model is trained with three loss
functions:
L C- BC,
(4)
where a and B are hyperparameters that control the relative importance of the individual loss functions. In our experimental setup, we set both and /3 to a value of 0.3.
Other Methods to Improve Diversity. To show that our method yields sophisticated concept lection beyond regular filtering, we compare it a simple synonym filtering on top of MoKGE, ap- plied during the inference step, that yields a set of unique KG concepts for generating outputs. This baseline prevents the model from selecting similar concepts when generating the outputs. Second, we consider the common pruning approach, which removes irrelevant paths from the potentially noisy subgraph, following KagNet (Lin et al., 2019). To measure the quality of the path, the path is decom- posed into a set of triples. Each triple is scored based on the scoring function of the knowledge graph embedding technique, TransE (Bordes et al., 2013) and the score for each path is the product of its triple scores. The threshold for pruning is a hyperparameter and set to 0.15 following Lin et al. (2019).
3.4 Diverse Generation based on MoE
To encourage more diverse outputs, we follow pre- vious work (Shen et al., 2019; Cho et al., 2019; Yu et al., 2022) and use mixture of experts (MoE).
We use K experts, where each expert is responsi- ble for generating a unique set of KG concepts. The model is trained using hard-FM algorithm (Demp- ster et al., 1977). Since it is similar to (Yu et al., 2022)), we put the details in Appendix E. In Fig- ure 2, the nodes in the 4th graph highlighted in green, red, and blue colors indicate the K 3 respective experts assigned handle different con- cepts. The utilization of our compressed graph version helps the model better prioritize the crucial concepts during output generation, as we demon- strate in our experiments.
Large Language Model (LLM). Lastly, we compare to Vicuna-13h (Chiang et al., 2023). This

---

ComVE
self-bleu-3 (U)
self-bleu-4 (4)
distinct-2 (1))
entropy-4 (ft)
bleu-4 (ft)
rouge-l (1t)
MoE, embed
33.64uz
28.21a.
46.57uz
9.61a.
18.66as
43.72az
MoKGE, embed
35.361.1
29.711.2
47.51e4
9.63o.1
19.13au
43.7a.1
+ SAG OT (ours)
32.19pm
26.28ks
49.05p.1
9.696.0
19.08a2
43.65ax
MoE, prompt
33.42na
28.4a3
46,93n.2
9,6a2
18.9
43.71as
MoKGE, prompt
30.93p9
25.31
48.44ma
9.670.2
19.01a1
43.83aa
+ filtering
34.01ns
28.92as
47.490.
9.640.1
19.02a4
43.48a6
+ pruning
33.4320
28.272.2
48.260.
9.640.0
18.67a2
43.10as
+ SAG (ours)
28.46us
22.81
48.33us
9.660.
19.00a6
43.800s
SAG OT (ours)
27.32a.
21.94a.
48.94u.1
9.696.0
19.31os
44.16a.1
o-NLG
self-bleu-3 (U)
self-bleu-4 (1)
distinct-2 (1)
entropy-4 (ft)
bleu-4 (1)
rouge-1 (1))
MoE, embed
29. 29.02
24.19
36.22p.
10,84
14.31az
38.91o2
MoKGE, embed
29.1715
24.04.6
38.15
10.9..
13.74az
38.06u2
+ SAG + OT (ours)
24.98n
19,83a.
38.93n.
10.93n.o
13.06aa
37.77a3
MoE, prompt
28.0520
23.181.
36.710.1
10.85u.
14.26a3
38.780.
McKGE, prompt
27.4020
22.4324
38.01as
10.88
14.17a2
38.82a7
+ filtering
31.3829
26.3628
37.95us
10.78p.6
13.89aa
38.07a1
+ pruning
31.842.
26.7224
38.21n.2
10.78pa
13.73c.
38.01o2
+ SAG (ours)
28.49ps
23.59as
38.05p4
10.86p.0
13.41as
38.0003
+ SASG+OT (ours)
23.99p
18.80
39.02p.7
10.880.0
14.21as
38.93u2
Table 1: Diversity and quality evaluation on ComVE and a-NLG datasets. All experiments are run three times with different random seeds, standard deviations are reported subseript.
large LM was built upon LLaMA-13b (Touvron et al., 2023), transformer-based LM trained on trillions of tokens exclusively sourced from pub- licly available data. Vicuna-13b performs par with ChatGPT (Chiang et al., 2023). We employ Vicuna- 13b 2-shot setup (see Appendix for the prompts).
tribution within the generated sentence.
Quality. We use standard metrics for automatic evaluation of generative tasks: BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), which are based on n-gram overlap between the generated sentences and human-written reference outputs. They assess the highest accuracy by comparing the best generated sentences to the target sentences.
4.3 Metrics
Following the same evaluation setting in previous works, we assess the performance of the generated sentences in terms of both diversity and quality.
5 Results and Discussion
Comparison to Baselines, Table 1. We observe similar trends for the two datasets and across the two model series, based on embedding and prompts. Overall, the differences are strongest for self-BLEU and Distinct-2, two aspects that are par- ticularly important for diverse generation. This suggests that our model is able to reason about different possible contexts. On both datasets, method, MoKGE+SAG+OT, outperforms the mix- tures of experts by large margins in terms of diver- sity and, at the same time, achieves comparable or better performance terms quality. Note that, on Com the quality differences between the best and our, second-best model are within stan- dard deviation.
Pairwise Diversity. Self-BLEU (Zhu et al., 2018) is used to evaluate how each sentence is similar the other generated sentences based on n-gram overlap. SeIf-BLEU-3/4 are diversity scores based on 3/4-gram overlap. The metries compute the av- erage sentence-level self-BLEU scores between all pairwise combinations of generated outputs. Hence, lower self-BLEU scores indicate greater variety between the sentences in the set generated for each input sample.
Corpus Diversity. Distinct-k (Li et al., 2016) is calculated by counting the number of unique k- grams in the generated sentence and dividing by the total number of generated tokens, to prevent preference towards longer sentences. Additionally, we report entropy-k (Zhang et al., 2018), which evaluates the evenness of the empirical n-gram dis-
The effectiveness of our approach especially evident from the comparison the filtering and pruning baselines. Recall that these approaches similarly aim at better exploiting the KG by im-

---

proving diversity and removing noise, respectively. However, we observe considerable decrease in diversity and nearly always also slightly in quality, This shows that simple solutions, unrelated to the task at hand, are seemingly not able to identify the most relevant knowledge. More specifically, for the filtering baseline, we observed that the model is unable to learn what concepts to choose for unseen data. As result, its ability generalize unseen data is limited, resulting in lower diversity scores on the test data. Altogether, this demonstrates that our approach, based on the compressed graph, is effective in suppressing redundant information present in the KG and promoting other knowledge that is more relevant in the given context.
Setf
Distinc22
ROUSEL
40-
38.236.6
38,33885
40.0
38.139.2
38.0
35-
Scores
24.3
23.0
0.3
0.5
0.65
0.7
0.9
Assignment Ratio
Figure 3: Self BLEU-3, Distinct-2, and ROUGE-I per assignment ratio o-NLG dataset. MoKGEE-rrompl with Self Attention and Optimal Transport is used for experiment.
We additionally confirm that our optimal trans- port loss helps the model to keep the KG subgraph more coherently; see especially the a-NLG results.
outputs and shows that the LLM is still prone to generating sentences with similar structure (e.g. "I wore wig to ..."", as it can be with a-NLG. Furthermore, while the generated sentence "I wore wig party and felt great" explains the first observation "I always wondered why loved wear- ing wigs", fails to explain the second observation got beat up and reminded of why I shouldn't tt". In the CO task, the generated sentences are diverse in terms of both sentence structure and word usage, but the model sometimes generates sentences that are less logical, such "Writing in a paper with eraser is not permanent". In contrast, our approach enables MoKGE to generate a wider range of sentences that incorporate rele- vant concepts and enhance the interpretability of the generation process.
Generation Examples, Figure 4. Observe that MoKGE tends to generate sentences with simpler structure and fewer concepts, whereas our model employs broader range of KG concepts. This makes the generations effectively more similar to the human-written ones, where each of the three sentences addresses different context. We show more examples in Appendix B.
Testing Robustness with Potentially more Re- dundancy and Noise, Table 2. We created a more challenging scenario by extending the ex- tracted subgraphs with additional, related knowl- edge potentially including more of both relevant and redundant information. This was done by ap- plying COMET (Bosselut al., 2019), trans- former that was trained generate KG triples (i.e.. entity-relation-entiy tuples) based on given entity- relation pairs. The original MoKGE model seems to struggle in this scenario: its performance de- creases without exception in terms all metrics. In contrast, our approach, applied on top of MoKGE, is successful in both retaining the performance of MoKGE alone and even the improvements of MoKGE+SAG+OT.
Analysis
Compression Ratios, Figure 3. This hyperpa- rameter determines the amount of concept nodes to be kept in the compressed subgraph. Maintaining 65% of the nodes in the subgraph yields the optimal performance in terms of both diversity and qual- ity on both datasets (see Appendix C for Com VE dataset). Interestingly, we not observe great negative impact on performance, even up to a ratio of 30%. This shows that ConceptNet apparently contains much information that is not necessarily beneficial for diverse generations in the context of particular task and hence justifies our proposal.
Comparison with Large Language Model, Ta- ble 3 & Figure 4. We compare our method to Vicuma-13b Most interestingly, our proposal out- performs the LLM Distinct-2 and Entropy-4. Note that even MoKGE alone is slightly better than the LLM in these aspects, yet our method is ef- fective extending the gap by better exploiting the externl knowledge. Figure 4 gives example
Unique Concepts in the Output, Ap- pendix The comparison of MoKGE and MoKGE+SAG+OT shows that MoKGE tends to generate more sentences containing 0

---

ComVE
self-bleu-3 (U)
self-bleu-4 (1)
distinct-2 (1t)
eatropy (1))
bleu-4 (1)
rouge-1 (1))
MoKGE
30.93a9
25.3
48.44uz
9.67az
19.01e.1
43.83u.
+COMET
32.731.s
27.45t.
48.32az
9. 9,6400
18.68a.
43.5
+COMET+SAG+OT
27.23
21.681.5
48.65as
9.68po
19.38
43,990.
a-NLG
self-bleu-3 (U)
self-bleu-4 (4)
distinct-2 (ft)
entropy-4 (tt)
bleu-4 (1t)
rouge-1 (1t)
MoKGE
27.402.
22.43
38.01as
10.88,
14.17pa
38.82
+COMET
31.4124
26.3224
37.99e2
10.77a.
13.87a.
37.960.1
+COMET+SAG+OT
25.48
21.141.3
38.36p3
10.8400
14.07p.
38.6504
Table 2: Performance with COMET and COMET with Self Attention and Optimal Transport. MoKGE-prompt is used for experiments.
Com VE
sclf-bleu-3 (1)
self-hlcu-4 (0)
distinct-2 (ft)
entropy-4 (1)
bleu-d (1)
rouge-1 (1)
Vicuna- 13b
18.0
12.74uo
48 18.4
9. 9.65uu
17.6 17.65up
43. u.
MOKGE+SAG+OT
27.32o.
21.94u4
48.94..1
9.69uu
19.31u3
44.16u.
a-NLG
self-bleu-3 (4)
self-bleu-4 (U)
distinct-2 (1t)
entropy (1)
bleu-4 (ft)
rouge-I (ft)
Vicuna-13b
33.23o.
27.39ue
37.97p.
10.38oo
17.30u0 14.21ps
40.58
MoKGE+SAG+OT
23.99
18.80a6
39.02o.7
10.88.a
38.93aa
Table Comparison between Vicuna- with 2-shot setup and MoKGE with SAG Pooling. MoKGE-prompt is used for experiments. Vicuna-131 ran 1 time.
always wondered why loved wearing wigs. [likely explanation between two events] got beat up and reminded of why shouldn't.
He used an eraser to write in paper. [explanation for sense-making)
Human:
Human:
1. but my parents told me was wrong because
1. An eraser used to erase something written in a paper.
im male.
2. wore a wig to school.
Erasers rub out write marks.
My friends caught me wearing moms wig. People didn't liked when looked like girl.
Eraser doesn't write.
MoKGE:
MoKGE:
KG-concepts; l'paper', 'pencil', 'use', 'mark']
concepts:
l'fight', 'wear', 'wig'l wig in fight. One day, decided wear wig. went and got a wig for Halloween.
1. eraser is used to erase pencil marks on paper.
1. decided to
The eraser is used to erase pencil marks.
3. Eraser is used to erase pencil marks on paper.
Vicuna-13h:
Vicuna-133
1. eraser is used to erase mistakes, not to write.
1. wore wig to a party and felt great.
2. Writing in paper with an eraser not permanent.
2. wore wig to work and felt confident. queen.
3. wore a wig to hed and
3. Eraser writing it an erasing tool.
MoKGE+SAG-OT:
MoKGE+SAG+OT:
KG-concepts: ['paper', 'pencil', 'use', 'cause', 'thing',
concepts: ['fight', 'wig', "party'. 'group', 'wear']
'mark', 'write]]
1. wore wig in front group of people. got in fight one day wearing a wig. wig to party and lost all my wigs.
1. eraser used to erase writing paper.
2. Eraser is used to erase pencil marks on paper.
3. You can't write with eraser because eraser is used to erase things.
d-NLG
ComVE
Figure 4: Examples of model generated sentences using MoKGE, Vicuna- 13b, MoKGE with Self Attention Optimal Transport.
concepts only. This indicates that the lower diversity scores MoKGE may be due to the selection of irrelevant concepts during output generation, showing the model's inability to effectively utilize them. The table shows that our method increases the numbers of KG knowledge
actually used by the model and thus its success injecting external bias into LMs.
Human Evaluation, Table 4. We conducted human evaluation on the outputs produced by our model MoKGE+SAG+OT and the baseline MoKGE for the a- n-NG task. We randomly se-

---

Model
diversity
quality
ConceptNet, was collected through crowdsoureing and may also introduce bias to our model (Mehrabi et al., 2021). However, we only use single word nodes from ConceptNet, which limits the impact of such bias.
MoKGE
1.88
1.93
MOKGETSAG+OT
2.10
2.08
Table Human evaluation performance on 30 randomly selected a-NLG samples.
Models
The generative models presented in the paper trained large-scale publicly available web corpus and may also bring some bias when generating sentences.
lected 30 generations from each model. The anno- tation was performed by researchers in the lab, We instructed the annotators to score the diversity and correctness (quality) of each generation on scale of 0 to 3. Table 4 shows a consistent per- formance improvement across both diversity and quality when comparing our model the baseline.
Acknowledgements
This work was funded, in part, by the Vector Insti- tute for AI, Canada CIFAR Al Chairs program, an NSERC discovery grant, and research gift from A12.
7 Conclusion
We present differentiable graph compression al- gorithm that enables the model to focus on crucial information. Through experiments on two com- monsense explanation generation tasks, we show that our approach not only improves the diversity but also maintains the quality of outputs. Moreover, our graph compression helps the model regain per- formance when and potentially noisy informa- tion is added to graphs. Our work opens up future research in effectively selecting and incorporating symbolic knowledge into NLP models.
References
Chandra Bhagavatula, Ronan Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Antoine Bordes, Nicolas Usunier, Alberto Gareia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Limitations
Use of Single Word Concept. Since ConceptNet contains mostly single words, we limit additional KG concepts to single words only. However, it can easily be extended into phrases and we leave it to future work to investigate how to effectively utilize longer phrases.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties. pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Use of Relations. When additional KG concepts are added to the model, we focus more on the con- cept nodes, not the edges. However, relation edges may provide additional insight. We leave the ex- ploration of this for future work.
Facze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2021. Learning to rationalize for monotonic reasoning with distant supervision. Pro- ceedings of the AAAÍ Conference on Artificial Intelli- gence, 35:12592 12601.
Ethics Statement
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. Advances in Neural Information Processing Systems, 31.
Data The datasets used in our work, SemEval- 2020 Task 4 Commonsense Validation and Explan- tation (ComVE; Wang et al., 2020) and Abduc- tive Commonsense Reasoning (a-NLG; Bhagavat- ula al., 2020), are publicly available. The two datasets aim to produce commonsense explanations and do not include any offensive, hateful, or sex- ual content. The commonsense knowledge graph,
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph F, Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90% chatgpt quality.

---

Jaemin Cho, Minjoon Seo, and Hannaneh Hajishirzi. 2019. Mixture content selection for diverse sequence generation.
ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 37744..333. PMLR.
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neu- ral information processing systems, 26.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-segeeeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics. pages 7871-7880 Online, Association for Computa- tional Linguistics.
pre-training
A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Soci- ery: Series B, 39:1-38.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational I Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promonte ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang. Jun Yan, and Xiang Ren. 2020. Scalable multi- hop relational reasoning for knowledge-awan tion answering. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295-1309, Online. As- sociation for Computational Linguistics.
ques-
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph net- works for commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (EMNLP-JJCNL)), pages 2829-2839, 2839, Hong Kong. China. Association for Computational Linguistics.
John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 3414- 3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations.
Tengfei Ma and Jie Chen. 2020. Unsupervised learning of graph hicrarchical abstractions with differentiable coarsening and optimal transport.
Pavan Kapanipathi, Veronika Thost, Siva Sankalp Patel. Spencer Whitehead, Ibrahim Abdelaziz, Avinash Bal- akrishnan, Maria Chang. Kshitij P. Fadnis, R. Chu- laka Gunasekara, Bassem Makni, Nicholas Mattei, Kartik Talamadupula, and Achille Fokoue. 2020. In- fusing knowledge into the textual entailment task using graph convolutional networks. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirtt-Secoodd Innovative Applica- tions of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAT Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8074-8081. AAAI Press.
Ninareh Mchrabi, Pei Zhou, Fred Morstatter, Jay Pu- jara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 5016-5033 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318,, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In 5th International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Gabriel Peyré and Marco Cuturi. 2019. Computational optimal transport. Foundations and Trends in Ma- chine Learning, 11 (5-6):355-602.
Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019. Self-attention graph pooling. In Proceedings of the 36th International Conference on Machine Learning.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense

---

reasoning. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4932-4942, Florence, Italy. Association for Computational Linguistics.
2023. Car: Conceptualizationaaugge augmented reasoner for zero-shot commonsense question answering, In Findings of the Association for Computational Lin- guistics: EMNLP 2023.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if- then reasoning. In Proceedings of the AAAI con- ference on artificial intelligence, volume 33, pages 3027-3035.
Weiqi Wang, Tianqing Fang, Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. 2023. Cat: A contextualized conceptualization and instan- tiation framework for commonsense reasoning. In Proceedings of the 6l Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2023.
Zhen Wang. Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by trans- lating on hyperplanes. In Proceedings of the AAAI conference on artificial intelligence, volume 28.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In The Semantic Web: 15th Inter- national Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593- 607. Springer.
Siwei Wu, Xiangqing Shen, and Rui Xia. 2023, Com- monsense knowledge graph completion via con- trastive pretraining and node clustering.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip Yu. 2021. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4-24. 24.
Tianxiao Shen, Myle Ott. Michael Auli, and Mare' Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In- ternational Conference on Machine Learning.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.3: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovee. 2021. QA-GNN: Reasoming with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- Language Technologies, pages 535-546, Online. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Zhitao Ying. Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018. Hi- erarchical graph representation learning with differ- entiable pooling. Advances in neural information processing systems, 31.
Quan Tu. Yanran Li, Jianwei Cui, Bin Wang. Ji-Rong Wen, and Rui Yan. 2022. MISC: A mixed strategy- aware model integrating COMET for emotional sup- port conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 308-319, Dublin, Ireland. Association for Computational Lin- guistics.
WenHao Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, and Meng Jiang. 2022. Diversi- fying content generation for commonsense reasoning with mixture of knowledge graph experts. In Find- ings of the Association for Computational Linguis- rics: ACL 2022, pages 1896-1906, Dublin, Ireland. Association for Computational Linguistics.
Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang. 2020. SemEval- 2020 task 4: Commonsense validation and explana- tion. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 307-321, Barcelona (online). International Committee for Computational Linguistics.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018. Generating informative and diverse conversational sponses via adversarial information maximization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-Adapter: Infusing Knowledge into Tre-Trained Models with Adapters. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1405-1418, Online. Association for Computational Linguistics.
Yaoming Zhu, Sidi Lu, Lei Zheng. Jiaxian Guo, Weinan Zhang, Jun Wang. and Yong Yu. 2018 Texygen: A benchmarking platform for text generation models.
A Prompt used with Vicuna-13b
Weiqi Wang*, Tianqing Fang*, Wenxuan Ding, Baixuan Xu, Xin Liu, Yanggiu Song, and Antoine Bosselut.
We present the prompts that we used for Vicuna- 13b for ComVE (Figure 5) and n-NLG (Figure 6).

---

E Mixture of Experts
few-shot examples
input sentence
Given input sentence ? and target sentence y, MoE employs a multinomial latent variable E {1, 2.., K} and decomposes the marginal likeli- hood as:
Give three reasons explaining why the above sentence does not make sense:
1. < reference1 >
2. < refeernnc22 >
3. < reference3
...
K
target example input sentence
Ply 9x) = Σ P(olz, P(y)0, 9.)
Give three reasons explaining why the above
sentence does not make sense:
Each å represents expert, which is responsible for explaining (ar, G' v) observation.
Figure 5: Vicuna prompt for the ComVE dataset.
With the above decomposition, the model mini- mizes the loss function (Eq.(4))
#t few-shot examples
Write three sentences that likely happened in between the past event: past event and the future event: < future event
V Plyir, (1) EK P(8 v.) Vlog 6lr, ()
1. referencel >
2. < reference2
and is trained using hard-EM algorithm (Dempster et al., 1977) as follows:
3. < #reference3
...
target example
E-step: choose expert ghest with minimal loss.
Write three sentences that likely happened in between the event: past event and the
future event: future event
ghest argmin log P(y, 6|0, G.)
Figure 6: Vicuna prompt for the a-NLG dataset.
M-step: update the parameters of the chosen expert ghest from E-step.
# of KG Concepts
Data
Model
1
3<=
Hyper-parameters
MoKGE
23.2
28.9
42.1
Co
+SAG+OT
+0.1
-3.1
+1.5
+1.0
We used BART-base (Lewis et al., 2020), which is built the Transformer architecture with 6 layer encoder-decoder. For model training, used Adam optimizer with a batch size of 60 and a learning rate of 3e-5. For the ComVE dataset. the warmup steps are set to 5000. For the a-NLG dataset, the weight decay is set to 0.01. For optimal transport, 7 is set to 1.0. As to the weights in the discrete distributions, {a:} are set evenly as 1 1/m, and {b; } are all set as 1/n, where and are number of nodes in the graphs.
MoKGE
16.8
31.9
29.0
22.3
o-NLG
+SAG+OT
-2.0
-1.1
+1.7
+1.4
Table Comparison models with MoKGE and MoKGE with Self Attention and Optimal Transport on the number of unique concepts in generated outputs. All KG concepts are lemmatized.
B Additional Generation Examples
We show additional sentences generated by MoKGE and MoKGE+SAG+OT for COmVE (Fig- ure 7) and a-NLG (Figure 8).
Assignment Ratio for ComVE
We show the performance on Com VE with varying node assignment ratios in Figure 9.
D Concept Inclusiveness
We analyze how well the model incorporates KG concepts in output generation in Table 5.

---

A bra is for shaving mans face.
like to buy clothes in the gym.
[explanation for sense-man maning)
[explanation for sense-making)
Human:
Human:
1. A.A bra is meant for woman's chest, and can not shave hair.
1. The gym does not sell clothes. Gyms don't don'tusually sell clothing. Gym place where cannot buy clothes.
2. A bra
chest.
Bra doesn'l has blades.
MoKGE:
MoKGE:
concepts: |'face', 'man', 'shaving', 'bra']
concepts: ['sell', 'buy', 'gym']
A bra not for shaving
faco.
You can't buy clothes the gym, not clothing.
2. not for shaving man's face.
3. Bra not meant for shaving mans face.
Gym doesn't have clothes to buy in the gym
3. gyms don't sell clothes, they sell shoes.
MOKGE+
M
concepts: ['dress', 'face', 'use', 'man', 'bra', 'shave"]
concepts: ['store', 'sell', 'buy', 'workout', 'gym'l
1. Bra is not for shaving mans face, it is for dressing up.
1. gym doesn't sell clothes. You go to the gym to workout.
2. bra not for shaving
face. 3. Men do not shave their faces using bras. bras
2. You can't buy clothes the gym. You go the
are not used for shaving.
3. Gyms don't sell clothes, they sell shoes.
ComVE
Figure 7: Additional examples of model generated sentences with MoKGE and MoKGE+SAG+OT Com VE dataset.

---

Roy loved everything about rice. [likely explanation between two events]
Kelly decided she wanted to paint some plaster. [likely explanation between two events] Kelly had to redo it.
Now Roy produces millions of pounds of rice and has sales worldwide!.
Human:
Human:
1. However, she realized she did it wrong.
1. Roy begin growing his own rice.
2. Kelly applied the plaster wrong. She finished her project, but then the dog over it and ruined
Roy decided become rice farmer.
3. Roy decided to grow rice.
4. Roy started growing his own rice and selling it.
5. Roy wanted to start producing rice.
MoKGE:
MoKGE:
concepts: ['company', 'rice', 'farm']
concepts: ['turn', 'paint']
Roy decided start his own rice company with his passion.
1. Kelly painted it hadly and it turned out hadly.
2. Kelly painted it badly. She did not realize how bad it looked.
2. Roy decided to start his own rice farm with his rice beans.
She not correctly and turned out badly.
decided start rice company with his passion.
MOKGE+SAG+OT:
concepts: ['good', 'white', 'plaster', 'wall', 'paint'] She painted it badly and it didn't look good.
MoKGE+SAG+OT:
concepts: ('company,, 'money', 'production', 'rice', 'farm']
2. Kelly painted the walls with white paint. looked bad.
Ray's passion gut him into the field of rice production.
3. Kelly did not like the look of the plaster.
2. Roy started his rice farm. Roy learned everything.
Rey decided start his own rice company with his own money.
d-NLG
Figure 8: Additional examples of model generated sentences with MoKGE and MoKGE+SAG+OT on o-NLG dataset.

---

Assignment ratio
49.1
66.3
44.1
41.
ROUGHEL
39-
30
25
Figure 9: Self BLEU--3 Distinct-2, and ROUGE-1 per assignment ratio ComVII dataset.


Reviews:
[{"review_id": "XtnUG1rUrQ", "reviewer": "Reviewer_wZ5V", "summary": "", "questions": "Question A: The prompts and inputs used for the large language model do not contain concepts, which may lead to an unfair comparison. The given concepts could have a significant effect on the generated results. If suitable concepts are identified for the large model using the method in the paper and incorporated into the prompts, would it lead to better reasoning performance?\n\nQuestion B: Do the authors think the quality of the knowledge graph would affect the quality of generation? For example, ConceptNet is constructed via crowdsourcing and may contain biases.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our contributions and for pointing out critical discussion points! We have made those more clear in the paper now.\n\n**This work does not take into account the relations when training concept embeddings**\n\nThis seems to be a misunderstanding. In fact, the relational graph convolutional network (R-GCN) (see Sec 3.1), which is commonly used in the KG context [1] and which we use for learning concept embeddings, takes the relations in the neighborhoods of the concept nodes into account very directly. In this way, they are also considered in both our losses, the KG concept loss and optimal transport loss. We could add the relations explicitly, but since R-GCN has been proven generally, we resorted to this more straightforward solution.\n\n[1] Modeling relational data with graph convolutional networks. Michael Schlichtkrull et al. In The Semantic Web: 15th International Conference, ESWC 2018.\n\n**Missing human evaluation**\n\nThis is a very fair point. We have tried our best to consider a broad range of metrics as suggested in prior works, covering pairwise diversity, corpus diversity, and quality. Overall, the metrics capture diversity - our main objective - well. We totally agree about the value of human evaluation to include more subtle logic. We now ran a small human evaluation with 30 random samples from alpha-NLG. The samples are randomly permuted and put on M-Turk with their model information removed. The annotators are asked to score the diversity and correctness/quality from 0 to 3. Due to the short time, we only have two annotators and the results are as follows: \n\ndiversity score: baseline avg: 1.81, our model avg: 2.03\n\nquality score: baseline avg: 1.87, our model avg: 2.05. \n\nThis hints at a consistent performance improvement with our model in terms of both diversity and quality metrics. We will extend this initial study and add more human evaluation results in the updated version, but the conclusion is supposed to be the same based on current observations.\n\n**No concrete descriptions or demonstrations of applications of the approach in the paper.**\n\nOur approach is particularly applicable to tasks that benefit from additional, external commonsense knowledge, such as CommonsenseQA, CommonGen, and multihop reasoning over commonsense KGs. In the paper, we focus on commonsense generation, we made this more clear now. But we added a more general discussion as an outlook.\n\n\n**LLM does not contain concepts, may leading to unfair comparison. If suitable concepts are identified for llm, would it lead to better reasoning performance?**\n\nOur intention in the LLM experiment was to compare to a “vanilla” LLM in a realistic scenario, a standard prompting with instructions in a few-shot setup. It is a valid, but different research question if we also can improve LLM performance by combining an LLM with the components of our approach (such as keyword extraction, path extraction, etc.). However, we assume that we see less direct impact here since most of these models were likely trained on knowledge similar to the one in available KGs. For this reason, we also don’t consider the comparison to be unfair.\n\n**Do the authors think the quality of the knowledge graph would affect the quality of generation?**\n\nThis is actually the point we intend to analyze in our experiment in Table 2, where we add potential noise generated by COMET. We made this more clear now. Generally, the quality of the KG will likely affect models using the KG out of the box, and we can see this with MoKGE, it is strongly impacted by the noise. In contrast, our adaptive, supervised approach, especially the OT part, is able to prune noise and irrelevant knowledge. Hence it is rather robust.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work proposes a differentiable graph compression algorithm, which can automatically identify and retain key conceptual information relevant to a specific task.\n2. This work utilizes self-attention and optimal transport regularization to reduce redundant and irrelevant information, which can lead to better generation of outputs.\n3. Through the proposed method in the paper, better results are demonstrated across the three metrics of pairwise diversity, corpus diversity, and quality.", "reasons_to_reject": "1. This work does not take into account the relations when training concept embeddings. Relations are an essential component that needs to be considered in the knowledge graph.\n2. This work lacks human evaluation. The current evaluation relies primarily on automated metrics. Human evaluation can better judge the quality and logicality of the generated outputs. MoE and MoKGE both conducted human evaluations.\n3. The authors propose that their generation approach can be applied to other NLP tasks, but do not provide concrete descriptions or demonstrations of this in the paper.", "questions_to_authors": "Question A: The prompts and inputs used for the large language model do not contain concepts, which may lead to an unfair comparison. The given concepts could have a significant effect on the generated results. If suitable concepts are identified for the large model using the method in the paper and incorporated into the prompts, would it lead to better reasoning performance?\n\nQuestion B: Do the authors think the quality of the knowledge graph would affect the quality of generation? For example, ConceptNet is constructed via crowdsourcing and may contain biases.", "paper_topic_and_main_contributions": "The main work of this paper is compressing the commonsense knowledge graph to facilitate a better generation of commonsense explanations. The framework of this article extends from MoKGE and utilizes self-attention scores to select highly relevant entities in the subgraph. It further employs transport loss to eliminate semantically similar duplicate entities. This work compares its method on two datasets with previous models, and notably, it also conducts comparisons with a large language model. Additionally, the paper presents some case studies to demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R8yCEbTIwh", "reviewer": "Reviewer_BbGf", "summary": "", "questions": "A. Can you explain 1) what training signal tells which concept we should choose and 2) how this signal is back-propagated to your attention network?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments! We would love to address additional questions during the discussion period if anything is unclear.\n\n**Is it still necessary to select concepts for LM in the first place. If LM contains the required knowledge, I do not see a problem for LM to figure out the concepts by itself. If LM contains the required knowledge, I do not see a problem for LM to figure out the concepts by itself.**\n\nWe intended to address this question in our LLM comparison (Table 3 and Figure 4), given that it is indeed likely that similar KG knowledge is included in the pretraining data. Our method’s focus is particularly on diversifying the output generations. Interestingly, our evaluation shows clearly that the LLM, Vicuna-13B in a 2-shot setup, cannot beat our approach in this respect. It fails to fully use its knowledge and generate diverse sentences or better-quality sentences. More generally, hallucination and lack of commonsense are still well-known problems of LLMs nowadays, so they need to be made more focused. Our research targets this direction by focusing on the aspect of learning to select KG knowledge to diversify generation.\n\n**The work assumes that whether the concept appears in the final sentence is indicative of what is a good concept. What training signal tells which concept we should choose and how this signal is back-propagated to your attention network.**\n\nThat’s a good question and we will clarify this in the paper. We don’t assume that only concepts that are included in the generated sentence are helpful. Instead, the final sentence provides indirect supervision to select the most relevant concepts (i.e., the sentence may contain concepts X and Y which are connected in the KG through concept Z which is not included in the sentence). We choose the relevant nodes based on self-attention scores and then apply optimal transport, also minimizing duplicate concepts, both of which are trained.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tObtaining contextualized knowledge is an important question which may benefit not just commonsense generation but broadly language understanding and reasoning.\n2.\tThe work explores some interesting ideas for pruning the graph while maintaining the quality.\n3.\tExtensive experiments and analyses are conducted to verify the effectiveness of the proposed method.", "reasons_to_reject": "1.\tThe work invests the major efforts in selecting the concepts. However, it seems at the end, we still rely on the knowledge encoded in LM (e.g., a fish cannot talk) to solve the task. This raises the question of whether it is still necessary to select concepts for LM in the first place. If LM contains the required knowledge, I do not see a problem for LM to figure out the concepts by itself.\n2.\tIn the proposed method, it is unclear how the training objectives would guide the model to select relevant concepts (and also how the training signals are back-propagated). It seems that the work assumes that whether the concept appears in the final sentence is indicative of what is a good concept. But I think many concepts that do not appear in the final sentence are also important as they introduce background knowledge (potentially multi-hop).", "questions_to_authors": "A. Can you explain 1) what training signal tells which concept we should choose and 2) how this signal is back-propagated to your attention network?", "paper_topic_and_main_contributions": "The work aims to solve the task of commonsense generation via commonsense KG. Given a sentence, the work first extracts from KG a local graph containing potentially relevant concepts. To filter irrelevant concepts, the work adopts self-attention to score the concepts. To ensure quality, the work adopts optimal transport to preserve the most information of the original graph. The final selected concepts serve as additional inputs to a seq2seq model for generating the explanation. Results on two datasets demonstrate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6UNwWE6z3n", "reviewer": "Reviewer_1hNR", "summary": "", "questions": "Please find my comments above. I will consider raising my scores if the authors address the issues I have raised in their rebuttal.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very detailed feedback! We would love to address additional questions during the discussion period if anything is unclear.\n\n**Why not other CSKGs? & Improve paper flow in the introduction**\n\nOur focus is indeed on evaluating the impact of reasoning for concept selection. To see that best, we chose the most straightforward scenario, using ConceptNet, which is also the one that allows for direct comparison with the most closely related works. In this context, we apply COMET (see also our response to the comment below), to highlight our approach’s usability in a very realistic scenario, that is, by including multiple and potentially redundant data sources. We did not intend to cause confusion or recommend a particular KG. This is good feedback and we made the motivation more clear now. Thank you also for the additional references and we will add them to the revised version! \n\n**Include a comparison to the performances of the original COMET (from ATOMIC2020) as another baseline for evaluation.**\n\nWe are unsure if we correctly understand this suggestion. COMET is not designed to generate multiple diverse sentences involving various concepts, so we would need to adapt it considerably for a direct comparison. Alternatively, we could replace ConceptNet with ATOMIC and use this as an external KG. This is indeed an interesting experiment to learn more about the impact of different KGs, but it requires some adaptation. Since our study’s goal is to evaluate our approach in comparison to the related works, we didn’t consider it directly. But please note that the results in Table 2 (MoKGE vs. MoKGE+COMET vs. MoKGE+COMET+SAG+OT) do include COMET and the knowledge graph expanded from COMET can be considered as another KG.\n\n**Improve the readability of Figure 3 by enlarging the font and possibly placing the legend at the top or bottom of the figure.**\n\nWe will update Figure 3 in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of compressing commonsense knowledge graphs is a novel and well-motivated approach. Although the authors developed this method with the explicit purpose of commonsense generations, I believe it has the potential to be applied to other tasks, such as commonsense QA and multi-hop reasoning. \n\nThe experiments conducted in this study are comprehensive, and the results demonstrate good performance. It is commendable that the authors not only analyzed the overall performance on two tasks but also demonstrated their improvements on COMET and included the comparison against a large language model. These findings have significant implications.\n\nThe paper is well-written and presents clear details, enabling readers to understand the methodology and results of the study. The case studies and figures provided are particularly helpful in illustrating the authors' points.", "reasons_to_reject": "Overall, I find the method and experiments in this paper to be generally sound, and I am leaning towards accepting it. However, I do have some suggestions and questions that I would like to bring up.\n\nFirstly, I believe that the introduction could be significantly improved. Currently, there is a large leap from discussing CSKG to concept-level CSKG, which is difficult for me to follow. I cannot help wondering why you insist on using ConceptNet but not other CSKGs such as ATOMIC or GLUCOSE. The reason seems to be, in lines 38-39, the authors argue that free-form CSKGs, such as ATOMIC, represent nodes in natural language, which adds flexibility but also introduces redundancy and noise. While this is a key point that supports the use of ConceptNet, it is not always true and requires further evidence to be convincing. Instead of relying solely on previous works, which may make the paper seem incremental, I suggest providing a more compelling motivation for the study. Specifically, it highlights the importance of concept-level understanding for realizing commonsense reasoning and pointing out the gap in the literature where previous works only focused on conceptualizing or instantiating commonsense knowledge with concepts but failed to consider their relevance or plausibility in the path of reasoning towards a specific contextual task. By doing so, the paper would have a stronger foundation and flow more smoothly. I have attached some references below for your consideration.\n\nAnother suggestion is to include a comparison to the performances of the original COMET (from ATOMIC2020) as another baseline for evaluation. This would provide additional context and help readers better understand the strengths of the proposed method.\n\nFinally, I recommend improving the readability of Figure 3 by enlarging the font and possibly placing the legend at the top or bottom of the figure. Alternatively, placing the numbers inside the bars could also make it easier to read.\n\n========================\n\nAfter rebuttal: it seems that I have misunderstood the content regarding my second reason-to-reject. The authors responded to my other questions adequately. I raised my scores.", "questions_to_authors": "Please find my comments above. I will consider raising my scores if the authors address the issues I have raised in their rebuttal.", "paper_topic_and_main_contributions": "This paper proposes to use a differentiable graph compression algorithm to extract salient and relevant knowledge from a commonsense knowledge graph for generating diverse commonsense and abductive explanations. The compressed graph is then used to generate diverse explanations using a mixture of expert models (MoE). The authors evaluate their approach on two tasks: the ComVE and abductive NLG tasks. Empirical results show that their approach outperforms baseline methods in terms of generated quality and diversity of generated explanations. \n\nThe primary contribution of this paper is the development of the CSKG compression algorithm, which represents a compelling NLP engineering experiment.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {"Missing_References": "Murphy, G. (2004). The big book of concepts. MIT press.\n\nHe, M., Fang, T., Wang, W., & Song, Y. (2022). Acquiring and modelling abstract commonsense knowledge via conceptualization. arXiv preprint arXiv:2206.01532.\n\nWang, W., Fang, T., Xu, B., Bo, C. Y. L., Song, Y., & Chen, L. (2023). CAT: A contextualized conceptualization and instantiation framework for commonsense reasoning. arXiv preprint arXiv:2305.04808.\n\nAllaway, E., Hwang, J. D., Bhagavatula, C., McKeown, K., Downey, D., & Choi, Y. (2022). Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions. arXiv preprint arXiv:2205.11658.\n\nWang, W., Fang, T., Ding, W., Xu, B., Liu, X., Song, Y., & Bosselut, A. (2023). CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering. arXiv preprint arXiv:2305.14869.\n\nLiu, J., Chen, T., Wang, C., Liang, J., Chen, L., Xiao, Y., ... & Jin, K. (2022). VoCSK: Verb-oriented commonsense knowledge mining with taxonomy-guided induction. Artificial Intelligence, 310, 103744."}}]

Author rebuttal:

Meta-review: This work aims to facilitate a better generation of commonsense explanations using compressed KGs. In the proposed framework, self-attention scores to select highly relevant entities in the subgraph of the KG. The experimental results on two datasets show that the proposed method can outperform conventional baselines in diversity while keeping the comparable or better relevance to the references. Reviewer wZ5V and 1hNR gave positive scores, whereas Reviewer BbGf pointed out the unnecessity of selecting concepts from the KG for LM due to its knowledge obtained in pretraining. Considering that the authors addressed the question by referring to Table 3 and Figure 4, we can judge that there is no major problem in accepting this paper.
Decision: Accept-Main