DIFFUCOMET: Contextual Commonsense Knowledge Diffusion (2024)

Abstract: Inferring contextually-relevant and diverse commonsense to understand narratives remains challenging for knowledge models. In this work, we develop a series of knowledge models, DIFFUCOMET, that leverage diffusion to learn to reconstruct the implicit semantic connections between narrative contexts and relevant commonsense knowledge. Across multiple diffusion steps, our method progressively refines a representation of commonsense facts that is anchored to a narrative, producing contextually-relevant and diverse commonsense inferences for an input context. To evaluate DIFFUCOMET, we introduce new metrics for commonsense inference that more closely measure knowledge diversity and contextual relevance. Our results on two different benchmarks, ComFact and WebNLG+, show that knowledge generated by DIFFUCOMET achieves a better trade-off between commonsense diversity, contextual relevance and alignment to known gold references, compared to baseline knowledge models.1

heading: 1 Introduction
text: Identifying the commonsense inferences that underlie narratives, such as stories or dialogues (Guan et al., 2019; Zhou et al., 2022), is crucial to understanding those same narratives. For example, to understand why “Hank ... got the shopping bags” in the context in Figure 1, a model would need to infer that (1) Hank was not finished wrapping gifts, and so (2) would need to buy more wrapping paper. However, comprehensively inferring these diverse, yet implicit, commonsense inferences that are relevant to a context remains a challenging task.
Recent methods for identifying contextuallyrelevant commonsense inferences (Bosselut et al.,
†Corresponding author. 1We release our code to the community at https://
github.com/Silin159/DiffuCOMET
2021; Tu et al., 2022; Peng et al., 2022) use knowledge models (Bosselut et al., 2019; West et al., 2022) to generate commonsense facts. While knowledge models have been less brittle than previous retrieval-based methods for commonsense inference, they have two major shortcomings. First, they are trained to verbalize tuples from general commonsense knowledge graphs (Sap et al., 2019; Hwang et al., 2021), leading them to produce valid, but often contextually-irrelevant, commonsense inferences when applied out-of-the-box to real narratives. Second, because they are trained using autoregressive training objectives, they subsequently decode high-likelihood, non-diverse sequences that only identify limited collections of commonsense inferences relevant to an input context.
In this work, we address these challenges of contextual commonsense knowledge generation by developing Diffusion (Ho et al., 2020) COMmonsEnse Transformer (Bosselut et al., 2019) models. DIFFUCOMET models (shown in
4809
Figure 1) uses diffusion-based decoding to generate relevant knowledge embeddings that are constrained to the narrative context. Over multiple iterations of constrained diffusion, our models refine a latent representation of the semantic connections between a context and its relevant facts, ensuring that it generates commonsense knowledge that is more contextually relevant to the narrative. At the same time, by jointly refining multiple fact embeddings during diffusion, DIFFUCOMET also generates more diverse inferences than comparable-size autoregressive knowledge models.
We evaluate DIFFUCOMET models using traditional NLG metrics (e.g., BLEU; Papineni et al., 2002) commonly used for evaluating knowledge models. However, these metrics focus on surface form matching to gold references, and fall short of measuring the diversity of commonsense inferences and their semantic relevance to real narrative contexts. Our second contribution is a novel set of metrics that assess the diversity and contextual relevance of knowledge generated by knowledge models. Using both the traditional evaluation metrics and our new suite, we evaluate our models on a commonsense inference linking benchmark (Gao et al., 2022a) that covers both social and physical knowledge, and a second knowledge generation benchmark that involves extracting RDF triplets from language, WebNLG+ (Ferreira et al., 2020).
Our result show that DIFFUCOMET models generate knowledge that achieves a better balance of diversity and contextual relevance compared to other knowledge models. DIFFUCOMET models also more robustly generalize to generate knowledge for out-of-distribution narratives, and are better at producing novel knowledge tuples that are not in their initial training set. Finally, on our second benchmark, WebNLG+, we verify that our diffusion modeling method also generalizes well to a completely new factual knowledge generation task beyond the commonsense domain.

heading: 2 Background: Diffusion Models
text: Diffusion models learn to construct synthetic data from random noise. They use a forward process to gradually corrupt real data samples with additive noise, and learn a reverse process to recover (or de-noise) the corrupted data samples. Through the de-noising of corrupted data, diffusion models learn to map from a random noise distribution to their target data distribution, which grounds their
synthetic data generation. In this paper, we adopt the DDPM2 (Ho et al., 2020) formulation of the forward and reverse diffusion processes. Specifically, based on a sample z0 from a continuous input data distribution q(z0), the forward process constructs noisy sample zt over a sequence of time steps t ∈ {1, 2, ..., T}. In DDPM, zt is sampled from a Gaussian distribution conditioned on the previous sample zt−1, given by:
q(zt|zt−1) = N (zt; √ 1− βtzt−1, βtI) (1)
where βt is a noise schedule hyperparameter unique to each diffusion step.
In the reverse process, diffusion models learn an inverse distribution q(zt−1|zt) to de-noise samples created by the forward process. To more precisely couple the intermediate states of the reverse process with the final de-noised sample z0, Diffusion-LM (Li et al., 2022) reformulates the task of predicting zt−1 as directly predicting z0 (based on zt), and uses a mean-squared error training loss on the z0 prediction at each time step3:
Lz0-mse = T∑
t=1
E∥z0 − fθ(zt, t)∥2 (2)
where fθ(zt, t) = ẑt−10 denotes the model’s learned prediction of z0 at the reverse stage of step t to t−1. To formulate ẑt−10 as a refinement of the former reverse stage’s output ẑt0, Bit-Diffusion (Chen et al., 2022) improves the model function of predicting z0 with self-conditioning, i.e., ẑt−10 = fθ(ẑ t 0, zt, t). At inference time, the noisy sample at step t is predicted from ẑt0 via the Eq. (1) forward process, denoted as ẑt to replace the unknown gold input zt, while the initial input zT is pure Gaussian noise sampled from N (0, I).

heading: 3 Contextual Knowledge Diffusion
text: In this section, we first introduce the task of contextual commonsense knowledge generation, and then propose DIFFUCOMET, our diffusion approach for this task. The overview of our method is presented in Figure 1.
Task Description Given a narrative sample S as context, e.g., a story snippet or a dialogue, the model needs to generate commonsense inferences
2Denoising Diffusion Probabilistic Models 3We include more detailed formulation of the reverse dif-
fusion training in Appendix A.
as a set facts K = {k1, ..., kn, ..., kN}, which are relevant for understanding the situation described in the context. Each fact kn = (hn, rn, an) is represented as a triple containing a head entity hn, a tail (attribute) entity an, and a relation rn connecting them, e.g., (wrapping paper, used for, wrap gifts), as shown in Figure 1. We denote the set of unique head entities, relations and tail entities in K as H, R and A, respectively.
Contextualization We ground knowledge diffusion on the given context S by using encoderdecoder cross attention, inspired by SeqDiffuSeq (Yuan et al., 2022). In particular, we use a BART (Lewis et al., 2020) encoder fθs to learn the context encoding that represents S as hidden state zS :
zS = fθs(S) (3)
Then, a BART decoder fθz , serving as the diffusion module, learns to predict the de-noised data sample z0. Given the context hidden state zS (via cross-attention to the encoder fθs), fθz makes a prediction of z0 at time step t-1 (i.e., ẑt−10 ) based on its former prediction ẑt0 and time step t’s noisy sample zt:
ẑt−10 = fθz(ẑ t 0, zt, t|zS) (4)
Unlike traditional transformer decoders (Vaswani et al., 2017), the diffusion module fθz applies a bi-directional self-attention to ẑt0 and zt, since all positions of ẑt−10 are decoded simultaneously, i.e., in non-autoregressive manner.4
Discrete Knowledge Diffusion We consider two formulations for representing discrete knowledge in continuous embedding spaces for diffusion: DIFFUCOMET-Fact, where we learn to reconstruct continuous representations of facts kn using diffusion, and DIFFUCOMET-Entity, where we use separate diffusion processes to reconstruct head hn and tail an representations and then predict the relation between them to complete the fact. We highlight these differences in Figure 2.
For diffusion on the fact-level embedding space (DIFFUCOMET-Fact), we first pre-train a BART encoder fθe to produce an embedding en of each fact kn in the knowledge set K (with embedding size d same as the hidden state size of BART):
en = fθe(kn) ∈ Rd (5)
where we input the concatenation of each fact’s head, relation and tail tokens to the encoder fθe , and take the output hidden state of a start token 
 as the embedding of the fact. The initial input z0 of the forward diffusion process is then sampled from a Gaussian centered on the concatenation of all fact embeddings e = [e1; e2; ...; e|K|] ∈ Rd×|K|, formulated as qe(z0|e) = N (z0; e, β0I).
In the reverse process, the diffusion module fθz is trained to generate the final output ẑ00 (using time step 1’s input z1 and ẑ10) as its predicted fact embeddings ê, i.e., ê = ẑ00 = fθz(ẑ 1 0, z1, 1|zS). Finally, we pre-train another BART decoder fθg to generate the synthetic fact k̂n with conditioned on the diffusion module’s predicted n-th embedding ên = ê[:][n], (n = 1, 2, ..., |K|)5:
k̂n = fθg(ên) (6)
For diffusion on the entity-level embedding space (DIFFUCOMET-Entity), we use a pipeline to generate head entities, tail entities and their relations. First, to generate head entities, we use a similar process as in DIFFUCOMET-Fact, i.e., pretrain a BART encoder to produce a gold embedding
4More implementation details of the diffusion module fθz are presented in Appendix B.1.
5At inference time, the maximum value of n (number of generated facts) can be arbitrary depending on the user’s choice. In Appendix B.2, we introduce how we control the number of facts that our models generate for each context.
of each unique head entity hi ∈ H (for training the diffusion module), and then pre-train a BART decoder to generate synthetic head entities ĥi from the diffusion module’s predicted embeddings. Each predicted head entity ĥi is then appended to the context (i.e., S in Eq. 3), expanding the context to Si = [S, ĥi]. A second diffusion module predicts embeddings of synthetic tail entities âj related to Si (trained using gold embeddings of tail entities aj ∈ A that possess relations rij ∈ R to the gold head hi). A final BART model predicts the relation r̂ij between each pair of generated head and tail entities, grounded on the context.
Embedding Module Training We pretrain the embedding modules (fθe , fθg ), which focus on modeling generic knowledge representations independent to the context, before the diffusion modules (fθs , fθz ), which learn the specific mapping from the context to its relevant knowledge. When training the diffusion modules, we freeze the pretrained embedding modules.
To pretrain the fact (or entity) embedding modules, we minimize the decoder’s negative loglikelihood of re-constructing facts k (or entity h or a) in the full set of knowledge Kfull involved in the whole narrative dataset (or domain), based on its embedding given by encoder fθe :
Lθe,θg = − log pθg(k|fθe(k)) (7) Diffusion Module Training We optimize a dual loss to train the diffusion modules. First, we consider the mean-square error loss of the diffusion module’s de-noised sample prediction ẑt0 at each time step t, compared to the reference sample z0 (for t > 0) and gold embeddings e (for t = 0):
Lmseθs,θz = E∥e− ẑ00∥2 + T−1∑
t=1
E∥z0 − ẑt0∥2 (8)
We also use an anchor loss (Gao et al., 2022b) to supervise the final fact (or entity) generation. For each time step t, we minimize the negative loglikelihood of the embedding module decoder (with frozen parameters θg) generating each fact kn in knowledge set K, based on the diffusion module’s predicted de-noised sample ẑt0:
Lgenθs,θz = T−1∑
t=0
|K|∑
n=1
− log pθg(kn|ẑt0[:][n]) (9)
where ẑt0[:][n] is the predicted de-noised representation of kn. The final loss is Lθs,θz = Lmseθs,θz + γLgenθs,θz , where γ is a tunable hyperparameter.
Inference At inference time, the reverse diffusion process is initialized with noise sampled from the Gaussian distribution N (0, I), while the embedding module encoder fθe , which provides gold diffusion references for training, is not used.

heading: 4 Evaluation
text: Prior work in commonsense knowledge generation (Hwang et al., 2021; Da et al., 2021) evaluated knowledge models using traditional NLG metrics (e.g., BLEU; Papineni et al., 2002) in controlled studies with KGs, where the inputs to the models were head entities and relations and the knowledge model produced tail attributes. In practice, however, knowledge models are used to generate implicit commonsense inferences for natural language contexts (Ismayilzada and Bosselut, 2023), requiring generated inferences to be relevant to a more complex input than a basic KG head entity, and necessitating diverse generated inferences that comprehensively augment the context. However, traditional NLG metrics fall short of measuring these important dimensions because they measure surface form overlap between model outputs and references, which rewards generating facts with similar or duplicated semantics, limiting diversity.
Motivated by these shortcomings, we propose novel evaluation metrics that assess the diversity and contextual relevance of generated knowledge. First, to eliminate the effect of knowledge repetition in generations, we cluster similar facts and treat each fact cluster (instead of each single fact) as a unit piece of knowledge. In particular, we use the DBSCAN (Ester et al., 1996) algorithm to group gold facts K = {k1, k2, ..., kN} and generated facts K̂ = {k̂1, k̂2, ..., k̂N̂} into clusters C = {c1, c2, ..., cM} and Ĉ = {ĉ1, ĉ2, ..., ĉM̂}, respectively. We test two methods for measuring the similarity of facts for clustering: word-level edit distance (Levenshtein et al., 1966), which measures the difference of two facts’ surface-form tokens, and Euclidean distance of Sentence-BERT (Reimers and Gurevych, 2019) embeddings, which measures the semantic difference of two facts. Based on these clusters, we develop three metrics to measure the diversity of generated facts, their contextual relevance, and their alignment to gold references, as shown in Figure 3.
Diversity. To measure the diversity of generated facts (i.e., amount of distinctive knowledge being generated), we count the number of fact clusters
(# Clusters), i.e., M̂ (or M for gold references). We also report the number of facts (# Facts), i.e., N̂ (or N for gold references), to compare the number of fact clusters to the number of generated facts produced by the models.
Relevance. We measure the relevance of the fact clusters to the narrative context using a fact linker6 trained on the ComFact dataset (Gao et al., 2022a) that scores the relevance of each fact k̂n to the context S, denoted as rel(k̂n,S) ∈ [0, 1]. The relevance score of a fact cluster ĉm is defined as the average relevance score of its facts, i.e.,∑
k̂n∈ĉm rel(k̂n,S)/|ĉm|. Finally, we measure the average relevance over all fact clusters in Ĉ:
rel(Ĉ,S) = 1 M̂
∑
ĉm∈Ĉ
1 |ĉm| ∑
k̂n∈ĉm
rel(k̂n,S)
(10) We note that Relevance can be viewed as a precision measure for generated facts, which tends to decrease as more facts are generated because irrelevant facts are more likely to be generated.
Alignment measures the average similarity of generated facts to gold fact clusters. Specifically, we define a function sim(k̂i, kj) ∈ [0, 1] to measure the pairwise similarity between a generated fact and a gold reference (using similar distance functions to define clusters above7). Using this function, we measure the maximum pairwise similarity of generated facts to references in each gold cluster cm∈ C, which serves as the alignment score to the gold cluster. Finally, we average the alignment scores of generated facts to all gold clusters:
sim(K̂, C) = 1 M
∑
cm∈C max k̂i∈K̂, kj∈cm sim(k̂i, kj) (11)
6Fact linking models predict the relevance of knowledge tuples to textual passages (Gao et al., 2022a)
7Further details on exact definitions are in Appendix C.1.
We note that Alignment can be viewed as the generated facts’ recall of gold fact clusters, which tends to increase as more facts are generated because more facts will be aligned to gold clusters. Given this trade-off between Relevance and Alignment, we also present the harmonic mean of Relevance and Alignment as an overall evaluation of the two dimensions, denoted as RA-F1.

heading: 5 Experimental Settings
text: Datasets First, we evaluate our approach on the ComFact (Gao et al., 2022a) benchmark, where models need to generate ATOMIC2020 (Hwang et al., 2021) social commonsense facts that are relevant to narrative contexts sampled from four diverse corpora: PERSONA-CHAT (Zhang et al., 2018), MuTual (Cui et al., 2020), ROCStories (Mostafazadeh et al., 2016) and CMU MovieSummaries (Bamman et al., 2013). We only use training data from the ROCStories portion of ComFact, to enable the evaluation of zero-shot generalization on the other three partitions of the dataset. Our fact embedding module is pretrained on the full ATOMIC2020 knowledge base, which contains ∼ 972K commonsense facts after preprocessing.8 We also evaluate our approach in a conceptually different setting, the WebNLG+ 2020 (Ferreira et al., 2020) dataset, which consists of RDF (Ora, 1999) facts sampled from the DBpedia (Lehmann et al., 2015) knowledge base, with corresponding natural language texts verbalizations. The task is to generate the sampled RDF facts given their verbalizations. We use ∼13k facts from the training data to pretrain the fact embedding module.
Baselines We train DIFFUCOMET using BARTbase and BART-large as pretrained models, and compare with three baselines developed on the same backbones: a) a Greedy baseline that is
8More data preprocessing details are in Appendix D.
trained to autoregressively generate the concatenation of all relevant facts,9 b) a Sampling baseline that uses nucleus sampling (Holtzman et al., 2019) to generate multiple individual facts in parallel, and c) a Diverse Beam search baseline that uses diverse beam search to generate multiple inferences in parallel. We also compare our models trained using BART-large to baselines developed on models of similar scale: d) the aforementioned greedy decoding, sampling and beam search baselines trained from COMET-BART (Hwang et al., 2021), a BART-large model further pre-trained on ATOMIC2020 for commonsense knowledge completion, and e) Grapher (Melnyk et al., 2022), which trains a T5-large (Raffel et al., 2020) model to generate entities (nodes) related to the context, followed by a MLP classifier to predict the relations (edges) between entities.
Metrics We evaluate these methods on our clustering-based metrics described in Section 4. As the clustering algorithm (i.e., DBSCAN) used in our metrics has an adjustable clustering granularity controlled by a distance threshold, we consider a range of distance thresholds and take the average of evaluation results across all thresholds in the range, allowing us to avoid biasing our metrics to a specific distance threshold.10 For ComFact, we also test on the metrics from Hwang et al., 2021 for evaluating commonsense knowledge generation, including BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and ROUGE-L (Lin, 2004). For evaluation on WebNLG+ 2020, we also report the official metrics for this dataset’s challenge (Ferreira et al., 2020), which construct
9Facts are concatenated by a special token <fsep>. 10We include more details of our clustering threshold selec-
tion in Appendix C.2.
optimal pairings between predicted facts and gold references, and then compute precision, recall, and F1 scores based on the surface-form matching of paired facts. We denote these WebNLG metrics as Web-Prec., Web-Rec. and Web-F1.11

heading: 6 Results and Analysis
text: Table 1 shows evaluation results on the ROCStories portion of the ComFact benchmark for our DIFFUCOMET models developed based on BARTlarge.12 On our new cluster-based metrics, DIFFUCOMET models demonstrate a better balance between diversity and accuracy in contextual knowledge generation. Specifically, DIFFUCOMET models achieve Relevance and Alignment scores that are both comparable to the best baseline results, contributing to their higher overall RA-F1 measures, while also producing a larger number of distinct knowledge clusters. By contrast, the Greedy, Sampling and Grapher baselines significantly sacrifice one or two dimensions of diversity and quality w.r.t. # Clusters, Relevance and Alignment. Beam baselines consistently underperform DIFFUCOMET on cluster metrics.
For evaluation on the traditional NLG metrics, we find that DIFFUCOMET models score higher overall than most baseline models on metrics that check the alignment with gold references, i.e., BLEU, METEOR and ROUGE-L, except for the Greedy decoding baseline, whose higher scores are artificially high because it generates very little knowledge, i.e., only ∼2 facts per context. We also include further comparisons of models with
11More details of WebNLG metrics are in Appendix C.3. 12Presented results of our metrics are based on fact clustering w.r.t. embedding Euclidean distance. Results based on word-level edit distance are included in Appendix F.1, and promote the same conclusions.
BART-base backbones in Appendix F.1, where our models outperform baselines by a larger gap, i.e., ∼15% absolute RA-F1 improvement on average.
We also test DIFFUCOMET’s ability to generalize to out-of-domain contexts using the other portions of ComFact with contexts sampled from PersonaChat, MuTual and MovieSummaries. We report main generalization results to the above three portions in Table 2, and observe similar results where DIFFUCOMET-Entity outperforms baselines by ∼5% RA-F1 and produces ∼20% more knowledge clusters.13 Interestingly, DIFFUCOMET-Fact struggles to outperform beam search baselines on the longer context narratives from MovieSummaries, showing that entity-level diffusion is more robust to the shift of narrative contexts, likely due to the more fine-grained multi-step learning of context-to-knowledge mapping.
The results of our automatic evaluation are also supported by our human evaluation. We hire Amazon Mechanical Turk workers14 to evaluate the validity and contextual relevance of models’ generated knowledge on the ROCStories portion of ComFact. Specifically, given a narrative context
13Full evaluation results on out-of-domain contexts are reported in Appendix F.2.
14Details on workers and their payment are in Appendix E

heading: Model # Novel Facts # Novel Clusters
text: and a list of commonsense facts that a model generates about the context, we ask three workers to independently judge whether each fact is valid (i.e., plausible and natural-sounding) and relevant15 to the context, and take their majority vote as the assessment. In Table 3, we see that DIFFUCOMET models produce valid facts at about the same rate as the best baseline, but produce facts that are far more relevant to the narrative context.
Novelty DIFFUCOMET models also produce more novel commonsense inferences. A historical advantage of knowledge models (e.g., COMET) was their ability to generate knowledge beyond the graphs they used for pretraining (Bosselut et al., 2019), making them powerful tools to generate commonsense knowledge for unseen narratives. To test the novelty of generated commonsense knowledge from DIFFUCOMET, we develop a heuristic method that identifies knowledge as novel if its maximum pair-wise (Sentence-BERT embedding) cosine similarity to ComFact gold references is lower than 0.45. However, as this cut-off would likely cause invalid and irrelevant facts to be considered novel, we only include facts whose relevance score is higher than 0.97.16 In Table 4, we see that
15invalid facts are automatically labeled irrelevant 16Thresholds are tuned by a manual check of 100 sampled
results to ensure a decent cutoff of novel and relevant facts.
DIFFUCOMET models produce more novel facts and clusters compared to baselines.
This pattern is observed in our qualitative analysis too. In Table 5, we provide examples of novel facts generated by Beam-COMET and DIFFUCOMET-Entity for an example narrative. We find that our model produces more diverse novel facts covering both physical entities (e.g., solitaire game) and complex actions (e.g., drawing cards to play solitaire). By comparison, Beam-COMET’s novelty is mainly restricted to physical knowledge.17
Diffusion Steps To investigate how DIFFUCOMET’s multiple rounds of knowledge representation refinement through the diffusion process affect the quality of generated knowledge, we record the performance of our DIFFUCOMET models as a function of diffusion steps conducted during inference. Figure 4 shows how DIFFUCOMET’s performance varies when knowledge is generated at earlier time steps.
We find that DIFFUCOMET models gradually produce more facts and more diverse facts (i.e., # Clusters) as the number of diffusion steps increase, indicating that the multiple rounds of diffusion produce a more separable representation capable of representing more facts. While the greater number of facts leads to a slight drop in contextual relevance across the generated facts, a greater corresponding increase in alignment to the gold clusters (as observed by the increase in Alignment and RAF1) also emerges. On RA-F1, DIFFUCOMET-Fact surpasses Beam-COMET18 as the diffusion steps increase to larger than 200, and DIFFUCOMETEntity consistently scores higher and continues benefiting from further diffusion, even after 1000 dif-
17We provide more analysis of example generated facts in Appendix F.3.
18To make the comparison intuitive, for each test context, we dynamically set the beam size of Beam-COMET to the number of facts generated by DIFFUCOMET-Entity.

heading: Model Web-Prec. Web-Rec. Web-F1
text: fusion steps. These results shows that multi-step refinement of facts via diffusion effectively improves contextual knowledge generation.

heading: 6.1 WebNLG+ 2020 Benchmark
text: Finally, to test whether our method generalizes outside the domain of generating commonsense inferences, we present our evaluation results on the WebNLG+ 2020 dataset in Table 6. DIFFUCOMET models achieve better performances on the WebNLG factual knowledge generation task, verified by the official metrics of the benchmark.19 This results suggests that our diffusion approach to knowledge graph construction could be adapted to other knowledge generation tasks.
19We also include the evaluation results on traditional NLG and our proposed clustering-based metrics in Appendix F.4.

heading: 7 Related Work
text: Commonsense Knowledge Grounding To augment NLP systems with commonsense knowledge, various systems for question answering (Zhang et al., 2022; Yasunaga et al., 2021, 2022) and narrative generation (Ji et al., 2020; Zhou et al., 2022) use retrieval methods based on heuristics to link relevant facts from commonsense knowledge graphs (Speer et al., 2017; Sap et al., 2019; Gao et al., 2023). However, these systems typically have low precision when adapted to more general and complex commonsense linking (Hwang et al., 2021; Jiang et al., 2021). Gao et al., 2022a developed commonsense fact linking to improve retrieval precision, but this requires inefficiently traversing all candidate facts to check their contextual relevance.
Due to above limitations of retrieval-based knowledge grounding, one line of research (Bosselut et al., 2021; Tu et al., 2022) uses knowledge models (Bosselut et al., 2019; West et al., 2022) to generate tail inferences from narrative statements. However, these methods often produce irrelevant facts as the knowledge models are pre-trained for context-free knowledge graph completion. Finally, developing new knowledge models to learn contextual commonsense generation turns out to be a promising track of research, while current works are limited to simple physical (Zhou et al., 2022) or RDF-style factual (Melnyk et al., 2022) knowledge. We build new models to address contextual commonsense generation in a more general scope.
Diffusion Models Considerable recent works (Gao et al., 2022b; Lin et al., 2022; Han et al., 2024) have developed methods to improve text generation with diffusion models (Sohl-Dickstein et al., 2015; Song and Ermon, 2019; Ho et al., 2020). However, the potential of diffusion models in textto-knowledge generation is still under-explored. In this paper, we introduce diffusion models for the task of contextual knowledge generation.

heading: 8 Conclusion
text: In this work, we leverage the power of diffusion models for contextual commonsense knowledge generation, and formulate novel metrics to highlight important dimensions of diversity and contextual relevance for this task. Our diffusion knowledge models, DIFFUCOMET, outperform various autoregressive knowledge models, producing more diverse, novel, and contextually-relevant com-
monsense knowledge, and achieving better out-ofdistribution performance. Finally, our analysis reveals how DIFFUCOMET refines implicit knowledge representations over the course of the diffusion process to produce more relevant and diverse inferences, hinting at our method’s potential benefit in other text-to-graph generation tasks.

heading: Limitations
text: We outline the following limitations in our work. First, narrative samples in our training datasets, i.e., ComFact (Gao et al., 2022a) and WebNLG+ 2020 (Ferreira et al., 2020), have short context windows (five sentences at maximum). Therefore, our knowledge models trained on these datasets may have limited inference capacities if applied to longer narratives that involve richer commonsense grounding. Moreover, our models are trained on solely English corpora, and may need additional resources to be adapted to other languages or multilingual settings. Finally, our diffusion modeling method is tested on an encoder-decoder model structure, i.e., BART (Lewis et al., 2020), with maximum model size 406M (BART-large). We leave the feasibility of our method on other model structures, e.g. decoderonly GPT (Radford et al., 2019), and larger model scales, to future work.

heading: A Backward Diffusion Process
text: Inverting from the forward diffusion process formulated as Equation (1), the backward diffusion process follows a Gaussian posterior distribution q(zt−1|zt, z0):
q(zt−1|zt, z0) = N (zt−1; µ̃(zt, z0), β̃tI)
µ̃(zt, z0) = √ αt−1βt 1− αt z0 + √ αt(1− αt−1) 1− αt zt
β̃t = 1− αt−1 1− αt βt
(12)
where αt = 1 − βt and αt = ∏t
i=1 αi are weight hyperparameters of the posterior Gaussian defined by the noise schedule βt. The posterior formulation indicates that only the mean µ̃ of zt−1 is correlated to the condition zt and z0. So the training loss for diffusion models, derived from the KL-divergence between gold and learned posterior distributions, is typically defined as a mean-squared error loss on the posterior Gaussian mean:
Lmse = T∑
t=1
E∥µ̃(zt, z0)− µθ(zt, t)∥2 (13)
where model (with parameter θ) learns the function µθ(zt, t) to predict the mean of zt−1. DiffusionLM (Li et al., 2022) further re-weights the meansquared error as Eq. 2 to enforce direct prediction of z0 in every loss term, which is shown to be more efficient at tuning the model to precisely predict the final de-noised sample.

heading: B Model Implementation Details

heading: B.1 Diffusion Module
text: To conduct the diffusion process defined by Eq.(4) using Transformers (Vaswani et al., 2017), zt and ẑt0 are first concatenated at the hidden-state dimension and projected by a MLP layer to form their joint representation. The positional encoding layer of Transformers is applied to the time step t (same for every position of self-attention), whose output time step embedding is added to the joint representation of zt and ẑt0. The decoder fθz takes the joint representation (with time step embedding added) as its bi-directional self-attention input, to ground its decoding of refined z0 prediction ẑt−10 .

heading: B.2 Number of Generated Facts
text: To enable our diffusion module (fθz ) to control the number of facts (or entities) generated for each context, we also pre-train our fact (or entity) embedding module (fθe and fθg ) to learn the representation of a special token kend :=<eok>, by adding it as a special fact (or entity) to the pre-training data, which indicates the end of a knowledge set. During the training of diffusion module, kend is appended to the end of knowledge set K, whose embedding and decoding also contributes to the training loss. At inference phase, we post-process our model’s generations to keep only the facts that are at positions before kend.

heading: B.3 Noise Schedule
text: For the noise schedule hyperparameter of diffusion process, we adopt the sqrt initialization (Li et al., 2022) to set αt = 1− √ t/T + s, where s = 1e−4 that sets the initial variance of noise (β0) to be 0.01. Based on that, we follow SeqDiffuSeq (Yuan et al., 2022) to implement an adaptive noise schedule, which dynamically adjusts αt for each sample position n (n = 1, 2, ...|K|) of the knowledge set K (the adjusted αt for position n is denoted as αnt ), according to the diffusion mean square error (MSE) loss Lmseθs,θz defined in Eq. (8). Specifically, for an adaptive noise schedule update, we first record the MSE loss at each time t and position n as:
Lnt = E∥z0[:][n]− ẑt0[:][n]∥2 (14)
Then we use a linear interpolation function to update the adjusted noise schedule, formulated as:
Fnt (L) = αnt − αnt−1 Lnt − Lnt−1 (L − Lnt−1) + αnt−1 (15)
where new loss value Ln,newt is re-arranged across time step t with equal interval between mint(Lnt ) and maxt(Lnt ), which is finally given to the update function to get αn,newt = F n t (Ln,newt ). The noise schedule is adjusted every 2000 training steps.

heading: B.4 Model Training
text: For the loss weight hyperparameter γ used to combine mean-square error and anchor losses defined by Eq. (8) and (9), we use γ = 1 for training our DIFFUCOMET models based on BART-base, while γ = 0.01 for training our models with BARTlarge backbone, which achieve the best convergence results, respectively. For training DIFFUCOMET based on BART-large, we also follow Difformer (Gao et al., 2022b) to amplify the standard deviation of diffusion noise by a factor of A = 4, i.e., to change the forward process as:
q(zt|zt−1) = N (zt; √ 1− βtzt−1, βtA2I) (16)
where t = 1, 2, ...T , which effectively avoids model collapse in training. The total diffusion steps T is set to 2000. We use AdamW (Loshchilov and Hutter, 2018) as our training optimizer, with learning rate 1e−5 and no weight decay. A linear learning rate scheduler is adopted with warmup steps 2000 and total training steps 150000 and 200000 for models based on BART-base (139M) and BART-large (406M), respectively. We train our base-scale DIFFUCOMET on 4 Tesla V100SXM2 (32GB) GPUs with batch size set to 4, while for large-scale DIFFUCOMET, we use 4 NVIDIA A100-SXM4 (40GB) GPUs, with batch size set to 2 instead. 15 and 36 hours are required to train base-scale and large-scale DIFFUCOMET models, respectively.
For the pre-training of our fact embedding module with loss described in Eq. (7), we adopt the same hyperparameter setting as training our diffusion module, except for learning rate changed to 2e−6 and batch size set to 128 and 64 for basescale and large-scale models, respectively. For pretraining large-scale (i.e., BART-large) fact embedding module, we add a weight decay of 0.01, which leads to better convergence. In DIFFUCOMETEntity, the two diffusion modules trained for generating contextual relevant head and tail entities share the same pre-trained entity embedding module.

heading: C Evaluation Metrics

heading: C.1 Clustering and Similarity Function
text: For our evaluation based on fact clustering w.r.t. edit distance, we define the similarity function in our Alignment metric as sim(k̂i, kj) = 1 − Edit(k̂i, kj)/MaxLen(k̂i, kj), where Edit denotes the word-level edit distance of two facts, and MaxLen denotes the length of the longer fact of the two, i.e., the maximum possible edit distance for normalization. Our distance measure for clustering also adopts the normalized edit distance, i.e., Edit/MaxLen. For evaluation based on fact clustering w.r.t. Sentence-BERT embedding, we define the similarity function in our Alignment metric as sim(k̂i, kj) = max(CoS(k̂i, kj), 0), where CoS denotes the cosine similarity of two facts’ SentenceBERT embeddings. We assume that facts with opposite meanings, i.e., negative similarity, are not considered as aligned with each other, so we cut off the negative values of cosine similarity. While for the distance measure of clustering, we use the Euclidean distance of two facts’ embeddings instead, which is typically adopted in DBSCAN (Ester et al., 1996) clustering algorithm.

heading: C.2 Clustering Threshold Selection
text: For our proposed clustering-based metrics as described in Section 4, we use DBSCAN (Ester et al., 1996) algorithm to group facts into clusters. To avoid bias on a specific clustering granularity, we consider a range of DBSCAN thresholds and take the average evaluation results across all thresholds in the range. We consider a range with equal interval of 0.05, where the number of gold fact clusters significantly changes from near the maximum (i.e., each fact as a cluster) to near the minimum (i.e., all facts grouped into one cluster). Figure 5 shows the number of gold clusters as a function of the DBSCAN clustering threshold, and our selection of threshold ranges (red square) on each dataset.

heading: C.3 WebNLG Metrics
text: In the evaluation of WebNLG 2020 Challenge (Ferreira et al., 2020), each generated RDF fact (i.e., subject-predicate-object triple) is paired to a gold reference to compute its precision, recall and F1 based on named entity matching (Segura-Bedmar et al., 2013). Three types of matching criterias are considered, including: a) each named entity in generated RDF needs to exactly match an entity in gold reference in order to be counted as true-positive,
while its type in the RDF (i.e., whether it is in subject, predicate or object) does not need to match (Exact Match), b) each entity in generated RDF only needs to partially match an entity in gold reference, and its type does not matter (Partial Match), and c) each named entity in generated RDF needs to exactly match an entity in gold reference, and its type also needs to match (Strict Match). For each matching criteria, optimal pairing (with the highest F1 score) between generated facts and gold references is searched by enumerating all possible permutations. We report Strict Match scores in the main body of our paper in Table 6, and include all three kinds of match scores in Table 20.

heading: D Data Preprocessing
text: The ComFact (Gao et al., 2022a) benchmark contains social commonsense knowledge linked from the ATOMIC2020 (Hwang et al., 2021) knowledge graph, which contains ∼1.33M facts covering physical entities, daily events and social interactions. ATOMIC2020 commonsense relations considered in our experiments are listed in Table 7. We preprocess ComFact and ATOMIC2020 to filter out facts
that have invalid tail entity “none” or contain fillable blank “___”, i.e., we do not consider facts with relation “IsFilledBy”. After preprocessing, ∼972K facts are involved in the training of our fact embedding and diffusion modules. The original ComFact training data in the ROCStories portion only has ∼ 1K contexts with gold annotations of relevant facts. Due to the limited supervised data, we augment the training data with ∼ 50K additional contexts sampled from the ROCStories corpus, and use a DeBERTa (He et al., 2020) fact linker developed from the ComFact benchmark to extract silver annotations of relevant facts from ATOMIC2020 to each additional context.
For preprocessing WebNLG+ 2020 (Ferreira et al., 2020) dataset, we follow Grapher (Melnyk et al., 2022) to remove underscores and surrounding quotes appeared in the dataset, and convert non-English characters into their closest available English characters, e.g., “õ” and “å” are mapped to “o” and “a”. After preprocessing, We develop our models based on the ∼ 35K WebNLG training texts and their linked RDF facts.

heading: E Human Evaluation Details
text: Our annotator pool for human evaluation contains 58 Amazon Mechanical Turk workers who are located in the USA and have been previously qualified by us for other similar tasks. To prepare the
workers for the new tasks of assessing the validity and relevance of knowledge in a given context, we share the instructions with them beforehand and do a small pilot run where we evaluate the quality of the worker annotations and give feedback if needed. We pay each worker $0.10 for each task. Figure 6, 7 and 8 show screenshots of our acceptance/privacy policy and instructions for knowledge validation and relevance tasks. Our data collection protocol follows Amazon Mechanical Turk regulations, and
is approved by our organization’s human research ethics committee.

heading: F Full Results of Knowledge Generation

heading: F.1 ROCStories
text: In Table 8 and 9, we present our full evaluation results of contextual commonsense knowledge generation on the ROCStories portion of ComFact benchmark. For evaluating Sampling and Beam baseline models, we test two sampling or beam search sizes around the average number of gold facts per context, i.e., 10 and 15 as indicated by the suffix numbers, and adopt the size which achieves better F1 results. On both base and large model scales, DIFFUCOMET models achieve consistently better balance between the diversity (i.e., # Clusters) and accuracy (i.e., RA-F1) of knowledge generation, compared to baseline models that typically perform generation in the autoregressive manner.

heading: F.2 Context Generalization
text: In this section, we present zero-shot evaluation results of models (trained on the contexts of ROCStories) generalizing to the contexts of other three ComFact portions, including PersonaChat (Table 10 and 11), MuTual (Table 12 and 13) and MovieSummaries (Table 14 and 15).
We observe that both DIFFUCOMET models
generalize well to the contexts of PersonaChat and MuTual, whose generated knowledge possesses comparable diversity (i.e., # Clusters) and better accuracy (i.e., RA-F1) than the strongest baseline model Beam-COMET. More interestingly, we find that DIFFUCOMET-Entity achieves larger points of improvements over baselines on the more challenging MovieSummaries-style contexts, while DIFFUCOMET-Fact struggles to outperform the strongest baseline Beam-COMET, showing that entity-level diffusion is more robust to the shift of narrative contexts, likely due to the more fine-grained multi-step learning of contextto-knowledge mapping.

heading: F.3 Case Study and Knowledge Types
text: Table 16 showcases the knowledge generation results of DIFFUCOMET models in a narrative context sampled from ComFact ROCStories, compared to the sampling and beam search baselines Sample-COMET and Beam-COMET. Facts that are novel (i.e., beyond the coverage of gold references) and relevant to the context are labeled in bold. We find that both DIFFUCOMET-Fact and DIFFUCOMET-Entity can generate facts that are rich in diversity, covering both physical entities
(e.g., baseball cap) and social events (e.g., go on vacation). Novel facts generated by DIFFUCOMET models also uncover implicit inter-connections between entities or events in the narrative context, e.g., “vacation” and “family” are associated because “X goes on vacation” to “spend time with family”. By contrast, Beam-COMET model mainly generates simple facts about physical entities, and Sample-COMET model generates many facts that are irrelevant to the context, e.g., “field is used for playing baseball”.
We also conduct a study on the proportion of different knowledge types that each model generates per context, based on the ROCStories portion of ComFact benchmark. In particular, we divide commonsense facts into three types according to their relation groups under ATOMIC2020 knowledge scheme, as shown in Table 7, including facts that are centered on physical entities, events and social interactions. Table 17 shows the results of knowledge proportion generated by DIFFUCOMET and baseline models, with gold references. Compared to Sampling-COMET and Beam-COMET baselines, DIFFUCOMET models generate a larger proportion of facts that reveal complex event or social inter-connections. The proportion of social-based facts generated by DIFFUCOMET even significantly surpasses the gold references. All above results imply that diffusion models have the potential to uncover more in-depth and implicit commonsense inferences from narrative contexts, which may not be easily extracted from existing knowledge bases.
F.4 WebNLG+ 2020
We present our full evaluation results on the WebNLG+ 2020 benchmark in Table 18, 19 and 20. For evaluating Sampling and Beam baselines,
we set both sampling and beam search sizes as 5, which is around the average number of gold facts per context. Consistent with the evaluation results on ComFact, DIFFUCOMET models keep achieving better performances on the WebNLG task of factual knowledge generation, implying that our method of diffusion-based contextual knowledge generation can generalize well to knowledge beyond commonsense.

heading: F.5 Comparison of Fact and Entity Diffusion
text: For the comparison in between our two diffusion models, DIFFUCOMET-Entity in general outperforms DIFFUCOMET-Fact on our proposed metrics, which may benefit from more fine-grained multi-step learning of knowledge construction in pipeline. However, DIFFUCOMET-Fact is computational cheaper, i.e., only requires a single step of fact diffusion instead of two steps of (head and tail) entity diffusion and a relation prediction.

heading: Future Work
text: , 2019), and larger model scales, to future work