Harnessing Black-Box Control to Boost Commonsense in LM's Generation
Authors: Yufei Tian
Felix Zhang
Nanyun Peng
Venue: EMNLP 2023 Main (2023)
Keywords: controllable text generation
generative commonsense reasoning
TL;DR: We present a computation-efficient framework that steers a frozen PTLM towards more commonsensical generation (i.e., producing a plausible output that incorporates a list of concepts in a meaningful way).

Abstract: Large language models (LLMs) such as GPT-3 have demonstrated a strong capability to generate coherent and contextually relevant text. However, amidst their successes, a crucial issue persists: their generated outputs still lack commonsense at times. 
Moreover, fine-tuning the entire LLM towards more commonsensical outputs is computationally expensive if not infeasible.
In this paper, we present a computation-efficient framework that steers a frozen Pre-Trained Language Model (PTLM) towards more commonsensical generation (i.e., producing a plausible output that incorporates a list of concepts in a meaningful way). Specifically, we first construct a reference-free evaluator that assigns a sentence with a commonsensical score by grounding the sentence to a dynamic commonsense knowledge base from four different relational aspects. We then use the scorer as the oracle for commonsense knowledge, and extend the controllable generation method called NADO to train an auxiliary head that guides a fixed PTLM to better satisfy the oracle.
We test our framework on a series of GPT-2-, Flan-T5-, and Alpaca-based language models (LMs) on two constrained concept-to-sentence benchmarks. Human evaluation results demonstrate that our method consistently leads to the most commonsensical outputs.

BooST: Harnessing Black-Box Control to Boost Commonsense in LMs' Generation
Yufei Tian Felix Zhang Nanyun Peng
Department of Computer Science, University of California, Los Angeles (yufeit, vis wcw.. edu
Abstract
(a) Concepts
hand, øyster, glove
woman is opening an oyster and then she puts the oyster in her glove.
Large language models (LLMs) such as GPT-3 have demonstrated strong capability gen- erate coherent and contextually relevant text. However, amidst their successes, crucial issue persists: their generated outputs still lack com- monsense at times. Yet fine-tuning the entire LLM towards more commonsensical outputs is computationally expensive if not infeasible. In this paper, we present computation-effit framework that steers frozen Pre-Trained Lan- guage Model (PTLM) towards more common- sensical generation (i.e., producing a meaning- ful and plausible output that incorporates a list of concepts).
Alpaca
She opened her hand reveal oyster glove.
in yellow raincoat latex gloves opens an oyster with hand.
Ours
(b) Concepts
wear, sunglasses, night
GPT-2
A young woman wearing long dress and sunglasses at night.
Alpaca
sunglasses at night and enjoyed the stars.
Someone wears sunglasses night to avoid bright lights of approaching car.
Ours
Specifically, we first construct a reference-free evaluator that assigns sentence with a com- monsensical score by grounding the sentence to dynamic commonsense knowledge base from different relational aspects. then use the scorer the oracle for commonsense knowledge, and extend the controllable genera- method called NADO to train an auxiliary head that guides fixed PTLM to better satisfy the oracle. We test our framework on a series of GPT-2-, FLAN-T5- and Alpaca-based language models (LMs) on two constrained concept-to- sentence benchmarks. Human evaluation re- sults demonstrate that method consistently leads to the most commonsensical outputs."
(c) Concepts
food, customer, watch, employee, prepare
GPT-2
Two employees watch customers prepare food in the
GPT-3
The employee watched the customer prepared their food.
Davinci-003
Several employees preparing food while customer waits and watches.
Ours
Figure 1: LMs such as GPT-2 finetuned, Alpaca-7b fewshot, and GPT-3 Davinci-903 fail to incorporate the concepts in a commonsensical way. We highlight the insensible phrases in purple. (c) illustrates that they are also vulnerable to perturbations of the input prompt as simple as the swap of two concept positions. Our system which uses auxiliary model to steer frozen PTLM generates the most commonsensical outputs.
1 Introduction
impotence (Marcus, 2020; Elazar et al., 2021; Ma- howald et al., 2023), and a discrepancy in what LLMs generate in the wild versus in question an- swering (Chen et al., 2023).
Recent years have witnessed remarkable progress in massively Pre-Trained Language Models such as GPT-3 (Brown al., 2020), Llama (Touvron et al., 2023) and instruction following models such as Flan-T5 (Chung et al., 2022), ChatGPT (OpenAI, 2022), and Alpaca (Taori et al.. 2023). However, one significant drawback is the lack of common- sense knowledge in their generated texts. There have been criticisms around their commonsense
In this paper, we explore the task of generative commonsense reasoning: a constrained text generation task aiming to generate plausible sentence given list of concepts input. As depicted in Figure language models should generate sentence that incorporates 'open, hand, oyster, glove' a meaningful way that aligms with our commonsense. We unveil that LLMs are
Source code will he available at https: http::///gtttuubooo com/ PlusLabNLP PBOONLPP//OSTL.... EMN_P__

---

Pre-tr Pre-trained Language Model p(y|x)
y": A harzs is being lassoed by a 52M
y"; The costoy used his lasso to catch the
Input Concepts =x):---+ lasso, horse, cow
y*: A waran is pulling horsa and a man lessoing cos.
Final Output Token Distribution: 4(ylx)
Auxiliary NADO Model (x)
(K, y score) (6, y score)
Oracle
TRAIN
Comonsense
(x, y*, score)
(CS) Scorer
The process of BOOST to steer frozen PTLM with an additional neural model and oracle commonsense scorer. The solid lines indicate the training process, while the dashed lines indicate inference. practice, we combine commonsense scorer with lexical checking rules, and use the joint signal train the auxiliary model.
Figure 2:
unreliable and fail generale commonsensical outputs when the input concepts get complicated. another case depicted in Figure I(c), when we swap the position of two input concepts 'customer' and 'employee', LLMs such as Davinci-009 are vulnerable to the change and generate 'employee watched a customer prepare food' despite being instructed not consider the concept appearance order, which is far from plausible.
Note that our training process is generalizable and only requires access the output probability of the PTLMs, which is also efficient due to the smaller size of the auxiliary model.
We test our method on gpt-2, Alpaca, and Flan-T5 on two datasets: 1) CommonGen (Lin et al., 2020) that focuses daily concepts (e.g., <open, hand, oyster, glove>) and 2) CSK-PN (Chen et al., 2023) that contains concepts linked with negated commonsense relations (e.g. <wear sun- glasses, at night>).
Various knowledge- auggeeeedd systems have been previously proposed to incorporate external knowledge into the model (Liu et al., 2021; He et al., 2022) for more plausible generation outputs. However, they all require updating model weights at the scale of hundreds millions of parameters such BART (Lewis et al., 2020). As PTLMs continue evolve and scale up to hundreds of billions of parameters in size, finetuning the entire LM be- comes computationally prohibited for many parties academia and the industry.
Our contributions are two-fold. First, we pro- pose reference- evaluator to assess how com- monsensical sentence is, which achieves on-par performance with referenced- metrics such as BERTScore (Zhang al., 2019) in terms of correlation with human judgment. Second, we ex- tend a controllable generation approach to improve commonsense for black-box PTLM. Experimental results show that our method consistently results in the most commonsensical outputs.
In this work, we propose BOOST, a framework to boost the commonsense of PLTMs' generation in plug-and-play manner (Figure 2), which is in- spired by the recent development of controllable generation to use small auxiliary model to control PTLM by training on its self-generated samples (Meng al., 2022). Specifically, to better integrate commonsense knowledge, we first build scorer that evaluates how commonsensical sentence is. The commonsense scorer, called O-Scorer, extracts tuples of commonsense-relates concepts (e.g., <customers, prepare their food>) from a sen- tence, and scores the extracted tuples grounding the tuples to a dynamic commonsense knowledge base (CSKB) (Bosselut al., 2019; Ghazarian al., 2023). Next, we use the signal from the O-Scorer to train an auxiliary model that steers the PTLM toward more commonsensical outputs.
Methodology
2.1 Overview
Figure provides overview of approach, BOOST. During training, BoOST first generate numerous samples (y),... (y',..,,y') from the PTLM conditioned on the input constraint x (e.g., 'lasso horse cow'). We then construct an oracle to give commonsense scores on all of these self-sampled generations. Next, for each y" of length T,, we train the auxiliary model called NADO which es- sentially learns to predict the expected cs score of the complete sequence y' given x and an incom- plete sequence y inference time is illustrated in dashed lines: both the PTLM and NADO take x and the generated sequence (prefix) y-ı. as input, from which we
[1, 2, ..., T.l). The flow

---

Sentences:
Tuple Extraction We present labor and cost efficient way extract all tuples from a target sen- tence, including both commonsensical and nonsen- sical tuples. LLMs such as GPT-3 and ChatGPT (Brown et al., 2020; Ouyang et al., 2022) have demonstrated remarkable ability of few-shot in- context learning for semantic parsing tasks (Dong and Lapata, 2016; Dunn et al., 2022). Motivated by such progress, instead of asking human workers to annotate a training set of sentences, we leverage OpenAT's GPT3.5-Turke model to parse the rele- vant tuples. We hand-crafted 9 examples for our few-shot prompt such that the LLM can accurately extract both sensical tuples (e.g.. girl is Capable Of blowing candles) and nonsensical tuples (e.g., horse is Capable Of riding bikes) from the input sentence. The complete instruction and prompt be found in Appendix A.
Peel an apple with drill and peeler. girl is blowing out candles cake.
horse riding bikes on river.
Extracted Tuples:
Dynanic CSKB
Score
Drill
Peelor
UsedFor
UsedFor
Peel Apples
Apples
0.7
Girl
Candles
Candles
Eapabledg
UsedFar
AtLocation
1.0
Blow Candles
Be blown
Cake
Horse
Bike ArLocation
Horse
Capabl e0f
AtLocation
0.1
Ride Bikes
River
Figure An example of our oracle commonsense scorer. We first extract tuples from a target sentence, and assign each extracted tuple with commonsensical score using COMET (Bosselut al., 2019), a dynamic com- monsense knowledge base. The sentence-level score is then obtained by aggregating tuple-Ievel scores.
obtain the final output distribution q(y/x).
However, in practice, using GPT-3.5-Turbe parse all sentences needed to train our auxiliary model is costly and unreliable when dependent on the unpredictable traffic of OpenAT's API. To obtain an extractor that can parse ~ a million sen- tences at reasonable cost, we finetune T5 large model (Raffel et al., 2020) 6,000 GPT-3.5 an- notated sentences for the same task. We show the performance of both tuple extractors in $3.2.
The rest of this section is organized as follows. In $2.2, we first introduce details to construct the commonsense scorer. Then, in $2.3, we provide the theory and practices to train the auxiliary model on PTLM's self- geffraenee data towards the oracle.
2.2 Constructing Commonsense Scorer
We use commonsense relation tuples the inter- mediate representation of sentence. Specifically, we get rid of human annotation and leverage on the results of few-shot LLMs. We then check whether these extracted tuples are sensible. To this end, we assign each parsed tuple with a compatibility score based on its maximum similarity with the numer- ous valid accepted answers generated by COMET, dynamic commonsense knowledge base (CSKB). Scores for all tuples in target sentence then aggregated to obtain the sentence-level common- sense score. Figure provides illustration of our oracle scorer.
2.2.2 Generative Commonsense Scoring
After extracting relation tuples from a sentence, we need to assess how commonsensical they arc. To this end, we follow the compatibility test proposed by Ghazarian et al. (2023) and leverage COMET (Bosselut et al., 2019), a pre-trained generative commonsense transformer that can predict sensi- ble tails given the heads and relations as input. Compared to other fixed and predefined knowledge bases, COMET dynamic and much more flexible when dealing with original and unseen inputs.
Formally, given a tuple T, = (h4, ri. 44) and a dynamic CSKB denoted by Cay. we query Cay with the head h and relation to obtain diverse list of conditionally generated tails with beam decoding:
2.2.1
Commonsens--Relation Extraction
Tuple Format We leverage the format of Con- ceptNet (Speer et al., 2017), a widely used knowl- edge graph connecting concepts or events with commonsense relations to represent general world knowledge. Specifically, each tuple T contains head concept/event h (c.g., driller) and tail con- cept/event (e.g.. drill hole), which are connected through a commonsensical relation (e.g.. is Used For). We consider four crucial relation types that dominantly exist: is UsedFor, Capable Of, At Location, and is Part Of.
{ Cet (hi, rı, beam k)) The common- sense score for T is computed by
(T))) max cos(emb(t:), emb (mm()))), ISick
(1)
where emb(-) is the vector representation from sentence embedding model (Reimers and Gurevych, 2019). Finally, we need to aggregate the compatibility scores computed from different

---

triplets extracted from a single sentence. The sentence-level commonsense score is denoted as the O-score. One rationale is that single non- sensical tuple can result in a nonsensical sentence, while the other is that one mistake will he mitigated by other reasonable tuples. We hence take the I) minimum and 2) average compatibility scores, and study their correlation with human judgement in $3.3 and Table 2.
Please refer to Meng et al. (2022) for details of the proof. From Eq.4 we see that when both x and Yt<r" are fixed, the optimal auto-regressive model is factorized into D and p at step T':
q" (yr |x,) ) i nt (x.y (or |x,err) (5)
Approximation As cannot enumerate y that contains infinite number of sequences, the well- defined RP is intractable. A neural model called NADO is hence introduced to approximate R by training on numerous samples y generated by p.
2.3 Commonsense- Generation
In this subsection, we describe how we use our derived commonsense oracle to steer the PTL) toward more commonsensical outputs through a neurally-decomposeed head (NADO). In 12.3.1, we summarize the theoretical solution of Meng et al. (2022) to decompose the sequence-level oracle into token-level guidance with frozen PTLM, such that when generating the i-th token, the auxiliary neural network modifies the original output logits predicted by the PTLM. Then, $2.3.2, we lever- age this method to generate more commonsensical outputs. Note that our model only trains the addi- tional NADO head which has much smaller size than the PTLM and does not require access to the parameters inside the PTLM.
2.3.2 NADO-Guided Generation
Given a pre-trained language model such as the GPT-2 and Alpaca model, we first ask p to generate numerous samples to obtain an approximation of y with various inputs concepts x E X. We then use the oracle O to assign each sample score, which is used to train the NADO model.
Training During training, the NADO model takes x, as input, and learns to predict from R (x. yr=o) till R ytgT)) Here, T is the com- plete sequence length and the sentence-level value (xx y) is used as the labels for all steps, from t = 0 till t = T. We emphasize that in order for O to learn R successfully, all (x, pairs must be self-sampled by the base model p instead of come from the CommonGen training data.
2.3.1 Token-Level Guidance with NADO
Notation Suppose have sub-optimal PTLM p(y our goal to obtain an optimal auto-regrossive model from such that q gener- ates outputs better satisfying the oracle scorer O (for example, q's generated outputs achieve higher O-scores than p). We now define a predictive function R Yt<T)) that predicts the expected O O-soores of the complete sequence y given input x and the currently generated tokens yterr-
We use cross entropy loss as the objective func- tion. Given a particular input x, the cross entropy loss is
CCE(x)
= Σ ply xx)LCE (x,
yEy
= Σ CF
CF (RO (x,ysi), O(x,ys)
(6)
Ro (ver)) = Expysplylx) (0(x,)) 10(x)) Yerl
In practice, also add regularization term to the loss. In order satisfy the def-
(2) inition that Eg, R (x. (yi x,)
no (x, 1), our regularization loss is measured by the KL divergence of the following:
= EE ply Xxerr)
yey
(3)
(x)g(x) = KL() Σ R (x,y<)) p x.)))
The unique closed formed solution of the optimal q is (namely, generates most common- sensically according to O):
Solution
(7)
RO (x,ysi-1)
(8)
R (x.y
q (y-T xxYeTr)
RD (x) (y=Tr xxyer)
Then, the final training loss is LCE + XCreg (x). (4) where is a peper-parameter to balance these two

---

terms. In practice, use grid search and choose the best À from [0.1. 0.5, 1.0].
At Lo-
Used For
Capa- ble Of
Purt Of
Relation Type
All
cation
T5-Large base
71.0
67.1
65.6
76.8
70.1
Inference At inference time, there are two for- ward passes as shown in Eq.5 and Figure 2. The decoding efficiency roughly remains unchanged be- cause the NADO head has much smaller size than the base PTLM.
T5-Large aug
72.4
67.9
68.9
79.2
72.1
T5-l.arge all
73.4
70.5
69.4
79.6
73.2
Few-Shot GPT-3.5
83.5
71.1
78.7
82.2
78.9
Table I: The performance of different tuple extractors, measured by Fl-score. The last row indicates the upper bound that our T55 models can achieve.
3 Experimental Results for the Oracle
In this section, we show the results of the com- monsense scorer described in $2.2. The experi- ments and results of commonsense-guided genera- tion ($2.3) can be found in $4 and $5.
Reference- min mean
Reference-Based
O-Score GPT-3.5 O-Score
0.276 0.284
METEOR-an
0.214
0.281 10,299
BERTScore-one
0.280
Gold O-Score
0.346 0.365
BeRTScore-all
0.302
Table 2:
Spearman correlation between human common- sense ratings and six automatic metries: our O-Score with tuples extracted by T5, GPT-3.5-Turbo, and the gold tuples, plus METEOR and BERTScore.
3.1 Tuple Extraction Data
We use the GPT-3.5-Tuiie 5-Tuoboo model provided by Ope- nAl to extract the tuples of 6,000 sentences (with a total cost of S S12.4), based on which we train the T5-large based tuple extractor. Since our goal is to parse all possible commonsense tuples whether they are sensical or not, need both sensical and less reasonable sentences. To this end, we randomly select 3,000 sentences from the Com- monGen (Lin et al., 2020) train split (we consider them as more sensical) and another 3,000 sampled from a basic gpt-2 model (we consider them as less coherent and sensical).
our extractors perform worse on UsedFor and Ca- pableOf than on AtLocation and PartOf, which is partially due to the errors of the training signal (i.e., labels are inaccurately annotated by GPT-3.5).
3.3 Oracle Commonsense Scorer Results
To compute the machine-generated compatibility score in Eq. we set beam size = 128. Mean- while, we instruct human annotators to evaluate the target sentences on how commonsensical they are. Each sentence is annotated by 3 workers with a scale of (least) to 4 (best). We also ask every an- notator to specify which part of the target sentence is nonsensical. We find out that explicitly asking the workers to pay detailed attention and point out the erroneous parts helps to increase the inter an- notator agreement (IAA, measured by Spearman's 's correlation) from 0.56 to 0.67. The final sentence- level commonsense score annotated by humans is the average of 3 individual ratings.
3.2 Tuple Extractor Results
Following the rationale in $3.1, we study the ben- efit brought by augmenting the training data with tuples extracted from less coherent and sensical sentences. Specifically, we compare the following three settings: 1) base: trained on the 3,000 sen- sical sentences; 2) aug: trained on 1,500 sensical sentences and 1,500 less sensical sentences; 3) all: trained on all 6,000 sentences. We test the model performance a held-out set of 350 sentences that is mix of both types. To obtain the gold labels on the test set, we start with the fhot GPT-3.5's annotation. After that, two human annotators itera- tively checked and fixed any error they see.
Table 2 shows the correlations between human ratings and automatic scores. For our proposed O-Score, we report the correlations of taking the minimum (min) and average (mean) of all tuple- level compatibility scores. Taking the average consistently result in higher correlation, reflect- ing that one mistake of a nonsensical tuple can be mitigated by other sensical ones. Therefore, we use the mean score train the auxiliary model. We also compare with reference-based metries such as METEOR (Banerjee and Lavie, 2005) and BERTScore(Zhang et al., 2019). Since there are, on average, 4 references per candidate in the Com-
For each relation type, we report the average fl- score in Table 1. Here, if the lemmatized tokens in generated triplet has over 50% overlap with those in the ground-truth triplet, we consider it as correct. Otherwise, we consider it as wrong. Com- paring La-Large aug with T5-Large base Table 1, we see improvements across all four relation types. Besides, increasing the train data size also boosts the extractor's performance. We also notice that

---

monGen dataset, we select the first reference to compute BERTSScoreonee, and all available refer- ences to compute BERTScore-all. We show that our reference- free scorer performs on par with the best reference-based metric, BERTScorr-all, and outperforms the same when use gold tuples ex- tracted by human.
on the CommonGen training data for epoch. well before the finetuning converges. We call this process warm up, as the goal is mainly to get the smaller base model onboard with our task format. For instruction-following models such as Alpaca, we still add this warm up process for a fair compar- ison. In total, we apply our commonsense- generation method to 5 different base models: gpt-2-large with warm up, zero-shot Alpaca-7b, few-shot Alpaca-7b, Alpaca-7b with warm up. and zero-shot Flan-75-large.
4
Experiments about Guided Generation
4.1 Data
Training Data As is illustrated in Figure 2, we train our auxiliary model on the PTLM's self- sampled data. For each set of input concept, we use top-p sampling (p=0.95) with temperature T=0.7 generate N samples. In theory, the larger the N. the more accurate approximation Rº can learn. practice, due to limitations in computational re- sources, we set to 48 when the base model is gpt-2, and 10 for Alpaca. In total, we have 1.5M training instances self-sampled by 2pt-2 and 0.3M training instances self-sampled by Alpaca.
Auxiliary Models. The auxiliary R models are 4-layer transformer decoders with the same di- mension and number of heads as the base mod- els. 3 They are 1/9, 1/8, and 1/12 the size of gpt-2-large, Alpaca-7b, and Flarge We train the auxiliary models for 10 epochs with a learning rate of le - 5 on a single NVIDIA A100 80GB GPU. In comparison, it is not possible to fine- tunc Alpaca-7b using only one 80GB GPU with- out any memory-saving technique such as LoRA (Hu et al., 2021).
Test Data We test on two different datasets. The first is the CommonGen dev split (Lin et al., 2020) which contains 993 lists of keywords focusing on daily concepts (e.g.. <open, hand, oyster, glove>). Each list of keywords is paired with more than one human written references. Our second test data is distilled from CSK-PN (Chen et al., 2023), which sources challenging triples from Concept- Net (Speer et al., 2017) and tags them with posi- tive/negative relation labels. We randomly select 993 triples with negative relations from CSK-PN (e.g. <wear sunglasses, at night>). There is no human reference for the second set. To reduce the effect of data leakage in GPT-3 and Alpaca, we ran- domly shuffled the keywords within each entry.?
4.3 Compared Systems
A &esque Decoding (Lu et al., 2022) A Neuro- logic decoding algorithm that injects constraints into a neurologic process with a look ahead heuris- tic, which results in more plausible outputs.
Gelato (Zhang et al., 2023) A tractable probabilis- tic model (TPM) impose constraints in language models such as gpt2-large. It achieves state-of- the-art (SOTA) performance on constraint satisfac- tion. Because it is non-trivial to train new TPMs on Alpaca-bases models, we use the authors' original TPM which is trained on the gpt2-large model that is finetuned on CommonGen.
Lex (Meng et al., 2022) The vanilla NADO method trained only with lexical constraints as the sequence-level Boolean oracle. Namely, the scorer returns if all lexical constraints are satisfied, and 0 otherwise.
4.2 Experimental Setup
Choice of Base Models. Although our framework does not require fine-tune PTLMs, it does require access to the PTLM's output distribution. Hence, we cannot apply our method to some popular but close-sourced LLMs such as ChatGPT. We choose Alpaca, Flan-T5, and gpt2 instead. In addition, because the pre-trained gpt2 has no instruction fol- lowing abilities, we have to train it to learn the task of "generating a commonsensical sentence given these input concepts'. Specifically, we finetune
BOOST (Ours) Our method that uses the common- sense oracle steer the auxiliary NADO model. We compare two variations: 1) BooST CS using only the commonsense oracle introduced in $2.2, 2) BoOST Joint: multiplying the lexical checking Boolean function (the same used in Lex) with the commonsense oracle score.
"Wo know that GPT-3 eext-davinci-003 is trained on recent dataset up to 2021 which likely contains the ConceptNet (Speer et al., 2017) and CommonGion (Lin et al., 2020) data.
'For Flan-T5,, the encoder-docoder model, the auxiliary head is 4-layer T5 decoder that takes in the input constraints as hidden steps from fixed, pretrained encoder.

---

Test Data
CommonGen (Lin et al., 2020)
CSK-PN (Chen et al., 2023)
Automatic Coverage
Human CS
Automatic Score
Human
Evaluation Metric
Score
BLEU4
Overall
Coverage
CS
Overall
Setting: gpt2
*eequu (Lu al., 2022)
0.469
97.2%
28.1
2.37
2.72
0.489
63.0%
3.14
3.09
GcLaTTo (Zhang al., 2023)
0.592
99.3%
33.0
2.45
2.78
Base Model
0.514
90.7%
23.2
2.31
2.80
0.53
83.9%
3.10
3.04
Lex (Meng al., 2022)
0.538
96.1%
2.38
0.544
92.1%
3.14
3.06
BOOST CS
0.615
90.9%
23.6
2.64
3.12
0.595
89.2%
3.33
3.13
BOOST Joint
0.597
96.1%
30.1
2.54
0.587
92.0%
3.28
3.18
Setting: F1an-T5 zero-shot
Base Model
0.571
84.6%
17.5
2.86
2.80
0.555
80.7%
2.78
2.71
Lex
0.577
93.7%
26.0
3.04
2.92
0.569
89.6%
2.97
2.88
BOOST CS
0.619
91.3%
21.6
3.14
3.05
0.613
88.9%
3.07
3.03
BOOST Joint
0.606
93.1%
25.6
3.06
0.601
89.6%
3.08
3.00
Setting: Alpaca warm up
Base Model
0.563
91.5%
20.9
3.02
2.84
0.523
93.9%
3.07
3.06
Lex
0.584
95.9%
30.5
3.12
3.00
0.535
95.0%
3.14
3.10
BOOST CS
0.611
93.6%
28.9
3.36
3.11
0.558
94.4%
3,21
3.19
BooST Joint
0.592
95.7%
30.3
3.32
3.11
0.543
94.8%
3.23
3.21
Sefting: Alpaca zero-shot
Base Model
0.509
90.4%
21.3
2.98
3.07
0,536
93.2%
3,26
3.09
Lex
0.566
95.3%
30.1
3.03
3.05
0.547
95.5%
3.21
3.11
BOOST CS
0.603
92.1%
24.4
3.36
3.17
0.565
93.9%
3,51
3.28
BOOST Joint
0.588
95.0%
29.7
3.32
3.23
0.559
95.4%
3.40
3.19
Setting: Alpaca few-sho few-shot
Base Model
0,552
92.2%
22.5
3.19
3.03
0,546
92.4%
3.27
2.89
Lex
0,581
95.7%
30.8
3.26
3.03
0,551
95.3%
3,26
2.92
BooST CS
0.608
94.6%
28.4
3,38
3.18
0.584
94.8%
3,40
3.10
BooST Joint
0.591
95.7%
30.1
3.36
3.18
0.572
95.2%
3.37
3.14
Table Intra- Srorpp evaluation results on two benchmarks: CommonGen (with reference) and CSK-PN (without reference). Here, we define group multiple systems under the same setting (i.e.. base model) and on the same dataset. We use boldface to denote the best scores within each group, and underlines to denote the second best. Our model BooST consistently achieves the most commonsensical ratings as annotated by humans. The gap between BooST and the corresponding Base Model is statistically significant (p<0.05) measured by Student's t-test. Note that the human ratings across groups are not directly comparable as they are conducted in separate batches.
GPT3/ChatGPT We instruct OpenAl's 3.5-turbo and text-davinci-003 to gener- ate plausible sentence given the constraints, stating that the keywords do not necessarily have to remain in the same order. Note that these models are likely be trained our data already. For all compared systems, we decode with top (k: = 30) sampling with a temperature 0.7.
like the sentence overall (e.g.. being interesting and informative). An example questionnaire with the full instructions can be found in Appendix C. We pay the MTuurkers $18 per hour, and the annotation process is the same mentioned in $3.3.
Inter-Group and Intra-Group Comparison. Our human evaluation is relative, meaning that the human evaluators are asked to compare the quality of different aecchne--gnnertte outputs given the same input constraint. Since we have five base models and each entails group of systems compare with, first conduct human evaluation within each group. Then, we select representative systems for inter-group group comparison.
4.4 Evaluation Setup
We use the keyword cover- age ratio (after lemmatization) and the score as automatic metrics to assess the quality of generated texts. For the CommonGen benchmark which contains human written sentences, we also report the n-gram overlap (BLEU-4). Considering that systems are trained towards higher O score, we also conduct human annotation for unbiased evaluation. Specifically,, we instruct the MTurrkes to evaluate 1) how commonsensical each sentence is from 1 1-4 Likert scale, and 2) how much they
Evaluation Metrics
Result and Analysis
5.1 Intra-Group Results
We compile the results on the CommonGen and CSK-PN benchmark Table We find out that,

---

Test Data
CommonGen
CSK-PN
Constraint
table, dog, game, walk, fireplace (from Com- monGen)
Human Eval
Overall
Overall
Gelato
A dog is playing a game on a table next to fireplace.
*esque
3.07
2.81
3.09
3.04
GeLaTo
3.15
2.78
A group of people are walking and playing video games at their dining room with fire- places, tables, and dogs.
BOOSTga12
3.27
2.95
3.24
3.00
A Decoding
Alpaca warm up
3.27
3,05
3.21
3.01
Alpaca few-shot
3.32
3.20
3.20
3.10
The dog walked around the table playing game by the fireplace.
Davinci-003
BOOSTaIpac varn
3.40
3.17
3.41
3.16
BOOSTлuрас tew-shot
3.44
3.28
3.38
3.18
The dog walked around the table while we played game by the fireplace.
BOOST Joint
Text- SattDavinci-003
3.33
3.19
3.33
3.10
The dog plays the game of walking from the table to the fireplace.
ChatGPT
3.46
3.09
3.49
2.95
Reference
Human
3.49
2.99
Constraint
statue, liberty, alive (from CSK-PN)
Table 4: Inter-Group human ratings. The scores of all models are comparable within the same test benchmark. We color human performance in grey background, and use boldface/underlines to denote the best/second -eett scores among all machines.
A Decoding
There are still some people who want to see statues of liberty as living creatures.
The Statue of Liberty became alive on bright and sunny day.
Alpaca
The statue of Liberty is alive and stands
Lex
proudly in New York City.
Davinci-003
The Statue of Liberty stands alive and proud.
BLEU-4 has a high correlation with the keyword coverage ratio (r = 0.914 measured by Pearson Correlation), but has close to zero correlation with human judgment on commonsense (r 0.088 and overall preference (r 0.04). We therefore hypothesize that BLEU-4, coverage ratio, and other metries measuring the superficial lexical overlap with ground truth, cannot identify meaningful and commonsensical outputs at least in our setting, Moreover, in all eight groups of human evaluation, BooST successfully improves the com- monsense level and overall preference. Comparing Flan-T5 with gpt2, we see that our approach is more effective on instruction-tuned models than similarly-sized decoder only models. addition, although BOOST Joint achieves slightly lower commonsense ratings than BooST CS, the later is a lot worse in the keyword coverage, indicating that BooST CS has higher risk to generate reasonable sentences without satisfying the input constraints. Hence, in the constrained generation setting, still consider BooST Joint as the best model.
The Statue of Liberty is a symbol of freedom and justice that is alive and well in the hearts of all Americans.
BooST Joint
Constraint
ant, cat, telephone (from CSK-PN)
Lex
The ant was cating the phone as if it were a delicious snack.
Davinci-003 BooST Joint
The ant was seen cating a telephone.
An ant cating dead fly on the telephone. A black ant eating on the side of brown telephone.
BoOST CS
Table 5: Example generations by different systems. Full outputs of all compared models can be found in Table 7 in the Appendix.
less preferred than the few-shot setting where high- quality in-context examples are carefully selected.
5.3 Case Study
We show three example generations by our systems and the baselines in Table 5 to further understand the advantage of BOOST. In the first example, the baselines connect different constraints logically, but in a less plausible way (e.g., all concepts are bonded to the same object). Our system on the other hand describes a scene where people play games while dogs walk around. In the second and third example, we all know that the Statue of Lib- erty is not alive and telephone is inedible. Instead of directly adding negations, tends to provide more contexts to make its output reasonable. In contrast, other baselines wrongly acknowledge that the Statue of Liberty can be alive or the ant can cat a telephone.
5.2 Inter-Group Results
The inter-group evaluation results are shown in Ta- ble 4. Our model BooST outperforms all baselines, including Davinci-003. We leave the comparison with ChatGPT in $6 as a separate discussion.
observe BoOST
Surprisingly, although human written refer- ences are still the most commonsensicall they are less preferred by our annotators compared with Alpaca/BoosT generations. Upon further inspec- tion, we find out that the gold references in Com- monGen are relatively short and flat (e.g., "The car drove through the snow.") which may also ex- plain why Alpaca warmed up on CommonGen are
6 Has ChatGPT solved this task?
Pair-wise comparison with BOOST. According to Table 4, our BOOST model may not surpass

---

Winning System
BOoST CS
Same
ChatGPT
GPT's language model, too. Unfortunately, due API limitations, direct verification remains infeasi- ble as we do not have access to its output logits.
CS
30%
17%
53%
Overall
47%
25%
28%
Table 6: Which system has better commonsense (CS) and overall human preference? Pair-wise comparison between BooST and ChatGPT shows that our model earns more overall pick while the ChatGPT have higher commonsense.
Related Works
Controllable Generation with Frozen PTLMs
There are two major lines: modifying the decoding algorithm and guiding PTLMs with auxiliary mod- els. Recently, Lu et al. (2021, 2022) propose neu- rologic decoding with look ahead heuristic, and (Qin et al., 2022) propose mergy-baseee constrained decoding. One drawback of this line that the in- ference slow due the large search space. In the other line, Dathathri et al.; Krause et al. (2021); Yang and Klein (2021) guide the generation pro- cess with an auxiliary model in a plug-and-play fashion by leveraging statistical principles such as the Bayesian rule. Meng et al. (2022) propose to solve the distributional discrepancy of training data and PTLM's generated tokens by training with data directly sampled from the base model. However, mistakes in commonsense are neglected when pre- vious works formulate the whole task as a lexical constrained generation game.
ChatGPT in terms of commonsense, but it excels in overall preference. On the CSK-PN eval set where the gap between our model and ChatGPT is larger, we randomly select 100 pairs of outputs and conduct pairwise comparison monsense and overall preference. Results can be found in Table 6. Specifically, each pair is first ran- domly shuffled and then annotated by least two annotators. If the two annotators disagree, third annotator is introduced for the final judge. They can also provide an optionally justification for their choice, which can earn them small bonus.
both com-
Analysis of human's feedback reveal that Chat- GPT tends to generate sentence with highly com- mon scenarios (e.g., "It not advisable wear sunglasses at night as can impede your vision and increase the risk of accidents.") making the raters less interested. On the other hand, our model tends to provide more creative context (e.g., "Some- one wears sunglasses at night to avoid the bright lights of the approaching car.", earning human an- notators' overall preference without sacrificing the commonsense too much. As one annotator com- mented, "I am fed up with those sentence with the so-called better commonsense because they are unimpressive". Such tendency of ChatGPT re- sults in a higher commonsense rating yet noticeably lower overall preference. In short, we highlight that ChatGPT has not entirely solved the task.
Commonsense Metries Zhou et al. (2022) mea- sures the commonsense of dialogue turns by hard and soft matching the relations across each turn to ConceptNet. ACCENT (Ghazarian al., 2023) propose unsupervised metric measure the event commonsense of dialogue responses via the ATOMIC knowledge graph (Hwang et al.. 2020). Our commonsense oracle is inspired by ACCENT but we primarily focused on factoid commonsense in a constrained generation setting. A concurrent work of ours is Vera (Liu et al., 2023), a super- vised model that learns to estimate the plausibility of statements. On the other hand, metric is unsupervised and neuro-symbolic, thus more inter- pretable.
The (so far) impossible fair comparison. Last, we would like to list two points regarding why eval- uating ChatGPT and our model may not be a fair comparison: (/) Test Data Contamination: Chat- GPT, which is trained on data up to 2021, likely have been trained on both datasets we tested on, including the test set. (2) Size and Trick Differ- ences: Diffcrent from BOOST, ChatGPT is more than plain language model and benefits largely from RLHF and many engineering tricks unknown to the public. It is also much larger than our largest PTLM, which is alpaca-7b. Nonetheless, our ap- proach is technically complementary with Chat-
Conclusion
We present BOOST, a framework to boost the com- monsense in PLTMs' generation by training an aux- iliary model with commonsense scorer as the ora- cle. Our O-Scorer is task-agnostic and reference- free, meaning that it is generalizable to many down- stream tasks such as dialogue and open-ended text generation. For such application, one may need replace the vanilla PTLMs with task-specific mod- cls and then train the NADO head. The O-Scorer can also be combined with task-specific guidance.

---

Acknowledgement
Ji. Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingua,, multimodal evaluation of chatgpt reasoning. hallucination, and interactivity. arXiv preprint arXiv: 040034433.
We thank PlusLab members from UCLA and the anonymous EMNLP reviewers for their construc- tive feedback and suggestions that helped to im- prove the paper.
Antoine Bosselut, Hamnah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762 -779,, Flo- rence, Italy, Association for Computational Linguis- tics.
Limitations
We discuss the limitations of our work. First, our tuple extractor covers only four relation types and can míss many other important relation types such causal, temporal order, etc. These later rela- tion types are more sophisticated such that LLMs are strong as gpt-3.5-turbo will fail at (Gao al.. 2023; Yuan et al., 2023; Chan et al., 2023; Bang et al., 2023). Second, we find out that the co- sine similarities of sentence embeddings used in Eq. 1 to compute the compatibility scores some- times do not align with human judgement. The errors incurred during the generative scoring pro- cess is then propagated into the training process of NADO, which negatively affect the output's qual- ity. Last, although the auxiliary models have much smaller size than the PTLMs, the number of sam- ples needed to train R still large in order to guarantee a good approximation of the closed form solution derived in Eq. 4.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voon Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, 004/2005.14165
Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang. Tianqing Fang, Xin Liu, and Yangqiu Song. 2023. Chatgpt evaluation on sentence level relations: A focus temporal, causal, and discourse relations. arXiv preprint arXiv:2304. 14827.
Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng. Lei Li, and Yanghua Xiao. 2023. Say what you mean! large language models speak too positively about negative commonsense knowledge. ACL 2023.
Ethics Statement
It is known that the generated results by PTLMs could capture the bias reflected in the training data (Sheng et al., 2019; Wallace et al., 2019). Our model BoOST is build upon PTLMs including T5 (Raffel et al., 2020), GPT-2 (Radford et al., 2019), and Alpaca (Taori et al., 2023), which may poten- tially generate offensive content for certain groups or individuals. We suggest to carefully examine the potential biases before deploying the models to real-world applications.
Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling esstrcction-inetuned language models. arXiv preprint arXiv:2210.11 1416.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In ternational Conference on Learning Representations.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 33 Berlin, Germany. Association for Computa- tional Linguistics.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL. Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarizatio,, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoo Lee, Andrew S. Rosen, Gerbrand Ceder. Kristin Persson, and Anubhav Jain. 2022. Structured information extraction from complex scientific text with fine-tuned large language models.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schütze,

---

and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- rions of the Association for Computational Linguis- tics, 9:1012-1031.
Ye Liu, Yao Wan, Lifang He. Hao Peng, and Philip S. Yu. 2021. Proceedings of the aaal conference on artificial intelligence.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lian- hui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2022. NeuroLogic a a decoding: Constrained text generation with lookahead heuris- tics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 780-799, Seattle, United States. Associa- tion for Computational Linguistics.
Jinglong Gao, Xiao Ding. Bing Qin, and Ting Liu. 2023. Is chatgpt a good causal reasoner? a comprehensive evaluation. arXiv preprint arXiv:2305.07375.
Sarik Ghazarian, Yijia Shao, Rujun Han, Aram Gal- styan, and Nanyun Peng. 2023. Accent: An au- tomatic event commonsense evaluation metric for open-domain dialogue systems. Proceedings of the 2023 Association for Computational Linguistics.
Xingwei He, Yeyun Gong, A-Long Jin, Weizhen Qi, Hang Zhang, Jian Jiao, Bartuer Zhou, Biao Cheng, Sm Yiu, and Nan Duan. 2022. Metric-guided distilla- tion: Distilling knowledge from the metric to ranker and retriever for generative commonsense reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 839- 852, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro- Logic decoding: (un)supervised neural text genera- tion with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Compulational Linguistics: Human Language Technologies, pages 4288-4299, Online. Association for Computational Linguistics.
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought in large language models: a cognitive perspective.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. 2021. Lora; Low-rank adaptation of large lan- guage models. In International Conference on Learn ing Representations.
Jena D. Hwang. Chandra Bhagavatula, Ronan Le Bras, Jeff Da. Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. AAA! Conference Artificial Intelligence.
Gary Marcus. 2020. The next decade in Al: four steps towards robust artificial intelligence. CoRR, abs/2002.06177.
Tao Meng, Sidi Lu, Nanyun Peng. and Kai- Wei Chang. 2022. Controllable text generation with neurally- decomposed oracle. In Advances in Neural Informa- tion Processing Systems.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shatiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. Gedi: Genera- tive discriminator guided sequence generation. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4922-
OpenAL. 2022. Introducing chatgpt.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin. Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoye.. 2020. BART: Denoising for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online. Association for Computa- tional Linguisties.
seeeence-t--sqqeeeee pre-training
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. 2022. Cold decoding: Energy-basee constrained text generation with langevin dynamics. NeurlPS 2022.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text gen- eration challenge for generative commonsense rea- soning. In Findings of the Association for Computa- rional Linguistics: EMNLP 2020, pages 1823-1840, Online. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazcer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, and Hanna Hajishirzi. 2023, Vera: seneral-purposs plausibility estimation model for commonsense statements. ArXiv, abs/2305.03666

---

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert, Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Appendix
Few-Shot GPT-3.5 Tuple Extractor Prompt
The prompt used to query gpt-3.5-turbo is dis- played in Figure 4. Recall that need the LLM to accurately extract both sensical tuples (e.g.. girl Capable Of blowing candles) and nonsensical tuples (e-g. horse Capable Of riding bikes) from the input sentence. Hence, not all sentences in the prompt are reasonable.
Emily Sheng. Kai Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as babysitter: On biases in language generation. arXiv preprint arXivv1909.0133:
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings the Thirty-Firs First AAAI Conference on Artificial Intelligence, 17
page 4444-4451. AAAI Press.
B Full Results of Case Study
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B, Hashimoto. 2023. Stanford alpaca: An instruction- foslloition-followingg llama model. https:// gi thub. com/ ttts/tttsuuuuae http:/stanfddd_aaaa
The full results of the case study with outputs of all compared systems can be found in Table
C Human Evaluation Questionnaire
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971.
Figure 5 and Figure 6 are screenshots of the ques- tionnaire used in the human evaluation.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial trig- gers for attacking and analyzing nlp. arXiv preprint 1eXiv:1908.00 0715..
Kevin Yang and Dan Klein. 2021. Fudge: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 351 1-3155
Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023. Zero-shot temporal relation extraction with chatgpt. arXiv preprini S.Xv:23340005555
Honghua Zhang, Meihua Dang, Nanyun Peng. and Guy Van den Broeck. 2023. Tractable control for autore- gressive language generation. The Fortieth Interna- tional Conference Machine Learning.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675.
Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu. and Dilek Hakkani-Tür. 2022. Think before you speak: Explicitly generating implicit commonsense knowledge for response generation. In ACL 2022.

---

Constraint
table, dog, game, walk, fireplace (from Com- monGen)
Gelato
A dog is playing fireplace.
on table next to
A group of people are walking and playing video games their dining room with fire- places, tables, and degs.
A Decoding
The dog walked around the table playing gume by the fireplace.
Davinci-003
Lex
dog is walking around the fireplace and playing game on the table.
Alpaca
The dog walks around the table while playing game by the fireplace.
BooST CS
Two dogs playing game of Frisbee by the fireplace next to the table.
The dog walked around the table while played a game hy the fireplace. the table the fireplace
BooST Joint
ChatGPT
dog played after walk.
dog plays game of walking from the table the fireplace.
Reference
Constraint
statue, liberty, alive (from CSK-PN)
A Decoding
There are still some people who want to see statues of liberty as living creatures.
Duvinci-003
The Statue of Liberty stands alive and proud. The statue of Liberty is alive and stands proudly in New York City.
Lex
The Statue of Liherty became alive a bright and sunny day.
Alpaca
The statue of liberty stands alone symbol of liberty and awakening alive. The Statue of Liberty is symbol of freedom
BooST CS
BooST Joint
and justice that alive and well in the hearts of all Americans.
ChatGPT
The Statue Liberty looked alive in the glow- ing sunset.
Constraint
ant, eat, telephone (from CSK-PN)
A Decoding A man is feeding ants to an antennac on top of his head, so they can be cating from the telephone.
Davinci-003
The ant was seen eating telephone.
Lex
The ant was eating the phone as if were delicious snack.
Alpaca
The ant ate the telephone. black ant eating on the side of brown telephone.
BOOST CS
BooST Joint
An ant cating dead fly the telephone. black ant eating the side of brown telephone.
BooST CS
ChatGPT
The ant tried the speaker miniature telephone.
Table 7: Full results of case study by different systems.

---

<-- Instruction:
Extract tuples (A, B) from the sentence for the relations based on the description below. Do not infer anything. Only extract tplles that explicitly mentioned in the sentence. Put None if there are tuples extract.
Isus (an object) is used to do B (a goal). AtLocation: A at the location larger area B. CapableOf: living) is capable of doing B (an event)
PartOf: A part of B.
Examples: -->
The runner because wanted win the car
IsUsedFor: None
AtLocation: None
CapableOr: (runner, run), (runner, win the car race)
PartOf: None
The small plates add dimension and depth
baked zucchinis carrots.
IsUsedFor: (small plates, adding dimension to this dish)
oocation None
CapableOF: None
ParOF: (baked zucchini, dish), ( (carrot, dish)
The grinning boy put his foot into the sock to dress himself..
IsUsedFor: None
AtLocation: (foot, sock)
CapableOf: (hoy. grin), (grinning boy, put his foot into the sock)
PartOf: (foot, grinning boy)
The judges give high scores to the woman wearing long dress who sings beautifully into microphone on the stage.
IsUsedFor: (microphone, singing)
AtLocation: (woman, stage), (microphone, stage), (Jong dress, stage) ) CapableOf: (judges, give high scores), (woman, wear a long dress), (woman, sing beautifully
PartOf: None
A man is kicking soccer ball with his head.
IsUsedFor: None
AtLocation: None
CapableOf: (man, kick snccer ball with his hcad)
PartOf: (head, man)
I used chisel and hammer
piece of wood.
IsUsedFor: chisel, carving piece wood), (hammer, carving piece of wood)
AtLocation:
None
CapableOf: (I,
piece of wood), (I, chisel hammer)
PartOf: None
Spewing volcano with waterfalls flowing results in an idyllic uncontaminated environment summer the mountains.
IsUsedFor: None
AtLocation: (volcano, mountains), waterfalls, mountains)
CapableOf: (volcano, spew), waterfall flow)
PartOf: waterfall, spewing volcano). idyllic uncontaminated environment, mountains)
A argues with stewards after being to leave the pitch.
IsUsddorr: None
AtLocation: (fan, pitch), (stewards, pitch)
CapableOr: (fan, argue with stewards), stewards, tell fans to leave pitch)
PartOf: None
The soldier is driving the smiling tank across the bridge to save people.
IsUsedFor: (tank, driving)
AtLocation: soldier. across the bridge), (tank, across the bridge)
CapableOf: soldier, drive the tank), (tank, smile)
PanOf: None
Figure 4:
Few-Shot prompt used to query gpt-3.5-turte extract tuples from sentence. We purposefully select few non-sensical sentences in the prompt.

---

Survey Instructions (Click expand)
commonsense annotation, you given sentences. sentence aligns with your commonsense knowledge the workd).
'commonsensical" (namely,
define
knowledge' following:
Commonawnsse knowledge includess the basic facta about evantss (inluting actiooss and thiir affects, laccta about knowledde and how it déstiined, facts about balieta and also
objects thoir propertios.
each sentence
4 means the most 'commoneensicat; Helly you cannot find any issues that hurts cemmonsensee
mostly "commonsensical but
has minor mistakes; you probably
miror weird, would sentence overall still
commorsvrsst
2 means somewhat "commonsensical with noticable mistakes too ganeral fua, would be helpffll to add more context)) for ecampil,, abmoot half the sentancee you Indiostes the least commonsensioal. ; for example, ic, most the contence doosn't 't maki too much sense to
doean't
When you do not choose 4, you will be asked Indicate whilhh word phrase hurting the commonsense. Please focus on the commonsesonse part and Ignore the exist. After that.
minor grammatical mistakes
rate like the sentencee overall, Now let's take look an example taok that you are waked to complete.
consider other
Example
Sentence: Dog palying with trisbee outdoorrs
How commonsenskall this sontence?
commonsensical
somewhat
mistakes)
mostly (can strange)
very commonsensical
Which part of this centence not making sense: N/A
aigns we would like to give a scare of
dege play dogs go green grase. Although
Example
leughed hard, couldn't keep my foot eticking my face.
commonsensicall this sontence?
1. least commonsersical
2, somewhat (but with noticable mistakes)
mostly (can feel bit strange)
very commonsensical
Which part of this sentence not making sense: foot sticking face
Exp ianaaiionn on: This sontence strange because people don't always keep thair foot sticknng to their facos, Bazod on your personal toolings, we consider both scoras of or ressonable.
Example
supennarket the employee watchhdd as the customer prepared their food to perfection.
commonnennical sentence??
least
somewhet noticable mistakes)
feel strange)
commonsensical
part sentence making sense: customer prepared
Exp lanationn This sentence not wall
superaaketss, should food for However,
Figure 5: The instructions for human evaluation (page 1).

---

least commonsensical
2. somewhat (but with notioable mistakes)
S. mostl (can feel a bit strange)
4. very commonsensical
Which sentence
prepared
Explanation: This sentenca doss not align well with our commonsennia bacause in superma kuts, emplayees should parpare the food for the customar. However, we do not rale least' commonsensical is somewhat reasonable that because customer and
concepts related suporaakeets.
Example
A man holding tree and sitting on
How commonsensiall santonce?
neensiall
somewhat noticable mistakes)
mostl (can bit
commonsensiicl
Which
Explanation: First, treos grow from the ground and largo, so that humans cannot hold them. Second, it not possible hold objest while sittngg the same object
you the you are anoouraged consider aspects, including but not liitted to ceherence, funniness, creativeness, grommar, and the commonsensical score you provicusty rated.
Hint: An ufficiant boost HIT's approval rata
1) Reed the sentiences carefilly. HIT consists groupe owntencess Each group of tences talkngg about the some topic. We kindly ask you to compare d dffrrennt sontences.
2) have taken meesures prevent theesting and you do not com complete taok honeetly will know and the HIT will rejected:
carefully
thoughts
Sentence $(sentence_
1. least commonsensical
2. somewhat (but with noticable mistakes)
O 3. mostly (con feel bitt strange)
4. very commonsensical
Which this sentence is making
you
1. little
2. somewhat
3. good
best
2. Sentence $(sentence_2 __
cammonsensical
least commonsensiical
somewhat with noticable mistakes))
mostly feel commonsensical
part making
much you santence
O somewhat
good
Figure 6: instructions for human evaluation (page 2).


Reviews:
[{"review_id": "YlvtxPcQ27", "reviewer": "Reviewer_Kqgt", "summary": "", "questions": "1. how to train the additional NADO head?\n2. since O-score is a reference-free evaluator, then what does O(x,y) in Eq.(2) mean?\n3. what does y_i mean in Eq.(3)?\n4. In Table 3, on the CommonGen, BOOST CS achieves the majority of SOTA according to O-score and Human CS & OVERALL metrics, but its BLEU4 score only performs better than base model, how to explain the phenomenon?\n5. In Table 5, why are the baseline models listed in the three examples inconsistent?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your valuable review! We’ve noticed that there might have been some misunderstandings, and we believe that most of your questions and clarity concerns are to be adequately addressed in our forthcoming rebuttal. We sincerely hope for your re-evaluation of our merits and strengths based on the responses. \n\nWe carefully address your concerns and questions as follows:\n\n- **[Q1]. how to train the additional NADO head?**\n\nYou can find the details of the training method (input, output, notation, formulation, loss, etc.) in Section 2.3.2 line 256-284. You can also find the training data which is generated by self-sampling  in Section 4.1 line 372-382, and auxiliary model architecture in Section 4.2 line 418-427. \n\nPlease kindly let us know if you have more specific questions. If so, we are more than happy to address them in detail during the author-reviewer discussion period.\n\n- **[Q2]. since O-score is a reference-free evaluator, then what does O(x,y) in Eq.(2) mean?**\n\nIn Eq 2 and the entire paper, the $x$ is an input constraint (e.g., ‘lasso, horse, cow’) and $y$ is a  ``machine-generated sentence`` that follows the input constraints (e.g., A horse is being lassoed by a cow.), and O(x,y) means the score assigned to the input-output pair (x, y). Namely, **_O-score is indeed reference-free_** because it does not require any ground-truth sentences to judge the commonsense. If we only care about the commonsense but not lexical constraint, O(x,y) is reduced to O(y). We keep using O(x,y) in Eq.(2) because in our experiments, we also combine the lexical checking Boolean function Lex(x,y) with the O-score.\n\nYou may find the official notations (e.g., meaning of x and y) in Section 2.1 line 110-124. \n\n- **[Q3]. what does y_i mean in Eq.(3)?**\n\nThis is a typo, and it should be $y_{T'}$ which means $y_{=T'}$. In Eq.(3) we haven’t introduced what $i$ is.  \nWe sincerely apologize for such a typo in an important equation and _**thank you so much**_ for spotting it/pointing it out!! \n\n- **[Q4]. In Table 3, on the CommonGen, BOOST CS achieves the majority of SOTA according to O-score and Human CS & OVERALL metrics, but its BLEU4 score only performs better than base model, how to explain the phenomenon?**\n\nThank you for your insightful question! We do have a few hypotheses to explain this phenomenon which we unfortunately weren't able to touch upon in the submission due to the page limit. The short answer is that ``BLEU-4, which was originally introduced for machine translation, is not a good automatic metric for our task focusing on commonsense and overall text quality (such as being meaningful and informative)``. \n\n**First**, for example, you can see in Table 3 that the keyword Coverage ratio has an exceptionally _**high correlation**_ with BLEU-4 (``0.914`` measured by Pearson Correlation and ``0.785`` measured by Spearman Correlation). This indicates that in our constrained generation task, BLEU-4 is biased towards favoring machine outputs that merely cover all the input concepts, ignoring aspects of commonsense. **On the other hand**, we also find the references in the CommonGen dataset mediocre. As is stated in Line 509-513, we find out that the references in CommonGen are relatively short and flat (e.g., The car drove through the snow.), which may also explain why Alpaca warmed up on CommonGen are less preferred by human annotators than few-shot Alpaca. In fact, BLEU-4 has close to zero correlation with human judgment on commonsense (``-0.08`` measured by Pearson; ``0.11`` measured by Spearman) and overall quality (``0.04`` measured by Pearson; ``-0.05`` measured by Spearman). \n\nBased on the above, we hypothesize that BLEU-4, or any other metric that measures the superficial lexical overlap with ground truth, cannot differentiate high-quality and commonsensical outputs in our settings.\n\n- **[Q5]. In Table 5, why are the baseline models listed in the three examples inconsistent?**\n\nApologies for the confusion. ``Due to page limit, we select the better/more informative baselines for each case, and that’s why they are different``. Namely, we have to be selective only to fit in the page limit. An exception goes to Gelato (and references) as we do not have the model’s output (or ground truth) on the CSK-PN eval set.\n\nIn case you are curious, we compile the full results below:\n| **Constraint**    | **table, dog, game, walk, fireplace (from CommonGen)** |\n| -------- | ------- |\n| Gelato  | A dog is playing a game on a table next to a fireplace.    |\n| A* Decoding | A group of people are walking and playing video games at their dining room with fireplaces, tables, and dogs.     |\n| Lex    | A dog is walking around the fireplace and playing a game on the table.   |\n| Base Model | The dog walks around the table while playing a game by the fireplace.|\n| BOOST CS| Two dogs are playing a game of Frisbee by the fireplace next to the table.|\n| BOOST Joint| The dog walked around the table while we played a game by the fireplace.|\n| Davinci-003 | The dog walked around the table playing a game by the fireplace.|\n| Reference| The dog plays the game of walking from the table to the fireplace.|\n|ChatGPT| A dog played near the table by the fireplace after a walk.|\n\n\n\n| **Constraint**    | **statue, liberty, alive (from CSK-PN)**|\n| -------- | ------- |\n| A* Decoding | There are still some people who want to see statues of liberty as living creatures.   |\n| Lex    | The statue of Liberty is alive and stands proudly in New York City.  |\n| Base Model | The Statue of Liberty became alive on a bright and sunny day.|\n| BOOST CS|  The statue of liberty stands alone as a symbol of liberty and awakening alive.|\n| BOOST Joint| The Statue of Liberty is a symbol of freedom and justice that is alive and well in the hearts of all Americans.|\n| Davinci-003 | The Statue of Liberty stands alive and proud.|\n| ChatGPT| The Statue of Liberty looked alive in the glowing sunset.|\n\n\n| **Constraint**    | **ant, eat, telephone (from CSK-PN)**|\n| -------- | ------- |\n| A* Decoding |   A man is feeding ants to an antennae on top of his head, so they can be eating from the telephone.|\n| Lex    | The ant was eating the phone as if it were a delicious snack.  |\n| Base Model | The ant ate the telephone.|\n| BOOST CS|  A black ant eating on the side of a brown telephone.|\n| BOOST Joint| An ant eating a dead fly on the telephone.|\n| Davinci-003 | The ant was seen eating a telephone.|\n| ChatGPT| The ant tried to eat the speaker of a miniature telephone.|\n\n\n\n\n- **6. Reproducibility:** We will open-source all codes upon acceptance.\n\n\nIn conclusion, we sincerely hope that the above response helps to clarify your questions and concerns. With the inclusion of an extra page upon acceptance, we shall be able to incorporate more details to adequately address all your comments. In the camera ready version, we also intend to present the methodology section (especially Section 2.3 talking about NADO) and the experimental results more comprehensively. _**Finally, we hope you agree that these adjustments are fixable by revisions of writings instead of another round of submission. We sincerely hope you could kindly reconsider our merits and contributions.**_ Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new evaluation method O-score is proposed to check the commonsense for the generated text.\n2. Based on NADO, a concept-to-sentence model is proposed to improve the commonsense by considering the O-score.", "reasons_to_reject": "1. Some details are not clear enough, especially the section 2.3.\n2. The experimental results are not convicing.", "questions_to_authors": "1. how to train the additional NADO head?\n2. since O-score is a reference-free evaluator, then what does O(x,y) in Eq.(2) mean?\n3. what does y_i mean in Eq.(3)?\n4. In Table 3, on the CommonGen, BOOST CS achieves the majority of SOTA according to O-score and Human CS & OVERALL metrics, but its BLEU4 score only performs better than base model, how to explain the phenomenon?\n5. In Table 5, why are the baseline models listed in the three examples inconsistent?", "paper_topic_and_main_contributions": "To generate more commonsensial text based on large language model (LLM), this paper proposes a reference-free evaluation method O-score for checking the commonsense knowledge in the generated text, and utilizes the O-score to guide the controllable generation method NADO by training a neural network-based model to approximate the O-score. Experiments are conducted on two constrained concept-to-sentence benchmarks. The contribution can be summarized:\n1. this paper proposes a reference-free evaluator to assess how commonsensial a setence is.\n2. this paper extends a controllable generation method to improve commonsense for pretrained language model.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 216, \"PTL)\" -> \"PTLM\""}, "extra_text": {}}, {"review_id": "jnsuaO6DnG", "reviewer": "Reviewer_DEwH", "summary": "", "questions": "Question A: Lack of some instruction-tuned baselines for comparison (e.g., text-davinci-003 and flan-t5), instead of instruction tuning again on gpt2 model. text-davinci-003 api (as well as flan-t5) can return the output distribution and fulfill the requirement stated in line 400. \n\nQuestion B: ChatGPT whether solving this task remains a question and hasn’t been included in this paper. Try to include this evaluation to validate the ability of ChatGPT to generates a plausible sentence given a list of concepts as input and show the gap between ChatGPT and the suggested method in this paper.\n\nQuestion C:  Whether these generated commonsense outputs can be applied to benefit any downstream tasks to emphasize the significance of this task (such as the open-domain dialogue task).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you so much for your valuable review and constructive feedback! We address your questions and comments as follows:\n\n**[Question A]. Lack of some instruction-tuned baselines for comparison (e.g., text-davinci-003 and flan-t5), instead of instruction tuning again on gpt2 model. text-davinci-003 api (as well as flan-t5) can return the output distribution and fulfill the requirement stated in line 400.**\n\nThank you for your constructive suggestion! We agree it would be nice to test our approach on instruction-tuned models, too. To our knowledge, however, we couldn't access the output logits of text-davinci-003 through the OpenAI API; we only have access to the top 5 tokens per step. Hence, we quickly implemented our approach with Flan-T5. Under this setting, the NADO head is a 4-layer t5 decoder that takes in the input constraints as hidden steps from a fixed, pretrained encoder. Due to time constraints, we are only able to test our approach on zero-shot Flan-T5-large (780M), and conduct automatic evaluation.\n\nPlease find the automatic evaluation results in the following table.\n\n| Test   Data       | CommonGen       |          |       | CSK-PN          |          |\n|-------------------|-----------------|----------|-------|-----------------|----------|\n|       Metric      | CS Score - Mean | Coverage | BLEU4 | CS Score - Mean | Coverage |\n| flan_t5 zero-shot |                 |          |       |                 |          |\n|    Base Model    |      0.571      |   84.6%  |  17.5 |      0.555      |   80.7%  |\n|        Lex        |      0.577      |   93.7%  |   26.0  |      0.569      |   89.6%  |\n|    Boost CS   |      0.619      |   91.3%  |  21.6 |      0.613      |   88.9%  |\n|   Boost Joint  |      0.606      |   93.1%  |  25.6 |      0.601      |   89.6%  |\n\nTable 1: Automatic eval results. We implement the same algorithm on Flan-T5.\n\n**Analysis**: Comparing the above results with gpt2-large with warm up (774M, in Table 3 in the original paper), we see that our approach is actually more effective on instruction-tuned models than similarly-sized decoder only models.   \n\n---\n\\\n**[Question B]. ChatGPT whether solving this task remains a question and hasn’t been included in this paper. Try to include this evaluation to validate the ability of ChatGPT to generates a plausible sentence given a list of concepts as input and show the gap between ChatGPT and the suggested method in this paper.**\n\nFirst, in response to including the comparison with ChatGPT, we'd like to address several points regarding why evaluating ChatGPT and our model may not be a fair comparison:\n\n- **Test Data Inclusion**: ChatGPT, which is trained on data up to 2021, likely have been trained on both datasets we tested on (CommonGen is published in 2020 while CSK-PN is a subset of ConceptNet which is published in 2017), including the test set. \n\n- **Size and Trick Differences**: Different from ours, ChatGPT is more than a plain language model and benefits largely from RLHF and many unknown engineering tricks. It is also much larger than our largest PTLM, which is alpaca-7b. \n\n**Experimental Results**: Upon request, we still obtained the results with ChatGPT below. Following previous decoding settings, we use sampling with a temperature T=0.7.\n- **Result 1: Human evaluation results across multiple baselines.**\n\n|Human Eval| CommonGen - CS |CommonGen - Overall | CSK-PN - CS |  CSK-PN - Overall|\n| -------- | ------- |  ------- |------- |  ------- |\n|A*esque| 3.07 |2.81| 3.09 |3.04|\n|GeLaTo| 3.15| 2.78 |/ |/|\n|BOOST gpt2| 3.27| 2.95| 3.24| 3.00|\n| -------- | ------- |  ------- |------- |  ------- |\n|Alpaca warm up| 3.27| 3.05| 3.21 |3.01|\n|Alpaca few-shot| 3.32 |[3.20]| 3.20| 3.10|\n|BOOST Alpaca warm up| [3.40] |3.17| **3.41**| [3.16]|\n|BOOST Alpaca few-shot| **3.44** |**3.28**| [3.38] |**3.18**|\n| -------- | ------- |  ------- |------- |  ------- |\n|Text-Davinci-003| 3.33 |3.19| 3.33| 3.10|\n|ChatGPT| 3.46 |3.09| **3.49**| 2.95|\n|Human |**3.49** |2.99| /| /|\n\nTable 2: Evaluation results with ChatGPT. We use boldface to denote the best scores within each group, and brackets [] to denote the second best.\n\n- **Result 2: Pairwise comparison**\n\nFurthermore, on the CSK-PN eval set where the gap between our model and ChatGPT is larger, we randomly select 100 samples and then conduct pairwise comparison on both commonsense and overall quality. Specifically, each pair is first randomly shuffled and then annotated by at least two Turkers. If the two annotators disagree, a third person is introduced for the final judge.\n\n\n**(a) Which system has better commonsense?**\n\n|Boost CS|Same |ChatGPT|\n| -------- | ------- |  ------- |\n|30%|17%|53%|\n\nTable 3 (a). Pairwise comparison between Boost CS and ChatGPT about commonsense.\n\n\n**(b) Which system has better overall quality?**\n\n|Boost CS|Same |ChatGPT|\n| -------- | ------- |  ------- |\n|47%|25%|28%|\n\nTable 3 (b). Pairwise comparison between Boost CS and ChatGPT about overall quality.\n\nFrom both Table 2 and Table 3, we see that our model BOOST has better overall quality than ChatGPT, but slightly worse in commonsense. Upon further inspection, we notice that on the CSK-PN eval set that contains concepts linked with negated commonsense relations (e.g.``<wear sunglasses, at night>``), ChatGPT tends to generate a short sentence with negation (e.g., ``It is ridiculous to wear sunglasses at night.``). Such tendency of ChatGPT results in a higher commonsense rating yet noticeably lower overall preference. On the other hand, our model tends to provide more context to make its output reasonable (e.g., ``Someone wears sunglasses at night to avoid the bright lights of the approaching car.``), earning human annotators’ overall preferences without sacrificing the commonsense too much.\n\n**Conclusion**: In short, it's crucial to highlight that ChatGPT hasn't entirely \"solved\" the problem, and has much room for improvement. Last, we want to emphasize that ``our approach can be complementary with ChatGPT’s language model, too``. Unfortunately, due to API limitations, direct verification remains infeasible as we do not have access to its output logits.\n\n---\n\\\n**[Question C]. Whether these generated commonsense outputs can be applied to benefit any downstream tasks to emphasize the significance of this task (such as the open-domain dialogue task).**\n\nYes! Our O-Scorer is task-agnostic and reference-free, meaning that our framework is generalizable to many downstream tasks including but not limited to dialogue and open-ended text generation. For such application, we just need to replace the vanilla PTLMs with task-specific models and then train the NADO head. In addition, the O-Scorer can also be combined with task-specific guidance such as the lexical checking function in our task. We shall add related discussions into our paper to emphasize the significance of this work. \n\n**Missing reference**: Thank you, we shall add these to our paper! \n\nWith the above, we sincerely hope you could kindly reconsider our merits and experimental soundness. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper conducts an interesting task by generating a plausible sentence given a list of concepts as input.\n* The performance of this proposed method outperforms baselines\n* This paper proposes a novel reference-free commonsense metric to assess how commonsensical a sentence is.", "reasons_to_reject": "* Lack of some instruction-tuned baselines for comparison (e.g., text-davinci-003 and flan-t5), instead of instruction tuning again on gpt2 model. \n* Moreover, ChatGPT whether solving this task, which generates a plausible sentence given a list of concepts as input, remains a question and hasn’t been included in this paper.\n* Whether these generated commonsense outputs can be applied to benefit any downstream tasks to emphasize the significance of this task (such as the open-domain dialogue task stated in [1])\n\n[1] Sarik Ghazarian, Yijia Shao, Rujun Han, Aram Galstyan, and Nanyun Peng. 2023. Accent: An automatic event commonsense evaluation metric for open-domain dialogue systems.", "questions_to_authors": "Question A: Lack of some instruction-tuned baselines for comparison (e.g., text-davinci-003 and flan-t5), instead of instruction tuning again on gpt2 model. text-davinci-003 api (as well as flan-t5) can return the output distribution and fulfill the requirement stated in line 400. \n\nQuestion B: ChatGPT whether solving this task remains a question and hasn’t been included in this paper. Try to include this evaluation to validate the ability of ChatGPT to generates a plausible sentence given a list of concepts as input and show the gap between ChatGPT and the suggested method in this paper.\n\nQuestion C:  Whether these generated commonsense outputs can be applied to benefit any downstream tasks to emphasize the significance of this task (such as the open-domain dialogue task).", "paper_topic_and_main_contributions": "This paper proposes a reference-free commonsense metric to assess how commonsensical a sentence is, first extracting tuples from a sentence and then grounding the extracted tuples to a dynamic CSKB (i.e., COMET). Moreover, the paper extends a controllable generation approach to improve commonsense for black-box PTLM by presenting a framework to boost the commonsense in PLTMs’ generation through training an auxiliary model with a commonsense scorer as the oracle.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {"Missing_References": "There are some missing references in lines 578-582 to evidence the statements.\n\nGao, Jinglong, et al. \"Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation.\" arXiv preprint arXiv:2305.07375 (2023).\n\nYuan, Chenhan, Qianqian Xie, and Sophia Ananiadou. \"Zero-shot temporal relation extraction with chatgpt.\" arXiv preprint arXiv:2304.05454 (2023).\n\nChan, Chunkit, et al. \"Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations.\" arXiv preprint arXiv:2304.14827 (2023).\n\nBang, Yejin, et al. \"A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.\" arXiv preprint arXiv:2302.04023 (2023)."}}, {"review_id": "PH0n3ci84f", "reviewer": "Reviewer_bDkz", "summary": "", "questions": "Question A: In Table 2, it is shown that the Reference-Based matrix \"BERTScore-all\" is more consistent with human rating. Besides, I guess computing the BERTScore-all will be faster than the model inference. So I wonder why you choose a model-based method (i.e., train a T5-large) to serve as the Commonsense Scorer? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you so much for your valuable review! We would like to address your questions and concerns as follows:\n\n**[QA].  In Table 2, it is shown that the Reference-Based matrix \"BERTScore-all\" is more consistent with human rating. Besides, I guess computing the BERTScore-all will be faster than the model inference. So I wonder why you choose a model-based method (i.e., train a T5-large) to serve as the Commonsense Scorer?**\n\n- It is infeasible to use BERTScore-all as the Commonsense Score exclusively, simply because of its reference-based nature. In our settings, 1) all data used to train the NADO head, which are self-sampled by PTLMs, and 2) some of the eval data (i.e., CSK-PN dataset) do not come with a reference. In light of these, we have to train a T5 model to serve as the reference-free Commonsense Scorer. After that, BERTScore-all serves as a compared system when test our newly proposed reference-free Scorer rigorously – on a small subset of eval data which is actually paired with ground truth.\n\n**[Contribution/Novelty/Significance of Work]**\n\n- First, we would like to kindly reiterate that proposing a novel framework for controllable generation is not one of our claimed contributions. Rather, we are the first to extend NADO from a simple, binary-rule based setting to a continuous, neural-evaluator based setting and show its efficacy. To this end, we also propose a new reference-free commonsense evaluator that can be used as an off-the-shelf tool. To the best of our knowledge, the idea of using automatic eval metric to guide language models towards more commonsense-compliant outputs is novel and ``hasn't been explored before``.\n\n- Moreover, commonsense-compliance is a known challenge among LMs. Many downstream fine-tuned models such as open-domain chatbots and story writing systems would largely benefit from improved commonsense abilities. Our proposed framework stands out for its model-agnostic nature. Added to the task-agnostic and reference-free nature of our O-Scorer, \t``Boost can be easily applied to benefit various downstream tasks for better commonsense-aligned generation``. \n\nWith the above, we sincerely hope you could kindly reconsider our merits. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-structed and easy to follow. The details of the proposed framework are also well illustrated.\n\n2. The experiments are comprehensive and convincing. The authors evaluate their framework on a series of GPT-2-based and Alpaca-based language models on two constrained concept-to-sentence benchmarks. They also utilize various metrics to assess the performance. ", "reasons_to_reject": "1. The method lacks novelty to some extent. BOOST is composed of two main part, the first part is to construct a commonsense scorer by grounding the sentence to a dynamic commonsense knowledge base from four different relational aspects, the second part is to use the scorer as the oracle for a method called NADO [1], which has been proposed for controllable generation. Therefore, the method can be regarded as NADO applied to controllable commonsense generation, with the oracle is constructed based on the commonsense domain knowledge. \n\n[1] Meng, T., Lu, S., Peng, N., & Chang, K. W. (2022). Controllable text generation with neurally-decomposed oracle. Advances in Neural Information Processing Systems, 35, 28125-28139.", "questions_to_authors": "Question A: In Table 2, it is shown that the Reference-Based matrix \"BERTScore-all\" is more consistent with human rating. Besides, I guess computing the BERTScore-all will be faster than the model inference. So I wonder why you choose a model-based method (i.e., train a T5-large) to serve as the Commonsense Scorer? ", "paper_topic_and_main_contributions": "This paper aims to address the generative commonsense impotence problem of Pre-Trained Language Models (PLTMs). To this end, the authors propose a framework named BOOST, which first build a scorer to evaluate how commonsensical a sentence is, and then use the score to train an auxiliary model that steers the PTLM toward more commonsensical outputs. The contributions are three-fold:\n\n1) The authors propose a reference-free scorer to evaluate how commonsensical a sentence is.\n2) The authors extend the controllable generation method called NADO to the controllable commonsense generation field.\n3) Extensive experiments validate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel approach to improve the commonsensical output of pre-trained language models (PLTMs). The research focuses on (1) the creation of a reference-free evaluator called the O-score to assess the commonsense nature of generated text and (2) guides the extension of a controllable generation method named NADO, aiming to steer PLTMs towards producing more commonsensical text.

Pros:
1. Introduces a new metric, the O-score, to evaluate the commonsensicality of generated text without requiring reference sentences.
2. Extends the NADO method to focus on controllable commonsense generation. The proposed method is model-agnostic. 
3. The paper is well-structured and comprehensive in its experimentation, and the results surpass baselines.

Cons:
1. The initial manuscript misses out on comparing with more diverse and strong LLMs such as instruction-tuned LLMs. However, I can see these experiments are added during rebuttal. I hope the author will add these new results in the final version and provide more details discussions. For example, the reply to QA of Reviewer DEwh should have more narrative to explain the results and comparisons. 
2. The paper does not clarify how the generated commonsense outputs could be useful for downstream applications, which could attest to the method's significance. The related discussion in rebuttal should be included in the final version.
Decision: Accept-Main