CASE: Commonsense-Augmented Score with an Expanded Answer Space
Authors: Wenkai Chen
Sahithya Ravi
Vered Shwartz
Venue: EMNLP 2023 Findings (2023)
Keywords: Commonsense Reasoning
Question Answering
Language Model
Zero-shot

Abstract: LLMs have demonstrated impressive zero-shot performance on NLP tasks thanks to the knowledge they acquired in their training. In multiple-choice QA tasks, the LM probabilities are used as an imperfect measure of the plausibility of each answer choice. One of the major limitations of the basic score is that it treats all words as equally important. We propose CASE, a Commonsense-Augmented Score with an Expanded Answer Space. CASE addresses this limitation by assigning importance weights for individual words based on their semantic relations to other words in the input. The dynamic weighting approach outperforms basic LM scores, not only because it reduces noise from unimportant words, but also because it informs the model of implicit commonsense knowledge that may be useful for answering the question. We then also follow prior work in expanding the answer space by generating lexically-divergent answers that are conceptually-similar to the choices. When combined with answer space expansion, our method outperforms strong baselines on 5 commonsense benchmarks. We further show these two approaches are complementary and may be especially beneficial when using smaller LMs.

CASE: Commonsense-Augmeeneed Score with an Expanded Answer Space
Wenkai Chen and Sahithya Ravi and Vered Shwartz
University of British Columbia Vector Institute for AI (wkchen, sahiravi, wshwartz wwc. ca
Abstract
The woman hired a lawyer because
A. she decided to sue her employer.
LLMs have demonstrated impressive zero-shot performance NLP tasks thanks to the knowl- edge they acquired in their training. In multiple- choice QA tasks, the LM probabilities as an imperfect measure of the plausibility of each answer choice. One of the major limita- tions of the basic score is that it treats all words as equally important. We propose CASE, a Score with an Ex- panded Answer Space. CASE addresses this limitation by assigning importance weights for individual words based their seman- tic relations to other words in the input. The dynamic weighting approach outperforms ba- sic LM scores, not only because it reduces noise from unimportant words, but also be- cause it informs the model of implicit com- monsense knowledge that may be useful for answering the question. then also follow prior work in expanding the answer space by generating lexically-divereent answers that are conceptually-simile the choices. When com- bined with answer space expansion, method outperforms strong baselines on 5 common- sense benchmarks. We further show these two approaches complementary and may be es- pecially beneficial when using smaller LMs.
B. she decided to run for office.
C. she wanted to sue her former employer.
used
sbe
docided
tпployer 2.70
LM
2.53
CASE
sbe
decided
to
701
for
office
score = 2.35
CA score 2.76
she
wartled
S18
het
Soctuee
employe:
LM score - 1.80
SE score 1.89
Figure I: An example from COPA. A and are the orig- inal options, while option C was generated by GPT-2 part of the answer space expansion step. The top line in each heatmap represent the LM (cross-entropy) score and the bottom line represents our score. Higher scores and blue blocks correspond to lower plausibility. CASE correctly predicts option (and option C which expansion of A) more plausible than option B, while the LM-score incorrectly predicts option B.
1 Introduction
Large language models (LLMs) have demonstrated strong few-shot and zero-shot performance across various NLP tasks, with the larger models often matching earlier fine-tuned approaches that relied on task-specific labeled data (Radford et al., 2019; Brown et al., 2020a; Touvron et al., 2023). We focus on the zero-shot setup, which assumes that the knowledge needed to perform specific task is already present in the LLM (Petroni et al., 2019; Zhou al., 2020; Saha et al., 2022). Zero-shot learning has been employed for tasks such trans- lating between unseen language pairs (Zhang et al., 2020), summarization (Brown et al., 2020a), com- monsense reasoning (Shwartz et al., 2020; Klein
and Nabi, 2021; Liu et al., 2022; Fang et al., 2022). and more.
In multiple-choice question answering (MCQA) tasks, zero-shot methods typically rely on the lan- guage model (LM) probabilities as proxy for plau- sibility, predicting the answer choice with the high- est probability conditioned on the question. LM score naïve proxy for plausibility, since it con- founds factors such as length, unigram frequency, and more (Holtzman et al., 2021: Niu et al., 2021). Indeed, in Figure 1, a GPT-2 based LM score in- correctly predicts that the woman hired a lawyer

---

because she decided to run for office, rather than because she decided to sue her employer.
2 Background
2.1 Plausibility Scoring
In this paper, we propose address one of the major limitations of the LM score. By summing or averaging the token-level probabilities, the LM score treats all tokens as equally important. A per- son reading this question would likely pay attention to option A because the word "sue" is highly rele- vant in the context of lawyer. This signal might be weaker in a basic LM score where the word "sue" is conditioned on each other token in the question and previous tokens in the answer. Furthermore, the LM might miss non-trivial connections between related words.
Although the plausibility score of a sentence be easily calculated by accumulating the proba- bility assigned by the LM for each token, this ap- proach suffers from various statistical biases such as sensitivity to the number of tokens, subword tok- enization, and word frequency (Abdou et al., 2020; Holtzman et al., 2021). To address these biases, several improvements have been proposed. With respect to the length bias, prior work normalized the score by length (Mao al., 2019; Brown al., 2020b), or focused the conditional probabilities of the question, which unlike the answer choices has fixed length (Trinh and Le, 2018; Tamborrino et al., 2020). To factor out word frequency, Holtz- man et al. (2021) proposed Domain Conditional Pointwise Mutual Information (DCPMI), which normalizes the conditional probability of the an- given the question by the prior probability of the answer. This is computed as the conditional probability of the answer given a domain -specff prefix such as "The sentiment the movie is" for sentiment analysis or "The answer is" for general QA tasks. SEQA (Niu et al., 2021) mitigates the sensitivity to word choice by generating answers using GPT-2, and selecting the answer choice most similar to the generated answers.
To address this challenge, we propose CASE: a Commonsonse-Augmenniee Score with an Expanded Answer Space. CASE a post-hoc dy- namic weight scoring algorithm that prioritizes im- portant words in the sentence. The importance of each individual word is determined based on its relationship with other words in ConceptNNt (Speer al., 2017). For example, ConceptNet pro- vides the information that "sue requires having a lawyer". We use the word-level importance scores to re-weigh the LM probability scores. Indeed, in the second line of option A in Figure 1, the impor- tance of the word "sue" increases the score of the entire sentence, leading to correctly predicting A the correct answer.
Existing methods solely focus the relationship between words in the choices and words in the question, ignoring the importance of each word for the decision. In this paper, we propose a new token- level weighting method to consider the importance of different words within the sentence based on their relationship to other words.
We further adopt the strategy suggested by Niu et al. (2021) to expand the answer space by using LM to generate additional answers and then map- ping semantically-simill generated answers into the original space. This mitigates the LM score's sensitivity to infrequent words. Figure 1 demon- strates that generated option C, "she wanted to sue her former employer", which is conceptually similar to A, further yields a higher probability score with our method.
2.2 Knowledge-Enhancee Models
Zero-shot LM-based scoring methods implicitly reason about which answer is more likely based on the token-level probabilities. However, many tasks require multiple steps of reasoning to reach the cor- rect answer (e.g., Mihaylov et al., 2018; Yang et al., 2018; Khot et al., 2020). A common approach is to retrieve relevant commonsense knowledge from knowledge bases (KBs) such as ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019a; Hwang al., 2021), in order to enhance the neu- ral model and explicate the reasoning steps (e.g.. Bauer et al., 2018: 0018; Xia et al., 2019; Lin et al., 2019; Guan et al., 2019; Chen et al., 2020; Huang et al., 2021). More recent work used the COMET model
We tested CASE on 5 popular commonsense MCQA datasets. CASE outperformed the broad range of strong baselines that we compared with, confirming that it an effective method for zero- shot MCQA. We further study the impact of dif- ferent model sizes, answer candidates of varying qualities, and different weight assignment strate- gies on the performance.!
'Our code is available at Github.

---

Group
Q: The woman hired lawyer because
she decided her employer
Cp. she wanted to sue her former employer.
A. she decided to sue her employer.
B. she decided to run for office.
Space Expenting
Group B
candidate genemttonn
B sbe decided to run for office.
Cp
Cy. she was wrongfully accused of crime.
Cy. she felt she was being treated unfairly.
Keyword Extraction
Weight Assigning
Cy. she wanted to sue her former employer.
ConceptNet
C
A nsw with weight
Group
Sooide
tv
enploye
Group
Fusea
snshop
LM scoring
A. 0.067
C₁ 0.151
Group
Group
0.063
Cp
Figure 2: Overview of CASE, illustrated with an example from the COPA datasct. Groups A and B correspond to original choices and and any generated answers mapped them ($3.3). Each word each answer is scored based on its ConceptNet relationships to other words in the instance (83.2). The score for each answer is based the word probabilities ($3.1), weighted by the word-level scores. Finally, CASE predicts the answer choice with the highest scoring answer its group.
(Bosselut et al., 2019; Hwang et al., 2021), which is a LM fine-tuned on the aforementioned KBs, to enhance models with high-coverage contextu- alized commonsense inferences (e.g.. Majumder al., 2020; Bosselut al., 2021; Kim al., 2022; Chakraburty al., 2022; Ravi et al., 2023).
CASE can be used for zero-shot MCQA tasks. It is based on LM score (Section 3.1). However, rather than treating all words in the context and answers as equally important, we propose weighted score where the conditional probability is weighed by the importance word. The weights are determined using a commonsense KB in order to provide infor- mation that humans might implicitly be reasoning about when answering such questions (Section 3.2). Following Niu et al. (2021), we expand the set of answer candidates by generating free-text answers, to increase the scorer's robustness to lexical vari- ability (Section 3.3). An overview of the method is shown in Figure 2.
An alternative recent approach which doesn't rely on external KBs prompts a LM to generate additional knowledge which is then incorporated back into the LM to make the prediction. Shwartz al. (2020) and later Liu al. (2022) used LM to generate questions and answers about an MCQA instance. The answers to the questions are then incorporated into the LM-based scoring model as additional knowledge. Wei et al. (2022) proposed the popular thain-of-thoughh (COT) prompting ap- proach in which the LM is taught through examples generate multiple steps of reasoning followed by the answer the question. In the zero-shot sion, the LM instructed to "think step-by-step" Finally, following concerns about the faithfulness CoT inferences, Creswell et (2022) proposed to iteratively select parts of the inputs and draw inferences on them.
3.1 Basic Scoring Method
The basic scoring method directly uses the LM score, which is calculated by accumulating the con- ditional probabilities assigned by the LM for each token given the prefix. Given a question Q Qu...Iug and an answer choice A, = Q..1...Q.,AA.> we convert Q into a declarative statement (see Appendix A), and define the LM score of answer choice A; as follows:
PA. P(Ai(s)
3 Method
DA Placsls,aas "...)))
II
n. nA.
We propose CASE, Aaammonsense-Augmenntne Scoring method with an Expanded Answer Space.
where n, is the number of tokens in s.

---

Finally, we can determine the most plausible choice A among the answer choices based on their corresponding scores:
denote Paths We convert the path set of sentences by expressing each edge as a natural language sentence, based on relation templates (see (2) Appendix B). For example, the path sue related to, in the context of law". We use the LM to score a single path Parig as follows. First, the score S Ea) of edge E, = (I., Ra,yi) is calculated as the conditional probability of generating the second node y/ following the textual template of relation R2 to which we assign the first node 2i, such P(lawisue is related to). We use the chain rule for conditional probability compute the score of the entire path:
= arg max PA:
law <th context of lawyer is expressed S "sue is related to law" and S2 = "lawyer is word used
3.2 Commonsense Augmented Scoring
The importance of individual words in the question and their contribution to choosing the correct an- swer varies greatly. Take for example the instance in Figure 1, taken from the COPA dataset (Gordon et al., 2012). Determining the cause of the event "The woman hired a lawyer" involves reasoning about the circumstances in which one might hire a lawyer, such as if they are suing someone. In this case, the keywords "lawyer" from the context and "sue" from the answer choice, and the semantic re- lation between them (i.e., suing someone requires a lawyer), supports the correct prediction. To that end, CASE first identifies important keywords from the question and answer choices (Section 3.2.1). Each keyword is assigned an importance score, and the conditional probability PA is updated by con- sidering the importance of each token in the answer choice (Sec 3.2.2).
Paree Σ
((Pa)))) = Paninl +
log S(R.) + log S(A")
where E' is an artificial summary edge from to YPa-gg with the "is related to" relation, such as "sue is related to lawyer".
To get an aggregated score for token a, we sum the scores of all paths in Paths
Spathson
Σ
S(Pargg)
(5)
3.2.1 Keywords Extraction
Given a question Q and answer choice A, use YAKE (Campos et al., 2018), an unsupervised automatic keyword extraction method, to extract a set of keywords Keyq C Q and Kсy C A. In particular, we are interested in finding the key- words from each answer choice that are important in the context of the question Q, which we denote ConceptNet
Finally, the weight for each token aij A; is com- puted as follows.
1 + XSpanao
if oo. E Key A.12 (6)
War =
1,
if of $ Key Key A,10
where is a hyperparameter ($4.3).
Key Key A. To that end, (Speer et al., 2017), commonsense knowledge base, find paths between terms in Keyq and Key A+ and include in Key blr answer choice that are connected ConceptNet to keywords from the question:
With the weights for each token, we can now update the LM score defined in Equation to a weight-based plausibility score as follows:
Aay keywords from the
PA = II Wo,; P(aisls,an... ,.ij 1) (7)
j=1
3g Keyo A
KeyAiq
a Key
3p =p=ca a CN CNA
(3)
3.3 Expanded Answer Space
|pl S
The final addition to our model aims at reducing the LM sensitivity to the phrasing of the correct answer. For example, an infrequent word in the correct answer choice of the choice and make the LM predict another option more plausible (Holtzman al., 2021). To mitigate this issue, we follow Niu et al. (2021) and expand the set of answer candidates by using a causal LM to generate open ended answers A*
where p denotes path in ConceptNet (CN) with up edges.
reduce the overall probability
3.2.2 Weight Assigning
We assign a weight to each token Key AlQ based on the strength of its connection to keywords in Keyg- To that end, we look at all the ConceptNet paths that connect a with keywords in Keyg- which

---

J. The idea is allow the model to consider various phrasings of the same conceptual answer. For example, in Figure 2, the generated answer C1 is a paraphrase of answer choice A.
(Ai, Ah, A
The Social Interaction Question Answering (SocialIQA; Sap al., 2019b) dataset tests models on their understanding social situa- tions and human behavior. Each question presents a hypothetical scenario followed by a question and 3 answer choices.
SocialIQA.
We treat the generated answer choices A" the same as the original answer choices A and com- pute the score for each answer A E A" using Equation 7. To map the answer choices back into the original answer space A, we attempt to match each A E A A, E A based on two criteria: sentence similarity and keyword connections.
ARC. The A12 Reasoning Challenge (ARC; Clark et al., 2018) consists of 7,787 science exam questions drawn from variety of sources. The questions are divided into Easy (ARC-E) and Chal- lenging (ARC-C) sets.
Sentence Similarity. We use the Sentence- Transformer package (Reimers and Gurevych, 2019) to represent the answers, and compute the cosine similarity between the representations of each generated answer in A* and original answer in A. The similarity score between the sentence pair should be above 8sim-
OBQA. The OpenBookQA (OBQA; Mihaylov et al., 2018) dataset contains questions that require multi-step reasoning, use of commonsense knowl- edge, and rich text comprehension. The dataset has roughly 6,000 questions.
Since the test set of SCT and SocialIQA are not publicly-available, we report the accuracy on the development set for all datasets.
Keyword Connections. We calculate the connec- tion score between the keywords in each generated answer in A and each original answer in A using the method introduced in Sec 3.2.2. We require the connection score be greater than O.
4.2 Baselines
We compare our proposed method with the basic LM-based scoring method described in Section 3.1, as well as more advanced LM-based scoring meth- ods described below.
candidate can only be assigned to a group it meets both thresholds, and we discard generated answers that are not mapped into answer choices in A. Once we mapped generated answers to original answers, the final prediction of the model modi- fies Equation 2 to select the highest scores of all answers within the same cluster:
Self-talk (Shwartz al., 2020) consists of two causal LMs. The knowledge generator LM gen- erates clarification questions conditioned on the context and pre-defined prefixes, and their corre- sponding answers. The scoring LM computes the probability of each answer choice conditioned on the context and question well the additionally generated knowledge.
Å arg max arg max PAUS
(8)
j
where As, is the jth answer in cluster A;.
DC-PMI (Holtzman et al., 2021) aims to elim- inate the effect of the number of synonyms and the word frequency on the LM score by dividing the conditional probability (Eq 1) by a domain- conditional prior probability for the answer choice.
4 Experimental Setup
4.1 Datasets
We evaluated our method on five multiple-choice commonsense question answering datasets de- scribed below.
SEQA (Niu al., 2021) uses LM to generate a set of answer candidates. These candidates then "vote" for an original answer candidate based on their semantic similarity to each candidate, and the top-voted answer is selected as the final answer. For fair comparison with the other model, we changed the voting model from SRoBERTa N.I the origin SRoBERTa that was further fine-tuned on an NLI dataset.
COPA. The goal in the Choice of Plausible Alternatives dataset (COPA; Roemmele et al., 2011) is, given a premise event, to choose the more plausible cause or effect among two alternatives.
SCT. The Story Cloze Test dataset (SCT; Mostafazadeh et al., 2016) is a collection of four- sentence stories with two possible endings. The goal is to predict which ending is more plausible following the beginning of the story.
We don't compare with follow-up work by Liu et al. (2022) since they targeted a diffcrent set of tasks.

---

Methods
LM
COРA
SCT
SocialIQA
ARC-E
ARC-C
OBQA
Scoring
Generating
LMoum
GPT2
69.0
67.6
43.1
53,5
25,4
22.4
LMavy
GPT2
68.4
71.5
45.8
47.4
28.7
30.8
Self-talk
GFT2
GPT2
66.2
70.4
47.5
DCPMI
GPT2
70.8
68.6
39.2
36.0
25.1
31.4
SEQA
SROBERTa
GPT2
55.8
57.4
36.4
32,1
23.7
21.2
SEQAGPTS
SRoBERTa
GPT3
66.2
64.4
40.3
54.4
34.8
22.2
CDG
GPT2
COMET
72.2
71.5
45.4
ArT
GPT2
GPT2
69.8
71.6
47.3
CA8
GPT2
70.4
73.0
46.0
55.8
28.8
32.6
CASEGPT2
GPT2
GPT2
73.8
76.1
46.1
54,4
30.8
30.2
CASEØPTS
GPT2
GPT3
78.2
83.2
48.5
63.2
36.5
35.2
Table I: Accuracy (%) of the scoring various methods on the dev sets. All scoring methods are based on GPT- Zztarge: CASEGPT2 and CASEGrT3 denote CASE with candidate generation by GPT-2ztarge and GPT-3 respectively. Takeaway: Weighting leads to substantial improvements. When combined with candidate generation, it outperforms all baselines by a large margin.
CDG (Bosselut al., 2021) uses knowledge from COMET (Bosselut al., 2019) to construct local commonsense knowledge graph for reasoning and inference.
5 Results
5.1
Main Results
The performance of the various scoring methods on the benchmarks are presented in Table 1. For fair comparison with the baselines, the table shows the performance when GPT2ziarge is used. We report the accuracy on the dev set. CAS stands for Commonsense-Auggeene Scoring, i.e. it execuudess the candidate generation.
ArT (Wang and Zhao, 2022) consists of two steps: notes taking and reverse thinking. In the notes taking step, the LM generates templated in- ferences pertaining key phrases in the context, which are later added as additional knowledge. The reverse thinking step aggregates the scores of dif- ferent orders of the answer and question (e.g. "x because y" vs. therefore x").
The performance of CAS shows that weighting leads substantial improvements upon the simpler baselines. CAS also stands out the competition with DCPMI, which can also be regarded as spe- cial weight-scoring method.
4.3 Setup and Hyper-parameters
We used GPT-2 via the HuggingFace Transformers library (Wolf al., 2020) for the scoring part, and GPT-2 XL and GPT-3 davinci-008 for the answer space expansion step. In the keyword extraction step ($3.2.1), we included ConceptNet paths with up to k 3 edges. In the weight assigning step ($3.2.2) set the coefficient À to 10.
When combined with candidate generation, CASE outperforms nearly all baselines, except for the SociallQA dataset, on which ArT and Self-talk perform better. Notably, both baselines rely on human husan-designed prompts to generate additional in- formation, which might give them an advantage.
The gap performance from SEQA, which also expands the answer space by generating candidate answers, further demonstrates the effectiveness dynamic weighting.
In the answer space expansion step ($3.3), we generated n 100 answers from GPT-2 and nA. 50 answers from GPT-3 for cach ques- tion. Similarly to SEQA,
used nucleus pling (Holtzman et al., 2021) with p 0.9 and set a maximum length of 1.5 tokens for both LMs. We set the sentence similarity threshold to Saim = 0.5 for GPT2 x-large and 8yim 0.6 for GPT-3.
5.2 Effeet of the Scoring LM Size
Table shows the performance of CAS, CASE and the simple baselines when using different sizes of GPT-2 models in the scoring part.
flyper-paramete values were selected based on preliminary experiments on the training sets and were not tuned on the dev sets.
Bigger better. Across the various methods, big- ger LMs perform better than smaller LMs.

---

Datasel
Methods
GFT2,
GFT2
GPT2,
GPT2x
78
opt2.s
gpt2-m
LMuum
600
66.6
69.0
gpt24
COPA
LM
CAS
62.6
62.0
672
63.4
67.0
70.4
68.4
76
gpt3
gpt2-d
CASECPT2
69.6
72.0
72.2
73.8
Accuracy CORN 190)
CASEGPTa
754
76.4
77.4
78.2
74
L.
58.2
62.7
64.4
67.9
LMang
60.4
66.4
68.8
71.5
SCT
CAS
61.9
67.5
70.9
73.0
72
CASE
75.2
CASEGTTS
76.7
78.6
79.0
83.2
LM
39.7
41.4
42.0
43.1
LMang
SIQA
CAS
42.8
44.6
45.7
46.0
68
CASEOPTS
CASEOPTS
47.6
48.4
48.5
48.5
40
60
100
LMaum
44.2
48.8
50.4
53.5
Number candidates
LMasp
37.9
47.4
ARC-E
CAS
46.1
49.8
53.0
55.8
Figure 3: Accuracy curve of CASE the COPA dev set, with different numbers candidates generated from various LMs. The dotted line represents the baseline method LM which uses GPT2z Takeaways: Generating more candidates leads to higher accuracy. but larger scoring LMS require fewer candidates.
CASEGIT2
46.5
49.6
52.0
54.4
CASROPTS
LM
19.7
23.1
22.7
25.4
LMase
23,4
23.7
25.4
28.7
ARC-C
CAS
2644
26.4
27.4
28.8
CASEGPTT
28.1
29.4
27.8
30.8
CASEGPTa
LMrum
16.2
18.2
21.8
22,4
LMa.y
23.0
26.8
30.0
30.8
CASEGPTT
26.0
26.6
27.4
30.2
be filtered out. This means that CASE effectively degenerates CAS. Thus, it's important to gener- ate a large number of candidates. This reassesses the findings in Niu et al. (2021).
32.2
35.4
37.4
35.2
Table 2: Accuracy when using GPT2 models with differ- ent sizes for the scoring. Takeaways: CAS consistently outperforms standard LM scoring methods, and is out- performed by CASE. For CASE, the best performance is achieved when using large GPT2 models for scoring and more importantly, GPT3 for candidate generation.
Larger models require fewer candidates. Larger LMs generate higher quality text which is more likely be fluent, relevant the context, log- ically correct, and consistent with commonsense knowledge. Therefore, we can expect fewer candi- dates to filtered out. In addition, the generated candidates may conceptually similar and better phrased than the original choice.
Smaller LMs gain more from candidate genera- tion. While all LMs benefit from weighting and candidate generation, smaller LMs gain bigger im- provements. For example, candidate generation with GPT-3 adds 13.4 points on COPA a GPT2s CAS scorer, but only 8.2 points for GPT2x1- We hypothesize that the model performance is more sensitive to the LM quality when a single sentence is considered, while expanding the answer space makes even the lower-quality LMs more robust.
5.4 Effect of the Weighting Strategy
Table compares the COPA performance of differ- ent weighting strategies. Two baselines, LM and LMagg, already introduced in Section 3.1, treat all tokens equally, summing or averaging the token- level probabilities. Conversely, the static weighting strategy (SW and SWC, with without candidate generation), assigns static number (1.5) each selected key token. Finally, the dynamic weighting strategies (CAS and CASE) not only distinguish key tokens from unimportant ones but also assign differ- ent scores to each key token based on its semantic relevance to the question.
5.3 Efffec of the No. of Generated Candidates
Figure shows the effect of the number of gener- ated candidates on the performance, focusing on CОРА. We summarize the findings below.
Generating more candidates leads to higher ac- curacy. When generating few (< 20) candidates, the model's performance is unstable and relatively low. This might happen due to the generated an- swers being conceptually different from the orig- inal candidate answers, in which case they might not meet the mapping thresholds in Section 3.3 and
The results show that while the static weighting strategy outperforms the baseline when no addi- tional candidates are generated (sW vs. LM), these strategies perform similarly when additional candi- dates are generated (sWC vs. LM+c). In both cases,

---

GPT2,
GPT2m
GPT2,
GPT2,t
Jim is new driver and has never been pulled over before. Yesterday he was pulled over for speeding.
LMaum
60.0
66.6
69.2
69.0
The officer explained to him why he was being given ticket.
+SW
61.2
66.6
70.0
69.6
Jim will have to work overtime to pay for the ticket.
CAS
62.0
67.2
69.4
70.4
+C
69.2
70.4
72.4
milted
69.2
71.2
72.4
72.2
releted
+ CASE
69.6
72.0
72.2
73.8
Jim is very upset.
depression
couse
stress
Table Accuracy on the COPA dev set when using dif- ferent weight-assigning methods. The methods below the dotted line expand the answer space by generating additional answer candidates. Takeaway: keyword se- lection improves the performance, especially when it is informed by commonsense knowledge.
Figure 4: An SCT example, along with the correct an- swer predicted by CASE, and example ConceptNet path that increased the weight of the important word upset.
Priya decided to try restaurant. She drove to new cafe opened. Priya sat booth.
She ordered mimosa and breakfast burrito.
the static weighting strategy underperforms com- pared to the dynamic strategy. This result confirms that commonsense knowledge can help inform the model about the keywords that are important for the current question.
manner of
Priya decided not to eat and left.
run
Figure A eecorrectl-ppredcctee SCT example, along with the incorrect answer predicted by CASE, and an example ConceptNet path that increased the weight of the word left.
6 Qualitative Analysis
We focus on CASE and look at the individual to- ken scores and corresponding ConceptNet paths to better understand the model decision-making process.
for knowledge retrieval from the KB, and adapt CASE for additional NLP tasks.
Figure 4 shows example from SCT where CASE predicted the correct answer. The word "up- set" in the correct answer choice was assigned a high weight by CASE thanks to ConceptNet paths such as upset related to depression stress related to work.
Limitations
Computational complexity. CASE is com- putationally expensive than using a basic LM score, as involves finding relevant paths from an ex- ternal knowledge base and then estimating their likelihood with a LM, in order to gauge the impor- tance of keywords.
causes
Conversely, in Figure 5, CASE predicted the in- correct answer choice for another SCT example. The model focused on the word "left" due to its semantic relation to the word "drove", failing to understand that Priya drove and not away from the restaurant.
Concept coverage. The weight assignment strat- egy CASE is based on ConceptNet. The knowl- edge in KBs such as ConceptNet not contextu- alized, which means that some facts pertaining to concepts in the instance might not be relevant to the specific context. In addition, it has limited cov- erage. COMET (Hwang et al., 2021) has been used in prior work (Majumder al., 2020; Chakrabarty et al., 2020; Ravi et al., 2023) to overcome this limitation. However, finding relevant paths using COMET requires an iterative multi-hop reasoning approach (Arabshahi et al., 2021) which is more complex, and more computationalll-intenssi aim to explore efficient ways to achieve this in future work.
7 Conclusion
We presented CASE, a novel LM-based plausibil- ity score for zero-shot MCQA tasks. CASE uses a commonsense KB to assign importance weights to words in the input. The weighting strategy outperforms basic LM scoring methods. When combined with generating additional answer candi- dates, CASE outperforms the baselines on 5 popular MCQA benchmarks. We further showed that the two approaches are complementary and are espe- cially beneficial when using smaller LMs. the fu- ture, we plan to explore a more selective approach
We
Answer format. Since our method assigns a weight for each word in the input, is only ap-

---

plicable for MCQA tasks in which the answer is a sentence. The weighting would be trivial for tasks with single word answers such as Common- senseQA (Talmor et al., 2019) and BoolQ (Clark et al., 2019).
Lisa Bauer, Yicheng Wang, and Mohit Bansal, 2018. Commonsense for generative multi-hop question an- swering tasks. In Proceedings of the 2018 Confer- ence on Empirical Methods Natural Language Processing, pages 4220-4230, Brussels, Belgium. Association for Computational Linguisties.
Performance limit. Our model demonstrates a significant performance improvement over other zero-shot baselines across a majority of datasets. However, it is worth noting that the state-of-te of-the- art performance on the datasets in this paper is achieved with more supervision (i.e. supervised or few-shot models).
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2021. Dynamic neuro-sy couro-symbolic knowledge graph construc- tion for zero-shot commonsense question answering. In Proceedings of the AAAI conference on Artificial Intelligence, volume 35, pages 1923-4931
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
Ethies Statement
Data. All the datasets and knowledge bases used in this work are publicly available. We used Con- ceptNet as source of commonsense knowledge. Since ConceptNet was crowdsourced, some of the knowledge may contain societal biases or preju- dices held by the annotators (Mehrabi et al., 2021).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models few-shot learners. In Advances in Neural Information Processing Syxtems (NewrIPS). volume 33, pages 1877-1901.
Models. The GPT-2 models are publicly acces- sible via HuggingFace, while GPT-3 is closed model behind an API. All language models may generate offensive statements if prompted with spe- cific inputs, however, our model only generates text internally while the end output is a choice between human-written answer candidates.
A Acknowledgements
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 33: 1777-190..
This work was funded, in part, by the Vector Insti- tute for AI, Canada CIFAR AI Chairs program, an NSERC discovery grant, and research gift from A12.
Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Célia Nunes, and Adam Jatowt. 2018. Yake! collection-indepeedent automatic key- word extractor. In Advances in Information Retrieval: 40th European Conference on IR Reseurch, ECIR 2018, Grenoble, France, March 26-29, 2018, Pro- ceedings 40, pages 306-810. Springer.
References
Mostafa Abdou, Vinit Ravishankar, Maria Barrett, Yonatan Belinkov, Desmond Elliott, and Anders Sø- gaard. 2020. The sensitivity of language models and humans to Winograd schema perturbations. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7590- 7604, Online. Association for Computational Lin- guistics.
Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022. It's not rocket science: Interpreting figurative language in narratives. Transactions of the Associa- tion for Compulational Linguistics, 10:589-606.
Forough Arabshahi, Jennifer Lee, Antoine Bosselut, Yejin Choi, and Tom Mitchell. 2021. Conversa- tional multi-hop reasoning with neural commonsense knowledge and symbolic logic rules. Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 7404-7418, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Mure- san, and Nanyun Peng. 2020. R": Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7976-7966 Online, Association for Computa- tional Linguistics.

---

Qianglong Chen, Feng Ji. Haiqing Chen, and Yin Zhang. 2020. Improving commonsense question answer- ing by graph-based iterative retrieval over multiple knowledge sources. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 2583-2594, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7038-7051, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Canming Huang, Weinan He, and Yongmei Liu. 2021. Improving unsupervised commonsense reasoning ing enoweedge-enabled natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4875 4885, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 9924-9936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In AAAI.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try are, the ai2 reasoning challenge, arXiv preprint arXiv: 88.5555577
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal, 2020. Qasc: A dataset for question answering via sentence compo- sition. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5082-8090
Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint 2tti::2205007111
Seungone Kim, Se June Joo, Hyungjoo Chae, Chae- hyeong Kim, Seung- won Hwang. and Jinyoung Yeo. 2022. Mind the gap! injecting commonsense knowl- edge for abstractive dialogue summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6285-6300, Gyeongju, Republic of Korea. International Com- mittee on Computational Linguistics.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP), pages 1173-1178, Hong Kong, China, Association for Computational Linguistics.
Tassilo Klein and Moin Nabi. 2021. Towards zero- shot commonsense reasoning with self-supervised refinement of language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P Processing, pages 8737-8743, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yuwei Fang. Shuohang Wang. Yichong Xu, Ruochen Xu, Siqi Sun, Chenguang Zhu, and Michael Zeng. 2022. Leveraging knowledge in multilingual com- monsense reasoning. In Findings of the Axsociation for Computational Linguistics: ACI. 2022, pages 9237-3246, Dublin, Ireland. Association for Compu- tational Linguistics.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019, KagNet: Knowledge-awant graph net- works for commonsense reasoning. Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829-28333 Hong Kong, China. Association for Computational Linguistics.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEval-2012 task 7: Choice plau- sible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Seman- tics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- uation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguisties.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3154-3169, Dublin, Ireland. Association for Compu- tational Linguistics.
Jian Guan, Yansen Wang, and Minlie Huang. 2019. Story ending generation with incremental encod- ing and commonsense knowledge. Proceedings of the AAAI Conference on Artificial Intelligence, 33701)66433-6400
Bodhisattwa Prasad Majumder, Harsh Jhamtani, Tay- lor Berg- Kirkpatrick, and Julian McAuley. 2020. Like hiking? you probably enjoy nature: Persona- grounded dialog with commonsense expansions. In
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form com- petition: Why the highest probability answer isn't

---

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9194-9206, Online. Association for Computa- tional Linguisties.
Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023. VIe-bert: Visual question answering with contextualized common- sense knowledge. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion (WACV), pages 1 115--1165.
Huanru Henry Mao, Bodhisattwa Prasad Majumder, Julian McAuley, and Garrison Cottrell. 2019, Im- proving neural story generation by targeted common sense grounding. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP). pages 5988-5993, Hong Kong, China, Association for Computational Linguistics.
Nils Reimers and Tryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing GEMNLP-IJCNLL pages 3982-3992, Hong Kong. China. Association for Com- putational Linguistics.
Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pu- jara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 5016-5033, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In AAAI spring symposium: logical formal- izations of commonsense reasoning, pages 90-95.
Amrita Saha, Shafiq Joty. and Steven C.H. Hoi. 2022. Weakly supervised neuro-symbolie module networks for numerical reasoming over text. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10): 1238-11247.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2399 Brussels, Belgium. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula. Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019a. Atomic: An atlas of machine commonsense for if- then reasoning. In Proceedings of the AAAI con- ference on artificial intelligence, volume 33, pages 3027-3035.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839 849, San Diego, California. Association for Computational Linguis- tics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Com- monsense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIP), pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Yilin NIu, Fei Huang, Jiaming Liang, Wenkai Chen, Xiaoyan Zhu, and Minlie Huang. 2021. A semantic- based method for unsupervised commonsense ques- tion answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3037-3049, Online. Association for Computational Linguistics.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages
Online. Association for Computa- tional I. Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence arrificial intelligence, volume 31.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

---

4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Proceedings the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1628- 1639, Online. Association for Computational Linguis- tics.
Alexandre Tamborrino, Nicola Pellicand, Baptiste Pan- nier, Pascal Voitot, and Louise Naudin. 2020. Pre- training is (almost) all you need: An application to commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1878-3887, Online. Association for Computational Linguistics.
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pre- trained language models. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9733-9740.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Trieu H. Trinh and Quac V. Le. 2018. A simple method for commonsense reasoning.
Jiawei Wang and Hai Zhao. 2022. ArT: All-round thinker for unsupervised commonsense question an- swering. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1490-1501, Gyeongju, Republic of Korea. Interna- tional Committee Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
formers: State-of-the
Jiangnan Xia, Chen Wu, and Ming Yan. 2019. Incor- porating relation knowledge into commonsense read- ing comprehension with multi-task learning. Pro- ceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2393-2396.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen- nrich. 2020. Improving massively multilingual neu- ral machine translation and zero-shot translation. In

---

A Question Prompts
B Relation Templates
Table 4 shows the prompts used for each dataset. For tasks with several specific question type such as COPA and SocialIQa, we convert each question type to a natural language proxy following previ- ous work (e.g. Shwartz et al., 2020). For tasks that present open-ended question, the prefix "The answer is". Finally, for tasks that are already designed to expect the next word or sentence (such as SCT), we use the instance as is.
Table displays the templates we used to convert edges with different relation types in ConceptNet to natural language sentences, following Davison et al. (2019).
append
Relation Type
Template
Sunuef B
form
244
A is
part
A
A has B
Dataset
Question
ued B
used for capable of
My body cast shadow over the grass [hecause] The physician misdiagnosed the patient [so]
COPA
cspable
oxpile
R
capable
Tyler went to a baseball game. He saw his favorite team! His team played hard. His team won! []
SCT
location
B
happans as subevent A
SociallQa
Tracy didn't go home that evening and resisted Rilcy's attacks, [Refore, Tracy needed to]
A fist subeical
B
begins with
Which technology was developed most re- cently? [the answer is]
A mlcessike
B is a dependency of he descrihed
B
ARC
A green plant absorbs light. A frog cats fliss.
progett
con not be deseribed as
These are both examples of how organisms [] the
Teehraal by and
B
Someone does A because want result
person can grow cabbage January help what product? [the answer is]
by
a
way
doiies
A desires B
OBQA
Gas can fill any container it is given, and liquid
A daiie
B
do not desire B created by simmilar to B
[]
Table 4: Question Formats used for each dataset. The red words square brackets are additions to the context, designed specifically for each dataset.
4
A opposite to B distinct from
Claivs
B
A
Strinol
symhol of B
detood
dafined
spocific way to
A
Jocasl ICE
B
A near
B
word used in the context
B
similar
riymadoghallyy wassad
have common origin with B
dynskgicalyyzzistiiccn
B derived
doiee
makes someone want B
A
sode of
B
A made of B
A
B
can be done to A
Table 5: Natural language templates for each relation
type in ConceptNet.


Reviews:
[{"review_id": "qhcgaT05WK", "reviewer": "Reviewer_CJBH", "summary": "", "questions": "1. In section 3.2.1, it is unclear what the automatic keyword extraction is optimizing for when picking the initial keywords before the paths are searched for in ConceptNet. Could you clarify?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. \n\n**While interesting, the results seem to suggest that the majority of the gains are due to the candidate expansion rather than the common-sense re-weighting scheme.**\n\nAs we show in Table 1, when candidate expansion and commonsense-reweighting are combined, they lead to outperforming the baselines by a large margin. While the gains from candidate expansion are bigger, Table 1 shows that each of these components yields improvements, in particular the gap in performance from the LM base scores shows the benefit of weighting. These two components complement each other. \n\n**Further, it is unclear if this method generalizes beyond GPT-2 to newer low-to-moderate-sized LLMs. Evaluations with multiple models would be interesting to see.**\n\nThanks for the suggestion to test our method on multiple models. We will add baselines based on other low/moderate sized LLMs (e.g. T5, BART) in the camera-ready version. \n\n**In section 3.2.1, it is unclear what the automatic keyword extraction is optimizing for when picking the initial keywords before the paths are searched for in ConceptNet. Could you clarify?**\n\nWe used YAKE, an unsupervised keyword extractor, to extract salient words from a document before we search paths on ConceptNet. Yake doesn’t rely on any external corpus or dictionary, and it is based on features such as casing, position, word frequency, and word relatedness to context. Our motivation was to only focus on words that provide important information for the sentence, and to reduce the overall search time. We will clarify this in the paper.\n\n**In L257, S is being used to denote the scoring function while the same symbol was used in L199 to denote the declarative statement.**\n\nThanks for pointing this out, it is indeed confusing, and we will change the notation in the camera-ready version.\n\n**In Equation 1, is there a reason for operating in the probability space instead of the more typical log probability space?**\n\nThanks for pointing this out. We wrote the equation in the probability space for the sake of simplicity, but in practice all computations were done in the log space for numerical stability. We will clarify it in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel common-sense augmented re-weighting scheme couple with candidate answer expansion that shows positive empirical performance in the task of multi-choice question answering. ", "reasons_to_reject": "While interesting, the results seem to suggest that the majority of the gains are had due to the candidate expansion rather than the common-sense re-weighting scheme. Further, it is unclear if this method generalizes beyond GPT-2 to newer low-to-moderate-sized LLMs. Evaluations with multiple models would be interesting to see. ", "questions_to_authors": "1. In section 3.2.1, it is unclear what the automatic keyword extraction is optimizing for when picking the initial keywords before the paths are searched for in ConceptNet. Could you clarify?", "paper_topic_and_main_contributions": "This paper addresses problems arising from scoring sequences of text using token-level probabilities output by a language model in the context of multiple-choice question answering. In particular, the authors make the claim that the performance of standard sequence scoring is negatively affected due to equal weights being assigned to all tokens. To overcome this problem, this work proposes a dynamic importance-weighting scheme that increases the probability scores of words that are deemed to be more relevant to the task/question at hand. The signal for importance, or relevance, is based on word-level scores derived from descriptions contained in the ConceptNet KG. Lastly, the authors propose LM-based answer space expansion to generate additional answers, which are then semantically clustered and mapped back to the original answer space.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Equation 1, should $s$ should be $S$?\n2. In Equation 1, is there a reason for operating in the probability space instead of the more typical log probability space?\n3. In L257, S is being used to denote the scoring function while the same symbol was used in L199 to denote the declarative statement."}}, {"review_id": "phiqXJFULH", "reviewer": "Reviewer_sJkV", "summary": "", "questions": "I have examined the results report in the SEQA paper (https://aclanthology.org/2021.acl-long.237.pdf), but upon comparison, I have observed discrepancies between the reported results in the current paper and those presented in the SEQA paper's Table 2.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. \n\n**The scope of the proposed method is conventional and constrained, as it is solely applicable to answer selection tasks, without extending its utility to the enhancement of answer generation tasks.**\n\nWe note that many NLP datasets are designed as multiple-choice question answering tasks, hence our method is broadly applicable to many NLP datasets. While adapting our method to generative tasks is beyond the scope of this paper, we hypothesize that it would be straightforward to adapt it by treating the answer expansion phase as a way to generate the initial answer choices. With that said, generative tasks are harder to evaluate automatically, so we leave this for future work.  \n\n\n**Furthermore, the applicability of the proposed method within a few-shot setting remains a pertinent consideration. The inclusion of experiments conducted within a few-shot scenario would be advisable, as it has the potential to yield practical advantages for real-world applications.\nNotably, GPT-3 davinci-003 serves as the basis for answer space expansion. However, it is noteworthy that a comprehensive evaluation of the performance of GPT-3 davinci-003 is absent from the reported findings.**\n\nFirst, our paper focuses on the zero-shot setting, however, we acknowledge the strengths of the few-shot setup and we plan to explore extending our method to this setup in the future. Second, we note that our model requires access to the individual token probabilities so it only supports open models or models that provide this information.  \n\n\n**I have examined the results report in the SEQA paper (https://aclanthology.org/2021.acl-long.237.pdf), but upon comparison, I have observed discrepancies between the reported results in the current paper and those presented in the SEQA paper's Table 2.**\n\nThe results in the SEQA paper are better because they used sentence-RoBERTa, which is fine-tuned on NLI datasets. For this reason, SEQA doesn't strictly conform to the zero-shot setting. Therefore, we re-ran the SEQA experiments with the original sentence-RoBERTa model. With that said, we will include the original SEQA results, which are comparable with CASE, in the paper. We will also include a version of CASE that is fine-tuned on an NLI dataset for complete comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposed an effective method called CAS, which has demonstrated its efficacy in the context of GPT-based answer selection tasks.\n2. Extensive experiments has been undertaken to validate the effectiveness of CAS, besides, it yields noteworthy enhancements in the performance of the previously established SEQA method.\n3. The presented methodology holds a broad applicability, capable of integration into diverse architectures, the intuition behind the method is sound.", "reasons_to_reject": "1. The scope of the proposed method is conventional and constrained, as it is solely applicable to answer selection tasks, without extending its utility to the enhancement of answer generation tasks.\n2. Furthermore, the applicability of the proposed method within a few-shot setting remains a pertinent consideration. The inclusion of experiments conducted within a few-shot scenario would be advisable, as it has the potential to yield practical advantages for real-world applications.\n3. Notably, GPT-3 davinci-003 serves as the basis for answer space expansion. However, it is noteworthy that a comprehensive evaluation of the performance of GPT-3 davinci-003 is absent from the reported findings.", "questions_to_authors": "I have examined the results report in the SEQA paper (https://aclanthology.org/2021.acl-long.237.pdf), but upon comparison, I have observed discrepancies between the reported results in the current paper and those presented in the SEQA paper's Table 2.", "paper_topic_and_main_contributions": "This article primarily introduces a rationality score for a zero-shot Multiple Choice Question Answering (MCQA) task based on a commonsense knowledge base, known as CASE. This score addresses the limitation of treating all words equally important by assigning importance weights to words in the input. CASE reduces the noise from less important words through dynamic weight assignment, and leverages implicit commonsense knowledge for answering questions. Furthermore, the article proposes a method to expand the answer space by generating diverse yet conceptually similar answers at the lexical level, aiming to enhance the model's performance. Across five common commonsense benchmark tests, the proposed approach outperforms baseline models. Additionally, the article demonstrates the complementarity of these two methods, especially when using a smaller language model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q6WgwIbr5n", "reviewer": "Reviewer_pNxV", "summary": "", "questions": "There is mention that few-shot and supervised sota is higher for these tasks. How much higher? Perhaps it is worth including these numbers in the appendix. Ideally, including results for GPT2 and/or GPT3.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. \n\n**The authors claim to make fair comparison to previous baselines, although I would consider using ConceptNet to be an advantage. This is most blatant when the authors the SEQA baseline that leverages NLI. Perhaps it should be explained more why using NLI is considered unfair but ConceptNet is fair game.**\n\nThanks for bringing this up, this is a fair point. Our distinction was for zero-shot vs. supervised methods, and we considered pre-training the LM on NLI datasets an unfair advantage compared to using an off-the-shelf LM. With that said, we will include the original SEQA results, which are comparable with CASE, in the paper. We will also include a version of CASE that is fine-tuned on an NLI dataset for complete comparison. \n\n**As the authors mentioned, conceptnet is somewhat limited in coverage especially compared with the training data of modern LMs. Perhaps there is a simple extension of this work that utilizes word embeddings and/or model probabilities and doesn't need to rely on a KB like Conceptnet. Even using a bag-of-words-based reweighting would be an informative baseline and/or alternative. That being said, seeing good results from conceptnet alone is encouraging.**\n\nThanks for the suggestions! Definitely, we agree there is room for improvement and acknowledge the limitations of using ConceptNet directly (i.e., lack of coverage and contextualization). We plan to work on more advanced methods in the future. \n\n**The approach works for condition probabilities of the answer, but it's not clear if it would work for conditional probability of the question.**\n\nThat’s a good question. Although it’s not immediately clear to us what would be the advantage of measuring the probability of the question instead of the answer, we will investigate this.  \n\n**There is mention that few-shot and supervised sota is higher for these tasks. How much higher? Perhaps it is worth including these numbers in the appendix. Ideally, including results for GPT2 and/or GPT3.**\n\nThat’s a good idea, we will  add them to the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. The re-weighting approach using conceptnet is clever and leads to nice results. It's also encouraging to see this is complementary to answer expansion.\n\nA2. The main experiments are on a large number of datasets (5 datasets) and includes multiple baselines. There is extensive analysis on model size and ablations for weight assignment.\n\nA3. The paper is upfront about some limitations.", "reasons_to_reject": "R1. The authors claim to make fair comparison to previous baselines, although I would consider using ConceptNet to be an advantage. This is most blatant when the authors the SEQA baseline that leverages NLI. Perhaps it should be explained more why using NLI is considered unfair but ConceptNet is fair game.\n\nR2. As the authors mentioned, conceptnet is somewhat limited in coverage especially compared with the training data of modern LMs. Perhaps there is a simple extension of this work that utilizes word embeddings and/or model probabilities and doesn't need to rely on a KB like Conceptnet. Even using a bag-of-words-based reweighting would be an informative baseline and/or alternative. That being said, seeing good results from conceptnet alone is encouraging.\n\nR3. The approach works for condition probabilities of the answer, but it's not clear if it would work for conditional probability of the question.", "questions_to_authors": "There is mention that few-shot and supervised sota is higher for these tasks. How much higher? Perhaps it is worth including these numbers in the appendix. Ideally, including results for GPT2 and/or GPT3.", "paper_topic_and_main_contributions": "The authors present a new technique for LM-based scoring, evaluated on multiple choice QA (MCQA). Typically, LM-based scoring is done by measuring the joint probability of LM-based next token probabilities (i.e. summing the log probabilities) and optionally applying length normalization. As others have demonstrated, this approach has weaknesses in the zero-shot setting especially due to bias towards specific tokens (as discussed in Holtzman et al). To address this weakness, the authors introduce a re-weighting scheme that assigns higher or lower weight to tokens in the answer based on their relevance to the question. To assign relevance weights, the authors use ConceptNet and measure the path length between concepts in the question and answer.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "NHgcpTqCP1", "reviewer": "Reviewer_8bm5", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. \n\n**First, while the proposal of a new token-level weighting method to consider the importance of different words within a sentence is effective, other zero-shot QA pipelines, such as fine-tuning models on knowledge bases or graphs, have stronger performance than the author's proposed method.**\n\nThis is a good question. While fine-tuning the LM on a knowledge base may lead to higher performance, it would be at a significant computational cost compared to our zero-shot model. Additionally, by accessing the KB during inference, our method doesn’t need to be re-trained to adapt to changes in the KB. Furthermore, our method provides interpretability. By looking at  the token-level weights (Figure 1) and the ConceptNet paths (Figures 4 and 5), we are able to interpret the model’s decision making and the knowledge that was used during the process. This would be less straightforward for LMs fine-tuned on a KB.. \n\n**Additionally, with the recent emergence of large language models that are strong QA solvers, the impact of the proposed method towards the community may be limited. While I do not reject the proposed method, I believe it is more likely to be incremental in the vanilla LM scoring pipeline compared to other zero-shot QA methods or models. It might be interesting to explore using the generated embeddings of GPT-3 to strengthen the scoring step to enhance the added value of this paper in the current NLP era.**\n\nThose are great ideas for future work!\n\n**There are some confusing notations in Table 1 and 2, where a frequently appeared baseline, LM, is not explicitly stated. I assume it is GPT-2 (from line 417), but there is no clear instructions.**\n\nGPT2 in Tables 1 and 2 is GPT-2-xlarge, as we mentioned in line 417. We will mention it explicitly in the captions of the two tables.\n\n**I appreciate the idea of answer set expansion, and it makes sense that it is effective. However, I suggest that the authors conduct a further analysis regarding the quality of the generated answers. For example, they could assess how many of the generations are plausible and how many are correct by using expert annotation on a sample of the generated datasets. Furthermore, it would be beneficial to show that sentence similarity and keyword connections are playing their expected roles when linking the generations.**\n\nThank you for recognizing our idea! We will try to include at least a small-scale annotation of the quality of generated answers in the camera-ready version.\n\nWe will fix all the typos in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed framework appears to be well thought-out, with an interesting motivation behind it. The authors have effectively integrated the score of each answer and utilized vanilla pretrained language models, including large language models such as GPT3. The experiment results demonstrate the superiority of CASE in comparison to a diverse range of zero-shot QA systems. The comparison is comprehensive, covering most of the prior methods that have been discussed. The analyses are well-designed and reflective, providing valuable insights into the performance of the proposed framework.", "reasons_to_reject": "- First, while the proposal of a new token-level weighting method to consider the importance of different words within a sentence is effective, other zero-shot QA pipelines, such as fine-tuning models on knowledge bases or graphs, have stronger performance than the author's proposed method. Additionally, with the recent emergence of large language models that are strong QA solvers, the impact of the proposed method towards the community may be limited. While I do not reject the proposed method, I believe it is more likely to be incremental in the vanilla LM scoring pipeline compared to other zero-shot QA methods or models. It might be interesting to explore using the generated embeddings of GPT-3 to strengthen the scoring step to enhance the added value of this paper in the current NLP era.\n- There are some confusing notations in Table 1 and 2, where a frequently appeared baseline, LM, is not explicitly stated. I assume it is GPT-2 (from line 417), but there is no clear instructions.\n- I appreciate the idea of answer set expansion, and it makes sense that it is effective. However, I suggest that the authors conduct a further analysis regarding the quality of the generated answers. For example, they could assess how many of the generations are plausible and how many are correct by using expert annotation on a sample of the generated datasets. Furthermore, it would be beneficial to show that sentence similarity and keyword connections are playing their expected roles when linking the generations.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main contributions of this paper are the introduction of CASE, a Commonsense-Augmented Score with an Expanded Answer Space, which improves the performance of language models on multiple-choice QA tasks. CASE reduces noise from unimportant words and informs the model of implicit commonsense knowledge that may be useful for answering the question. Additionally, by generating lexically divergent answers that are conceptually similar to the choices, CASE expands the answer space and outperforms strong baselines on 5 commonsense benchmarks. The main contribution type of this paper lies in the scope of NLP Engineering experiments.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Relevant zero-shot CSQA methods:\n[1] Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering, 2020\n[2] Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization, 2021\n[3] CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering, 2023 (optional, as it's contemporaneous)", "Typos_Grammar_Style_and_Presentation_Improvements": "In the final sentence of the abstract, the author states, \"when using smaller LLMs.\" I believe it would be more appropriate to say, \"when using smaller LMs.\" Additionally, in the experimental section, the author employs different terms to refer to the same approach, using \"answer expansion\" and \"candidate generation\" interchangeably. This inconsistency may lead to confusion during the initial reading."}}]

Author rebuttal:

Meta-review: The paper tackles the issue of answer scoring based on LM probabilities for common-sense related tasks. The main idea is to use a weighting sum of answer token probabilities, where the weights are derived by path-based scores calculated from ConceptNet. The answer set is also expanded to allow for better scoring. 

The problem and solution are well-motivated. The main issue of obtaining answer probabilities via an aggregation of token probabilities is an important one. The proposed method of finding weights on the answer token probabilities is reasonable.  The experimental evidence is mostly sound with some minor caveats. The experiments were are all done on GPT-2 a relatively weaker model compared more recent ones (e.g. T5, UnifiedQA) of similar sizes. The paper uses ConceptNet as extra information during answer scoring. A tighter claim on answer token weighting would have been to include ConceptNet paths via other means in the model (e.g. as inputs to the model).  There are adequate analyses of the components in the paper. The writing is clear overall. 

The work has significance within the scope of selection technique in common-sense QA tasks. The main contribution is in showing that weighting token probabilities helps for answer selection. This is a neat idea that is relatively simple to implement and use. Another plus for significance is that it adds to methods that tackle calibration of LM probabilities, which allow us to get more out of LMs when used as black-boxes.

I am less excited about the broader significance of the work for the following reason. The overall idea seems similar to rescoring methods in generation tasks, where a base LM generates multiple candidates and some scoring method rescores the outputs, which can also be integrated into the beam search. It is not clear if there was some new challenge or twist to address in implementing the idea for answer selection.
Decision: Accept-Findings