BUCA: A Binary Classification Approach to Unsupervised Commonsense Question Answering
Year: 2023

Abstract:
Unsupervised commonsense reasoning (UCR) is becoming increasingly popular as the construction of commonsense reasoning datasets is expensive, and they are inevitably limited in their scope. A popular approach to UCR is to fine-tune language models with external knowledge (e.g., knowledge graphs), but this usually requires a large number of training examples. In this paper, we propose to transform the downstream multiple choice question answering task into a simpler binary classification task by ranking all candidate answers according to their reasonableness. To this end, for training the model, we convert the knowledge graph triples into reasonable and unreasonable texts. Extensive experimental results show the effectiveness of our approach on various multiple choice question answering benchmarks. Furthermore, compared with existing UCR approaches using KGs, ours is less data hungry. Our code is available at https://github.com/probe2/BUCA

heading: 1 Introduction
text: Commonsense reasoning has recently received significant attention in NLP research (Bhargava and Ng, 2022), with a vast amount of datasets now available (Levesque, 2011; Gordon et al., 2012; Sap et al., 2019; Rashkin et al., 2018; Bisk et al., 2020; Talmor et al., 2019). Most existing methods for commonsense reasoning either fine-tune large language models (LMs) on these datasets (Lourie et al., 2021) or use knowledge graphs (KGs) (Pan et al., 2017) to train LMs (Liu et al., 2019a; Yasunaga et al., 2022). However, it is not always possible to have relevant training data available, it is thus crucial to develop unsupervised approaches to commonsense reasoning that do not rely on labeled data.
In this paper, we focus on the unsupervised multiple choice question answering (QA) task: given a question and a set of answer options, the model is expected to predict the most likely option. We
propose BUCA, a binary classification framework for unsupervised commonsense QA. Our method roughly works as follows: we first convert knowledge graph triples into textual form using manually written templates, and generate positive and negative question-answer pairs. We then fine-tune a pretrained language model, and leverage contrastive learning to increase the ability to distinguish reasonable from unreasonable ones. Finally, we input each question and all options of the downstream commonsense QA task into BUCA to obtain the reasonableness scores and select the answer with the highest reasonableness score as the predicted answer. Experimental results on various commonsense reasoning benchmarks show the effectiveness of our proposed BUCA framework. Our main contributions are:
• We propose a binary classification approach to using KGs for unsupervised commonsense question answering.
• We conduct extensive experiments, showing the effectiveness of our approach by using much less data.

heading: 2 Related work
text: Language models are widely used in unsupervised commonsense inference tasks, e.g. as an additional knowledge source or as a scoring model. Rajani
376
et al. (2019) propose an explanation generation model for the CommonsenseQA dataset. Self-talk (Shwartz et al., 2020) uses prompts to stimulate GPT and generate new knowledge. SEQA (Niu et al., 2021) generates several candidate answers using GPT2 and then ranks each them.
Another research direction in unsupervised commonsense reasoning is the use of e.g. commonsense KGs (Speer et al., 2016; Romero et al., 2019; Malaviya et al., 2020) to train the model (Chen et al., 2021; Geng et al., 2023). In Banerjee and Baral (2020), given the inputs of context, question and answer, the model learns to generate one of the inputs given the other two. Ma et al. (2021) update the model with a margin ranking loss computed on positive and negative examples from KGs. MICO (Su et al., 2022) uses the distance between the positive and negative question-answer pairs obtained from the KG to calculate the loss. However, all of the above approaches demand a large amount of training data, sometimes reaching million of training samples, while BUCA only needs tens of thousands, cf. Table 2. The most similar to our work is NLI-KB (Huang et al., 2021), which trains a model on NLI data, then applies the corresponding knowledge to each question-answer pair on the downstream task. Our paper, instead, shows that is not the NLI data but the retrieved knowledge that helps.

heading: 3 Methodology
text: We focus on the following multiple choice question answering (QA) task: given a question q and a set of options A, the model should select the most likely single answer Ai ∈ A. We consider an unsupervised setting in which the model does not have access to the training or validation data. Our BUCA approach first trains the model with a knowledge graph and then uses the trained model to test on multiple QA downstream tasks. Formally, a knowledge graph (KG) (Pan et al., 2017) G is a tuple (V,R, T ), where V is a set of entities, E is a set of relation types and T is a set of triples of the form (h, r, t) with h, t ∈ V the head and tail entities and r ∈ R the relation of the triple connecting h and t.
Our approach has three main components: knowledge graph transfer to training data, training loss design, and downstream task testing:
Converting Triples into Binary Classification Training Data. Inspired by previous work (Su
et al., 2022), each KG triple is converted into question-answer pairs by using pre-defined templates, so that the obtained pairs are then used as the input of the classification task. We use the templates provided in (Hwang et al., 2020). For example, the ATOMIC triple (PersonX thanks PersonY afterwards, isAfter, PersonX asked PersonY for help on her homework) can be converted to “After PersonX asked PersonY for help on her homework, PersonX thanks PersonY afterwards”. In the appendix we show the distribution of the converted sequence pairs. Along with the correct QA pairs created from the KG triples, our framework is also trained on negative QA pairs, so it can better discriminate between reasonable and unreasonable QA pairs. More precisely, in the training dataset, each correct QA pair generated from a triple tp = (h, r, t) has a corresponding negative pair obtained from a variation of tp in which t is substituted by t′, which is randomly drawn from the existing tails in the KG.
Training Loss. For our binary classification model, we add a classification head with two nodes to the pre-trained language model. After normalizing the values on these two nodes, we can obtain reasonable and unreasonable scores for the QA pairs. From the triple conversion step, we obtained n training examples, each consisting of a question q, correct answer ac, and incorrect answer aw. For each question-answer pair, we can then obtain the reasonable and unreasonable scores r+i and r − i after applying a softmax layer. In each loss calculation, we jointly consider the correct and incorrect answers. For binary classification, we use two kinds of losses: Traditional Binary Loss (TBL).
L = − n∑
i=1
(log(p+ac) + log(p − aw))
where p+ac and p − aw are the probabilities of correct and incorrect answers, respectively corresponding to reasonable and unreasonable scores.

heading: Margin Ranking Loss.
text: L = n∑
i=1
max(0, η − log(p+ac) + log(p+aw))
+max(0, η − log(p−aw) + log(p−ac))
where η is a margin threshold hyper-parameter. In order to pull the representational distance between reasonable question-answer pairs as close as possible and to push the representational distance
between reasonable and unreasonable ones as far as possible, we use supervised contrastive learning (Gunel et al., 2021) along with the binary classification. This is done by considering as positive examples of a given example within a category, all those examples within the same category. Contrastive Loss of the i-th QA pair
Lscl = N∑
j=1
1yi=yj log esim(hj ,hi)τ
∑N k=1 1i ̸=ke sim(hk,hi)/τ
where τ is the temperature parameter and h denotes the feature vector.
Inference. In the prediction phase for each candidate answer, we calculate its reasonableness score. We choose the answer with the highest reasonableness score as the predicted answer.

heading: 4 Experiments
text: In this section, we first describe our experiments on five commonsense question answering datasets, followed by ablation studies and data analysis.

heading: 4.1 Datasets and Baselines
text: We use two well-known commonsense KGs for training our framework: ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2018). For evaluation, we use five commonsense QA datasets: COPA (Gordon et al., 2012), OpenBookQA (Mihaylov et al., 2018), SIQA (Sap et al., 2019), CSQA (Talmor et al., 2019), and SCT (Mostafazadeh et al., 2017), covering a wide range of topics within commonsense reasoning. We compare our approach with various baselines:
RoBERTa-Large (Liu et al., 2019b), GPT2 (Radford et al., 2019), Self-talk (Shwartz et al., 2020), Dou (Dou and Peng, 2022), Wang (Wang and Zhao, 2022) and other unsupervised systems using KGs: SMLM (Banerjee and Baral, 2020), MICO (Su et al., 2022), NLI-KB (Huang et al., 2021) and Ma (Ma et al., 2021). Most reported results are collected from the literature. For NLI-KB, we used their publicly available code to get the results.
Details of the KGs and datasets, as well as implementation details, can be found in the appendix.

heading: 4.2 Main results
text: Table 1 shows the results for the five benchmarks. Overall, BUCA achieves the best performance on all datasets. More precisely, our results respectively outperform baselines on the validation and test sets as follows: MICO by 6.8% and 13.2% on COPA; Dou by 4.2% and 7.8% on OpenbookQA. We also outperform MICO by 5.4% on SIQA; NLIKB by 13.3% on CSQA, and NLI-KB by 16.3% on SCT. Ma does not provide results for COPA,
OpenBookQA and SCT, but it achieves state-ofthe-art results on CSQA 67.4 and on SIQA 63.2, while BUCA’s best results respectively are 65.4 and 61.4. However, Ma uses multiple KGs to train a single model, ConceptNet, WordNet, and Wikidata for CSQA and ATOMIC, ConceptNet, WordNet, and Wikidata for SIQA, with a total training data of 662,909 and 1,197,742, while BUCA only uses 65,536 and 61,530, cf. Table 2. Considering the difference on used training data and the closeness of results, BUCA’s approach clearly demonstrates its effectiveness. We can also observe the same trend as in MICO: ConceptNet is more helpful for CSQA and ATOMIC is more helpful for SIQA. This is explained by the fact that SIQA is built based on ATOMIC and CSQA is built based on ConceptNet. On other datasets our framework shows similiar behavior with both KGs. As for the loss functions, the margin ranking loss is on average 0.8% higher than the binary loss on ConceptNet, and 0.1% higher on ATOMIC. These results are explained by the fact that the ranking loss separates more the scores between reasonable and unreasonable answers. In light of this, we will only consider margin ranking loss in the below analysis.

heading: 4.3 Ablation Studies
text: In this section, we analyze the effects of the backbone models, the effect of contrastive learning, and explore the vocabulary overlap between the knowledge training set and the downstream task as well as the accuracy of our BUCA method.
Backbone Pre-trained LMs Our experiments using different backbone models show that in general the stronger the PLM the better the perfor-
mance on the downstream task. Regarding the KGs, in the BERT-base and RoBERTa-base variants, the ATOMIC-trained models perform better than the ConceptNet-trained models, while in the RoBERTa-large one they perform similarly. This might be explained by the fact that as the model capacity increases it has more inherently available event-like commonsense knowledge, necessary in the ATOMIC-based datasets. Detailed results are shown in Table 3.
Effects of Contrastive Learning Our experiments show that the RoBERTa-large variant with contrastive learning outperforms the version without it on all datasets, regardless of the used KG. Detailed results are shown in Table 4.
Accuracy of the Binary Classifier Inspired by Ghosal et al. (2022), we evaluate how often input sequences corresponding to correct and incorrect answers are accurately predicted. To this end, we use the RoBERTa-large variant trained on ATOMIC. Table 5 shows that our model tends to predict all answers as reasonable since in our training set the negative examples are randomly selected, many QA pairs are semantically irrelevant or even ungrammatical. For the manually crafted candidate answers, many of them are semantically relevant and grammatical, so our model predicts them as reasonable. We also see that the accuracy metrics for SCT and COPA are the highest. Our findings are consistent with Ghosal et al. (2022).

heading: 4.4 Data Analysis
text: To better understand why transfer learning from CKGs is more suitable than from other datasets
(i.e. MNLI or QNLI) in the commonsense QA task, we performed an analysis on the training data in NLI-KB (Huang et al., 2021) and the used CKGs. Following (Mishra et al., 2021), we first compare the vocabulary overlap of ConceptNet, ATOMIC and MNLI (training data) with our evaluation QA datasets. We follow the definition of overlap introduced in (Mishra et al., 2021). Table 6 shows that MNLI has higher vocabulary overlap with all the evaluation datasets than both used CKGs. However, the results for NLI-KB in Table 1 show that the vocabulary overlap is not a key factor for performance as otherwise, NLI-KB fine-tuned with the NLI datasets (before injecting knowledge) should perform better that the other models in the downstream task due to the high lexical similarity.
We also analyze the distance to the sentence embeddings. Our results show that the MNLI entries performed poorly in commonsense knowledge
retrieval for SIQA-queries as they are not reasonable answers. In contrast, the sentences generated from ATOMIC and ConceptNet successfully pair the SIQA-questions with reasonable answers. This reveals that, although MNLI has a higher lexical coverage, MNLI does not have suitable examples to match SIQA questions. Thus models fine-tuned with the NLI dataset hardly get any benefit for downstream commonsense reasoning tasks. Tables 7 and 8 present a random sample showing this, where reasonable alternatives are in bold.

heading: 5 Conclusion
text: We presented a framework converting KGs into positive/negative question-answer pairs to train a binary classification model, discriminating whether a sentence is reasonable. Extensive experiments show the effectiveness of our approach, while using a reasonably small amount of data. For future work, we will explore how to better select negative cases.

heading: Limitations
text: The method to select negative examples could be improved, as randomly selecting negative examples for training might lead to identifying most of examples in the evaluation datasets as reasonable. Secondly, we did not explore using other number of candidates in the training set, we always use 2 candidate answers for each question.

heading: Acknowledgments
text: This work is supported by the Chang Jiang Scholars Program (J2019032).

heading: Appendix

heading: A KGs, Datasets, and Implementation
text: This section contains more experimental details. In particular, we give details of the used KGs and datasets. We also discuss implementation details.

heading: ConceptNet
text: ConceptNet (Speer et al., 2017) is a traditional KG that focuses on taxonomic, lexical and physical relations (e.g., IsA, RelatedTo, PartOf ). In our experiment, we employed the CN-82K version which is uniformly sampled from a larger set of extracted ConceptNet entity-relations (Li et al., 2016).

heading: ATOMIC
text: The ATOMIC KG (Sap et al., 2018) focuses on social-interaction knowledge about everyday events, and thus has a higher coverage in the field of commonsense query answering. It consists of 880K knowledge triples across 9 relations (e.g. xNeed, oEffect, xReact). This includes mentions of topics such as causes and effects, personal feelings toward actions or events, and conditional statements. The ATOMIC dataset is collected and validated completely through crowdsourcing.
As seen in Table 2, in comparison to related works: Ma (Ma et al., 2021) and MICO (Su et al., 2022), our methods used much less data from the CKGs (~5-8x Ma, ~2-20x MICO) while still maintaining competitive performance on the evaluation dataset.

heading: A.1 Generation of QA pairs
text: The QA pairs were generated using the templates in the ATOMIC paper (Hwang et al., 2020), which is compatible with relations in both ConceptNet and ATOMIC. These templates help to convert KG triples into natural sentences, examples shown in Table 9. The head entity and mapped relation phrases are joined as a question. The correct tail entity and a randomly sampled tail from the dataset are used as the positive and negative answers, respectively, for contrastive learning.

heading: A.2 Evaluation Datasets
text: We evaluate our framework using five downstream QA tasks: COPA, OpenBookQA, SIQA, CSQA, and SCT, which covere a wide range of topics within commonsense reasoning. Accuracy is used
as the evaluation metric. All experiments are perform in an unsupervised setting, where our model are not train on the source task.
Choice of Plausible Alternatives (COPA) (Gordon et al., 2012) is a two-choice question-answer dataset designed to evaluate performance in opendomain commonsense causal reasoning. Each entry contains a premise and two possible answers, the task is to select the answers that most likely have a causal relationship with the premise. The dataset consists 500 questions for both debvelopment and test sets.
OpenBookQA (Mihaylov et al., 2018) is inspired from open book exams that assess human understanding in real life. This QA task requires a deeper understanding about both open book facts (e.g., metals is a heat conductor) and a broad common knowledge (e.g., a steal spoon is made of metal) to answer questions like: Which of these objects conducts the most heat: A metal spoon, pair of jeans, or cotton made clothing? It contains 500 multiple-choice science questions for both development and test sets.
SocialIQA (SIQA) (Sap et al., 2019) contains multiple-choice questions with topics concerned with emotional and social interactions in a variety of everyday situations. Each entry comes with a context, a question, and 3 candidate answers. The questions are generated using the ATOMIC KG by converting triples into question sentences using predefined templates, and the answers are crowdsourced. The dataset’s development split is used as evaluation dataset, containing 1,954 questions.
CommonsenseQA (CSQA) (Talmor et al., 2019) contains questions focused on various commonsense aspects. Each entry contains a question and five candidate answers. The questions are constructed by crowd workers. The answer candidates include distractors comprised of hand-picked ones or nodes from ConceptNet. The development set is used as evaluation set, containing 1,221 questions.
Story Cloze Test (SCT) (Mostafazadeh et al., 2017) is a LSDSem’17 shared task, evaluating story understanding and script learning. Each entry contains a four-sentence story and two possible fifth sentences, where the model has to pick the most suitable ending for the story. The development set is used as the evaluation set, containing 1572 different stories.
A.3 Implementation details Our experiments are run on a single A100 GPU card. We use RoBERTa-Large as our backbone model. The training batch size is 196, and the maximal sequence length for training is 64. The learning rate is set to 5e-5 for all experiments. For experiments with the margin ranking loss, η is set to 1. The validation set is evaluated by accuracy and used to select a best model for further evaluation. The models are trained for 20 epochs and early stopped when the change of validation loss is within 1%.

heading: B Ablation Studies
text: We present the full results for the ablation studies discussed in Section 4.3. Table 3 for the backbone models study; Table 4 for the influence of contrastive learning; and Table 5 for accuracy.

heading: C Data Analysis
text: In the analysis of the distance to sentence embeddings, we treat each entry in the CKG datasets as possible answers and encode them using the SBERT pre-trained model (all-mpnet-basev2) (Reimers and Gurevych, 2019, 2020). Then, the cosine-similarity between the SIQA question and the encoded sentences is calculated to rank their semantic relatedness.
We retrieved the top 3 answers for each source and listed by similarity score at descending order. Table 10 extends the results presented in Section 4.4; Table 11 show the alternative answers from CKG datasets COPA questions.
SIQA Example Question: After a long grueling semester, Tracy took the final exam and finished their course today. Now they would graduate. Why did Tracy do this? Answer: complete their degree on time
MNLI
Because I had a deadline. This entails I had to finish by that time. The professors went home feeling that history had been made. This entails The professors returned home. They got married after his first year of law school.This entails Their marriage took place after he finished his first year of law school.
ATOMIC Tracy wants finish before time expires. because Tracy takes the exam Tracy wanted to get a degree. as a result Tracy finishes Tracy’s test Tracy graduates with a degree. but before, Tracy needed get pass with good marks.
ConceptNet pass class causes graduation study ends with the event or action graduate graduation because take final exam
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Section Appendix

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section Appendix
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section Appendix
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section Appendix
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we will explore how to better select negative cases