\documentclass[11pt,a4paper]{article}
\usepackage{acl}
\usepackage{times}
\usepackage{latexsym}
\usepackage[TABBOTCAP]{subfigure}
\usepackage[shortlabels]{enumitem}

\usepackage{tikz-dependency}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{multirow}

\usepackage{color}
\usepackage{helvet}
\usepackage{textcomp}
\usepackage{graphicx}
\graphicspath{ {images/} }
\usepackage{amsmath}
\usepackage{float}
\usepackage{booktabs,amsfonts,dcolumn}
\usepackage{hyperref}
\usepackage{url}

\usepackage[]{collab}
\collabAuthor{yt}{teal}{Yintong Huo}

\def\AM{{\mathcal A}}
\def\BM{{\mathcal B}}
\def\CM{{\mathcal C}}
\def\DM{{\mathcal D}}
\def\EM{{\mathcal E}}
\def\FM{{\mathcal F}}
\def\GM{{\mathcal G}}
\def\HM{{\mathcal H}}
\def\IM{{\mathcal I}}
\def\JM{{\mathcal J}}
\def\KM{{\mathcal K}}
\def\LM{{\mathcal L}}
\def\MM{{\mathcal M}}
\def\NM{{\mathcal N}}
\def\OM{{\mathcal O}}
\def\PM{{\mathcal P}}
\def\SM{{\mathcal S}}
\def\RM{{\mathcal R}}
\def\TM{{\mathcal T}}
\def\UM{{\mathcal U}}
\def\VM{{\mathcal V}}
\def\WM{{\mathcal W}}
\def\XM{{\mathcal X}}
\def\YM{{\mathcal Y}}
\def\ZM{{\mathcal Z}}
\def\ZB{{\mathbb Z}}
\def\RB{{\mathbb R}}

\def\A{{\bf A}}
\def\a{{\bf a}}
\def\B{{\bf B}}
\def\b{{\bf b}}
\def\C{{\bf C}}
\def\c{{\bf c}}
\def\D{{\bf D}}
\def\d{{\bf d}}
\def\E{{\bf E}}
\def\e{{\bf e}}
\def\f{{\bf f}}
\def\G{{\bf G}}
\def\H{{\bf H}}
\def\I{{\bf I}}
\def\k{{\bf k}}
\def\o{{\bf o}}
\def\K{{\bf K}}
\def\L{{\bf L}}
\def\M{{\bf M}}
\def\m{{\bf m}}
\def\n{{\bf n}}
\def\p{{\bf p}}
\def\Q{{\bf Q}}
\def\q{{\bf q}}
\def\R{{\bf R}}
\def\S{{\bf S}}
\def\s{{\bf s}}
\def\T{{\bf T}}
\def\U{{\bf U}}
\def\u{{\bf u}}
\def\V{{\bf V}}
\def\v{{\bf v}}
\def\W{{\bf W}}
\def\w{{\bf w}}
\def\X{{\bf X}}
\def\x{{\bf x}}
\def\Y{{\bf Y}}
\def\y{{\bf y}}
\def\Z{{\bf Z}}
\def\z{{\bf z}}
\def\0{{\bf 0}}
\def\1{{\bf 1}}

\def\name{{\bf CIKQA}}

\usepackage{xcolor}
\usepackage{soul}
\newcommand{\hlc}[2][yellow]{{%
    \colorlet{foo}{#1}%
    \sethlcolor{foo}\hl{#2}}%
}

\newcommand{\Red}[1]{\textcolor[rgb]{1.00,0.00,0.00}{#1}}
\newcommand{\Blue}[1]{\textcolor[rgb]{0.00,0.00,1.00}{#1}}
\newcommand{\Green}[1]{\textcolor[rgb]{0.00,0.80,0.00}{#1}}
\newcommand{\Black}[1]{\textcolor[rgb]{0.00,0.00,0.00}{#1}}
\newcommand{\Purple}[1]{\textcolor[rgb]{0.6,0.3,0.9}{#1}}
\newcommand{\Cyan}[1]{\textcolor[rgb]{0.039,0.72,0.71}{#1}}

\newcommand{\reviseyq}[1]{\Red{#1}}
\newcommand{\yqc}[1]{\textcolor{red}{[YQ: #1]}}
\newcommand{\yq}[1]{\textcolor{red}{#1}}

\newcommand{\revisehm}[1]{\Blue{#1}}
\newcommand{\reviseyt}[1]{[\Cyan{#1}]}
\newcommand{\xr}[1]{[\Green{xr: #1}]}
\newcommand{\ye}[1]{\textcolor{purple}{Yanai: #1}}

\def\aclpaperid{*} %
\newcommand\BibTeX{B\textsc{ib}\TeX}

\title{CIKQA: Learning Commonsense Inference with a Unified \\ Knowledge-in-the-loop QA Paradigm
}

\author{Hongming Zhang$^{1,2}$, Yintong Huo$^3$, Yanai Elazar$^{4,5}$, Yangqiu Song$^1$, Yoav Goldberg$^{4,5}$, Dan Roth$^2$\\
$^1$HKUST, $^2$UPenn, $^3$CUHK, $^4$AI2, $^5$University of Washington, $^6$Bar Ilan University\\
\texttt{\{hzhangal,yqsong\}@cse.ust.hk}, \texttt{ythuo@cse.cuhk.edu.hk} \\
  \texttt{\{yanaiela,yoav.goldberg\}@gmail.com}, \texttt{danroth@seas.upenn.edu}}
\date{}

\begin{document}
\maketitle
\begin{abstract}

Recently, the community has achieved substantial progress on many commonsense reasoning benchmarks. However, it is still unclear what is learned from the training process: the knowledge, inference capability, or both? 
We argue that due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task to cover all commonsense for learning. Thus we should separate the commonsense knowledge acquisition and inference over commonsense knowledge as two separate tasks. 
In this work, we focus on investigating models' commonsense inference capabilities from two perspectives: (1) Whether models can know if the knowledge they have is enough to solve the task; 
(2) Whether models can develop commonsense inference capabilities that generalize across commonsense tasks.
We first align commonsense tasks with relevant knowledge from commonsense knowledge bases and ask humans to annotate whether the knowledge is enough or not.
Then, we convert different commonsense tasks into a unified question answering format to evaluate models' generalization capabilities.
We name the benchmark as Commonsense Inference with Knowledge-in-the-loop Question Answering (\name). 
\end{abstract}

\section{Introduction}\label{sec-introduction}
\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{figure/CIKQA-intro-demo.png}
    \caption{\name~ demonstration. All tasks are converted into a unified format such that we could easily evaluate the generlization capability of all models. We also equip all questions with auto-extracted knowledge graphs from existing KGs and ask humans to annotate whether the knowledge is gold or not. In this example, we expect models to first identify the quality of the knowledge and then conduct inference over the knowledge to solve the question. }
    \label{fig:intro_demo}
    \vspace{-0.2in}
\end{figure*}
Understanding human language requires both the language knowledge (e.g., grammar and semantics) and world knowledge, which can be further divided into factual and commonsense knowledge \cite{Katz1963-KATTSO-3}.
Recently, the community has made great progress on helping machines acquire and apply language and factual knowledge. 
However, how to help machines acquire and infer over commonsense is still unclear. 
To answer this question, many commonsense reasoning datasets~\cite{DBLP:conf/aaaiss/RoemmeleBG11,DBLP:conf/aaai/SakaguchiBBC20,DBLP:conf/naacl/TalmorHLB19,DBLP:conf/cvpr/ZellersBFC19,DBLP:conf/emnlp/LinLKR20} have been proposed. Even though they target different knowledge types, modalities, and come in different formats, they often follow a standard supervised learning setting, which aims at helping machines to solve a specific task with the training data.
However, two limitations of this learning paradigm have restricted the development of commonsense reasoning systems.

First, there is no clear separation between knowledge and inference. As discussed in~\cite{DBLP:journals/corr/abs-2104-08161}, a common phenomenon is that larger training data will lead to better performance, mainly because richer knowledge is covered. However, due to the large scale of commonsense knowledge, it is infeasible to annotate a large enough training set for each task, and the responsibility of the training data should be teaching models how to do inference rather than acquire the commonsense knowledge. 
Several recent works have explored using structured knowledge for commonsense reasoning tasks~\cite{DBLP:conf/emnlp/LinCCR19,DBLP:conf/aaai/LvGXTDGSJCH20,DBLP:conf/emnlp/PaulF20}. However, as these works did not clearly analyze the coverage of the structured knowledge (i.e., knowledge graphs (KGs)), it is still unclear what the performance means, better knowledge coverage or better inference capability. 
To dig into what is behind this learning process, we propose to equip each question with auto-extracted knowledge and ask humans to annotate whether the knowledge is gold (i.e., sufficient to answer the question).
By doing so, we could evaluate whether models can know if the provided knowledge is gold or not and how well they can conduct inference over the provided knowledge to solve the task.

Second, the supervised learning may force the model to learn the distribution of the training data rather than a universal inference model. As a result, the model may perform well on the test set that follows the same distribution but fail on other tasks~\cite{DBLP:journals/corr/abs-2011-09159}.
Previously, as different tasks have different formats, it is hard to evaluate the generalization ability of commonsense reasoning models.
Motivated by the existing trend of using a unified format (i.e., question answering) for different tasks~\cite{DBLP:conf/emnlp/KhashabiMKSTCH20}, we propose to convert various commonsense reasoning tasks into a unified QA format such that we can easily and fairly evaluate the generalization ability of learned commonsense reasoning models.

 
Combining these two lines of effort, we propose a new commonsense inference evaluation benchmark Knowledge-in-the-loop Commonsense Inference with QA (\name).
An example is shown in Figure~\ref{fig:intro_demo}. We first convert several popular commonsense reasoning tasks into a unified QA format and equip them with the relevant knowledge from existing commonsense knowledge graphs.
We leverage human annotation to label whether the provided knowledge is gold to answer the question.
With \name, we are interested in answering two questions: (1) Whether current models can distinguish the knowledge is gold or not; 
(3) Can current commonsense inference models generalize across different commonsense reasoning tasks.

Experiments with several recent knowledge-based commonsense reasoning models show that even though current deep models could learn to conduct simple inference after training with a few examples when gold knowledge is provided, they still cannot learn to distinguish gold knowledge very well.
Moreover, even though current models demonstrate an encouraging generalization ability across the three tasks we consider, they still cannot learn complex inference (e.g., abductive reasoning) very well.
We hope that our benchmark\footnote{Available at https://github.com/CogComp/CIKQA.} can motivate more advanced commonsense inference methods in the future.
 
\section{Dataset Construction}\label{sec:definition}

In \name, to encourage a generalizable commonsense inference model, we follow previous work~\cite{DBLP:conf/emnlp/KhashabiMKSTCH20,DBLP:journals/corr/abs-2010-04829,DBLP:conf/acl/WuWYWL20,DBLP:conf/emnlp/DuC20} to unify all selected tasks as a binary question answering problem, and equip each question with a supporting knowledge graph $G$ retrieved from existing commonsense KGs.
We leverage crowd-sourcing workers to annotate whether the knowledge is gold (i.e., accurate and enough) for answering the question.
Details about task selection, format unification, support knowledge extraction, and annotation are as follows.

\begin{table*}[t]
    \small
    \centering
    \begin{tabular}{l||p{4.0cm}|p{4.5cm}|p{3.5cm}}
    \toprule
        Task Name & Original Assertion & Transformed Question &  Answer \\
    \midrule
       HardPCR  & The fish ate the worm. It was hungry. & The fish ate the worm. It was hungry. What was hungry? & {(A) \Blue{Fish}; (B) \Red{Worm}} \\
       \hline
       CommonsenesQA  & What is a place that someone can go buy a teddy bear? & What is a place that someone can go buy a teddy bear? & (A) \Blue{Toy store}; (B) \Red{Shelf}\\
       \hline
       COPA  & I drank from the water fountain. & I drank from the water fountain. What was the cause of this? & (A) \Blue{I was thirsty.}; (B) \Red{I felt nauseous.} \\
       \hline
       ATOMIC & PersonX buys the bike. & Before PersonX buys the bike, what did PersonX want? & (A) \Red{To be social.}; (B) \Blue{To have transportation.}\\
    \bottomrule
    \end{tabular}
    \caption{Demonstration of the original assertion, transformed questions, and answers. Correct and wrong answers are indicated with blue and red, respectively.}
    \vspace{-0.1in}
    \label{tab:Commonsense_Task_Demonstration}
\end{table*}

\subsection{Task Selection}\label{sec:task_selection}
In \name, we select the following four popular commonsense reasoning tasks:
\begin{enumerate}[leftmargin=*]
        \item HardPCR~\cite{DBLP:journals/corr/abs-2009-12721}: The hard pronoun coreference resolution (HardPCR) task is one of the most famous commonsense reasoning tasks. For each question, a target pronoun and two candidate mentions are provided, and the task is to select the correct mention that the pronoun refers to. Careful expert annotations are conducted to get rid of the influence of all simple linguistic rules and the models are required to solve the problem with commonsense reasoning. In \name, we include instances from WSC~\cite{levesque2012winograd}, DPR~\cite{DBLP:conf/emnlp/RahmanN12}, and WinoGrande~\cite{DBLP:conf/aaai/SakaguchiBBC20}. 
        To create a question regarding the target pronoun, we first find the sentence that contains the target pronoun and then determine whether the participating pronoun refers to a person or an object.%

    \item CommonsenseQA~\cite{DBLP:conf/naacl/TalmorHLB19}: CommonsenseQA is a commonsense question answering dataset. For each question-answer pair, four relevant but wrong concepts are used as the other candidates, and the models are required to select the correct one out of five candidates. In \name, we randomly sample a negative answer to make it a binary choice task, which is consistent with other datasets. 
    \item COPA~\cite{DBLP:conf/aaaiss/RoemmeleBG11}: COPA focuses on evaluating the understanding of events causality. For a target event, two candidate followup events are provided, and models are asked to predict the one caused by or the reason for the target event.
    
    \item ATOMIC~\cite{sap2019atomic}: The last one is the commonsense knowledge base completion. Given a head concept (e.g., ``eat food'') and a relation (e.g., ``cause''), we want to predict the tail concept. In \name, we focus on predicting edges of ATOMIC. 
\end{enumerate}

In COPA and ATOMIC, where the task is to predict the relations between two events or states (e.g., ``PersonX eats''-\textit{Causes}-``PersonX is full''), for each triplet, we randomly sample another event or state as the negative tail and ask the model to select the correct one. 
To make the task challenging and avoid sampling irrelevant events or states, we require the sampled negative event or state to be connected with the head event or state with a different triplet (e.g., ``PersonX is hungry'' from the triplet ``PersonX eats''-\textit{CausedBy}-``PersonX is hungry''). 
For each type of relation, we write a pattern to generate the question. For example, for the ``Causes'' relation, we will ask ``What can be caused by `PersonX eats'?''. 
Examples of instances in the original datasets and their transformed questions and candidate answers are presented in Table~\ref{tab:Commonsense_Task_Demonstration}. 
\subsection{Supporting Knowledge Extraction}\label{sec:knowledge_extraction}
As discussed in Section~\ref{sec-introduction}, a limitation of existing commonsense reasoning benchmarks is that there is no clear boundary between knowledge and inference. As such, it is unclear what is learned from the training data, the knowledge, or how to perform inference.
To address this issue and encourage models to learn inference rather than knowledge from the training data, we propose to equip each question with supporting knowledge. 
The question is selected as part of the dataset only if we find supporting knowledge to answer the question.
Note that this procedure serves as an improved evaluation setup than pure supervised learning, and not as a solution to commonsense reasoning.
This section introduces the selected commonsense knowledge graphs and then introduces how we extract the corresponding commonsense knowledge for each question.

\subsubsection{Commonsense KG Selection}

Many commonsense knowledge graphs were developed to enhance machines' commonsense reasoning abilities, including ConceptNet~\cite{liu2004conceptnet}, ATOMIC~\cite{sap2019atomic}, GLUCOSE~\cite{mostafazadeh-etal-2020-glucose}, and ASER~\cite{zhang2019aser}.
Among these four, ConceptNet, ATOMIC, and GLUCOSE were constructed via crowd-sourcing while ASER was constructed automatically with information extraction techniques.
Besides ATOMIC, which is used as one of the tasks, we use the other KBs as supporting knowledge resources.
\subsubsection{Supporting Graph Extraction}
Here we introduce how to extract the supporting knowledge from external commonsense knowledge bases.
For each question, we need to obtain a sub-graph from supporting knowledge graphs such that it contains the relevant commonsense knowledge about the question. The sub-graph extraction process includes the following three steps: (1) Pre-processing: Convert each question into several key sentences; (2) Matching: Match the sentences into nodes in the KG; (3) Extraction: Retrieve the relevant sub-graphs from the KG. 
\noindent \textbf{Data Pre-processing}: For each question and the associated candidate answers, we first replace the question words (e.g., ``What'') with the two candidate answers such that it becomes two declarative sentences.
For instance, if the question is ``The fish ate the worm. It was hungry. Who is hungry?'' and the candidates are ``Fish'' and ``Worm,'' we will convert the question into the declarative sentence: ``The fish is hungry'' and ``The worm is hungry.'' 
As a result, we will get three sentences for this question: ``The fish ate the worm,'' ``The fish is hungry,'' and ``The worm is hungry.''

\begin{table*}[t]
    \small
    \centering
    \begin{tabular}{l||c|c|c||c|c|c}
    \toprule
        \multirow{2}{*}{Task Name} & \multicolumn{3}{c||}{\# Instance by Knowledge Resource} & \multirow{2}{*}{\# Total Instance}& \multirow{2}{*}{Avg Sub-graph Size} & \multirow{2}{*}{\# Gold Instance} \\
        & ASER & ConceptNet & GLUCOSE  &  & & \\
    \midrule
   HardPCR & 2,030 & 202 & 2,143 & 4,375 & 2.85 & 670 \\
   CommonsenseQA  & 530 & 31 & 37 & 598 & 3.19 & 59\\
   COPA & 103 & 41 & 149 & 293 & 3.03 & 78\\
   ATOMIC  & 5,655 & 212 & 3,466 & 9,333 & 2.67 & 2,200\\
       \midrule
       Total & 8,318 & 486 & 5,795 & 14,599& 2.75 & 3,007\\
    \bottomrule
    \end{tabular}
    \caption{\name ~statistics. ``Avg Sub-graph Size'' is the average graph size, which is measured by the number of edges. ``\# Gold Instance'' means the number of instances supported by different knowledge resources and annotated gold (i.e., Accurate and Enough) knowledge. }
    \label{tab:dataset_statistics}
    \vspace{-0.2in}
\end{table*}

\noindent \textbf{KG Matching}: After getting the declarative sentences that contain the question and key answers, to extract the relevant knowledge, we map them to nodes in knowledge graphs. Considering that each sentence may have multiple words and it is often hard to find an exact match, we adopt an embedding-based fuzzy matching technique. 
For each sentence and node in the KG, we treat them as a sentence and get the corresponding representations with SimCSE~\cite{DBLP:conf/emnlp/GaoYC21}. For each input sentence, SimCSE encodes the sentence into a vector. A close distance between two vectors indicates that the two sentences are similar to each other.
We use cosine similarity on the obtained representations to measure the similarity between two sentences.\footnote{We also tried other techniques such as string match, ROUGE~\cite{lin2004rouge}, and BLEURT~\cite{DBLP:conf/acl/SellamDP20}, but found them to be either inaccurate or too slow for our scale.}
Since there are 287 thousand nodes in GLUCOSE and 194 million nodes in ASER, it is computationally infeasible to compute the cosine similarity between sentences pair by pair.
Thus we use an approximation. 
For each extracted sentence, we first apply Faiss~\cite{DBLP:journals/corr/JohnsonDJ17}, a large-scale similarity-based matching algorithm that first clusters all KG nodes in the vector space to increase the matching efficiency when finding the top $N$ nodes in the KG. 
We encode all the nodes of the graph and index them using Faiss~\cite{DBLP:journals/corr/JohnsonDJ17}. Then, we can perform fast and quick retrieval of the most-similar nodes with each query sentence.
After that, we sort the $N$ nodes based on the cosine similarity to find the top $K$ similar nodes.
We set $N$ and $K$ to be 60 and 1, respectively.
On average, it takes 25 seconds to retrieve the relevant nodes for each question.

\noindent \textbf{Graph Extraction}: Next, we extract the sub-graph that contains all the relevant nodes. We denote the extracted $m$ nodes as $n_1, n_2, ..., n_m$, and for each of them, we find $K$ similar nodes from KG. The resulting matched node sets are denoted as $\NM_1, \NM_2, ..., \NM_m$. For any pair of nodes $n \in \NM_i$ and $n^\prime \in \NM_j$ ($i \neq j$), if there exist a path in the KG between $n$ and $n^\prime$, we will keep that path. After adding all paths together, we will get the final sub-graph.
On average, it takes less than two seconds to construct a graph for each question.
\noindent \textbf{Knowledge Quality Annotation}: 
Since our extraction method is an automatic one, some of the subgraphs may be irrelevant or insufficient for answering the questions. We use crowdsourcing to annotate whether the extracted knowledge is gold (i.e., accurate and enough). For each question, we invite five annotators to provide the annotation. The average Inter-annotator agreement (Cohen’s kappa statistic) is 0.83, which indicates the high-quality of our annotation. In the end, we apply a strict standard (at least four of five annotators need to vote for gold) to select the gold knowledge.
More annotation details could be found in Appendix Section~\ref{sec:annotation}.
\subsection{\name~ Statistics}
We report the dataset statistics in Table~\ref{tab:dataset_statistics}. 
In total, we collect 14,599 instances, and among which Hard PCR and ATOMIC provide the most questions because their original datasets are much larger than others.
According to the annotation, 16.69\% of the supporting knowledge graphs are gold knowledge. 
Based on our analysis, annotators hold a very strict standard for selecting the gold knowledge.
For each task, we randomly split the dataset into training, development, and testing set with a standard 8:1:1 splitting.
As a result, we get 11,678 training, 1,459 development, and 1,462 testing instances.
\section{Experiment Setup}\label{sec:experiment}

We present the performance of following commonsense inference models on \name:

\noindent \textbf{(1) Vanilla LM}: We use the language model (LM) based multiple-choice (MC) model as the basic baseline. For each candidate answer, we concatenate it with the question and feed it to the model. After getting the sentence representation, a linear layer is used to obtain a score and trained with a cross-entropy loss. 

    \noindent \textbf{(2) KagNet}: As one of the pioneering works that utilized structured knowledge for solving commonsense reasoning tasks, KagNet~\cite{DBLP:conf/emnlp/LinCCR19} first uses a graph convolution network to encode the knowledge graph and then apply an LSTM based hierarchical attention mechanism to encode the knowledge paths that start with the nodes corresponding to the question and end with nodes corresponding to the answer. At the same time, KagNet encodes the question and answers with pre-trained LMs. In the end, it concatenates all representations for the final prediction. 
    
    \noindent \textbf{(3) Graph Based Reasoning (GBR)}: Instead of only encoding paths starting with the question nodes and ending with answer nodes, in GBR~\cite{DBLP:conf/aaai/LvGXTDGSJCH20}, they proposes to run a depth-first algorithm over the knowledge graph to generate a sequence of paths as the supporting knowledge paths.
    
    \noindent \textbf{(4) Multi-Head Knowledge Attention (MHKA)}: To further utilize the knowledge, MHKA~\cite{DBLP:conf/emnlp/PaulF20} uses a transformer network to model the paths from the question nodes and answer nodes, then concatenates the knowledge and context representation for the final prediction. 
    
    \noindent \textbf{(5) Graph-to-Text (G2T)}: In the end, we also evaluate a simple yet effective approach of combining structured knowledge and language models: Graph-to-Text~\cite{DBLP:conf/aaai/BianH0021}, which first verbalizes knowledge into a sentence and then concatenates the knowledge sentence and target question together. On top of that, a transformer-based model is used to encode the input the sentence and make the final prediction.

 
\paragraph{Implementation Details}
We implement all experiments with Huggingface~\cite{DBLP:journals/corr/abs-1910-03771}.
We select BERT-base
~\cite{DBLP:conf/naacl/DevlinCLT19} as the base language model for all models. The batch size is set to be 16. All models are trained for 10,000 steps\footnote{All models converge at 10,000 steps.}, and the best-performing checkpoints on the dev set are evaluated.
For our model, we set both the number of random walk paths and walk length to be five.
Considering that the auto-extracted knowledge could contain noise or miss certain knowledge, we add a ``gold knowledge'' setting, where only examples with the gold knowledge are used for training and testing, for all models as the upper bound of their model.
All other hyper-parameters are the same as the base language model. 
All models are trained with GTX 2080 and the average running time is 12 hours.

\section{Result Analysis}
\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{figure/all_instances.pdf}
    \caption{Learning curves of all evaluated models on all instances of \name. }
    \label{fig:all_instances}
\end{figure}

We first conduct analysis experiments to evaluate to what extent the provided knowledge could help existing models. 
For each model, we train it with different numbers of training instances and report the average performance and standard deviation\footnote{Due to the space limitation, we put the detailed experimental results in Appendix Section~\ref{sec:detailed_experimental_results}.} of five trails.
Experiment results with all instances and the gold subset of \name, where only instances with gold knowledge are used for training and testing, are presented in Figure~\ref{fig:all_instances} and~\ref{fig:gold_instance}, respectively.
From the results, we can make the following observations. First, when explicitly including the knowledge, all inference models outperform the baseline model that has no support of the knowledge, especially G2T. When the auto-extracted knowledge and gold knowledge are provided, G2T outperforms the baseline Vanilla LM model by 4.17 and 15.34 accuracy, respectively.
It supports our assumption that it is hard to learn all knowledge from the limited training data and external structured knowledge could help.
At the same time, we also notice that there is a significant gap between auto-extracted knowledge and gold knowledge. For example, models could learn to answer the questions with only a small number of examples if gold knowledge is available. 
This indicates that the knowledge quality can significantly impact models' performance, which further shows the importance of distinguishing whether the knowledge is gold or not automatically.
Last but not least, we can see that G2T outperforms other inference models among most settings, which shows that with the help of current large-scale LMs, jointly encoding question and knowledge is more efficient and a more effective strategy than acquiring them separately. Due to the simplicity and efficiency of G2T, we will conduct the rest analysis experiments with G2T.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{figure/gold_instance.pdf}
    \caption{Learning curves of all evaluated models on the gold subset of \name, where only instances with gold knowledge are used for training and testing. }
    \label{fig:gold_instance}
    \vspace{-0.1in}
\end{figure}
\subsection{Distinguishing the Gold Knowledge}
\begin{table*}[t]
	\centering
    \small
    \vspace{-0.05in}
    \subtable[Full Dataset (Vanilla LM (without knowledge)$\rightarrow$ G2T (with knowledge))]{
  \begin{tabular}{l||c|c|c|c}
    \toprule
     \multirow{2}{*}{Training Task}                  & \multicolumn{4}{c}{Testing Task}\\
 \cline{2-5}
    &Hard PCR  & CommonsenseQA & COPA  & ATOMIC \\
         \midrule
    Hard PCR   &  -  &  37.50 $\rightarrow$ 52.30  & 75.00 $\rightarrow$ 53.24  &  44.13 $\rightarrow$ 53.32  \\
    CommonsenseQA   &  50.00 $\rightarrow$ 50.14  &  -  &  62.50 $\rightarrow$ 56.67  &  56.34 $\rightarrow$ 70.56  \\
    COPA    &  45.95 $\rightarrow$ 51.26  &  62.50 $\rightarrow$ 58.33 &  -  &  49.77 $\rightarrow$ 62.96  \\
    ATOMIC   &  39.19 $\rightarrow$ 50.76  &  50.00 $\rightarrow$ 76.67  &  62.50 $\rightarrow$ 73.33  &  -  \\
    \bottomrule
    \end{tabular}
	}

    \subtable[Gold Subset (Vanilla LM (without knowledge)$\rightarrow$ G2T (with knowledge)) ]{
    \begin{tabular}{l||c|c|c|c}
    \toprule
     \multirow{2}{*}{Training Task}                  & \multicolumn{4}{c}{Testing Task}\\
 \cline{2-5}
    &Hard PCR  & CommonsenseQA & COPA  & ATOMIC \\
         \midrule
    Hard PCR  &  -  &  46.67 $\rightarrow$ 51.67  &  63.33 $\rightarrow$ 56.67  &  51.85 $\rightarrow$ 55.78  \\
    CommonsenseQA   &  49.32 $\rightarrow$ 50.32  & - & \hlc[orange]{ 50.00 $\rightarrow$ 75.00 } & \hlc[green]{ 60.39 $\rightarrow$ 91.08 }\\
    COPA     &  52.51 $\rightarrow$ 54.79  & \hlc[orange]{ 56.67 $\rightarrow$ 87.50 } &  -  & \hlc[green]{ 53.01 $\rightarrow$ 76.06 }\\
    ATOMIC   &  50.46 $\rightarrow$ 51.35  & \hlc[green]{ 68.33 $\rightarrow$ 93.75 } & \hlc[green]{ 56.67 $\rightarrow$ 87.50 } &  - \\
    \bottomrule
    \end{tabular}
	}
	\vspace{-0.1in}
	\caption{Generalization ability demonstration. 
	We report the performance on both the full dataset and gold dataset (i.e., only questions with gold knowledge are selected for training and testing)  to show the generalization ability. Strong and moderate generalization settings are indicated with the  \hlc[green]{green} and \hlc[orange]{orange} background, respectively.} 
\label{tab:Generalization_ability}
\end{table*}
Humans have the capability of saying ``I do not know'' when they find out that they cannot answer a question with their knowledge.
To investigate whether current deep models have a similar capability, we use G2T as an example to test whether these deep models can distinguish the gold knowledge.
For each (question, answer, and knowledge) triplet, we train and test G2T with annotated knowledge quality labels.
To address the imbalanced distribution problem, we randomly 
select the same number of ``Not Gold'' examples as the ``Gold'' ones to make the dataset balanced.
From the results in Figure~\ref{fig:IDK_results}, we can see that the performance of G2T can be improved  slightly with the increase of training data.
However, after seeing thousands of examples, it still can only achieve 0.65 accuracy on a binary classification problem.
It shows that knowing when to say ``I do not know'' is still a challenging task for current deep models, which is consistent with the observations in previous literature that deep models cannot understand the reasons and knowledge they used to answer questions~\cite{DBLP:conf/acl/ZhangZS20,DBLP:journals/corr/abs-2110-08207}.
We hope that \name~could motivate more future work on this important research problem.
\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{figure/IDK.pdf}
    \caption{The learning curve of G2T on the gold knowledge identification task.}
    \vspace{-0.2in}
    \label{fig:IDK_results}
\end{figure}
\subsection{Generalization Ability}
An important assumption and motivation behind the unified problem design of \name~is that even though the commonsense could be enormous, the inference rules over commonsense knowledge can be limited.
As a result, even though we could not learn all the commonsense from limited training data, we can learn how to conduct inference with several tasks and then generalize to others.
In this section, we conduct experiments with both the ``Without Knowledge'' and ``With Knowledge'' models to show that with our unified formulation, we can gain such generalization ability across different tasks.
We conduct experiments on two settings: (1) Full Set: We train and test the model with the whole dataset; (2) Gold Subset: We only train and test the model on questions, where the supporting graph is annotated as gold.
We train the model with questions from a specific task and test it on all tasks. The results are in Table~\ref{tab:Generalization_ability}.

\begin{figure*}
    \centering
    \includegraphics[width=0.95\linewidth]{figure/exp-case-study.png}
    \vspace{-0.01in}
    \caption{\name~ Case Study. Mapped nodes for the question/answers are in blue/pink. Other nodes are white. Edge weights are in brackets. We only show the relevant parts of the graphs for clear representation. }%
    \vspace{-0.1in}
    \label{fig:case_study}
\end{figure*}

From the results, we can see that the knowledge can help models to generalize well among CommonsenseQA, COPA, and ATOMIC. The only exception is HardPCR.
This is mainly because the inference needed for solving HardPCR is more complex than the other tasks, where we do not only need to find the relevant knowledge but also need to replace the target pronouns with the entity in the provided knowledge.
As shown in Figure~\ref{fig:case_study}, two paths can be found relevant to question: (1) ``I am drunk''$\rightarrow$\textit{Co\_Occurrence}$\rightarrow$``I hit someone''; (2) ``I am drunk''$\rightarrow$\textit{Co\_Occurrence}$\rightarrow$``That is not fair''$\rightarrow$\textit{Co\_Occurrence}$\rightarrow$``You kick me''. For the correct inference, we need to know when there is a conflict, we should trust the one-hop inference more because the additional node in the two-hop path may introduce extra noise.
As a comparison, for other tasks, the main inference we need is to find the relevant paths, which is relatively easy.
How to train a model that can learn to conduct such complex reasoning is a problem worth exploring in the future.

In general, the observed generalization ability is encouraging because if we can learn a good model on \name, based on the assumption that there are limited types of inference, potentially we can solve any commonsense reasoning task as long as the needed inference types are covered by \name. At the same time, we also notice that models typically generate better when gold knowledge is provided, which further proves the importance of the gold knowledge identification task.
\section{Related Work}\label{sec:related_works}
To help machines understand commonsense, the community has devoted great efforts in constructing commonsense knowledge bases with either crowdsourcing (e.g., ConceptNet~\cite{liu2004conceptnet} and ATOMIC~\cite{sap2019atomic}) or information extraction techniques (e.g., ASER~\cite{zhang2019aser}).
Typically, crowd-sourced knowledge bases are of higher quality, and the auto-constructed ones have larger coverage.
Besides acquiring commonsense knowledge, the community also developed many commonsense reasoning datasets to train and test models' commonsense reasoning abilities. Even though these datasets may have different \textit{formats} (e.g., slot fitting in Winogrande~\cite{DBLP:conf/aaai/SakaguchiBBC20} and question answering in CommonsenseQA~\cite{DBLP:conf/naacl/TalmorHLB19}), \textit{knowledge types} (e.g., causal commonsense in COPA~\cite{DBLP:conf/aaaiss/RoemmeleBG11} and numerical commonsense in NumerSense~\cite{DBLP:conf/emnlp/LinLKR20}), or \textit{modalities} (e.g, visual commonsense in VCR~\cite{DBLP:conf/cvpr/ZellersBFC19} and textual commonsense in many others), they follow a standard supervised learning setting, and aim at helping machines to solve a specific commonsense task in an end-to-end manner.
Given this setting, it is often difficult to tell what has been learned during the training process. Was it used to acquire commonsense knowledge, learn to conduct commonsense inference, or both?
Such ambiguity limits our progress in solving these commonsense reasoning tasks.
In this work, we connect the efforts on commonsense acquisition and inference by creating a commonsense inference benchmark \name~, where models can focus on learning to identify the gold knowledge and perform inference over the supporting commonsense knowledge.

Answering questions in natural language based on a knowledge base (KB) is a mature research topic in the NLP community, which is also known as the KBQA problem~\cite{clark1999knowledge,DBLP:conf/acl/YihCHG15,DBLP:conf/acl/YihRMCS16,DBLP:conf/esws/UsbeckNHKRN17,DBLP:journals/pvldb/CuiXWSHW17}.
Previous work mainly focuses on factual knowledge, which is stored in the format of triplets, and the main challenge is to parse the question and then precisely and effectively identify the correct path over a large-scale KB to do the inference.
Compared with inference over factual knowledge, inference over commonsense knowledge brings the following unique challenges: 
(1) Commonsense is a kind of preference rather than fixed knowledge. As a result, the ideal commonsense reasoning process could involve the comparison of multiple candidates . For example, both ``drink coffee'' and ``drink bear'' could happen in the morning, but a normal person will prefer ``drink coffee;''
(2) Beyond named entities, commonsense knowledge also covers daily entities and events, and thus it is difficult to find an exact node from the commonsense KB that matches the question and we may need to conduct inference based on the partial match (i.e., the extracted nodes are relevant but not identical).

\section{Conclusion}\label{sec:conclusion}

In this paper, we present \name, a unified commonsense inference benchmark. 
Specifically, we first convert several popular commonsense tasks into a unified QA format and then equip each question with a supporting commonsense knowledge graph.
We also leverage humans to annotate the quality of auto-extracted knowledge.
Experiments show that even though models can better learn how to do commonsense inference with a few examples and significantly outperform the baseline method that does not use structured knowledge in the data-scarce setting, how to identify the gold knowledge is still an unsolved problem.
More interestingly, with our unified formulation, models demonstrate the encouraging generalization ability across tasks.
As both the format unification and supporting graph extraction are automatic, we can easily extend to other commonsense reasoning tasks in the future. 
All used code and data are submitted in the submission system.

\section*{Acknowledgements}

The authors of this paper were supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program, and by contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA).
The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
This paper was also supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong with special thanks to HKMAAC and CUSBLT, and  the Jiangsu Province Science and Technology Collaboration Fund (BZ2021065).
Yanai Elazar is grateful to be supported by the PBC fellowship for outstanding PhD candidates in Data Science and the Google PhD fellowship.
\bibliography{main}

\clearpage
\appendix
\section{Annotation Details}\label{sec:annotation}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\linewidth]{figure/survey_demo.png}
    \caption{An example of the used survey.}
    \label{fig:survey_demo}
\end{figure}

The annotation goal is to determine whether the supporting graph can help answer the question or not. Thus, for each QA pair, we present the question, candidate answers, and the supporting sub-graph to annotators\footnote{All annotations follow the ethical guidelines.}, and then ask them two questions: (1) What is the correct answer for this question; (2) Whether the provided commonsense knowledge contains all the essential commonsense for answering this question. The purpose of the first question is to assess the annotation quality. A survey example is shown in Figure~\ref{fig:survey_demo}. In beginning of each survey, we also provide detailed instructions and examples to help annotators understand our task. We employ annotators from Amazon Mechanical Turk to provide annotations. To improve the annotation quality, we require the annotators to be English native speaker and to have an overall acceptance rate above 90\%. For each survey, we invite five annotators to provide the annotations and pay them \$0.1.
The average Inter-annotator agreement (Cohen's kappa statistic) for Q1 and Q2 are 0.87 and 0.83, respectively.
The annotation results show that humans could provide consistent annotation about whether the knowledge could be used to answer the questions.
\section{Detailed Experimental Results}\label{sec:detailed_experimental_results}

Detailed experimental results are presented in Table~\ref{tab:Commonsense_Task_Results}.

\begin{table*}[t]
    \small
    \centering
    \begin{tabular}{l||c|c|c|c|c|c|c}
    \toprule
        \multirow{2}{*}{Model} & \multicolumn{7}{c}{Number of Training Instances} \\
        & 5 & 10 & 100 & 500 & 1,000 & 5,000 &  11,678 \\
    \midrule
    Chance Performance & 50.00 (0.00) & 50.00 (0.00) & 50.00 (0.00) & 50.00 (0.00) & 50.00 (0.00) & 50.00 (0.00) & 50.00 (0.00)\\
    \midrule
Vanilla LM & 51.16 (1.92) & 55.88 (2.41) & 56.52 (2.37) & 63.67 (2.19) & 66.76 (1.37) & 70.04 (0.58) & 70.11 (0.28)\\
    \midrule
    KagNet~\cite{DBLP:conf/emnlp/LinCCR19} & 53.29 (2.16) & 55.47 (2.74) & 59.92 (3.05) & 61.97 (1.19) & 65.90 (1.54) & 68.90 (1.21) & 71.50 (1.29)\\
    GBR~\cite{DBLP:conf/aaai/LvGXTDGSJCH20} & 51.77 (1.75) & 56.57 (3.13) & 59.92 (2.34) & 63.36 (1.62) & 68.06 (0.35) & 67.10 (0.17) & 71.34 (0.31)\\
    MHKA~\cite{DBLP:conf/emnlp/PaulF20} & 54.89 (2.34) & 60.47 (1.13) & 61.70 (0.41) & 63.82 (0.78) & 67.85 (0.32) & 69.29 (1.58) & 71.30 (1.14)\\
    G2T~\cite{DBLP:conf/aaai/BianH0021} & \textbf{57.25} (0.21) & \textbf{62.41} (0.97) & \textbf{64.02} (0.99) & \textbf{68.54} (0.47) & \textbf{71.55} (0.75) & \textbf{72.36} (0.56) & \textbf{74.28} (0.21)\\
    
    \midrule
    KagNet-gold& 55.21 (3.21) & 64.36 (0.83) & 68.65 (1.64) & 74.28 (1.31) & 79.05 (0.57) & 80.21 (0.84) & 80.20 (0.21)\\
    GBR-gold & 50.53 (1.62) & 66.34 (1.82) & 69.31 (1.33) & 72.94 (0.35) & 76.24 (0.21) & 80.86 (0.21) & 78.85 (0.13)\\
    MHKA-gold & 58.35 (2.67) & 78.54 (1.32) & 78.55 (0.72) & 79.23 (0.64) & 80.53 (0.50) & 80.52 (0.52) & 81.85 (0.15)\\
    G2T-gold & \textbf{61.39} (2.56) & \textbf{80.85} (1.35) & \textbf{82.18} (0.33) & \textbf{82.51} (0.50) & \textbf{84.32} (0.42) & \textbf{85.81} (0.45) & \textbf{85.48} (0.17)\\
    \bottomrule
    \end{tabular}
    \caption{Demonstration of different models with different training instances. We report the average performance of five different random seeds and standard deviation (in brackets). ``-gold'' indicates that the models are trained and tested with instances with gold knowledge. We cannot directly compare them with the normal setting, but it could serve as the upper-bound for our learning paradigm. Best performing models under both settings are indicated with the \textbf{bold} font.}
    \label{tab:Commonsense_Task_Results}
\end{table*}
\end{document}


Peer reviews:

content: confidence: 3: The reviewer is fairly confident that the evaluation is correct
rating: 7: Good paper, accept
review: The paper proposes CIKQA, a commonsense benchmark, which unifies several commonsense task into QA format and associates them with relevant knowledge. Experiments shows that models can better learn inference and generalize across tasks with proposed formulation and usage of knowledge.

Strength:
1. the proposed benchmark can be a useful resource for the field. 
2. the experiments and analysis are comprehensive and give interesting insights. 
3. the writing is clear and easy to follow.

Weakness:
1. the coverage of proposed benchmark is limited, it doesn't include any physical or social commonsense tasks, like PIQA or SocialIQA.
2. the idea isn't entire novel - unified task formulation and knowledge injection have already been well-studied in QA domain. 
3. missing related works: 
[1] Liu, Jiachen et al. “Generated Knowledge Prompting for Commonsense Reasoning.” ArXiv abs/2110.08387 (2021): n. pag.
[2] Shwartz, Vered et al. “Unsupervised Commonsense Question Answering with Self-Talk.” ArXiv abs/2004.05483 (2020): n. pag.

title: a good benchmark paper overall

content: confidence: 3: The reviewer is fairly confident that the evaluation is correct
rating: 7: Good paper, accept
review: Overall the paper seems comparatively complete and solid to me. The authors propose the benchmark CIKQA with a clear task formulation and detailed steps on how to extract supporting knowledge, as well as a strong baseline that takes advantage of its format. Experiments are also well-executed to answer the authors' questions regarding leveraging provided knowledge, distinguishing gold knowledge, and model generalization ability on different tasks.

Here are my comments:
- It is a bit overclaimed to me that CIKQA can focus on learning to do the inference with the current task setting. 
- The related work discussion seems a bit sparse to me.
- I am a bit concerned about the novelty as the unified format and injecting knowledge have all been discussed widely.
- From Table 1, there are only 3,007 instances with gold knowledge, but for experiment results in Figure 3, even models with the suffix `_gold` could still be trained with $10^4$ training instances. Hope the authors could address the issue there.
title: Commonsense benchmark paper - simple and well-presented

content: confidence: 3: The reviewer is fairly confident that the evaluation is correct
rating: 6: Marginally above acceptance threshold
review: The authors propose a reformulation of commonsense reasoning QA tasks
that attempts to separate knowledge (e.g., facts as specified by a KG)
from inference (i.e., reasoning over a given set of facts). Their
setup is to pair a small knowledge graph with each question that
contains the relevant knowledge to answer the question. They report
experimental results in this setting, showing that their model,
JointI, 1) effectively incorporates the knowledge graph information in
a fewer-shot setting (e.g., 100-1000 points); and 2) transfers between
tasks better than if the model didn't have explicit knowledge handed
to it.

I commend the authors for their attempt to solve a difficult problem:
indeed, the distinction between factual knowledge and inference over
that knowledge is rather illspecified in the commonsense domain.  The
proposed approach: converting and then augmenting existing QA datasets
with all the knowledge the might need gives a potentially nice
solution to this problem: i.e., by conditioning on "all of the
knowledge", the algorithms can focus entirely on inference; similarly,
by retrieving knowledge as a first step. I also think the results here
regarding generalization are quite interesting! Because we expect that
the inference required for commonsense reasoning tasks may be shared,
the transfer results suggest that, moreso than a model without
explicit knowledge provided, an inference-focused model may generalize
better.

My biggest concern is that I'm not entirely convinced that this setup,
as the authors claim, fully separates the knowledge versus inference
question. While the approach makes sense in theory (i.e., conditioning
on all needed knowledge), 1) there are still pieces of commonsense
knowledge required to, e.g., interpret the small KGs that are paired
with each question. To take the example in Figure 2: one simple case:
an algorithm must know that sleeping is a type of resting. And 2)
models could simply ignore the given knowledge graph in this setup,
e.g., if one was to use a pretrained language model that was already
imparted with both knowledge and inferential capacity. The authors do
use BERT-Small in some experiments and performance improves when the
graph is included in the input, but, I suspect that if more powerful
pretrained models were used the large performance gaps presented
between Table 2 (a) vs. Table 2 (b) might vanish.

Overall, the authors report some interesting results for their new
setup, which may have practical promise for few-shot learning with
small models. However, I do worry that CIKQA has limitations that need
to be addressed if larger models were to be applied to such a task.
title: review