\pdfoutput=1

\documentclass[11pt]{article}

\usepackage[]{emnlp2021}

\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{graphicx}

\usepackage[T1]{fontenc}

\usepackage[utf8]{inputenc}

\usepackage{microtype}

\usepackage{enumitem}

\newcommand{\textapprox}{\raisebox{0.5ex}{\texttildelow}}
\newcommand{\cissq}{\textsc{Cis\textsuperscript{2}}}
\interfootnotelinepenalty=10000
\title{\cissq: A Simplified Commonsense Inference Evaluation for Story Prose}
\author{Bryan Li, Lara J. Martin, \and Chris Callison-Burch \\
  University of Pennsylvania \\
  Philadelphia, PA, USA \\
  \texttt{\{bryanli, laramar, ccb\}@seas.upenn.edu}}

\begin{document}
\maketitle

\begin{abstract}
\textit{Contextual Commonsense Inference (CCI)} is the problem of inferring causal relations
between the events of a text, such as a story.
Like other commonsense reasoning tasks, CCI is a problem of language understanding, rather than language generation. We show that prior work, in using language generation to perform CCI, trains models that struggle on the CCI task in isolation. This \textit{conflation} of tasks is further exacerbated by evaluating with word-matching based metrics such as BLEU.
In order to isolate CCI from language generation, we reframe CCI as a classification problem.
Our system, which we call \cissq, forces the model to focus on CCI directly by providing it the original text of the story to use for understanding while having it generate only the bare minimum: indices to sentences.
We look at the GLUCOSE~\cite{mostafazadeh-etal-2020-glucose} dataset and compare against their task for predicting CCI between story sentences.
We find that models trained on \cissq{} index labels achieve a 4.3\% higher CCI accuracy than those trained for generating full phrases, such as in the original GLUCOSE task.
\end{abstract}
\section{Introduction}
Transformer-based language models \cite{transformer}---particularly off-the-shelf models---have shown mixed success with story generation~\cite{see-etal-2019-massively, Wang2019, ippolito-etal-2020-toward}. Language models (LMs) lose coherence as their output length increases, and are prone to meandering, losing the plot of a story over time. This can be largely attributed to the LM generating each token by sampling from a probability distribution, failing to distinguish between statistical correlation (how frequently event A and event B are seen together) and causal reasoning (event A causes event B to occur).

Since causal events across sentences in stories help people understand and retain story information \cite{Trabasso1984}, we posit that the inability of language models to perform commonsense inference leads them to output less coherent long-form text. 
Commonsense inference is still an open problem in NLP, especially when the commonsense information is unstructured and provided in the form of natural language.
We refer to this task of grounding commonsense inference relations within prose as \textit{contextual commonsense inference (CCI)}, a sub-task within commonsense reasoning.
Due to storytelling being deeply intertwined with causal understanding, improving CCI will yield both more accurate story generation evaluation metrics and better story generation.

\begin{figure}
    \centering
    \includegraphics[width=0.48\textwidth]{figures/io_conflation.png}
    \caption{Motivation for \cissq,
    illustrating how the original GLUCOSE task conflates commonsense inference and text generation. Input and output are exactly as seen by finetuned T5. \textcolor{blue}{Blue}: selected sentence \textit{X} is always paraphrased. \textcolor{orange}{Orange}: dimension specifies the position of \textit{X}, and the relation. \textcolor{green}{Green}: commonsense inference is needed here to select the other sentence \textit{Y}.}
    \label{fig:io_conflation}
\end{figure}

Current methods in CCI for story understanding often include the use of generative LMs. While LMs might be helpful for encoding the textual information, they are less suited to operating on and making decisions based on this information due to their probabilistic way of generating text. This leads to a tendency to focus on grammar rather than meaning \cite{Martin2018AAAI}. Furthermore, commonly-used language generation evaluation metrics like BLEU put emphasis on exact word usage and grammar. In this paper, we look at what it would mean to de-emphasize generation and paraphrasing for understanding tasks like CCI.

Our contributions in this paper are twofold. First, we critique an existing method addressing the \textit{contextual commonsense inference} (CCI) task by using the GLUCOSE~\cite{mostafazadeh-etal-2020-glucose} dataset and teasing apart their associated CCI task formulation.  We designed several diagnostic tasks which selectively omit sentences of the input and investigate which sentences contribute the most to paraphrasing/generation. We replicate their results, then finetune T5 models \cite{t5} on each of our diagnostic tasks, to show the significant conflation of language understanding and generation in the original GLUCOSE T5 model. 

Second, we propose \cissq~(Contextual Commonsense Inference in Sentence Selection), a simplified task for more fairly evaluating commonsense inference in storytelling, which abstracts away the natural language generation component almost entirely. We develop a heuristic to convert story sentences into \cissq{} tags and show that a language model, when trained on this data, outperforms the original GLUCOSE task formulation on forming the correct causal relations between sentences in stories. Our findings reinforce that while the GLUCOSE dataset encodes useful commonsense information, we urge that future work should carefully disentangle language generation when performing language understanding tasks. Our code, data, and models are available at \url{https://github.com/manestay/cis2}.
\section{Related Work}

\label{sec:related}
Commonsense inference is the ability to use prior knowledge based on real world experiences to infer what has happened or will happen.
While lived experiences vary from person to person, there are still significant commonalities as we live and interact within the same physically- and temporally-constrained world.

\subsection{Commonsense Knowledge Graphs}
\citet{hwang2021comet} formalized the \textit{commonsense inference task} (CI) for AI systems as a knowledge three-tuple, to predict the \textit{object} of a relation given the \textit{subject} and \textit{relation}.
This formulation of commonsense inference can be structured as a graph, where the subjects and objects are nodes and the relations are the edges connecting the entities. These
commonsense knowledge graphs (CKGs) explicitly encode the structure of inference relationships between entities.
ATOMIC~\cite{ATOMIC} is one such CKG dataset that organizes everyday events into if-then relationships.  COMET~\cite{Bosselut2019} is a transformer language model designed on top of ATOMIC relations, showing language models can encode and generalize commonsense information.

However, \citet{Wang2021} show that language models struggle to perform generalizable commonsense inference across three popular CKG datasets: ConceptNet~\cite{speer2017conceptnet}, TupleKB~\cite{dalvi-mishra-etal-2017-domain}, and ATOMIC~\cite{ATOMIC}. They found that LMs trained on several CKGs have limited ability to transfer knowledge to unseen CKGs, and that adaptation generalizes well to unseen subjects, but less so on unseen objects.

Although these graphs do well at representing facts and their relations, their statements lack context and would need to be adapted to a textual domain, such as story prose. Using them to generate a story as-is would fail to engage readers since the ``story'' would simply be a series of facts. Our work goes beyond the explicit structure of CKGs, focusing on finding and leveraging commonsense relations in natural language short stories.
\subsection{Commonsense Inference for Storytelling}
\label{ssec:CIstories}

Early research on automated story generation research focused on designing systems that create \textit{coherent} stories \cite{Lebowitz1986, Turner1986, Liu2002, Young2013}.
Despite the success of neural networks for AI tasks, commonsense and coherence remain big issues for story generation systems.

Applying commonsense reasoning to the events of a story has been proposed as one way to tackle the difficult problem of assessing the quality of machine-generated stories. The Story Cloze Test~\cite{mostafazadeh-etal-2016-corpus} formulates story ending generation as a multiple-choice task, having systems look at several possible endings and predict the one that is most reasonable. \citet{Guan2019} integrated commonsense reasoning directly into their Story Cloze model by building context clues and using implicit knowledge.

Commonsense reasoning can also help story generation with issues in plot coherence. \citet{Martin2021Thesis} created a neurosymbolic system that leveraged VerbNet~\cite{Brown2019} facts to ground neural story generation in commonsense reasoning. They did this by tracking the story state and pruning out impossible options that a neural network provided as candidate next sentences for the story. Similarly, the Commonsense inference Augmented neural StoryTelling (CAST) framework \cite{Peng2021} modeled interactions between multiple characters using ATOMIC. The stricter, more explicit generation constraints of CAST produced more coherent and on-topic two-character stories than generating via sampling from a distribution alone.

TellMeWhy \cite{lal-etal-2021-tellmewhy} is a dataset built on top of ROCStories~\cite{mostafazadeh-etal-2016-corpus}, consisting of 30k questions on why characters perform their actions and the corresponding answers. They found that current state-of-the-art models performed far worse than humans, especially on questions whose answers are external to the narratives. This contrasts with the findings discussed in \citet{mostafazadeh-etal-2020-glucose} that language models can approach human performance. 

\section{The GLUCOSE Dataset and Task}
\label{ssec:original-dataset}

\begin{table}[t]
    \centering
    \small
    \setlength{\tabcolsep}{4pt}
    \begin{tabular}{p{0.18cm}p{4.6cm}p{2cm}}
    \textbf{\#} & \textbf{Description} & \textbf{Relation Text}\\
    \toprule
    1 & Event that causes or enables X & >Causes/Enables> \\
    2 & Emotion/basic human drive that motivates X & >Motivates> \\
    3 & Location state that enables X & >Enables>\\
    {4} & Possession state that enables X & >Enables>\\
    {5} & Other attributes enabling X & >Enables>\\
    \midrule
    {6} & Event that X causes or enables & >Causes/Enables>\\
    {7} & An emotion that is caused by X & >Causes>\\
    {8} & A change in location that X results in & >Results in>\\
    {9} & A change of possession that X results in & >Results in>\\
    {10} & Other changes in property that X results in & >Results in>\\
    \bottomrule
    \end{tabular}
    \caption{The ten GLUCOSE dimensions and the corresponding relation text connecting statements~\cite{mostafazadeh-etal-2020-glucose}.}
    \label{tab:dimensions}
\end{table}
Our work follows from GLUCOSE (GeneraLized and COntextualized Story Explanations)~\cite{mostafazadeh-etal-2020-glucose}. In this section we briefly describe their dataset and experiments; for more details, refer to the original paper.
The GLUCOSE dataset contains 670K crowdsourced annotations identifying
causal reasoning relations between the sentences within stories from ROCStories~\cite{mostafazadeh-etal-2016-corpus}---a collection of crowdsourced five-sentence everyday stories in English.
The authors structured the collected data around ten different dimensions, shown in Table~\ref{tab:dimensions}, of causal relations between a pre-selected sentence \textit{X} from the story and another statement \textit{Y}, which can either be another story sentence or some external commonsense knowledge. The relationship between these statements can be formalized as:
\begin{equation}
\text{{\em statement\textsubscript{1} REL statement\textsubscript{2}}}
\end{equation}

\textit{X} can be in either \textit{statement} position, depending on the particular dimension chosen:
Dimensions 1-5, specify events that \textit{caused X} (i.e., \textit{X} is \textit{statement\textsubscript{2}}%
), and dimensions 6-10 specify events \textit{caused by X} (i.e., \textit{X} is \textit{statement\textsubscript{1}}). %

\begin{table}[t!]
    \centering
    \small
    \setlength{\tabcolsep}{2pt}
    \begin{tabular}{lp{4.8cm}}
    \textbf{Parameter}           & \textbf{Text} \\
    \toprule
    Story               & Fred woke up late. He just missed his bus. He then went to his mom's room. His mom then drives him   to school. He makes it to first class on time. \\
    \midrule
    Selected Sentence (\textit{X}) & Fred woke up late. \\
    \midrule
    Dimension           & 6\\
    \midrule\midrule
    Specific Rule       & Fred wakes up late   >Causes/Enables> Fred misses his bus   \\
    \midrule
    General Rule        & Someone\textsubscript{A} wakes up   late >Causes/Enables> Someone\textsubscript{A}  misses Something\textsubscript{A}     \\     
    \bottomrule
    \end{tabular}
    \caption{Example GLUCOSE entry~\cite{mostafazadeh-etal-2020-glucose}. The top three rows (story, \textit{X}, dimension) are input, and the bottom two rows (specific rule, general rule) are output.}
    \label{tab:GLUCOSE_example}
\end{table}
\begin{table*}[ht]
    \centering
    \small
    \begin{tabular}{lp{.405\textwidth}p{.405\textwidth}}
    \toprule
    \textbf{Task} & \textbf{Input} & \textbf{Output} \\
    \midrule
    \textsc{Original} & 1: My mother told me to fix the car. I was unable to do this right away. \textbf{* I could not find my tools. *} I looked everywhere for them. It turns out they were stolen the night before. & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A}  \\ \hline
    
    \textsc{History} & 1: My mother told me to fix the car. I was unable to do this right away.  & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} \\ \hline
    
    \textsc{Mask X} & My mother told me to fix the car. I was unable to do this right away. \texttt{<masked>} I looked everywhere for them. It turns out they were stolen the night before. & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} \\\hline
    
   \textsc{History+X} & 1: My mother told me to fix the car. I was unable to do this right away. \textbf{* I could not find my tools. *} & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} \\\hline\hline
    \cissq & 1: My mother told me to fix the car. I was unable to do this right away. \textbf{* I could not find my tools. *} I looked everywhere for them. It turns out they were stolen the night before. & \texttt{<s\textsubscript{4}> >Causes/Enables> <s\textsubscript{2}>} \\
    \bottomrule
    \end{tabular}
    \caption{Task formulations of the same GLUCOSE entry. The output is split into a specific rule and a general rule by ``**'', and the selected sentence \textit{X} (``I could not find my tools'') is surrounded by single asterisks. In this table, we also \textbf{bolded} the selected sentence, and special tokens are \texttt{monospace}. The ``1:'' at the beginning of the input specifies the GLUCOSE dimension; ``1'' corresponds to the Causes/Enables relation. The diagnostic tasks \textsc{History}, \textsc{Mask X}, and \textsc{History+X} are variations on the original task, \textsc{Original}. \cissq{} is our proposed task.}
    \label{tab:tasks}
\end{table*}
\subsection{Contextual Commonsense Inference Task}
\label{ssec:task}
GLUCOSE addresses the task of predicting relationships between statements explicitly or implicitly expressed within a text, a task we term \textit{contextual commonsense inference} (CCI).
An example GLUCOSE entry can be found in Table~\ref{tab:GLUCOSE_example}.
The entries are organized to reflect the CCI task and are formalized as input-output tuple pairs,
with input tuple
\begin{gather} \label{eq:input}
    \langle \text{\textcolor{blue}{story \textit{S}, selected sentence \textit{X}, dimension \textit{D}}} \rangle,
\end{gather}
where a \textcolor{blue}{story \textit{S}} consists of five sentences
[\textit{s\textsubscript{0}, s\textsubscript{1}, s\textsubscript{2}, s\textsubscript{3}, s\textsubscript{4}}],
the \textcolor{blue}{selected sentence \textit{X}} is the sentence on which the rule is centered, and
the number \textcolor{blue}{dimension \textit{D}} is one of the ten dimensions from Table \ref{tab:dimensions}---and output tuple
\begin{gather} \label{eq:output}
     \langle \text{\textcolor{olive}{specific rule \textit{R\textsubscript{S}}, general rule \textit{R\textsubscript{G}}}} \rangle,
\end{gather}
where the \textcolor{olive}{specific rule \textit{R\textsubscript{S}}} is the relation between \textcolor{blue}{\textit{X}} and \textit{Y}. \textit{Y} can be either (1) another sentence in the story or (2) an implicit statement from outside the text. %
The \textcolor{olive}{general rule \textit{R\textsubscript{G}}} is the same rule as \textcolor{olive}{\textit{R\textsubscript{S}}} but using generalized tags for named entities (e.g., Someone\textsubscript{A} instead of Fred).
To summarize, the GLUCOSE task is: given \textcolor{blue}{\textit{S}, \textit{X}, and \textit{D}}, predict/generate \textcolor{olive}{\textit{R\textsubscript{S}} and \textit{R\textsubscript{G}}}.

In this paper, we compare to their best model, a finetuned T5 model~\cite{t5}, which achieved a 71.26 average SacreBLEU~\cite{post-2018-call} across the 10 dimensions on predicting general rules and a 75.65 average for the specific rules.\footnote{Our best-effort replication of their experiments achieves slightly lower BLEU scores (66.2 \& 70.7, respectively) due to resource limitations (detailed in Appendix \ref{ssec:repro}).}
The models were also rated for ``correctness'' using crowdsourcing, where their T5 model scored 2.5/3 averaged across all 10 dimensions on a 4-point Likert scale mapped to a numerical scale of 0-3. For context, their closest baseline got a 2.21/3 average and the gold standard was 2.8/3.
\subsection{Issues with the GLUCOSE Task for CCI}
\label{ssec:issues}
We find that the GLUCOSE dataset is well-designed and of good annotation quality. However, we take issue with the GLUCOSE task, which asks a model to perform two tasks simultaneously: commonsense inference and language generation.
Due to this \textit{conflation} of tasks, the model, in generating its output, would rely heavily on the already-good language generation ability of T5 and neglect learning enough CCI.
T5~\cite{t5} and other transformer LMs were designed to perform language {\em generation} tasks. Therefore, by including text generation as part of CCI, T5 will focus on paraphrasing or even copying story sentences. %

There are several one-to-one correspondences between parts of the input and output in the original GLUCOSE task (illustrated in Figure~\ref{fig:io_conflation}). For example, for all GLUCOSE entries, the output contains at least one paraphrased sentence from the input. Conflation with paraphrasing worsens with BLEU as the evaluation metric, where incorrect commonsense inferences can score partial credit if they have words in common.



\section{Diagnostic Tests}
\label{ssec:diagnostic}
In this section, we describe our three diagnostic tests---variations on the original GLUCOSE task with altered input---to isolate different factors that influence T5's generation. Through these tests, we investigate the extent to which language models rely on paraphrasing to generate the commonsense rule output for GLUCOSE.
For each of the following diagnostic tests, we finetune the same T5~\cite{t5} model, a pretrained model using the same hyperparameters as in the GLUCOSE paper, to generate the same output as in Equation~\ref{eq:output}. The diagnostic tests differ only in the format of the input. The purpose of these tests was to assess how reliant the model is on language generation when performing CCI. More detailed training setup and hyperparameters for these models can be found in Appendix \ref{sec:hyperparams}.

Because these tasks are measured with BLEU, conflation between CCI and language generation will always occur. But by deleting different parts of the input, these diagnostic tasks analyze which sentences contribute the most to performance, thus resulting in more conflation.

An overview of the tests' different data formats can be found in rows 2, 3, and 4 of Table~\ref{tab:tasks}.  We describe them in this section using the following terminology for brevity:\\
\textit{Dimension (dim)}: the causal dimension\\
\textit{Pre-context}: sentences before selected sentence X\\
\textit{Selected sentence (X)}: the story sentence of interest\\
\textit{Post-context}: sentences after selected sentence X

\paragraph{\textsc{Original}.} This experiment is the same as in \cite{mostafazadeh-etal-2020-glucose}, which we described in Section~\ref{ssec:task}. We report results on our own replication of the finetuned T5 model, implemented with the \texttt{transformers} package~\cite{wolf2019huggingface}.

\paragraph{\textsc{History}.} This experiment gives as input only the pre-context (the sentences before sentence \textit{X}) and the dimension. This model must generate the output without knowing the target sentence \textit{X}, nor the events happening afterwards. Here, we test the model's ability to generate two (specific) statements given only what happened before. This difficult task serves as a lower bound to contextual commonsense inference performance. Conflation with language generation is absent.

For all dimensions, the model must first speculate what \textit{X} might be given the pre-context. Based on this predicted {X}, it generates a statement \textit{Y} that follows from the causal relationship: either a paraphrase from the input or an implied statement.

\paragraph{Masked Selected Sentence (\textsc{Mask X}).} This experiment gives as input the pre-context, post-context, and the dimension. The selected sentence is replaced with a token \texttt{<masked>}.  Here, we test the commonsense ability to generate two (specific) statements given most of the story---4 out of 5 sentences---but not the selected sentence \textit{X}. This will let us see how much of a performance boost the model is given by copying \textit{X} from the input.

As with \textsc{History}, for all dimensions, the model must first predict \textit{X}, then generate a paraphrased or implied statement \textit{Y} that is causally consistent.

\paragraph{History and Selected Sentence (\textsc{History+X}).} This experiment gives as input the pre-context, selected sentence, and dimension. This is used as a direct comparison to \textsc{History} except with selected sentence \textit{X} given as part of the input. Statement \textit{Y} is generated as it is in \textsc{History}.

For this diagnostic test, we drop entries in which the modifications result in input identical to the original task. For example, for \textsc{History+X}, we omit those entries where \textit{X} is the last sentence.

\begin{table}[t!]
    \small
    \setlength{\tabcolsep}{1.8pt}
    \begin{tabular}{l|ccc|ccc}
    \toprule
             model &  spec &  spec1-5 &  spec6-10 &  gen  &  gen1-5 &  gen6-10 \\
    \hline
     \textsc{Original} &           70.7 &             67.1 &              74.4 &         66.2 &            62.3 &             70.0 \\   
     \textsc{History} &          35.9 &             36.9 &              34.9 &         50.4 &            50.1 &             50.7 \\
    \textsc{Mask X} &           41.6 &             38.8 &              44.4 &         49.6 &            50.4 &             48.8 \\
    \textsc{History+X} &          68.3 &             66.2 &              70.4 &         65.5 &            61.8 &             69.3 \\
    
    \bottomrule
    \end{tabular}
    \caption{Test SacreBLEU scores for the diagnostic tasks. \textsc{Original} performs the best since it can access the entire input. As we keep the output and underlying T5 LM consistent but vary the input, the results' trends demonstrate how omitting different parts of the input affect BLEU scores.}
    \label{tab:results}
\end{table}

\subsection{Diagnostic Task Results}
Table~\ref{tab:results} compares the results of T5 models trained on the diagnostic tasks. We report test set results on the averaged dimensions 1-10, as well as averaged dimensions 1-5 (\textit{X} is the second statement), and 6-10 (\textit{X} is the first). Following \citet{mostafazadeh-etal-2020-glucose}, we use SacreBLEU~\cite{post-2018-call} with equal weights up to 4-grams. We report results for both specific and general rules, but focus on specific.

\textsc{Original}, of course, performs the best as its input has the most available information.  \textsc{History} and \textsc{Mask X} perform similarly to each other and far worse than the other diagnostic tasks. \textsc{History}, with only the pre-context, has a a 35-point BLEU gap for specific rules (16 for general) compared to \textsc{Original} averaged across all dimensions.
\begin{figure*}[ht]
    \centering
    \includegraphics[width=0.75\paperwidth]{figures/CIS2.png}
    \caption{Generation of \cissq{} labels from a GLUCOSE entry.
    The input story is highlighted in orange. Each story sentence is indexed by its position in the story. For example, the selected sentence \textit{X} (*Fred woke up late.*), surrounded with asterisks, is assigned the tag $\texttt{<s\textsubscript{0}>}$.
    The relation \texttt{>Causes/Enables>} is given automatically from the dimension. 
	The ``other'' sentence \textit{Y} is compared to each story sentence; the dashed lines represent sentence similarity scores, with the darkest line being the highest similarity. $\texttt{<s\textsubscript{1}>}$ is selected as the Sentence \textit{Y} tag.}
    \label{fig:glucose_cis2}
\end{figure*}
Adding to \textsc{History} multiple sentences of the post-context gives \textsc{Mask X}, and modest score gains (35.9 vs 41.6 specific). 
However, adding to \textsc{History} just the one selected sentence \textit{X} gives \textsc{History+X}, which performs very closely to \textsc{Original} for both specific and general rules (70.7 vs 68.3 specific). Furthermore, comparing trends between dimensions 1-5 and 6-10, we find that 6-10 scores are mostly higher, for both general and specific, than 1-5.

These results and their trends show that BLEU scores are highly contingent on having \textit{X} as input over all other sentences. 
Conflation always occurs for \textit{X}, since this is copied from the input, and conflation is also worse in cases where an incorrect statement \textit{Y} was generated but contains tokens that match the correct statement.
We believe it is unlikely that achieving \textapprox 35.9 BLEU on specific rules for \textsc{History} would mean that it is half as good at CCI than \textsc{Original}, with 70.7 BLEU specific.
We found that the fine-tuned T5 models perform some CCI, but BLEU scores are hard to interpret and can be unreliable. 
\paragraph{Specific vs. General Rule Performance}
Table~\ref{tab:results} shows that both \textsc{Original} and \textsc{History+X} perform better for specific rules than general. This matches the results seen in \cite{mostafazadeh-etal-2020-glucose}.
However, for \textsc{History} and \textsc{Mask X}, which both omit \textit{X}, the opposite trend occurs; general is higher than specific. This shows that copying and paraphrasing from the original text is in fact a conflating factor in the LM's BLEU performance.

\section{Contextual Commonsense Inference in Sentence Selection (\cissq)}
\label{ssec:cis2}
Given the extensive paraphrasing present in both the GLUCOSE task and the evaluation method, we design the Contextual Commonsense Inference in Sentence Selection (\cissq) task to abstract away language generation.
We recast the task as a classification problem, with the same 3 inputs as in \textsc{Original} (Equation~\ref{eq:input}), while the output becomes

\begin{equation} \label{eq:output_cis2}
\langle \texttt{<s\textsubscript{a}>}~\texttt{REL}~\texttt{<s\textsubscript{b}>} \rangle
\end{equation}

where \texttt{<s\textsubscript{a}>} and \texttt{<s\textsubscript{b}>} are tags corresponding to sentences from the original story, $a$ and $b$ are indices from $[0,4]$ and $a\neq b$. The output sequence comes from a limited vocabulary of 5 sentence index tokens, 5 causal dimension tokens,\footnote{\texttt{>Causes/Enables>}, \texttt{>Causes>}, \texttt{>Enables>}, \texttt{>Results in>}, \texttt{>Motivates>}} and the sentence index token corresponding to the selected sentence \textit{X} can be before or after the REL token, depending on what causal dimension is being used. The classification task is to choose the correct sequence of 100 possible output sequences.\footnote{20 (5P2) sentence tag combinations  * 5 relations = 100}

The abstracted output avoids the prior conflation issue since there are no partial matches within tokens of statements. Furthermore, there is no explicit correspondence between input and output. Note that \cissq{} does not distinguish between specific and general rules.

Finetuned \cissq{} models are forced to only learn the commonsense inference task. The input is kept the same, so the models see the same information as with the original task formulation. Therefore, we argue that \cissq{} is a simpler and fairer measurement of commonsense inference performance.

\subsection{GLUCOSE Entries to \cissq{} Tag Heuristic Conversion}
\label{ssec:ciss_gen}

To evaluate the \cissq{} formulation, we need to convert story sentences into \cissq{} output labels, as in Equation~\ref{eq:output_cis2}. See Figure~\ref{fig:glucose_cis2} for the conversion process.
Each sentence of an input story corresponds to a tag $\texttt{<s\textsubscript{0}>}$ to $\texttt{<s\textsubscript{4}>}$ with indexes corresponding its position in the story. To get the three \cissq{} output labels, we do the following: (1) Identify selected sentence \textit{X} from the input since it always be denoted as the sentence with the asterisks surrounding it. The input dimension informs the position of sentence \textit{X} in the output---whether is \texttt{<s\textsubscript{a}>} or \texttt{<s\textsubscript{b}>}; (2) Get the relation REL from the output directly; and (3) Calculate the similarity of ``other'' sentence \textit{Y} from the output to every other sentence in the input story and select the closest match.

To find the remaining token, we look at the specific rule from the original GLUCOSE task output, which consists of two statements separated by relation \texttt{REL}. We will call them \textit{P\textsubscript{0}} and P\textsubscript{1}. Suppose \textit{X} corresponds to \textit{P\textsubscript{0}}, and we need to find which sentence \textit{Y} corresponds to \textit{P\textsubscript{1}}. We do this by iterating over the sentences (excluding X), for each calculating its similarity with P\textsubscript{1}.  We take the index of the sentence with the highest similarity to \textit{P\textsubscript{1}} as \texttt{<s\textsubscript{b}>}. We describe our experiments with several sentence similarity metrics in Section~\ref{ssec:cis2_results}.

Being a heuristic approach, generated \cissq{} labels are not perfect. However, our manual inspection finds most labels are reasonable for GLUCOSE entries that have an explicit \textit{Y} (from the story). \cissq{} labels do not exist for those GLUCOSE entries with implicit relationships\footnote{\citet{mostafazadeh-etal-2020-glucose} estimate these are a minority.}, i.e. \textit{Y} is not in the original story. We attempted to filter these out by removing any training examples that did not pass a threshold\footnote{0.16 is the mean SBERT value across the train set.} of SBERT $\leq0.16$ for any sentence in the story. However, this resulted in a slight drop in the final evaluation, so these examples were kept.

We run the conversion method on the GLUCOSE train set and train a T5 model using the same hyperparameters used for our other models with the task of generating the three-token \cissq{} label, given the GLUCOSE input. We refer to this model as  \textsc{Cis\textsuperscript{2}-T5}. Note that although using \cissq{} tags turns this into a classification problem, the model is still doing generation to predict the output.

\subsection{\cissq{} Classification Task \& Results}
\label{ssec:cis2_results}
\begin{figure}[t]
    \centering
    \includegraphics[width=0.48\textwidth]{figures/cis2_results.png}
    \caption{\cissq{} accuracy results for Original and diagnostic GLUCOSE task models, and \cissq\textsc{-T5}. The dashed line shows Random Y Selection, a baseline that derives \textit{X} and the relation text from the input, and randomly selects \textit{Y}.}
    \label{fig:cis2_results}
\end{figure}
In Section~\ref{ssec:diagnostic}, we showed that BLEU is not an appropriate metric for the CCI task, given the GLUCOSE models' extensive copying and paraphrasing.
Furthermore, \cissq-T5 generates \cissq{} tags instead of full sentences, making it non-trivial to compare to the \textsc{Original} GLUCOSE T5 model.

We run the conversion method from Section~\ref{ssec:ciss_gen} on each model's specific rule output to obtain its predicted \cissq{} labels, and on the GLUCOSE test set to obtain the \cissq{} test set.\footnote{For future work we plan to obtain ground-truth test labels via crowdsourcing.} Both are now formatted as in Equation~\ref{eq:output_cis2}. This enables us to do an exact-match comparison between the model labels and the test set labels, and removes the associated issues with evaluating generated text.
In effect, the \cissq evaluation considers requires {\em the correct sentence \textit{Y} to be chosen}; there is no partial credit for those outputs that can easily be inferred from input: the selected sentence \textit{X}, and \texttt{REL}.

The sentence similarity metric used is crucial in the process of heuristically generating \cissq{} labels. We experimented with both BLEU scores of lemmatized tokens, as well as Sentence-BERT (SBERT)~\cite{reimers2019sentence}. By using BLEU for sentence similarity, GLUCOSE \textsc{Original} achieves 66.0\%, whereas \cissq-T5---despite being trained on these \cissq{} labels converted with BLEU---only achieves 57.2\% accuracy.
This stems from same issues of BLEU measuring language generation, rather than CCI, as discussed in Section~\ref{ssec:diagnostic}. Also, this shows that the \cissq{} classification task does not favor our \cissq{} system by default. 

Therefore, for the final evaluation we opt for SBERT, a more context-dependent similarity metric. Results for this evaluation are shown in Figure~\ref{fig:cis2_results}.
We compare all of our results to a random baseline which is the probability one of the 4 other story sentences is randomly selected for the index of \textit{Y}; this would have an accuracy of 25\% (the dashed horizontal line in Figure~\ref{fig:cis2_results}).
Out of all the models, \cissq-T5 achieves the highest score at 66.2\%, while \textsc{Original} is not far behind at 61.9\%. As for the diagnostic tasks, we see the same score ordering of models with BLEU evaluation. \textsc{History+X} scores 8\% lower than \textsc{Original}. \textsc{History} and \textsc{Mask X} perform even worse than random, indicating that their BLEU performance was largely due to partial token matches.\footnote{Experiments comparing \cissq~to models that are trained to generate only specific rules can be found in Appendix \ref{app:spec}.}

The best GLUCOSE model \textsc{Original} achieves 70.7 specific BLEU, but only 61.9\% \cissq{} accuracy. Although we cannot directly compare BLEU of generated output, and \cissq{} exact match accuracy, we have shown that \cissq{} provides a fairer estimate of CCI performance of these fine-tuned T5 models by removing language generation from evaluation. These \cissq{} results are promising, but there is still much room for improvement.
\section{Discussion}
The diagnostic tasks we discussed in the paper investigated the extent to which the original GLUCOSE task conflates language generation and contextual commonsense inference (CCI). We found that the most significant sentence of the input is the selected sentence \textit{X}, and if omitted, BLEU scores drop significantly compared to omitting other story sentences. This shows that the language model is relying on \textit{X} for CCI, as it should.
It is worth discussing how ``fair'' it is to remove \textit{X}---after all, without \textit{X}, the LMs have little to condition their predictions on. While this is true, we emphasize that our diagnostic tasks are intended to be taken together to analyze the extent of conflation. The main takeaway is that by including \textit{X}, trained models will rely on copying instead of good commonsense inference.

We have also shown evidence for extensive copying and paraphrasing as seen from the higher performance on specific rules relative to general rules for \textsc{Original} and \textsc{History+X}. These trends hold for \cissq{}  evaluation as well, but are even more marked since there is no inflation from matching tokens.

Lastly, we have shown that the T5 model trained on the GLUCOSE task (to maximize BLEU on the specific and general rules) performs only 4.3\% worse on the \cissq{} than one trained directly on \cissq{} labels. This shows that T5 can still learn significant CCI from the GLUCOSE data, and can further improve performance with \cissq{} converted labels, abstracting away with language generation.
\subsection{Future Work}
We plan to collect ground-truth \cissq{} labels via crowdsourcing for the entire test set, and for some training examples. To simplify the task, we will have workers verify, and correct if necessary, the heuristic \cissq{} labels.

Future work can further explore utilizing GLUCOSE and related datasets for story generation tasks.
One promising avenue to extending our CCI evaluation to story generation settings is incorporating our approach with the COINS framework \cite{paul-frank-2021-coins}, which generates contextualized inference rules to guide future output sentences. Abstracting these inference rules through \cissq{} would likely allow the language model to better capture and learn CCI.

We also resonate with question-answering based approaches to commonsense inference for stories \cite{lal-etal-2021-tellmewhy, Castricato2022}. \citet{lal-etal-2021-tellmewhy} trained large language models on their dataset, finding that they only perform well when the answers are present in the narrative. This finding goes hand in hand with our finding that the original GLUCOSE task formulation allows for easy paraphrasing and thus inflated performance.
\section{Conclusion}

This work investigated the extent to which language models learn contextual commonsense inference (CCI), utilizing the GLUCOSE~\cite{mostafazadeh-etal-2020-glucose} dataset and the T5~\cite{t5} language model as case studies. We showed how the original GLUCOSE task conflates language generation and CCI tasks, causing over-estimation of true CCI performance. We then formulated diagnostic tasks by permuting the original task and found that LMs rely on paraphrasing the selected sentence and context in making their predictions. 

We proposed \cissq~as an alternative task to structure and evaluate language models for CCI. \cissq{} evaluation is a simplified, fairer measurement of CCI performance than BLEU. By finetuning a T5 model on our \cissq~task, it correctly selects the causal statement 4.3\% more than a model trained on the original GLUCOSE task. We note this is using heuristically converted \cissq{} labels, and collecting ground-truth \cissq{} labels for training would lead to even better performance.

Overall, we found that GLUCOSE indeed encodes contextual commonsense information, and T5 has capacity to learn this. Therefore, the challenge for future researchers is to leverage GLUCOSE and other contextual commonsense inference datasets' knowledge representations appropriately and avoid conflation of language generation.


\bibliography{custom,anthology}
\bibliographystyle{acl_natbib}

\appendix
\clearpage

\begin{table*}[t]
    \setlength{\tabcolsep}{3pt}
    \begin{tabular}{llrrrrrrrrrrr}
    \toprule
            Model & Level &  avg &    1 &    2 &    3 &    4 &    5 &    6 &    7 &    8 &    9 &   10 \\
    \midrule
         \cite{mostafazadeh-etal-2020-glucose} & Specific & N/A &72.5 	&73.8 &	70.5 &	81.1 &	71.7 &	73.9 &	79.3 &	80.2 &	86.6 &	66.9 \\
        \cite{mostafazadeh-etal-2020-glucose} & General & N/A & 66.4 	&68.5 &	69.8 &	76.8 &	68.6 &	67.6 &	73.0 &	77.0 &	86.8 &	57.5 \\
        \midrule
         GLUCOSE TF-checkpoint & Specific & 75.7 & 71.9 & 69.8 & 75.8 & 75.9 & 73.3 & 75.2 & 79.8 & 80.2 & 85.5 & 69.9 \\
        GLUCOSE TF checkpoint & General & 70.1 & 66.4 & 66.4 & 70.1 & 72.1 & 70.0 & 69.2 & 71.6 & 72.4 & 82.0 & 61.0 \\
        \midrule
         replicated t5-large & Specific & 70.7 & 65.9 & 60.4 & 63.8 & 76.5 & 69.0 & 66.7 & 72.6 & 74.0 & 82.4 & 76.0 \\
        replicated t5-large & General& 66.2 & 61.3 & 59.9 & 60.4 & 68.8 & 61.3 & 60.5 & 65.0 & 68.1 & 75.8 & 80.4 \\
    \bottomrule
    \end{tabular}
    \caption{Test Set Results for the original GLUCOSE task. The first rows are the original results, the second are decoded by us using the provided GLUCOSE TF checkpoint, and the third are our best-effort replications.}
    \label{tab:replicated}
\end{table*}
\section{Appendix}
\label{sec:appendix}
\subsection{Acknowledgements}
We thank the authors of GLUCOSE, in particular Or Biran and Lori Moon, for their helpful assistance in working with the GLUCOSE dataset and codebase. We also thank Daphne Ippolito and the anonymous reviewers for their comments and suggestions.

This material is based upon work supported by the National Science Foundation under Grant \#2030859 to the Computing Research Association for the CIFellows Project.

\subsection{Ethical Considerations and Broader Impacts}
The methods used in our paper build in large part upon work by prior researchers. The T5~\cite{t5} language model we used was pretrained on a massive dataset for many days. Despite the energy usage, T5 has proved be a valuable tool that can be used for countless downstream NLP applications, ours included. As for our own trained models, we note that we further fine-tuned T5 on an array of diagnostic and custom tasks. During development, we made sure to pilot any experiments on smaller datasets, and we carefully managed our GPU and CPU usage throughout.

As for the data used, the ROCStories \cite{mostafazadeh-etal-2016-corpus} and GLUCOSE \cite{mostafazadeh-etal-2020-glucose} datasets, in which our work builds on, involved a great deal of careful task design and interaction with crowd-source workers. We thank these researchers for their ethical treatment of their crowdsource workers, with fair pay and two-way communication~\cite{moon-glucose-data}.

We will publicly release all our code, from data preprocessing, to model training, to final evaluation, to ensure that our work is fully reproducible.

The broader impacts of our work outside its immediate subject are several. First, our work takes a step towards analyzing stories, which are something fundamentally human, and that machines have yet to master. Second, we have encouraged NLP researchers in general to think more carefully about the structure of a task, before defaulting to the latest state-of-the-art language model. For example, we found that our \cissq{} task, which is simpler and thus requires less training resources than the language generation task, performs better on capturing contextual commonsense inference.

\subsection{Reproducing Our Work}
We make our code publicly available at \url{https://github.com/manestay/cis2}. The codebase includes complete preprocessing, training, and evaluation scripts, to take the raw GLUCOSE CSVs and T5 checkpoints, and train both diagnostic and \cissq{} models. We will also release the final trained checkpoints.

We also include our code to reproduce the original GLUCOSE experiments. We model this closely to the original GLUCOSE paper, starting from their provided code repository.

\subsection{Reproduction Results}
\label{ssec:repro}
We report the results we obtained on the original GLUCOSE task in Table~\ref{tab:replicated}. We report per-dimension BLEU, as was done prior, as well as the weighted average BLEU across all dimensions. We find that the reported numbers from ~\cite{mostafazadeh-etal-2020-glucose} and their provided Tensorflow checkpoint are essentially consistent.

Our replication results (done with the \texttt{transformers} package~\cite{wolf2019huggingface}) achieve 4-5 BLEU points lower, due to resource limitations and slight differences in experimental setup (i.e. we had far less GPU resources and and training time). For consistency's sake all of our experiments use the same setup as replicated t5-large (termed Original in the main text), and thus use this as the baseline.

We report results on the test set, but choose to evaluate BLEU on only the first of the three provided references for each test set entry. This is because the GLUCOSE train set only has one reference per entry, not 3, and we carved a small development set out of the train set, since no train/development split was provided. We evaluate our custom development and the original test set the same way, with 1 reference per entry.

\subsection{Training Setup and Hyperparameters}
\label{sec:hyperparams}
We trained our models on 2 NVIDIA Quadro RTX 6000 GPUs, with 24 GB vRAM each.
We train up to 10 epochs, early stopping after 10 checkpoints without improvement on the validation set. Depending on the task, the models finish training between 6 to 34 hours. The GLUCOSE authors trained their model far more -- for 72 hours on 8 TPUs -- which can explain our lower BLEU scores.

We use the exact same hyperparameters as in~\citet{t5}, following~\citet{mostafazadeh-etal-2020-glucose}, with one major exception: we use a learning rate of 1e-4 instead of 1e-3, which we found to converge too quickly. 

\subsection{Specific-Only Results}
\label{app:spec}

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.7\textwidth]{figures/cis2_results_appendix.png}
    \caption{\cissq{} accuracy results, comparing specific+general models vs. specific-only models. The specific+general results are the same as in Figure~\ref{fig:cis2_results}.}
    \label{fig:cis2_results_appendix}
\end{figure*}

\begin{table}[t]
    \small
    \setlength{\tabcolsep}{1.8pt}
    \begin{tabular}{l|ccc|ccc}
    \toprule
             model &  spec &  sp1-5 &  sp6-10 &  gen  &  ge1-5 &  ge6-10 \\
    \hline
     \textsc{Original} &           70.7 &             67.1 &              74.4 &         66.2 &            62.3 &             70.0 \\   
     \textsc{History} &          35.9 &             36.9 &              34.9 &         50.4 &            50.1 &             50.7 \\
    \textsc{Mask X} &           41.6 &             38.8 &              44.4 &         49.6 &            50.4 &             48.8 \\
    \textsc{History+X} &          68.3 &             66.2 &              70.4 &         65.5 &            61.8 &             69.3 \\\hline
    \textsc{Original-Spec} &          67.6 &             60.5 &              74.8 &         NA &            NA &             NA \\
    \textsc{History-Spec} &          37.6 &             36.1 &              39.0 &          NA &             NA &              NA \\
    \textsc{Mask X-Spec} &          42.5 &             41.3 &              43.8 &          NA &             NA &              NA \\
    \textsc{History+X-Spec} &          65.6 &             62.0 &              69.3 &          NA &             NA &              NA \\
    \bottomrule
    \end{tabular}
    \caption{Test SacreBLEU scores for all tasks. The first 4 rows are the same as in Table~\ref{tab:results}---the models that outputted both specific and general rules. The last 4 rows are for models outputting specific rules only.}
    \label{tab:results_spec}
\end{table}

Given that \cissq{} only considers the specific rule, one may ask how the GLUCOSE models trained to generate only specific rules would perform. We therefore train 4 ``specific-only'' models, one for each of the 4 diagnostic tasks of Section~\ref{ssec:diagnostic}. We denote specific-only models with the suffix \textsc{-Spec} and we compare the results to the specific+general models (as in the main text) without a suffix.

Table~\ref{tab:results_spec} compares the BLEU results, whereas Figure~\ref{fig:cis2_results_appendix} compares the \cissq{} results.
We see that the specific+general models and the specific-only models perform similarly. This confirms the findings of~\citet{mostafazadeh-etal-2020-glucose}, where T5 can effectively learn both specific and general rules jointly. As both BLEU scores and \cissq{} classification accuracy are similar, we report the specific+general model results in the main paper to be consistent with prior work.
\begin{table*}[ht]
    \centering
    \small
    \setlength{\tabcolsep}{3pt}
    \begin{tabular}{lp{.36\textwidth}p{.36\textwidth} l l}
    \toprule
    \textbf{Task} & \textbf{Input} & \textbf{Output} & \textbf{Specific} & \textbf{General} \\
    \midrule
    \textsc{Original} & 1: My mother told me to fix the car. I was unable to do this right away. \textbf{* I could not find my tools. *} I looked everywhere for them. It turns out they were stolen the night before. & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} & 70.7 & 66.2 \\ \hline
    
    \textsc{History} & 1: My mother told me to fix the car. I was unable to do this right away.  & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} & 35.9 & 50.4\\ \hline
    
    \textsc{Mask X} & My mother told me to fix the car. I was unable to do this right away. \texttt{<masked>} I looked everywhere for them. It turns out they were stolen the night before. & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} & 41.6 & 49.6\\\hline
    
   \textsc{History+X} & 1: My mother told me to fix the car. I was unable to do this right away. \textbf{* I could not find my tools. *} & They were stolen the night before >Causes/Enables> \textbf{I could not find my tools} ** 
    Something\textsubscript{A}  is stolen >Causes/Enables> Someone\textsubscript{A} cannot find Something\textsubscript{A} & 68.3 & 65.5 \\\hline\hline
    \cissq & 1: My mother told me to fix the car. I was unable to do this right away. \textbf{* I could not find my tools. *} I looked everywhere for them. It turns out they were stolen the night before. & \texttt{<s\textsubscript{4}> >Causes/Enables> <s\textsubscript{2}>} \\
    \bottomrule
    \end{tabular}
    \caption{Table with I/O \& BLEU}
    \label{tab:tasks_bleu}
\end{table*}

\end{document}


Peer review:
The paper pinpointed a valid issue with the existing CCI task formulation. However, the findings are somewhat obvious, and the newly designed task might not generalize to settings where the CS inference is implicit and not part of the given story.
This paper critiques existing methods on Contextual Commonsense Inference (CCI), which conflates generation and reasoning tasks. The authors propose reframing the CCI task as a classification task (called Cis2) to isolate commonsense reasoning from the generation. This helps in evaluating the commonsense inference ability of a model irrespective of its generation performance. For this, they convert story sentences into output tags which avoids a partial match between input and output sequences. The model is then required to generate an abstracted output which contains story sentences' tags instead of full sequences. 

Pros:
* It is important to evaluate the reasoning abilities of models in isolation from their generation abilities and the author pinpointed a valid issue with original GLUCOSE task formulation.


Cons:
* Most of the findings from the diagnostic tests are obvious and expected (see comments). 
* Too much content is provided about Mostafazadeh et al. 2020 which could be easily skipped and referred to the original paper.
* It is not clear how the newly designed task formulation handles cases where the inference output Y is not explicitly stated in the given story.

Comments:
1- How does your task reformulation handle cases where the inference output Y is not explicitly stated in the story? As the authors mention in Line 196, the CS inference Y might or might not be part of the story. And example from the original GLUCOSE paper is:
“Gage wants safety” Causes/Enables “Gage turned his bike”, while “Gage wants safety” is never stated in the story and should be inferred thus can not be replaced by a tag from the story.

2- Line 259: while I agree with the authors that the original task formulation suffers from conflation of CCI and language generation tasks, I think this can be solved mostly by 1) removing the selected sentence X from the output, and 2) including a better evaluation metrics that accounts for semantic similarity such as BertScore. 

3- Line 367-369: Isn't this obvious? if in the training data, output always copy/paraphrase X, it's expected that the model learns this pattern and consequently the BLEU score would be high. The issue is why the model should generate X in the first place. Without including X in the output (no matter if X is in the input or not) the evaluation using n-gram overlap would be less unreliable.

4- Line 380: In my opinion copying is an easier task.

5- The first 5.5 pages are allocated for background and related work and only on page 6 the author started to talk about their proposed task Cis2. 

6- It is helpful to explicitly mention somewhere in the paper that you are using a generative classifier where the model GENERATES one of the 100 possible output sequences (using T5) and it’s not a 100-way classification task.

7- Line 449: For what portion of the original data could the authors find output Y explicitly mentioned in the input story? And why not discard those with very low similarity scores?

Typo: 

Line 406: the The → The

Line 407: footnote after punctuation.
Rating: 6: Marginally above acceptance threshold
Confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct