Introduction
Although deep neural networks have achieved remarkable successes (e.g., BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ), their dependence on supervised learning has been challenged as a significant weakness. This dependence prevents deep neural networks from being applied to problems where labeled data is scarce. An example of such problems is common sense reasoning, such as the Winograd Schema Challenge BIBREF0 , where the labeled set is typically very small, on the order of hundreds of examples. Below is an example question from this dataset:
Although it is straightforward for us to choose the answer to be "the trophy" according to our common sense, answering this type of question is a great challenge for machines because there is no training data, or very little of it.
In this paper, we present a surprisingly simple method for common sense reasoning with Winograd schema multiple choice questions. Key to our method is th e use of language models (LMs), trained on a large amount of unlabeled data, to score multiple choice questions posed by the challenge and similar datasets. More concretely, in the above example, we will first substitute the pronoun ("it") with the candidates ("the trophy" and "the suitcase"), and then use LMs to compute the probability of the two resulting sentences ("The trophy doesn’t fit in the suitcase because the trophy is too big." and "The trophy doesn’t fit in the suitcase because the suitcase is too big."). The substitution that results in a more probable sentence will be the correct answer.
A unique feature of Winograd Schema questions is the presence of a special word that decides the correct reference choice. In the above example, "big" is this special word. When "big" is replaced by "small", the correct answer switches to "the suitcase". Although detecting this feature is not part of the challenge, further analysis shows that our system successfully discovers this special word to make its decisions in many cases, indicating a good grasp of commonsense knowledge.
Related Work
Unsupervised learning has been used to discover simple commonsense relationships. For example, Mikolov et al. BIBREF15 , BIBREF16 show that by learning to predict adjacent words in a sentence, word vectors can be used to answer analogy questions such as: Man:King::Woman:?. Our work uses a similar intuition that language modeling can naturally capture common sense knowledge. The difference is that Winograd Schema questions require more contextual information, hence our use of LMs instead of just word vectors.
Neural LMs have also been applied successfully to improve downstream applications BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 . In BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , researchers have shown that pre-trained LMs can be used as feature representations for a sentence, or a paragraph to improve NLP applications such as document classification, machine translation, question answering, etc. The combined evidence suggests that LMs trained on a massive amount of unlabeled data can capture many aspects of natural language and the world's knowledge, especially commonsense information.
Previous attempts on solving the Winograd Schema Challenge usually involve heavy utilization of annotated knowledge bases, rule-based reasoning, or hand-crafted features BIBREF21 , BIBREF22 , BIBREF23 . In particular, Rahman and Ng BIBREF24 employ human annotators to build more supervised training data. Their model utilizes nearly 70K hand-crafted features, including querying data from Google Search API. Sharma et al. BIBREF25 rely on a semantic parser to understand the question, query texts through Google Search, and reason on the graph produced by the parser. Similarly, Schüller BIBREF23 formalizes the knowledge-graph data structure and a reasoning process based on cognitive linguistics theories. Bailey et al. BIBREF22 introduces a framework for reasoning, using expensive annotated knowledge bases as axioms.
The current best approach makes use of the skip-gram model to learn word representations BIBREF26 . The model incorporates several knowledge bases to regularize its training process, resulting in Knowledge Enhanced Embeddings (KEE). A semantic similarity scorer and a deep neural network classifier are then combined on top of KEE to predict the answers. The final system, therefore, includes both supervised and unsupervised models, besides three different knowledge bases. In contrast, our unsupervised method is simpler while having significantly higher accuracy. Unsupervised training is done on text corpora which can be cheaply curated.
Using language models in reading comprehension tests also produced many great successes. Namely Chu et al. BIBREF27 used bi-directional RNNs to predict the last word of a passage in the LAMBADA challenge. Similarly, LMs are also used to produce features for a classifier in the Store Close Test 2017, giving best accuracy against other methods BIBREF28 . In a broader context, LMs are used to produce good word embeddings, significantly improved a wide variety of downstream tasks, including the general problem of question answering BIBREF19 , BIBREF29 .
Methods
We first substitute the pronoun in the original sentence with each of the candidate choices. The problem of coreference resolution then reduces to identifying which substitution results in a more probable sentence. By reframing the problem this way, language modeling becomes a natural solution by its definition. Namely, LMs are trained on text corpora, which encodes human knowledge in the form of natural language. During inference, LMs are able to assign probability to any given text based on what they have learned from training data. An overview of our method is shown in Figure 1 .
Suppose the sentence $S$ of $n$ consecutive words has its pronoun to be resolved specified at the $k^{th}$ position: $S = \lbrace w_1, .., w_{k-1}, w_{k} \equiv p, w_{k+1}, .., w_{n}\rbrace $ . We make use of a trained language model $P_\theta (w_t | w_{1}, w_2, .., w_{t-1})$ , which defines the probability of word $w_t$ conditioned on the previous words $w_1, ..., w_{t-1}$ . The substitution of a candidate reference $c$ in to the pronoun position $k$ results in a new sentence $S_{w_k\leftarrow c}$ (we use notation $n$0 to mean that word $n$1 is substituted by candidate $n$2 ). We consider two different ways of scoring the substitution:
which scores how probable the resulting full sentence is, and
which scores how probable the part of the resulting sentence following $c$ is, given its antecedent. In other words, it only scores a part $S_{w_k\leftarrow c}$ conditioned on the rest of the substituted sentence. An example of these two scores is shown in Table 1 . In our experiments, we find that partial scoring strategy is generally better than the naive full scoring strategy.
Experimental settings
In this section we describe tests for commonsense reasoning and the LMs used to solve these tasks. We also detail training text corpora used in our experiments.
Main results
Our experiments start with testing LMs trained on all text corpora with PDP-60 and WSC-273. Next, we show that it is possible to customize training data to obtain even better results.
The first challenge in 2016: PDP-60
We first examine unsupervised single-model resolvers on PDP-60 by training one character-level and one word-level LM on the Gutenberg corpus. In Table 2 , these two resolvers outperform previous results by a large margin. For this task, we found full scoring gives better results than partial scoring. In Section "Partial scoring is better than full scoring." , we provide evidences that this is an atypical case due to the very small size of PDP-60.
Next, we allow systems to take in necessary components to maximize their test performance. This includes making use of supervised training data that maps commonsense reasoning questions to their correct answer. Here we simply train another three variants of LMs on LM-1-Billion, CommonCrawl, and SQuAD and ensemble all of them. As reported in Table 3 , this ensemble of five unsupervised models outperform the best system in the 2016 competition (58.3%) by a large margin. Specifically, we achieve 70.0% accuracy, better than the more recent reported results from Quan Liu et al (66.7%) BIBREF26 , who makes use of three knowledge bases and a supervised deep neural network.
Winograd Schema Challenge
On the harder task WSC-273, our single-model resolvers also outperform the current state-of-the-art by a large margin, as shown in Table 4 . Namely, our word-level resolver achieves an accuracy of 56.4%. By training another 4 LMs, each on one of the 4 text corpora LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and add to the previous ensemble, we are able to reach 61.5%, nearly 10% of accuracy above the previous best result. This is a drastic improvement considering this previous best system outperforms random guess by only 3% in accuracy.
This task is more difficult than PDP-60. First, the overall performance of all competing systems are much lower than that of PDP-60. Second, incorporating supervised learning and expensive annotated knowledge bases to USSM provides insignificant gain this time (+3%), comparing to the large gain on PDP-60 (+19%).
Customized training data for Winograd Schema Challenge
As previous systems collect relevant data from knowledge bases after observing questions during evaluation BIBREF24 , BIBREF25 , we also explore using this option. Namely, we build a customized text corpus based on questions in commonsense reasoning tasks. It is important to note that this does not include the answers and therefore does not provide supervision to our resolvers. In particular, we aggregate documents from the CommonCrawl dataset that has the most overlapping n-grams with the questions. The score for each document is a weighted sum of $F_1(n)$ scores when counting overlapping n-grams: $Similarity\_Score_{document} = \frac{\sum _{n=1}^4nF_1(n)}{\sum _{n=1}^4n}$
The top 0.1% of highest ranked documents is chosen as our new training corpus. Details of the ranking is shown in Figure 2 . This procedure resulted in nearly 1,000,000 documents, with the highest ranking document having a score of $8\times 10^{-2}$ , still relatively small to a perfect score of $1.0$ . We name this dataset STORIES since most of the constituent documents take the form of a story with long chain of coherent events.
We train four different LMs on STORIES and add them to the previous ensemble of 10 LMs, resulting in a gain of 2% accuracy in the final system as shown in Table 5 . Remarkably, single models trained on this corpus are already extremely strong, with a word-level LM achieving 62.6% accuracy, even better than the ensemble of 10 models previously trained on 4 other text corpora (61.5%).
Discovery of special words in Winograd Schema
We introduce a method to potentially detect keywords at which our proposed resolvers make decision between the two candidates $c_{correct}$ and $c_{incorrect}$ . Namely, we look at the following ratio: $q_t = \frac{P_\theta (w_t | w_1, w_2, ..., w_{t-1}; w_k \leftarrow c_{correct})}{P_\theta (w_t | w_1, w_2, ..., w_{t-1}; w_k \leftarrow c_{incorrect})}$
Where $1 \le t \le n$ for full scoring, and $k +1 \le t \le n$ for partial scoring. It follows that the choice between $c_{correct}$ or $c_{incorrect}$ is made by the value of $Q = \prod _tq_t$ being bigger than $1.0$ or not. By looking at the value of each individual $q_t$ , it is possible to retrieve words with the largest values of $q_t$ and hence most responsible for the final value of $Q$ .
We visualize the probability ratios $q_t$ to have more insights into the decisions of our resolvers. Figure 3 displays a sample of incorrect decisions made by full scoring and is corrected by partial scoring. Interestingly, we found $q_t$ with large values coincides with the special keyword of each Winograd Schema in several cases. Intuitively, this means the LMs assigned very low probability for the keyword after observing the wrong substitution. It follows that we can predict the keyword in each the Winograd Schema question by selecting top word positions with the highest value of $q_t$ .
For questions with keyword appearing before the reference, we detect them by backward-scoring models. Namely, we ensemble 6 LMs, each trained on one text corpora with word order reversed. This ensemble also outperforms the previous best system on WSC-273 with a remarkable accuracy of 58.2%. Overall, we are able to discover a significant amount of special keywords (115 out of 178 correctly answered questions) as shown in Table 6 . This strongly indicates a correct understanding of the context and a good grasp of commonsense knowledge in the resolver's decision process.
Partial scoring is better than full scoring.
In this set of experiments, we look at wrong predictions from a word-level LM. With full scoring strategy, we observe that $q_t$ at the pronoun position is most responsible for a very large percentage of incorrect decisions as shown in Figfure 3 and Table 7 . For example, with the test "The trophy cannot fit in the suitcase because it is too big.", the system might return $c_{incorrect} = $ "suitcase" simply because $c_{correct} = $ "trophy" is a very rare word in its training corpus and therefore, is assigned a very low probability, overpowering subsequent $q_t$ values.
Following this reasoning, we apply a simple fix to full scoring by normalizing its score with the unigram count of $c$ : $Score_{full~normalized} = Score_{full} / Count(c)$ . Partial scoring, on the other hand, disregards $c$ altogether. As shown in Figure 4 , this normalization fixes full scoring in 9 out of 10 tested LMs on PDP-122. On WSC-273, the result is very decisive as partial scoring strongly outperforms the other two scoring in all cases. Since PDP-122 is a larger superset of PDP-60, we attribute the different behaviour observed on PDP-60 as an atypical case due to its very small size.
Importance of training corpus
In this set of experiments, we examine the effect of training data on commonsense reasoning test performance. Namely, we train both word-level and character-level LMs on each of the five corpora: LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and STORIES. A held-out dataset from each text corpus is used for early stopping on the corresponding training data.
To speed up training on these large corpora, we first train the models on the LM-1-Billion text corpus. Each trained model is then divided into three groups of parameters: Embedding, Recurrent Cell, and Softmax. Each of the three is optionally transferred to train the same architectures on CommonCrawl, SQuAD and Gutenberg Books. The best transferring combination is chosen by cross-validation.
Figure 5 -left and middle show that STORIES always yield the highest accuracy for both types of input processing. We next rank the text corpora based on ensemble performance for more reliable results. Namely, we compare the previous ensemble of 10 models against the same set of models trained on each single text corpus. This time, the original ensemble trained on a diverse set of text corpora outperforms all other single-corpus ensembles including STORIES. This highlights the important role of diversity in training data for commonsense reasoning accuracy of the final system.
Conclusion
We introduce a simple unsupervised method for Commonsense Reasoning tasks. Key to our proposal are large language models, trained on a number of massive and diverse text corpora. The resulting systems outperform previous best systems on both Pronoun Disambiguation Problems and Winograd Schema Challenge. Remarkably on the later benchmark, we are able to achieve 63.7% accuracy, comparing to 52.8% accuracy of the previous state-of-the-art, who utilizes supervised learning and expensively annotated knowledge bases. We analyze our system's answers and observe that it discovers key features of the question that decides the correct answer, indicating good understanding of the context and commonsense knowledge. We also demonstrated that ensembles of models benefit the most when trained on a diverse set of text corpora.
We anticipate that this simple technique will be a strong building block for future systems that utilize reasoning ability on commonsense knowledge.
Recurrent language models
The base model consists of two layers of Long-Short Term Memory (LSTM) BIBREF31 with 8192 hidden units. The output gate of each LSTM uses peepholes and a projection layer to reduce its output dimensionality to 1024. We perform drop-out on LSTM's outputs with probability 0.25.
For word inputs, we use an embedding lookup of 800000 words, each with dimension 1024. For character inputs, we use an embedding lookup of 256 characters, each with dimension 16. We concatenate all characters in each word into a tensor of shape (word length, 16) and add to its two ends the <begin of word> and <end of word> tokens. The resulting concatenation is zero-padded to produce a fixed size tensor of shape (50, 16). This tensor is then processed by eight different 1-D convolution (Conv) kernels of different sizes and number of output channels, listed in Table 8 , each followed by a ReLU acitvation. The output of all CNNs are then concatenated and processed by two other fully-connected layers with highway connection that persist the input dimensionality. The resulting tensor is projected down to a 1024-feature vector. For both word input and character input, we perform dropout on the tensors that go into LSTM layers with probability 0.25.
We use a single fully-connected layer followed by a $Softmax$ operator to process the LSTM's output and produce a distribution over word vocabulary of size 800K. During training, LM loss is evaluated using importance sampling with negative sample size of 8192. This loss is minimized using the AdaGrad BIBREF37 algorithm with a learning rate of 0.2. All gradients on LSTM parameters and Character Embedding parameters are clipped by their global norm at 1.0. To avoid storing large matrices in memory, we shard them into 32 equal-sized smaller pieces. In our experiments, we used 8 different variants of this base model as listed in Table 9 .
In Table 10 , we listed all LMs and their training text corpora used in each of the experiments in Section "Main results" .
Data contamination in CommonCrawl
Using the similarity scoring technique in section "Customized training data for Winograd Schema Challenge" , we observe a large amount of low quality training text on the lower end of the ranking. Namely, these are documents whose content are mostly unintelligible or unrecognized by our vocabulary. Training LMs for commonsense reasoning tasks on full CommonCrawl, therefore, might not be ideal. On the other hand, we detected and removed a portion of PDP-122 questions presented as an extremely high ranked document.

Question: Which of their training domains improves performance the most?
Answer: documents from the CommonCrawl dataset that has the most overlapping n-grams with the question

Question: Do they fine-tune their model on the end task?
Answer: Yes