Pre-trained language representation models, including feature-based methods BIBREF0 , BIBREF1 and fine-tuning methods BIBREF2 , BIBREF3 , BIBREF4 , can capture rich language information from text and then benefit many NLP tasks. Bidirectional Encoder Representations from Transformers (BERT) BIBREF4 , as one of the most recently developed models, has produced the state-of-the-art results by simple fine-tuning on various NLP tasks, including named entity recognition (NER) BIBREF5 , text classification BIBREF6 , natural language inference (NLI) BIBREF7 , question answering (QA) BIBREF8 , BIBREF9 , and has achieved human-level performances on several datasets BIBREF8 , BIBREF9 .However, commonsense reasoning is still a challenging task for modern machine learning methods. For example, recently BIBREF10 proposed a commonsense-related task, CommonsenseQA, and showed that the BERT model accuracy remains dozens of points lower than human accuracy on the questions about commonsense knowledge. Some examples from CommonsenseQA are shown in Table 1 part A. As can be seen from the examples, although it is easy for humans to answer the questions based on their knowledge about the world, it is a great challenge for machines when there is limited training data.We hypothesize that exploiting knowledge graphs for commonsense in QA modeling can help model choose correct answers. For example, as shown in the part B of Table 1 , some triples from ConceptNet BIBREF11 are quite related to the questions above. Exploiting these triples in the QA modeling may benefit the QA models to make a correct decision.In this paper, we propose a pre-training approach that can leverage commmonsense knowledge graphs, such as ConceptNet BIBREF11 , to improve the commonsense reasoning capability of language representation models, such as BERT. And at the same time, the proposed approach targets maintaining comparable performances on other NLP tasks with the original BERT models. It is challenging to incorporate the commonsense knowledge into language representation models since the commonsense knowledge is represented as a structured format, such as (concept $_1$ , relation, concept $_2$ ) in ConceptNet, which is inconsistent with the data used for pre-training language representation models. For example, BERT is pre-trained on the BooksCorpus and English Wikipedia that are composed of unstructured natural language sentences.To tackle the challenge mentioned above, inspired by the distant supervision approach BIBREF12 , we propose the “align, mask and select" (AMS) method that can align the commonsense knowledge graphs with a large text corpus to construct a dataset consisting of sentences with labeled concepts. Different from the pre-training tasks for BERT, the masked language model (MLM) and next sentence prediction (NSP) tasks, we use the generated dataset in a multi-choice question answering task. We then pre-train the BERT model on this dataset with the multi-choice question answering task and fine-tune it on various commonsense-related tasks, such as CommonsenseQA BIBREF10 and Winograd Schema Challenge (WSC) BIBREF13 , and achieve significant improvements. We also fine-tune and evaluate the pre-trained models on other NLP tasks, such as sentence classification and NLI tasks, such as GLUE BIBREF6 , and achieve comparable performance with the original BERT models.In summary, the contributions of this paper are threefold. First, we propose a pre-training approach for incorporating commonsense knowledge into language representation models for improving the commonsense reasoning capabilities of these models. Second, We propose an “align, mask and select" (AMS) method, inspired by the distant supervision approaches, to automatically construct a multi-choice question answering dataset. Third, Experiments demonstrate that the pre-trained model from the proposed approach with fine-tuning achieves significant performance improvements on several commonsense-related tasks, such as CommonsenseQA BIBREF10 and Winograd Schema Challenge BIBREF13 , and still maintains comparable performances on several sentence classification and NLI tasks in GLUE BIBREF6 .Language representation models have demonstrated their effectiveness for improving many NLP tasks. These approaches can be categorized into feature-based approaches and fine-tuning approaches. The early Word2Vec BIBREF14 and Glove models BIBREF0 focused on feature-based approaches to transform words into distributed representations. However, these methods suffered from the insufficiency for word disambiguation. BIBREF15 further proposed Embeddings from Language Models (ELMo) that derive context-aware word vectors from a bidirectional LSTM, which is trained with a coupled language model (LM) objective on a large text corpus.The fine-tuning approaches are different from the above-mentioned feature-based language approaches which only use the pre-trained language representations as input features. BIBREF2 pre-trained sentence encoders from unlabeled text and fine-tuned for a supervised downstream task. BIBREF3 proposed a generative pre-trained Transformer BIBREF16 (GPT) to learn language representations. BIBREF4 proposed a deep bidirectional model with multi-layer Transformers (BERT), which achieved the state-of-the-art performance for a wide variety of NLP tasks. The advantage of these approaches is that few parameters need to be learned from scratch.Though both feature-based and fine-tuning language representation models have achieved great success, they did not incorporate the commonsense knowledge. In this paper, we focus on incorporate commonsense knowledge into pre-training of language representation models.Commonsense reasoning is a challenging task for modern machine learning methods. As demonstrated in recent work BIBREF17 , incorporating commonsense knowledge into question answering models in a model-integration fashion helped improve commonsense reasoning ability. Instead of ensembling two independent models as in BIBREF17 , an alternative direction is to directly incorporate commonsense knowledge into an unified language representation model. BIBREF18 proposed to directly pre-training BERT on commonsense knowledge triples. For any triple (concept $_1$ , relation, concept $_2$ ), they took the concatenation of concept $_1$ and relation as the question and concept $_2$ as the correct answer. Distractors were formed by randomly picking words or phrases in the ConceptNet. In this work, we also investigate directly incorporating commonsense knowledge into an unified language representation model. However, we hypothesize that the language representations learned in BIBREF18 may be tampered since the inputs to the model constructed this way are not natural language sentences. To address this issue, we propose a pre-training approach for incorporating commonsense knowledge that includes a method to construct large-scale, natural language sentences. BIBREF19 collected the Common Sense Explanations (CoS-E) dataset using Amazon Mechanical Turk and applied a Commonsense Auto-Generated Explanations (CAGE) framework to language representation models, such as GPT and BERT. However, collecting this dataset used a large amount of human efforts. In contrast, in this paper, we propose an “align, mask and select" (AMS) method, inspired by the distant supervision approaches, to automatically construct a multi-choice question answering dataset.The distant supervision approach was originally proposed for generating training data for the relation classification task. The distant supervision approach BIBREF12 assumes that if two entities/concepts participate in a relation, all sentences that mention these two entities/concepts express that relation. Note that it is inevitable that there exists noise in the data labeled by distant supervision BIBREF20 . In this paper, instead of employing the relation labels labeled by distant supervision, we focus on the aligned entities/concepts. We propose the AMS method to construct a multi-choice QA dataset that align sentences with commonsense knowledge triples, mask the aligned words (entities/concepts) in sentences and treat the masked sentences as questions, and select several entities/concepts from knowledge graphs as candidate choices.This section describes the commonsense knowledge base investigated in our experiments. We use the ConceptNet BIBREF11 , one of the most widely used commonsense knowledge bases. ConceptNet is a semantic network that represents the large sets of words and phrases and the commonsense relationships between them. It contains over 21 million edges and over 8 million nodes. Its English vocabulary contains approximately 1,500,000 nodes, and for 83 languages, it contains at least 10,000 nodes for each of them, respectively. ConceptNet contains a core of 36 relations.Each instance in ConceptNet can be generally represented as a triple $r_i$ = (concept $_1$ , relation, concept $_2$ ), indicating relation between the two concepts concept $_1$ and concept $_2$ . For example, the triple (semicarbazide, IsA, chemical compound) means that “semicarbazide is a kind of chemical compounds"; the triple (cooking dinner, Causes, cooked food) means that “the effect of cooking dinner is cooked food", etc.In this section, we describe the details of constructing the commonsense-related multi-choice question answering dataset. Firstly, we filter the triples in ConceptNet with the following steps: (1) Filter triples in which one of the concepts is not English words. (2) Filter triples with the general relations “RelatedTo" and “IsA", which hold a large proportion in ConceptNet. (3) Filter triples in which one of the concepts has more than four words or the edit distance between the two concepts is less than four. After filtering, we obtain 606,564 triples.Each training sample is generated by three steps: align, mask and select, which we call as AMS method. Each sample in the dataset consists of a question and several candidate answers, which has the same form as the CommonsenseQA dataset. An example of constructing one training sample by masking concept $_2$ is shown in Table 2 .Firstly, we align each triple (concept $_1$ , relation, concept $_2$ ) from ConceptNet to the English Wikipedia dataset to extract the sentences with their concepts labeled. Secondly, we mask the concept $_1$ /concept $_2$ in one sentence with a special token [QW] and treat this sentence as a question, where QW is a replacement word of the question words “what", “where", etc. And the masked concept $_1$ /concept $_2$ is the correct answer for this question. Thirdly, for generating the distractors, BIBREF18 proposed a method to form distractors by randomly picking words or phrases in ConceptNet. In this paper, in order to generate more confusing distractors than the random selection approach, we request those distractors and the correct answer share the same concept $_2$ or concept $_1$ and the relation. That is to say, we search ( $\ast $ , relation, concept $_2$ ) and (concept $_2$0 , relation, $_2$1 ) in ConceptNet to select the distractors instead of random selection, where $_2$2 is a wildcard character that can match any word or phrase. For each question, we reserve four distractors and one correct answer. If there are less than four matched distractors, we discard this question instead of complementing it with random selection. If there are more than four distractors, we randomly select four distractors from them. After applying the AMS method, we create 16,324,846 multi-choice question answering samples.We investigate a multi-choice question-answering task for pre-training the English BERT base and BERT large models released by Google on our constructed dataset. The resulting models are denoted BERT_CS $_{base}$ and BERT_CS $_{large}$ , respectively. We then investigate the performance of fine-tuning the BERT_CS models on several NLP tasks, including commonsense-related tasks and common NLP tasks, presented in Section "Experiments" .To reduce the large cost of training BERT_CS models from scratch, we initialize the BERT_CS models (for both BERT $_{base}$ and BERT $_{large}$ models) with the parameter weights released by Google. We concatenate the question with each answer to construct a standard input sequence for BERT_CS (i.e., “[CLS] the largest [QW] by ... ? [SEP] city [SEP]”, where [CLS] and [SEP] are two special tokens), and the hidden representations over the [CLS] token are run through a softmax layer to create the predictions.The objective function is defined as follows: $$L = - {\rm logp}(c_i|s),$$   (Eq. 10) $${\rm p}(c_i|s) = \frac{{\rm exp}(\mathbf {w}^{T}\mathbf {c}_{i})}{\sum _{k=1}^{N}{\rm exp}(\mathbf {w}^{T}\mathbf {c}_{k})},$$   (Eq. 11) where $c_i$ is the correct answer, $\mathbf {w}$ are the parameters in the softmax layer, N is the total number of all candidates, and $\mathbf {c}_i$ is the vector representation of the special token [CLS]. We pre-train BERT_CS models with the batch size 160, the initial learning rate $2e^{-5}$ and the max sequence length 128 for 1 epoch. The pre-training is conducted on 16 NVIDIA V100 GPU cards with 32G memory for about 3 days for the BERT_CS $_{large}$ model and 1 day for the BERT_CS $_{base}$ model.In this section, we investigate the performance of fine-tuning the BERT_CS models on several NLP tasks. Note that when fine tuning on multi-choice QA tasks, e.g., CommonsenseQA and Winograd Schema Challenge (see section 5.3), we fine-tune all parameters in BERT_CS, including the last softmax layer from the token [CLS]; whereas, for other tasks, we randomly initialize the classifier layer and train it from scratch.Additionally, as described in BIBREF4 , fine-tuning on BERT sometimes is observed to be unstable on small datasets, so we run experiments with 5 different random seeds and select the best model based on the development set for all of the fine-tuning experiments in this section.In this subsection, we conduct experiments on a commonsense-related multi-choice question answering benchmark, the CommonsenseQA dataset BIBREF10 . The CommonsenseQA dataset consists of 12,247 questions with one correct answer and four distractor answers. This dataset consists of two splits – the question token split and the random split. Our experiments are conducted on the more challenging random split, which is the main evaluation split according to BIBREF10 . The statistics of the CommonsenseQA dataset are shown in Table 3 .Same as the pre-training stage, the input data for fine-tuning the BERT_CS models is formed by concatenating each question-answer pair as a sequence. The hidden representations over the [CLS] token are run through a softmax layer to create the predictions. The objective function is the same as Equations 10 and 11 . We fine-tune the BERT_CS models on CommonsenseQA for 2 epochs with a learning rate of 1e-5 and a batch size of 16.Table 4 shows the accuracies on the CommonsenseQA test set from the baseline BERT models released by Google, the previous state-of-the-art model CoS-E BIBREF19 , and our BERT_CS models. Note that CoS-E model requires a large amount of human effort to collect the Common Sense Explanations (CoS-E) dataset. In comparison, we construct our multi-choice question-answering dataset automatically. The BERT_CS models significantly outperform the baseline BERT model counterparts. BERT_CS $_{large}$ achieves a 5.5% absolute improvement on the CommonsenseQA test set over the baseline BERT $_{large}$ model and a 4% absolute improvement over the previous SOTA CoS-E model.The Winograd Schema Challenge (WSC) BIBREF13 is introduced for testing AI agents for commonsense knowledge. The WSC consists of 273 instances of the pronoun disambiguation problem (PDP). For example, for sentence “The delivery truck zoomed by the school bus because it was going so fast.” and a corresponding question “What does the word it refers to?”, the machine is expected to answer “delivery truck” instead of “school bus”. In this task, we follow BIBREF22 and employ the WSCR dataset BIBREF23 as the extra training data. The WSCR dataset is split into a training set of 1322 examples and a test set of 564 examples. We use these data for fine-tuning and validating BERT_CS models, respectively, and test the fine-tuned BERT_CS models on the WSC dataset.We transform the pronoun disambiguation problem into a multi-choice question answering problem. We mask the pronoun word with a special token [QW] to construct a question, and put the two candidate paragraphs as candidate answers. The remaining procedures are the same as QA tasks. We use the same loss function as BIBREF22 , that is, if c $_1$ is correct and c $_2$ is not, the loss is $$\begin{aligned}
L = &- {\rm logp}(c_1|s) + \\
&\alpha \cdot max(0, {\rm logp}(c_2|s)-{\rm logp}(c_1|s)+\beta ), \end{aligned}$$   (Eq. 16) where $p(c_1|s)$ follows Equation 11 with $N=2$ , $\alpha $ and $\beta $ are two hyper-parameters. Similar to BIBREF22 , we search $\alpha \in \lbrace 2.5,5,10,20\rbrace $ and $\beta \in \lbrace 0.05,0.1,0.2,0.4\rbrace $ by comparing the accuracy on the WSCR test set (i.e., the development set for the WSC data set). We set the batch size 16 and the learning rate $1e^{-5}$ . We evaluate our models on the WSC dataset, as well as the various partitions of the WSC dataset, as described in BIBREF24 . We also evaluate the fine-tuned BERT_CS model (without using the WNLI training data for further fine-tuning) on the WNLI test set, one of the GLUE tasks. We first transform the examples in WNLI from the premise-hypothesis format into the pronoun disambiguation problem format and then transform it into the multi-choice QA format BIBREF22 .The results on the WSC dataset and its various partitions and the WNLI test set are shown in Table 5 . Note that the results for BIBREF21 are fine-tuned on the whole WSCR dataset, including the training and test sets. Results for LM ensemble BIBREF25 and Knowledge Hunter BIBREF26 are taken from BIBREF24 . Results for “BERT $_{large}$ + MTP" is taken from BIBREF22 as the baseline of applying BERT to the WSC task.As can be seen from Table 5 , the “BERT $_{large}$ + MCQA" achieves better performance than “BERT $_{large}$ + MTP" on four of the seven evaluation criteria and achieves significant improvement on the assoc. and consist. partitions, which demonstrates that MCQA is a better pre-processing method than MTP for the WSC task. Also, the “BERT_CS $_{large}$ + MCQA" achieves the best performance on all of the evaluation criteria but consist., and achieves a 3.3% absolute improvement on the WSC dataset over the previous SOTA results from BIBREF22 .The General Language Understanding Evaluation (GLUE) benchmark BIBREF6 is a collection of diverse natural language understanding tasks, including MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, of which CoLA and SST-2 are single-sentence tasks, MRPC, STS-B and QQP are similarity and paraphrase tasks, and MNLI, QNLI, RTE and WNLI are natural language inference tasks. To investigate whether our multi-choice QA based pre-training approach degenerates the performance on common sentence classification tasks, we evaluate the BERT_CS $_{base}$ and BERT_CS $_{large}$ models on 8 GLUE datasets and compare the performances with those from the baseline BERT models.Following BIBREF4 , we use the batch size 32 and fine-tune for 3 epochs for all GLUE tasks, and select the fine-tuning learning rate (among 1e-5, 2e-5, and 3e-5) based on the performance on the development set. Results are presented in Table 6 . We observe that the BERT_CS $_{large}$ model achieves comparable performance with the BERT $_{large}$ model and the BERT_CS $_{base}$ model achieves slightly better performance than the BERT $_{base}$ model. We hypothesize that the commonsense knowledge may not be required for GLUE tasks. On the other hand, these results demonstrate that our proposed multi-choice QA pre-training task does not degrade the sentence representation capabilities of BERT models.In this subsection, we conduct several comparison experiments using different data and different pre-training tasks on the BERT $_{base}$ model. For simplicity, we discard the subscript $base$ in this subsection.The first set of experiments is to compare the efficacy of our data creation approach versus the data creation approach in BIBREF18 . First, same as BIBREF18 , we collect 606,564 triples from ConceptNet, and construct 1,213,128 questions, each with a correct answer and four distractors. This dataset is denoted the TRIPLES dataset. We pre-train BERT models on the TRIPLES dataset with the same hyper-parameters as the BERT_CS models and the resulting model is denoted BERT_triple. We also create several model counterparts based on our constructed dataset:Distractors are formed by randomly picking concept $_1$ /concept $_2$ in ConceptNet instead of those sharing the same concept $_2$ /concept $_1$ and the relation with the correct answers. We denote the resulting model from this dataset BERT_CS_random.Instead of pre-training BERT with a multi-choice QA task that chooses the correct answer from several candidate answers, we mask concept $_1$ and concept $_2$ and pre-train BERT with a masked language model (MLM) task. We denote the resulting model from this pre-training task BERT_MLM.We randomly mask 15% WordPiece tokens BIBREF27 of the question as in BIBREF4 and then conduct both multi-choice QA task and MLM task simultaneously. The resulting model is denoted BERT_CS_MLM.All these BERT models are fine-tuned on the CommonsenseQA dataset with the same hyper-parameters as described in Section "CommonsenseQA" and the results are shown in Table 7 . We observe the following from Table 7 .Comparing model 1 and model 2, we find that pre-training on ConceptNet benefits the CommonsenseQA task even with the triples as input instead of sentences. Further comparing model 2 and model 6, we find that constructing sentences as input for pre-training BERT performs better on the CommonsenseQA task than using triples for pre-training BERT. We also conduct more detailed comparisons between fine-tuning model 1 and model 2 on GLUE tasks. The results are shown in Table 6 . BERT_triple $_{base}$ yields much worse results than BERT $_{base}$ and BERT_CS $_{base}$ , which demonstrates that pre-training directly on triples may hurt the sentence representation capabilities of BERT.Comparing model 3 and model 6, we find that pre-training BERT benefits from a more difficult dataset. In our selection method, all candidate answers share the same (concept $_1$ , relation) or (relation, concept $_2$ ), that is, these candidates have close meanings. These more confusing candidates force BERT_CS to distinguish synonym meanings, resulting in a more powerful BERT_CS model.Comparing model 5 and model 6, we find that the multi-choice QA task works better than the masked LM task as the pre-training task for the target multi-choice QA task. We argue that, for the masked LM task, BERT_CS is required to predict each masked wordpieces (in concepts) independently and for the multi-choice QA task, BERT is required to model the whole candidate phrases. In this way, BERT is able to model the whole concepts instead of paying much attention to the single wordpieces in the sentences. Comparing model 4 and model 6, we observe that adding the masked LM task may hurt the performance of BERT_CS. This is probably because the masked words in questions may have a negative influence on the multi-choice QA task. Finally, our proposed model BERT_CS achieves the best performance on the CommonsenseQA development set among these model counterparts.In this subsection, we plot the performance curve on CommonsenseQA development set from BERT_CS over the pre-training steps. For every 10,000 training steps, we save the model as the initial model for fine-tuning. For every of these models, we run experiments for 10 times repeatedly with random restarts, that is, we use the same pre-trained checkpoint but perform different fine-tuning data shuffling. Due to the unstability of fine-tuning BERT BIBREF4 , we remove the results that are significantly lower than the mean. In our experiments, we remove the accuracy lower than 0.57 for BERT_CS $_{base}$ and 0.60 for BERT_CS $_{large}$ . We plot the mean and standard deviation values in Figure 1 . We observe that the performance of BERT_CS $_{base}$ converges around 50,000 training steps and BERT_CS $_{large}$ converges around the end of the pre-training stage or may not have converged, which demonstrates that the BERT_CS $_{large}$ is more powerful at incorporating commonsense knowledge. We also compare with pre-training BERT_CS models for 2 epochs. However, our model produces worse performance probably due to over-fitting. Pre-training on a larger corpus (with more QA samples) may benefit the BERT_CS models and we leave this to the future work.Table 8 shows several cases from the Winograd Schema Challenge dataset. Questions 1 and 2 only differ in the words “compassionate" and “cruel". Our model BERT_CS $_{large}$ chooses correct answers for both questions while BERT $_{large}$ chooses the same choice “Bill" for both questions. We speculate that BERT $_{large}$ tends to choosing the closer candidates. We split WSC test set into two parts CLOSE and FAR according as the correct candidate is closer or farther to the pronoun word in the sentence than another candidate. As shown in Table 9 , our model BERT_CS $_{large}$ achieves the same performance on CLOSE set and better performance on FAR set than BERT $_{large}$ . That's to say, BERT_CS $_{large}$ is more robust to the position of the words and focuses more on the semantic of the sentence.Questions 3 and 4 only differ in the words “large" and “small". However, neither BERT_CS $_{large}$ nor BERT $_{large}$ chooses the correct answers. We hypothesize that since “suitcase is large" and “trophy is small" are probably quite frequent for language models, both BERT $_{large}$ and BERT_CS $_{large}$ models make mistakes. In future work, we will investigate other approaches for overcoming the sensitivity of language models and improving commonsense reasoning.In this paper, we develop a pre-training approach for incorporating commonsense knowledge into language representation models such as BERT. We construct a commonsense-related multi-choice question answering dataset for pre-training BERT. The dataset is created automatically by our proposed “align, mask, and select" (AMS) method. Experimental results demonstrate that pre-training models using the proposed approach followed by fine-tuning achieves significant improvements on various commonsense-related tasks, such as CommonsenseQA and Winograd Schema Challenge, while maintaining comparable performance on other NLP tasks, such as sentence classification and natural language inference (NLI) tasks, compared to the original BERT models. In future work, we will incorporate the relationship information between two concepts into language representation models. We will also explore other structured knowledge graphs, such as Freebase, to incorporate entity information into language representation models. We also plan to incorporate commonsense knowledge information into other language representation models such as XLNet BIBREF28 .The authors would like to thank Lingling Jin, Pengfei Fan, Xiaowei Lu for supporting 16 NVIDIA V100 GPU cards.

Q: How do they select answer candidates for their QA task?