SlowGuess/ACL_2020_OCR huggingface dataset:
Model	Property (32)	Object (82)	Eventuality (88)	Spatial (64)	Quantity (20)	Others (48)	Overall (273)
BERT (base)	56.25%	64.63%	50.00%	57.81%	50.00%	45.83%	56.04%
BERT (large)	56.25%	62.20%	62.50%	67.19%	45.00%	52.08%	61.90%
RoBERTa (base)	43.75%	51.22%	56.82%	51.56%	55.00%	39.58%	51.65%
RoBERTa (large)	50.00%	51.22%	52.27%	48.44%	65.00%	56.25%	52.75%
GPT-2 (small, full)	56.25%	51.22%	55.68%	51.56%	60.00%	47.92%	52.75%
GPT-2 (small, partial)	43.75%	60.98%	53.41%	51.56%	60.00%	54.17%	53.48%
GPT-2 (large, full)	68.75%	68.29%	61.36%	53.13%	55.00%	45.83%	59.34%
GPT-2 (large, partial)	65.63%	75.61%	72.73%	62.50%	65.00%	60.42%	69.23%
BERT (base) + WSCR	71.88%	64.63%	55.68%	59.38%	65.00%	45.83%	59.71%
BERT (large) + WSCR	81.25%	75.61%	73.86%	67.19%	85.00%	64.58%	71.43%
BERT (base) + Grande	65.63%	58.54%	60.23%	59.38%	55.00%	56.25%	60.34%
BERT (large) + Grande	75.00%	70.73%	77.27%	79.69%	75.00%	68.75%	73.63%
RoBERTa (base) + WSCR	62.50%	60.98%	57.95%	64.06%	55.00%	64.58%	63.00%
RoBERTa (large) + WSCR	84.38%	84.15%	79.55%	76.56%	70.00%	81.25%	80.95%
RoBERTa (base) + Grande	75.00%	67.07%	72.73%	75.00%	80.00%	70.83%	72.16%
RoBERTa (large) + Grande	90.63%	84.15%	93.18%	84.38%	90.00%	89.58%	87.55%
Table 4: Performances of different models on WSC questions. Questions are grouped by their major knowledge types. If one question contains more than one knowledge types, it will be counted in all categories. If one question contains only 'Others' knowledge, it will be grouped into 'Others'. Numbers of questions are shown in brackets.
Model	Single (222)	Multiple (51)
BERT (base)	56.31%	54.90%
BERT (large)	63.06%	56.86%
RoBERTa (base)	53.15%	45.10%
RoBERTa (large)	54.05%	47.06%
GPT-2 (small, full)	51.80%	56.86%
GPT-2 (small, partial)	53.48%	54.90%
GPT-2 (large, full)	58.56%	62.74%
GPT-2 (large, partial)	70.27%	64.71%
BERT (base) + WSCR	59.91%	58.82%
BERT (large) + WSCR	70.27%	76.47%
BERT (base) + Grande	61.26%	56.86%
BERT (large) + Grande	72.52%	78.43%
RoBERTa (base) + WSCR	64.86%	54.90%
RoBERTa (large) + WSCR	81.53%	78.43%
RoBERTa (base) + Grande	72.97%	68.63%
RoBERTa (large) + Grande	86.94%	90.20%
Table 5: Performances of different models on different sets of WSC questions. Questions are grouped by the number of essential knowledge types (i.e., single or multiple). Numbers of questions are shown in brackets.
similar and even better performance on the complicated questions. In general, this observation is consistent with our previous observations that large models are capable of solving complex questions from the 'Others' category with the support of suitable fine-tuning datasets.
5 WinoWhy Experiments
In this section, we conduct experiments to investigate whether current models can understand how human beings solve WSC questions.
5.1 Unsupervised Setting
Experiment Details: To evaluate whether pretrained language representation models, which achieve the state-of-the-art performance on the WSC task, can distinguish the plausible reasons against the wrong ones, following (Kocijan et al., 2019; Radford et al., 2019; Sakaguchi et al., 2019), we first connect the questions and candidate reasons into single sentences, put them into the models, and take the returned probability as the prediction. Higher probability indicates higher plausibility prediction. Best thresholds are selected for different models to calculate the final accuracy. Similar to Section 4, we evaluate BERT (base), BERT (large), GPT-2 (small), GPT-2 (large), RoBERTa (base), and RoBERTa (large) on WinoWhy. For GPT-2 models, as the partial setting has been proved more useful, we only report the performances based on the partial setting. Besides these two, we also consider BERT/RoBERTa + WSCR/Grande combinations as additional unsupervised approaches because they are not directly optimized towards the WinoWhy task.
Result Analysis: Based on the results shown in Table 6, we can observe that even though pre-trained language representation models have achieved significant improvement over the original WSC task, they are still struggling on the WinoWhy task. Moreover, experimental results on different knowledge types prove that such a conclusion is universal rather than for a specific
5742
Model	Property (337)	Object (856)	Eventuality (928)	Spatial (674)	Quantity (206)	Others (496)	Overall (2865)
Majority Voting	54.30%	56.31%	56.47%	52.67%	52.43%	55.24%	55.67%
BERT (base)	56.97%	56.54%	56.25%	54.01%	51.94%	55.44%	55.92%
BERT (large)	56.38%	57.24%	56.14%	53.41%	51.94%	56.65%	56.13%
RoBERTa (base)	54.30%	56.31%	56.90%	52.67%	52.91%	55.44%	55.78%
RoBERTa (large)	54.30%	56.43%	56.47%	52.67%	52.43%	55.04%	55.67%
GPT-2 (small)	56.68%	54.91%	57.11%	54.45%	59.71%	57.66%	56.37%
GPT-2 (large)	57.57%	54.44%	54.42%	55.93%	54.85%	54.84%	55.77%
BERT (base) + WSCR	55.49%	56.31%	56.90%	52.97%	51.94%	55.04%	55.71%
BERT (large) + WSCR	56.97%	56.31%	56.79%	53.12%	52.91%	55.04%	55.99%
BERT (base) + Grande	57.27%	56.43%	57.22%	53.41%	52.91%	55.24%	55.99%
BERT (large) + Grande	54.90%	56.07%	56.57%	52.67%	52.91%	55.44%	55.71%
RoBERTa (base) + WSCR	52.82%	55.61%	58.41%	53.26%	56.31%	55.04%	56.19%
RoBERTa (large) + WSCR	54.90%	58.06%	56.90%	52.08%	52.91%	56.85%	56.23%
RoBERTa (base) + Grande	56.08%	58.88%	58.19%	55.64%	57.28%	57.66%	58.05%
RoBERTa (large) + Grande	56.08%	58.06%	59.59%	56.82%	56.80%	58.06%	58.18%
kind of knowledge. One possible reason is that even though the designers of WSC are trying to avoid any statistical correlation between the answer and the trigger word, such statistical correlation still exists. As a result, pre-trained language representation models can learn such correlation from large-scale training corpus and thus can answer WSC questions without fully understanding the reasons behind. Besides that, another interesting finding is that GPT-2 (large), as the best unsupervised model on WSC, performs poorly on WinoWhy. One possible explanation is that a lot of negative examples are generated with GPT-2 (large), and thus the dataset brings extra challenges for GPT-2 (large). Last but not least, we can find that fine-tuning over similar dataset (i.e., WSCR and WinoGrande) can slightly help RoBERTa, but the effect is still quite limited. This is probably because such a fine-tuning procedure only teaches pre-trained models to better answer WSC questions rather than understand the commonsense knowledge behind.
5.2 Supervised Setting
Besides the unsupervised setting, we are also interested in whether a model can learn to distinguish reasons through supervised learning.
5.2.1 Experiment Details
Here, we randomly divide the annotated dataset into five groups and conduct five-fold cross-validation. We tried two different splitting meth
Table 6: Performances of different models on WinoWhy questions. We report performances of different reason sets based on the required knowledge types. Reasons could belong to multiple categories as the original WSC questions could contain more than one knowledge types. Numbers of questions are shown in brackets.
Setting	Model	Accuracy	std
Five-fold (q)	Glove + LSTM	59.74%	1.04%
BERT (base)	77.48%	2.06%
BERT (large)	77.39%	1.54%
RoBERTa (base)	75.01%	2.48%
RoBERTa (large)	75.04%	1.97%
GPT-2 (small)	74.48%	2.43%
GPT-2 (large)	75.89%	1.35%
Five-fold (r)	Glove + LSTM	64.92%	1.76%
BERT (base)	77.77%	1.54%
BERT (large)	77.50%	2.43%
RoBERTa (base)	74.41%	1.35%
RoBERTa (large)	74.66%	1.75%
GPT-2 (small)	76.19%	3.69%
GPT-2 (large)	76.13%	4.30%
Table 7: Accuracy and the standard deviation (std) results of evaluated supervised models.
ods, one is based on the WSC questions and the other one is based on the reasons. We denote these two settings as Five-fold (q) and Five-fold (r) respectively. As WinoWhy can be viewed as a text classification task, we adopt the traditional encoding+classification framework and leverage a two-layer feed-forward neural network as the classification module. Seven different encoding methods (Bi-LSTM (Hochreiter and Schmidhuber, 1997), BERT (base), BERT (large), GPT-2 (small), GPT-2 (large), RoBERTa (base), and RoBERTa (large)) are evaluated. For LSTM, we choose the number of layers to be two, the hidden embedding dimension to be 300, and Glove (Pennington et al., 2014) to be the word embedding. All models are trained for ten epochs. Average accuracies over folds and standard deviations are reported.
The results in Table 7 demonstrate that in general, WinoWhy is a challenging task as the best supervised model can only achieve  $77.77\%$  accuracy on a two-class classification task. Besides that, we also notice that all models are getting relatively large standard deviations, especially under the 'Five-fold (r)' setting, which may imply that these supervised models are sensitive to the dataset distribution. Both of these observations show that training a supervised model on WinoWhy is not enough to fully understand the reasons behind WSC decisions and we may need to include reasoning over more complex knowledge to solve this challenging problem.
Based on the observations that fine-tuning over WSCR and WinoGrande can only help solve WSC rather than WinoWhy and the machine-learning based models over WinoWhy can be sensitive to the dataset distribution, it is reasonable to suspect that the improvement achieved by fine-tuning over a similar or same dataset might come from better dataset fitting rather than better commonsense reasoning. As the original purpose of proposing both WSC and WinoWhy is to evaluate how good current AI systems can understand commonsense knowledge rather than solve these questions by fitting the dataset, the unsupervised setting might be the more reasonable evaluation setting.
As an important knowledge resource for many artificial intelligence systems, commonsense knowledge covers various knowledge categories like causality (Sap et al., 2019), reasoning (Schubert, 2015), property (Liu and Singh, 2004), and quantity (Elazar et al., 2019), and has been proven crucial in many downstream tasks like question answering (Lin et al., 2019), dialogue system (Zhou et al., 2018), reading comprehension (Wang et al., 2018), and pronoun coreference resolution (Levesque et al., 2012). Among all these tasks, Winograd Schema Challenge (WSC) (Levesque et al., 2012) is viewed as one of the most challenging ones because solving WSC questions typically requires inference over various kinds of commonsense knowledge. Conventionally, people tried to solve WSC questions in an unsupervised way by leveraging either search
engines (Emami et al., 2018), linguistic knowledge (Zhang et al., 2019, 2020), or language representation models (Kocijan et al., 2019). Experimental results showed that these models still cannot fully solve the problem but we are not clear about how to further improve them. One important reason behind this is that the conventional definition of commonsense knowledge is too vague and thus we are not clear about what kinds of knowledge are still challenging for current commonsense reasoning models. In this paper, we use the WSC task as the breaking point to conduct a deep diagnosis of essential commonsense knowledge types, which sheds some light on how to achieve a better commonsense reasoning system in the future.
In this paper, we presented the first deep diagnosis of essential commonsense knowledge for answering Winograd Schema Challenge questions. By doing so, we better understand the strengths and limitations of current commonsense reasoning models. More importantly, we better know about what kinds of commonsense knowledge are required to be acquired for better commonsense reasoning. On top of the collected reasons, we develop a new task called WinoWhy, which requires models to select the plausible reasons for answering WSC questions. Experiments show that even though current models have gained significant improvement over the original WSC task, they still cannot fully understand the reasons behind.
This paper was supported by the Early Career Scheme (ECS, No. 26206717) from the Research Grants Council in Hong Kong and the Tencent AI Lab Rhino-Bird Focused Research Program.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171-4186.
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In Proceedings of ACL 2019, pages 3973-3983.
5744
Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018. A knowledge hunting framework for common sense reasoning. In Proceedings of EMNLP 2018, pages 1949-1958.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Ray Jackendoff, editor. 1990. Semantic Structures. Cambridge, Massachusetts: MIT Press.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the winograd schema challenge. In Proceedings of ACL 2019, pages 4837-4842.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of KR 2012.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of EMNLP-IJCNLP 2019, pages 2829-2839.
Hugo Liu and Push Singh. 2004. Conceptnet: a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
Quan Liu, Hui Jiang, Zhen-Hua Ling, Xiaodan Zhu, Si Wei, and Yu Hu. 2017. Combing context and commonsense knowledge through neural networks for solving winograd schema problems. In Proceedings of AAAI Spring Symposia 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica, 22(3):276-282.
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. Semeval-2018 task 11: Machine comprehension using commonsense knowledge. In Proceedings of SemEval-2018, pages 747-757.
Alexander P. D. Mourelatos. 1978. Events, processes, and states. Linguistics and Philosophy, 2:415-434.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP 2014, pages 1532-1543.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The winograd schema challenge. In Proceedings of EMNLP-ConL L 2012, pages 777-789.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of AAAI 2020.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: an atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI 2019, pages 3027-3035.
Lenhart K Schubert. 2015. What kinds of knowledge are needed for genuine understanding. In Proceedings of Cognition 2015.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of AAAI 2017, pages 4444-4451.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT 2019, pages 4149-4158.
Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. CoRR, abs/1806.02847.
Liang Wang, Meng Sun, Wei Zhao, Kewei Shen, and Jingming Liu. 2018. Yuanfudao at semeval-2018 task 11: Three-way attention and relational knowledge for commonsense machine comprehension. arXiv preprint arXiv:1803.00191.
Hongming Zhang, Hantian Ding, and Yangqiu Song. 2019. SP-10K: A large-scale evaluation set for selectional preference acquisition. In Proceedings of ACL 2019, pages 722-731.
Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. 2020. Aser: A largescale eventuality knowledge graph. In Proceedings of WWW 2020, pages 201-211.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of IJCAI 2018, pages 4623-4629.
5745