SlowGuess/ACL_2020_OCR huggingface dataset:
include them into our fine-tuning study to be sure to compare results for which models learn the actual premise-hypothesis commonsense reasoning task.
Comparison settings
In order to make fair comparisons, we train and compare three different model settings:
- Our scoring method with target premise mode, 1-gram, margin-based loss, full-text
format (ours).
- A randomly initialized classifier with cross-entropy loss and separated-sentence format (head  $CE$ ). The cross-entropy loss is computed on the probability of the correct candidate, normalized over all candidates in the set (see Equation 1 in (Li et al., 2019)).
- A randomly initialized classifier with margin-based loss and full-text format (head margin)
The head margin setting is an ablated version of our scoring method to verify that our reuse of the MLM head actually provides a significant advantage over a randomly initialized head. For our method, we report results only for the best performing scoring method which is the target premise mode. Experiments showed us that varying the number of grams produce comparable re
sults, so we use the 1-gram setting for computational efficiency. We reckon that the enriched bidirectional context granted by N-gram score can be directly learned when fine-tuning on the task.
For each dataset, we train the three model settings for 20 random seeds each. For each seed, we pick the best performing model on the validation set and report its accuracy on the Test set. We then compute the max accuracy, mean accuracy and standard deviation of each model setting on the Test set. For all model settings, following the recommended hyper-parameters to fine-tune RoBERTaLARGE (Liu et al., 2019), we set a learning rate of 1e-5, a warm-up ratio of  $6\%$  of the total number of training steps, a linear learning rate decay and a weight decay of 0.01. We use a batch size of 8 for COPA (4 for the  $10\%$  training percentage setting) and 16 for CommonsenseQA. For the margin-based loss (ours and head margin), we set  $\eta = 0.5$  after a few trials.
COPA and CommonsenseQA results
On both COPA and CommonsenseQA, our method outperforms both the head CE and head margin methods in terms of mean accuracy and max/best accuracy (see Figure 2 and Figure 3). Moreover, we find that a progressive decrease of the training dataset size results in a progressive increase of the best accuracy gap between our method and the other ones. This confirms our intuition that our methods is the most advantageous when few training data is available.
For example, when using  $1\%$  of training data of CommonsenseQA, our method achieves an accuracy of  $56.7\%$  on the Test* set (vs.  $40.2\%$  for the head CE approach). Using the whole training data, our approach still outperforms other methods but by a lower margin ( $76.4\%$  accuracy versus  $75.4\%$  for head CE). In addition, when evaluated on the CommonsenseQA private Test set, our approach gets  $71.6\%$  accuracy which is close to  $\mathrm{RoBERTa}_{LARGE}$  cross-entropy (Liu et al., 2019) under an important hyper-parameter grid search $^4$  ( $72.1\%$  accuracy).
When using  $100\%$  of the COPA training set (400 train samples), our method outperforms the head CE setting per 5 points and the head margin setting per 3 points, achieving an accuracy of  $92.4\%$  on the Test set. This result allows our approach to reach the second place in the Su
perGLUE leaderboard $^5$  (Wang et al., 2019) between RoBERTa $_{LARGE}$  (Liu et al., 2019) and the T5 model composed of 11 billions of parameters (Raffel et al., 2019) (respectively 90.6 and  $94.8\%$  accuracy on the Test set).
We also notice that our method provides a much more stable training relative to the random seed as shown by the box plots in Figure 2 a) and 3 a). When training on the full COPA dataset, our method exhibits a  $\times 10$  standard deviation reduction on the test accuracy compared to the head CE setting (1.35% versus 12.8%). Our intuition is that the improved stability is due to the better reuse of the pre-trained model priors and the absence of new randomly initialized weights. This is important result towards easier experiment comparisons as fine-tuning BERT-like architectures is known to be unstable across random restarts as shown in (Phang et al., 2018).
In this work, we presented a new method for plausibility ranking tasks, specifically targeting commonsense ranking problem. We define a scoring function that leverages the MLM head of large pre-trained bidirectional transformer models. We establish strong results in a zero-shot setting on four commonsense reasoning datasets, comparable to supervised approaches. We then fine-tune such model using a margin-based loss on the proposed scoring function, and provide a comparative study with state of the art randomly initialized head methods. Our study demonstrates that the direct use of MLM over custom head yields increasingly superior performance gain when decreasing training data size. The proposed approach outperforms state-of-the-art training methods in terms of both test accuracy and training stability.
Future works include applying such scoring method on broader classification tasks like Natural Language Inference and Sentiment Analysis. We also think that our token-level scoring method could be used during the self-supervised pretraining phase to extend traditional next sentence prediction and sequence ordering tasks, bringing more commonsense knowledge in the model.
4https://github.com/pytorch/fairseq/tree/master/examples/roberta/commonsense_qa
<sup>5</sup>https://super.fluebenchmark.com/leaderboard
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. ArXiv, abs/1904.09751.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the Winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837-4842, Florence, Italy. Association for Computational Linguistics.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR'12, pages 552-561. AAAI Press.
Zhongyang Li, Tongfei Chen, and Benjamin Van Durme. 2019. Learning to rank for plausible plausibility. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4818-4823, Florence, Italy. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as
knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.03681.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5926-5936, Long Beach, California, USA. PMLR.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. ArXiv, abs/1806.02847.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3261-3275. Curran Associates, Inc.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference.
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.