SlowGuess/NAACL_2022_OCR huggingface dataset:
Edward Loper and Steven Bird. 2002. NLTK: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63-70, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand? Transactions of the Association for Computational Linguistics, 9:1047-1060.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Kanishka Misra, Allyson Ettinger, and Julia Rayz. 2021. Do language models learn typicality judgments from text? In Proceedings of the 43rd Annual Conference of the Cognitive Science Society.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106.
Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 922–938, Online. Association for Computational Linguistics.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 4444-4451. AAAI Press.
Oyvind Tafjord and Peter Clark. 2021. General-purpose question-answering with macaw.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. ProofWriter: Generating implications, proofs, and
abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621-3634, Online. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020a. oLMpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758.
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020b. Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge. In Advances in Neural Information Processing Systems, volume 33, pages 20227-20237. Curran Associates, Inc.
Aaron Traylor, Roman Feiman, and Ellie Pavlick. 2021. AND does not mean OR: Using formal languages to study language models' representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 158-167, Online. Association for Computational Linguistics.
Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932-948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. Symbolic knowledge distillation: from general language models to commonsense models.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth. 2020. Transomcs: From linguistic graphs to commonsense knowledge. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4004-4010. International Joint Conferences on Artificial Intelligence Organization. Main track.
Hongming Zhang, Xin Liu, Haojie Pan, Haowen Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song. 2021. Aser: Towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities. arXiv preprint arXiv:2104.02137.
Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. 2021. RICA: Evaluating robust inference capabilities based on commonsense axioms. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7560-7579, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pretrained language models. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9733-9740.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV).
A BERT Hyperparameters
We train the BERT-base architecture (12 layers, 12 attention heads, hidden size of 768) following the original pre-training hyperparameters: a batch size of 256, sequence length of 128, and train for 1 million steps. We use the Adam optimizer and linearly warmup the learning rate to 1e-4 over the first 10,000 steps of pre-training, and then linearly decay the learning rate.
Our code builds on the Huggingface Transformers (Wolf et al., 2020) and MegatronLM (Shoeybi et al., 2019) implementations of BERT. The pretraining corpus is uncased and pre-processed using the MegatronLM pre-processing. Training takes four days on eight V100 GPUs.
Our conclusions are based on the training dynamics of BERT-base, and future work might investigate if scaling model size allows for more systematic inferences.
B Leap-of-Thought Fine-tuning Results
The original Leap-of-Thought evaluation consists of fine-tuning BERT to classify if a sub-statement is true given supporting premises. In the explicit reasoning evaluation, all supporting premises are given at test time (e.g., the model must determine if "robins can fly" is true given the context "robins are birds and birds can fly"). In the implicit reasoning evaluation, the class relation is withheld (e.g., the model must determine if "robins can fly" given only the context that "birds can fly." This inference relies on the implicit knowledge that robins are birds). We fine-tune for four epochs following Talmor et al. and otherwise use default hyperparameters.
Our main purpose in running this evaluation is to validate our pre-training procedure; however, we also evaluate all intermediate BERT checkpoints in order to understand how the performance changes across pre-training. Interestingly, we find performance increases log-linearly with pre-training iterations in the implicit reasoning test, but performance of the explicit reasoning evaluation peaks at just  $15\%$  of pre-training (Figure 4). Numerical results are presented in Table 2.
Figure 4: Accuracy on Talmor et al. (2020b)'s original Leap-of-Thought evaluation across pre-training iterations (from 50K to 1M).
Iteration	Implicit	Explicit
0	0.507	0.493
5000	0.507	0.493
10000	0.490	0.490
15000	0.571	0.621
20000	0.625	0.636
30000	0.710	0.763
40000	0.798	0.900
50000	0.814	0.965
100000	0.838	0.971
150000	0.860	0.992
200000	0.843	0.953
250000	0.855	0.973
300000	0.870	0.958
350000	0.863	0.978
400000	0.850	0.931
450000	0.867	0.937
500000	0.859	0.933
550000	0.874	0.951
600000	0.867	0.943
650000	0.880	0.931
700000	0.877	0.937
750000	0.874	0.929
800000	0.872	0.949
850000	0.877	0.979
900000	0.875	0.967
950000	0.894	0.945
Table 2: Fine-tuning accuracy on the original Leap-of-Thought evaluation across pre-training checkpoints.