[1] Z. Ankner, N. Saphra, D. Blalock, J. Frankle, and M. L. Leavitt. Dynamic masking rate schedules for mlm pretraining. arXiv preprint arXiv:2305.15096, 2023.
[2] F. Bai, A. Ritter, and W. Xu. Pre-train or annotate? domain adaptation with a constrained budget. arXiv preprint arXiv:2109.04711, 2021.
[3] I. Beltagy, K. Lo, and A. Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676, 2019.
[4] L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC. CiteSeer, 2009.
[5] C. Blakeney, J. Z. Forde, J. Frankle, Z. Zong, and M. L. Leavitt. Reduce, reuse, recycle: Improving training efficiency with distillation. arXiv preprint arXiv:2211.00683, 2022.
[6] E. Bolton, D. Hall, M. Yasunaga, T. Lee, C. Manning, and P. Liang. Biomedlm. https://crfm.stanford.edu/2022/12/15/biomedlm.html, December 2022. Accessed on 15th May 2023.
[7] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017.
[8] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[9] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020.
[10] I. Dagan, O. Glickman, and B. Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, pages 177-190. Springer, 2006.
[11] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16344-16359, 2022.
[12] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933–941. PMLR, 2017.
[13] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[15] N. Dey, D. Soboleva, F. Al-Khateeb, B. Yang, R. Pathria, H. Khachane, S. Muhammad, R. Myers, J. R. Steeves, N. Vassilieva, et al. BtIm-3b-8k: 7b parameter performance in a 3b parameter model. arXiv preprint arXiv:2309.11568, 2023.
[16] B. Dolan and C. Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
[17] H. El Boukkouri, O. Ferret, T. Lavernne, and P. Zweigenbaum. Re-train or train from scratch? comparing pre-training strategies of bert in the medical domain. In LREC 2022, pages 2626-2633, 2022.
[18] J. Geiping and T. Goldstein. Cramming: Training a language model on a singlegpu in one day. In International Conference on Machine Learning, pages 11117-11143. PMLR, 2023.