Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Creators: name: Srivastava, Aarohi
name: Rastogi, Abhinav
name: Rao, Abhishek
name: Shoeb, Abu Awal Md
name: Abid, Abubakar
name: Fisch, Adam
name: Brown, Adam R.
name: Santoro, Adam
name: Gupta, Aditya
name: Garriga-Alonso, Adrià
name: Kluska, Agnieszka
name: Lewkowycz, Aitor
name: Agarwal, Akshat
name: Power, Alethea
name: Ray, Alex
name: Warstadt, Alex
name: Kocurek, Alexander W.
name: Safaya, Ali
name: Tazarv, Ali
name: Xiang, Alice
name: Parrish, Alicia
name: Nie, Allen
name: Hussain, Aman
name: Askell, Amanda
name: Dsouza, Amanda
name: Slone, Ambrose
name: Rahane, Ameet
name: Iyer, Anantharaman S.
name: Andreassen, Anders
name: Madotto, Andrea
name: Santilli, Andrea
name: Stuhlmüller, Andreas
name: Dai, Andrew
name: La, Andrew
name: Lampinen, Andrew
name: Zou, Andy
name: Jiang, Angela
name: Chen, Angelica
name: Vuong, Anh
name: Gupta, Animesh
name: Gottardi, Anna
name: Norelli, Antonio
name: Venkatesh, Anu
name: Gholamidavoodi, Arash
name: Tabassum, Arfa
name: Menezes, Arul
name: Kirubarajan, Arun
name: Mullokandov, Asher
name: Sabharwal, Ashish
name: Herrick, Austin
name: Efrat, Avia
name: Erdem, Aykut
name: Karakaş, Ayla
name: Roberts, B. Ryan
name: Loe, Bao Sheng
name: Zoph, Barret
name: Bojanowski, Bartłomiej
name: Özyurt, Batuhan
name: Hedayatnia, Behnam
name: Neyshabur, Behnam
name: Inden, Benjamin
name: Stein, Benno
name: Ekmekci, Berk
name: Lin, Bill Yuchen
name: Howald, Blake
name: Orinion, Bryan
name: Diao, Cameron
name: Dour, Cameron
name: Stinson, Catherine
name: Argueta, Cedrick
name: Ramírez, César Ferri
name: Singh, Chandan
name: Rathkopf, Charles
name: Meng, Chenlin
name: Baral, Chitta
name: Wu, Chiyu
name: Callison-Burch, Chris
name: Waites, Chris
name: Voigt, Christian
name: Manning, Christopher D.
name: Potts, Christopher
name: Ramirez, Cindy
name: Rivera, Clara E.
name: Siro, Clemencia
name: Raffel, Colin
name: Ashcraft, Courtney
name: Garbacea, Cristina
name: Sileo, Damien
name: Garrette, Dan
name: Hendrycks, Dan
name: Kilman, Dan
name: Roth, Dan
name: Freeman, Daniel
name: Khashabi, Daniel
name: Levy, Daniel
name: González, Daniel Moseguí
name: Perszyk, Danielle
name: Hernandez, Danny
name: Chen, Danqi
name: Ippolito, Daphne
name: Gilboa, Dar
name: Dohan, David
name: Drakard, David
name: Jurgens, David
name: Datta, Debajyoti
name: Ganguli, Deep
name: Emelin, Denis
name: Kleyko, Denis
name: Yuret, Deniz
name: Chen, Derek
name: Tam, Derek
name: Hupkes, Dieuwke
name: Misra, Diganta
name: Buzan, Dilyar
name: Mollo, Dimitri Coelho
name: Yang, Diyi
name: Lee, Dong-Ho
name: Schrader, Dylan
name: Shutova, Ekaterina
name: Cubuk, Ekin Dogus
name: Segal, Elad
name: Hagerman, Eleanor
name: Barnes, Elizabeth
name: Donoway, Elizabeth
name: Pavlick, Ellie
name: Rodola, Emanuele
name: Lam, Emma
name: Chu, Eric
name: Tang, Eric
name: Erdem, Erkut
name: Chang, Ernie
name: Chi, Ethan A.
name: Dyer, Ethan
name: Jerzak, Ethan
name: Kim, Ethan
name: Manyasi, Eunice Engefu
name: Zheltonozhskii, Evgenii
name: Xia, Fanyue
name: Siar, Fatemeh
name: Martínez-Plumed, Fernando
name: Happé, Francesca
name: Chollet, Francois
name: Rong, Frieda
name: Mishra, Gaurav
name: Winata, Genta Indra
name: de Melo, Gerard
name: Kruszewski, Germán
name: Parascandolo, Giambattista
name: Mariani, Giorgio
name: Wang, Gloria
name: Jaimovitch-López, Gonzalo
name: Betz, Gregor
name: Gur-Ari, Guy
name: Galijasevic, Hana
name: Kim, Hannah
name: Rashkin, Hannah
name: Hajishirzi, Hannaneh
name: Mehta, Harsh
name: Bogar, Hayden
name: Shevlin, Henry
name: Schütze, Hinrich
name: Yakura, Hiromu
name: Zhang, Hongming
name: Wong, Hugh Mee
name: Ng, Ian
name: Noble, Isaac
name: Jumelet, Jaap
name: Geissinger, Jack
name: Kernion, Jackson
name: Hilton, Jacob
name: Lee, Jaehoon
name: Fisac, Jaime Fernández
name: Simon, James B.
name: Koppel, James
name: Zheng, James
name: Zou, James
name: Kocoń, Jan
name: Thompson, Jana
name: Wingfield, Janelle
name: Kaplan, Jared
name: Radom, Jarema
name: Sohl-Dickstein, Jascha
name: Phang, Jason
name: Wei, Jason
name: Yosinski, Jason
name: Novikova, Jekaterina
name: Bosscher, Jelle
name: Marsh, Jennifer
name: Kim, Jeremy
name: Taal, Jeroen
name: Engel, Jesse
name: Alabi, Jesujoba
name: Xu, Jiacheng
name: Song, Jiaming
name: Tang, Jillian
name: Waweru, Joan
name: Burden, John
name: Miller, John
name: Balis, John U.
name: Batchelder, Jonathan
name: Berant, Jonathan
name: Frohberg, Jörg
name: Rozen, Jos
name: Hernandez-Orallo, Jose
name: Boudeman, Joseph
name: Guerr, Joseph
name: Jones, Joseph
name: Tenenbaum, Joshua B.
name: Rule, Joshua S.
name: Chua, Joyce
name: Kanclerz, Kamil
name: Livescu, Karen
name: Krauth, Karl
name: Gopalakrishnan, Karthik
name: Ignatyeva, Katerina
name: Markert, Katja
name: Dhole, Kaustubh D.
name: Gimpel, Kevin
name: Omondi, Kevin
name: Mathewson, Kory
name: Chiafullo, Kristen
name: Shkaruta, Ksenia
name: Shridhar, Kumar
name: McDonell, Kyle
name: Richardson, Kyle
name: Reynolds, Laria
name: Gao, Leo
name: Zhang, Li
name: Dugan, Liam
name: Qin, Lianhui
name: Contreras-Ochando, Lidia
name: Morency, Louis-Philippe
name: Moschella, Luca
name: Lam, Lucas
name: Noble, Lucy
name: Schmidt, Ludwig
name: He, Luheng
name: Colón, Luis Oliveros
name: Metz, Luke
name: Şenel, Lütfi Kerem
name: Bosma, Maarten
name: Sap, Maarten
name: ter Hoeve, Maartje
name: Farooqi, Maheen
name: Faruqui, Manaal
name: Mazeika, Mantas
name: Baturan, Marco
name: Marelli, Marco
name: Maru, Marco
name: Quintana, Maria Jose Ramírez
name: Tolkiehn, Marie
name: Giulianelli, Mario
name: Lewis, Martha
name: Potthast, Martin
name: Leavitt, Matthew L.
name: Hagen, Matthias
name: Schubert, Mátyás
name: Baitemirova, Medina Orduna
name: Arnaud, Melody
name: McElrath, Melvin
name: Yee, Michael A.
name: Cohen, Michael
name: Gu, Michael
name: Ivanitskiy, Michael
name: Starritt, Michael
name: Strube, Michael
name: Swędrowski, Michał
name: Bevilacqua, Michele
name: Yasunaga, Michihiro
name: Kale, Mihir
name: Cain, Mike
name: Xu, Mimee
name: Suzgun, Mirac
name: Walker, Mitch
name: Tiwari, Mo
name: Bansal, Mohit
name: Aminnaseri, Moin
name: Geva, Mor
name: Gheini, Mozhdeh
name: T, Mukund Varma
name: Peng, Nanyun
name: Chi, Nathan A.
name: Lee, Nayeon
name: Krakover, Neta Gur-Ari
name: Cameron, Nicholas
name: Roberts, Nicholas
name: Doiron, Nick
name: Martinez, Nicole
name: Nangia, Nikita
name: Deckers, Niklas
name: Muennighoff, Niklas
name: Keskar, Nitish Shirish
name: Iyer, Niveditha S.
name: Constant, Noah
name: Fiedel, Noah
name: Wen, Nuan
name: Zhang, Oliver
name: Agha, Omar
name: Elbaghdadi, Omar
name: Levy, Omer
name: Evans, Owain
name: Casares, Pablo Antonio Moreno
name: Doshi, Parth
name: Fung, Pascale
name: Liang, Paul Pu
name: Vicol, Paul
name: Alipoormolabashi, Pegah
name: Liao, Peiyuan
name: Liang, Percy
name: Chang, Peter
name: Eckersley, Peter
name: Htut, Phu Mon
name: Hwang, Pinyu
name: Miłkowski, Piotr
name: Patil, Piyush
name: Pezeshkpour, Pouya
name: Oli, Priti
name: Mei, Qiaozhu
name: Lyu, Qing
name: Chen, Qinlang
name: Banjade, Rabin
name: Rudolph, Rachel Etta
name: Gabriel, Raefer
name: Habacker, Rahel
name: Risco, Ramon
name: Millière, Raphaël
name: Garg, Rhythm
name: Barnes, Richard
name: Saurous, Rif A.
name: Arakawa, Riku
name: Raymaekers, Robbe
name: Frank, Robert
name: Sikand, Rohan
name: Novak, Roman
name: Sitelew, Roman
name: LeBras, Ronan
name: Liu, Rosanne
name: Jacobs, Rowan
name: Zhang, Rui
name: Salakhutdinov, Ruslan
name: Chi, Ryan
name: Lee, Ryan
name: Stovall, Ryan
name: Teehan, Ryan
name: Yang, Rylan
name: Singh, Sahib
name: Mohammad, Saif M.
name: Anand, Sajant
name: Dillavou, Sam
name: Shleifer, Sam
name: Wiseman, Sam
name: Gruetter, Samuel
name: Bowman, Samuel R.
name: Schoenholz, Samuel S.
name: Han, Sanghyun
name: Kwatra, Sanjeev
name: Rous, Sarah A.
name: Ghazarian, Sarik
name: Ghosh, Sayan
name: Casey, Sean
name: Bischoff, Sebastian
name: Gehrmann, Sebastian
name: Schuster, Sebastian
name: Sadeghi, Sepideh
name: Hamdan, Shadi
name: Zhou, Sharon
name: Srivastava, Shashank
name: Shi, Sherry
name: Singh, Shikhar
name: Asaadi, Shima
name: Gu, Shixiang Shane
name: Pachchigar, Shubh
name: Toshniwal, Shubham
name: Upadhyay, Shyam
name: Shyamolima
name: Debnath
name: Shakeri, Siamak
name: Thormeyer, Simon
name: Melzi, Simone
name: Reddy, Siva
name: Makini, Sneha Priscilla
name: Lee, Soo-Hwan
name: Torene, Spencer
name: Hatwar, Sriharsha
name: Dehaene, Stanislas
name: Divic, Stefan
name: Ermon, Stefano
name: Biderman, Stella
name: Lin, Stephanie
name: Prasad, Stephen
name: Piantadosi, Steven T.
name: Shieber, Stuart M.
name: Misherghi, Summer
name: Kiritchenko, Svetlana
name: Mishra, Swaroop
name: Linzen, Tal
name: Schuster, Tal
name: Li, Tao
name: Yu, Tao
name: Ali, Tariq
name: Hashimoto, Tatsu
name: Wu, Te-Lin
name: Desbordes, Théo
name: Rothschild, Theodore
name: Phan, Thomas
name: Wang, Tianle
name: Nkinyili, Tiberius
name: Schick, Timo
name: Kornev, Timofei
name: Tunduny, Titus
name: Gerstenberg, Tobias
name: Chang, Trenton
name: Neeraj, Trishala
name: Khot, Tushar
name: Shultz, Tyler
name: Shaham, Uri
name: Misra, Vedant
name: Demberg, Vera
name: Nyamai, Victoria
name: Raunak, Vikas
name: Ramasesh, Vinay
name: Prabhu, Vinay Uday
name: Padmakumar, Vishakh
name: Srikumar, Vivek
name: Fedus, William
name: Saunders, William
name: Zhang, William
name: Vossen, Wout
name: Ren, Xiang
name: Tong, Xiaoyu
name: Zhao, Xinran
name: Wu, Xinyi
name: Shen, Xudong
name: Yaghoobzadeh, Yadollah
name: Lakretz, Yair
name: Song, Yangqiu
name: Bahri, Yasaman
name: Choi, Yejin
name: Yang, Yichi
name: Hao, Yiding
name: Chen, Yifu
name: Belinkov, Yonatan
name: Hou, Yu
name: Hou, Yufang
name: Bai, Yuntao
name: Seid, Zachary
name: Zhao, Zhuoye
name: Wang, Zijian
name: Wang, Zijie J.
name: Wang, Zirui
name: Wu, Ziyi
Publisher: name: arXiv
Year: 2022
Subjects: subject: Computation and Language (cs.CL)
subject: Artificial Intelligence (cs.AI)
subject: Computers and Society (cs.CY)
subject: Machine Learning (cs.LG)
subject: Machine Learning (stat.ML)
subject: FOS: Computer and information sciences
subject: FOS: Computer and information sciences

description: Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
description: 27 pages, 17 figures + references and appendices, repo: https://github.com/google/BIG-bench