CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods
Authors:   The Critical Assessment of Genome Interpretation Consortium, Jain Shantanu , Bakolitsa Constantina , Brenner Steven E. , Radivojac Predrag , Moult John , Repo Susanna , Hoskins Roger A. , Andreoletti Gaia , Barsky Daniel , Chellapan Ajithavalli , Chu Hoyin , Dabbiru Navya , Kollipara Naveen K. , Ly Melissa , Neumann Andrew J. , Pal Lipika R. , Odell Eric , Pandey Gaurav , Peters-Petrulewicz Robin C. , Srinivasan Rajgopal , Yee Stephen F. , Yeleswarapu Sri Jyothsna , Zuhl Maya , Adebali Ogun , Patra Ayoti , Beer Michael A. , Hosur Raghavendra , Peng Jian , Bernard Brady M. , Berry Michael , Dong Shengcheng , Boyle Alan P. , Adhikari Aashish , Chen Jingqi , Hu Zhiqiang , Wang Robert , Wang Yaqiong , Miller Maximilian , Wang Yanran , Bromberg Yana , Turina Paola , Capriotti Emidio , Han James J. , Ozturk Kivilcim , Carter Hannah , Babbi Giulia , Bovo Samuele , Di Lena Pietro , Martelli Pier Luigi , Savojardo Castrense , Casadio Rita , Cline Melissa S. , De Baets Greet , Bonache Sandra , Díez Orland , Gutiérrez-Enríquez Sara , Fernández Alejandro , Montalban Gemma , Ootes Lars , Özkan Selen , Padilla Natàlia , Riera Casandra , De la Cruz Xavier , Diekhans Mark , Huwe Peter J. , Wei Qiong , Xu Qifang , Dunbrack Roland L. , Gotea Valer , Elnitski Laura , Margolin Gennady , Fariselli Piero , Kulakovskiy Ivan V. , Makeev Vsevolod J. , Penzar Dmitry D. , Vorontsov Ilya E. , Favorov Alexander V. , Forman Julia R. , Hasenahuer Marcia , Fornasari Maria S. , Parisi Gustavo , Avsec Ziga , Çelik Muhammed H. , Nguyen Thi Yen Duong , Gagneur Julien , Shi Fang-Yuan , Edwards Matthew D. , Guo Yuchun , Tian Kevin , Zeng Haoyang , Gifford David K. , Göke Jonathan , Zaucha Jan , Gough Julian , Ritchie Graham R. S. , Frankish Adam , Mudge Jonathan M. , Harrow Jennifer , Young Erin L. , Yu Yao , Huff Chad D. , Murakami Katsuhiko , Nagai Yoko , Imanishi Tadashi , Mungall Christopher J. , Jacobsen Julius O. B. , Kim Dongsup , Jeong Chan-Seok , Jones David T. , Li Mulin Jun , Guthrie Violeta Beleva , Bhattacharya Rohit , Chen Yun-Ching , Douville Christopher , Fan Jean , Kim Dewey , Masica David , Niknafs Noushin , Sengupta Sohini , Tokheim Collin , Turner Tychele N. , Yeo Hui Ting Grace , Karchin Rachel , Shin Sunyoung , Welch Rene , Keles Sunduz , Li Yue , Kellis Manolis , Corbi-Verge Carles , Strokach Alexey V. , Kim Philip M. , Klein Teri E. , Mohan Rahul , Sinnott-Armstrong Nicholas A. , Wainberg Michael , Kundaje Anshul , Gonzaludo Nina , Mak Angel C. Y. , Chhibber Aparna , Lam Hugo Y. K. , Dahary Dvir , Fishilevich Simon , Lancet Doron , Lee Insuk , Bachman Benjamin , Katsonis Panagiotis , Lua Rhonald C. , Wilson Stephen J. , Lichtarge Olivier , Bhat Rajendra R. , Sundaram Laksshman , Viswanath Vivek , Bellazzi Riccardo , Nicora Giovanna , Rizzo Ettore , Limongelli Ivan , Mezlini Aziz M. , Chang Ray , Kim Serra , Lai Carmen , O’Connor Robert , Topper Scott , van den Akker Jeroen , Zhou Alicia Y. , Zimmer Anjali D. , Mishne Gilad , Bergquist Timothy R. , Breese Marcus R. , Guerrero Rafael F. , Jiang Yuxiang , Kiga Nikki , Li Biao , Mort Matthew , Pagel Kymberleigh A. , Pejaver Vikas , Stamboulian Moses H. , Thusberg Janita , Mooney Sean D. , Teerakulkittipong Nuttinee , Cao Chen , Kundu Kunal , Yin Yizhou , Yu Chen-Hsin , Kleyman Michael , Lin Chiao-Feng , Stackpole Mary , Mount Stephen M. , Eraslan Gökcen , Mueller Nikola S. , Naito Tatsuhiko , Rao Aliz R. , Azaria Johnathan R. , Brodie Aharon , Ofran Yanay , Garg Aditi , Pal Debnath , Hawkins-Hooker Alex , Kenlay Henry , Reid John , Mucaki Eliseos J. , Rogan Peter K. , Schwarz Jana M. , Searls David B. , Lee Gyu Rie , Seok Chaok , Krämer Andreas , Shah Sohela , Huang ChengLai V. , Kirsch Jack F. , Shatsky Maxim , Cao Yue , Chen Haoran , Karimi Mostafa , Moronfoye Oluwaseyi , Sun Yuanfei , Shen Yang , Shigeta Ron , Ford Colby T. , Nodzak Conor , Uppal Aneeta , Shi Xinghua , Joseph Thomas , Kotte Sujatha , Rana Sadhna , Rao Aditya , Saipradeep V. G. , Sivadasan Naveen , Sunderam Uma , Stanke Mario , Su Andrew , Adzhubey Ivan , Jordan Daniel M. , Sunyaev Shamil , Rousseau Frederic , Schymkowitz Joost , Van Durme Joost , Tavtigian Sean V. , Carraro Marco , Giollo Manuel , Tosatto Silvio C. E. , Adato Orit , Carmel Liran , Cohen Noa E. , Fenesh Tzila , Holtzer Tamar , Juven-Gershon Tamar , Unger Ron , Niroula Abhishek , Olatubosun Ayodeji , Väliaho Jouni , Yang Yang , Vihinen Mauno , Wahl Mary E. , Chang Billy , Chong Ka Chun , Hu Inchi , Sun Rui , Wu William Ka Kei , Xia Xiaoxuan , Zee Benny C. , Wang Maggie H. , Wang Meng , Wu Chunlei , Lu Yutong , Chen Ken , Yang Yuedong , Yates Christopher M. , Kreimer Anat , Yan Zhongxia , Yosef Nir , Zhao Huying , Wei Zhipeng , Yao Zhaomin , Zhou Fengfeng , Folkman Lukas , Zhou Yaoqi , Daneshjou Roxana , Altman Russ B. , Inoue Fumitaka , Ahituv Nadav , Arkin Adam P. , Lovisa Federica , Bonvini Paolo , Bowdin Sarah , Gianni Stefano , Mantuano Elide , Minicozzi Velia , Novak Leonore , Pasquo Alessandra , Pastore Annalisa , Petrosino Maria , Puglisi Rita , Toto Angelo , Veneziano Liana , Chiaraluce Roberta , Ball Mad P. , Bobe Jason R. , Church George M. , Consalvi Valerio , Cooper David N. , Buckley Bethany A. , Sheridan Molly B. , Cutting Garry R. , Scaini Maria Chiara , Cygan Kamil J. , Fredericks Alger M. , Glidden David T. , Neil Christopher , Rhine Christy L. , Fairbrother William G. , Alontaga Aileen Y. , Fenton Aron W. , Matreyek Kenneth A. , Starita Lea M. , Fowler Douglas M. , Löscher Britt-Sabina , Franke Andre , Adamson Scott I. , Graveley Brenton R. , Gray Joe W. , Malloy Mary J. , Kane John P. , Kousi Maria , Katsanis Nicholas , Schubach Max , Kircher Martin , Mak Angel C. Y. , Tang Paul L. F. , Kwok Pui-Yan , Lathrop Richard H. , Clark Wyatt T. , Yu Guoying K. , LeBowitz Jonathan H. , Benedicenti Francesco , Bettella Elisa , Bigoni Stefania , Cesca Federica , Mammi Isabella , Marino-Buslje Cristina , Milani Donatella , Peron Angela , Polli Roberta , Sartori Stefano , Stanzial Franco , Toldo Irene , Turolla Licia , Aspromonte Maria C. , Bellini Mariagrazia , Leonardi Emanuela , Liu Xiaoming , Marshall Christian , McCombie W. Richard , Elefanti Lisa , Menin Chiara , Meyn M. Stephen , Murgia Alessandra , Nadeau Kari C. Y. , Neuhausen Susan L. , Nussbaum Robert L. , Pirooznia Mehdi , Potash James B. , Dimster-Denk Dago F. , Rine Jasper D. , Sanford Jeremy R. , Snyder Michael , Cote Atina G. , Sun Song , Verby Marta W. , Weile Jochen , Roth Frederick P. , Tewhey Ryan , Sabeti Pardis C. , Campagna Joan , Refaat Marwan M. , Wojciak Julianne , Grubb Soren , Schmitt Nicole , Shendure Jay , Spurdle Amanda B. , Stavropoulos Dimitri J. , Walton Nephi A. , Zandi Peter P. , Ziv Elad , Burke Wylie , Chen Flavia , Carr Lawrence R. , Martinez Selena , Paik Jodi , Harris-Wai Julie , Yarborough Mark , Fullerton Stephanie M. , Koenig Barbara A. , McInnes Gregory , Shigaki Dustin , Chandonia John-Marc , Furutsuki Mabel , Kasak Laura , Yu Changhua , Chen Rui , Friedberg Iddo , Getz Gad A. , Cong Qian , Kinch Lisa N. , Zhang Jing , Grishin Nick V. , Voskanian Alin , Kann Maricel G. , Tran Elizabeth , Ioannidis Nilah M. , Hunter Jesse M. , Udani Rupa , Cai Binghuang , Morgan Alexander A. , Sokolov Artem , Stuart Joshua M. , Minervini Giovanni , Monzon Alexander M. , Batzoglou Serafim , Butte Atul J. , Greenblatt Marc S. , Hart Reece K. , Hernandez Ryan , Hubbard Tim J. P. , Kahn Scott , O’Donnell-Luria Anne , Ng Pauline C. , Shon John , Veltman Joris , Zook Justin M. 
Published: February 2024

Abstract
Background
The Critical Assessment of Genome Interpretation (CAGI) aims to advance the state-of-the-art for computational prediction of genetic variant impact, particularly where relevant to disease. The five complete editions of the CAGI community experiment comprised 50 challenges, in which participants made blind predictions of phenotypes from genetic data, and these were evaluated by independent assessors.

Results
Performance was particularly strong for clinical pathogenic variants, including some difficult-to-diagnose cases, and extends to interpretation of cancer-related variants. Missense variant interpretation methods were able to estimate biochemical effects with increasing accuracy. Assessment of methods for regulatory variants and complex trait disease risk was less definitive and indicates performance potentially suitable for auxiliary use in the clinic.

Conclusions
Results show that while current methods are imperfect, they have major utility for research and clinical applications. Emerging methods and increasingly large, robust datasets for training and assessment promise further progress ahead.