Critical assessment of variant prioritization methods for rare disease diagnosis within the rare genomes project
Authors: Stenton Sarah L. , O’Leary Melanie C. , Lemire Gabrielle , VanNoy Grace E. , DiTroia Stephanie , Ganesh Vijay S. , Groopman Emily , O’Heir Emily , Mangilog Brian , Osei-Owusu Ikeoluwa , Pais Lynn S. , Serrano Jillian , Singer-Berk Moriel , Weisburd Ben , Wilson Michael W. , Austin-Tse Christina , Abdelhakim Marwa , Althagafi Azza , Babbi Giulia , Bellazzi Riccardo , Bovo Samuele , Carta Maria Giulia , Casadio Rita , Coenen Pieter-Jan , De Paoli Federica , Floris Matteo , Gajapathy Manavalan , Hoehndorf Robert , Jacobsen Julius O. B. , Joseph Thomas , Kamandula Akash , Katsonis Panagiotis , Kint Cyrielle , Lichtarge Olivier , Limongelli Ivan , Lu Yulan , Magni Paolo , Mamidi Tarun Karthik Kumar , Martelli Pier Luigi , Mulargia Marta , Nicora Giovanna , Nykamp Keith , Pejaver Vikas , Peng Yisu , Pham Thi Hong Cam , Podda Maurizio S. , Rao Aditya , Rizzo Ettore , Saipradeep Vangala G. , Savojardo Castrense , Schols Peter , Shen Yang , Sivadasan Naveen , Smedley Damian , Soru Dorian , Srinivasan Rajgopal , Sun Yuanfei , Sunderam Uma , Tan Wuwei , Tiwari Naina , Wang Xiao , Wang Yaqiong , Williams Amanda , Worthey Elizabeth A. , Yin Rujie , You Yuning , Zeiberg Daniel , Zucca Susanna , Bakolitsa Constantina , Brenner Steven E. , Fullerton Stephanie M. , Radivojac Predrag , Rehm Heidi L. , O’Donnell-Luria Anne 
Published: April 2024

Abstract
Background
A major obstacle faced by families with rare diseases is obtaining a genetic diagnosis. The average "diagnostic odyssey" lasts over five years and causal variants are identified in under 50%, even when capturing variants genome-wide. To aid in the interpretation and prioritization of the vast number of variants detected, computational methods are proliferating. Knowing which tools are most effective remains unclear. To evaluate the performance of computational methods, and to encourage innovation in method development, we designed a Critical Assessment of Genome Interpretation (CAGI) community challenge to place variant prioritization models head-to-head in a real-life clinical diagnostic setting.

Methods
We utilized genome sequencing (GS) data from families sequenced in the Rare Genomes Project (RGP), a direct-to-participant research study on the utility of GS for rare disease diagnosis and gene discovery. Challenge predictors were provided with a dataset of variant calls and phenotype terms from 175 RGP individuals (65 families), including 35 solved training set families with causal variants specified, and 30 unlabeled test set families (14 solved, 16 unsolved). We tasked teams to identify causal variants in as many families as possible. Predictors submitted variant predictions with estimated probability of causal relationship (EPCR) values. Model performance was determined by two metrics, a weighted score based on the rank position of causal variants, and the maximum F-measure, based on precision and recall of causal variants across all EPCR values.

Results
Sixteen teams submitted predictions from 52 models, some with manual review incorporated. Top performers recalled causal variants in up to 13 of 14 solved families within the top 5 ranked variants. Newly discovered diagnostic variants were returned to two previously unsolved families following confirmatory RNA sequencing, and two novel disease gene candidates were entered into Matchmaker Exchange. In one example, RNA sequencing demonstrated aberrant splicing due to a deep intronic indel in ASNS, identified in trans with a frameshift variant in an unsolved proband with phenotypes consistent with asparagine synthetase deficiency.

Conclusions
Model methodology and performance was highly variable. Models weighing call quality, allele frequency, predicted deleteriousness, segregation, and phenotype were effective in identifying causal variants, and models open to phenotype expansion and non-coding variants were able to capture more difficult diagnoses and discover new diagnoses. Overall, computational models can significantly aid variant prioritization. For use in diagnostics, detailed review and conservative assessment of prioritized variants against established criteria is needed.