OpenEBench. TCGA: A community-led benchmarking use case.
Creators: name: Ventas, Javier Garrayo
name: Sundesha, Vicky
name: Porta-Pardo, Eduard
name: Gonzalez, José María Fernández
name: Codó, Laia
name: Valencia, Alfonso
name: Jospe Ll Gelpi
name: Capella-Gutiérrez, Salvador
Publisher: name: F1000 Research Limited
Year: 2020

description: The Cancer Genome Atlas (TCGA) community is a joint effort to characterize cancer driver genes in 33 different cancer types from nearly 10,000 exomes [1]. In such an effort, several methods for predicting cancer genes from genomic data are constantly being developed and improved.<br><br>OpenEBench (https://openebench.bsc.es), is the ELIXIR benchmarking and technical monitoring platform for bioinformatics tools, web servers and workflows. OpenEBench is part of the ELIXIR tools platform ecosystem (https://www.elixir-europe.org/platforms/tools). Its development is led by the INB/ELIXIR-ES (https://inb-elixir.es) Central node at the Barcelona Supercomputing Center (BSC) in collaboration with different partners across Europe. It is focused on providing an infrastructure where end-users can learn from different available options and select the one best fitting their scientific needs; bioinformatics software developers can find relevant datasets and meaningful scientific challenges to evaluate their own developments, and communities interested in a particular scientific domain can easily define which datasets and metrics are relevant for developers to work on, which in turn will allow the field to move ahead.<br><br>TCGA, as a engaged community in OpenEbench, provides an ideal use-case to demonstrate the latest developments at OpenEBench including the deployment of use of a Virtual Research Environment (VRE). End-users can have access to the software tools who has participated in the TCGA effort to identify cancer driver genes across 33 different cancer types while developers have the opportunity to evaluate their latest developments by submitting their predictions to the OpenEBench VRE. A predefined workflow including format and data validation, metrics computation, results consolidation and visualization is executed for each submission. Results can be visualized at the VRE, downloaded and/or extended by using a number of scripts provided via the OpenEBench repository [2].<br><br>In the case of TCGA, users can test their performance against 8 different tools. A list of gene identifiers should be submitted for doing so. They are then compared against several reference datasets defined by the TCGA community depending on the different cancer types, which results in two assessment metrics (true positive rate and precision). This allows the submitted predictions to be benchmarked against the rest of the participants currently present within the TCGA community. The benchmark results are provided in a compressed file (containing the assessment data for all participants and an SVG image with the visualization) and can also be visualized in a scatter-plot where the user can then apply several classification methods implemented in OpenEBench in order to transform the results to table format, which are easier to interpret by non-expert users.<br><br>[1] Bailey, M. H., Tokheim, C., Porta-Pardo, E., Sengupta, S., Bertrand, D., Weerasinghe, A., … Mariamidze, A. (2018). Comprehensive Characterization of Cancer Driver Genes and Mutations. Cell, 173(2), 371–385.e18.<br>[2] https://github.com/inab/TCGA_visualizer