Scientific Communities in OpenEBench
Creators: name: Sundesha, Vicky
name: Garrayo, Javier
name: Fernández, José M
name: Codó, Laia
name: Repchevsky, Dmitry
name: Juergen  Haas
name: Altenhoff, Adrian
name: Dessimoz, Christophe
name: Cuesta, Isabel
name: Monzon, Sara
name: López, Daniel
name: Florido, Javier Perez
name: Dopazo, Joaquín
name: Valencia, Alfonso
name: Gelpi, Josep LI
name: Capella-Gutierrez, Salvador
Publisher: name: F1000 Research Limited
Year: 2020

description: OpenEBench is the ELIXIR benchmarking and technical monitoring platform for bioinformatics tools, web servers and workflows. OpenEBench is part of the ELIXIR Tools platform and its development is led by the Barcelona Supercomputing Center (BSC) in collaboration with partners within ELIXIR and beyond.<br>Scientific benchmarking helps determining the precision, recall and other metrics of bioinformatics resources in unbiased scenarios, which have been set up through reference databases, <i>ad-hoc</i> input and test data sets reflecting specifying scientific challenges. Chosen metrics allow to objectively evaluate the relative scientific performance of the different participating resources. It is even possible to understand what are the software potential biases, strengths and weaknesses and/or under which conditions do they perform better or worse.<br><br>Unbiased and objective evaluations of bioinformatics resources are challenging to set-up and can only be effective when built and implemented around community driven efforts. Several communities from different scientific domains collaborate with OpenEBench in order to set-up, host and further develop their scientific efforts. Communities can focus on specific problems, e.g. Quest for Orthologs (QfO); or having a broader spectrum e.g. Spanish Network of Biomedical Research Centers on Rare Diseases (CIBERER); or covering different challenges on each of their editions, e.g. DREAM Challenges. Benchmarking efforts led by scientific communities might have a national scope e.g. CIBERER; or a global one e.g., Global Microbial Identifier Initiative (GMI).<br><br>Most communities have similar needs in terms of reference data sets, metrics, and benchmarking results accessible within the community and beyond, independently of the scientific challenges tackled by each community and their geographical scope. Data sets should reflect existing challenges of the scientific community in terms of size, complexity, and content. Moreover, data sets are used for producing predictions by participants and to compute the performance of each participant when comparing their predictions against a previously agreed, often private, data sets that are referred many times as golden data sets. Metrics are used to measure the performance of individual participants and should reflect the common practices in the field. Finally, making results available to the community and beyond is as relevant as generating them. It is important that researchers can access these results at any time, and have the tools to assist them in understanding them. Moreover, associated data and metadata to any benchmarking efforts should fulfill the FAIR principles and be available in long-term repositories such as Zenodo and/or EUDAT with permanent digital identifiers for further use and re-use.<br><br>Thus, OpenEBench has engaged with different communities offering assistance to bring their previously generated data and activities into the platform. Communities can make use of any of the three available levels in the OpenEBench architecture. However, how communities use the platform depends on their specific needs and resources. To ensure the long-term sustainability of OpenEBench, we are implementing a co-production model to accelerate the incorporation of new communities and the maintenance of the existing ones.