Hitting The Bull'S Eye With Darts: Artifact Evaluation In Computer Science
Creators: name: Wagner, Michael
Publisher: name: Zenodo
Year: 2017
Subjects: artifact evaluation
dagstuhl
reproducibility
research artifact
research data
research data management
research evaluation

description: The publishing of research data and artifacts is currently in the general focus of the scientific community and funding agencies. In its 2012 Communication, the European Commission criticised the lack of citation mechanisms for and access to research data, subsequently making citable and retrievable publication of research data one of the cornerstones of the Open Science policy of the Horizon 2020 program. This goes in line with the most recent recommendations of the German Research Foundation DFG regarding the handling of research data.

Evaluation and publishing of research data and artifacts has increasingly become a topic of discussion in computing research also, since researchers were complaining about poor repeatability, reproducibility, correctness, and buildability. Beginning already in 2011, an increasing number of conferences (especially in software sciences) have established artifact evaluation committees to ensure artifacts submitted along with research papers to be consistent with the paper, complete, well documented and easy to reuse.

The Dagstuhl Artifacts Series (DARTS) - established in close cooperation with stakeholders from the computing research disciplines  and maintained by Schloss Dagstuhl - Leibniz Center for Informatics - publishes evaluated research data and artifacts in all areas of computing research. Especially, DARTS is taking the conference-focused publication culture in computing research into account.

An artifact can be any kind of content related to computing research, e.g., experimental data, source code, virtual machines containing a complete setup, test suites, or tools. In contrast to existing repositories for research data and artifacts like Zenodo or figshare, DARTS focuses on artifacts that underwent an evaluation process before their publication which is done by peers of the community.

This talk will give an introduction to the publication culture in computing research, the setup and implementation of artifact evaluations and the publishing of the artifacts in the DARTS series.
description: {"references": ["Moshe Vardi, Comm. ACM, vol 54(1)", "David Patterson, Lawrence Snyder, Jeffrey Ullman (1999). \"Evaluating Computer Scientists and Engineers For Promotion and Tenure\". Available online: http://cra.org/resources/best- practice-memos/evaluating-computer-scientists-and-engineers-for-promotion-and-tenure/", "Ainsley Seago, doi:10.1371/journal.pbio.1001779.g001", "July 2012 Commission's Recommendation on access to and preservation of scientific information. Available online: http://ec.europa.eu/research/science- society/document_library/pdf_06/recommendation-access-and-preservation- scientific-information_en.pdf", "PLOS Data Availabitity: http://journals.plos.org/plosone/s/data-availability", "Whyte, A., Tedds, J. (2011). 'Making the Case for Research Data Management'. DCC Briefing Papers. Edinburgh: Digital Curation Centre. Available online: http://www.dcc.ac.uk/resources/briefing-papers", "\"Reality Check on Reproducibility.\" 2016. Nature. Springer Nature. doi:10.1038/533437a.", "Grigori Fursin (2017). 'CGO/PPoPP'17 Artifact Evaluation Discussion (enabling open and reproducible research)'. Available online: https://www.slideshare.net/GrigoriFursin/cgoppopp17-artifact-evaluation-discussion-enabling-open-and-reproducible-research", "DataCite Metadata Working Group. 2016. \"DataCite Metadata Schema Documentation for the Publication and Citation of Research Data v4.0.\" Edited by Jan Ashton, Amy Barton, Tina Bradford, Anne Ciolek-Figiel, Stefanie Dietiker, Jannean Elliot, Berrit Genat, et al. DataCite e.V. doi:10.5438/0012."]}