DOI: 10.5061/dryad.k6djh9w5h
title: Survey of software engineering in code used in published papers

description: Background: Computer code underpins modern science, and at the
 present time has a crucial role in leading our response to the COVID-19
 pandemic. While models are routinely criticised for their assumptions, the
 algorithms and the quality of code implementing them often avoid scrutiny
 and, hence, scientific conclusions cannot be rigorously justified.
 Problem: Assumptions in programs are hard to scrutinise as they
 are rarely explicit in published work. In addition, both algorithms and
 code have bugs, effectively unknown assumptions that have unwanted
 effects. Code is fallible. Any model interpretation that relies on code is
 therefore fallible, and if the code is not published with adequate
 documentation, the code cannot be scrutinised. In turn, the scientific
 claims cannot be properly scrutinised. Solutions: Code can be
 made much more reliable using software engineering good
 practice. Three specific solutions are proposed. First, professional
 software engineers can help and should be involved in critical research.
 Secondly, “Software Engineering Boards” (supplementing and analogous to
 Ethics or Institutional Review Boards) must be instigated and used.
 Thirdly, code, when used, must be considered an intrinsic part of any
 publication, and therefore must be formally reviewed by competent software
 engineers. The paper’s Supplementary Material includes a summary of
 professional software engineering best practice, particularly as applied
 to scientific research and publication.
description: The web sites of journals were accessed for their current (July
 2020, double-checked January 2021) issues. Peer reviewed research article
 (i.e., excluding commentary, letters, etc) were selected on the basis that
 their titles implied code had been used in the
 published research. A list of full citations, including
 DOIs, is provided in the supplementary material. The
 associated articles (PDFs) were read by the author, and all links to
 supplementary material examined. Where possible, code from the links was
 downloaded and assessed.  In addition, the code
 policies, where available, of the relevant journals (at the time of the
 survey) were copied into the supplementary material.
 The article’s supplementary material has further details of the
 assessment and the individual results, and is therefore an essential part
 of this Dryad data.
 <strong>NOTE</strong>
 It is important that this survey is not taken as an evaluation,
 whether praise or criticism, of any specific paper: any measurement is
 subject to error, and individual paper assessments are inevitably noisy.
 Instead, by evaluating a set of diverse papers, the noise will tend to
 average out. Mean scores, as summarised in the paper, are more reliable
 measurements.
description: Raw data is stored in JSON in the file data.js
 The file data.js defines the data format and includes JavaScript
 that validates and analyses the data. If you run
 data.js as a program - eg by using <pre> node
 data.js</pre> you will generate all data files in various
 formats, and Latex files for all analyses used in the paper.
 In particular, the file allData.csv (generated by running
 node.js) can be read in Excel or other applications. It provides all data
 used in the paper. For convenience, an HTML file which
 can be read in a browser is produced - this makes it easy to follow the
 DOIs back to the original sources. A NB file (Mathematica) is also
 generated, which can be used for analysing and visualising the data in
 many other ways (eg using ML). The Latex file
 summarise.tex (typeset as summarise.pdf) will format all data from node.js
 as used in the paper. Note that summarise.tex does not attempt to format
 NB and HTML files that node.js also generates, as these are not in Latex
 formats.