Artifact for Article #5907 Assessing The Effectiveness of Test Suites AlBalwi et al. Abstract—In this document we introduce the data that we 1 generated for our experiment, and explain how readers can 2 duplicate our experiment using the data. 3 Index Terms—Test suites, test suite effectiveness, syntactic cov- 4 erage, semantic coverage, detector sets, absolute correctness, 5 relative correctness, partial correctness, total correctness. 6 I. DATA ORGANIZATION 7 Semantic Coverage is the measure we introduce in our paper 8 to reﬂect the effectiveness of a test suite to reveal the failures of 9 a program with respect to a speciﬁcation. The purpose of our 10 empirical study is to illustrate how we compute the semantic 11 coverage of test suites for a benchmark program; to relate 12 semantic coverage to existing measures of test suite effective- 13 ness, we also show how these test suites are ranked according 14 to their ability to kill mutants of the sample program. To this 15 effect, we generate two sets of mutants of the base program, 16 and for each set, we rank the test suites by set inclusion of 17 the sets of mutants that they kill. 18 The data is organized into eight directories: 19 • Source. The benchmark program we are using is a 20 simulator of the VT100 terminal (called jTerminal, jT 21 for short), written in Java and often used in testing 22 experiments. This program comes with a test class that 23 includes 35 tests; we have added two tests to this class, for 24 a total of 37 tests, for the deliberate purpose of tripping 25 jTerminal, i.e. causing it to diverge. The reason for this is 26 that we want to distinguish between total correctness and 27 partial correctness, and obtain different values of semantic 28 coverage depending on the standard of correctness that 29 we consider. 30 This directory includes two ﬁles, the source code of 31 jTerminal, and the source code of the test class. 32 • TS: The 20 Test Suites. From this expanded test suite of 33 37 tests (whose elements we number from 1 to 37), we 34 generate twenty (20) subsets as follows: 35 – T 1...T 5: Five subsets obtained by removing ﬁve ran- 36 dom elements (and ensuring that they are distinct). 37 – T 6...T 10: Five subsets obtained by removing ten ran- 38 dom elements (and ensuring that they are distinct). 39 – T 11...T 15: Five subsets obtained by removing ﬁfteen 40 random elements (and ensuring that they are distinct). 41 – T 16...T 20: Five subsets obtained by removing one 42 random element (and ensuring that they are distinct). 43 Directory TS contains the 20 test suites T 1... T 20. 44 • TOT25. The semantic coverage of a test suite T to test 45 a program P depends not only on T and P, but also 46 on the standard of correctness (partial, total) and on the 1 speciﬁcation with respect to which correctness is tested. 2 For the purpose of this experiment, we have used two 3 mutants of jTerminal, namely M25 and M50, as the 4 speciﬁcations with respect to which correctness is tested. 5 This directory contains the semantic coverage of test 6 suites T 1 ... T 20 for total correctness with respect to 7 M25; these ﬁles are named TOT25T1 ... TOT25T20. 8 This directory also contains the graph that ranks the 9 test suites T 1 ... T 20 by their semantic coverage; this 10 graph is represented in two formats, namely the list 11 of arcs (ArcsTOT25) and the graphical representation 12 (GraphTOT25) (also shown in Figure 5). 13 • PAR25. This directory contains the semantic coverage 14 of the twenty test suites T 1 ... T 20 of jTerminal for 15 partial correctness with respect to M25; these ﬁles are 16 named PAR25T1 ... PAR25T20. This directory also 17 contains the graph that ranks the test suites T 1 ... T 20 by 18 their semantic coverage; this graph is represented in two 19 formats, namely the list of arcs (ArcsPAR25) and the 20 graphical representation (GraphPAR25) (also shown in 21 Figure 4). 22 • TOT50. This directory contains the semantic coverage 23 of the twenty te