DOI: 10.5281/zenodo.3610821
title: Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package

description: <strong>Does Unit-Tested Code Crash? A Case Study of Eclipse: Replication Package</strong> This is a replication package associated with the paper titled “Does Unit-Tested Code Crash? A Case Study of Eclipse”. Below is a description of the package’s contents. <strong>Data</strong> Data files associated with the paper are provided in the <code>data</code> directory. <strong>Text file <code>tested-crashed.txt</code></strong> Data specifying whether methods were tested and whether they crashed (according to the criteria adopted in the study). Extracted from <code>matches.xlsx</code>. The data are used as input for Fisher’s test (RQ1). <strong>Spreadsheet <code>matches.xlsx</code></strong> Test coverage data and calculations associated with failed methods, class coverage, and matching method coverage results are provided in an Excel spreadsheet. Below is the description of the spreadsheet’s contents. Worksheet <em>Test Coverage</em> Contains the data regarding the JaCoCo test code coverage analysis. Class: The name of the class in which a method appears in JVM internal form notation Method: The method’s name Parameters: The method’s arguments in JVM parameter descriptor format; required to handle Java’s {} polymporphism Class Has Unit Test: Whether the corresponding class has associated unit test code Class Unit-Test Line Density: The ratio of lines in class’s test code over those in the class’s implementation code Covered Instructions / Branches / Lines: As reported by JaCoCo Total Instructions / Branches / Lines: As reported by JaCoCo Covered Instructions / Branches / Lines ratio: The ratio between the two preceding values; 1 for methods without any branches Top-1 / Top-6 / Top-10 : In how many stack traces the method appears within; the top-10 / top-6 / the very first stack frame(s) Tested: TRUE if the method is considered tested by having a test code coverage above the median (0.966) and an associated test class Crashed: TRUE if the method has crashed as evidenced by its appearance on the topmost stack frame Stack trace file names: in which the method appeared Worksheet <em>Test Existence</em> Contains the data of the analysis regarding the existence of test code. Class: Class containing implementation code TestClassNames: Classes that contain tests for the above Number of relevant tests Lines in class test code Lines of class Class Unit-Test Line Density: The ratio between the two above Worksheet <em>Metrics</em> Contains the derivation of metrics reported in the paper. In the cases of tables these are formatted in LaTeX for direct incorporation into the text. <strong>Spreadsheet <code>jacoco.xlsx</code></strong> Complete test coverage data obtained from JaCoCo are provided in an Excel spreadsheet. Below is the description of the spreadsheet’s contents. Worksheet <em>Data</em> Contains the following method code coverage fields as reported by JaCoCo, as well as the calculated percentages. Class Method Parameters Covered Instructions Total Instructions % Covered Instructions Covered Branches Total Branches % Covered Branches Covered Lines Total Lines % Covered Lines Worksheet <em>Metrics</em> Contains the derivation of numbers reported in the preliminary quantitative analysis and Figure 2. Compressed tar archive <code>eclipse-src.tar.gz</code> Contains the Eclipse source code used for running the Eclipse tests with JaCoCo code coverage analysis. It was obtained from the Eclipse source code repositories as follows. Clone the Eclipse aggreagator repository into a directory named z by running: <code>git clone -b master --recursive git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.aggregator.git z</code> In the <code>z</code> directory, checking out the used release by running <code>cd z && git submodule foreach git checkout M20160212-1500</code> Checking out the release for the main repository by running: <code>git checkout M20160212-1500</code> Applying the patch <code>eclipse-src.diff</code> <strong>Patch file <code>eclipse-src.diff</code></strong> See above. <strong>Zip file <code>incidents.zip</code></strong> Contains the 126,026 incidents (crash report stack traces and meta-data) associated with <em>EclipseProduct</em> <code>org.eclipse.epp.package.java.product</code> and <em>BuildID</em> <code>4.5.2.M20160212-1500</code>. This is a subset from the two million incidents available as the AERI stack traces data set. The subset of incidents was extracted from the full AERI data set with the following command. <pre><code class="language-bash">for f in *; do grep -q org.eclipse.epp.package.java.product $f && grep -q 4.5.2.M20160212-1500 $f && mv $f selected-files/ done</code></pre> <br> <strong>Compressed file <code>jacoco.xml.gz</code></strong> Contains the results of the JaCoCo code coverage analysis over the Eclipse testing. <strong>Code</strong> The following scripts are provided in the <code>src</code> directory <code>extract.py</code>: script for extracting crash (incidents) and coverage (JaCoCo) data <code>unit-tested-classes.py</code>: script for finding the classes with associated unit test code <code>merge.py</code>: script for matching crash (incidents) with coverage (JaCoCo) data <code>fisher.r</code>: R script for running Fisher’s test