DOI: 10.6084/m9.figshare.29666057
title: Diagnostics for State-Based Specifications dataset

description: This dataset accompanies the paper "Diagnosing Violations of State-Based Specifications in iCFTL", by Cristina Stratan, Claudio Mandrioli, and Domenico Bianculli.Together with the dataset (), we provide:the source code (iCFTL-Diagnostics-tool.zip) of the tool developed as part of the contribution of this paper. The instructions on usage can be found in the README in the iCFTL-Diagnostics.zip file.a projects.zip file that contains the 1 folder and 2 files. The folder <i>Project</i> contains projects selected from DyPyBench benchmark that have the specifications in their corresponding folders. We kept the original naming of the projects, i.e., project2.<br>In order to know the path of the specifications and their executables we have file project_details.csv, where we have columns:<b>ID </b>- Unique identifier for each of the 112 specifications<b>Project </b>- Short identifier for th rporject the specification belongs to, (e.g, P2 for project2)<b>Specification </b>- Path to the specification file within the project.<b>Program</b> - Path to the program file associated with the specification.<b>Module_path</b> - Path to the module where the specification is applied.<b>Project_path </b>- Original path of the project.<b>Executable </b>- The test that triggers execution of the instrumented program based on the specification.<br>We have the ground truth file (GroundTruth.csv) that we built in order to verify the effectiveness of our tool. The file has the columns:<b>ID </b>– Unique identifier for each specification.<b>Project</b> – Short identifier for the project (e.g., P2).<b>Specification</b> – Path to the specification file.<b>Program</b> – Path to the program file that the specification is written over.<b>Function</b> – Name of the function where the relevant statement is located.<b>Line_number</b> – Line number of the relevant statement.<b>Statement</b> – The actual code statement identified as relevant.<b>Cause</b> – Explanation or rationale for why the statement is considered relevant. It can be either because a statement is relevant to another line number or because it is relevant to the atom or quantifier.<b>Function_proximity </b>– Proximity of the statement to the specification violation, measured at the function level.<br><br>The experiment_result.zip has four folders:<b>rq1</b> for research question RQ1<b>rq2</b> for research question RQ2<b>rq3</b> for research question RQ3<b>rq4</b> for research question RQ4In folder <b>RQ1</b>, we have a file (rq1.csv) that contains the data used to answer <b>RQ1: </b><i>How effective is \icftlDiagnostics in identifying the statements relevant to a specification violation?</i>, and it has the following columns:<b>Specification</b> – The path of the specification being analysed.<b>Precision_0.0</b> – Precision score computed for proximity level 0.0.<b>Recall_0.0</b> – Recall score computed for proximity level 0.0.<b>Precision_1.0</b> – Precision score computed for proximity level 1.0.<b>Recall_1.0</b> – Recall score computed for proximity level 1.0.<b>Precision_2.0</b> – Precision score computed for proximity level 2.0.<b>Recall_2.0</b> – Recall score computed for proximity level 2.0.<b>Precision_3.0</b> – Precision score computed for proximity level 3.0.<b>Recall_3.0</b> – Recall score computed for proximity level 3.0.<b>Precision_4.0</b> – Precision score computed for proximity level 4.0.<b>Recall_4.0</b> – Recall score computed for proximity level 4.0.<b>Precision_5.0</b> – Precision score computed for proximity level 5.0.<b>Recall_5.0</b> – Recall score computed for proximity level 5.0.<b>Precision_inf</b> – Precision score computed when considering all program points (i.e., proximity infinity).<b>Recall_inf</b> – Recall score computed when considering all program points (i.e., proximity infinity).In folder RQ2 we have rq2_sloc.csv and rq2_func.csv files that contain the data used to answer <b>RQ2</b>: <i>To what extent does \icftlDiagnostics reduce the complexity of understanding specification violations?</i>. We quantified the reduction in complexity at two levels of granularity: Fine-grained, measured in terms of lines of code (SLOC) and Coarse-grained, measured in terms of functions.We used two separate files to capture these perspectives:In rq2_H_E_fam.csv we have columns:<b>specification</b> – the path of the specification being analysed.<b>program</b> – the name of the program the specification is written over.<b>GT_sloc</b> – the total number of source lines of code (SLOC) in the procedures that have at least one statement in the Ground Truth.<b>diagnosis_sloc</b> - the number of statements identified by iCFTL-Diagnostics as relevant to the specification violation.<b>diagnosis_halstead_effort </b>- the Halstead Effort computed on the portion of the program that contains only statements in the diagnosis.<b>GT_halstead_effort </b>- the Halstead Effort computed on the statements counted in <b>GT_sloc</b>.<b>complexity</b> – the complexity score computed with the ratio <b>diagnosis_halstead_effort/</b><b>GT_halstead_effort</b>.<b>reduction_complexity</b> – the reduction of complexity obtained by computing 1 - <b>complexity</b>.In rq2_H_E_unfam.csv we have columns:<b>specification</b> – the path of the specification being analysed.<b>program</b> – the name of the program the specification is written over.<b>program_sloc </b>- the total number of source lines of code (SLOC) in the program.<b>diagnosis_sloc </b>- the number of statements identified by iCFTL-Diagnostics as relevant to the specification violation.<b>diagnosis_halstead_effort </b> - the Halstead Effort computed on the program statements included in the diagnosis.<b>program_halstead_effort </b>- the Halstead Effort computed on the whole program.<b>complexity</b> – the complexity score computed with the ratio <b>diagnosis_halstead_effort</b>/<b>program_halstead_effort</b>.<b>reduction_complexity</b> – the reduction of complexity obtained by computing 1 - <b>complexity</b>.<br>In folder <b>RQ3</b>, we have a file (rq3.csv) that contains the data used to answer <b>RQ3: </b><i>How much time and memory does the iCFTL-Diagnostics approach require?</i>, and it has the following columns:<b>file_path</b> – for each specification we save the time and memory that it takes to execute the insrumented program (saved in time-mem-x.json).<b>mean_time</b> – the mean time (in milliseconds) taken by iCFTL-Diagnostics to analyse the trace, computed over 5 runs.<b>std_time</b> – the standard deviation of the execution time across the 5 runs.<b>mean_memory</b> – the mean memory (in kB) taken by iCFTL-Diagnostics to analyse the trace, computed over 5 runs.<b>std_memory</b> – the standard deviation of memory usage across the 5 runs.<b>min_time</b> – the minimum execution time recorded across the 5 runs.<b>max_time</b> – the maximum execution time recorded across the 5 runs.<b>min_memory</b> – the minimum memory usage recorded across the 5 runs.<b>max_memory</b> – the maximum memory usage recorded across the 5 runs.<b>trace_length</b> – the number of events in the trace file.<b>bindings</b> – the number of bindings (generated by a specification) processed by \icftlDiagnostics, which may influence both time and memory consumption.In folder <b>RQ4</b> we have memory.csv and time.csv files that contain the data used to answer <b>RQ4</b>: <i>What is the overhead in terms of time and memory consumption introduced by iCFTL-Diagnostics </i><i>instrumentation?</i>. Both files have the following columns:<b>ID</b> – a unique identifier for each of our 112 specifications.<b>Spec</b> – the file name of the specification being analysed.<b>D</b> – the time (in seconds) or memory (in kB) consumption of the diagnostics instrumentation.<b>N</b> – the time (in seconds) or memory (in kB) consumption of the non-instrumented program.<b>V</b> – the time (in seconds) or memory (in kB) consumption of the vanilla instrumentation.<b>delta_D_V</b> – the normalized overhead of diagnostics instrumentation relative to vanilla instrumentation.<b>delta_D_N</b> – the normalized overhead of diagnostics instrumentation relative to the non-instrumented program.<br>