DOI: 10.5281/zenodo.3730598
title: The dataset of the ASE'20 paper titled "Automated Patch Correctness Assessment: How Far are We?"

description: This is the experiment result of the ASE'20 paper titled "<strong>Automated Patch Correctness Assessment: How Far are We?</strong>". If you use our data for academic research, please cite our paper as: <pre><code class="language-html">@inproceedings{wang2020automated, title={Automated Patch Correctness Assessment: How Far are We?}, author={Wang, Shangwen and Wen, Ming and Lin, Bo and Wu, Hongjun and Qin, Yihao and Zou, Deqing and Mao, Xiaoguang and Jin, Hai}, booktitle={Proceedings of the 35th International Conference on Automated Software Engineering (ASE)}, year={2020}, organization={ACM} }</code></pre> The file <strong><em>Patches.zip</em></strong> includes all the patches we take into consideration in this study. Note that 269 patches come from "Automated Patch Assessment for Program Repair at Scale (Ye et al.), Technical report 1909.13694, arXiv, 2019". The file <em><strong>Patches_for_Static</strong></em> include all the class files we used for static method. The file <em><strong>Tests-oracle </strong></em>includes all the test cases generated by <strong>Evosuite</strong> and <strong>Randoop</strong> on the fixed version programs. The file <em><strong>Tests-buggy</strong></em> includes all the test cases generated by <strong>Evosuite</strong> and <strong>Randoop</strong> on the buggy version programs. The file <em><strong>DiffTGen-result</strong></em> includes ingredients and output information of <strong>DiffTGen</strong>. The file <em><strong>Daikon-output</strong></em> includes inferred invariants of each patch and its corresponding ground-truth. The file <em><strong>PATCH-SIM_result</strong></em> includes the output vector files from <strong>PATCH-SIM</strong> and<strong> E-PATCH-SIM</strong>. The file <em><strong>Training_result</strong></em> includes the output of six ML algorithms with or without oracle. <pre><code class="language-xml">Chart: 1-26; Closure: 14, 18, 31, 33, 38, 40, 57, 62, 63, 70, 73, 86, 92, 93, 115, 123, 126; Lang: 6, 7, 10, 16, 20, 21, 22, 24, 26, 27, 33, 35, 38, 39, 41, 43, 44, 45, 50, 51, 55, 57, 58, 59, 60, 61, 63; Math: 2, 3, 4, 5, 6, 8, 20, 22, 25, 28, 30, 31, 32, 33, 34, 35, 39, 41, 49, 50, 53, 56, 57, 58, 59, 60, 61, 63, 65, 68, 70, 71, 73, 74, 75, 79, 80, 81, 82, 85, 86, 88, 89, 90, 93, 97, 98, 99, 104; Time: 4, 7, 11, 14, 15, 19. </code></pre> Please note that for bugs in the above table, the Evosuite tests on the fixed version programs are reused from a previous study. We thank <strong>He Ye</strong>, <strong>Matias Martinez</strong>, and <strong>Martin Monperrus</strong> so much for sharing their data. <strong>Notice!</strong> For patches under the folder <em>Patches_ICSE</em>, those under <em>Ddifferent</em> and <em>Dsame</em> folders are all correct patches. <em>Different</em> and <em>Same</em> only indicate whether the patch is syntactically identical to the ground truth patch. Patches generated for Mockito project (2 in total): Kali-A-Mockito-10; Arja-Mockito-10 Patches do not pass plausibility check (6 in total): Kali-Closure-133; kPAR-Chart-12; FixMiner-Chart-12; patch1-Lang-6-SketchFix-plausible; patch2-Lang-6-SketchFix-plausible; patch1-Math-2-SOFix Patches that are mistakenly labeled (12 in total): patch2-Lang-51-Jaid; patch1-Lang-43-CapGen; patch2-Lang-43-CapGen; patch2-Math-53-CapGen; patch2-Math-53-Jaid; jKali-Lang-7; ACS-Lang-35; Arja-Math-35; SimFix-Math-72; SimFix-Closure-19; Arja-Math-50; SimFix-Lang-60 Detailed reasons for the mislabeled patches: 1. the ground-truth patch modifies multiple locations while the generated patch only modifies one of them (2/12, SimFix-Math-72, SimFix-Lang-60); 2. the edit points in the generated patch are different from those in ground-truth patch (8/12, patch2-Lang-51-Jaid, patch2-Math-53-Jaid, patch1-Lang-43-CapGen, patch2-Lang-43-CapGen, patch2-Math-53-CapGen, ACS-Lang-35, SimFix-Closure-19, Arja-Math-50); 3. the generated patch doesnot fulfill the intended function in ground-truth (2/12, jKali-Lang-7, Arja-Math-35). Take <em>Arja-Math-50</em> as an example, this patch deletes a conditional statement which deals with an unexpected input (<em>null</em>) in the method <em><strong>verifyBracketing</strong></em>.<em><strong> </strong></em>However, in the oracle program, this conditional statement still exists. Then, <strong>Randoop</strong> generated a test case by calling <strong><em>verifyBracketing</em></strong> with a <em>null</em> argument. This test passed on the ground-truthpatch but failed on the patch generated by Arja due to the removeof the exception handling statements. As a result, this patch is actually overfitting but mistakenly labeled as correct. We have confirmed this case with Kui Liu, the first author of the recent ICSE'20 paper (Title: <em>On the Efficiency of Test Suite based Program Repair</em>) which makes up our patch benchmark. Border line Patches (3 in total): ACS-Lang-7; kPAR-Lang-7; TBar-Lang-7. Reasons for overfitting: Evosuite generates some tests that fail on those patches, e.g., test049 in Seed 1; the Java documentation above the function states that it needs to deal with the situation where the input cannot be converted. Reasons for correct: it synthesizes the correct modification; currently, in the program, <em>createBigDecimal()</em> is not called directly in other part of the production code except <em>createNumber()</em> and the test code. In our paper, we consider these three patches as correct and that's why Evosuite has 3 false positives.