DOI: 10.5281/zenodo.4768772
title: Effectiveness Assessment of Open-Source Static Analysis Tools

description: This spreadsheet presents data regarding the effectiveness of four open-source static analysis tools (SATs), namely PMD, SpotBugs, SonarLint, and Infer. These tools were evaluated considering a set of 35 Java projects collected from well-known benchmarks, such as Defects4J, QuixBugs, and NpeFix Dataset. The data set shows the number of true/false positives/negatives identified from the analysis performed by each SAT. Assessing the number of false positives/negatives is particularly essential for analyzing SATs since they might report warnings about faults that do not exist or miss warnings about actual faults. While false negatives impact the reliability of SATs because faults are missed, false positives lead developers to waste time on investigating incorrect warnings in the program under analysis, thus having a greater potential of negatively affecting the usability of SATs. These measures of true/false positives/negatives can be used to determine both precision and recall of each SAT, which are relevant metrics in this context as SATs are known for generating false positives/negatives, which may limit their adoption. For the effectiveness assessment, a manual analysis was carried out upon the execution output of each tool for each sample. In the 134 executions finished without tool-related errors, a process of classification and partitioning was carried out to filter out only indications of NPE-related faults, which are the object of interest of this work. False negatives were considered as the amount of known NPE-related faults that were not detected in a sample. This amount was obtained by considering the number of known faults informed by the selected benchmarks or the ones identified by manually inspecting the small-sized samples.
description: {"references": ["Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source software. In Proceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering. IEEE, USA, 470\u2013481. https://doi.org/10. 1109/SANER.2016.105", "Thomas Durieux, Benoit Cornu, Lionel Seinturier, and Martin Monperrus. 2017. Dynamic patch generation for Null Pointer Exceptions using metaprogramming. In Proceedings of the 24th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, USA, 349\u2013358. https://doi.org/10.1109/SANER. 2017.7884635/", "Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don't software developers use static analysis tools to find bugs?. In Proceedings of the 35th International Conference on Software Engineering. IEEE, USA, 672\u2013681.", "Ren\u00e9 Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. ACM, USA, 437\u2013440. https://doi.org/10.1145/2610384.2628055.", "Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017. QuixBugs: A multi-lingual program repair benchmark set based on the Quixey Challenge. In Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity. ACM, USA, 55\u201356. https://doi.org/10.1145/3135932.3135941.", "Diego Marcilio, Carlo A. Furia, Rodrigo Bonif\u00e1cio, and Gustavo Pinto. 2020. SpongeBugs: Automatically generating fix suggestions in response to static code analysis warnings. Journal of Systems and Software 168 (Oct. 2020). https: //doi.org/10.1016/j.jss.2020.110671.", "PMD \u2013 An extensible cross-language static code analyzer. Available at https://pmd.github.io/.", "SonarLint \u2013 Fix issues before they exist. Available at https://www.sonarlint.org.", "SpotBugs \u2013 Find bugs in Java programs. Available at https://spotbugs.github.io/.", "Infer Static Analyzer. Available at https://fbinfer.com."]}