Identification and characterization of software faults with application on regression testing
Creators: name: LAALI, Mohsen
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 461208
subject: Software testing, verification and validation

description: The knowledge about fault characteristics can be explored to better target the most fault-prone part of the software. This information includes identifying and grouping similar faults together to model their behaviour to predict faults. This also enables developers following best practices to avoid code smells, which are common software patterns which indicate the possibility of a fault. To this aim, this thesis identifies and characterizes software faults and applies its findings in software testing, as an application of utilising fault characteristics. Characterizing faults is not only useful for software testing, but also the other areas of software engineering such as software reliability, to which information about faults is critical.

Past research estimates that one-fourth of software development cost is related to identify software faults. Faults are not evenly spread through the software, and it is more likely that most of the defects are clustered in a particular part of the software, which is also referred to as the Pareto principle in software engineering. Although the Pareto principle has been used in many software engineering methods, the underlying rationale for this principle yet has not been fully explored empirically with the aim of minimising the costs. A crucial step towards optimising the costs, first and foremost, requires characterizing and understanding fault behaviours in the development of software.

To be enable investigation of fault characteristics on a large scale, this thesis uses version control and issue tracker data, where most of the software developers keep the history of their code developments and discussions, respectively. This research utilises the issue tracker as metadata to classify software changes stored in version control as fix and non-fix related changes. Fix changes lead us to find a faulty section of the software. This approach helps us to identify thousands of faults during software development, where we can investigate the behaviour and characteristics of faults, as the main goal of this thesis.

To better understand faults, the first contribution focuses on finding any relation between a fault and a software change, in other words, how a change in software may result in introducing a fault. There is a positive correlation between fault and software change. However, these correlations vary among different granularities of a change. It is discovered that commits obtain a higher correlation to faults compared to added or deleted lines. This high correlation may indicate that the number of commits, a close representation of tasks, is more influential on software faults compared to the length of tasks as reflected by the number of added or deleted lines. This experiment is further extended to multiple projects from six C-family languages. The findings show that Java, as an object-oriented programming language, is the least sensitive language to faults regarding the number of commits. In contrast, PHP as a scripting language has the highest correlation between commits and faults. Furthermore, the spread of faults in software project files is investigated. The results manifest that 80% of software project faults are densely located in less than 20% of the code, which empirically validates the Pareto principle in software engineering. Further extending the experiment using multiple projects from six C-family languages indicates different density rates with the location of faults in project files for various programming languages. Specifically, scripting languages, which are usually lengthy with a complex code kept in few files, have a higher density of faults in fewer files compared to Java as an object-oriented language where each file represents only one class. This finding may suggest that programming languages have different levels of difficulty.

Following the latter findings, the next contribution of this thesis explores the association between faults and programming language keywords to discover the under- lying reason for various difficulty levels of programming languages. This contribution finds out that the use of different programming language keywords is associated with software faults to different extents. Moreover, programming language keywords are categorised into different groups based on their role in software development which also shows various fault associations. For instance, conditional language keywords like "if" are more likely to be involved in software faults compared to the loop language keywords such as "while". Finally, this research discovers that the fault associations with language keywords are consistent among different software projects.

The last contribution applies the findings of the previous two contributions to pro- pose a more effective test case prioritization technique. Former contributions imply that faults appear in same approximation and that code coverage could be improved by using weighted schema on programming language keywords based on their association to software faults. These verdicts guide us to introduce a family of test case prioritization techniques prioritizing test cases that cover parts of the code which already have discovered a fault.

Overall, the findings of the fault characteristics help software engineering methods to be more effective and efficient in areas to which the fault information is crucial, which has been illustrated through applications in software testing.