Automated Software Process Performance Analysis and Improvement Recommendation

Software development processes can generate significant amounts of data that can be periodically analyzed to identify performance problems, determine their root causes and devise improvement actions. However, there is a lack of methods and tools for helping in that kind of analysis. Conducting the analysis manually is challenging because of the potentially large amount of data to analyze, the effort and expertise required and the lack of benchmarks for comparison. Hence, the goal of this dissertation is to develop methods, models and tools for automating the analysis of process performance data and identifying and ranking performance problems and their root causes, reducing effort and errors and improving user satisfaction as compared to previous approaches. The main contributions of the dissertation are a novel method for process performance analysis and improvement recommendation (the ProcessPAIR method), a support tool (the ProcessPAIR tool), and a performance model for instantiating ProcessPAIR for the Personal Software Process (the ProcessPAIR model for the PSP). In the ProcessPAIR method, the analysis of the performance data of an individual process user is based on a performance model that is defined by an expert in the process under consideration and calibrated automatically from the performance data of many process users. To enable the automatic identification of performance problems, the process expert has to define the relevant (top-level) performance indicators (PIs); recommended performance ranges, needed for classifying and ranking the values of PIs according to three semaphors (red, yellow or green), are derived from calibration data sets. To enable the automatic identification of potential root causes, the process expert has to indicate hierarchical cause-effect relationships between the top-level and more detailed performance indicators. To rank the identified root causes, a novel ranking method is proposed based on the computation of a ranking coefficient that represents a cost-benefit estimate of improvement efforts; the needed data comes from the training data set (statistical distribution of individual PIs and regression models between related PIs). The support ProcessPAIR tool is able to automatically identify and rank performance problems and their potential root causes, so that subsequent manual analysis for the identification of deeper causes and improvement actions can be properly focused. Performance models for a process under consideration are defined as tool extension and calibrated automatically based on the performance data of many process users. The results of the analysis of the performance data of an individual process user are shown in multiple views. A performance model was developed to enable the application of ProcessPAIR for the Personal Software Process (PSP). The model addresses time estimation accuracy, quality, and productivity as top-level PIs. A PSP data set from the Carnegie Mellon Software Engineering Institute, referring to more than 30,000 projects, was used to validate and calibrate the model. Two experiments were conducted to validate the approach. In a postmortem experiment in which we compared the results of manual performance analysis (by 20 students from Instituto Tecnológico de Monterrey in Mexico) and automatic performance analysis, ProcessPAIR was able