Framework for measuring program comprehension

Program comprehension is a major human factor in software development. It often makes the difference between success and failure of a software product, because maintenance programmers spend most of their time with understanding code, and because maintenance is the main cost factor in software development. Thus, if program comprehension is not supported properly, time and cost for software development increase significantly. Although program comprehension has such importance, researchers and practitioners do not consider it properly, but use plausibility arguments to claim positive or negative effects on program comprehension. Program comprehension is an internal cognitive process, so we cannot rely on plausibility arguments only, but need to conduct controlled experiments to empirically evaluate what improves and what impairs program comprehension. We believe that one reason for this mismatch between the importance of program comprehension and its insufficient evaluation is the effort of conducting controlled experiments. Thus, a plethora of new programming paradigms and techniques exist that supposedly improve program comprehension, but have not been evaluated empirically to confirm a positive effect on program comprehension. Although there is effort to improve program comprehension, it is not clear whether this effort has success. To improve the current situation, we define two goals in this thesis: First, reducing the effort for conducting controlled experiments, and, second, creating first empirical evidence how one of the plethora of new approaches, feature-oriented software development, affects program comprehension. To address our first goal, we first evaluate whether software measures represent a suitable alternative to controlled experiments. Software measures rely only on sourcecode properties and do not require observing human participants. However, the results indicate that software measure are not a reliable substitute for controlled experiments with human participants. Hence, we aim at reducing the effort for controlled experiments with human participants. To this end, we conduct a literature survey of controlled experiments of the last ten years published in seven major journals and conferences of the software-engineering domain. This way, we get an overview of the state of the art of measuring program comprehension, on which we base a framework to support planning, conducting, and replicating experiments. Specifically, we provide a list of confounding parameters for program comprehension, including control techniques, so that researchers know how the validity of experiments is threatened and how they can address the threats. Doing so, we reduce the major obstacle for conducting controlled experiments with human participants. Furthermore, we develop and evaluate a questionnaire to reliably and conveniently measure programming experience, the most important confounding parameter for program comprehension. Additionally, we provide a tool that helps researchers to design and replicate experiments. With our second goal, we analyze how a specific programming approach, featureoriented software development, affects program comprehension. To this end, we conduct a series of controlled experiments, in which we show how different facets of featureoriented software development affect program comprehension. Specifically, we show