For instance, firms may benefit from training or, to the extent that STW programs may act as driving engines of successful school reform, entire school systems and non-STW students might also benefit. Answering the first of these is in principle a simple data collection task, though a number of difficulties can arise. Answering the second is conceptually more difficult because the counterfactual information it calls for is simply not available, and must be artificially constructed. The two common comparison methodologies, randomized trials and observational analysis , differ mainly in the way they answer this question. The choice between these is a fundamental step in all evaluations, and directly affects factors such as credibility and cost.
Experimental studies rely on random selection of two subgroups of the target population.
One is labeled the treatment group and participates in the program, while the other is called the control group and is excluded from treatment. The control group's performance is an estimate of how well the treatment group would have done if they had not received the treatment. The crucial point about random selection is that, if the samples are sufficiently large, the average characteristics of the two groups will be very similar, except in terms of whether or not they received treatment. This implies that a simple comparison of the results for the two groups will yield a good measure of program effects. An observational study, in contrast, is generally used as a comparison methodology when the treatment group was not randomly selected.