Summary report on the methodological framework for counterfactual development
Creators: name: Artell, Janne
name: Aakkula, Jyrki
name: Toikkanen, Heini
Publisher: name: Zenodo
Year: 2015

description: The main issue in constructing the counterfactual is programme participants self-selecting to the programme. In such a case, a simple comparison between participants and non-participants gives a biased estimate of the environmental effects of the programme. The logic model for constructing a counterfactual in evaluating the environmental public good impact determines the possible routes of analysis with piece-by-piece evaluation of the available data. The logic model identifies methods for constructing a counterfactual, and explicitly forces the evaluator to assess the possibilities and weaknesses of the available data. There are three general level options for counterfactual construction: evaluation options without comparison groups, qualitative and naïve quantitative evaluation options with an ad-hoc approach to sample selection, and statistics-based evaluation options with an explicit approach to sample selection. A comparison of experiences from the ENVIEVAL public good evaluation case studies shows that the logic model serves the purpose of identifying issues and methodological options opened by existing data. Further, the experiences show that despite the general level guidance of the logic model, data-related issues, and in some cases problems in the methods themselves, can complicate evaluation. As problems are case specific, so are the solutions. However, this report serves to give examples of different cases in constructing a counterfactual. The report does find, however, that current data can be used for evaluation, and can be improved to allow rigorous evaluation procedures - Data availability, timing of monitoring data and its spatial resolution all need actions. Generally, the case studies using direct impact indicators (i.e. the actual environmental impact) had more often data availability related problems. Monitoring is expensive, leading to relatively small data repositories which may not cover non-participant farms or areas. Further, actual environmental impacts are often slow processes which may not exhibit during the evaluation period even with good data availability. In these cases modelling approaches, qualitative analysis or using proxy indicators (e.g. indirect pressure indicators) in statistical analysis may be preferable. The logic model shows promise in finding different approaches for constructing a counterfactual with varying data availability. As some case studies discovered, data quality as initially perceived did not always allow the intended method for constructing a counterfactual, but the logic model could in those cases be used to trace back other methodological options. Additionally the case<br> studies show that sometimes an ad-hoc approach to sample selection may help in improving the evaluation quality from simple comparisons. Ad-hoc approaches rely on the evaluator’s ability to understand the underlying selection process to measures. However, such approaches are not easily comparable over time and across measures warranting improved monitoring that specifically aims to help evaluation process. The structure of the methodological framework for counterfactual development will inform the development of the methodological handbook for the evaluation of environmental impacts of rural development programmes, which will pay particular attention to the practical problems encountered as well as the solutions applied in the environmental public good case studies.