<figure><img src="image_1.png" /></figure>

HAL
open science

# Impact of Decentralized Electrification Projects on Sustainable Development: A Meta-Analysis

Jean-Claude Berthélemy, Arnaud Millien

► To cite this version:

Jean-Claude Berthélemy, Arnaud Millien. Impact of Decentralized Electrification Projects on Sustainable Development: A Meta-Analysis. 2018. halshs-01965653

HAL Id: halshs-01965653

https://shs.hal.science/halshs-01965653

Submitted on 26 Dec 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
<figure><img src="image_2.png" /></figure>

Impact of Decentralized Electrification Projects on
Sustainable Development: A Meta-Analysis

Jean-Claude BERTHELEMY, Arnaud MILLIEN

2018.39

<figure><img src="image_3.png" /></figure>
# Impact of Decentralized Electrification Projects on Sustainable Development: A Meta-Analysis

Jean-Claude Berthelemy, Professor at Paris 1 University Pantheon-Sorbonne, Centre d'Economie de la Sorbonne (CES), Programme Director at FERDI - France.

Email: jean-claude.berthelemy@univ-paris1.fr

Arnaud Millien, PhD candidate at Paris 1 University Panthéon-Sorbonne, Centre d'Economie de la Sorbonne (CES), consultant at FERDI - France.

Email: arnaud.millien@univ-paris1.fr

19 december 2018.

## Abstract

This paper is the first product of a project which aims at building a Collaborative Smart Mapping of Mini-grid Action (CoSMMA), whose principal objective is to identify best practices of decentralized electrification projects.

Using evaluations of 421 projects, from published research papers, we built a pilot CoSMMA which proves its feasibility. Its relevance is demonstrated by a meta-analysis, which reveals the principal characteristics of decentralized electrification projects which have positive impacts on sustainable development.

Four main characteristics were considered: technology (source or energy), system size (power), decision level (from local to country level), geographic location. When searching for best practices, technology and system size must be considered together, because the chosen technology may constrain the power, which is provided by the system. We find that the most popular projects, which are based on Solar Home Systems (SHS) are not the most effective. The problem with SHS is not the use of solar energy, but the small system size often chosen for SHS. Mini-grids, of larger size, especially those which use hybrid renewable sources of energy, have more positive impacts, because these systems combine the benefits of sustainability and flexibility. In terms of decision level, we find that both top-down and bottom-up approaches have advantages, with the observation of a U-shaped curve for the influence of the decision level on the probability of obtaining positive impacts. Geographical location matters, as it is very often the key to system feasibility. We find that DEPs are more effective in Latin America than in Asia, and more effective in Asia than in Africa.

We also attempted to study the type of effects resulting from DEPs. Descriptive data suggest that for some types of effects, positive impacts are more likely than for others. Decentralized electrification projects have a more positive impact on Lifestyle & NICT or Household agenda than on Economic transformation or Community life. However, this pilot CoSMMA does not contain
enough information to study precisely the types of effects, because some types of effects have not been studied frequently in the existing literature. This is the case, for instance, for environmental effects, which have been rarely measured scientifically.

Finally, we attempted to broaden our information set by including expert data, which was entered into the CoSMMA meta-analysis. We define expert data as data that are not supported by statistical tests with measures of significance, whereas the evaluations based on scientific data were supported by statistical tests of significance. The expert data may be valid, but our attempt to include it in the analysis failed at this stage. The determinants of unproven effects appear to be quite different from the determinants of proven effects in our meta-analysis, and using expert data would imply merging proven and unproven effects, which would totally blur the conclusions.
## Acknowledgments

The authors thank Thomas André, Christophe Angely, Rémi Bazillier, Jean-Bernard Châtelain and Josselin Thuilliez for useful comments and discussions on this project; Fatou Cisse for her key support for data collection and data quality control; Olivier Santoni for his expertise dealing with geographical data. Both authors gratefully acknowledge support from Fondation pour les Etudes et Recherches sur le Développement International (Ferdi) and the French National Research Agency under program ANR-10-LABX-14-01.

JEL: L93, O13, O18, O22

**Keywords:** Decentralized electrification, sustainable development, impact assessment, meta-analysis.

## Table of contents

<table><tr><td>Abstract</td><td>1</td></tr><tr><td>Acknowledgments</td><td>3</td></tr><tr><td>Table of contents</td><td>3</td></tr><tr><td>List of Tables</td><td>5</td></tr><tr><td>List of figures</td><td>5</td></tr><tr><td>List of Annexes</td><td>5</td></tr><tr><td>1. Introduction: The CoSMMA project</td><td>6</td></tr><tr><td>2. Research question and literature review</td><td>7</td></tr><tr><td>2.1. Definition of DEPs</td><td>7</td></tr><tr><td>2.2. The need to identify best practice of DEPs</td><td>7</td></tr><tr><td>2.3. The potential contribution of a meta-analysis</td><td>8</td></tr><tr><td>2.4. Previous mapping of DEP effects</td><td>8</td></tr><tr><td>2.5. Previous meta-analyses</td><td>10</td></tr><tr><td>3. Sources of data</td><td>13</td></tr><tr><td>3.1. Methods used to build the CoSMMA</td><td>13</td></tr><tr><td>3.2. Descriptive statistics</td><td>17</td></tr><tr><td>4. Methodology for a meta-analysis of DEP impacts</td><td>18</td></tr><tr><td>4.1. Objects of the meta-analysis</td><td>18</td></tr><tr><td>4.2. Source of heterogeneity across control variables</td><td>19</td></tr><tr><td>4.3. Specification and estimation strategy</td><td>20</td></tr><tr><td>4.4. Excluding outliers</td><td>21</td></tr></table>
5. Which characteristics of electricity projects yield positive impacts on sustainable development?

22

5.1. Role of source of energy.................................................................................................................. 22

5.2. Role of system size................................................................................................................................. 24

5.3. Role of project decision level.................................................................................................................. 25

5.4. Role of location ................................................................................................................................. 25

5.5. Significant pairs of size and technology .......................................................................................... 26

5.6. Factors of success by type of effects .............................................................................................. 28

5.7. Extending knowledge of effects with expert data ................................................................................ 31

6. Discussion ................................................................................................................................................................. 33

6.1. Possible selection bias.............................................................................................................................. 33

6.2. Cycle between funding and evaluation.............................................................................................. 34

7. Concluding remarks.................................................................................................................................................. 34

References ........................................................................................................................................................................ 36

ANNEXES........................................................................................................................................................................ 38
## List of Tables

<table><tr><td>Table 1 - Previous studies addressing a mapping of the socio-economic effects of electricity</td><td>10</td></tr><tr><td>Table 2 - Review of literature or meta-analysis about electricity economics (as of Oct 12<sup>th</sup> 2018)</td><td>12</td></tr><tr><td>Table 3 - Listed studies in electricity economics (as of 12<sup>th</sup> Oct 2018)</td><td>13</td></tr><tr><td>Table 4 - Project inclusion criteria</td><td>15</td></tr><tr><td>Table 5 - Scientific vs. Expert data</td><td>19</td></tr><tr><td>Table 6 - Effectiveness characteristics of DEPs - Average Marginal Effects (AME)</td><td>23</td></tr><tr><td>Table 7 - AME of impacts - Highest Significant Pairwise Comparisons</td><td>27</td></tr><tr><td>Table 8 - Distribution of effects by type (scientific data)</td><td>28</td></tr><tr><td>Table 9 - Distribution of impacts by effect type (scientific data)</td><td>28</td></tr><tr><td>Table 10 - AME - Impacts - scientific data - separated regressions by subset of Effect Type</td><td>29</td></tr><tr><td>Table 11 - AME- with expert data (N &gt;= 0)</td><td>32</td></tr></table>

## List of figures

<table><tr><td>Figure 1 - Number of papers by publication period (4 years)</td><td>14</td></tr><tr><td>Figure 2 Map of DEPs registered in CoSMMA</td><td>17</td></tr></table>

## List of Annexes

<table><tr><td>Annex 1 - Methods used by research to evaluate DEP effects</td><td>38</td></tr><tr><td>Annex 2 Distribution of scientific data</td><td>39</td></tr><tr><td>Annex 3 - Distribution of effects by direction and significance of effects</td><td>40</td></tr><tr><td>Annex 4 - Distribution of subsets of positive impacts</td><td>41</td></tr><tr><td>Annex 5 - Distribution of effects by project objective</td><td>42</td></tr></table>
# 1. Introduction: The CoSMMA project

Decentralized Electrification Projects (DEPs) are booming in developing countries as a response to the deficiencies of on-grid electrification in many parts of the developing world, particularly in rural areas, and due to the new possibilities offered by technological progress in renewable sources of energy. This process potentially has a lot of promise for sustainable development in developing countries, but it may be curbed by a lack of visibility of what works and what does not work, which may in turn become an obstacle to financing such projects.

This paper is part of a project to build a collaborative database on decentralized electrification, named CoSMMA (Collaborative Smart Mapping of Mini-grid Action), with the objective of identifying best practice from the point of view of sustainable development (i.e. to identify the project characteristics that maximize positive impacts on sustainable development).

To this end, DEPs are described in the CoSMMA in several dimensions:

* Basic technical characteristics of the project, such as the energy source and system size (electrical power)

* Project goal and expected impact

* Ex-post evaluated effects

Additional types of information include conditions of evaluation and document sources.

At the current stage of our project, we have built a pilot CoSMMA, with the objective of testing its feasibility and proving its relevance. We used information on 421 DEPs available in published research papers, which we analyzed and coded into variables describing the projects and their effects. This information was gathered by a structured search from 4 principal academic sources - Academic Search Premier, Business Source Complete, EconLit, and GreenFILE. The information was then processed through a meta-analysis regression, whose results shape the core of this paper.

Our principal tool is a multi-probit meta-regression, which shows which factors led to which effects of DEPs on sustainable development. We also attempt to break down the analysis by type of effect, but at this stage the available information limits our analysis. Only a few types of effect of DEPs have been sufficiently explored in the papers registered in the CoSMMA to allow uncovering their specific determinants. Finally, we attempted to enlarge the data base used in the meta-regression by including so-called expert data (i.e. evaluations provided by experts but not supported by statistical tests of validity). However, we were unsuccessful in this attempt, because statistically proven effects and unproven effects appear to have quite different determinants.

In section 2, we develop in more detail our research question and relate it to the existing literature. In section 3 we document our sources of data and methods used to build the CoSMMA and we report descriptive statistics on project characteristics and project effects registered in the CoSMMA.
In section 4, we describe the econometric methodology used to perform our meta-analysis. In section 5, we discuss our empirical results and their possible extensions. Section 6 is devoted to a discussion of the possible bias that could affect our results, section 7 concludes and proposes some possible areas for further research.

## 2. Research question and literature review

### 2.1. Definition of DEPs

Defining decentralized electrification is not simple, because many field practitioners and scientists refer to decentralized project as an obvious notion, although to the best of our knowledge, no clear criteria has been established so far.

CoSMMA is limited to off-grid or individual solutions, with no connection to the national grid. Our definition also includes a size limitation: any project above 100 MW is not considered as decentralized, because it could be involved in clearing price exchanges (Dillig et al, 2016).

### 2.2. The need to identify best practice of DEPs

A variety of DEP projects have been implemented and evaluated so far, with a focus on solar nano solutions, the so-called SHS (Solar Home Systems). This focus comes from a convergence of interest between funders and developers, as they offer a low-commitment solution for the funder, and a low-cost market test for the developer. This focus does not imply that SHS represent the best practice in terms of positive impact on sustainable development. Institutions working in the sector frequently face the reality of economic or technical failures (Ikejemba et al, 2017). Failures are also repeatedly reported by NGOs promoting DEPs, with estimates of failure rates of project commonly being above one third. Clarifying the question of DEP performance and identifying best practice is thus important.

The development of DEPs faces three major challenges:

* Because projects are not connected to the grid, they show a large heterogeneity of economic and technical design

* So far, no unified framework of knowledge and data on DEPs can offer a complete vision on the variety of field experiments, and qualify their ability to yield sustainable favorable impacts

* There is no clear consensus on the types of effects that matter and the primary types of impacts that a DEP should address first

In this study, "best practice" is defined as the project characteristics that produce significant favorable effects on sustainable development. Significant favorable effects are also called "positive impacts" in this study.
## 2.3. The potential contribution of a meta-analysis

Identifying best practice in DEPs requires an innovative methodology, because the focus on energy in Sustainable Development Goals (SDGs) is recent. Few DEPs have been assessed in a rigorous evaluation framework, although many observations of DEP effects are available in other areas of research. Using these observations is complex given their heterogeneity, however it can provide an approach to delivering an early assessment of DEP strategic choices.

A meta-analysis adds to the understanding of a phenomenon by combining results obtained by researchers using a variety of data and methods (Stanley, 2001). In conducting a meta-analysis with published results for DEP effects, we expect, like Carré et al (2015), to have more robust conclusions than a mere review of separate regressions. Using a systematic selection from research databases, a meta-analysis avoids the classic pitfalls of a literature review, which could be unbalanced due to selection bias, or reflect the beliefs of authors who might tend to reject papers that run against their convictions (Stanley, 2001).

To the best of our knowledge, this study is the first meta-analysis which attempts to relate DEP characteristics to their impact on sustainable development, and hence which addresses clearly the question of best practice in decentralized electrification.

In order to base our contribution on previous literature, we review below two branches of research: first we consider what has been proposed so far in terms of mapping DEP effects. Second, we analyze the methodological references for meta-analysis.

## 2.4. Previous mapping of DEP effects

To the best of our knowledge, there is no previous study which proposes a complete mapping of DEP effects in developing countries. Several studies have been done with more specific research questions, as shown in Table 1. The CoSMMA offers an original contribution, mapping a wide scope of DEP effects in developing countries with observed data.

Special attention must be paid to SE4ALL (2017) and Katre and Tozzi (2018).

SE4ALL (2017), *Why wait?*, was the first study to assess the effects of access to electricity appliances in developing countries on SDGs, using the multi-tier framework defined by ESMAP (2015). There was a similarity with our objectives, although CoSMMA covers more countries (69) and indicators (765). Furthermore, by considering DEP effects published by researchers, we evaluate DEP impacts without any preconceived assumption about which impact should be evaluated first or might be expected to arise initially. Effects in the CoSMMA were collected for 157 dimensions, grouped in 15 topics and 8 types.

Katre and Tozzi, (2018) propose a complete comprehensive scorecard for DEP evaluation, but do not use field data. Using observed or reported effects of DEPs, we are able to feed a large
comprehensive framework with effects of various dimensions, and gather observational or
experimental data in a common database.<fnref n="1" />

<footnote n="1">¹ Some data are experimental, yielded by evaluations of DEPs in a natural experiment (Randomized Control Trial) or in quasi-natural experiment conditions (DiD).</footnote>
<table><caption>Table 1 - Previous studies addressing a mapping of the socio-economic effects of electricity</caption><thead><tr><td>Reference</td><td>Converging feature</td><td>Differentiating feature</td></tr></thead><tbody><tr><td>Kanagawa and Nakata (2008)</td><td>Socio-economic impact of access</td><td>Macro study, no project</td></tr><tr><td>Hayn et al (2014)</td><td>Socio-demographic factors</td><td>In Europe</td></tr><tr><td>Bell et al (2015)</td><td>Electricity effect on sociability</td><td>131 customers in United Kingdom</td></tr><tr><td>Marszal-Pomianowska et al (2016)</td><td>35 electrical appliances</td><td>In Denmark. Looking at the impact of appliances on the system, not on socio-economic household behaviors</td></tr><tr><td>Thopil and Pouris (2015)</td><td>Externalities on environment, health and employment, in South Africa</td><td>1 country, 3 types of effects, 9 indicators</td></tr><tr><td>Holtorf et al (2015)</td><td>Consider success criteria of SHS</td><td>Technology constrained (SHS only) No data (a comprehensive framework)</td></tr><tr><td>SE4ALL (2017)</td><td>Quantify the access dividends according to the multi-tier framework of appliances. Relate tiers of appliances and research on effects</td><td>3 countries (Bangladesh, Ethiopia, Kenya), 21 indicators</td></tr><tr><td>Katre and Tozzi, (2018)</td><td>Build a scorecard relating tiers of appliances with dimensions of yielded effects.</td><td>No field data. Calibration made with data from individual assessment by other researchers.</td></tr></tbody></table>

## 2.5. Previous meta-analyses

Frequently used in medical studies, meta-analyses were popularized in social science (Carré et al (., 2015), and were widely used as a quantitative method of research synthesis to calibrate structural models, examine patterns of publication bias, and explain differences in the results of individual studies (Frequently used in medical studies, meta-analyses were popularized in social science (Carré et al, 2015), and were widely used as a quantitative method of research synthesis to calibrate structural models, examine patterns of publication bias, and explain differences in the results of individual studies (http://meta-analysis.cz/).

In a seminal work, Stanley (2001) provides clear and comprehensive advice on the steps to follow and pitfalls to avoid, when conducting a meta-analysis that "employs conventional statistical methods and criteria to summarize and evaluate empirical economics". We follow this methodology, especially in the important step of defining the objective of the meta-analysis.

Doucouliagos and Paldam (2009) conducted a referral meta-analysis for development economics, in which they assessed the publication bias in aid effectiveness evaluation. More details on the methodology and meta-equations. They used 97 research papers on aid effectiveness, from 4 databases. Their main research questions were to determine whether aid increases accumulation in the recipient country, and if so, by how much? The spirit of our research question is similar to this approach because we are examining whether the theoretical favorable effects of DEP on sustainable development have been proven by the literature.
However, classical meta-analyses like these, address only one parameter of interest at a time (aid effectiveness in Doucouliagos and Paldam (2009), Ricardian equivalence in Stanley (2001)), and usually a continuous parameter. We propose an original extension to these classic approaches, by testing simultaneously a relatively large number of parameters, related to the contribution of project characteristics

To clarify to what extent our study fills a gap, we investigated 4 sources specialized in conducting international meta-analysis, a website http://meta-analysis.cz/ and reviews of Journal of Economic literature, Journal of economic perspective, Journal of economic surveys.

As shown in Table 2 there is no meta-analysis about access to electricity ("electrification"). Our research shows that our paper is the first meta-analysis on electrification effects.

We found 12 meta-analyses about "electricity", which proves the growing importance of the electricity economics field, as each meta-analysis is based on a populated set of underlying studies. Those studies address topics so different from CoSMMA, that we can hardly use them as reference, but we can highlight 2 findings:

* Meta-analyses about electricity economics are feasible

* CoSMMA fills a gap in off-grid electrification assessment.

It is worth noting that 6 of the existing meta-analyses are about USA electricity economics, 2 about developing countries, and 1 about renewable electricity.

Several literature reviews about energy economics were also investigated. Table 3 shows the number of articles reviewed. These numbers are small compared to the number of papers used in the CoSMMA (123).
<table><caption>Table 2 - Review of literature or meta-analysis about electricity economics (as of Oct 12th 2018)</caption><thead><tr><th>Review</th><th>Key words</th><th>Response/reference</th><th>Title</th><th>Qualification with respect to CoSMMA objectives</th></tr></thead><tbody><tr><td>Journal of Economic Literature</td><td>"electrification"</td><td>0</td><td></td><td></td></tr><tr><td>Journal of Economic Perspectives</td><td>"electrification"</td><td>0</td><td></td><td></td></tr><tr><td>Journal of Economic Surveys</td><td>"electrification"</td><td>2. Of which, responses to consider: 0</td><td></td><td>Off-topic: railroad electrification and cliometrics</td></tr><tr><td>http://meta-analysis.cz/</td><td></td><td>Havranek et al (2018)</td><td>Does Daylight Saving Save Electricity? A Meta-Analysis</td><td>Off-topic and reverse causality: the authors study the impact of daylight saving time on electricity consumption (44 studies)</td></tr><tr><td>Journal of Economic Literature</td><td>"electricity"</td><td>(Zheng and Kahn (2013)<br/>Zheng and Kahn (2013)</td><td>Understanding China's Urban Pollution Dynamics</td><td>Off-topic: on-grid analysis and only one dimension studied.<br/>Underlying studies unclear (an assembly of datasets).<br/>Authors study the impact of electricity consumption on environmental externalities, notably air quality</td></tr><tr><td>Journal of Economic Perspectives</td><td>"electricity"</td><td>8</td><td></td><td></td></tr><tr><td></td><td></td><td>Joskow (2003)<br/>Davis (2012)</td><td>Creating a Smarter U.S. Electricity Grid<br/>Prospects for Nuclear Power</td><td>Off-topic: US<br/>Off-topic: US, Nuclear Power</td></tr><tr><td></td><td></td><td>Borenstein (2012)</td><td>The Private and Public Economics of Renewable Electricity Generation</td><td>The author aims to evaluate the pricing of (positive) externalities from renewable electricity generation.<br/>Off-topic: Discussion in the literature.</td></tr><tr><td></td><td></td><td>Wolfram et al (2012)<br/>Borenstein (2002)</td><td>How Will Energy Demand Develop in the Developing World?<br/>The Trouble With Electricity Markets: Understanding California's Restructuring Disaster</td><td>Off-topic: Impact of growing energy demand on the grid<br/>Off-topic: US</td></tr><tr><td></td><td></td><td>Bazelon and Smetters (1999)<br/>Winston (1998)</td><td>Discounting Inside the Washington D.C. Beltway</td><td>Off-topic: US</td></tr><tr><td></td><td></td><td>Joskow (1997)</td><td>U.S. Industry Adjustment to Economic Deregulation<br/>Restructuring, Competition and Regulatory Reform in the U.S. Electricity Sector</td><td>Off-topic: US</td></tr><tr><td>Journal of Economic Surveys</td><td>"meta-analysis electricity"</td><td>18. Of which, responses to consider: 2</td><td>Note: retrieved papers strongly orthogonal to our research (i.e. not in electricity economics field) are not shown.</td><td></td></tr><tr><td></td><td></td><td>Stern (2012)</td><td>Interfuel Substitution: A Meta-Analysis</td><td>Off-topic: underlying studies are macro-economics<br/>The author studies inter-fuel substitutability (47 studies)</td></tr><tr><td></td><td></td><td>Heshmati (2014)</td><td>Demand, Customer Base-Line and Demand Response in the Electricity Market: A Survey</td><td>The authors study models used in the literature to evaluate the demand for electricity (and its impact on reliability)</td></tr></tbody></table>
<table><caption>Table 3 – Listed studies in electricity economics (as of 12th Oct 2018)</caption><thead><tr><td>Reference</td><td>Number of listed studies</td></tr></thead><tbody><tr><td>Praktiknjo et al (2011)</td><td>16</td></tr><tr><td>Jamasb et al (2017)</td><td>18</td></tr><tr><td>Bonan et al (2014)</td><td>20</td></tr><tr><td>Brenneman and Kerf (2002)</td><td>40<sup>2</sup></td></tr><tr><td>Peters and Sievert (2015)</td><td>9</td></tr><tr><td>Thopil and Pouris (2015)</td><td>15</td></tr><tr><td>(van Gevelt (, 2014)</td><td>90</td></tr></tbody></table>

# 3. Sources of data

## 3.1. Methods used to build the CoSMMA

Research papers used to document DEPs in the CoSMMA were taken from 4 economic research academic databases: Academic Search Premier, Business Source Complete, EconLit, GreenFILE.

The studies on the impact of electrification show a wide scope of methodologies, data, and projects. Following Stanley (2001), "differences in quality, data or methods do not provide a valid justification for omitting studies. Rather, such difference provide the underlying rationale for doing a meta-regression analysis in the first place."

A systematic collection of research papers was made, with no *ex-ante* exclusion, but the topic relevance (Stanley, 2001) : "after reducing the sample of studies to those that contain some relevant empirical estimate, test or finding"). Off-topic studies (e.g. electrification of railways), macroeconomic studies, studies focused only on potential and barriers, *ex-ante* cost/benefit analyses, or technical feasibility studies) were not used for the CoSMMA. Papers with a developed country in the title were excluded. Only papers with a publication date later than 1980 were selected. This time span was set to avoid missing any important precursor publications about decentralized electrification. However, because the growing interest in decentralized electrification is recent, papers before 1990 are scarce (see Figure 1).

Publication conditions were also checked. Documents had to use a common language (English) and be peer-reviewed, or designed for such a process (e.g. working papers of research institutions). A few economic reports (from financing institutions or companies) were included because they had been through a quality control process before public dissemination. They represent 7% of the current primary sources of the CoSMMA.

<footnote n="2">² Only papers about the impacts of energy are counted. Papers about the impact on growth are not counted.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 1 - Number of papers by publication period (4 years)</figcaption></figure>

A key sentence containing words usually used to analyze decentralized electrification projects was defined and parsed through EBSCO for the 4 databases. Keywords were automatically reweighted by a smart text mining function in EBSCO. Some variants were also used. Finally 6 main queries were defined which gave 6 sets of documents, called "packs". For the most complex queries, a common set of additional keywords was used in order to limit the study more closely to decentralized electrification projects.

Reweighted queries were saved to keep track of the search, allowing for possible external replication.

Being keyword-based and systematic, this methodical sampling aims to define a neutral collection of papers, which is not influenced by the researcher's knowledge or a specific direction of research. The keyword-based sampling approach warrants providing a random selection of papers related to the DEP effectiveness field of research. However, the ability of an algorithm to fit accurately to a field of research cannot be guaranteed, and so *ex-post* human checks were performed on the EBSCO selection results. Possible duplicates were eliminated, and a final check of the application of *ex-ante* selection criteria was made (e.g. residual macroeconomic studies or other off-topic papers were eliminated).

Within each of the 6 packs, keywords defined specific branches. Inside each branch, some articles with large bibliography were used to define sub-branches, in which some of the papers quoted in the bibliography of the head article were collected as well. However, the bibliographies of initial articles were used with parsimony, because too many papers from sub-branches could have introduced a
bias toward the past into the meta-analysis, and also a direction bias: at a given point in time, a researcher can only cite previously published papers, and papers strongly related to his or her own research direction.

For reasons related to the research project's origin, 32 articles were used in addition, following a classic approach based on research about the econometric evaluation of decentralized electrification. These articles did not duplicate the EBSCO extraction. They constitute an additional pack in the meta-base. Additional papers (from sub-branches or historical pack) are 18% of all collected papers.

After this selection process, the dataset ready for statistical analysis contains 421 projects, evaluated in 123 papers, with a total of 2605 effects.

The inclusion criteria applied to project characteristics are presented in Table 4 below. Note that the number of exclusions results from the simultaneous application of criteria, and the number of exclusions is thus not the total number resulting from each individual criterion.

Figure 2 shows the geographical distribution of DEPs registered in the CoSMMA, showing that CoSMMA is based on a wide variety of experiences.

<table><caption>Table 4 - Project inclusion criteria</caption><thead><tr><th>Projects in sample must -</th><th>Initial number of observations - 2818 effects</th><th>Number of excluded observations</th></tr></thead><tbody><tr><td>be operative (or eventually have been operative)</td><td>Projects must have a commission date before 2018. Defaulted projects are accepted.</td><td>105</td></tr><tr><td>not be in OECD and must have an understandable continental location</td><td>Effects from projects in OECD countries or with unclear continental location ("worldwide" studies) are excluded</td><td>15</td></tr><tr><td>deliver capacity below 100 MW</td><td>Application of Dillig et al (2016) criteria: projects with capacity below 100 MW are not involved in balancing, nor in market exchanges nor in clearing</td><td>3</td></tr><tr><td>use a clear specified technology</td><td>Effects from studies based on existing energy mix are excluded</td><td>85</td></tr><tr><td>be deployed in rural area</td><td>Effects from projects in urban area are excluded</td><td>1</td></tr><tr><td>be evaluated with samples that do not show unusually high extreme values</td><td>Observations above 5 times the usual threshold for Dfbeta were excluded</td><td>2</td></tr><tr><td>be evaluated with samples of normal size</td><td>Samples with observation number larger than the 99% quantile of this number were excluded. The threshold was 352,800 observations</td><td>2</td></tr><tr><td></td><td>Number in large sample - 2,605 observations</td><td>Total number of deleted observations (effects) - 213</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Figure 2 Map of DEPs registered in CoSMMA</figcaption></figure>

## 3.2. Descriptive statistics

The CoSMMA covers a variety of evaluations of DEPs, from well-identified econometric estimations to mere descriptive observations.

**Annex 1** lists the various effects by quality of estimation. About two-thirds of the reported effects were submitted to statistical tests by their authors. We call scientific data the data associated with tested effects. We call the remaining data expert data. For reasons explained later, our conclusions from the meta-analysis are based only on scientific data. For this reason, we restrict at this stage our description of CoSMMA data to the sub-sample of scientific data.

**Annex 2** describes the distribution of this sub-sample by technology, system size, decision level, and continent. Solar systems account for the vast majority (over 70%) of the DEPs in this subsample; other sources are mainly renewables, and there is a small number of DEPs which use fuel.

The system size of DEPs can vary from less than 1 kW to 100 MW. Our classification by size is based on the following definitions:

* Nano: < 1 kW

* Micro: 1 to 100 kW

* Mini: 100 kW to 100 MW

More than 70% are nano systems, and these nano systems are mostly solar based: the most frequent type is Solar Home System (SHS).
The most frequent decision levels are at country level or at local (municipality) level. This corresponds to two vastly different approaches: top down or bottom up.

As for geographical distribution, approx. 50% of projects are in Africa, approx. 40% are in Asia, and 10% are in Latin America.

**Annex 3** describes the distribution of effects by direction. Some effects are qualified as favorable to sustainable development (e.g. they increase energy availability, develop income generating activities, save time for households, improve health or education, reduce environmental damage). Some are qualified as unfavorable. About 2/3 are favorable and 1/3 are unfavorable (with a small proportion of inconclusive studies). This ratio of 2 to 1 corresponds to anecdotal evidence reported by NGOs on their success rates with DEPs. However, the most striking observation is that whatever the direction of the effects, a majority of effects are unproven: the usual statistical tests show that the probability that reported evidence is false is above 5%. Only about 20% of reported effects are proven, this proportion is a little higher for favorable effects than for unfavorable effects. As a consequence, out of the sample of 1,486 effects measured with scientific data, only 225 are proven favorable effects, which we call henceforth "positive impacts", and 81 are proven unfavorable effects, which we call "negative impacts".

**Annex 4** shows the distribution of characteristics associated with positive impacts. Comparing this distribution with the distribution of observed effects in Annex 2 gives a primary assessment of where the best practices are. We observe that there are, relatively speaking, more positive impacts in projects using hybrid technology with renewables and less positive impacts in projects using fuel or hybrid technology with fuel. There are also a few more solar-based projects with positive impact than in the full sample. We observe also that there are more positive impacts in projects based on nano size systems. The proportion of positive impacts is the highest at provincial decision level. Finally, there are more positive impacts in Africa than in Asia. These descriptive conclusions may however be misleading, because the different characteristics considered one by one are actually correlated. For instance, most solar systems are SHS, of nano size. We will show in section 5 that the predominance of hybrid and renewable technologies is confirmed by a multivariate analysis, but that nano size systems are not best practice, and that the provincial decision level is not as effective as the country decision level. As for geographical considerations, the multivariate analysis will not confirm the better performance of Africa than Asia suggested by simple descriptive statistics.

# 4. Methodology for a meta-analysis of DEP impacts

## 4.1. Objects of the meta-analysis

Because our meta-analysis does not use directly data from the field, it is important to define what are the objects of the analysis. As pointed out by Glass (1977): "*the design of a study is a complex*
judgmental process that produces as many different studies as there are researchers and settings in which they work".

The objects of this meta-analysis are the effects of DEPs observed from previous published evaluation studies, which used experimental or observational data. An evaluation study of electrification project is a document that:

* Describes the characteristics of the project

* Describes the general purpose of the project

* Documents or measures the effects of the project

## 4.2. Source of heterogeneity across control variables

A meta-study aims to exploit the variance along a common dimension across a set of various studies; but because each research is unique, it seems paradoxical to pretend to identify a common dimension from all the features that make every study unique. As noted by Stanley (2001), "because [...] most studies entail a unique combination of techniques, independent variables, data, time periods and other research choices, not every study characteristic can be coded and analyzed. Nor should a researcher wish to do so. Variation due to minor modeling choices may be treated as part of the random study-to-study background."

To achieve this separation between genuine sources of heterogeneity and heterogeneity that arises from modeling choices, we attempted to establish a clear distinction between a measured phenomenon (effects of a DEP), and the measurements performed to capture this phenomenon (estimation methods), or measurement conditions (Number of observations).

Some meta-analyses capture the number of observations, others capture the T-statistic (Doucouliagos and Paldam, 2006b) or standard error (Havranek et al, 2015), some even include the date of collection, which gives a panel of studies (Havranek et al, 2018).

First, in this study, we controlled for the number of observations (N) when it was available. A clear distinction is made between using scientific data (N>1) and studies using expert data (N ≤ 1), as illustrated in Table 5.

<table><caption>Table 5 - Scientific vs. Expert data</caption><thead><tr><td>Denomination</td><td>Type</td><td>Number of obs. (N)</td></tr></thead><tbody><tr><td>Scientific data</td><td>Quantified effect with variance</td><td>n > 1</td></tr><tr><td>Expert data</td><td>Quantified effect without variance</td><td>n=1</td></tr><tr><td>Expert data</td><td>Documented effect from Research</td><td>n=0</td></tr><tr><td>Expert data</td><td>Unmeasured effect</td><td>n=0</td></tr></tbody></table>
Second, we controlled for the methodology of evaluation. A large variety of research methodologies has been used by researchers to gather evidence of DEP effects (as reported in Annex 1), from the least sophisticated ones (citing others' results) to the most advanced ones (robust econometric evaluations which permit statistical inference).

The use of expert data merits a specific discussion. As shown by the dataset used in Havranek et al (2018), the absence of statistical tests in some studies, though infrequent, is not necessarily an obstacle to conducting a rigorous meta-regression, although it might be expected that introducing too high a proportion of studies without significance testing could weaken the ability to arrive at conclusions. Carré et al (2015) also conducted a meta-regression using data without variance of the estimates, which confirmed that the methodology is feasible with expert data. They introduced a dummy for the quality level of observations.

In this study, we initially conducted a baseline meta-regression using scientific data only, reproducing the classic framework of meta-analysis. Then we introduced expert data in an attempt to enlarge the estimation sample, using the pool of expert knowledge on DEPs. However, as discussed later, this attempt to include expert data was inconclusive, because we observed large differences in best practice revealed by regressions to separate proven and unproven effects, and by regressions mixing them.

Third, we controlled for the time lag between the year of implementation of the system and the year of publication of its evaluation. This time lag may reflect the short term vs. long term nature of effects, but may also reflect other factors such as the difficulty to collect data which relies on the memory of survey respondents.

### 4.3. Specification and estimation strategy

As noted by Stanley (2001), "*the independent variables -often called "moderator variables"- are those study characteristics that are thought to be consequential*". In this meta-analysis, project specifications are expected to be the essential channel of DEP impact. First, this is in line with our objective of exploring best practice in DEPs. Second, DEPs show highly different characteristics, because in decentralized electricity there is no grid playing a structuring role in terms of balancing, interconnection standards, or demand response. The heterogeneity of DEP characteristics is higher across projects chosen for research evaluation, because evaluated projects are often the most innovative ones, either in terms of technological features or in terms of socio-economic environment and organizational features.

The outcome variable is the direction of the effect of a DEP, which is a categorical variable; we aim to explore the determinants of the probability of observing a given direction of effect. Basically, we could consider a dichotomous outcome, i.e. whether evaluated DEPs had favorable effects on sustainable development or not. However, given the large number of studies reporting unproven directions of the effects (Annex 3), our set of information would be too fuzzy in the absence of a
distinction between proven and unproven conclusions. In an attempt to avoid this shortcoming, we considered 5 distinct categories of effect directions, as reported in Annex 3, in which proven and unproven effects are separated.

The baseline estimation links a set of project characteristics and of methodological control of evaluations to the probability of achieving a given outcome. The possible outcomes are defined by the combination of the direction of the effects (favorable or unfavorable) and their nature (proven or unproven). The parameters are estimated with a multi-probit estimator, which yields simultaneously all equations, one for each direction and nature of outcome.

$$P(outcome_{ip} = k) = \text{constant} + c.EvalCond_{ip} + s.ProjectSpec_{p} + \text{error-term}_{ip}$$

Where:

* p is a project

* i is an observed or reported effect

* outcome = k is one of 5 possible outcomes

* EvalCond<sub>ip</sub> is a vector of control variables of an evaluation's conditions, as developed in the next section;

* ProjectSpec<sub>p</sub> is a vector of a project's specifications

This equation provides an assessment of best practice by evaluating s, a vector of parameters which describes the influence of project specifications (ProjectSpec) on the probability of obtaining a positive impact, after controlling for conditions of evaluation in underlying studies (EvalCond). Although our interest is focused on the positive impact outcome, estimating the full set of parameters associated with all 5 outcomes in a multi-probit regression provides a way to enrich our estimation, because estimating the determinant of other outcomes conveys indirectly information about the DEP characteristics that limit their ability to have positive impacts.

Most of the variables in the vector ProjectSpec are categorical. We consider 4 different types of characteristics:

* Technology: the sources of energy and technics used to produce electricity

* System size: electric power available for users connected to the system

* Organization: decision level - from local to country or multi-country, at which the DEP was conceived

* Geographical location.

The variance estimator uses clusters by project objectives, whose classification is described in Annex 5.

### 4.4. Excluding outliers
In the same way as controls were introduced *ex-ante* in the estimation strategy, some essential robustness checks were performed before the estimation, to exclude the worst cases that could have spoiled the estimation. These checks correspond to the bottom 2 criteria in Table 4.

The most atypical observations were excluded from the estimation sample, because they could have a too high influence on the meta-estimation, and hide other more frequent relationships. Atypical data appear from uncheckable errors during the collection process, or due to abnormal observations at the extreme end of the variables' distributions.

The outliers were filtered based on a Dfbeta estimation and a quantile-based filter. Outliers above a limit for Dfbeta were excluded: from a simple LPM using all candidate covariates, Dfbeta were computed for all observations. Observations with a Dfbeta more than 5 times the usual threshold (5 x 2/√N) were dropped. Observations above the 99% quantile of their distribution were also dropped.

**5. Which characteristics of electricity projects yield positive impacts on sustainable development?**

**Table 6** shows the role of the characteristics in project effectiveness, after controlling for conditions of evaluation. This table presents average marginal effects (AME) of the probability of generating the different outcomes. As our variables of interest are categorical, estimated AMEs represent the difference between the probability that a given category generates the outcome and the probability associated with a reference category, which is denoted as "ref. =".

Below, we discuss the role of source of energy, system size (power), decision level, and location

**5.1. Role of source of energy**

Many different sources of energy can be utilized in DEPs. The different sources of energy have different unit costs, intermittence, reliability, or maintenance requirements. At this stage we can only measure the average performance of the different sources, and we cannot compare them because in practice performance is conditioned by many other factors such as geography.

We chose as reference hydroelectric power projects, which were historically among the first DEPs based on renewable energy deployed in developing countries.

Solar power based on photovoltaic panels is by far the most popular technology for DEPs. Solar projects have +12.3 percentage points (pp) higher chance of generating positive impacts than hydroelectric power projects (Table 6 col 1). It is also interesting to note that the risk that solar-based DEPs create negative impacts is significantly lower (-10.7 pp) than for hydroelectric power projects.
<table><caption>Table 6 – Effectiveness characteristics of DEPs - Average Marginal Effects (AME)</caption><thead><tr><th></th><th>Positive</th><th>Unproven favorable</th><th>Proven unfavorable</th><th>Negative</th><th>Inconclusive</th></tr></thead><tbody><tr><td>No. of Observations (N)</td><td>-0.000</td><td>0.000<sup>**</sup></td><td>-0.000<sup>***</sup></td><td>0.000<sup>*</sup></td><td>-0.000<sup>***</sup></td></tr><tr><td>Delay of evaluation</td><td>-0.010<sup>***</sup></td><td>-0.013</td><td>0.021<sup>***</sup></td><td>0.001</td><td>0.002<sup>*</sup></td></tr><tr><td>Method (ref. = Econometrics without inference)</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Identification</td><td>0.335<sup>***</sup></td><td>-0.242<sup>*</sup></td><td>-0.012</td><td>-0.040</td><td>-0.041</td></tr><tr><td>Econometrics without inference</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>No inference</td><td>-0.158<sup>***</sup></td><td>0.307</td><td>-0.128<sup>***</sup></td><td>0.039</td><td>-0.060</td></tr><tr><td>Technology: (ref. = Hydroelectric)</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Hydroelectric power source</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Solar</td><td>0.123<sup>***</sup></td><td>-0.107<sup>*</sup></td><td>0.027</td><td>0.027</td><td>-0.070<sup>***</sup></td></tr><tr><td>Hybrid with Fossil fuel</td><td>0.320<sup>***</sup></td><td>-0.176<sup>*</sup></td><td>0.017</td><td>-0.119<sup>***</sup></td><td>-0.042</td></tr><tr><td>Hybrid renewables</td><td>0.418<sup>***</sup></td><td>-0.177<sup>*</sup></td><td>-0.015</td><td>-0.157<sup>***</sup></td><td>-0.069<sup>***</sup></td></tr><tr><td>Biomass (and related tech.)</td><td>0.212<sup>***</sup></td><td>0.035<sup>*</sup></td><td>-0.029</td><td>-0.133<sup>***</sup></td><td>-0.086<sup>***</sup></td></tr><tr><td>Fossil Fuels</td><td>0.238<sup>***</sup></td><td>-0.352</td><td>-0.040</td><td>0.241</td><td>-0.086<sup>***</sup></td></tr><tr><td>Power : (ref. = Nano)</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Nano: &lt;1 kW</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Micro: 1 to 100 kW</td><td>0.365<sup>***</sup></td><td>-0.188<sup>**</sup></td><td>-0.051<sup>***</sup></td><td>-0.102</td><td>-0.025<sup>***</sup></td></tr><tr><td>Mini: 100 kW to 100 MW</td><td>0.319<sup>***</sup></td><td>-0.162<sup>***</sup></td><td>-0.059<sup>***</sup></td><td>-0.086</td><td>-0.012<sup>***</sup></td></tr><tr><td>Programme Decision Level (ref. = Local)</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Country</td><td>0.086<sup>***</sup></td><td>-0.099<sup>***</sup></td><td>-0.008<sup>**</sup></td><td>0.058<sup>*</sup></td><td>-0.037<sup>**</sup></td></tr><tr><td>Province</td><td>-0.173<sup>***</sup></td><td>0.034<sup>**</sup></td><td>0.014</td><td>0.155<sup>***</sup></td><td>-0.031</td></tr><tr><td>County</td><td>-0.089<sup>***</sup></td><td>-0.237<sup>***</sup></td><td>0.109<sup>***</sup></td><td>0.251<sup>***</sup></td><td>-0.034<sup>**</sup></td></tr><tr><td>District</td><td>-0.011<sup>**</sup></td><td>0.050<sup>*</sup></td><td>-0.029<sup>***</sup></td><td>0.030</td><td>-0.041<sup>**</sup></td></tr><tr><td>Local</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Geographical Area (ref. = Asia) :</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Africa</td><td>-0.139<sup>**</sup></td><td>0.074</td><td>-0.000</td><td>0.084</td><td>-0.019</td></tr><tr><td>Asia</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Latin America</td><td>0.059<sup>***</sup></td><td>-0.038</td><td>-0.014</td><td>-0.038</td><td>0.031</td></tr><tr><td>Total number of obs. in Mprobit</td><td>1,486</td><td>1,486</td><td>1,486</td><td>1,486</td><td>1,486</td></tr><tr><td>Obs. number of outcomes</td><td>225</td><td>797</td><td>81</td><td>356</td><td>27</td></tr></tbody></table>
However solar energy is not necessarily best practice. All technologies have a higher chance to bring positive impacts than hydroelectric power projects, between +41.8 pp (hybrid renewables) and +12.3 pp (solar) (Table 6, col 1). Hydroelectric power projects are highly constrained by resource location; other technologies used for DEPs could be more flexible and mobile, with a better ability to adapt accurately the energy supply to population needs, combining supplied power with local network of distribution. Hydroelectric power projects are constrained by the topography of hydrological basins, which can increase the cost of local power lines due to mountainous terrain and distance to local populations, and make a substantial difference between the theoretical potential of Small Hydro Power (SHP) and a feasible answer to electricity needs UNIDO (2010).

The most promising solutions appear to be hybrid renewables, suggesting positive combined roles played by sustainability and flexibility. However, hybrid renewable projects are very innovative, and for this reason could result in more attention being paid to the project's inclusion in its socio-economic environment. Technology could thus hide other unobservable factors of project organization and governance, with which they might be correlated.

The fact that fossil fuel technologies also have a significantly high positive impact (higher than solar projects) suggests that flexibility is a major determinant of positive impact. Although environmental negative impacts could be expected, diesel generators are highly mobile: they have the capacity to fill a missing link in energy supply networks, and provide short term solutions in emergency situations. They prove here to bring positive impacts to populations, basically by bringing missing energy: 62% of effects observed with fuel technology are related to the Energy & energy market type of effect.

Renewable technologies cover a much broader scope of socio-economic and environmental effects, with 76% of effects appearing in all types of effects except Energy & energy market, with also a relatively small number of effects on environment (4.1%). This suggests the importance of taking into account the type of effects in further analysis (see section 5.6).

Biomass technology has mixed results for development: projects using biomass technology have 21.2 pp more chance of having positive impacts than hydroelectric power (Table 6, col. 1). But projects based on biomass also show a higher risk of having negative impacts (+3.5 pp). As a matter of fact, the dependency of biomass solutions on the availability of biomass resource limits their ability to respond to energy demand.

## 5.2. Role of system size

As reference we use the nano size, which is the system size most commonly associated with SHS. All projects bigger than nano size have a high chance of generating positive impacts. Micro-grids have the highest chances to generate positive impacts (+36.5 pp, Table 6, col. 1).

The difficulty for nano size solutions to bring positive impacts may come from the observation that many projects based on nano systems fail, because they do not generate enough new income
to cover their cost Roche and Blanchard (2018). However, this observation may be wrong due to a lack of accurate evaluation of such projects. As shown by Table 6, col 2, favorable effects of nano projects have a significantly higher probability of remaining unproven than those of other project sizes.

### 5.3. Role of project decision level

DEPs can be decided at many different levels, from the local to the country level (or even the multi-country level, which we aggregated with the country level, due to lack of sufficient number of observations of multi-country projects).

The level of project decision could have different types of consequences. On the one hand, a locally decided project might take population needs better into account; it might also be based on a governance structure attentive to promoting cooperation in resource management, thereby preventing the emergence of free-riding issues. On the other hand, projects decided at country level, or at multi-country level, could benefit from a higher degree of expertise, experience, and scalability. Economies of scale in knowledge accumulation and a higher level of expertise can help to find, at least from a technical point of view, the most efficient solutions; public management and supervision systems provide country authorities with accurate feedback from the field, which can be used to identify good and bad practices in the project cycle.

The combination of these two sets or arguments suggests that both bottom-up approaches and top-down approaches can trigger positive impacts, which may lead to a U-shaped relationship between the level of decision and the probability of obtaining positive impacts. This is what we observe in Table 6, col 1. Taking the local level as reference, we observe that the farther the decision level is from the local level, the lower the probability of a positive impact when we compare decision levels from local level to county level. Above the county level, the probability of obtaining positive impacts increases again, and reaches its maximum at the country (or multi-country) level (+8.6 pp as compared to the reference local level). However, this last result could be biased, because country or multi-country programs may have more resources to implement proper *ex-post* evaluations. The probability of obtaining unproven favorable effect is significantly lower at country level than at local level (Table 6, col 2). The higher probability of positive impacts at country level could thus result from a higher probability of conclusive evaluation.

### 5.4. Role of location

Location is considered at the scale of continents and introduced as a broad control of project context.

An important issue of DEP deployment is to know whether the local context can alter the effectiveness of technical features. Addressing this question, for small geographic areas would be affected by many unobservable variables, which have been identified in the literature as important
factors. Factors like distance to raw material, cost of resource transportation, light intensity, solar incidence, wind speed, cost of local power line extension according to the morphology of terrain, and population density may influence the total costs of production and/or system performance. We do not mean that these geographical factors are unimportant, they are important, but we cannot disentangle them from other unobservable factors that affect the outcome of DEPs.

Continental location gives some information about the area of economic influence. For instance, the literature shows a genuine know-how of rural electricity development in China, where 400,000 villages have been electrified using small hydroelectric power (SHP) systems NRGExpert (2013) and UNIDO (2010). This experience is spreading into neighboring countries. Also, experts and engineers are different on different continents, and this difference might lead to different practices in electrification projects.

Asia was chosen as the reference location, due to its long experience in developing rural electricity based on DEPs.

Our model shows that projects in Latin America have a significantly higher probability to of having positive impacts (+5.9 pp, Table 6 col 1) than those in Asia, Projects in Africa have a significantly lower probability of having positive impacts (-13.9 pp, Table 6, col 1).

### 5.5. Significant pairs of size and technology

Table 7 shows the most contrasted interactions of system size and technology, by replacing the variables technology and size from Table 6 with their interaction. Only interactions with more than 30 observations were kept, and only the most contrasted pairs are shown. A positive value means that the interaction on the left has a higher probability of impact than the one on the right.

The two biggest contrasts are when comparing mini hybrid fossil projects with nano solar projects (+42.5 pp) and micro fuel projects with nano solar projects (+41.9 pp). As suggested from the separated effects in Table 6, solar *per se* may have a positive impact (although moderate), but solar projects are less efficient when the power is delivered through nano systems (i.e. in SHS), as compared to other combinations of size and technology of off-grid electricity production. The lower probability of positive impacts of nano solar projects is due to their size limitation: a too low supplied power does not permit access to all types of electrical appliances.

Half of the biggest contrasts between interactions of size and technology involve hybrid projects, with or without fuel. Deploying a micro size project with hybrid renewable technologies has +40.4 pp higher chance of generating positive impacts than a nano solar solution, and +28.5 pp more than a micro hydroelectric power project. Even nano projects based on hybrid renewable electricity have a higher probability of having a positive impact than micro projects based on a hydroelectric power source (the "classic" SHP) (+24.8 pp)% or nano projects based on solar source (+36.7 pp).
<table><caption>Table 7 – AME of impacts - Highest Significant Pairwise Comparisons</caption><thead><tr><td></td><td>Pairwise delta</td><td>p-value</td></tr></thead><tbody><tr><td>Hybrid with Fossil Mini 100 kW to 100 vs Solar Nano &lt;1 kW</td><td>0.425</td><td>0.000</td></tr><tr><td>Fossil Fuels Micro 1 to 100 kW vs Solar Nano &lt;1 kW</td><td>0.419</td><td>0.000</td></tr><tr><td>Biomass and related Micro 1 to 100 kW vs Solar Nano &lt;1 kW</td><td>0.411</td><td>0.000</td></tr><tr><td>Hybrid renewables Micro 1 to 100 kW vs Solar Nano &lt;1 kW</td><td>0.404</td><td>0.000</td></tr><tr><td>Hybrid renewables Nano &lt;1 kW vs Solar Nano &lt;1 kW</td><td>0.367</td><td>0.000</td></tr><tr><td>Solar Micro 1 to 100 kW vs Solar Nano &lt;1 kW</td><td>0.329</td><td>0.000</td></tr><tr><td>Hybrid with Fossil Mini 100 kW to 100 vs Hydropower source Micro 1 to 100 kW</td><td>0.306</td><td>0.000</td></tr><tr><td>Fossil Fuels Micro 1 to 100 kW vs Hydropower source Micro 1 to 100 kW</td><td>0.300</td><td>0.000</td></tr><tr><td>Biomass and related Micro 1 to 100 kW vs Hydropower source Micro 1 to 100 kW</td><td>0.292</td><td>0.000</td></tr><tr><td>Hybrid renewables Micro 1 to 100 kW vs Hydropower source Micro 1 to 100 kW</td><td>0.285</td><td>0.000</td></tr><tr><td>Hybrid renewables Nano &lt;1 kW vs Hydropower source Micro 1 to 100 kW</td><td>0.248</td><td>0.000</td></tr><tr><td>Solar Micro 1 to 100 kW vs Hydropower source Micro 1 to 100 kW</td><td>0.210</td><td>0.000</td></tr></tbody></table>

Values as observed in CoSSMA sample - delta > 20 pp,%, pvalue < 10%. Interactions with less than 30 observations were dropped
## 5.6. Factors of success by type of effects

The distribution of effects observed with scientific data by type of effect is shown in Table 8. The highest concentration is for Energy & energy market and the lowest is for Environment. The relatively low number of scientific measurement of effects on the environment may be due to the relatively recent emphasis in development policies on this core aspect of sustainable development. Economic transformation is also an effect for which observations are relatively few, perhaps due to the concentration of observations on SHS, which do not target productive uses of electricity.

<table><caption>Table 8 - Distribution of effects by type (scientific data)</caption><thead><tr><th rowspan="2"></th><th colspan="2">Type of Effect</th></tr><tr><th>No.</th><th>%</th></tr></thead><tbody><tr><td>Household Agenda</td><td>185</td><td>12.4</td></tr><tr><td>Community, Migration, Security</td><td>104</td><td>7.0</td></tr><tr><td>Economic transformation</td><td>105</td><td>7.1</td></tr><tr><td>Education</td><td>225</td><td>15.1</td></tr><tr><td>Energy and energy market</td><td>377</td><td>25.4</td></tr><tr><td>Environment</td><td>68</td><td>4.6</td></tr><tr><td>Health</td><td>206</td><td>13.9</td></tr><tr><td>Lifestyle & NICT</td><td>216</td><td>14.5</td></tr><tr><td>Total</td><td>1,486</td><td>100.0</td></tr></tbody></table>

A comparison with the distribution of positive impacts by type reported in Table 9 suggests that the effect types that are the most likely to result in success of DEPs are Lifestyle & NICT and Household Agenda: the proportion of positive impacts in observed effects is respectively 28% and 24%. For Community, Migration, Security, and Economic transformation the positive impacts account for 6% and 5%, respectively. For the other effect types the proportion of positive impacts is a little below the average (15%).

<table><caption>Table 9 - Distribution of impacts by effect type (scientific data)</caption><thead><tr><th rowspan="2"></th><th colspan="2">Type of Effect</th></tr><tr><th>No.</th><th>%</th></tr></thead><tbody><tr><td>Household Agenda</td><td>44</td><td>19.6</td></tr><tr><td>Community, Migration, Security</td><td>6</td><td>2.7</td></tr><tr><td>Economic transformation</td><td>5</td><td>2.2</td></tr><tr><td>Education</td><td>29</td><td>12.9</td></tr><tr><td>Energy & energy market</td><td>45</td><td>20.0</td></tr><tr><td>Environment</td><td>8</td><td>3.6</td></tr><tr><td>Health</td><td>28</td><td>12.4</td></tr><tr><td>Lifestyle & NICT</td><td>60</td><td>26.7</td></tr><tr><td>Total</td><td>225</td><td>100.0</td></tr></tbody></table>
As a consequence of this structure of effects by effect type, identifying factors that lead to positive impacts on the various types of effects is constrained by the small number of available observations for most of the types. Table 9 shows that only three effect types have more than 30 positive impacts in the CoSMMA. Due to the peculiar heterogeneity of these impacts in each effect type, estimates could be computed for only two types of effect: Energy & energy market, and Lifestyle & NICT. Table 10 shows separated regression results. For each type of effects, only observations with effects of this type were selected, thereby defining separated multi-probit estimations. Only parameters related to the probability of positive impacts are reported.

<table><caption>Table 10 - AME - Impacts - scientific data - separated regressions by subset of Effect Type</caption><thead><tr><th></th><th>Energy and Lifestyle and NICT</th></tr></thead><tbody><tr><td>No. of Observations (N)</td><td>0.000</td></tr><tr><td>Method (ref. = Econometrics without inference)</td><td>-0.000<sup>***</sup></td></tr><tr><td>Identification</td><td>0.521<sup>***</sup></td></tr><tr><td>Econometrics without inference</td><td>0.000</td></tr><tr><td>No inference</td><td>-0.211<sup>***</sup></td></tr><tr><td>Technology : (ref. = Hydro) :</td><td></td></tr><tr><td>Hydroelectric power source</td><td>0.000</td></tr><tr><td>Solar</td><td>-0.103</td></tr><tr><td>Hybrid with Fossil fuel</td><td>0.076</td></tr><tr><td>Hybrid renewables</td><td>0.007</td></tr><tr><td>Biomass (and related tech.)</td><td>0.038</td></tr><tr><td>Fossil Fuels</td><td>0.077</td></tr><tr><td>Power : (ref. = Nano)</td><td></td></tr><tr><td>Nano: &lt;1 kW</td><td>0.000</td></tr><tr><td>Micro: 1 to 100 kW</td><td>0.066</td></tr><tr><td>Mini: 100 kW to 100 MW</td><td>0.237<sup>***</sup></td></tr><tr><td>Decision Level (ref. = Local) :</td><td></td></tr><tr><td>Country</td><td>0.008</td></tr><tr><td>Province</td><td>-0.048<sup>***</sup></td></tr><tr><td>County</td><td>-0.159<sup>***</sup></td></tr><tr><td>District</td><td>0.305<sup>***</sup></td></tr><tr><td>Local</td><td>0.000</td></tr><tr><td>Geographical Area (ref. = Asia) :</td><td></td></tr><tr><td>Africa</td><td>-0.139<sup>***</sup></td></tr><tr><td>Asia</td><td>0.000</td></tr><tr><td>Latin America</td><td>0.025</td></tr><tr><td>Total number of Obs. in Mprobit</td><td>377</td></tr><tr><td>Obs. Number of positive impact outcome</td><td>45</td></tr></tbody></table>
Due to the limited number of observations, the set of controls was restricted to two essential controls, the number of observations (N) and the method of estimation.<fnref n="3" />

<footnote n="3">³ The diagnosis of colinearity was made with the sub-sample of effects in Energy market, which is the largest sub-sample (384 observations before estimation).). The number of observations (N) had a first occurrence at second rank in the Condition Number matrix for the set of explanatory variables: it should thus not be dropped in first intention. The delay of evaluation was previously used as a qualification variable, not as a strong control: it was thus preferably dropped, as compared to and the variable Method was used.</footnote>
The parameters associated with technology in the Lifestyle & NICT effect type could not be computed. They were estimated in the Energy & energy market effect type, but none were significant.

The effects of system size are in opposition depending on the effect type: mini-grids have a higher chance of yielding impacts in Energy & energy market (+23.7 pp, col1), but in Lifestyle & NICT, there is a higher probability of obtaining positive impacts with nano solutions (+30.3 pp) as compared to mini-grids or micro projects (+28.8 pp).

For effects for Energy & energy market, decisions at local and district levels have higher chances of generating positive impacts than intermediate levels of decision, partly confirming the U-shaped pattern for this type of effect. The parameters for the decision level could not be estimated for Lifestyle & NICT.

Projects which impacted Energy & energy market and Lifestyle & NICT were more efficient in Asia than Africa (+13.9 pp and +22.1 pp, col 1 and 2)

Finally, the negative sign of a number of observations for Lifestyle & NICT suggests that these results could be affected by a publication bias.

## 5.7. Extending knowledge of effects with expert data

Given the shortage of information based on scientific data, it is tempting to try to expand our information base with expert data. Expert Data (ED) has two levels of data quality: effects that rely on observations without variance (N=1) (i.e. without confidence interval), and effects that are solely documented from other research papers, or simply mentioned in institutional reports (N=0).

In both cases, including ED in the estimation does not allow the separation of proven favorable (unfavorable) outcomes and unproven favorable (unfavorable) outcomes, as ED do not provide confidence intervals.

Hence the use of ED in a model, which enlarges the observation sample, limits the precision of the model because the number of possible outcomes is reduced to 3 (favorable, unfavorable, inconclusive). This modifies the results. Merging proven and unproven favorable effects hides the specific information provided by proven favorable effects. In our data unproven favorable effects are more numerous (940 versus 225), and Table 6 shows that proven and unproven favorable effects have different explanatory factors. As a result, as shown by the differences between Table 6 and Table 11, with Expert Data included, the unproven favorable effects dominate and lead to a totally different picture of DEP effectiveness.
<table><caption>Table 11 - AME- with expert data (N >= 0)</caption><thead><tr><th></th><th>Favorable</th><th>Unfavorable</th><th>Unknown direction</th></tr></thead><tbody><tr><td>Delay of evaluation</td><td>-0.003</td><td>0.003</td><td>-0.000</td></tr><tr><td>Method (ref. = Econometrics without inference)</td><td></td><td></td><td></td></tr><tr><td>Identification</td><td>0.026</td><td>-0.051</td><td>0.025<sup>**</sup></td></tr><tr><td>Econometrics without inference</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>No inference</td><td>0.079</td><td>-0.073</td><td>-0.007</td></tr><tr><td>No measurement</td><td>0.187</td><td>-0.203</td><td>0.016</td></tr><tr><td>Technology (ref. = Hydro):</td><td></td><td></td><td></td></tr><tr><td>Wind</td><td>-0.395<sup>**</sup></td><td>0.290</td><td>0.104</td></tr><tr><td>Geothermal/Tidal</td><td>0.108<sup>***</sup></td><td>-0.131<sup>***</sup></td><td>0.024</td></tr><tr><td>Hydroelectric power source</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Solar</td><td>-0.137<sup>***</sup></td><td>0.129<sup>**</sup></td><td>0.008</td></tr><tr><td>Hybrid with Fossil fuel</td><td>-0.058</td><td>0.056</td><td>0.002</td></tr><tr><td>Hybrid renewables</td><td>0.037<sup>*</sup></td><td>-0.019</td><td>-0.018<sup>*</sup></td></tr><tr><td>Biomass (and related tech.)</td><td>-0.078<sup>***</sup></td><td>0.072<sup>***</sup></td><td>0.007<sup>***</sup></td></tr><tr><td>Fossil Fuels</td><td>-0.198</td><td>0.223</td><td>-0.026<sup>**</sup></td></tr><tr><td>Power (ref. = Nano)</td><td></td><td></td><td></td></tr><tr><td>Nano: &lt;1 kW</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Micro: 1 to 100 kW</td><td>-0.063<sup>*</sup></td><td>0.046</td><td>0.017</td></tr><tr><td>Mini: 100 kW to 100 MW</td><td>-0.029</td><td>-0.024</td><td>0.054<sup>***</sup></td></tr><tr><td>Programme Decision Level (ref. = Local)</td><td></td><td></td><td></td></tr><tr><td>Country</td><td>-0.075<sup>*</sup></td><td>0.087<sup>*</sup></td><td>-0.012</td></tr><tr><td>Province</td><td>-0.116<sup>*</sup></td><td>0.135<sup>**</sup></td><td>-0.019</td></tr><tr><td>County</td><td>-0.362<sup>***</sup></td><td>0.394<sup>***</sup></td><td>-0.032<sup>***</sup></td></tr><tr><td>District</td><td>0.101<sup>***</sup></td><td>-0.068<sup>***</sup></td><td>-0.033</td></tr><tr><td>Local</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Geographical Area (ref. = Asia)</td><td></td><td></td><td></td></tr><tr><td>Other non-OECD</td><td>0.261<sup>***</sup></td><td>-0.218<sup>***</sup></td><td>-0.043<sup>***</sup></td></tr><tr><td>Africa</td><td>-0.018</td><td>0.032</td><td>-0.013<sup>*</sup></td></tr><tr><td>Asia</td><td>0.000</td><td>0.000</td><td>0.000</td></tr><tr><td>Latin America</td><td>0.026</td><td>0.009</td><td>-0.035<sup>***</sup></td></tr><tr><td>Europe non-OECD</td><td>-0.734<sup>***</sup></td><td>-0.176<sup>**</sup></td><td>0.909<sup>***</sup></td></tr><tr><td>Total number of obs. in Mprobit</td><td>2,532</td><td>2,532</td><td>2,532</td></tr><tr><td>Obs. Number of outcomes</td><td>1,854</td><td>586</td><td>92</td></tr></tbody></table>

As shown on Table 11, Solar technology would be beaten by most alternatives, except Wind (whose parameter could not be estimated in Table 6). The most effective technologies for development would be Geothermal, whose parameter also could not be estimated in Table 6 (+10.8 pp, Table 11, col1) and Hybrid renewables (+3.7 pp).

Nano solutions would be a significantly better choice than Micro-grids (+6.3 pp, col 1) and would have a similar performance to Mini-grids.
Local decisions (whether from municipalities or districts) would be more efficient than any other level (col 1), and country levels of decision would bring more unfavorable effects (col 2).

These results are in contrast to the conclusions reached from Table 6. They are however fragile, as the significance of parameters is very sensitive to sample changes. When estimated on the sub-sample of scientific data, parameters associated with sources of energy and system size become non-significant (table not shown).

# 6. Discussion

## 6.1. Possible selection bias

The selection procedure with EBSCO ensures that no bias from a focus in search of a specific research frontier, which would result in a narrowly oriented selection of papers, remains. However, some selection bias could still occur.

First of all, impacts of DEPs can be evaluated only under 3 conditions:

- Effects arise from implemented projects

- Researchers tested these effects

- Effects were measured with observed data with heterogeneity.

Effects are not observable if researchers have not considered evaluating them: some effects were not considered relevant, or not of interest, at the time of evaluation, or not surveyed due to budget constraints on the collection of field data. This might be the case for environmental effects, which are rarely measured with field data in the papers entered in the CoSMMA. Some effects are only documented from other pieces of research, or quantified without any sample of observations. In both cases, the significance of an effect is not computable, and whether the reported effect is an evidence of impact remains unknown. Our results suggest that evaluations without statistical tests may lead to conclusions at odds with conclusions obtained from evaluations with statistical tests. Resolving this issue is beyond the scope of this paper, but future developments of the CoSMMA, whose aim is to broaden the scope and depth of DEP evaluations, could contribute to the solution.

In addition, research on DEP impacts can be affected by publication bias, because research publications are driven by the need to show innovation and tend to favor significant positive results. In our case, the relatively small proportion of positive impacts reported in evaluations may suggest that this publication bias is limited.

The publication bias can also be magnified by the possible lack of independence of the project assessor. Organizations implementing funded projects need to demonstrate *ex-post* that positive impacts occur, and might be tempted to resort to assessors dedicated to showing positive impacts. This may explain why a large proportion of papers that we have collected in the CoSMMA report favorable effects without providing scientific evidence in support of their conclusions. At this stage
of development of the CoSMMA, our choice of using evaluations based on scientific data, and not on expert data, is the only way to deal with this issue and identify best practice.

Finally, sample size could also be the origin of a publication bias. The cost of field evaluations puts a budget constraint on the sample size that researchers might be able to collect: as a result, small studies with limited samples might show significant effects only for those studies with the largest magnitude effects. The reason for this is that the critical size sample is a convex decreasing function of the magnitude of the assessed effect Châtelain (2010).

## 6.2. Cycle between funding and evaluation

Because projects are risky, donors or lenders tend to commit funds to new projects that show comparable specifications to previous successful projects, especially when they have defined risk management policies based on project characteristics. Publication bias might therefore sustain conservative commitment strategies, repeating the funding of the same type of projects as those which have shown large effects with small samples. In the absence of any third-party evaluations of electrification projects, the cycle of decision/evaluation/judgment could continue unabated. There could be a virtuous/vicious circle between publication bias and project commitment, each nurturing the other, a cycle that is all but random.

## 7. Concluding remarks

This research is the first step of the CoSMMA project, towards a better understanding of the potential contribution of DEPs to sustainable development, with the aim of identifying best practice. In this pilot CoSMMA we have assembled a database of the characteristics of 421 DEPs and their effects on sustainable development.

The results of our meta-regression highlight the roles of energy source and system size. There is clearly a trade-off between the choice of new sources of renewable energy, especially solar energy, and system size. Solutions relying on solar energy alone bring positive impacts, but these impacts are reduced because solar electricity is mainly delivered through nano systems, whose positive impacts are much less frequent than positive impacts of larger systems such as Mini-grids. Hybrid systems may provide an interesting compromise, because they can be larger than SHSs, and also help solve other technical issues such as intermittence. Our results also suggest an important role for the organizational characteristics, as evidenced by the U-shaped curve describing the influence of the decision level.

So far, we have been able to use only scientific data, and not expert data, for lack of comparable precision of data provided by experts and data reported in econometric research. As a consequence, the sample size available for meta-analysis is smaller than what we have in the CoSMMA, which limits the breadth of our exercise. In particular, data limitations prevented us from exploring systematically the different types of effects of DEPs. In particular, we could not reach any
conclusions in many fields that are critical for sustainable development, such as environment, health, and education. Also, we could not assess best practice related to the objective of economic transformation. Hopefully, the fast development of DEPs may remove these limitations in the future.

One possible direction for further research could be related to the assessment of different uses of electricity. The CoSMMA project could sustain further studies to measure which uses of electricity matter in terms of economic development, based on the proven effects of DEPs, in relation to the power (system size) which is supplied to customers. As a result, electrification project developers could quantify the need for power given the socio-economic conditions of the targeted off-grid area, and thus project sizing could be optimized, increasing the survival probability of projects.

Measuring the latent demand is important because the development path of electrical appliances that was followed by households in advanced countries cannot be replicated today in developing countries. European consumers started to buy fridges during the 1950's and mobile phones in the 2000's. In contrast, African households have reached a 60% equipment rate in mobile phones in the last 5 years, but rarely own a fridge. It is thus crucial to further analyze what will be the household preferences for electrical appliances.

Our meta-analysis of DEPs effects gives a preliminary contribution to the measurement of latent demand for electricity, as positive impacts of electrification may be considered as proxies of electricity uses in developing countries.

In addition, our meta-analysis emphasizes which project characteristics have the highest probability of having positive impacts on sustainable development, and this should help developers to relate project design to expected electricity uses.

Presenting best practices of decentralized electrification may both encourage better sizing of projects, and also provide first indications for further research on the measurement of latent demand for decentralized electricity.
References

* Bazelon, C., Smetters, K., 1999. Discounting Inside the Washington D.C. Beltway. J. Econ. Perspect. 13, 213-228.
https://doi.org/10.1257/jep.13.4.213

* Bell, S., Judson, E., Bulkeley, H., Powells, G., Capova, K.A., Lynch, D., 2015. Sociality and electricity in the United Kingdom: The influence of household dynamics on everyday consumption. Energy Res. Soc. Sci. 9, 98-106.
https://doi.org/10.1016/j.erss.2015.08.027

* Bonan, J., Pareglio, S., Tavoni, M., 2014. Access to modern energy: a review of impact evaluations.

* Borenstein, S., 2012. The Private and Public Economics of Renewable Electricity Generation. J. Econ. Perspect. 26, 67-92.
https://doi.org/10.1257/jep.26.1.67

* Borenstein, S., 2002. The Trouble With Electricity Markets: Understanding California's Restructuring Disaster. J. Econ. Perspect. 16, 191-211.
https://doi.org/10.1257/0895330027175

* Brenneman, A., Kerf, M., 2002. Infrastructure & Poverty Linkages. Lit. Rev. World Bank Wash. DC.

* Carré, E., Couppey-Soubeyran, J., Dehmej, S., 2015. La coordination entre politique monétaire et politique macroprudentielle. Que disent les modèles dsge ? Rev. Économique 66, 541-572.
https://doi.org/10.3917/reco.663.0541

* Châtelain, J.-B., 2010. Comment bien régresser: La statistique peut-elle se passer d'artefacts?, in: Séminaire Probabilismes. pp. 1-38.

* Davis, L.W., 2012. Prospects for Nuclear Power. J. Econ. Perspect. 26, 49-66.
https://doi.org/10.1257/jep.26.1.49

* Dillig, M., Jung, M., Karl, J., 2016. The impact of renewables on electricity prices in Germany – An estimation based on

historic spot prices in the years 2011-2013. Renew. Sustain. Energy Rev. 57, 7-15.
https://doi.org/10.1016/j.rser.2015.12.003

* Doucouliagos, H., Paldam, M., 2009. The aid effectiveness literature: The sad results of 40 years of research. J. Econ. Surv. 23, 433-461.

* Doucouliagos, H., Paldam, M., 2006a. Aid effectiveness on accumulation: A meta study. Kyklos 59, 227-254.

* Doucouliagos, H., Paldam, M., 2006b. Aid effectiveness on accumulation: A meta study. Kyklos 59, 227-254.

* ESMAP, 2015. Beyond Connections Energy Access Redefined. ESMAP.

* Glass, G.V., 1977. 9: integrating findings: the meta-analysis of research. Rev. Res. Educ. 5, 351-379.

* Havranek, T., Herman, D., Irsova, Z., 2018. Does Daylight Saving Save Electricity? A Meta-Analysis. Energy J. 39.

* Havranek, T., Horvath, R., Irsova, Z., Rusnak, M., 2015. Cross-country heterogeneity in intertemporal substitution. J. Int. Econ. 96, 100-118.

* Hayn, M., Bertsch, V., Fichtner, W., 2014. Electricity load profiles in Europe: The importance of household segmentation. Energy Res. Soc. Sci. 3, 30-45.
https://doi.org/10.1016/j.erss.2014.07.002

* Heshmati, A., 2014. Demand, Customer Base-Line and Demand Response in the Electricity Market: A Survey. J. Econ. Surv. 28, 862-888.
https://doi.org/10.1111/joes.12033

* Holtorf, H., Urmee, T., Calais, M., Pryor, T., 2015. A model to evaluate the success of Solar Home Systems. Renew. Sustain. Energy Rev. 50, 245-255.
https://doi.org/10.1016/j.rser.2015.05.015

* Ikejemba, E.C.X., Mpuan, P.B., Schuur, P.C., Van Hillegersberg, J., 2017. The
empirical reality & sustainable
management failures of renewable
energy projects in Sub-Saharan Africa
(part 1 of 2). Renew. Energy 102, 234–240.
https://doi.org/10.1016/j.renene.2016.10.037

* Jamasb, T., Nepal, R., Timilsina, G.R., 2017. A Quarter Century Effort Yet to Come of Age: A Survey of Electricity Sector Reform in Developing Countries. Energy J. 38. https://doi.org/10.5547/01956574.38.3.tja m

* Joskow, P., 2003. Electricity Sector Restructuring and Competition: Lessons Learned. Cuad. Econ. 40, 548–558.

* Joskow, P.L., 1997. Restructuring, Competition and Regulatory Reform in the U.S. Electricity Sector. J. Econ. Perspect. 11, 119–138.
https://doi.org/10.1257/jep.11.3.119

* Kanagawa, M., Nakata, T., 2008. Assessment of access to electricity and the socio-economic impacts in rural areas of developing countries. Energy Policy 36, 2016–2029.
https://doi.org/10.1016/j.enpol.2008.01.041

* Katre, A., Tozzi, A., 2018. Assessing the Sustainability of Decentralized Renewable Energy Systems: A Comprehensive Framework with Analytical Methods. Sustainability 10, 1058. https://doi.org/10.3390/su10041058

* Marszal-Pomianowska, A., Heiselberg, P., Kalyanova Larsen, O., 2016. Household electricity demand profiles – A high-resolution load model to facilitate modelling of energy flexible buildings. Energy 103, 487–501.
https://doi.org/10.1016/j.energy.2016.02.159

* NRGExpert, 2013. Renewable Energy and the Global Recession 2008-2013. Glob. Renew. Rep. 1–596.

* Peters, J., Sievert, M., 2015. Impacts of Rural Electrification Revisited: The African Context.

* Praktiknjo, A.J., Hähnel, A., Erdmann, G., 2011. Assessing energy supply security: Outage costs in private households. Energy Policy 39, 7825–7833. https://doi.org/10.1016/j.enpol.2011.09.028

* Roche, O.M., Blanchard, R.E., 2018. Design of a solar energy centre for providing lighting and income-generating activities for off-grid rural communities in Kenya. Renew. Energy 118, 685–694. https://doi.org/10.1016/j.renene.2017.11.053

* SE4ALL, 2017. Why wait? Seizing the energy access dividend. SE4ALL.

* Stanley, T.D., 2001. Wheat from Chaff: Meta-Analysis as Quantitative Literature Review. J. Econ. Perspect. 15, 131–150.

* Stern, D.I., 2012. Interfuel Substitution: A Meta-Analysis. J. Econ. Surv. 26, 307–331.
https://doi.org/10.1111/j.1467-6419.2010.00646.x

* Thopil, G.A., Pouris, A., 2015. Aggregation and internalisation of electricity externalities in South Africa. Energy 82, 501–511. https://doi.org/10.1016/j.energy.2015.01.059

* UNIDO, E.G., 2010. UNIDO Projects for the Promotion of Small Hydro Power for Productive Use.

* van Gevelt, T., 2014. Off-grid energy provision in rural areas: a review of the academic literature.

* Winston, C., 1998. U.S. Industry Adjustment to Economic Deregulation. J. Econ. Perspect. 12, 89–110.
https://doi.org/10.1257/jep.12.3.89

* Wolfram, C., Shelef, O., Gertler, P., 2012. How Will Energy Demand Develop in the Developing World? J. Econ. Perspect. 26, 119–138.
https://doi.org/10.1257/jep.26.1.119
ANNEXES

<table><caption>Annex 1 - Methods used by research to evaluate DEP effects</caption><thead><tr><th rowspan="2"></th><th colspan="2">Groups of Methods</th></tr><tr><th>No.</th><th>%</th></tr></thead><tbody><tr><td>Identification</td><td>721</td><td>27.7</td></tr><tr><td>Econometrics without inference</td><td>68</td><td>2.6</td></tr><tr><td>No inference</td><td>932</td><td>35.8</td></tr><tr><td>No measurement</td><td>884</td><td>33.9</td></tr><tr><td>Total</td><td>2605</td><td>100.0</td></tr></tbody></table>

Sample: CoSMMA
Annex 2 Distribution of scientific data

<table><thead><tr><th colspan="4">Technology</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Hydroelectric power source</td><td>66</td><td>4.4</td><td>4.4</td></tr><tr><td>Solar</td><td>1,079</td><td>72.6</td><td>77.1</td></tr><tr><td>Hybrid with Fossil fuel</td><td>97</td><td>6.5</td><td>83.6</td></tr><tr><td>Hybrid renewables</td><td>121</td><td>8.1</td><td>91.7</td></tr><tr><td>Biomass (and related tech.)</td><td>68</td><td>4.6</td><td>96.3</td></tr><tr><td>Fossil Fuels</td><td>55</td><td>3.7</td><td>100.0</td></tr><tr><td>Total</td><td>1,486</td><td>100.0</td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="4">System size</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Nano: &lt;1 kW$</td><td>1,032</td><td>69.4</td><td>69.4</td></tr><tr><td>Micro: 1 to 100 kW</td><td>335</td><td>22.5</td><td>92.0</td></tr><tr><td>Mini: 100 kW to 100 MW</td><td>119</td><td>8.0</td><td>100.0</td></tr><tr><td>Total</td><td>1,486</td><td>100.0</td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="4">Decision Level</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Country</td><td>519</td><td>34.9</td><td>34.9</td></tr><tr><td>Province</td><td>396</td><td>26.6</td><td>61.6</td></tr><tr><td>County</td><td>132</td><td>8.9</td><td>70.5</td></tr><tr><td>District</td><td>63</td><td>4.2</td><td>74.7</td></tr><tr><td>Local</td><td>376</td><td>25.3</td><td>100.0</td></tr><tr><td>Total</td><td>1,486</td><td>100.0</td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="4">Continent</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Africa</td><td>716</td><td>48.2</td><td>48.2</td></tr><tr><td>Asia</td><td>621</td><td>41.8</td><td>90.0</td></tr><tr><td>Latin America</td><td>149</td><td>10.0</td><td>100.0</td></tr><tr><td>Total</td><td>1,486</td><td>100.0</td><td></td></tr></tbody></table>
Annex 3 - Distribution of effects by direction and significance of effects

<table><thead><tr><th rowspan="2"></th><th colspan="2">Direction of effects</th></tr><tr><th>No.</th><th>%</th></tr></thead><tbody><tr><td>Inconclusive direction</td><td>27</td><td>1.8</td></tr><tr><td>Proven - Favorable</td><td>225</td><td>15.1</td></tr><tr><td>Proven - Unfavorable</td><td>81</td><td>5.5</td></tr><tr><td>Unproven - Favorable</td><td>797</td><td>53.6</td></tr><tr><td>Unproven - Unfavorable</td><td>356</td><td>24.0</td></tr><tr><td>Total</td><td>1,486</td><td>100.0</td></tr></tbody></table>
Annex 4 – Distribution of subsets of positive impacts

<table><thead><tr><th colspan="4">Technology</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Hydroelectric power source</td><td>5</td><td>2.2</td><td>2.2</td></tr><tr><td>Solar</td><td>165</td><td>73.3</td><td>75.6</td></tr><tr><td>Hybrid with Fossil fuel</td><td>6</td><td>2.7</td><td>78.2</td></tr><tr><td>Hybrid renewables</td><td>32</td><td>14.2</td><td>92.4</td></tr><tr><td>Biomass (and related tech.)</td><td>14</td><td>6.2</td><td>98.7</td></tr><tr><td>Fossil Fuels</td><td>3</td><td>1.3</td><td>100.0</td></tr><tr><td>Total</td><td>225</td><td>100.0</td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="4">System size</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Nano: &lt;1 kW</td><td>198</td><td>88.0</td><td>88.0</td></tr><tr><td>Micro: 1 to 100 kW</td><td>25</td><td>11.1</td><td>99.1</td></tr><tr><td>Mini: 100 kW to 100 MW</td><td>2</td><td>0.9</td><td>100.0</td></tr><tr><td>Total</td><td>225</td><td>100.0</td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="4">Decision Level</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Country</td><td>44</td><td>19.6</td><td>19.6</td></tr><tr><td>Province</td><td>105</td><td>46.7</td><td>66.2</td></tr><tr><td>County</td><td>16</td><td>7.1</td><td>73.3</td></tr><tr><td>District</td><td>6</td><td>2.7</td><td>76.0</td></tr><tr><td>Local</td><td>54</td><td>24.0</td><td>100.0</td></tr><tr><td>Total</td><td>225</td><td>100.0</td><td></td></tr></tbody></table>

<table><thead><tr><th colspan="4">Continent</th></tr><tr><th></th><th>No.</th><th>%</th><th>Cum. %</th></tr></thead><tbody><tr><td>Africa</td><td>133</td><td>59.1</td><td>59.1</td></tr><tr><td>Asia</td><td>66</td><td>29.3</td><td>88.4</td></tr><tr><td>Latin America</td><td>26</td><td>11.6</td><td>100.0</td></tr><tr><td>Total</td><td>225</td><td>100.0</td><td></td></tr></tbody></table>
Annex 5 - Distribution of effects by project objective

<table><thead><tr><th rowspan="2"></th><th colspan="2">Project objective</th></tr><tr><th>No.</th><th>%</th></tr></thead><tbody><tr><td>Access</td><td>2,216</td><td>85.1</td></tr><tr><td>Default cause</td><td>84</td><td>3.2</td></tr><tr><td>Duration of access</td><td>17</td><td>0.7</td></tr><tr><td>Shutdown</td><td>6</td><td>0.2</td></tr><tr><td>Capacity</td><td>274</td><td>10.5</td></tr><tr><td>Energy substitution</td><td>8</td><td>0.3</td></tr><tr><td>Total</td><td>2,605</td><td>100.0</td></tr></tbody></table>

Sample: CoSMMA