Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served? If so how, and if not why not?

Protocol written by John Hine, Maysam Abedin, Richard Stevens, Tony Airey, Tamala Anderson, Cardno IT Transport

EPPI-Centre Social Science Research Unit Institute of Education University of London

July 2014 The authors are part of Cardno IT Transport and were supported by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre).

This protocol should be cited as: Hine J, Abedin M, Stevens RJ, Airey T, Anderson A (2014) Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served? If so how, and if not why not? A systematic review. Protocol. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

© Copyright Authors of the systematic reviews on the EPPI-Centre website (http://eppi.ioe.ac.uk/) hold the copyright for the text of their reviews. The EPPI-Centre owns the copyright for all material on the website it has developed, including the contents of the databases, manuals, and keywording and data extraction systems. The centre and authors give permission for users of the site to display and print the contents of the site for their own non-commercial use, providing that the materials are not modified, copyright and other proprietary notices contained in the materials are retained, and the source of the material is cited clearly following the citation details provided. Otherwise users are not permitted to duplicate, reproduce, re-publish, distribute, or store material from this website without express written permission. Contents

01. Background .................................................................................................................. 1 1.1 The need for a better analysis of rural road impact .............................................. 1 1.2 Building on a recent systematic review ................................................................. 3 1.3 Methodologies used to identify impact ................................................................. 4 1.4 The outcomes and impacts of rural road investments ........................................... 5 1.5 Developing an effective theory of change ............................................................. 6 1.6 Important issues to take into account ................................................................. 7
02. Search strategy ............................................................................................................. 9 2.1 Initial searches ....................................................................................................... 9 2.2 Refinements to the search strategy following initial screening ............................ 10 2.3 Reference management ....................................................................................... 12
03. Selection criteria ......................................................................................................... 13 3.1 Language ............................................................................................................. 13 3.2 Country ............................................................................................................... 13 3.3 Area .................................................................................................................... 13 3.4 Interventions ....................................................................................................... 13 3.5 Comparators ....................................................................................................... 13 3.6 Approaches and methods .................................................................................... 13 3.7 Outcomes ............................................................................................................ 13 3.8 The process of selection ..................................................................................... 14
04. Study quality assessment ............................................................................................ 15
05. Information and data extraction .................................................................................. 17 5.1 Case studies ......................................................................................................... 17 5.2 Review articles .................................................................................................... 18
06. Synthesis ..................................................................................................................... 19
07. Review limitations ....................................................................................................... 20
08. Reporting ..................................................................................................................... 21
09. Schedule ...................................................................................................................... 22
10. Bibliography ............................................................................................................... 23 Appendix 1: Authorship of this review ........................................................................... 25
11. Background

1.1 The need for a better analysis of rural road impact

Approximately one billion people, or about 40% of the rural population served by the International Development Association (IDA), the World Bank’s fund for the world’s poorest countries, lack reliable access to the road network. As a result, rural road investment is a significant component of government and aid agency budgets. The World Bank alone spends in the region of US$ 1 billion per year on rural roads; this excludes expenditure on main and secondary roads (World Bank, 2007).

Despite the importance of the topic, there is some dissatisfaction with the evidence to demonstrate the impact of rural road investment. Although there have been quite a number of studies of impact, giving varying results, much of it has been anecdotal and in 2008 one reviewer came to the conclusion that there were relatively few studies that had been carried out with proper controls and subject to rigorous analysis and statistical testing (De Walle, 2008).

Although the evidence of impact studies has been broadly supportive of rural road investment, most studies tend to treat the topic as a ‘black box’, without identifying how, and in what circumstances, rural road investment is likely to have the most, or least, impact. Because of the lack of a consistent analysis, rural road impact studies have had very little influence on the planning and choice of standards for rural road investment. For example, showing that rural road investment in general has an impact on rural development provides little or no guidance on exactly what engineering measures to take. In fact, there are a wide range of possible interventions, including, for example, spot improvements, basic access, gravel standard roads, low-cost sealed roads. In practice, the engineering measures chosen tend to be driven by a combination of available budgets, rules of thumb, crude prioritisation indices and simple transport user cost analysis. In order to better inform decision makers we need to develop a more appropriate planning methodology from the evidence of impact studies through an appropriate theory of change.

1.1.1 Intervention

Rural road investment can take many forms: spot improvements to an existing track (including the construction of small bridges and culverts), the construction of an earth road, the upgrading of an earth road or new construction of a gravel road, or the construction of a bitumen sealed road. In most instances, road impact studies usually relate to the upgrading of an earth road to gravel standard or the rehabilitation of a gravel road. Sometimes completely new vehicle access will be established. The condition of the road or track prior to the investment can be very variable, although the final engineering standards of a newly constructed gravel roads might be relatively similar. From the users’ point of view, the key issues are passability during the wet season and the roughness of the road (measured according to the international roughness index, IRI), which affects vehicle operating costs and transport fares and tariffs.

Earth and gravel roads deteriorate quickly with traffic and rainfall and they need frequent maintenance, which may cover activities like drainage maintenance, surface grading and pothole filling. Unfortunately there is no precise distinction between ‘investment’ and ‘maintenance’. However where comparisons are made between intervention and non-intervention roads, it is usually assumed that routine maintenance will be carried out on both categories of roads. Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

The review will cover studies of all types of road investment, provided that the intervention studies cover a significant change in rural accessibility and that socio-economic data relating to the rural population is analysed. So studies relating to the impact of maintenance activity, where there appears to be little impact on accessibility, will be omitted.

1.1.2 Theory of change

The process of moving from initial rural road investment to the final impact on livelihoods of the local population is thought to be relatively complex and not fully understood. In the course of preparing the review, it is hoped that further refinements can be made. The process as articulated through a ‘theory of change’ may be broken down as listed below; however, it should be remembered that the outcomes are of variable magnitude and have variable time lags.

The initial construction process: The direct impact of construction may have a slight disruption on traffic flows as the road is being built; there will also be negative local environmental effects on water courses, the establishment of borrow pits, disposal of spoil and the movement of materials. However the main effects on livelihoods will arise from local employment during construction. These will obviously be greater if labour-intensive, or labour-based methods of construction are employed. Incomes can result from employment, letting out rooms and providing food for construction staff. Sometimes significant incomes can be generated for the local population. It has frequently been observed that household ownership of assets such as bicycles can result.

Change in transport costs, fares and tariffs: The main benefits from road investment are believed to result from an underlying change in transport costs; this may be from reduced travel times, a change in transport mode (e.g. from headloading to vehicle transport) or from reduced vehicle operating costs arising from reduced road roughness or fewer delays or diversions because roads are seasonally closed roads or boggy. The Highway Development and Management Model (HDM4) provides a framework for predicting how underlying transport costs will change with road investment. The absolute change in transport costs will depend upon the difference in road condition, the length of the road improvement (the size of the impact will vary along the length of the road) and the possibility of a change in transport mode - for example, headloading can cost, per ton/km, 30 times more than using a truck. Although the final condition of typical gravel road construction may be fairly uniform across the world, the initial condition of the road or track before being improved will be extremely variable. Lastly the extent to which changes in underlying transport costs are passed on to customers will also vary because of the competitive nature of transport services. Research has shown that in general, African transport services are very monopolistic and much higher fares and tariffs are charged than in Asia.

Changes in the reliability of transport services: Improved roads lead to improvement in the reliability of access and transport services; there may be far less chance of a road becoming impassable. This effect may not be captured by an analysis of transport costs. Improved resilience of the transport system may have wide-ranging effects on livelihoods, and may lead to a greater chance of employment outside of the area.

Changes in transport volumes: Often the most obvious change following road investment is an increase in traffic volumes. If underlying costs are cheaper and journeys quicker, there is often a strong incentive for transporters to offer more services and for the local population to travel more. In time there may be a response in other areas of economic activity which will also increase traffic volumes.

Changes in agricultural activity and produce marketing: If transport costs are reduced, transport channels more reliable, and market prices in the towns remain broadly constant, then one may expect farm gate prices to rise. Farmers are likely to respond by increasing agricultural production for external sale. However, these effects may be relatively small for small-scale road improvements. For example, it was estimated that, in Ghana, a 5 km improvement of an existing accessible vehicle track might only increase farm gate prices by 0.01 per cent for maize. However, the effect might be a hundredfold greater if a change in transport is involved. The extent to which farmers can respond to changes in farm gate prices depends upon the gross elasticities of agricultural supply. These in turn will be dependent upon the availability of suitable agricultural land, labour credit and external markets.

Changes in non-agricultural activity: It is very common for villages and small towns to respond to increased passing traffic by increasing the selling of produce and services, such as shops, hairdressing, shoe repairs, mobile phone services and wood and metal working.

Changes in the quality and availability of government, NGO and extension services: Improved roads can lead to improvements in the availability of external services. External organisations are unlikely to settle in locations that have unreliable access. They will be far more willing to locate in areas that have all-year-round vehicle access.

Changes in health and education outcomes: There is evidence to suggest that better access will lead to better supervision of schools and hence better educational outcomes. The same is likely to be true for health outcomes. Women frequently suggest that the biggest perceived benefit to them from improved roads is a better chance of getting to a health centre or hospital in times of an emergency, most obviously during childbirth. It can be matter of life or death.

Possible adverse effects: Beside possible environmental effects due to road construction, other adverse effects may be an increase in road accidents with more and faster traffic (accidents may also fall if there is better design), a possible increase in crime as more outsiders visit the area, an increase in prostitution (most likely in towns and around work camps) and an increase in the incidence of HIV/AIDS.

Changes in household incomes, expenditures, assets and livelihoods: The final impact of road investment on the local communities will be on incomes, expenditures, assets and livelihoods. Obviously if transport fares and tariffs fall and agricultural and non-agricultural activity increases, this will increase available expenditure and cash incomes which can be spent on other things. Livelihoods may also be improved by increased social mobility and improved education and health, through increasing social and physical capital.

1.1.3 Objectives

The objectives of this systematic review therefore include the systematic collection of evidence from existing reviews and rural road impact studies in low- and middle-income countries. By doing so, the review aims to answer the following questions:

1. What are the conditions, and what type of rural road interventions, are most likely to have a positive, or minimal, impact on poverty reduction and resilience in the local population?

2. What is the likely range and scale of impact for different interventions?

3. What is the most appropriate theory of change of rural road impacts that can assist with planning rural road interventions?

1.2 Building on a recent systematic review

A recent systematic review was undertaken that investigated the impact of infrastructural investments in roads, electricity and irrigation on agricultural productivity (Knox et al., 2013). Although the topic of this review is not identical, and roads only form part of the Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

material, the 2013 review is clearly very relevant. The 2013 review searched back to 1990 and only 27 relevant papers relating to roads were found.

We propose to build on the earlier systematic review by:

a) Broadening the scope of the review by looking for a wider range of poverty outcomes (i.e. not just agriculture and rural income), including health, education, access to social facilities, consumption and expenditure

b) Searching the literature back to 1980. The 2013 systematic review covered literature as far back as 1990, but it is anticipated that there will be relevant literature dating back to 1980. There also may be a few items of literature dating back to periods prior 1980; however, demographic and technological and changes will probably render such material obsolete

c) Developing from the data a theory of change that can be more readily used by planners, economists and engineers that relates the extent and quality of road intervention to the degree of impact. This should help to identify exactly what types of intervention (e.g. basic access, improved gravel roads, low-volume sealed roads) are likely to provide the best value for money in different circumstances.

d) Investigating the evidence of the effect of infrastructure investment on resilience. This will cover engineering resilience (i.e. road passability), transport service resilience and any identifiable long term effects on the resilience on the livelihoods of local communities.

1.3 Methodologies used to identify impact

A variety of methods have been used to determine the impact of rural roads, but the array of tools and methodologies are not equal in scope and sophistication. This systematic review will aim to utilise all relevant literature that passes a critical threshold of academic rigour. Further discussion regarding the literature base is provided in Section 5 (Study Quality Assessment).

The variety of methods include:

Identifying impact from memory: This is the simplest approach. After a road has been built, people along the road are asked to identify the difference. For a combination of reasons, this is probably the most unreliable approach.

Before and after study: Two surveys are carried out before and after road investment. Impact is identified as the difference. Because there are no controls - and we don’t know what happened to the rest of the economy and whether there were changes in agriculture resulting from variations in weather patterns and crop diseases during the intervening period between the different surveys - this is also perceived as unreliable.

Cross-sectional (or with and without) study: In this case, impact is identified by comparing places with different degrees of access or road condition. The surveys are carried out at broadly the same time. Sometimes modelling based on transport costs or land rents is used. Care needs to be taken to rule out other factors which may influence the result, such as soil fertility, variations in weather or population density. This approach cannot easily pick up the direct effects of employment generation due to construction activity.

Double-difference approach: This approach combines the two earlier approaches, combining the ‘before and after’ with the ‘with and without’. This is reckoned to be more reliable, as there is much greater use of control data, so historic trends can be identified and eliminated.

Propensity score matching with double-difference or cross-sectional approach: Propensity score matching is used to select households or roads and areas on a more scientific basis so that control observations can be more carefully matched with intervention observations.

**Regional econometric analysis:** In this approach, impact is identified via complex mathematical modelling from data drawn from surveys undertaken at different times in different areas of the country. National Living Standard Measurement Surveys (LSMS) are usually the source of data. Researchers from the International Food Policy Research Institute (IFPRI) have carried out about eight studies of this sort.

**Partial studies:** Sometimes partial studies are reported; these are mostly just ‘before surveys’ with an expectation that the full results will be published later after another round of surveys has been undertaken. So although there may be reference to terms such as ‘double-difference’, ‘before and after’ and ‘propensity score matching’, there may be no description or estimation of final impact. However, initial studies can have a ‘first go’ at attempting to identify impact through a cross-sectional analysis. Overall, unless there is an attempt to estimate impact (even if no impact is found), these partial studies should be omitted from a systematic analysis.

In practice, the simplest studies (‘identification from memory’ and ‘before and after studies’) will be less likely to be found in published literature; however, they will be present in the grey literature.

### 1.4 The outcomes and impacts of rural road investments

As previously described in Section 2.1.2 on theory of change, a wide range of outcomes and impacts can be derived from road investments. These are described further in detail below. As can be seen there are many similarities with the theory of change discussion; however, with the theory of change, the process from investment to outcomes and impacts is described, while in this section, the emphasis is on the end result (outcome and impact).

This review will be focusing on these outcomes and impacts:

**Extent of road network and engineering condition:** Road investment can affect both the length and condition of the network. The changes in length may be in road type (track, earth road, gravel road, paved road) or in terms of road classification (unclassified, community, district, secondary, main road etc.). Changes in condition may be specified in type, in road surface roughness, or in terms passability or impassability (e.g. impassable three months per year).

**Effects on employment and economic activity during road construction:** Road construction itself can have an important impact on the local community. This will be particularly so if labour-intensive methods are used.

**Transport outcomes:** After roads have been improved, the immediate effect will be on transport. Examples of changes may include:

- changes in transport tariffs and fares
- changes in calculated vehicle operating costs (calculated using road planning models hdm4 or Road Economic Decision – RED)
- changes in traffic composition from, say, walking to vehicles (including changes in seasonal composition)
- changes in traffic volume along roads
- changes in household trip making
- changes in trip distance or journey time
- changes in vehicle accident rates. Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

**Agricultural impact:** A wide range of agricultural impacts are often reported by impact studies. These may be in the form of changes in the production of key crops (or livestock) or changes in the marketing of key crops. Ideally one would like estimates of the change in the value of gross outputs, but this is not so common, because of the difficulty of valuing subsistence consumption. Sometimes impact studies will have details on the change in agricultural inputs, i.e. changes in the area of cultivated land, the use of labour and particularly the use of modern inputs such as fertiliser or insecticide and extension advice.

**Changes in provision of services and non-agricultural activity:** Greater accessibility will often lead to economic opportunities in the provision of services (including marketing), artisan employment and within the transport sector itself (including an increase in transport services).

**Changes in consumption or welfare:** Sometimes there may be reported changes in food consumed or the number of meals eaten per day (an important measure of welfare). It is possible that some health and nutrition outcomes may be identified.

**Changes in household cash incomes or gross expenditure:** Household financial impact is usually measured via changes in household cash expenditure or household cash income. Cash expenditures are usually more reliable than cash incomes as respondents can be sensitive to disclose sensitive financial information, however, as far as possible it is important to understand the composition of income in order to trace the impact from changes in accessibility.

**Changes in consumer behaviour:** Sometimes studies will identify changes in the pattern of consumer behaviour, such as an increase in expenditures on expensive food items (cooking oil, soft drinks, beer, meat, tinned fish) or on other item, such as batteries, clothing, shoes, cigarettes.

**Changes in household assets:** Ownership of assets is usually a good guide to overall welfare. The ownership of radios, TVs, mobile phones, bicycles, motorcycles, tin roofs and electricity supply are common examples.

**Changes in paid employment:** Often studies will identify how the pattern of employment changes with road improvements. However, this effect will need to be broken down into temporary changes and more permanent changes.

**Changes in access to important social facilities:** Studies will often mention improved access to and attendance at schools, health centres and markets, as well as use of extension services.

**Changes in health and educational outcomes:** Sometimes studies directly report on health and education outcomes. These may be positive (such as improved supervision of schools or improved maternal mortality) or negative (such as injuries or death from road accidents or an increase in HIV/AIDS).

**Other effects:** It is sometimes reported that increased accessibility and the resulting mobility can lead to a range of adverse effects, including crime, prostitution, the spread of disease, and road accidents.

### 1.5 Developing an effective theory of change

Earlier reviews that synthesise the literature will be particularly useful in helping to develop an effective theory of change so that we can more clearly identify how, and in what circumstances, engineering interventions will have an impact. From economic theory, it is currently believed that the absolute change in transport costs will be crucial, and that many of the effects may be estimated from demand and supply elasticities. It is anticipated that, ideally, the following information from case studies will be required:

1. Background

- background socio-economic data (e.g. population density, rural economy, patterns of agriculture)
- the nature, costs and extent of road engineering intervention undertaken
- local employment generated during construction and maintenance
- identified change in engineering characteristics (e.g. increased road length, change in road type, reduced road roughness, reduced incidence of impassability)
- change in measures of accessibility (changes in transport fares, costs, travel time, travel distance)
- identified impact on longer-term incomes, employment, welfare and poverty

However, it is recognised that very few (if any) studies will provide the complete range of data.

1.6 Important issues to take into account

Length of road network: The terms of reference and title of this work refer to the extension of the rural road network. Unfortunately, the length of the road network is an ambiguous concept. Roads are gradually improved over time, so a track will be upgraded to an earth road, an earth to a gravel road, a gravel to a paved road. As tracks are upgraded, so they will be formally classified and registered, and the local government or road authority will be given responsibility for maintaining them. So the classified network may increase although the total length of vehicle access may remain the same. Rural roads can be reclassified as secondary roads and secondary roads reclassified as main roads. Sometimes a completely new road will be built across land where no vehicle access existed before - but this tends to be rare. Most rural road impact studies relate to the upgrading or rehabilitation of tracks, earth and gravel roads. This may mean that culverts and bridges are built and that the running surface is improved. Earth and gravel roads deteriorate quickly, and in order to maintain their engineering properties, they will need to be graded at least every year. The gravel surface will need to be renewed every four to eight years. For rural roads and tracks, seasonal passability is a critical issue. So the route may only be open for say eight months a year. And often a road may be open but the journey is so heavy going that very few operators will attempt it.

For rural roads there is no absolute distinction between investment and maintenance. The same types of intervention may be classified in different ways. Unfortunately in Africa less than half of rural roads receive any systematic maintenance, and ‘gravel’ roads will typically be regravelled every 20 to 30 years - way beyond the standard recommended gravelling cycle. So after the initial improvement, it is very common for roads to deteriorate and offer poorer service both in terms of road roughness (affecting vehicle operating costs), and in terms of passability.

These are some of the typical complexities when analysing the types of rural transport intervention. This systematic review will address the issue through an organised process of coding and classifying different types of intervention.

Case studies that concentrate purely on the effects of routine and periodic maintenance will be excluded from the systematic review. However, as the work progresses it will be necessary to review the title and consider whether ‘Extension of the Rural Road Network’ best reflects the bulk of the evidence collected.

Resilience: The terms of reference and title refer to ‘poverty reduction and resilience for the rural areas served’. It may be difficult to separately identify resilience issues within most studies. Resilience implies longer-term sustainability, which may not be covered by the survey procedures. Nevertheless we may look out for long-term improvements in income and access to transport services and social facilities, and interventions that Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

include significant improvements in all-year-round passability, including significant work on structures such as bridges and culverts. As the study progresses, this issue will be considered.

Reporting and finding no impact: One challenge we have is that findings of little or no impact may well be initially reported in project progress reports, but these findings are less likely to receive wider publicity and be published in final project reports or review articles. It is believed that there is a substantial bias in the reporting process.

Similarly, of the studies that are published, those that demonstrate less impact are likely to be less detailed than studies that identify substantial impact. For this reason, we need to review studies carefully that, for example, place an emphasis on minor changes in consumer behaviour. It is possible that these studies collected a great deal more information, including estimation of cash incomes and expenditures or agricultural output and attempted to find a range of impacts, but because little impact was found, these important findings were left out or barely reported.

Reverse causality or endogeneity: Road investment is subject to planning and political processes. It is not randomly distributed. Using conventional road planning models, planners are more likely to invest in roads in areas where traffic and the economy are growing strongly; similarly, the planning models will not favour areas where demand is weak and the economy static. Against this, there will also be political pressures to provide some degree of equity of access for the more remote communities. Therefore, any correlation between road investment and development does not automatically imply causality.

Within the review, we can try to see if there are any other obvious explanations for impact in the different case studies, besides the road investment. However, this is an extremely difficult issue to disentangle. In the course of the work, this issue will be reviewed and it will be addressed in the final report. 2. Search strategy

2.1 Initial searches

The three main elements of the search strategy (A, B and C) will be used to capture references on the three concepts ‘rural’, ‘road’ and ‘poverty reduction OR resilience’. For the purposes of the mechanics of searching, it was thought unnecessary to make ‘resilience’ a separate concept from ‘poverty reduction’, but instead to group it with the other terms denoting some sort of positive effect of an extension of the rural road network.

The three sets will be combined as follows to arrive at the references which will be screened for relevance. Where possible, the title (TI), abstract (AB) and any descriptor (DE) fields will be searched. Where field searching is not possible or practical, then ‘all text’ of each record will be searched.

A and B and C

Where:

- **A**: (TI=transport\* or TI=road\* or TI=track\* or TI=highway\* or TI=bridge\* or TI=access\* or TI=network\* or TI=infrastructure or TI=passab\* or TI=impassab\*) or (AB=transport\* or AB=road\* or AB=track\* or AB=highway\* or AB=bridge\* or AB=access\* or AB=network\* or AB=infrastructure or AB=passab\* or AB=impassab\*) or (DE=transport\* or DE=road\* or DE=track\* or DE=bridge\* or DE=access\* or DE=network\* or DE=infrastructure or DE=passab\* or DE=impassab\*)

- **B**: (TI=rural or TI=village\* or TI=settlement\*) or (AB=rural\* or AB=village\* or AB=settlement) or (DE=rural\* or DE=village\* or DE=settlement)

- **C**: (TI=poverty or TI=impact\* or TI=income\* or TI=welfare or TI=agricultur\* or TI=output\* or TI=wealth\* or TI=economic\* or TI=development\* or TI=benefit\* or TI=asset\* or TI=employ\* or TI=outcome\* or TI=price\* or TI=earning\* or TI=wage\* or TI=wealth\* or TI=growth or TI=consumption or TI=salar\* or TI=resilien\* or TI=evaluat\* or TI=sustainab\*) or (AB=poverty or AB=impact\* or AB=income\* or AB=welfare or AB=agricultur\* or AB=output\* or AB=wealth\* or AB=economi\* or AB=development\* or AB=benefit\* or AB=asset\* or AB=employ\* or AB=outcome\* or AB=price\* or AB=earning\* or AB=wage\* or AB=wealth\* or AB=growth or AB=consumption or AB=salar\* or AB=resilien\* or AB=evaluat\* or AB=sustainab\*) or (DE=poverty or DE=impact\* or DE=income\* or DE=welfare or DE=agricultur\* or DE=output\* or DE=wealth\* or DE=economi\* or DE=development\* or DE=benefit\* or DE=asset\* or DE=employ\* or DE=outcome\* or DE=price\* or DE=earning\* or DE=wage\* or DE=wealth\* or DE=growth or DE=consumption or DE=salar\* or DE=resilien\* or DE=evaluat\* or DE=sustainab\*)

Where possible the search will be limited to developing countries by using a field-specific search, for example by combining the search with “AND DE=developing countries”. In some cases it may be necessary to search for the full list of developing countries in the title, abstract and descriptor fields.

In some databases, websites or other resources, there may be no need to include C at all. Instead, a simple ‘A and B’ search may be sufficient to gather a small number of relevant resources for screening. Accordingly, the A + B search will be run on all databases first, then C will be performed if necessary. But C will not be used in cases where its use would narrow the search so much that potentially useful studies may be lost. Similarly, in Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

transport-specific databases the term ‘transport\*’ may be excluded as it would produce too many irrelevant hits.

Detailed notes on the searching process will be kept. This record keeping will include details of who carried out each search, the databases or other resources searched, the date, the total number of results found and any problems encountered.

2.2 Refinements to the search strategy following initial screening

It is expected that minor refinements to the search strategy will be needed following an analysis of the results of the initial searches. For example, some terms may not be deemed appropriate because they are too broad in scope and hence retrieve far too many results.

We will search a wide range of potentially relevant resources for primary studies and systematic or good-quality reviews:

Bibliographic databases:

- ANTE (Abstracts in New Technologies and Engineering)
- CIS (Construction Intelligence Service)
- EconLit
- GEOBASE (Engineering Village)
- Health and Safety Science Abstracts
- PsycINFO
- Scopus
- Transport database
- TRID (Transport Research Information Services)
- Web of Science
- Google Scholar

Books and book chapters:

- Amazon.com
- COPAC book catalogue
- Google.com
- Googlescholar.com
- Zetoc

Internet gateways:

- Bubl
- Intute (social science)
- ELDIS (international development)
- SciDev.Net (science and development network)
- AGORA (Access to Global Online Research in Agriculture)

Organisational websites:

- AFCAP

2. Search strategy

- African Development Bank
- ASANRA (Association of National Road Agencies)
- Asian Development Bank
- AusAID (Australian Government Overseas Aid Program)
- Campbell Collaboration database of systematic reviews
- Cardno IT Transport
- CDB (Caribbean Development Bank)
- CIDA (Canadian International Development Agency)
- CSIR (Centre for Scientific and Industrial Research)
- DANIDA (Danish International Development Agency)
- DFID Systematic Reviews
- EuropeAid (European Commission Cooperation Office)
- German Technical Cooperation, GIZ
- gTKP (global Transport Knowledge Partnership/Practice)
- IADB (Inter-American Development Bank)
- IFRTD (International Forum for Rural Transport Development)
- International Food Policy Research Institute
- International Fund for Agricultural Development (IFAD)
- IRC (Indian Roads Congress)
- IRF (International Road Federation)
- Irish Aid
- JICA (Japan International Cooperation Agency)
- Millennium Challenge Corporation
- R4D
- REAAA (Road Engineering Association of Asia and Australasia)
- SLoCaT (Partnership on Sustainable Low Carbon Transport)
- SSATP (Sub-Saharan Africa Transport Policy Program)
- Tanzania Transportation Technology Transfer (TanT2) Centre
- Transport Links
- Transport Research Board
- Transport Research Laboratory (TRL) Ltd
- US Agency for International Development
- World Bank
- World Health Organisation
- WRA (World Road Association - PIARC)

Grey literature resources: Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

- SIGLE database
- Conference proceedings
- Dissertation abstracts
- Index to theses

The references listed in existing systematic or good-quality reviews of relevant literature will also be used to gather any studies not found in the searches (e.g. those listed by Knox et al., 2013).

2.3 Reference management

Appropriate reference management software (e.g. EPPI-Reviewer, Reference Manager, EndNote or Excel) will be used to create databases of (a) studies found in the searches prior to screening and (b) included studies following the screening and quality assessment process. The software will also be used in the data extraction and study synthesis processes. 3. Selection criteria

The inclusion criteria for the review will be as follows.

3.1 Language Only studies presented in English will be considered.

3.2 Country Only material relating to developing countries will be included. This will be defined according to the World Bank Atlas definition: a GNI per capita of less than $12,616 in 2012, which includes both low- and middle-income countries.

3.3 Area Only studies relating to predominantly (or exclusively) rural areas will be included. So, for example, studies relating to urban areas will be excluded.

3.4 Interventions Only studies relating to the extension of, or significant longer-term improvement to the rural road network (including bridges) will be included. So, for example, studies relating to new rural roads, and upgrading of roads or tracks will be included. However studies relating to improvement of footpaths, waterways, urban roads or the basic maintenance of the network will be excluded. Similarly, studies relating to transport services will be excluded unless they are associated with an improvement of the rural road network. Studies of secondary or main roads will be included if the analysis of effect is rural based.

3.5 Comparators The studies must make, or draw on, comparisons of socio-economic data relating to:

- ‘before and after’ a road investment
- ‘with and without’ a road investment
- areas with different degrees of accessibility
- some combination of the above.

3.6 Approaches and methods The studies will employ a range of approaches and methods to identify impact. These will include pre- and post-evaluation studies of road investments, econometric analysis of road programmes, geographic studies and modelling of areas with different degrees of accessibility. It is expected that studies will be drawn from international development banks, country aid agencies, country reports (including grey literature), NGOs, research institutes, and academic research journals and conferences (such as Transport Reviews and the Low Volume Roads Conference).

It is recognised that the studies will include both case studies and reviews. Both types will be initially selected according to the same general criteria; however, after selection, detailed analysis and data extraction will be different.

3.7 Outcomes The main focus of the work is on the effect of road infrastructure on poverty. For inclusion, studies must include socio-economic survey data that either directly measure Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

Income or provide significant indicators of livelihoods, poverty, welfare or income. The vast majority of this data will be quantitative, but it can include some qualitative data, e.g. via focus groups. Examples of such data include cash incomes, food consumption, household expenditures, employment, agricultural output, produce sales, agricultural prices, agricultural inputs and use of technology, household assets, transport fares and tariffs, journey frequencies (for example to work, markets, clinics and schools), educational attainment, infant and maternal mortality. Studies that do not include socio-economic data, for example, exclusively focusing upon engineering condition or project execution, will be excluded. For inclusion, the studies should draw on structured survey data; those that just draw upon anecdotal or incidental socio-economic data will be excluded.

3.8 The process of selection

Initial filtering will be piloted on the title and abstract (where available) by two researchers (Hine and Stevens). Once 90% agreement is achieved in the pilot, studies can then be screened independently at the initial stage. After initial filtering, the full text of the selected items will be reviewed by two researchers (Stevens, Hine) working independently. Cross comparisons will again be made to ensure consistency in the acceptance and rejection criteria. Where there is no agreement, the study will be discussed and, if necessary a third reviewer (Abedin) will be consulted. 4. Study quality assessment

Study quality can refer to several different things. For example, Petticrew and Roberts (2006) highlight the relevance of the research question, the internal and external validity of the study, the appropriateness of the data analysis and presentation, and the ethical implications of the intervention evaluated. The same authors note that in some settings, quality assessment may include wider issues, such as transparency, accuracy, whether the study was ‘fit for purpose’ or ‘fit for use’ and propriety in terms of ethical and legal considerations. However, they conclude that quality assessment in systematic reviews often focuses primarily on methodological problems. In this context, the most methodologically appropriate studies are assigned the greatest weight in terms of quality assessment. Studies undergo a process of critical appraisal in order to assess their scientific quality in terms of how they have minimised bias.

It is recognised that rural road impact studies are subject to a very wide range of quality. Hence the findings are not all equally valid. For example, it is generally accepted that larger, statistically representative repeated surveys with controls will be more reliable than small ‘one-off’ surveys with no controls. Furthermore some studies may treat certain components and outcomes very well but other components may be ignored or treated badly. A checklist will be developed to help score the different components and outcomes of individual case studies. The checklist will cover the following:

- overall methodology of identifying change including:

  - single survey using just interviewer recall to identify change
  - ‘before and after’ surveys with no controls
  - ‘with and without’ survey
  - surveys involving change in geographic access
  - double-difference approach involving both ‘before and after’ and ‘with and without’
  - use of propensity score matching
  - econometric analysis of regional data

- survey size, survey procedures and reported statistical significance

- other outcomes, including:

  - identification of the infrastructure intervention or comparison (e.g. road quality before and after, network length change, costs of intervention)
  - identification of the change in accessibility, transport costs, traffic and mobility
  - identification of the change in employment
  - identification of the change in consumption and welfare
  - identification of the change in household assets and consumer behaviour
  - identification of the change in social outcomes of health, education, gender etc.
  - identification of the change in agricultural production and marketing
  - identification of the change in incomes and expenditure Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

- rigour and transparency in the presentation of the final results.

It is unlikely to be possible to provide an overall weighting system that can rank the quality of all studies against one scale; rather, the focus will be on identifying the quality of treatment of individual components and outcomes. Thus a marking system will be developed to score the different aspects of the case studies. A simple hierarchy will be developed relating to the methodology adopted. For the other components and outcomes of the checklist, marks will be given to show how well each is covered.

The quality assessments will be carried out by three researchers (Hine, Abedin and Stevens) and the data compiled with other extracted data. Cross comparisons will be made to ensure consistency of approach. 5. Information and data extraction

Information on road impact will come from a combination of individual case studies and earlier review articles. As we are only aware of one systematic review, mentioned above, most review articles will be ‘unsystematic’.

5.1 Case studies

For individual case studies, the following information data will be extracted and tabulated:

- study descriptors: name of study, authors, country, publication date
- dates of and nature of surveys: purpose-built or extracted from national surveys
- types of surveys: household (including farm surveys), focus groups, transporters, markets, traffic counts, road condition, officials etc.
- number of different interviews undertaken
- type of impact study: ‘before and after’, ‘double-difference’ etc.
- background on the area: regional location, population, agro-climate zone, major crops, any important non-agricultural economic activity
- road engineering: type of road intervention; length of roads studied after the intervention; costs of intervention in US$; access condition before and after; maintenance activity before and after; road roughness before and after; passability before and after.
- transport outcomes: identified changes in transport tariffs and fares, traffic volumes in wet and dry seasons, traffic composition, household trip making, accidents
- key impacts (or lack of impact) identified
- other impacts:
  - agriculture and marketing
  - consumption, welfare
  - education and health outcomes
  - cash incomes and expenditure
  - consumer behaviour
  - household assets
  - paid employment
  - access to social facilities.

As far as possible, an attempt will be made to summarise the data into a uniform numerical form. However, it is recognised that this is only likely to be achieved for a small proportion of the data set. Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

5.2 Review articles

The review articles will be analysed to help develop a theory of change and to provide better structure to synthesise the results. The following key information will be abstracted:

- study descriptors: name of study, authors, publication date, areas or countries covered
- findings on the methodologies of identifying impact, including strengths and weaknesses of the different approaches
- identification of the circumstances most likely, or least likely, to influence the scale of impact
- most important (defined by impacts that would have the largest impact on livelihoods and household incomes) and most common impacts found
- Key causal factors and pathways to impact leading to a theory of change.

6. Synthesis

The synthesis of the evidence found by the review aims to transform the data in the primary studies into a single cohesive message which answers our question. The final message is of course partly framed by the theory and conceptual framework (outlined in the background section above) which underpin how the data is collected from the studies and organised for analysis.

In carrying out the synthesis, account will be taken of how well different outcomes are treated by the different studies. So for different outcomes (e.g. transport outcomes, agricultural output or household incomes) more weight will be given to those studies that provide a better analysis of the relevant outcomes. Extracted data will be analysed to find common patterns. The most important and most frequent impacts will be identified. The circumstances (e.g. engineering choices and socio-economic background) most likely to lead to major impacts will be identified, as will the circumstances leading to little or no impact. The results will be compared with earlier reviews. An attempt will be made to fit the case studies into an appropriate theory of change that ties in impact with economic background, engineering choices and changes in accessibility and transport costs. Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

7. Review limitations

The limitations of the systematic review will be identified. These are likely to be based on:

- the quality, comprehensive nature and number of the case studies
- the consistency of the findings and the degree to which the findings can be fitted into a plausible theory of change framework
- the geographical dispersion of the case studies and their associated socio-economic characteristics
- the range of engineering interventions identified
- the possibility of reverse causality or endogeneity affecting the findings.

8. Reporting

The report(s) will be reviewed by principal consultants of Cardno IT Transport, led by Mr Gary Taylor.

Working with DFID, the results will be submitted to portals such as DFID’s R4D, and gTKP. In addition, dissemination will take place through the International Forum for Rural Transport and Development (IFRTD), the International Road Federation and AFCAP. The results may also be submitted to journals and conferences such as Transport Reviews, the Journal of Transport Economics and Policy, and the Low Volume Roads Conference. Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

9. Schedule

The following represents the research team’s schedule according to the original proposal. There will need to be modifications due to the protracted period of protocol submission:

- submission of protocol (this document): 16 May 2014
- DFID and external review: 19 May to 18 July 2014
- study search: 21 July to 1 August 2014
- assessment of study relevance: 4 to 15 August 2014
- obtaining documents: 18 to 22 August
- extraction of data: 25 August to 5 September 2014
- synthesis and/or statistical analysis: 8 to 19 September 2014
- preparation of draft report: 22 September to 17 October
- DFID and external review of draft report: 20 October to 28 November 2014
- revision of draft report: 1 to 12 December 2014
- preparation of evidence brief for policy: 15 to 24 December 2014
- publication of final report and evidence brief: 1 to 14 January 2015

10. Bibliography

Ahmed R, Donovan C (1992) *Issues of infrastructural development: a synthesis of the literature*. Washington, DC: International Food Policy Research Institute.

Asian Development Bank, Independent Evaluation Department (2009) *Special evaluation on Asian Development Bank’s contribution to inclusive development through assistance for rural roads*. SST:REG 2009-35. [http://www.adb.org/documents/asian-development-banks-contribution-inclusive-development-through-assistance-rural-roads](http://www.adb.org/documents/asian-development-banks-contribution-inclusive-development-through-assistance-rural-roads) (accessed 15 July 2014).

Bryceson DF, Bradbury A, Bradbury T (2008) Roads to poverty reduction? Exploring rural roads’ impact on mobility in Africa and Asia. *Development Policy Review* 26 (4): 459-482.

Campbell Collaboration (2013) *International Development Group*. [http://www.campbellcollaboration.org/int_dev_our_group/index.php](http://www.campbellcollaboration.org/int_dev_our_group/index.php) (accessed 1 October 2013).

Cook P, Cook C (1990) *Methodological review of the analysis of the impacts of rural transportation in developing countries*. Transportation Research Record No. 1274. Washington, DC: National Research Council.

Chamberlain J (2007) *Interim guidelines for enhancing poverty reduction impact of road projects*. Kathmandu: Geo-environment and Social Unit (GESU), Department of Roads, Ministry of Physical Planning and Works, Government of Nepal.

De Walle D (2008) *Impact evaluation of road investment projects*. Washington, DC: World Bank.

Gough D, Oliver S, Thomas J (2012a) *Introduction to systematic reviews*. London: Sage.

Gough D, Thomas J, Oliver S (2012b) Clarifying differences between review designs and methods. *Systematic Reviews* 1: 28.

Gough D, Oliver S, Thomas J (2013) *Learning from research: systematic reviews for informing policy decisions*. [http://www.alliance4usefulevidence.org/assets/Learning-from-Research-A4UE.pdf](http://www.alliance4usefulevidence.org/assets/Learning-from-Research-A4UE.pdf) (accessed 15 July 2014).

Grootaert C (2002) *Socioeconomic impact assessment of rural roads: methodology and questionnaires*. Draft report. Washington, DC: World Bank.

Hettige H (2006) *When do rural roads benefit the poor and how: an in-depth analysis based on case studies*. Manila: Asian Development Bank.

Hine J (1982) *Road planning for rural development in developing countries: a review of current practice*. LR 1046. Crowthorne: Transport Research Laboratory.

Howe J (1981) *The impact of poverty alleviation: a review of the literature*. World Employment Programme Research. Geneva: International Labour Organisation. Does the extension of the rural road network have a positive impact on poverty reduction and resilience for the rural areas served?

Howe J, Richards P (1984) *Rural roads and poverty alleviation*. London: Intermediate Technology Publications.

Kingombe C (2011) *Achieving pro-poor growth through investment in rural feeder roads: the role of impact evaluation*. Background Note. London: ODI.

Knox J, Daccache A, Hess T (2013) *What is the impact of infrastructural investments in roads, electricity and irrigation on agricultural productivity?* CEE Review 11-007. [http://r4d.dfid.gov.uk/pdf/outputs/systematicreviews/CEE11-007_SystematicReview.pdf](http://r4d.dfid.gov.uk/pdf/outputs/systematicreviews/CEE11-007_SystematicReview.pdf) (accessed 15 July 2014).

Petticrew M, Roberts H (2006) *Systematic reviews in the social sciences: a practical guide*. Chichester: Wiley.

Sieber N (1997) *An annotated bibliography on rural transport*. London: The International Forum for Rural Transport Development.

Warr P (2005) *Roads and poverty in rural Laos*. Departmental Working Papers 2005-04. Canberra: Australian National University.

World Bank (2007) *IDA at work: rural roads: linking people to markets and services*. Washington, DC: World Bank. Appendix 1: Authorship of this review

Authors

Mr John Hine (Cardno IT Transport) Mr Maysam Abedin (Cardno IT Transport) Dr Richard Stevens (Cardno IT Transport) Dr Tony Airey (Cardno IT Transport) Miss Tamala Anderson (Cardno IT Transport)

Contact details

Address: IT Transport Ltd, The Old Power Station, Ardington, Nr Wantage, OX12 8QJ Telephone: + 44 (0) 1235 833753 Email: gary.taylor@itttransport.co.uk

Conflicts of interest

John Hine, Tony Airey and Maysam Abedin are transport professionals engaged in providing ongoing advice on transport issues and interventions.

John Hine is currently preparing a report *The Planning and Prioritisation of Rural Transport Infrastructure and Services* for the SSATP/World Bank. Similarly Tony Airey is currently preparing a Working Paper for the SSATP/World Bank called *Filling the Knowledge Gaps in Transport Monitoring and Evaluation*. In both papers, the methodology and results of road impact studies are discussed.

Both John Hine and Tony Airey have been authors of road impact studies and John Hine undertook an early (1982) review of such studies.

Maysam Abedin is providing Monitoring and Evaluation expertise for the Trans-Gambia bridge in Gambia and the PAST programme in Nicaragua. The work involves providing an evaluation of transportation schemes.

Richard Stevens is engaged in other systematic review projects in transport and international development.

There is unlikely to be any conflict of interest. The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) is part of the Social Science Research Unit (SSRU), Institute of Education, University of London.

The EPPI-Centre was established in 1993 to address the need for a systematic approach to the organisation and review of evidence-based work on social interventions. The work and publications of the Centre engage health and education policy makers, practitioners and service users in discussions about how researchers can make their work more relevant and how to use research findings.

Founded in 1990, the Social Science Research Unit (SSRU) is based at the Institute of Education, University of London. Our mission is to engage in and otherwise promote rigorous, ethical and participative social research as well as to support evidence-informed public policy and practice across a range of domains including education, health and welfare, guided by a concern for human rights, social justice and the development of human potential.

The views expressed in this work are those of the authors and do not necessarily reflect the views of the EPPI-Centre or the funder. All errors and omissions remain those of the authors.

This document is available in a range of accessible formats including large print. Please contact the Institute of Education for assistance:

telephone: +44 (0)20 7947 9556 email: info@ioe.ac.uk