Ex-post Assessmentof Transport Investments
and Policy Interventions
Roundtable Summary and Conclusions

19

Discussion Paper 2014 • 19
Tom Worsley

Institute for Transport Studies,
University of Leeds, United Kingdom

Discussion Paper No. 2014-19

Summary and Conclusions of the Roundtable:
Ex-post Assessment of Transport Investments
and Policy Interventions:
Prerequisites for ex-post assessments andmethodological challenges(15-16 September 2014, OECD, Paris)

Tom WORSLEY
Institute for Transport Studies
University of Leeds
United Kingdom

January 2015

THE INTERNATIONAL TRANSPORT FORUM
The International Transport Forum at the OECD is an intergovernmental organisation with
54 member countries. It acts as a strategic think-tank, with the objective of helping shape
the transport policy agenda on a global level and ensuring that it contributes to economic
growth, environmental protection, social inclusion and the preservation of human life andwell-being. The International Transport Forum organises an annual summit of Ministers
along with leading representatives from industry, civil society and academia.
The International Transport Forum was created under a Declaration issued by the Councilof Ministers of the ECMT (European Conference of Ministers of Transport) at its Ministerial
Session in May 2006 under the legal authority of the Protocol of the ECMT, signed in
Brussels on 17 October 1953, and legal instruments of the OECD.
The Members of the Forum are: Albania, Armenia, Australia, Austria, Azerbaijan, Belarus,
Belgium, Bosnia and Herzegovina, Bulgaria, Canada, Chile, People’s Republic of China,
Croatia, Czech Republic, Denmark, Estonia, Finland, France, Former Yugoslav Republic ofMacedonia, Georgia, Germany, Greece, Hungary, Iceland, India, Ireland, Italy, Japan, Korea,
Latvia, Liechtenstein, Lithuania, Luxembourg, Malta, Mexico, Republic of Moldova,
Montenegro, the Netherlands, New Zealand, Norway, Poland, Portugal, Romania, RussianFederation, Serbia, Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Turkey, Ukraine,
United Kingdom and United States.
The International Transport Forum’s Research Centre gathers statistics and conducts
co-operative research programmes addressing all modes of transport. Its findings arewidely disseminated and support policymaking in Member countries as well as contributing
to the annual summit.

Discussion Papers
The International Transport Forum’s Discussion Paper Series makes economic research,commissioned or carried out at its Research Centre, available to researchers andpractitioners. The aim is to contribute to the understanding of the transport sector and toprovide inputs to transport policy design.
ITF Discussion Papers should not be reported as representing the official views of the ITF orof its member countries. The opinions expressed and arguments employed are those of theauthors.
Discussion Papers describe preliminary results or research in progress by the author(s) andare published to stimulate discussion on a broad range of issues on which the ITF works.
Comments on Discussion Papers are welcomed, and may be sent to: International Transport
Forum/OECD, 2 rue André-Pascal, 75775 Paris Cedex 16, France.
For further information on the Discussion Papers and other JTRC activities, please email:itf.contact@oecd.org
The Discussion Papers can be downloaded from:www.internationaltransportforum.org/jtrc/DiscussionPapers/jtrcpapers.html
The International Transport Forum’s website is at: www.internationaltransportforum.org
This document and any map included herein are without prejudice to the status of or sovereignty over any territory,
to the delimitation of international frontiers and boundaries and to the name of any territory, city or area.

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

1. Introduction

Ex-post evaluation can be used to serve multiple purposes at the core of which is the
improvement of ex-ante analysis:






it can help policy-makers better identify the kinds of projects that work best incertain situations;
it can identify the effect of concurrent non-transportation investments and theirinteraction with transportation investments;
it can contribute to establishing the time frames in which we expect to see
economic impacts materialize, thereby helping to set realistic expectations for theeffect of investments and economic development;
we can make use of findings of ex-post assessments to support communication
with the public, improve the information provided, and support consensus-building
efforts.

But ex-post evaluation is often also perceived as a weak link in the assessment process fortransport infrastructure and policy initiatives. The assessment methods have tended to rely
on ex-ante appraisal, making predictions of how a scheme or policy might perform rather
than being based directly on the outcomes of past decisions. Substantial resources have been
devoted by those countries which rely on economic appraisal as a means of informing
decision-makers to improving and updating appraisal methods. Many of these developments,for example research on the valuation of crowding on public transport or on the valuation of
agglomeration and other wider economic benefits, have been the subject of ITF discussion
papers and Roundtables (OECD 2015, 2014, 2011) and most countries have access toextensive national guidelines on transport investment appraisal.
Many countries also have legislation and guidelines in place that require ex-post evaluation
but few actually enforce it. In the UK, for example, the National Audit Office (NAO) in its
reports regularly “reminds” the Department for Transport to conduct ex-post evaluation. The
non-execution of ex-post evaluation is not unrelated to the lack of dedicated funds.
Although one might think that ex-post evaluation is a far more straightforward exercise thaninvestment appraisal and the forecasting of what might happen in the future, the experienceof evaluators and an increasing volume of research literature would beg to differ. There are anumber of reasons for this. When funds are available, there may be limitations in terms ofavailable data. Data collection protocols are seldom introduced before project starts in orderto enable ex-post analysis. Our investment/intervention is very likely not the only changethat happened to a certain location or region. Multiple other changes probably occurred in theyears since our project became operational, which can make it very difficult to ascertain what
the net contribution of our investment/intervention was. This is can be a substantial
methodological challenge, even when data are available.

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 20153

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONS

These challenges involve methodological and practical issues for the execution of ex-post
evaluation. A more fundamental question is: are there other roles for ex-post evaluation?
The primary role is informing and improving the ex-ante appraisal process, but can an expost analysis also serve accountability in decision-making?
Four discussion papers were prepared for the roundtable to address these questions. Each
discussion paper examines either a role ex-post evaluation can play in enhancing decisionmaking or an approach to improving the execution of ex-post assessment.
The structure of this summary is as follows:







Chapter 2 outlines the meaning of ex-post vs. ex-ante evaluation.
Chapter 3 examines the accuracy of the conventional ex-ante methodology.
Chapter 4 describes two basic systems/approaches to data collection andevaluation.
Chapter 5 describes the challenges in expanding the conventional investment
appraisal to broader development effects.
Chapter 6 presents statistical tools that are available to evaluate the broader
development effects in ex-post evaluation.
Chapter 7 discusses the need to expand the concept of ex-post evaluation tomonitoring and corrective actions to be taken already during project execution.

The report ends with the concluding remarks in Chapter 8.

4

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

2. The roles of ex-ante appraisal and of ex-post evaluationin the assessment of transport investments and initiatives

This focus on ex-ante appraisal in transport sector planning provides a contrast to themethods adopted by politicians and their advisors in many other fields of evidence-based
public sector decision-making. Here the approach is to conduct well specified trials or
experiments and to monitor the effects of the intervention. Such trials often requireestablishment of two or more otherwise similar groups of people or places and administering
the treatment to one segment of this population and not to another. Comparison between
the two groups provides evidence of the outcome; if successful, the treatment or policy canthen be rolled out to other similarly affected groups. Conversely, if the test cannot be shownto deliver the desired outcomes, it can be discontinued and the experiment either abandonedor alternative treatments can be tested.
Examples of such an approach to decision-making are found in the health sector, where drug
trials are administered by testing two otherwise similar populations, with one given thetreatment to be tested and the other group a placebo. Education provides another example
of public policy decisions being based on experiments to test different teaching methods andmonitoring the outcomes.
Governments may provide guidance to ensure consistent and cost effective evaluation
methods, such as the UK Treasury’s Magenta Book1. Guidance in these fields is concernedwith setting up the trials, selecting a control group, correcting for the placebo effect andavoiding or eliminating confounding effects.
Transport investments and policy initiatives are generally different from the examples inhealth and education quoted above. Location is an important consideration: places differ
from each other in terms of their populations, their transport networks, the quantity andlocation of household and economic activity. The solution to any problem, its cost and its
impacts on transport users and others tends to be specific to the location. What works in one
place may not work so well in another.
Investment in transport is lumpy and the assets generally cannot be moved to anotherlocation if the initial outcome is not a success. So the evaluation ‘model’ of trialling an
initiative, careful monitoring and rolling it out if successful, perhaps modified if the evaluationsuggests such changes, is not usually a practical option for transport. It is not an option tobuild a short section of a high speed line, evaluate this trial length and deduce from this the
economic and other impacts of a full length city to city scheme. So in the transport field
there is a need for good quality appraisal methods to be used in advance any decision so as
to inform the decision-maker of the likely outcomes of that decision.

1

https://www.gov.uk/government/publications/the-magenta-book
Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

5

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONSThere is another sound reason for this emphasis on ex-ante appraisal in the transport sector.
There is good evidence about how and why people travel and use the transport networks
which are often congested, overcrowded and slow and which decision makers, responding totheir voters, want to improve. Based on this evidence, the well-established theory of
consumer behaviour can be used to estimate the benefits – their willingness to pay and thesurplus in excess of their willingness to pay – that transport users and others derive from anintervention.
Most countries collect transport use and traffic data which include information on the use thatbusiness travellers and industry make of their transport networks. From these data estimates
can be made of the transport cost savings and hence economic benefits gained by the
productive sector of the economy that result from providing more capacity, reducing
congestion and providing for quicker, more reliable journeys. Offsetting these benefits are
the costs that agents responsible for the provision and operation of the infrastructure andassociated services incur, estimated using the judgement of the engineering professionals
who are responsible for the design, construction and operation of the scheme.
The evidence about how and why people travel is complemented by information gained fromresearch – Stated Preference and Revealed Preference based studies – about how people
value those impacts of a scheme which do not result in direct cost savings, including savingsin non-working time, reductions in crowding on trains and improvements in reliability.
Incorporating these values into the utility maximising model of transport users’ behaviour
shows how transport users will respond, in terms of changing their choice of mode,
destination or route, when presented with the option of a new scheme. Research continues to
update and improve the values that make up this model and to extend it to cover theexternal impacts of transport investments, including the impacts on the environment and on
accessibility, economic mass and hence on the productivity of urban agglomerations.
The existence of a widely accepted theoretical basis and rich set of behavioural based
evidence has had an interesting consequence which singles transport out from many otherareas of public spending. Most of the resources devoted to research in transport economics
have been aimed at methods of improving the ex-ante appraisal process: fewer resources
have been targeted on carrying out ex-post evaluation studies. If the appraisal methods are
judged to be fit for purpose because of the plausibility of the theory on which the methods
are based and because of the evidence which supports that theory, there is, arguably, noneed for evaluation. Moreover, on those occasions when an evaluation is carried out, theresults of the evaluation become available long after the scheme has opened. In theintervening years, many improvements have been made to the modelling, forecasting andappraisal methods and the findings of the evaluation may have little relevance to the choicesthat currently face decision-makers.

6

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

3. Shortcomings of the appraisal process – errors in forecasts

The ex-ante cost benefit appraisal methods have, quite rightly, been questioned. There havebeen many cases of errors in forecasts of patronage of a public transport scheme or in
forecasts of road traffic volumes. Project costs are often underestimated2. Changes are madeto the scope of a project, sometimes after decision-makers have approved it, without any
assessment of the incremental costs and benefits of this change. Questions have been raised
about some of the key values in the cost benefit appraisal. For example, criticism has been
made of the UK’s approach to valuing rail business passenger time savings in the context ofthe economic case for high speed rail, on the grounds that business passengers often work
productively while on a journey.
Some errors are due to exogenous changes – changes in the reference case, for example on
account of the failure to anticipate at the time of the appraisal future changes in fuel prices,
GDP or in local or regional economic or territorial development assumptions. Many of thetransport models used to appraise schemes assume that land use changes are not inducedby the scheme, since the relaxation of this assumption complicates the attribution of thebenefits of the scheme. Differences between forecast and actual values also occur because oferrors in the traffic or transport model’s prediction of mode, destination or route choice.
More recent studies3 also suggest that the systematic errors in forecasting of traffic and costestimation may at least in part be the inherent result of the characteristics of the methods
applied.
A further source of error arises because the economic appraisal generally omits any
assessment of the responses of transport providers to the investment or policy intervention.
There has been a systematic failure to anticipate the growth of low cost air carriers and their
response to high speed rail, or to consider the scenario in which an open access operator
might provide on-rail competition to the operator of a new high speed line either by using thefreed up capacity of the existing line or on the new one. Nor is any account generally takenin the appraisal of transport schemes of technological change – for example the implicationsof self-guided cars on the economic case for the additional highway capacity.

2

Flyvbjerg, B. (2004): Procedures for Dealing with Optimism Bias in Transport Planning. Department
for Transport. London; Flyvbjerg, B., Holm, M.K., Buhl, S.L. (2005): How (In)accurate Are Demand
Forecasts in Public Works Projects? The case of transportation (Journal of American Planning
Association, Spring 2005, Vol. 71, No.2).
3
Eliasson, J., and Fosgerau, M. (2013). Cost-overruns and demand shortfalls – deception or selection?
Transportation Research Part B: Methodological, 57, 105-113; Rose, J. M., & Hensher, D. A. (2013).
Toll roads are only part of the overall trip: the error of our ways in past willingness to pay studies.
Transportation, 1-19; Makovšek, D. (2014). Systematic construction risk, cost estimation mechanism
and unit price movements. Transport Policy, 35, 135-145.
Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

7

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONS

4. Systems for Evaluation

These deficiencies in the appraisal process provide a clear motivation for carrying out
evaluation studies. They can help decision-makers and their analysts understand some of thecauses of error in the appraisal and learn from these. However, evaluation studies needplanning well in advance. Transport schemes take many years to plan and construct. The
impacts expected of a scheme evolve over a long period and take time to develop. Time
series data are essential for any evaluation. Rent-seeking agents respond well before ascheme has opened, while other effects materialise some years after its completion. Data on
networks, services and rail fares, while publicly available at the time of travel, are multidimensional, evanescent and complex. As a result they are often impossible to reconstructpost-hoc.
In France there is a long tradition of setting up observatories, established by law (LOTI of1982), for collecting data to facilitate a detailed evaluation of all major transport schemes.
Information is published setting out key indicators of ex-post performance, including data on
actual and forecast costs, patronage or, in the case of highway schemes, road traffic
volumes. The LOTI audit provides an estimate of the predicted and actual economic internalrate of return for the investment. In addition, the LOTI audits include a high levelassessment of the causes of any observed differences between forecast and outturn values
as a means informing analysts about potential improvements to appraisal methods. The audit
is published so that decision-makers and the public have some understanding of the impactsof the project and any errors in the forecasts. LOTI imposed a legal obligation on theinfrastructure companies to fund an audit of each scheme, to be carried out by a different
department in the organization to the one that prepared the ex-ante analysis. The report isthen reviewed by another body, although within the same line ministry, which may limit theindependence of the reviewers.
Despite the discipline imposed by the LOTI-based audit process, certain limitations wereidentified, in particular on account of the difficulty of attributing local and regional
development impacts to the scheme and comparing these ex-post estimates with the ex-ante
aspirations. In addition, the time period over which data were collected and the coverage ofthe data were both judged to be inadequate for a full evaluation of projects.
The establishment of permanent observatories has greatly improved the quality andcoverage of data on major transport schemes in France. For the largest projects, anobservatory is now set up at the same time as the project is approved, so as to ensure thatthe ‘before’ conditions are adequately recorded. Data which are available for only a shortperiod of time – for example the fare charged by a rail or air operator using yield
management methods to maximise revenue – can only be collected through a systematic andtimely monitoring process. Information on attributes of demand such as the price actually
paid by the passenger and the restrictions on the availability of the ticket improves the basis
8

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

for analysing and understanding trends in patronage and can explain some causes of errors
in the forecasts of demand. Figure 1 illustrates the volume of information – price shifts inyield management on a train line Bordeaux-Paris – which becomes unavailable after a few
months as nobody stores the information.
Figure 1. An example of the fleeting nature of information: Minimum fares for all
Bordeaux-Paris trains, based on length of time elapsed since reservation to
departure date

Source: Bonnafous 20144.
A different approach to evaluation has been followed in the United States. A Transport
Project Impact Case Studies (T-PICS) database5 has been set up, based on 100 projects thatvary by size, location and motivation. The on-line database contains details of thecharacteristics of each scheme and the objectives it was intended to achieve. Information isprovided on the indicators of change used to measure its success when compared withotherwise comparable locations, the regional and local area context and data from local
interviews about the extent of complementary policies and their role in delivering theoutcomes in terms of economic development and local employment.

4

Bonnafous, A. (2014). Permanent observatories as tools for ex-post assessment: the French casestudy. Discussion Paper No 2014-10. International Transport Forum at the OECD. Paris.
5
http://www.tpics.us/
Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

9

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONSThe T-PICS database6 is intended to provide decision makers and the public with information
about the range of outcomes delivered by a sample of projects which are broadly comparable
to one that these decision makers might be considering for their locality. The databaseprovides information on the type of scheme and the circumstances in the area served, so as
to facilitate the matching of the project under consideration with a number of broadly
comparable projects in the database. Information is provided on the factors that are judged
to have contributed to the success or failure of the schemes in the sample. Fifteen of the 100
projects failed to deliver any direct employment effects although these projects did haveother impacts on property prices and productivity.
The database is intended to be used well in advance of any detailed design or appraisal of a
project, to be considered at the early stage of the decision process as a means of sifting
potential options. It also helps decision makers and the public understand whether, incircumstances similar to theirs, complementary policies have been a necessary condition fordelivering the objectives of the project or whether the transport scheme alone is has in theseother examples been sufficient.

5. Challenges in extending the forecasting methodology– development impacts

Conventional appraisal methods estimate, at the best, the first round impacts of a scheme ontransport users – the travel time/generalised cost savings. These time savings make itpossible for users of the improved transport network to change where they live, work andcarry out the whole range of activities that every household partakes in. The savings in travel
time estimated and measured in the ex-ante appraisal are converted into changes in landuse, with second round effects on house prices, labour markets and wages. Firms use thecost savings to re-organise the location of their production, distribution and retail networks,
passing cost savings on to consumers. And these changes result in a change in the spatial
distribution of economic activity and, in some cases, its overall national level.
Policy-maker’s concerns – in this age of austerity – are with the economic development
impacts of transport. The impact of a transport scheme on economic development is far more
important to decision-makers than the present value of the time savings – and of moreinterest to voters. Transport appraisal methods are largely silent about these second roundeffects. Transport appraisal experts regard these impacts as difficult to estimate and in any
case largely accounted for in terms of the overall size of economic benefits by measuring the6

Fitzroy, S.; Weisbrod, G.; Stein, N. (2014). TPICS, TIGER and US Experience: A Focus on Case-BasedEx Post Economic Impact Assessment. Discussion Paper No 2014-11. International Transport Forum at
the OECD. Paris.

10

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

direct impacts of the project on users. Some progress has been made through theincorporation in the appraisal of Wider Economic Benefits, covering agglomeration, labour
market and imperfect competition effects within the cost benefit framework. But thesedevelopments of the cost benefit model provide only limited information on possible changes
in the location of economic activity. They do not provide decision-makers with acomprehensive assessment of the local or regional impacts of the project on economic
development. While there have been advances in integrating models of land use change into
the conventional transport model, these land use transport interaction models are costly to
set up, complex and the predictions from these models have not been subject to extensive
testing.
Evaluation, as is shown in the US and French papers, has the potential to provide decisionmakers with information about the economic development effects of a scheme. It can tellpolicy-makers what they want to know about the decisions on transport investment that theyor their predecessors have taken. The US paper describes the process of collecting anadequate sample of completed schemes and data on them. It describes a process ofidentifying the extent of any economic development effects in the area affected by thescheme and the circumstances that might have facilitated those impacts. While this does notdemonstrate causation, it can help decision-makers and the public understand what schemes
and associated policies might have a chance of delivering the desired economic development
effects. The evaluations carried out by the French observatories provide similar evidence
about the effects of the investment on land-use change and economic development.

6. Evaluation - causation

Decision-makers need to be confident that a transport investment or initiative they approve
will be the cause of the outcomes that they are elected to deliver. Yet establishing that any
observed correlation can be interpreted as causation is a challenge. Decisions about where tointervene are not made across a random set of choices. Investment is targeted on thoselocations where congestion or overcrowding is at its worst or where the accident record ishighest, or where there is a particular need to stimulate economic activity. Methods thatassume interventions are made at random will produce biased results if applied to transport.
In addition, the specific nature of each transport scheme – each place is different in terms of
its transport networks on the supply side and of the population and economic circumstances
determining demand for that network – means that data on the counterfactual, thehypothesis that the investment had not taken place, is unobtainable.

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 201511

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONS

The paper by Dan Graham7 describes a means of avoiding these potential sources of bias
through the use of statistical inference methods. The aim of the approach is to identify thedifferences between the areas which benefit from transport interventions and other areas
where no such intervention takes place. The effect of the scheme, or of a sample of schemes,
is identified by netting off the influence of these confounding effects when comparing theoutcome in locations benefiting from transport investment with what has happened
elsewhere. The use of causal inference models to identify the impacts of transport schemes
avoids imposing on the evaluation the assumptions about transport user behaviour that
underpin the conventional cost benefit based economic appraisal model.
Figure 2. An example of a causal inference method: Differences in differences
Income per capita

Treatment Group

Binary Treatment



Parallel trend assumption  Control
potential heterogeneity

Average treatment effectControl Group

Treatment

Time

8

Source: Graham et al. 2014 .
The paper provided examples of applying these techniques to a sample of highway schemesin the US and to high speed rail in Spain. In the case of the US study, the evidence did notsupport the hypothesis that providing more road capacity in urban areas reduced roadcongestion or increased productivity in the areas where the investment took place. Thesefindings should not be interpreted as implying that there were no benefits from theinvestment. The trips induced by the schemes delivered benefits dependent on increasing
mobility. But if the objective of the programme was to reduce congestion, it failed to deliver.
The same method was used to assess the impact on regional economic performance of the
Madrid-Barcelona high speed rail line. The results showed that, once confounding factors
had been accounted for, there was no significant difference between economic growth rates
in the target corridor and in other unaffected provinces in Spain. While these examples show

7

Graham, D. J.; Brage-Ardao, R.; C. Melo, P. (2014). Causal influence for ex post evaluation oftransport interventions. Discussion Paper No 2014-13. International Transport Forum at the OECD.
Paris.
8

Graham, D. J.; Brage-Ardao, R.; C. Melo, P. 2014. Quantifying the Economic Development Impacts ofMajor Transport Infrastructure Projects: A Case Study of High-Speed Rail in Spain Presentation at theRoundtable Ex-post Assessment of Transport Investments and Policy Interventions, 15-16 September
2014, OECD, Paris.

12

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

the application of the technique, they do not provide any more general lessons about the roleof transport in facilitating economic development as they depend on the nature of the
projects assessed.

7. Evaluation to inform the process of procurement,
implementation and financing a transport project

The term “ex-post” immediately suggests that something will take place after anaction/event is completed. With regard to evaluation of transport investments, “ex-post”
usually refers to evaluation after the completion of the construction phase or later. But theterm is generic. An evaluation can be executed at any point during the project development
and operation.
Indeed, one part of the work of the UK’s National Audit Office is concerned with monitoring
the progress of major infrastructure schemes or policy interventions prior to their completion.
Barker et al. (2014)9 notes that the NAO provides the UK Parliament’s Public Accounts
Committee with the evidence that they need to understand whether government
departments and their ministers are obtaining value for money in their spending andinvestment programmes. The PAC has the right to investigate spending decisions and to ask
questions of departmental officials and ministers about whether they are delivering or are
likely to deliver value for money.
Major infrastructure projects take many years to deliver and the NAO will often review
progress during the implementation and construction phases if in its view such anintervention might be beneficial. The NAO seeks to ensure that the discipline imposed on
project sponsors by the process of cost-benefit appraisal is not relaxed once the project has
been approved. For example, changes in the scope of a major project frequently arise after it
has been approved and, while such variations might not be subject to economic appraisal,
the NAO’s examination can act as an incentive on project sponsors and funders to manage
the costs of such alterations to the initial contract.
The NAO’s role is not restricted to reviews during the phase of project implementation: it also
carries out evaluations of a sample of major projects after they have opened to understandwhether they have delivered the expected benefits in terms of traffic volumes, time savings
and regeneration impacts. These post opening audits provide the NAO with the opportunity

9

Barker, G.; Beardsley, G.; Parsons, A. (2014). The National Audit Office’s value-for-money
assessment of transport investments. Discussion Paper No 2014-12. International Transport Forum at
the OECD.

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 201513

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONS

to make recommendations to the Department for Transport about improvements to its
appraisal methods.
In general it can be noted, that both the NAO and T-PICS approaches to ex-post assessment
rely heavily on interview and interaction with people and organizations directly affected by
the investment(s) being assessed.

8. Conclusions

While economic appraisal tends to follow a standard approach based on the theory of
consumer behaviour, there are several different approaches to conducting an ex-post
evaluation. The model chosen depends on the purpose of the evaluation and the linkage
between the purpose of the evaluation and the method for the evaluation is examined here.

Purpose of evaluation – audit during implementation
Audit of the stages of implementation, procurement, management and financing: By
monitoring these stages of project development and delivery, evaluation can provide those
responsible for approving the funding the project (Parliament, in the case of the UK’s NAO)
with an account of the management of the project by the transport department or other
responsible authority. The evaluation can highlight risks to successful project delivery andrecommend changes to tighten overall management so as to reduce the risk of project delays
and of cost overruns. This also provides lessons for management of future projects.
There is a strong case for ensuring that such an audit is carried out by an organisation that is
independent of the project sponsor. The UK NAO has noted that independent audit at an
early stage in the delivery of a project reduces the risk of any overrun only becoming
apparent when it is too late to take remedial action.
Experience in the UK suggests that the threat of external audit imposes a stronger disciplineon Department for Transport and supports the Department’s process of internal audit.
Many countries have within government an audit department which is independent of thedepartment responsible for transport and other spending departments. Not all are as active
as the US GAO or the UK’s NAO or have the support of the UK Parliament’s PAC to ensurethat the legislature is aware of their findings. There is a case for strengthening the role ofindependent government audit departments in many jurisdictions and encouraging them tocontribute to the objective of better management of the delivery of transport projects,
thereby reducing cost overruns and encouraging those responsible to improve cost anddemand forecasts.

14

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

Purposes of appraisal – did the project deliver its objectives?
Policy makers need to know whether a project has delivered its objectives. Reassurance thatthe outcomes delivered by the investment are in line with impacts that were predicted at thetime the decision was made provides policy makers with the confidence to continue with orextend the transport investment programme. It can also provide evidence to support the exante appraisal methods used to justify the initial decision to approve the investment.
Evaluation can serve a number of objectives. These include:
1. Identifying the differences between forecast and actual values for project costs, roadtraffic or public transport patronage volumes and other key impacts such as carbon
dioxide emissions.
One approach is to provide a high level assessment of the possible causes of any
differences, such as the impact of the economic downturn, not anticipated when thescheme was approved. While such an approach might not provide the level of detailneeded for a full comparison of the counterfactual with observed demand, it can help
to improve the allowance to be made for risk and uncertainty in future decisions.
2. Validation of the transport model used to estimate mode, route and destination
choice.
Evaluation is only used infrequently for the purpose of validating the transport model
adopted to appraise a scheme. In many circumstances models become obsolete by
the time the scheme has opened and become established, or the data and model runs
are no longer available. Decision-makers are unlikely to be willing to prioritise funding
for a new best-practice model to represent a counterfactual which, because thescheme has been built, could not exist.
3. Understanding some of the key relationships between transport investment and its
outcomes through econometric analysis of a sample of schemes.
Several studies have been carried out in the UK to examine the link between changes
in transport costs and changes in employment and productivity.10
4. Determining the extent to which the decision-makers’ objectives for the scheme or
programme were delivered.
These objectives might include economic development, defined in terms of additionaljobs in the area, increases in productivity or take-up of industrial sites, population
growth, encouraging housing developments, shift from car to public transport or toactive modes, and reductions in carbon intensity or other environmental goals. This
forms the main focus of most evaluations of transport schemes11.

10

https://www.esrc.ac.uk/my.../1eae75a5-2f1f-41f3-90e0-deb0c11f5a0e
A point mentioned in the roundtable discussions is also the need to properly define the project’s
objectives in the first place. For example, one can only speak about having a problem if there is at least
11

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 201515

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONS

It should be noted that the objectives of some projects do not necessarily coincidewith the value-for-money concept in economic theory or with the purpose of
maximizing socio-economic welfare. For example, it is entirely possible that it is in the
public interest to improve a transport connection to a dislocated community even if
this intervention does not increase the social welfare of the nation (i.e. the project isassessed to have a negative net present value). In this context the challenge for the
ex-post evaluator is how to devise a clear measure of success, which would
differentiate between abuse of power and social necessity.
Evaluation to determine whether an investment or a programme of projects has delivered onpolicy objectives does not necessarily establish any causal link between the project and theoutcomes. Even when the evaluation takes the form of validating the transport model andthe responses to the changes in transport costs as a result of the investment, there is an
implicit assumption in such an evaluation about the structural form and behaviouralrelationships within the model. The use of causal inference methods to attribute causation isparticularly suited to transport investments because the decision about where and when toinvest is not a statistically random one but one based on the greater perceived need in thelocation where the investment is made. In addition, the approach does not require any
hypothesis about the counterfactual or about transport users’ responses to changes intransport costs. While its application to transport schemes is still at the research stage, it has
the potential to provide a valuable method for better evaluation.
Data are essential in any evaluation method. The French permanent observatories have set a
new standard for monitoring changes over the long period during which a major new scheme
influences travel demand and territorial development. The process of data collection needs toencompass the technological developments that have made possible changes in the way
public transport is priced and accessed and how travel time is used.
Each approach has strengths and weaknesses – in terms of costs of collecting the data, thetarget audience and how it helps to improve decision-making and provide for more informed
decision-makers. Case-study approaches in particular help transport ministers understandwhat complementary policies (land use planning, skills and training, institutional changesetc.) might increase the likelihood of transport programmes or schemes delivering theirobjectives.

one solution available. If this is not the case, then our problem is only a fact. An example would be aroad connection, which is congested and winding and the objective was to reduce congestion. An
investment into a straight road would reduce travel time, but it would not reduce congestion (due toinduced traffic for example). Thus the success of an intervention is also dependant on the proper
definition of the objectives.

16

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENT AND POLICY INTERVENTIONS

The way forward?
The discussion papers presented at this roundtable each introduce a unique element:NAO’s assessments, already in the project preparation and execution stages, enable
monitoring on the spot, while the project is still running and corrective action is still
possible, and accountability still enforceable.



The T-PICS case-study approach, among other things, involves field research andinterviews. Potentially it provides a source of information that can explain therelations between cause and effect.

Transport observatories represent a robust approach to ensuring that short-lived data
can be preserved and used in the ex-post evaluation.

Causal inference methods provide a tool for the objective assessment of effects (butdo not explain the functional relations which lead to them),

The latter three approaches also enable an improvement or calibration of structural models
used in ex-ante assessments in terms of functional relations applied as well as the accuracyof predicted outcomes.
Combining these elements together may represent a holistic approach to ex-post evaluation,
creating a high-resolution image of the outcomes we might expect from transport
investments and other interventions. What perhaps remains to be determined in the future isa standard or best practice of ex-post evaluation that would combine these methods andtheir unique elements in the most efficient and cost effective way.

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 201517

SUMMARY AND CONCLUSIONS: EX POST ASSESSMENT OF TRANSPORT INVESTMENTS AND POLICY INTERVENTIONS

References

Barker, G.; Beardsley, G.; Parsons, A. (2014), The National Audit Office’s value-for-money
assessment of transport investments. Discussion Paper No 2014-12. International
Transport Forum at the OECD, Paris.
Bonnafous, A. (2014), Permanent observatories as tools for ex-post assessment: the Frenchcase study. Discussion Paper No 2014-10, International Transport Forum at the OECD,
Paris.
Fitzroy, S.; Weisbrod, G.; Stein, N. (2014), TPICS, TIGER and US Experience: A Focus on
Case-Based Ex Post Economic Impact Assessment. Discussion Paper No 2014-11,
International Transport Forum at the OECD, Paris.
Graham, D. J.; Brage-Ardao, R.; C. Melo, P. (2014), Causal influence for ex post evaluationof transport interventions. Discussion Paper No 2014-13. International Transport Forumat the OECD, Paris.
LOTI (1982), Loi d'orientation des transports intérieurs, France,
http://www.legifrance.gouv.fr/affichTexte.do?cidTexte=LEGITEXT000006068730
OECD (2011), Improving the Practice of Transport Project Appraisal, International Transport
Forum Roundtable 149, OECD publishing.
OECD (2014), Appraising Transformational Projects: The case of the Grand Paris Express,
International Transport Forum and Société de Grande Paris, OECD Publishing.
OECD (2015), Major Transport Infrastructure Investments and Economic Development,International Transport Forum Roundtable 154, OECD publishing.

18

Tom Worsley, Discussion Paper No. 19 — © OECD/ITF 2015

International Transport Forum
2 rue André Pascal
75775 Paris Cedex 16
itf.contact@oecd.org
www.internationaltransportforum.org