CPB

Broadening Transport
Appraisal
Summary and Conclusions

188

Roundtable

Corporate Partnership
Board

188

Roundtable

Broadening Transport
Appraisal
Summary and Conclusions

The International Transport ForumThe International Transport Forum is an intergovernmental organisation with 64 member countries. It acts
as a think tank for transport policy and organises the Annual Summit of transport ministers. ITF is the only
global body that covers all transport modes. The ITF is politically autonomous and administratively
integrated with the OECD.
The ITF works for transport policies that improve peoples’ lives. Our mission is to foster a deeper
understanding of the role of transport in economic growth, environmental sustainability and social
inclusion and to raise the public profile of transport policy.
The ITF organises global dialogue for better transport. We act as a platform for discussion and prenegotiation of policy issues across all transport modes. We analyse trends, share knowledge and promoteexchange among transport decision makers and civil society. The ITF’s Annual Summit is the world’s largest
gathering of transport ministers and the leading global platform for dialogue on transport policy.
The Members of the Forum are: Albania, Armenia, Argentina, Australia, Austria, Azerbaijan, Belarus,
Belgium, Bosnia and Herzegovina, Bulgaria, Cambodia, Canada, Chile, China (People’s Republic of),
Colombia, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Georgia, Germany, Greece,
Hungary, Iceland, India, Ireland, Israel, Italy, Japan, Kazakhstan, Korea, Latvia, Liechtenstein, Lithuania,
Luxembourg, Malta, Mexico, Republic of Moldova, Mongolia, Montenegro, Morocco, the Netherlands,
New Zealand, North Macedonia, Norway, Poland, Portugal, Romania, Russian Federation, Serbia,
Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Tunisia, Türkiye, Ukraine, the United Arab Emirates,
the United Kingdom, the United States and Uzbekistan.
International Transport Forum
2 rue André Pascal
F-75775 Paris Cedex 16
contact@itf-oecd.org
www.itf-oecd.org

ITF Roundtables
ITF Roundtables bring together international experts to discuss specific topics on economic and regulatory
aspects of transport policies in ITF member countries. Findings of ITF Roundtables are published in a
Summary and Conclusions paper. Any findings, interpretations and conclusions expressed herein are thoseof the authors and do not necessarily reflect the views of the International Transport Forum or the OECD.
Neither the OECD, the ITF nor the authors guarantee the accuracy of any data or other information
contained in this publication and accept no responsibility whatsoever for any consequence of their use.
This document and any maps included herein are without prejudice to the status of or sovereignty overany territory, to the delimitation of international frontiers and boundaries or to the name of any territory,
city or area. The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli
authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East
Jerusalem and Israeli settlements in the West Bank under the terms of international law.
Cite this work as: ITF (2022), Broadening Transport Appraisal: Summary and Conclusions, ITF Roundtable
Reports, No. 188, OECD Publishing, Paris.

ACKNOWLEDGEMENTS

Acknowledgements
This report summarises expert discussions during the International Transport Forum (ITF) Roundtable“Broadening Appraisal to Capture the Full Impact of Transport Investments”, convened virtually from 29
September to 1 October 2021.
Rex Deighton-Smith (ITF) and Professor Roger Vickerman (Imperial College, London) drafted the reportwith inputs from Roundtable participants from 21 ITF member countries. Stephen Perkins (ITF) provided
direction and quality control. David Prater (ITF) copy-edited the report and prepared it for publication.
Special thanks are due to the authors of the four input papers prepared for and presented at theRoundtable: Alain Bonnafous, Armando Carteni, Ennio Cascetta, Karel Martens, Morten Skou Nicolaisen,
Nir Sharav, Yoram Shiftan and Yuval Shiftan. Thanks are also due to the authors of three additionalpresentations made during the Roundtable: Floridea Di Ciommo, Morten Welde and Tom Worsley.
Finally, the ITF would like to thank the 44 participants (listed in Annex A) and in particular Professor Roger
Vickerman for chairing the discussions.

GLOSSARY

Glossary
Accessibility

The ease with which destinations can be reached to enable accessto fundamental needs such as work, education, healthcare andshopping. Accessibility measures consider the time and costrequired to reach destinations, using different modes, as well asfactors such as comfort and safety.

Benefit–cost ratio

The ratio of the total dollar value of the expected benefits of a
proposed project and the total dollar value of the expected costs.

Cost–benefit analysis

A systematic method for summing all the expected benefits andcosts arising from a particular action (e.g. a proposed transport
project) and comparing them using discounting to make benefits
and costs accruing at different times comparable.

Discounting

A method used to convert future values occurring over different
periods of time to a present value so that alternative future values
can be compared on the same basis.

Discount rate

The specific percentage rate(s) used in conducting discounting.

Diversity, equity and inclusion

A term used to describe policies and programmes that promote
the representation and participation of different groups ofindividuals (e.g. people of different ages, races and ethnicities,
abilities, genders, religions, cultures and sexual orientations).

Deliberative monetary valuation

An interactive valuation method, which brings different actors
together to form value judgements in an open dialogue. It allows
consideration of ethical beliefs, moral commitments and social
norms rather than being based solely on the aggregation ofindividual utility values.

Ex-ante evaluation

Evaluation undertaken before an investment decision is made.

Ex-post evaluation

Evaluation undertaken after an investment decision is made.

Internal rate of return

A measure of the profitability of an investment. The IRR is thediscount rate that makes the net present value (NPV) of all cash
flows equal to zero in a discounted cash flow analysis.

Mobility

The ability and ease of moving people and goods.

Monetise

To convert a benefit or cost into an equivalent monetary value.

GLOSSARY

Multi-criteria analysis

A means of systematically comparing the attractiveness ofdifferent investments or other actions by explicitly identifying andascribing weights to performance criteria, scoring options against
each criterion and summing up the resulting scores to provide anaggregate measure.

Needs case

Assessment undertaken to determine whether investment isrequired in a particular context.

Net present value

Difference between the discounted value of all expected benefitsof a project and the discounted value of all expected costs.

Participatory value evaluation

A method for assessing the relative desirability of a set of potential
investments. In a participatory value evaluation, participants are
given information about the impacts of a range of potential
projects and a constrained public budget in an experimental
environment. They are asked to choose their preferred projects,
while respecting the public budget constraint. The trade-offs madecan be used to establish individuals’ preferences to rank theseprojects in terms of their desirability.

Shadow carbon priceAn investment analysis tool that adds a hypothetical surcharge tothe price of projects that involve the creation of carbon emissions.
The shadow price is intended to enable the harms associated withemissions (or benefits of reductions) to be taken into account ininvestment decisions, making for more comprehensive analyses.

Stochastic cost estimation

A technique for estimating the level of uncertainty of an economic
result. The simulation technique makes use of probability
distributions.

Subjective well-being

A self-reported measure of well-being, typically obtained byquestionnaire. It includes both emotional reactions and cognitive
judgements.

Utilitarianism

An ethical perspective based on the view that the best available
outcomes is one that leads to the greatest happiness of thegreatest number of people.

Welfare economics

A branch of economics that uses micro-economic techniques toevaluate well-being (welfare) at the aggregate level. Application of
the principles of welfare economics gives rise to the field of public
economics, the study of how government might intervene toimprove social welfare. Welfare economics provides thetheoretical foundations for certain instruments of public
economics, including cost-benefit analysis.

Willingness to pay

The maximum price a person is willing to pay for a good or service.

TABLE OF CONTENTS

Table of contents
Executive summary ............................................................................................................................. 8
The objectives of transport planning ................................................................................................. 11
Addressing equity, sustainability and other strategic objectives ........................................................ 13
Modifying cost–benefit analysis .................................................................................................... 14
Cost–benefit analysis and climate impacts.................................................................................... 17
Integrating cost–benefit analysis and complementary analytical tools......................................... 20
Addressing the gender dimension of equity ................................................................................. 25
Improving appraisal through ex-post evaluation ................................................................................ 27
Reviewing evaluation outcomes.................................................................................................... 27
Lessons for appraisal practice ....................................................................................................... 28
Conclusions on ex-post evaluation ................................................................................................ 29
Country examples of ex-post evaluation ............................................................................................ 31
France............................................................................................................................................ 31
Norway .......................................................................................................................................... 34
The United Kingdom...................................................................................................................... 36
The need for wider reform of transport planning .............................................................................. 38
The role of strategic planning........................................................................................................ 38
Decision processes and transport planning practice ..................................................................... 42
Conclusions and recommendations ................................................................................................... 48
Conclusions ................................................................................................................................... 48
Recommendations ........................................................................................................................ 49
References ........................................................................................................................................ 52
Notes................................................................................................................................................. 57
Annex A. List of Roundtable participants ........................................................................................... 58

6

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

TABLE OF CONTENTS

FiguresFigure 1. Government guidance on carbon values for use in project evaluation, United Kingdom..... 18
Figure 2. Accessibility maps: before and after the proposed transit system in Tel Aviv...................... 22
Figure 3. Accessibility change by socio-economic decile, Tel Aviv ...................................................... 22
Figure 4. The Norwegian Quality Appraisal scheme for large government projects ........................... 34
Figure 5. Assessment of transport infrastructure investments in Norway.......................................... 40
TablesTable 1. Commissioning delays and profitability losses in French high-speed rail projects................. 32
Table 2. Criteria used in ex-post evaluations, Norway........................................................................ 35
Table 3. An example of an appraisal summary table .......................................................................... 42
Table 4. A taxonomy of uncertainty in decision making ..................................................................... 44
Boxes
Box 1. Incorporating accessibility analysis in transport appraisal in Israel .......................................... 21
Box 2. The ITF Gender Analysis Toolkit for Transport ......................................................................... 26
Box 3. National Audit Office insights on major programme management ......................................... 37

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

7

EXECUTIVE SUMMARY

Executive summary
What we did
This report examines what ongoing shifts in the objectives of transport policy mean for transport projectappraisal and planning processes. Many countries are rethinking their transport policy and planningobjectives. Their focus is increasingly shifting from providing mobility to ensuring accessibility; giving
greater priority to equitable access for all; recognising the urgency of decarbonising transport; and making
urban environments healthier, safer and more liveable. The report summarises the findings of a discussion
among 44 experts from 21 countries at an ITF Roundtable held from 29 September to 31 October 2021.

What we foundAppraisal techniques should consider the impacts of potential investments on all transport policy
objectives. Indirect valuation techniques have progressively broadened the scope of cost–benefit analysis(CBA) to capture more impacts that are harder to express in monetary values. However, changing policy
objectives – in particular, increased interest in equity, access and climate change issues – require further
enhancements to CBA and the use of complementary appraisal methods.
CBA sometimes addresses equity concerns by incorporating distributional weights. However, factoring
equity issues directly into CBA requires disaggregated assessment (e.g. by income, gender or location) andweighting the results. Choosing weights to reflect policy objectives is challenging. Moreover, opportunities
for dialogue on the relative importance of distributional and efficiency impacts are lost if results arepresented only in aggregated form.
There are questions regarding the analytical robustness of CBA in incorporating climate change
imperatives in decision making. However, recent CBA guidance has done much to address this issue. The
conceptual basis for determining shadow carbon prices increasingly reflects the cost of achieving nationalemission-reduction commitments.
Supplementary analyses alongside CBA results can greatly enhance the information available to decision
makers, even where state-of-the-art CBA practice is applied. These include various kinds of accessibility
analyses which can shed light on social, regional and gender-equity issues.
Many experts advocate using multi-criteria analysis (MCA) to integrate the results of the diverse analyses.
However, presenting the results of multiple analyses as a single set of MCA scores is often problematic,
since views can differ on the weights allocated to different objectives. Presenting results as aggregated
MCA scores reduces the information readily available to decision makers and stakeholders. Shortsummaries that highlight the results of the CBA, the supplementary analyses and their implications offer
more insights for decision making.
To address shifts in transport policy objectives, a broad response that encompasses the entire transport
planning process should accompany improvements to appraisal. Project identification, in particular, must

8

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

EXECUTIVE SUMMARY

respond to changing transport policy objectives. Critical steps are better framing of project proposals inlight of broader policy objectives identifying alternatives consistent with these broader framings, andimproving the way projects are reviewed against broader policy objectives in the initial needs-caseassessment. Adopting a strategic planning framework can improve outcomes by offering a co-ordinatedand consistent approach to infrastructure investment.
Ex-post evaluation can do much to refine appraisal by identifying systemic biases and enhancing
methodologies. However, evaluation results are not always transferable between projects or countries. A
major problem in evaluation practice is the fleeting nature of data. Thus, data for ex-post studies must begathered from the start of a project to avoid data loss. Systematic and comparable evaluations can begreatly facilitated by establishing permanent observatories, as in France.
Decision processes should account for the complexity of the problems that major infrastructure
investments address and the uncertainty involved. Extensive stakeholder engagement can reduce the riskof planning failures by improving information, identifying and addressing concerns, enhancing thelegitimacy of decisions and avoiding the withdrawal of consent by important parties. Multi-stage decision
making improves outcomes as decisions are made progressively as more and better information becomes
available. Sound decision making should integrate three elements: technical assessment, process
management and public engagement.

What we recommend
Develop long-term strategic infrastructure plans that explicitly identify transport policy objectives
Strategic plans should explain the importance of each objective and set out the policies and investments
required to achieve them. This helps ensure project appraisals reflect governments’ strategic policy
orientations.
Broaden project appraisal to ensure its processes and practices take account of all transport policy
objectives, as embedded in strategic infrastructure plans
CBA should make use of indirect valuation techniques and other methodological innovations. Aggregate
CBA should be supplemented with a disaggregated appraisal of project costs and benefits for populationgroups of concern (e.g. because of income, gender, or location). Other analytical tools can yield additional
insights. They should be adopted as needed.
Incorporate accessibility indicators, or other relevant tools, to assess equity impacts in transport projectappraisals
The increasing focus on accessibility as the central objective of transport policy requires that appraisals
incorporate purpose-specific measures to capture and assess accessibility and other relevant dimensionsof equity. Methods should be chosen for their ability to clarify the specific issues at stake.
Provide detailed guidance on accounting for climate change impacts in transport project appraisals,
incorporating clear linkages between shadow carbon prices and emissions reductions commitments
CBA is a robust method for addressing climate change impacts if all relevant impacts are assessed andrealistic estimates of the cost of meeting emissions reduction targets are used. Social willingness
willingness-to-pay measures – that is, measures of the values society collectively places on key outcomes– should be incorporated in the analysis.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2020

9

EXECUTIVE SUMMARY

Present the results of transport project appraisals in a transparent and concise format that highlights needscase assessments
The results of broader project appraisals that assess performance against a wider range of policy objectives
cannot readily be expressed in terms of a single metric. Decision makers and stakeholders differ on which
impacts are important and what weights should be attached to them.
This implies that, to support well-informed discussion and decision making, appraisal results should bepresented in a multi-dimensional format that separately presents the performance of potential projects
across different measures, and in terms of different underlying policy objectives.
Ensure decision-making processes for large investments in transport systems account for uncertainties andthe need for broad stakeholder support
Processes that are properly adapted to capture decision-critical factors minimise the risk of planning
failures. For large investments, the decision process should be multi-staged, with feedback loops thatincorporate previous outcomes into subsequent decisions. Stakeholder engagement should commence atthe early stages of the process and be maintained throughout.
Integrate technical assessment, process management and public engagement into decision processes for
major transport infrastructure investments
Ensuring that these three elements are well-integrated will help ensure that decisions meet efficiency,
equity and other underlying objectives and enjoy wide support, enabling them to be implemented
effectively.
Undertake systematic ex-post evaluation for all transport infrastructure projects entailing expenditure above
an identified level
Ex-post evaluation can contribute significantly to improved appraisal methods. Increasing evaluation
activity, and standardising methods and processes, is essential to ensure these gains are realised.
Consider the merits of the permanent observatory model as a means of maximising the quality of evaluations
Meeting the substantial data requirements is a major challenge when conducting evaluations. The factthat much data is fleeting in nature, and difficult or impossible to obtain ex-post, underlines this problem.
Permanent observatories, established at project inception, appear to be a promising means of addressing
data and other quality control issues and enhancing evaluations of larger infrastructure projects.

10

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2020

THE OBJECTIVES OF TRANSPORT PLANNING

The objectives of transport planning
Transport policy and planning objectives are being fundamentally rethought in many OECD countries. A
central element of this shift is the increasing move to replace the traditional mobility focus of transport
planning with an accessibility-based perspective. The accessibility perspective recognises that transport isa derived demand and that the purpose of passenger transport policy is to enable people to reachdestinations to participate in various activities. As the ITF has previously argued, “mobility does not in itself
ensure accessibility, and can even detract from it. Greater mobility for some can reduce mobility andaccessibility for others. This can, in turn, lead to significant deterioration of certain aspects of the qualityof life and an inefficient use of economic and natural resources” (ITF, 2020a).
A core focus of the accessibility perspective is on providing equitable access, ensuring everyone enjoys an
acceptable minimum level of accessibility. This focus implies that the distribution of the benefits oftransport investment should be a central consideration for policy makers. In many significant domains ofgovernment intervention (e.g. health care, education and housing), delivering a particular good to citizens
is the overriding concern. Equity is a central goal of government policies, and policy interventions –including regulations, taxes and subsidies – are designed to deliver equity in each domain. There is a strong
argument that transport planning should adopt the same perspective, as adequate accessibility is afundamental requirement for achieving an acceptable standard of living (Shiftan et al., 2021).
Different dimensions of equity are relevant within an accessibility paradigm. Equity can be considered at
geographical and household levels and in terms of characteristics such as gender and disability. Addressing
the different dimensions of equity has a range of implications for transport planning goals. In particular,transport planners need to ensure that:•access to local services in regional areas is adequate and consistent, and that regional
connections to the national transport network are also adequate•transport links within metropolitan areas provide adequate accessibility for those without accessto private vehicles•

transport networks provide sufficient accessibility to address a wide range of journey types andpurposes•transport modes and infrastructure facilities are considered safe by all user groups•

transport modes and infrastructure facilities are accessible to all, including people with restrictedmobility and those with other disabilities that can limit accessibility.

The second major shift in transport planning is towards recognising that individual project appraisals often
do not result in the delivery of transport infrastructure investment programmes which, taken as a whole,
contribute effectively to achieving governments’ strategic transport objectives. For example, a major
finding of the recent review of the United Kingdom’s project appraisal guidance (HM Treasury’s GreenBook) was that appraisals frequently do not address proposals’ impacts on strategic government
objectives, including equity (“levelling up”) and climate change (HM Treasury, 2020b).

11

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE OBJECTIVES OF TRANSPORT PLANNING

This conflict between the outcomes of project-level appraisal and the need to optimise the performanceof larger investment programmes is becoming more apparent – and critical. Governments are increasingly
adopting strategic planning processes to guide the delivery of major infrastructure programmes, moving
away from practices that typically saw infrastructure investment decisions made in response to specificbottlenecks.
This disconnect between individual project appraisals and the strategic objectives of transport planning isclear in the context of the imperative need to address climate change. This imperative has two essential
dimensions in relation to long-term transport policy. First, it involves giving priority to the urgent need todecarbonise the transport system to meet international commitments to limit greenhouse gas emissions,and thus temperature rises. This implies a need for fundamental changes to current transport systems anda substantial modal shift away from private vehicles, notwithstanding the progressive electrification of thevehicle fleet. Second, it involves ensuring that transport infrastructure is sufficiently resilient to expected
increases in the frequency and severity of extreme weather events. Resilience issues must be addressed
when designing and constructing new infrastructure and in decision making on maintenance and upgrade
programmes for existing infrastructure (OECD, 2018).
Traditional appraisal processes and techniques have undergone significant evolution in recent decades. In
particular, as the ITF has recently highlighted (2021a), the external impacts of major transport projects
have increasingly been brought within the scope of appraisal through analysis of wider economic benefits.
These external benefits derive from the positive impacts on productivity of increased urban agglomeration,
which some investments in improved transport infrastructure can enable. In addition, cost–benefit
analysis (CBA) has evolved methods for quantifying, monetising and incorporating a wider range of benefits
and costs within the central analysis.
Despite these changes, significant further review and modification of appraisal practice is required toensure that it appropriately considers the broader policy objectives identified above. A central issue is thattraditional “mobility-based” CBA appraisals do not give due weight to the effects of transport
infrastructure choices on health, local and global environmental impacts, or quality of life factors such as
the aesthetic quality of locations in the public domain (ITF, 2020a). The underlying requirement is toaddress concerns that “[t]he disconnect between urban transport policy making and other policy areas,such as public health or spatial planning . . . [is] causing serious negative social impacts such as chronic
disease, premature mortality (Khreis et al., 2016), or community severance” (Shortall and Mouter, 2021).
Roundtable participants considered means of expanding and reforming the practice of transport appraisal
to ensure that the results generated support decisions that lead to the effective achievement ofgovernments’ strategic objectives. However, a major finding is that achieving this outcome requires widerchanges, extending beyond appraisal. These include changes in wider transport planning, decision making,and stakeholder engagement practices and processes.

12

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

Addressing equity, sustainability and otherstrategic objectives
Transport appraisal has long centred on the use of cost–benefit analysis (CBA). This reflects CBA’s
foundation in welfare economics and consequent ability to provide clear information on the relative
welfare impacts of different options and its ability to express outcomes in terms of clear, comparable
metrics. Given the political nature of many major infrastructure investment decisions, these are crucial
advantages. It can thus function as a countervailing power against the pursuit of projects for essentiallypolitical purposes (Mouter, 2017).
The traditional approach to CBA in transport appraisal adopts the aggregated value of travel-time savingsas the central benefit measure for infrastructure investments. Business travel is accorded higher values,
with little attention paid to the distribution of travel-time savings. CBA practice has developed
progressively, and the use of indirect valuation techniques, which provide monetary values for a wide
range of impacts, is now more common. Public policy objectives have broadened to include a greater focus
on environmental and social issues. However, CBA is still frequently criticised as unduly narrowly focused.
The broader range of policy objectives that transport appraisal must consider has increasingly raisedquestions about whether the CBA-focused approach remains appropriate and adequate. Transport
planning increasingly emphasises accessibility, and the size and distribution of accessibility gains havebecome central considerations. As the decarbonisation of the transport system becomes a pressing
priority, it is essential that the relative performance of different options against this criterion is accurately
analysed and the results conveyed clearly to policy makers. The impact of investments on the resilience ofthe transport system in the face of climate and other threats is increasingly critical. The transport
investment programme as a whole must also contribute to strategic objectives such as modal shifts that
support decarbonisation and urban liveability goals. In all of these areas, CBA-based appraisal continues
to be criticised. Its critics see it as providing insufficient information on the relative performance ofdifferent potential investments across the expanded range of relevant objectives.
Some transport planners argue that broadening the goals of transport planning necessarily increases
misalignment between people’s personal choices and their preferences regarding the allocation of
government transport funding. For example, Manaugh et al. (2015) observe that for most of the 20th
century, transport objectives were almost entirely mobility-based. The associated goals of congestion
reduction, travel-time savings and safety improvements for motorists were relatively easy to value byobserving people’s (hypothetical) private consumer choices. In this context, CBA captured all the essential
goals of transport planning identified by policy makers at that time. However, urban transport projects
now pursue both traditional goals (e.g. costs, travel-time savings, safety and reduction of noise pollution)
and a wide range of non-traditional goals. Examples include long-run sustainability, urban liveability,
decarbonisation, resilience and social equity. Many authors argue that CBA is far less well-equipped toaddress these non-traditional goals, which are either not included in CBA or under-valued because they
are not fully quantified or monetised (Shortall and Mouter, 2021).

13

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVESIn response, many jurisdictions require one or more supplementary analyses as part of standard appraisal
processes. The use of multiple analyses raises the issue of presenting the results coherently andsystematically to effectively inform decision makers. A common approach is to use multi-criteria analysis(MCA), which usually involves applying explicit weights to performance scores across different policy
objectives to reflect views of their relative importance. However, as discussed below, MCA is often
criticised for arbitrariness in setting weights. It is also sometimes criticised because presenting a single
score as the outcome of the analysis can reduce transparency. This is arguably a criticism of common MCA
practice rather than an intrinsic weakness of the methodology, as it is also possible to draft MCA reportsthat present the entire matrix of impact scores and weights, thus providing a highly transparent
information set.
Some critics argue that, even where MCA is adopted, CBA often continues to take a predominant role indecision making. For example, Shiftan et al. (2021): argue that “the practice of MCA usually includes CBA
as the main criterion with often little attention for other criteria in the policy and political discussions ontransport investments”. Di Ciommo and Shiftan (2017) argue that “[e]ven when [equity] is at the core of
the appraisal, the utilitarian approach at the base of this evaluation disables a proper consideration ofequity”.
Ensuring that project appraisal, and ultimately project selection, adequately address the expanded rangeof policy priorities requires appraisal techniques to be adapted, expanded or supplemented. Broadening
appraisal is essential to ensure that all dimensions of the benefits sought through transport investments
are included in the analysis and appropriately weighed, including by using scenario analysis to addresssignificant uncertainties.
Roundtable participants identified and discussed several improvements to standard appraisal techniques,some already relatively well-established and others less developed. Improvements canvassed included
greater rigour in addressing the “needs case”, which informs the threshold decision as to whether
investment is required in a given context. However, a core finding is that broader changes to transport
planning processes are needed to account for recent shifts in transport policy objectives.

Modifying cost–benefit analysis
While indirect valuation techniques have enabled an increasing range of environmental and other values
to be monetised and included in the CBA, a major concern remains over CBA’s ability to compare the
distributional impacts of potential investments. The increased focus on the accessibility perspective has
made this issue especially acute.
Bonnafous (2021) notes a marked contradiction between equity and profitability in most transport
investments. This contradiction is due to increasing returns to scale, which implies that the most profitable
investments are those undertaken along heavily used axes and in regions that generate high traffic levels.
Yet these areas already tend to enjoy high initial levels of accessibility. Indeed, Bonnafous and Masson(2003) have proposed a formal proof of the contradiction. Policy makers must clearly understand theequity impacts of investment options. This is essential to enable them to understand the nature and extent
of the equity/efficiency trade-offs and make informed choices between project options.
Several other modifications to CBA have been proposed, and some implemented to greater or lesser
extents. Some enable CBA to address equity issues by changing the weights given to impacts on different
groups to reflect policy makers’ equity objectives. Many involve departing from values based on willingness
to pay (WTP) for benefits such as travel-time savings. This is because WTP-based values necessarily favour

14

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

projects that predominantly benefit higher-income groups, whose greater ability to pay typically yields
higher WTP estimates (Martens, 2006). As Shiftan et al. (2021) note, this implies a deviation from thewelfare-theoretic underpinning of CBA. However, Jara-Diaz (2007) argues it is consistent with people’s
general public preferences to give a higher weight to the interests of low-income groups when considering
potential social expenditures.
One alternative to WTP-based valuations is to apply a uniform “equity” value of time, to all users of the
proposed service. Another is to use reverse weightings, which ascribe higher monetary values to benefits
accruing to relatively disadvantaged groups and lower values to benefits reaped by relatively advantaged
groups. Doing so increases the likelihood that projects providing more significant benefits to deprived
groups will be selected. The primary rationale for this approach derives from the economic concept ofdiminishing returns. That is, if accessibility is considered as a good or service, a given increase inaccessibility will be less highly valued by someone who already enjoys high levels of accessibility thansomeone who does not.
Critics argue that adopting such weights is insufficient to address equity issues effectively and ensure
adequate standards of accessibility for all. For example, Martens and Di Ciommo (2017) argue that lowincome groups, people with limited mobility and other groups characterised by small numbers of realised
trips and low travel-time values effectively remain invisible if CBA approaches based on measuring thevalue of travel-time savings remain central to project appraisal.
Nahmias-Biran and Shiftan (2016) propose a method for addressing this issue, based on a measure of
subjective well-being (SWB). As low-income groups gain a greater increase in well-being from a given
accessibility improvement, the SWB measure, which the authors call “the subjective value of accessibility”,
compensates for income differences by including subjective well-being factors. Shiftan et al. (2021) note:“By applying this in an ‘equity benefit analysis’ as a variation on a pure welfare-based cost-benefit analysis,
projects that direct accessibility gains to population groups with relatively low levels of accessibility or
relatively low incomes are likely to perform better than projects that would do the opposite.”
Di Ciommo (2021) highlights another potential approach to integrating equity considerations into CBA.
This involves replacing the use of a single benchmark discount rate, as recommended in most
governments’ CBA guidance documents, with a range of rates that reflect the rate of time preference ofthe different groups affected by the proposed project. Providing decision makers with CBA results
calculated using both approaches to discounting demonstrates the impact of including this measure ofequity considerations. However, this approach also introduces several difficulties. One is that it appears to
imply a presumption that the appropriate conceptual basis for setting the discount rate is that of the social
rate of time preference. By contrast, many countries’ CBA guidance material argues for (or requires) theuse of an opportunity cost of capital-based approach. Moreover, while the conceptual basis for setting
discount rates and the specific rates recommended varies, most appraisal guidance documents argue forconsistent approaches to discounting across sectors and project contexts. This emphasis reflects the factthat the choice of the discount rate should be determined by a broader range of factors than those relating
to the assessment of performance against specific objectives in a given sector (OECD, 2009).
Some criticisms of CBA question the reliability of revealed preference-based valuations as an accurate
gauge of citizens’ preferences in a public policy context. Recent work in welfare economics has also
addressed these criticisms, adopting the view that people may have different preferences as citizens,considering broader social welfare, than those they express in their roles as consumers. Empirical researchsupports this view, showing that individuals value the impacts of transport projects (e.g. travel-time savings
and accident risk) differently when assessed against the public budget versus their private budgets(Mouter et al., 2017, 2018, cited in Shortall and Mouter, 2021).

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

15

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

Participatory value evaluation (PVE) and deliberative monetary valuation (DMV) are two examples of thiswork. Proponents of PVE and DMV propose these methods as complements, rather than alternatives, to
CBA.
In PVE, individuals receive information about the impacts of several possible public projects and aconstrained public budget in an experimental context. The exercise gathers data on their preferences forallocating scarce public resources (Mouter et al., 2021) rather than inferring these preferences based on
their private expenditure choices, as is done in traditional CBA. Evaluators can then rank these projects bydesirability (Shortall and Mouter, 2021). PVE potentially enables individuals to express a broader range ofpreferences about government policies than existing methods (Mouter, Koster and Dekker, 2019).
DMV “raises concepts of social willingness to pay and accept which are distinct from an aggregate ofindividual values” (Spash 2007). In this approach, differences in valuation result from individuals adopting
different valuation processes in group and individual contexts (see also Shortall and Mouter, 2021). DMV
seeks to recognise the importance of social interaction in preference formation by asking individuals to
express their preferences after exposure to one or more deliberative mechanisms. Examples include group
discussions, consulting expert witnesses, or a forum. This deliberative aspect allows citizens to learn fromeach other, form reasoned opinions and evaluate positions. It also addresses critiques of the individual
approach traditional CBA takes to preference formation (Kenter et al., 2016).
Both methods seek to amend standard welfare economics. However, there is little experience with theirimplementation to date. Hence, their practical ability to address critiques of CBA remains unknown(Shortall and Mouter, 2021). Moreover, both draw conclusions based on people’s statements of theirpreferences in hypothetical situations. While this potentially makes it possible to gather data on a broader
range of issues, the reliability of the results depends on the quality of the hypothetical scenarios. For thisreason, the results of stated-preference studies are generally considered less robust than those based on
analysing actual consumer behaviours. This distinction argues for caution in using these emerging
methodological tools.
Some critiques of CBA challenge the utilitarian perspective underlying the welfare economics paradigm.
For example, Van Wee et al. (2014) argue that utilitarianism may not address crucial moral dimensions oftransport policies, such as freedom of choice or alleviating inequality. Thus, several proposed alternative
or complementary analytical tools (e.g. some deliberative approaches) are not grounded in welfare
economics.
Another criticism of attempts to address equity issues within CBA lies in the relative lack of transparency
that these approaches entail, given the aggregative nature of CBA. Even where equity-based weights areapplied, decision makers do not necessarily receive information about the values adopted. This makes it
difficult or impossible for them to compare the performance of different options on this specificdimension. The weights used to derive these equity values are also frequently poorly justified. Projects
with poor distributional outcomes may still score more highly overall (i.e. in net present value or benefit–
cost ratio terms) but decision makers may remain unaware of this unless they receive information on
distributional impacts alongside aggregated CBA outcomes (Shiftan et al., 2021).
These shortcomings arguably make using a modified CBA to address equity issues a poor solution, at leastin contexts in which distributional impacts are central to the policy objectives. Moreover, if multiple policy
objectives are relevant to the analysis, choosing the weights to apply within the CBA can becomechallenging. In practice, identifying the impacts of proposals on specific groups in unweighted terms, anddiscussing their different distributional impacts alongside the aggregate CBA results, remains the morecommon means of addressing equity issues in CBA. This method is then supplemented by a qualitative
discussion highlighting the nature and extent of the equity/efficiency trade-offs between proposals.
16

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVESFor example, the guidelines issued by the Netherlands Bureau for Economic Policy Analysis (2013) statethat if the distributional effects of proposed investments are substantial, the CBA should clarify these
distributions across societal groups, so that policy makers can determine what weight to give these effects.
As Gomijn and Renes (2013) point out: “The outcome of this exercise can then provide a basis for deciding,for example, that a measure that delivers a negative rate of return should still go ahead because of its
distributional effects. Conversely, CBA can be used to rank measures intended to bring about a morefavourable distribution of income or welfare according to their economic welfare impact.”
This approach improves the transparency of the various weighting approaches by presenting informationon the impacts on individual groups in a disaggregated format. In practice, this means elected policy
makers rather than officials assign these weights. Conversely, there is arguably also a loss of transparency
because the weights policy makers apply in deciding between proposals are necessarily implicit. Where
multiple distributional impacts (or claims) are involved, it may be challenging to understand what
distributional claims have been weighed, and to what extent.
However, the fundamental challenge is to include distributional issues in the analysis and the policy
process. In practice, if accessibility is not an explicit part of transport policy objectives, this (rather thanmethodological limitations or divergences of view) may be why CBA fails to address distributional issues
adequately.

Cost–benefit analysis and climate impacts
CBA’s ability to address the implications of transport investments for climate change imperatives has also
been the subject of much discussion. Theoretically, in a perfectly functioning market economy, the pricesof goods and services internalise the total cost of producing them, including environmental costs. This
assumption also implies that the price attached to greenhouse gas emissions is sufficiently high to preventthe devastating consequences of climate change. However, emissions are often not priced; and where
taxes and other interventions establish effective carbon prices, these have invariably been too low toprovide the needed incentives.
Transport appraisal methods have been adjusted to account for these shortcomings via efforts to calculate
shadow prices of carbon, and apply them in CBA calculations. This relies on the feasibility of identifying
robust shadow carbon prices that are widely accepted as reflecting the full costs of damage from, or ofadapting to, climate change. This is a dynamic concept: CBA for long-term projects must incorporate
credible future values for shadow carbon prices.
Recent developments have raised significant issues in this regard. CBAs have, in the past, typicallyincorporated shadow carbon prices based on those observed in carbon markets, notably the European
Union’s Emissions Trading Scheme. However, these prices depend on how much carbon is defined asavailable to buy – a figure determined by political negotiation – and the market conditions in industries
that buy them. Neither are fixed in relation to global carbon targets. Prices on the European carbon market
have remained low, despite rapid increases in recent years. From around EUR 5 per tonne before the 2018
reforms, they rose an order of magnitude by mid-2021 (to approximately EUR 56/tonne on 1 July), andwere predicted to reach over EUR 90/tonne by 2030. Nonetheless, the combined effects of these prices inthe traded sectors, and the use of similar prices in cost–benefit studies in the non-traded sectors (including
most of transport) will not have sufficient effect on decision making to enable climate change targets, andthe carbon emission targets they imply, to be met.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

17

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

Carbon-emission targets increasingly reflect the perspective of the maximum stock of atmospheric carbon
that is consistent with meeting policy goals on temperature increases rather than the flow of annual
emissions. This approach reflects, in particular, the scientific work of the Intergovernmental Panel on
Climate Change (IPCC), as reported to the 26th United Nations Climate Change Conference of the Parties(COP26) in Glasgow in 2021. It implies significantly higher shadow prices, given the speed with which
concentrations of CO2 in the atmosphere will exceed maximum target levels were the current rate ofemissions to be maintained.
Reflecting this, recently revised CBA guidance in some countries has adopted much higher shadow prices
for CO2 for appraisal purposes. For example, in 2015 France adopted carbon values rising fromEUR 56/tonne in 2020 to EUR 100/tonne in 2030 (Carbon Pricing Leadership Coalition, 2017). Most
recently, revised editions of the UK Treasury’s Green Book and associated guidance documents (BEIS
2021), published following a 2020 review, adopted a “central” scenario CO2 valuation for 2021 of
GBP 241/tonne. This rises to GBP 280/tonne by 2030 and GBP 378/tonne by 2050. A “high” scenario valueof over GBP 500/tonne (see Figure 1) is also included.
Figure 1. Government guidance on carbon values for use in project evaluation, United Kingdom

Source: UK Department for Business, Energy and Industrial Strategy (2021).

The UK Department for Transport has adopted this pricing guidance for transport appraisal and extended
the series to the year 2100. By that time, the high-scenario carbon price approaches GBP 1 000/tonne (at2010 prices). The current guideline value for 2021 emissions is thus more than an order of magnitude
higher than the 2018 value, and the guideline value for 2030 emissions is 3.5 times the equivalent value inthe 2018 edition (BEIS, 2018). The guidance also requires the use of low- and high-price scenarios. The
high-price scenario now includes prices of GBP 361/tonne in 2021, rising to GBP 568/tonne by 2050 (BEIS,
2021).
18

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVESThe updated shadow carbon prices in France and the UK seek a target-consistent marginal abatement costto achieve agreed global targets aiming to limit the average global temperature increase to 1.5 degrees
Celsius (°C). These targets imply net-zero carbon emissions by 2050, with an interim target of a 78%
reduction in emissions by 2035 compared with 1990 levels. The scientific literature shows some variationin shadow prices calculated as meeting these conditions. However, there is broad agreement. The 2030
value in the UK’s Green Book is marginally lower than the IPCC median cited result (GBP 147/tonne versus
GBP 163/tonne). The 2050 value, meanwhile, is much higher than the corresponding IPCC median(GBP 576/tonne versus GBP 326/tonne), while still within the interquartile range of IPCC carbon values.
The most recent work by Stern and Stiglitz (2021) also supports the use of significantly higher carbon
prices. Like the BEIS guidance, it points to an increasing focus on the target of keeping temperature
increase to 1.5°C and the widespread adoption of “net zero by 2050” commitments as major drivers of
this conclusion. The authors note that the report of the 2017 Commission they chaired (Carbon Pricing
Leadership Coalition 2017) recommended carbon prices for 2030 of USD 50–100/tonne but argue in their2021 paper that subsequent tightening of international targets would justify adopting a figure at the topof this range. They also cite Kaufmann et al. (2020), who derived a carbon price of USD 125/tonne for 2030
based on a social cost of carbon methodology, noting that “The two approaches (exemplified by SternStiglitz and Kaufman et al.) are different but in either case the numbers likely to emerge would be more in
the region of USD 100 per ton by 2030 rather than the USD 50 per ton (in 2007 prices, USD 60 in 2018
prices) estimated by the Interagency Group in the Obama administration” (Stern and Stiglitz, 2021).
In sum, while carbon values adopted for policy appraisal purposes continue to vary, there is a clear trend
towards applying significantly higher values. This trend reflects greater appreciation of the urgency ofclimate action and the consequent adoption of more specific and challenging abatement targets. It also
recognises that rational policy requires a risk-averse approach in the face of severe or existential risks(Stern and Stiglitz, 2021).
These trends are consistent with a shift in transport planning perspectives from a “predict and provide” toa “decide and provide” paradigm – that is, one based on identifying preferred future outcomes and taking
policy decisions to support their achievement, as previously recommended by the ITF (2021b). The
Nationally Determined Contributions adopted in response to climate-change targets agreed at successive
UN summits constitute the required policy outcomes, with shadow carbon prices set at levels consistent
with their achievement.
Changes in carbon pricing of this extent and rapidity will likely lead to large changes in the rankings of setsof proposed transport infrastructure projects. With higher shadow carbon prices, investments in lowcarbon modes become more attractive, in relative terms, than increases in road capacity to cope withmore travel by high-carbon modes. This, in turn, indicates that traditional CBA-based appraisal methods
can be made consistent with the achievement of the significant modal shift and changes in traffic volumes
that the strategic policies of ITF member country governments require.
However, this will only occur if appropriate values for major variables such as greenhouse gases are usedconsistently. Appraisals must include and quantify all relevant impacts (e.g. the health impacts of increased
use of active transport). Of course, cost–benefit calculations alone will not reduce carbon impacts.
Decisions taken need to be consistent with those calculations. This is not always the case, as otherconsiderations may cause decision makers to proceed with investments that undermine the achievementof the targets.
A second methodological issue relevant to climate change and other long-term environmental policy issues
is the appropriate use of discounting in CBA. The OECD has previously discussed the argument that theapplication of discount rates favours short-term policy-making perspectives. It noted that an increasing
BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

19

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

number of appraisal guidelines recommended using lower discount rates for projects with very long time
horizons. However, different guidance documents provided diverging justifications for this move. These
included the need to account for the preferences of future generations; expected declines in per-capita
income growth rates; and the need to acknowledge increasing uncertainty about appropriate discountrates in the distant future (OECD, 2009).
Some guidelines conclude that discounting is not the preferred way to address the conceptual issues
associated with assessing impacts over long time horizons. In the UK, the BEIS supplemented its recent
review of the Green Book guidance with a purpose-specific review of the application of discount rates tolong-term environmental impacts. The review engaged with a range of academic experts to determine
whether there was a theoretical justification for applying a lower discount rate for projects withenvironmental impacts.
The resulting report (HM Treasury, 2021) found that, on balance, a differentiated discount rate should not
be applied to environmental impacts. It reasoned that discounting constitutes an imprecise way ofaccounting for effects such as lack of substitutability and increasing relative scarcity. It concluded that theexisting Green Book methodologies could instead be modified via relative price adjustments and increasing
values over the appraisal period to better address these issues. It noted that these conclusions wereconsistent with those of the Treasury’s review of the economics of biodiversity (Dasgupta, 2021). 1
The complexities involved when addressing the issues of shadow carbon prices and discount rates suggestthat addressing climate change-related impacts continues to pose significant problems for CBA. However,the conceptual framework of CBA can accommodate the costs and benefits of carbon and contribute tosuccessful climate-change policies.
That said, while CBA is well placed to analyse the aggregate benefits and costs of climate-related policies,
much policy making now focuses on the distributional dimension of climate change. Climate change isincreasingly being understood as having a significant social justice dimension, as it becomes more apparent
that many of the most severe impacts will fall on the most vulnerable populations. This recognition has
several aspects. Many of the geographical areas that will be most heavily affected by changes, including
sea level rises and extreme weather, are in the Global South. In many developed country contexts, lowincome groups are more vulnerable to climate change impacts because of living in lower-quality housing.
At the same time, the aged are more susceptible to the health impact of extreme heat exposures. Lowerincome groups (e.g. tenants as opposed to owner-occupiers) are also made more vulnerable by being lessable to fund and undertake mitigation actions (Preston et al, 2014).
The importance of this dimension of climate policy suggests that applying additional analyses to generatedetailed information on the incidence of climate impacts on groups of concern will also be important. Thiscan help to inform decision makers adequately on the social justice implications of different policy choices.

Integrating cost–benefit analysis and complementary analytical
tools
Many government project appraisal processes now require additional analytical tools to be applied inparallel with CBA. These seek to better assess impacts that CBA is seen as unable to address adequately.
They include a variety of accessibility measures, environmental impact analyses and assessment of thegender impacts of different investment options (see Box 1).

20

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

Box 1. Incorporating accessibility analysis in transport appraisal in Israel
In response to Israel’s 2012 national plan for developing public transport, a group of government
ministries, experts and consultants collaborated to produce a strategic plan for public transport in the
country’s largest metropolitan area, Tel Aviv.
The methodology combined strategic, long-term tools, more detailed transport planning, and network
simulations using the Tel Aviv activity-based transport model. The plan included a shift away from thecurrent road-dominated model and towards a high level of metro use. The three categories of strategic
goals were (a) level of service (including accessibility, system and service coverage, and speed), (b) levelof transit usage and (c) level of investment required. The design chosen for the transit network reflected
these strategic goals, demand characteristics, current and expected future land use, and mass-transit
systems’ shape in similar cities.
Policy makers in the Mass Transit Committee (MTC) then set the goals, criteria and weights needed toenable the conduct of an MCA to assess the identified alternatives. The four main criteria considered in
the analysis, and the weights assigned to each, were transport (40%), economic criteria (30%), qualityof life and the environment (20%) and feasibility (10%). Equity was analysed separately as a uniquedimension in evaluating the plan, with a focus on accessibility.
The impact of the strategic plan options was analysed using a travel-demand model that divided themetropolitan area into 1 310 zones. Demographic data was produced for each zone (e.g. the number ofhouseholds, population size, gender and age distribution, car ownership and employment). Long-term
spatial and housing plans were then used to forecast these zonal attributes for the target years 2030,
2040 and 2050.
This forecasting enabled an estimate of the change in accessibility for each zone for each target year. In
addition, changes in accessibility levels were calculated for each socio-economic group. These
calculations showed that implementing the plan would yield major accessibility gains for all zones servedby the proposed lines. These gains were substantially higher for the lowest socio-economic groups.
Policy makers selected a radial transit network based on three subway lines and three light rail lines.
Three rapid-transit bus lines complement this network, forming an outer ring and providing local service
on the edges of the urban area. Figure 2 summarises the accessibility gains. It shows that, following theplan’s implementation, all socio-economic groups will have a level of accessibility that meets or exceeds
the target level of 50% of destinations within 45 minutes. Figure 3 shows the difference in the
proportion of destinations reachable, on average, by each socio-economic decile due to the plan. The
levels of accessibility are very similar for all groups, in contrast to the current position.
Additional equity analysis included a comparison of Lorentz curves and Gini coefficients for the pre- andpost-metro cases. Thus, the analysis demonstrates the nature and extent of equity and accessibility
benefits. Transport appraisers often present this type of analysis to decision makers alongside CBA
results. Thus, CBA remains influential, especially if governments have adopted threshold values thatmust be met (e.g. positive net present value, minimum benefit–cost ratio) before project approval.
However, providing a separate accessibility analysis enables a comparison of the size and incidence of
the accessibility gains with the results of a traditional CBA. This provides transparency to decision
makers about the nature and extent of any equity/efficiency trade-offs. By contrast, incorporating
equity weights within the CBA is likely less transparent. See Shiftan et al. (2021) for additional details.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

21

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

Figure 2. Accessibility maps: before and after the proposed transit system in Tel Aviv

Note: The Grey lines in the right-hand map are metro routes. Areas with darker shading have greater
accessibility.
Source: Shiftan et al. (2021).

Figure 3. Accessibility change by socio-economic decile, Tel Aviv

Source: Shiftan et al. (2021).

22

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVESThe OECD has documented similar developments in the appraisal of proposed regulation. In this context,
many countries have supplemented CBA requirements with mandates to address specific policy issues ofconcern explicitly. Examples include the impacts of regulation on competition, trade, small business,
administrative burdens, the environment, sustainable development, regional development, gender equity,
poverty, and native populations.
The main rationale for adopting most of these additional requirements is the concern that traditional CBA
does not adequately address crucial distributional questions (OECD, 2009). However, this development
raises the issue of integrating the results of these analyses to provide a coherent information set todecision makers. The OECD has previously identified the risk of losing analytical coherence as a core
concern when adopting multiple complementary analyses (OECD, 2009).
Multi-criteria analysis

Shiftan et al. describe multi-criteria analysis (MCA), with CBA results incorporated as one dimension of theanalysis, as “the main alternative approach” to sole reliance on an expanded CBA. They also identify themain difference between the two approaches: “In contrast to cost–benefit analysis, multi-criteria analysis
does not express all the effects in one dimension (money) but uses several dimensions concomitantly”(2021). They argue that, unlike CBA, MCA is open-ended and can absorb additional criteria.
MCA can be used to assess the impacts of proposed investments in terms of regional, social and genderdimensions of equity. In relation to social equity, it can incorporate various indices of income inequality,
such as the Gini or Palma indexes, to provide different perspectives on the distributional consequences ofpotential projects. Including a broader range of equity indicators in the analysis may lead to this criterion
being weighed more heavily overall by contributing to a better understanding of the specific equity issues
at stake and their extent.
Shiftan et al. (2021) also cite the composite Sustainable Mobility Inequality Indicator (SUMINI), created byThomopoulos and Grant-Muller (2013). SUMINI uses an MCA approach to quantify a range of equity
impacts of infrastructure investments and incorporates criteria weights derived from stakeholders as partof an analytic hierarchy process. Also of note is the “transportation equity scorecard” approach of Williams
et al. (2020), which provides a framework to address equity during project screening and ranking. Theybase their approach on a review of the literature and practice in integrating equity factors into project
evaluation and priority setting. It includes a range of equity-related criteria and makes it possible to ascribe
weights to each after consulting with relevant stakeholders.
The practical impact of MCA depends on how planners communicate their results to decision makers andstakeholders. Shiftan et al. (2021) identify three broad approaches:
1. adopting criterion weights to reflect the relative political priority accorded to each impact andsumming the weighted scores to obtain an overall performance measure, enabling a single
ranking of the options
2. presenting policymakers with a matrix of impacts, without summing the scores across thedifferent criteria, to provide a richer database to inform decision making
3. providing a set of unweighted scores for the various options across each of the performance
criteria, enabling different stakeholders to determine their own weights (reflecting their own
priorities) and thus their own rankings of project options, facilitating a political decision-making
process.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

23

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVESThe use of these approaches partly responds to the difficulties of assigning criteria weights as the basis forderiving an aggregated result. As suggested above, stakeholders’ views on appropriate weights may differ
significantly. Thus, there can be tension between producing an integrated analysis and achieving a highlevel of acceptance of the analysis by stakeholders.
Shiftan et al. (2021) argue that, regardless of the approach taken, an MCA which includes CBA as one
dimension of the analysis provides substantially more information to decision makers than an expanded
CBA. This greater transparency also provides the basis for broader engagement on priorities and choices.
However, MCA has long been criticised for bringing subjectivity into the analysis and lacking transparency.
These criticisms centre on how MCA determines weights and scores and whether these elements aretransparent in the published MCA and thus able to be analysed and criticised.
The allocation of criteria weights is open to manipulation by proponents of particular options (OECD,2009). Subject-matter experts and focus groups consulted about weighting and scoring impacts often selfselect (if sent a general invitation) or belong to specific interest groups. This results in a risk that interest
groups and special pleading have an undue influence over MCA results (Dobes and Bennett, 2009).
Reflecting these concerns, some government analytical guidance limits MCA use to specific circumstances.
For example, the UK Green Book explicitly rejects the use of simple MCA “because of its lack of transparent
objectivity” (HM Treasury, 2020a). Instead, it accepts only the use of a more sophisticated variant, which
it calls multi-criteria decision analysis (MCDA), stating that “MCDA should not be confused with simple
weighting and scoring, sometimes referred to as Multi Criteria Analysis (MCA). This latter is not arecognised Green Book approach because of its lack of transparent objectivity.” Moreover, the use of
MCDA is only authorised when considering unmonetised trade-offs at the longlisting stage of project
identification and not as a substitute for CBA in appraising shortlisted proposals.
The Green Book (HM Treasury, 2020a) also highlights the need to give significant attention to quality
control: “To work effectively MCDA requires top level decision makers, senior experts and stakeholders tobe assembled in a workshop, facilitated by an independent expert facilitator experienced in MCDA, andthe use of swing weighting.” This cautious approach to using MCA differs markedly from earlier editionsof the Green Book. The 2003 edition described “weighting and scoring (sometimes called multi-criteria
analysis)” as the most common technique for comparing unvalued costs and benefits. It provided
substantial guidance on how to adopt the method and references a detailed manual (HM Treasury, 2003).
The current sceptical approach perhaps reflects a reaction to problems encountered with the prior use of
MCA in practice.
Other critiques of MCA are more fundamental. For example, Dobes and Bennett (2009) argue that MCA
breaches the principle of dimensionality by attempting to compare unlike properties on a common scale.
They note that this problem is compounded when MCA effectively multiplies cardinal numbers (thequantities of whatever is being measured) by an ordinal scale (the “score” allocated on each criterion) andan interval scale (the weightings used), then treats the result as a cardinal number that can be added toother cardinal numbers.
In sum, MCA suffers from its own methodological limitations. Like CBA, it requires significant inputs ofexpert resources to ensure a high-quality result. Thus, while MCA can potentially provide a framework for
integrating various analytical tools, its effective use requires detailed guidance, transparency provisions
and independent oversight. Shortall and Mouter (2021) argue that while complementary tools add value
by addressing one or more critiques of CBA, no single method will address all of them, while none can
include all aspects of policy appraisal (environmental, economic, social, ethical). They conclude that further
research is needed into how the different methods can be effectively combined and their respective
strengths harnessed. Preparing a summary appraisal document for decision makers highlighting the results
24

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVESof the CBA and supplementary analyses will likely communicate vital considerations more effectively. Sucha document is also more transparent than a single MCA result or score.

Addressing the gender dimension of equity
In addition to the regional and social dimensions of equity, there is increasing recognition of the need toaddress access for people with reduced mobility, which has been discussed in several recent ITF reports(e.g. ITF 2020a, ITF 2017), and the gender dimensions of transport demand (e.g. ITF and FIA Foundation,2022). The ITF has recently published a Gender Analysis Toolkit for Transport (see Box 2), which considers
gender issues in transport planning in the analysis of infrastructure investment options and women’s rolesin the transport workforce. The toolkit provides readily usable tools to enable transport planners toconsider gender issues and identify the gender-specific data necessary for the analysis, design, and datacollection processes and create indicators of the level of gender equity.
Di Ciommo (2021) highlights major data needs for addressing the gender dimension of equity. This includes
data on the breakdown of trip numbers by purpose; differences in the distribution of trips by purpose
between genders; and differences in attitudes to the use of different modal and service type choices
between genders. Di Ciommo also highlights crucial differences in transport usage between genders basedon data from two very different urban contexts in Spain and the United States. These show that the highest
proportion of trips relate to caring activities (over 40% of the total in both cases), while women undertakea disproportionate share (two-thirds) of caring-related journeys.
Conversely, the orientation of transport planning leans strongly towards commuter demands, where men
undertake higher numbers of trips. A re-orientation of transport planning to better cater to unmet needs
for caring-related trips would imply a greater focus on local-level improvements (e.g. making areas morepedestrian and cycling friendly), rather than optimising large-capacity axes. It would also have considerable
potential to address the current bias towards the needs of men in transport planning.
Different patterns of trip purposes constitute one explanation for the observation of significant genderbased differences in the propensity to use different transport modes, which reflect personal choices andaccess. Recent data show that women are more likely than men to use public and active transport and lesslikely to use private vehicles. Women also have a less positive attitude to shared mobility options, greater
wariness of new and unproven technologies and a less positive attitude toward autonomous vehicles (DiCiommo, 2021).
These differences necessarily lead to gender-based differences in the pattern of benefits associated withdifferent transport investment choices. Decision makers increasingly seek to account for these differencesin project selection and subsequent evaluations. An example relates to the M-30 road tunnel project inMadrid. Analysis showed that women, who made less use than men of private transport, would benefit
less from the project. Furthermore, their greater use of active transport meant they were more exposedto the negative pollution impacts generated by the proposal.
However, Di Ciommo argues that effectively addressing gender inequities in transport infrastructure
programmes requires a broader approach. In particular, transport planners should incorporate diversity,
equity and inclusion (DEI) in the initial stages of the planning process and consider broader issues such as
the organisation of transport services and the design of public space from the DEI perspective.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

25

ADDRESSING EQUITY AND SUSTAINABILITY OBJECTIVES

Box 2. The ITF Gender Analysis Toolkit for Transport
The ITF Gender Analysis Toolkit for Transport provides a straightforward method for incorporating agender-inclusive perspective into transport projects, plans and policies. It offers governments,
international organisations, contractors, and all those who design, manage, implement or evaluate
transport projects three uncomplicated tools for carrying out their own gender analyses.
ITF’s approach to gender analysis aims to answer three questions:
1. How are travel behaviours and patterns of women and men affected by their social roles andthe level of accessibility of transport services?
2. How will transport policies, programmes and projects affect women and men differently?
3. How will greater gender equality in the transport workforce enhance transport infrastructure,systems and modes to benefit women and other users?
The Gender Analysis Toolkit for Transport helps ITF member countries and other stakeholders identify
the gender-specific data necessary for the analysis; create essential indicators to determine the level ofgender equality; and design the data collection processes, including surveys.
The Toolkit comprises three tools:
1. The Gender Checklist makes it easy to assess a project’s gender-inclusivity and createstransparency by providing a simple Gender Equality Score as a benchmark.
2. The Gender Indicators help project leads and policy makers to select the metrics most suited tomeasure gender equality in their project and to identify relevant data.
3. The Gender Questionnaire offers a ready-made template to design surveys and data collection
processes.
Together, these three tools offer practitioners a simple yet comprehensive set of tools to advancegender neutrality in transport.
Access the Toolkit online: www.itf-oecd.org/gender-toolkit.

26

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

IMPROVING APPRAISAL THROUGH EX-POST EVALUATION

Improving appraisal through ex-post evaluation
Evaluating a completed project (ex-post evaluation) is a widely acknowledged means of improving
appraisal practice. It can help identify systemic biases and unanticipated impacts, thus providing a basis
for enhancing ex-ante appraisal methodologies. However, a range of challenges must be understood andaddressed for the potential benefits of ex-post evaluation in improving appraisal to be fully realised inpractice.

Reviewing evaluation outcomes
Nicolaisen (2021) reviews the literature on ex-post evaluation to identify major conclusions and lessons
for the design and practice of ex-ante appraisal. Despite studies pointing to the potential benefits ofconducting systematic project evaluations and assessing the results against ex-ante appraisals, in practice,
only a relatively limited number of evaluations occur. Citing Laird et al. (2012), Nicholson notes thatmandated public-sector evaluations rarely receive rigorous academic attention. Existing studies typically
include only a small number of observations, raising questions of availability bias, and have suffered fromother methodological difficulties, which limit their usefulness. A further issue is that a completed project
often differs from the proposal subjected to ex-ante evaluation.
However, Nicolaisen also argues that, despite these shortcomings, existing studies provide valuable
insights. These insights have implications for ex-ante appraisal and transport planning practice. The
literature on evaluation also includes studies by national auditing bodies, parliamentary committees andmultilateral lending institutions, including the World Bank. Surveying this literature, he identifies fiveconclusions on improving ex-ante appraisal practice.
1. Construction costs are underestimated (as is the case for most larger projects). This systemic bias
appears to have several causes, including unforeseen construction obstacles, especially in relation
to underground projects; design alterations, often resulting from ongoing discussion of theappropriate trade-offs between different objectives; optimism bias among project proponents;and strategic misrepresentation of expected costs by project proponents to increase the likelihoodof approval.
2. Demand forecasts are biased (the mean inaccuracy is non-zero). Inaccuracy in demand forecasts
varies widely between evaluations, but systemic biases are evident. This is true of road projects(tolled and untolled) and rail projects. Build (i.e. “do something”) options typically underestimate
demand for new, untolled roads, while “do-nothing” or base-case scenarios overestimate thatdemand. By contrast, demand for new rail and toll road projects is typically over-estimated.
Typically, fewer passengers and vehicles than predicted use these services on completion (Bain,
2009). Over-estimation of demand for toll roads and rail projects implies the net benefits of theseprojects will tend to be lower than ex-ante appraisals suggest. However, the situation in respectof non-tolled roads is more complex. Under-estimation of demand in the “do-something” option
may imply that actual project benefits are higher than anticipated: this will be the case if capacity
27

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

IMPROVING APPRAISAL THROUGH EX-POST EVALUATION

constraints are not reached. However, project benefits may be lower than forecast if capacity
constraints occur and significant congestion results. Similarly, over-estimation of the level of traffic(hence, congestion) in the base (“do-nothing”) scenario will lead to over-estimation of project
benefits. Nicolaisen identifies potential causes for these biases, including inadequate transportdemand models and a failure to deliver other external changes (e.g. land zoning) that affectdemand. However, their relative importance is unclear.
3. Demand forecasts are highly imprecise (the standard deviation of inaccuracy is substantial). While
forecasts are systematically biased towards over- or under-estimation for different project types,
considerable differences exist between results for individual projects. Nicolaisen and Driscoll(2014) observe large differences between projects (and for all project types) regarding the extent
and direction of forecasting inaccuracies. This variation makes it difficult to identify ways toimprove forecasting, as several causal mechanisms may be involved.
4. Appraisals do not adequately address economic effects (e.g. property value, lost opportunity cost).
Appraisals typically do not consider associated land-use impacts (Börjesson et al., 2014), partly
due to difficulties in estimating these impacts. However, research indicates rail projects (and othermajor transit investments) frequently give rise to considerable non-transport economic benefits
in the longer term when supporting investments have been made. For example, Banister andThurstain-Goodwin (2011) found that non-transport economic benefits were up to three times aslarge as transport economic benefits for some projects. Conversely, the economic costs of newroad projects often relate to the tendency of such projects to favour more dispersed development.
While these costs are also difficult to estimate and rarely incorporated in the appraisal, their
existence suggests that project benefits may be significantly overestimated in some cases.
5. Appraisals do not adequately address non-economic effects (e.g. environmental and social effects).
While the value of travel-time savings is usually the primary rationale for transport infrastructure
investments, projects also frequently address social and environmental objectives that are notdirectly related to economic growth (Beukers et al., 2012). While impact analyses assess someenvironmental effects, it is rare for appraisals to include the full range of relevant social andecological impacts.

Lessons for appraisal practiceNicolaisen (2021) finds that the significant observed differences between ex-ante forecasts of project
impacts and actual outcomes result from a complex range of technical/methodological andorganisational/political factors. However, they do not depend on the travel demand model used toforecast traffic volumes. He concludes that, rather than investing resources in developing new
sophisticated modelling software, attempts to improve appraisal practice should focus on reducing theimpact of these inaccuracies on strategic planning decisions for transport infrastructure. He proposes five
strategies to achieve this end.
1. Systematic evaluation schemes. Adopting consistent methods and approaches to evaluation isarguably the most crucial potential improvement to its practice. While evaluations are now morecommon, systematic approaches remain rare. There is little standardisation of methodologies andwide variation in the quality, coverage and comprehensiveness of evaluation between countries
and project types. Systematic evaluation provides the direct and regular feedback essential tolearning from past mistakes (Kahneman, 2011). In their absence, forecasters will likely remain
unaware of systematic biases and inaccuracies. Only systematic evaluation can ensure that fleeting
28

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

IMPROVING APPRAISAL THROUGH EX-POST EVALUATION

data is reliably captured, providing the necessary inputs to quality evaluation. It addresses
availability bias and data noise resulting from inconsistent methodologies (Nicolaisen and Driscoll,2014). Establishing systematic evaluation schemes would enable governments to improve theirperformance assessments, identify inaccuracies and locate their causes, all of which would help
enhance ex-ante appraisal.
2. National transport plans. Many countries focus on case-by-case appraisals of individual projects
without addressing the collective impact of transport projects on achieving strategic transport
objectives. Given the dominance of CBA in project appraisal, projects that contribute to achieving
strategic objectives in ways not readily captured by CBA are less likely to receive funding. An
example is increasing the modal share of public and active transport. The biases highlighted above
and the long-term nature of many of the benefits imply that these types of projects will fare poorly
in appraisals. Adopting a national transport plan with explicit strategic objectives enables
assessment of projects’ ability to contribute to achieving those objectives, in addition to individual
project-level appraisal.
3. Strategic scenario planning. Insights from evaluations can lead to methodological improvements.
These, in turn, can significantly improve the accuracy of ex-ante appraisal. However, forecasts forlarge projects that form part of a complex system are subject to high levels of uncertainty. These
uncertainties sometimes make it difficult to provide confidence in CBA-based outcomes. Adopting
strategic scenario planning can add substantial value when assessing fundamentally different
solutions. It allows the exploration of different development trajectories to see which solutions
are most robust across a range of likely scenarios. It also helps identify risk factors for different
projects, making essential success criteria clear to decision makers. The ITF (2021a) has previously
noted that scenario analyses could add considerable value to project decision-making processes.
4. Accounting for induced demand effects. Arguably the largest systematic bias found in appraisals is
the overestimation of road-traffic growth in the “do nothing” scenario. This, in turn, implies an
exaggeration of congestion-reduction benefits and an underestimation of carbon impacts. The
combination of this effect and the general tendency to underestimate traffic growth implies thatbuilding new road infrastructure creates strong induced demand effects unaccounted for inappraisals. Accurately estimating future traffic volumes is essential when deciding whether adding
capacity is the best approach to addressing congestion. Incorporating recognition of this dynamic
in appraisals could reduce an apparent systemic bias in favour of investing in road projects.
5. The holistic assessment approach. Significant differences in appraisal practice exist at different
levels of government. National-level approaches typically focus on travel-time savings andconstruction costs. This reflects the fact that these are easier to estimate than other impacts suchas changes in land value, which are often a critical focus at the local level. Nicolaisen (2021) argues
for adopting a consistent and holistic approach, which incorporates as wide a variety of impacts
as possible. This approach should improve dialogue and help to achieve agreement on projectpriorities and pipelines.

Conclusions on ex-post evaluation
Experts have identified numerous ways in which better evaluation practice could yield better ex-anteappraisal. However, substantial challenges exist and have meant that the potential contribution ofevaluation to improved appraisal remains largely unrealised. One fundamental problem is that too little
evaluation occurs. A corollary is that when evaluation does occur, it is rarely systematic.
BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

29

IMPROVING APPRAISAL THROUGH EX-POST EVALUATION

Methods and analytical frameworks are not usually standardised, and policy makers rarely undertake
cohesive programmes to evaluate significant numbers of comparable projects. This lack of cohesion isparticularly problematic because evaluation lessons may vary according to project type, mode or otherfactors. Hence, there are few opportunities to draw conclusions based on comparisons of the results of aset of evaluations of similar projects. A related issue is that the scope of the evaluation must be
appropriate. If it is too narrow, it risks failing to identify significant project impacts. If it is too broad, it candetract from the evaluation’s ability to identify clear lessons. The absence of a systematic approach toevaluation increases the difficulty of making appropriate choices about scope.
Addressing these issues will only be possible with political support. This applies to enhancing therobustness of evaluation practice by providing more resources and requiring better processes, and to using
the insights that better evaluation can provide. Such support is often lacking, as concern about the politicalconsequences of negative evaluations of past projects can result in unwillingness to expand the scope andperceived credibility of appraisal.
An additional risk is that improved evaluation programmes may tend to undermine the credibility of exante appraisals, both with decision makers and the public, by highlighting their inaccuracies. This could risk
making appraisal less influential in decision making, potentially making for a less rationally-driven process.
This risk can only be addressed if there is a sustained commitment to using the insights from evaluation toenhance appraisal and demonstrating the improvements made.
These factors suggest the need to better promote the benefits of improved evaluation to decision makers
and, where evaluation resources and expertise remain limited, to orient evaluation efforts toward areas
with the potential for them to be most influential. This could involve assessing the potential benefits andappropriate scope of a specific ex-post appraisal at the planning stages of each major project.
Consideration could be given to the context of each project, likely changes in the background environment(i.e. independent of transport considerations), potential unforeseen changes in travel behaviour, broader
economic and political trends, the level of confidence in demand forecasts and the potential importanceof the evaluation on the broader policy context.
Examples of improvements to ex-ante appraisal resulting from the insights from evaluation and theresulting benefits in terms of better policy outcomes should also be highlighted and widely shared. While
increasing the extent of evaluation activity and making better use of the data and insights derived from it
could significantly improve appraisal practices and methods, the Roundtable discussion also highlighted
its potential to lead to more transformative change. Rather than simply leading to incremental
improvements in appraisal methods and processes, better integration of evaluation into the broadertransport planning process could help to catalyse wider changes to transport planning. As discussed in thenext section, the Roundtable concluded that wider changes to transport planning are essential to ensuring
the achievement of the new objectives of transport policy, within the broader context of contributing to
liveable cities and societal and individual well-being.

30

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNINGCountry examples of ex-post evaluation
While relatively few ITF member countries evaluate the impacts of major transport investments, France,
Norway and the UK provide partial exceptions.

FranceBonnafous (2021) reviews French transport appraisal practice and highlights important lessons. In France,
major transport infrastructure investments are evaluated systematically. Since the 1982 adoption of theLoi d’Orientation sur les Transports Intérieurs [Domestic Transport Guidance Law, LOTI] and its successor,
the 2010 Code des transports [Transport Code], all projects above a threshold value (currently EUR 83
million) require an ex-post evaluation.
The aims of the ex-post reviews are to (a) inform the public on project outcomes, especially the extent andcauses of any differences between outcomes and initial estimates; (b) account for the use of public funds
by evaluating the effectiveness of the investments made; and (c) provide feedback to support the
improvement of future ex-ante appraisal practice.
The evaluation process is largely independent of the entity responsible for the ex-ante appraisal, although,in some cases, independence is achieved by allocating responsibility to different parts of the same entity.
For example, the Audit and Risks Department within SNCF Réseau (the network manager) performs LOTI
evaluations for rail projects. In all cases, however, LOTI reports are supplemented by an independent
opinion from the Conseil-General de l’Environment et du Developpement Durable [General Council for theEnvironment and Sustainable Development]. Most completed evaluations relate to motorway and rail
projects, meaning it is only possible to draw precise, mode-specific lessons for these two modes.
Motorways

Motorway project appraisals demonstrate systematic underestimation of construction costs, with asubstantial majority of the projects identified by Bonnafous (2021) experiencing cost overruns. Traffic also
exceeded forecasts in around two-thirds of cases. However, most of these results occurred whenmotorway concessions were allocated on a non-competitive basis to one of five motorway companies(four of which were government-owned). Financially unviable projects received an implicit cross-subsidy,
with one of the concession holder’s current projects receiving an extension to its concession. Thus,
incentives for cost control and timely delivery were lacking.
The transposition of European Community (EC) Directive 93-37 substantially changed these arrangements.
The Directive abolished the protection from competition previously provided to motorway companies.
Private providers can now respond to calls for motorway tenders, and an explicit subsidy regime is now in
operation. The first evaluation of a project completed under the new system (the Rouen–Alençon sectionof the A28 motorway) showed that both costs and traffic levels were in line with initial estimates, and therate of return was higher than predicted. However, without further evaluations, it is impossible to draw

31

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

firm conclusions on performance improvements resulting from the improved incentive structures andenhanced transparency of the new concession system. Conversely, evaluations of ex-ante traffic forecasts
show much greater forecasting accuracy for projects completed after 2000 due to improvements in trafficmodelling and, in particular, the estimation of factors determining the growth of mobility.
High-speed rail

Data from evaluations of six high-speed rail (HSR) projects in France shows that ex-post rates of returnwere, in all cases, well below those predicted in ex-ante analysis. In four cases, traffic levels falling short ofestimates constituted a major explanatory factor, while traffic was above forecast in the other two cases.
Costs were higher than estimated in all cases. However, Bonnafous (2021) shows that construction delays
have contributed significantly to lower-than-expected internal rate of return (IRR) performances. Table 1
shows the delay experienced for each project (expressed as a percentage of the planned duration of thework) and the consequent percentage reduction in the IRR.
Bonnafous (2021) argues that this experience of systematic cost over-runs and delays is likely to have beena major factor in decisions to build the three most recent HSR projects as public-private partnerships(PPPs). However, France’s high public debt/GDP ratio was also likely a factor, given the limits establishedby the Maastricht Treaty and the exclusion of the debts of PPP entities from accountings of public debt.
The recent success of PPPs for motorway construction may also have contributed. In all three projects,
construction was completed on time and within budget. If this limited evidence of significantly improved
on-time performance due to the adoption of more competitive bidding processes proves robust, one could
expect similar improvements in the accuracy of ex-ante appraisal.
Permanent observatories in the road and rail sectors

Wider economic benefits also fall within the scope of the LOTI ex-ante appraisal requirements (and thoseof its successor legislation). Ex-post evaluations therefore also address these benefits. However, assessing
whether projects delivered on ex-ante claims in practice proved difficult in the evaluations reviewed by
Bonnafous (2021).
For example, while the Ligne à Grand Vitesse Nord [North High-Speed Line, LGV Nord] was expected toprovide economic benefits to locations in Lille, the evaluation concluded this was “very difficult toestablish”. Bonnafous notes the challenge of identifying statistics to confirm or invalidate ex-anteestimates. Data – especially about tariffs offered to various users at different times – is typically “fleeting”
in nature. That is, if it is not captured at the time of an activity, it requires great effort to reconstruct, if thisis even possible. The loss of memory of central actors is also a well-known difficulty in undertaking
retrospective research.
Table 1. Commissioning delays and profitability losses in French high-speed rail projects

Delay (%)

Atlantique

NordEurope

Interconnexion Îlede-France

RhôneAlpes

Méditerranée

Est(Phase 1)

13 %

7%

50 %

9%

75 %

28 %

-4.4 %-10.0 %

-7.3 %-2.9 %-3.9 %-7.2 %

Internal rate of
return loss (%)

Source: Bonnafous (2021).

32

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

This experience motivated the creation of permanent observatories (in French, observatoires) to develop
expertise and record data in real time. In the 1970s, the University of Lyon opened France's first permanent
motorway observatory. It began its work when two motorways (A43 and A48) were opened. The
observatory monitored demographic and economic change indicators in 245 municipalities potentiallyaffected by the motorways. The results illustrated the greater effectiveness of permanent observationcompared to ex-post studies. They also effectively responded to the problems of fleeting data and memoryloss.
The French Ministry of Equipment and motorway concession companies have since gradually expanded
the number of motorway observatories, with varying success. While seen by concessionaires as important
inputs to the required LOTI balance sheet, their design and operation were neither regulated nor
standardised.
Bonnafous (2021) notes that several observatories developed with university teams have delivered
valuable lessons. A central example, the A39 motorway observatory (1993–2004), was established at the
beginning of the construction phase and continued for six years after the motorway entered service. This
meant that it could obtain detailed data on the impacts on the local economy, the operation of theinfrastructure, and the level and distribution of traffic. It also made it possible to analyse the effects on
local production and development.
The benefits of permanent observatories for highway investments are thus well established in France,
especially in identifying and assessing indirect effects. However, this is not the case for HSR investments.
None of the six projects mentioned in Table 1 benefited from a permanent observatory. Conversely, thelow quality of the data gathered by retrospective academic studies of these projects highlights the limitsof such retrospective analysis. This strengthens the case for adopting a permanent observatory model for
new HSR projects. Increased local opposition to such projects underlines the role of observatories inquantifying their benefits.
Contracts for the two most recent HSR projects required the private operators to establish and financepermanent observatories to measure their environmental and economic effects. The observatory for the
Bretagne–Pays de Loire line (a PPP valued at EUR 3.3 billion) must operate for ten years from contract
signing. The l’Observatoire Socio-Économique de la Ligne à Grand Vitesse Sud-Europe Atlantique [SocioEconomic Observatory for the South-Europe Atlantic High-Speed Rail Line, OSE-LISEA] between Tours andBordeaux (EUR7.8 billion) runs for ten years from the line’s entry into service. The OSE-LISEA observatory’s
longer duration reflects the view that shorter analyses cannot capture such a project’s full effects. This, inturn, echoes the conclusions of many previous evaluations.
The OSE-LISEA’s stated purpose is to assess the direct and indirect effects of the LGV SEA on mobility, thelocal economy and regional planning. 2 Local actors (e.g. local governments) must be able to use this work
to maximise the project's benefits for the local economy and society. Such actors also contribute tonational reviews and discussions on the impact of HSR lines.
As a result, the observatory involves national agencies and local partners. It is managed by a specificdepartment of LISEA, assisted by a scientific committee comprising academics who help develop the work
programme. State services and local partners participate in a monitoring committee, which validates thisprogramme and evaluates the work. This work of OSE-LISEA is also the subject of regular feedback open
to all interested local partners. It addresses a range of impacts across six major axes: construction,transport and traffic (including price effects and modal impacts), development near stations, the behaviourof local actors and organisations, urban and regional dynamics, and tourism.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

33

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

Lessons from ex-post evaluation in France have led to clear improvements in the quality of CBA
methodology. Progress can be seen, for example, in the substantial reduction in inaccurate traffic
forecasts. LOTI assessments have probably favoured the use of PPPs by demonstrating considerable
benefits in meeting implementation time and cost targets. That said, the small number of observations of
PPPs with exclusively private operators to date necessarily limits the robustness of this result.
Bonnafous (2021) concludes that the growing number of permanent observatories has improved
understanding of the economic impacts of major transport projects on the local areas involved, particularly
for motorway projects. It also seems that the effectiveness of observatories in capturing and processing
ephemeral data, especially price changes for different modes, will deliver new types of analyses. However,assessing the effectiveness of France’s permanent observatories will remain difficult until the publicationin 2022 of the first evaluation reports produced under the legislative requirements.

NorwayIn Norway, ex-ante evaluation is compulsory for projects with an estimated investment cost exceeding
EUR 100 million. This requirement, introduced in 2000, responded to a series of cost overruns and project
failures. It is a stage-gate process that aims to ensure quality-at-entry through external quality assuranceof project proposals and cost estimates.
The starting point is a conceptual appraisal by the responsible agency, which reviews the strategic case forinvestment and includes a CBA of different alternatives. External consultants conduct a quality appraisal(QA1) before the Cabinet decides whether to carry on with planning or not. After further planning andappraisal of the selected alternative, the full business case is reviewed (QA2). External consultants assess
the quality of cost estimates and other crucial management parameters at this stage and advise theresponsible department on the appropriate budget. QA2 must occur before the project is submitted toparliament for approval. The entire process (see Figure 4) helps ensure projects are sufficiently welldeveloped before a funding decision is made and reduces the risk of optimism bias.
Around 300 projects have been subject to this scheme, of which 120 have so far been completed (mostwithin the budget approved after QA2). Because the strategic case is assessed at the outset, and large
projects require an explicit and early political mandate, the potential for politically driven projects to obtainbudget funding without being subject to proper appraisal and QA is reduced. That said, Eliasson et al.
(2015) found that despite the significant emphasis placed on CBA of project proposals in Norway, CBA
outcomes have little impact overall on the probability a project will be selected. They argue that the main
effect of CBA may be to screen out projects with very low benefit/cost ratios at an early stage, while final
project selections are more heavily influenced by other factors (e.g. regional policy considerations).
Figure 4. The Norwegian Quality Appraisal scheme for large government projects

Source: M. Welde (supplied).

34

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

Table 2. Criteria used in ex-post evaluations, Norway
Level ofsuccess

Evaluation
criterion

Definition

Operational

Efficiency

Project implementation and performance in terms of cost, time and quality.

TacticalEffectiveness

Whether the agreed goals have been obtained and to what extent the project hascontributed to this outcome.

Strategic

Other impacts

All consequences beyond the agreed outcome that can be attributed as the result of the
project, positive and negative, short-term and long-term, for different stakeholders.

Relevance

A project is relevant if there is a need for what the project delivers. Project relevance ismeasured in relation to national political priorities and stakeholders’ preferences.

Sustainability

A project is sustainable if its benefits are likely to persist throughout its lifetime. The total
impacts (financial, environmental and social) should be acceptable in the long run.

Benefit–cost
efficiency

Total willingness to pay for what the project delivers in relation to cost, as measured by thecost–benefit analysis.

Source: Welde (2021).

Another aspect of the Norwegian ex-ante appraisal system is that significant changes in project scope andexternal conditions often occur between QA1 and QA2. This sometimes results in projects being rejected
before, during or after QA2. However, there is no formal requirement to revisit the strategic case after the
initial assessment.
Projects are also subject to ex-post evaluation. The ex-post evaluation is not part of the QA scheme itself.
Rather, since 2012, systematic ex-post evaluation has been undertaken as part of a research programme
funded by the Ministry of Finance, the Concept Programme. The evaluations are based on a goal-oriented
framework originally developed by the OECD Development Assistance Committee (DAC), which includes
five criteria (OECD DAC, 1991). The framework has been expanded to include benefit–cost efficiency as asixth criterion (Volden, 2018). Definitions of the six criteria are presented in Table 2.
A central feature of this evaluation framework is that it covers multiple perspectives. All evaluations are in
the same format, with results summarised by applying a score for each criterion on a six-point scale. So
far, 29 projects have been evaluated, of which 14 are transport projects. All are considered large-scale in
the Norwegian context (i.e. cost more than EUR 100 million). The main findings of the evaluation
programme to date, as summarised by Samset and Volden (2022), relate to operational, tactical andstrategic success.
1. Operational success. Most projects receive an acceptable score on efficiency (the median score is
4 for both railway and road projects). In particular, cost performance is good in the majority of
projects. There is much to suggest that the QA scheme, which requires the use of stochastic costestimation and external scrutiny of the estimates, has improved the accuracy of cost estimatesand project management practices. A study of 78 completed projects subjected to external quality
assurance (Volden, 2018; Welde, 2017) confirms these promising results.
2. Tactical success. Tactical project success, as measured by effectiveness, was high for road projects(median score 5) whereas the four railway projects performed less well (median score 3). This ispartly explained by the fact that some road projects have trivial goals (e.g. increased mobility as
measured by time savings). Rail project goals, by contrast, have been over-ambitious, with goals

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

35

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

related to improved rail competitiveness and a green shift in the transport sector. The latter typeof goals have not generally been reached, and evaluations suggest that achieving them would haverequired greater efforts from the railway authorities, undertaken as part of a wider strategy.
3. Strategic success. Four criteria measure the strategic purpose of the project. The results show thatstrategic success is multi-dimensional and that a project may score well on one criterion and not
on another. Most transport projects were rated as acceptable in terms of other impacts because
they had no major positive or negative external effects. Some could have done more to avoid
negative side-effects on nature and the local environment. Only one project, a subsea road tunnel,
clearly indicated wider economic benefits. On relevance and sustainability, both railway and roadprojects performed well. In particular, the railway projects were well aligned with thegovernment’s strategy for sustainable transport. The infrastructure was also considered to havesufficient capacity within planned lifespans (40 to 75 years). However, in some cases, evaluators
suspected that other, more relevant (at least more cost-efficient) solutions to the problem existed(e.g. more effective route choices for some railway projects).
On benefit–cost efficiency, as measured by the CBA, road projects generally scored better thanrailway projects in the sample. The road projects all generated higher traffic volumes and lower
accident frequencies, while the investment cost was approximately on budget. Generally, mostroad projects had better benefit–cost ratios than estimated ex-ante. In several cases, the ratio was
greater than 1 in the ex-post evaluations, in contrast to ex-ante estimates of ratios less than 1. Bycontrast, all four rail projects showed low benefit–cost efficiency, and their ex-ante CBAs seem tohave been subject to systemic positive bias. However, the shadow carbon price used in CBA wasrelatively low, meaning that the monetised benefits of a modal shift (if achieved) would be
moderate at best. Notably, the low benefit–cost efficiency for some projects (especially rail) was
known by decision makers ex-ante. This indicates that the results of the ex-ante CBA had not been
crucial to project selection.
Norway’s overall experience with the generic six-criteria evaluation model is positive. The model implies abroader perspective on project success than value for money as measured by the CBA. This is expected toencourage planners to focus on a wider set of strategic issues at the project selection stage. Moreover,the systematic evaluation of many projects using the same model will, over time, lead to the developmentof a large database that can be used to determine whether estimates made ex-ante are systematicallybiased and as a basis for improving estimation tools and methods.

The United Kingdom
The National Highways (previously Highways England) post-opening project evaluation (POPE)
programme, and National Audit Office (NAO) reviews are two examples of ex-post evaluation in the UK.
Post-opening project evaluations

National Highways and its predecessor organisations have carried out the POPE programme for more than
15 years. Under the programme, analytical reviews to evaluate the impact of infrastructure schemes orprogrammes occur between one and five years after a project opens to traffic. National Highways
describes them as an early mechanism to ensure the project is on track to deliver the anticipated benefits
over its lifecycle (typically 60 years). 3 All projects on the strategic road network with a value of more than
GBP 10 million are evaluated. The POPE process aims to assess whether schemes deliver the anticipated

36

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

value for money; validate the accuracy of the estimated scheme costs, impacts and benefits identified in
the business case; use the ex-post impact assessment to improve future appraisals; and promotetransparency and accountability to stakeholders.
A recent review of POPE (Hyperion 2020) found that its highly standardised review format and clear focus
have significantly increased its impact. The clear link between the ex-post evaluation and the ex-ante
project appraisal provides a high level of transparency, particularly as evaluations are routinely published.
The publication of the methodology used to conduct a POPE (Highways England 2022) also contributes tothe transparency of the process. The evaluation is also conducted using the same methodology as the exante evaluation, despite any changes in methodology adopted in the interim. Another feature of the
programme is that independent “meta reports” are produced periodically to identify and analyse trends
across the programme as a whole, thus potentially deepening the insights gained. 4
The National Infrastructure Commission has concluded that the POPE programme has led to more accurate
ex-ante project appraisals (National Infrastructure Commission 2018). A major recent change to the designof the POPE programme was the move to conduct evaluations within National Highways itself through a
dedicated Evaluation Group. This has allowed the organisation to retain knowledge and conduct longerterm, thematic and programme-level evaluations (Hyperion 2020). In recent years the programme has also
adopted a wider view of benefits, moving away from its previous focus on direct user benefits (e.g. travel
time savings and accident costs). This occurred in the context of the adoption of a new Benefits
Management Process, which also includes an early identification of essential data needs for evaluation
purposes, as part of a new Benefits Realisation and Evaluation Plan. The review found the new process has
improved the scope, governance and quality assurance of evaluations (Hyperion 2020).
National Audit Office reviews

While NAO reviews focus on value for (public) money, they contain a broader set of ex-post evaluations ofprojects (and, in some cases, programmes). The NAO has also undertaken strategic analyses, which seek
to leverage the knowledge acquired by undertaking many such reviews over an extended period. The NAO
has published reports on strategic topics, including optimism bias, quality assurance, and lessons learned
from major rail infrastructure programmes. It has also developed a framework for reviewing major
programmes (NAO, 2021), a tool for understanding the challenges in delivering project objectives and aguide for senior managers on challenging project costs. A recent synthesis report identified lessons fromits reviews of major expenditure programmes (NAO, 2020) derived from eight specific insights (see Box 3).
Box 3. National Audit Office insights on major programme management
1. Ensure that the programme’s scope clearly aligns with its strategic objectives.
2. Understand the limits of cost and schedule estimates.
3. Critically examine and be realistic about the schedule.
4. Develop detailed plans to achieve anticipated savings.
5. Ensure interdependencies are identified and managed.
6. Consider operational planning from the start.
7. Ensure that programme management changes as the project develops.
8. The importance of transparency and honesty in major programmes.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

37

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

The need for wider reform of transport planning
The previous section identified a range of potential improvements to appraisal methods, some of which
are increasingly being adopted in practice, while others remain relatively little used. Roundtableparticipants agreed that, while these were valuable in themselves, a broader response to the changing
objectives of transport policy is required. This should extend beyond project appraisal to encompass theentire transport planning process.
Improved project appraisal cannot change outcomes if the right projects are not selected for appraisal.
This highlights the importance of ensuring that project identification reflects, or responds to, the changing
objectives of transport policy. Improving the way problems are framed in light of broader policy objectives,and identifying project alternatives in line with these broader framings, are therefore critical steps before
embarking on typical appraisal processes.
Addressing this issue in the specific context of accessibility and equity, Shiftan et al. (2021) argue: “Analternative to the traditional approach to transport planning would take equity seriously from the very
start of the transport planning process”. This would align the transport planning process more closely withpractices in other core government policy fields (e.g. healthcare, education and housing). Improving theway projects are reviewed against broader policy objectives in the initial “needs case” is therefore critical.
Extensive stakeholder engagement is crucial to achieving broad understanding of, and support for, a new
set of transport policy objectives focussed on accessibility, equity, decarbonisation and urban liveability.
This engagement is especially important given the extent of the change involved. The expected result is a
substantial move away from the decades-long predominance of investment in road infrastructure.
Broadening transport appraisal alone is not enough; the scope of decision making must also be broadened.
This requires an improved understanding of the issues that transport planning seeks to address. It implies
that appraisal processes should be multi-staged and timed to begin early in the planning process, with an
initial needs assessment. This needs assessment, in turn, needs to be linked to and consistent with theapproaches taken in subsequent stages.

The role of strategic planning
Explicitly adopting a strategic planning process is likely to facilitate and accelerate changes in transport
planning objectives. A recent ITF Working Group report, Developing Strategic Approaches to Infrastructure
Planning (ITF, 2021a), recommended that governments adopt strategic approaches to infrastructure
planning and communicate them clearly via an explicit, detailed and periodically updated strategic
infrastructure plan. It noted the increasing use of independent advisory bodies to provide transparent,
expert advice on long-term policy and planning, and reduce instability in the planning process due to shortterm political considerations. It also recommended that governments consider establishing such bodies asa low-cost and effective means of improving and complementing existing processes.
An example of a strategic planning approach at the metropolitan level is Tel Aviv’s 20-year Strategic Plan
(Tel Aviv–Jafo Municipality, 2017; see also Box 1). As noted above, the plan uses a broad range of explicitly
established criteria and sub-criteria, with explicit weightings, to assess investment options using an MCA.
Policy makers set the goals, criteria, and weights (Shiftan et al., 2021). Strategic plans encourage a coordinated and consistent approach to transport infrastructure investment that reflects long-term

38

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

objectives and priorities. This is likely to yield better outcomes than a simple aggregation of project-byproject appraisals, especially when moving towards a new and broader set of transport planningobjectives.
Strategic plans must be reviewed and updated periodically to ensure their continued relevance andconsistency with public policy priorities. Strategic infrastructure plans should also be linked to explicit
infrastructure funding envelopes. This helps ensure that they can be delivered, that conscious and soundly
based priority setting is undertaken, and that project pipelines are at least broadly identified. There should
also be clear linkages between strategic infrastructure plans and individual project selection processes.
The wider use of strategic planning also poses implementation challenges, including building public
support for the strategic plan's objectives. Even with substantial stakeholder engagement, extreme
divisions between stakeholders and the public regarding long-term future visions may persist. Opinion
polling over time often demonstrates large medium-term shifts in the public’s expressed policy priorities.
This further complicates the task of developing a plan that can be maintained and delivered over a longtime horizon.
However, it is likely that a “virtuous circle” may exist in this regard. That is, sustained engagement efforts
should yield progressively better results over time, as stakeholders become better informed and morefamiliar with the dynamics of the consultation process, and gain more confidence that decision makers
address their inputs. Convincing the public of the need for major strategic initiatives minimises objections
based on so-called not-in-my-backyard (NIMBY) responses. However, doing so means explaining problems
rather than simply focusing on proposed solutions.
The “design–announce–defend” model is poorly adapted to the task of convincing the public of the needfor major initiatives, and recent years have seen a more general move towards more participative decision
making across much of Europe and, to some extent, in the United States. Such participation is increasingly
seen as a condition of legitimacy in decision making.
Despite the difficulties of ensuring high levels of stakeholder support for strategic plans, some experience
shows that this can be a crucial factor in ensuring support across political parties. This support increases
the chances that long-term projects will retain support and continue to be developed despite changes ofgovernment. To take one example, the development of the Grand Paris Express project included extensive
stakeholder engagement, notably with the mayors of disadvantaged municipalities. The project design was
modified to give priority to completing lines serving these areas over other project elements with higher
benefit–cost ratios (e.g. the transversal line linking the two Paris airports). While this suggests that greater
weight was given to factors other than direct user benefits, the widespread support for the project saw it
survive a global financial crisis and a change of president and governing party (ITF, 2017b).
It is also essential to highlight the links between transport investment and land use in the strategic planning
context. The ITF has previously argued that a critical factor in effectively addressing accessibility objectives
is ensuring that changes in land use and transport planning are co-ordinated and mutually supportive (ITF,
2020a). One approach involves developing a comprehensive land-use and transport strategic plan,
incorporating high-level objectives, setting broad policy directions based on these objectives, andidentifying relevant quantitative indicators.
Identifying a needs case is a central aspect of strategic planning in relation to project identification,
appraisal and selection. The needs case determines the criteria for justifying a decision to undertake an
investment. Figure 5 describes current Norwegian practice in this respect. The investment decision ispredicated on an assessment of long-term user benefits and wider economic impacts. Longer-term

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

39

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

impacts on strategic objectives such as sustainability also constitute a vital part of the assessment. Of noteis the explicit acknowledgement of the increasing importance of uncertainty as the time horizon extends.
Many project failures stem from decisions made in the project planning phase (Morris, 2011; Edkins et al.,2013). Norway’s experience provides clear evidence that robust planning and preparation in the earlystages of a project’s lifecycle are essential for successful delivery. Ex-ante appraisal in Norway is based ona staged process that aims to ensure that proposed projects solve real problems, meet the needs of users
and other stakeholders, and deliver value for money. Norway’s assessment process, like the “five-casemodel” in the UK (discussed below) incorporates a significant focus on the strategic case for the project,in addition to the narrower CBA-based value for money assessment.
A standardised model for ex-ante appraisal, combined with an external review, sets out the requirementsthat must be met for projects and programmes to continue to be developed. Norway has assigned
resources to developing uniform data sets for estimating costs and benefits, better enabling comparisonsacross projects and sectors. This helps to identify and eliminate poor-quality proposals at an early stage
and reduces the tendency for assessments to suffer from optimism bias.
However, a lengthy project development process can lead to increasing expectations from stakeholders
and escalation of commitment by decision makers. In other words, there may be a tendency to justify aproject proposal even if new evidence suggests that the project may be economically and strategicallyunsuitable. This can mean that final approval becomes inevitable: once large projects gain momentum,
they may be almost unstoppable. Moreover, once the investment decision has been taken, the scope formaking changes is limited. Therefore, the choice of conceptual solution is the most crucial decision in theprocess. Evidence from Norway shows that most projects subjected to early conceptual appraisal are
implemented sooner or later, regardless of the results of subsequent value-for-money analyses. Thissuggests a need for intervention before the responsible agency commits funds, to identify schemes
unlikely to either satisfy needs cost-efficiently or meet strategic objectives (Edkins et al., 2013).
Figure 5. Assessment of transport infrastructure investments in Norway

Source: Welde (2021).

40

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

The requirement to justify proposed investments against a broad range of strategic policy objectives has
a long history in the UK. For example, in 2000 the UK Department for Transport (DfT) released its ten-year
transport plan. The transport planning objectives identified in the plan included contributing to industrial
competitiveness, boosting regional economic development, enhancing access and opportunity in rural
areas, reducing social exclusion and lessening the impact of transport on the environment (DfT, 2000).
Implementation was to be guided by a series of multi-modal studies, conducted according to the following
principles of transport appraisal, which reflect the underlying strategic objectives:•clear definition of the transport problem, formulated in terms of all relevant transport objectives•

clear exposition of the transport objectives relevant to the project proposals•

a multi-measure and multi-modal focus, to replace a “road-building mentality”•

exploration of reasonable alternatives, including identification of synergies between
components, in preference to “obvious solutions”•

enhanced stakeholder involvement.

This approach was also applied at the local level, where authorities were placed under obligations todeliver an integrated transport system via local transport plans. Funding requests for major public
transport or road schemes needed to include evidence that they were necessary to achieve the objectivesof the applicable plan and that those objectives could not be achieved in other ways. Schemes were also
to be integrated with measures to promote modal shift, and consider the scope for alternatives to major
construction, including demand management (European Conference of Ministers of Transport, 2004).
As an early example of a strategic plan, the DfT plan was not linked to an explicit budget constraint. By
contrast, the programme of priority infrastructure investments recommended in the National
Infrastructure Commission’s 2018 National Infrastructure Assessment was based on the indicative funding
envelope (of 1.2% of GDP per annum) identified by the British Government (ITF, 2021a). Further reforms
to the UK’s transport appraisal processes led to the adoption of the “five-case model”, which requires ananalysis of all proposed investments in terms of a five-part business case (HM Treasury 2018).
The content of the five cases is directly relevant to effective project appraisal, selection and delivery.
However, a proliferation of analytical requirements risks obscuring core messages and reducing decision
makers’ ability to assimilate the information provided and focus on vital strategic considerations and tradeoffs. This risk was countered by establishing a requirement to prepare a summary appraisal table (seeTable 3 for an example) to present the main results of CBA and other dimensions of appraisals side by side.
The table also includes commentary on aspects of the project that support or run counter to the overall
strategic objectives of the government of the day and highlights critical trade-offs.
The summary table approach, which is also recommended in Israel’s appraisal guidelines, has important
advantages as a tool for communicating with decision makers and stakeholders. In contrast to summarising
performance against all dimensions via MCA weightings, it enables the performance of each option oneach of the four performance dimensions to be seen at a glance, with separate reporting of the scores for
impacts expressed in monetary, other quantitative, and qualitative terms, as well as a distributional impact
score.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

41

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

Table 3. An example of an appraisal summary table
Assessment

Impacts

Summary of impacts

Quantitative

QualitativeNet
present
value

Distributional(7pt scale)

Economic
Business users, transport
providers, reliability impact,
regeneration, wider
economic benefits
Environmental
Noise, air, greenhousegases, landscape,townscape, historic
environment, biodiversity,water
SocialCommuters and other users,
reliability impact, physicalactivity, journey quality,
accidents, security, access,affordability, severance,
option/non-use values
Public accounts
Cost to transport budget,
indirect tax revenues
Note: Scores for each sub-heading are presented separately in the original table. They are presented together
here for reasons of legibility.
Source: Adapted from https://www.gov.uk/government/publications/webtag-appraisal-tables.

The summary table format enables an assessment of the overall performance of the proposed investment
against the core elements of the original needs case. This allows updates to the needs case to incorporate
new data or changes to the scenario(s) on which it was originally based. As Welde (2021) notes, in Norway,
the initial project appraisal decision judges the project's likely success in terms of the business case. Thisbusiness case identifies all the project goals, rather than the (less comprehensive) CBA. A well-structured
summary table can facilitate this process, ensuring that decision makers focus on the overall strategic
rationale when deciding whether to proceed with investments.

Decision processes and transport planning practice
As part of the discussion of the need for broader reform to transport planning processes, the Roundtablediscussed the fitness for purpose of the decision models used in transport planning and how these could
be reformed to support better outcomes. The central challenge is the extreme complexity of the decision
42

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

environments for most major transport infrastructure projects. Cascetta and Carteni (2021) cite Rittel andWeber (1973), who characterise the problems of social policy as “wicked problems”, to which there is no
definitive solution:
The search for scientific bases for confronting problems of social policy is bound to fail, because of
the nature of these problems. They are “wicked” problems, whereas science has developed to deal
with “tame” problems. Policy problems cannot be definitively described. Moreover, in a pluralistic
society there is nothing like the undisputable public good; there is no objective definition of equity;
policies that respond to social problems cannot be meaningfully correct or false; and it makes nosense to talk about “optimal solutions” to social probIems unless severe qualifications are imposed
first. Even worse, there are no “solutions” in the sense of definitive and objective answers. (Rittel
and Weber 1973)
Cascetta and Carteni (2021) argue that these characteristics mean that such problems entail a highprobability of planning failure. Technical limitations and errors, which constitute one cause of thesefailures, can be at least partially addressed by methodological and process improvements. However, asecond fundamental cause of failure is uncertainty about key input variables such as demand behaviour
parameters, the project context (including political cycles) and the attitudes of major stakeholders. Theseissues are partly outside the control of transport planners. At the same time, a high-quality planning
process must recognise their importance and act to minimise the risks they pose to planning outcomes.
Approaches to stakeholder engagement constitute a central factor. Major transport investments, incommon with other large infrastructure projects, typically involve a diverse range of stakeholders with awide range of perspectives and objectives. Moreover, as Cascetta and Carteni (2021) note, somestakeholders (e.g. officials from the area in which a project will be built) typically hold a degree of vetopower. Stakeholder opposition is therefore a substantial risk and a major cause of planning failures,
rendering the “decide–announce–defend” approach to decision making unfit for purpose, or at least highly
risky. Thus, decision-making styles that incorporate effective and sustained stakeholder engagement arelikely to reduce the risk of such failures substantially. The need to obtain consensus on pursuing asubstantially modified set of transport planning objectives further underlines this point.
The technical rational decision-making modelTransport planning is dependent on both technical elements and societal responses. This poses the
problem of ensuring that decisions reflect both an adequate degree of “technical rationality” and an
acceptable level of stakeholder consensus. Cascetta et al. (2015) argue that many of the problems intransport planning are related to the gap between the traditional technically based approach and thenature of real-life decision-making processes. As a result, “Transport-related decisions can be a-rational or‘sub-optimal’ with respect to stated, formal objectives and still ‘rational’ in the context of a wider and lessdefined set of contrasting objectives including maximizing consensus and/or minimizing opposition to
proposed solutions” (Cascetta et al. 2015).
Much of the research literature on decision making under uncertainty adopts a four-part taxonomy ofuncertainty types and levels (see Table 4). Transport planning decision processes should take account of
this taxonomy to maximise the probability of delivering good decisions about which set of potential
transport investments to adopt. Cascetta and Carteni (2021) propose that good decisions address widely
recognised problems or needs; do so effectively and efficiently; have widespread support, or at least
acceptance, among decision makers; and be based on the best available information on the expected
impacts of the decision options.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

43

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

Table 4. A taxonomy of uncertainty in decision making
Level

Description

1

Futures are essentially predictable, either because the time horizon is a short-term one or because
the evolution of the system can reasonably be anticipated.

2

There are a limited number of possible futures to which occurrence probabilities may be attached.

3

A large number of possible futures does not allow for the calculation of relative occurrence
probabilities.

4a

There is a potentially infinite number of possible futures, with outcomes lying between extreme
values.

4b

There is no information at all (i.e. the completely unknowable future) and not even the direction of
future key variables may be known.

Source: Cascetta and Carteni (2021).

Decision-making processes should also deliver rational decisions. However, the uncertainties highlighted
above, including questions over stakeholder responses, imply that strictly rational decisions, in thetechnical sense, may not be achievable. Recognising this, Cascetta et al. (2019) propose that rational
decisions, in such contexts, are those that are:•consistent, both internally (not contradicting the stated objectives and existing constraints), andexternally (non-conflicting with decisions taken in other interacting contexts or at different
moments in time)•comparative, considering more than one alternative (including the option of “not deciding” orlooking for other possibilities not yet included in the choice set)•aware, based on information about the options, the context and their likely impacts wheneverknowable•

flexible, open to changes due to new information on alternative options and their effects, tochanges in the economic, physical, and institutional contexts, and taking into account “decision
opportunity costs” (i.e. postponing unnecessary decisions).

Recognising that major, long-term infrastructure investments are subject to major uncertainties has
implications for the type of decision-making model, or set of processes, that should be adopted in orderto maximise the quality of the resulting decisions.
Cascetta and Carteni (2021) note that strongly rational models implicitly assume stable preferencefunctions and constraints and expect economic actors to choose utility or profit-maximising courses ofaction accordingly. Such models require a unique system of preferences to order the possible alternatives(typically identified via CBA-based processes) and are strongly sequential in process terms. They can beapplied where there are low levels of uncertainty regarding impacts, all alternative options can be
identified and analysed effectively, and decision-making power is located at a single point, or within a smallcohesive group.
However, this model comes under pressure as problem complexity arises. It may be impossible tocomprehensively analyse large numbers of alternative options or achieve sufficient consensus among
stakeholders with varying or conflicting objectives and the ability to exercise a veto. As uncertainty rises
through the levels identified in Table 4, the problem becomes progressively more acute. A large proportion

44

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNINGof major transport decisions are made using this model. In such circumstances, transport planning
decisions have a high risk of planning failure.
Formal project identification, appraisal and selection processes in most developed countries largely reflect
the strongly rational model described above. However, many of the most important decisions they inform,
especially in relation to major, long-term transport infrastructure investments, are taken in contexts
characterised by high levels of uncertainty and complex stakeholder environments. A modified decision
process, operating according to a different decision logic, could yield better planning outcomes in suchcircumstances.
The cognitive or multi-stage rational decision-making model

Cascetta and Carteni (2021) propose a “cognitive” or multi-stage rational decision-making model, which isiterative in nature, in response to this problem. The essence of the model is to capture the dynamic natureof the decision process by dividing it into sequential decision stages and delaying decisions where sufficientdata, agreement or both may be lacking. The model draws on the learning theory of dynamic decisionmaking processes. It is fundamentally a multi-step decision-making process, with each decision based onmonitoring and evaluating the outcome of the previous decision step. The model responds to theuncertainties inherent in the transport planning process by reducing the opportunity cost of ill-informed
decisions and providing enhanced opportunities to build larger coalitions via stakeholder management at
the sequential decision stages.
The model is “cognitive” in that it depends substantially on the learning undertaken by the actors during
the process about the effects of prior decisions, what objectives are achievable in practice and to what
extent, and what possible trade-offs exist. It remains “rational” to the extent that the four conditions forrational decision making identified in Cascetta et al (2019), and cited above, are met.
A central element of the model is that stakeholders are involved in all process stages. Cascetta and Carteni(2021) argue that:. . . it’s important that stakeholders are involved at the beginning of the “planning stage” whereproblems, objectives, constraints and the “rules of the game” are discussed before any specificproposal/evaluation is considered…. [They benefit from]…the “learning process” as the decisionmaking process develops within stages and between stages and in doing so they develop a senseof participation and commitment to the choices made that legitimates the process and stabilises itwith respect to changes in political and\or context changes (reduces context related uncertainty
levels).
This engagement extends as far as participating in defining and choosing between options. At each stageof the process, planning scenarios are formulated and assessed, in both technical terms and in terms ofjudgements regarding the degree of support or opposition among participating stakeholders.
Technical analyses can be based on different techniques, with the choice of technique(s) at any given time
being based on judgements as to the level of (un)certainty that exists with respect to key variables. Thus,
if high levels of uncertainty (i.e. level 3 or level 4 in Table 4) are identified, techniques such as scenario
discovery or approaches based on regret theory may be used. Engaging stakeholders in this context allows
the risk attitudes of the communities involved to be taken into account in decision making. It also provides
opportunities to convince communities of the need for major initiatives, thus helping to avert NIMBY
dynamics. This is likely to constitute a major advantage of the model, when contrasted to processes basedon the design–announce–defend paradigm.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

45

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

A significant element of the process is therefore the interaction of decision makers, officials and experts
in planning activities, as informed by the combination of the results of technical analyses and stakeholder
values and expectations. Fundamentally, the model is based on the three connected elements of technicalassessment, process management and public engagement. The central role of public engagement implies
that appraisals must identify and model all impacts relevant to stakeholders and that the results of analyses
must be presented in ways that they understand readily.
The iterative decision process will therefore continue until decision makers are sufficiently confident thatthe conditions for success have been met in respect of each of the three connected elements. That is,there must be:•adequate information to underpin confidence in the results of the technical assessment•a sufficiently robust and detailed planning process (both technical and in terms of stakeholder
engagement) to be generally regarded as legitimate•

confidence that there is a sufficiently broad and durable consensus in favour of the project amongmajor stakeholders as to minimise the risk it will subsequently fail due to the withdrawal ofconsent.

Cascetta (2021) identifies several lessons, or success factors, in the practical application of the model,
based on the experience of its application in the development of the Regional Transportation Plan of
Veneto, Italy. These are:
1. The support of an efficient and committed regional administration is critical to the successful
implementation of the model.
2. The high level of stakeholder engagement implies the need for a regional community
accustomed to participation in such processes and comfortable with them.
3. The stakeholder engagement process is likely to be useful in helping identify additional sourcesof uncertainty that were not visible a priori, as well as some potential solutions to these otherproblems.
4. Objective distinctions between variants and options can be useful in defusing conflicts about
specific choices, and enabling the deferral of some decisions until more complete information
can be obtained and analysed.
5. The long-run/short-run and variant/option framework allowed the regional administration to be
more flexible, and the plan more robust, with respect to the relatively high levels of uncertainty
recognised at the outset.
In sum, the cognitive, rational model proposed by Cascetta and Cartegni (2021) responds to the major
causes of transport planning failures in four main ways. First, it implies expanding the range of analyses
undertaken during the process, and fitting the choice of analytical tools used to the project context –notably the time horizon involved and the associated nature and extent of uncertainty regarding criticalvariables. The scope of the analysis undertaken must include qualitative variables such as the degree ofconsensus in favour of different options.
Second, stakeholder engagement is a core element, at all stages of the planning process, with stakeholders
having multiple roles. Third, the model incorporates multiple feedback loops as a core element. It formallyacknowledges that decision making in the presence of high levels of uncertainty and long project timelines
should be a multi-stage process and that prior decisions may need to be revisited in the light of new
information.
46

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

THE NEED FOR WIDER REFORM OF TRANSPORT PLANNING

Fourth, the changed nature of the decision model – notably its features of multi-staged or iterativedecision making and sustained stakeholder engagement – implies that the role of quantitative data must
expand beyond those it plays in traditional decision-making contexts. There should be a focus identifying
and modelling the impacts of high relevance to stakeholders, together with presentation and explanationof results for non-experts. Assessment methods that allow the more effective evaluation of impacts suchas levels of vertical and horizontal equity and the degree of consensus should be used. In addition, themodel’s explicit recognition that many crucial decisions are taken in a context of high levels of uncertainty
implies the need to adopt new assessment tools, such as scenario discovery and minimal regret analyses.
The authors propose the model as an alternative to the strongly rational model, based on the primacy of
CBA and embodying a design–announce–defend approach. They argue it is appropriately used where there
are high levels of uncertainty regarding technical variables and the level of stakeholder support. This isoften the case when decisions on major, long-term transport investments must be made.
Arguably, adopting the cognitive rational model would largely have the effect of formalising and makesome existing dynamics of the transport planning process more predictable. That is, while the strongly
rational model formally constitutes the basis of project appraisal and choice, it is widely acknowledgedthat political decision makers frequently make choices other than those the model recommends. This is
typically because they perceive and seek to avoid the political problems that strict conformity to its dictates
would cause. In particular, elected decision makers will necessarily be sensitive to the views of core
stakeholders, at least where the issue in question may have wider ramifications.
Welde and Volden (2021) note that, in practice, decision makers are often more interested in impacts
other than those that are central to CBA. For example, local and regional impacts are often consideredparamount, even though they frequently constitute transfers between regions rather than net economic
benefits to the economy as a whole. Thus, a central property of the cognitive rational model is arguablythat it formalises the use of political rationality as a factor in appraisal and decision making, along witheconomic rationality. By more formally accommodating political dynamics, the use of the model could thus
tend to favour more rational outcomes.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

47

CONCLUSIONS AND RECOMMENDATIONS

Conclusions and recommendations
Conclusions
The strategic objectives of transport planning are undergoing fundamental change. The previous focus on
ensuring high levels of mobility is shifting towards an accessibility-based approach that emphasizes theimportance of social, regional and gender equity. At the same time, the imperative of responding toclimate change means that decarbonisation and enhanced infrastructure resilience are increasingly
important objectives, while climate change policy also contains an increasingly prominent social justicefocus. Finally, the decarbonisation imperative combines with greater focus on the liveability of major cities
and increased awareness of the health impacts of different transport choices to make achieving a shift
towards public and active transport a core objective. Growing recognition of the need to take account of
the complex interactions between transport and land-use planning increases the complexity of theplanning task in both areas.
There is also increasing recognition that the transport planning process, at least in relation to large-scale,
long-term infrastructure projects, has long been subject to a high probability of failure. In many cases,
projects are abandoned or substantially modified after extensive development and expenditure, while
numerous completed projects fail to deliver the full anticipated benefits.
The combination of these factors points strongly toward the need for substantial changes to transport
planning and appraisal practice. Numerous opportunities exist to expand and improve the specificmethodologies and processes employed in transport appraisal in ITF countries. However, changes tospecific methodologies will not be sufficient to address the identified challenges to transport planning.
Instead, a broader set of reforms is needed to improve both the appraisal process and the wider practiceof transport planning.
Governments and transport planners should adopt a strategic approach to assess the ability of transport
investments to contribute to achieving long-term policy objectives, in addition to their specific, projectlevel impacts. This is most likely to be achieved if governments adopt long-term strategic plans that
explicitly identify and explain these objectives and their inter-relationships.
In addition, the decision-making processes by which projects are identified, assessed, and chosen should
not lose sight of the overall transport and broader policy objectives the investment is intended to advance.
This means that they should recognise the high level of uncertainty that typically exists in relation to longterm infrastructure decision making and the fundamental importance of processes to achieve a high levelof agreement among stakeholders on the objectives and means of achieving them.

48

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

CONCLUSIONS AND RECOMMENDATIONS

Recommendations
Develop long-term strategic plans that explicitly identify transport policy objectives
Strategic plans should explicitly identify crucial long-term objectives, explain their importance and set out
the major directions of policy and investment required to achieve them. Explicitly stating these objectives
can help ensure the appraisal of individual projects systematically accounts for broader strategicorientations. This is especially vital as a wider range of objectives – including accessibility; social, regionaland gender equity; decarbonisation; and infrastructure resilience – assume increasing importance andurgency.
Ensuring an adequate level of support for the objectives and orientations of the strategic plan is essential.
This is often difficult to achieve due to the widely differing objectives and priorities of different stakeholder
groups. Thus, consistent engagement, based on transparency, adequate data and information, andresponsiveness to stakeholder inputs, is crucial. Revising strategic plans at regular (but infrequent)intervals helps ensure their continued policy relevance.
Broaden project appraisal to ensure its processes and practices take account of all transport policy
objectives, as embedded in strategic infrastructure plans
Performance in relation to many of these additional objectives can be assessed effectively throughenhanced CBA, using a range of indirect valuation techniques and recent methodological innovations. CBA
guidance and practice should require the adoption of these broader approaches. Aggregate CBA should
be supplemented with a disaggregated appraisal of project costs and benefits on population groups ofconcern because of income, gender, location or other factors. However, additional analytical tools canyield further insights in many cases, and should be adopted as required.
Incorporate accessibility indicators, or other relevant tools, to assess equity impacts in transport projectappraisals
The increasing focus on accessibility as transport policy's fundamental objective means that appraisals
should incorporate purpose-specific measures to ensure accessibility and any other relevant dimensionsof equity are clearly identified and effectively assessed. The methods adopted should be chosen for theirability to clarify the specific issues at stake.
Provide detailed guidance on appropriately accounting for climate-change impacts in appraisal,
incorporating clear linkages between shadow carbon prices and emissions reduction commitments
CBA can constitute a robust methodology for addressing climate-change impacts during appraisal.
However, its ability to do so depends on ensuring all relevant impacts are incorporated in the analysis andusing appropriate shadow prices. In particular, it is critical to ensure that recent moves in some countries
to ensure shadow carbon prices reflect realistic estimates of the cost of meeting emissions reductions
agreements over time are widely adopted. Social willingness to pay measures must also be incorporated
in the analysis. Addressing resilience requirements for new and upgraded infrastructure in the light of
expectations about future extreme weather is another core requirement.
Present the results of transport project appraisals in a transparent and concise format that highlights needscase assessments
The results of broader project appraisals that assess performance against a wider range of policy objectives
cannot readily be expressed in terms of a single metric. Decision makers and stakeholders will necessarily

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

49

CONCLUSIONS AND RECOMMENDATIONS

differ as to what impacts are important and what weights should be attached to them. This implies that,to support well-informed discussion and decision making, appraisal results should be presented in a multidimensional format that separately presents the performance of potential projects across different
measures, and in terms of different underlying policy objectives.
Multi-stage appraisal processes are widely used in assessing transport investment and are necessary to
examine all the dimensions relevant to the decision to invest. However, periodic review and streamliningof procedures is recommended to counter any loss of focus on the needs case at the outset. This moststrategic part of the appraisal must be protected from the risk of becoming simply a preliminary exercise
in CBA that sees added detail in successive rounds of assessment of economic, financial and business cases.
The results of the needs-case assessment and the summary appraisal report are the most importantelements of effective communication of meaningful results to decision makers. Preserving this focus is as
important as innovating with new methods of assessment.
Ensure decision-making processes for large investments in transport systems account for the uncertainties
involved and the need for broad stakeholder support
Major, long-term infrastructure investments are subject to high levels of uncertainty, much of which isexogenous to the transport planning process and so not susceptible to being treated via improvements toappraisal processes or methods. This uncertainty, combined with the critical need to ensure adequate
stakeholder support for major projects to enable them to proceed, constitutes a major cause of transport
planning failures. Ensuring that decision-making processes are properly adapted to these criticalenvironmental factors can help to avoid transport planning failures and support the timely delivery of
projects that contribute to strategic transport policy objectives.
For such investments, the decision process should be multi-staged, with feedback loops ensuring theresults of monitoring and evaluation of the outcomes of previous decisions are systematically incorporated
into subsequent decisions. Stakeholder engagement should commence at the early stages of the process
and be maintained throughout. Adequate information should be provided, while stakeholders should beseen as vital sources of information and potential problem-solving resources, rather than as interest
groups whose consent is required to realise plans. Scenario analysis should be adopted as part of theappraisal, to provide decision makers with a sophisticated understanding of the implications of thedifferent uncertainties identified.
Integrate technical assessment, process management and public engagement in decision processes for
major transport infrastructure investments
Effective stakeholder engagement can significantly reduce uncertainty in complex, long-term
infrastructure investment projects, both by acting as a source of additional information and insight into
policy options during project identification and appraisal and by increasing the level of support for, andlegitimacy of, project decisions. Successful stakeholder engagement requires careful process
management, ensuring that all relevant voices are heard and discussions are as fully informed as possible.
The results of technical assessments should be made available promptly and effectively communicated to
non-technical audiences. Integrating these three elements of the decision process will help ensure thatdecisions meet efficiency, equity, and other underlying objectives and enjoy broad support, thus enablingthem to be implemented effectively and without unnecessary delay.

50

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

CONCLUSIONS AND RECOMMENDATIONS

Undertake systematic ex-post evaluation for all transport infrastructure projects entailing expenditure above
an identified level
Ex-post evaluation can contribute significantly to improving appraisal methods over time. Increasing thecurrently limited number of evaluations and moving to standardise the methods and processes is essential
to ensure these gains are realised in practice. Ensuring evaluations cover a sufficiently long period toenable longer-term impacts to be identified and weighed is also essential to evaluation quality. Evaluations
should be conducted by entities independent of project sponsors or stakeholders.
Consider the merits of the permanent observatory model as a means of maximising the quality of evaluations
Meeting the substantial data requirements is a major challenge when conducting evaluations. The factthat much data is fleeting in nature, and difficult or impossible to obtain ex-post, underlines this problem.
Permanent observatories, established at or soon after project inception, appear to be a promising meansof addressing this and other quality control issues and maximising the value of evaluations of larger
infrastructure projects.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

51

REFERENCES

References
Banister, D. and M. Thurstain-Goodwin (2011), “Quantification of the non-transport benefits resulting
from rail investment”, Journal of Transport Geography, Vol. 19/2, pp. 212–23,
http://doi.org/10.1016/J.JTRANGEO.2010.05.001.
BEIS (2021), “Valuation of greenhouse gas emissions: for policy appraisal and evaluation”, Policy Paper,
2 September 2021, UK Department for Business, Energy and Industrial Strategy, London,
www.gov.uk/government/publications/valuing-greenhouse-gas-emissions-in-policy-appraisal/valuationof-greenhouse-gas-emissions-for-policy-appraisal-and-evaluation.
BEIS (2018), “Updated short-term traded carbon values used for modelling purposes (2018)”,
UK Department for Business, Energy and Industrial Strategy, London,
www.gov.uk/government/publications/updated-short-term-traded-carbon-values-used-for-modellingpurposes-2018.
Beukers, E., L. Bertolini and M.T. Brömmelstroet (2012), “Why Cost Benefit Analysis is perceived as aproblematic tool for assessment of transport plans: A process perspective”, Transport Research Part A,
46/1, pp. 68–78, http://doi.org/10.1016/J.TRA.2011.09.004.
Bonnafous, A. (2021), “Analysis of the system of transport appraisal implemented in France and somemethodological innovations”, Paper commissioned for the ITF Roundtable “Broadening Appraisal to
Capture the Full Impact of Transport Investments”, 29 September–1 October 2021 [available on
request].
Bonnafous, A. and S. Masson (2003), “Evaluation des politiques de transports et équité spatiale”[Evaluation of transport policies and spatial equity], Revue d’Economie Regionale Urbaine, 2003/4,
pp. 547–72, http://doi.org/10.3917/reru.034.0547.
Börjesson, M. et al. (2014), “Land-use impacts in transport appraisal”, Research in Transport Economics,
47, pp. 82–91, http://doi.org/10.1016/J.RETREC.2014.09.021.
Carbon Pricing Leadership Coalition, “Report of the High-Level Commission on Carbon Prices”,
International Bank for Reconstruction and Development and International Development Association(World Bank), Washington, DC, www.carbonpricingleadership.org/report-of-the-highlevel-commissionon-carbon-prices.
Cascetta, E. (2021), “Broadening quantitative appraisal in transportation decision making processes
through effective stakeholders engagement”, Presentation at ITF Roundtable “Broadening Appraisal to
Capture the Full Impact of Transport Investments”, 29 September–1 October 2021, www.itfoecd.org/broadening-quantitative-appraisal-transportation-decision-making-processes-througheffective.
Cascetta, E. and A. Carteni (2021), “Strengthening and broadening quantitative appraisal in transportdecision-making processes through effective stakeholder engagement”, Paper commissioned for the ITF

52

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

REFERENCES

Roundtable “Broadening Appraisal to Capture the Full Impact of Transport Investments”, 29 September–
1 October 2021 [available on request].
Cascetta, E., A. Cartenì and M. Montanino (2019), “Decision-making processes in transportation
planning: a taxonomy and four theoretical models”, in Venezia, E. (ed.), The Decision-making Process for
Infrastructural Investment Choices, Franco Angeli, Milan, pp. 112–43.
Cascetta, E. et al. (2022), “A cognitive and participative decision-making model for transportation
planning under different uncertainty levels”, Transport Policy, Vol. 116/2, pp. 386–98,
https://doi.org/10.1016/j.tranpol.2021.12.013.
Dasgupta, P. (2021), The Economics of Biodiversity: The Dasgupta Review, H.M. Treasury, London,
www.gov.uk/government/publications/final-report-the-economics-of-biodiversity-the-dasgupta-review.
Di Ciommo, F. (2021), “Broadening appraisal to incorporate consideration of gender impacts”,
Presentation at ITF Roundtable “Broadening Appraisal to Capture the Full Impact of Transport
Investments”, 29 September–1 October 2021, www.itf-oecd.org/broadening-appraisal-incorporateconsideration-gender-impacts.
DfT (2000), “Transport Ten Year Plan 2000”, UK Department for Transport, London,
www.railwaysarchive.co.uk/documents/DTLR_10YearPlan.pdf.
Di Ciommo, F. and Y. Shiftan (2017), “Transport equity analysis”, Transport Reviews, Vol. 37/2,
pp. 139–51, https://doi.org/10.1080/01441647.2017.1278647.
Dobes, L., and J. Bennett (2009), “Multi-Criteria Analysis: ‘Good Enough’ for Government Work?”,
Agenda: A Journal of Policy Analysis and Reform, Vol. 16/3, pp. 7–29, www.jstor.org/stable/43200260.
Edkins, A. et al. (2013), “Exploring the front end of project management”, Engineering Project
Organization Journal, Vol. 3/2, pp. 71–85, https://doi.org/10.1080/21573727.2013.775942.
Eliasson, J. et al. (2015), “Does benefit–cost efficiency influence transport decisions?”, Journal ofTransport Economics and Policy, Vol. 49/3, pp. 377–96, www.jstor.org/stable/jtranseconpoli.49.3.0377.
ECMT (2004), Assessment and Decision Making for Sustainable Transport, European Conference ofMinisters of Transport OECD Publishing, Paris, https://doi.org/10.1787/9789282113134-en.
Gomijn, G. and G. Renes (2013), General Guidance for Cost-Benefit Analysis, Netherlands Bureau for
Economic Policy Analysis and Netherlands Environmental Assessment Agency, The Hague,
www.cpb.nl/sites/default/files/publicaties/download/cba-guidance.pdf.
HM Treasury (2021), “Environment Discount Rate Review: Conclusion”, 2 September 2021,
UK Government, London, www.gov.uk/government/publications/green-book-supplementary-documentenvironmental-discount-rate-review-conclusion.
HM Treasury (2020a), The Green Book: Appraisal and Evaluation in Central Government, UK Government,
London, www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-centralgovernent.
HM Treasury (2020b), “Green Book Review 2020: Findings and response”, UK Government, London,
www.gov.uk/government/publications/final-report-of-the-2020-green-book-review.
HM Treasury (2018), Guide to Developing the Project Business Case: Better business cases for better
outcomes, UK Government, London,
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/749
086/Project_Business_Case_2018.pdf.
BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

53

REFERENCES

HM Treasury (2003), The Green Book: Appraisal and Evaluation in Central Government, UK Government,
London, https://webarchive.nationalarchives.gov.uk/ukgwa/20080305121602/http:/www.hmtreasury.gov.uk/media/3/F/green_book_260907.pdf.
Hyperion Infrastructure Consultancy (2020), “Reviewing Highways England’s evaluation of its capital
investment programme’s benefits: Final report”, UK Office of Rail and Road, London, May 2020,www.orr.gov.uk/sites/default/files/om/ORR-CT-19-20_Reviewing-Highways-Englands-Evaluation-ofBenefits_FINAL-REPORT_ISSUED.pdf.
ITF (2021a), Developing Strategic Approaches to Infrastructure Planning, ITF Research Report, OECD
Publishing, Paris, www.itf-oecd.org/developing-strategic-approaches-infrastructure-planning.
ITF (2021b), Travel Transitions: How transport planners and policy makers can respond to shifting
mobility trends, ITF Research Report, OECD Publishing, Paris, www.itf-oecd.org/travel-transitions-policy
makers-respond-mobility-trends.
ITF (2020a), Accessibility and Transport Appraisal: Summary and Conclusions, ITF Roundtable Reports,
No. 182, OECD Publishing, Paris, www.itf-oecd.org/accessibility-and-transport-appraisal.
ITF (2020b), Reforming Public Transport Planning and Delivery, ITF Research Report, OECD Publishing,Paris, www.itf-oecd.org/reforming-public-transport-planning-and-delivery.
ITF (2017a), The Economic Benefits of Improving Transport Accessibility: Summary and Conclusions,
ITF Roundtable Reports, No. 165, OECD Publishing, Paris, https://itf-oecd.org/economic-benefitsimproved-accessibility-transport-systems.
ITF (2017b), Strategic Infrastructure Planning: International Best Practice, Case-Specific Policy Analysis,
OECD Publishing, Paris, www.itf-oecd.org/strategic-infrastructure-planning.
ITF and FIA Foundation (2022), Gender Equality and the Role of Women in Decarbonising Transport,
International Transport Forum and FIA Foundation, Paris and London, www.itf-oecd.org/gender-equalitywomen-decarbonising-transport.
Jara-Díaz, S.R. (2007), Transport Economic Theory, Elsevier Science, Amsterdam.
Kahneman, D. (2011), Thinking, Fast and Slow, Farrar, Straus and Giroux, New York.
Kaufman, N. et al. (2020), “A near-term to net zero alternative to the social cost of carbon for setting
carbon prices”, Nature Climate Change, Vol. 10, pp. 1010–1014, https://doi.org/10.1038/s41558-0200880-3.
Khreis, H. et al. (2016), “The health impacts of traffic-related exposures in urban areas: understanding
real effects, underlying driving forces and co-producing future directions”, Journal of Transport & Health,Vol. 3/3, pp. 249–267, https://doi.org/10.1016/j.jth.2016.07.002.
Laird, J. et al. (2012), “Ex Post Appraisal: What lessons can be learnt from EU Cohesion funded transport
projects”, Paper presented at the European Transport Conference, Glasgow, October 2012,https://aetransport.org/past-etc-papers/conference-papers-2012?abstractId=3945&state=b.
Martens, K. (2006), “Basing transport planning on principles of social justice”, Berkeley Planning Journal,
Vol. 19/1, pp. 1–17, https://doi.org/10.5070/BP319111486.
Martens, K. and F. Di Ciommo (2017), “Travel time savings, accessibility gains and equity effects in cost–benefit analysis”, Transport Reviews, Vol. 37/2, pp. 1–18,
http://doi.org/10.1080/01441647.2016.1276642.

54

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

REFERENCES

Mouter, N., P. Koster and T. Dekker (2019), “An Introduction to Participatory Value Evaluation”,
Tinbergen Institute Discussion Paper 2019-024/V, http://dx.doi.org/10.2139/ssrn.3358814.
National Highways (2022), “POPE Methodology Manual: Post-opening project evaluation for major
projects”, January 2022, https://nationalhighways.co.uk/media/exypgk11/pope-methodology-note-jan2022.pdf.
NAO (2021), “Framework to review programmes: Update April 2021”, National Audit Office, London,
www.nao.org.uk/report/framework-to-review-programmes-update-april-2021/.
NAO (2020), “Lessons learned from Major Programmes”, Report by the Comptroller and AuditorGeneral, UK National Audit Office, London, www.nao.org.uk/report/lessons-learned-from-majorprogrammes.
National Infrastructure Commission (2018), “National Infrastructure Assessment”, UK Government,
London, https://nic.org.uk/app/uploads/CCS001_CCS0618917350-001_NIC-NIA_Accessible-1.pdf.
Nicolaisen, M.S. (2021), “Ex-post Evaluation of Transport Projects: Key challenges and good practices”,
Paper commissioned for the ITF Roundtable “Broadening Appraisal to Capture the Full Impact ofTransport Investments”, 29 September–1 October 2021 [available on request].
Nicolaisen, M.S. and P.A. Driscoll (2014), “Ex-post Evaluations of Demand Forecast Accuracy: A Literature
Review”, Transport Reviews, Vol. 34/4, pp. 540–57, https://doi.org/10.1080/01441647.2014.926428.
OECD (2018), “Climate-resilient Infrastructure”, OECD Environment Policy Papers, No. 14,
OECD Publishing, Paris, https://doi.org/10.1787/4fdf9eaf-en.
OECD (2009), Regulatory Impact Analysis: A Tool for Policy Coherence, OECD Publishing, Paris,
https://doi.org/10.1787/9789264067110-en.
OECD Development Assistance Committee (1991), “Principles for Evaluation of Development Assistance”,
OECD, Paris, www.oecd.org/development/evaluation/2755284.pdf.
Preston, I. et al. (2014), “Climate Change and Social Justice: An evidence review”, Joseph Rowntree
Foundation, York, www.jrf.org.uk/report/climate-change-and-social-justice-evidence-review.
Rittel, H.W.J. and M.M. Webber (1973), “Dilemmas in a general theory of planning”, Policy Sciences,Vol. 4, pp. 155–69, https://link.springer.com/article/10.1007/BF01405730.
Samset, K. and G.H. Volden (2022), “Closing the loop: Ex-ante and ex-post evaluation in order to learn
from mistakes and successes”, in Williams, T.M, K. Samset and G.H. Volden (eds), The Front-end of Large
Public Projects: Paradoxes and Ways Ahead, Routledge, Abingdon.
Shortall, R. and N. Mouter (2021), “Social and distributional impacts in transport project appraisals”, in
R.H.M. Pereira and G. Boisjoly (eds), Social Issues in Transport Planning, Advances in Transport Policy andPlanning, Vol. 8, pp. 243–71, https://doi.org/10.1016/bs.atpp.2021.07.003.
Shiftan, Y. et al. (2021), “Accounting for equity in transport appraisal”, Paper commissioned for the
ITF Roundtable “Broadening Appraisal to Capture the Full Impact of Transport Investments”,
29 September–1 October 2021 [available on request].
Spash, C.L. (2007), “Deliberative monetary valuation (DMV): Issues in combining economic and politicalprocesses to value environmental change”, Ecological Economics, 63/4, pp. 690–99,
https://doi.org/10.1016/j.ecolecon.2007.02.014.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

55

REFERENCES

Stern, N. and J. Stiglitz (2021), “The Economics of Immense Risk, Urgent Action and Radical Change:
Towards New Approaches to the Economics of Climate Change”, NBER Working Paper Series, No. 28472,
February 2021 [revised February 2022 by Stern, Stiglitz and Taylor],
www.nber.org/system/files/working_papers/w28472/w28472.pdf.
Tel Aviv–Yafo Municipality (2017), “The Strategic Plan for Tel Aviv–Yafo: The City Vision”, December
2017, www.tel-aviv.gov.il/Residents/Development/DocLib1/City%20Vision%202017.pdf.
Van Wee, B., M. Hagenzieker and W. Wijnen (2014), “Which indicators to include in the ex anteevaluations of the safety effects of policy options? Gaps in evaluations and a discussion based on an
ethical perspective”, Transport Policy, Vol. 31, pp. 19–26,
https://doi.org/10.1016/J.TRANPOL.2013.11.002.
Volden, G.H. (2018), “Public project success as seen in a broad perspective: Lessons from a metaevaluation of 20 infrastructure projects in Norway”, Evaluation and Program Planning, Vol. 69,
pp. 109–17, https://doi.org/10.1016/j.evalprogplan.2018.04.008.
Welde, M. (2017), Kostnadskontroll i store statlige investeringer underlagt ordningen med eksternkvalitetssikring [Cost performance in large government investment projects that have been subjected toexternal quality assurance], Concept Research Programme Report Series, No. 51, Ex Ante Academic
Publishers, Trondheim,
www.ntnu.no/documents/1261860271/1262010703/Concept_reeport_51_eng_web.pdf/4e9e286f03f7-491a-810d-aabcdfa28aee?version=1.0.
Welde, M. and G.H. Volden (2021), “Norwegian perspectives on ex post evaluation”, Presented at the
ITF Roundtable “Broadening Appraisal to Capture the Full Impact of Transport Investments”,
29 September–1 October 2021, www.itf-oecd.org/norwegian-perspectives-ex-post-evaluation.
Williams, K.M. et al. (2020), “Transportation Equity Scorecard: A tool for project screening andprioritisation”, Center for Transportation, Equity, Decisions, and Dollars, University of Texas at Arlington,https://rosap.ntl.bts.gov/view/dot/55950.

56

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

NOTESNotes

1 A further tension exists, which makes choosing a specific discount rate for use in the context of policies responding to climate change
problematic. On the one hand, these appraisals necessarily consider very long timeframes. The costs and benefits to future generations in the far
future are of great importance, but will only influence the results of the analysis significantly if very low (or even zero) discount rates are applied.
On the other hand, the pace of climate change is currently very fast and not in the right direction. This creates a level of urgency for action which,if reflected in discount rates, would require a high rate to encourage immediate policy responses.
2 See https://www.lisea.fr/observatoire-socio-economique-de-la-lgv-sea/.
3 See https://nationalhighways.co.uk/our-roads/post-opening-project-evaluation-pope-of-major-schemes/.
4 See https://www.gov.uk/government/collections/post-opening-project-evaluation-pope-of-major-schemes.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

57

ANNEX A

Annex A. List of Roundtable participants
Roger VICKERMAN (Chair), Emeritus Professor of European Economics, University of Kent and Visiting
Professor, Imperial College London, United Kingdom
Ernest ALBUQUERQUE, Principal Advisor, New Zealand Transport Agency
Alex ANAS, Professor of Economics, University of Buffalo, United States
Danielle BASSAN, Principal Adviser, Investments, Ministry of Transport, New Zealand
Alain BONNAFOUS, Professor Émérite, Université de Lyon, France
Robin CAMBERY, Chief Transport Modeller and Head of Modelling and Appraisal Methods, Department
for Transport, UK
Armando CARTENI, Associate Professor, Transport Planning, University of Campania, Italy
Ennio CASCETTA, Professor of Transportation Systems Planning, University of Napes Federico II, Italy
Rodrigo CONTRERAS, Ministry of Transport, Chile
Yves CROZET, Professor, University of Lyon, Laboratoire Aménagement Economie Transports, France
Rex DEIGHTON-SMITH, Project Manager, ITF
Gerard DE JONG, Research Professor, Institute of Transport Studies (ITS) Leeds, UK
Floridea DI CIOMMO, CambiaMO |changing MObility Research Institute, Spain
Guido FARA, European Court of Auditors, Luxembourg
Sandy FONG, Acting Manager, Domain Strategy, Ministry of Transport, New Zealand
Pat GILBERTSON, Transport planning consultant, Australia
Jose GOMEZ IBANEZ, Professor of Urban Planning & Public Policy, Harvard University, USA
Dan GRAHAM, Professor of Engineering, University College London, UK
Wakana HARA, Japan Institute of Country-ology and Engineering
Ilaria HENKE, Researcher, University of Naples Federico II
Haruo ISHIDA, Professor, University of Tsukuba, Japan
Asuka ITO, ITF Secretariat
Mikko JÄÄSKELÄINEN, Senior specialist, Ministry of Transport and Communications, Finland
Hiromichi KITAMA, Director for Overseas Road Projects, International Affairs Office, Planning Division,
Road Bureau, Ministry of Land, Infrastructure, Transport and Tourism, Japan
Atsushi KOIKE, Professor, University of Kobe, Japan
Martin KONING, Splott Laboratory at Université Gustave Eiffel, France
Joanne LEUNG, Chief Economist, Ministry of Transport, New Zealand
58

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

ANNEX A

Hiroshi MAKINO, Director of Road Policy Group, Japan Institute of Country-ology and Engineering, Japan
Vittorio MARZANO, Associate Professor, Department of Civil, Architectural and Environmental
Engineering, University of Naples Federico II
Waka MATSUDA, Research Institute for Road and Street, JapanDavid MEUNIER, Associate Researcher, École Nationale des Ponts et Chaussée, France
Corinne MULLEY, Professor Emerita, University of Sydney, Australia
Morten Skou NICOLAISEN, Programme Manager, Green Mobility, Aarhus Kommune, Denmark
Toshimori OTAZAWA, Professor, University of Kobe, Japan
Fabio PASQUALI, Head, Energy Efficiency and Energy Management Unit, Anas Group, Italy
Stephen PERKINS, Head of Research and Policy Analysis, ITF
Yoram SHIFTAN, Professor of Transportation System Planning, Israeli Institute of Technology, Israel
Yuval SHIFTAN, Technion, Israeli Institute of Technology, Israel
Juha TERVONEN, Senior specialist, Ministry of Transport and Communications, Finland
Diana VAZQUEZ, ITF Secretariat
Gro Holst VOLDEN, Project Manager, Department of Civil and Environmental Engineering, Norwegian
University of Science and Technology
Glen WEISBROD, Chairman, EBP Group, USA
Morten WELDE, Senior Researcher, Norwegian University of Science and Technology, Norway
Tom WORSLEY, Research Fellow, University of Leeds, UK
Pauline WORTELBOER VAN DONSELAAR, Programme Manager, Netherlands Institute for Transport Policy
Analysis
Participants provided their affiliations at the time of their participation in the Roundtable meeting.

BROADENING TRANSPORT APPRAISAL © OECD/ITF 2022

59

Broadening Transport Appraisal

International Transport Forum
2 rue André Pascal
F-75775 Paris Cedex 16+33 (0)1 73 31 25 00
contact@itf-oecd.org
www.itf-oecd.org

Published: 09/2022 | Photo credit: Igor Plotnikov / ShutterstockThis report examines what ongoing shifts in the objectives oftransport policy mean for transport project appraisal and planning
processes. Many countries are rethinking their transport policy
and planning objectives. Their focus is increasingly shifting fromproviding mobility to ensuring accessibility; giving greater priority
to equitable access for all; recognising the urgency of decarbonisingtransport; and making urban environments healthier, safer and more
liveable. The report summarises the findings of a discussion among
44 experts from 21 countries at an ITF Roundtable held from 29
September to 31 October 2021.