The National Audit Office’s
Value-for-Money Assessmentof Transport Investments

12

Discussion Paper 2014 • 12
Geraldine Barker,
Grace Beardsley, Annie Parsons
The National Audit Office, London,
United Kingdom

The National Audit Office’s Value-for-Money
Assessment of Transport Investments

Discussion Paper No 2014-12

Prepared for the Roundtable:
Ex-post Assessment of Transport Investments
and Policy Interventions(15-16 September 2014, OECD, Paris)

Geraldine BARKER,
Grace BEARDSLEY, Annie PARSONSThe National Audit OfficeLondonUnited Kingdom

August 2014

THE INTERNATIONAL TRANSPORT FORUM
The International Transport Forum at the OECD is an intergovernmental organisation with
54 member countries. It acts as a strategic think-tank, with the objective of helping shape thetransport policy agenda on a global level and ensuring that it contributes to economic growth,environmental protection, social inclusion and the preservation of human life and well-being. The
International Transport Forum organises an annual summit of Ministers along with leading
representatives from industry, civil society and academia.
The International Transport Forum was created under a Declaration issued by the Councilof Ministers of the ECMT (European Conference of Ministers of Transport) at its Ministerial Sessionin May 2006 under the legal authority of the Protocol of the ECMT, signed in Brussels on 17
October 1953, and legal instruments of the OECD.
The Members of the Forum are: Albania, Armenia, Australia, Austria, Azerbaijan, Belarus, Belgium,
Bosnia and Herzegovina, Bulgaria, Canada, Chile, People’s Republic of China, Croatia, Czech
Republic, Denmark, Estonia, Finland, France, Former Yugoslav Republic of Macedonia, Georgia,
Germany, Greece, Hungary, Iceland, India, Ireland, Italy, Japan, Korea, Latvia, Liechtenstein,
Lithuania, Luxembourg, Malta, Mexico, Republic of Moldova, Montenegro, the Netherlands, New
Zealand, Norway, Poland, Portugal, Romania, Russian Federation, Serbia, Slovak Republic,
Slovenia, Spain, Sweden, Switzerland, Turkey, Ukraine, United Kingdom and United States.
The International Transport Forum’s Research Centre gathers statistics and conducts co-operative
research programmes addressing all modes of transport. Its findings are widely disseminated andsupport policymaking in Member countries as well as contributing to the annual summit.

Discussion Papers
The International Transport Forum’s Discussion Paper Series makes economic research,commissioned or carried out at its Research Centre, available to researchers and practitioners. The
aim is to contribute to the understanding of the transport sector and to provide inputs to transport
policy design.
ITF Discussion Papers should not be reported as representing the official views of the ITF or of its
member countries. The opinions expressed and arguments employed are those of the authors.
Discussion Papers describe preliminary results or research in progress by the author(s) and are
published to stimulate discussion on a broad range of issues on which the ITF works. Comments on
Discussion Papers are welcomed, and may be sent to: International Transport Forum/OECD, 2 rueAndré-Pascal, 75775 Paris Cedex 16, France.
For further information on the Discussion Papers and other JTRC activities, please email:itf.contact@oecd.org
The Discussion Papers can be downloaded from:www.internationaltransportforum.org/jtrc/DiscussionPapers/jtrcpapers.html
The International Transport Forum’s website is at: www.internationaltransportforum.org
This document and any map included herein are without prejudice to the status of or sovereignty
over any territory, to the delimitation of international frontiers and boundaries and to the name ofany territory, city or area.

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Table of Contents
Summary ................................................................................................................................ 4
1.

The Role of the National Audit Office ........................................................................ 6

2.

Our Approach to Assessing Value for Money ............................................................ 8

3.

The Challenges ............................................................................................................ 9

4.

Our Recommendations Leading to Improvement ..................................................... 13

Bibliography ........................................................................................................................ 16
Annex 1: Recent NAO value for money studies on transport ............................................. 18
Annex 2: Analytical framework for assessing value for money.......................................... 19

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 20143

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Summary

The UK National Audit Office (NAO) scrutinises public spending on behalf of Parliament, helping
it to hold government departments to account and helping public bodies improve performance anddelivery. We publish around 60 value for money studies each year across a range of government
activities, of which, around three of these usually cover transport topics. Our reports look at how
government projects, programmes and initiatives have been implemented and makerecommendations on how it can be improved.
Our value for money work is not strictly ex-post assessment in the usual sense of assessing a
programme once it has been in operation for some time. Due to the length of time needed tocomplete major transport investments and our remit to focus on accountability, we often carry out
an assessment of a project before its completion. In some cases, particularly for significant
infrastructure investments, a series of value for money reports is appropriate as the programme
will develop over time. These tend to focus on how the programme is being delivered, in terms of
the planning, procurement or construction phases of infrastructure projects.
We have had cause to comment on the Department for Transport’s (the Department) failure tocarry out full ex-post assessment of the UK transport projects that we have examined. Ex-post
assessments have a valuable role to play in capturing learning and feeding into decision making
about current and future projects. Given the long lead times such projects and programmes often
have before they deliver their intended benefits, ex-post assessments should be one part of a wider
programme of reviews throughout a project or programme’s life.
In carrying out our work our three main aims are to:Obtain robust evidence and analysis in order to draw sound conclusions about whether theinvestment of public money provided value for money;
Address the issues which taxpayers and their elected representatives believe are pertinent inmaking that assessment;
Draw out lessons for future programmes both within the transport sector and elsewhere ingovernment.

The first and third of these – the need for robust evidence and the desire to learn for the future –clearly have strong parallels in ‘classic’ ex-post assessment.
We base our work on a standard approach which we apply across the range of central
government’s activities and services. All our value for money work refers to an analyticalframework shown in Appendix 2 of this paper. Audit teams use this framework as a starting point
and the basis on which to develop their detailed methodologies. They flex the application of theframework according to the particular topic, applying their professional judgment and experience.
The analytical framework examines the economy, efficiency and effectiveness with which
resources are used. We also consider factors such as whether the use of resources was optimal andkey decisions were reasonable at the time they were taken.
4

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

This paper focuses on four of the most significant and most recurring challenges we encounter
when carrying out value for money work in the transport sector:








Choosing the appropriate point or points at which to assess a programme. We need tobalance a number of considerations such as allowing the programme sufficient time tobecome established, the expectations of our stakeholders for timely evaluation and the scopefor our recommendations to influence the programme going forward;
Evaluating the wider economic impacts of transport investment. Where wider economic
impacts are cited as the reason why a transport project is required, decision makers andevaluators have found it hard to quantify these impacts. However, we have seen increasing
interest recently in carrying out full ex-post assessments, including effects such as economic
growth and regeneration, and using the learning from these to inform approaches to planningand delivering future programmes. For example, the Department for Transport has also
published an evaluation strategy and an evaluation and monitoring programme.
Obtaining and interrogating data and information. While our statutory rights give us
unique access to data and documentary evidence, we too face issues of having to makejudgments on a programme’s success where there is insufficient data or information. We
provide an example of how we have used available data to test and validate decisions made;andAssessing the impact of government interventions where there are many other factors
at play. For example, transport regulatory bodies have some influence on the rate of roadaccidents through their work to enforce vehicle safety standards but road safety is also
influenced by, for example, the weather and drivers’ health.

While most of our discussion focuses on the examination of major infrastructure programmes, we also
refer to examples of other interventions by government agencies in the transport sector. In assessing
such activities we have used techniques such as benefit-cost calculations to conclude on whether
agency interventions are value for money.
Our value for money reports contain recommendations for the audited body, however the extent towhich they have been implemented is variable. Certain recommendations often recur, such as therobustness of underlying data and quality of risk management. We seek to address this by taking asystematic approach to following up our recommendations, with the aim that the Department and its
agencies will increase their focus on these areas and improve performance over time.

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

5

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

1. The Role of the National Audit Office

The work of the National Audit Office is part of the UK accountability process for funds granted by
Parliament (Figure 1).
Figure 1: The accountability process

Source: National Audit Office
The National Audit Office is headed by the Comptroller and Auditor General (C&AG) who has thepowers to undertake financial audits of government accounts and value for money audits. We produce
around 60 value for money reports each year, across the range of government’s activities. Each value
for money report will:

consider the economy, efficiency and effectiveness with which government departments andother public sector bodies have used their resources;
form a judgment on whether value for money has been achieved; andmake recommendations on how it could be improved.

Most value for money reports are presented to the Committee of Public Accounts (the Committee) in
Parliament. The Committee uses our report as the basis of a hearing at which the Committee willquestion the government officials responsible about our findings. It subsequently publishes its own
reports, including recommendations to which the Government must respond.

6

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Each year around three NAO reports are published on transport topics (see Appendix 1). Wheretransport projects or activities are directly funded by the Department for Transport (the Department),
we have unique access to conduct value for money audits. We also examine projects which the
Department delivers via local government or other partners in the private and third sectors. We selectprojects and areas of expenditure for audit on the basis of various criteria, including their financial
value, strategic importance, level of risk and topicality. Our work in recent years has included:







Major investments in rail infrastructure, including the construction and sale of the highspeed rail line linking London and the Channel Tunnel (‘High Speed 1’), and the setting upof the programme to build a high-speed rail line linking London and northern England(‘High Speed 2’);Major procurements such as the purchase of new trains for intercity services and the‘Thameslink’ service across London;
Operational activities of the Department and its agencies, for example the inspection ofheavy goods vehicles by the Vehicle and Operator Services Agency;
Interventions by the Department, for example its decision in 2009 to terminate thefranchise for the East Coast Mainline rail service, held by a private company andsubsequently transferred to be run by a public operator;
Delivering transport improvements at the local level, for example the maintenance ofroads by both the Highways Agency and local authorities.

Our reports are not strictly ex-post assessments of the Department’s programmes, in the sense of
assessing a programme once it is fully in operation, as defined by Campbell and Rozsnyai (2002). We
tend to examine major investments before they have been completed and their benefits fully realised,focusing on the planning, procurement and construction phases. This is due in part to the interests of
our audience – Parliament and the taxpayer - in holding departments to account for the way they use
public money, at a sufficiently early stage to influence the remainder of the programme, and to ouraim to help improve performance and service delivery. It is also due to the scale and duration of the
programmes that we examine which take many years to come into operation. Of our recent work, ourthird report on the Channel Tunnel Rail Link, now known as High Speed 1, published in March 2012,
most resembles a classic ex-post assessment of transport investment. It involves a reworking of theoriginal business case using available data. We also commented specifically on whether the project
had been completed to time and cost and on whether forecasts of passenger demand had beenachieved.
Within our reports, when relevant, we have commented on the extent to which the Department has
evaluated its transport programmes. In general we have found a lack of ex-post assessment. For
example, in our review of the Department’s funding of local authority major capital schemes we foundthat the Department did not enforce requirements for local authorities to evaluate their schemes, and it
had received evaluation reports for only two of seven projects which required them. More recently, theDepartment has taken positive steps. In March 2013, the Department produced a monitoring andevaluation strategy, followed by a document in October 2013 setting out its monitoring and evaluation
programme which it will update annually.

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 20147

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

2. Our Approach to Assessing Value for Money

Key to our approach is the requirement to conclude on value for money. Our analytical framework(described below and reproduced in Appendix 2) helps us to define value for money and how we willassess it, at the beginning of a piece of work. This will lead ultimately to the value for money
conclusion in the published report.
The analytical framework has three key components:

Establishing exactly what we will examine; 1
Identifying what good performance would look like, taking into consideration theenvironment at the time which could constrain what is achievable; andAssessing actual performance against ‘what good looks like’ to draw conclusions andidentify recommendations.

The framework draws on traditional approaches to assessing value for money, examining the:

Economy with which resources are used;
Efficiency of the relationship between the output of goods, services or other results and theresources used to produce them; andEffectiveness, i.e. the relationship between the intended results and the actual results of the
projects, programmes and services.

The value for money conclusion, included in all of our value for money reports, is the C&AG’s viewon how well resources have been used to achieve particular objectives. It is usually founded on some
type of comparative assessment such as cost-benefit analysis where we evaluate the net benefits of aprogramme against a counterfactual; performance benchmarking against appropriate comparator
programmes, or comparison to a model of good practice. There are times when there is insufficient
information to conclude on value for money. In these cases, we may decide how far the audit body is
at fault, or whether data limitations are a feature of the environment in which the body operates.

1

We may choose to examine the objectives and rationale for the programme; progress in delivering the
programme to time and budget; and looking forward, whether risks to delivering the next stage of the
programme are being managed effectively. This includes whether the Department is collecting theinformation and establishing the baselines to enable it to conduct ex-post assessments in the future.

8

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

3. The Challenges

There are a number of specific issues related to transport investments and projects which present uswith challenges when conducting our value for money assessments and could also present problems
for ex-post assessments. We discuss these further below, with reference to case examples taken fromour reports. The issues and case examples we will cover include:

Determining when is the right time to assess the value for money of a programme: Department for Transport: The completion and sale of High Speed 1
 Department for Transport: High Speed 2;\



Evaluating whether wider economic impacts have been achieved: Department for Transport: The completion and sale of High Speed 1;

Making an assessment when there is insufficient data or documentary evidence: Department for Transport: The InterCity East Coast Passenger Rail Franchise;Assessing the impact of an intervention when there are other factors and agencies at play: Vehicle and Operator Services Agency: Enforcement of regulations on commercial
vehicles Determining when is the right time to assess the value for money of a programme.

It takes time to construct major transport projects and bring them into service. For example, theCrossrail service which is currently being constructed east-west across London will open fully in
2019, ten years after construction began in 2009. Phase 1 of High Speed 2, a new rail link from
London to northern England, is expected to be operational in 2026, some 14 years after the
department’s decision to go ahead with the project. When considerable sums of public money are
spent there is a natural desire to establish whether that expenditure is worthwhile. Crossrail will cost
£14.8 billion, while High Speed 2 is expected to cost around £42.6 billion2. Our key stakeholders are
eager for information to inform other projects and are not inclined to wait for a full ex-post assessment(which might be many years hence).
Our challenge is to establish at what point an NAO assessment will be most valuable. We are now
taking the approach of examining the Department’s most significant programmes at key stages during
their initiation, development and delivery. This enables the Department to address the risks and issues
we identify at an earlier stage. We talk more about this in Part III of this paper. For example we firstexamined High Speed 1 in 2001, focusing on the financing of the project. We returned to the topic in
2005, reviewing progress in construction and revised expectations for public funding and in 2012 toreport on the completion of the programme and the sale of the operating company, High Speed One
Limited.

2£42.6 billion is the cost of both Phase 1 and Phase 2 of High Speed 2.

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 20149

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

We are adopting a similar approach to High Speed 2, a programme to develop a new high-speed rail
network between London, the West Midlands, Manchester and Leeds. The Department for Transport
took the decision to develop Phase 1 of the scheme in January 2012 and the line is expected to be
operational in 2026. We published our first report in May 2013, on the Department for Transport’s
progress in putting in place the foundations for successful programme delivery, examining
specifically:

the Department’s case for building a high-speed railway;
the Department’s cost estimate for Phase 1 of the programme, between London and the West
Midlands and its assessment of the programme’s affordability; andhow the Department has set up the programme.

We evaluated the Department’s performance against the key elements of success which we had
identified in our Guide to Initiating Successful Projects, which is based on our experience of 40 major
government projects. We examined specifically whether:

the programme had a clear rationale and objective. We concluded that the Department had
poorly articulated the strategic case for high-speed rail;the programme was well costed and affordable. We concluded that cost estimates were at an
early stage and there were risks to affordability;plans for programme delivery were realistic and feasible. We concluded that the timetablefor planning phase one, from London to the West Midlands, was challenging; androles and governance arrangements were clear. We concluded that there were weaknesses inprogramme management which the Department was taking steps to improve.

During the construction phase of a project we would typically examine a major programme once a key
stage had been completed, looking in more depth at issues such as whether the programme is being
delivered on time and within budget; and whether risks to delivery are being effectively managed. Our
recent reports on Thameslink and Crossrail are examples of such examinations.
Even when the project is completed and is operational an issue remains for us as to when our final and,in effect, ex-post assessment should be carried out, as it can take years for the full benefits to berealised. However experience has shown that some of the information that we need for our assessment,
particularly on costing is most likely to be available shortly after the completion of the project.
Our March 2012 report on the completion and sale of High Speed 1 included a cost benefit analysis
which involved reassessing the costs and journey time saving benefits of the project. The analysis
enabled us to determine the likely benefits and costs to the taxpayer. We made the following
conclusion which illustrates the difficulties in drawing a value for money conclusion at this stage:
In assessing whether a project will deliver value for money, the Department considers a wide range ofimpacts that a project might have, some of which it can quantify and others on which it has to makemore qualitative judgements. The original business case in 1998 was based on benefits to transport
users, from faster journey times and increased rail capacity, and regeneration benefits. The data
available only allows us to estimate that the value of journey time savings benefits, over a 60-year
appraisal period to 2070, would be £7,000 million. We estimate that the net costs to the taxpayer to
2070 would be £10,200 million. On these measures we would conclude that the project is not value formoney. When including other impacts from the project, some of which are unmeasurable, we accept
that such a clear conclusion is not possible. The Department, however, would need to demonstrate
10

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

that these benefits are going to be at least £8,300 million, giving a higher contribution than originally
expected, to achieve the benefit-cost ratio of 1.5 to 1 estimated in 1998.
At the time of publishing the report in March 2012, the Department was developing a plan to evaluate
the High Speed 1 project. The Department’s view was that a robust assessment of transport benefits
from the high speed line could only be made after December 2012, three years after the introduction of
the domestic high speed services. This was based on standard industry demand forecasting guidance
which states that the expected change in demand would be complete three years after a major service
change, with the majority of change occurring by the end of the first year. However, we believed andstated that the Department should already have had an evaluation plan in place which identified thedata it needed to collect and monitor to measure project benefits. We acknowledged that the
Department had started work to identify the method it would use to evaluate wider economic impacts
and regeneration benefits and how it would establish a counterfactual. We felt however that there wasa risk that the Department would not be able to measure robustly the impact of the project because it
was not able to demonstrate that it had collected the information it would need. The Department iscurrently carrying out its evaluation of the project, and a report is expected to be published in 2014.

Evaluating whether wider economic impacts have been achieved
One of the challenges for both ex-ante and ex-post assessments of transport projects is themeasurement of wider economic impacts. This is illustrated by the High Speed 1 programme. The
main project benefits the Department identified in 1998 were benefits to transport users, from fasterjourney times and increased passenger rail capacity, and regeneration benefits. The Department choseto route the line through east London to stimulate regeneration. Including a monetary value forregeneration was unconventional for a public transport project at the time because the Department did
not have an agreed method for calculating such benefits. The Department’s approach for other projects
such as Thameslink, Crossrail and High Speed 2 has been to base the initial benefit cost ratio on thetransport benefits and to then produce an additional benefit cost ratio which includes an assessment ofwider economic impacts. In the case of High Speed 1 it valued the expected regeneration benefits
based on the 50,000 jobs that it originally estimated the line would create at sites around the threeinternational stations, and the amount that the Government would be prepared to pay to create thesejobs through other interventions.
When we examined the project in 2012, we found that London and Continental Railways (whodelivered the project) had commissioned analysis of the approved developments at all three locations
and estimated they would support at least 70,000 jobs. The Department had yet to review theregeneration benefits and told us that they intended to do so after the 2012 Olympic Games, when thelegacy plan for the Olympic Park had been implemented. This was because the high speed line has astation at Stratford, where the Olympics took place. Under its transport analysis guidance theDepartment would need to identify the impact, for example, on unemployment in areas served by thehigh speed line to quantify regeneration benefits. The Department told us that these impacts are not aseasy to isolate as the impact on transport benefits, where the Department already collects data, and a
specific study would be required to assess the wider economic and regeneration benefits.
As can be seen in paragraphs 26 and 27 above we did not attempt to quantify the wider economic
impacts ourselves, but used our reworking of the cost benefit analysis to show the scale of benefit
needed if the project was to deliver value for money.

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201411

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

It appears to us that in the UK there is an increasing interest in ex-post assessments and in making
investment decisions using benefit-cost ratios which capture the full benefits of the project. For thelatter, this involves having a better understanding of the wider impacts that infrastructure projects suchas those in transport generate. For example, the July 2014 the Committee of Public Accounts report on
Crossrail recommended that the Department should improve its understanding of the wider economic
benefits of transport projects and include this in its investment decisions. The Department told theCommittee that it agreed it needed to do more work on understanding wider economic benefits, suchas changing land use, since these could not currently be quantified in the benefit-cost ratio.

Making an assessment when there is insufficient data and documentary evidence
The availability of data will usually determine what methods we use and the focus of the report. While
we can carry out or commission primary research, in the transport sector we generally analyse existingdatasets held by the Department, regulatory bodies or other stakeholders such as the rail infrastructure
manager, Network Rail. These datasets vary in their complexity, completeness and comparability, andwe plan our work taking into account their limitations.
In 2011, we published a report which examined the Department’s decision to terminate the InterCity
East Coast franchise, in response to the National Express Group stating that it would no longer
financially support the franchisee, National Express East Coast. We wanted to determine whether the
Department’s decision to terminate the franchise offered better value for money than renegotiating theterms of the contract with National Express or negotiating a consensual exit from the contract. The
Department had not carried out this analysis and believed that relaxing contract terms would haveencouraged the operators of other rail franchises to seek similar deals, exposing the taxpayer toincreased costs. The Department’s data was not held in a way that would easily allow analysis of theavailable options which were to renegotiate the terms of the franchise with the operator; negotiate aconsensual exit; or terminate for contract default. We therefore drew together the Department’s dataon the actual and forecast financial performance of train operators facing financial difficulties, andadjusted the data so that it was comparable. We used this evidence to develop a financial model thatallowed us to calculate the potential costs to the taxpayer of the three options available to the
Department. The analysis helped to support our overall value for money conclusion that the
Department’s decision to terminate the franchise was the best means of protecting the taxpayer, whencompared to other potential options.

Assessing the impact of an intervention when there are other factors and agencies at play
A challenge in assessing the operations of government transport agencies is attributing ultimate
outcomes to their activity. For example, a number of factors contribute to road safety. Although the
Vehicle and Operator Services Agency’s work (described below) had an impact on road safety, there
are many other factors and agents other than the Agency which affect the rate of road accidents,
including weather patterns, economic growth rates, drivers’ health and drivers’ behaviour patterns.
In January 2010, we published a report which examined the work of the Vehicle and Operator
Services Agency (VOSA). Until its replacement by the Driver and Vehicle Standards Agency in April
2014, VOSA was the executive agency of the Department for Transport which was responsible for
ensuring that Heavy Goods Vehicles (HGVs) and Public Service Vehicles (PSVs) complied with awide variety of roadworthiness and traffic regulations covering physical maintenance, weight limits
and drivers’ hours. We wanted to determine whether the benefits from the Agency’s enforcement
activities met its costs. One aspect of our methodology was to conduct a benefit-cost calculation of its
12

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

enforcement work. This enabled us to conclude that the benefits are likely to exceed the Agency’s
expenditure, but in our opinion it could deliver significantly better value for money. To carry out ourassessment, we:estimated the proportion of vehicles with defects for which the Agency checks that go on
to cause accidents;
used this to estimate the number of accidents prevented by VOSA’s inspections;estimated the average benefit of preventing an accident involving an HGV; andapplied this to the number of accidents prevented by VOSA’s activities to estimate theirvalue to the economy.

We calculated that if the Agency’s roadside checks prevented 283 accidents and the average value ofeach of these was £143,529, the roadside checks would have delivered £40.7 million of benefits. This
compares to the Agency’s expenditure of £32.9 million on HGV enforcement in 2008-09. We also
performed sensitivity analysis by varying the number of accidents prevented by the Agency and theaverage benefit of a prevented accident by up to +/- 50 per cent and then looking at the effect different
combinations of these changes had on the benefits delivered by roadside checks. This suggested thatroadside checks could deliver a minimum of £10 million and a maximum of £91 million benefits. The
benefit exceeded the Agency’s expenditure in 2008-09 in 44 per cent of the combinations analysed.
This assessment enabled us to conclude that the benefits are likely to exceed the Department’s
expenditure and the result was included in our value for money conclusion on VOSA’s enforcement
activities. The conclusion, which found that the Agency had achieved ‘satisfactory results’ also
highlighted other issues such as the potential to improve value for money through system
improvements such as staff deployment, better location of the sites for checking vehicles and closer
working with the Department. This example also serves to illustrate how our value-for-money
conclusions take account of multiple aspects to reach an overall judgment. Benefit-cost or otherquantitative analysis plays an important part but is not the whole picture.

4. Our Recommendations Leading to Improvement

Although our assessments are not strictly ex-post evaluations, our value for money reports include anumber of recommendations for the audited body. As explained earlier, the Committee of Public
Accounts will use our reports as the basis of a hearing to question the government officials responsible
and subsequently publish its own report. Its report will also contain recommendations which the
Department must respond to and a summary of its response is made public in HM Treasury’s Treasury
Minutes.
In our reports on major transport projects, the recommendations generally address the same areas
identified by De Jong et al. (2013):

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201413

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS






Improvement of cost and benefit estimation approaches;
Risk management measures;
Increasing accountability; andClarifying project scope and objectives.

Some examples of how we have addressed these themes include:







Improvement of cost and benefit estimation approaches: Our recommendations in thisarea have focused on the Department’s work in ensuring and making more transparent therobustness of its cost benefit analysis.
Our March 2012 report on High Speed 1, recommended that the Department ensures its
demand forecasts, which feed into the benefit cost estimation, are subject to rigorous scrutiny
and scepticism. We also recommended that it should assess the benefits under a range ofdifferent scenarios, perform a sensitivity analysis of key assumptions and a sense check tounderstand the reality of meeting forecast demand. In the November 2012 Treasury Minutes,
the Department stated that it now takes greater account of downside risks, and typicallyundertakes extensive sensitivity analysis to test the robustness of the business case to varying
input assumptions.
Our May 2013 report on High Speed 2 highlighted a number of issues with the calculation of
the benefit cost ratio in the economic case including errors in earlier calculations, the need to
update the data underpinning key assumptions and the lack of analysis of the effect ofpremium pricing on forecast passenger demand. The revised business case in October 2013
did seek to address some of these concerns by updating some of the data and revising someassumptions for example around journey time savings, and by quoting the benefit cost ratio
as a range rather than point estimates to recognise explicitly the uncertainty of the economic
case.
Risk management measures: The importance of risk management was particularly evident
in our examination of the failure of Metronet in 2009. Metronet was a private infrastructurecompany responsible for the maintenance and upgrade of sections of the London
Underground. It went into administration in July 2007. Although Transport for London (TfL)
had guaranteed 95 per cent of Metronet’s borrowing, the Department had also informallygiven assurances to investors that it would guarantee the borrowing3. When Metronet failed,the Department had to make a grant payment of £1.7 billion to help London Undergroundpurchase Metronet’s debt obligations, a sum that would otherwise have been repaid over the
30 year life time of the contracts. The Department was exposed to this risk but lacked directways of gaining assurance over the management of the risk.4 We advised that the
Department should: collect and analyse a range of financial and performance data held byparties to the contract or available independently; request regular risk reports from London
Underground and TfL as the contracted clients; and review the devolved body’s
understanding of the key risks to the project to allow it to identify and investigate any issues
relevant to the management of its own risk.

3

The Department provides a grant to TfL. London Underground is a subsidiary of TfL.

4

Under the Greater London Authority Act 1999, strategic and investment responsibility for London
Underground was devolved to TfL and the Mayor of London. The Secretary of State of the
Department could only direct the Mayor to make changes to transport strategy where it would beinconsistent with national policy and have an adverse effect outside London. DfT was not a party to
the contracts and had no direct influence over performance.

14

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS







The quality of the Department’s oversight of large programmes is an issue which we
continue to examine. For example, our recent report on Crossrail commented favourably on
the Department and Transport for London’s oversight of that programme, highlighting theuse of a probability-based approach to forecasting the delivery date and final cost, and tomonitor and manage risks which allows the sponsors and Crossrail Limited to identify when
there are risks to delivery and to take action to mitigate those risks.
Increasing accountability: In our December 2012 report on the cancellation of the InterCity
West Coast franchise competition, we found that staff in the project team reported todifferent parts of the organisation which meant no one person oversaw the whole process, or
could see patterns of emerging problems. We recommended the Department appoint
someone with sufficient seniority to oversee each significant commercial transaction andmajor project, with the knowledge, skills and the authority in the Department to take action if
things are going wrong. The Department has since taken action to review its existing Senior
Responsible Owners (SROs) to ensure they have the right seniority, experience and expertisefor the projects for which they are currently responsible. Additionally, it has provided
training for SROs to ensure they fully understand their responsibilities.
Clarifying project scope and objectives: As can be seen in paragraph 23 above, one of ourcriticisms of the early preparations for High Speed 2 was the Department’s poor articulationof its strategic case for the route. The Committee of Public Accounts subsequently called forthe Department to publish detailed evidence which clearly showed why it considered High
Speed 2 to be the best option for increasing rail capacity into London, improving
connectivity between regional cities and rebalancing the economy. The Department sought
to address the Committee’s concerns in its revised strategic case for the project in October
2013.

The extent to which recommendations are implemented has varied and in recent years we have sought
to address this by following up implementation more systematically. As can be seen in paragraph 39
above, some issues such as the robustness of underlying data for business cases and the quality of risk
management are recurring themes in our work. By returning to these themes, we hope to increase
focus on them within audited bodies and raise standards over time. Moreover, where we haveconducted early examinations of programmes, we examine specifically in subsequent reports whether
earlier recommendations have been implemented. One example of this is our final report on High
Speed 1 in May 2012, in which we commented that the Department had not yet reassessed the project
costs and benefits since 2001 despite making a commitment to the Committee of Public Accounts to
do so.

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201415

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Bibliography

Campbell, C. & Rozsnyai, C., 2002, Quality Assurance and the Development of Course Programmes.
Papers on Higher Education Regional University Network on Governance and Management ofHigher Education in South East Europe Bucharest, UNESCO.
Comptroller and Auditor General (all reports available at www.nao.org.uk):
July 2014, Department for Transport: Procuring new trains, Parliamentary Session 2014-15,
HC 531
June 2014, Department for Transport and Highways Agency: Maintaining strategic
infrastructure: roads, Session 2014-15, HC 169
May 2013, Department for Transport: High Speed 2: A review of early programme
preparation, Session 2013-14, HC 124
December 2012, Department for Transport: Lessons from cancelling the InterCity West Coastfranchise competition, Session 2012-13, HC 796
October 2012, Department for Transport: Funding for local transport: an overview, Session
2012-13, HC 629
March 2012, Department for Transport: The completion and sale of High Speed 1, Session
2010-2012, HC 1834
December 2011, Guide: Initiating successful projects
March 2011, Department for Transport: The Intercity East Coast passenger rail franchise,
Session 2010-2011, HC 824
June 2010, Department for Transport: Increasing passenger rail capacity, Session 2010-11,
HC 33
January 2010, Vehicle and Operator Services Agency: Enforcement of regulations oncommercial vehicles, Session 2009-2010, HC210
June 2009, Department for Transport: The failure of Metronet, Session 2008-2009, HC 512
July 2005, Department for Transport: Progress on the Channel Tunnel Rail Link, Session
2005-2006, HC 77
March 2001, Department of the Environment, Transport and the Regions: The Channel
Tunnel Rail Link, Session 2000-2001, HC 302.
16

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

De Jong et al. (2013), How to Build Major Transport Infrastructure Projects within Budget, in Timeand with the Expected Output; A Literature Review, Vol. 33, Issue 2, pp. 195-218.
Department for Transport:
October 2013, Monitoring and Evaluation Programme;
March 2013, Monitoring and evaluation strategy
HM Treasury (all reports available at: www.gov.uk/government/collections/treasury-minutes):
May 2013, Treasury Minutes, Cm 8613, pages 34-38;
November 2012, Treasury Minutes, Cm 8467, pages 25-29.
House of Commons Committee of Public Accounts, July 2014, Crossrail, Eighth Report of Session
2014–15, HC 1062
National Audit Office, May 2011, Department for Transport: Local Authority Major Capital Schemes

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201417

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Annex 1
Recent NAO value for money studies on transport

All reports available at www.nao.org.uk:
Procuring new trains (July 2014);



Maintaining strategic infrastructure: roads (June 2014);



Crossrail (January 2014);Progress in delivering the Thameslink programme (June 2013);;

High Speed 2: a review of early programme preparation (May 2013);

Lessons from cancelling the InterCity West Coast franchise competition (December 2012);;Funding for local transport: an overview (October 2012);



The completion and sale of High Speed 1 (March 2012);

Reducing costs in the Department for Transport (December 2011);



Local Authority Major Capital Schemes (May 2011);



Regulating Network Rail’s efficiency (April 2011);



The Intercity East Coast passenger rail franchise (March 2011);

Procurement of the M25 private finance contract (November 2010);



Increasing passenger rail capacity (June 2010);

Highways Agency: Contracting for Highways Maintenance (October 2009);

The Department for Transport: The failure of Metronet (June 2009).

18

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Annex 2Analytical framework for assessing value for money

The framework below is a key reference source for NAO auditors scoping and planning Value forMoney work. It provides a guide to the types of questions we will need to answer to draw clear Valuefor Money conclusions on topics across the range of government activities. The framework isnecessarily broad and acts as a foundation for the development of audit teams’ work. Its application toa particular topic will be informed by auditors’ professional judgment and experience. Neither this nor
any other tool can provide a simple, ‘mechanical’ conclusion on value for money.

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201419

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

20

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201421

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

22

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 2014

THE NATIONAL AUDIT OFFICE’S VALUE-FOR-MONEY ASSESSMENT OF TRANSPORT INVESTMENTS

Geraldine Barker et al. — Discussion Paper 2014-12 — © OECD/ITF 201423

International Transport Forum
2 rue André Pascal
75775 Paris Cedex 16
itf.contact@oecd.org
www.internationaltransportforum.org