Reviewing Highways England's evaluation of its capital investment programme's benefits: Final Report ORR/CT/19-20 Hyperion Infrastructure Consultancy Ltd 
 

## Contents

Executive Summary ........................................................................................................................... 3 
Findings and Recommendations .................................................................................................... 4 
1 
Background ............................................................................................................................... 7 
1.1 
History and purpose of post-opening project evaluation .................................................... 7 
1.2 
Benefits Management within Highways England ................................................................ 7 
1.3 
Benefits Management Manual ........................................................................................... 9 
2 
Project Methodology ............................................................................................................... 11 
3 
Highways England's Evaluation of Benefits............................................................................... 12 
3.1 
Evaluation process for Major Projects .............................................................................. 12 
3.2 
Application of evaluation process to different programmes ............................................. 16 
3.3 
Changes to evaluation process ......................................................................................... 17 
4 
Specific Lines of Enquiry 
........................................................................................................... 20 
4.1 
Resources and capabilities ............................................................................................... 20 
4.2 
Governance and assurance .............................................................................................. 21 
4.3 
Evaluation data ................................................................................................................ 22 
4.4 
Evaluation of outturn costs .............................................................................................. 22 
4.5 
Publication of evaluation reports ..................................................................................... 23 
4.6 
Feedback for future investment decisions ........................................................................ 25 
4.7 
Future developments ....................................................................................................... 25 
5 
Scheme Review........................................................................................................................ 26 
5.1 
Overview of the evaluation of RIS1 schemes .................................................................... 26 
5.2 
Detailed review of sample schemes.................................................................................. 27 
5.3 
Findings from scheme review ........................................................................................... 28 
6 
Comparison with other organisations ...................................................................................... 29 
6.1 
UK Public Sector ............................................................................................................... 29 
6.2 
International practice 
....................................................................................................... 29 
 

## Executive Summary

Evaluation of capital investment programmes is an important mechanism in assuring the investment and assessing whether the expected costs and benefits have been achieved, and to learn lessons for future investment decisions. In 2019, Hyperion Infrastructure Consultancy Ltd (Hyperion) - working with associates Elliott Asset Management Ltd, Infrata Ltd and Pragmatex Ltd - was commissioned by ORR to carry out a review of Highways England's processes for evaluating and assessing the benefits realised from its capital investment, and how these processes are being implemented, including the publication of the postopening project evaluation (POPE) reports. 

The review covered three strands of Highways England's capital portfolio: 

 
Major projects, including large-scale complex infrastructure projects (CIP), regional investment programme (RIP) and the smart motorway programme (SMP);  
 
Smaller scale enhancements such as those delivered through its congestion relief programme; and 
 
Ring-fenced or 'designated' investment funds. 
The review found that Highways England has a well-established approach to evaluating the benefits delivered by major schemes through the post-opening project evaluation, or POPE, process.  The purpose of the POPE process is to: 

1. 
Assess whether schemes have delivered the anticipated value for money; 
2. 
Validate the accuracy of the estimated scheme costs, impacts and benefits which were agreed as part of the business case for investment, and use this to improve future scheme appraisals; and 
3. 
Promote transparency and accountability to Highways England's stakeholders through the publication of POPE reports. 
The POPE process compares well with other organisations in the UK and internationally and Highways England are considered "best in class" by DfT when it comes to evaluation. 

There have been three significant changes that have impacted on Highways England's evaluation approach during RIS1: Evaluation was brought in-house during 2017 having formerly been carried out by an external consultant acting on behalf of Highways England. This required Highways England to create, resource and upskill an internal evaluation team but should enhance internal capabilities and improve knowledge retention, allowing Highways England to take a wider view of evaluation. Introduction of a benefits management approach in 2018 has provided a common framework for benefits management and evaluation across all projects, programmes and portfolios and, ultimately, will contribute to a benefits management culture across the organisation.  The aim of benefits management is to help projects to deliver the intended outcomes by focussing on the identified benefits, then ensuring the solution is designed and implemented to realise these benefits. 

Highways England is introducing a new process for approving and publishing POPE reports in line with the Company's transparency policy, which should reduce the time between the completion of an evaluation and publication. Overall, the evaluation of benefits from its capital programme is something that Highways England does well.  The evaluation process is well considered and appears to be working well across both Major Projects (MP) and Operations Directorate (OD). Highways England's approach compares well with other organisations.  The various changes that have occurred during RIS 1 should improve the evaluation process further but the transition led to an interruption to the publication of POPE reports which Highways England is working hard to address. 

Going forward, a wider, more sophisticated approach to evaluation is being taken with bespoke studies, thematic reviews and programme-level analysis being used, proportionately, to build the evidence base to support future investment decisions across the whole portfolio.  The challenge will be to evolve the process, while ensuring consistency with previous evaluations. 

## Findings And Recommendations The Key Findings Of The Project Are Summarised Below.

 
In line with many public sector organisations, Highways England is in the process of increasing the focus on benefits as part of project delivery. Partly due to its long-established POPE evaluation process, Highways England compares well with other organisations and is considered "best in class" by DfT and others.  The challenge now will be to continue to evolve the evaluation process going forward while ensuring consistency.  
 
The evaluation process is intended to (i) identify whether schemes, programmes or portfolios have had the expected impacts and whether there are any lessons that could inform future investment decisions, and (ii) to provide transparency and accountability through the publication of the POPE reports and meta-analyses. 
 
There has been a significant amount of change in how Highways England carries out evaluation 
during RIS1.  This has included the bringing in-house of the evaluation process and subsequent 
creation and building of the Evaluation Group, introducing a new process for approving and publishing POPE reports, and introducing the new benefits management manual and processes.  
There is evidence that Highways England is moving out of this transition period. 
 
Bringing the POPE process in house has enabled Highways England to retain knowledge and allow longer-term, thematic and programme-level evaluations to be undertaken. Amongst other things, this will also allow Highways England to take a wider view of benefits and focus evaluation resources proportionately, for example where they provide most value in terms of future learning. 
 
The move from Highways Agency to Highways England and the associated governance changes, interrupted the publication of POPE reports.  Highways England is working hard to catch-up and remain committed to providing transparency through the continued timely publication of POPE reports going forward. 
 
The review found that, since 2017, the average time between completion and publication of POPE reports is 27 months.  Now that Highways England has ownership of the sign-off and 
 
publication process, there should be a significant improvement in the time taken to publish POPE reports going forward. 
 
In terms of the major schemes in RIS1, at the time of this study, 37 had been opened to traffic, of which 20 had have been opened to traffic for more than one year.  Six POPEs had been 
completed for these schemes and two reports published.  Three evaluations were either on hold or had been combined with other schemes, and evaluation of the remaining 11 schemes was in progress. 
 
The new benefits management process as set out in the Benefits Management Manual is aligned with Infrastructure Projects Authority (IPA) best practice and takes a much wider view of 
benefits than the previous method which was more narrowly focussed on transport impacts (journey times, accidents, etc.) and aims to introduce flexibility whilst at the same time ensuring consistency. 
 
The new benefits management process, which includes evaluation, appears to be an improvement on the previous process in terms of scope, governance and assurance.  The early identification of evaluation evidence requirements and inclusion of appraisal data in the new Benefits Realisation and Evaluation Plan (BREP) streamlines evaluation planning.  However, the long lead-in time for schemes means that only one has followed the new BREP process within the timeframe of this study, but it is planned to retrofit the BREP process to schemes on a case by case basis. 
 
A proportionate approach is taken to the evaluation of small-scale improvements which builds on the sample-based approach adopted for the former Pinch Point Programme.  A similar approach is being adopted for similar small-scale improvements in the future, as well as Designated Funds. 
 
Due to their scale and impact, Complex Infrastructure Projects (CIPs) provide significant opportunities for learning and process improvements, but may present a challenge in terms of scope of benefits, management of change and extended lifecycle.   There may be a case for a 
multi-year evaluation programme which provides and early assessment of outcomes as well as monitoring benefits over a longer timeframe. 
 
Highways England is considering the most effective way to disseminate the findings from evaluation within the organisation and in the published POPE reports to ensure transparency. 
There are a number of recommendations that are made for Highways England's consideration, these have been grouped into themes. 

Resourcing and capabilities 
R1 
Highways England should ensure that project teams have received adequate training about the importance of benefits management and their role in the evaluation process as it continues to evolve. 
R5 
Highways England should ensure that the Evaluation Group remains fully resourced and skilled to meet the requirements of RIS2, and beyond, and the evolving evaluation process. 

Evaluation process 
R2 
Highways England should continue to develop the evaluation process to make sure it is sufficiently flexible and scalable to cover the wide range of schemes and programmes whilst ensuring consistency and comparability. 
R10 
Highways England should continue to use BREP in preference to SEP and monitor its effectiveness and further improve the evaluation process as required. 
R3 
Highways England should consider whether there are ways to provide an early assessment of scheme outcomes, as well as monitoring benefits over the longer term, so that lessons can be identified and shared in a timely way. 
R4 
Highways England should ensure that the requirements for benefits realisation and evaluation remain consistent and up to date in the respective processes and products across all projects and programmes. 
R8 
Highways England should continue to develop the evaluation methodologies to consider a broader range of other scheme impacts including impact on the wider economy and customer satisfaction. 
Publication and awareness R9 
POPE reports contain a lot of useful information and should be published in a timely manner in order to maximise their value.   Highways England should consider what steps it can reasonably take to improve the time taken between report completion and publication without compromising quality. 
R6 
 As part of the review of future reporting formats for scheme and programme evaluations that better meet the needs of their respective audiences, Highways England could consider what information this will generate that may be of use to the ORR. 
R7 
Given the potential value of the information they include, Highways England should consider 
how to raise awareness of the POPE and other evaluation reports amongst the full range of potential audiences. 
 

## 1 Background

1.1 History and purpose of post-opening project evaluation The post opening project evaluation or 'POPE' is a long-established process that has been carried out by Highways England, and Highways Agency before that, for 15-years.  POPEs are analytical reviews to evaluate the impact that infrastructure schemes or programmes have had, carried out one and five years after opening to traffic.  POPEs are carried out for every major scheme1.  For small-scale improvement programmes such as the previous Local Network Management Schemes (LNMS) or Pinch Point Programme, POPEs are carried out on a sample basis and a meta-analysis carried out for the programme as a whole. 

Over the years, POPE of major projects has developed to monitor key impacts which have been identified within the scheme's business case.  It provides a consistent evaluation methodology, applied to schemes which enables thematic or portfolio level meta-analysis to be undertaken. The POPE process aims to: 

1. Assess whether schemes have delivered the anticipated value for money; 2. Validate the accuracy of the estimated scheme costs, impacts and benefits which were agreed as 
part of the business case for investment, and use this to improve future scheme appraisals; and 
3. Promote transparency and accountability to Highways England's stakeholders. 
Independent reviews have cited the impact of POPE as an example of good practice. 

"POPE has had a much greater impact through its standardisation and clear focus ... Evaluation of Roads now follows a standardised procedure which feeds back into pre-project appraisal in a transparent way". 

Institute for Government 2017 'What's wrong with infrastructure decision making?' 
 
"Highways England routinely publish outturn project evaluations of major investments. This system has led to more accurate estimates of the likely costs of future projects". 

National Infrastructure Commission 2018, National infrastructure Assessment 
1.2 Benefits Management within Highways England In common with many public sector organisations, Highways England is in the process of implementing a benefits management approach as recommended by the Infrastructure and Projects Authority (IPA) which sets out a five-stage lifecycle for benefits management consisting of: identify, value, plan, realise and review. Benefits management is therefore a continuous activity that runs throughout a scheme's lifecycle and the POPE relates to stage 5 'Review' as illustrated in Figure 1. Benefits management encompasses all of Highways England's imperatives of safety, customer service and delivery, making benefits management a fundamental activity for Highways England. The objective of benefits management is to help schemes to deliver the intended outcomes through providing focus on the identified benefits required, then ensuring the solution is designed and implemented to realise these benefits. 

It is widely accepted that successful benefits management should be an integral element of good project and programme management practices throughout the whole project lifecycle. As a practice it standardises the approach and tools used to document the benefits, plan for their realisation, measure and track their successful delivery throughout the lifecycle of a project. In addition, for Highways England it is expected that a proactive focus on benefits management will: 

 
Enable efficiency of resource and effort 
 
Remove non 'value added' or non 'benefit focused' expenditure 
 
Enhance the value for money of projects and programmes 
 
Ensure a clear and transparent identification and management of benefits 
 
Enhance outcomes and experiences for customers 
 
Ensure the legacy of a project are clear to see after delivery 
 
Enable portfolio management decisions to be benefit led as the link to outcomes and benefits are clearly aligned 
 
Ensure a clear and transparent identification and effective mitigation of dis-benefits 
This approach focusses on the delivery and realisation of benefits throughout the scheme lifecycle, and post-opening evaluation naturally forms part of this new benefits management process.  Both benefit management and evaluation are key elements of Highways England's Benefit and Value for Money framework as illustrated below. 



The evaluation process generates evidence to demonstrate accountability for investment in major projects and benefits realisation, and POPE is a requirement of the Major Project's (MP) programme and project governance processes. However, evaluation is wider than just the production of POPE reports, and evaluation questions are now being designed to look at wider/longer-term benefits through case studies and themed metaanalysis.  In particular, Highways England recognises that wider evidence is needed about the impact of schemes on the wider economy and customer experience. As its approach to benefits management matures, Highways England is intending to take a wider and more sophisticated view of benefits and focus evaluation resources proportionately, for example, where they provide most value in terms of future learning.  The aim is to introduce flexibility and scalability whilst at the same time ensuring consistency of analysis. More generally, benefits management will allow Highways England to better develop and track the benefits case through the scheme lifecycle. The new approach can be applied to schemes, programmes and portfolios. 

## 1.3 Benefits Management Manual Highways England Produced Its Benefits Management Manual (Bmm) In 2018 Which Sets Out The Approach To Benefits Management To Be Used Across All Our Portfolios, Programmes And Projects.

The BMM describes: 

 
How the different stages of benefits management map onto the PCF stages 
 
Roles and responsibilities for benefits management activities 
It also provides templates for key benefit management products (benefit maps, benefit register and Benefits Realisation and Evaluation Plans). 

 

Highways England's benefit categories are also described including primary and secondary benefits and dis-benefits.  Primary benefits encompass all benefits in the original scheme appraisal and are therefore covered by the POPE report. Secondary (dis)benefits are additional indirect (dis)benefit on scheme(s) that either 

 
Stems from a primary benefit; or 
 
Is a softer (dis)benefit that is not perceived to significantly impact the scheme(s) or the success of the benefit category objective. 
The purpose of the BMM is to ensure there is a common purpose, structure and delivery of project benefits across Highways England; whilst ensuring flexibility to allow management of diverse projects.

## 2 Project Methodology

The six-month project comprised four steps: 

## 1. Review Documentation

- Review of Highways England documentation describing the evaluation and benefits 
management processes, reports from scoping studies and wider contextual documentation, as well as management information about the current evaluation programme.

## 2. Stakeholder Interviews

- Interviews with Highways England stakeholders involved in the evaluation and benefits 
management processes, including:
- Members of the Evaluation Group within the Chief Analysts Office in Strategy & Planning 
Directorate
- Benefits leads responsible for:
- Complex infrastructure projects and regional investment programme (both fall within 
Major Projects Directorate)
- Ring-fenced investment funds (Safety, Engineering and Standards Directorate)
- Small-scale improvements (Operations Directorate)
- Interviews with stakeholders at DfT responsible for evaluation across the department and 
within the strategic roads team.

## 3. Sample Scheme Review

- High-level review of evaluation status of all RIS1 schemes openned to traffic
- Detailed review of a sample of RIS1 schemes that have gone through, or are currently going 
through, the evaluation process

## 4. Comparison With Other Organisations

- Discussions with other publicly funded UK organisations about how they manage and 
evaluate the delivery of benefits, including: - DfT Rail Group - Transport for London - NHS
- West Midlands Police
- Comparison with international practice
Acknowledgements Hyperion would like to thank Highways England, and the other organisations, for taking part in this review and providing evidence openly and collaboratively. We would also like to thank the ORR and Highways England Project Managers for their valuable support in facilitating this review. Disclaimer This report has been prepared by Hyperion on the basis of the Form of Agreement with ORR dated 7th October 2019, in relation to contract CT/19-20. This report is for the benefit and information of ORR.   All surveys, observations, analysis and forecasts contained in the report have been made on the basis of the information available at the time of the study. In preparing the report, Hyperion has relied upon, and assumed the accuracy of, information obtained from a variety of sources.  Hyperion accepts no responsibility and will not be liable in the event that information provided during the course of the project is subsequently found to be inaccurate. 


## 3 Highways England'S Evaluation Of Benefits

3.1 Evaluation process for Major Projects POPE for major projects has typically been carried out one and five years after the scheme has opened for traffic, followed by periodic meta-analyses.  The process evaluates whether the scheme, or programme, has delivered the anticipated benefits and focuses on monitoring the following metrics over time: 

 
Safety 
 
Journey times and reliability 
 
Environment, including biodiversity, landscape, air quality and noise 
 
Delivery efficiency and cost effectiveness, including outturn costs and outturn value for money 
This is an established process that was inherited by the Evaluation Group when it was created in 2017.  Initially the new group looked to replicate the existing process but will be reviewing the most efficient and effective way of delivering evaluation as the team matures. Alignment to scheme lifecycle The Project Control Framework (PCF) was developed by Highways England to standardise the way projects are developed, managed and governed within Major Projects. For Major Projects the PCF is based on a scheme lifecycle and consists of 8 (0 to 7) stages within 4 phases: 
 
Each stage has a suite of accompanying products which cover all aspects of governance and accountability and are designed to build on each other towards scheme construction, handover to Operations Directorate (OD) and closeout. They also provide the evidence for a project's evaluation. 

A product matrix defines when products are produced, updated, refined, and reviewed. In order to progress between phases Highways England carries out a Stage Gate Assessment Review (SGAR) with key stakeholders.  A similar process is carried out under the OD PCF framework for capital renewals. Evaluation planning for POPE has traditionally formed part of the PCF; with the introduction of the Benefits Management Manual the requirements have been widened to include the consideration of benefits realisation management.  The following diagram shows how benefits management and evaluation activities and products align with the PCF stages, the key products and activities relating to evaluation are shown in red. 

As the purpose of the evaluation is to determine whether capital investments are delivering the outcomes identified in the original appraisal, there are two mandatory benefits products that must be produced for each scheme; 

 
Appraisal Summary Table (AST) and benefits register; and  
 
Benefits Realisation and Evaluation Plan (BREP). 
In addition, depending on the scale, complexity and value of investment, there are a number of optional products that may be produced including; benefits maps - which show the relationship between outcomes, benefits and outputs - benefits management strategy, benefit reports, etc. 

Scheme appraisal Once benefits have been identified and categorised using benefit maps, where applicable, and registers, they must be valued and appraised. These activities are typically completed within Highways England as part of other VfM framework activities including appraisal methods and analytical assurance work. Value and appraisal is an iterative process during development and refinement of a project's business case and, in PCF terms, happens continuously throughout PCF Stages 1-5. For an outline business case a selection of the most important benefits identified will need to be valued, to ensure the project is justified on economic grounds.  Scheme appraisal is undertaken using DfT's Transport Appraisal Guidance which, in turn, is based on the HM Treasury Green Book. Schemes are appraised to realise benefits over 60 years.  The results of the scheme appraisal are summarised in the Appraisal Summary Table (AST). For projects that are part of the PCF process, the analytical work that is required to appraise (quantify) the impacts from a scheme is identified in the Analytical Requirements Report (ARR) which sets out the analytical requirements throughout the project lifecycle, including Evaluation. The ARR is prepared by the project manager in consultation with the business partner from the Transport Planning Group. The Transport Planning Group will advise on the extent of modelling required which is usually proportionate to the size, importance and sensitivity of the project and benefit. When the final business case (PCF Stage 5) is completed, a plan for benefits realisation and evaluation is developed and aligned to these documents. Benefits Realisation and Evaluation Plan The benefits realisation and evaluation plan (BREP) is a collation of benefit details and is intended to be a short document signposting to other benefit related products and documents. It is not intended to repeat detail from benefit registers and profiles for a given project. It will summarise outputs from elsewhere. It will summarise the resulting plan and set out how success will be tracked and evaluated throughout project delivery and benefit realisation. For Major Projects, BREPs are produced at PCF stage 5: Construction Preparation, based on benefits mapping and registers produced and updated through the earlier PCF stages.  The BREP sets out the scope and objectives of the scheme, assures data availability for evaluation, and identifies any additional data that must be collected (e.g. traffic counts on local roads).  As mentioned above, the ARR also sets out the analytical requirements for each stage of the scheme, and the section of evaluation must be signed off by the Evaluation Group. BREPs are produced by the Project Teams responsible for delivery of the scheme with support from the Evaluation Group.  The role of the Evaluation Group is to set standards, provide advice and assurance, as well as carry out the evaluation itself. 

## Benefits Realisation And Evaluation Plan

| Description    | Collation of benefit details in single, signposting document            |
|----------------|-------------------------------------------------------------------------|
| Purpose        | To ensure all data is available for post-opening benefit monitoring and |
| evaluation.    |                                                                         |
| Applies to     | MP schemes                                                              |
| Produced at    | PCF Gate 5 'Construction Preparation'                                   |
| Responsibility | Produced by Project Teams with support from Evaluation Group            |

Once the BREP has been approved, baseline data and the forecasts used to inform the original investment decision are collected prior to construction 

R1 Highways England should ensure that project teams have received adequate training about the 
importance of benefits management and their role in the evaluation process as it continues to evolve. 
Post Opening Evaluation PCF Stage 7: Close Out includes the initiation of the post opening evaluation process.  This is carried out by the Evaluation Group after the scheme has been opened to traffic for one and five years, in accordance with the BREP or SEP and using the identified data sources.  A scheme has to be open to traffic for at least 12 months before an evaluation can be carried out to allow time for the traffic impacts to settle and for other data to be available (e.g. safety). 

Once the scheme has opened, Senior Analysts meet with the Project Team, plus the Benefits Lead from the relevant programme and other business partners, to discuss the scheme, any changes during construction and revise the evaluation scope if there are any changes to the earlier plan. The Senior Analyst reviews and refines the requirements based on the earlier evaluation plan and creates the evaluation project plan based on the resources available as advised by the Evaluation Programme Manager.  The analysts will then carry out the evaluation. Evaluation of benefits for each of the focus areas is typically based on the following: 

| Evaluation Focus Area                                                   |
|-------------------------------------------------------------------------|
| Safety                                                                  |
|                                                                        |
|                                                                         |
| Comparison of forecast and outturn changes in collision numbers,        |
| rates and severity using STATS19 data published by DfT                  |
|                                                                        |
|                                                                         |
| Safety trends are compared in the five years before the scheme is       |
| constructed with trends up to five years after the scheme has been      |
| fully open to traffic.                                                  |
|                                                                        |
|                                                                         |
| Change in safety levels are compared with an estimate of what might     |
| have happened on the road if the scheme had not been built to control   |
| for the wider factors which impact on road safety.                      |
| Traffic                                                                 |
|                                                                        |
|                                                                         |
| Comparison of traffic model outputs (from appraisal) with actual        |
| outturn traffic flow data, assessment of journey times and reliability  |
|                                                                        |
|                                                                         |
| Specially commissioned traffic surveys on local roads if the scheme was |
| predicted to impact on traffic flows on the local road network and      |
| other data sources are not available                                    |
| Environment                                                             |
|                                                                        |
|                                                                         |
| Non-traffic based impacts.  Site visits to confirm that the mitigations |
| identified in the Environmental Impact Assessment are in place and      |
| operating effectively                                                   |
|                                                                        |
|                                                                         |
| Traffic-based environmental impacts (e.g. noise, air quality Green      |
| House Gas (GHG) emissions are based on observed and forecast traffic    |
| data.                                                                   |
| Economic                                                                |
|                                                                        |
|                                                                         |
| Monetised journey time and accident benefits (as of recently, this is   |
| only undertaken Five Years After opening)                               |
|                                                                        |
|                                                                         |
| Comparison of predicted and outturn scheme costs, although the remit    |
| of POPE is not to provide analysis for reasons of over/under-spend.     |
| Costs are converted to a present value comparable with the benefits to  |
| facilitate the calculation of the benefit cost ratio (BCR)              |

As can be seen, the evaluation process is a resource intensive process of data collection, site visits and analysis which can take at least nine months to complete.  The timing of the analysis is dependent on: 

 
Whether traffic management is still being undertaken on the road 
 
If traffic flows on the road are temporarily constrained by nearby schemes under construction or external factors (e.g. local road schemes) 
 
Availability of validated safety data 
 
 
Need for temporary traffic counts during 'neutral' travel periods (i.e. outside holiday periods) and ideally during a consistent period to the baseline 
 
Seasonality factors related to the environmental site visit 
 
Quality of the data and whether any analytical assurance issues have been raised. 
The final POPE report includes the detailed results of the evaluation for each of the focus areas, as well as a summary of the scheme impacts and a comparison with the original scheme objectives. 

## 3.2 Application Of Evaluation Process To Different Programmes

CIP and SMP schemes POPEs are produced for each individual scheme that falls within Major Projects (i.e. CIP, RIP and SMP). At the time of this review, all CIP schemes are either in construction or planning stages, however, given the unique nature of CIP schemes, a bespoke approach to appraisal and evaluation is often required, although POPE reports are still likely to be produced one year after fully opening which, given the long duration of these projects, can be some time after parts of the schemes have been opened to traffic.  CIP schemes also provide significant opportunities to learn lessons and identify wider benefits and process improvements both within Highways England and the supply chain. These are outside the evaluation process but are captured through Highways England's continuous improvement processes. POPE is undertaken for smart motorways using the consistent evaluation method and a number of POPE reports have been published for smart motorway schemes (e.g. M1 J10–13, M1 J39–42, M25 
J5–7 and M62 J25-30), and the new approach to benefits management and evaluation is being rolled out across the smart motorway programme.  In addition, the monitoring and evaluation evidence base for smart motorways includes specific trials and bespoke safety monitoring studies which are outside the scope of this review but is reflected within DfT's Smart Motorway Stocktake which was published in March 2020. 

Small-scale improvements For small-scale improvements such as the Congestion Relief Programme that are delivered by Operations Directorate, and where it is difficult to justify the effort of evaluating every scheme, a sample-based approach is taken, which builds on the established approach used for the former Local Network Management Schemes (LNMS) and Pinch Point Programmes whereby POPEs are produced for a representative sample of schemes and a meta-analysis carried out to assess the impact of the programme as a whole. Ring-fenced investment funds The ring-fenced or 'Designated' investment funds are managed by Safety, Engineering and Standards Directorate (SES) and were introduced in RIS1.  These relate to: 

 
Environment 
 
Air Quality 
 
Cycling, Safety and Integration 
 
Innovation 
 
Growth and Housing 
 Highways England have undertaken reviews to determine the most appropriate way of evaluating schemes delivered under these programmes and derive an overall evaluation approach which could be tailored to the requirements of each fund. Five evaluation approaches were considered for each fund as shown in Table 3. 

| Evaluation Approach                                                     | Details                                                              |
|-------------------------------------------------------------------------|----------------------------------------------------------------------|
| 1.                                                                      |                                                                      |
|                                                                         |                                                                      |
| Process evaluation                                                      | Helps establish whether the programme is being implemented as        |
| planned, and what is working more or less well                          |                                                                      |
| 2.                                                                      |                                                                      |
|                                                                         |                                                                      |
| Outputs evaluation                                                      | Shows what was delivered by the programme                            |
| 3.                                                                      |                                                                      |
|                                                                         |                                                                      |
| Outcomes evaluation                                                     | Shows the effect of the programme                                    |
| 4.                                                                      |                                                                      |
|                                                                         |                                                                      |
| Economic evaluation                                                     | Provides an ex-post assessment of programme-level benefit cost ratio |
| 5.                                                                      |                                                                      |
|                                                                         |                                                                      |
| Quasi-experimental                                                      |                                                                      |
| evaluation                                                              |                                                                      |
| Shows how much of any change in key metrics is directly attributable to |                                                                      |
| the programme                                                           |                                                                      |

The results of the studies suggested a combination of process evaluation, outputs evaluation and outcomes evaluation for all funds with some elements of economic and quasi-experimental evaluation if appropriate and methodologically feasible. In terms of reporting, it is expected that they take a similar, sample-based approach to that used for small-scale improvements focussing on the specific objectives of each fund and sub-fund. 

R2 
Highways England should continue to develop the evaluation process to make sure it is 
sufficiently flexible and scalable to cover the wide range of schemes and programmes whilst ensuring consistency and comparability. 
R3 
Highways England should consider whether there are ways to provide an assessment of early 
scheme outcomes, as well as monitoring benefits over the longer term, so that lessons can 
be identified and shared in a timely way. 
R4 
Highways England should ensure that the requirements for benefits realisation and 
evaluation remain consistent and up to date in the respective processes and products across all projects and programmes. 

3.3 Changes to evaluation process Prior to the publication of Highways England's Benefits Management Manual (BMM) in October 
2018, the POPE process was largely a standalone exercise that was carried out for each major scheme and, on a sample basis, at a programme level for small scale improvements with periodic or programme level meta-analyses. The anticipated benefits, as quantified during the original scheme appraisal, and how they were to be measured were set out in a Scheme Evaluation Plan (SEP) which was produced by Highways England analysts or consultants (before the formation of the Evaluation Group) prior to construction beginning. Although POPE has always been a requirement of the MP PCF, with the introduction of the new benefits management process as set out in the BMM, Highways England formally incorporated the existing post-opening evaluation process into the a wider benefits management approach that runs throughout the scheme lifecycle.  The BMM aims to ensure consistency in benefits management across all of Highways England's portfolios, programmes and projects, and clearly defines how evaluation fits into the wider benefits management process, with clearly defined responsibilities for planning and delivering the evaluation. Under the new process, responsibility for producing the evaluation plan (which is now part of the wider Benefits Realisation and Evaluation Plan, or BREP, which is replacing the SEP) sits with the Project Team but is assured and signed-off by the Evaluation Group.  The BREP is more comprehensive than the SEP and clearly sets out the evaluation data requirements, including baseline data, additional data that needs to be collected (e.g. local road traffic surveys), and links to the original benefits register and benefits mapping. On its introduction, the new BREP process was applied to schemes that hadn't yet passed project control framework (PCF) Stage 5 'Construction Preparation' (although is being retrofitted to some schemes on a case-by-case basis) which means that, given the long lifecycle of a scheme, there is currently a mixture of schemes that are going through the old SEP process and the new BREP process. The key changes affecting evaluation during RIS1 are summarised in the table below. 

| Change                  | Original situation (pre    |
|-------------------------|----------------------------|
| 2018)                   |                            |
| Current situation (post |                            |
| 2018)                   |                            |
|                        |                            |
|                         |                            |
| POPE carried out by     |                            |
|                        |                            |
|                         |                            |
| Evaluation carried out  | Responsibility for         |
| evaluation              | external consultants       |
|                        |                            |
|                         |                            |
| New steering group      |                            |
|                        |                            |
|                         |                            |
| Standalone POPE         |                            |
|                        |                            |
|                         |                            |
| POPE part of wider      |                            |
| process                 |                            |
|                        |                            |
|                         |                            |
| Evaluation              |                            |
| Introduction of         |                            |
| benefits                |                            |
| management              |                            |
| approach                |                            |
|                        |                            |
|                         |                            |
| Evaluation              | requirements captured      |
| in the SEP as part of   |                            |
| PCF process             |                            |
|                        |                            |
|                         |                            |
| SEP produced by         |                            |
|                        |                            |
|                         |                            |
| BREP produced by        |                            |
| consultants             |                            |
|                        |                            |
|                         |                            |
| Publication of POPE     |                            |
|                        |                            |
|                         |                            |
| POPE reports approved   | Approval and               |
| publication of          |                            |
| POPE reports            |                            |
| reports approved by     |                            |
| DfT and published on    |                            |
| .GOV website            |                            |
|                        |                            |
|                         |                            |
| Reports are shared      |                            |
Impact 
 Need to create, 
resource and upskill evaluation team 
by Highways England internal Evaluation Group 
 Enhanced internal 
capabilities 
 Ability to take wider 
and governance arrangements 
view of evaluation 
 Improved knowledge 
retention 
 Focus on benefits 
management culture 
benefits management approach 
 Better integration of 
evaluation into scheme lifecycle 
requirements captured in the BREP 
 Better definition of 
evaluation requirements, including data needs 
project team with signoff from Evaluation Group 
 Improved ownership of 
benefits management and evaluation requirements by the business 
 Increased control over 
publication process and timescales 
 Independent, 4th line 
assurance provided by DfT 
by Highways England's Chief Analyst and agreed for publication by the Corporate Publication Steering Committee then published by Highways 
England with DfT for information prior to publication  

## 4 Specific Lines Of Enquiry

This section describes the results of a number of specific lines of enquiry that the review was asked to consider. 

## 4.1 Resources And Capabilities

Evaluation Group As mentioned before, the POPE process has been in existence for 15-years, beginning with Major Projects.  Historically, the POPE of Major Projects was carried out by external consultants but in 2017 Highways England took the decision to bring a number of routine processes, including production of the POPE, in-house.  This required the creation, resourcing and upskilling of a new Evaluation Group, all of which took approximately one year to complete. The Evaluation Group sits alongside Transport Planning Group, Economics Group and Performance Analysis and Modelling Group within the Chief Analyst's Division (CAD) of Strategy and Planning Directorate and is therefore independent of any of the delivery arms of Highways England.  The Evaluation Group consists of ten people (Head of Evaluation, Principal Evaluation Manager, plus eight analysts covering safety, traffic, environment and Smart Motorways) and the structure is shown in the figure below. 

The new team has brought together technical specialists from within Highways England, for instance in environmental impacts of schemes, alongside experienced analysts who joined Highways England to be part of the newly formed function. Economic evaluation is carried out by the Evaluation Group with support from other groups within CAD as required, including advisory and assurance support from Transport Planning and Economics Groups.  Where required, additional specialist resources are brought in (for example Sustrans have been advising on benefits from cycling schemes).   

The initial priority of the new Evaluation group has been to replicate the POPE process while looking for methodological improvements, and not hold up the MP evaluation programme.  The team is currently fully occupied with evaluation of RIP and SMP schemes but the Head of Evaluation does provide an advisory role for other programmes and there are requests for bespoke evaluations. Overall priorities are set by the MP Evaluation Steering Group and evaluation is managed as a programme. Bringing the evaluation process in-house is anticipated to provide Highways England's analysts with a broader perspective and allow longer-term insights to be provided, as well as, improving knowledge management.   
A further role of the Evaluation Group is to develop standards and work with the benefits leads within MP to encourage a benefits culture ensures that the focus remains on benefits realisation throughout the life of schemes. 

R5 
Highways England should ensure that the Evaluation Group remains fully resourced and skilled to meet the requirements of RIS2, and beyond, and the evolving evaluation process. 
Major Projects evaluation programme management Management of resources within the Evaluation Group is the responsibility of the Principal Evaluation Manager. Management of the evaluation programme includes a scheme tracker which tracks the progress of schemes prior to opening.  The tracker shared with the review team includes in excess of 90 schemes. Once opened to traffic, schemes form part of the current evaluation work programme.   A senior analyst from the Evaluation Group is allocated to each scheme to act as the evaluation manager, supported by a project team of analysts for each of the focus areas, i.e. safety, traffic, environmental and economy.  Ideally the same senior analyst would be involved throughout the process. 

Small Scale Improvements evaluation management Evaluation of small-scale improvements, such as the current Congestion Relief Fund programme, is carried out on behalf of Highways England by external consultants on a representative sample basis. The evaluation programme is managed by Operations Directorate (OD).  Once the programme has been delivered, a meta-analysis of the POPEs will be carried out by the Evaluation Group. 

## 4.2 Governance And Assurance

Major Projects Evaluation Steering Group Overall delivery of the Evaluation Programme, including delivery against milestones, quality assurance and implementation of lessons learnt is overseen by the Evaluation Steering Group. The Steering Group forms a key part of the governance structure for the evaluation programme and oversees and steers its delivery, providing accountability to the Chief Analyst and Executive Director of MP. The Steering Group approves and assures evaluation outputs and acts as focal points within their programme areas to embed evaluation findings and lessons learnt back within the business. The group is chaired by the Head of Evaluation and membership includes benefits management leads from each of the MP programmes (RIP, CIP and SMP), MP Programme Office, Transport Planning Group, Environment Group and Operations Directorate, as we as programme and technical leads from the Evaluation Group. 

Analytical Requirements Committee At a scheme level, the Analytical Requirements Committee (ARC) sits throughout the scheme lifecycle.  The ARC reviews the analytical assumptions being made for the scheme in the Analytical Requirements Report (ARR) which sets out the analytical requirements at each stage of the scheme lifecycle (discussed below).  The evaluation section within the ARR must be approved by the Evaluation Team who are represented on the ARC.  This committee can flag any issues with the Evaluation Group at any stage during the scheme lifecycle. 

Assurance As well as the Evaluation Steering Group, all output from the evaluation process is reviewed by someone within the Chief Analyst Division who was not involved in the evaluation.  This forms part of the overall assurance framework based on Four Lines of Defence: 

1. Peer review of data analysis 2. Senior review of evaluation within the Evaluation Group (e.g. individual focus-area such as 
Environment) 
3. Evaluation Steering Group and member of Chief Analyst Division not involved in the 
evaluation reviews output for scheme evaluation as a whole 
4. DfT review of Tier 1 schemes (i.e. high value or high priority schemes) 
Overall responsibility for Highways England's approach to evaluation sits with the Chief Analyst, who along with Exec Members, approves POPE reports. 

4.3 Evaluation data As can be seen above, the evaluation process is dependent on the quality and coverage of the data that it uses and, in particular traffic data.  This data is drawn from Highways England's centralised Roads Information Framework (RIF) and, if necessary, can be supplemented by local monitoring data if the scheme is forecast to impact on traffic flows on the local road network. Where there are issues with the quality of the automated collection of traffic data (for example the loops in the road have become faulty), this limits the extent of the traffic evaluation which can be undertaken. Going forward, Evaluation Group will continue to feed their requirements into the ongoing development of the RIF There have been legacy issues with carrying out evaluations of older in particular the historic handover of detailed project data to support the summary information in the AST.  This particular issue was addressed in the design of the BREP which includes the original appraisal data. 

4.4 Evaluation of outturn costs Although, as mentioned above, the remit of POPE is not to provide analysis of the reasons for overunder spend, the current POPE process does include core metrics relating to delivery efficiency and cost effectiveness. This includes a comparison of outturn construction cost against those agreed in the Business Case as well as outturn Value for Money (VfM) assessment made using data profiles from the first five years post opening. 

Outturn cost metrics The metrics that outturn cost influences and which are monitored for each scheme and evaluated in the scheme POPE include: 

 
The outturn scheme cost against the Final Target Cost captured in the Final Business Case prior to construction; 
 
The impact of outturn cost in the scheme Benefit Cost Ratio (BCR); 
 
The impact of outturn cost on the scheme VfM after evaluating other scheme objectives such as accident savings and journey time improvements. 
4.5 Publication of evaluation reports Although it is not a specific licence requirement, as mentioned above, a key aspect of the evaluation process, as well as learning for future schemes or other schemes within a programme, is to provide transparency and accountability through the publication of the POPE reports.  The reports have a number of potential audiences including: 

 
The public; 
 
HM Treasury and the Department for Transport; 
 
Supply chain and other external partners; 
 
ORR; and 
 
Internal Highways England stakeholders. 
Evaluations begin after a scheme has been fully opened to traffic for at least 12 months and, although there is no formal target date for publication of POPE reports, Highways England aim to disseminate findings internally and publish the reports in a timely manner. ORR monitoring requirements for RIS2 From discussions with ORR, it is understood that, with regards to evaluation, their focus from a monitoring perspective during RIS2 will be: 

 
The completion of POPEs and timely publication of reports 
 
The outturn costs and benefits achieved for each major scheme or programme; and 
 
The overall benefits delivered by RIS2 as a portfolio. 
Approval and publication process Previously, POPE reports were reviewed and approved for publication on the .GOV website by DfT. The Evaluation Group has been working with the Executive Committee and DfT stakeholders to agree a new process for assuring and signing off POPE reports ahead of publication in line with the Company's transparency policy.  Key aspects of this process are:  

 
The process builds in analytical assurance and sufficient time to review the findings internally. 
 
The results will be formally cleared by the Chief Analyst (to ensure appropriate analytical assurance) and the Executive Director of Corporate Affairs and Communications.  
 
Once officially signed off, the timing of the publication will be agreed by the Executive Director of Corporate Affairs and Communications and the Executive Director of Strategy and Planning as part of the routine publication process within the organisation. 
 
Media handling plans will be developed for each study in collaboration with Strategic Communications business partners and Press Office.  
 
 
Highways England share the completed studies and handling considerations with the Department for Transport ahead of publication. 
However, the move from Highways Agency to Highways England and the associated governance changes, interrupted the delivery of POPE reports.  Highways England are working hard to catch-up and remain committed to providing transparency through the continued timely delivery and publication of POPE reports going forward. In addition to the POPE reports, and meta-analysis of smaller schemes delivered as part of a particular programme, periodically Highways England carries out a meta-analysis of all evaluations to date, to take an overview of trends across the programme as a whole.  For example, in 2019 
Highways England published an Evaluation Insight Paper based on a meta-analysis of the 85 evaluations that had been undertaken of major schemes between 2002 and 2014.  The ability to produce such Insight reports highlights the importance of maintaining a consistent and comparable evaluation methodology. Since 2015, Highways England have published the results from more than 30 evaluation studies in full and summary formats.  The full reports contain detailed technical information whereas the summary reports have focussed on the main findings of the study and the extent to which the original scheme objectives have been met.  It is understood that Evaluation Group are reviewing the style of POPE reports and will be considering new approaches for the evaluations they have completed in-house. Stakeholder awareness and application of POPE reports To gauge the level of awareness and application of POPE reports, as part of this study, more than 60 organisations were invited to complete an online survey which was also promoted on social media. Responses were received from eight organisations which comprised: 
3 x Highways England Tier 1 supply chain partners 2 x Highways England Tier 2 supply chain partners 
1 x Sub-national Transport Body 1 x National Transport Body 1 x wished to remain anonymous Out of the eight respondents, five were aware that Highways England published POPE reports and three were not.  Of the five that were aware of the POPE reports, only one respondent had made use of a POPE report to "see the outcomes and scheme benefits for projects that we'd been involved with" and had found the report very useful as it "helps tailor the design and construction planning for subsequent schemes".  The respondent thought the report was very detailed and didn't have any suggestions for how it could be improved. Of the seven respondents who had not made us of a POPE report, reported that they were interested in the results of post-opening evaluation of major projects and three of the seven were either 'Very Likely' or 'Likely' to make use of a POPE report in the future. Although the number of responses was relatively small and is not suitable for making generalisations, the results do suggest that there is interest in POPE reports amongst the wider range of stakeholders - and in particular the supply chain - but that more could be done to raise awareness that the reports exist and where they can be found. 

R6 
As part of the review of future reporting formats for scheme and programme evaluations that better meet the needs of their respective audiences, Highways England could consider what information this will generate that may be of use to the ORR. 
R7 
Given the potential value of the information they include, Highways England should consider how to raise awareness of the POPE and other evaluation reports amongst the full range of potential audiences. 
4.6 Feedback for future investment decisions Lessons learned are captured and feed into future scheme appraisal processes.  As mentioned previously one of the roles of the Evaluation Steering Group is to ensure that lessons learned are fed back into future investment decisions and there are numerous examples where this has happened ranging from enhancements to traffic models and supporting research into the accuracy of cost estimates through to providing evidence to inform the RIS2 business case. More generally, the Evaluation Group is represented on the Appraisal Methods Group and the intention is the learning is fed back as a matter of routine rather than as a specific process. 

## 4.7 Future Developments As Its Approach To Benefits Management Matures, Highways England Intends To Take A Wider And More Sophisticated View Of Evaluation Which Is Likely To Include:

 
Developing methodologies to evaluate the impact on wider economy and/or customer satisfaction; 
 
Introducing greater flexibility into the evaluation process e.g. bespoke or interim evaluations, focussing effort where there is most to learn, whilst ensuring consistency and 
comparability; 
 
Extending robust evaluation processes across the full range of portfolios and programmes proportionately; and 
 
Developing more bespoke output formats for the evaluation findings to support internal dissemination and improve external transparency. 
In evolving its evaluation approach, Highways England is continuing to work closely with DfT to ensure alignment with good practice and consistency with other transport modes. 

R8 
Highways England should continue to develop the evaluation methodologies to consider a broader range of other scheme impacts including impact on the wider economy and customer satisfaction. 

## 5 Scheme Review

To provide an evidence base for the review of HE's Major Projects evaluation process the schemes within RIS1 were examined and a sample of nine schemes was selected from the MP portfolio. Schemes were selected to investigate and highlight different aspects of the evaluation process and how it has evolved over time.  The analysis of the sample schemes was largely qualitative and involved reviewing scheme evaluation documentation and the associated processes. The purpose of the scheme review was to: 

1. Provide evidence on how effectively the schemes move through the evaluation process.  In 
particular the development and publication of one year and five-year POPEs. 
2. To illustrate how the evaluation process has evolved over time and whether these changes 
are driving continuous improvement in the scheme evaluation process and the associated outputs from scheme evaluation. 
5.1 Overview of the evaluation of RIS1 schemes Since the beginning of RIS1, Highways England have published more than 30 reports on evaluation studies including two 'one year after' POPEs for RIS1 schemes, nine 'one year after' and nine 'five year after' POPE reports for pre-RIS1 schemes as well as a number of meta-analyses and the Insight report. In terms of the MP schemes in RIS1, at the time of this study, 37 had been opened to traffic.  Of these, 17 had not been open to traffic for more than 12 months.  The evaluation status of the remaining 20 schemes as of January 2020 is shown in the table below. 

| Evaluation Status                                |   MP Schemes  |
|--------------------------------------------------|---------------|
| On Hold (due to ongoing schemes in the locality) |            1  |
| Combined with other schemes                      |            2  |
| Data Collection                                  |            8  |
| Analysis                                         |            3  |
| Assurance                                        |            4  |
| POPE completed and Published                     |            2  |
| TOTAL                                            |           20  |

The pipeline of evaluation work demonstrates the progression of schemes from data collection, through analysis, assurance and ultimately publication of a POPE.  Whilst this data is only a snapshot provided by Highways England's evaluation team the volume of work appears to be consistent across the different stages of the evaluation process. The review found that, since 2017, the average time between completion and publication of POPE reports is 27 months.  Now that Highways England has ownership of the sign-off and publication process, there should be a significant improvement in the time taken to publish POPE reports going forward. 

R9 
POPE reports contain a lot of useful information and should be published in a timely manner 
in order to maximise their value.   Highways England should consider what steps it can reasonably take to improve the time taken between report completion and publication without compromising quality. 
Despite the long lead time for the publication of POPEs, the project obtained assurances from Highways England that the lessons from evaluation were acted upon swiftly and findings were disseminated to the relevant internal stakeholders upon the completion of evaluation - prior to the publication of the corresponding POPE report. 

## 5.2 Detailed Review Of Sample Schemes The Commission Examined A Sample Of Nine Individual Schemes.  These Schemes Were Selected To:

 
Span the significant changes in the evaluation approach.  Specifically, the change from using an external consultant to undertake evaluation and produce POPE reports to this work being brought in house with the formation of Highways England's evaluation team; 
 
Span the change in evaluation process from the production of SEPs to BREPs; and 
 
Include different project types e.g. Smart Motorway and Complex Infrastructure Projects. 
The table below provides a list of the sample of schemes and the corresponding evaluation process used.  A short description of each scheme paraphrased from the information provided from the evaluation is provided in the subsequent subsections. 

| Carried out By    | Scheme           | Programme       | Opened to Traffic    |
|-------------------|------------------|-----------------|----------------------|
| Evaluation        |                  |                 |                      |
| Process           |                  |                 |                      |
| A453 Widening     | RIP              | July 15         | SEP                  |
| RIP               | Apr 15           | SEP             | External             |
| A14 Kettering     |                  |                 |                      |
| Bypass            |                  |                 |                      |
| SMP               | In Construction  | SEP             | External             |
| M6 J2-4 Smart     |                  |                 |                      |
| Motorway          |                  |                 |                      |
| RIP               | Sep 17           | SEP             |                      |
| Transition from   | A21 Tonbridge to |                 |                      |
| Pembury           |                  |                 |                      |
| External to       |                  |                 |                      |
| Internal          |                  |                 |                      |
| M20 J10a          | RIP              | In Construction | SEP                  |
| M271 Redbridge    | RIP              | In Construction | SEP                  |
| SMP               | In Construction  | SEP             | Internal             |
| M20 J3-5 Smart    |                  |                 |                      |
| Motorway          |                  |                 |                      |
| A500              | RIP              | In Construction | BREP                 |
| CIP               | Dec-19           | SEP / BREP      | Internal             |
| A14 Cambridge -   |                  |                 |                      |
| Huntingdon        |                  |                 |                      |

5.3 Findings from scheme review The detailed scheme review found good conformity with the process and consistency in the application of the evaluation process across the different programmes under both the SEP and BREP model.  Furthermore, the review of the sample schemes found a good standard of consistency between the evaluation approach undertaken by the external consultant and the in-house Highways England evaluation team. The BREP is a considerable improvement over SEP in terms of capturing and collating information and tying this back to the scheme objectives and the appraisal, which should improve evaluation process overall. 

## R10 Highways England Should Continue To Use Brep In Preference To Sep And Monitor Its Effectiveness And Further Improve The Evaluation Process As Required. 6 Comparison With Other Organisations 6.1 Uk Public Sector As Part Of The Study, The Following Comparator Organisations Agreed To Be Interviewed About Their Approach To Benefits Management And Evaluation.

 
DfT Evaluation Centre of Excellence and Roads Economics, Modelling and Evaluation Group 
 
DfT Rail Group 
 
Transport for London 
 
NHS 
 
West Midlands Police 
Across government, all organisations have been encouraged to adopt a benefits-focussed approach to project delivery but the majority of the organisations interviewed were at an early level of maturity. There were some examples of good practice in providing feedback from post-project reviews to future investment decisions but only Highways England had a well-established, standardised process for post-opening evaluation.  Indeed, largely for this reason, Highways England is considered "best in class" by DfT and the POPE process has been cited as good practice by the Institute for Government (2017) and the National Infrastructure Commission (2018). Because many organisations are at a similar level of maturity when it comes to benefits management, there is very active community of practice that provides support and shares lessons and good practice.  Highways England are an active participant in this community. 

## 6.2 International Practice

Government commissioned evaluations A desktop study and contact with various government and private sector organisations reveals that there are only a handful of countries that conduct Post Opening Evaluations in an equivalent manner to the POPE methodology. The research has shown however that there are similar post-opening evaluations commissioned by government agencies in Norway, France, USA, and New Zealand as shown in the table below. 

| Country     | Infrastructure type       | Project Size                 | When done              |
|-------------|---------------------------|------------------------------|------------------------|
| France      | Road, rail, port, airport | Over £70m                    | 5 years post opening   |
| Norway      | Road                      | over £20m                    | 5 years post opening   |
| USA         | Rail                      | All projects (funded by FTA) | 2 years post opening   |
| New Zealand | Road, busways             | Over £5m                     | 1-3 years post opening |

In terms of scope, these post-opening studies have differing evaluation criteria including cost, traffic, travel times, safety and environment as shown in the table below: 

 
Country 
Cost 
Traffic 
Travel times 
Safety 
Environment 
France 
 
 
 
 
 
Norway 
 
 
 
 
 
USA 
 
 


New Zealand 
 
 
 
 
 

In France post-opening studies are undertaken by *Loi d'Orientation des Transports Interieurs* and cover: costs (and reasons for overruns), traffic, travel times, safety, ecology, water quality, environmental impacts, and public feedback. In Norway, *Statens Vegvesen* commissions these evaluations which include a review of costs, traffic, travel time savings, vehicle operating costs, noise and air pollution. Non monetised benefits including environmental, but changes in land use are not included. In USA, Federal law requires sponsors of transit projects receiving major capital project funding from the *Federal Transit Authority* to produce a Before-and-After Study to cover construction costs, service levels, project scope and ridership. In New Zealand, the *New Zealand Transport Agency* undertakes Post Implementation Reviews on project cost and timeframe, traffic, travel times, safety (crashes), and pedestrian/ cycle facilities, As can be seen, the scope of most evaluations focusses on monetised cost-benefit criteria such as construction costs, traffic forecasts and accident reductions. Environmental evaluations are less prevalent despite the fact that most developed countries do perform some kind on Environmental Impact Assessment at project inception. 

Government-backed lending agencies Many of the government-backed lending institutions, including the World Bank and the European Bank for Reconstruction and Development (EBRD) conduct post-opening project reviews of transport projects. World Bank's Independent Evaluation Group produces Project Performance Assessment Reports (PPAR) that evaluate projects according to bank performance and borrower performance as well as outturn demand based on one-year-after opening traffic counts. EBRD carries out ex post evaluations of projects on a case study basis.  These evaluations are primarily focussed on outturn cost in terms of updating the original cost benefit analysis with a qualitative approach taken to assessing the outturn benefits. 

## Other Evaluations

Aside from the formal assessments outlined above, the project team has reached out to other contacts overseas to get a picture of what might be happening elsewhere: 

| Organisation                                                                           | Comment                                                                              |
|----------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Netherlands                                                                            |                                                                                      |
| highways                                                                               |                                                                                      |
| Discussions with organisations on two recently completed highway schemes (A27/A1       |                                                                                      |
| at Hilversum and A1/A6 just outside Amsterdam) indicate that a Post Completion         |                                                                                      |
| Report is completed by the contractor which looks primarily at the construction        |                                                                                      |
| phase and includes:                                                                    |                                                                                      |
| (i)                                                                                    |                                                                                      |
|                                                                                        |                                                                                      |
| a summary of significant technical changes                                             |                                                                                      |
| (ii)                                                                                   |                                                                                      |
|                                                                                        |                                                                                      |
| construction issues/risks manifesting themselves                                       |                                                                                      |
| (iii)                                                                                  |                                                                                      |
|                                                                                        |                                                                                      |
| creation of work opportunities                                                         |                                                                                      |
| (iv)                                                                                   |                                                                                      |
|                                                                                        |                                                                                      |
| traffic volumes                                                                        |                                                                                      |
| (v)                                                                                    |                                                                                      |
|                                                                                        |                                                                                      |
| achievement of project milestones                                                      |                                                                                      |
| There are also market consultation workshops between the Dutch highways                |                                                                                      |
| authority                                                                              | Rijkswaterstaat                                                                      |
| cooperation aspects (how can they improve cooperation). For example, on the A9         |                                                                                      |
| IXAS highway tunnel project, RWS has engaged University of Delft to hold workshops     |                                                                                      |
| and do interviews to report back on how cooperation was (i) during construction (ii)   |                                                                                      |
| after civils completed (iii) after tunnel opening.                                     |                                                                                      |
| USA highways                                                                           | Apart from the FTA conducted reviews on rail projects, nothing similar is undertaken |
| for highway schemes although it is understood some States do conduct their own         | ad                                                                                   |
| hoc                                                                                    | studies but nothing USA-wide is known about. For instance, in Florida, FDOT          |
| review outturn costs and traffic data whereas in Ohio, ODOT sends out a lessons        |                                                                                      |
| learned questionnaire about one-year-after- opening.                                   |                                                                                      |
| German highways                                                                        | Nothing similar to a POPE undertaken. On accidents, every year there is a regional   |
| meeting between Authority, police and concessionaires/ maintainers to discuss          |                                                                                      |
| issues, but this is not project-based more on a regional basis. With regard to traffic |                                                                                      |
| again this is reviewed annually on a network rather than project basis.                |                                                                                      |
| Canada highways                                                                        | Nothing similar to a POPE undertaken. There are general                              |
| Authority and contractor/ concessionaire at a high-level on how things are going, and  |                                                                                      |
| any lessons learned but nothing is published.                                          |                                                                                      |

Summary Despite some evidence of the use of post project evaluation schemes in the international transport there remains significant differences in terms of coverage and scope of the schemes in different countries and Highways England's approach compares well.