Development of a single Official House Price Index

Office for National Statistics, Land Registry, Registers of Scotland and Land & Property Services Northern Ireland

1. Summary This article builds on the October 2014 consultation on the development of a definitive house price index and subsequent published response to provide final details of the methodology being used in the development of the new, single official house price index (HPI). This article will focus on explaining the sources of data being used in the production of the new HPI; define the final methodology which is being used to calculate the new HPI; consider further the findings and recommendations from both the user consultation and United Kingdom Statistics Authority special phase 1 assessment of the new UK HPI before finally discussing the initial plans for transition to the new HPI in 2016.

2. Introduction The importance of the housing market to the UK economy and society as a whole means that there is a need for clarity, completeness and coherence in the official statistics published on this topic. Previous reviews of official statistics in this area, such as the 2010 National Statistician’s Review of House Price Indices, have highlighted this and formally recommended the production of a single official HPI.

The current official measures of house prices are published by the Office for National Statistics (ONS), Land Registry, Registers of Scotland (RoS) and Land & Property Services Northern Ireland (LPSNI). However, differences in coverage, the source data and methodology used have led to a divergence in the published results, which can cause users difficulty in understanding and interpreting the data. As an example of this, Figure 1 compares the average house price for London published by Land Registry and ONS. In order to address these issues, work has been taking place over the past two years to develop a single, official HPI. Progress towards this has been reported on periodically during this time and the work is now moving towards the publication of such an index.

Taking on board previous user feedback and requirements, it is clear the target for the proposed new HPI (and indeed that of the current official published HPIs) is a price index that reflects the final transaction price for sales of residential property in the UK. The price index should cover purchases at market value for owner-occupation and buy-to-let, excluding those purchases not at market value (such as remortgages), where the ‘price’ represents a valuation. As such, the new index should be weighted using the most appropriate set of transaction weights, which is explained further in section 3.3 of this article.

This article brings together the work which has been carried out to date and provides some detail regarding the next phase of the development and the plans for publication. Progress over the duration of 2015 has been hindered by the legal and Parliamentary processes required to secure the necessary access to input data. This process was completed in December 2015, and work is now underway to finalise the processes required to produce and quality assure of the new HPI.

2.1 Timeline and reviews of the housing market

Over a number of years, there have been various reviews and consultations associated with the publication of house price statistics, reflecting both the importance and interest in the housing market. A number of these have a direct connection with the development of a new single official HPI. This section summarises the reviews and consultations that have impacted on the development of the new HPI, providing a timeline of the development work which has taken place and assessing how the requirements/recommendations from each review have been taken forward.

The main review of house prices was carried out by the National Statistician in 2010 leading to the publication of the National Statistician’s review of official house price indices in December 2010. The review report concluded that there was a clear need for official house price statistics but neither of the official measures available then (the current ONS or Land Registry HPIs) fully met user requirements. The main recommendation from the review was ‘a single definitive house price index and accompanying statistics should be produced by the official statistics producer community’. Additional recommendations were also made in relation to improving the publication and understanding of house prices. The focus of development work to date has been in taking forward the recommendation to produce a single house price index. However, improvements have been made in response to the other recommendations, such as improving commentary and coherence in published statistics and the publication of ‘Official House Price Statistics Explained’ detailing differences in the current official HPIs.

Following publication of the 2010 review, although not directly linked to the recommendations, the HPI previously published by the Department for Communities and Local Government (DCLG) was transferred to the ONS. The transfer of the HPI was completed in April 2012.

A follow up National Statistician’s review was carried out in 2012, extending the scope to wider housing market statistics. This second review was published in September 2012 and built on the content of the first review, recommending more accessible housing market statistics and the development of official statistics covering the private rental sector. This second review added support to the ongoing work to develop a new single official HPI and as a result of the recommendations, a new private rental price index was developed and published for the first time; along with an article covering trends in the UK housing market.

Following the transfer of the HPI from DCLG to ONS, a Working Group was formed comprising those organisations responsible for producing official house price statistics and an independent member from the Department for Business, Innovation and Skills. From late 2012, the Working Group investigated options for developing a house price index to meet the requirements from the National Statistician’s review. This included reviewing the methodology used in the production of current house price statistics, investigating the available data sources and carrying out feasibility work for the development of a new, single HPI. The culmination of this work was a proposal to produce a new, single official house price index that would ultimately replace the HPIs currently published by ONS, Land Registry and LPSNI. This proposal was formally consulted on with users in late 2014, and followed up with an official response in March 2015. The consultation response noted that the development of a new, single official HPI was met with support from users but detailed a number of areas for further analysis, which this article describes.

At the same time as the user consultation, 3 specific issues were taken to the Government Statistical Service Methodological Advisory Committee (GSS MAC) for their views. These related to the number of years’ transactions that should be used when constructing the weights; the most appropriate method for averaging prices; and the presentation of average prices. GSS MAC were supportive of the development but formally recommended the publication of analysis to justify the use of 1 year’s worth of transactions in the calculation of index weights and to assess the quality of the new index based on the sources of data being used. Again, these requirements are addressed below.

Finally, in light of the above work, the UK Statistics Authority announced in February 2015 that they would assess the development of a new UK HPI against compliance with the Code of Practice for Official Statistics initially, and follow this up with a second phase assessment after the new UK HPI is published to assess designation of the new index as a National Statistic. The first phase of the assessment was published in July 2015 and put forward a number of recommendations to ensure the new UK HPI fully complies with the Code of Practice. These are detailed in Table 1, along with an action plan for complying with the requirements.

Table 1: Action plan to meet UK Statistics Authority assessment requirements, December 2015:

| Finding | Requirement | Action plan | |---------|-------------|-------------| | ONS and partners have consulted users about proposed methods, and commissioned some early peer appraisal, but continuing analysis means that some questions remain unanswered. ONS and partners should: | 1: Publish a methods document that: Details the methods that will be used to produce the new UK HPI and related indices for the four countries of the UK Explains the rationale for the choices of methods within the context of how the statistics will be used and the decision that they will inform. Explains how the methods comply with international standards Addresses the key methods questions raised by users and expert peer appraisal (paragraph 3.8) As part of meeting this requirement, ONS and partners should commission a second phase of expert peer appraisal. | Progress: Partially met by publication of this article. Further detail behind methods will be published alongside new index and a second phase of peer review will be carried out in 2016. | | ONS and partners are employing a range of new methods and administrative data sources. It is important that they make informed judgements about the strengths and limitations of the statistics, including the assurance of the quality of the data sources, and communicated these judgements to users. ONS and partners should: | 2: Publish information about the quality of the new UK HPI that explains the strengths and limitations of the statistics in relation to uses and describes potential sources of error and bias and how it will mitigate for them, including explaining the different sources of volatility in the estimates. Ensure that all service level and data sharing agreements between ONS and data suppliers clearly set out arrangements for the assurance of the data sources used to produce the new UK HPI. Publish information about the quality assurance arrangements for the administrative data sources that will be used to produce the new UK HPI and related indices, taking into consideration the Authority’s Administrative Data Quality Assurance Toolkit. | Progress with 2a and 2c - The departments involved in the development of a new HPI have held a workshop to review the quality and assurance of the datasets feeding into the new HPI, further work is being undertaken following this workshop. Details will be published alongside the new HPI although some information is included within this article. Progress with 2b - Service/Data Sharing Agreements are in the final stages of being agreed between departments (where these have not already been implemented) and should be in place by January 2016. These will be reviewed to ensure the necessary assurance statements are included. | | Registers of Scotland will continue to publish Quarterly House Prices Statistics in addition to the new HPI. It is important that ONS and partners explain the coherence of these accompanying statistics. ONS and partners should: | 3: Publish prominently alongside the statistics information about the coherence of the new UK HPI, and associated indices, with other published official house price statistics. | Progress: This requirement will be fulfilled alongside publication of the new HPI. Coherence with all house price data will be fully explained. | The development of a coherent set of official house price statistics is positive. However, it will remain important to describe for users the relationship with non-official indices. ONS and partners should:

4: Publish information about how they have sense-checked the new official house price statistics against other non-official sources, and how they will continue to inform users about the relationships between these different measures, and clarify the distinctive purpose that these statistics serve.

Progress: This requirement will be fulfilled as the production of the new HPI (including back series) takes place and will be fully documented alongside the first publication of the new HPI in 2016.

3. A new house price index

Based on the feasibility work carried out by the Working Group and presented in the October 2014 User Consultation, it was proposed that a new, single official HPI could be produced jointly by the current producers of official house price statistics.

This section will describe the data sources and the final methodology for the proposed new index, along with justification of the approach chosen. At this stage, this article will not fully consider the quality aspect of all data sources but will lay the foundation for this clarification as the development progresses. An assessment of the quality and limitations for each source will be published alongside the new HPI in mid-2016.

It should be noted at this point that due to data confidentiality restrictions, individual record level Northern Ireland data cannot be shared beyond LPSNI. Therefore, LPSNI will produce aggregated price data and indices, which will then be shared and combined with the GB prices/indices to produce UK estimates.

3.1 Data sources

The development of the new official HPI brings together a number of comprehensive administrative data sources. This section briefly summarises each data source and details how the source of data will be used in the new HPI. There is an increasing reliance on the use of administrative data in the production of official statistics. This has been recognised by the UK Statistics Authority, who published a Regulatory Standard(^1) for the quality assurance of administrative data. At this stage of development, this article does not attempt to fulfil this requirement. Each administrative source used in the new HPI will be assessed against the Regulatory Standard, using the Administrative Data Quality Assurance Toolkit. Full details of this assurance will be published alongside the new HPI in 2016.

The data sources used to produce the new HPI fall into two distinct categories; price data or property attributes data. The price data provides details of the final transaction price at which a residential property has been sold (this source of data does include limited property attributes data).

(^1) [http://www.statisticsauthority.gov.uk/assessment/monitoring/administrative-data-and-official-statistics/index.html](http://www.statisticsauthority.gov.uk/assessment/monitoring/administrative-data-and-official-statistics/index.html) whilst the property attributes data provides details regarding the actual property, such as the type of property (detached, semi-detached etc.), the size of the property (number or rooms or floor space) and the location of the property. Combining the detailed property attributes data with the price data provides a comprehensive and robust dataset required for use in a hedonic regression model, which is defined further in section 3.2.

3.1.1 Price data The main sources of price paid data used in the new HPI are the Land Registry for England and Wales, Registers of Scotland and Her Majesty’s Revenue and Customs Stamp Duty Land Tax data for the NI RPPI (this NI data is to be provided directly by LPSNI from their Residential property prices index. Full details of the LPSNI residential property prices index can be found via the attached methodology report).

Land Registry for England & Wales: Information on residential property transactions for England and Wales, collected as part of the official registration process, will be provided by Land Registry for properties that are sold for full market value. The dataset will contain the sale price of the property, the date when the sale was completed, full address details, the type of property (i.e. detached, semi-detached, terraced or flat), if it is a newly built property or an established residential building and a variable to indicate if the property has been purchased as a financed transaction (i.e. using a mortgage) or as a non-financed transaction (cash purchase). All this information is currently published by Land Registry in its Price Paid Dataset (PPD)², with the exception of the cash/mortgage indicator which is not publically available as this data is considered personal information and would breach data protection rules.

Given the need to measure residential property transaction prices, there are a number of registrations excluded from the version of the PPD used in the new HPI. These include sales of residential property that were not at full market value such as the sale of part of a property, a transfer between parties on divorce, ‘Right to buy’ sales at a discount, remortgages etc. In addition, all commercial transactions of residential properties are excluded from the dataset (where transactions of residential properties involve a transfer to a corporate body, company or business).

Whilst the PPD offers a comprehensive and robust source of residential property transaction data, there are some notable limitations with the dataset. The first limitation relates to the amount of time between the sale of a property and the registration of this information with Land Registry. It typically takes between 2 weeks and 2 months to complete the registration process, with purchases of new properties taking longer than pre-existing properties. As a result HPI data for the two most recent months will be subject to revision. Secondly, the registration of property transactions with the Land Registry do not require details regarding property attributes to be included. For this reason, the PPD needs to be supplemented with a suitable source of property transactions data (such as the number of rooms or total floor space). This supplementary data is described in section 3.1.2. The PPD is the primary source of data used in the production of the current Land Registry HPI.

² https://www.gov.uk/government/collections/price-paid-data Registers of Scotland house price data

Similar to the Land Registry’s PPD, residential property sales for Scotland will be provided by Registers of Scotland (RoS) for use in the calculation of the new HPI. The Scotland data follows a similar process to the PPD where the data provided for the HPI is a by-product of the Scottish property registration process, whereby property sales are submitted to RoS on completion of a sale. Data are gathered from these applications and entered into the RoS Land Registration System (LRS). A weekly extract of all sales data is then taken from LRS and undergoes a quality assurance process to check the price paid, date of entry, property type (land, commercial, residential, forestry, agricultural and other) and to identify market value residential sales (as opposed to non-market sales similar to those detailed for PPD). This data is then combined with further details, such as full address and house type (detached, semi-detached, terrace or flat), using the RoS Geographical Information System database to provide a comprehensive source of data that can be provided each month for use in the new HPI.

In addition, a cash/mortgage indicator and a new build indicator are also added to the final dataset for use in the HPI. The cash/mortgage indicator is derived by identifying whether the applications for registration of a market value residential sale also contain an application for registration of a standard security mortgage deed. All applications with a mortgage deed are mortgage sales, while all those without an accompanying mortgage deed are marked as being a cash sale. The new build indicator is given to every sale of an individual property from a builder’s development title. The coverage of the RoS data differs to the PPD slightly in that transactions relating to residential properties where the buyer or seller is a corporate body, company or business are included within this dataset. Its coverage is similar to that of the LPSNI as these transactions cannot be separately identified on the dataset.

The RoS price data has similar limitations to the PPD for its use in a price index. There are similar lags in the registration process, so the data used to calculate the new HPI for the two most recent months will be subject to revision. Additionally, there are limited property attributes data collected during the registration process so, like the PPD, the RoS price data will be supplemented with property attributes data - from the Scottish Energy Performance Certificates. This data is described further in section 3.1.2.

Council of Mortgage Lenders – Regulated Mortgage Survey

The Regulated Mortgage Survey (RMS) is the Council of Mortgage Lender’s version of the Mortgage Product Sales Data (PSD) that all regulated lenders report to the Financial Conduct Authority (FCA). This is detailed transaction level data on mortgage completions. Starting in April 2005, the RMS now contains over 12 million individual mortgage sale records.

It is collected electronically, with all reporting firms submitting according to FCA-defined data definitions, and in a standard xml format, ensuring consistent data structure. Data coverage is estimated to be around 95% of all regulated mortgages currently advanced. However, some lenders do not permit their data to be transferred to third parties, like ONS, so the share of the market covered by data received by ONS is about 70%.

Reporting fields include purchase price, completion date of property sale, type of borrower (first time buyer (FTB) or home mover) new or second hand property, type and size of dwelling. The RMS is the only comprehensive source of data available for the type of borrower and provides the necessary data to allow the new HPI to be produced according to whether the buyer is a first time buyer or an existing owner. The RMS is also a key source of data used in the production of house prices for inclusion in the Retail Price Index and will continue to be for the foreseeable future.

3.1.2 Property attributes data

As detailed above, there is comprehensive data available on the price of transacted property across the UK; however, this price data is limited in terms of details regarding the physical characteristics of the property. The use of price data alone is insufficient for the calculation of an inflationary index for house prices due to the fact the composition and type of property sold can differ vastly between periods. Therefore, in line with recognised international best practice hedonic regression and mix-adjustment is used to account for the change in composition (section 3.4.1 of ‘Official House Prices Explained’ provides a useful worked example of the need for mix-adjustment). For a robust hedonic regression model to be fitted, sufficient details regarding the attributes of properties sold are required to supplement the price paid data. For the production of the new HPI, these attributes are available from a number of official sources described below.

Valuation Office Agency Council Tax Valuation list

The main source of property attributes data that will be used to supplement PPD for England and Wales is administrative data taken from the Council Tax Valuation list maintained by the Valuation Office Agency (VOA). The VOA has been responsible for banding properties for Council Tax since the tax was first introduced in 1993; before then, the VOA was responsible for the earlier system of domestic rates. The Council Tax Valuation list is a robust source of property attributes (such as the size of the property) data that covers, in principle, all residential properties in England and Wales. The data is collected as efficiently as possible using the following sources:

- Verification through the taxpayer for data VOA already holds.
- Third party sources, of which there are a number.
- Local Authority planning records where the information is available to the public.
- Local Authority building control where this information is available through VOA’s Finance Department.
- Subscription to third party data sources who are able to satisfy VOA data accuracy standards.
- External inspection of the property where attributes visible from the road can be confirmed.
- Internal inspections of the property where attributes internal to the property need to be confirmed or a re-measurement is required.

Following collection, the data is input into the VOA central operational database. This input is subject to quality checking and undergoes a number of validations to ensure the properties’ attributes are compatible.

3 http://www.ons.gov.uk/ons/guide-method/user-guidance/prices/hpi/official-house-price-statistics-explained.pdf The VOA are an executive agency of Her Majesty’s Revenue and Customs (HMRC), and is therefore subject to the Commissioners for Revenue and Customs Act 2005 (CRCA) which means disclosure of the relevant Council Tax information is precluded except in certain limited circumstances, such as a legislative gateway. A permissive legal gateway for the sharing of the relevant Council Tax information with the UK Statistics Authority (for use in the calculation of a new HPI) has been created through the Statistics and Registrations Service Act 2007 (Disclosure of Revenue Information) Regulations 2015.

In securing access to the data, the request for attributes had to be assessed as proportional to the need, so only those attributes held on the Council Tax list that were deemed as necessary for inclusion in the production of a new HPI could be provided. For each residential property appearing on the Council Tax valuation list, ONS are provided with total floor area (meters squared), number of rooms, number of bedrooms, property type and full address details (including postcode). Ongoing updates to the Council Tax list (or amendments where necessary) will be provided on a monthly basis to ensure the list remains up-to-date and representative.

As with the PPD data, there are frequently lags in updating the information held on the VOA database, following the completion of a property sale or registration of a new property. This is another reason for permitting revisions to the new HPI for the two most recently published months.

**Land and Property Services Northern Ireland Valuation List**

Land and Property Services maintains the list of Northern Ireland properties which are valued for rating purposes. The data are maintained and validated similarly to the VOA data described above. The Valuation List database contains all the property attributes required for the hedonic regression used in the production of the NI RPPI.

**Energy Performance Certificates Scotland**

Scottish Energy Performance Certificates (EPC) will be used to provide the floor area of the property and the number of habitable rooms required for calculation of the new HPI. This will be matched against the price paid data provided by RoS (using the address details of the property). EPCs were introduced to comply with European legislation which requires that an EPC be provided on construction, sale or rental of a building to a new tenant. EPCs are produced by members of ‘Approved Organisations’ (AO) who have been appointed by Scottish Ministers to deliver certification services. As part of their appointment each AO must operate within the terms of the operating framework, which sets out how the organisation must work in terms of Code of Conduct, quality assurance etc.

EPC fall into two categories, dwellings and non domestic buildings. EPCs for newly constructed dwellings are produced using the Standard Assessment Procedure (SAP) and submitted to the verifier (Local Authority) as part of the completion certificate process. For existing dwellings a reduced version of SAP is used. EPCs for non-domestic buildings are produced using the Simplified Building Energy Model (SBEM). The format of the domestic and non-domestic EPC differs; example certificates can be viewed on the [Scottish Government website](http://www.scottishgovernment.gov.uk).

______________________________________________________________________

4 [http://www.ons.gov.uk/ons/about-ons/business-transparency/statistical-legislation/draft-regulations-to-authorise-the-disclosure-of-information-on-residential-properties-in-england-and-wales/index.html](http://www.ons.gov.uk/ons/about-ons/business-transparency/statistical-legislation/draft-regulations-to-authorise-the-disclosure-of-information-on-residential-properties-in-england-and-wales/index.html)

5 [http://www.nisra.gov.uk/housepriceindex/LPSHousePriceIndexMethodologyFinal.pdf](http://www.nisra.gov.uk/housepriceindex/LPSHousePriceIndexMethodologyFinal.pdf) The legislation establishing the EPCs, The Energy Performance of Buildings (Scotland) Regulations 2008, came into force towards the end of 2008, meaning that EPCs exist for all residential property transactions that have taken place since January 2009. As with the VOA data, this legislation restricts the purposes for which the data can be used, and does not include their use in the calculation of the HPI. However, the Regulations were amended by the Scottish Government in November 2015, meaning that ONS could receive the required data with effect from 19 December 2015.

Property details recorded for EPCs have to be obtained prior to the completion date of the property sale. It is therefore anticipated that there will not be any delays associated with the use of these data. Further details regarding the Scottish EPC data can be found via the Scottish Government website.

**Energy Performance Certificate England and Wales**

A further source of property attributes data for England and Wales is available through Energy Performance Certificate (EWEP) data. This EWEP data will be used to supplement the VOA Council Tax Valuation List (CTVL) to provide better coverage of new properties being built (and sold) as there is a lag in getting this information onto the CTVL. Access to the EWEP data has yet to be agreed but the data is not critical to the production of the new HPI. Further work is taking place alongside the development to secure the access before the publication of the new HPI in mid-2016.

**ACORN classification**

A key determinant of house prices is the demographic characteristics of the area in which the property is located, such as the affluence of those people living in the area. A well established geodemographic segmentation of the UK is available through the ACORN dataset, produced and licensed by CACI Ltd. ACORN segments postcodes into categories and groups by analysing significant social factors and behaviours.

ACORN is a key price characteristic used in the production of LPSNI HPI and the current ONS HPI. The ACORN dataset will continue to form part of the regression model being used in the new HPI, where the appropriate ACORN classification will be matched to each property using the Postcode variable.

**3.2 Methodology**

The methodology for the new UK HPI has been developed to best utilise the sources of data currently available. In addition, recommended international best practice, as defined in the Eurostat Residential Property Price Indices handbook has been used to guide the development, along with the current methodology used in the NI RPPI. The biggest improvement to methodology (as compared with the current ONS and Land Registry HPIs) is the use of additional property attributes data, in particular floor space of a property, in the modelling of house price data. Not only will the attributes data increase the amount of records which can be used in the calculation of the UK HPI.

______________________________________________________________________

6 [http://acorn.caci.co.uk/what-is-acorn](http://acorn.caci.co.uk/what-is-acorn)

7 [http://ec.europa.eu/eurostat/web/hicp/methodology/housing-price-statistics/residential-property-handbook](http://ec.europa.eu/eurostat/web/hicp/methodology/housing-price-statistics/residential-property-handbook) each period, but it will also provide greater comparability with HPIs produced internationally, where the use of floor space is more prevalent than in the UK.

The new house price index is calculated by valuing a fixed ‘basket’ of property transactions at two different points in time. The ratio of the two valuations shows by how much property prices have changed in percentage terms. A time series for the HPI is built up by valuing the same fixed basket in successive time periods (usually months).

The ‘basket’ is fixed in that it comprises all properties that are sold in a particular time period; in the case of the new HPI, this covers a 12 month period.

The mix of properties sold is not constant from one period to the next, so the basket is kept up-to-date by changing it each year. The value of successive baskets cannot be directly compared because they comprise a different set of properties. Instead, each basket is priced for a 13 month period running from January to the following January. An ‘in-year’ price index for each basket is then constructed, with the first January in each case set equal to 100. These in-year indices can then be chained together to give a continuous time series because they overlap, with the month 13 January index for one basket being the month 1 index of the following basket.

The properties in the basket are valued by considering a number of price-determining characteristics, and then each month determining the effect of each of these characteristics on the price. In the case of the new HPI, the price-determining characteristics are as follows:

- Local authority district in GB and housing market area in Northern Ireland
- ACORN area classification variable
- Property type (i.e. detached, semi-detached, terraced, flat)
- Floor area (metres squared)
- Number of rooms
- New or old property

The valuation of the price-determining characteristics is done using a statistical technique, called hedonic regression. Mathematically, a semi-log model is used, of the form:

$$\\log(p_i) = k + \\sum_j \\beta_j x\_{ij} + e_i$$

Where: $p_i$ is the price of property $i$

$k$ is a constant ( x_i^j ) indicates whether property ( i ) has the characteristic ( j ) (e.g., detached property). If so, it takes the value 1, otherwise it takes the value 0 (except for floor area where it takes the floor area in m(^2)).

( \\beta_j ) is the coefficient associated with characteristic ( j ).

( e_i ) is the statistical error.

In this equation, the logarithm of the price paid is used because house prices tend to be log-normally distributed – i.e., the frequency distribution of the log of the price is bell-shaped.

When the regression analysis is run, one local authority is chosen as the reference district and given a coefficient of zero; every other district is then valued relative to this reference district. So, a property in a district with a coefficient greater than zero is, on average, more expensive than one with a coefficient less than zero. The hedonic regression calculates a coefficient for each separate value that the variables can take. This is done for each of the price determining variables. In the case of floor area, the coefficient is calculated as a rate per metre squared.

The price determining characteristics can then be combined together to give a predicted price for each property in the fixed basket. These predicted prices are then averaged using a geometric mean, which involves multiplying the ‘n’ predicted prices together, and then taking the ( n^{th} ) root. (In practice, the geometric mean is calculated by summing the logs of the predicted prices, dividing by ( n ), and taking the exponential of the result.) The ratio of the geometric mean average prices in successive time periods then gives the price index.

The ACORN area classification code is available at different levels of detail. The group level, comprising 18 groups, is used in the regression analysis for GB, whereas the six category level is used in NI due to the smaller population. The room’s variable is treated as a categorical variable, where a coefficient is calculated for each number of rooms in the property, up to a maximum of eight, rather than a rate per room. The quality of the rooms data in NI is insufficient to allow it to be included in the NI model.

The current ONS regression model uses the buyer status (first-time or former owner-occupier) as one of the price-determining characteristics. It has been decided not to use this in the new HPI model, primarily because it accounts for very little of the variation in price. It has also been argued by some users that it is a characteristic of the buyer, rather than the property. Nonetheless, it is recognised that there is considerable interest in the price pressures faced by first time buyers, and results by buyer status will be published each month. It will be possible to produce this analysis for GB because the base set of transactions that form the weights will, for each transaction, indicate the buyer status. Alongside this will be the predicted price of each property (determined from the characteristics used in the hedonic regression) from which an average price by buyer status can be produced. This can then be used to produce an index in the same way as for, say, a breakdown by property type.

The financing status of the buyer (cash or mortgage) was also considered for inclusion in the model. However, like buyer status, it accounted for very little of the variation in price and is a characteristic of the buyer, rather than the property, so was not included in the model. However, results by financing status will be published each month. In a similar way to the analysis by buyer status, it will be possible to do this for GB because the base set of transactions that form the weights will, for each transaction, also include the financing status of the buyer.

The amount of variation in price (the “R-squared”) captured by the hedonic regression model is about 80 per cent.

When running the hedonic regression model, some properties may be missing one or more of their price determining characteristics – for instance, floor area may not be available. These properties are still used in the regressions, but are given less weight in the calculations depending on the importance of the missing variable as a price determinant. For instance, floor area is found to have more of a bearing on price than whether the property is new or old, so a property with missing floor area will have a lower weight than one missing the new/old indicator.

It is not possible to produce a predicted price for a property that has a missing value for any of its price determining characteristics. So, the properties that are included in the fixed basket must all have a complete set of characteristics. This is achieved by imputing for missing values using a ‘nearest neighbour’ method, whereby the missing value is replaced by a non-missing value ‘donated’ from a randomly chosen property with the same values for the non-missing variables. It should be noted that imputation also takes place for buyer status (i.e. first time/ former owner-occupier) and cash/mortgage which means that average predicted prices, and hence price indices, can be calculated for these variables. Buyer status is obtained in the first instance by matching data from the Council of Mortgage Lenders with data from the Land Registry and Registers of Scotland. This will, by definition, not cover cash purchases; for these, it is assumed that they are all purchases by former owner-occupiers.

The hedonic regressions will be run each month for the latest month and the two previous months. They will be run for the two earlier months because of delays in some properties being registered with the land registries, and being entered or updated on the VOA Council Tax List.

Full details regarding the methods used to produce the new HPI will be given in an article accompanying the first publication of the new index.

3.3 Weighting the price index

As noted in the previous section, the new price index will be mix-adjusted to allow for the fact that different houses are sold in different periods; by annually updating the fixed basket of properties (section 3.4.1 of ‘Official House Prices Explained’ provides a useful worked example of the need for mix-adjustment). The resulting mix-adjustment weights will be calculated annually and will be derived using the latest complete year’s worth of transactions. This proposal is a change to the methodology used in the current ONS HPI, where a rolling 3 years worth of transactions are used, but is in line with the LPSNI HPI which uses 1 year.

The question about how many years’ worth of transactions to use for weighting was raised in both the User Consultation (section 2.45) and the Peer review by the Government Statistical Service Methodological Advisory Committee (GSS MAC). This question has been considered further by the

______________________________________________________________________

8 http://www.ons.gov.uk/ons/guide-method/user-guidance/prices/hpi/official-house-price-statistics-explained.pdf 9 http://www.nisra.gov.uk/housepriceindex/LPSHousePriceIndexMethodologyFinal.pdf Working Group in terms of justifying the robustness and potential variability in using 1, 2 or 3 years worth of transactions for the weighting. A summary of the analysis is presented here.

Analysis was carried out using published data from the Land Registry Price Paid data set (PPD) to investigate the impact of using 1, 2 or 3 years worth of transactions for the weights on the variability in average prices. Initially, a dummy house price index was set up using the PPD. A weighting structure was calculated for property type within Local Authority level (where property type is one of detached, semi detached, terrace or flat) based on the number of transactions that occurred during a period. Three iterations of weights were calculated for:

- The latest complete years worth of transactions (so for 2014, this would be 2013 transactions)
- The latest complete 2 years worth of transactions (so for 2014, this would be 2012 and 2013 transactions)
- The latest complete 3 years worth of transactions (so for 2014, this would be 2011, 2012 and 2013 transactions)

Monthly average prices were calculated by property type for each Local Authority district and combined with their appropriate weights to calculate weighted average prices for each Local Authority. This process was repeated for each iteration of the weights detailed above, and worked backwards to January 2009 (so 2009 prices would be calculated using either the number of transactions from 2008, 2007 and 2008 or 2006 to 2008). Starting the analysis in 2009 ensured the calculation of weights encompassed the economic downturn of 2008/09.

For each year, a set of average prices were calculated for each Local Authority using weights based on either 1, 2 or 3 years transactions and the average growth in these prices over the 12 months derived. Table 2 shows the average percentage change in price for each Local Authority (over the period January to January) for each iteration of weights. The table also includes the maximum recorded percentage change in price within a Local Authority during the period: Table 2: Change analysis at Local Authority level based on 1 year, 2 year or 3 year transaction weights:

| Index Period | Weights period | 1 year | 2 year | 3 year | |-----------------------|----------------|--------|--------|--------| | January 13 to January 14 | Average % change | 7.75 | 7.83 | 7.84 | | | Maximum % change | 35.32 | 36.55 | 36.69 | | January 12 to January 13 | Average % change | 6.97 | 6.99 | 6.98 | | | Maximum % change | 51.07 | 51.64 | 52.04 | | January 11 to January 12 | Average % change | 6.72 | 6.70 | 6.60 | | | Maximum % change | 33.68 | 33.55 | 32.91 | | January 10 to January 11 | Average % change | 7.07 | 6.40 | 7.01 | | | Maximum % change | 44.78 | 40.19 | 41.82 | | January 09 to January 10 | Average % change | 12.63 | 12.53 | 12.66 | | | Maximum % change | 66.43 | 65.54 | 66.51 |

Table 2 shows that the impact of each different weighting approach on the average 12 month percentage price growth (of each Local Authority in England and Wales) is minimal.

Further analysis was carried out by comparing the difference in average price calculated using each set of weights. As expected, there was marginally more variability in price levels calculated using 1 year’s worth of weights compared to 2 or 3 years, as the impact of using more than 1 year leads to a smoothing effect in the calculations. However, this variability was not deemed to be significant.

Finally, the robustness in terms of the number of transactions available within each Local Authority was considered. The analysis showed that the majority of Local Authorities would have at least 1,000 transactions available based on 1 year’s worth of transactions. The number of transactions in only one Local Authority gave rise to any concern and this Local Authority is a clear outlier in that the number of transactions are limited when using either 1, 2 or 3 years. Further consideration will be given regarding the treatment of this Local Authority going forward.

In conclusion, the analysis carried out has indicated that the choice of 1 year’s worth of transactions to calculate weights for the new HPI is robust and importantly, is in line with the methodology currently used in the production of weights for the LPSNI residential property price index.

3.4 Seasonal adjustment

The 2010 National Statistician’s review of house price statistics defined the availability of a seasonally adjusted HPI as one of the criteria for a definitive official house price index. The 2014 user consultation exercise highlighted that the publication of seasonally adjusted house price data can cause confusion for users, and in particular that seasonality is present in property sales as opposed to property prices. However, there was also feedback from users to request that a seasonally adjusted HPI continues to be published.

In light of the feedback, it was agreed that a seasonally adjusted HPI will continue to be produced and published as part of the move to the new HPI. The level of geography at which the seasonally adjusted series will be calculated and published has yet to be decided, and will be assessed once a sufficient period of new data is available from the new HPI to carry out a formal seasonal adjustment review. Based on previous analysis, it is expected that the most suitable level of geography to produce and publish a seasonally adjusted HPI will be at the regional and national level. Further information on the seasonal adjustment will be presented in early 2016.

Publication of a seasonally adjusted HPI will sit alongside the publication of non-seasonally adjusted data. The new HPI will include full guidance for users to ensure they understand the differences between the two sets of house price data.

3.5 Incorporating the Northern Ireland data The LSPNI house price index will continue to be calculated and published quarterly. Northern Ireland data will be provided to ONS quarterly, and will be combined with the Great Britain data to give overall figures for the UK. This will be done by assuming that house prices in Northern Ireland are constant for each month in a quarter. In the two months following the end of a quarter, when Northern Ireland results for the most recent quarter are not yet available, indices and house prices for Northern Ireland will be carried forward from the previous quarter. They will subsequently be revised when the quarter is complete.

It is recognised that the assumption about Northern Ireland house prices being constant during the quarter is a simplification. However, sensitivity testing of alternatives – e.g. imputing a monthly path – shows that this has minimal impact on the overall UK HPI.

4. Publication Feedback from the user consultation made it clear that the new single official HPI should be published in a central location to make data access easier for users, and to remove any potential confusion for users in having the same data published across a number of different departmental websites. There was also a strong user requirement for the publication of a full back series to accompany the new HPI and publication of consistent house price levels, allowing comparisons of price levels over time. This section will look at the planned process for publication, the proposed method for producing average house price levels, plans for publishing a consistent back series for the new HPI and what additional house price data will continue to be published by departments alongside the new HPI.

4.1 Data to be published It is proposed that the new HPI will be published monthly using GOV.UK. This central point of publication will be managed by the Land Registry for England and Wales and provide users with headline prices for UK, country and regional data, along with sub-regional data (to a Local Authority level) for England, Scotland and Wales. Each quarter, a lower geographical breakdown for Northern Ireland will also be available through the LPSNI release, although users will be linked to this data directly from GOV.UK. The input data and new methodology will allow for the publication of the following breakdowns at all levels of geography detailed above:

- Type of dwelling (detached house, semi-detached house, terrace or flat)
- Type of property (new build or old)
- Volume of sales
- Cash or mortgage purchases (not available for Northern Ireland)
- Type of buyer (first-time buyer or existing owner; not available for Northern Ireland)

The monthly publication of data will be accompanied by an accessible (i.e. PDF download) monthly report/bulletin providing commentary and background information for the latest published data. This will also be accompanied by full tables of data for breakdowns detailed above (Northern Ireland detail will be available on a quarterly basis) along with interactive analysis tools, although the exact specification of these additional tools will be defined over the next few months and explained to users during the transition to the new HPI in 2016.

Where possible, a full back series for the new HPI will be published. However, this will be constrained by the availability of data, and is explained further in section 4.3. For example, the production of house price data broken down by whether the sale is completed using cash or mortgage is new analysis that will be available for the first time. The source data for this series is only available from 2011, so this breakdown can only be calculated from this year.

Publication will take place around 6 weeks after the period to which the most recent period relates. This is a similar timescale to the current ONS schedule but is later than the Land Registry currently publish.

4.2.1 Method of calculating average prices

There are different ways of calculating average prices. These include the arithmetic mean, the geometric mean and the median. The current ONS HPI uses the arithmetic mean, whereas the Land Registry uses the geometric mean and Registers of Scotland publish results for both arithmetic mean and median.

International experts and the GSS Methodological Advisory Committee were consulted as to which was the most appropriate method for the HPI. It was also a question in the User Consultation. The general view was that the geometric mean was the preferred measure, as it is less distorted by high values, it is in line with international best practice, and is consistent with the method used in the Consumer Prices Index.

4.2 Publication of average price levels

A limitation with the current ONS HPI is the method in which average price levels are published each period. The ONS approach is to publish in-year mix-adjusted average prices. The process of mix-adjustment requires that, in each January, a fixed basket of properties is updated to reflect changes in the composition of properties being sold. This basket is then used to produce modelled prices for the year, before the basket is then updated again in the subsequent January. This means that the average prices produced from a fixed basket in 2014 are not directly comparable with the average price produced using the 2013 basket as they will reflect a different mix of properties (this process is explained fully in section 3.4 of the article ‘Official House Price Indices Explained’).

This lack of comparison between years has been identified as an issue for users, with feedback from the User Consultation exercise supporting the publication of a comparable set of price levels over time. As a result, the new HPI will publish a comparable set of price data by up-rating a base set of transactions with the price index11, a similar approach to that used in the current Land Registry HPI. To ensure the base set of transactions remains representative, the base period for the price series will be updated every five years, and the whole of the average price series rescaled to align with the new base period. For example, assume the initial price series calculated for the new index will use January 2010 as the base for average prices. This base will be updated every 5 years (so the next update will move the base to January 2015). This is illustrated in table 2 (in reality, the publication of the new HPI will use January 2015 as the base, but the same approach will hold):

Table 3 - Example of re-referencing for the new price level series

| Price Index (2010=100) | 2010 base year price series | actual average price | 2015 base year price series | |------------------------|-----------------------------|----------------------|-----------------------------| | 2005 | 96 | 192,000 | 195,840 | | 2006 | 93 | 186,000 | 189,720 | | 2007 | 90 | 180,000 | 183,600 | | 2008 | 85 | 170,000 | 173,400 | | 2009 | 95 | 190,000 | 193,800 | | 2010 | 100 | 200,000 | 200,000 | | 2011 | 105 | 210,000 | 214,200 | | 2012 | 110 | 220,000 | 224,400 | | 2013 | 115 | 230,000 | 234,600 | | 2014 | 120 | 240,000 | 244,800 | | 2015 | 125 | 250,000 | 255,000 | | 2016 | 130 | 260,000 | 265,200 |

In the example (which is based on annual data, as opposed to monthly for ease of presentation), the index remains unchanged by the re-referencing of the price base period from 2010 to 2015 but the average prices are all scaled up by 2 per cent, which is the percentage difference between the actual price for 2015 and the 2010-based average price for that year. This approach will ensure

______________________________________________________________________

10 http://www.ons.gov.uk/ons/guide-method/user-guidance/prices/hi/official-house-price-statistics-explained.pdf 11 The option detailed in section 2.40 of the user consultation: http://www.ons.gov.uk/ons/about-ons/get-involved/consultations-and-user-surveys/consultations/consultation-on-the-development-of-a-definitive-house-price-index/consultation-on-the-development-of-a-definitive-hpi.pdf that a set of comparable average prices are published, and that these prices remain representative of the current market.

There was also interest from users in the continued publication of the mix-adjusted prices (i.e. the modelled prices used in the production of the price indices). The Working Group acknowledge the interest in this set of data and will consider options for making available the modelled mix-adjusted price data on a less frequent basis.

4.3 Presentation of a consistent back series

A key requirement for users is the publication of a consistent back series to accompany the new HPI. Particularly to cover the time-series already available from the house price indices which the new HPI will ultimately replace (i.e. the Land Registry and ONS house price indices). The current Land Registry HPI publishes at a country, regional and county/unitary authority level back to January 1995. The ONS house price index publishes data at a national and regional level back to 1968, although there are some regions where data is not available back to this date. The ONS data are published quarterly prior to 2002 and there are methodological changes during the time series. A commitment has been made to produce back series for the new HPI to reflect the availability of data that is currently published. This will need to be calculated in two stages:

Firstly, historic data will be produced using the data and new methodology detailed in section 3 (i.e. recalculated based on the new approach). However, this approach can only be used to produce a time-series based on the availability of data (for example, the Land Registry data for England and Wales is only available from 1995).

Secondly, where there is insufficient data available to produce historic data using the new methodology, a suitable method will be applied to derive a back-series, for example, prior to 1995 the path of the current ONS HPI could be used to construct the period from 1968 to 1995.

It should also be advised that a full back series will not be available for each of the various breakdowns available in the new HPI – for example, the distinction between cash and mortgage sales is only available from 2011.

The following table summarises the likely approach and availability of historic data:

| Published geography | Recalculated back series | Derived back series | |--------------------------------------|--------------------------|---------------------| | United Kingdom | 1995 to present | 1968 to 1995 | | England | 1995 to present | 1968 to 1995 | | Wales | 1995 to present | 1968 to 1995 | | Scotland | 2003 to present | 1968 to 2003 | | Northern Ireland | 2005 to present | 1968 to 2005 | | English regions | 1995 to present | 1968 to 2005 | | England sub-regional (Local Authority)| 1995 to present | Not available | | Wales sub-regional | 1995 to present | Not available | | Scotland sub-regional | 2001 to present | Not available | | Northern Ireland sub-regional | 2005 to present | Not available | 4.4 Ongoing publication of existing house price data

The development of the new HPI brings together the producers of official house price statistics (where official is defined as those statistics produced by Government bodies). This development does not cover the production of house price statistics by private sector companies such as Nationwide and Halifax. These well respected house price indices will continue to play an important role in the assessment of the housing market.

In addition to the private sector data, there will continue to be other sources of official house price data published alongside the new HPI. User feedback gathered during the 2014 consultation exercise reflected a strong demand for the continued publication of such data. The new HPI will include better explanation of the data available (both by Government and private producers) and full explanations of the differences. A summary of the additional house price data that will continue to be published by the departments involved in this development are:

**Registers of Scotland**

Registers of Scotland (RoS) will continue to publish their Quarterly House Price Statistics (QHPS), which provides simple average and median house prices, volumes of sales and market value for all sales of residential properties with RoS in a quarter where the selling price of the property lies between £20,000 and £1,000,000. The raw data feeding into the production of QHPS will be derived from the same source as the data used in the production of the new HPI although different methodology will be applied to produce a monthly house price index, and the HPI will cover the full range of property prices. A full explanation of the differences between QHPS and the new HPI will be published in each release once the new HPI is published. RoS is reviewing whether to request assessment of QHPS as a National Statistic alongside the new HPI.

**Land Registry House Price Data**

The Land Registry for England and Wales will continue to publish monthly Price Paid Data, which includes information on all residential sales in England and Wales logged for registration. The Price Paid Data will also form the basis of price data feeding into the new HPI (for England and Wales) although there are some notable differences with the HPI dataset, such as the inclusion of a cash or mortgage variable and the timing at which the dataset is produced. For the purpose of the new HPI, the Price Paid Data will be matched with property extract data from the Valuation Office Agency (not publically available) for use in a regression model to produce modelled price data each month. This process means the ongoing publication of Price Paid Data will not allow users to replicate the production of the new HPI.

**Land and Property Services, Northern Ireland**

Land and Property Services, Northern Ireland (LPSNI) will continue to publish their quarterly residential property prices index for Northern Ireland (RPPI) statistics which include information on volume of sales. The same published data will be used in the construction of the new UK HPI, with LPSNI providing the aggregated price and index data for inclusion. The RPPI will be assessed as a National Statistic alongside the new HPI.

**House Price Statistics for Small Areas**

ONS will continue to publish House Price Statistics for Small Areas (HPSSAs). These statistics report the median price paid for residential property at a middle layer super output area. This set of house price data provides a representation of the price paid for property in small geographical areas that are not available in headline indices. They are useful for assessing the affordability of housing at this level of geography as well as broad patterns in prices and the number of sales over time.

**Quarterly and annual housing analysis – ONS**

ONS publish a number of quarterly and annual analyses of the housing market alongside their current HPI. This includes distributional analysis of mortgages using the Regulated Mortgage Survey source. It is intended to continue publishing this analysis, which is based on mortgage only transactions.

ONS will also consider publishing additional annual analyses based on the new HPI. This may include average prices per metre squared and average prices based on the number of rooms in a property.

**5. Transition**

The Working Group understands there will be an impact for users in the publication of a new HPI. In particular, the new HPI will replace the house price indices currently published by ONS, Land Registry and LPSNI and therefore users will need to fully understand the transition from these current outputs to the new HPI.

Further details on the transition to the new HPI will be made available in early 2016, including a series of user events to provide users with full details of the new HPI and the impact of changing to the new series. It is planned to first publish the new index in June 2016 (the ONS HPI and Land Registry HPI will cease to be published from this point). There is still a large amount of work to take forward to reach this date summarised by the following high-level plan:

- Finalising methods, systems and test production of new HPI (including production of back series) - January and February 2016
- Seasonal adjustment review of new HPI – February/March 2016
- Parallel run of new index alongside current official HPIs – March, April and May 2016
- User transition – article and events – March and April 2016
- Full documentation of methods – March, April and May 2016
- Launch of new HPI – June 2016
- 2nd phase assessment of new HPI as a National Statistic (including quality assurance of administrative data sources) – June 2016 onwards

A finalised set of publication dates for the new HPI will be published in early 2016 as part of the initial user transition work. The Working Group would welcome input from users regarding the transition to the new HPI; in particular what assistance would be required to ensure the switch to the new HPI is as smooth as possible for all users. Please get in touch using the following email address in the first instance:

Email: Hpi@ons.gsi.gov.uk

The UK Statistics Authority will continue to monitor progress with the development of a new HPI ahead of publication and ultimately assessment of the new HPI as a National Statistic.