DELIVERING AND MEASURING DATA QUALITY IN 

UK NATIONAL STATISTICS 

Marta F. Haworth and Jean Martin 

Office for National Statistics, UK 

Email: marta.haworth@ons.gov.uk jean.martin@ons.gov.uk 

Abstract 

A new Framework for UK National Statistics was launched in June 2000. Key objectives are to improve the quality, 

timeliness and relevance of official statistics to customers within government and the wider community, and to 

demonstrate that the statistics are produced to the best professional standards as well as minimising the burden on 

those whose supply the information on which they are based. The emphasis on quality presents major challenges to 

methodologists in developing methods and tools to deliver high quality statistics. The methodologists are also 

contributing to the development of measures and indicators of quality to establish whether statistical products and 

services meet the required standards and to evaluate the progress and effectiveness of new developments and quality 

improvements. 

The paper describes the quality strategy for UK National Statistics and the various approaches to delivering quality 

products and services. A number of specific projects to improve quality and to provide better measures of both 

output and process quality are described. 

Keywords: data quality, process quality, output quality 

Introduction 

Following a wide-ranging public debate about UK official statistics, a Framework for UK 

National Statistics was launched in June 2000 (http://www.statistics.gov.uk.). The core values 

are quality, efficiency and public confidence in UK National Statistics. The aim is to provide an 

accurate, timely, comprehensive and meaningful picture of the economy and society to support 

public debate, research, decision making and the formulation and monitoring of economic and 

social policies. The quality of National Statistics is thus key to meeting this aim. 

This paper describes the methodological work currently being undertaken to deliver and measure 

data quality for National Statistics. This is set in the context of the overall UK National Statistics 

Quality Strategy and the Programme of National Statistics Quality Reviews launched in 2000. 

The measures of data quality include both those for data sources such as surveys and those for 

more complex outputs. The paper also discusses quality reporting which is being developed 

within the European quality management framework. 

National Statistics 

In the UK National Statistics are produced by staff working both in the Office for National 

Statistics (ONS) and in other government departments and agencies under the UK devolved 

statistical system. The term 'National Statistics' refers to the collective set of outputs produced 

within this system. All public access databases and publications produced by ONS and many 

key public interest statistics produced by other government departments are included. 
2 

UK Statistics Commission 

The UK Statistics Commission has been set up as part of the National Statistics Framework in 

order to provide independent, reliable and relevant advice on National Statistics and an 

additional safeguard on the quality and integrity of National Statistics. Its role is to be a source 

of high quality independent advice on statistical issues. The Commission plays a key role in 

advising on the quality, quality assurance and priority setting for National Statistics. 

The Importance of Quality 

Since the launch of National Statistics, there have been strong drivers for strengthening the focus 

on quality. In addition to the implementation of National Statistics Quality Strategy (see below), 

the UK Statistics Commission has put the measurement and reporting of reliability of National 

Statistics outputs at the top of the list of issues in which it takes particular interest. 

For the UK Office for National Statistics, the implementation of a new business strategy will 

lead to its transformation to a quality statistical organisation. The key objectives of the new 

arrangements are to improve quality, timeliness and relevance to customers within both 

government and the wider community; also to demonstrate that official statistics are produced to 

the best professional standards while minimising the burden on those who supply the information 

on which they are based. The strong focus on minimising respondent burden and compliance 

costs, and on minimising non-response, demands continuing improvement in data collection 

methods and questionnaire design, and on editing procedures which involve recontacting 

respondents. 

New developments have also been taking place at European level, through the setting up of an 

EU Leadership Expert Group on Quality. This group has produced specific recommendations on 

how quality should be taken forward within the European Statistical System (ESS) as a whole. A 

list of dimensions of output data quality has been developed for use in the ESS. These are: 

• Relevance 

• Accuracy 

• Timeliness 

• Accessibility and clarity 

• Comparability 

• Coherence 

• Completeness 

From the above it is clear that quality of statistical products has more dimensions then just 

accuracy and that some dimensions can be difficult to measure. In practice the main emphasis is 

on accuracy and timeliness which can be measured quantitatively. Other components of the 

quality may require qualitative assessments. 
3 

National Statistics Quality Strategy 

The National Statistics Quality Strategy aims to deliver business excellence, to be responsive to 

user needs, to respect respondents, particularly with regards to maintaining the confidentiality of 

the information they provide, and to have a strong user focus for National Statistics outputs and 

services. The strategy has three components: 

• Protocols for the National Statistics Code of Practice 

• Building quality up front: quality management 

• National Statistics quality reviews 

More detailed quality strategies and quality plans are being developed as part of implementation 

of the strategy. 

National Statistics Code of Practice Protocols 

A Code of Practice for National Statistics has been drafted and is awaiting ministerial approval. 

This is to be underpinned by a series of protocols which are currently under development. These 

are statements which will describe to users the standards they can expect from National Statistics 

and set out the practice that should be followed by those working to deliver National Statistics. 

Building Quality Up Front: Quality Management 

This is at the heart of delivering quality in National Statistics and includes: 

• User focus and user consultation 

• Quality process design leading to quality results 

• Provision of indicators of quality to accompany statistics 

• Quality evaluation and reporting 

• Review and continuous improvement 

• Documentation 

• Management tools and the use of the Business Excellence Model 

• Risk management 

National Statistics Quality Reviews 

A programme of National Statistics Quality Reviews of key outputs has been implemented. The 

reviews have independent inputs and bring together producers, independent experts and users. 

The reports of the reviews are published and a formal response on behalf of the National 

Statistician is also made publicly available. In each case implementation plans are developed by 

the business area responsible for the relevant statistics. The business areas are responsible for 

introducing the recommended changes and improvements. The plan is that key reviews will 

cover all main outputs within a period of about five years. 

Standardised Tools Leading to Quality Results 

Further initiatives to improve quality include the development and use of common tools and 

methods which are key ingredients to providing high quality statistics and services. Variation in 

tools and processes often leads to variation in product characteristics with potential risks to 
4 

output quality. The use of common tools improves efficiency while the quality assurance of tools 

and techniques facilitates documentation, training of new staff, internal staff moves and changes 

in processes or process re-engineering. ONS is carrying out a major project to improve its 

statistical infrastructure, standardising the use of best tools and methods to deliver quality results 

and greater efficiency. 

Quality improvement initiatives 

In this section we provide some examples of specific initiatives carried out by ONS to improve 

various aspects of quality. 

Harmonising concepts, definitions and classifications 

Some seven years ago ONS embarked on a major initiative to harmonise concepts and 

definitions in common use on government household surveys. This was in response to the 

recognition that minor differences between surveys was a threat to coherence of survey estimates 

from different sources. The differences were also a threat to the quality of information collected 

in surveys since in general the same interviewers, working on a range of different surveys, had to 

learn all the minor differences in definitions and individual questions between the different 

surveys, which increased the chances of some interviewers making mistakes. 

The initiative covered the harmonisation of both inputs and outputs. A booklet was produced 

(GSS, 1996) (now a web publication - ref.) documenting the agreed harmonised concepts and 

definitions, classifications, individual questions and output variables to be used across all major 

government surveys. Clearly many surveys had to introduce changes to conform to the agreed 

harmonised standards. Such surveys are carried out for a range of government departments and 

by a number of different survey organisations, so getting agreement to changes from the key 

stakeholders was a major undertaking and implementation had to be phased in gradually 

depending on the constraints on individual surveys. There are two sets of standard questions: the 

primary set applies to all surveys (basic demographic information, household composition, 

economic activity etc) while the secondary set covers those which apply only to particular groups 

of surveys. Some surveys ask more detailed versions of the questions but need to be able to 

collapse the detail to form the standard variable which should then be comparable across 

surveys. 

The range of topics covered by the harmonisation initiative has increased over time, with an 

interdepartmental committee co-ordinating agreement on new areas for harmonisation and 

implementation arrangements. Further topics for inclusion are under review and updates are 

agreed when necessary (e.g. to take account of major changes in classifications, external changes 

which affect questions etc.) 

Initially this initiative applied only to household surveys. Attention was, however, paid to 

designing questions for the 2001 Census so that as far as possible the outputs would be

comparable with the harmonised survey definitions, although the different requirements of the 

Census and the constraints of the short self-completion form used necessitated some 

compromises. 
5 

More recently attention has turned to the business surveys with the acceptance in principle of the 

goal of having a library of standard definitions and questions for use across the range of surveys. 

These currently have many minor differences in definitions and questions used for key concepts 

such as income and turnover. Not only is this inefficient in terms of questionnaire design and 

processing, but also confusing for the large businesses which are in the samples for a number of 

different surveys. Thus harmonisation of definitions and questions can be expected to lead to 

improvements in survey data quality and in the coherence of estimates. 

Improving questionnaire design 

ONS, like many other NSIs, set up a cognitive question testing laboratory some years ago. 

Initially most of the work was carried out to develop and test questions for the 2001 Census and 

for some household surveys. A well established unit which develops and tests questions and 

survey instruments for household surveys has been in operation for around six years and 

attention is now being extended to business surveys, again mirroring experiences in other NSIs. 

Some early work done on surveys of employers revealed the need to examine not just 

understanding of questions on forms but the whole process of data collection, from the point of 

arrival in the organisation of a survey letter or questionnaire, to what the business has to do to 

supply information and make a return to ONS. A facility to provide expertise in sound principles 

of question and questionnaire design and testing methods for business surveys is now under 

development. 

A further aspect of questionnaire design for business surveys is the plan to create customised 

questionnaires which take account of features of the responding unit (industrial sector, size of 

establishment, information from previous rounds of data collection). This is intended to improve 

data quality and reduce the burden on respondents by asking them to respond only to relevant 

questions. 

Improving cost effectiveness of editing 

On household surveys the need for and therefore the cost of editing reduced dramatically with 

the move from paper interviewing instruments to computer assisted interviewing. The 

computerised interviewing instrument automatically routes interviewers to the next appropriate 

question; checks on consistency of answers can be programmed to run during the interview

leaving only the most complex editing to be carried out in the office. ONS was an early adopted 

of CAI and had completed the transition of all major surveys to this mode of interviewing by the 

mid 90s. On business surveys, by contrast, most data collection is still based on paper 

questionnaires although touch-tone data entry and Internet based data collection are becoming 

more prevalent. Editing paper questionnaires accounts for a large proportion of business survey 

costs (up to 40%) and frequently involves re-contacting respondents to check their answers 

which both takes time and resource and increases the burden on businesses. Thus improving the 

efficiency of editing without adverse effects on data quality is being given high priority. 

ONS has been carrying out work to evaluate the potential of introducing two changes to editing 

procedures: automatic editing to deal with common, easily recognisable errors; and selective 

editing to target for editing cases which will have most impact on survey estimates. This 

approach involves, for each case which fails one or more validation checks, computing a score 

which is designed to reflect the importance of editing that record. Cases with scores above a pre-
6 

determined threshold are selected for scrutiny and editing (often involving recontacting the 

business) while those below the threshold are accepted without further scrutiny. Both 

approaches have been tested and are being implemented on most business surveys in the course 

of the next few years. 

Measuring and Reporting Quality 

Measuring quality is important in order to establish whether statistical products and services 

meet the required standard, to evaluate progress and effectiveness of new developments and for 

quality improvement programmes. It is also a valuable tool for ensuring that statistical 

processes meet user needs and priorities, and deliver quality results. Quality reports and 

declarations are important as a means of communicating the quality standard of statistics, their 

strengths and limitations. 

Measuring quality across statistics is a complex undertaking, with few commonly accepted 

guidelines or standards. Quality as a concept is not easy to define with clarity. The ESS list of 

output quality attributes provides a framework for data quality assessment, although for some of 

the attributes this poses considerable difficulties of measurement. Documenting aspects of 

survey processes is also a valuable component of quality reporting – the UK has developed a 

Statistical Quality Checklist (GSS, 1997) to aid producers. This is designed for application 

mainly in the context of surveys and covers issues which should be addressed in reports: 

objectives, design, coverage, data collection, processing, estimation and analysis. 

In addition, guidelines have been prepared for documenting processes so as to ensure that

practices are regularly evaluated and continuously improved. A major new initiative is to build a 

standards and guidance database: a source of information on current practices for operational use 

by survey managers and analysts, and as a tool for disseminating and developing guidance on 

best practice. 

In the past measurement of quality concentrated on attempting to estimate accuracy in terms of 

mean square error - a combination of measures of bias and variance. The estimate of variance is 

still a key feature of quality measurement but poses major methodological challenges in 

obtaining robust measures for the more complex statistics. The recognition of the difficulty of 

measuring other sources of error satisfactorily has lead to attempts to develop measures of less 

direct aspects of quality, and in particular on the measurement of the quality of processes which 

are relevant to assessing output quality. The interest in output quality goes beyond surveys and 

data sources, and encompasses the full range of the National Statistics outputs. 

ONS has carried out a number of projects to develop better measures of both output and process 

quality which are described below. 

Variance estimation for complex statistics 

Estimates of sampling variance are recognised measures of the impact of sample design on 

survey precision and are thus key measures of output quality. However, methodological work in 

this area faces some difficult challenges. (Platek and Sarndall, 2001). Specifically, variance 

estimation for the Average Earnings Index, the Index of Production and the Consumer Price 
7 

Index (both levels and changes) has to solve complex computational problems and assessing 

overall accuracy has not been possible, although significant progress has been made on coverage 

aspects. 

Accuracy Assessment of National Accounts Statistics 

There is also a continuing demand for producing quality indicators for National Account 

statistics. This is a task of even greater complexity. In the UK, the approach has traditionally 

been based on analysis of revisions; in addition some assessments of relative quality of different 

National Accounts indicators (usually in terms of banding) have been published in the past. 

More recently in response to growing user demands for quality indicators for National Accounts 

a European Task Force was set up to review accuracy assessment for National Accounts. The 

European Court of Auditors’ proposed measure for the accuracy of GNP was by means of 

confidence intervals. However, the Task Force concluded that the confidence interval approach 

was not feasible and instead recommended focusing on quality measurement through processes 

which lead to the resulting statistical estimates. The Task Force has proposed a tabular analysis 

which quantifies the adjustments made by type (validation, conceptual, exhaustiveness, 

balancing) and source (surveys/censures, administrative data, models/extrapolation and other). 

Specific aspects of compilation are also included. Such tabular analysis has to be completed 

during compilation of the accounts. The UK is planning to produce such a table for its National 

Accounts publication for 2002 which will refer to estimates for 2000. Some other European 

countries will also be piloting this approach. In addition, further work has been done on 

presenting revision analysis and comparisons of estimates with and without balancing. 

Developing better indicators of process and output quality for surveys

Given the practical and methodological difficulties in constructing comprehensive measures of 

output quality, ONS, like many other organisations, recognises the importance of measures and 

indicators of process quality on the basis that these are powerful indicators of the quality of the 

eventual outputs. 

For household surveys ONS has been developing proposals for a number of different indicators 

of the quality of survey processes. (Haselden and White, 2001). These have been developed 

following detailed work to identify key processes and the outputs of each process, in order to 

develop appropriate indicators of the quality of each which might impact either on the next 

process in sequence or at a later point in the processing chain. Some, like response rates, are 

quantitative measures but many are qualitative descriptions of the processes undertaken which 

are likely to have an impact on quality. Examples include: 

• Effectiveness of sample design in relation to cost of fieldwork 

• Quality of sampling instructions 

• Procedures followed to develop and test questions 

• Ease for respondents of understanding and answering questions 

• Accuracy of programming survey instruments 

• Efficiency of allocation of work to interviewers 

• Training and monitoring of interviewers 

• Response rates 
8 

• Interview length 

• Efficiency of interviewer work patterns 

• Accuracy of data entry and coding 

It will be apparent from the list that some indicators will be used mainly within the organisation 

to monitor the efficiency and effectiveness of its survey operations but most will be provided 

along with other metadata to users of the survey data and results so they are in a position to judge 

the quality of survey data and its suitability for their intended uses. 

A European research project on Model Quality Reports in Business Statistics has been set up to 

develop guidelines for producing quality reports for business survey outputs. (Eurostat, 1997) 

The study concluded that quality reports should be organised into three sections; a summary of 

quality assessments; the more detailed quality report and a description of the survey and its 

processes. Other recommendations that were made included the following points: 

• Both survey managers and methodologists should be involved in the quality assessment 

work; 

• The tools for quality assessment need to be available and shared, to save resources and to 

ensure that users of different surveys are presented with a similar approach to quality 

assessment; 

• Availability of data is critical. Careful planning is needed to ensure that all of the necessary 

datasets are available when needed; 

• Quality assessment should be part of the results process to ensure that maximum use is made 

of the information on quality. 

The project was set up to develop a detailed description of the methods for assessing the quality 

of surveys with particular application in the context of business surveys. The approach was to 

apply these methods in some example surveys to evaluate their quality, and then to produce 

guidelines for producing quality reports for business survey outputs more generally. 

Defining and measuring non-response 

Response rates are one of the most important and widely used measures of survey quality. 

Response rates are not in themselves direct measures of survey quality (they do not provide 

measures of bias or variance of estimates) but are important quantitative indicators of quality of 

survey outputs, but they actually measure the quality of a key part of the survey process rather 

than outputs. Comparisons hampered by inconsistencies in the ways in which response outcomes 

are defined and response rates are calculated. Building on work in the US to standardise 

response rates for RDD telephone of response rates between different surveys, survey 

organisations and countries has been surveys, in-person household surveys and mail surveys of 

named persons (AAPOR, 2000), ONS and the National Centre for Social Research have recently 

completed a project to standardise response outcomes and the calculation of a number of 

different rates for the types of in-person household surveys common in the UK, and which 

indicate different aspects of survey quality. These are in the process of being implemented on 

most major government household surveys. Initial investigations reveal similar problems of lack 

of standardisation in the definition and calculation of response rates on business surveys so a 

project has been set up to tackle these next. 
 

9 

Conclusions 

This paper has attempted to describe the new environment for the production of high quality 

National Statistics for the UK. The increased emphasis on the importance of quality has lead to a 

number of important initiatives in developing a quality strategy, carrying out a programme of 

quality reviews, improving the statistical infrastructure in the ONS and in a number of 

methodological projects aimed at both improving various aspects of quality of statistical outputs 

and in developing better measures of quality so that users have appropriate information about the 

quality of our outputs and services. 

References 

AAPOR (2000) Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for 

Surveys. American Association of Public Opinion Research. 

Eurostat (1997) Model Quality Report for Business Statistics. Eurostat. 

Government Statistical Service ( 1997) Statistical Quality Checklist. Office for National 

Statistics. 

Government Statistical Service (1996) Harmonised Concepts and Questions for Government 

Social Surveys. Office for National Statistics. 

Haselden L and White A (2001) Developing new quality indicators in social surveys. Paper 

presented at the International Conference on Quality in Official Statistics, Stockholm, May 14– 

15 2001. 

Platek R and Sandall C-E (2001) Can a Statistician Deliver? Journal of Official Statistics, Vol 

17, No. 1. 

Office for National Statistics (2000) A Framework for National Statistics.