The Quality Assurance Reviews at Statistics Canada 

Laurie Reedman, Claude Julien 

Statistics Canada 

100 Tunney’s Pasture Driveway, Ottawa (Ontario), K1A 0T6, Canada

1. Description of the Quality Assurance Reviews 

1.1 Origin 

Since 2007, Statistics Canada has developed and implemented an annual quality review process through which 

quality assurance practices in a set of statistical programs (e.g. Consumer Prices) or functions (e.g. seasonal 

adjustment) are subjected to an independent internal assessment. All of these reports and assessments are presented 

to the Policy Committee (Statistics Canada’s topmost management committee). 

In the first two waves of reviews, the focus was on assessing the measures taken in the execution of programs to 

prevent erroneous data from being released to the public, i.e., to assure the accuracy of the data. The objectives of 

the review were to identify best practices that should be promoted to other programs at Statistics Canada as well as 

areas where action is needed. In the third and subsequent waves of reviews the scope has been broadened to include 

the implementation phase and automated systems. 

The mandate of the reviews is to demonstrate how quality is being managed. Risks to quality are explored and are 

distinguished from project management risks such as cost or time overruns. The quality reviews are one element in 

Statistics Canada’s Integrated Strategic Process, that includes the Departmental Project management Framework and 

the Integrated Risk Management Framework, and they also provide input to the audit and evaluation process. 

1.2 Objectives 

The first objective of the reviews is to identify risks that could prevent the statistical program from being able to 

deliver its regular product. A risk event is defined as the event or occurrence that would hinder the production or 

delivery of the regular product. The risk of that event occurring is described in terms of the likelihood of it 

happening within the next two years, and the impact on Statistics Canada (not just the impact on the local program) 

if it did. A three point scale, as is typically found in risk management literature, is used to represent the likelihood 

and the impact of the risk. Finally, a three point scale is used to reflect the composite risk, taking both the likelihood 

and impact into account. 

The second objective is to highlight good practices already in use, so that they can be shared by other programs. 

Some or all of the risk associated with a particular risk event could be mitigated by practices already in place. The 

residual risk is addressed through recommendations for improvement. The recommendations should have 

measureable impacts, and can be quality assurance practices or specific actions to mitigate the risks.

1.3 Scope 

In 2007, the first wave of reviews included nine key economic indicator programs. Six more programs were 

reviewed in 2008. At this time, the focus was on assessing the measures taken in the execution of programs to 

prevent erroneous data from being released to the public. A detailed questionnaire was developed to guide the 

review. The questionnaire covered all the steps in a typical statistical program and posed questions designed to 

stimulate discussion of the strengths and vulnerabilities of the program with respect to the accuracy of its data 

products. In 2009 the scope was broadened to include risks in the implementation of the design of the program as 

well as risks in its automated systems. 
Regular survey programs, cost recovery programs and programs based on administrative data are in scope for the 

Quality Assurance Reviews. Also, horizontal topics such as knowledge transfer, coding methods and practices and 

seasonal adjustment practices have been the subject of Quality Assurance Reviews. 

1.4 Participants, roles and responsibilities 

The Quality Secretariat at Statistics Canada is mandated to promote and support the use of sound quality 

management practices across the Agency. Coordinating the quality reviews is the responsibility of the Quality 

Secretariat. This includes preparing materials, facilitating communication and liaising with upper management. 

Reviewers are Statistics Canada employees at the middle management level, and they are nominated by their 

superiors. They are assigned to review programs outside of their current area of responsibility, so as to ensure 

independence and an unbiased assessment. Reviewers are not provided any additional budget to carry out the 

review. The workload for each reviewer is approximately twelve working days, spread over an eight month period. 

Members of Statistics Canada’ Policy Committee, the Agency’s most senior executive committee, propose programs 

from within their own area of responsibility to be reviewed. This includes programs that can be proposed by the 

Chief Statistician. Programs about to undergo redesign could be good candidates, in order to get an independent 

assessment of where to focus attention. Programs that have experienced quality issues or those with known 

vulnerability such as aging infrastructure, lack of experienced personnel or unstable funding, could also be 

proposed. The programs are also selected in coordination with the auditing and program evaluation functions of the 

Agency to avoid duplication and excessive burden on program managers and staff. Recommendations from a quality 

review are a key input to the allocation of resources to address program or corporate risks. Typically three to six 

reviews are conducted at the same time, on an annual basis. 

Reviewers are guided and facilitated by a steering committee made up of higher level managers, at the Director 

General level, who ensure that relevant and timely information is shared with the reviewers. Programs under review 

are not required to produce any new documentation expressly for the reviewers; however they are expected to make 

available all existing documentation. 

1.5 The review process 

Two reviewers are typically assigned to review each program, or a larger team could be assigned in the case of a 

horizontal review requiring consultation with several different groups or areas. Information is gathered primarily by 

meeting with program area staff and through documentation. Typically three to five meetings take place. Staff at all 

levels is invited, as well as internal service providers such as systems engineers and methodologists, where 

appropriate. 

The reviewers are provided with background information about the program to be reviewed. They are given access 

to any and all existing documentation. This could include annual reports, data product descriptions, quality reports, 

working papers, design descriptions, and everything available on the division’s internal communications network. 

The objectives are clearly stated in terms of what the reviewers should be looking at, and what they should be 

looking for. The Generic Statistical Business Program Model (GSBPM) is used to clarify which processes are in 

scope for the review and to provide a common terminology. A timeframe is given for performing the review and a 

deadline is set for presenting the findings to Policy Committee and submitting deliverables. 

1.6 Deliverables 

A summary of the most significant risks, best practices and recommendations is presented to the Policy Committee 

at the conclusion of each review. A formal report with much more detail is also prepared. This report becomes the 

property of the Director managing the reviewed program. While the summary documents are made public to anyone 

in Statistics Canada, the detailed reports are made available only upon request. 
The Quality Secretariat acts as custodian of the various documents produced by each quality assurance review. 

Over the years, this has become a rich repository of quality assurance risks and practices. These have been 

categorized, summarized and made available to all Statistics Canada employees. 

2 Benefits 

2.1 Benefits to reviewers 

Being a reviewer is an excellent on-the-job learning opportunity. Given that reviewers are assigned to programs 

outside of their current area of responsibility, at the very least they are given the chance to learn about a new 

program. New professional relationships can be forged, expanding personal networks and knowledge of the 

organization. Leadership and communication skills are developed through meeting with the program area staff 

members. Discussions about risks and mitigating actions sometimes lead to the disclosure of sensitive information. 

Discretion and good judgment are required in these situations. A professional relationship built on trust and 

common values is necessary to ensure that the issues are fully explored. Brain storming and problem solving skills 

are exercised through the process of crafting relevant recommendations with measureable impacts. 

Analytical skills are honed through the experience of examining a program and assessing its strengths and 

weaknesses. These skills can be used on the reviewers’ regular projects. Good quality assurance practices 

discovered through being a reviewer can be brought back to the reviewer’s home division. 

2.2 Benefits to programs being reviewed 

Program managers are afforded a variety of benefits from having their programs reviewed. If a program is heading 

into a redesign phase, the recommendations from the review provide valuable direction. In some cases, program 

managers are already aware of risk mitigating work that needs to be done, but they lack the resources to do the 

required development. A quality assurance review can raise awareness of such situations and lend strength to a 

request for additional funding. 

If on the other hand the review finds few unmitigated risks, the good practices already in place are highlighted and 

shared with other programs. This provides a welcome opportunity to acknowledge employees making positive 

contributions to their respective projects. 

Although the quality review process is completed upon the delivery of the findings, this is not the end of the story 

for program managers. It is their responsibility to address the recommendations from the quality review and to 

report progress on related initiatives in subsequent annual reports. 

 2.3 Benefits to the whole organization 

There is a net gain to the entire organization for investing in a few quality reviews every year. Sharing the role of 

reviewer among the middle managers ensures a wide dispersion of this on-the-job training opportunity. The 

professional relationships formed through the course of a review strengthen the connections between the various 

branches of the organization. Placing the emphasis on improving quality assurance practices rather than laying 

blame for exposure to risk ensures a continued willingness to share information and demonstrates the organization’s 

commitment to quality. All program areas are encouraged to “learn by example”, by taking up good practices 

initiated in other areas. 

3 Challenges 

3.1 Choosing programs 

The Quality Assurance Review process was initiated in response to several unfortunate risk events occurring at 

Statistics Canada. The first wave of reviews was aimed at mitigating the most significant risks, and targeted key 

economic indicator programs. Since then the scope has been broadened to include essentially all programs, whether 

they are based on survey or administrative data, cross-cutting services such as seasonal adjustment or automated 
coding, or professional activities such as peer and institutional review of analytical products or the transfer of 

corporate knowledge. 

While the programs proposed for review in any given year do not constitute a representative sample of in-scope 

programs, there is attention given to selecting programs from the various fields, and achieving a balance between 

high profile programs, cost-recovery programs, and cross-cutting services. A more systematic approach to program 

selection could result in a more uniform coverage; however it is desirable to maintain the flexibility to include 

programs for which the timing is strategically appropriate for an independent assessment. 

3.2 Governance and follow-up

The institutional environment at Statistics Canada fosters a commitment to quality and high ethical standards. In 

this light, employees are willing to participate in the quality assurance reviews and they treat the information that 

surfaces through the reviews with professionalism and discretion. The four stakeholders – the Quality Secretariat, 

reviewers, steering committee and program managers – understand and respect each other’s roles and 

responsibilities. If ever the reviewers are not getting the information they need, it is the role of the steering 

committee to facilitate communication. 

Over the seven years that quality reviews have been conducted, several reviewers have also experienced the other 

side of the coin, having had their own programs undergo a review. This shared experience has a positive effect on 

the overall review process. 

Although the Quality Secretariat coordinates the reviews, it is not mandated to do any formal follow-up. However 

this is something that could be done in the future. A follow-up could be done to measure take-up of key 

recommendations. One challenge with this is that the program area would have to be afforded sufficient time and 

resources before the follow-up. Another challenge is that allowance should be made in the case where a mitigating 

action different from the recommendation from the review is implemented. 

4 Findings 

4.1 Common areas of risk, and mitigating actions 

The risks discovered to date fall into the following categories: staffing; project management; systems and

processing; standardization; and documentation. The practices to mitigate these risks are summarized below. 

Staffing 

 Investing in training throughout an employee’s career is vital. Training includes both formal and informal 

methods. 

 The use of learning plans ensures that employees continue to expand their skill set with relevant and 

practical tools. 

 Job shadowing, cross-training and overlap periods are initiatives used specifically to reduce the risk of 

losing knowledge when employees change jobs or take their retirement. 

 Maintaining a dialogue with Canadian universities and actively recruiting graduates trained in specialized 

programs helps to ensure the availability of staff with expert, up to date knowledge. 

Project management 

 The practice of having someone “sign-off” each step once it is completed is recommended. 

 The practice of developing a risk assessment plan is recommended. 

 Having a quality assurance team or group is recommended to suggest, implement and monitor the use of 

quality assurance practices. 

 Establish working groups to study particular issues or challenges. 

 Establish resource centres or centres of expertise to maximize the efficient use of expert knowledge and 

scarce skill sets. 

Systems and processing 
 Automation of steps that are repeated and not requiring complex decision making is very highly 

recommended. 

 A system renewal plan should be in place. The plan for system backups and safeguards should cover five 

years into the future. 

 The use of corporately available services and software is highly recommended to mitigate processing risks 

and relieve dependence on a few individual with specialized skills or knowledge. 

 Protocols for file and variable naming conventions are highly recommended, as well as a version control

strategy for managing data files and software programs. 

 The use of extensive production schedules and checklists are straightforward methods for minimizing the

likelihood of errors during production, and also for detecting errors if they do occur. 

 During a period of system upgrades or changes, parallel runs of production processing are recommended. 

Standardization 

 Standardization of questionnaires, for example to collect financial information, often requires compromise. 

However the benefits make the compromise worthwhile, in terms of reduced response burden, improved 

coherence and consistency in the data, and increased reusability of systems and tools. 

 Use standard methodologies and software products. 

 Use a standard process for making revisions. 

Documentation 

 Up to date, complete and accessible documentation is recognized as essential. 

 Use automated verification of data to be disseminated and checklists to ensure that all steps are executed 

successfully. 

5 Conclusions 

In conclusion, at Statistics Canada the Quality Assurance Reviews are a low cost, high benefit quality initiative. 

They complement the other reporting mechanisms and increase overall awareness and application of quality 

assurance practices. They also provide an opportunity for practical and valuable on-the-job training for the middle 

manager community. 

References 

Reedman, L. and Julien, C. (2010) “Current and Future Applications of the Generic Statistical Business Process 

Model at Statistics Canada”, Q2010 Conference proceedings. 

UNECE Secretariat (2009) “Generic Statistical Business Process Model: Version 4.0 – April 2009”, Joint 

UNECE/Eurostat/OECD Work Session on Statistical Metadata (METIS).