TE
D
IN
PR
EN
H

ED

W

The Food and Environment Research Agency

O
N

TR
O

LL

Protocol for Proficiency Testing Schemes

N

O

T

C

Version 3, January 2012

Part 1 – Common Principles© Crown Copyright 2012

PREFACE
This Protocol is a series of inter-related documents. Part 1, this document, sets out an overview of,and the principles common to, all of the PT schemes provided by the Food and Environment Research
Agency. Subsequent parts give scheme specific details. It follows that neither Part 1, nor any of theother parts, can be used in isolation. Part 1 must always be read in conjunction with a scheme
specific supporting part and vice versa.

VERSION HISTORYThis Protocol was completely revised in 2009, superseding all proficiency testing scheme Protocols
previously published by the Food and Environment Research Agency (previously, Central Science
Laboratory), i.e. all previous editions of the separate FAPAS and FEPAS Protocols.

TE
D

Version 3 of January 2012, this version, supersedes Version 2 of December 2010. The changes are
as follows;PR

1.1 Addition of text and reference taken from LEAP® Reports.

IN

1 Change LEAP™ to LEAP®

3. Expanded references for guidance on the level and frequency of participation

EN

3.3.1 Clarification that homogeneity is performed on the subsamples

H

3.3.1 Release of information pertaining to test material preparation

W

4.1.3 Addition of sMAD

N

O

T

C

O
N

TR
O

LL

ED

4.1.3 Clarification of details presented in Reports and information only z-scores

Page 2 of 18

CONTENTS
1. INTRODUCTION

4

1.1. What is PT?

4

1.2. Accreditation and PT

4

2. ORGANISATION of SCHEMES

5

2.1. Administration

5

2.2. Confidentiality

5

2.3. Typical Timetable

5

2.4. Management System

6

2.5. Subcontractors

6

TE
D

3. PARTICIPATION in SCHEMES
3.1. Enrolment and Fees
3.3. Dispatch and Receipt of Test Materials

EN

3.3.1. Test material preparation and homogeneity testingH

3.4. Analysis of Test Materials

ED

3.5.1. Collusion and Falsification of Results

LL

4. PERFORMANCE ASSESSMENT

TR
O

3.7. Follow-Up Services

7
7
7
8
8
8
8
8
9
9
9

4.1.3. Consensus assigned value

10

4.1.4. Standard deviation for proficiency

11

N

O

4.1.2. Types of scoresTC

4.1.1. Why score?

6

9

O
N

4.1. ScoringW

3.5. Submission of Results and Outline Methodology
3.6. Report Distribution

PRIN

3.2. Agents6

4.2. Interpreting Scores

12

4.2.1. Interpreting z-Scores

12

4.2.2. Interpreting Q-scores

13

4.3. Appeals

13

5. REFERENCES

14

APPENDIX 1 GLOSSARY OF TERMS

15

Page 3 of 18

1. INTRODUCTION
The Food and Environment Research Agency (Fera) is an Executive Agency of the UK Government
Department for Environment, Food and Rural Affairs (Defra). Fera provides a wide range of
proficiency testing (PT) schemes.
The management of these PT schemes is the sole task of one of many teams within Fera. Known
internally at Fera as the Proficiency Testing Group (PTG), this team provides Fera’s PT schemes
globally under the brand name of FAPAS PT.

IN

For the purpose of this Protocol we use FAPAS PT to mean Fera PTG.

TED

FAPAS® is an acronym for Food Analysis Performance Assessment Scheme. The other branded PT
schemes run by PTG are the Food Examination Performance Assessment Scheme (FEPAS®),
Laboratory Environmental Analysis Proficiency scheme (LEAP®), Genetically Modified Materials
Analysis performance scheme (GeMMA) and plant health diagnostics (PhytoPAS). This Protocol,
Part 1, should be read in conjunction with the scheme-specific parts. For FAPAS® in its entirety, seealso Part 2 of the Protocol. For FEPAS®, see also Part 3. For GeMMA scheme, see also Part 4. For
LEAP® scheme, see also Part 5. PhytoPAS is still in its infancy, hence, there is no independent
Protocol part for PhytoPAS yet.

PR

1.1. What is PT?

H

EN

ISO/IEC 17043:2010 [1] defines PT as the evaluation of participant performance against preestablished criteria by means of interlaboratory comparisons.

T

C

O
N

TR
O

LL

ED

W

The demand for independent demonstration of competence, from regulatory bodies and customers,
means that proficiency testing is relevant to all laboratories testing samples for quality and safety.
Hence, it is a requirement of accreditation to ISO/IEC 17025 [2] that the laboratory takes part in a PT
scheme, if a suitable scheme exists. In particular, for laboratories entrusted with the official control
of food and feeds, Article 12 of EU Regulation (EC) 882/2004 [3] requires such laboratories to beassessed and accredited in accordance with ISO/IEC 17025. This is reinforced internationally under
Codex guidelines [4]. PT is an important requirement of the EU Council Directive 98/83/EC [5] on thequality of water intended for human consumption. With the increasing demands for independent
proof of competence from regulatory bodies and customers, proficiency testing is relevant to alllaboratories testing water for quality and safety in every country. Proficiency testing is therefore alegal requirement for these laboratories. Thus, together with the use of validated methods andinternal quality control, proficiency testing is an essential element of laboratory quality assurance.

NO

The analysis of an external quality check sample as part of a laboratory’s routine procedures provides
objective standards for individual laboratories to perform against and permits them to compare theiranalytical results with those from other laboratories. In summary, PT is a way of checking theaccuracy [6] of results from laboratories.

1.2. Accreditation and PT
Accreditation is a completely separate concept to PT. Accreditation requires the formal, external,
assessment of an organisation’s documented procedures against a relevant International Standard.
The relevant conformity standard for laboratories in the field of testing is ISO/IEC 17025:2005.
Compliance with ISO/IEC 17025 alone cannot guarantee that the procedures give accurate results.
Only the external check of a proficiency test can confirm that the results are accurate – hence therequirement within ISO/IEC 17025 for laboratories to take part in PT schemes.
It must be stressed that taking part in a PT scheme does NOT confer accreditation upon a laboratory.
This applies even if the PT provider is, as is FAPAS PT, accredited for the provision of PT schemes.

Page 4 of 18

2. ORGANISATION OF SCHEMES
2.1. Administration
All PT schemes provided by FAPAS PT are administered in keeping with internationally agreed
principles, in particular those set out within the IUPAC International Harmonized Protocol for theProficiency Testing of Analytical Chemistry Laboratories [7]. The original (1994) version of thisInternational Harmonized Protocol was derived from the entire first (1991) FAPAS® Protocol while therecent revision (2006) drew heavily upon the experience of FAPAS PT in delivering PTs in theintervening years.

TE
D

Each PT scheme has its own Advisory Committee, which meets at least annually. The Advisory
Committees comment upon the relevant programme of PTs planned by FAPAS PT for the forthcoming
year and discuss any scientific issues arising from PTs conducted in the current year. Committee
members are available to advise FAPAS PT staff at any point during the year and group email
correspondence is frequently used to facilitate discussions. A list of current Advisory Committee
members and the terms of reference are available on request from FAPAS PT.

W

H

EN

PR

IN

The day to day running of an individual PT is the responsibility of a designated member of staff, the‘Round Co-ordinator’. Ultimate responsibility for all FAPAS PTs lies with the Head of Group. Expert
advice to support all staff in these duties is readily available from within Fera and from a variety of
external sources. External advisors are selected on the basis of their personal expertise and not theiraffiliation; they need not be members of the relevant Advisory Committee. When consulting experts,
FAPAS PT will not disclose any participant information, purely scientific information will be exchanged,
see below.

ED

2.2. Confidentiality

O
N

TR
O

LL

All information held by FAPAS PT about participants, including their z-scores, is confidential and will
not be disclosed to anyone unless explicitly agreed by the participant for a particular purpose. To
preserve this confidentiality participants receive reports giving all the results for that PT but without
identifying individual laboratories. The laboratory code numbers used in reports are assigned in orderof receipt of results from participants. Participants will be assigned the same code number indifferent PTs only by chance.

N

O

T

C

To avoid any conflict of interest / breach of confidentiality, if any of the various analytical testingteams elsewhere within Fera wished to participate in a PT they will be treated in exactly the samemanner as any other participant. They will not have access to details of any other participants.
Likewise, when FAPAS PT seeks expert advice from other parts of Fera (or indeed any externalsource) it will not disclose any information that would breach participant confidentiality.
Once reports are issued to participants they are regarded as being in the public domain but all PT
reports issued by FAPAS PT are UK Crown Copyright, which cannot be assigned to other publishers.
Anyone wishing to use data from within FAPAS PT reports for their own publications should first seek
permission from FAPAS PT. It should be noted that this request for respect of copyright cannot
preclude publications exploiting FAPAS PT data being distributed without the prior knowledge orapproval of FAPAS PT.

2.3. Typical Timetable
FAPAS PT provides on-going PT schemes, where test materials are distributed on a regular basis
every year. FAPAS PT also provides bespoke ‘closed’ PT schemes, where the test materials are
distributed at the time and request of a commissioning client.
For ease of planning and timetabling FAPAS PT advertise the on-going schemes in annual blocks,
from 1 April to 31 March the following year. These annual programmes of proficiency tests arecompiled by FAPAS PT in conjunction with the Advisory Committee for each Scheme. They are
Page 5 of 18

generally published in December in anticipation of the following April-March.
formats are published, the UK convention of DD/MM/YY is employed.

Where short date

The outline process of conducting a single proficiency test is as follows:a) Preparation of test materials, including homogeneity testing.
b) Dispatch of test materials on advertised date from FAPAS PT, York, UK.
c) Participants analyse test materials and report results by a given date. Generally the closing
date is six to eight weeks from the dispatch date, though for certain analyses where theanalyte/matrix combination potentially are unstable a much shorter time scale may be set.
d) Results subjected to statistical analysis by FAPAS PT.
e) Distribution of final report to all participants. Generally the report is issued within a month of
the PT closing date but FAPAS PT reserves the right to extend this period in cases where thestatistical evaluation proves to be atypical.
Participants will be kept informed by email if a delay arises at any of these stages.

TED

2.4. Management System

EN

PR

IN

The quality management system for the whole of Fera is certified to ISO 9001 [8]. In addition, themajority of the work of FAPAS PT is accredited by UKAS. The formal accreditation certificate isavailable on the FAPAS PT web site [9] (Adobe PDF format), while the current formal scheduledetailing the scope of this accreditation can be obtained from the United Kingdom Accreditation
Service (UKAS) web site (Adobe PDF format) [10].

ED

W

H

During 2010, the full standard ISO/IEC 17043:2010 was published. This supersedes the ISO/IEC 431:1997 guide [11]. Until the publication of ISO/IEC 17043:2010, the United Kingdom Accreditation
Service (UKAS) conferred accreditation on PT Providers in accordance with ISO Guide 43 throughassessment against ILAC G13: 2007 [12].

ON

2.5. Subcontractors

TR
O

LL

The scheme specific supporting parts to this document include the accreditation status of each PT
scheme.

N

O

T

C

FAPAS PT does not have any laboratory facilities of its own. Test material preparation andhomogeneity testing is carried out by subcontractors. Homogeneity testing may be carried out by adifferent laboratory to the one that prepares the test material. FAPAS PT maintains a list of approved
subcontracting laboratories and regularly reviews the service received. Where possible, FAPAS PT will
only use subcontracting laboratories who hold accreditation to recognised international standards(ISO/IEC 17025 [2], for example). Subcontracting laboratories may also participate in FAPAS PTs. In
this situation, the subcontracting laboratory participation will be treated in exactly the same way as all
the other participants, and the same rules of confidentiality will apply.

3. PARTICIPATION IN SCHEMES
None of the FAPAS PT schemes stipulate a minimum level and frequency of participation. Advice on
the level and frequency of participation in PT schemes may be obtained from other sources, [13, 14]for example. Participants do not necessarily have to analyse for all the analytes in a test.

3.1. Enrolment and Fees
The programmes for FAPAS PTs are available on the web site, www.fapas.com. Customers place
their orders on-line by browsing these programmes and compiling a ‘wish list’. If the customer is aprevious participant and has access to the secure pages of our web site they can convert their ‘wish
Page 6 of 18

list’ into a formal order on-line. New customers can use the ‘wish list’ to request a quote.
Alternatively, PDF files of the programmes are available from FAPAS PT, at the address shown on the
final page of this document.
PT order confirmations are automatically emailed to customers on completion of the ordering process.
The confirmation email contains a link to a printer friendly version of the order, held within thecustomer’s secure pages on our web site. It is the responsibility of the customer to check that FAPAS
PT has processed their requests correctly, i.e. that they are enrolled in the correct PTs.
Details of all fees are available on request. FAPAS PT reserves the right to withhold test materials
and/or PT reports from participants if payment is delayed.
Formal Fera Standard Terms and Conditions for Proficiency Testing Schemes are available, either
from the address given at the end of this document or from our web site (PDF format) [15].

3.2. Agents

PR

IN

TE
D

Agents are appointed by FAPAS PT in some countries. The advantages to participants of using theagent are to register locally to participate in FAPAS PTs and the facility to pay in local currency.
Agents will also liaise with FAPAS PT on the participant’s behalf for any queries or problems. Agents
may also be able to help samples pass more easily through customs. Details of participants’performance in the PTs are not disclosed to the agents. The list of agents is available from thewebsite, www.fapas.com.

EN

3.3. Dispatch and Receipt of Test Materials

ED

W

H

All test materials are distributed with a generic compliments slip. The compliments slip provides
details on how to access instructions from our website about reporting of results and method details.
Instructions specific to the PT with regard to storage on receipt, type of analysis required, etc. will be
included in these instructions.

TR
O

LL

It is the responsibility of participants to read these instructions and follow them. FAPAS PT cannot beheld responsible for any problems arising from failure to comply with these directions.

ON

It is the responsibility of the participant to contact FAPAS PT if they have not received the test
material within agreed timescales, as set out in the Fera Standard Terms and Conditions for
Proficiency Testing Schemes.

O

T

C

Delays to the dispatch of test materials occasionally arise. If the dispatch of a test material has to bedelayed for any reason, then participants will be notified of this fact by email prior to the advertised
dispatch date. FAPAS PT cannot be held responsible if participants overlook this notice of delay.

N

3.3.1. Test material preparation and homogeneity testing
The determinands in test materials may either be at natural levels, incurred or spiked at a particular
requested formulation level. Details of test material preparation are retained by FAPAS PT but not
published in PT Reports, except where pertinent to the statistical analysis of the results.
Test materials in FAPAS PTs will not be distributed until testing demonstrates that the individual
subsamples are of sufficient homogeneity. FAPAS PT uses the statistical procedure developed by
Fearn and Thompson [16]. Details of the homogeneity testing data are retained by FAPAS PT but not
published in PT Reports.
Participants may contact FAPAS PT to request details of test material preparation and homogeneity
testing. Such details may be released on request, except where this compromises data which iscommercial in confidence or where such knowledge is scientifically invalid in the interpretation ofassessments.

Page 7 of 18

3.4. Analysis of Test Materials
If the PT is to yield maximum benefit as an external check on the routine working of participants’methods then the sample should not be given any special treatment. Hence, participants are free to
use whatever method of analysis they wish. On the occasions where the method is known to be
empirical (i.e. the result is dependent on technique) participants are still free to use whatever method
they wish. In order to obtain a comparable set of results for statistical assessment, however, FAPAS
PT may advise participants that only the results submitted for a given method will be used to derive
an assigned value by consensus.

3.5. Submission of Results and Outline Methodology
The reporting of results within the requested time scale and in the specified units is part of theperformance assessment.

PR

IN

TE
D

Participants are requested to submit their results and methods via the secure pages on our web site.
Each participant will confidentially be provided with a unique UserID and Password required to accessthese pages. While the submission of a result is a prerequisite for a performance assessment,
participants are not obliged to submit their methodology. However, where an assigned value derived
by consensus is dependent on a particular aspect of methodology, some specific questions may berequired with the result submission.

H

EN

Acceptance, or otherwise, of results submitted after the closing date is at the discretion of the Round
Co-ordinator.
Where extenuating circumstances have prevented timely results submission,
participants should contact FAPAS PT to discuss acceptance of late results.

W

3.5.1. Collusion and Falsification of Results

TR
O

LL

ED

Collusion, either between participants or between individual participants and the scheme provider, iscontrary to professional scientific conduct. It serves only to nullify the benefits of proficiency testing tocustomers, accreditation bodies, and analysts alike.
Collusion is, therefore, to be strongly
discouraged.

CON

As a preventive measure FAPAS reserves the right to distribute more than one test material within a
PT so that participants cannot compare results directly. Ultimately, though, it is the responsibility ofthe participating laboratories to avoid collusion or falsification of results. Laboratories found to befalsifying results may be refused participation in subsequent proficiency tests.

OT

3.6. Report Distribution

N

Participants are advised in the PT instructions when to expect the publication of the report. FAPAS PT
aims to do this as soon as is practical after the closing date of the PT. Participants should note that
our quality procedures involve extensive cross-checking and scrutiny by several FAPAS PT staff under
the guidance of the Round Co-ordinator. There is no fixed way of generating an assigned value.
Consequently this means the process takes anywhere between 3-8 weeks depending on thecomplexity of the data.
All reports are distributed in Adobe PDF format. They are both password secured and digitally signed
to ensure that they cannot be altered in any way. The digital signature automatically validates when
the PDF file is opened using Adobe Reader v7 or higher on a PC with access to the Internet. Reports
are only available for download to the named contact(s) for the PT in question.

3.7. Follow-Up Services
If a participant wishes to obtain advice on any aspect of their performance they should contact
FAPAS PT by email in the first instance. Participants must note that FAPAS PT is most likely to offer
assistance in the form of a broker service whereby FAPAS PT will either anonymously or subsequent

Page 8 of 18

to all parties agreeing to waive their confidentiality, pass on the participant’s inquiry to an expert
laboratory / external advisor.
Surplus test materials from the batch used for the PT may be available for purchase. These samples
are not Certified Reference Materials. However, reference materials for the food analysis sector are
not numerous and surplus FAPAS PT test materials may be the only source of a suitable quality
control material.
Outline details on the availability or otherwise of such quality control materials are given in therelevant report. The exact stock level of any given quality control material can be checked via our
web site.

4. PERFORMANCE ASSESSMENTTE
D

The statistical model used by FAPAS PT is set out fully within the International Harmonized Protocol[7]. In summary, as indicated in the Introduction, the purpose of a FAPAS PT is to check theaccuracy of results submitted by the participating laboratories. This check is achieved typically bycomparing participants’ results to some estimate of the ‘true’ value.

IN

If the results submitted are quantitative then this comparison will be in the form of a numericalscore. Semi-quantitative (< or >) data are not assessed, except where detailed in the relevant
scheme specific supporting parts of this Protocol.

PR

The comparison for qualitative results will be against the answer anticipated by formulation or bytaking account of the consensus of participants’ results.

W

H

EN

The results submitted to a single PT represent the final product in a complex string of actions carriedout by the participants, from sample receipt to results reporting. As such they encompass all aspectsof a laboratory’s performance. A mistake, however trivial, at any stage will contribute to the final
outcome.

LL

ED

It is unwise to view any performance assessment as anything other than a snapshot of the wholelaboratory performance at the time of the PT.

TRO

4.1. ScoringON

4.1.1. Why score?

C

The advantages of expressing participants’ results as a standardised score are that:they are simple and transparent,•

they present participants’ results in a readily understood form,•

they permit comparison over time,•

when tabulated and charted, they place individual performancein the overall context of the PT.

N

O

T

•

When the standardised score incorporates a prescribed value that represents limits of acceptablevariation for the analysis in question then the score embodies the concept of fitness-for-purpose, i.e.
the balance between expending considerable time and effort (= expense) on an analysis to get ahighly accurate result vs. carrying out a rough and ready procedure that only provides an indicationof the level present and so be of limited use/require further analysis.

4.1.2. Types of scores
A variety of standardised scores is available. This Protocol presents only two such scores but thisdoes not preclude FAPAS PT from adopting alternatives, if so advised by our statistical experts.

4.1.2.1. z-Scores
Page 9 of 18

FAPAS PT favours the use of z-scores because when the standard deviation is based on a fit-forpurpose criterion, i.e. it is a prescribed ‘standard deviation for proficiency’, then the significance of theperformance assessment is immediately apparent, no matter what the concentration or identity of theanalyte, the nature of the test material or the physical principle underlying the analytical measurement.
By assessing a participant’s performance by way of a z-score, both the trueness and the precision of
their result are addressed. Use of an objective, fit-for-purpose standard deviation for proficiency
requires the measurement uncertainty of a participant’s result to be in keeping with this level.
A z-score combines an estimate of the error of a result with a standard deviation:z=( x − xa )

σp

where

x = the result reported by the participant
xa = the assigned value
σp = the standard deviation for proficiency

and

IN

TE
D

The derivation of the assigned value and the choice of fit-for-purpose standard deviations forproficiency are more complex (see sections 4.1.3 and 4.1.4 below). The report for each PT will givefull details on the choice and calculation of both the assigned value and the standard deviation for
proficiency assessment.

PR

4.1.2.2. Q-scoresW( x − xa )
xa

ED

Q=

H

EN

On very rare occasions, where FAPAS PT is unable to set an appropriate standard deviation for
proficiency assessment, it may be appropriate to calculate a ‘Q-score’:x = the result reported by the laboratory
xa = the best estimate of the ‘true’ value

LL

where

TR
O

This type of score only indicates the relative error of a result.

ON

4.1.3. Consensus assigned value

TC

In all FAPAS PTs, the ‘assigned value’, xa , is the best estimate available to FAPAS PT of the ‘true’
value. The assigned value can be set as a:consensus value•

formulation level•

certified reference value

N

O

•

Suitable algorithms for the derivation of a consensus value are readily available [17, 18, 19]. A
consensus value is almost invariably taken by FAPAS as the assigned value. The procedure used toderive the consensus will involve, as a minimum:•

removing invalid data, i.e., results reported as approximately 10, 100 or 1000x greater orsmaller than the majority of submitted results (considered to be reporting errors).

•considering the symmetry, or otherwise, of the distribution of results.

•

where the results form a roughly symmetric distribution (outliers aside), minimising theinfluence of outliers by the use of a robust statistical procedure to derive the mean [17].

•where there is a degree of asymmetry, scrutinising the results with a procedure that
estimates the mode or, in some instances, helps to identify multimodality (by a procedureknown as ‘bump-hunting’ [18]).

Page 10 of 18

•comparing the robust mean, median and mode(s). The median or mode may be used as theconsensus if FAPAS PT considers that sufficient supporting evidence is available to justify
such action.

Additional procedures may be adopted for particular PTs when results have to be submitted withsupporting information, for example, on recovery correction. This will be detailed in each specific PT
report.
An estimate of the uncertainty of the consensus is also required. For n results, the uncertainty u of arobust mean is taken as its standard error,u = σ̂

nTE
D

where σ̂ is the robust standard deviation of the results. For a mode, the standard error is calculated
directly by the bootstrap method [18]. For a median, the standard error is taken as the medianabsolute deviation (sMAD). Where the test of u/σp is equal to or greater than the critical value of 0.4,
the effect of the uncertainty on z-scores will be taken into consideration when issuing z-scores.
Where the uncertainty is too high, z-scores may be issued for information only and should not be
used by participants as fully evaluative of performance.

IN

The statistics for the derivation of the assigned value will be summarised in each PT report. Reports
will detail any complications in the derivations, as necessary.

PR

4.1.4. Standard deviation for proficiency

H

EN

The standard deviation for proficiency (informally, the ‘target sd’, σp) determines the limits ofsatisfactory performance in a PT. It is set at a value that reflects fitness-for-purpose for the analysis
in question. Fit-for-purpose standard deviations for proficiency can be obtained from:predictive models, e.g. modified Horwitz Equation [20]•collaborative trials / method performance studies•

legal definition•

expert opinion.

TR
O

LL

ED

W

•

T

C

O
N

The Horwitz function, describing the trend of standard deviation of reproducibility found incollaborative trials, represents fitness-for-purpose in the food sector over a wide range ofconcentrations. It is therefore used by FAPAS PT in the majority of instances. In some ranges,
however, a more appropriate precision is required and, in those instances, statistics from relevant
collaborative trials or other sources are used to derive the standard deviation.

NO

The appropriate form of the modified Horwitz equation [20] used by FAPAS PT requires the analyte
concentration c to be expressed as a mass fraction, e.g., 10-6 ≡ 1 ppm ≡ 1 mg kg-1, or 10-2 ≡ 1%. It
specifies the following;
For analyte concentrations less than 1.2x10-7 (120 ppb),

σp = 0.22c
For analyte concentrations between 1.2x10-7 (120 ppb) and 0.138 (13.8%),

σp = 0.02c 0.8495
For analyte concentrations greater than 0.138 (13.8%),

σp = 0.01c 0.5
FAPAS PT uses the assigned value xa as the concentration in these equations. The raw σp values are
mass fractions and have to be converted to the required units before use in calculating z-scores. Thisis easily achieved by dividing the result by the mass fraction appropriate to the units used.
Example, sodium in canned meat meal. The robust mean is 0.27 g/100g, where the mass fraction is10-2. Hence;

Page 11 of 18

σp = 0.02 x (0.27x10-2)0.8495 = 0.00013
Convert to g/100g units,

σp = 0.00013 / 10-2 = 0.013 g/100g
When collaborative trial statistics are used to determine σp , the value at the relevant concentration is
obtained by interpolation, using an appropriate model, usually the assumption of a constant relative
standard deviation. The function usually applied to derive σp is;

σp =

RSDR
×c
100

where RSDR is the Relative Standard Deviation of Reproducibility from collaborative trial,
expressed as %and c is concentration, i.e., the assigned value.

TED

4.2. Interpreting ScoresIN

4.2.1. Interpreting z-Scores

LL

ED

W

H

EN

PR

The guiding principle of scoring in FAPAS PT is fitness-for-purpose. This means that the standard ofaccuracy required is based on an uncertainty that is independently determined to be appropriate forthe analysis in question.
A hypothetical laboratory performing exactly according to thispredetermined standard will obtain z-scores like a random selection from a normal distribution.
However, most laboratories will use methods with both a bias and a repeatability standard deviationthat differs from the fitness-for-purpose uncertainty. Accordingly, the collected z-scores from a
FAPAS PT often deviate from the normal distribution. The deviation may take the form of heavy tails
and outliers and, occasionally, asymmetry or multimodality. Because the scoring is based on an
independently-prescribed uncertainty, it is logical to interpret z-scores on the basis of the normal
distribution.

N

O

T

C

O
N

TR
O

The properties of a normal distribution are such that, over time, about 95% of observations lie
between ±2 standard deviations. Performance in a FAPAS PT PT, therefore, is considered fit-forpurpose if a z-score lies within the range ±2. It follows that an exactly-conforming participant’s zscores will fall outside this range with a probability of 1 in 20. Occasional scores in the range
2 < |z| < 3 may therefore be of no importance. Such z-scores require consideration and appropriate
action, in the context of the other scores obtained by that laboratory. However, the probability of aconforming participant’s z-score falling outside |z| >3 is less than about 1 in 300. Given this rarity,
such scores therefore represent results that are probably not fit-for-purpose and should be used totrigger investigation and remedial action.
The consideration of a set or sequence of z-scores over time provides more useful information than asingle z-score. Examples of suitable methods of comparison are provided in the International
Harmonized Protocol [7].
NB. In the past, terms such as ‘satisfactory’, ‘questionable’ and ‘unsatisfactory’ have been applied toz-scores within certain ranges. This approach categorises the z-score when it is not appropriate to doso and is likely to be misleading. The limits z = ±2, z = ±3, must not be regarded as strict
boundaries but should be treated as action limits. z-Scores are statistics and MUST be interpreted assuch [21].
A note on homogeneity and z-scores: The requirement of distribution units to be ‘sufficiently
homogeneous’ means that any variation detected between the units by the homogeneity test should
be of negligible magnitude in relation to fitness-for-purpose and thus too small to affect z-scores.
FAPAS PT therefore takes no account of between-unit uncertainty in its scoring.

Page 12 of 18

4.2.2. Interpreting Q-scoresThis method of scoring has the disadvantage that the significance of any result is not immediately
apparent. Q-scores cannot and must not be interpreted in the same way as z-scores.
Q-scores are not based on fitness-for-purpose. They are only an indication of the bias of each result
relative to all of the submitted results.
FAPAS PT will only ever issue such scores on rare occasions and then only as indicative measures andnot to be used for performance evaluation purposes.

4.3. Appeals

N

O

T

C

O
N

TR
O

LL

ED

W

H

EN

PR

IN

TE
D

FAPAS PT undertakes to correct any mistakes attributable to errors on its part promptly andsympathetically. If a participant has any concerns about any aspect of the PT they should contact
FAPAS PT by email in the first instance. An investigation will be conducted in accordance with ourmanagement system and the participant advised of the outcome.

Page 13 of 18

5. REFERENCES
ISO/IEC 17043:2010, Conformity assessment – General requirements for proficiency testing.

2

ISO/IEC 17025:2005, General requirements for the competence of testing and calibrationlaboratories.

3

Regulation (EC) 882/2004 of the European Parliament and of the Council of 29 April 2004 on
official controls performed to ensure the verification of compliance with feed and food law,animal health and animal welfare rules, Official Journal L165, 30/04/2004, 0001-0141.

4

CAC/GL 27-1997, Guidelines for the assessment of the competence of testing laboratories
involved in the import and export control of food, Codex 2006.

5

Council Directive 98/83/EC of 3 November 1998 on the quality of water intended for humanconsumption, Official Journal of the European Communities L 330/32, 05/12/1998.

6

AMC Technical Brief 13, 2003, Terminology – the key to understanding analytical science. Part 1:
Accuracy, precision and uncertainty.

7

Thompson, M., Ellison, S.L.R. and Wood, R., 2006, The International Harmonised Protocol for
the Proficiency Testing of Analytical Chemistry Laboratories, Pure Appl. Chem., 78 (1), 145–196.

8

http://www.fera.defra.gov.uk/aboutUs/commitmentToQuality.cfm

9

http://www.fapas.com/pdfpub/certificate.pdf

10

http://www.ukas.com/about-accreditation/accredited-bodies/proficiency-testing-organisations.asp

11

ISO/IEC Guide 43-1:1997, Proficiency testing by interlaboratory
1:Development and operation of proficiency testing schemes.

12

ILAC G13:08/2007, Guidelines for the Requirements for the Competence of Providers ofProficiency Testing Schemes.

13

EA-4/18: 2010, Guidance on the level and frequency of proficiency testing participation.

14

Selection, Use and Interpretation of Proficiency Testing (PT) Schemes, Eurachem, 2011, second
Edition15

http://www.fapas.com/terms.cfm

16

Fearn, T. and Thompson, M., 2001, A new test for sufficient homogeneity, Analyst, 126, 14141417.

17

Analytical Methods Committee, 1989, Robust statistics - how not to reject outliers: Part 1 Basicconcepts, Analyst, 114, 1693-1697.

18

Lowthian, P.J. and Thompson, M., 2002, Bump-hunting for the proficiency tester-searching for
multimodality, Analyst, 127, 1359-1364.

19

ISO 13528: 2005, Statistical Methods for use in Proficiency Testing by Inter-Laboratory
Comparisons.

20

Thompson, M., 2000, Recent trends in inter-laboratory precision at ppb and sub-ppb
concentrations in relation to fitness for purpose criteria in proficiency testing, Analyst, 125, 385386.

21

AMC Technical Brief 11, 2002, Understanding and acting on scores obtained in proficiency testing
schemes.

comparisons

–

Part

N

O

T

C

O
N

TR
O

LL

ED

W

H

EN

PR

IN

TE
D

1

Page 14 of 18

APPENDIX 1 GLOSSARY OF TERMS
This glossary includes terms not specifically mentioned in this Protocol but which may be used in the
supporting parts of the protocol or PT report. Participants may find this glossary useful in relation to
proficiency testing in general.
Accuracy
The closeness of agreement between a test result and the accepted reference value.
NOTE.

The term “accuracy”, when applied to a set of test results, describes a combination of
random components and a common systematic error or bias component.

Assigned value
The value to be used as the “true” value by FAPAS PT in the statistical treatment of results. It is thebest available estimate of the true value of the analyte in the matrix.

TE
D

Bias

The difference between the expectation of the test results and an accepted reference value.

IN

Bias is due to systematic error, not random error. There may be one or more systematic
error components contributing to the bias. A larger systematic difference from theaccepted reference value is reflected by a larger bias value.

PR

NOTE.

EN

Bias of the measurement method

An example of this is where a method purporting to measure the sulfur content of acompound consistently fails to extract all the sulfur, giving a negative bias to themeasurement method. The bias of the measurement method is measured by thedisplacement of the average of results from a large number of different laboratories allusing the same method. The bias of a measurement method may be different at different
analyte concentrations.

TR
O

LL

ED

NOTE.

W

H

The difference between the expectation of test results obtained from all laboratories using thatmethod and an accepted reference value.

O
N

Certified Reference Material (CRM)

N

O

Consensus value

T

C

A reference material, one or more of whose property values are certified by a technically valid
procedure, accompanied by or traceable to a certificate or other documentation which is issued by acertifying body.
The assigned value, as generated from valid participant’s results. Some participants’ results may be
excluded from the consensus calculation where they fail to meet specific criteria. The consensus
value may be the robust mean, median or mode.
Distribution unit
One sample of the test material which is sent to a participant.
Error
The difference between a reported result and the assigned value.
FAPAS PT
Organisation providing the FAPAS®, FEPAS®, GeMMA, LEAP®, and PhytoPAS and specialised
proficiency testing schemes.
FAPAS®
Food Analysis Performance Assessment Scheme

Page 15 of 18

FEPAS®
Food Examination Performance Assessment Scheme
Fera
The Food and Environment Research Agency
Fitness for Purpose
The precision and accuracy of analytical data must be sufficient to enable the end-user of the data to
make sound decisions as to whether the results/samples analysed are fit for the intended purpose.
GeMMA Scheme
Genetically Modified Materials Analysis performance scheme
Interlaboratory test comparisons
Organisation, performance and evaluation of tests on the same or similar items or materials by two or
more different laboratories in accordance with pre-determined conditions.

TE
DInternal Quality Control (IQC)

PR

IN

The set of procedures undertaken by the laboratory staff for continuous monitoring of operations andresults in order to decide whether the results are reliable enough to be released; IQC primarily
monitors the batch-wise trueness of results on quality control materials, and precision on replicate
analysis of test materials.

EN

Laboratory bias

W

H

The difference between the expectation of the test results from a particular laboratory and an
accepted reference value.
Laboratory component of bias

LL

(1) The laboratory component of bias is specific to a given laboratory and the conditions ofmeasurement within the laboratory, and it may be different at different analyte
concentrations.

TRO

NOTES.

ED

The difference between the laboratory bias and the bias of the measurement method.

ON(2) The laboratory component of bias is relative to the overall average result, not the trueor reference value.

C

LEAP® SchemeO
N

PhytoPAS

T

Laboratory Environmental Analysis Proficiency scheme
Plant health Performance Assessment Scheme
Precision
The closeness of agreement between independent test results obtained under prescribed conditions.
NOTES.

(1) Precision depends only on the distribution of random errors and does not relate to theaccepted reference value.
(2) The measure of precision is usually expressed in terms of imprecision and computed asa standard deviation of the test results. Higher imprecision is reflected by a larger
standard deviation.
(3) “Independent test results” are defined as results obtained in a manner not influenced
by any previous result for the same or similar material.

Proficiency Testing Scheme (Performance Assessment Scheme)
The system for objectively checking laboratory results by means of an external agency (e.g.
FAPAS PT). It includes comparison of a laboratory's results at intervals with those of other

Page 16 of 18

laboratories, the main object being the establishment of trueness. Proficiency testing is designed toassess the accuracy of a laboratory's results. Proficiency testing is sometimes referred to as "external
quality assessment" (EQA).
QC materials
Surplus test materials from the batch used for a PT. Useful for internal quality control (QC) in alaboratory but these are not CRMs.
Quality Assurance System/Programme (QAS)
The sum total of a laboratory's activities aimed at achieving the required standard of analysis. While
IQC and proficiency testing are very important components of a quality assurance programme it mustalso include staff training, administrative procedures, management structure, auditing, etc.
Accreditation bodies judge laboratories on the basis of their quality assurance programme plus peer
review of technical competence for a specific technical capability.
Reference Material (RM)

σ
x

×100

EN

RSD =

PR

The standard deviation expressed as a percentage of the mean:IN

Relative Standard Deviation (RSD) / (Coefficient of Variance)TED

A material or substance one or more properties of which are sufficiently homogeneous and wellestablished to be used for the calibration of an apparatus, the assessment of a measurement method,or for assigning values to other materials.

W

H

where σ is the standard deviation and x is the arithmetic meanED

Robust mean

LL

The mean of results calculated by a robust statistical method, for example Huber’s H15 algorithm asused by FAPAS PT.

TROStandard deviation (for proficiency, target sd)

ON

A numerical value for the standard deviation of a measurement result, which has been designated asa goal for measurement quality.
Test materialO

Test method

TC

The matrix/analyte combination to be tested that is distributed to participants in the proficiency test.

N

A defined technical procedure to determine one or more specified characteristics of a material or
product.
Testing laboratory
A laboratory that measures, examines, tests, calibrates or otherwise determines the characteristics or
performance of materials or products.
True value
The actual concentration of the analyte in the matrix. Very often, the true value is unknown.
Trueness
The closeness of agreement between the average value obtained from a large series of test results
and an accepted reference value.
NOTE.

The measure of trueness is usually expressed in terms of bias.

Page 17 of 18

TE
D
IN
PR
EN
H
W
ED
LL
TR
O
O
N
C
T

N

O

FAPAS®
Proficiency Testing GroupThe Food and Environment Research Agency
Sand Hutton, YorkYO41 1LZ
UK
Tel:
Fax:
e-mail:web:+44 (0)1904 462100+44 (0)1904 462040
info@fapas.com
www.fapas.com

Page 18 of 18