Research Notes
Issue 39/February 2010
A quarterly publication reporting on research, test development and validation
Cambridge ESOL Editors
Dr Sacha DeVelle
Dr Hanan Khalifa
Dr Fiona Barker
Editorial Board
Dr Nick Saville, Director Research and Validation, Cambridge ESOL
Roger Johnson, Director Assessment and Operations, Cambridge ESOL
Dittany Rose, Quality and Evaluation Manager, Cambridge ESOL
Martin Nuttall, European Projects Officer, ALTE Secretariat
Production Team
Caroline Warren, Research Support Administrator
Sally Downes, Marketing Services Manager
George Hammond Design

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0|1

Research Notes
Contents
Editorial Notes

1

Setting the standard: Quality management for language test providers : Dittany Rose

2

Implementing an internal audit process: A case study from Cambridge ESOL : Nick Beresford-Knox& Dittany Rose

8

Using conversation analysis to investigate the interaction of QMS audits : Glyn Hughes

10

Working with examination centres to encourage continuous high standards: The centre
inspections programme : Claire McCauley & David Collett

20

Auditing the quality proﬁle: From code of practice to standards : Nick Saville24

Training auditors: The ALTE system : Francesca Parizzi

28

Auditing Cambridge ESOL’s Main Suite and BEC examinations : Sibylle Bolton

31International accreditation of quality in language learning and teaching : Maria Matheidesz

33

IELTS Masters Award winner 200939

37th ALTE meeting and conference, Maynooth, Ireland

40

Editorial Notes
Welcome to issue 39 of Research Notes, our quarterly publication reporting on matters relating toresearch, test development and validation.
The theme of this issue is quality assurance and its impact on language assessment, language
and teaching. We open with an article by Cambridge ESOL’s Quality and Evaluation Manager
Dittany Rose who evaluates the history of quality management to further understand the role ofquality assurance within assessment generally, and Cambridge ESOL in particular. Our next article,
authored by Nick Beresford-Knox and Dittany Rose, focuses on the implementation of a Cambridge
ESOL internal audit. This case study approach describes the internal audit system process using
examples taken from three distinct audit reports. Glyn Hughes then presents an innovative
application of conversation analysis to the spoken interaction of QMS (quality management
system) audits. His article describes the relationship between auditor and auditee via
conversational patterns taken from three internal audits and discusses the practical implications
for quality managers. This is followed by Claire McCauley and David Collett who assess theCambridge ESOL centre inspections programme that ensures all examination centres are regularly
monitored and receive feedback on performance.
The next three articles examine the auditing process within the context of the Association of
Language Testers in Europe (ALTE). ALTE membership involves 32 member organisations
representing 27 European languages, and as such requires a (QMS) capable of meeting therequirements of such membership. Nick Saville provides an overview of auditing procedures with aparticular focus on the outcomes from audits implemented in 2007 and 2008. Next Francesca
Parizzi discusses the training of auditors within the ALTE auditing system. Her work focuses on thenecessary competencies and knowledge needed for both new and experienced auditors throughout
the audit cycle. This is followed by Sibylle Bolton who describes the audit process for Cambridge
ESOL Main Suite and Business English exams using the ALTE Code of Practice Procedures for
Auditing. The ﬁnal article for this issue focuses on language learning and teaching. Maria
Matheidesz describes the development of a new manual for the EAQUALS (European Association for
Quality Language Services) inspection scheme and includes the organisation’s charters, inspection
and assessment methods to explain the basic principles behind EAQUALS accreditation.
We ﬁnish this issue with the winner of the IELTS Masters Award for 2009 and a conference reportfrom the 37th ALTE meeting and conference held in Ireland from 11–13 November 2009.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

2|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Setting the standard: Quality management for
language test providers
DITTANY ROSE RESEARCH AND VALIDATION GROUP, CAMBRIDGE ESOLIntroduction
Test providers assess, and in turn are assessed by theircustomers. An exam board assesses candidates’ level oflanguage ability and they in turn assess whether the tests
will help them to achieve their goals. In a true market,
customers have a choice and can take the test they feel isof the greatest quality. However, in many language testing
situations, for example with less widely taught languages,
there is only one possible exam for candidates to take. Thismeans that the emphasis is squarely on the test providers
themselves to make sure quality is upheld. On the other
hand, when competition exists between test providers,candidates must be able to distinguish between the quality
proﬁles of different tests and the organisations that
produce them.
Cambridge ESOL has a long history of providing language
tests which are reliable, relevant and with a positive
educational impact. We are also one of the few examboards to be certiﬁed to the international quality
management standard ISO 9001. Our commitment toquality is further evidenced by our founding membership of
the Association of Language Testers in Europe (ALTE). In thisarticle, we explain how certiﬁcation to ISO 9001 helps us tobe conﬁdent about our quality proﬁle, and how thiscomplements our long-standing commitment to test
validity.
A distinction is often made between quality assurance(i.e. focus on quality of the product) and quality
management1 (i.e. a wider focus on process) and this article
looks at the history of quality management to see howthese two strands have developed. It then discusses somestandards that test providers can be judged against or canself-assess against. There is also an attempt to deﬁne what
‘quality’ means in regard to assessment and to describe
how quality management can help to ensure this isachieved and that processes are continually improved.

A history of quality management
In the last 50 years or so there has been a plethora ofquality management (QM) approaches, standards andawards, including total quality management (TQM), ISO
9001, European Foundation for Quality Management(EFQM) Excellence model, Zero Defect approach, Statistical
Process Control, Price of Non-conformance (PONC), Lean
Manufacturing and Six Sigma. These names are bafﬂing to

1 In fact deﬁnitions in different sources are often blurred. Those in ISO 9000:2005
Quality Management Systems – fundamentals and vocabulary, are clear but
distinctions are ﬁne between ‘quality management’, ‘quality control’ and ‘quality
assurance’.

many people, who may also have heard stories of quality
management failures, such as the company that thinks thatusing the word ‘quality’ in its title will fool customers into
believing it embodies the said attribute. There are also
stories of organisations where, at some point in the past,
there has been a ‘big push’ for one of the above quality
management approaches. Often, this ‘push’ is largely
forgotten and unsupported by top management and staff
are left with empty phrases and rhetoric but no real
beneﬁts. A further issue is the over-emphasis on standards
and certiﬁcation at the expense of quality and improvement
in a wider sense. By their very nature standards can create
standardisation, which can lead to lack of choice for thecustomer. It can also be argued that they lead to a loweringof quality across an industry because some organisations
stop when they reach the standard, rather than attempting
to surpass it. The other problem with the image of quality
has been with its widespread use for external checking bygovernment agencies and other bodies. This often turns
into a ‘checklist approach’, and if used in a heavy handed
manner can be seen as a stressful waste of time by those
involved.
To the uninitiated (and some of the initiated), therefore,
QM can seem like a closed club and a very modern form of
madness. In fact the key concepts of quality management,such as consumer laws, standards and inspection, havebeen with us for millennia. If one goes back to the crafts
workshops of the Western Zhou Dynasty in ancient China
we ﬁnd sophisticated rules for the quality assurance of
products, including a ‘department for formulating andexecuting standards’ (Qiupeng, Meidong and Wenzhao
1995:12). As soon as the human race was able to write,
documenting expected standards became a natural andessential activity for trade and commerce. Quality was, forthousands of years, controlled and maintained byinspection against these standards. Rejected items were
scrapped or in some cases reworked.
In the last 300 or so years, however, the Western world
has experienced an industrial and scientiﬁc revolution,and quality management has moved into a new cycle.
It has moved from an activity with a focus on maintaining
standards, to a deﬁned ﬁeld with a speciﬁc focus on
improving standards. The founders of this ﬁeld, suchas Taylor (1911) and Shewhart (1931, 1939), saw
management as a science and began to show how systems
could be experimented with and improved. They, andothers, began to describe manufacturing activities (andlater services) in terms of processes with inputs andoutputs, thus enabling the links between different partsof an organisation to be seen more clearly. They also
demonstrated the costliness of relying solely on inspection
and rework, and argued that the focus needed to move to©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

reducing the number of defective items that reach the ﬁnal
stage.2 Another key ﬁgure was the consultant W Edwards
Deming who built on these ideas to create one of the ﬁrst
coherent QM approaches, now known as total quality
management (TQM). TQM, which Deming wrote about in1986 in Out of the Crisis, is regarded as ‘the father of all
the quality management methods and standards that havecome since’ (Kemp 2006:177), and has had a huge
inﬂuence on the development of modern management
standards. Its greatest and most publicised success was inpost-war Japan, where Deming’s approach was taken up
whole heartedly by a nation who needed to change theirreputation for producing cheap poor-quality products. The
approach worked, and by the 1960s companies who had
implemented TQM, such as Sony, Toyota and Hitachi, werehousehold names in the West with a reputation for quality.
Despite the success of TQM, many people still equate
quality management purely with compliance to standards.
This is a misunderstanding, and as Simon Feary, CEO of theChartered Quality Institute says on their website ‘Quality is
not certiﬁcation. It is not inspection. It is not a science,
restricted to a department of experts … It is an approach,a set of tools, a philosophy, the substance of which is no
more than an articulation of what all good managers
recognise to be good business practices’ (2009).
Therefore, it can be argued that if your organisation is
performing functions that do not relate to quality (or to thecustomer’s perception of quality), it is wasting time, effort
and resource. Quality is, by deﬁnition, about everything thatan organisation does. An organisation that understands thiswill see that quality is not purely the domain of the ‘quality
department’ but in fact relates to the work of eachindividual. In this sense QM covers the following activities:• quality assurance: those activities related directly to
product/service quality, e.g. monitoring andmeasurement of product/service• cross-business quality management activities and tools,
including document control, records management, risk
management, internal audit, liaison with external
auditors, conformance to relevant standards, business
continuity planning, centrally led process improvement
activities• quality of other support processes used acrossorganisations such as recruitment, ﬁnance, IT,purchasing, project management, sales and marketing.

Quality matters for customers
One of the key tenets of modern quality management is
the importance of meeting customer requirements. In
order for a test provider to be conﬁdent that they are able todo this, they need to ﬁrst deﬁne who their customers are(i.e. stakeholders within the assessment community), andwhat their requirements are. Once that is done the tester
can deﬁne their processes and see what improvements canbe made.
2 It has been shown that the earlier in a process that this can be tackled the better,as early errors are compounded and become more serious errors later in theprocess (Kemp 2006:79).

|

3

Ultimately, in assessment, the end customer is the test
taker. However, in many cases the candidate may not be
the person who pays for, or makes the decision to enter for,a particular test. That may be a parent, teacher, school or
employer. In some cases governments will make thedecision to use a test, perhaps to increase language
attainment throughout their country. There are also otherstakeholders in testing, namely those who use the test
results. These include university admissions staff,
immigration departments, employers and publishers oftest preparation books.
What each of these stakeholders wants from the sametest will vary; sometimes their needs may conﬂict, and more
importantly, sometimes their needs may be unstated. This
could be because the end customer cannot, or does not,make their needs known. This can be due to a lack of directcontact with the provider, such as when school candidates
are entered for a test by their school, or it could be because
the customer does not know or understand what they need.
This is particularly relevant when talking about technicalaspects of testing, such as the concepts of validity andreliability. These will be discussed later in the article.
It is also useful to look again at what it is we think ourcustomers are ‘buying’. Is it the exam paper or the results?
Is it how they are treated whenever they need to use theresults? In fact, the whole candidate experience is our‘product’, and we impact on that at a number of stages asshown below:• when they study for the exam
• when they enter for the exam
• when they take the exam
• when they get and use their results and/or language
skills.
Building on the above, we can suggest that the following
areas will be of importance to the stakeholder:• Impact – What effect will studying for and taking theexam have on the skills and life goals of the candidate
and also on wider society?
• Accessibility – Can the candidate take the exam when
and where they want to?
• Fairness – Does the experience of taking the test feel fairto the candidate, both in terms of what is on the actual
question paper and in terms of the administration of the
test? Are results delivered in good time, and do they feelfair to the candidate and other stakeholders? Do goodresults ‘open doors’ for the right candidates?
• Customer service – If things don’t go to plan, how well
are they dealt with? What customer service is in place todeal with complaints and appeals?
Based on the above points we can see that there are
qualities relating to test items; those related to theadministration of the test, and those related to customer
service. Some aspects of administration and customer
service will impact on test validity, others will impact on
candidate experience but not necessarily have an impact on
the validity of the test. To enable an organisation to deliver
these areas, they will need an effective and efﬁcient
structure of management and administration as well as©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

4|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0quality objectives focused on the products and services
they are delivering.

ISO 9001 and its use at Cambridge ESOL
Cambridge ESOL has been certiﬁed to ISO 9001 since May
2007. A few years prior to that, top management took thedecision to gain certiﬁcation as they were conﬁdent that
Cambridge ESOL’s quality proﬁle was strong. It was also feltthat certiﬁcation to a management standard would not only
complement existing quality assurance procedures, but also
actively contribute to the development of the validity
argument. ISO 9001 was chosen above other standards andawards for a number of reasons. It is now the most wellknown quality management standard worldwide,
implemented by over a million organisations in 175
countries. This means that our global stakeholders are most
likely to have an understanding of what compliance to thisstandard means to them. The fact that the standard is
aimed at all types of organisations, whether manufacturing
or service, proﬁt or not-for-proﬁt, was a further contributing
factor in the decision.
The latest incarnation of the certiﬁcation standard is
ISO 9001:2008 Quality Management System Requirements,
which speciﬁes requirements to be met to achievecertiﬁcation. Its sister standard is ISO 9004:2009 Managing
for the sustained success of an organisation – A quality
management approach. This ‘provides a wider focus on
quality management than ISO 9001 … and provides
guidance for the systematic and continual improvement of
the organisation’s overall performance’ (2009:v). ISO 9004
is not designed to be used for certiﬁcation purposes.
Both standards are based on a series of eight principles
which ﬁt within the goals of Cambridge ESOL as set out next:
Principle 1: Customer focus(Taken from ISO 9001:2008 Quality Management System
Requirements 2008:vi)
If one looks at the description of PDCA above, it can beseen that there are some key questions that need to be
asked at the Check stage. These are as follows:• Are policies, objectives and requirements deﬁned?
Or in other words, do we ‘say what we do’?
• If they are deﬁned, are they being followed and met?
Or, in other words do we ‘do what we say’?
Of course, simply doing what one says, namely complying
with your own procedures, is not enough. An organisationmust be sure that what it does is effective in meeting therequirements of customers and other stakeholders. Even
where the organisation is both compliant and effective,
it should ask itself ‘Can we improve?’.
At Cambridge ESOL we incorporate such questions, bothin terms of checks on our own processes at head ofﬁce (seethe article in this edition by Beresford-Knox and Rose on ourinternal audit process) and our centres network. In regard to
the latter, we have a system of centre inspections which is
discussed by McCauley and Collett in this issue. It is
important to note that these inspections are complemented
by extensive training, documentation and other support for
centres and centre staff. In other words, we make sure that
inspection is an important check on our processes, rather
than the key method for assuring quality.
Table 1 provides selected clauses from ISO 9001 andshows how they are operationalised at Cambridge ESOL.
In order to avoid the dangers inherent in QM being
equated purely with compliance, top management at
Cambridge ESOL emphasises the fact that compliance tothe standard is a) a starting point for further improvement
and b) complements our existing focus on quality
assessment.

Principle 2: Leadership
Principle 3: Involvement of people

Quality assurance of tests

Principle 4: Process approach

ISO 9001 and other quality management standards arealso supported by other technical standards to provide
quality assurance. The oldest and most established are the1999 Standards for Educational and Psychological Testing(AERA/APA/NCME), the forerunner for which was ﬁrst
published in 1954. As the title states, these standards areintended to cover a range of test types. For language tests
in particular we turn to the Association of Language Testersin Europe (ALTE) who ﬁrst developed their Code of Practicein 1994. This, along with the 17 Minimum Standards forestablishing quality proﬁles in ALTE examinations and thePrinciples of Good Practice (2001), form the ALTE quality
assurance approach. For more information on ALTE quality
management, see articles in this edition by Saville, Parizzi
and Bolton.
None of the above standards currently operates as arequirement for certifying against. For this reason mostawarding bodies will choose to use these standards for the
purpose of internal benchmarking and improvement. In theUK, however, awarding bodies must comply with theregulatory monitoring and reporting rules of the Ofﬁce of
the Qualiﬁcations and Examinations Regulator (Ofqual),

Principle 5: System approach to management
Principle 6: Continual improvement
Principle 7: Factual approach to decision making
Principle 8: Mutually beneﬁcial supplier relationships(Taken from ISO 9004:2009 Managing for the sustained
success of an organisation – A quality management
approach 2009:38)
Another key methodology underlying the ISO 9001
standard is the Plan, Do, Check, Act (PDCA) cycle shown next:• Plan: establish the objectives and processes necessary todeliver results in accordance with customer requirements
and the organisation’s policies• Do: implement the processes• Check: monitor and measure processes and productagainst policies, objectives and requirements for the
product and report the results• Act: take actions to continually improve process
performance.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0|

5

Table 1: Interpretation of ISO 9001 standard in Cambridge ESOL
Selected clauses fromISO 9001:2008

Notes and comments on
interpretation of standard in
Cambridge ESOL

Selected clauses fromISO 9001:2008

Notes and comments on
interpretation of standard in
Cambridge ESOL

Clause 4 Quality management
system (including documentation
requirements)• The standard requires that theorganisation understands what it
does as a series of processes anddescribes how these interact witheach other. Cambridge ESOL has
done this in the form of a Process
Interactions Map with further layersof documentation beneath thisdescribing individual processes.

Clause 6 Resource management

In effect this clause applies to moststaff, temporary staff, outsourced staff
or consultants, as ‘Personnel
performing work affecting productquality’ actually covers most of theactivities we do. In an exam board, ofcourse this clause also applies to
examiners, and so at Cambridge ESOL
we need to be able to show external
auditors that training has taken place
where necessary and that examiners’
work is checked to show that they aremarking to the correct standards.

4.1 General requirements
The organisation shall establish,
document, implement and maintaina quality management system andcontinually improve its
effectiveness in accordance with therequirements of this International
Standard.
The organisation shall:a) determine the processes needed
for the quality management
system and their application
throughout the organisationb) determine the sequence andinteraction of these processes.
4.2 Documentation requirements
4.2.1 General
The quality management system
documentation shall include:a) documented statements of aquality policy and quality
objectivesb) a quality manualc) documented procedures andrecords required by thisInternational Standard andd) documents, including records,
determined by the organisation
to be necessary to ensure theeffective planning, operation andcontrol of its processes.

• Processes in a testing organisation
can be complex and technical, and adocument such as this can help staff
to get an overview of how what they
do impacts on further stages. Where
one part of the organisation wishes
to make changes, it can see which
other parts of the organisation thosechanges may impact on.

A criticism of ISO 9001 is that it canlead to ‘over-documentation’. It is
important to note therefore that the
number of required documents in thecurrent standard is very low:
• Quality Manual including a Quality
Policy• Six documented procedures• 18 types of record including‘appropriate records of education,
training, skills and experience’.
• 4.2.1 d allows the organisation todecide how many other documents
it needs. In Cambridge ESOL those
who work on the process decide if
documents are needed, althoughinternal audits may sometimes
recommend creation or
improvement of documentation
where this has been shown to be arisk.

6.2.2 Competence, awareness andtraining• The organisation shall:a) determine the necessarycompetence for personnel
performing work affecting
product qualityb) provide training or take otheractions to satisfy these needsc) evaluate the effectiveness of
the actions takend) ensure that its personnel areaware of the relevance andimportance of their activities
and how they contribute to theachievement of the quality
objectives, ande) maintain appropriate recordsof education, training, skills
and experience.

Clause 7 Product realisation1
7.2 Customer-related processes
7.2.1 The organisation shalldetermine:a) requirements speciﬁed by thecustomer, including therequirements for delivery andpost-delivery activitiesb) requirements not stated by thecustomer but necessary forspeciﬁed or intended use, where
knownc) statutory and regulatory
requirements related to the
product, andd) any additional requirements
determined by the organisation.

A further point to note is that
controlled documentation is crucial
to the smooth running of a process.
For example, many hours can bewasted if staff have to search for theright document to use. In Cambridge
ESOL we have put in place a simple
system of document registers which
list all published documents.

Clause 5 Management
responsibility
5.1 Management commitment
Top management shall provide
evidence of its commitment to thedevelopment and implementationof the quality management systemand continually improve its
effectiveness by:a) communicating to theorganisation the importance ofmeeting customer as well as
statutory and regulatory
requirementsb) establishing the quality policyc) ensuring that quality objectives
are establishedd) conducting management reviewse) ensuring the availability ofresources.

In a successful organisation,
management will support theimplementation and improvement of
the quality management system(QMS) far beyond achievement ofcertiﬁcation.
In Cambridge ESOL the Chief Executive
and Group Directors work through therole of the Quality and Evaluation
Manager to implement and improve
the QMS. This role reports to the
Director of Research and Validation,
thus drawing together the principles ofquality management of process, andtechnical evaluation of product.

continued overleaf
1 A note in Clause 3 states that ‘throughout
the text of this international standard,
wherever the term “product” occurs, it canalso mean “service” (2008:1)’.

We also have in place a system of
quality management training sessions,worksheets and guidance documents
for internal staff, including:• general introduction to quality
management concepts• risk management• self assessment and process
improvement.
This is a key clause as it covers
product development and provision of
product and service including any
product release, delivery and postdelivery activities. It includes a focus
on the customer, looking both at thecustomer requirements (clause 7.2.1
as below) and also at the processes inplace for customer communication.
a) A candidate may for example, tell usthat they want ‘a fair test with fairadministration, at a price I canafford’. However, in most cases
exam boards do not have one-onone relationships with individual
candidates, so it tends to be
centres, schools, commercial
businesses and other organisations
who mediate and specify
requirements. In most cases thiswill be about choosing an existingproduct, but where the need isshown to be currently unfulﬁlled
this process can lead to a productdevelopment process and thecreation of a new or adapted test fora speciﬁc client.
b) This includes elements of thevalidity argument that the testing
provider knows are necessary but
the stakeholder may not specify.
To deﬁne these in more detail weneed to look at speciﬁc industry‘product’ standards and frameworks
such as the ALTE Code of Practiceand Minimum Standards, or theStandards for Educational andPsychological Testing (AERA/APA/
NCME). See the following section forfurther information.
c) There may be further regulations
that awarding bodies need to takeaccount of, for example, in the UK
issues of access to tests for people
with disabilities are covered in the
Disability Discrimination Act. Issues
regarding the information that we
keep about our candidates iscovered by the Data Protection Act.
d) Then there are requirements that
may not impact on individual
stakeholders but that the test
provider will want to consider ingeneral and for the long term. For
example, we could argue that
impact studies designed to look at©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

6|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Table 1: Interpretation of ISO 9001 standard in Cambridge ESOL – cont.
Selected clauses fromISO 9001:2008

Notes and comments on
interpretation of standard in
Cambridge ESOLthe effects of washback are notnecessary for the intended use of
the exam, but they are deﬁnitely
necessary in terms of evaluation ofand continual improvement of theexam.

7.2.3 Customer communication
The organisation shall determine
and implement effective
arrangements for communicatingwith customers in relation to:a) product informationb) enquiries, contracts or order
handling, including
amendments, andc) customer feedback, including
customer complaints.

a) For a language test provider productinformation will refer to information
about the test, made available sothat users can decide if it is the righttest for them and so that teachers
can teach appropriate content.
b) Cambridge ESOL does this in a
number of ways including throughour public websites, our programmeof international seminars forteachers, and publication of examhandbooks and supporting
materials.
c) Enquiries that cannot be answeredwith the information supplied
through the channels mentioned
above are dealt with through our
network of centres and also throughour Centre Support Ofﬁcers andCustomer Service Desk.
d) The Customer Service Desk also acts
as a ﬁrst port of call for any
complaints we receive. In testingorganisations complaints may be
about quality of service but in somecases candidates will question theaccuracy of the results they havereceived. When these cases go
forward they are called appeals andare dealt with through a deﬁned
process as set out on our public
website. The UK regulator Ofqualalso speciﬁes requirements forappeals processes.

formerly the Qualiﬁcations and Curriculum Authority3 (QCA).
The QCA published guidance for Awarding Body Selfassessment (2006) and The Statutory Regulation of External
Qualiﬁcations (2004). These regulations remain in force andcover aspects of organisational effectiveness such as
governance, as well as more technical exam related
aspects. Various frameworks such as those developed by
Bachman and Palmer (1996) and Weir (2005) attempt todeﬁne the full range of attributes at test development, test
review and evaluation stages. Although these have notspeciﬁcally been developed into standards or codes ofpractice, they have inﬂuenced the development of thestandards mentioned in this article.
It could be argued that some testing situations areperhaps more straightforward to standardise than others.
However, the Standards for Educational and Psychological
Testing (1999) highlight that a tick box approach to quality
management of tests and test items will, in most cases,
not be appropriate (1999:4):‘Evaluating the acceptability of a test or test application does not rest on
the literal satisfaction of every standard in this document, andacceptability cannot be determined by using a checklist. Speciﬁc
circumstances affect the importance of individual standards, andindividual standards should not be considered in isolation. Therefore,
evaluating acceptability involves:a) Professional judgement that is based on a knowledge of behaviouralscience, psychometrics, and the community standards in theprofessional ﬁeld to which the tests apply.
b) The degree to which the intent of the standard has been satisﬁed by
the test developer and user.
c) The alternatives that are readily available.
d) Research and experiential evidence regarding feasibility of meetingthe standard’.

Clause 8 Measurement, analysis
and improvement
8.2.2 Internal audit
The organisation shall conduct
internal audits at planned intervals
to determine whether the quality
management system:a) conforms to the plannedarrangements, to therequirements of this International
Standard and to the quality
management system
requirements established by theorganisationb) is effectively implemented andmaintainedThis clause closes the loop in
Shewhart’s PDCA cycle by covering theCheck and Act phases. It includes therequirements to have a system ofinternal audits, to measure customer
satisfaction and to control
non-conforming product2.
It also deals with the taking of‘preventive action’ and at Cambridge
ESOL we have a deﬁned risk
management process that deals with
this.

An internal audit programme shallbe planned
8.2.4 Monitoring and measurementof product
The organisation shall monitor andmeasure the characteristics of the
product to verify that productrequirements have been met …

Tests and test items are monitored andmeasured at a number of stages
throughout the test development andtest production processes and there isa dedicated Research and Validation
Group whose task it is to do this.

2 Non-conforming product is any product orservice that does not conform torequirements, be they those as in 7.2.1,
stated by the customer or unstated.

Language testing is one of the more complex types oftesting. As Jones (2009:7) notes ‘tests do not simply
measure discrete atoms of knowledge, but attempt to tap
hidden mental processes’. For this reason, the standards
mentioned here focus on methods of evaluation and typesof evidence required, rather than speciﬁc values
recommended for evidence. They talk about building an‘argument’ regarding validity, and emphasis is on theprofessional, educated judgement of the evaluator.
However, evidence is needed. For example, as Weir (2005:
1) notes, ‘test validation is the process of generating
evidence to support the well-foundedness of inferences
concerning trait from test scores’.
A key task, therefore, of all testing providers is to develop
this validity argument for each assessment and to provide
evidence that shows the argument is met. Evidence can beprovided to staff internally so they fulﬁl the aim ofcontinually improving assessment. Alternatively, it may beprovided to auditors, as in the case of ALTE audits (seeSaville, Parizzi and Bolton in this issue), or regulatory
assessments by bodies such as Ofqual in the UK. Beyond
this, the test provider will also have a responsibility to

3 In 2008 the QCA was split into two organisations: QCDA and Ofqual. In April
2010, pending an act of parliament, these changes become ofﬁcial.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0communicate the argument appropriately to its
stakeholders. This in turn helps them to make the rightchoices about which assessment to take. It is worth notingthat certain stakeholders may be more sophisticated intheir understanding of relevant concepts, however testingorganisations can do much to educate stakeholders about
these concepts.
This is not to say that each ‘product’, or exam, cannot
have set standards: they can and should. However,awarding bodies will need to show how the speciﬁc context
and use of each exam inﬂuences the standards that aremet, for example in terms of values for reliability, or size ofcandidate samples for pretesting. In time, similar types of
exams may be compared. However, in most cases contexts
are unique to each exam and comparisons are difﬁcult tomake.
A method that most test providers use, to checkstandards have been met, is that of the Test Speciﬁcation
document. Checking against such a document is done atvarious stages of the test development and production
cycle before the test is released and taken by candidates.
There will also be further checks post live administration
to check whether the test items worked as expected.
Examples of the type of internal quality controls include
training of item writers and test constructors, pretesting oftest items, and additional vetting and checking of test
items. The types of attributes checked at these stages
include level of difﬁculty of items, presence of bias,
content and format of items.

|

7

only the beginning. It is important to put theory into
practice and move to the ‘act’ phase of PDCA – to makechange happen. The other attribute that is required is awillingness to admit when improvements can be made.
This includes a degree of honesty about the value of quality
management systems and practices themselves. They are,as in most human endeavours, ﬂawed systems in need of
their own continual improvement. That is why at Cambridge
ESOL we work hard to continually improve the effectivenessof our QMS, including our internal audit system. See thearticles in this issue by Beresford-Knox and Rose for
evidence of how we do this, and by Hughes for aninteresting study into the dynamics at play in an internal
audit.

References and further reading
AERA/APA/NCME (1985/1999) Standards for Educational andPsychological Testing, Washington: AERA.
ALTE (2001) Principles of Good Practice for ALTE Examinations.
ALTE Code of Practice and Minimum Standards, retrieved fromwww.alte.org
Bachman, L and Palmer, A (1996), Language Testing in Practice,
Oxford: Oxford University Press.
BSI (2005) ISO 9000:2005 Quality Management Systems –fundamentals and vocabulary, British Standards Institution.
BSI (2008) ISO 9001:2008 Quality Management Systems –Requirements, British Standards Institution.
BSI (2009) ISO 9004:2009 Managing for the sustained success of anorganisation – A quality management approach, British Standards
Institution.

Conclusion
In terms of quality assurance of assessments and quality
management of processes, many tools and approaches
are now successfully embedded within testingorganisations. Product and management standards areboth useful as guides and one can complement the otherto assure customers that a test provider understands themeaning of ‘quality’. The aim should not only bemaintenance of standards but also improvement. This canhappen in many ways, including the use of internal andexternal audits and assessments. Inspection remains a
useful tool if used appropriately, for example whenlooking at compliance with certain aspects of customer
service. Where it is clear what the service level should be
it is possible to operate a check box approach toinspection.
For quality of tests and test items a more nuanced
approach is required, and an understanding of the need toclearly deﬁne the context and use of the exam. However,this should not preclude test providers from gathering
relevant evidence and presenting this to stakeholders in
an appropriate format and context. Within testing
providers there is a growing understanding of the need to
do so, whether there is a local need for accountability
such as a move to a more regulated system, or whether
the desire comes from within the organisation itself.
For quality management to work it requires continuingcommitment; certiﬁcation to a standard such as ISO 9001
or self assessment against the ALTE minimum standards is

Deming, W E (1986) Out of the Crisis, Cambridge: Cambridge
University Press.
Feary, S (2009) Time for Quality, retrieved from www.thecqi.org
Jones, N (2009) A comparative approach to constructing a
multilingual proﬁciency framework: constraining the role ofstandard setting, Research Notes 37, 6–9, Cambridge: Cambridge
ESOL.
Kemp, S (2006) Quality Management Demystiﬁed, New York, McGraw
Hill.
Oakland, J (2000) TQM Text with Cases, Oxford: Elsevier Butterworth
Heinemann.
QCA (2004) The statutory regulation of external qualiﬁcations in
England, Wales and Northern Ireland, Qualiﬁcations andCurriculum Authority.
QCA (2006) Awarding body self-assessment. Guidance for evaluating
and improving performance, Qualiﬁcations and Curriculum
Authority.
Qiupeng, J, Meidong, C and Wenzhao, L (1995) Ancient China’s historyof managing for quality, in Juran, M (Ed.) A History of Managing for
Quality: The Evolution, Trends, and Future Directions of Managing
for Quality, ASQC Quality Press: Milwaukee, Wisconsin.
Shewhart, W A (1931) Economic Control of Quality of Manufactured
Product, New York: D Van Nostrand Company.
Shewhart, W A (1939) Statistical method from the viewpoint of quality
control. Washington: The Graduate School, the Department ofAgriculture.
Taylor, F W (1911) The Principles of Scientiﬁc Management, New York:
Harper and Brothers Publishers.
Weir, C J (2005) Language Testing and Validation: An evidence-based
approach, Basingstoke: Palgrave Macmillan.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

8|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Implementing an internal audit process: A case study
from Cambridge ESOLNICK BERESFORD-KNOX RESEARCH AND VALIDATION GROUP, CAMBRIDGE ESOLDITTANY ROSE RESEARCH AND VALIDATION GROUP, CAMBRIDGE ESOLIntroduction
Cambridge ESOL achieved certiﬁcation to the internationallyrecognised quality standard ISO 9001 in May 2007. This
standard outlines a set of requirements for a quality
management system. Internal auditing is a corerequirement of ISO 9001; the standard speciﬁes thatinternal audits should be conducted at planned intervals.
An internal audit system was implemented in August 2006
and at the request of the external assessment body allareas of the organisation were internally audited prior to theISO 9001 external certiﬁcation visit in May 2007.

Building up internal auditor skills
An internal audit team requires a wide variety of
individuals, from varying levels of authority andprofessional backgrounds, to be capable of tacklingdifferent types of audit. The team must have sufﬁcient
resources to ensure that the internal auditor is independentof the process being audited. ISO 9001 provides us withguidance to this end, stating that: ‘The selection of auditorsand conduct of audits shall ensure objectivity andimpartiality of the audit process. Auditors shall not audit
their own work’ (British Standards Institution 2008:12).
This point is also highlighted in internal auditing training
materials: ‘Auditors should be independent of the activity
being audited and free from bias and conﬂict of interest’(Abbassi 2009:10). The following could be considered as
key competencies of an internal auditor: good timekeeping,
impartiality, communication skills, diplomacy, sensitivity
and politeness. Styles and techniques will vary between
individuals and this variation creates a rounded team.
The UK implementation body of ISO 9001, the British
Standards Institution, provides guidance on auditing of
quality management systems in ISO 19011. With reference
to the training of internal auditors the standard suggests:‘They should have completed auditor training that
contributes to the development of knowledge and skills …
This training may be provided by the person’s own
organisation or by an external organisation’ (British
Standards Institution 2002:26). Cambridge ESOL internal
auditor training involves classroom based training with an
external training organisation and hands-on training in theworkplace. An ISO 9001 introduction training session is ﬁrst
attended to gain a detailed understanding of the standard,followed by a 2-day internal Auditor Training course.
Whereas the ﬁrst focuses on the detail of the standard, thesecond looks at internal auditing techniques including the
responsibilities of the internal auditor from pre-audit topost-audit activities. Many training organisations suggestthat having attended the two courses, the trainee is now
ready to embark on their ﬁrst internal audit. We, however,add a ﬁnal stage to the training. To build conﬁdence, anddraw together skills, the shadowing of a more experienced
internal auditor is arranged by the internal audit team.
This could entail observing an entire internal audit frompre- to post-audit activities, or even actual participation
if the trainee and internal auditor feel it is appropriate.
Following this shadowing activity the trainee will undertake
their ﬁrst internal audit with guidance and shadowing froman experienced internal auditor.
Internal auditor training is a continual process at
Cambridge ESOL and we aim to improve with regularinternal sessions at internal auditor meetings and further
external training such as Lead Auditor courses. For example,
research by Hughes (2010), highlighted evidence that
effective feedback to auditees during an audit is not often
sufﬁciently fulﬁlled. This issue was subsequently raised anddiscussed at an internal auditor meeting, with techniques ofeffective feedback suggested.
Internal auditing is a skill that is reﬁned and developed
over time, and as mentioned by Hughes (2010), a traineeinternal auditor will initially be inexperienced in the coretechniques. This is relevant to Cambridge ESOL and manyother organisations, and so we attempt to assist internal
auditor development through regular training, shadowing
and close supervision by the management of the internal
audit system.

Creating a system
The internal audit system was designed around a familiar
model as illustrated in Figure 1. The ﬁrst stage of theprocess is the planning of the internal audit schedule forthe year. This details the internal audits for the coming year,
including scope, internal auditor, auditee and date. Initially,
for the ﬁrst round of internal audits, this programme had
the objective of internally auditing every area of thebusiness as explained above. Over subsequent years thiswas reﬁned to a smaller number of internal audits in key
areas to avoid over-auditing of each department. Factors
considered when highlighting areas for internal audit
include areas of high risk, areas where problems haveoccurred in the past and new processes.
The internal audit programme consists of high levelaudits looking at the process as a whole, and more focusedaudits analysing a deﬁned part of a process. These will all
focus on compliance (conformance to requirements),
effectiveness (process produces intended results/output)©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0|

9Figure 1: The internal audit system

Internal audit
programme· Scope· Date· Auditee· Auditor

Pre-audit
activities· Confirmationof scope· Document
review· Processreview· Audit plan
· Preparation ofchecklistand efﬁciency (are there opportunities for improvement?),
but some will focus more heavily on one than another.
The system outlined in Figure 1 was documented in astandard operating procedure with an additional set of
guidelines detailing the interactions between parties
involved. Templates were drawn up for the internal audit
plan, checklist, internal audit and non-conformance reportto ensure commonality between internal auditors. The
auditees were also considered in the implementation of theprocess with a document produced to assist auditees in
preparation for an internal audit called Frequently Asked
Questions (FAQs). This was later followed by an additional
document focusing on the process approach, with abackground to the aims of process auditing and to serve asa self-assessment and preparation tool. In addition to the
documents that assist auditees and internal auditors,
a number of templates were issued for the administrationof the system, again to ensure standardisation ofcommunication in pre- and post-audit activities.
Alongside the implementation of ISO 9001 a web-based
document management system was introduced in
Cambridge ESOL. This document management system
houses the internal audit records, including internal audit
plans and internal audit reports, from implementation of
the internal audit system in August 2006 to present. The
document management system contains records,
documents and templates that are mentioned in theprevious paragraph. To track the current status of internal
audits, a spreadsheet was designed highlighting key dates
and outstanding actions.
The quality and evaluation manager takes overall
responsibility for the internal audit system, with frequent
input from senior management. All internal audit reports
are reviewed by the quality steering group overseeing theprocess, and a 2-monthly report on the quality
management system is sent to the senior management
team.

Audit

· Openingmeeting· Meetings withauditees

Post-audit
activities

Close out
and follow-up· Report writing· Closingmeeting· Distribution ofreport· Monitoring ofaudit actions· Review
effectivenessof actions· Closure ofactions

internal audit, with commentary from the writer of thisarticle and using the report written by the internal auditor.
Examination administration

Scope: A high level process audit, from the processing of
entries to the despatching of examination materials tocentres, including Oral Examiner management and centre
management.
Summary: The internal auditor reported that the process
had three distinct sub-processes managed in different areasof the business.
Findings:• The process consistently delivers the required outputs inall three areas.
• There is a clear culture of improvement across all threegroups internally audited.
• The auditees were aware of areas where improvement
was required and plans were in place to satisfy theseneeds.
• The process is monitored satisfactorily at various levels
through reports to applicable meetings.
• Issues with administrative support meant that somedocumentation was not up-to-date and ﬁled correctly.
• The internal auditor felt that further analysis could takeplace from the outcomes of the Team Leader system.
Conﬁdential despatch

Scope: The process of despatching conﬁdential materials
from the Cambridge Assessment warehouse facility,
including the cross-departmental interface.
Summary: The internal auditor felt the process worked
well given the large number of despatches, but noted that
there were some risks in the process which could be
addressed.
Findings:

ExamplesBelow are three examples of internal audits which havetaken place over the last two years in the areas ofrecruitment, examination administration and conﬁdential
despatch. The examples provided are an overview of each

• A new group formed to overview the despatch process
had been very successful, implementing a number of
positive developments.
• A fortnightly report had recently been introduced toprovide a summary of measures within the process(i.e. percentage of despatches sent).

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

10|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0• Risk assessments had been conducted.

• Suggestions were made regarding the updating of therecruitment manual for managers.

trained internal audit team that identiﬁes relevant andbeneﬁcial improvements to a process.
A recent staff message highlighted some of the beneﬁts
of internal auditing and also included testimonials frominternal auditors. Comments on the beneﬁts of being aninternal auditor included gaining an insight into the work ofdifferent parts of the organisation, meeting and working
with colleagues from other departments, and seeing how
the processes in the organisation ﬁt together. The focus of
the message was to promote the potential improvementsinternal auditing can bring to a process, and theadvantages of being an internal auditor. Further
communication tools have included staff seminars on
internal auditing and also coverage in induction of new staff
through worksheets and presentations.
As well as communicating the beneﬁts of internal
auditing, it is also important to review the process itself andseek feedback from those involved in the system. Since
implementation the process has been improved regularly,
with simpliﬁed administration procedures and clearer
documentation following feedback from internal auditors
and auditees. Feedback is sought from auditees after eachinternal audit via a questionnaire, and internal auditors
attend a post-audit meeting with the quality manager.
The quality management team looks at best practice inother organisations and communicates with other quality
managers. Conferences and training events are regularly
attended to keep informed of the latest developments inthe ﬁeld.

• It was suggested that a formal mechanism for feedbackon the recruitment process should be introduced.

References and further reading• Several service level agreements were in place uponwhich it was suggested could be harmonised into fewer
documents.
Recruitment

Scope: A high level process audit of the recruitment process
for permanent staff.
Summary: The internal auditors found the process to beeffective in appointing candidates to available posts andalso reported that a future internal audit could cover the
online recruitment system which was to be implemented.
Findings:• Auditees have a sound knowledge of the process andassociated procedures, and are committed to ensuring
these are adhered to. The implementation of an online
recruitment system should streamline the process.
• There are clear guidelines and ﬂow charts for recruitment
managers.
• Positive comments were received regarding the feedback
from Human Resources to candidates.
• Some staff were unable to locate documentation relating
to recruitment on the intranet, for example therecruitment criteria document.

Abbassi, N (2009) Internal Auditing of Quality Management Systems(July 2009), Abbassi Ltd.

Communicating the beneﬁts
The philosophy behind the Cambridge ESOL Internal Audit
System is not one of box ticking or rubber stamping, tosatisfy the criteria of ISO 9001, but to add value to theorganisation and to promote continual improvement. This isachievable through the use of an experienced and well

British Standards Institution (2002) Guidelines for quality and/or
environmental management systems auditing (ISO 19011:2002).
British Standards Institution (2008) Quality Management System –Requirements (ISO 9001:2008).
Hughes, G (2010) Using conversation analysis to investigate theinteraction of QMS audits, Research Notes 39, 10–20, Cambridge:
Cambridge ESOL.

Using conversation analysis to investigate theinteraction of QMS audits
GLYN HUGHES ASSESSMENT AND OPERATIONS GROUP, CAMBRIDGE ESOLIntroduction
Internal quality audits are an integral part of most quality
management systems (QMS), performing the function ofverifying that a company’s QMS is being implemented.
They are designed to provide an opportunity for staff andmanagement to assess and report on the effectiveness ofprocesses in an organisation. Much of the literature on
internal auditing focuses on the fact that there is asigniﬁcant gap between what a QMS audit could be andwhat it is; a particular problem is the relationship between
the auditor and the auditee. This article explores this issue,
using conversation analysis (CA) to carry out a detailed
sequential analysis of transcriptions of three audit
interviews undertaken at Cambridge ESOL1.
In this article I analyse how the relationship between
auditors and auditees develops in real time during a number
1 This article is based on the author’s MSc dissertation in Educational Management
in TESOL, at the School of Languages and Social Sciences, Aston University, UK.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0of audits. This analysis is contextualised in the literature
on quality management and internal auditing, before
summarising the principles behind CA and presenting themethodology. The conclusions focus on what lessons canbe learned by internal auditors and discuss possibilities
for future research in this area.

Quality management systems
In a competitive marketplace, all organisations seek toensure and to publicise the quality of their products andservices. A common way of doing both of these is byimplementing a quality management system and obtaining
certiﬁcation to an externally recognised standard, mostcommonly ISO 9001 which is used by more than a million
companies in over 170 countries around the world (Francis
2008:16). The advantages of such a system are clear: theimplementation of a robust quality management system canensure that a company operates effectively in pursuit of its
business objectives and can allow it to manage risk.
As Pfannerstill writes, ‘The purpose of the internal audit is
to investigate, analyse and report on activities related to thequality management system. Considering the potential forsome inaccuracies in our procedures and inconsistencies in
our activities, you would think this activity would havesigniﬁcant importance in the organisation … Based on what
I read in the literature and my personal experience, it doesnot’ (2005:67–8). This sums up a problem that lies at theheart of the quality audit – a general feeling that, in manyorganisations, there is a signiﬁcant gap between what a
QMS audit could be and what it is.
Central to the quality audit is the relationship between theauditor and the auditee (the individual being audited). In an
ideal world, both auditor and auditee are working towards
the common goal of reviewing processes and procedures inorder to ensure that nothing is seriously wrong and to ﬁndopportunities for improvement. However, as Asbury andAshwell report: ‘In many organisations … auditors are stillviewed with suspicion and even a degree of disdain by those
who are likely to be audited’ (2007:174). Rather than arelationship of mutual trust built on the common goal of
ensuring quality in an organisation, they refer to the‘antipathy between the hunter and the hunted – thepoliceman and the offender’ (ibid).

Quality management and internal auditing
The internal quality audit traces its roots from two distinct
places, both of which combine to give its current role. Firstly,
the notion of auditing as a form of control has existed since
Roman times. This original form of auditing focused on acompany’s accounts and is still very common today.
Companies regularly carry out audits of their ﬁnances.
However, as Moeller and Witt (1999:22–3) explain, quality
auditing is a distinct type of audit that has developed a very
different focus. It retains the same overall purpose of
ensuring that systems are under control, but here thesystems being audited are operational and managerial
instead of ﬁnancial.

|

11

The other root of the internal quality audit is in the growthof quality management techniques and systems that havebeen applied worldwide since the Second World War. The
pioneers in the ﬁeld of quality management were Deming(1982) and Juran (1988, 1989), who developed new
approaches to quality in Japan in the 1950s and thenexported them to the US in later decades. Quality was also a
focus during and after the war in the British and US military
where errors in production had very obvious implications,such as warheads not exploding when intended. The British
Military Standard became a British Standard (BS 5750) in
1970, and this British Standard evolved into the ﬁrst
international ISO standard in 1987. ISO 9001:2008 in turn isthe most widely used successor to the 1987 ISO standard,
and the standard used by Cambridge ESOL. Internal quality
audits have become an established part of almost all quality
management systems and are required by ISO 9001:2008 as
outlined in section 8.2.2 of the ISO standard, which statesthat the organisation shall conduct internal audits atplanned intervals to determine whether the quality
management system:• conforms to the planned arrangements, to therequirements of this International Standard and to thequality management system requirements established by
the organisation, and• is effectively implemented and maintained.

Quality management
A quality management system ‘deﬁnes all activities that areimportant and agreed upon to ensure the quality of the
product or service’ (Pfannerstill 2005:40). W Edwards
Deming is often referred to as the founder of modern quality
management. His fundamental contributions to quality
management were the notions of ‘systemic control’ and‘continuous improvement’. The ﬁrst of Deming’s 14
management principles that run throughout much of his
writing is ‘Create constancy of purpose toward improvementof product and service’ (1982:23). Fundamental to Deming’s
notion of improvement was the Shewhart cycle of ‘plan, do,check, act’ (1982:88). This has become axiomatic to manybusinesses today as the Deming Wheel (Asbury and Ashwell
2007:35).
Although Deming was instrumental in laying thefoundations of modern quality management, he did not
discuss auditing in detail. Juran (1989), however, viewed
audits as a crucial means of quality control. To Juran, quality
audits are an integral part of what he refers to as strategic
quality management (SQM) (1989:203). Juran emphasises
the strategic role of the quality audit in his list of questions
that an audit is designed to answer, the ﬁrst of which is:‘Are our quality policies and goals appropriate to ourcompany’s mission?’ (ibid:204). The notion of using aquality management system for the strategic purpose of
ensuring that the broader objectives of the organisation are
met is a recurrent theme in the literature on quality
management. The evolution of the role of the QMS, and byextension the internal audit, from one of checking technicalmatters to one of dealing with management issues is dealt©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

12|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0with by many writers on quality management (see Asbury
and Ashwell 2007, Juran 1988, Pfannerstill 2005).

about the audits on their own terms, rather than compared
to an ideal audit. It will also ensure that any areas that cause
difﬁculty in the area of auditor–auditee relationship are
genuine, rather than contrived to prove a point.

The role of the internal auditor
Taormina and Brewer (2002:1–2) summarise thefundamental tenets of ISO 9001 as:• document what you do• do what you documented• verify that you are doing it.
The job of the internal auditor is to perform the job of
veriﬁcation, a job that, as Pfannerstill’s insights make clear,is both essential to a successful quality management system
and at the same time fraught with difﬁculty. Pfannerstill
describes an internal audit as ‘a reoccurring activity, thepurpose of which is to verify that systems and processes are
implemented as intended and to identify variations somanagement can determine whether they should beincorporated or corrected’ (2005:89). In other words, thegoal of an internal audit is to review the effectiveness ofprocedures, and to verify that they are happening.
However, this process is not as straightforward as it may
appear. Asbury and Ashwell allude to the complexities of theaudit process when discussing their ‘Audit Process Roller
Coaster©’ (2007:82–8). This illustrates how, on top of all
the interpersonal matters that auditors need to consider,they also need to move between high-level analysis anddetailed review and veriﬁcation within an audit, while
maintaining the link between the two. An audit plan is an
ideal that is realised through the inﬁnitely messier businessof an actual audit.
Relationships between individuals, people and systems
are at the heart of internal auditing. A point of conﬂict in anaudit may be differing perceptions of organisational cultureon the part of the participants. This can in turn lead todiffering views on what the audit is for: whether it isprimarily about detecting deviations or whether it is also
about evaluating the effectiveness of processes.
Much of the literature on quality management and internal
auditing focuses on the gap between the aspirations ofinternal auditing and the reality. The goal of much of theliterature on auditing is to attempt to make the audit work
more effectively, or to prevent it from deviating from the‘ideal’ audit that exists in the minds of the authors. To do so,
the authors provide charts such as those produced byAsbury and Ashwell (2007), or systems such as Pfannerstill’s
progressive audit with its scoring guide (2005:73–144).
Auditor training can also take a similar, and indeed,sometimes somewhat more extreme approach. An exampleof this is training material that asks trainee auditors to
identify the shortcomings of a comically incompetent‘auditor from hell’ (Meakin no date:4–10) who is rude andincompetent in a way that no real auditor would ever be.
This study follows a different approach from that used inmost of the literature about quality management andinternal auditing. It will use the emic perspective of CA toobserve and discuss what is actually happening in internal
audits as they occur. This will enable us to draw conclusions

Conversation analysis
Conversation Analysis, as deﬁned initially by Sacks (1992),
grew out of the sociological discipline of ethnomethodology
pioneered by Garﬁnkel (1984). He rejected the sociologicalapproach prevalent at the time that starts from a position of‘conceptual abstraction’ and seeks evidence about how, andthe extent to which, individuals successfully conform (Boden
1994:43). Ethnomethodology, by contrast, adopts an emic
perspective (ten Have 2007:34–5) in which the researcher
views the social action (such as a conversation) as thestarting point, and uses how the participants demonstrablyorientate themselves to their actions as the basis foranalysis. In theory at least, the researcher does not bringprior categorisation to bear on the analysis but rather uses
the analysis to create categories according to how theanalyst believes the participants themselves view theiractions.
Sacks (1992) applied the emic perspective ofethnomethodology to the study of spoken discourse,
rejecting the idea of rule-governed behaviour and insteadholding the view that conversation, like other behaviour,
is both context-shaped and context-renewing and that
context is ‘both the project and the product of theparticipants’ own actions and therefore inherently locally
produced and transformable at any moment’ (Drew and
Heritage 1992:18).
The essential features that a conversation analyst will
study are: turn taking organisation, sequence organisation,
repair organisation and turn design (ten Have 1999:107).
In addition, and in spite of its determinedly bottom-up
approach, CA’s focus on classifying sequences has produced
a ﬁxed terminology and also a number of generalisations,or ‘regularities’ (ten Have 2007:37), that occur inconversation. These include adjacency pairs, turnconstructional units, pre-sequences, response tokens,
accountable silences and many others. Practitioners of CA
have been able to use patterns found in one piece of
research to elucidate a very different situation. This research
will attempt not to prejudge the motivations of the speakers,
but where relevant, will refer to patterns found by other CA
researchers to explain the actions of the participants.
Institutional interaction

A major use of CA is the analysis of institutional interaction
as its tools seem especially appropriate in gaining a fresh
perspective on a range of different institutional procedures.
Boden in particular makes the case that the principles of
ethnomethodology can reveal much about how processes
within an organisation are, to use Weick’s term, ‘enacted’(Boden 1994:41). Boden’s view is that ‘what looks – fromoutside – like behaviour controlled by rules and norms is
actually a delicate and dynamic series of interactionallylocated adjustments to a continual unfolding of “just what”
is going on’ (1994:41). She argues that organisations are not©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

best deﬁned by their rules and procedures but by what
people do and, by extension, what they say to each other;stating ‘Talk is not just talk but rather the mode and medium
through which the structure of the organisation isconstituted and reconstituted’ (ibid:202).
Research that is particularly relevant to this study
illustrates how certain conversational norms can beanalysed with the institutional context in mind and canreveal how the participants are orientating themselves toeach other and why they may react in a particular way.
Bergmann (1992:137–62), for example, discusses how
attempts by psychiatrists to elicit problems from patients infact carry ‘a veiled morality’ which can account for the
hostile reaction they receive. Simply by mentioning an event,a psychiatrist turns it into something worth mentioning, or a
‘reportable event’. Also relevant to this research is the studyof questions and answers (see, for example, Bergmann
1992, Drew 1992, Heritage and Seﬁ 1992). The type ofquestioning ranges from hostile cross examinations (Drew
1992) to advice given by health visitors to ﬁrst-time mothers(Heritage and Seﬁ 1992). A common thread running throughthese studies is the position of power that the ‘questioner’ isin, and the extent to which choices made by the person
asking the questions, in terms of sequence organisation andof turn design, determine the manner in which the recipient
responds and the tone of the proceedings in general.
Ten Have examines what can happen when questioners
deviate from pre-planned and pre-scripted routines, forexample when dealing with repair sequences in scriptedinterviews (1999:187–8). He draws a distinction that is notdissimilar to Asbury and Ashwell’s earlier (2007) Audit
Process Roller Coaster© mentioned earlier when he posits
that a useful focus of what he calls ‘applied CA’ is to
examine the ‘tensions between local practices and any
larger structure’ – in this case the difference between thehigher level script and the locally realised interview.
This research draws on CA to explore internal audits fromthe bottom up; rather than starting with what should be
happening, the research generalises from what ishappening. It is of course essential with any qualitative
research, but especially so with CA, which by its nature
focuses on a narrow sample and rejects broad notions ofgeneralisation, to authenticate and legitimate any ﬁndings,
as well as to aid their transferability to other contexts.

Analytical approach
The approach used here aimed to draw conclusions that canbe used by quality professionals as well as CA specialists:
it is therefore very much a piece of applied
ethnomethodology. The early stages are best described,
as they have been by Psathas, as ‘unmotivated looking’(1995:45), meaning that an analyst should not, for example,
start with the preconception that simply because we are
analysing an audit the participants view themselves purely
as ‘auditor’ and ‘auditee’. It is essential to ﬁnd features of
the talk that show that this is how they are orientatedtowards the interaction. In fact, in the context of thisresearch, auditor and auditee are only temporary roles, evenin the institutional context. Other roles, such as senior|

13

manager, junior staff member or even ‘colleague withwhom I disagreed in a meeting yesterday’, may also play arole.
Any attempt to apply CA to institutional interaction and to
draw conclusions from the research must retain the‘specimen perspective’ essential to the approach. As ten
Have points out, transcripts are ‘to be considered as
specimens of their kind and not … as either statements
about or reﬂections of a reality “out there”’ (2007:36). This
research therefore is an attempt to get at what really
happened in a sample of audits. The hope is that using anemic perspective can reveal something about the process as
it is realised, which can then be of use to those involved inplanning and carrying out audits.
A common CA approach, often taken when analysing
institutional interaction, is to focus on one interaction as a
primary example of relevant features and to look to otherinteractions for examples of similarities and differences(Drew 1992, ten Have 1999); this is the approach taken
here.
It can be argued that the context of the audits being
analysed plays a role in determining the approach of theanalyst. In some ways therefore, prior knowledge of theaudit process, particularly some of the issues that arise withregard to the relationships between the participants, could
almost be regarded as a disadvantage. The aim of thisresearch has therefore been to, as ten Have writes, ‘bracket
conventional ideas during the actual analysis of the data’(1999:186–7). However, the second stage of CA is theelaboration of these sequences in order to come up with ‘aset of formulated “rules” or “principles”, which participants
are demonstrably oriented to in their natural interactions’(2007:150). In this second stage, reference to the broader
context is invaluable. A discussion of context is essential
because, as a piece of applied CA research, this article
hopes to draw broader conclusions about internal auditing,and offer tentative advice to auditors based on the ﬁndings.
However, due to the very nature of CA, this piece of researchdoes not and indeed cannot, make claims about the
typicality of the audits it is using for its sample.
This research followed the conventions of CA as outlined
above and started with an attempt at ‘unmotivated looking’at the transcripts without, consciously, taking contextual
features into account. Broader contextual features were only
considered after the initial notes had been made, and wereused to aid the elaboration of the analysis and to draw
broader conclusions. However, analysis and explication
have been combined below as is standard practice in CA
research, to avoid repetition and to make it easier for non
CA specialist readers to interpret the ﬁndings.

Context of research and audit procedures
Cambridge ESOL instituted a formal quality management
system, complete with a system of internal audits, in 2006,
as part of a project aimed at gaining ISO 9001 certiﬁcation
which was achieved in May 2007. The internal standard
operating procedure for internal audits states that the
purpose of internal audits is to evaluate whether:
organisational procedures and policies are being followed;©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

14|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

the QMS meets the requirements of ISO 9001; therequirements of the organisation’s Quality Policy and Key
Business Objectives are being achieved; that processes are
effectively managed and implemented so that they deliver
the required output; and that there are opportunities for
improvement (OFIs). Most internal audits will therefore checka process for conformance, for effectiveness and for OFIs,
also reporting on evidence of good practice.
The audit programme for the organisation is determinedby the quality manager, in consultation with senior
management and other staff. A responsible auditee is
identiﬁed for each audit; this is the person who is identiﬁed
as being the owner of the process being audited (although
in a complex organisation with processes that stretch across
many parts of the organisation this person is often only
nominally ‘in charge’ of the whole process). The audit begins
with a pre-meeting, which takes place 2–3 weeks before
the audit and at which the scope of the audit is discussed
and names are put forward for people to be interviewed.
Following the pre-meeting, the auditor will read
documentation about the process and will formulate achecklist of questions to be asked during the audit.
The audit itself usually takes place on a single day, startingand ending with a meeting between the auditor and the
responsible auditee. During the day, the auditor interviews anumber of individuals involved in the process, asking
questions from the checklist and asking to see evidence,
where necessary, that procedures are being followed andthat the process is effective. Following the audit, the auditor
prepares a draft summary report containing details ofstrengths, opportunities for improvement and nonconformances. This report is discussed at a closing meetingthat is held between the auditor and the responsible auditee
within seven days of the audit interviews taking place. The
report is then ﬁnalised and is circulated to all relevant
parties. There are further procedures in place to ensure that
all non-conformances are addressed.

Data collection and transcription
For this study, three internal audits were recorded in their
entirety, a total of over 7 hours of recorded material. One
interview was selected from each audit for transcription.
All three interviews (A,B,C) are with people who have someform of supervisory role in their process and all interviews
have two auditors: a lead auditor (OR) who planned theaudit and took the lead in asking questions and a secondary
auditor (O2) who occasionally asked follow-up questions.
All lead auditors were relatively inexperienced at the time of
the recording: 1.5 years (A), two years (B), and a ﬁrst audit
as lead auditor (C). None of the auditors had experience ofauditing prior to becoming internal auditors in Cambridge
ESOL. The lack of experience raises some issues oftransferability with regard to organisations that have very
experienced auditing teams; issues such as this can beaddressed through further research.
Each auditee (EE) has either a supervisory or managerialrole which enabled the analysis to focus on how auditors
and auditees deal with the planning of processes and not
merely their execution. The audits are of different processes

and the auditees are at different levels within the company.
In Audit A the auditee has been working with this particular
process for about 10 years within an area that wasundergoing signiﬁcant reorganisation at the time of theaudit. Auditee A supervises administrative staff and plays arole in determining how the process is carried out. In Audit B
the auditee is quite involved in policy-making and this audit
attempts to take a high-level approach to the ‘core’ process
being audited, whereas the other two audits concern subprocesses. Auditee C is at a similar level in the organisation
to Auditee A, however she was recruited into this position ayear before the audit took place, and had been instrumental
in suggesting and implementing changes to procedures.
Another difference manifested itself in the attitudes of theauditors and auditees; the dynamic between them was quite
different in the three audits, something outside the scope of
this article.
The data was transcribed using standard CA conventions
developed by Jefferson (2004) with some minor
amendments to make them readable to non-CA experts.
The analysis focuses on the central information-gathering
part of the audit, where the auditor asks questions ofindividuals involved in the process.

Analysis
The summary of the analysis presented below focuseson the role that auditors and auditees play in an audit.
It demonstrates that all participants clearly orientate
themselves to the role of auditor and auditee, and looks inmore detail at similarities and differences in the way thatdifferent auditors and auditees perceive their role in theaudit which is important, as these perceptions in turn reﬂect
participants’ beliefs about the purpose of the audit and of
the quality management system. We look in turn at
asymmetry, membership categorisation, reaching mutual
understanding, pursuing the agenda through questioning,
the use of ‘sorry’ to assert auditor authority, and explicit
feedback. Further areas were described in the full study.
Asymmetry

All three data extracts are immediately identiﬁable asinstitutional through the asymmetry of the interaction(Heritage 1995:164). The vast majority of talking in all three
extracts is done by the auditees. This is consistent with thefact that the auditee is describing the process. The auditor’s(OR, O2) role is to ask questions; in fact, this is so much theauditors’ role that auditees very rarely do it. In these
extracts, auditees only ask questions for two reasons:• to initiate repair:OR
EE

and can you very brieﬂy give me an overview of the [name]2
process.
an overview of the whole process?

• to ask a question of a third party in order to giveinformation to the auditor:

2 Please note that information in square backets [ ] is anonymised material.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

EE
E2
EE

[E2]?
yeah
[E2], remember the problem we had with, with er SYSTEM notworking?

Audits A and C begin with auditor questions, prior to any
description of the process by the auditee. This clearly
indicates the auditor taking charge of the process andlimiting the scope of the discussion. Audit B is slightly
different in that the auditor does not ask his ﬁrst question
until line 76, by which time the auditee has spent several
minutes explaining the process. However, the intervention inline 76 shows that the auditor is willing to interrupt quite
assertively to direct the audit:

76

OR
EE
OR

Can I, sorry
sorry
um so do you::: (2) do you have um, do you have um targets,
performance indicators? things to, to measure the processagainstIt is possible to account for the auditor’s prior lack ofquestioning by arguing that the auditee was simply saying
what the auditor wanted to hear. As soon as the auditee’sturn became irrelevant to the auditor’s perception of thegoals of the audit, he interrupted. What is more, EE accepts
the interruption without question and answers the question,
thereby indicating that he understands this to be theprerogative of the auditor.
There is strong evidence that the participants are fulﬁlling
the roles of ‘auditor’ and ‘auditee’ with a very high degree of
uniformity. The person asking the questions is the personthat tends to hold the power in an interaction. Further
evidence of the extent to which the auditor retains thispower throughout the audit will be examined below, as will
the use the auditor makes of the advantage the institutionalsituation gives him/her.
Membership categorisation

Looking in more detail at how roles are established, themanner in which auditors use membership categorisation
devices to introduce the audit varies across the audits. We
will now look in more detail at what is said before the ‘audit
proper’ starts, in order to examine how the auditor andauditee establish the nature of the process that they areabout to enter into. The analysis will argue that although themanner of membership categorisation is quite different,
what it does is essentially the same.
Audit A has the longest preparatory sequence prior to thestart of the audit. It is not until line 36 that the ﬁrst questionis asked:
OR

31

36

O2
EE
OR
OR

=no not really. it’s just, it’s just kind of interesting because
mm uhm obviously you’re going to have a different uhm sortof outlook from [Manager] and a different outlook from fromfrom the subject ofﬁcer or whatever so it’s quite interesting
to see the different um sort of facets. um the ﬁrst question
I have [EE] °was° oh sorry I should mention the timing
shouldn’t I (2) um (paper rustling)(inaud) 10.50.
I’ve got your timings according to that.
10.50 yeah
is what planning activities go into the [name] process.

|15

The main purpose of this introduction seems to be to set
the context for the audit and to reassure the auditee. The
auditor is very scrupulous about ensuring that the auditee isin full possession of the facts before the audit starts. The
auditor’s zeal in ensuring that all is correctly done leads to
him interrupting his own question by initiating a repair
sequence (line 31: ‘oh sorry I should mention the timing
shouldn’t I’). Auditor A seems happiest working from anassumption that he and the auditee are starting from scratch
and that EE has no prior knowledge of the audit procedure;
this continues even when EE suggests that in fact he does
have prior experience. The auditor’s decisions to stick withhis predeﬁned script are a feature of this audit. There is also,
however, a palpable sense of the auditor making sure thatboth participants are completely sure of, and comfortable in,their roles before the audit starts. Auditor C takes a very
different approach to membership categorisation:
OR OK (3) we’re going to look at planning and day-to-day
management of the [name] process. I’m sorry we’re a bit late.
EE That’s alright.
OR Umm can you just tell me brieﬂy what planning activities
you’re involved in.
EE Um, well really it’s …

The introduction is very cursory: there is a reminder of the
purpose of the audit and an apology for running late, whichis accepted. After that, OR asks the ﬁrst question. Here,OR’s use of ‘OK’ in line one serves as a membershipcategorisation device. Words such as ‘OK’, ‘so’ or even ‘um’
(as in the previous extract in line 31) act as a break in theconversation and signal a change of topic, signalling in thiscase to the other participants that the real business is about
to start. It should be pointed out here that there may havebeen small talk going on before the recording started, but
the switching on of the recording prompts the ‘OK’ and thestart of the audit ‘proper’. The explanation for the difference
between these two audits can be explained by analysing thestart of Audit B:
OR

6

EE
OR

EE
OR
EE
OR
EE

OK (.) umm (3) now w, we talked a little bit about this before
but as I mentioned this, the, the purpose of this audit is thatit’s (.) it’s a high level process audit of the whole [process]system. so (.) um what we want what we what we I’m gonna
try and stop myself from doing is going into too much detailabout the nuts and bolts. but what I’m interested to talk to
you about as the, the senior manager of the process is thereporting[]
mm hmmmechanisms of the process. so how you know about what’s
happening. u::m and about[]
Mm
things like process improvements um and things like that.
Yeah
Um
(2)
OK, well, (2) I mean there are a number of means by which …

As in Audit C, Auditor B starts with ‘OK’, to establish thatthe audit has started and that the appropriate roles should
be adopted. Following this OR explicitly starts with areference to a previous conversation. It is clear that all
participants come with prior knowledge to these audits,©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

16|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0often as a result of prior conversations, and it is this priorknowledge that both this auditor and the auditor in Audit C
are cueing with the ‘OK’. Although the auditor in Audit B
does go on to give background about the audit, crucially his
explanation is based on the difference between a high levelprocess audit and other types of audit in which he would go
into detail about ‘nuts and bolts’ (line 6). What is clear here,
therefore, is that OR in audit B assumes that EE has
knowledge of what audits are normally like; this enables
him to make the contrast that he makes.
Although done in very different ways, reﬂecting thedifferences in the approach of the auditors, there isevidence in all three audits of the auditor taking necessary
steps to preface the start of the audit with devices formembership categorisation, ranging from the minimal ‘OK’
to a lengthy explanation.
Reaching mutual understanding

Both participants in an audit have to work towards mutual
understanding of what is being said. On some occasions thisrequires the use of repair sequences; the most common
repair type is ‘other initiated self repair’, where the auditor
requests clariﬁcation from the auditee who then supplies it.
This is sometimes brief:
EE
OR
EE

=so they are created bank by bank.
do you mean paper by paper or pr…
paper by paper.
[]
paper by paper.

These repair sequences can also be signiﬁcantly longer,
an example in Audit A lasted 26 lines.
The auditees in particular seem keen to ensure that theyleave open the fact that what they are telling the auditor iseither already known to the auditor, or doesn’t need saying.
The method used for this is the use of ‘obviously’ which isused in all three audits, occurring twice in Audit A and threetimes in Audit C. In Audit B in contrast, the auditee uses theword ‘obviously’ 25 times. The use of ‘obviously’ is verymuch the same as in other audits, although it is on two
occasions repeated in the same turn, for example:
EE

so that’s obviously one indicator and obviously what what
what we’re looking at there is number of entries against you
know the comparable session last year. and whether it’sThis increased use may simply be a feature morecommonly employed by this individual when explaining
something.
In addition to reaching mutual understanding regarding
the auditee’s description of the process, there is also aneed to reach mutual understanding of what it is that theauditor wants. On one level, this occurs through theauditor initiating repair sequences by rephrasing hisquestions if he feels that EE has not interpreted thequestion as it was meant. Interestingly, these repair
sequences can also result in clariﬁcation of a point of fact
rather than of the auditor’s question, as occurs in Audit B:
EE

OR
EE

so that’s obviously one indicator and obviously what what
what we’re looking at there is number of entries against you
know the comparable session last year. and whether it’s[]
mm hmmmore or less than we would ex, expect. y’know so that’s a

102
OR
OR

EE

OR
EE

[
]
mm hmmso is that, that’s the performance of the organisation as awhole I suppose.
[]
it is, it is as a wholeyeah but I mean it’s also an indicator that we’re actually
getting the entries on the system as well you know, I mean if
there’s, if there’s a shortfall for instance on [EXAM] you know
we would ask (1) is this because we’re failing to get theentries on the system or is it because we haven’t got enoughentries, I mean that’s a bit crude, but[]
mm hmmthat’s, those are basically the two reasons why you haven’t
got enough entries

On this occasion OR indirectly initiates a repair sequencein line 102 by hinting that EE has misunderstood his initialquestion, asked in line 75 about targets and performanceindicators to measure the process against. EE is able toclarify in his next turn that in fact, what he is describing, isan answer to OR’s question. OR accepts EE’s reinterpretationand continues with his questioning.
Pursuit of agenda through questions

Boden points out that it can also be valuable to isolate one
person’s turns, in order to shed light on their strategy andsee what their agenda is and how they set about achieving it(1994:175–7). This section will look at how Auditor A tries
to achieve his agenda through his questioning style. In his
ﬁrst six turns OR is focusing his question:

T2
T3
T4

OR
OR
OR
OR

T5

OR

T6

OR

is what planning activities go into the [name] process.
=right= right
do you mean paper by paper or pr…
paper by paper.
ok (6) um obviously there’s, I gather there’s quite a ha halarge volume of through put through this ofﬁce. u:::m
uhm is there any other planningi, is there any other planning which goes into sort of um
looking at that workload for the coming year
mm hm. I think I was thinking in in in terms of like, I gather
that you know sometimes there’ll be new products coming
along, and how do you sort of plan for them, for those or or
do you just deal with them when they appear

Turn 6 is dealing with an element of planning notpreviously mentioned; OR is using the question to focus EE’s
explanations in a different direction. OR’s turn design in thissection, namely the use of the past tense, ‘I was thinking interms of …’, indicates that he is pursuing a predeﬁned
agenda and that he is merely clarifying to EE what this is.
Auditor A is essentially indicating that Auditee A’s previous
response was not what he wanted. If OR is pursuing anagenda that seems to be predeﬁned, it is worth asking why
he did not simply ask the question he wanted answered in
the ﬁrst place. This is a situation in which the auditor may
have a preference for co-implicating the auditee in his
judgements by giving the auditee the opportunity tospontaneously bring up the issues that the auditor would
like raised. If the attempt to co-implicate fails, the auditor
can pursue his agenda anyway.
In turns 7, 8 and 9 Auditor A follows up with questions
about the impact of what EE has said, and further questions
about how it is evaluated:©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

AT7 OR
T8 OR
T9 OR

mm hmmand what sort of issues does that create for you.
and how is that sort of (.) erm the impact of those erm sort ofshort (2) the impact that those sort of late products have
how is that evaluated. y’know cos it sounds as if that might
erm imply a risk to

In turn 8 OR is asking EE to expand on any issues he may
have. In his next turn, OR asks how the impact is evaluated.
OR’s question is complex, leading to his recapping it at theend: ‘how is it evaluated’. Interestingly OR then goes on tojustify his question: ‘y’know cos it sounds as if that might
erm imply a risk to’. EE then picks up on the follow up, rather
than the initial question: ‘it it it is there is a risk’. However,this creates some ambiguity regarding the question, as
illustrated by EE’s response:
EE

A85

it it it is there is a risk. (1) um they all appear, because
they are given tailored schedules to the need (.) to getthem done in a short space of time they all then start
appearing on late lists. (.) erm and late reports that we
generate from [SYSTEM]. (.) and they have to just be
watched and monitored as they go through. (.) aah(y’know) the only thing I would say p’raps is that
fortunately we don’t have lots of these all the time. nh
they just appear occasionally. er but it it they just haveto be managed with care.

AT10 OR
T11
T12
T13
T14

OR
OR
OR
OR

can you show me um the obs of how how a late report from
[SYSTEM] …
uh huhwhat’s the two ha ha, ha ha, ha haso why isn’t necessarily a paper which is overdue not late.
so there’s sort of contingencies

In turns 15 to 20 OR returns to the topic of evaluation andpresses the issue with very speciﬁc questions:
AT15 OR
T16 OR
T17 OR
T18 OR
T19 OR
T20 ORand who, i i is it you who does this evaluation.
=who’s we sorry. you and (inaud)
who, who’s management.=
what about (.) [Director].
What about [Manager].
thank you.=

Auditors use ‘sorry’ to assert their authority as the controllerof the agenda and to announce the fact that normal rules are
suspended. Using a word normally associated with apology
enables the auditor to do this in a manner that acknowledges
that he is ‘interrupting’ something that might otherwise
continue. A broader look at how the term is used supports
this view. Across all three recordings, ‘sorry’ appears 16
times, used by the auditor on 13 occasions, making ‘sorry’appear to be very much an auditor device. It is possible tocategorise all auditor uses as apologies of a sort, often as ameans for the auditor to manage deviations from the normthat are caused by the situation of the audit. In addition to
examples of ‘sorry’ used in very conventional ways, such as
when auditor A is rephrasing a question (line 218) or when
auditor C is apologising (line 2), there are also several
distinct uses of ‘sorry’ which perform interesting roles interms of how the auditor exerts his authority. These include:• asking for repetition due to making notes:
OR=so what was the process sorry? I was still making my noteson the previous thing.

• making requests and asking questions, indicating acertain degree of awkwardness:
U::m , which is, which is, which is actually on my, on my listof things to ask you about. um can I have a look °sorry°?
close to but within [EXAM] closing dates (inaud – reading the
document)

This is both a change of topic and could be seen as an‘interruption’ example or could be seen as a request to see
an important document, which is usually only seen by
senior management within the organisation.
• interrupting to pursue agenda:
EE

OR
OR
EE
OR

that particular amount of problems. (2) ah::m so (2) so that’s
the sort of the conventional (1) issue reporting. In terms of[]
Can I,Can I, sorry
Sorry
um so do you::: (2) do you have um, do you have um targets,
performance indicators? things to, to measure the processagainstOn this occasion, EE repeats OR’s ‘sorry’, suggesting that
he is aware that he has stepped outside what he should bedoing in his role. He is essentially acknowledging OR’s rightto interrupt.
• abrupt change of topic to pursue agenda:
EE

OR

The technique of drilling down to get to the point inquestion, with some detours for clariﬁcation, is foundthroughout all three audits. It is possible, therefore, todiscern a consistent technique that the auditors in these
extracts use to achieve their agenda. Within this context, we
will now examine what is revealed about the role of theauditor and auditee while this agenda is being achieved, as
well as how the means the auditor chooses to achieve his
agenda have an impact on the relationship between theauditor and the auditee.

17‘Sorry’ to assert auditor authority

B401 OREE has picked up on the last thing said. By giving therationale for his question, OR has moved the focus onto
the risk rather than the method of evaluation. EE’s response
reinforces the sense of ambiguity regarding OR’s question.
He does not feel it necessary to ask further questions
about the risks involved, but he does not feel the need torephrase his initial question either. This ambiguity is not
resolved until turn 15. Instead OR seeks in turns 10–15 toclarify the different types of report produced by thedepartment.

|

EE

obviously with other things that are being proposed by[Person] at the moment we are in a state of ﬂux.
(8)
and (.) um (.) what’s, sorry [EE] what’s your responsibility
with regard to the [name] process?
awawa in in a nutshell I would say to ensure that all thepapers are ultimately printed and checked and printed by therequired date.

Here the auditor is shifting the topic away from an issue
that EE has raised as a concern in order to pursue his own
broader agenda, using ‘sorry’ to acknowledge the fact that
he is doing this. The auditors’ use of ‘sorry’ throughout theaudits shows sensitivity to the issue of the auditees’ ‘face’(Goffman 1972). The use of a term usually used for©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

18|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

apologising allows the auditor to assert authority over theproceedings, while at the same time acknowledging with aform of apology that this assertion of authority places theauditee in a subservient role. The use of ‘sorry’ helps toameliorate this effect and maintain relationships throughout
the audit.
Given the importance of ‘sorry’ as described above, it is all
the more surprising that there is only one instance of it in
Audit C, a conventional apology:

Nevertheless, OR gives no feedback as to whether he issatisﬁed with this explanation.
In fact, in only one of the three audits are opportunities for
improvement mentioned at all – in Audit B where two comeup in quick succession:
EE
OR
OR

OR

OK::: (3) so we’re going to look at planning and day-to-day
management of the pretesting process. I’m sorry we’re a bit
late.
EE
OR

The reason ‘sorry’ does not appear elsewhere in this audit
is that the auditor does not attempt to assert her authority
using any of the methods stated above; this means that
there are no occasions where ‘sorry’ might be used.
Although there are topic shifts, they tend not to be abrupt
and seem to occur when a line of questioning on a topic
has been ﬁnished, rather than due to the auditor ‘forcing’
the issue.

EE
OR

Explicit feedback

Audit ﬁndings are delivered in terms of strengths,
opportunities for improvement (OFIs) and nonconformances. Asbury and Ashwell (2007:178) argue that aneffective auditor gives feedback on ﬁndings as they occur,
thereby encouraging buy-in from all participants, and in fact,
the guidelines for Cambridge ESOL internal audits state that
feedback should be given when OFIs and non-conformancesarise during the audit. There are several instances when
explicit feedback could be given but is not, for example:
EE

OR
EE
OR
EE
136

142

OR
EE

OR
EE

OR

we. [Administrator] and I. (1) [process]. °shall I
say° (1) um and pass copies to:: management.
(6)
who, who’s management.==[Person A]. [Person B]. an::d [Person C] is it at least that’s
the [position in organisation] or whatever they’re called.
what about (.) [Person D].
[Person D] doesn’t have them any more. (4) he used to.
(5)
What about [Person E].
[Person E] doesn’t have them on a regular basis I believe.
I don’t believe [Administrator] sends them to [Person E]
I think she sends them occasionally. but [Administrator] will
conﬁrm that. cos [Administrator] deals with them each week
(5)
thank you.==we do separate reports that erm, through [Person E] and[Person A] that goes to [Person D] highlighting particularareas of concern as to y’know routine this is a couple of dayslate.
so is this the way that you predict problems to prevent theiroccurrence. u:::m

The fact that OR mentions [Person E] in line 136 indicatesthat he believes that the fact that she wasn’t mentioned as arecipient of the report is an issue. This fact is backed up by
EE’s extended response in lines 142–144. However, OR
makes no explicit comment, and in fact initially attempts toclose out the questioning with a ‘thank you’ indicating that
he is ready to move on. EE keeps the issue alive andindicates that he feels he needs to justify the fact that
[Person E] does not receive the reports with his further turn.

…

EE
OR
…
EE

OR
445 OR

EE
OR

it’s always risky taking an example about checking your data
ﬁrst.
it is isn’t it.
(general laughter)n, not, not risky in the sense that now y, now I mean I’ve put
this down as an opportunity for improvement. but it’s a,y’know it’s on it’s on the minor very easy to ﬁx sort of
[
]
[
]
Yeah
yeah
basically, make sure it’s, that the peop, that the, that the,I mean on the one hand it’s basically is it updated onto, is it
uploaded onto [SYSTEM] often enough but on the other handit’s basically is everybody looking at the same document all
the time, because if[]yeahdifferent people are looking at different documents then thatcan be, that could be more[]that’s right yeahof an issue. so this has gone up to the [SYSTEM]?
…
°well it didn’t prove anything much anyway°. No,
unfortunately I don’t have a, I don’t have the actual link oranything to it.
[]
u::::mso now it’s gone to the, so now it’s gone to the um, I’ll putthat down as an opportunity for improvement when I writethat down. um now, now I, now, now that it’s on the[SYSTEM][]mm, ah right[SYSTEM] so it’s it goes, goes to the [meeting] presumably.
(1) umm

As discussed earlier, OR brings up the ﬁrst opportunity for
improvement in a somewhat apologetic way. He seeks
almost to dismiss it as soon as it is mentioned: ‘but it’s a,y’know it’s on it’s on the minor very easy to ﬁx sort of’. OR
does however go on to give a rationale for why there is an
issue with documentation: ‘because if different people arelooking at different documents then that can be, that could
be more of an issue’. Later in 443, he notes the fact that EE
cannot link to a certain document and puts that down as an
OFI as well. Even more so than on the last occasion, this OFI
is mentioned in a way that gains it as little attention as
possible; OR includes it as an aside in the middle of anotherutterance that starts beforehand with ‘so now it’s gone to’and ﬁnishes with ‘the [SYSTEM]’. OR’s tentativeness isfurther illustrated by the frequent repetition of ‘now’ in his
turn. EE acknowledges the OFI and the interaction swiftly
moves on.

Summary of ﬁndings
Institutional nature of interaction
The audit interviews analysed share clear features whichmake it reasonable for us to label and analyse them as aparticular type of institutional interaction, namely the©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0internal audit interview. It is clear that the participants bringcertain expectations to the process and that these form thebasis of the interaction. These are:
• Participants assume the role of ‘auditor’ and ‘auditee’,independent of other roles in the organisation. The taking
on of these roles is marked at the beginning of theinteraction.
• The auditor asks the questions, unless clariﬁcation isrequired, and therefore is in control of the interaction.
• The auditor controls the agenda and is free to follow up
what they view as important (although the auditee’s ‘face’must be considered). The auditee accepts that the auditor
will choose what to follow up on, and that it is theauditor’s right not to follow up on concerns raised by theauditee.
Auditor–auditee relationshipThis research was carried out with a view to gaining insights
into the relationship between auditors and auditees and inparticular why, ‘in many organisations … auditors are stillviewed with suspicion and even a degree of disdain by those
who are likely to be audited’ (Asbury and Ashwell
2007:174). The audits analysed here ﬂag up some pertinentissues in this area. In all three audits, auditors did not
generally give feedback to auditees regarding the opinions
they were forming of a process. Asbury and Ashwell refer tothe ‘no surprises’ technique (2007:133). Their view is thatkeeping people informed of ﬁndings at the earliest possible
opportunity, even during the interview stage, is the mosteffective way of ensuring buy-in from auditees. In the audits
analysed here, this seems like something of an opportunity
lost.
From the perspective of the auditor–auditee relationship,a number of issues were identiﬁed, mostly from Audit A.
Firstly, analysis revealed that the auditor was primarily
interested in management structure and systems within theprocess being audited. However, this was not made explicit
to the auditee. This resulted in the auditor rather abruptly
cutting off lines of questioning to the evident confusion of
the auditee. Secondly, as a result of the auditor’s focus on
management issues, he fails to follow up on issues raised by
the auditee; the auditee raises many explicit problems withthe process but these are generally dealt with ﬂeetingly, if at
all. This has obvious negative implications in terms of theauditee feeling he has a stake in the process.

Implications
As this is a piece of applied ethnomethodology, the present
research has practical implications for quality managers
rather than methodological implications for conversation
analysts. The aim is to use these audits to hold a mirror up
to some issues that may arise in the process of internal
quality auditing in any organisation. Furthermore, thisresearch does not seek to draw broad conclusions aboutauditor behaviour in general. Instead, it give examples ofauditor behaviour from a limited data sample. It is the job of
the reader to determine the extent to which issues that arisein these audits may also arise in their own context.

|

19

The most apparent use for these ﬁndings would be as a
training tool for quality managers and audit teams within
and beyond Cambridge ESOL. For example, quality managers
can use the research as a starting point for someethnographic research into internal audits that focus on, forinstance, whether there is evidence that auditees feel they
are not given enough feedback during audits, or whether
they feel that their own concerns are not given enoughweight by the auditor. Quality managers could also use theissues identiﬁed as an awareness-raising exercise for traineeand practising auditors, who could discuss the decisions
made by the auditors in the extracts and how it reﬂects on
their notion of what an audit is. They could then compare
this to what they believe they do, and what they believe theyshould be doing. This could lead to a rich discussion of what
an auditor’s role ought to be and how this should be
realised. Unlike the externally produced training material
mentioned earlier, there is room for disagreement andambiguity, and the potential for all participants to develop
their views on internal auditing outside a prescribed norm byusing extracts from real audits.
Finally, the issues that arise in the audits studied here
bring up the issue of organisational culture. Although theauditors make an effort to establish the roles of auditor andauditee, there is evidence that, in spite of the auditor’s very
thorough explanation of the audit process, auditor andauditee have different views about the broader purpose of
the audit. These different views may be a result of differing
perceptions of how the hierarchy of an organisation works.
It may be instructive for quality managers to explore how
explicit the quality manual is about the nature of decision
making within an organisation and, within that framework,
the precise role played by the internal auditor and auditee.

Conclusion
There are many possibilities for further research in this ﬁeld.
A broader ethnographic perspective may give more insights
into the auditing process. Whilst this research has focusedon the audit interactions themselves, there are othersources that could be used as a basis to compare ﬁndings
from a CA approach with the views of the participants andthe outcomes of the audits. For example, it may be possible
to use the internal audit summary reports that detail themain ﬁndings of the audits to corroborate the ﬁndings of theresearch regarding the auditor’s agenda, and also theauditor’s views on what the auditee has said. It could also
be fruitful to conduct post-hoc interviews with participants
to discuss their attitudes towards the audit. This may
provide more information about whether an auditee was
genuinely frustrated by an auditor’s actions.
A broader piece of research may choose to use a larger
number of recordings, possibly from a range of sources indifferent companies. In particular, as mentioned above,
Cambridge ESOL auditors were all relatively inexperienced at
the time of the study. Research that focused on moreexperienced auditors may yield information about whether
the ﬁndings of this research are related to the inexperienceof the auditor group. This would enable the researcher todraw broader conclusions about internal auditing and may©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

20|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

make the research more easily transferable. Although thisresearch is limited in scope, it is hoped that the ﬁndings arerobust and transferable enough to provide food for thoughtfor those involved in internal auditing, and help to develop
their ideas about how best to ride the ‘audit roller coaster’in their own organisations (Asbury and Ashwell 2007:82–8).
References and further reading
Asbury, A and Ashwell, P (2007) Health and Safety, Environment andQuality Audits: A risk-based approach, Oxford: ButterworthHeinemann.
Bergmann, J R (1992) Veiled morality: Notes on discretion in
psychiatry, in Drew, P and Heritage, J (Eds) Talk at Work: Interactionin Institutional Settings, Cambridge: Cambridge University Press,
137–62.
Boden, D (1994) The Business of Talk, Oxford: Blackwell.

Goffman E (1972) Interaction Ritual: Essays on Face-to-Face Behaviour,
London: Allen Lane.
Heritage, J (1995) Conversation analysis: Methodological aspects,
in Quasthoff, U M (Ed.) Aspects of Oral Communication, Berlin:de Gruyter, 391–418.
Heritage, J and Seﬁ, S (1992) Dilemmas of advice: Aspects of thedelivery and reception of advice in interactions between healthvisitors and ﬁrst-time mothers, in Drew, P and Heritage, J (Eds) Talk
at Work: Interaction in institutional settings, Cambridge: Cambridge
University Press, 359–417.
Jefferson, G (2004) Glossary of transcript symbols with an
introduction, in Lerner, G H (Ed.) Conversation analysis: Studies
from the First Generation, Amsterdam: John Benjamins, 13–34.
Juran, J M (1988) Juran on Planning for Quality, New York: The Free
Press.
Juran, J M (1989) Juran on Leadership for Quality, New York: The Free
Press.

Boden, D and Zimmerman, D H (1991) Structure in action: An
introduction, in Boden, D and Zimmerman, D H (Eds) Talk and Social
Structure: Studies in Ethnomethodology and Conversation Analysis,
Cambridge: Polity Press, 3–21.

Meakin, S (no date) Internal QMS Auditor Course, Case Study Manual –“The Auditor from Hell”, Pera.

BSI (2008) Quality management systems – Requirements: BS EN ISO
9001:2008, London: British Standards Institution.

Pfannerstill, R (2005) The Progressive Audit: A Toolkit for Improving
Your Organisational Quality Culture, Milwaukee, Wisconsin: ASQ
Quality Press.

Deming, W E (1982) Out of the Crisis: Quality, Productivity andCompetitive Position, Cambridge, Mass: The Massachusetts Instituteof Technology, Center for Advanced Engineering Study.
Drew, P (1992) Contested evidence in courtroom cross-examination:
the case of a rape trial, in Drew, P and Heritage, J (Eds) Talk at Work:Interaction in Institutional Settings, Cambridge: Cambridge
University Press, 470–520.
Drew, P and Heritage, J (1992) Analysing talk at work: An introduction,
in Drew, P and Heritage, J (Eds) Talk at Work: Interaction in
Institutional Settings, Cambridge: Cambridge University Press, 3–65.
Francis, F (2008) ISO 9001: The history, in Quality world, October
2008, 16–20.
Garﬁnkel, H (1984) Studies in Ethnomethodology, Cambridge: Polity
Press.

Moeller, R and Witt, H N (1999) Brink’s Modern Internal Auditing,Somerset, NJ: John Wiley and Sons.

Psathas, G (1995) Conversation Analysis: The Study of Talk inInteraction, Thousand Oaks, CA: Sage.
Sacks, H (1992) Lectures on Conversation Vols 1 and 2, Oxford: BasilBlackwell.
Taormina, T and Brewer, K A (2002) Implementing ISO9001:2000: The
journey from conformance to performance, Upper Saddle River, NJ:
Prentice Hall PTR.
ten Have, P (1999) Doing Conversation Analysis: A Practical Guide,
London: Sage Publications.
ten Have, P (2007) Doing Conversation Analysis: A Practical Guide
Second Edition, London: Sage Publications.

Working with examination centres to encouragecontinuous high standards: The centre inspections
programme
CLAIRE MCCAULEY CENTRE MANAGEMENT MANAGER, CAMBRIDGE ESOLDAVID COLLETT CENTRE INSPECTIONS OFFICER, CAMBRIDGE ESOLIntroduction
It is essential that Cambridge ESOL works closely withexamination centres to ensure standardised examination
experiences around the world maintain security of
conﬁdential materials, and allow candidates to have theopportunity to perform at their best in the examination.
Cambridge ESOL works closely with centres by runningconsultations for all major changes and developments andby encouraging continuous self improvement in centres. The
inspection programme plays an important part in ensuringthat all centres are regularly monitored and given feedbackon their performance. Inspectors are given training to makesure that ratings and feedback are standardised. The resultsof inspections are analysed and fed into the developmentof new documentation, regulations and training.

The purpose and scope of centre
inspections
Inspections play an important part in Cambridge ESOL’s
quality assurance. They help to assess the quality of thedelivery of its examinations around the world. The visible
process of an inspection provides public assurance thatexamination security and integrity is upheld and©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

encouragement to centres to maintain high standards.
The centre inspections programme inspects live test
administration sessions to ensure accordance with theregulations as set out in the Handbook for Centres, which
is provided to all centres each year, and is available todownload from CentreNet (Cambridge ESOL’s extranet for
centres). As such, an Inspector will evaluate:|

21

Figure 1: The inspections cycle• the security of conﬁdential materials• the examination administration, including the checking ofcandidates’ photographic identiﬁcation• the examination rooms.
In addition to ensuring compliance with regulations,inspections promote improvement in centre performance.
An inspection may include observing a written paper, or
may focus on the delivery of the Listening test. An
inspection may also cover the Speaking tests, in which casean Inspector cannot be in the examination room during thetest, but checks the set up of the test room, thearrangements for marshalling candidates, the identity
checks and the timetabling. Inspections may also include a
computer-based test and involve checking that theconditions are appropriate, and there are sufﬁcient PCs.
An inspection may survey all the rooms in a venue, or
may concentrate on just one room, depending on thecircumstances. Inspectors also request to see the securestorage facilities and arrangements. The Inspector observes
the instructions given to candidates at the start of thepaper, the identity checks and the invigilation of theexamination. For written papers, the Inspector must stay forat least one paper, and often stays for subsequent ones,
especially if non-compliances have been observed. For
some non-compliances, it is possible for the centre to makechanges to its examination administration immediately andtherefore, where possible, the Inspector stays to observe
the improvements and include these in their report. Eachcentre is inspected at least once every three years. Morefrequent inspections occur if a centre needs additional
support such as advice on correct examination
administration.
In 2009, Cambridge ESOL introduced a programme ofgiving centres advance notiﬁcation of routine inspections.
By working together with centres, Cambridge ESOL aims toencourage self-improvement. Although the date and time
are not announced, centres may be advised that they will beinspected in the next examination session. Inspections arecarried out by a cadre of Inspectors around the world. Someare external consultants who work with Cambridge ESOL ina number of roles, others are based at local ofﬁces or
Cambridge itself. Each Inspector is given comprehensive
training.
The centre is ultimately responsible for quality assurance
at all its venues. This responsibility is laid out in the
Handbook for Centres and the Centre Agreement. An
Inspector may visit any venue run by any centre. Inspectionsof all venues should reveal the same degree of adherence
to Cambridge ESOL standards.
In many regions, there is a local co-ordinator who
manages the Inspectors in their region and who reports
directly to Cambridge ESOL’s Centre Inspections Unit.

Figure 1 summarises the inspections process. When acentre submits entries, the inspections team reviews whether
the centre is due for inspection, and contacts an Inspector tocheck availability and arrange a visit. The inspection planning
involves matching the availability of an Inspector and theopportunities for inspection, and where possible co-ordinates
visits to make best use of resources. After the inspection,a report is passed to the Centre Inspections Unit for review
and inputting to the customer relationship management
database. These records feed into the inspections planning
process in terms of the next date a centre is due for its
routine visit, and also in terms of whether a centre requires
early reinspection. Follow up actions are also carried out; forexample, writing to the centre and also following up with theCentre Support Unit. This process is a continuous cycle ofexamination entries, inspections and following up on theoutcomes of inspection visits.

Helping centres to prepare for inspection
Centre documentation

Cambridge ESOL gives the regulations to centres in
November of the preceding year. All examination centres
receive a copy of the Handbook for Centres, and otherrelevant documents, including the Examination Instructions
booklets, Notice to Candidates and Regulations. Centres
also have access to a comprehensive range of Frequently
Asked Questions (FAQs), and can call the Cambridge ESOL
Helpdesk if they have any further queries about
examination administration.
Training for centres

Cambridge ESOL offers a comprehensive training
programme to centres, including:• Face-to-face training: a number of training events are
conducted each year. These are normally held in
Cambridge, although they may be offered in otherlocations if required. Centres are trained on how toadminister the examinations according to Cambridge
ESOL regulations. For example, centres are shown how to©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

22|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

make entries, and how to ensure that venues meetrequirements. Excellent feedback has been received from
attendees at these events. The training gives new centres
and new Centre Exams Managers (CEMs) conﬁdence tostart running the tests. Established centres also ﬁnd it
useful to attend training for updates and refreshers.
• Support is also offered via telephone or personal visitsfrom Centre Support Ofﬁcers, experienced CEMs from thesame region, and also via web conferencing.
• Interactive training tutorials are available on Cambridge
ESOL Online which explain how to use the site for
examination administration.
In 2009/10 an online training site for centres and theirstaff will be launched. The site will contain modules andactivities designed to help centres learn and practise theirapplication of the knowledge gained.

must meet minimum professional requirements, and aretrained to record fair and accurate ﬁndings and to fulﬁl a
supportive role towards centres; encouraging improvement
where needed. All Inspectors are trained face-to-faceinitially and shadow an experienced Inspector before
conducting an inspection. Improved global internet accessmeans that Cambridge ESOL is planning to launch an online
training site for Inspectors. This will enable Inspectors toreceive the latest training and information and provide aplatform for sharing ideas/experience across the globe aswell as promoting a standardised approach worldwide.
In addition to this, Inspectors are also monitored once
every three years. The Inspector is shadowed as they
undertake a live inspection and assessed according to achecklist of key criteria. Feedback is given following theinspection and additional training given if necessary.
On the day of the examination

Self assessment

Cambridge ESOL provides self assessment checklists on
CentreNet. These checklists (one for written papers, one for
Speaking tests) include all the points from the inspections
form in a simple format. Centres can use these forms totrain supervisors and invigilators and to check that a venue
has been set up correctly on the day of the test. Feedback
about these self assessment tools has been
overwhelmingly positive.

The Inspector observes the administration of at least one
examination paper as well as checking the arrangements forthe storage of examination materials. During the inspection,
the Inspector completes an inspection report form that
promotes a standardised approach and provides feedback
in a structured manner to both centres and Cambridge
ESOL. The report form includes a checklist of 50 items thatcover key Cambridge ESOL regulations, and is broken down
into the following ﬁve sections:• Security of materials

Inspectors – recruitment and training
To ensure that standards are applied equally throughout
the world, all Inspectors undergo RITME (Recruitment,
Induction, Training, Monitoring and Evaluation). Inspectors• Examination rooms• Computer-based test arrangements• Conduct and supervision• Speaking tests.
Figure 2: Online training site forInspectors (under development)©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

The report form has space for Inspectors to write
additional information regarding each fault that occurs.
For example, if the Notice to Candidates is out of date, theInspector can note this and remind the centre to check they
have a current version ready for examination day. At theend of the inspection the Inspector advises centre
representatives of the ﬁndings. Feedback is positive andconstructive. An inspection provides an opportunity forcentres to raise queries and receive advice about
examination security and administration. Many Centre
Exams Managers use this face-to-face opportunity tocomment on matters relating to examinations. They
can note these comments on the report form. However,Inspectors are not required to make professional
judgements relating to syllabus, paper-speciﬁc elementsor teaching practices, and the Centre Inspections Unitpasses such queries to the appropriate department.
A copy of the report form is left with the centre so thatthey have a written record of the outcome and any noncompliances found. A copy of the report form is also
forwarded to Cambridge ESOL and the information isrecorded and used for reporting.

Following an inspection
Written feedback

All centres receive written feedback from Cambridge ESOL
about the outcome of their inspection. Centres awarded‘Satisfactory’ receive an email congratulating them on theiradministration of the examinations, while centres awarded‘In need of improvement’ are sent an email acknowledging
that while the majority of regulations were met, there aresome areas that need to be rectiﬁed.
Centres awarded ‘Unsatisfactory’ are sent an Action Plan
that lists the regulations that were not met. The centre must
write on the plan the actions it will take to put right the noncompliances identiﬁed. When the centre returns the Action
Plan, it is passed to the Centre Support Unit who can offer
further advice and/or training. A copy of the Action Plan is
also sent to the Inspector to take with them at the follow-up
inspection so they can check that these measures havebeen implemented.
Analysing inspection ﬁndings

Over 600 inspections take place each year, which
represents about one third of centres. This performance isin line with Cambridge ESOL’s aim to inspect all centres atleast once every three years. The ﬁndings from eachinspection are entered into a central database. This allows
Cambridge ESOL to monitor trends, identify regulations orcountries with a high frequency of non-compliance, and to
identify patterns across regions and examinations. Reports
can also show the ratings awarded by each Inspector, which
examinations are inspected most frequently and so on.
These reports are reviewed regularly by Cambridge ESOL’s
Centre Inspections Unit and also by senior management at
Cambridge ESOL in order to monitor centre and Inspector
performance, and to identify potential areas of|

23

Table 1: Most common faults when monitoring centre performance
Regulation

Number of occurrences

Full instructions read out

92

Current Notice to Candidates visible outside
examination room, not inside81

Current and correct Examination Instructions
booklet in examination room

70

Unauthorised material in designated area

70

improvement. An example of the most common faults areshown in Table 1.
Cambridge ESOL monitors these reports and consults
centres and Inspectors about why non-compliances are
occurring. This feedback is analysed and measures
are put in place to prevent/discourage non-compliance infuture.
The analysis of non-compliances and the reasons theyoccur feed into the development of new policies,
redrafting of current policies, rewriting of centre
documentation and the training of Centre Exams Managers.
Reports can also show the effectiveness of the centre
inspections programme. For example, reports have shownthat centres who receive regular inspection visits performbetter than those who are visited less often. This suggeststhat the inspections programme is successful in meeting
its aim of encouraging continuous high performance
among centres. The inspections team also works closelywith other teams in Cambridge ESOL to give feedback
about centre performance. For example, feedback about
centre performance is given to business development
teams.

Conclusion
Cambridge ESOL works closely with examination centres
to ensure that the administration of examinations meetsrequirements, and to protect the security of conﬁdential
materials. As part of this, Cambridge ESOL aims forinspections to be a means of assessing whether centres
are complying with regulations, and for providing feedback
to centres on any improvements required to theiradministration of examinations. All centres are inspected
at least once every three years, and they are given training
and support in order to successfully run the examinations.
This training can be face-to-face or via a website. Centre
inspections play an important role in the development ofpolicies, training and documentation to ensure that
centres have the best support and guidance possible,
and that candidates receive a high quality examination
experience. Any non-compliances noted at inspection arelisted on the report form which is left with the centre,
thus allowing for immediate action. They may also be
followed up in the form of an Action Plan which thecentre submits to Cambridge ESOL. The inspections teamcan produce reports which show patterns and trends
across regions and examinations, and can help identify
areas for improvement.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

24|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Auditing the quality proﬁle: From code of practice tostandards
NICK SAVILLE RESEARCH AND VALIDATION GROUP, CAMBRIDGE ESOLIntroduction
This paper reports on the work of the Association of
Language Testers in Europe (ALTE) and in particular the
ALTE Procedures for Auditing and the outcomes of audits
conducted between 2007 and 2009. It summarises the
background to the approach, presents data collected so far
and reﬂects on the lessons to be learned in implementing
this approach. The need to justify the use of language
assessments to stakeholders using codes of practiceand quality management systems (QMS) forms part of this
discussion.
The ALTE Procedures for Auditing derive from ongoingwork on the ALTE Code of Practice (1994) and draw on
approaches to auditing adopted by other organisations
such as ISO – the International Organisation forStandardisation.
The formal scrutiny of standards is the culmination of along process of working towards establishing audited‘quality proﬁles’ across the ALTE examinations. The aim of
the process is to allow ALTE members to make a formal,
ratiﬁed claim that a particular test, or suite of tests, has aquality proﬁle appropriate to the context and use of thetest, based on 17 parameters for establishing minimum
standards. Ultimately, this is to ensure that the assessment
is fair and meets the needs of the stakeholders inappropriate ways.
It is important to remember the context in which this work
has been carried out and in particular the wide range anddiversity of ALTE members, associates and afﬁliates:• 32 full and associate members: these are organisations
which include government departments, universities,
consortia and examination boards which have a role inassessing their own language.
• 27 languages are currently represented, including many
less widely taught languages.
• In addition, there are nearly 40 institutional afﬁliates withan interest in language education and language
assessment. A list of members is shown in Table 1.
The description of the development and application ofthis system ties in with recent discussions in the language
testing literature on the use of argumentation to support
claims of validity. The system is also an example of anotherkind of theory, ‘a theory of action’ or change processes,(see Fullan 1993, 1999).
A theory of action, capable of bringing about positiveand sustainable change, was needed by ALTE members
in order to raise standards and improve the quality of
their examinations. I would argue that the QMS approachand the auditing system have now provided such a theory.

Table 1: ALTE membership (2009)
27 languages

32 membersorganisations comprising government departments,
universities, consortia and examination boards

BasqueBasque Government

Bulgarian

University of Soﬁa ‘St Kliment Ohridski’

CatalanGeneralitat de Catalunya

CzechCharles University Prague

DanishDanish Language Testing Consortium

Dutch

CITO
Certiﬁcaat Nederlands als Vreemde Taal

English

University of Cambridge ESOL Examinations

Estonian

National Examination and Qualiﬁcation Centre

Finnish

University of Jyväskylä and National Board of Education

FrenchAlliance Française
Centre international d’études pédagogiques

Galician

Xunta de Galicia

GermanGoethe-Institut, telc GmbH, TestDaF-Institut

Greek

University of Athens

HungarianInstitute of the Eötvös Loránd University

Irish

National University of Ireland

Italian

Università per Stranieri di Perugia

LatvianNaturalisation Board of Latvia

Lithuanian

University of Vilnius

Luxembourgish Centre de Langues Luxembourg
Norwegian

University of Bergen

PolishJagiellonian University

Portuguese

University of Lisbon

RussianRussian Language Testing Consortium

Slovenian

University of Ljubljana

SpanishInstituto Cervantes and University of Salamanca

SwedishStockholm University

Welsh

Welsh Joint Education Committee

Ethics and principles of good practice
The early work of ALTE members in this area took place inthe 1990s when they addressed the question of what acode of practice might be like and what the underlying
principles should be.
This led to the publication of the ALTE Code of Practice in
1994, and at about the same time, the adoption of the ALTE
Principles of Good Practice.
A Code of Practice Working Group was set up in 2000 totake the early work forward in light of changes to the ALTE
membership, and advances in the ﬁeld of language testingin the 1990s. Since then this group has met regularly andthe main outcomes include the production of detailed
documentation updating the Principles of Good Practice,proposals for minimum professional standards and theimplementation of appropriate checklists and procedures
for monitoring those standards. The most recent work,
starting about three years ago, has focused on thedevelopment of the ALTE auditing system.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Methodological framework
The methodological framework which this work represents
is shown in Figure 1, with relevant reference documents
listed on the right:
The overriding aim can be summarised as follows:|25• deﬁne your mission, role of institution, future
ambitions/aims• assess what you currently do• identify areas in need of improvement• decide on measures of improvement and an action plan‘… to strive for overall fairness in the testing process, always acting in• carry out action to bring about improvement

good faith and avoiding negative impact, in order to provide a high• review progress and revise the action planquality of service to the clients and other test users’.

• carry out action to bring about improvement
However, the ALTE Code was an inspirational document
and not designed to assist language testing practitioners incarrying out their day-to-day work of writing andadministering tests, or in agreeing on what might beacceptable in terms of minimum standards for their work.
Having established the principles, and having provided
some practical tools to help ALTE members improve their
examination systems, the working group addressed theissue of how to put the principles into practice, how
improvements could be monitored and whether adequate
standards were in fact being met. In order to address thisproblem, and to seek consensus, it was decided that theappropriate paradigm for this activity would be that ofquality management systems.

Ethic

Code of Fair Testing Practices in
Education – 1988
ALTE Code of Practice – 1994
ILTA Code of Ethics – 2000

PrincipleStandard

AERA/APA/NCME Standards for
Educational and Psychological Testing –1999
ALTE Principles of Good Practice –1993/2001

Auditing• and so on …
The ALTE Code set out the public position in terms of theaspirations and ambitions of the association, as explained
above, but in adopting the QMS approach, members
undertook:• to understand the nature of their organisations better• to involve their stakeholders in making improvements tothe quality of their examination systems.
In effect, this involved an ongoing commitment to‘change management’ (innovation processes). It isaxiomatic in this approach that improvement is always
possible, even where good practice may already exist.
In QM systems standards are not imposed from ‘outside’,
but are initially established through the system itself andthe procedures to monitor standards are based onawareness raising and self-assessment in the ﬁrst instance.
External monitoring is introduced at a later stage to conﬁrm
that the minimum standards are being met.
In order to provide a practical tool to raise awareness of
those areas where change was necessary, the original Codeof Practice was reworked to function as Quality
Management Checklists; this re-designed format reﬂected
the four aspects of the testing cycle with which testdevelopers are very familiar:• examination development• administration of the examinations

ALTE QM System and checklists – 2002

• processing of the examinations including the marking,grading and issue of results

ILTA CoP – 2005

• analysis and post-examination review.

Figure 1: Methodological framework

A quality management system (QMS)
QM systems seek to improve the products and/or servicesof an organisation in order to meet the requirements of its
customers in the most effective way, and they go about
doing so in a well-planned and focused way. There aremany examples of QM systems being used in educational
contexts, and several of these were thoroughly reviewed by
the working group. This was summarised and thenextensively discussed by the full membership (for moredetails see van Avermaet, Kuijper and Saville 2004).
Effective QM systems usually involve a public statementin the form of a code of practice or charter, and an ongoingcommitment to the change process (improvement) typicallyinvolving the following steps:

The revised format provided four checklists which were
put into excel spreadsheets for ease of use and formembers to use as evaluation tools. These excel checklists
are available from the ALTE website (www.alte.org). In
summary, the QMS approach provided a supportive tool
and allowed those working with it:• to enhance the quality of their examinations in theperspective of fairness for the candidates• to engage in negotiations with their senior management
and sponsors in a process of organisational change
where necessary, and to ensure that resources are madeavailable to support ongoing improvements• to move from self-evaluation to external veriﬁcation inorder to agree and monitor acceptable standards.
By proceeding in this way ALTE members were madeconstantly aware of the different contexts in which theywork and of the various backgrounds from which the©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

26|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

different members come. Extensive discussions have takenplace around the question of how to achieve reconciliation
between diversity and professional standards which areacceptable to all. Since 2007 this has continued throughthe implementation of the auditing system.

Monitoring standards – Auditing thequality proﬁle
It was always envisaged that self-evaluation would need tobe supplemented by an external ‘auditing’ system. This wasdeveloped and piloted starting in 2005/6 and continued
with the ﬁrst cycle from 2007/8 seeing audited ‘quality
proﬁles’ established across a wide range of ALTE
examinations.
Taking the Code of Practice and QMS checklists into
account, 17 parameters for establishing minimum
standards were agreed, with the aim of establishing a
Quality Proﬁle for each exam or suite of exams, shown in
Table 1 in the article by Bolton in this issue (page 32).
The Quality Proﬁle is created by explaining how theexamination meets the minimum standards, and byproviding adequate evidence.
The formal external scrutiny of these parameters in theauditing process is intended to ensure that adequate
standards are being set and achieved.
ALTE members are required to make a formal, ratiﬁed
claim that a particular test or suite of tests has a quality
proﬁle appropriate to the context and use of the test,
bearing in mind the following points:• Different tests are used in different contexts, by different
groups of test users. There is no intention to impose asingle set of uniform quality standards across all ALTE
exams.
• Members requesting an audit of their quality systems andprocedures are invited to build an argument that thequality standards within a test or suite of tests are
sufﬁcient and appropriate for that test or suite of tests.
• It is the argument which is the subject of the audit, rather
than the organisation itself (which is often dealt with byother systems of regulation, e.g. ISO 9001, government
regulators etc.).
• Each audit considers one test, suite of tests or testing
system.
• The audit has both a consultancy and quality control role.
• The audit aims to establish that minimum quality
standards are being met in a way that is appropriate to
the context of a test, and also to offer recommendations
towards best practice where, though quality standards
are appropriate, there is still room for improvement.
• If quality standards are not being met, ALTE members will
collaborate with the audited organisation to implement
an action plan aimed at working towards and ultimately
reaching the quality standards.
The argumentation structure is as follows. A claim is
made about each of the 17 parameters; the claims support
an argument that minimum standards are being met for thetest in question. Information is provided to support theclaims; this information is provided in the form of
explanations. A justiﬁcation is also needed to provide
legitimacy for this information; this must be based on therelevant language testing theory with reference to the Codeof Practice and the Principles of Good Practice, and may
also take into account prior experience and best practicemodels where appropriate. This justiﬁcation, in turn, needs
to be backed up with appropriate evidence which has beencollected as part of the validation process. This approach isconsistent with Toulmin’s (2003) argument structure andBachman’s (2005) application of his work to language
testing. In Toulmin, the justiﬁcation is known as a warrant,
and the evidence is known as the backing. Bachman (2005)
suggests that the test developer must be prepared to deal
with rebuttals (alternative explanations and counter claims)
and to provide additional evidence to reject them (rebuttal
data). The QM checklists provide a practical tool for
structuring the explanations supporting the claims, and forcollecting and presenting the necessary evidence.
The auditor and an elected standing committee scrutinise
the claims and the evidence; they can challenge whether
the claims adequately meet minimum standards, and if
necessary can ask for additional information to be provided.
An audit remains ‘ongoing’ or ‘in progress’ until such points
have been clariﬁed or alternative procedures have been put
in place which are deemed acceptable.
In 2006/7 a manual setting out the ‘procedures forauditing’ was developed, a small group of auditors wasrecruited and appointed from within the ALTE membershipand an initial induction and training programme wasdeveloped. All those eligible to conduct audits had to go
through this training process. It was recognised, however,that the procedures would need to be improved and that
better co-ordination and standardisation would be
required in future as the system develops (i.e. an ongoingand iterative process in its own right – see Parizzi in thisissue).
The membership as a whole is the arbiter of decisions
arising from the auditing process; this takes place throughthe Council of Members as a whole and in particularthrough the smaller, elected Standing Committee which hasdelegated responsibility to oversee the auditing process.
During the ﬁrst cycle of auditing overall outcomes of audits
were either stated as satisfactory or not satisfactory. Whenoutcomes of an audit are considered satisfactory, theStanding Committee may recommend that an action plan be
developed for that examination to improve practice on any
weaker parameters. Members are asked to provide anaction plan with a timescale and a rationale for addressing
the issues in a particular way. Even in cases of goodpractice, opportunities for improvement (OFIs) can be
identiﬁed and suggested.

Results of the auditing
After the initial piloting of the system, further reﬁnements
were made and implemented in the audits carried out todate.
The ﬁrst cycle of auditing is still going on, this cycle aims

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

at having at least one examination of each member audited.
This ﬁrst cycle also provides the necessary experience for
improving the audit procedures and the training system forthe future. To summarise:• 46 examinations of 23 members have been audited todate• eight are still in a negotiation phase for setting up theaudit• three new member audits have recently been completed.
The outcomes of the completed audits (at the time ofwriting) are as follows:• three remain in progress (not yet satisfactorily meeting
minimum standards)|

27

The completed audits have also provided a useful ‘snap
shot’ of the state of affairs across the examinations of ALTE
members. The information now functions as an input for
further training, and for organising well targeted workshops
to help improve examinations.
Ways of improving the auditor training itself have also
emerged, based on the experiences of the auditing
described above. This process of improvement will continue
in the future (see Parizzi in this issue), but so far it has
already resulted in an improved training programme that ismore ﬁrmly based on ‘real life auditing data’.

Conclusion• two satisfactory

In conclusion, the QMS and the auditing procedures
provide a dynamic system which forms the necessary basis
for action in establishing and monitoring standards. It isuseful for:• four have been submitted but are still to be discussed by
the Standing Committee.

• clarifying the quality demands of examinations in relation
to their functions and purposes• satisfactory with an action plan• satisfactory with suggestions for improvements (OFIs)• providing ALTE members with valuable information of thestate of affairs in the examinations in their frameworks

Reﬂections on the auditing experience
After each audit the auditor has to write an audit report,
which is ﬁrst discussed with the auditee and then sent to
the Standing Committee to be discussed and ratiﬁed.
Despite the fact that all the auditors followed the guidelines
as described in the ‘procedures for auditing’ each report
differed from the others to a certain degree and insigniﬁcant ways. This probably had to do with the following
issues:• differences in background of the auditors, leading to adifferent focus of attention• a need for greater elaboration of the core elements withineach minimum parameter to achieve better
standardisation of the pre-audit, the audit and thereporting• a need for more clarity in, and agreement on, which coreelements have to be met in order to meet the minimum
standard for each parameter.
Comparisons of the information in the different audit
reports, and the way different auditors came to their
judgments, have already resulted in a more accurate
description of the core elements of each minimum
parameter, as shown in Minimum Parameter 13:• providing ALTE members with concrete possibilities ornecessities for improving their own examinations• setting priorities for training, workshops and mutual
consultancy and support• accounting for the validity of the examinations tostakeholders• improving the system itself.
References and further reading
AERA/APA/NCME (1985/1999) Standards for Educational andPsychological Testing, Washington: AERA.
ALTE (1993/2001) Principles of Good Practice for ALTE Examinations.
ALTE (1994) Code of Practice.
ALTE (1998) Handbook of Language Examinations and Examination
Systems.
ALTE/Council of Europe (1997) Users Guide for Examiners Strasbourg.
Bachman, L F (1990) Fundamental Considerations in Language
Testing, Oxford: Oxford University Press.
Bachman, L F (2005) Building and supporting a case for test use,
Language Assessment Quarterly 2(1), 1–34.
Bachman, L F and Palmer, A (1996) Language Testing in Practice
Oxford: Oxford University Press.

‘You collect and analyse data on an adequate and representative

Fullan, M (1993) Change Forces: Probing the Depths of Educational
Reform, London: Falmer Press.

sample of candidates and can be conﬁdent that their achievement is a

Fullan, M (1999) Change Forces: The Sequel, London: Falmer Press.

result of the skills measured in the examination and not inﬂuenced by

ILTA (1995) Task Force on Testing Standards.

factors like L1, country, gender, age and ethnic origin.’

Minimum Parameter 14 states:‘If you make a claim that the examination is linked to an externalreference system (e.g. the Common European Framework) then you canprovide evidence of alignment to this system.’Working in this way it is possible to make audits morecomparable and transparent and less dependent on
individual interpretations of the different auditees andauditors.

ILTA (2000) Code of Ethics, Language Testing Update, Issue 27.
Kane, M T (1992) An argument-based approach to validity,
Psychological Bulletin 112, 527–535.
Kunnan, A J (2000) (Ed.) Fairness and Validation in Language
Assessment, Cambridge: UCLES/Cambridge University Press.
Kunnan, A J (2004) Test fairness, in Milanovic, M and Weir, C (Eds),
Studies in Language Testing 18: European Language Testing in a
Global Context, 27–45.
Messick, S A (1980) Test validity and the ethics of assessment,
American Psychologist 35, 1,012–1,027.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

28|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Messick, S A (1989) Validity, in Linn (Ed.) Educational Measurement,
New York: ACE/Macmillan.
Messick, S A (1994) The interplay of evidence and consequences inthe validation of performance assessments. Educational
Researcher 32(1), 13–23.
Mislevy, R J (1994) Evidence and inference in educational
assessment, Psychometrika 59, 439–483.
Mislevy, R J, Steinberg, L and Almond, R (2003) On the structure of
educational assessments, Measurement: Interdisciplinary Researchand Perspectives 1(1) 3–62.

Saville, N, van Avermaet, P and Kuijper, H (2005) Setting andMonitoring Professional Standards: A QMS approach, paper
presented at the 2nd EALTA Conference, Voss, June 2005.
Toulmin, S E (2003) The Uses of Argument. Cambridge: Cambridge
University Press.
van Avermaet, P, Kuijper, H, and Saville, N (2004) A code of practiceand quality management system for International examinations,
Language Assessment Quarterly 1(2). Special Issue: The Ethics ofLanguage Assessment, 137–150.

Training auditors: The ALTE system
FRANCESCA PARIZZI CENTRO PER LA VALUTAZIONE E LA CERTIFICAZIONE LINGUISTICA (CVCL), UNIVERSITÀ PER STRANIERI DI PERUGIA1Introduction
This article outlines a training path which aims to develop
knowledge and competencies auditors need in order toaccomplish their task in accordance with the requirementsof the ALTE procedures. Although training sessions for bothauditors and auditees have been organised since 2006,
the experiences gathered so far, including feedback fromtrainees and recent changes in the procedures for recruiting
auditors, make the topic of training a very relevant one.
What is presented here is the result of an ongoing
discussion where all ALTE members are invited to take part.
Although suggestions and descriptions included in thisarticle relate to the ALTE auditing system in particular, they
may also be applied in other auditing contexts.

The ALTE auditing system: an overview
A description of the ALTE auditing system, as well as theapproach and the principles sustaining its implementation,
has already been the object of several articles andpresentations (see Saville, van Avermaet and Kuijper 2005;
Saville 2008, van Avermaet, Kuijper and Saville; 2004 andSaville in this issue). The ALTE website offers a wide
overview of the system and includes part of the
documentation which forms its reference points. What will
be described here is a brief history and outline of the main
characteristics of the system to provide a generalbackground, and to establish a conceptual context to what
will follow this article.
The ALTE auditing system is the outcome of a process
which dates back to the beginning of the 1990s. Since theassociation has been founded, establishing high standards
in language assessment, as well as striving for fairness inrespect of candidates, have been key issues and the main

1 The Centre for Evaluation and Language Certiﬁcation of the Università per
Stranieri di Perugia.

focus of the debate among ALTE members. The ﬁrst draft of
the ALTE Principles of Good Practice (1993), as well as the
publication of the ALTE Code of Practice (1994), represents
important milestones in this process. In order to support
organisations in applying those principles in their own
contexts, practical tools have been developed. Amongthese, the ALTE check-lists give a detailed guidance to test
providers in order to self-assess the extent to which
acceptable standards are met throughout all phases of thetesting cycle. The way towards the implementation of asystem has gradually been put in place, leading to a formal
deﬁnition of procedures and a common agreement aboutparameters. In 2006 the ﬁrst version of Procedures for
Auditing was released, along with a list of 17 parametersagainst which minimum standards are established. Far from
representing a deﬁnitive outcome, these procedures havebeen the object of an ongoing process of revision whichaims at giving to the various stakeholders involved a clear
representation of the system, and guidance to its
implementation. As usual in the ALTE modus operandi, thework conducted so far is the result of a joint effort where allmembers are invited to take part in the discussion and offer
their own contribution. The ALTE Code of Practice (CoP)Working Group, which was formed in 2000, has met
regularly since then, and co-ordinates the ALTE activities inthis area.
The approach, which is at the basis of the ALTE auditing
system, is derived from the paradigm of the quality
management system (QMS), as exempliﬁed by thestandards of ISO 9000 series. Adopting a QMS for an
organisation, whatever its mission and business area,
implies a strategic decision which aims to guarantee and,
where possible, improve customer satisfaction. This aim
can be achieved by maintaining and improving the overall
organisational performance.
For language exam providers such as ALTE members,‘customer’ means a range of stakeholders, such as
candidates, public institutions, schools and employers.
Assuring their satisfaction, by meeting their expectations,©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

is principally a question of providing exams which are valid,reliable and practical: valid because scores must be an
accurate reﬂection of the candidate’s language ability;
reliable, because they are stable, consistent and free of
error; practical, because the resources necessary to
produce and administer them must be appropriate to thecontext. Besides these characteristics which relate mainly
to the nature of the exams, a transparent and exhaustive
communication as well as efﬁciency and reliability inadministrative and logistical operations, are also
fundamental to guaranteeing quality.
In the ALTE auditing system, 17 parameters have been
established which speciﬁcally refer to the above mentioned
aspects. They are divided into ﬁve areas which are:• test construction• administration & logistics|

29• Reading and analysing the argument put forward by theauditee2, judging its validity, checking all the documents
and/or other material produced by the auditee as
evidence to sustain the argument.
• Maintaining3 constant communication with the auditee inorder to request further explanation or documentation
before the audit visit.
• Conducting the audit visit.
• Writing a report to be submitted to the ALTE Standing
Committee4 where it is stated whether and to what extent
the minimum standards have been met by the auditee.
The report must also include recommendations andsuggestions in order to improve the auditee’s system,
even if standards are met. The audit is namely intended
not only to have a control function, but also a consultancyrole.

• marking & grading• test analysis• communication with stakeholders
These parameters are the reference points against which
ALTE members must establish a quality proﬁle for eachexam or suite of exams they intend to submit for an audit.
This is accomplished by ﬁxing standards that they consider
as sufﬁcient and appropriate to the context and the use ofthat particular exam or suite of exams. In other words, theyrefer back to standards that meet the expectations of all the
stakeholders involved and guarantee their satisfaction.
Taking into consideration the nature of an association like
ALTE, in which members operate in different contexts anddeal with different ranges of stakeholders, establishingﬁxed common standards and forcing all members to meetthem would have been inappropriate.
Once the organisation has planned and put in place all
the procedures considered as necessary to meet the ﬁxed
minimum standards, and has gone through a process of
self-assessment2, a formal request for an audit issubmitted to the ALTE secretariat. In this phase theorganisation builds an argument arguing that the exam orsuite of exams have quality standards sufﬁcient andappropriate according to the context in which they areadministered. It is the argument, and not the exam or theorganisation as itself, which is the object of the audit.
This is a key issue in the ALTE auditing system and isoutlined in more detail below.
In the ﬁrst phase of experimenting with the ALTE
procedures for auditing, a system of ‘peer-review’ was put
in place, as auditors were recruited from ALTE members andconsequently trained. In order to aim at standardisation in
the system as a whole, ALTE is now developing newprocedures of recruitment which will lead to the constitutionof a permanent group of auditors who, after going through a
programme of induction and training, will be involved in asystem of co-ordinated activities including regularstandardisation sessions. That is one of the reasons why
auditor training is now an object of discussion among ALTE
members.
The role of an auditor in the ALTE system includes aseries of duties and responsibilities which can besummarised as follows:

What competencies and knowledge
auditors needSome key issues have emerged from the outline of the ALTE
auditing system which has been sketched above. Theyshow characteristics which relate to the nature of ALTE.
These are, among others:• It is neither the organisation nor an exam, but theargument which is the object of the audit.
• No standards are imposed. Quality standards are
considered as sufﬁcient and acceptable according to thecontext where they are applied.
• The audit not only has a quality control function, but also
one of consultancy.
Taking these three points as a reference, the focus now
draws to the auditors and consideration of what knowledge
and competencies are needed and what elements ingeneral should characterise their behaviour and attitude
when conducting an audit. This constitutes the basis fordeveloping a training programme.
The three main issues listed above can be reformulated
as three questions that auditors put to themselves
throughout the entire audit process:• is the argument put forward by the auditee valid?
• are the standards met sufﬁcient and acceptableaccording to the context?
• how can I help the organisation improve its system and,
where necessary, reach the minimum standards?
Finding suitable answers to these questions, which canbe considered as the core of an auditor’s responsibilities,
represents a big challenge. In order to meet this challenge,
knowledge and competencies are not the only attributes
2 ALTE check-lists are the appropriate tool produced by ALTE to help members inthis phase of self-assessment.
3 In order to gain standardisation in the way arguments are structured, newdocumentation has been developed by the ALTE CoP Group which provides theauditee with a guide to be followed to cover all aspects related to the testdevelopment cycle.
4 The Standing Committee is made up of ALTE members who are elected by themembership as a whole. It has the function of overseeing the auditing process
and taking decisions about audit outcomes.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

30|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

required. Good communication, and the ability to consider
and balance a range of factors, are also fundamental.
As for knowledge required, it is clear that a deep
understanding of theoretical principles and practicalapplication in the ﬁeld of language testing is vital. Being in
the position to assess whether a certain method adopted
by the auditee to guarantee consistency to their exams issuitable and appropriate to the context can mean applying
a range of other methods to reach a better outcome. Or
again, assessing the relevance of a theoretical model as thebasis of a construct deﬁnition can be facilitated by knowing
other similar examples of that model’s application in orderto weigh its appropriateness. A theoretical understanding ofwhat a validity argument is, and how it can be evaluated,
represents a fundamental competence an auditor should
develop. Criteria and procedures for evaluating arguments
have been discussed in the literature (Bachman 2005, Kane
1992, Kane, Crooks and Cohen 1999). In this phase of theirjob, auditors can be supported by instruments andmethods to evaluate whether the argument put forward by
the auditee has been stated clearly, whether it is coherent
and plausible, and whether the evidence given is sufﬁcient
to support the assumptions.
Mentioning coherence and plausibility in the argument
gives the chance to bridge the discussion to the second
point to focus on. An auditor should be able to assess
whether the standards met by the auditee are acceptable
and sufﬁcient for the context. This aspect can be seen as avery delicate one because of the constant risk of
subjectivity in formulating judgements. One way to reduce
this risk is to systematically review information related tothe context where the exam has been developed andadministered. As will be shown later, a benchmarking
approach to standardisation, using concrete cases toestablish a shared understanding of context, may also be away forward.
The last of the three points outlined above is related tothe ability of auditors to maintain a proactive andcollaborative conduct throughout the whole auditing
process. This kind of approach is exempliﬁed by an open
and transparent way in communicating with the auditee, aninterest in detecting possible areas of improvement in theaudited system, and in giving their opinion about possible
action to be undertaken. In short, auditors should conduct
the audit in such a way that it will not be perceived by theauditee as a kind of inspection but, on the contrary, as away to improve through continuous interaction.

Training auditors
If the points listed above aim to focus on knowledge andcompetencies which should characterise auditors, thefollowing are suggestions for a possible training path inorder to develop these knowledge and competencies.
As mentioned in the introduction, this is the result of an
ongoing discussion among ALTE members in order tocontinuously improve auditor training, also taking into
account some recent changes to the procedure for recruiting
auditors.
A ﬁrst proposal, currently under discussion, includes twostages of training: a ﬁrst basic one, where all new auditors
and also auditees are invited to take part5, and advancetraining, which will involve experienced auditors only.
The basic training can be considered as a kind of
induction into the ALTE system. Participants are provided
with a complete picture of the ALTE auditing system. Eachcomponent is presented and analysed in order to detect
their main function, and the way all components link
together in the construction of the system is discussed. The
purpose is to develop for auditors and auditees a clear ideaof how the system works, and awareness about their duties
and responsibilities. At this stage the 17 parameters
represent a focus in the discussion. An explanation about
what each one refers to and the areas of the examination
cycle they cover is fundamental.
Once the picture of the system has been deﬁned, a key
issue must be addressed: the concept and the areas ofapplication of a validity argument. Although a description ofwhat a validity argument is, and an outline of its elements
should be presented during training, it is clear thatparticipants need to develop deeper understanding of this
topic by referring to the available literature. This is why,
before the training session, some useful reading on validity
arguments will circulate. This allows trainees to take part
actively in the discussion about possible models to be
applied in the evaluation of a validity argument, andsome practical activities can accordingly been organised.
Concrete cases of construction of a validity argument can bepresented at this stage. The ideal would be to provide
participants with examples of validity arguments alreadysubmitted for an audit, taking care to preserve anonymity.
Trainees will then be invited, by working in groups or pairs,
to detect whether the argument has been structured in a
coherent way, covering all the areas of the 17 parameters,
and whether the evidence provided is sufﬁcient to support
the argument. Discussing the outcomes in plenary would
complete the task by collecting different points of view andconveying the participants, where possible, towards acommon conclusion.
Activities like the one just suggested above are to beconsidered in a stage of advanced training. One of the main
responsibilities of the auditors, as already mentioned, is
not only judging the validity of the argument per se, butalso assessing whether, and to what extent, minimum
standards have been met. Working on isolated concrete
cases may not be sufﬁcient. In order to develop thisparticular competence, auditors must be trained through asocial moderation approach. Several concrete cases can bepresented, which would refer to different contexts. Type of
exam provider (universities, private companies, public
institutions, etc.), use of exams (high-stake vs. low-stakeexams), number of candidates (exams with thousands ofcandidates a year vs. exams with a few dozen) can be,among others, criteria to refer to when establishing theplausibility of the argument and the appropriateness of thestandards met. Referring to a range of examples can makethe task of auditors easier when making a decision about
the ﬁnal outcome of an audit. This can also be a possible

5 In the present article only auditor training will be considered.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0|

31

way to improve standardisation among outcomes of audits
conducted by different auditors.
Practical issues are also to be considered during training,and suggestions on how to maintain good communication
with the auditee must also be part of the discussion. In
order to make the audit visit as efﬁcient as possible,
auditors should be trained to plan by collecting in advanceas much information as needed. Poor communication
during the months preceding the visit6 could compromise
its success. Providing participants with practical check-listsof all the aspects they should consider before the audit visit
is important. Considering that the audit is accomplished by
the auditor through the production of a report to beaddressed to the Standing Committee, some training on
how to write it is also necessary. In this case, good
examples of previous reports can be the basis on which toopen a discussion. Points such as how to state the positive
and negative aspects detected throughout the audit, how towrite the judgement clearly by providing the rationale, andhow to formulate recommendations can be objectives forpractical session training.

strength of this approach is that auditors can be an
important reference point to look at when improving thewhole system.

Conclusion

Saville, N (2002) Striving for fairness: The ALTE Code of Practice andquality management systems, Research Notes 7, 12–14.

Auditors play a key role in the ALTE auditing system.
The direction taken is the one to form a group which candevelop competencies and expertise throughout acontinuous exchange of opinions and sharing ofexperiences. Far from representing isolated occasions,training and standardisation sessions should frequently
be repeated and all auditors are invited to attend. The

Saville, N (2008) From code of practice to standards: Auditing thequality proﬁle, paper presented at the 30th LTRC, Hangzhou, June2008.

6 Three months elapse between the time when the auditor receives the auditee
documentation and the audit visit.

References and further reading
ALTE (1993) Principles of Good Practice for ALTE Examinations, draft
manuscript.
ALTE (1994) Code of Practice.
Bachman, L F (2005) Building and supporting a case for test use,
Language Assessment Quarterly 2(1), 1–34.
ISO International Organisation for Standardisation (2000) Quality
management systems: Guidelines for performance improvements.
ISO (2009) The ISO Standards collection – ISO 9000 Quality
management on CD.
Kane, M T (1992) An argument-based approach to validity,
Psychological Bulletin 112, 527–535.
Kane, M T (2006) Validation, in Brennan, R L (Ed.) Educational
Measurement, 4th Edition, Westport, CT: American Council on
Education and Praeger Publishers, 17–64.
Kane, M T, Crooks, T and Cohen, A (1999) Validating measures of
performances, Educational Measurement: Issues and Practice,517.

Saville, N, van Avermaet, P and Kuijper, H (2005) Setting andmonitoring professional standards. A QMS approach, paper
presented at the 2nd EALTA Conference, Voss, June 2005.
Sireci, S G (2007) On validity theory and test validation, Educational
Researcher 36(8), 477–481.
van Avermaet, P, Kuijper, H and Saville, N (2004) A code of practiceand quality management system for international examinations,
Language Assessment Quarterly 1(2/3) 137–50.

Auditing Cambridge ESOL’s Main Suite and
BEC examinations
SIBYLLE BOLTON ALTE CONSULTANT, MUNICH, GERMANY

Introduction
The ALTE auditing system is based on ongoing work in the
Code of Practice Working Group and in ALTE as a whole1.
The process of setting and monitoring standards for ALTE
members’ examinations began in the 1990s with the
publication of the ALTE Code of Practice and the ALTE
Principles of Good Practice. To help ALTE Members in
improving their examination systems, the Code of Practice
Working Group was set up in 2000. Since then, this working
group has met regularly and has produced a number ofworking tools such as the Code of Practice and Quality
Management checklists and the Minimum standards forestablishing quality proﬁles in ALTE examinations, both ofwhich can be found on the ALTE website2. Such documentsallow ALTE members to reﬂect on and assess the aspects ofthe testing cycle: test construction, administration andlogistics, marking, grading and results, test analysis andpost-examination review.
Since 2005, the Code of Practice Working Group has
focused on establishing the ALTE auditing system (i.e. theformal external scrutiny of ALTE members’ examinations).
All ALTE members, as well as ALTE afﬁliates wishing tobecome a full or associate member, need to apply to the
1 For more detailed information see Saville, van Avermaet and Kuijper (2005) andthis issue.
2 For the Quality Management checklists see www.alte.org/cop/copcheck.php
For the Minimum Standards go to www.alte.org/standards/index.php©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

32|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

ALTE Secretariat to have their examination(s) audited. The
audit may consider one test, a suite of tests or a testing
system. The ALTE Code of Practice Procedures for Auditing
specify how the audit should be prepared and carried out by
the audited organisation and the auditor. The audit is basedon the 17 Minimum Standards shown in Table 1, which mustall be covered in the audit. The auditing process has threephases: the pre-audit, which usually last three to fourmonths and during which the audited organisation and theauditor prepare the audit, the audit visit and the post-audit,
when the auditor writes the report. Testing bodies
requesting an audit of their quality systems and procedures
are invited to build an argument that shows the quality
standards within a test or a suite of tests are sufﬁcient andappropriate. It is this validity argument that is the subject of
the audit, rather than the organisation itself.

The audit
In April 2007, the ALTE Secretariat asked me to audit thefollowing Cambridge ESOL examinations: Lower Main Suite(KET, PET), Upper Main Suite (FCE, CAE, CPE), and theBusiness English Certiﬁcates (BEC Preliminary, BEC Vantage,
BEC Higher). Cambridge ESOL appointed two members ofstaff to prepare the audit, write the report on theexaminations to be audited (the validity argument) and put
together the documentation. This information was thenpassed on to me. The report explained in detail theprocedures followed and how the audited examinations
meet the requirements for each of the 17 MinimumStandards. In addition, the following supporting
documentation was sent:

Table 1: Minimum standards for establishing quality proﬁles in ALTE
examinations
TEST CONSTRUCTION
1

The examination is based on a theoretical construct, e.g. on amodel of communicative competence.

2

You can describe the purpose and context of use of theexamination, and the population for which the examination isappropriate.

3

You provide criteria for selection and training of test constructors
and expert judgement is involved both in test construction, and in
the review and revision of the examinations.

4

Parallel examinations are comparable across different
administrations in terms of content, stability, consistency andgrade boundaries.

5

If you make a claim that the examination is linked to an externalreference system (e.g. Common European Framework), then you
can provide evidence of alignment to this system.

ADMINISTRATION & LOGISTICS
6

All centres are selected to administer your examination according
to clear, transparent, established procedures, and have access toregulations about how to do so.

7

Examination papers are delivered in excellent condition and by
secure means of transport to the authorised examination centres,
your examination administration system provides for secure andtraceable handling of all examination documents, andconﬁdentiality of all system procedures can be guaranteed.

8

The examination administration system has appropriate support
systems (e.g. phone hotline, web services etc.).

9

You adequately protect the security and conﬁdentiality of results
and certiﬁcates, and data relating to them, in line with current
data protection legislation, and candidates are informed of theirrights to access this data.

10

The examination system provides support for candidates withspecial needs.

• units 1–4 of the ALTE Quality Management checklists
specifying the procedures for the different examinations

MARKING & GRADING• handbooks (test speciﬁcations)• item writer guidelines• handbooks for candidates

11

Marking is sufﬁciently accurate and reliable for purpose andtype of examination.

12

You can document and explain how marking is carried out andreliability estimated, and how data regarding achievement ofraters of writing and speaking performances is collected andanalysed.

• reports on research carried out• examination reports• sample tests• mark schemes

TEST ANALYSIS
13

You collect and analyse data on an adequate and representative
sample of candidates and can be conﬁdent that theirachievement is a result of the skills measured in the examinationand not inﬂuenced by factors like L1, country of origin, gender,
age and ethnic origin.

14

Item-level data (e.g. for computing the difﬁculty, discrimination,
reliability and standard errors of measurement of the examination)is collected from an adequate sample of candidates and analysed.

• demographic data on candidature• handbooks for examination centres• information for candidates with special needs• information on recognition of the examinations• legal agreements for printing and despatch.
During the audit I was given the opportunity to study
further documents that, for security reasons, could not be
included in the documentation previously described.
The pre-audit phase and audit visit focused on whether
the validity argument put forward in the report was clear
and exhaustive for each of the 17 Minimum Standards.
Speciﬁcally, it stated not only how Cambridge ESOL isdealing with all the aspects of the Minimum Standards but
also why the procedures followed are appropriate, given
their purpose and potential decisions based on the results.

COMMUNICATION WITH STAKEHOLDERS
15

The examination administration system communicates the resultsof the examinations to candidates and to examination centres(e.g. schools) promptly and clearly.

16

You provide information to stakeholders on the appropriate
context, purpose and use of the examination, on its content,
and on the overall reliability of the results of the examination.

17

You provide suitable information to stakeholders to help theminterpret results and use them appropriately.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

I also focused on the interdependence of the separate
Minimum Standards (MS). All Minimum Standards
contribute to an overall argument for the test or suite of
tests. For example, if it is claimed in MS2 (purpose andcontext of use) that a particular test score may be taken tomean that the candidate can study at a university, this has
an impact on other MSs, for example on the theoreticalconstruct (MS1), the stability of parallel examinations (MS4),
the reliability of the marking (MS11 and MS12), and so on.
The audit visit took place on 27 and 28 September 2007.
A time-table was established beforehand specifying when
I could speak to members of staff responsible for testdevelopment, training (internal staff, item writers, raters
and examiners) and statistics (analyses and setting of
grade boundaries). The audit ran very smoothly due to theexcellent preparation by Cambridge ESOL, and members ofstaff involved in the audit who provided all the necessary
information.

Outcomes
The audit report stated the outcome for each MinimumStandard. For the examinations audited, most outcomes
were reported as ‘Good practice’. Examples of good practice
are as follows:• The theoretical construct underpinning the examinations(MS1) is well documented in books and in articles and thetest tasks reﬂect this construct in an appropriate manner.
Ongoing research is carried out and the tests tasks arerevised accordingly.
• MS1 impacts on MS2, and the purpose and context of use
stated for the audited examinations supports theargument put forward in MS1. And purpose and context ofuse are well documented in handbooks and on the
Cambridge ESOL website.
• The arguments put forward in MS1 and MS2 impact on
MS4 (comparability of parallel versions) and MS5|

33(alignment to an external reference system) and support
the arguments put forward in both. All tests areconstructed to set speciﬁcations in terms of content, text
types, tasks and statistical properties. Test linkage to the
CEFR is well documented, and ongoing research
programmes provide validity evidence in support ofalignment claims.
• The arguments put forward in MS4 and MS5 also impact
on and support the arguments put forward in MS14 (itemlevel data). All objectively scored and clerically marked
papers are pretested and analysed (Classical and Raschanalyses). Analyses are double-checked. In addition, posttesting is carried out.
Besides stating the outcome for each Minimum Standard,
the audit report also stated the overall outcome. In 2007,
the overall outcome of an ALTE audit was reported in two
ways: satisfactory or unsatisfactory. For Main Suiteexaminations and Business English Certiﬁcates, theoutcome of the audit was ‘satisfactory’ due to thecomprehensive and clear validity argument and supporting
documentation.

References and further reading
ALTE (1994) Code of Practice.
ALTE (1998) Handbook of Language Examinations and Examination
Systems.
Bachman, L F (2005) Building and supporting a case for test use,
Language Assessment Quarterly 2(1), 1–34.
Saville, N, van Avermaet, P, and Kuijper, H (2005) Setting andmonitoring professional standards: a QMS approach, paper
presented at the 2nd EALTA Conference, Voss, June 2005.
van Avermaet, P, Kuijper, H and Saville, N (2004) A code of practiceand quality management system for international examinations,
Language Assessment Quarterly 1(2/3), 137–50. Special issue:
The Ethics of Language Assessment. Guest Editor, Alan Davies,
137–150.

International accreditation of quality in language
learning and teaching
MARIA MATHEIDESZ OPERATIONS MANAGER, EAQUALSWhat is EAQUALS?
Founded in 1991, EAQUALS (European Association for
Quality Language Services) is an association that fosters‘excellence through accreditation of quality in language
services’. It is a not-for-proﬁt international association of
language training providers who aim to attain andguarantee high quality in language teaching. The primary
aim of EAQUALS is to encourage and accredit language
teaching and learning in any type of institution, in any
language being taught, and whatever the national context.

EAQUALS is internationally recognised for its quality
standards in language education and training and for its
accreditation services. The main purposes of EAQUALS are:• through its Charters and accreditation scheme, to provide
guarantees of high quality for language learners, as well
as for the learners’ sponsors, employers, parents, or thefunding agencies which ﬁnance their study• to bring together providers of language education
services, whether they are privately or publicly funded,
and wherever they are located©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

34|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0• to seek to encourage greater awareness of consumer
rights and quality issues in language learning, both
among those providing language education services andamong those using them• to assist national and international bodies to develop
accreditation schemes for those providers that may not
yet be ready to meet EAQUALS standards• to provide support through self-assessment,
management training and consultancy for any language
teaching institution which has a genuine desire to raise
its standards• to provide information on leading-edge developments inthe ﬁeld and create a lively network of professionals
involved in language education.
To support these aims, EAQUALS has established andpublished a demanding set of accreditation criteria. These
are laid out in the form of four charters – a General Charter,a Charter for Course Participants, a Staff Charter and an
Information Charter. These are displayed in EAQUALSaccredited schools, and are also used as a basis for
EAQUALS’ rigorous inspection scheme. The complete text of
the EAQUALS Charters can be found on the EAQUALS
website: www.eaquals.org
In this article I will outline the basic principles behind
EAQUALS accreditation. I will refer to the challenges
involved in developing an international scheme foraccrediting quality, the ways in which EAQUALS hasaddressed these challenges, and also describe themethods and means of assessment and the process of
EAQUALS accreditation.

organisation (ISO 9001:2000), which mainly refer tomanagement processes. EAQUALS, on the other hand,includes all elements of the management of language
service provision. In addition to checking on documents andprocedures, it also requires teaching and the support
services provided for course participants to be observed,
and looks at the ‘institution in action’. EAQUALS has liaison
status with two ISO committees, one of which has developed
a set of requirements for ‘learning service providers’ (ISO DIS
29990), and continuously contributes to the further work on
the standards overseen by these committees. EAQUALS was
also actively involved in developing a standard for‘Language Study Tour Providers’, under the auspices of CEN(Comité Européen de Normalisation), the European subgroup of ISO member organisations.
EFQM (The European Foundation for Quality Management)
was founded in 1988 by leaders from 14 companies who
were convinced that a new membership organisation basedin Europe was necessary to promote higher standards ofmanagement through shared knowledge and mutual
recognition. EFQM has developed an excellence model
specifying ﬁve fundamental concepts, which EAQUALS also
shares:• result orientation• customer focus• leadership and constancy of purpose• management by processes and facts• people development and involvement.

Challenges of an international scheme foraccrediting quality in language education
EAQUALS in the international context
EAQUALS is recognised as an international nongovernmental organisation and has been granted
participatory status by the Council of Europe. It subscribes
to the Council of Europe’s principles on language learning
for European citizenship:

The EAQUALS inspection scheme looks at all aspects oflanguage teaching services. This includes all processes ofcourse design, which means that EAQUALS accreditationrefers to the institution as a whole. At the same time thebody also guarantees that:• to facilitate the free movement of people and ideas
across Europe• courses are based on written course outlines stating theobjectives, content, materials and assessment
procedures• to increase mutual knowledge and understanding among
all European people• course design is sound and professional, and teaching iswell-planned and effective• to increase the quantity and quality of international
co-operation• there is systematic needs analysis – learners’ priorities
for use of the language are considered• to combat prejudice and intolerance towards people ofdifferent languages and cultures.

• courses are related to the levels of the Common European
Framework of Reference (CEFR)EAQUALS contributes to Council of Europe language
projects, and is consulted by the European Commission onmatters related to language education. It has memoranda ofco-operation with the European Centre for Modern
Languages (ECML) and with the Association of Language
Testers in Europe (ALTE). EAQUALS has many common
features with other quality control schemes, but it is uniquein being the only accreditation designed for language
learning and teaching.
ISO (International Organisation for Standardisation) for
example, has standards for quality management in any• course participants’ progress is regularly checked andregular feedback is given• the teachers used are qualiﬁed in foreign language
teaching• the performance of teachers is monitored through regularobservation.
In the context of quality control schemes, the EAQUALS
scheme has the following unique features. It is:• sector speciﬁc: it has been speciﬁcally developed for
language education purposes.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0• inclusive: the standards can be applied to institutions ofdifferent types, operating in different market and national
contexts and offering different languages.
• comprehensive and complete: it includes all areas ofacademic and general management.
• evidence-based: it focuses on evidence that is gathered
by looking at the performance of the institution in action,
as well as at systems and documentation.
• tangible: quality is assessed by applying speciﬁc
indicators relevant to the activity of language service
providers.
• non-prescriptive: although the basic requirements areclearly established, EAQUALS requirements can beapplied in different contexts, and it is the institution’stask to provide evidence which demonstrates they meetthe standards.
• reliable: inspector teams consist of two independent
inspectors with international professional backgrounds,
and inspection reports are moderated by an expert
committee to maintain consistency and impartiality.
In addition, inspectors attend regular standardisation
workshops.
• developmental: the inspection report has a strong
consultative element due to the recommendations thatare provided for the institution to consider, thus helping
them to maintain the momentum of continuous
development.
• ﬂexible: if institutions are ‘referred’ in one or more areas,
they have the opportunity to take action and prove at alater stage that they meet EAQUALS standards.
The above features have inherent controversies in them,which EAQUALS has had to address with innovative
solutions in its inspection scheme. Some basic questions
had to be answered:• how can standards be applied consistently across all
languages offered, and in different kinds of institutions indifferent national and market contexts?
• how can established standards be non-prescriptive andspeciﬁc at the same time?
• how can assessment be reliable if standards are notprescriptive?

Inspection methods and standards
The inspectors seek evidence from different sources toverify that EAQUALS standards are being met. This is done
mainly through a visit to the institution, lasting between
two and four days depending on the size of theinstitution. It includes assessment of all activities, using
such means as observation of teaching, meetings withkey staff and teachers, clients and partners, studying
documents, inspecting premises, and student
accommodation if applicable.
The EAQUALS assessment of institutions focuses on the
global picture. For example, it does not inspect theperformance of individual teachers but looks at the generallevel of teaching and support for learning being carried out|

35

at the institution. Inspectors collect evidence from as manysources as possible before and during the inspection toarrive at a global judgement of the institution’s
performance. The evidence collected by the inspectors foreach main area is then examined for coherence and the way
in which the different aspects and areas interrelate in thecontext of the institution being inspected.
The international teams of inspectors consist of language
teaching professionals who have extensive experience bothin academic and general management, as well as quality
assurance. The inspectors are selected through a rigorous
process and have to go through a speciﬁcally designed
training process which includes attending training sessions
and carrying out inspections with an experienced mentor
inspector as part of on-the-job training.
Guidelines on how to apply requirements are laid outtransparently in the EAQUALS Inspection Scheme Manual,and guidance on how to apply these in the various countryspeciﬁc contexts are summarised in the individual Country
Notes, which include local regulations on legal
requirements, teachers’ qualiﬁcations, labour law, etc. The
scheme is transparent and the Inspection Scheme Manual
has been designed to meet the needs of institutions,
including members of EAQUALS and prospective members,
as well as to assist EAQUALS inspectors in their work.
For the purposes of the EAQUALS inspection, the complex
activities of language teaching institutions are divided into
12 main areas, for each of which transparent and tangible
quality indicators have been formulated. These 12 mainareas contain altogether 50 key assessment criteria, whichconstitute the main requirements of the scheme. The more
speciﬁc focus points listed under the key assessment
criteria are quality indicators, which are not to beinterpreted either as comprehensive or as obligatory
checklists, but as deﬁnitions of various tangible elementsthat are susceptible to assessment.
The assessment criteria are not absolute: provided thatbasic requirements are complied with, it is accepted that
there may be room for further development, even in areasof excellence. Areas of excellence are identiﬁed and grade 1
is awarded by the inspectors if quality indicators
demonstrate innovative and creative ways of good practice.
The ﬁndings of the inspections are summarised in an
inspection report. Each assessment category is evaluatedby a verbal description and given a grade on a 3-point
scale where 1 signiﬁes excellence, 2 is awarded forcompliance with the relevant EAQUALS quality criteria, and3 indicates non-compliance. For areas in which an
institution does not meet EAQUALS standards,‘requirements’ are speciﬁed, which the institution has toact on in order to later gain accreditation. The inspection
report also includes several non-mandatory
recommendations made by the inspectors, which adds astrong consultative element to the inspection.
Coherence between verbal assessment, grade andrelevant requirements and recommendations ensures thatthe inspection report gives transparent and detailed
feedback to the institution about the level of quality it isattaining as related to EAQUALS standards.
The 12 main areas and 50 key criteria are outlined inTable 1 below.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

36|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Table 1: Overview of EAQUALS assessment criteria and standards (Taken from the EAQUALS Inspection Scheme Manual, released in January 2010.)
Main categories and assessment criteria in the Inspection Scheme

What inspectors will look for

1. Teaching• approach and content• teaching methods• course participants’ needs and learning

Inspectors aim to get evidence that effective learning is taking place inclasses, and to form an overall picture of the quality of teaching. They do
not inspect the performance of individual teachers.

2. Academic management – Curriculum and Syllabus• a structured course of study• levels that refer to the CEFR
• learning objectives that are adapted to learners’ needs• statements of content to be covered in a course and during a certain
periodInspectors seek evidence that there is coherence between what is laid out
in the publicity materials and academic management documents andwhat is going on in the classrooms. e.g. if a school claims that it provides
teaching based on a communicative approach, evidence of this needs tobe shown both in supporting documents and in the teaching.

3. Academic management – Assessment and Certiﬁcation• placement testing• assessment of progress and achievement• external exams and tests• reports, certiﬁcates and diplomas

The inspectors check whether the assessment procedures and methods
are valid and reliable, and are in line with the general approach toteaching. e.g. if learning aims are deﬁned by ‘can do’ statements, isassessment designed to assess what course participants can do with thelanguage?

4. Academic management – Quality assurance• system and procedures for lesson observation• support and guidance for teachers• coherence between outcomes of class observations and internal
training• professional development review of teaching staff• other procedures for quality assurance

The inspectors seek evidence that the standards of teaching aremonitored by well qualiﬁed and experienced academic managers, andthat appropriate action is taken to continuously improve standards where
and when needed. The professional development opportunities for staff
and systematic assessment of teachers’ work are also checked.

5. Academic resources
Availability and organisation of:• course books, software, reference materials and/or student library• supplementary materials produced in-house• hardware: OHPs, audio & video/DVD equipment, computers, accessto the internet etc.

The quality, appropriacy, availability and use of teaching resources arechecked, and also whether they correspond to the needs of learners andare in line with the statements on teaching philosophy in the institution’s
curriculum and syllabus documents.

6. Other services to course participants• fair dealing between the institution and course participants• advice and consultation• leisure and social programme• course participants’ welfare• accommodation services: residential and home-stay

The nature and number of other services offered to course participants
may vary considerably from institution to institution. Inspectors check
what is offered, whether the services are high quality and whether
systems and procedures for quality assurance and continuous
improvement are in place.

7. Staff contracts, terms and conditions• contracts• terms and conditions

Aspects of the EAQUALS Staff Charter are checked, based on local
conditions as laid out in the Country Notes, and terms and conditions are
veriﬁed for their fairness within the local context.

8. Qualiﬁcations, experience and training• competence, experience and qualiﬁcations of teachers• administrative and academic staff proﬁle• training and professional development

Inspectors look at the qualiﬁcations proﬁle of the teaching staff. The
qualiﬁcations and competence of all staff are checked, as well as thetraining opportunities available to them.

9. Communications• clear academic management systems• procedures for internal communication between staff and managers• information available to staff about the institution• grievance and disciplinary procedures• performance review for managers and admin staff

Channels of communication and the balance between formal andinformal communication are checked, as well as awareness of basic
company information among staff. Inspectors also look at theinstitution’s systems for managing staff performance.

10. Information
Completeness and accuracy; clarity and accessibility of:• all promotional materials• all contractual information• information about EAQUALS and the correct use of the EAQUALS
logo and Charters

Adherence to the EAQUALS Information Charter is checked. Publicity
materials, websites, contracts and information about EAQUALS should be
accurate, clear, comprehensive and easy to access.

11. Premises• teaching and study facilities• other facilities• hygiene• health and safety• welfare

All teaching and learning premises and facilities used by the institution
are checked to see whether their condition corresponds to what is statedin the institution’s publicity materials and whether they offer a safe andpleasant learning environment. Safety features are checked forcompliance with European directives and local requirements.

12. Management and administration• legality and integrity• fair dealing between the institution and its clients• management processes• course organisation and administration• enrolment and placement administration• quality assurance procedures• complaints• client feedback

This complex area comprises all elements of general management,
organisational and administrative procedures.
Requirements with reference to legality are preconditions of EAQUALS
accreditation, and these are checked very thoroughly during aninspection.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Applying the EAQUALS assessment
methods: An example
In the following simple case study a brief illustration is
given on how the assessment criteria are applied in a
concrete situation. The rather complex category of internal
quality assurance, a key area for external quality control,
has been selected for this purpose.

|37

The example below demonstrates that the quality
indicators do not prescribe the ways in which procedures
and systems are set up or carried out – this should dependon the institution’s size, management structure and culture.
However, coherence between the systems and theireffectiveness is checked by the inspectors, e.g. the role ofco-ordinators in the example was not fully clariﬁed andthus the system did not fully meet staff expectations.

Table 2: Sample extract from an EAQUALS inspection report

4. Academic management – Quality assurance
Key assessment criteria and quality indicators

Description of the case – ﬁndings of the inspectors

System & procedures for lesson observation• an appropriate mix of some, or all, of the different types of lesson
observation for different purposes:– quality control (buzz observations)– individual professional development (by academic management,peer)– institutional development• criteria and procedures for setting up observations• feedback and follow up systems• The academic manager observes each teacher at least once a year
• A feedback session, based on self-review (supported by a welldesigned template) is held and this is appropriately documented
and ﬁled for further reference• Level co-ordinators carry out systematic lesson observations (1–2 aterm) in their group of teachers and written feedback is given on this
• The DOSs carry out buzz observations in the third week of the course
to check on placement and group composition• Peer observation is encouraged and the AM and DOSs are available
for cover if requested

Support and guidance for teachers• induction of new teachers• professional support for teachers (mentoring, consultation, etc.)• New teachers are selected on the basis of their applications and CVs
by the DOSs
• New teachers are given a Staff Handbook, which includes all thenecessary information on curriculum and syllabus, as well as on
administrative duties for teachers• Level co-ordinators provide personal guidance and support to newteachers• New teachers are observed and detailed feedback is given in the ﬁrst
month of their teaching• If individual support is required a support scheme is set up by the
DOS and involving the level co-ordinators in the ﬁrst three months of
employment

Coherence between outcomes of class observations and internal
training• identifying and acting on areas of weakness• The AM organises a meeting with the DOSs every month anddiscusses the ﬁnding of lesson observations• The common weakness areas are selected for inclusion in theinternal training programme• Individual support is provided by the level co-ordinators to newteachers or to those who need it

Professional development review of teaching staff• a formal and regular system is in place• professional development interviews are held on a regular basis
and these are recorded in writing• the system includes:– feedback on lesson observation(s)– a self-assessment element– agreed self-development aims
– the signature of interviewer and interviewee• The lesson observation carried out by the AM forms the basis of theannual professional development review for all permanent staff• The interview includes feedback on teaching, on other work done by
the teacher (e.g. syllabus, test development) and agreed
development aims are set• The nearly 30% of hourly paid staff has no professional development
review and they are observed by the level co-ordinators only

Other procedures for quality assurance• internal assessment processes• feedback and consultation opportunities for staff• There are regular co-ordinators’ meetings in which students’ progress
and weaknesses are discussed and actions are planned on these• The AM and DOSs have a monthly meeting in which staff requests
and needs are discussed• The level co-ordinators do not attend these meetings and no formal
forum is available for them to relate staff requests to academic
management. This was conﬁrmed in the staff focus group as a sourceof communication problems between teaching staff and academic
management.

Assessment of this area: Academic management – Quality assurance
Overall assessment

Grade

It is obvious from the collected evidence that theinstitution has good systems in place for quality
management and these are organised and implemented
systematically and consistently. The systems andtheir implementation meet EAQUALS standards.

2

Requirements/Recommendations
As the institution meets EAQUALS standards, no requirement is set.
The inspectors recommend the following for the institution to consider
for further development:• Gradually include all teachers in the professional development
interview system• Set up a system for peer observation so that all teachers are involved• Create a forum for level co-ordinators to communicate staff requests
to academic management©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

38|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

Basic principles of the EAQUALS
inspection scheme
As the above example shows, the EAQUALS inspection
scheme is more than just a coherent and effective system
for accrediting institutions; it seeks to make a constructive,
consultative contribution to the improvement of theinstitutions inspected and to language learning andteaching in general.
Pre-inspection advisory visits and EAQUALS consultancies(as well as a comprehensive self-assessment pack and agreat variety of resource materials available to members on
the EAQUALS website) are all designed speciﬁcally to help
institutions reach the required standards. Some of theabove materials are also available to potential members
through the EAQUALS Secretariat.
The aims and ethos of EAQUALS govern the way in whichinspections are managed and carried out. The following
fundamental principles are respected by all EAQUALS
inspectors:
Conﬁdentiality

It is imperative for the integrity of EAQUALS that all written
and orally communicated information about institutions
undergoing accreditation is kept conﬁdential.
Professional colleagueship and supportivenessAt all times inspectors need to remember that those they
are dealing with are colleagues, with experience,
qualiﬁcations and know-how similar to their own. On theother hand, friendliness and misleading helpfulness andinformality are also to be avoided in the interests of
impartiality.
Absolute impartiality and fairness

While a spirit of colleagueship is essential, if the aim ofsupport and advice is to be achieved, impartiality isessential for a fair report and result. It is important to take
into account all relevant information about the history,
context and ‘special circumstances’ of the institution whenevaluating whether or not the requirements of the EAQUALSCharters are being met. EAQUALS rules forbid hospitality
between staff and inspectors before, during or at the end of
inspections.
Comprehensiveness

The EAQUALS Charters cover every aspect of theinstitution’s work. All the basic criteria speciﬁed in theinspection scheme manual need to be inspected andassessed with equal thoroughness.
Consistency

Inspectors must apply the same standards from inspection
to inspection. It is crucial that institutions undergoinginspection feel that the standards being applied areconsistent with those laid down in the current inspection
scheme manual and do not differ from those applied intheir previous inspection. When doing re-inspections,inspectors receive the previous report. Any signiﬁcantchanges in the Charters, and subsequently in theinspection scheme or in the inspection scheme manual,
are approved by EAQUALS membership at the annual
general meeting.
Clarity

The oral feedback given to managers at the end of theinspection, and the written report submitted to theinspections sub-committee, include clear information about
the institution and a straightforward evaluation of each areaof its work. Shortcomings in relation to EAQUALS standards
are identiﬁed and exempliﬁed as requirements.
Suggestions for further improvement are listed asrecommendations, the implementation of which is notmandatory for the institution.

The EAQUALS accreditation process
The EAQUALS accreditation process goes beyond theinspection itself. It includes considerable preparation by theinstitution and by the inspectors, and the follow-up on
inspections can also take some considerable time,
especially if the inspections sub-committee requires further
evidence to prove that EAQUALS standards have been met.
In some cases, even accredited institutions are asked tosend in evidence of steps they have taken to maintaincontinuous improvement.
Accredited members of EAQUALS are re-inspected every
three years. In the case of institutions which run seasonal
activities (e.g. junior and/or summer courses, teacher
training, specialised courses, etc.), it is a requirement thatevery second inspection takes place at a time when these
activities can also be observed. If these seasonal activities
are not run at the time of the inspection, the inspectors may
deem it necessary to inspect these areas during a follow-up
visit, involving an additional fee. This depends on thevolume, infrastructure and general management of these
activities.
It is an EAQUALS requirement that all activities within an
organisation that are eligible for inspection by EAQUALS(i.e. are related to the provision of language learning
services) are included in EAQUALS accreditation. These
include off-site courses and activities run at branches,annexes, or satellite schools which operate as part of thesame company or under the same trading name, and are
not separately accredited by EAQUALS. The organisation of
the inspections of complex operations with different
branches is prepared in consultation with the institution
and the most cost effective solutions are sought.
The inspection report written by one of the inspectors,
in consultation with the other, gives a full account of theevidence gathered and the ‘verdict’ proposed by theinspectors globally and in each main area. However, theresult of the inspection is conﬁrmed in a verdict letter
issued by the Chair of the inspection sub-committee after
the moderation process within the inspection subcommittee has been completed. This involves threemembers of the committee, who read the report forcoherence and consistency, and double check information
with the inspectors if needed and who may even proposechanges to the statements in the report and grades.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0|

39

The verdict letter conﬁrms that the institution meets
EAQUALS standards in all areas speciﬁed by the inspection
scheme, and indicates the expiry date of accreditation or
re-accreditation. In cases where there is a referral in one ormore areas, the verdict letter makes it clear what action isrequired in order to attain accreditation, and within whattime frame.
The verdict letter also summarises the result of theinspection in a standard format. This paragraph of theverdict letter can be used by the institution as reference inpublicity materials and on the website.

fully comply with EAQUALS standards can be accredited anddisplay the EAQUALS ‘accredited member’ logo. Accredited
members of EAQUALS are recognised as centres of
excellence, which offer a consistently high standard of
language education that meets the world’s language
learning needs.

International certiﬁcation of language
learners’ achievement

Inspector members:

Accredited members may (from 2010) issue certiﬁcates ofachievement indicating the CEFR level attained by each
student at the end of their course. This will involve a simple
additional accreditation process, in which institutions will
have to prove that their systems and procedures forassessing their students’ language levels are valid andreliable. Applications will be evaluated by an expert teamfrom an assessment panel. A rich set of resources has
already been developed by EAQUALS experts to assist
member institutions in developing good quality internal
systems for assessing language competence.

Associate members:This category of membership is for organisations withconsiderable involvement and professional achievement inlanguage education. Associate membership is by invitation
only. The essential criterion is a commitment to quality.

These are individuals who are trained and appointed
EAQUALS inspectors but are not afﬁliated to any EAQUALS
institution. Through their membership they become part of
networking and collaborate on EAQUALS internal projects,
as well as do their work as EAQUALS inspectors andadvisors.
At the time of writing, EAQUALS has 102 accredited
members, 20 associate members in 22 countries, as well as
two project partners. (The current list of members andproject partners can be found on the EAQUALS website:www.eaquals.org)

EAQUALS membership

Conclusion

Membership is open to any institution that is a provider ofhigh quality language training or is, in some other way,committed to the achievement of excellence in this area.
There are three types of members:

Accreditation by EAQUALS is the primary means by which a
provider of language courses can demonstrate theircompliance with international standards designed
speciﬁcally for language education services. Accreditationand full membership is achieved after a rigorous quality
assessment process. The main element of this process is an
EAQUALS inspection based on transparent assessment
criteria, reﬂecting the content of the EAQUALS Charters.
Inspections also provide professional consultancy to help
maintain the momentum of continuous improvement.

Accredited members:

To become an accredited member, language training
institutions prove their full adherence to the Charters bysubmitting themselves to a full and rigorous inspection,
which is repeated every three years. Only institutions that

IELTS Masters Award
Winner of the IELTS Masters Award 2009
The IELTS Research Committee, which comprises membersof the three partner organisations, met in October 2009 toreview the shortlisted submissions. After carefulconsideration, the Committee decided to announce Ruslan
Suvorov as the winner for his thesis entitled ‘Context visualsin L2 listening tests: The effectiveness of photographs andvideo vs. audio-only format’. Ruslan completed his thesis at
the Department of English, Iowa State University, USA andhis supervisor was Associate Professor Volker Hegelheimer.
Ruslan’s full abstract appears below:

Although visual support in the form of pictures and video

has been widely used in language teaching, there appears
to be a dearth of research on the role of visual aids in L2
listening tests (Buck 2000, Ockey 2007) and the absenceof sound theoretical perspectives on this issue (Ginther
2001, Gruba 1999). The existing studies of the role ofvisual support in L2 listening tests yielded inconclusive
results. While some studies showed that visuals canimprove test-takers’ performance on L2 listening tests(e.g., Ginther 2002), others revealed no facilitative effectof visuals on listening comprehension of test takers(e.g., Coniam 2001, Gruba 1993, Ockey 2007).
The given study, conducted at Iowa State University in
Spring 2008, investigated the inﬂuence of context visuals,
namely a single photograph and video, on test-takers’©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.

40|

C A M B R I D G E E S O L : R E S E A R C H N OT E S : I SS U E 3 9 / F E B R U A R Y 2 0 1 0

performance on a computer-based listening testdeveloped speciﬁcally for this study. The listening test,
consisting of six listening passages and 30 multiplechoice questions, was administered to 34 international
students from three English listening classes. In
particular, the study examined whether test takers
perform differently on three types of listening passages:
passages with a single photograph, video-mediatedlistening passages, and audio-only listening passages.
In addition, participants’ responses on the post-test
questionnaire were analysed to determine whether theirpreferences of visual stimuli in listening tests
corresponded with their actual performance on different
types of visuals.

The results indicated that while no difference was foundbetween the scores for photo-mediated and audio-onlylistening passages, participants’ performance onvideo-mediated listening passages was signiﬁcantly lower.
The Research Committee agreed the Ruslan’s research
project skilfully tackled a poorly researched area and makes
a signiﬁcant contribution to further work investigating theuse of context visuals within online listening tests.
Ruslan will be presented with his award (a cheque andcertiﬁcate) at the Annual Language Testing Research
Conference (LTRC) to be held in Cambridge from 14–16 April
2010. For further information about LTRC please visit
www.CambridgeESOL.org/LTRC2010/

37th ALTE meeting and conference, Maynooth, Ireland
The 37th ALTE meeting and conference took place inMaynooth, Ireland from 11–13 November 2009 and washosted by the National University of Ireland, Maynooth. The
ﬁrst two days included a number of Special Interest groupmeetings and workshops, for ALTE members and afﬁliates,
and the third day was an open conference day for anyone
with an interest in language testing.
Michael Corrigan from the ALTE Validation Unit ran aworkshop with Beate Zeidler (telc GmbH) on Linking Existing
Examinations to the CEFR with a particular focus on writing
tests. The workshop discussed the establishment of a
relationship between a local rating scale and the Common
Reference levels of the CEFR, and clariﬁed what the points
on the former mean when interpreted in terms of the CEFR.
The presenters demonstrated a method to establish this linkusing ranking and rating techniques and also provided aplatform for discussion about using this method; and moregenerally of linking pre-existing examinations to the CEFR.
In addition, Professor Jim Purpura from Columbia
University led a workshop on Assessing Grammar for
Learning in which he questioned the assumptions underlying
traditional grammar assessment and assessment practices,
and argued for a learning-oriented model of language
assessment. Dr Tony Green from the University of
Bedfordshire ran a workshop on CEFR Linking and Standard
Setting for Reading and Listening which looked atprocedures advocated in the Council of Europe’s manual for
relating tests to the CEFR levels but also discussed
alternative procedures that have been used by test providers
but which are not covered by the manual. Dr Lynda Taylor,a Consultant to Cambridge ESOL, ran a workshop entitled
Working with Oral Examiners – issues of professional
training, standardisation and monitoring in which she
explored a number of questions such as ‘What makes a good
examiner?’, ‘What sort of training should they receive?’ and‘How might we monitor their performance for purposes ofquality assurance and professional development?’.
Plenary sessions were given by Dr Neil Jones from
Cambridge ESOL who gave an update on The Work of
SurveyLang, Professor Barry O’Sullivan from the Universityof Roehampton who talked about Testing less widelyspoken languages in a multilingual context, Joanna McPakefrom the University of Strathclyde who spoke about Valuing
all languages in a multi-lingual Europe, Dr Steve Coleman
from NUI Maynooth who talked about The Gaeltacht as a
type of community, Siuan Ni Mhaonaigh from NUI
Maynooth who discussed The Development of Irish Tests for
Adults (Teastas Eorpach na Gaeilge), and Professor David
Little from Trinity College, Dublin, who spoke about
Learning, teaching, assessment: how should national
education systems respond to the challenges of the CEFR?
Seán Ó Cuirreáin, Ireland’s ﬁrst Language Commissioner
also gave a very informative presentation on the current
state of the Irish language and the challenges to be met insecuring the language’s future, and Paul Caffrey, a seniorinspector at the Department of Education and Science in
Ireland reported on the challenges of Ireland’s shift from abilingual to a plurilingual society.
Prior to the meeting and conference, Dr Lynda Taylor ran
a 2-day introductory course on the assessment of speaking
which was attended by 24 participants including two from
Cambridge ESOL. Other participants came from Brazil,
Bulgaria, Canada, the Czech Republic, Denmark, Estonia,
Germany, Hungary, Ireland, Israel, Japan, Sweden andWales. On the Saturday following the conference, ALTE
launched its ﬁrst foundation course in Testing Writing,which is part of a new series of short, half-day courses inlanguage testing aimed at new delegates from ALTE
members and afﬁliates, and those with little or no
experience and a limited technical knowledge ofassessment. This ﬁrst session was run by Jacky Newbrook
and Annie Broadhead, Consultants to Cambridge ESOL.
ALTE’s next meeting and conference will take place inRome from 12–14 May 2010 where the theme of theconference day will be on Assessment and Reference Level
Descriptions. The conference will be preceded by a 2-day
testing course (topic to be conﬁrmed) on 10–11 May, anda 1-day seminar on Language and Migration and Integration
on 11 May, and will be followed by one or more foundation
courses on 15 May 2010.

©UCLES 2010 – The contents of this publication may not be reproduced without the written permission of the copyright holder.