Studies of the IELTS test and Progetto Lingue 2000
Language teaching and testing programmes have long been considered toexert a powerful inﬂuence on a wide range of stakeholders, including
learners and test takers, teachers and textbook writers, test developers andinstitutions. However, the actual nature of this inﬂuence and the extent towhich it may be positive or negative have only recently been subject toempirical investigation through research studies of impact.
This book describes two recent case studies to investigate impact inspeciﬁc educational contexts. One analyses the impact of the International
English Language Testing System (IELTS) – a high-stakes English language
proﬁciency test used worldwide among international students; the second
focuses on the Progetto Lingue 2000 (Year 2000 Languages Project) – a
major national language teaching reform programme introduced by the
Ministry of Education in Italy.

Impact Theory and Practice

Impact Theory and PracticeRoger Hawkey

Studies in Language Testing 24

Studies in
Language
Testing

24

Impact Theory
and PracticeRoger Hawkey

Key features of the volume include:• an up-to-date review of the relevant literature on impact, including
clariﬁcation of the concept of impact and related terms such as
washback• a detailed explication of the process of impact study using actual cases
as examples• practical guidance on matters such as questionnaire design, interviews,
permissions and conﬁdentiality, data collection, management and analysis• a comprehensive discussion of washback and impact issues in relation tolanguage teaching reform as well as language testing.
With its combination of theoretical overview and practical advice, thisolume is a useful manual on how to conduct impact studies and will be ofparticular interest to both language test researchers and students oflanguage testing. It will also be relevant to those who are concerned withthe process of curriculum and examination reform.

Studies in
Language
Testing

24
Series Editors
Michael Milanovic
and Cyril J Weir

Impact Theory and PracticeStudies of the IELTS test
and Progetto Lingue 2000

Also in this series:
Performance testing, cognition and assessment: Selected papers from the 15th Language Testing
Research Colloquium, Cambridge and Arnhem
Edited by Michael Milanovic and Nick Saville
The development of IELTS: A study of the effect of background knowledge on reading
comprehension
Caroline Clapham
Verbal protocol analysis in language testing research: A handbook
Alison Green
Multilingual glossary of language testing terms
Prepared by ALTE members
Dictionary of language testing
Alan Davies, Annie Brown, Cathie Elder, Kathryn Hill, Tom Lumley, Tim McNamara
Learner strategy use and performance on language tests: A structural equation modelingapproach
James E. Purpura
Fairness and validation in language assessment: Selected papers from the 19th Language Testing
Research Colloquium, Orlando, Florida
Antony John Kunnan
Issues in computer-adaptive testing of reading proficiency
Micheline Chalhoub-Deville
Experimenting with uncertainty: Essays in honour of Alan Davies
Edited by A. Brown, C. Elder, N. Iwashita, E. Grove, K. Hill, T. Lumley, K. O’Loughlin, T. McNamara
An empirical investigation of the componentiality of L2 reading in English for academic purposes
Edited by Cyril J. Weir, Yang Huizhong, Jin Yan
The equivalence of direct and semi-direct speaking tests
Kieran O’Loughlin
A Qualitative Approach to the Validation of Oral Language Tests
Anne Lazaraton
Continuity and Innovation: Revising the Cambridge Proficiency in English Examination
1913–2002
Edited by Cyril Weir and Michael Milanovic
European Language Testing in a Global Context: Proceedings of the ALTE Barcelona Conference
July 2001
Edited by Cyril Weir and Michael Milanovic
A Modular Approach to Testing English Language Skills: The development of the Certificates inEnglish Language Skills (CELS) examinations
Roger Hawkey
Testing the Spoken English of Young Norwegians: A study of testing validity and the role of‘smallwords’ in contributing to pupils’ fluency
Angela Hasselgreen
Changing language teaching through language testing: A washback study
Liying Cheng
The impact of high-stakes examinations on classroom teaching: A case study using insights fromtesting and innovation theory
Dianne Wall
Issues in testing business English: The revision of the Cambridge Business English Certificates
Barry O’SullivanImpact Theory and PracticeStudies of the IELTS test
and Progetto Lingue 2000

Roger Hawkey
ESOL Consultant

CAMBRIDGE UNIVERSITY PRESS
Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo
Cambridge University Press
The Edinburgh Building, Cambridge CB2 2RU, UK
www.cambridge.org
Information on this title: www.cambridge.org/9780521680974© UCLES 2006
This publication is in copyright. Subject to statutory exception
and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.
First published 2006
Printed in the United Kingdom at the University Press, Cambridge
A catalogue record for this publication is available from the British LibraryISBN -13 978-0-521-68097-4
ISBN -10 0-521-68097-2
The author is grateful to the copyright holders for permission to use the copyrightmaterial reproduced in this book. Every effort has been made to trace the copyrightholders. Cambridge University Press apologises for any unintentional omissions andwould be pleased, in such cases, to add an acknowledgement in further editions.

iv

Contents

Acknowledgementsvi

Series Editors’ note

viii

Chapter 1
Impact, washback, evaluation and related concepts:
definitions and examples

1

Chapter 2
Impact study approaches

29

Chapter 3
Impact study objectives, designs and research questions

41

Chapter 4
Impact study instrumentation

54

Chapter 5
Collecting and managing the data

85

Chapter 6
The IELTS impact study: main messages

99

Chapter 7
The PL2000 Impact Study: main messages

133

Chapter 8
Related developments, further research and lessons learned

162

Appendices

179

References

215

Index

225

v

Acknowledgements

I am sincerely grateful to the many people who helped me, in various ways,
with the impact studies described here, and with the writing and editing of thebook.
I was fortunate enough to be able to meet, at Lancaster, members of the
Lancaster University team who played such a key part in the early phases ofthe study of IELTS impact. From Charles Alderson, Professor of Linguistics
and English Language Education in the Department of Linguistics and English
Language, Dr Jayanti Banerjee, researcher in language testing and assessment
and English for Academic Purposes (EAP), and Tania Horak, Research
Associate currently working on the New TOEFL (Test of English as Foreign
Language), I received valuable insights and advice on both historical andcurrent aspects of the study of impact. My thanks, too, to Dr Dianne Wall,also at my Lancaster meetings, for sharing her considerable and continuingexperience in the study of test impact.
The second group of individuals whose help and support I should like toacknowledge played key roles during the implementation stages of the studies.
Dr Raffaele Sanzo, Co-ordinator of the PL2000 in the Ministry of Education,
Italy, not only helped to facilitate the Cambridge ESOL Impact Study, butoffered encouragement and wise insights throughout its implementation.
Richard Turner, of Cambridge ESOL Research and Validation, who held anexpert camcorder during the video-ing of IELTS impact study preparation
classes, also played a major role in the data management and analysis for both
IELTS and PL2000 studies. Liam Vint, Cambridge ESOL Country Manager
for Italy, turned PL2000 Impact Study camera person, too, in addition toproviding vital contacts and background knowledge throughout planning andimplementation stages. Lee Knapp, Cambridge Development Manager, UK,accompanied me on some of my IELTS impact study visits to UK centres andprovided me with information and views from his many other IELTS-related
contacts. My thanks also to Tony Green, Validation Officer at Cambridge
ESOL, who shared data and validating operations with me for the IELTS
study. I should like in addition, to express my gratitude to Susan Chapman,of Cambridge ESOL Research and Validation, who gave invaluableadministrative support to both studies, and to Rowena Akinyemi and Sally
Downes, who co-ordinated the final editing and production of the book.

vi

Acknowledgements
I am grateful to those commissioned to read and make suggestions on the
manuscript, namely Alan Davies, Emeritus Professor, Department ofTheoretical and Applied Linguistics, University of Edinburgh and Peter
Falvey, Consultant to Cambridge ESOL. To Nick Charge, Subject Manager
for IELTS and Liam Vint, my thanks for reading and advising on parts of thebook of particular relevance to their work. May I acknowledge, too, Dr Mike
Milanovic, CEO of Cambridge ESOL, and Cyril Weir, Professor of English
Language Acquisition at the University of Luton, for guidance, beyond theirduties as SiLT series editors, on content and emphasis.
Finally I should like to acknowledge the constant interest in the projecttaken by Nick Saville and Lynda Taylor, Director and Assistant Director
respectively, of the Cambridge ESOL Research and Validation Group. Theirwise and helpful advice was based both on their interest in and readings of thisbook.
Last, and in some ways most, I should like to recognise all the many
participants in the studies described in this book. To the students and theirparents, the test takers, the teachers, the school and centre heads, the parents,
the administrators and managers, including Cambridge ESOL Local
Secretaries, involved in IELTS and the Progetto Lingue 2000, thank you foryour co-operation and helpfulness, for your contribution to the good cause of
impact study.

vii

Series Editors’ note

Cambridge has always taken a great interest in the educational and social
impact of its tests and assessments. For many years while testing experts
around the world were preoccupied with the quantitative aspects of assessment
worrying about impact was not considered to be particularly relevant. The
Cambridge attention to this area was probably even thought of as slightly
quirky and old fashioned. However, in recent years, the concept of identifying
and measuring how tests impact on the environment in which they operate has
been recognised as a very relevant concern. Indeed, three further volumes inthis series by Liying Cheng, who looks at washback in Hong Kong, Diane
Wall, who documents an impact study in Sri Lanka, and Tony Green, who
focuses on IELTS, demonstrate the growing importance of impact research as
an aspect of test validation. Languages in general and English in particular, areof ever growing importance, not only for economic reasons, but also for socialand political ones. Stakeholders in the language assessment processincreasingly require evidence on the interactions between examinations, thestakeholders involved and the outcomes expected. The effective
conceptualisation of the dimensions of test impact and its systematic study –within the context of test validation research – is one of the ways that will help
us to address this requirement better.
This volume is written from the perspective of an international language
testing agency although the issues discussed are of relevance in national andlocal assessment situations. Roger Hawkey, who has now conducted extensive
work in the area of test impact, considers its dimensions and why
understanding test impact is important. After some discussion of the conceptsof impact and washback and how they fit into a broader educational, researchand social context, he looks at the role of impact studies in the Cambridge
ESOL test development, validation and revision systems, with particularreference to the Progetto Lingue 2000 in Italy and the study of IELTS impact.
In the fields of language teaching and testing, the concepts of washback andimpact, as Hawkey explores in some depth, are a matter of both theoretical andpractical differentiation and concern. Through the 1980s and into the early
1990s attention focused on the concept of test washback and as such took arelatively narrow view, focusing largely on the teaching–learning relationshipwith some attention paid to the role of publishers and course materials. But
beyond the learners and teachers affected by the washback of a language testviii

Series Editors’ noteare a range of other stakeholders on whom an examination has impact eventhough they do not take the test or teach it. These stakeholders, for example,
parents, employers, university admissions officers and others, form what wemight refer to as the language testing constituency. Cambridge ESOL has
defined this constituency particularly in relation to candidates taking its own
examinations but the definition applies in other contexts too. Different tests
will have different constituencies and an examination board like Cambridge
ESOL will be dealing with numerous and varied constituencies, quite possibly
for the same test and at the same time. The stakeholders interact with the test
construct, format, conditions and assessment criteria in various ways.
Cambridge ESOL routinely conducts impact studies as part of the test
validation process on an ongoing basis. It is our view that an examination
board must be prepared to review and revise what it does in the light of how
its stakeholders use and feel about its examinations. As educational processes
and social needs change it is vital that examinations adapt to meet therequirements of these changes and the study of test impact facilitates thisprocess even if the interrelationships involved are complex and highly
context-dependent. Hawkey rightly points out that impact research is an
exemplification of the growing importance of evidence-based approaches to
education and assessment. Evidence-based education requires policy andpractice capable of being justified in terms of sound evidence about theirlikely effects. Given that education, or indeed assessment, is not an exact
science, it is too important to allow it to be determined by unfounded opinion,
whether of politicians, teachers, researchers or anyone else. Validation
research, including research into test impact, aims to seek out the evidence
necessary to develop, redevelop or indeed operate a testing system in an
appropriate and ethical manner.
The role of ethics in language testing has risen to the fore much moresignificantly in the last decade or so. The intention of those concerned withethical language testing is to implement codes of professionally and socially
responsible practice. These codes should provide tighter yet feasible
guarantees of test development rigour and probity, with properly-definedtargets, appropriate and reliable evaluation criteria, comprehensive,transparent and fair test interpretation and reporting systems, continuous
validation processes, and a keener regard for the rights of candidates and otherstakeholders (for example, see the ALTE Code of Practice, the ALTE quality
assurance work, and the IELTS Handbook and Annual Review).
An ethical approach to language testing is a must in the modern world andtest impact studies play an important role in demonstrating that language tests
are used ethically. However, impact studies can also help address some of theconcerns raised by the critical language testing lobby. The critical languageix

Series Editors’ note
testing movement characterises tests as, intentionally or not, biased,
undemocratic, and unfair means of selecting or policy-changing. It is argued
that the main actual impact of language tests is the imposition of constraints,
the restriction of curricula, and the possible encouragement of boring,mechanical teaching approaches.
Whether this is the case or not needs to be a matter of research rather thanopinion and in such a general context a focus on test impact is an importantarea of study. It is driven by considerations in the field of language testing ofwanting to do the job right and providing the appropriate evidence to back any
claims. It is also driven by a broader social, political, educational and evencognitive impetus, and we see again the growing movement in education todevelop the notion of basing what we do on sound evidence. Indeed, theevidence-based education manifesto argues that we need a culture in whichevidence is valued over opinion, and where appropriate action (or inaction) isvalued over action for the sake of being seen to do something. This applies justas much to what critical language testers have to say as it does to the claimsof examination boards, education departments, schools and so on.
This volume is intended to provide the reader with an approach to the studyof test impact which allows evidence to be gathered and displayed. It
documents in some detail aspects of two impact studies that have been
conducted in the Cambridge context and as such, we believe it makes a uniqueand much needed addition to the field. Its focus on the use of international
assessments in state systems in the Progetto Lingue 2000 is relevant as
English becomes a core subject in many countries around the world, and it isvital that there is a good understanding of what impact international
assessment may have. The focus on IELTS is no less significant asinternational mobility continues to increase. The extensive IELTS researchtakes us beyond a narrow focus on the test itself to the broader impact that it
has and demonstrates very clearly that IELTS impacts positively on language
learning and teaching in addition to its well known measurement attributes.
Two further volumes on IELTS will be published soon after this volume.
The first, entitled IELTS Collected Papers: Research in speaking and writing
assessment and edited by Lynda Taylor and Peter Falvey, documents a rangeof research studies with a particular focus on speaking and writing.
The second, written by Alan Davies and entitled Assessing Academic English:
Testing English proficiency 1950–2005 – the IELTS solution, documents
the development of the testing of academic English from the 1950s to thepresent day.
Michael Milanovic
Cyril Weir
2005

x

1

Impact, washback,evaluation and related
concepts: definitions andexamplesImpact is the main topic of this book, so this opening chapter will attempt toclarify the concept and its implications. It will also consider related terms, forexample evaluation, monitoring and washback.
Context for the discussion throughout will be via reference to the role ofimpact studies in the test development and validation systems of the
University of Cambridge English for Speakers of Other Languages (ESOL)
Examinations.
This book will refer to two Cambridge ESOL impact study projects inparticular.
One is the study of the impact of the International English Language
Testing System (IELTS), an examination ‘designed to assess the English
language ability of people whose first language is not English and who needto study, work or live where English is used as the language ofcommunication’ (www.ielts.org). The second is a study of the impact of theProgetto Lingue 2000 (Year 2000 Languages Project), a Ministry of
Education, Italy, state school foreign language education improvement
programme.
This should set the scene for Chapter 2, which considers different
approaches to the collection and analysis of impact data, and Chapter 3, on thedefinition of research objectives and questions. Chapter 4 then traces thedevelopment of impact study instrumentation, and Chapter 5 the collection,
management and analysis of data. In Chapters 6 and 7, some of the main
findings of the studies into IELTS and the Progetto Lingue 2000 impacts arepresented, in their own right and as examples of the outcomes that may be
expected from research into the foreign language learning and testing aspectsof educational impact. Chapter 8 traces research and other developments
related to the two studies, considers lessons to be learned, and suggests
approaches for the continuing study of educational impact.
But first some key terms need to be defined.

11 Impact, washback, evaluation and related concepts

Impact in educational research
Impact of process and productTaking an educational evaluation viewpoint, Weiss defines impact as ‘the net
effects of a programme (i.e. the gain in outcomes for program participants
minus the gain for an equivalent group of non-participants)’(1998: 331). She
then broadens this somewhat narrow definition by adding that ‘impact may
also refer to program effects for the larger community’, and admitting that
‘more generally it is a synonym for outcome’ [all italics mine]. This wider
view of the impact construct is reflected in a definition from developmental
education studies:
Impacts (also referred to as effects) may be planned or unplanned; positiveor negative; achieved immediately or only after some time; andsustainable or unsustainable … Impacts may be observable/measurable
during implementation, at project completion, or only some time after the
project has ended. Different impacts may be experienced by different
stakeholders (Department for International Development (DFID) Glossaryof terms 1998).

Note that this definition of impact appears to include a focus on processes aswell as outcomes or product, a distinction often at issue in impact andevaluation studies. Roy defines process and product studies as follows:
A study of the product is expected to indicate the pay-off value while astudy of the process is expected to indicate the intrinsic values of the
programme. Both are needed, however, to find the worth of the
programme (1998:71).

Weiss defines a process focus more straightforwardly as the study of ‘whatgoes on while a program is in progress’, whereas outcome studies measure anddescribe the ‘end results of the program’ (1998:334–335).
In the field of education, impact studies most commonly focus on theeffects of interventions, including both teaching programmes and tests, on thepeople participating in them in various ways. Given the formative nature of
education and learning, such studies seek to measure and analyse bothoutcomes, for example test results or subsequent performance on the criteria
the test is measuring, and processes, for example the learning and teachingapproaches and activities of programmes preparing candidates for a test.
The study of the impacts of the IELTS test is, by definition, a form ofsummative evaluation, concerned with outcomes such as candidate test
performances. But a test such as IELTS, used as an English language
qualification for academic studies in English-speaking countries and for
immigration, training and employment purposes, also has significant potential
impact on processes such as the ways candidates learn and prepare for the test2

Impact in educational researchitself and for their English language activities beyond it (there is further
discussion on this below). There are thus formative aspects (intended toprovide information to improve programmes or tests) as well as summative
aspects to impact studies. As for the Progetto Lingue 2000 (PL2000) Impact
Study, of a Ministry of Education project for the improvement of language
learning in the state sector, there is a focus on developments in areas where theProject is intended to have influence. These include, of course,
teaching/learning processes and foreign language performance outcomes.
Impact studies of tests, like impact studies of learning programmes, are likely
to be process- as well as product- or outcome-oriented.

Impact studies and evaluation studies
Varghese contrasts impact studies with evaluation studies, the latter, he feels,
tending to focus more closely on the immediate objectives of projects rather
than their longer-term development.
… An evaluation of adult literacy programmes may indicate the total
number of persons made literate by the programme. An impact study ofthe programme will focus on the social implications of the outcomes … It
will also ask, for example, whether the reading habits of the community
improved (1998:49).

Varghese (49–50) reminds us that impacts are changes (or effects) rather than
the achievement of project targets, which are often seen as the focus ofevaluation studies.
Weiss defines evaluation as the ‘systematic assessment of the operation
and/or outcomes of a program or policy, compared to explicit or implicit
standards, in order to contribute to the improvement of the program or policy’(1998:330). The term ‘evaluation’ refers to an overall process; an evaluation
study is, after all, an exercise to appraise (that is, measure the value of) an
educational programme. Impact may well be one of the areas of the
programme covered by an evaluation. So, evaluation and impact are linked,
with evaluation in some cases tending to include impact, in the sense of
programme effects which evaluators want to find out about as part of theirevaluation. Why? In order to make proposals to adjust these effects to‘contribute to the improvement of the program or policy’ (see Weiss above).
But the evaluation study literature (e.g. Agar 1986, Connell et al 1995,
Cronbach 1982, MacDonald 1974, Parlett and Hamilton 1972, Weiss 1998)
warns us regularly that the nature of evaluation should not be over-simplified
as it was following Scriven’s 1967 contrast between summative and formativeevaluation. ‘Evaluations that focus on outcomes’ says Cronbach, ‘can andshould be used formatively’ (1982). Parlett and Hamilton go further with theirconcept of ‘illuminative evaluation’ (1972). This has a primary concern ‘with3

1 Impact, washback, evaluation and related concepts
description and interpretation rather than measurement and prediction’, with
how innovation operates, ‘how it is influenced by the various school situations
in which it is applied; what those directly concerned regard as its advantages
and disadvantages; and how students’ intellectual tasks and academic
experiences are most affected’. Illuminative evaluation ‘aims to discover anddocument what it is like to be participating in the scheme’ (see Murphy andTorrance (eds.) 1987:60–61). Levine’s concept of evolving curriculum issimilarly dynamic, ‘where evaluation is an inherent aspect of the curriculum
planning process (evaluation in planning)’ with ‘the evaluation process itself
a perpetual and self-developmental inquiry process (evaluation as planning).
The curriculum evaluation process that emerges is flexible, yet methodical,
open, yet directive, and respectful of the diverse, complex curricular visions,needs and constraints encountered in schools and classrooms’ (2002:26).
There would seem to be much to learn from these definitions. The impact
studies discussed in this book attempt to combine ‘description andinterpretation’ with ‘measurement and prediction’. They seek to investigate
the influences of ‘the various school situations’ in which the IELTS test is
prepared for and the principles of the PL2000 are put into practice. They
certainly seek to discover ‘what those directly concerned’ regard as the‘advantages and disadvantages’ of the test and the curriculum reform project,
and also how students’ ‘intellectual tasks and academic experiences are mostaffected’. We shall also see, throughout this book, that the study of the impactof language tests or programmes, like the evaluation process, tends to be‘perpetual and self-developmental’ rather than single and monolithic.

Monitoring
Then there is the term monitoring, clearly related to both impact andevaluation, and actually suggested by Italian colleagues participating in the
study of the impact of the PL2000, as a synonym for impact study. Weissdefines monitoring as ‘[a]n ongoing assessment of program operations
conducted during implementation, usually by sponsors or managers, to assess
whether activities are being delivered as planned, are reaching the target
populations, and are using resources appropriately’ (1998:333). Judging by
this definition, there is considerable overlap between monitoring andevaluation, but the fact that monitoring takes place only during theimplementation of a programme may distinguish it.
A further distinction, suggested by Lynda Taylor (2005, personalcommunication) sees monitoring as primarily descriptive in function,
followed by evaluation, which is, naturally, mainly evaluative in function. As
will emerge from this chapter (see Figure 1.5 on page 20) the Cambridge

4Impact in educational research
ESOL model of test development includes, following the establishment of theneed for a new or revised test, the stages of:• design and initial specification• development through trialling, analysis, evaluation and review• monitoring, mainly through routine descriptive data for analysis, until adecision is made, based on particular monitoring information• reviewing and evaluating the test for possible further revision.
Weiss links monitoring with ‘process evaluation’, and adds a participant
dimension:… process evaluation is not very different from what is often called
monitoring. One key difference is that monitoring is done primarily onbehalf of the funders and other high-level officials to hold the program toaccount (1998:181).

One teacher/administrator participant in the study of IELTS impact seemed tosense that impact studies may be less top-down and judgemental when she
described them as more ‘user-friendly’ than evaluations or monitoring.
In the case of the PL2000 Impact Study there was, of course, no questionof the impact study ‘funders’, Cambridge ESOL, holding policy-makers,
designers, managers or officials of the PL2000 to account. Rather, theexaminations board was concerned with the two-way impacts (see below) of
Cambridge exams on participants in the Progetto, and of the Progetto on theseexams. The PL2000 Impact Study was carried out by Cambridge ESOL as aninterested party, selected alongside other international test providers (seebelow), to provide external certification for students who had been
participating in foreign language courses under the PL2000. Cambridge ESOL
was not the initiator or leader of the foreign language reform project itself; itsrole, through the PL2000 Impact Study which it ran with Ministry approval,
was to describe impacts rather than to evaluate the Project.

Insider and outsider roles
The question of the ‘evaluator-user and insider-outsider interface’ is often atissue in the evaluation literature. O’Dwyer summarises as follows:
Evaluators may remain distant and report findings in their own way withthe expectation that these may be used to improve a program; or, may be
actively involved in the program, working hand-in-hand with those in a
program, or stakeholders to the program, with a view to specifying theevaluation focus according to the needs of the users. The profile which an
external evaluator may adopt, therefore, could be of a complete outsider toa program, or, towards the other end of the spectrum, of a close ‘insider’
in relationship to the clients (2005).

5

1 Impact, washback, evaluation and related concepts
The differences in evaluator roles described here would appear to apply toimpact studies as well as evaluations. With the design, trialling andimplementational phases of the study of IELTS impact, which will be
described in detail in Chapters 3–6 below, the outsider-insider roles included,
at various stages, both external consultant teams and individuals
commissioned by Cambridge ESOL, and validation and implementation
expertise from within the organisation. Cambridge ESOL is, of course, one of
the three partners in the IELTS test, along with the British Council and IDP
Education Australia : IELTS Australia. Both these latter partners are fully
informed of the impact studies and themselves contribute to research insupport of IELTS through the IELTS funded-research programme (see thischapter and Chapters 4, 6 and 8). In the PL2000 Impact Study, it will be seen,
relationships between the impact study team and participants such as the casestudy school teachers and heads were close, though not quite ‘insider’.

Impact and washback in foreign language
teaching and testing
In this section of the chapter, the concepts of evaluation, monitoring andimpact are investigated within the fields of language teaching and testing,where similarities with and distinctions from the general educational literature
will be discovered.
In the language teaching and testing literature, the concept of impact aseffects or changes still stands but the term co-occurs frequently with the term
‘washback’ (or ‘backwash’) and it is the distinction between the two that isoften an issue of debate. In the context of studies of the effects of language
programmes or tests on those involved, the concepts of impact andwashback/backwash are often considered in terms of their:
•
•
•
•
•
•
•
•
•
•
•

6

logical location
definition and scope
positive and negative implications
intentionality
complexity
direction
intensity, emphasis
stakes and stakeholders
relationships with validity and validation
relationships with the Critical Language Testing viewrole in impact/washback models.

Impact and washback in foreign language teaching and testingThis chapter attempts below to cover all these aspects of impact andwashback.

Washback and impact‘Washback and the impact of tests more generally has become a major area ofstudy within educational research’ Alderson (2004a:ix) and as the washback
and impact net widens, so does the need for agreed labels for the kinds ofstudy we carry out to investigate the effects of tests or programmes in andbeyond the classroom context. Hamp-Lyons summarises the situation and theterminology well. She finds that Alderson and Wall’s ‘limitation of the term
‘washback’ to influences on teaching, teachers, and learning (including
curriculum and materials) seems now to be generally accepted, and thediscussion of wider influences of tests is codified under the term ‘impact’(Wall 1997), which is the term used in the wider educational measurement
literature’ (2000:586). In similar vein, Bachman and Palmer 1996 refer toissues of test use and social impact as ‘macro’ issues of impact, while
washback takes place at the ‘micro’ level of participants, mainly learners andteachers.
So the term ‘impact’ now appears to be used to describe studies whichinvestigate the influences of language programmes and/or tests on
stakeholders beyond the immediate learning programme context. An impact
study might, for example, investigate the effects of a programme or test on
school heads, parents, receiving institution administrators, high-stakes test
providers (all these stakeholders included in the two impact studies describedin Chapters 3–8 below).
Given that the term ‘impact’ is a word in everyday use in its meaning of‘influence or effect’ (e.g. Oxford School Dictionary, 1994), it is unsurprising
to find the term also apparently used non-technically. When Alderson (2004a:ix), for example, writes: ‘We now know, for instance, that tests will have more
impact on the content of teaching and the materials that are used than they willon the teacher’s methodology’, is he using the term in its lay sense, sincetechnically the content of teaching and the teacher’s methodology are
washback rather than impact matters? Or is he acknowledging that, for some,
impact, the broader construct, includes washback? Green notes that although‘the terms have been used to refer to the same concept, backwash is
distinguished from test impact by Bachman and Palmer (1996:30) who, with
McNamara (1996, 2000), Hamp-Lyons (1998) and Shohamy (2001) place
washback within the scope of impact’ (2003:6). This would presumably meanthat one could use the term ‘impact’ for all cases of influence from a language
test or language programme, whether on teaching and learning or on, say, auniversity’s admissions policy.

7

1 Impact, washback, evaluation and related concepts
Andrews, writing on washback and curriculum innovation, appears to
acknowledge the fragility of the washback : impact distinction:
The term washback is interpreted broadly … the present chapter uses
washback to refer to the effects of tests on teaching and learning, theeducational system, and the various stakeholders in the education process.
Where the word ‘impact’ occurs in this chapter, it is used in a nontechnical sense, as a synonym for ‘effect’ (2004:37).

In this book we shall try to be consistent in the use of terms:• to use ‘washback’ to cover influences of language tests or programmes on
language learners and teachers, language learning and teaching processes(including materials) and outcomes• to use ‘impact’ to cover influences of language tests or programmes on
stakeholders beyond language learners, teachers, except when it is theinfluences of a test or programme on learners and teachers outside theirlearning or teaching roles, for example on their attitudes to mattersbeyond language learning; in this case the book will tend to refer to
impact e.g. Research Question 4: What is the impact of IELTS on theparticipants who have taken the test?
In terms of these definitions, the two studies which are the focus of this bookcover both washback and impact. They are called ‘impact studies’ because ofthis breadth.

Washback/backwash
Hamp-Lyons notes that washback ‘is one of a set of terms that have been usedin general education, language education and language testing to refer to a setof beliefs about the relationship between testing and teaching and learning’(1997:295). Another of the ‘set of terms’ is ‘backwash’, but it would appear
that the terms ‘washback’ and ‘backwash’ are used interchangeably in thefield. ‘… to clarify the distinction between the terms backwash andwashback’, Alderson says (2004a:xi), ‘there is none’. Hughes admits that
there is interchangeable use of the two terms in his work but adds, (2003:57)‘Where “washback” came from I do not know. What I do know is that I canfind “backwash” in dictionaries, but not “washback”’. Cheng and Curtis
choose to use the term ‘washback’ ‘as it is the most commonly used in thefield of applied linguistics’ (2004:5). This book will follow suit, preferring theterm ‘washback’ as it does now appear to be in more common use in the field.

8Impact and washback in foreign language teaching and testing

Impact and validity
Saville and Hawkey cite ‘the implementation of new national curricula withnational achievement tests’ (2004:75) in the UK and New Zealand as
examples of a growing tendency for tests to be used to provide evidence of andtargets for change, thus having more significant influences on the lives ofindividuals and groups.’ Language tests such as IELTS are more and more
frequently used in a ‘gate-keeping’ role in decisions of crucial importance tocandidates such as the admission or otherwise to particular programmes,
professions or places. They thus earn the label of ‘high-stakes’ tests. The
social consequences of test use are a growing concern.
Messick insists on the inclusion of the outside influences or ‘consequential
validity’ of a test in its validation, ‘the function or outcome of the testing,
being either interpretation or use’ (1989:20). In an interesting personalcommunication to Alderson, however, Messick warns against too glib a viewof the relationship between test washback or impact and test validation.
Washback is a consequence of testing that bears on validity only if it canbe evidentially shown to be an effect of the test and not of other forces
operative on the educational scene … Washback is not simply good or badteaching or learning practice that might occur with or without the test, butrather good or bad practice that is evidentially linked to the introduction of
the use of the test (Alderson 1995:3).

Alderson (1995:4) himself takes ‘an agnostic position’ on the relationshipbetween test impact and test validity. He agrees that ‘test consequences areimportant and may relate to validity issues (bias being perhaps the mostobvious case)’ but has ‘difficulty seeing the washback and impact as central
to construct validity’ because of the ‘myriad factors’ impacting on a test:
teacher’s linguistic ability, training, motivation, course hours, class size,
extra lessons and so on. ‘This is not, of course, to deny,’ Alderson notes in his
paper written for Phase One of the study of IELTS impact (see also Chapters
2 and 4), ‘the value of studying test impact and washback in its own right, but
it underscores the need to gather evidence for the relationship between atest and its impact on the one hand, and of the futility, given current
understanding and data, of making direct and simplistic links between
washback and validity’Alderson (1995:3).
Green agrees that backwash ‘is not generally considered to be a standard for
judging the validity of a test’, because ‘backwash can only be related to a test
indirectly, as effects are realised through the interactions between, inter alia,
the test, teachers and learners’ (2003a). Green cites Mehrens (1998) on ‘thelack of agreed standards for evaluating backwash’ and the fact that ‘different
stakeholders may regard the same effects differently’. There is interestingfood for thought in Messick’s 1996 advice, also cited by Green: ‘rather thanseeking backwash as a sign of test validity, seek validity by design as a likely
basis for backwash’ (1996:252).
9

1 Impact, washback, evaluation and related concepts
Hamp-Lyons links the increasing importance attached to tests to thewashback/impact relationship, claiming that the ‘shift from washback to
impact suggests a growing awareness by language testers that the societies inwhich and for which we work are, whether we approve or not, using tests astheir levers for social and educational reform’ (2000:586). Actually, as Alan
Davies points out (personal communication), this is by no means a newphenomenon, being a feature, for example, of the Civil Service examinationsin India in the 19th century.
But Bachman (2004) still feels that validity and test use are not necessarily
accepted as related in language assessment, despite Messick (1989) andBachman’s own earlier view (1990). In the Bachman and Palmer 1996
definition of test ‘usefulness’, entailing six qualities: reliability, constructvalidity, authenticity, interactiveness, impact and practicality, ‘both theconstruct validity of our score-based inferences and the impact, orconsequences, of test use need to be considered from the very beginning of test
design, with the test developer and test users working together to prioritise therelative importance of these qualities’ (2004:5). Bachman considers thatconsiderations of validity and impact are thus subsumed ‘under a unitaryconcept of test usefulness’. In Chapter 8, Bachman’s case for the articulationof assessment use arguments, in terms of claims, warrants, backing andrebuttals is discussed. These could well feature in a model of test impact study.
Whether impact is intended or unintended, it would seem to be a legitimate
and crucial focus of research, both micro and macro, to ‘review and change’
tests and programmes in the light of findings on, among other aspects ofprogrammes or tests, ‘how the stakeholders use the exams and what they think
about them’ (Saville 2003:60). This is a justification, of course, for studies ofthe effects of exams as part of the test validation process, that is ‘the processof investigating the quality of test-based inferences, often in order to improve
this basis and hence the quality of the test’ (McNamara 2000:138).

The location of impact studies in programme and testdevelopment
Figure 1.1 suggests a sequence of washback and impact events in the context
of a new educational programme.

10Impact and washback in foreign language teaching and testing
Figure 1.1 Sequence of washback and impact occurrences and their study inrelation to a new educational programmeNew
programme

Study of washbackof known curricular
content andapproaches of new
programme(including
promised tests) on:
student learning,
attitudes,
performance andteacher, teaching,attitudes, methods,
materials,
performance etc.

Programme
tests andscoresStudy of washbackof test performanceon contents andapproaches of future
courses
Study of impact
of test content, task
formats on candidate
results, attitudes,
futures, programme
implementation,
support and othersystems

Changes
to programmes
tests, othersystems on thebasis ofwashback andimpactfindings

As the chart indicates, washback (that is the effects of an educational
intervention, for example a new foreign language programme such as thePL2000 in Italy, on language teaching, learning, performance) occurs
throughout a first implementation of a new language course because thecurriculum based on the principles of the Project is known before the deliveryof the courses. Washback also takes place in subsequent courses in the newprogramme because of the experience of the processes of the first
implementation and because of the test performances and scores of learners on
the tests at the end. The first occurrence of washback in Figure 1.1 is a ‘bowwave’ from the programme curriculum and the prospects of the test rather thanbackwash from a previous programme. The second occurrence is indeed
washback from the course and test(s) that have already taken place.
The examples of ‘impact’ in Figure 1.1 indicate that the term is used toinclude the effects of a programme or test beyond washback, to include
stakeholders, domains and systems in addition to language learners andteachers and their immediate context. The main point of Figure 1.1. is to see
the washback and impact concepts in action and to suggest where they occur
in the sequence of impact study events. Notice here, as in Figure 1.2, the final
arrow indicating continuing washback and impact and the iterative nature ofits study.
Figure 1.2 similarly indicates the potential washback (here the effects of aninternational gate-keeping language test such as IELTS on test preparation
programmes) and impact (effects of the test on candidates’ futures at receiving
institutions (RI) and the admissions policies of those institutions). Again,
where washback and impact occur, they may be studied. The data collected on
test washback and impact may inform changes designed to improve the test
and related systems.

11

1 Impact, washback, evaluation and related concepts

Figure 1.2 Sequence of washback and impact occurrences and theirstudy in relation to a high-stakes test
Need totake future
IELTS test

Study of washbackof upcoming IELTS
test on prep course,
student learning,
attitudes,
performance andteacher, teaching,attitudes, methods
materials,
performance

IELTS
Tests andscoresStudy of washbackof test performanceon contents andapproaches of future
courses
Study of impact
of test content, task
formats on candidate
results, attitudes,
futures

Changes
to programmes,
tests, othersystems on thebasis ofwashback andimpact findings

Impact of test andresults on RI
teachers, RI
administrator
attitudes, admissions,policy-makers etc.

Washback: negative or positive – intended or unintended?
Cheng and Curtis claim that it was Alderson, in 1986, who, in a context oflanguage testing innovation, ‘identified washback as a distinct – and at thattime emerging area within language testing, to which we needed to turn ourattention’ (2004:5). Alderson refers to ‘potentially powerful influence offsets’(1986:104) from testing to curricular change. Hamp-Lyons (2000) is also
informative on both the implications, the scope and the status of the term
‘washback’ which, she claims, need not imply ‘curricular alignment’, that is anarrowing of curricula to fit closely and exclusively with the demands of atest. In the Alderson and Hamp-Lyons interpretations (as in, for example,
Bachman and Palmer 1996, Davies et al 1999) we see test washback as leading
to both positive and negative effects. If this is the case, washback itself should,
logically, be regarded as neutral. Alderson and Wall (1993:121) suggest as
such, with washback a ‘metaphor which is useful in that it encourages us to
explore the role of tests in learning and the relationship between teaching andtesting’. Alderson notes, however, that: ‘Although the possibility of positive
washback has also often been mooted, there are, interestingly, few examplesof this having been demonstrated by careful research’ (2004a:x). Perhaps theimpact studies reported in this book will demonstrate further such examples.
Green and Hawkey illustrate a historical expectation of negative washbackfrom modern language tests, ‘the kind of harmful influence that poorly
designed tests are said to exert on the classroom’ (2004:66) and, it is implied,
into the target language use domain (TLU, defined by Bachman and Palmer

12Impact and washback in foreign language teaching and testing
1996:44–45 as “a set of specific language use tasks that the test taker is likely
to encounter outside the test itself”). Andrews (2004:39) traces the opposite
trend, towards positive washback, most clearly, if controversially, exemplified
in measurement directed instruction (MDI) ‘when a high-stakes test of
educational achievement influences the instructional program that prepares
students for a test’ (Popham 1993, cited in Chapman and Snyder 2000:460).
MDI is a clear example of intentional washback, with the effect of the testplanned in order to influence learning and teaching towards positive outcomes(and, perhaps, processes).
In foreign language teaching and learning terms, in particular in terms oftests (such as IELTS) claiming a communicative language construct, a view oflanguage as purposeful, dynamic, interactive, involving users’ grammatical,
sociolinguistic and discourse competences, washback might be intended. Suchtests might be attempting to encourage language learning and teaching where
the communicative activities of the target language domain, and language
teaching, learning and use on the course are similar or overlap significantly.
Figure 1.3 adapted from Green indicates a test washback/construct overlap
model.
Figure 1.3 Green’s model of test washback: target language
construct overlap
TLU
constructs

Source: Green (2003a:23)

OVERLAP

Test design
and formats

Potential for
positive washback

An example of the issues raised by this model would, for instance, be a test
whose construct and approach led to integrated rather than discrete item test
formats, the first closer to the target language use constructs, the latter,
perhaps, more reliably scored.
But, of course, the intention of construct/format overlap and an intended
positive washback on learning and teaching activity towards target language
performance which meets the communicative requirements of the test and of
the learner’s needs, may be too good to be true. Washback is complex, with agreat many independent, intervening and dependent variables. Alderson andWall (1993) famously state fifteen washback hypotheses, suggesting a test’s
potential influence on: the teacher, the learner, what and how teachers teach,
how learners learn, the rate and sequence of learning, attitudes to teaching and

13

1 Impact, washback, evaluation and related concepts
learning methods. In this interpretation, washback is complex, broad andmulti-faceted indeed. Milanovic and Saville also emphasise the broad scope ofand complicated relationships within the washback concept, including as it
does the ‘complex interactions between the factors which make up theteaching/learning context (including the individual learner, the teacher, theclassroom environment, the choice and use of materials etc)…’ (1996:2).

Stakeholders
For many involved with studies into the effects of language tests or language
educational programmes, research questions are raised which concern a whole
range of people, usually called ‘stakeholders’ ‘because they have a direct orindirect interest (stake) in a program or its evaluation’ (Weiss, 1998:337).
Weiss goes on to specify stakeholders as ‘people who conduct, participate in,fund, or manage a program’, then widens the net still further by adding ‘or
who may otherwise affect or be affected by decisions about the program or
evaluation’.
Rea-Dickins attempts a comprehensive list of potential stakeholders inlanguage testing, including:… language testers, teachers, parents, administrators, teacher educators,
sponsors and funding bodies, government bodies, the public, various
national and international examination authorities, members of working
parties and curriculum committees, test takers (and the larger group oflearners of whom they form part) … and to this list we should add test
administrators as well as test users, for example university admission
officers who need to interpret scores on national and international
proficiency tests … (1997:305).

Of course, it is unlikely that all these stakeholders would be included in any
particular study, and there is always a danger of casting the evaluation or
impact study net too widely. Note, however, that the Rea-Dickins
stakeholder list already stretches beyond the immediate language learning,teaching, testing context. This is because, if studies of the effects ofprogrammes and tests are limited in their scope to the albeit complex andwide-ranging effects on teaching and learning, they would not coverprogramme and test effects on the full range of stakeholders whose reactions
and attitudes may be relevant to the programme or test’s validity andusefulness. As the stakeholder coverage widens, a study may be movingfrom washback to impact territory.
It will be seen that the studies of IELTS and PL2000 impact seek
information from more than learners, test candidates and teachers, yet by nomeans from all the stakeholders on Rea-Dickins’s list.

14Impact and washback in foreign language teaching and testing
The stakeholder concept is certainly an important one in the washback andimpact study literature. Figure 1.4 below shows the wide range of stakeholders
involved in various ways in the development, construction and administrationof high-stakes tests.
Figure 1.4 Stakeholders in the testing community

Learners

CAMBRIDGE ESOL

Parents/carers

Teachers
School owners

Teachers

Test construct

School owners

Test writers
Consultants

Test centre
administrators
Materials writers

Receiving
institutions

Test format

Examiners
RatersLearners

Test scores
Test conditions

Test assessment
criteria

Publishers
inter alia

Government
agencies
Professional
bodiesEmployers
Academic
researchersinter alia

Source: Taylor (1999)

As Saville indicates, such a ‘taxonomy of stakeholders’ demands of an
exam provider such as Cambridge ESOL, systems that ensure that it can‘review and change what it does in the light of findings on how the
stakeholders use the exams and what they think about them’ (2003:60). Thisis a strong justification, of course, for studies of the effects of exams as partof the test validation process, the views of the stakeholders concerned being
taken into account along with other impact data, in decisions on test revisionor renewal.

Critical language testing and the study of impact
As recently as the early 1990s, Alderson and Buck were lamenting that ‘there
has not (yet) been a call for public accountability on the part of the examining
boards’ (1993:21). The 1992 discussion paper by Milanovic and Saville,
Principles of Good Practice for UCLES Examinations, could, however, beseen as just such a pioneering step:15

1 Impact, washback, evaluation and related concepts
It is recognised that UCLES has a responsibility to be held accountable forall matters related to use of its examinations. This involves providing ahigh quality service to the users of UCLES examinations which meets anagreed code of practice … (1992:5).

Saville and Hawkey note that ‘[i]n tune with increasing individual and societal
expectations of good value and accountability, testers are expected to adhere
to codes of professionally and socially responsible practice’ (2004:75). This
tends to increase the concern of high-stakes exam providers with the ethics oflanguage testing. Such codes (for example, of the International Language
Testing Association (ILTA), 1997, or the Association of Language Testers in
Europe (ALTE), 1994), were intended to improve test development rigour andprobity, through measures such as the following:• properly-defined targets• appropriate and reliable evaluation criteria• comprehensive, transparent and fair test interpretation and reporting
systems• continuous validation processes• a keener regard for the rights of candidates and other stakeholders(For more information see the ALTE Code of Practice 1994, and the IELTS
Handbook 1997/8.)
As on so many key language testing matters, Spolsky himself is instructiveon the question of testing ethics, taking, as Hamp-Lyons points out, the stanceof a ‘humanistic sceptic’ (2000:587). Spolsky talks of ‘competing sets offorces’ namely the institutional context of a test, the need for general
acceptance, and the ‘unavoidable uncertainty’ or ‘probable error’. He
continues:
What I have labeled post-modern may be seen as an approach that accepts
the equal approach of all three factors … Post-modern testing adds asincere, ethically driven consideration of the potentially deleterious effects
of testing on the test taker, on the instructional process, and on other facetsof the social context in which we test (1995:354–357).

‘Sincere, ethically driven consideration’ of the effects (not only ‘potentially
deleterious’ but also positive) of testing ‘on the test taker, the instructional
process, and on other facets of the social context in which we test’ might beconsidered an admirable impetus for an impact study. We may recall,however, that the term impact is in itself neutral (see above). The common
conviction (on which both the studies described in this book will provide data)is that high-stakes tests always do seem to have washback.
As implied by the overlap model shown in Figure 1.3, there is a demand for
positive test washback and impact. A test may have optimal validity as ameasure of the target ability and thus satisfy the demands of usefulness,
16Impact and washback in foreign language teaching and testingalthough not representing entirely actual performance of that ability. The
demand is for test task formats as close as possible to the communicative
construct. A further reality, of course, as Green points out, is that ‘learners,
teachers, administrators and other participants may have competing goals.
Thus effects regarded as positive by one constituency within an educational
system may be seen as negative by another’ (2003:11). Green refers to thedisagreements over measurement-driven instruction in the USA (Airasian1988, Bracey 1987, Popham 1987, Ramirez 1999) to exemplify this.
The apparent non-neutrality of language tests is a concern for the
proponents of the critical language testing view. Shohamy considers that tests
are ‘powerful because they lead to momentous decisions affecting individuals
and programs … . They are conducted by authoritative and unquestioning
judges or are backed by the language of science and numbers’ (1999:711).
‘Critical testing’, Shohamy adds, ‘refers to the need to examine the use of tests
within a broader social and political critique of aspects of testing as a social
and institutional practice’ (1999:714). Given the definitions of impact study
emerging above, it should be possible to use such studies in the interests ofsolving some of the problems identified by the critical language testers.
Shohamy herself suggests this:Studies of the use of tests as part of test validation on an ongoing basis are
essential for the [testing] profession. Language tests fall at the crossroadsof many conflicts and should therefore be studied, protected and guarded
as part of the process of preserving and perpetuating democratic cultures,values and ethics (2001:390).

Shohamy’s two themes here, of the study of ‘the use of tests’ (which she also
calls ‘research on impact of language tests’) as an element in continuous test
validation systems and as part of the ethical validation of tests, recur in thisbook, both in the discussion of impact study policy in general, and in theprinciples and practices of the studies of IELTS and PL2000 impact. Later inthis chapter such studies are contextualised within the continuous validation
systems of Cambridge ESOL.
Alan Davies sounds a note of caution, however, on the ethical and criticallanguage testing case. Davies agrees that the language testing profession
should attempt to ensure the validity of its activities, and that it ‘may therefore
be sensible to equate ethics with validity’, but there are limits:
An ethical perspective for a language tester is, I have argued, necessary.
But I have also urged the need in all professional statements of morality fora limit on what is achievable or even perhaps desirable. In my view,
therefore, the apparent open-ended offer of consequential validity goes too
far. I maintain that it is not possible for a tester as a member of a profession
to take account of all possible social consequences (1997:335).

17

1 Impact, washback, evaluation and related concepts
A balance is clearly needed in impact research between two extremes. One
would tend to neglect the social consequences or consequential validity of ahigh-stakes test; the other would tend to enquire into its effects on too many
stakeholders, or pursue too many intervening variables for clear washback or
impact connections to be made.

Good practice and test validation
So, with international examinations of ever-higher stakes (the candidature for
IELTS has grown from 50,000 to 500,000 in ten years), and the growing
importance of codes of practice, examination boards need to accept
responsibility for the impacts of their tests on stakeholders. They do thisthrough systems that ensure good practice in test development andmanagement. For Cambridge ESOL, one way of enhancing these systems wasparticipation, with leading international partners, in the research andnegotiation leading to the Code of Practice of the Association of Language
Testers in Europe (ALTE). This code of practice (see previous) made strong
statements of provider responsibility for exam development, scoring, resultinterpretation, fairness and transparency.
Saville (2003:65–78) summarises the implications of such good practice as
the need to pursue test validation, namely to make every systematic effort toensure that a test or exam achieves:• appropriacy to the purposes for which it is used
• the ability ‘to produce very similar results in repeated uses’ (Jones 2001)• positive influence ‘on general educational processes and on theindividuals who are affected by the test results’ (Saville 2003:73), and• practicability in terms of development, production and administration.
In the theory and practice of Cambridge ESOL test research and development,these four exam targets are labelled validity, reliability, impact andpracticality (VRIP for short). The overlap with Bachman and Palmer’s 1996
six test usefulness qualities, reliability, construct validity, authenticity,interactiveness, impact and practicality (see previous) is neither insignificant
nor coincidental given the close relationship of Bachman with UCLES. On the2005 Cambridge ESOL website, reference is still made to activities planned asa follow up to the work of Lyle Bachman and colleagues, on what was known
as the Cambridge-TOEFL Comparability Study, carried out between 1987–9.
So impact, the main focus of this book, encompassing (in our interpretation
above) and closely associated with washback, is firmly set in the VRIP
context. According to Saville, impact studies cover three major groups ofstakeholders: the examination developer, the examination taker, and theexamination user, that is someone ‘who requires the examination for somedecision-making or other purpose’ (2003:60).

18

Impact studies and validation
Although there are four components to VRIP, it is clear that they are by nomeans independent. Test validity, for example, in the unitary constructproposed by Messick (1989:16), subsumes reliability, impact and practicality.
‘Individual examination qualities’, Saville agrees, ‘cannot be evaluatedindependently. Rather the relative importance of the qualities must be
determined in order to maximise the overall usefulness of the examination’(2003:61). Weir (2004), in his socio-cognitive framework for validating tests,
has a similar perspective. For him, test validity is the superordinate category
to theory-based validity (covering internal language ability processes), context
validity (the appropriateness of the communicative and administrative context
in which the test takers are called upon to perform) and scoring validity (thedependability of test results, subsuming the conventional category of
reliability). Test validity also subsumes the two post-test validities, concurrent
and consequential, the latter, of course, including the study of the impacts ofthe test on stakeholders.
But Weir reminds us that test ‘validity is perhaps better defined as theextent to which a test can be shown to produce data, i.e. test scores, which are
an accurate representation of a candidate’s true level of language knowledge
or skills’. Validity ‘resides in test scores (not tests)’ (2004:12). Despite thecommon shorthand reference to valid tests, it is, Weir notes, ‘inaccurate to talkof a test such as TOEFL or IELTS as being valid or not. It is the scoresproduced by a particular administration of a test on a particular sample ofcandidates that we are concerned with’. ‘Obviously,’ Weir adds, ‘over time ifvarious versions of a test or administrations of the same test provide similar
results then synthetically a case may be made for X or Y test being valid overtime and across versions and population samples’ (2004:11). McNamara
suggests that validity is about ‘the relationship between evidence from test
performance and the inferences about candidates’ capacity to perform in thecriterion that are drawn from the evidence’ (2000:138).

Impact studies and validation
Despite the complexities of washback and impact, and despite the multifaceted nature of even a unified model of language testing validity, impact
studies remain part of the test validation process. The tests need to takeaccount of Messick’s consequential validity. As he says, ‘the social values andthe social consequences [of a test] cannot be ignored in considerations ofvalidity’ (1989:20). And, as we have seen above, the study of test impact is but
one element in a continuous and iterative test validation process or, in
Shohamy’s words, ‘part of test validation on an ongoing basis’ (2001:390).
The cycles of exam development, validation and revision are the major
theme of three recent volumes in the Studies in Language Testing (SILT)

19

1 Impact, washback, evaluation and related concepts
series. Volume 15 traces the continuity, innovation and revisions of theCambridge Certificate of Proficiency in English (CPE) examination over 90
years, and Volume 16 describes the development of a modular exam, theCertificates in English Language Skills (CELS), developed from several
precursor exams. Volume 17 explains Cambridge ESOL’s position on
language testing for specific purposes and describes the rationale for changes
implemented in the revision of the Business English Certificates (BEC). In all
three cases, the continuing, iterative and increasingly transparent nature of testdevelopment, validation and revision is clear. So, increasingly, is the interrelated nature of validity, reliability, impact and practicality (VRIP), and thesuperordinacy of validity. As data relevant to the four inter-related validity
elements emerge with every administration of an international exam, so thevalidation of the exam continues through further iterations. Impact data,
whether from routine feedback systems, for example Cambridge ESOL’s
Candidate Information Sheets (CIS), or through test washback and impact
investigations such as the continuing IELTS study, the first three phases ofwhich are described in this book, feed into the validation process andcontribute, along with other validity, reliability and practicality data, to theexam evolution and revision process.
Figure 1.5 Cambridge ESOL model of the test development process
Perceived need for new
or revised test

➤

Start

Revision
Planning Phase

Design Phase
Initial speciﬁcations➤

Evaluation

Operational Phase

Monitoring Phase

➤

Live
Test

Trialling
Analysis
Evaluation/Review➤

Development Phase

Review➤
20

Indicates collection and consideration of washback/impact dataImpact studies and validation
The standard current Cambridge ESOL model of the test development orrevision process is illustrated in Figure 1.5, with indicators added ( ➤ ) of theuse of washback and impact study data in the process.
The indicators of test impact study data collection and use added to Figure 1.5
suggest the following about their role in the Cambridge ESOL testdevelopment process:• that impact data may inform decisions to revise a test
• that data from impact studies will be part of the test trialling process• that such data represent part of the test monitoring process, alongside, forexample, routine candidate profiling and test performance analyses acrosstest versions and tasks• that when a test is evaluated for possible revision, data from impact study,
based on questions on the evaluation research agenda, will be included in
the decision-making.
The aim of a test development project is, after all, to produce a test which will
have positive impact, or at least will have minimal negative impact. The
validation process, of which impact study is a key element, is to collectadequate evidence to support the test developer’s claims about the test’s
suitability for its intended purpose.
When Saville further defines Cambridge ESOL procedures for theachievement of good practice, he re-emphasises the important role of impact
studies:
The approach now being formally adopted recognizes the importance ofvalidation through the collection of data and the role of research anddevelopment in examination processes. In this respect, the principles …
will continue to evolve over time as research and development
programmes expand (2003:57).

We can envisage impact data of various kinds being taken into account at
different stages of the test development cycle, as indicated in Figure 1.5.
Washback and impact data would be considered alongside the descriptive
statistics, classical and Rasch-based analyses carried out on Cambridge ESOL
exams at trial and operational phases (see, for example, Saville: 2003:90–96).
It is significant, however, that Figure 1.5 does not yet contain the terms
washback or impact, which, as Saville appears to imply, is the most recent
element in the VRIP combination.
It would seem that the principles of impact study use in Cambridge ESOL
test development, validation and revision appeared before explicit reference toimpact. Clapham describes how, in 1986, the Revision Committee for the
English Language Testing Service (ELTS 1980–1989: see Alderson andClapham 1993, Criper and Davies 1988), the predecessor of IELTS, ‘sent out

21

1 Impact, washback, evaluation and related concepts
questionnaires to ELTS users such as test administrators, EAP teachers andtesters and receiving institutions such as universities and colleges asking forcomments on ELTS and for advice about the content and format of the future
test battery’ (1996:4). Clapham adds that interviews and discussions with
British Council personnel and English for Academic Purposes (EAP) teachers
and testers were held, and 1,000 past candidates’ test report forms studied.
These procedures, one is tempted to say, amount to the study of ELTS impact,by any other name. Cambridge ESOL was certainly involved in the study of
impact significantly before the term itself was routinely used.
Given the broad scope proposed in this chapter for test washback andimpact data and the iterative applications of such data in the test development
process outlined in Figure 1.5, a further question arises. Where, if anywhere,
do impact data end and the routine post-test monitoring and analysis of testscores and performance data begin? Are test impact and washback data
additional to routine validation data, and data requested by the test validators
because of particular emerging validation, or test revision-related queries? Or
could the routine test development and validation systems be regarded as‘intrinsic’ impact study, the non-routine additional study data as ‘extrinsic’?
At present, the ‘intrinsic’ and ‘extrinsic’ research do share some areas of datacollection, for example, test candidate profiling. The Cambridge ESOL test
validation process certainly benefits from both the routine, mainly post-test
validation investigations and analyses, and from the more ‘extrinsic’ butnevertheless planned and systematised study of test impact.
It seems likely, too, that different potentially useful impact data suitdifferent kinds of data collection contexts. Candidate test performance data,
for example, to be correlated with candidate baseline data, with inferences tobe drawn, for example, on test task validity, might seem more appropriately
handled through routine post-test validation systems. Candidate attitudes tothe test and to their preparation for it would, on the other hand, suit multifaceted investigation through the questionnaires, interviews and classroom
observations of an impact study.
Saville suggests that, from a validation perspective, Cambridge ESOL must
‘be able to monitor and investigate the educational impact that examinations
have within their contexts of use’. To this end, procedures are needed ‘tocollect information that allows impact to be examined’. These proceduresshould include ‘collecting data on: candidate profiles, exam result users andpurposes; test preparation courses; public and participant perceptions of theexam’. Saville refers to a ‘systematic approach to investigating impact’ as‘developed since the mid-1990s’ which includes ‘procedures to monitor the
impact of IELTS as part of the next revision cycle’, and representing an
example of ‘the continuous, formative, test consultation and validation
programme pursued by UCLES and which has also been a significant part of
the FCE and CPE Revision Projects’ (2003:73–76).

22

IELTS impact research in context
The two impact investigations used as case studies in this book illustrate
this means of contributing to the development and validation of high-stakes
tests in different ways. In the case of the study of IELTS impact, the data arebeing collected with both a pre- and a post-test validation intent. Along withresearch data from other sources, the results of our enquiries into candidate
profiles, their attitudes, test perceptions, preparation and performances; theexperiences and views of their teachers, the nature of test-specific preparationmaterials and the impact of the test on receiving institutions, are all being usedin the monitoring and review of the existing IELTS test. The data are thus
post-test but the impetus for the study of IELTS impact was, from the
beginning (see Saville 2003), intended for use in decisions on potential IELTS
revisions. In that sense, the study also has a pre-test perspective.
The study of the impacts of the PL2000 also has a test validation aspect.
One of the research questions for the study (see Chapter 3) was: ‘What
impacts is the PL2000 having on language evaluation and assessment?’ Datain response to this question (see Chapter 7) were informative not only on
which of the Cambridge ESOL Main Suite exams was suitable at which
student English language performance level, but also led to changes in theformat and use of certain exams (see also Chapter 7). Note again here both apost-test and test revision element in the study.
This book will also discuss limitations on the scope of impact studies, someof them practical, some more a matter of principle. Limitations on both our
example studies are described in Chapters 3–7, as is the constant concern with
impact study ethics.
This chapter has so far explored the concept of impact, including washback,with reference to related terms in the field of education, including evaluation,
monitoring and validation. The means and implications of language
programme and test washback and impact have been discussed in terms of
their logical research location, definition and scope, positive and negative
aspects, intentionality, complexity, stakeholders, and the relationships ofwashback and impact studies with test standards, ethics and validation. The
intensity of washback and impact, sometimes also included in lists of theircharacteristics (e.g. Cheng 1997, Watanabe 2000), would seem more logicallyleft until our actual washback and impact data are analysed.
The focus next will be on the role of the studies of IELTS and PL2000
impact in relationship with other Cambridge ESOL test research anddevelopment.

IELTS impact research in context
The impression should be emerging that some impact studies are not to bethought of as monolithic, for example as ‘The IELTS Impact Study’, as if there23

1 Impact, washback, evaluation and related concepts
could be once-and-for-all enquiry which brings closure to the quest foranswers to key impact questions on a high-stakes test that is still in operation.
As will be seen when the history of IELTS impact study is recounted in
Chapter 4, the impact study phases we shall be referring to in this book, anddescribing in some detail in Chapters 3–6, did indeed have their own planning,
trialling, data collection, analysis, and reporting phases. But, as suggested in
the previous section, these were only some of a range of continuing initiatives
designed to assist in the process of validating the IELTS test. The book will
therefore tend to refer to ‘the study of IELTS impact’, or ‘IELTS impact
study’ (with no ‘The’ and no capitalisation).
The test has been the subject of a number of studies to check its predictive
validity, usefulness (see Bachman and Palmer 1996) and positive impacts, and has
undergone several revisions since its debut in 1989 (IELTS itself being, of course,a development of the English Language Testing System or ELTS). The mostrecent modifications to the IELTS were in a major revision in 1995, which
implemented three areas of significant change suggested by routine research:• The three field-specific Reading and Writing modules were replaced with
one academic Reading module and one academic Writing module (seeCharge and Taylor 1997).
• General training Reading and Writing modules were brought into linewith the academic Reading and Writing modules in terms of time
allocation, length of written responses and the reporting of scores.
Since 1995 two revisions to the productive modules, Writing and Speaking,have been introduced. The new Speaking test was launched in July 2001,
involving a complete revision of both the format and assessment of the test. A
change to the assessment of the Writing test was introduced in January 2005.
Articles relating to both revisions can be found in Cambridge ESOL Research
Notes (www.CambridgeESOL.org/rs notes), for example, Shaw August 2002,
May 2004, Taylor February, July and November 2001.
So the routine monitoring and evaluation of the IELTS continues, and thevalidation systems are themselves monitored and modified, including themeasures introduced to gather data on test performance and candidatebackground so that issues of fairness relating to test use and users may be
more effectively monitored.
All IELTS research activities are co-ordinated within Cambridge ESOL’s
framework for research and validation, which is designed to cover three broad
areas of activity:• routine operational analyses for exam production, conduct,
marking/grading and post-exam evaluation• instrumental research involving small-scale projects to inform theoperational activities

24

IELTS impact research in context• research projects involving longer term assessment objectives relevant tobroader objectives and future developments.
The Research and Validation Group of Cambridge ESOL (established in1989) is responsible for the provision of quality assurance services for
Cambridge ESOL exams. This is achieved through work by specialised staff
on test validation matters involving, for example, statistical analyses ofcandidate performance, test task and item validation, Writing and Speaking
test corpus analysis. The group also advises on research design issues, andpresents and/or publishes information based on the work carried out. The
research and validation programme is designed to ensure that all ESOL
assessment products meet acceptable standards in relation to the four essential
VRIP qualities. The standard procedures established and implemented by theGroup are, it is claimed, themselves evaluated and refined in the light oftheoretical, technological and business developments. External stakeholders
are kept informed on issues of the quality and fairness of the examinations,and of the relevant research and validation work carried out by Cambridge
ESOL.
Routine and non-routine longer term research on IELTS is supplemented
by the two other partners in the IELTS test: IDP Education Australia:IELTS
Australia, and the British Council, who call for proposals and designate funds
for suitable research projects on IELTS. Of the 44 titles on the 1995–2001
IELTS funded research project list (see Research Notes 8, May 2002), six
contained the word ‘impact’, five the word ‘effect’, two ‘monitoring’ and one‘evaluation’. The labels are interesting given our discussions and definitions,but more important is the further clear evidence that impact-type research iscontinuous and focused rather than monolithic and one-off. The connection
between studies conducted under the IELTS funded-research programme andwashback or impact studies is clear – 15 are mainly concerned with the IELTS
skill modules (Reading, Listening, Writing, Speaking), 12 with IELTS
stakeholders (including candidates, examiners, receiving institutions), and 11
with IELTS preparation courses and candidates’ future language-related
needs. As the IELTS website (www.ielts.org) rightly claims, ‘such research
makes an important contribution to the monitoring and test development
process for IELTS; it also helps IELTS stakeholders (e.g. English language
professionals and teachers) to develop a greater understanding of the test’.
Further research under rounds 8–10 of the programme is discussed in Chapter
8, where we look at more recent IELTS impact research-related developments.
As we shall see in Chapter 6, responses to the research questions asked in the
study of IELTS impact provide data from learners, teachers and otherstakeholders on a wide range of impact and test validation areas. These include:• candidate profiles, strategies and test attitudes• perceptions of test motivation and stress25

1 Impact, washback, evaluation and related concepts• candidate and teacher perceptions of test relevance, difficulty andreliability• candidate and teacher views on test preparation course content,
approaches, materials and quality.
It is to be expected that such washback and impact data will be given
appropriate weight alongside the other continuously emerging information
processed through the research and validation systems already established forthe IELTS.

Progetto Lingue 2000 impact research in context
The PL2000 was a radical language education reform project of the Ministryof Education, then Ministero della Pubblica Istruzione (MPI, Ministry of
Public Education) now Ministero dell’Istruzione, dell’Università e della
Ricerca (MIUR, Ministry of Instruction, Universities and Research) in Italy.
The PL2000 aimed to provide foreign language education in state schools tomeet the communication and certification needs of students as defined by the
Common European Framework of Reference for Languages (2001). Courses
were to be delivered to homogeneous groups of learners through precisely
specified learning modules, using the most technologically informed taught
and self-access means. One of the Project’s key policies was to encourage theexternal certification of learners’ language proficiency through theexaminations of providers such as Cambridge ESOL. The PL2000 is describedin further detail in subsequent chapters.
During the 2001–2002 school year, with the encouragement of the
Education Ministry in Italy, Cambridge ESOL carried out a study of the
impact of the PL2000, collecting data from stakeholders including students,
teachers, parents, education managers and language testers. Given the fairly
broad range of stakeholders covered, the longitudinal, process-oriented natureof the study, and its constructively critical ethos, the study was called an
impact study rather than a washback study or an evaluation. The study is given
the capitalised format, ‘The PL2000 Impact Study’ as this was its title fromearly conceptualisation to completion and final report, and because it wasmore of a single, self-contained study than Phases 1, 2, 3 (and more) of thestudy of IELTS impact.
The PL2000 Impact Study, like the IELTS study, was, however, oneelement in an evaluation and validation system. It was a study carried out by
an external certifying agency, Cambridge ESOL, with the encouragement andapproval of the PL2000’s originator, the Italian Ministry of Education. The
willingness of the Ministry to permit a study by an external examination
provider, an officially appointed one (see below), was evidence of an interestin obtaining feedback on the Project from a variety of sources, including, of26

IELTS impact research in context
course, its own monitoring and research systems. These, according to the
Ministry’s PL2000 objectives and procedures statement included:… systematic monitoring of PL2000 at provincial, regional and national
levels to support the project and its evaluation, using quantitative andqualitative criteria such as: number of modules and languages taught;
number of students involved; time schedules (time organisation of themodules); certifed qualifications and qualification mode (internal/ external
to the school); content and organisation of modules; number ofcollaborative activities with foreign cultural agencies; learners’ success(PL2000 Impact Study Report 2003:11).

In addition, the PL2000 included a ‘national information module forprovincial referenti to permit regular comparison and verification of PL2000
trends within their districts and to encourage the sharing of organisational andmanagement strategies’ (PL2000 Impact Study Report 2003:10).
It was clear that the PL2000 Impact Study, relatively small-scale as it was(see Chapters 3 to 5 and 7 below), produced data and findings of use in areas
specified for the Ministry’s ‘systematic monitoring of PL2000’. These,comparing the contents of the PL2000 Impact Study Report itself (2003:11)
with the Ministry’s specified target monitoring areas included:• small-sample quantitative and qualitative impact information on thenumbers of students involved in the case study schools• analyses of teaching/learning module organisation, content,
methodologies, media and learners’ success.
These data could be added to other Ministry information on crucial focus areas
for the evaluation of the PL2000 and for the development of future initiatives
for the reform of school foreign language education in Italy. Chapter 7 will
describe, for example, revealing impact analyses of the learning/teaching
events in classrooms where courses under the PL2000 were taking place, andsummarise aspects of learner success as seen by the learners themselves, theirteachers, their school heads and their parents.
In addition to the information it provided for the Italian Ministry of
Education on the impacts of the PL2000, the study contributed data to
Cambridge ESOL test validation systems. One of the impact study’s statedaims was ‘to analyse the development of examinations for the certificationof language levels as appropriate to the needs of the growing range oflanguage learners in Italy’ (PL2000 Impact Study Report, 2003:13). As
Chapter 7 will show, the study provided data on test-validation matters suchas:
• student, teacher, school head and parent attitudes to Cambridge ESOL
exams including the Preliminary English Test (PET), the First Certificate
in English (FCE) and Certificate in Advanced English (CAE)27

1 Impact, washback, evaluation and related concepts• teaching/learning : test relationships, including student and teacher
perceptions and classroom observation data
• comparisons of PL2000 and non-PL2000 exam performances.
An interesting extra insight from the study concerned the effect of the PL2000
on the exam providers and their exams, an example of two-way washback, ofa programme on exams as well as of the exams on a programme.

Studies of IELTS and PL2000 impact
Although the two studies described in this book differ significantly in scale,status, focus and context, both are labelled impact studies. Both seek
information on the effects respectively of an education reform programme anda high-stakes test on stakeholders beyond, though including, theteaching/learning territory of washback. The PL2000 study hears the views
and attitudes of school heads, project officials and learner parents; the IELTS
study contacts receiving institution administrators for their reactions to thetest, as well as the candidate/student and teacher stakeholder constituencies.
Both studies examine outcomes as well as processes, and are seen as but one
element in a continuous and iterative test validation process.
As we look into both studies in subsequent chapters, the labels andsignificance of constructs such as impact, washback, evaluation and related
terms will be revisited. This chapter has attempted to define main constructs
and terms and to put the study of impact into perspective and context. Neither
the study of IELTS nor of PL2000 impact is seen as time-bound or selfsufficient. In fact (see also Chapter 8) both studies are already leading tofurther, related washback and impact research.

28

2

Impact study approaches

Impact, washback and related concepts are defined and exemplified, and theCambridge ESOL context for impact studies as part of the test validation
process described in Chapter 1. Chapter 2 will survey possible methodologies
for the collection and analysis of impact data. It will cover research
approaches, sampling, the pre- and post-validation of instrumentation, anddata strategies. Chapters 3, 4, and 5 will then examine, respectively: impact
study objectives and research questions; instrumentation; and the
management and analysis of impact data.

Qualitative and quantitative research approaches
Choices of research approach
The choice of type of research methodology normally involves a considerationof the advantages and feasibility of quantitative and qualitative researchmethods. Weiss usefully defines the two general approaches to research, as
follows:
Quantitative Research: research that examines phenomena that can be
expressed numerically and analysed statistically
Qualitative Research: research that examines phenomena primarily
through words, and tends to focus on dynamics, meaning and context.
Qualitative research usually uses observation, interviewing and document
reviews to collect data (1998:335).

Some of the discussion in Chapter 1 of types of evaluation reflects thedistinctions Weiss makes.
Lazaraton cites in Figure 2.1 (page 30) the generalised research typecontinua of Larsen-Freeman and Long (1991) to summarise aspects ofquantitative and qualitative research methodologies, including data collectionmeans, type and focus of data collected and findings derived.
Weiss distinguishes a quantitative and a qualitative research focus thus:
Quantitative evaluators tend to focus on whether and to what extent
change in x causes change in y. Qualitative evaluators tend to beconcerned with the process that connects x and y (1998:284).

29

2 Impact study approaches
Figure 2.1: Characteristics of quantitative and qualitative research
Quantitative Researchcontrolled
experimental

Qualitative Research
naturalistic
observational

objective

subjective

inferential

descriptive

outcome-oriented
reliableparticularistic‘hard’, ‘replicable’ dataprocess-oriented
valid
holistic‘real’, ‘rich’, ‘deep’ data

Source: adapted from Charles S Reichardt and Thomas D Cook (eds): Qualitative and Quantitative Methods in
Evaluation Research, page 10, copyright 1979 by Sage Publications, Inc (in co-operation with the Evaluation
Research Society). Reprinted by permission of Sage Publications, Inc.

The characteristics in Figure 2.1 are, of course, by no means mutually
exclusive. They may be combined within the same study; they may be present
in different strengths along their continua. Cronbach (1982:18–19) talks of a‘reconciliation’ between what he terms ‘scientistic’ and ‘humanistic’ research
and welcomes ‘multiple-method approaches’ (Saxe and Fine 1979:64)
combining qualitative and quantitative elements. In fact, it is nowadays quiterare to encounter evaluation or impact studies that do not, to a greater or lesser
extent, claim to combine qualitative and quantitative research approaches.
We have noted in Chapter 1 the complex nature of washback, and the nonmonolithic and iterative nature of impact studies. With such inherent complexity
and continuity, it is understandable that the investigation of washback and impact
will require a range of approaches. In a recent summary of the methodologies
employed in washback research (including the study of the impact of the IELTS
test which is described in this book) Watanabe suggests that since it is‘conceptualised on several dimensions’, ‘the methodology that attempts todisentangle the complexity has inevitably to be multifarious’ (2004:20).
The characteristics of impact research thus seem likely to show influencesfrom both ends of Lazaraton’s continua, but perhaps with an inclination towards
the qualitative end. Watanabe notes that ‘qualitative or ethnographic researchhas been increasingly widely used among researchers in the fieldof language teaching and learning’, such research, he considers, being
characterised by strategies ‘which are relevant to the research into washback’(2004:20). Watanabe’s apparent equation of qualitative and ethnographic
research might be disputed; classroom observation and case studies, forexample, though regularly used in applied linguistics studies, do not
usually constitute fully fledged ethnographic research (Heath 1982:36).
‘Ethnography’ confirms Lazaraton ‘requires a deeper and broader philosophical30

Qualitative and quantitative research approaches
and methodological commitment than does simple participant observation’(1995:460).
Duff (2002:1–10), also denies a polarity between quantitative andqualitative research. She notes important ‘research practices that can enhancequalitative applied linguistics research’ (2002:4), then suggests that suchresearch practices include:• conducting longitudinal research, when possible• eliciting participants’ perspectives on their own behaviour• using participants who know each other and have some familiarity withthe researcher• looking for recurrent patterns across data sets
• providing whenever possible methodological, analytical andperspective/epistemological triangulation (‘using several methods toreveal multiple aspects of a single empirical reality’ Denzin, 1978).
The research practices exemplified here by Duff and worth pursuing in thenext few sections with relation to our two studies, clearly derive from thenature of impact research rather than from an automatic methodologicalchoice. It is because of the complexity of the inter-relationships between
variables such as tests and test-related teaching or lesson plans and lesson
implementation (both key areas of attention in the studies of IELTS and
PL2000 washback) that complex and often qualitative research practices areentailed. It is also clear that Duff’s research practices such as ‘eliciting
participants’ perspectives on their own behaviour’ or ‘using participants who
know each other’, while likely to be enriched and enhanced by the collectionof open-ended, qualitative conversational data, will also benefit from morequantitative analyses, for Duff’s purposes of ‘triangulation wherever
possible’.

IELTS and PL2000 Impact Study approaches
It is appropriate here, using insights from both IELTS and PL2000 impact
studies, to examine further the match between impact research objectives,
methods and contexts.
The IELTS research (see Chapter 3 for a full specification of its aims,
objectives and research questions) seeks answers to questions on thewashback and impact of the test on:••••

the IELTS test taking population
classroom activity in IELTS-related classes
IELTS teaching materials, including textbooks
other users of the test.

31

2 Impact study approaches
The PL2000 Impact Study, the aims, objectives and research questions ofwhich are also defined in detail in Chapter 3, sought to:• ascertain the effects of the PL2000 foreign language reform measures on
English language learner performance in state schools in Italy• explore the significance of PL2000 for some main stakeholders• analyse the effects of the use of external examinations for the certificationof PL2000 learner language levels• identify areas of potential support for teachers teaching classes under the
PL2000.
The pursuit of answers in such question areas clearly required‘phenomenological data that represent the world view of the participants
being investigated’ (Watanabe, ibid.:22), and ‘eliciting participants’perspectives on their own behaviour’ (Duff above).’ It is also commonly
noted (e.g. Hawkey forthcoming, Hopkins 1985) that participant perceptions
may not always reflect what is actually happening. Thus, the studies of IELTS
and of PL2000 impact elicit perspectives on participants’ own and others’
behaviour both through more quantitative data, for example, from theadministration of the objective-item Language Learning Questionnaire(Purpura 1999), and more qualitatively through interviews, focus group
discussions and open-ended questionnaire items. The two studies also attempt
to frame their smaller scale data with broad quantitative comparative data.
IELTS band scores among the impact study candidate population, forexample, are related to global IELTS candidate results (see Chapter 6).
PL2000 course student results on the same exams are compared with those forcandidates from the rest of Italy and for non-Italy candidates (see Chapter 7).
The phases of IELTS impact study described in this book, seeking fairly
broad worldwide coverage, were conducted mainly at a distance, though withvisits to some participating centres for the collection of face-to-face data. The
distance data collection was intended to minimise disruption at the target
institutions, given the priorities and pressures affecting those involved with animminent or recent high-stakes English language assessment such as the
IELTS. The PL2000 Impact Study was undertaken with the permission andencouragement of the Ministry of Education in Italy. It was implemented at aselection of typical schools, again with the promise of minimum disruption todaily educational and administrative life. The institutional conditions, as wellas the research focus for both studies, tended to favour the collection of data,in the terms of Figure 2.1, through more ‘naturalistic’, ‘observational’,‘descriptive’ rather than controlled experimental means.
The PL2000 Impact Study was longitudinal in the sense that the main data
were collected from the selected schools on two visits, in October 2001 andApril 2002, that is at the beginning and towards the end of the school year.
Differences were then analysed using quantitative data from matched student

32

Qualitative and quantitative research approaches
and teacher background and attitude questionnaires, and English language
proficiency exam scores, along with more qualitative interview, focus groupand classroom observation data from the two school visits.
The PL2000 study in particular benefited from information supplied by‘participants who know each other and have some familiarity with theresearcher’ (Duff 2002:1–10). This was exemplified in teacher interviews andfocus groups, and in correspondence with some participants between study
visits to the schools. It is difficult to conceive of an impact study that does notlook for Duff’s’ ‘recurrent patterns across data sets’, but both the Cambridge
ESOL studies described here certainly did so, often when seeking
triangulation (Weiss’s ‘cross-check through different modes of inquiry’,
1998:263). Examples of triangulation in the two studies include comparisons
between:• closed and open-ended student questionnaire responses• teacher questionnaire responses and teacher interviews• student and teacher questionnaire responses and classroom observation
analyses.
The collection and analyses of these triangulated data are examined in somedetail in Chapters 5–7.

Experimental research
It is because of the complexity of washback and impact (see Chapter 1),
because of the consequent need to probe webs of inter-relationships between
variables and processes, and to seek insights from a range of stakeholders, thatthe experimental research paradigm is rarely evident in impact research. It issignificant that in Washback in Language Testing: Research Contexts andMethods (Cheng and Watanabe 2004) the sections on impact and washback
methodology focus on qualitative approaches to the study of washback andimpact, although with quantitative data analytical methods often used. The
eight ‘washback studies carried out in many different parts of the world’(2004:xiv) then described in the Cheng and Watanabe collection all use
survey, interview and observation techniques rather than experimental
research designs.
The experimental research approach is applicable where the researcher isin a position to control both independent variables being studied (for example,
the teacher and the classroom materials) and the conditions under whicha study takes place (for example, the class size and the target language
proficiency level of the learners). Then, to satisfy the conditions of a truly
experimental design, participants are assigned randomly to the experimentalgroup (all of whom receive the programme or test under investigation) andthe control group (selected randomly from the same population as the33

2 Impact study approaches
experimental group, but not receiving the programme or test concerned). The
differences in performance between the experimental and control groups canthen be measured and inferences drawn about the effects of the programme ortest concerned. Experimental studies with such parameters are relatively rarein educational research, especially where larger numbers of learners are
involved, and data are collected from numerous and distant centres.
Watanabe indicates a further reason for the rarity of controlled experiments
in impact studies, namely the inevitability of a role for the contexts in whicha test is administered:
Qualitative research also stresses gathering data in ‘real’, that is, nonexperimental settings. The test always plays a certain role in a specificcontext, so even if it were found that a test has some impact on teaching
and learning under controlled settings, it is likely that the result would
not apply to situations where the teaching is actually being done for test
preparation (2004:23).

Watanabe uses the term ‘selection’ rather than ‘sampling’ when discussing
participants invited to take part in an impact study’s quest ‘to examine thevalidity of predictions’ (2004:29). ‘The selection’, Watanabe continues ‘is not
to be made at random, but purposefully’, often the case, it would seem, when‘selecting cases for study in depth’ (Patton 1987:52). Watanabe also suggests,
basing his judgement on the eight washback or impact studies reported in
Washback in Language Testing (2004), that ‘it is normal to select various
groups of participants rather than one single population’. This is one way offacilitating the triangulation of data, of course, and the possible re-focusing ofsome research questions as new, potentially significant variables emerge.
In the case of the studies into IELTS impact, as we shall see in greater
detail in Chapter 4, centres were selected for the invitation to participate indata provision according to the results of a pre-survey of more than 300
institutions and according to the nationality proportions in the overall IELTS
candidate population. For the PL2000 study, participating schools wereselected (see also Chapter 4) according to strata such as school type(elementary, middle and high), geographical location (northern, central andsouthern Italy) and external exam certification (Cambridge ESOL exams atfour levels). These school selections were confirmed or modified throughcontacts made during pre-study visits. On the key related issue ofgeneralisation from qualitative data, Duff (2002) reminds us that qualitative
researchers should acknowledge any limitations in their participant groups
and the context-bound nature of their findings. Attempts are made in Chapters
4, 6, 7 and 8 to make such acknowledgements with regard to both studies.

34

Qualitative and quantitative research approaches

Local contexts
Miller suggests that more ethnomethodological studies ‘focus on the socialand political contexts within which members use available interpretative
methods to construct social realities’ (in Silverman 2001:29). This appears toaccord with Duff’s further proposal for qualitative studies. They should, shesuggests, seek to fulfil as far as possible their potential ‘to yield interestingunderstanding of local contexts and to examine the interplay of variables andissues to a greater extent than quantitative research typically sets out to do; thelatter, conversely, attempts to control as many extraneous variables as
possible’ (2002:6).
Weiss emphasises similar advantages in qualitative research approacheswith reference to the concept of the case study, ‘a research strategy thatinvestigates a phenomenon in its natural setting using multiple sources ofevidence’ (1998:328). In the study of a high-stakes test such as IELTS, thereceiving institutional context was important on key issues such as the IELTS
band score cut-off points accepted by particular departments (see Chapter 6).
There are findings, too, from the PL2000 Impact Study (see Chapter 7) which
relate interestingly to their particular contexts yet have potential
generalisability, for example the influence of parental attitudes to foreign
languages on student presence and progress on courses run under the PL2000
initiative.

Survey research
Both our example impact studies employ what are usually termed survey
research approaches especially, perhaps, in their use of questionnaires,interviews, and observation. Survey research methods are not confined tolarge sample studies and simple statistical analyses. They appear to suit
impact studies, which tend (see Baker 1997:35) to be seeking probabilistic
and interactive, rather than deterministic, relationships between individualand group characteristics such as: language background, attitudes, motivation;
language learning approaches, strategies; styles of language teaching, andtarget language performance. Baker also, however, reminds us of problems
sometimes associated with survey research. These can include: a lack of clear
aims; implicit rather than explicit theoretical input; un-established causal
relationships; inadequate sampling; instruments containing invalid items; lackof triangulation through other data collection methods; and interviewer or
researcher effects. It is clear that such validity problems are not confined tosurvey research but apply to other methods of enquiry, including experimental
research. Baker’s warnings, however, are salutary.
Both the studies described in this book attempt to minimise survey-related
problems. Both define and adhere to their objectives and research questions35

2 Impact study approaches(see Chapter 3). Both attempt to validate their data collection instruments (seethis chapter and Chapter 4). Both, as already indicated, triangulate data andacknowledge the limitations of their samples.

Pre- and post-validation of instrumentationand data
At the time when Cambridge ESOL (UCLES EFL as it was then) was indiscussion with Charles Alderson (consultant to the IELTS impact study
project from 1995–1996), it was agreed that one of the starting points for sucha project was an in-depth review of key constructs. The result was a paper
entitled Ideas for Research into Impact, Washback and IELTS, submitted
to UCLES by Alderson in June 1995. The paper (already cited in Chapter 1
and see also Chapter 4) reviews washback/impact definitions, reminds usof the complexity of relationships between both and any test, suggests thatthe primary goal of any washback/impact study is ‘to describe’, and stresses
the importance in the conduct of such studies of prediction, baselines andcomparisons, the collection of data on IELTS-related classroom activity
and teaching/learning materials, and on stakeholder attitudes. A further
example of the focus, throughout the study of IELTS impact, on the validityof data collected was the commissioned paper How might impact study
instruments be validated? (Alderson and Banerjee, 1996).
Chapter 4 describes the development, applications and analyses of impact
study instrumentation in detail, but it is relevant in this chapter to inferprinciples and practices which inform our discussion of impact study
approaches. It is certainly clear that, whether a study is to the left or the righton the Lazaraton quantitative … qualitative research continua, the means ofcollecting data should be as valid and reliable as possible.
Key recommended actions and approaches emerging from the 1995
UCLES brief and the resultant papers were:• a careful exploration of constructs and advance hypotheses by ‘insiders’and ‘outsiders’ before the drafting of instruments• checks that every item contributes to the corroboration or refutation of
washback and impact hypotheses• the prediction of participant responses to compare with actual responses
at piloting and trialling stages, with adjustments to instruments madeaccordingly• the use of expert and stakeholder judgements on draft instruments• the comparison of draft instruments with other instruments• the use of a combination of methods of data collection, approachingvalidation from a number of angles with data triangulation used as acheck.
36

Strategies for data analysis
The studies of IELTS and PL2000 impact, as will be seen in Chapters 3–6
below, included comprehensive instrument validation measures.

Strategies for data analysis
Validity is also, of course, an issue when it comes to data analysis, althoughsound work at the pre-validation stage should help reduce some of the
problems that can be associated with the production of valid data.
Weiss points to differences in the analysis strategies applied to qualitativeand quantitative data because of the ‘nature of the data and the analytic intent’(1998:283). Quantitative research, seeking a degree of generalisability
through larger sample size, will produce measures yielding numerical values
or narrative data coded into numerical values, analysed and post-validated
using statistical methods. As Weiss also notes, much of the quantitative
researchers’ data analysis work ‘they have already done through thedevelopment and/or selection of measures and data collection’. Once the data
are in, quantitative research will:… focus on locating significant relationships among them. Throughstatistical techniques, they identify associations among variables and thelikelihood that associations are real and not mere chance fluctuations.
Their aim is to model the system of cause and effect (1998:284).

Qualitative studies, on the other hand, seeking more holistic and dynamic
information, may well produce more, and more varied, data, and from smaller
numbers of participants. The data may be less objectively chronicled, as in thecase of field notes, narrative accounts of conversations, recordings ofinterviews or group discussions, school documents, video-recorded classroom
lessons and so on. Such data may be less susceptible to statistical analysis andthus require more qualitative interpretation and validation, often, however,through triangulation with more quantitative data on the same subject. Afterall, as we have suggested already, much social scientific research collects
and analyses both qualitative and quantitative data. Both of the studies
exemplified in this book certainly do so.
Davis suggests the following pattern of data analysis in applied linguistic
research: a search for patterns of generalisation across multiple sources of
data; and the establishment of credibility through systematic evidence fromthick description (1995:446). Watanabe, drawing on experience specific to thetest washback field, sees the researcher with an array of data to be analysed,
including classroom materials, audio and/or video recordings, field notes,
interview data, memos, e-mails and computer files. Throughout the data
analysis, Watanabe reminds us, research verification must also be taken ‘toestablish reliability (or the consistency of data analysis) and validity (therelevance of the data) in the quantitative research tradition, and credibility,37

2 Impact study approaches
transferability, dependability, and confirmability in the qualitative researchtradition’ (2004:34). His analysis steps include: reviewing the field notes;
completing observation analyses; qualitative refinement of relevant
categories; quantitative analyses of the extent of their relevance; interpretation
through the interplay of data and theory; checking rival interpretations;drawing final inferences (2004:34–35). These data analysis and interpretation
steps seem close to the ‘basic analytic strategies’ suggested by Weiss(1998:285) as used in both quantitative and qualitative data analysis, namely:
•
•
•
•
•
•
•
•
•
•

descriptions
counting
factoringclustering
comparingco-variation
finding commonalities
ruling out rival explanations
modelling, andtelling a story.

Descriptions from data analysed are used to describe a test such as the IELTS
or a foreign language learning reform project such as the PL2000. The
descriptions may be derived from quantitative analyses such as a bar chart ofpost-IELTS candidate closed-item responses to a questionnaire on factors
most affecting their performance in the test in Chapter 6, or qualitative
analyses such as those of teacher focus groups discussing the advantages anddisadvantages of the PL2000 in Chapter 7. In both cases, the presentation andinterpretation of impact study findings require description.
Weiss’s analytical category of counting is a part of the strategy of describing,
explicitly or implicitly, most other data analysis operations. The counting
involved in the establishment of population samples, and in response selection
frequencies on key issues, informs most parts of the data analyses of both
IELTS and PL2000 studies. Wherever it appears necessary, actual counts ofparticipants or responses are included to counteract tendencies to refer to‘more’, ‘a few’ and so on when the scale and proportions of such references are
not clear. Where participant opinions are cited in the PL2000 study, forexample, the numbers or prevalence of such opinions are also given.
Factoring in the sense of dividing aggregates into constituent parts,
applies, as used in both our example studies, to the coding of narrative
information into key words or categories. This technique is employed in theclassroom observation and interview analyses to facilitate inferences about
the content, frequency and co-variation of categories. Factor analysis is usedin the confirmation of constructs. The language learning and test-taking

38

Strategies for data analysis
strategies in the Language Learning Questionnaire (see Chapters 1, 4 andelsewhere), for example, are categories confirmed through such clustering
techniques. Qualitative data analysis also involves clustering, as exemplified,perhaps, in the close analysis of subjective data on learner motivation inferredfrom the PL2000 impact study teacher, school head and parent interviews(Chapter 7).
Comparison represents a central operation in impact studies, which are bydefinition concerned with how things are, with and without, before, during
and after, a programme or test. The PL2000 Impact Study compares student
baseline and post-Progetto course data. It also compares the performance ofstudents on these courses with the performance of those on courses not run
according to the principles of the Project. Analysis of variance (comparing
independent and dependent variables across several groups to see if there issignificant difference) and regression analysis (to explain and predict
relationships between independent and dependent variables) are statisticaltechniques often used to investigate such differences (see references in the
IELTS study in Chapter 4). Covariation is another key concept in impact
studies as it refers to the relating of changes in one phenomenon to those inanother. An example of this in both IELTS and the PL2000 studies is theprediction that the test and the new curricula respectively would lead todifferent lesson content and teaching approaches (see Chapters 6 and 7).
Covariance analysis, an extension of the analysis of variance, may be used tomeasure such phenomena quantitatively.
Davis’s search for patterns of generalisation (1995) and Weiss’s finding
commonalities, are basic strategies of data analysis, entailing theidentification of trends and significant effects. What are the most common
preparation course experiences of IELTS candidates? What are the main
impacts on head teachers of the PL2000? What are the common elements inlearner and teacher reactions to both? The examination of ‘deviant’ cases isthe corollary to finding commonalities, suggesting that it may not always beappropriate to drop such cases from one’s data in favour of concentrating on
the central tendency, since important impact information may ‘lurk at theextreme ends of a distribution’ (Weiss 1998:287). Analytic induction entails
a focus on any evidence that challenges hypothesised constructs, andencourages researchers to keep modifying these constructs until they cover all
the data; ‘outlier’ cases may indicate unexpected but nevertheless significant
impacts. Interesting, possibly contradictory, responses on IELTS stress andmotivation impacts from the teacher questionnaire used in this study were
investigated (see Chapter 6). Differences in student and teacher perceptions ofactivities in the same lessons were found to be revealing in the PL2000 study(see Chapter 7).
Watanabe’s ‘checking rival interpretations’ is clearly a crucial process inall kinds of research. In real-time educational research in particular, every

39

2 Impact study approaches
effort should be made to distinguish changes brought about by the programmeor test concerned from changes caused by extraneous factors. But with thecomplexity of factors involved in learning, teaching and testing, it is not at alleasy to rule out the possibility that factors or events other than the programmeor test are responsible for observed washback or impacts. The data analyst
must seek possible alternative explanations. To what extent is tension caused
in candidates for the IELTS test to do with the test itself, to what extent by
their dislike of any test? (See Chapter 6.) To what extent was the claimed
target language improvement of students under the PL2000 the result of theteaching approaches of the Project itself, and/or to what extent was the
improvement motivated by the introduction of higher stakes external exams at
the end of the course? (See Chapter 7.) The impact researcher can attempt toresolve such questions through closer examination of existing data, bycollecting new data on the issue, or, at least, by acknowledging the query andanalysing the possibilities.
Modelling and telling the story are the final data analysis strategies listed
by Weiss, and would, of course, include Watanabe’s drawing final inferences,a balanced final evaluation of which research hypotheses are supported, andwhich are not, to what extent, and why. Modelling, in this sense, means
pulling together the main threads of the data to support descriptions andexplanations of key processes and outcomes. The ‘telling of the story’ is thecommunication to readers of the findings of an impact study, clarifying results
and inferences, re-emphasising contextual and generalisability factors andlinking recommendations with future developments. A good story is most
likely to be told when the researcher fully understands the purpose, processes
and outcome of the study, and can present its conclusions with confidence,
openness and well-supported argument. The ‘story to be told’ also needs tosuit the book (or institutional context) in which it appears, to be accessible andto engage the readers (or stakeholders).
This chapter has attempted to summarise qualitative and quantitative datacollection and analysis approaches in general, illustrating them withreferences to studies of IELTS and PL2000 impact. We have examined, so far
mainly at the level of principle rather than practice, key research issues suchas the selection of participants, survey methods, pre- and post-data collection,
the validation of instruments, generalising, data analytic patterns andstrategies. In more human terms, these phases in the research process areintended to produce research that, as Weir suggests, should be believable,
logical, feasible, have value and interest. Even better, Weir adds, if theresearch is ‘important to the person doing it’ (2004:221–222).
It would seem appropriate now to consider the specification of aims andresearch questions, and the development of research designs, in pursuit ofanswers. These are the themes of Chapter 3.

40

3Impact study objectives,
designs and researchquestions

Chapter 2 illustrated certain qualitative and quantitative data collection andanalysis approaches with reference to the studies of IELTS and PL2000
impacts. It is logical now to progress to the first implementational step in suchstudies, that is the specification of aims, conventionally translated into
research questions and action plans. These are the themes of Chapter 3.
There would appear to be a hierarchy of preparation, planning and research
design procedures for impact studies (and other kinds of project research)which it might be helpful to discuss here, before moving to the instrumentationand implementation of such studies in Chapters 4–7. A reasonable context
from which to derive conceptual definitions and distinctions might beacademic research, for which the conventional hierarchy of preimplementational categories may be summarised as in Figure 3.1.
Figure 3.1 Hierarchy of research pre-implementation
Context and purpose
Aim
Objectives
Research Questions
Research Design/Action Plan
Hypotheses

The labels here are often flexibly defined and inconsistently used, as might be
expected, but there is a conceptual pattern to the pre-implementation planningand development process of research projects (Bunton 1998). As is the casethroughout this book, the relevant processes will be discussed here withreference to our two case studies.
41

3 Impact study objectives, designs and research questions

Context and purpose
In the case of practical research such as an impact study, ‘context’ will have adefinition close to the conventional academic research term ‘scope’, which isnormally used to define the broad subject area of a programme of research andits relationship with other work in the field. In a sense, we have alreadyestablished the context and purpose for our two studies. Their context is thevalidation (see Chapter 1) of high-stakes language tests through checks on theirvalidity, reliability, impact and practicality (Cambridge ESOL’s VRIP), in which
the investigation of the effects of such tests on various stakeholders has the
purpose of providing data and recommendations on test washback and impact.
Also to be included under the construct of context might be what Weisscalls ‘evaluability’ (1998:95), which entails making sure that a project,programme or test is amenable to washback and impact study. Both the IELTS
test and the Progetto Lingue were agreed to be appropriate for impact research.
The IELTS test, with its very high-stakes role and growing candidatureworldwide, was already being evaluated and monitored through routine andproject-based studies (see Chapter 1 above). What is more, a study into thetest’s washback and impact had been foreseen since the mid-1990s when ‘it
was agreed that procedures would be developed to monitor the impact of thetest and to contribute to the next revision cycle’ (Saville 2001:5).
As discussed in Chapter 1, the Italian Ministry of Education had included
in its plan for the PL2000 systematic monitoring at provincial, regional andnational levels. Cambridge ESOL saw the Progetto as appropriate for impact
study in a validation context because its own language tests had been approved
for the external certification of the foreign language proficiency and progressof students on PL2000 courses. Cambridge ESOL exams were selected for thispurpose along with the tests of other international language test providers suchas the Trinity International Examinations Board (also for students of English
language), the Alliance Française (for French), the Goethe Institut (forGerman) and the Instituto Cervantes (for Spanish). A study of the processes
and products of English language teaching, learning and assessment on
PL2000 English language courses would, it was felt, produce revealing dataon Cambridge ESOL Main Suite exams, especially the Key English Test
(KET), the Preliminary English Test (PET), the First Certificate in English(FCE) and the Certificate in Advanced English (CAE), which were being used
for the external certification, respectively, of PL2000 students at CEF levels
A2, B1, B2 and C1.

Aims and objectives
Aims and objectives are often seen as representing different levels in the
research project hierarchy, the aim representing the broad purpose of a projector study, the objectives defined as the desired or intended outcomes of the42

Aims and objectives
project. In some project planning, however, aims and objectives are nothierarchically distinct. Characteristic, though flexible, patterns for the preimplementation phase of academic research include the following, each witha differently named but similar initial step:• aim > purposes > research objectives > research questions (for eachobjective)• purpose/rationale/significance > research questions > hypotheses• general research question > related research questions > hypotheses.
In a less academic context but nevertheless referring to educational research
initiatives such as impact studies, Weiss specifies setting the scene, purposes,
understanding the program and specifying the key questions as the four initialsteps in a research project (1998:72).
The stated aim of the phases of IELTS impact study described in this book,in the context of Cambridge ESOL’s continuous and iterative IELTS
validation programme, was summarised by Saville as follows:
In order to understand the test impact better and to conduct effective
surveys to monitor it, it was decided that a range of standardised
instruments and procedures should be developed to focus on the following
aspects of the test:• the content and nature of classroom activity in IELTS-related classes• the content and nature of IELTS teaching materials, including
textbooks• the views and attitudes of user groups towards IELTS• the IELTS test taking population and the use of results (2001:5).

This statement of aim is broad because it includes reference to the context andpurpose of IELTS impact research likely to embrace more than one study.
As we shall see in Chapters 4 and 5, the statement is in a sense a meta-aim, of
which the phases described in this book are a rationalised part. The researchquestions derived from this broad IELTS impact study aim were, as we notein the next section of this chapter, framed to assist the achievement ofobjectives in the four research areas described by Saville above.
The PL2000 itself was originally presented by the Italian Ministry of
Education in terms of one aim and two objectives. Its aim was:To set up a continuous course of studies from elementary school to thesecond year of secondary school for the teaching/learning of foreign
languages with a view to the acquisition of a pragmatic-communicativecompetence in accordance with the directions contained in the Common
European Framework of Reference of the Council of Europe. By the end of
the programme, pupils should have learned at least two foreign languages
regardless of the type of school attended (Ministry of Education, Italy,
1999, translated).
43

3 Impact study objectives, designs and research questions
The Progetto’s stated objectives were:• to develop communicative competence in reading, written and oralinteraction and production appropriate to the learners’ age, level,domains and contexts of specific language use in the various schooltypes• to allow for the design of courses for specific purposes, including
literary and professional language, as well as the development of partial
competence in a single language skill (Ministry of Education, Italy,
1999, translated).

The language teaching and learning background to the PL2000 is describedvividly by Duguid in Anatomy of a Context: English Language Teaching in
Italy (2001). She sees as a factor motivating the Project the poor performancein English of Italian students in a context where the language was becoming
increasingly necessary to help Italians take advantage of study and work
opportunities in the European Union and global market. Before the PL2000,
foreign language teaching was dominated by grammar-translation, teachercentred approaches. These were often in the hands of untrained teachers sincequalifications were not a requirement until 1999. There was little or no formal
inspection of teachers, who tended to write their own syllabuses and do theirown language testing, often using discrete-item writing formats, and long oral
tests. Given all this, and the lack of co-ordination between primary andsecondary schools, students entered university with little ability tocommunicate in English.
The PL2000 was thus conceived as a project to develop standards forlanguage teaching in schools linked to the Common European Framework(CEF), subject to certification by selected external exams and providing
improved professional development support for teachers.
The PL2000 was first proposed as an appropriate focus for an impact studyin August 2000 by Dr Peter Hargreaves, then Chief Executive Officer ofCambridge ESOL. In March 2001, following discussions involving the Italian
Ministry of Education, senior Cambridge ESOL staff members and an
external consultant, the following impact study aims were presented for a
Cambridge ESOL impact study of the PL2000:• to ascertain the effects of PL2000 on English language performancethrough pedagogy, textbooks and other media, evaluation and assessment,
and resource centres• to explore the significance of PL2000 to main stakeholders, including
learners, teachers, teacher trainers, parents, evaluators, and educational
managers• to analyse the development of examinations for the certification of
language levels as appropriate to the needs of the growing range oflanguage learners in Italy

44

Research questions• to identify further areas of potential support for teacher in-service andtest-awareness programmes, resource centre development, and textbook
design• to provide a focus for the presentation, to other EU countries, of the
Italian model for state-sector adoption of the Common European
Framework of Reference for language learning.
It might be argued that these aims could have been specified in terms of one
aim, that is to ascertain ‘the effects of PL2000 on English language
performance through pedagogy, textbooks and other media, evaluation andassessment, resource centres’. This aim could then have been specified interms of four objectives. Meeting these objectives would entail seeking dataon the effects of the Project on:••••

a range of stakeholders in the language classroom and beyond
their relationships with external language exams
project support facilities, materials and training
other potential foreign language programme reformers.

The PL2000 Impact Study is in a way a two-layered study, investigating thewashback and impact of the PL2000, and, within these, the impacts andwashback of Cambridge ESOL tests used in the PL2000 for the certificationof English language learners’ performance and progress.
There are, it would appear, a variety of ways of specifying impact study
aims and objectives.

Research questions
Research questions appear consistently in the various research planning
outlines mentioned so far. This is hardly surprising given the element ofenquiry that is common to all forms of research. That research questions
should be dictated by the context, purpose, aims and objectives of the study isalso clear, which locates them after these levels in the sequence of interdependent research planning steps. However, experienced researchers warn ofthe dangers of asking too many questions, producing too many data to analyse,
framing questions which stray from the research aims or objectives. Light,
Singer and Willett advise that: ‘Well-crafted questions guide the systematicplanning of research. Formulating your questions precisely enables you todesign a study with a good chance of answering the questions’ (1990:13). Notehere the implied chronological precedence of research questions over research
design.
Weiss (1998:95) also emphasises the need for the researcher to frame theright research questions, that is questions that relate essentially to research
focus on programme or process, outcomes, and the links between them.
Decisions on research questions must also take account of practical matters45

3 Impact study objectives, designs and research questions
such as: access to participants and data; the interests of stakeholders; time,
personnel and financial resources, including available research expertise and,in the case of data to be collected in working institutions, the need to minimise
disruption to programmes and participants (see, especially, Chapter 5).
Ten research questions were framed to mediate the achievement of theobjectives of Phase 3 of the study of IELTS impact in the four research areas
cited above. The questions are presented here according to participant or
activity focus, namely:• IELTS pre- and post-test candidates, preparation courses, test impacts on
candidates• IELTS preparation course teachers, test impacts on these teachers• IELTS preparation materials and test impacts on these materials• IELTS preparation lessons• IELTS impacts on receiving institution administrators.
The research questions are sequenced in the order in which they are raised in
the five data collection instruments and corresponding face-to-face enquiry
activities to be described in Chapter 5.
Focus on IELTS candidates:• Research Question 1: What are the profiles of the candidates taking the
IELTS test?
• Research Question 2: What is the washback of the IELTS test on courses
preparing candidates to take it?
• Research Question 3: What are the profiles of the participants who havealready taken the IELTS test?
• Research Question 4: What is the impact of IELTS on the participants
who have taken the test?
Focus on IELTS preparation course teachers:• Research Question 5: What are the profiles of the teachers preparing
candidates to take the IELTS test?
• Research Question 6: What is the washback of the IELTS test on theteachers preparing candidates to take the test?
Focus on IELTS-related course materials:• Research Question 7: Which textbooks and other materials are used on
IELTS preparation courses?
• Research Question 8: What is the washback of IELTS on these
preparation course materials?

46

Research questions
Focus on IELTS preparation lesson observation:• Research Question 9: What do IELTS-preparation lesson observations andanalyses indicate about the washback of the IELTS test on the lessons?
Focus on receiving institutions:• Research Question 10: What is the impact of the IELTS test on receiving
institution administrators?
The PL2000 Impact Study sought to identify, describe and explain thewashback and impact of an important and radical educational reform
programme, the PL2000, on a range of participants. Given the aims of thestudy as defined above, the main research questions which the study set out toanswer were, according to the PL2000 Impact Study Main Report, thefollowing:• What washback is the PL2000 having on the pedagogy, materials, andmedia for language teaching and learning?
• What washback are changes in language teaching and learningpedagogy, materials, and media having on the performance andattitudes of the students and the teachers?
• What washback is the PL2000 having on language evaluation andassessment?
• What impacts is the PL2000 having on educational managers,
including Heads of Schools?
• What impacts is the PL2000 having on support for teacher in-service
and test-awareness programmes, resource centre development, andtextbook design? (Hawkey, 2003:17)

Apart from being couched in question form, the research areas indicated by the
PL2000 study aims are rationalised in these questions into more discrete andconsecutive elements. The rationalised questions imply that some of theanswers might constitute a potential response to the fifth aim of the study,
namely ‘to provide a focus for the presentation, to other EU countries, of the
Italian model for state-sector adoption of the ideas and proposals of theCommon European Framework of Reference for Languages: Learning,teaching, assessment’ (see above).
Chapter 4 will detail the long and sometimes tortuous route between theinitial specification of research questions and their ultimate representation in
the various impact study data collection instruments – but that belongs to theimplementation rather than the planning phase of a study. Plans

47

3 Impact study objectives, designs and research questions

Research designs and action plans
Most of the models of pre-implementation research steps discussed include what
may be called research designs and/or action plans, work plans or project
development plans, depending on whether a study is conducted as academic oras project research. Sometimes, of course, it may be both. The research ofClapham (1996), for example, on the effect of background knowledge on reading
comprehension, investigates English for specific purposes constructs underlying
the ELTS and IELTS Reading modules, and is described as ‘doctoral work
carried out … over a number of years and supported by UCLES’.
An action plan is concerned with translating the purpose, aim andobjectives of the study into a route to be followed, as far as possible, to reachthe intended destination, that is the achievement of the study objectives. The
action plan should demonstrate that the study team:• specified the main phases and activities of the project (e.g. instrument
development, participant identification and contacts, data collection
design, data management systems etc.) which may actually be called theresearch design, that term representing one part of the action plan)• defined the study management structure, staffing and responsibilities• obtained approved finance for personnel, supplies and equipment andoverhead resources• agreed study monitoring and reporting systems.
Weiss defines a research design as ‘the plan or structure a researcher develops
to guide the study’ (1998:330), which distinguishes it from an action plan, whichis intended to ensure the framework and resources for the study are defined andavailable. For Weiss, a research design should specify target participant groups
and numbers, selection criteria, data collection intervals, methods and means.
The study of IELTS impact described in this book, which, as we have seen,
is a less one-off, short-duration project than the PL2000 Impact Study, andpart of a continuous, formative test consultation and validation programme,
had a broad action plan envisaged as comprising three phases:
Phase 1 for the identification of areas to be targeted and development ofdata collection instrumentation; Phase 2 for the validation andrationalisation of these instruments, and Phase 3 for the collection andanalysis of impact data (Saville and Hawkey, 2004:76).

Each of these phases involved its own action plans (see Chapter 5 for plans
guiding the instrument development phase).
The IELTS impact study research questions shown above were
contextualised into the impact study implementation research design forPhases 2 and 3 of the project, as shown in Figure 3.2.

48

Research designs and action plans
Figure 3.2 IELTS impact study Phase 3 research design of datacollection chronology, types and inter-relationships
2001

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep

Oct

Nov

Dec

Rationalisation and finalisation of instruments
Design of pre-survey instrument,
sample and data management
systemAdministration ofpre-survey instrument

Analysis of pre-survey data

2002

May Jun

Sep

Jan

Feb

Mar

Apr

Design, preparation andorganisation of participant
sample

Jul

Aug

Oct

Nov

Dec

Administration of impact study instruments

Schedule of centre visits and classroom
observation
2003

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep

Oct

Nov

Dec

Finalisation of data management system, analysis of data and preparation of preliminary reports

Notice the use of a pre-survey (of 300+ IELTS centres worldwide) to collect
up-to-date preparatory data on centres currently administering IELTS,
including: their key contacts and schedules; course numbers, durations;participants’ backgrounds and numbers; teacher numbers; main IELTS
textbooks. This early implementational step (see Chapter 5) also provided an
opportunity for the centres concerned to indicate their willingness toparticipate in the subsequent data collection phase, which would involve someconsiderable time and effort on the part of their students, candidates andteachers.
Figure 3.3, which shows the very straightforward project planning
framework agreed for the PL2000 study, also illustrates the place of an action
plan in the project development phase. Figure 3.4 (page 50) reproduces thePL2000 Impact Study reseach implementation design.
Figure 3.3 PL2000 Impact Study project plan model
Project Development• State purpose and objectives

Project Implementation

Project Assessment• Communicate with partners• Report achievements• Describe outcomes, activities• Assure quality• Communicate lessons• Estimate budget• Keep to schedule• Determine next steps• Develop action plan

• Manage costs49

3 Impact study objectives, designs and research questions
Figure 3.4 PL2000 Impact Study research design of data collection
chronology, types and inter-relationships
February > September 2001

Profiles of selectedcase study schools• numbers of students,
classes, teachers• EL staff• PL2000 classes• hpw• foreign languages• EL tests• course books• resource centres• EL use• PL2000 TL impact,changeCase study classstudent backgrounddata• age, gender• foreign language
years, experiences,
attitudes• EL tests taken,
planned• student EL test andother TL level data
Pilot videos
Classroom, andstudent, head, teacher,parent interviews

October 2001

> early
2002

School headinterview 1• PL2000 experience• plans• attitudes• FL teaching andmanagement aims
Student
questionnaires 1• EL access, uses,
attitudes• future study andwork plans• Language Learning
Questionnaire(learning attitudes,
motivations,strategies)• Quick Placement
Test (EL level)

Teacher interviews 1
including:• PL2000• EL needs, methods,
materials• EL progress• FL exams
Case study
classroom videos 1
Parent interview(s) 1
including:• attitudes to FL,
PL2000• FL needs, aims,
methods, progress,
exams, futures

April 2002
School headinterview 2• PL2000 experience• plans• attitudes• FL teaching andmanagement aims
Student
questionnaire 2• EL learning
activities• EL access, uses,
attitudes• future, study andwork plans• EL gain, change
Teacherquestionnaire• age, qualifications,experience• EL classroom
activities• PL2000 aims,
achievement• PL2000 advantages,
disadvantages• attitude change
Teacher interviews 2• attitude change• PL2000 methods,
change/progress,
standards• EL status, purposes,
standards,change/
progress.
• FL exams
Case study
classroom videos 2
Parent interview(s) 2
including:• attitudes to FL,
PL2000, children’s
needs, methods,
progress,• external exams,

Experience of and views on PL2000 from eight teacher entries to
the Cambridge ESOL – Bell International School essay competition
Teacher information on student progress, sample work, test results, PL2000 management
etc.

50

> June 2002

Outcomes
and further
research
proposalsHypotheses
The development phase of the PL2000 Impact Study, from the first proposal
for a study in August 2000 to its approved action plan and agreement inNovember 2000, saw regular draft study proposals circulated, feedback
reports exchanged, Cambridge ESOL team policy and planning meetings heldat Cambridge and in Italy, and liaison and approval meetings with appropriate
Ministry of Education officials. The study attempted and, in the main, adhered
to the research implementation design specifying participants, types of data,
and data collection methods under the longitudinal study timeline.

Hypotheses
In the models of research planning cited, hypotheses tended to come at the endof the planning steps; that is when the study really begins. In fact, though,research, undertaken as it is because there are questions that appear to require
answers in some kind of empirical way, also starts, implicitly, fromhypotheses. In the lay sense of the term, as an unproved theory or proposition
tentatively accepted to explain certain facts or, as with a working hypothesis,
to provide a basis for further investigation, both the IELTS and the PL2000
studies began with hypotheses.
With the IELTS study, the research question: ‘What is the washback of the
IELTS test on courses preparing candidates to take it?’ implies a working
hypothesis:• the IELTS test affects courses preparing candidates for the test.
When Cambridge ESOL felt that a ‘study of the processes and products ofEnglish language teaching, learning and assessment on PL2000 English
language courses would produce revealing data on the VRIP of Cambridge
Main Suite exams’ (see previous), hypotheses were already implied, forexample:• the PL2000 affects English language teaching on courses under the
Project, or• the use of a Cambridge ESOL exam at the end of an English course under
the PL2000 affects English language learning.
The definition and status of hypotheses in research depend on the research
approach. Weiss, tending to espouse more qualitative approaches, defines ahypothesis as follows:
The researcher’s firm hunch, arising from theory or experience, that a
particular predictor (in the evaluator’s case a programme or an aspect of aprogramme) causes an outcome. Hypotheses are confirmed or denied
using empirical analysis (1998:331).

51

3 Impact study objectives, designs and research questions
In more quantitative research circles, the definitions are somewhat stricter.
As in Snodgrass, for example:
In hypothesis testing, we divide all the possible states of the world into
two competing hypotheses such that if one is true the other is not, and viceversa. One of these hypotheses is called the null hypothesis, and the otheris called the alternative hypothesis. The only hypothesis that is tested
directly is the null hypothesis, and the investigator either rejects, or fails
to reject, the null hypothesis (1977:187).

Restated as a null hypothesis, the IELTS study hypothesis above would be:• the IELTS test does not affect courses preparing candidates for the test.
It is clear that null hypotheses and their testing are most closely associatedwith the experimental research paradigm (see previous), where the researcher
controls the key variables and the conditions in which the research or
hypothesis-testing study takes place. Given the more qualitative researchparadigm adopted by both examples and by most impact studies, for thereasons given in Chapter 2, we tend to be dealing with research questions andworking hypotheses rather than null and alternative hypotheses.
Green (2003), in his doctoral thesis, a comparative study in backwash
between IELTS preparation and university pre-sessional courses, does present
his research hypotheses in explicitly null terms, for example: ‘IELTS
preparation courses do not reflect the English for academic purposes (EAP)
construct of academic writing’. The testing of this hypothesis involves the useof questionnaires to language teaching centres, repertory grid interviews (seeKelly, 1955), case study interviews and questionnaires, classroom observationand the analysis of classroom artefacts. Some of Green’s hypothesis-testingmeasures do not differ greatly from those used to collect data in response to
the research questions of the IELTS and PL2000 studies, both of which use
questionnaires, case study interviews and classroom observation and theanalysis of ‘classroom artefacts’. This will become clearer when we look
more closely at the implementation of both studies, in Chapters 5–7.
This chapter has illustrated how, both in theory and in practice, and in both
our impact studies, the constructs at the top of the research planninghierarchy, purpose, aims, objectives, though not always defined or usedconsistently, are as essential to the planning of impact studies as they are inother forms of research. There is clearly the need, in the pre-implementation
stage of an impact study, to state what the study is setting out to do, and how
it relates to its field. Research aims are analysed into research questions,specified rationally to take account not only of answers that the aims of theresearch imply are needed, but also of the practical constraints faced by thestudy. A research design specifies the target participant groups and numbers,
selection criteria, methods and means for the collection of data in response to52

Hypotheses
the research questions. Good research project action plans attempt to provide
the essential prompts and criteria for the resourcing and tracking of progressin an impact study. Hypotheses, whether of the null, alternative or working
kind, tend to represent the final step in the research planning process, andsignal the beginning of the research proper.
In Chapter 4, we shall look at study implementation. In particular we shalltrace the development of instrumentation for the collection of data in answer
to research questions and the support or refutation of hypotheses, all in theprocess of the fulfilment of research objectives for the achievement ofresearch aims.

53

4

Impact study instrumentation

Chapter 3 has illustrated the importance of analysing impact research aims
into questions, research designs, action plans, and hypotheses. In thischapter we shall report on the next steps in the implementation of our two
studies, in particular the development and validation of data collection
instrumentation.
The chapter will suggest examples of the conceptual and procedural
differences between language tests and impact data collection instruments,
then put into context the data collection instruments used in the sample
studies. The development, validation and use of the instruments for the twostudies will next be summarised. Throughout, there will be a focus on key
principles and processes, including operationalising, prediction,
brainstorming and review, piloting and trialling, post-trial validation, and, inthe case of the IELTS impact, the rationalisation of instrument use.

Distinctions and similarities between datacollection and test instruments and theirvalidation
In the paper commissioned as part of the early research for the study of the
impacts of IELTS, How might impact study instruments be validated? (seeChapter 2), Alderson and Banerjee express some surprise at the apparent lackof ‘a well-developed literature on how the reliability and validity’ of‘questionnaires, interviews, surveys and classroom observation schedules’
should be established (1996:1). They then discuss some of the distinctions andsimilarities between tests and data collection instruments and in theirvalidation. The insights provided by their paper and experience gained as
research progressed on both the impact studies under discussion here may
help to explain some of the relationships between the validation of language
tests and the validation of data collection instruments. In Chapter 2, we havealready seen, at the level mainly of principle rather than practice, how analytic
strategies such as the ones described here are essential to telling the story of
an impact study.

54

Distinctions and similarities

Distinctions• In data collection instruments, for example in open-ended questionnaire
items such as those in the student and teacher questionnaires for both ourstudies, it is the veracity of participant responses that is the focus of theanalysis rather than, as in language tests, their linguistic accuracy orappropriacy. However, since responses to questionnaire items may notalways, for a variety of reasons, reflect what participants actually think orfeel, checks on veracity need to be made, where feasible, whether throughrelated items or through triangulation from other data sources on thematters concerned.
• Item difficulty analysis, to ascertain ‘the relative difficulty of a test item
for a given group of test takers’ (McNamara 2000:134), would not seemto apply, in its comparative, language proficiency level-setting sense, toquestionnaires. The consistent ease of understanding of data-collecting
items, from both a conceptual and language difficulty point of view, is,however, crucial, and was a constant concern throughout the developmentof IELTS and PL2000 instrumentation and face-to-face data collection.
• Data collection questionnaire items may not be susceptible to certain
other language test item analysis operations. Internal consistency
measures, methods to estimate test reliability depending on the‘homogeneity of item variance as a reflection of consistency of scoring’(Henning 1987:193), may not be appropriate, for example. Questionnaires
such as the IELTS or PL2000 Impact Study student questionnaires
normally seek a range of information from members of a participantgroup. They may well thus contain single items on a particularautonomous matter such as these from the PL2000 study questionnaire,
requiring responses on the frequency of activities in participants’ foreign
language learning classes:

ver
Ne

es
tim
me
So
en
oft
ite
Qu
y
ntl
que
Fre

Comments

grammar exercises
discussions with thewhole class

There is no necessary reason why students’ responses on the two items
should coincide across classes and teachers, or be ‘consistent’.
• Item Response Theory (IRT), a form of statistical analysis that ‘gives anestimate score on a test for each individual, basing this estimate on that
individual’s performance on each of the items on the test’ (Hughes55

4 Impact study instrumentation
2003:42), may be used to validate test items by ascertaining their matchwith a test-taker’s ‘estimated performance’. With questionnaires seeking a
range of participant experiential or attitudinal matters, IRT may not be
useful for instrument validation. An individual’s ‘unusual’ responses may
not be caused by flaws in the items but rather represent the actual valueon the variable concerned.

Similarities
It was also becoming clear from our studies that the design of data collection
questionnaires and language tests shared certain similarities, including theneed for validation so that claims to be made on the basis of their findings
would be well-founded. Questionnaires, interviews, surveys and observations
are, as Alderson and Banerjee point out:
… not dissimilar to tests in that they too seek information about a person,
be it their spending behaviour, their attitudes towards corporal punishment
or the way they learn a new language. So, as with tests, it would not be
unreasonable to assume the importance of establishing the reliability andvalidity of data collection instruments i.e. whether they are capturing thesame information each time they are administered and whether these
instruments are actually capturing the information researchers expect themto capture (1996:7).

Some of the similarities between test and data collection instruments and theirvalidation potential are now discussed.
• Data instruments measuring cognitive or affective traits by inviting
participant responses to statements representing the trait concerned aresimilar to certain kinds of language tests, for example those using
multiple choice items. The Language Learning Questionnaires (LLQ), onesuch example, were originally developed, in collaboration with UCLESEFL, by Bachman, Cushing and Purpura (1993) and finalised (Purpura
1999) in the form of questionnaires on socio-psychological factors,
namely language learning attitudes, anxiety, motivation and effort. The
LLQ which was used to collect data in both the IELTS (see Appendix A)
and the PL2000 studies uses sets of items representing particular traits (orabilities). It is thus more suited to some of the validation operations
associated with language tests, for example:– Descriptive analyses e.g. totals; proportions; means; standard
deviation: skew and kurtosis (both these latter terms referring todepartures from the normal distribution of scores, skew so that ‘thepattern of scores is not symmetrical around the mean’, kurtosis so thatthe distribution is ‘flatter or more peaked than normal’, Henning56

Distinctions and similarities

•

•

•

•

1987:183, 197); such analyses led, for example, to proposals during
Phase 2 of the IELTS study (Gardiner 1999) for the modification of the
7-point scale on the Test-taker Background Questionnaire of sociocognitive and cognitive, and test-taking variables to five points (seealso below).
– Factor analysis, which refers to ‘a variety of multivariate, correlational
statistical procedures used to aggregate data into non-overlapping or
minimally overlapping categories or factors’ (Henning 1987:192), isused in two ways. The first, exploratory factor analysis, examines
patterns in the observed questionnaire data but assumes ‘no a priori
patterns in the data’. Confirmatory factor analysis is used ‘to determine
the extent to which items designed to measure a particular factor
actually do so’. Both these definitions are given by Purpura himself(2004:100) writing on the statistical procedures considered and used in
the validation of the Language Learning Questionnaires.
Questionnaires, like multiple-item language tests, are amenable tovalidation through correlation analyses within instruments, for example
PL2000 student baseline data performance on the Oxford University Press
Quick Placement Test (QPT) (see Chapter 5), and subsequent
performance on their First Certificate in English (FCE) exams (both also
see Chapter 7). Questionnaire item responses may also be checked forvalidity through triangulation across data sources, for example IELTS
student questionnaire responses on test module difficulty compared withstudent interview and focus group responses on the same matter.
Retrospective interviews, as in some of the instrument piloting for the
study of IELTS impact referred to in this chapter, and participant thinkaloud protocols are seen as possible validating tools for questionnaire
instruments, as they are for some test tasks.
Inter-evaluator (c.f. inter-rater) reliability may be checked, as it was in thetrialling of the Instrument for the Analysis of Textbook Materials(IATM), (see Appendix C), in the study of IELTS impact. Comparisonswere made between the responses of different evaluators on the sametextbooks and the responses of the same evaluators on different textbooks,
representing an attempt to establish convergent – divergent validity(Litwin 1995:44 cited in Alderson and Banerjee 1996:27).
Test-retest reliability validation is an ‘estimate of the likelihood that thetest would rank test takers in the same order from one administration toanother proximate one’ (Henning 1987:196). The technique is mentioned
as planned for the validation of the IELTS impact Test-taker Background
Questionnaire (precursor to the LLQ), when trial questionnaires were sent
to IELTS centres during 1997–8, with ‘a number of candidates
completing the questionnaire twice, so a test-retest reliability analysis

57

4 Impact study instrumentation
could be carried out’ (Gardiner 1999:4). No test-retest data were, in theevent, analysed from this trial, but there would appear to be questions
over the use of the technique with instruments seeking attitudinal data,
which are subject to fluctuation, although instruments such as the LLQ
would seem more amenable to test-retest analyses since they aremeasuring socio-psychological traits of some permanency.
• Multi-trait, multi-method validation procedures are also referenced byAlderson and Banerjee, premised, as they are, on ‘the idea that if aconstruct exists it should be possible to measure it in more than one way’in the expectation that ‘the correlation among these different measures ofthe same thing should be strong and positive’ Fitz-Gibbon and Morris(1987:15). Logically, this principle should apply both to tests and meansof data collection. It also appears to overlap triangulation (see Chapter 2
and this chapter) quite strongly. In the first three phases of the study of
IELTS impact study, attention was paid during instrument development
and validation, through piloting, trialling and use, to the comparison ofdata across, for example:• closed and open-items on the same construct in the same instrument
e.g. in the Teacher Questionnaire (see Appendix B):
If an IELTS score had not been a requirement would you have
prepared your students for their future studies in the same way?
Yes/NoWould your IELTS preparation course be a good way to learn English
for someone going to university but who is not going to take IELTS?
Why? Why not?
• responses on the same construct across different data collection means,e.g. Teacher Questionnaire and teacher interviews/focus groups (seethis chapter and Chapter 6).
Since, as Alderson and Banerjee (1996:7) suggest, impact study datacollection instruments tend sometimes not to be validated at all and since, itwould appear, no standard models of instrument validation for use in the studyof impact appear to exist, it seems natural to look to language test validation
principles and procedures for direction. As the examples above indicate, thereis clearly no exact fit across the two fields, given key differences of topic,
purpose and enquiry type. But there are lessons to be learned.
None of the validation techniques covered above will, of course, inthemselves prevent the fundamental problems of construct under- or overrepresentation, that is, making sure that data collection instruments actually
cover adequately the areas required by the research focus. It is only throughcomprehensive and thorough work on study aims and research questions,followed by their iterative verification using approaches such as those
discussed below (e.g. brainstorming, expert opinion, prediction and review),that this will be achieved.
58

Contextualising the instruments

Contextualising the instruments
IELTS impact study
As discussed in Chapter 2, work on IELTS impact studies was to begin by reexamining fundamental impact and washback concepts. Thus when, in June
1995, in response to a request by Dr Mike Milanovic, then Director of the
Validation Unit of UCLES EFL, Charles Alderson submitted the paper Ideas
for research into impact, washback and IELTS (see Chapter 2), it was
exploratory and explanatory on complex issues such as:••••

the major descriptive function of washback/impact studies
their role in test validation
high-stakes test stakeholders
the scope and limitations of impact studies.

The paper also covered ‘specific ideas for research’ such as:• predicting impacts, then comparing predictions with response data
• the need for baseline data (that is ‘measurements obtained to indicate
levels of performance before the intervention of an educational or otherprogramme against which to assess change’ (Satterly 1989:336))• the relevance of attitudinal data
• the need to distinguish between real impacts and coincidental policychanges• the importance of systematising test impact research including‘specifically set-up studies’ (1995:11) as a contribution to continuing test
revisions.
The difficult problem of identifying with any certainty cases of classroom
washback remained, of course. It is not easy to ‘prove’ that events in aclassroom or performances beyond it are actually washback from tests or
curricular innovation, rather than the result of other factors. These ‘otherfactors’ may include learner characteristics, attitudes, levels, preferences andso on. They may also include teachers’ characteristics, attitudes, levels,
preferences, approaches – or the changes in classroom processes and learnerperformance may be the result of extraneous influences not really to do withlearners or teachers (see also Alderson and Hamp-Lyons 1996, Andrews 1995,
Watanabe 1996). Given the complexity of washback relationships, washback
studies can only seek to ask the right people the right questions, analyse theanswers rigorously, present findings accurately and fairly, infer and explain
insightfully on such matters.
Alderson’s June 1995 paper ended with this ‘final thought’:59

4 Impact study instrumentation
One of the most significant ways in which UCLES can have impact on theprofession – of teachers as well as testers – is by showing that it takes
seriously its responsibility to have positive impact where at all possible
and that it is willing to investigate and establish this. In doing this, UCLES
will be taking a very important lead in a traditionally conservative area,
and I pay tribute to UCLES’ willingness to innovate, investigate, andhopefully report on its findings in the general area of impact and washback(1995:12).

A further obvious concern, from the outset of the IELTS impact study project,
was the closest possible attention to the validity of data collected, the themeof the further paper commissioned for the study: How might impact study
instruments be validated? (Alderson and Banerjee 1996), which proved
seminal to the IELTS impact study (see above).
The historical context for such concerns is of relevance to this book in thatit once again illustrates the systemic role of impact study in the validation oftests and programmes. The specifications document of the revised (1995) formof the IELTS test identifies the collection of test impact data as part of
UCLES’ continuing test validation programme. It was around this time, too,that UCLES was espousing the ‘expanded’ view of impact (see Chapter 1)
with four maxims of test impact to ‘achieve appropriate impact withinternational examinations’, namely:• plan (through a ‘rational and explicit approach to test development’)• support ( of ‘stakeholders in the testing process’)• communicate (by providing ‘comprehensive, useful and transparent
information’) and• monitor and evaluate (by collecting ‘all relevant data’ and ‘analysing’ as
required’) (Milanovic and Saville 1996:10).
Alderson was the obvious choice for a key role in the development of
instrumentation for studies of IELTS impact. He was co-editor, with Caroline
Clapham his colleague at Lancaster, of IELTS Research Reports 2 and 3 in the
English Language Testing Service (ELTS) revision project which led to therevised test, the IELTS, in 1989. Alderson also, of course, had experience ofthe practices as well as the theory of impact studies through the Sri Lankan
Impact Study (Wall and Alderson 1993) and a study of the washback of the
TOEFL test on preparation courses (Alderson and Hamp-Lyons 1996).
The 15 June 1995 commissioning letter from UCLES EFL requested
Alderson and his team to ‘carry out the next stage of activity’, namely to
specify the four main project areas as described by UCLES in terms ofstaffing and the delivery of a set of instruments for the collection of relevant
impact data. The original wording of the four project area descriptions is
significant in that it illustrates the breadth and continuity of the studies
envisaged by UCLES EFL:

60

Contextualising the instruments: the PL2000 Impact Study
1. The design of a range of instruments and procedures for the collectionof classroom data which will be applicable to IELTS and eventually toother UCLES EFL exams. This will be based on work done by
Lancaster staff and will include the following: a) a questionnaire forteachers; b) a questionnaire for students; c) an observation schedule forclassroom activity; d) a procedure for producing summaries of criticalclassroom activity; e) a procedure for setting up focus groups andrecording the data (questions, audio recording etc.).
2. An instrument for materials analysis which will focus on IELTS
materials which are currently available. This can be based on a numberof sources including the Colorado experiment [reference to theAlderson and Hamp-Lyons 1996 study of the washback of the TOEFL
test], work by post-graduate students at Lancaster and existing
checklists (such as the ALTE checklists).
3. The design of four questionnaires to collect views and attitudes about
IELTS from the 4 main user groups: a) test takers b) EFL teachers; c)administrators; d) lecturers in receiving institutions.
4. An instrument for collecting information about the test-taking
population and the use of test results i.e. in addition to the standard
demographic data collected using the [Cambridge ESOL] Candidate
Information System. This instrument focusing on IELTS candidates
can be adapted from a range of existing questionnaires (such as the
UCLES/UCLA questionnaires [reference to Bachman, Cushing and
Purpura (1993)].

Note then that an extensive range of instruments was foreseen for a series of
IELTS-related impact studies, and with the possibility of the eventual use of
the instruments beyond the IELTS study.

Contextualising the instruments: the PL2000 Impact
Study
The instrumentation for the Progetto Lingue 2000 Impact Study was derived
from the aims of the study as agreed between the Italian Ministry of
Education, senior Cambridge ESOL staff members and an external consultant(see Chapter 3). The study was envisaged as seeking data on influences ofPL2000 on English language performance, pedagogy, textbooks and othermedia, evaluation and assessment, and resource centres. The data were to becollected from stakeholders such as learners, teachers, teacher trainers,
parents, evaluators, and educational managers.
The data collection instruments designed for the study were:
1. A school profile pro-forma (see Appendix F).
2. A student questionnaire (in three parallel versions) (see Appendix G).

61

4 Impact study instrumentation
3. A teacher questionnaire (see Appendix H).
4. A classroom observation analysis form (see Appendix D).
5. Structured interview or focus group forms for use with teachers, schoolheads and parents (see Appendix I).

IELTS impact study instrument development andvalidation: principles and processes
At this stage of a chapter summarising the development and validation of the
instrumentation used in our two studies of impact, it seems appropriate to trace
how the instruments were developed and validated. Phase 1 of the study of
IELTS impact would cover ‘the identification of areas to be targeted anddevelopment of data collection instrumentation’, and Phase 2 ‘the validation
and rationalisation of these instruments’. Phase 3, as already noted, would
cover ‘the collection and analysis of impact data’ (all from Saville and
Hawkey 2004:76).
The instruments for use in the study of IELTS impact were characterised by
the following cycles of procedures, all contributing, it was agreed, to thedesign of valid instruments:
•
•
•
•
•
•

operationalisation
prediction
brainstorming and review
piloting and trialling
qualitative and quantitative validation, andrationalisation.

IELTS impact study: operationalisation
IELTS impact study aims (see Chapter 3) had been agreed and specified inadvance, as had the scope, main research questions, and appropriate
methodological options for the study. The constructs, that is the traits,characteristics or features hypothesised as relevant to the impact study
research questions had, of course, to be operationalised, that is converted into
traits that can be measured, ‘relating constructs defined theoretically to ourobservations of behaviour’ (Bachman 1990:42). The second stage of
operationalisation involves including the constructs in an instrument orprocedure for collecting data on them.
Figure 4.1 (page 67) will summarise the contents and item types of the datacollection instruments to be used in IELTS impact study Phase 3. To illustrate

62

IELTS impact study instrument development and validation
the processes entailed in developing, trialling and validating the instruments,
experience with particular IELTS impact study instruments (student
questionnaires, teacher questionnaires, textbook analysis instrument, classroom
observation form) will be cited here, then summarised in the Figure.

IELTS impact study: prediction
As noted in Chapter 2, a prominent role in the instrument development andvalidation process was given to clarifying constructs through predictions of
the washback/impact of the test on major test areas, e.g. reading, writing,listening, speaking. The study of IELTS impact also used response prediction
at the questionnaire item level to help ensure the relevance and workability ofitems. The Proposal for work to develop instruments to study the washback
and impact of IELTS (Alderson), notes that it ‘is essential to gather
predictions [my italics] as to the likely washback effects of IELTS andassociated materials from item writers, EAP teachers, teacher trainers andlanguage testers’ (3 August 1995). The April 1996 IELTS Impact Study
Report from Lancaster to UCLES notes that such predictions were collectedfrom participants at a British Council Linguistics seminar (September 1995),
which also included members of the Lancaster Testing Research Group(LTRG). Participants should, Alderson suggested, consider washback effects‘from the point of view of: topics, content, texts, skills, feedback types, testtaking and in-class use’ (ibid:1).
A further Phase 1 Lancaster discussion meeting on 24 October 1995
illustrated the importance attached to predicting washback, and the fact thatthe team pursued such predictions assiduously. The report of the meeting
notes that participants (13 researchers, an MA student and staff from theInstitute of English Language Education and the Department of Linguistics
and Modern English Language) were asked actually to take the IELTS sample
Listening test and receive an orientation briefing on the Reading module,
before brainstorming on: how they would expect students to prepare for thetests, what sort of test preparation techniques to expect, what sort ofpreparation would be appropriate, what methodology, and what they would
consider to be negative washback in preparation practice. The usefulness of
this exercise in the development of Phase 1 student and teacher questionnaire
items and in the design of classroom observation instruments will beappreciated.
An example of the useful debate and thought encouraged by the use of
response prediction was on the question of test fairness. Test taker perceptionsof the fairnesss of the IELTS test seemed to various impact study data
instrument developers relevant to a research question such as: What is theimpact of IELTS on the participants who have taken the test? But predicting
the response in the process of validating items brought a range of responses
634 Impact study instrumentationsuggesting the need for careful follow-up sub-items (for example on reasonsfor the response) and related items (for example on the test takers’ typical and
IELTS-specific views of their own test success).

IELTS impact study: brainstorming and review
Brainstorming, meaning discussion of evidence and the offering of different
views to enable movement towards problem solution (May and Sigsworth
1983:257–58) also proved important in the development of IELTS impact
study instruments. Brainstorming sessions like those mentioned in theprevious paragraph would typically cover test effects on Alderson’s (above)
teaching/learning topics, texts, tasks, skills, feedback types, test-taking andin-class use. In Phase 1, such matters as these were reviewed regularly by
project staff at Lancaster and the Project Co-ordinator (Alderson), and at
impact study liaison meetings between Alderson and relevant UCLES EFL
senior management and IELTS staff. Records of such meetings were
circulated at UCLES and Lancaster for further comment. The Notes on
Meeting at UCLES, IELTS Washback Project, 1 November 1995, exemplified
the combination of prediction and brainstorming methods of instrument
validation, as well as the fairly formalised dissemination and review
procedures:
The Lancaster Language Testing Research Group had already conducteda brainstorming on predicted washback of Listening and Speaking [seepreceding section]. The report would be edited by the Lancaster team, andthen sent to UCLES for comment and additions. A similar procedure
would be followed for the results of today’s meeting, with the Lancasterteam commenting on the edited version of today. The two edited sets ofpredictions would then be combined into one document which would besent to those who had not been able to be present at either meeting, as wellas other interested and affected parties in the UK and [Australia]. A
version of the predictions would be discussed at the [IELTS] Advisory
Committee in February [1996], but instrument construction would have tocommence using whatever sets of predictions were available when thework began (1 November 1995).

The May 1996 IELTS Impact Study Report from Lancaster to UCLES also
underlines the importance in instrument development of brainstorming on
item and scale-types. These sessions would attempt to ensure as far as possiblethat item wording enabled participants to understand the questions in the way
they were intended (taking account of the fact that learner participants in the
study would not, in the main, be first-language users of English) and to review
draft instruments. At the Lancaster Language Testing Research Group(LTRG) meeting of 23 April 1996, for example, draft instruments had beenreviewed by members with reference to four questions:
64

IELTS impact study instrument development and validation• is the instrument/procedure likely to measure IELTS washback
adequately?
• have important aspects of washback/impact been overlooked?
• can the instrument/procedure be shortened?
• are there any practical points on instrument administration or data
analysis?
Note the concern here for measurement efficiency, construct underrepresentation, economy of time, and practicality.

IELTS impact study: piloting and trialling draft
instruments
Alderson and Banerjee suggest that ‘pilot surveys involve the administrationof the instrument on a small number of respondents with the intention of
revealing problem areas in the instrument’(1996:7). They also feel that muchof the relevant literature on piloting concentrates on procedures for eliciting
information and views from participants that may be useful to ‘check theadequacy of questions’ rather than running ‘statistical analyses on the pilot
data’ (op. cit). Presumably, however, the latter procedures would depend on
the size of the pilot sample.
In the case of Phase 1 of the IELTS impact study, the piloting was carriedout on small numbers of participants but with intensive analysis of theirreactions to the draft instruments and items. Studies also involved subsequent‘wider scale international piloting’ (IELTS impact study report (1996:3), often
called ‘trialling’ in project documentation to distinguish it from the smallscale Phase 1 piloting. The trials used larger sample participant groups toallow for statistical analyses and instrument modifications based on these. So,
the piloting was carried out during the Phase 1 Lancaster development phaseof the IELTS study, the trialling by UCLES in Phase 2.

IELTS impact study: post-trial validation
Examples will be seen (Figure 4.1 below) of a wide range of techniques, bothquantitative and qualitative, used in the validation of draft data collection
instruments for the study of IELTS impact. The Figure will also describe theirmodification into the forms in which they were used to collect impact data.
Descriptive analyses (frequency statistics, mean, standard deviation, skew,
kurtosis, see above) were run using an Access database, with the data imported
into the Statistical Package for Social Science (SPSS), for example on theStudent and Teacher Questionnaires). Factor analyses used the EQS for
Windows programme, for example on the trial data from the Test Takers’65

4 Impact study instrumentation
Background Questionnaire (TTBQ). This instrument also made use ofquantitative and qualitative revision data in the form of reviews by UCLESEFL Validation Group staff, as well as outside consultants such as Hawkey
and Green (2000), Kunnan (2000), Purpura (1996b). These reviews led tomodifications to constructs, scope, scales and item formats. Convergent –divergent validation data on the draft Instrument for the Analysis of Textbook
Materials (IATM) were derived from more than one evaluator analysing thesame book, and the same evaluators analysing different books (see previous).
Rater responses from the trial analyses of the draft IATM were consolidated
on to a comparative item analysis form covering all draft items. The response
aggregates were investigated in particular for inter-rater differences andinconsistencies. Qualitative validating data on the draft IATM were alsocollected at the intensive brainstorming sessions between evaluators and thequestionnaire reviser. Brainstorming and review exercises took similar forms
in the validation of all the IELTS Phase 3 impact study instruments. The full,
finalised instruments appear at Appendices A–E.
Figure 4.1 (pages 67–72) now summarises main events in theoperationalisation, piloting, trialling and validation of instruments during
IELTS impact study Phases 1 and 2.

IELTS impact study: rationalisation
With phased projects such as the study of IELTS impact, which have an
iterative design and include instrumentation not necessarily intended for
immediate or all-at-once use, it is likely that the collection of data for any
particular study phase will entail rationalisation of instrument use. The
discussions of instrument use in Figure 4.1 will already have indicated thatsome instruments were not yet to be used in Phase 3 of the study, and that
others were to be used in modified form.

66

Operationalisation

5) Phase 3 SQ includes items from Phase 1 TTBQ
and Phase 2 LLQ development and trialling.

4) Item response formats:open-ended, Yes/No, option selection (inventory-categorial, proportion, frequency/time,
usefulness, intensity, Likert scales, rank order).

3) Validated TTBQ and LLQ items included
in SQ.

2) Most focus areas retained, i.e.:
Student:• L1, Target Language (TL) background• IELTS prep course type, activities,
materials, views• for post-test candidates: IELTS outcomes,
aims, pressure, fairness, difficulties,
performance self-assessment, likes/dislikes,
advice and general comment.

Student Questionnaire 1) Focus areas of Phase 3 version derived from
(SQ, see Appendix A)
Phase 1 draft versions of SQ i.e.:• Questionnaire 5 (Q5) for students preparing
for IELTS• Q8 for students who have taken IELTS• Test Taker Background Q12 (TTBQ) and
LLQ modified on basis of trial data from
Phase 2.

IELTS Study
Instrument• Phase 2 revised TTBQ, including
LLQ items, trialled on 500
candidates before their IELTS(Gardiner, 1999).

• TTQB (Herington 1996) draft
informed by LLQ , (Bachman,
Cushing, Purpura 1993), trialled
on 126 IELTS test takers at four
UK test centres.

• Final version of Phase 1 draft SQ(without LLQ) after local piloting(Horak 1996), then trialled on
379 students in nine countries
in Phase 2

Piloting and Trialling• Trialled instrument data interpreted throughdescriptive analysis by SPSS: value label,frequency, percentage, valid percentage, andcumulative percentage.
• Open-ended responses analysed qualitatively
for additional insights into research questions.
• Triangulation of quantitative and qualitative
data, backing or rebutting related data.
• Analyses included frequencies, percentages,
reliability estimates (Kuder-Richardson 20 for
internal consistency, acknowledged at the time
as not entirely appropriate since constructs represented in the TTQB by only two items each).
• Mean, standard deviation, skew, kurtosis,
frequency and factor analyses on trial data,with recommendations for revision.
• Revisions also suggested by UCLES EFL
Validation Unit, Kunnan, Purpura, Hawkey
and Green (see Hawkey and Green 2000,
Kunnan 2000, Purpura 1996b).
• Quantitative and qualitative analyses taken
into account in further revisions, incl. changes
to constructs, scope, scales and format.
• Phase 3 SQ subjected to expert analysis(Green, 2003).

Validation

Figure 4.1 Summary of main operationalisation, piloting, trialling and post-draft
instrument validation exercises for IELTS impact study instruments

IELTS impact study instrument development and validation

67

68

1) IATM Constructs from study aims and researchInstrument for thequestions.
Analysis of Textbook
Materials(IATM, see Appendix C) 2) Main sources for lists of categories: ALTE
Development and descriptive checklists fortasks and examinations (1995), Munby (1978)3) Item response formats: open ended, Yes/No,
option selection (inventory-categorial,
proportion, frequency/time/proportion,
usefulness, intensity) and rank order.

2) TQ contents: Teacher• background and experience• students: ages, countries, edn., target
countries, module, IELTS status• IELTS suitability, reliability, pressures,
washback• IELTS preparation likes and dislikes• candidate success features• advice on IELTS preparation• preparation course success• test features• IELTS prep classes: type, length, course
activities; test practice, materials; participant
success characteristics, suitability• further comments.

• Phase 2 analysis and revision,
final draft trialled with two expert
evaluators.

• Phase 1 (Bonkowski, 1996)
trialling through nine trial IATM
textbook rater analyses bypractising teachers.

• Final version after local piloting is
1) Constructs related to those in SQ by aims andeighth Phase 1 edition; this (Horak,research questions; some items identical e.g. on
1996) trialled in Phase 2 with 60
IELTS suitability, preparation course type,
teachers in sevenactivities, test module difficulty, for candidate
countries.
and teacher attitude comparisons.

Teacher
Questionnaire(TQ, see Appendix B)

Piloting and Trialling

Operationalisation

IELTS Study
Instrument

Figure 4.1 continued• Phase 1 convergent-divergent, multi-trait,multi-method validation data on draft IATM,
from four raters evaluating IELTS-oriented textbook, two a TOEFL prep book, two an upperintermediate book, one a different
further IELTS prep book.

• Trialled instrument data interpreted throughsame validating processes as Student
Questionnaire (SQ).

Validation

4 Impact study instrumentation

Classroom
Observation
Instruments(see Appendix D)

IELTS Study
Instrument• Phase 1 instrumentation: (Banerjee 1996),
with insights from Sri Lankan Impact Study(Wall, Alderson 1993), Nicaraguan, Impact
study (Luxon 1996), TOEFL washback study(Alderson, Hamp-Lyons 1996), IELTS
Specifications.
• pre-observation questionnaire re: observer,
teacher, venue, class, course, IELTS relationship, students, materials, lesson focus• observation schedule: six-column matrix
observation schedule for activities, timings,
teacher and students’ actions, interaction,
skills; inventories for selection re. text related
activities, homework• post-observation: teacher EL use, IELTS
impact, lesson feedback.

3) Teacher /evaluator/ textbook details• teacher evaluator background• textbook ID, type
• book organisation units• language features• enabling skills• question/tasking techniques• communicative opportunities• text types• text topics• authenticity• open-ended comment: listening, reading,writing, speaking• open-ended comment on whole book• open-ended comment on the relationshipbetween the book and test(s).

Operationalisation

Figure 4.1 continued• Phase 2 comparative use by
Green and Hawkey of COLT andsimpler draft alternative classroom observation analysis modelon the same two video-recorded
lessons collected for the IELTS
impact and Green’s own studies.

• Phase 1 instruments piloted on asmall scale with staff and students
at Lancaster University, and modified (six drafts) in line withextensive feedback from individuals with a research interest inclassroom observation.

Piloting and Trialling• Analyses in general agreement; two versions
reconciled and observation analysis model
agreed for use in Phase 3 of IELTS and
PL2000 studies.

• Phase 3 analysis, by two expert judges, of 52
test-related classroom lessons, including
IELTS and PL2000 Impact Study lessons, inthe course of the design of a Cambridge
ESOL classroom video database (see Chapter
5).

• Phase 1 instrument used by Hayes andRead (2004: 97–111) in their own IELTS
washback research.

• Phase 1 brainstorming, expert opinion andreview incl. by textbook writers.
• inter-rater discussion of IATM findings
included by Yue in thesis (1997).
• Phase 2 quantitative analysis, rater responses
given comparative item analysis, re. differences, inconsistencies, insights whether problems caused by interpretations of wording or
raters’ interpretation of textbook.
• Saville, Hawkey (2004: 81) note five kinds ofmodifications to draft IATM:
exclusion, items modified, items merged,
items moved, items supplemented.
• length of the Phase 1 version a major
consideration.

Validation

IELTS impact study instrument development and validation

69

70

Interviews and Focus
GroupsIELTS Study
Instrument

3) Informal, un- or semi-structured interviews
preferred (Farley Templeton 1996, Gunter
2001, Morrison 1998, Weiss 1998: 140–4).

2) Face-to-face data collection forms and methods
informed by survey, formal/structured interviews, focus group literature (Hawkey 2000,
2001, Weiss 1998: 140–144)

1) Close construct relationship with SQ, TQ; andreceiving institution questionnaire (RIQ
below).

2) Phase 3 instrument: more economic classroom
observation to collect IELTS-related lessons forobservation by video; categories: timing, level,
participants, macro-skills, mode, activity andcomment as key elements in classroom observation analysis for test washback.

• summary section• inter-observer discussion• teacher post-observation questionnaire: lesson
objectives, success, materials; IELTS impact,features; skills activities, likes, dislikes,
materials and handbook, activities, teacher
background• student post-observation questionnaire: background, IELTS knowledge, lesson feedback.

Operationalisation

Figure 4.1 continued• Phase 2: main interviewer/focus
group facilitator attends British
Council IELTS impact study survey sessions as observer

Piloting and Trialling• Phase 3 face-to-face data triangulated with
relevant questionnaire data.

• Sessions at five of the seven impact study UK
venues co-observed, at three video-recorded
and double reported.

Validation

4 Impact study instrumentation

Receiving Institution
Questionnire(RIQ, see Appendix E)

IELTS Study
Instrument

2) But early Phase 2 visits note staff often share
IELTS-related responsibilities: IELTS prep
course language teachers also IELTS administrators or admissions officers; some directors oflanguage schools or centres are also teachers.

1) Draft Phase 1 instruments foresee separate datacollection for IELTS administrators, admissions
officers, receiving institution subject teachers.

6) Probably easier to avoid or remedy suchproblems in less structured interviews or focus
groups; problem reactions can be seen as they
happen; in structured sessions responses may
be too controlled by the format to reveal
uncertainties or misunderstandings.

5) Weiss, on key face-to-face data collection
points:• relevance and specificity of questions• correct level and clarity of language• adequate definition of terms• use of categorial rather than just yes/no
responses where appropriate• avoidance of double negatives anddouble-barrel items• caution with second-hand information(1998: 140–142).

4) Interactions more like discussions; facilitatorwith list of key relevant topics but individual
or group may react to these freely though withprobes and follow-up questions to elicit further
information and views or to keep interactions
on track.

Operationalisation

Figure 4.1 continued• Trial data and review of Phase 1
administrators, admissions
officers, receiving institution
subject teachers’ questionnaires
used to develop single RIQ for
possible use in Phase 3 or later.

• Phase 2 trial data from 54 administrators in 21 countries.

Piloting and Trialling• Phase 3 data collected face-to-face andfrom secondary sources for impact study use
and for possible modifications of RIQ for
future use.

Validation

IELTS impact study instrument development and validation

71

72

IELTS Study
Instrument

6) Test information and other data from receiving
institution subject teachers deferred to a later
iteration of the study.

5) Focus areas in Phase 3:• IELTS general and skill module validity,
reliability and bias
• use of IELTS scores, cut-off band levels• administration and management issues(e.g. dates, venues, examiners, retakes,
security, cost).

4) Thus for Phase 3, one instrument developed,
the Questionnaire for Receiving Institutions(RIQ). But main data collection means at thisstage is face-to-face IELTS impact comment
on IELTS administrative matters.

3) Phase 3 team also note valuable receiving institution data collected by Cambridge ESOL outside strict limits of the study.

Operationalisation

Figure 4.1 continued
Piloting and Trialling

Validation

4 Impact study instrumentation

IELTS impact study instrument development and validation
In fact, as is indicated in Figure 4.2, the original 12 instruments developed inPhase 1, were rationalised to five, for use in Phase 3 of IELTS impact study(2002–2003).

Figure 4.2 Rationalisation of original data collection instruments
Phase 1 draft instruments
Instruments
1
2
3
4
5
6
7
8
9
10
11
12

Target participants
for classroom observers pre-, in- and post-observation
for teachers, post-classroom observation
for students, post-classroom observation
for teachers using student textbook materials
for students preparing for IELTS
for teachers preparing students for IELTS
for IELTS administratorsfor students who have taken IELTS
for admissions officers in receiving institutions
for receiving institution subject teachers
for teachers preparing students for academic study post-IELTS
for test takers’ background/characteristics

Phase 3 rationalised instruments
Instruments
SQ
TQ
IATM
CRO
RIQ

Target participants
test-taker students
IELTS prep teachers
IELTS prep teachers evaluating textbooks
classroom observation of IELTS prep class students and teachers
receiving institution administrators

A further increasingly prominent consideration at the rationalisation stage ofPhase 2 of the IELTS study was the scale of the demands to be made on
participants. Though the instruments had, as a result of the trial andrationalisation process, tended to be shortened (see IATM), they were still
extensive and demanding since so many dimensions needed to be included, as
relevant to the washback and impact research questions.
A revised Student Questionnaire (SQ in Figures 4.1 and 4.2), combined
items from three draft questionnaires from Phases 1 and 2 of the impact study:
the original questionnaire for students preparing for IELTS (instrument 5 in4.2), the one for students who have taken IELTS (8) and 12 on test takers’
background and characteristics. The new, composite instrument incorporated
Language Learning Questionnaire and Test Taker Background Questionnaire
constructs and validated items relevant to the IELTS impact study researchquestions. This Student Questionnaire became the main student data collection73

4 Impact study instrumentation
instrument for Phase 3 of the study. It was subject to a significant expert view
validation session with Tony Green, himself involved (Green 2003) at the timein his related doctoral research (see above). The option of one student
questionnaire instrument rather than three was appealing from a practical pointof view, of course. Hawkey’s August 2001 paper to UCLES EFL (summarisedin Figure 4.1) thus proposed a student questionnaire instrument incorporating
relevant coverage of language learner characteristics.
As Figures 4.1. and 4.2 indicate, the final version of the IELTS Teachers’
Questionnaire instrument (see Appendix B) differs much less from its pilot
version than does the Student Questionnaire.
The validation process for the Instrument for the Analysis of Textbook
Materials (IATM) is also detailed in Figure 4.1, the need to rationalise fromthe extremely long Phase 1 version being a priority. Work on the IATM
included rationalising to elicit summary, rather than over-specific, responses,
reducing numbers of sub-sections, rationalising most checklists throughmerging and deletion, but seeking more information on the intention, focus
and approaches of materials through enhanced open-ended comment sections.
As with the student and teacher questionnaires, however, the ‘uniformity ofitem format’ suggested by Kunnan (2000) in his Phase 2 instrument validation
report, was not attempted. Trialling and review had indicated that it was often
appropriate to include open-ended as well as closed items on key issues
because the open format was providing valuable elaborations of andtriangulation checks on the more quantitative questionnaire data.
The rationalising decision to use a single, simplified classroom observation
instrument (Q4) rather than the multi-part Phase 1 instrument was made forpurpose-specific Phase 3 reasons and after related validation operations (seeFigure 4.1). The original instrument proves insightful, well-constructed,
comprehensive and very useful for intensive research use as by Hayes andRead (see Figure 4.1 above and Banerjee 1996). However, it would be lesssuitable for a study such as IELTS impact study Phase 3, conducted mainly at
a distance from the participating centres. The purpose of the classroom
observations carried out in Phase 3 was, crucially, to triangulate observed dataon classroom activity and attitudes with learner and teacher perceptions ofsuch activities and attitudes as analysed from the IELTS candidate and IELTS
preparation class teacher questionnaires.
The rationalisation of Phase 1 instruments 7, 9 and 10 (for receiving
institution administrators, admissions officers and subject teachers,
respectively) was on practical grounds (see Figure 4.1) related to thedecision to limit Phase 3 of the IELTS study to receiving institution views,
collected mainly face-to-face but with the three instruments originating inPhase 1 ready for future use. A wide range of useful comment, spanning
both washback and impact matters, had emerged on receiving institution
visits made mainly to triangulate questionnaire data (see Figure 4.1 and74

PL2000 Impact Study instrument development and validation
Chapter 6). It also became clear on our own early visits to IELTS-related
institutions including universities and language schools, how valuable
receiving institution data were being collected by Cambridge ESOL
outside the strict limits of the study, for example by Lee Knapp, Cambridge
ESOL UK Development Manager (Knapp 2003). The breadth anddepth of such comment, on washback and impact matters, can be seenin Chapter 6, when the main messages from the face-to-face data arepresented. So it was decided that, given the already broad scope of thestudy, the pursuit of data from receiving institution subject teachers(instrument 10 in Figure 4.2) would be deferred to a later iteration of thestudy (see Chapter 8).
The final rationalisations of IELTS study instruments for use in Phase 3 ofthe project were made by the now IELTS impact study Project Co-ordinator,
Roger Hawkey, using all the trial data, proposals from a workshop at UCLES
led by Antony Kunnan in spring 1999, additional feedback from researchers,
including Green, working on related projects, and after consultations withUCLES senior management. It should be remembered that the original IELTS
impact study design had always seen IELTS impact research as continuous
and iterative, not necessarily involving the use of all instrumentation at onceor indeed for the same project.

PL2000 Impact Study instrument development
and validation: principles and processes
The PL2000 Impact Study was, as we have noted, an altogether smaller andmore compact study than Phase 3 of the study into IELTS impact. It
nevertheless used a range of data collection instruments (see Figure 3.3) aswell as collecting and recording face-to-face data through interviews, focus
groups and classroom observations. Figure 4.3, which summarises theoperationalisation, piloting and trialling and validation of the PL2000 Impact
Study instruments replicates Figure 4.1, which does this for the study of
IELTS impact.
The Cambridge ESOL PL2000 Impact Study did not have the same long
research group development and pre-validation phase as the IELTS study.
This was because the study was always recognised as more single-focus,
national rather than international, and as a study of a particular, time-boundproject rather than a high-stakes test with a long history and future. The mainprinciples and prompts for the development and validation of PL2000 Impact
Study instrumentation were thus:• the relevant literature• PL2000 documentation• the impact study project aims, research questions, target participants, and75

4 Impact study instrumentation

•
•
•

•

action plan as agreed by Cambridge ESOL management, headquarters andlocal, and endorsed by the Italian Ministry of Educationcase study school pilot visit interviews and observations (January to
March 2001)
regular draft study proposals, circulated among PL2000 Impact Study
project team members, with feedback reports exchanged
Cambridge ESOL team policy and planning meetings held at Cambridgeor in Italy, and liaison and approval meetings with appropriate Italian
Ministry of Education officials, including the PL2000 Director,
Dr Raffaele Sanzo
the use, where appropriate, of impact study approaches and datacollection items validated and used in other studies, including the study of
IELTS impact.

PL2000 Impact Study: operationalisation
The motivation for all the PL2000 Impact Study data collection was response
to our research questions (see Chapter 3). These, it will be recalled, concerned
PL2000 washback and impact on language teaching and learning: on students,
teachers, educational managers and other stakeholders; on textbooks,
language evaluation and assessment; on support for teacher in-service andtest-awareness programmes and resource centre development.
The study of IELTS impact, which, as we have seen, has been less a oneoff, short-duration project than the PL2000 Impact Study, and more anelement in a continuous, formative test consultation and validation
programme, had a broad action plan envisaged as comprising three phases (seeprevious). Each of these phases involved its own action plan (see Chapter 5,
for example, for plans guiding the Phase 2 instrument development).
The design and implementation of the PL2000 Impact Study, on the other
hand, were more concentrated and continuous, consisting of:• a relatively short planning, development and pilot phase, from initial
proposal in August 2000 to first case study school visits in October 2001
• a seven-month implementation phase (October 2001 to April 2002)• a further seven months for data analysis and final report publication (inJanuary 2003).
As with the aims of the study of IELTS impact, PL2000 Impact Study aims
had been agreed and specified in advance, as had the scope, main researchquestions, and appropriate methodological options (see Chapter 3). The study
constructs had, of course, to be operationalised, that is made measurable andincorporated into data collection instrumentation.
Given the narrower scope and scale of the PL2000 study, theinstrumentation was relatively simpler. The data collection instruments76

PL2000 Impact Study instrument development and validation
described in Figure 4.3 and beyond, with information on their design andvalidation were:
•
•
•
•
•

the school profile form
the student questionnaire (three versions)the teacher questionnaire
the classroom observation analysis form
the structured interview and focus group forms.

Since the design and timing of the PL2000 Impact Study precluded the kindof research group development used especially in Phase 1 of the IELTS study,
the ‘literature and related-project review’ (see above) was led by the study coordinator. He based the design of the attitudinal parts of the questionnaires (inthis case, the student and the teacher questionnaires) on the following key
principles (see Henerson, Lyons Morris, Fitz-Gibbon 1987):• determining specific information needed according to agreed PL2000
features, PL2000 study aims, research questions and construct
operationalisations• using response formats which combine open and closed responses
because of the potential advantage (revealed in IELTS Phase 2 instrument
trials) of obtaining measurable data enhanced and triangulated by opencomment• using closed response formats with scale categories or choices that require
relatively less subjective selections (e.g. frequently, never etc. as opposedto very good, good etc.)• becoming familiar with the frame of reference of the participants, i.e. in
the case of the PL2000 study, children, some young, for whom English
was a foreign language, and adjusting instrument item language
accordingly and/or allowing an explanatory and support role for L1
speakers, including the students’ teachers (see Chapters 5 and 7)• modifying and pre-validating instrument sections and items throughbrainstorming, team liaison and review meetings, and pilot-visit data
• keeping the instruments concerned as short and straightforward as
possible• agreeing the appearance of questionnaires taking account of factors suchas house style and user-friendliness in terms of:• appropriate introductory comments re. the purpose of, permissions forthe study (see Chapter 5) and anticipated response areas• minimised length, completion time, item numbers• clear and logical section and item sequencing and demarcation• succinct and appropriate wording• transparent and convenient response entry, including appropriate space
allocation to open-ended items.
77

4 Impact study instrumentation

PL2000 Impact Study prediction: brainstorming and review
As noted in Chapter 2, the prediction of the impacts of the PL2000 on theselected stakeholders and activities was used to help ensure relevant andworkable data collection means and items. Given that there was no permanentgroup of researchers working together on the development of the PL2000
Impact Study instruments, the brainstorming and review processes wereless intensive and regular than for the IELTS studies. However, brainstorming
and review, in the sense of the discussion of study objectives andinformation and the exchange of views on priorities and action for theirachievement, were attempted through regular Cambridge ESOL teammeetings, with agenda and reports, and attended by the UCLES EFL CEO, theHead of the Research and Validation Group, representatives from UCLESEFL Projects, Marketing and Events departments and the co-ordinating
consultant. All data collection instrumentation was also discussed with thethen UCLES Development Manager, Italy. The PL2000 Impact Study student,
teacher, and school profile instruments were also shared with and discussedwith the key contacts at the case study schools.
Figure 4.3 (pages 79–82) summarises main events in the operationalisation,piloting, trialling and validation of impact study instruments for the PL2000
Impact Study.
The use, performance and findings of the PL2000 Impact Studyquestionnaires are detailed in Chapters 5 and 7.

Other PL2000 Impact Study data
The baseline data on the students in the case study classes at the seven schools
were, in addition to their responses to the student questionnaire and their
videoed classroom action, supplemented by the use of the Oxford University
Press Quick Placement Test (QPT) (see Chapter 5), a test validated, before
publication, from performances by more than 5,000 students in 20 countries.
The test was used in all PL2000 Impact Study case study classes except thoseat elementary level, where it was considered that the pupils were too young toparticipate in providing data for the impact study, except through theobservation of their classes and interviews with their teachers and someparents.
The baseline data on the students in the middle and high school case study
classes at the seven schools were also supplemented by the use of theCambridge University Press Language Learning Questionnaire (LLQ) (seeabove). A selection of students completed the instrument in its fullcomputerised format. This instrument sought information, in a limited pilot
form, on their approach to learning English, profiled their language learning
attitudes and motivations, the time and effort they were willing to give to78

Student Questionnaire(SQ, see Appendix G)

1) Impact study longitudinal, collecting data early • Early 2001 pilot visits to schools
on in and towards end of school year, so three
confirm classroom activities on SQ
versions: one for first-time students, parallel(e.g. discussions with whole class,version for second-time end-of-year students,using computers) as
and third for (few) students present only for the
relevant for enquiry by interviews
second visit.
with PL2000 teachers, students andofficials and from observation of2) First student questionnaire mainly factual
PL2000 classes.
information i.e.:• background identity data
• home and school languages• years of English, foreign languages

1) Given the case study sample, instrument• Pre-validation of data areas through,designed to summarise the defining features of
Ministry PL2000 documentation
the seven case-study schools; informationand meetings, discussions.
entered on school profile forms to serve as context to responses to research questions.
• Data also from Jan to March 2001
recorded pilot visit interviews with
2) PL2000 Impact Study research questions indiPL2000-participant school teachers,cated form should collect objective backgroundheads, students and parents, local
and baseline information on:
PL2000 officials; also PL2000• numbers of students, classes
related information from Cambridge• teachers, English language teaching staff
ESOL Development Manager, Italy• PL2000 classes, hours per week
and contacts.
• other foreign languages• English tests used
• course books• resource centres.

School profile form
(see Appendix F)

Piloting and Trialling

Operationalisation

PL2000 Impact
Study Instrument• Activities in accordance with the ‘acquisition ofa pragmatic-communicative competence’ as in
Ministry PL2000 aims (e.g. working with apartner).
• Validated items from student / teacher
questionnaires as used in• Green (2003) (e.g. listening and taking notes)• Banerjee (1996) draft IELTS impact study
Phase 1 classroom observation instrument(e.g. students read the questions and guess
what listening passage is about)and• Single form to be completed one per school;
completers invited to query or discuss items, but
no problem encountered with completion orpresentation of data from form.

• School profile form, almost completely factseeking; no statistical validation required.

• Process also suggested form should include two
open-ended, more subjective questions: one on
PL student use of English outside and beyond
school, and one a general-opinion item on the
differences the PL2000 seemed to be making to
the schools concerned.

• Consultation process confirmed relevance ofschool profile information areas.

Validation

Figure 4.3 Summary of main operationalisation, piloting andtrialling and post-draft validation exercises for PL2000 Impact Study instruments

Other PL2000 Impact Study data

79

80

3) 2 sections of second-time student questionnaireseek attitudinal data through systematic use of
closed-item formats; section on student perceptions on relative frequencies of classroom
activities e.g.:• discussion with a partner; vocabulary
exercises, or taking practice exams• five multi-choice items on: perceptions oftarget language improvement over theyear (2 items), study time outside class,attitudes to external exams, main reason
for learning English.
• these areas related directly to researchquestions (Chapter 3) and key to any
evaluation of PL2000 washback /impact.

• exam history and intentions• overseas English language experience• English language self-access use• career intentions• self-access English language use item invites
some quantification (never, almost never,occasionally, often) for each activity, e.g.
reading English language newspapers• one open-ended question, i.e. ‘What are youropinions about learning English?’Operationalisation

Teacher Questionnaire 1) Case study school English teachers, key stake(TQ, see Appendix H)
holders in PL2000 and impact study, especiallyteachers of students responding to SQ (above)

PL2000 Impact
Study Instrument

Figure 4.3 continued• Early 2001 pilot visits to schools
confirm classroom activities on TQ
as relevant for enquiry throughinterviews with PL2000 teachers,
students and officials, and fromobservation of PL2000 classes.

Piloting and Trialling• SQ and TQ shared items facilitate triangulation;even if perceptions differed of what was happening on particular courses, similar overall patterns of activity should also be perceptible
across learner and teacher data (see Chapter 7).

• related items in validated IELTS impact
study student, teacher and IATM
questionnaires (see previous)

Validation

4 Impact study instrumentation

Classroom
Observation
Instruments(see Appendix D)

PL2000 Impact
Study Instrument

2) So, for PL2000 study, use of same instrument
to analyse 20 closely observed, video-recordedclassroom language lessons in case study
schools in terms of episodes, timings, activities,
participation, materials and comment; theseanalyses offered most direct evidence on
Progetto pedagogy and ethos, collected on two
visits seven months apart.

1) PL2000 Impact Study classroom observation
instrument shared objectives and developmentwith similar IELTS study instrument (see 4.1);
both seeking accurate analysis of what really
happened in classroom for insights into washback from a language education reform projectand high-stakes tests.

4) Second attitudinal section in TQ explores
teachers’ views of successful or otherwiseachievement of PL2000 stated objectives.
• Shared validation with IELTS study; analysis,by two expert judges, of 52 test-related classroom lessons, including from IELTS and
PL2000 studies.
• Analyses in general agreement; final observation analysis model agreed (see Appendix D).

• Decision to use, in both impact
studies, version of Hawkey’s classroom observation analysis framework (see Appendix D).

• The analyses of video-recordings of PL2000
classroom lessons (see below) would complete
the data triangle here.

Validation• See 4.1, trial comparative use by
Green and Hawkey of COLT andsimpler draft alternative classroom
observation model.

• All PL2000 teachers completing
TQ also interviewed and consulted
on the form.

2) TQ seeks factual participant background, thenscale responses on same list of classroom
activity frequency as on SQ. This permits
analyses of perceived PL2000 lesson content
and activities, and comparisons between learner and teacher perceptions of same lessons.

3) Two open-ended items allow further comparisons between student and teacher attitudes toProgetto; one seeking teacher views on how
PL2000 impacted on their students; second
paralleling student questionnaire item onchanges over year.

Piloting and Trialling

Operationalisation

Figure 4.3 continued

Other PL2000 Impact Study data

81

82

3) Key focus group and interview data collection
validation points noted for moderating/interviewing:• working mainly to formalised schedule of
topic headings to be addressed by participants• keeping discussion on track but without
inhibiting flow of ideas or emergence of
unanticipated views• ensuring all group members contributewith no one participant’s opinions dominating• avoiding overt support for or criticism ofviews• resolution of issues or achievement of goals• N.B. real aim of data-collecting interviews
and focus groups being to gather as much
valid information as possible.

1) Theory and practice of interviews and focus• As for IELTS study: maingroups surveyed (Farley Templeton 1996,
interviewer/focus group facilitator
Gunter 2001, Morrison 1998); account taken of
attends British Council IELTS
relevant experience in IELTS study.
impact study survey sessions asobserver.
2) As with IELTS study, key questions are:• generalisation• social nature of focus groups• facilitator role and actions.

Interviews and Focus
Groups

Piloting and Trialling

Operationalisation

PL2000 Impact
Study Instrument

Figure 4.3 continued• All sessions at all case study schools withobserver present, often as interpreter.
• All sessions but one video recorded; reportsof analyses of recorded sessions verified byobserver/interpreter.
• Face-to-face data triangulated with relevant
questionnaire data.

Validation

4 Impact study instrumentation

Other PL2000 Impact Study data
learning the language, the strategies they used and how they organised theirlearning (see Chapter 5 for further details of the management of the LLQ in
the PL2000 study).
Additional PL2000 Impact Study insights on pedagogy, materials, andmedia were drawn from information provided by some of the case study
school teachers writing direct to the study co-ordinator, and gained fromUCLES EFL : Bell International PL2000 teacher essay competition entries in
response to the question: ‘What the Progetto Lingue 2000 means to me as ateacher’. These had always been seen, by Dr Peter Hargreaves, then CEO of
Cambridge ESOL, as potentially useful triangulation data for the study.
It is possible that the PL2000 Impact Study, in which a relatively small
number of students and teachers were in direct contact with researchers, their
video camcorders and microphones, computerised tests and theirquestionnaires, was vulnerable to the Hawthorne Effect. This is defined by
Weiss as a ‘change in the outcome measure that is the result of the attention
that participants receive during intervention rather than a result of the program
under study’ (1998:331). The possibility of such an effect cannot, of course,
be ruled out. However, participants in the PL2000 Impact Study, like those in
the study of IELTS impact, were not actually subject to a treatment, in thesense of a continuous programme that they were aware other similar
participants were not receiving. Rather, they were subject to two or three datacollection contacts during an academic year, none of which, with the possible
exception of the QPT with its on-screen learner profile, actually left thestudents with anything their peers outside the impact study did not receive.
Given the relatively fleeting involvements of PL2000 participants (and theeven fewer face-to-face contacts of most IELTS impact participants)
Hawthorne Effects seem fairly unlikely to have played a part in the impact
studies.
There may well have been Hawthorne Effects from the PL2000 itself, ofcourse, where students at the same schools could be, for a whole school year,
attending courses under the project, when their friends were not. In the case of
the Progetto, the Ministry’s aim was, in a sense, to create a Hawthorne Effectof successful language learning and use.
The problem of ascertaining that classroom events and the English
language performance of learners are actually evidence of the washback oftests or curricular innovation rather than the result of other factors remains, ofcourse. It is the expectation, however, that research such as the PL2000 Impact
Study, with its inclusion of observational as well as survey data and its
triangulation of evidence, will succeed in identifying likely signs of washback
and impact (see Chapters 5 and 7). Chapter 5 will explain the approaches taken
to try to collect valid data. Chapter 7 will summarise and interpret the main
findings, and describe further attempts to come to valid impact conclusions.

83

4 Impact study instrumentation
Following the enquiry into impact research aims, questions, action plan,
research designs, and hypotheses in Chapter 3, this chapter has described thesecond step in the two studies, of the IELTS and PL2000, with a particularfocus on the development and validation of data collection instruments.
General and instrument-specific points have been made about the main datacollection tools of both studies, the validation of questionnaires and similar
instruments emerging as just as important as, though in many cases
procedurally different from, the validation of language tests.
The next logical step, for Chapter 5, is an investigation, exemplified as everwith reference to our two impact studies, of the project implementation steps.
This will cover data collection, bases, analyses, and validation.

84

5Collecting and managing
the data

Chapter 4 has traced the development, validation and production of impact
study instrumentation for studies of IELTS and PL2000 impact, in their own
right, and as examples of the processes that may be expected in the study ofthe language learning and testing aspects of educational impact. Chapter 5
now examines the collection, management and analysis of impact data. In
Chapters 6 and 7, the main findings of the two studies will be presented, while
Chapter 8 draws conclusions on the study of impact, seeks lessons to belearned, looks forward to future related research and speculates on future
developments in models of impact research.

IELTS impact pre-survey
Following the development of pilot data collection instruments in Phase 1 andtheir validation and rationalisation in Phase 2, steps were taken to ensure thatthe implementation of Phase 3 of IELTS impact study collected data from anappropriate sample of candidates and preparation course teachers. To this end,
it was felt necessary to update the information on IELTS centres. In May
2001, therefore, a pre-survey questionnaire was sent to over 300 University,
British Council, IDP Education Australia and other test venues worldwide (seeFigure 3.2 in Chapter 3 above). The survey, which achieved a high response
rate of over 65% from centres in 41 countries, ascertained up-to-date objective
baseline data such as the following:
•
•
•
•
•

the language tests for which the centres were running courses
the annual numbers, durations and dates of such courses
the numbers and nationalities of students
teacher strength
the textbooks and other materials used.

All these data were used to help select IELTS centres for the main datacollecting phase of the study. The numbers foreseen for Phase 3 were a casestudy sample of around 40 centres, representing IELTS test taker nationality
and language populations (see Chapter 6).
Points of interest from the pre-survey data, analysed using straightforward
descriptive statistics (frequencies, percentages, means), are summarised here.

85

5 Collecting and managing the data
Thirty per cent of the pre-survey returns were from university centres, 19%from British Council and IDP Education Australia centres, and 51% from othertypes of language teaching institutions. IELTS was, unsurprisingly given thecriteria for the selection of centres for the pre-survey and the objectives of thestudy, the test for which preparation courses were most frequently run at thesecentres (83% of the cases). However, courses were also being offered for the
Test of English as a Foreign Language (TOEFL) at 34 of the 203 centres offering
preparation for tests in addition to IELTS, and for the Test of English forInternational Communication (TOEIC) at 18. Forty-five per cent of the centres
were providing preparation courses for one or more Cambridge ESOL Main
Suite exams (see Chapter 3).
A further pre-survey question asked about the length of IELTS preparation
courses at the centres concerned. Figure 5.1 summarises the responses,
indicating the highest proportion of courses at between nine and 12 weeks inlength, but with significant percentages half that length or less.
Figure 5.1 Lengths of IELTS preparation courses at IELTS impact
study pre-survey centres
Weeks

Centres

1-2
3-5
6-8
9-12
13-16
Longer

27
35
26
73
10
22

From our contacts with the centres concerned we also gathered that theconcept of ‘IELTS’ courses might need clarifying as the preparation for thetest sometimes figured as part of other programmes, for example, ‘General
English’ or English for Academic Purposes (EAP) courses. This clarification
was made (see Chapter 6) in the questions asked of IELTS preparation
students in the questionnaire for Phase 3 of the study itself.
Figure 5.2 indicates that at the pre-survey centres most IELTS preparation
classes had relatively small student numbers, 6–10, and 11–15 being the mostfrequent class sizes reported.
Figure 5.2 Responses from 177 centres on approximate numbers ofstudents on each IELTS preparation course
Number of
students

86

1-5
6-10
11-15
16-20
More

Number ofcentres
21
65
47
23
21

Instrument delivery and data return
Final key figures from our pre-survey data, suggesting that the student
nationality groupings were fairly representative of the IELTS test takerpopulation, are given in Figure 5.3.
Figure 5.3 Responses from 193 centres on the main nationality groups
on their course(s)Regions

Centres claiming as
main nationality groups’

East and South East Asia
EuropeMid-East
Latin America
S AsiaAfrica

125
65
23
18
10
3

Centres were, of course, quite likely to name more than one nationality groupas ‘main’ participants on their courses.
The IELTS impact study pre-survey also provided a useful list of IELTS
preparation course textbooks in use. This informed the study’s investigation,
through the Instrument for the Analysis of Textbook Materials (IATM, see
Chapter 4), into teachers’ choice and opinions of such books.
Above all, the contacts with centres made through the pre-survey helped thestudy team to select appropriate centres for Phase 3, based on the factual data
provided (for the pre-survey was requesting only objective facts from thecentres, not opinions or attitudes).

Instrument delivery and data return
There was, at this stage of Phase 3 of the IELTS study, considerable
discussion of two important practical points in the management of
impact research projects, the decisions on which would affect both ourstudies. The first discussion point was the means of delivery of the datacollection instruments, the second the types of direct contact neededwith participants.

Phase 3 of the IELTS impact study
There was extensive debate within the Phase 3 IELTS impact study team, with
Cambridge ESOL senior management, with potential participants and withoutside consultancy contacts on how the impact study questionnaires should
be delivered, by what means they should be completed and how they should87

5 Collecting and managing the data
be returned for analysis. The pre-survey was administered using astraightforward two-page questionnaire, sent by post to the 313 centres
concerned, fronted by an explanatory invitation to participate, and pursued, if
necessary, with e-mail reminders. 43% of the returns were received by fax,
and the response rate, as stated above, was high, at 65%.
Frankfort-Nachmias and Nachmias (1996) refer to a 20–50% response rate
as typical of conventional mail surveys. However, there are indications thatelectronic questionnaires, that is instruments sent and returned by e-mail andwith responses entered electronically, can achieve higher response rates.
Walsh et al (1992) claimed a 76% response rate with a randomly selected
sample and 96% with a self-selected sample, although their survey was about
computer networks, a fact that, 14 years ago at least, would appear likely toinvite a high computerised response rate. Anderson and Gansneder (1995)
report a 68% response to their e-mail survey, with 76% of responses returned
via e-mail and 24% conventionally. Mehta and Sivadas (1995) compared aconventional mailed questionnaire with an e-mail equivalent, finding that theunsolicited mail questionnaire achieved a 45% rate of return, compared withthe 40% response to its e-mail equivalent. The e-mail response rate increased
to 63%, however, if an initial e-mail was sent requesting participation in the
study. The approach to initial invitations to participate, reminders and thankyou contacts in the studies of IELTS and PL2000 impact are discussed in thenext section of this chapter.
The deliberations on how to deliver IELTS impact study questionnaires andreceive the return data raised a range of issues before decision. These included
the following potential advantages and disadvantages.
The Internet’s ‘easy access to world-wide samples’, and potential relative‘unobtrusiveness’ to respondents are clear advantages (Selwyn and Robson
1998:1). However, the use of e-mail as a research tool may still be somewhat
constrained by the bias and limits of its user population in terms of variables
such as age, socio-economics or nationality. In 2001/2, of course, the periodof IELTS impact study Phase 3, with e-mail use many times more common
than it was in 1997, the danger was of information overload and that research
via e-mail was already running the risk of becoming marginalised as a form ofelectronic ‘junk mail’. In more recent research (Archer and Kendall 2003),
questionnaire topic, length and sending institution were found to be the main
factors that affected response rates. Furthermore, although ‘almost all’ schools
in the NFER survey of head teachers’ views on participating in research ‘hadinternet access and email, the number indicating that they would prefer tocomplete electronic questionnaires rather than paper-based versions wasconsiderably lower than this’ (ibid:1).
However, we were, in the main data collection operation of Phase 3,
contacting IELTS candidates or potential candidates through the centres

88Instrument delivery and data return
where they were studying. Since not all such centres provided e-mailaddresses for their students, the IELTS impact study team was not in as
advantageous a position as researchers contacting their sample populations
through independent or official institutional e-mail addresses. If this had been
the case, the convenience and reduced costs of e-mail contacts would havebeen a stronger factor in decisions on how to deliver and receive thequestionnaires.
E-mail questionnaire designs also offer the advantage of data that aretransferable in electronic form to the databases set up for the analyses
required. Miller explains:
Questionnaires may be posted online and submitted electronically. Thiscan significantly reduce the time involved in administering and analysing
questionnaires. Answers submitted on a spreadsheet can be entered on toa summary sheet and summary scores computed much more quickly when
the spreadsheet is submitted electronically (2002:21).

We see below the actual methods used in the analysis of IELTS impact study
data, applying spreadsheet operations but only after the data had been enteredindirectly from the completed questionnaires.
The decision finally to use hard copy instruments was influenced by thefollowing factors:• the global coverage of Phase 3 of IELTS impact study using centres withvarying IT facilities and equipment• the comprehensive nature of the data collection instruments involved,
with their significant number of open-ended items• the success of hard copy instrumentation in both the Phase 2 trials and thepre-survey.
Further such studies are nevertheless likely to use electronic instrument
delivery and collection modes, especially as the relevant technologies improveand spread so quickly.
The preliminary contact letters or e-mails to 72 centres selected from thepre-survey sample and others contacted since, offered the centre authorities
the option of receiving, by post or special delivery, either the number ofstudent, teacher and textbook assessment instruments they wished to use or
single copies of the instruments concerned, which they could copy as required.
In the event, the study obtained a satisfactory number of questionnaires
returned by mail (see Chapter 6 for details of the 572 student, 83 teacher and
45 textbook evaluator participants), in proportions reasonably close to the
IELTS test taker and preparation course teacher populations. The instruments
were, in general, comprehensively and conscientiously completed.

89

5 Collecting and managing the data

Instrument delivery and data return
PL2000 Impact Study
The PL2000 Impact Study, where similar discussions of the instrument
delivery and return options took place, used a combination of electronic andhard copy data collection. The main instruments concerned are re-listed here,
along with justification for decisions on how they should be administered:
1. School profile pro-forma (see Appendix F): This form, it will be recalledfrom Chapter 4, was to collect background information on student numbers,
teachers, PL2000 classes, external English language certification, course
books, foreign language resource centres, PL2000 experience. It seemed
clear that this form, one only per case study school, should be completed bysomeone in authority, with access to the information concerned. The data
were mainly documentary, to be reproduced in the PL2000 report as
background information rather than analysable data. Hard copy format was
the preferred option when the schools were consulted in advance. A singlepage form was thus used and seemed to suit the school authorities concerned.
The form was completed promptly, and received no negative comment.
2. Student questionnaires (in three parallel versions, see Figure 4.3 in Chapter
4 and Appendix G) were administered through the English language
teachers of the case study classes in hard copy format. The completed forms
were returned to Cambridge ESOL by hand during Impact Study venue
visits, or later by post, especially in cases where class English teachers
wished to arrange discussions on the questionnaire with their students (seeChapter 7).
3. PL2000 Impact Study participant teachers completed the hard copy
Questionnaires for Teachers (see Appendix H) during the second round of
school visits in April 2002.
4. PL2000 classroom video-recordings were made by camcorder anddirectional microphone on the February 2001 pilot visit by Cambridge
ESOL Video Unit specialists, who trained the Cambridge ESOL country
manager and the Impact Study co-ordinator to make the subsequent
recordings on the October 2001 and March 2002 visits.
5. The Oxford University Press Quick Placement Test (QPT), designed forlearners of all levels and all ages, was selected for use with the case study
school students near the beginning of their courses as it would give an
immediate broad assessment of their English language proficiency in thecase study classes at the seven schools. The computer-based adaptive
version of the test was selected. The computer estimates the level of the test
taker on the basis of his/her responses and selects from its bank items of theappropriate difficulty for that level. Because of this, each response90Permissions, confidentiality and accountability
contributes a maximum amount of information and the test can thus beshorter than equivalent paper and pen tests, without sacrificing reliability.
QPT results are generated immediately. On the PL2000 study, the test wassent to and administered by the case study schools on a networked system,
following detailed instructions from Cambridge ESOL Projects Office.
6. The UCLES Questionnaire (LLQ, see Chapter 4) was administered in its
Italian language version through CD-ROM delivered to selected case study
schools with full instructions for installation on the school computerlaboratory systems for student self-completion. Participants keyed in or
ticked electronically information about their attitudes and approaches tolearning English, and were able to view an immediate on-screen profile oftheir approach to learning a foreign language in terms of:• attitudes, concerns, motivations regarding English language• willingness to commit time and effort to learning the language• language learning strategies.
The LLQ was intended to take no more than 40 minutes to complete andhad an on-screen help option. Further details of arrangements andpermissions for LLQ use appear in the next section.
7. As mentioned in Chapter 4, additional PL2000 impact data were drawn
from competitor submissions to the UCLES EFL : Bell International
PL2000 teacher essay competition, arranged at the same time as thePL2000 Impact Study.
The PL2000 Impact Study combined hard copy and electronic instrument
delivery and data return. Decisions on which to use were made, as with the
IELTS study, on practical grounds, after discussion by the project team in
consultation with participating centres.

Permissions, confidentiality and accountability
Like all researchers seeking information from people, impact evaluators need
to be aware throughout the research process of the ethical implications of theirstudy for the participants. Tindall identifies the following ‘closely intertwined
ethical issues’ (2001:153): informed consent, protection of participants,
confidentiality and anonymity, and accountability. Weiss (1998:95) adds
reciprocity as a further ethical issue.
Informed consent from participants presupposes, of course, that they are
fully aware of the purposes and approaches of the research concerned and that
their initial consent to participate includes the right to withdraw at any time,even retrospectively. Participant protection is achieved mainly through theconfidentiality and anonymity of participant information.

91

5 Collecting and managing the data

IELTS impact study
The student questionnaire used in Phase 3 of the study of IELTS impact
attempted to obtain the informed consent of test candidates by:• telling them of the research purpose and accountability (in the sense ofreference to the sponsors to whom the researchers are answerable, and theresults they are expected to achieve)• defining the context, nature and potential participants in the study• assuring participants of the confidentiality of their responses• thanking them for their time and co-operation• inviting their consent to participate under conditions of:– anonymity– right to refuse or withdraw participation, and– guarantee of restriction to the research team of access to information
given.
The full IELTS study Student Questionnaire introduction and permission slip
was as follows (see also Appendix A):
Dear Participant,As part of the continuing programme to update and refine its International
English Language Testing System (IELTS), the University of Cambridge
Local Examinations Syndicate (UCLES) is conducting a study of the
impact of the test. We are contacting students, candidates, English
teachers and university administrators for information and comment.
Your responses to this questionnaire will be treated in confidence, andonly used for the stated purposes of the study.
Thank you very much for your time and co-operation. We should also be
grateful if you would complete and sign the consent note below.
Yours sincerely
Nick Saville
EFL Director
Research and Validation
University of Cambridge Local Examinations Syndicate
English as a Foreign Language
Cambridge
England92

Permissions, confidentiality and accountabilityYour consent to participate in the UCLES IELTS impact study
I understand that:•the purpose of the study is to collect and analyse information fromthose familiar with the IELTS•

my name will not appear in any project publication•

the information I give, but not my name, may be quoted•

I am free to refuse to participate in the study and may withdraw at any
time•

my completed questionnaire is for the study team only; it will not beshown to anyone not connected with the UCLES study.

Signature:

Date:

The teacher, textbook evaluation and administrator questionnaire permissions
took a similar form.

PL2000 Impact Study
The letter below to case study school heads from the PL2000 Impact Study coordinator characterises the question of participant permissions for the study,
which, unlike the study for IELTS, involved personal contact with the study
team at all venues. It noted that the careful, though not necessarily formalised,
approach to permissions (to enter schools, administer questionnaires etc.)
would be maintained, with the Cambridge ESOL Development Manager to be
responsible first-instance. The fact that the Impact Study was approved by the
PL2000 Co-ordinator at the Ministry of Education in Italy meant that generalprinciples of consent, confidentiality and accountability had already been
established. The following mid-project letter from the Cambridge ESOL
PL2000 Impact Study co-ordinator to the head of one of the case study schools
exemplifies the way in which key project contacts were maintained,arrangements pursued, purposes confirmed, and permissions sought.

93

5 Collecting and managing the data

18 March 02
Dear Prof xxxxxxx
As co-ordinator of the Cambridge Progetto Lingue 2000 Impact
Study, I should like to request your approval for thefollow-up visit to your school by the Cambridge ESOL
Development Manager for Italy and myself, as discussed when
we were at xxxxxx School last October. May I suggest Tuesday
April 15, 2002 for our visit. We should be in [two casestudy school towns] for one or two days and intend, if thedate is convenient, to be working in [town 1] on Monday
April 15, and at the xxxxx school on Tuesday morning.
We should like to achieve the following objectives during
the time we are with you:1) the video-recording of :•an interview with yourself on latest developments re thePL2000•a lesson in a PL2000 English class which would suitably
illustrate some of the effects of the Progetto•

interviews before or after the lesson with teacher(s)and, perhaps, some of the students concerned•a meeting of some of your English and other foreign
language teachers.

We would welcome your assistance in ensuring that
permissions to video and talk to students and staff havebeen obtained.
2) discussions with you and appropriate English language
staff of the use at your school of the Cambridge Language
Learners Questionnaire (LLQ) and the Oxford Quick
Placement Test (QPT).
We would, of course, makedisruption during the visit.

every

efforttominimise

I do hope the proposed visit on 15 April is acceptable toyou. Your agreement represents a further step in the
excellent support you have already given to the PL2000 and
the Cambridge Impact Study. The classroom video, interview
and questionnaire data we collected at xxxx school lasttime have proved very useful indeed for our Study.
I trust that the academic year has been going well for yourschool and you, and look forward to hearing from you assoon as possible on our planned visit so that we can takethe necessary action on arrangements.
Yours sincerely
Dr Roger Hawkey
Co-ordinator, PL2000 Impact Study

94Permissions, confidentiality and accountability
When an independently produced data collection instrument is being used, as
in the case, for example, of the LLQ in the PL2000 Study, agreements would
be signed covering both providers and users. Letters to PL2000 Impact Study
schools in January 2002, for example, included:• a statement that the agreement covered terms and conditions of use of the
LLQ computer-based software for the purposes of the Cambridge PL2000
Impact Study in Italy• Cambridge ESOL agreement to provide one set of LLQ materials to eachof the Italian centres identified in the project (including LLQ CD-ROMS
and Italian installation manuals)• agreement that the school, as one of the schools participating in the
Impact Study project, would install the LLQ software on its own
equipment• a statement that the licensed LLQ software and documentation containconfidential and proprietary information of the licensor and that all
copyright trade marks and other intellectual property rights in the licensed
software and the documentation are the exclusive property of Cambridge
ESOL• agreement that material would be used for the PL2000 project alone andthat it would remain the property of UCLES EFL throughout• agreement that the LLQ would be completed by students in the selectedclasses in the PL2000 Impact Study by the middle of March 2002, after
which all material would be returned to Cambridge ESOL
• agreement that users of the software would not copy, modify or merge thewhole or any part of the licensed software or documentation; nor use
them on behalf of or make them available to any third party.
In cases such as this, permissions and confidentiality are clearly a two-way
process. There is little doubt that issues of informed consent, participant
protection, confidentiality, anonymity, and accountability in research willcontinue to grow in significance and require increased formalisation. In theirtemplate for agreements to permit the use of examination data for academic
research purposes, for example, Cambridge ESOL Examinations seeks
signature to a statement of the following:
•
•
•
•
•
•
•

purpose, use, request context of datadetailed description of data requested
specification and limitations on use of the data provided
permanent ownership of data
date of return of data to ownerassurance of safe-keeping of data on loan
guarantee of restriction of use of data by signing party only unless withprior permission of owner95

5 Collecting and managing the data
• agreement by temporary user of the data to offer to owner for priorcomment all papers or forms of report using data, with owner right torefuse permission to publish• data user undertaking to maintain anonymity and confidentiality of data
participants unless agreed otherwise in writing with Cambridge ESOL.

Reciprocity
Both the IELTS and PL2000 studies made significant demands on people’s
time and information. With the Italian project, the case study schools tended
to be establishments where Cambridge ESOL had already made beneficial
contacts, through, for example, teacher-support programmes. With Phase 3 ofthe IELTS impact study, however, carried out mainly at a greater distance, it
was felt that the collaboration of participating centres should be reciprocatedin some way. It was thus agreed that the letters inviting centres to participate
would reiterate the data provision tasks they were being invited to perform,
then offer the project sponsor’s tokens of gratitude in the following words:
We know that this is asking rather a lot. However, we intend to support
your participation in the IELTS impact study as much as we can with any
advice needed, by providing all the questionnaires you require and bysending you the results of the analysis of the data you submit. We are alsooffering a book award to every teacher completing the textbook evaluationquestionnaire.

The promised data report was individualised where appropriate for each centre
and sent by e-mail. The book award was in the form of an international booktoken sent by post. The IELTS impact study team was pleased to receive
acknowledgements from centres for both tokens of gratitude.

Data analysis and management
IELTS
The means of analysis used as appropriate to the validation of Impact Study
data collection instruments have been discussed in some detail in Chapter 4.
The data management and analysis system for Phase 3 of the IELTS study was
designed by Richard Turner of the Cambridge ESOL Research and Validation
Group, in consultation with the co-ordinator of the study and author of thisbook, and with assistance from Research and Validation Group staff. The
system covered the analysis of the data from the questionnaires, closed andopen-ended items, the interviews and focus groups, and the classroom
observations.

96

Data analysis and management
Since Phase 3 of the study of IELTS impact was a fairly long-term project,with data-collection through the questionnaires and face-to-face modes
spanning several months, it was important to have prepared a data
management system which would permit the entry and monitoring of data toidentify early trends and possible problem areas. The Microsoft Access 97
Database for Windows was used for this purpose, with its system of:• tables, for data storage• queries, a function permitting the user to extract and draw together
specific parts of the stored data (e.g. the number of female IELTS
candidates with Cantonese as their first language, the highest frequency
first language, the relationship between candidates’ expected and required
IELTS band scores)• forms, for the viewing, editing, inputting and control of data.
To aid data evaluation and reporting, a link was developed between
Microsoft Access and the Microsoft Excel spreadsheet-based programme with
its strong chart-creation facilities. The link enabled a query posed in Access on
the most recent data held in the database, to be pulled by Excel into a
spreadsheet table for chart display. Once set up, the whole data action chain
could be updated with a touch of the Excel ‘refresh’ button.
It was also possible to link the more powerful quantitative research package
SPSS (Statistical Package for Social Sciences) once the limits of Access werereached. The SPSS package had already been used in Phase 2 of the IELTS
study (see Chapter 4), although the data analysis operations remain, as may beseen from the data presentations in Chapter 6, mainly at the descriptive
statistical level. It will be recalled from Chapter 2 that the IELTS study was
categorised as a mainly descriptive study towards the qualitative side of thequantitative/qualitative continuum, the inferential statistics being reserved forthe validation of some data collection instruments (see Chapter 4).
Open-ended data from the IELTS impact study questionnaires were also
entered into the Access system and submitted to keyword analyses. Semistructured interviews and focus groups were summarised through manualnote-taking, in some cases double-checked across versions taken by tworecorders.
To store and enable further analyses of the IELTS and PL2000 videorecordings, a dynamic Access-platform video-data management system wasdesigned by Richard Turner, with the assistance of Roger Hawkey and theCambridge ESOL Video Unit (see Hawkey, Thompson and Turner
forthcoming). This system permits the electronic retrieval of classroom video
clips, which have been analysed according to 11 fields: project, date, location,
data source, time code, participation, level, mode, activities, quality andcomment. The now completed, but expandable, Impact Study research video97

5 Collecting and managing the data
database covers 55 lessons, from the IELTS and PL2000 studies and theFlorence Language Learning Gain Project (see Chapter 8). Twenty videoedinterviews and focus groups from the IELTS and PL2000 studies are also inthe database.

PL2000
The PL2000 Impact Study was, as has already been indicated, smaller in scaleand handling less complex data. The school profile form, student and teacher
questionnaires and structured interviews and focus group data were analysedmanually and summarised using Microsoft Excel data analysis andpresentation operations (see Chapter 7). PL2000 study videoed semistructured interview and focus group data were summarised in note form,
cross-checked with observer (or interpreter) and cited in the PL2000 Impact
Study report (see Chapter 7) with appropriate attribution.
This chapter has detailed some of the key features of IELTS and PL2000
Impact Study data collection, management and analysis approaches. The twostudies demonstrate both hard-copy and electronic data collection, manualand computer-programmed data analysis and management systems. The
importance of introductory, follow-up, permission-seeking or granting andacknowledging correspondence with participants has also been recognised.
Chapters 6 and 7 now present some of the main findings of the IELTS and
PL2000 studies respectively.

98

6

The IELTS impact study:
main messages

In Chapters 6 and 7 we summarise some of the main findings of our two
studies. This is intended both to present key messages for stakeholders andto illustrate how these messages emerged from the design, development,
implementation and analysis stages of the two studies as described in Chapters
1–5. This chapter will focus on Phase 3 of the study of IELTS impact, Chapter
7 on the PL2000 Impact Study.
Phase 3 of the IELTS study aimed, it will be recalled from Chapters 2 and
3, to respond to a series of washback and impact research questions seeking
data on:• profiles of IELTS test candidates and teachers preparing candidates totake the IELTS test
• the washback of the test on courses preparing candidates to take it• the impact of IELTS on participants who have taken the test
• the impact of the IELTS test on institutions receiving IELTS candidates.
The chapter will first examine evidence from the study towards answers to theresearch questions on candidate participant profiles, then on the IELTS
preparation courses and their teachers. Evidence will next be presented onparticipant perceptions of IELTS test fairness, likes and dislikes, pressures andmotivation, test module difficulty and validity. Finally, mainly through ananalysis of the study’s face-to-face data, questions of receiving institution use
of IELTS band scores and issues of test administration will be considered.

Impact study candidate profile
In pursuit of profiles of pre- and post-test IELTS candidates, a case study
sample of IELTS centres representative of the IELTS nationality population
was selected using data from the pre-survey (see Chapter 5). IELTS
candidates and teachers at these centres were invited, through the individuals
already nominated as key contacts on the basis of pre-survey responses, torespond using the appropriate data collection instruments (also see Chapter 5).
The chart in Figure 6.1 summarises the IELTS impact study Phase 3 candidate
population in terms of regional background compared with the full 2002
IELTS candidature, indicating a reasonable match between the two.

99

6 The IELTS impact study: main messages
Of course, this candidature is changing constantly. The 2003 IELTS
Annual Review notes, for example, a rise of 34%, from 355,000 to 475,000
candidates that year, with ‘particularly impressive’ growth ‘in South Asia andthe Middle East’ (2003:1). As established in Chapter 1, a ‘high-stakes, global
English language test, accepted as fulfilling English language requirementsfor people whose first language is not English and who need to study, workor live where English is used as the language of communication’(www.ielts.org home page), the IELTS test is naturally subject to changes inpolicies and procedures both in the home and destination countries ofcandidates. There is therefore never likely to be a perfect sample of the test’s
candidate population.
Figure 6.1 IELTS impact study and IELTS test candidate regional
backgrounds%
8070

Impact study

60

Official statistics
2002

50
40
30
20
10
0
East Asia
Pacific

South
Asia

WesternEuropeCentralandEasternEuropeCentralandSouth
America

MiddleEast andNorth
AfricaCentralandSouthern
AfricaNorth
AmericaandMexicoThe closer profile of the candidates participating in IELTS impact study Phase
3 is summarised in Figure 6.2. in terms of participant gender, age, English
language background, academic level and IELTS test status. The figures are
based on the full sample of 572 participants, percentages, however, calculated
without missing entries, unless stated. Further baseline information on the test
takers, including LLQ data, is available in the main report with the Cambridge
ESOL Research and Validation Group.

100Impact study candidate profile
Figure 6.2 Profile of candidate participants (n=572)
Gender:No.

% (rounded)female

305

55

male

252

45

% of statedAge:
15 – 20181

34.7

21 – 25192

36.8

26 – 3084

16.1

31 – 3533

6.3

36 – 4023

4.4

41 – 456

1.2

46 – 502

0.4

Not stated51

First languages

32

Countries of origin

34

Years of English:kindergarten to college/university
primary school to college/university
secondary school to college/university
kindergarten to secondary
kindergarten only
primary only
primary + secondary only
secondary only
college/ university

51
90
210
17
3
28
28
89
26

9
16
38
3
1
5
5
16
5

outside classes only

8

1

above levels with extra language classes?

310

56

Educational levels*
pre-university

7

6

undergraduate

56

46

post-graduate

59

48

Pre-IELTS

368

64

Post-IELTS

204

36

academic

165

89

general training

20

11

IELTS test status

IELTS Module taken** Only post-IELTS candidates asked

The fields of study of the impact study IELTS participant candidates are
summarised in Figure 6.3.
101

6 The IELTS impact study: main messages
Figure 6.3 Participant candidate fields of study
4%
Law
2%
Hotels and Tourism

3%
PublicAdministration

6%
Engineering
9%
ITC

14%
Education/Teaching
14%
Finance

32%
Business
16%
Health and Social
services

Recent evidence from the Cambridge ESOL IELTS Candidate Information
Sheets (CIS) suggests that this distribution of fields is fairly typical of current
IELTS entries.
The profiling of the candidates responding to the Phase 3 Student
Questionnaires serves two main purposes. It is necessary, first of all, as adata baseline against which to evaluate our later analyses of the impacts of
the IELTS test on these candidates, and, secondly, it provides further
evidence that the participants concerned are not untypical of the IELTS
candidate population in general. When we begin to discuss how candidates
are affected by the test, we shall be aware who the candidates are who arethus affected.

IELTS washback on preparation courses
The data analyses in this section are in response to the research question: Whatis the washback of the IELTS test on courses preparing candidates to take it?

The courses
96% of the IELTS impact study Phase 3 candidate participants wereattending or had attended a preparation course for the test. According to
the responses to the questionnaires completed by the 83 preparation course

102

IELTS washback on preparation courses
teachers participating in the study, most of whom, of course, were teaching
the participating IELTS candidates or their colleagues, the main
destination countries for academic study or other entry purposes were theUK (36%), Australia (28%), Canada (12%), New Zealand (12%) and theUSA (5%).
When asked what kind of IELTS preparation classes they were attendingor had attended, the participating candidates responded as in the table in
Figure 6.4.
Figure 6.4 Student experience of IELTS preparation course types (%)
Course categories% of responses

Course with ‘IELTS’ in its title

40%

EAP/English language study skills course

42%

General English course with IELTS preparation elements

17%

These course categories (by no means mutually exclusive, naturally) are ofparticular interest in the context of an investigation of test washback andimpact. A key issue is the extent to which students are participating in a course
to prepare for a test or participating to improve their target language
proficiency. The teachers in the study responded to a parallel item in theirquestionnaire, on the kinds of IELTS preparation classes they were currently
teaching. According to the table in Figure 6.5 they, too, were involved with asimilar variety of IELTS-related course types.

Figure 6.5 Teachers’ preparation course types (%)
Course categories% of responses

Course with ‘IELTS’ in its title

53%

EAP/English language study skills course

35%

General English course with IELTS preparation elements

12%

The table in Figure 6.6, using Teacher Questionnaire response data, indicatesthe class sizes in IELTS preparation courses, groups of 11–15, 16–20 and
6–10, in descending order, apparently the most common.

103

6 The IELTS impact study: main messages
The figures from the impact study centres differ somewhat from the range andfrequencies of the class sizes reported in the pre-survey (see Chapter 5). There,
it may be recalled, class sizes of 6–10 were more common than 11–15.
However, the three most common class sizes in both the pre-survey and thestudy are 6–10, 11–15, and 16–20, the three sizes accounting for 76% and 81%of the class sizes respectively.
Figure 6.6 Average numbers of students in IELTS-related classes
NumberNo answer

%

2

1–5

4

5

6–10

15

19

11–15

34

42

16–20

16

20

More

12

14

Total (without no answer)

81

100

The IELTS preparation teachers
In response to research question 5: What are the profiles of the teachers
preparing candidates to take the IELTS test? the objective data from the 83
IELTS preparation course teachers completing the teacher questionnaire give
key baseline information (see Figure 6.7, page 105).
The teacher profile here rather suggests that IELTS preparation course
teaching at the centres concerned was normally in qualified, though notalways IELTS-experienced, hands. Like the candidate profile preceding it, theteacher profiling is useful as baseline data. It will be important as we reportfindings on IELTS impact on the teachers below, for example on their viewsof and attitudes to the test, to be aware of the fact that the reactions are comingfrom an apparently well-qualified group. The baseline data will also be
informative, as IELTS impact studies progress through Phase 3 and beyond,on how typical the participants are of IELTS preparation course teachers ingeneral.

104

IELTS washback on preparation courses
Figure 6.7 Profile of IELTS preparation teachers
Gender:No.

%

Female

48

58

Male

35

42

Under 30

10

12

31–40

30

36

41–50

19

23

51–60

20

24

61+

4

5

Years teaching

No.

%

Age:

No answer

5

1–5

11

14

6–10

21

27

11–15

14

18

16–20

5

6

20+

27

35

Total excluding no answer

78

100

Qualifications (more than one possible)No.

%

No answer

7

Teacher training college

1

1

Post-grad certificate

4327

Diploma40

25

Bachelor degree

3623

Master degree

3321

Doctorate/PhD

4

3

Total (excluding no answer)

157

100

Position at institution

No.

%

Teacher64

80

Director

3

4

Manager

45Administrator

2

3

Professor

2

3

Other

5

6

No answer3

Total (excluding no answer)

80

100

Trained as an IELTS examiner:
Yes

52

No

48

Received training in preparation of students for IELTS:
Yes

32

No

68

105

6 The IELTS impact study: main messages

Teacher perceptions of the influence of IELTS on
preparation course content, methods
Pursuing answers to the research question What is the washback of the IELTS
test on courses preparing candidates to take it? we find, according to teacher
questionnaire responses, that 90% of the participant teachers agreed that thetest influences the content of their lessons, 63% that it also influences their
methodology. (The two questions were: Does the IELTS test influence yourchoice of the content of your IELTS preparation lessons (i.e. what you teach),and Does the IELTS test influence your choice of methodology (i.e the way
you teach) for IELTS preparation lessons?
Both these percentages are high, the second especially so given one ofAlderson and Wall’s findings in their 1993 study of washback from a revised
O Level English exam in Sri Lanka. This was that ‘the examination had had
considerable impact on the content of English lessons and on the way English
teachers designed their classroom tests (some of this was positive and some
negative), but it had little or no impact on the methodology they used in theclassroom …’ (1993:348). Note that Alderson retains this view, presenting it
now almost as a given: ‘We now know, for instance, that tests will have more
impact on the content of teaching and the materials that are used than they willon the teacher’s methodology’ (2004a:1). In the same piece, Alderson relates
this phenomenon of impact (or washback) with ‘the teacher factor in
washback’, adding that ‘it is at least as much the teacher who brings about
washback, be it positive or negative, as it is the test’ (ibid). Other factors
matter too, of course, as we inferred from the discussion of the complexity of
washback and impact in Chapter 1. Note also how this focus on the teacher
factor further emphasises the need for teacher baseline data in test impact
studies.
Liying Cheng, reporting a study on the (intended) washback of a Hong
Kong secondary school exam on teaching and learning, concludes thatteachers’ ‘perceptions of the underlying teaching methodology associatedwith the new [Hong Kong Certificate Examinations in English] remained
relatively unchanged’ (2004:163).
Judging from teacher responses to the follow-up open-ended items intheir questionnaire on IELTS washback on preparation courses, thedistinction between course content and methodology is not always clearcut.
Whereas there was only a single no-response to the question ‘… please notehere how the [IELTS] test influences your decisions on lesson content (i.ewhat you teach)’, 29 of the teachers gave no response to the parallel item on
how the IELTS tests influences teacher choice of methodology for IELTS
preparation lessons. The implication is that they were not easily
distinguishing between content and methods. Three of the 79 teachers

106

IELTS washback on preparation courses
responding actually wrote ‘see 3.1’, that is referring the question on methods
to their response on content.
Both the perceived overlap between course content and methods and an
apparently strong focus on the IELTS test itself are clear from the teacher
responses. Fifty-five of 82 responses to the question on course content (67%)
referred specifically to teaching for IELTS test elements, including:
•
•
•
•
•
•
•
•
•
•
•
•
•

three references to ‘teaching for the test’
five to focusing on the test
five to content relevant to the test
three to content related (exactly, directly) to the test
seven to content dictated by, concentrating on, oriented to, influenced by,consistent with the test
four to work specifically for the testeight responses which referred to teaching with time pressures or time
management in mind, reflecting that aspect of the IELTS test
four stating that a test focus was insisted on by the students
one claiming a 100% focus on the test, one to the test being ‘all we do’,one to it being 90%
five mentioning test practicenine referring to test techniques/strategies
four to predicting test items
three to old/sample papers.

These responses in the context of questions on test content compare witheasily the highest number of responses (23) referring to teaching IELTS
formats, techniques, skills in response to the methodology question, and sevenreferences to administering practice tests. Three teachers also referred,
responding to the methodology question, to student demands for a close test
focus on their preparation courses.
The message from these Teacher Questionnaire items is that there appears
to be IELTS washback on the preparation courses in terms of content andmethodology, a conclusion apparently differing from that drawn by Alderson
and Hamp-Lyons (1996), Watanabe (1996) and also by Green (2003), whose
observations of IELTS preparation and EAP classrooms indicate that ‘course
content was very clearly influenced by the test, but any influence on teaching
and learning methods was less obvious and was mediated by participant
beliefs’ (2003:45). Green also reminds us, however, of the need to triangulate‘potentially misleading interview/questionnaire data’. ‘Wall and Alderson’(1993:65), Green notes, ‘could find no evidence of observable changes in
instructional methodology, despite the teachers’ assertions that the new test
affected how they taught.’ IELTS study Phase 3 classroom observation data
are cited in the next section, suggesting again the importance of the teacher
107

6 The IELTS impact study: main messages
factor and implying, like Green, a typical eclectic methodological approach:‘For the individual teachers, methodology did not emerge as a core issue,
perhaps varying little across the two course types. Teachers did not tend toconstrue class activities in terms of methodologies…’ (op. cit.:128–29). As
the classroom observation data indicate, however, the eclectic methods tended
to be influenced by the Communicative Approach to Language Teaching(CALT) (see Chapter 7).
In case the IELTS study Phase 3 teacher quotes above might suggest such
relationships are too straightforward when we have noted above how complex
they actually are, it should be noted that the teacher responses to the course
content and methodology questions also featured the following views andpractices:• Six of the 83 teachers (7%) responding to the two items indicated that thetest did not influence the choice of content of their IELTS preparation
lessons.
• 27 (32%) of the teachers claimed that IELTS did not affect their
methodology on such courses.
• Five responses emphasised the teaching of general English or improving
students’ general proficiency, two improving their study skills, two
encouraging wide reading.
• Three responses emphasised the teachers’ student-centred view, trying tochallenge learners and to build their confidence.
• There were 10 references to a focus on communicative micro-skills (e.g.
prediction, reading and listening for (key) information, recognising text
types, organising text, expressing ideas).
• Nine teachers claimed an emphasis on grammar/structure, five on
vocabulary.
• Five of the responses each referred to the courses being more structured
because they were preparing for IELTS, and to their being task-based,
three to a focus on topics.
• There were two references to pair work, two to group work.
• Five teachers described their approach as ‘structured’, three as‘intensive’,’ two as ‘focused’, one as ‘serious’ (without specific reference
to IELTS as a cause of these features).
• Two teachers said there was more teacher talk and input than on other(non-IELTS) courses.
• Teachers referred once each to their IELTS prep courses as being less‘entertaining’, ‘dynamic’, ‘personal’, ‘reflective’, and having less‘discovery’, ‘interaction’, ‘flexibility’.
The Phase 3 Teacher Questionnaire asked more targeted questions relating tolesson content and methodology in addition to the two general items on IELTS108

IELTS washback on preparation courses
preparation course content and methodology (see above). Figure 6.8 in thenext section shows prominent IELTS-preparation course activities identified
by teachers and students. These are taken from an inventory of 28 suchactivities developed and validated during Phases 1 and 2 for the originalclassroom observation analysis instrument (Banerjee 1996) and the student
and teacher questionnaires (Horak 1996, Winetroube 1997). All four macroskills are covered and the activities specified have a methodological tone (e.g.
listening and taking part in seminar/workshop activities; interpreting
statistics/graphs/diagrams; short report writing; group discussions/debates).
These are similar to methodology-oriented questions to the teachers in
Cheng’s Hong Kong study (1998:170), for example: explain the meaning of atext; organise group work or discussion. Green refers helpfully to categories
such as ‘participant organisation, content control or student modality’ indescribing the ‘broad similarities in teaching methods across courses’ (2003a:339) revealed by his observation of IELTS and EAP preparation, his
categories surely supporting the potential overlap between content andmethodology. Cyril Weir (personal communication) wonders whether, by
their very academic reading and writing emphasis, both course types are likely
to be target domain and, thus, content influenced.
The richness of the Phase 3 data on course content and methodology so far
certainly justifies the inclusion of open-ended questions following the
Yes/No/Don’t know items. As anticipated in Chapter 4, the open items served
both a triangulation and a data-enhancing function.

Summary of preparation course contents, skills, activities
We now move from teachers’ reactions to questions designed to encouragethem to evaluate the influence of the IELTS test on their preparation courses,
to attempts to discover what students and teachers perceived as the actual
content of such courses. The table in Figure 6.8 (page 110) summarises
activities selected by high percentages of both the teachers and the students asprominent in their IELTS preparation courses. The activities concerned have,it will be noted in connection with discussion in the previous section, bothcontent and methodological implications.
The influences of the IELTS test may still be seen in these activities, forexample in their relationships with the four modules (Reading, Listening,
Writing, Speaking). The prominent items selected may also appear quite likely
to make for interesting classroom lessons, including the use of a rather
appropriate and lively set of communicative activities, relevant to students’needs.
As for the sources of IELTS preparation lesson content, a Teacher
Questionnaire item asked what materials, other than a main textbook, theteachers used. The answer was additional textbooks (in 19 cases of the 62
109

6 The IELTS impact study: main messages
teachers specifying materials in response to this question, 10 of theseidentifying books offering extra test hints or practice, the rest textbooks (20)targeted at specific language skills or components). However, supplementarymaterials from the press (11), TV and radio (7), video and audio (11) and theInternet (5) were also mentioned by the teachers, as were centre in-house orteachers’ own materials (11), mostly not further specified.
Figure 6.8 Candidate and teacher perceptions of prominent IELTSpreparation course activities

Activities
Reading questions and predicting listening text andanswer types

Students %Teachers %89

86

Listening to live or recorded talks and note-taking

83

63

Analysing text structure and organisation

74

90

Interpreting and describing statistics/graphs/diagrams

74

90

Learning quick and efficient ways of reading texts

73

93

Reading quickly to get main idea of text

77

96

Learning how to organise essays

82

99

Practising making a point and providing supporting examples

78

88

Group discussion/debates

83

76

Practising using words to organise a speech

74

83

There are related data of interest from the Instrument for the Analysis of
Textbook Materials (IATM, see Chapter 4), from items seeking answers to
Research Question 7: Which textbook and other materials are used on IELTS
preparation courses? and Research Question 8: What is the washback of
IELTS on these preparation course materials? Teacher textbook evaluatorscompleting the instrument on IELTS-related textbooks, most of which (66%)
were language teaching and test-preparation books combined, found that they
covered micro-skills such as those listed, in rank order of selections by the 43
teacher-evaluators, in Figure 6.9.
As the chart in Figure 6.10 indicates, too, there is a post-structural,communicative ethos to the organisational framework of most of the books
analysed by the 43 teachers concerned.
IELTS coursebook activities considered by the teacher evaluators toencourage communicative opportunity were, in rank order: essays, group
work, pair work, reports, reviews, reading, listening, viewing for personal
interest, and role plays.

110

IELTS washback on preparation courses
Figure 6.9 Perceived micro-skills coverage in IELTS preparation books
Micro-skills

No. of selections(N=43)

identifying main points

40

identifying overall meaning

38

predicting information

36

retrieving and stating factual information

34

planning and organising information34

distinguishing fact from opinion

31

drawing conclusions

30

making inferences

29

evaluating evidence

27

identifying attitudes

23

understanding, conveying meaning through stress and intonation

15

recognising roles

12

Figure 6.10 Teachers’ views of organisational units of their IELTS
preparation coursebooks
30

Number

25
20
15
10
5
0

topics,themeslanguageskillsgrammaticalstructures

tests,
tasks

notions,functionsArea

As we have seen in Chapters 2, 3 and 4, it is part of the IELTS claim to theorybased validity that it reflects the language macro- and micro-skills needed bycandidates to carry out their target communicative activities. The student andteacher data already presented in this chapter indicate that the preparation
courses and their purpose-designed textbook materials are perceived as bothinfluenced by IELTS and characterised by an emphasis on communicative
activities and micro-skills. If this is definitely the case, it could again be
evidence of Green’s washback : construct overlap model (Figure 1.3 in111

6 The IELTS impact study: main messages
Chapter 1), where the target language domain and the language teaching,learning and use coincide, at least partly due to washback from the constructsof the test.
A useful check on whether the IELTS preparation courses really are as thestudents and teachers describe them in their questionnaire responses could bedata from IELTS preparation lessons recorded and analysed using classroom
observation instrumentation (see Chapter 4 and Appendix D). The Phase 3
instrument used categories such as participations, timings, materials, andactivities to describe the lessons concerned. It was also felt appropriate toinclude a student communicative opportunity time criterion for listening,speaking, reading and writing activities in an attempt to detect significant
differences in how much learning and communicating time individual
students were actually getting.
The findings from the 10 IELTS preparation lessons analysed, supported
by analyses of 20 recorded PL2000 lessons, should, of course, be treatedwith caution, given the limited number of recordings and analyses so far in
the video database concerned (see Chapter 5). This is already being
expanded, however, through related impact study research (see Chapter 8).
Impressions from the IELTS preparation lessons recorded and analysedsuggested that, while communicative activities across the four skills in linewith the test tasks were evidence of CALT influences (see previous), theopportunities for learners to communicate on their own behalf seemed to varyconsiderably, with teacher approach and attitude a particular influence. Thiswas further evidence, of course, of the influence of the teacher factor.
Communicative opportunity for learning in the target language was enhanced
where teachers were willing and able to allow learners the time and space totry to communicate, even when they were still uncertain of their ability to doso. The observed IELTS lessons included significant moments, when, thanks
to a teacher’s well-prepared but flexibly learner-centred lesson, a learner wasable to strain towards adequate communication of a message with the help of
a fellow-student partner also communicating for that purpose in the target
language.
General impressions from the analyses of the IELTS preparation lessons
may be summarised as follows:• learners who are motivated, but sometimes to the extent of wanting, evendemanding, a narrower IELTS focus than their teacher would otherwise
tend to offer• teachers’ preference for task-based, often inter-related, macro-skills
activities, involving micro-skills relevant to IELTS• the use of materials from within and beyond the textbook• multicultural learning and communicating between learners often one of
the most engaging features in mixed nationality classes

112

IELTS washback on preparation courses• an ethos of focused activity within a coherent, often learner-centred,
institutional approach to preparation for IELTS, but• teacher willingness to try a range of teaching methods and approaches.

Student and teacher preparation course satisfaction
An open-ended item invited candidates already studying in an Englishmedium situation to comment on whether their IELTS preparation courses
provided them with the language knowledge and skills they needed. There isa high 83% positive response from the 282 participants concerned. But thisapparently positive view of preparation courses did not necessarily mean thatcandidates were satisfied with their own performance on the courses. When all(N=431) students attending preparation courses were asked whether they felt
they were successful on the courses, the responses were less positive (seeFigure 6.11), with only 49.5% of those who were sure of their answer saying
yes.
Figure 6.11 Candidate views on their success on IELTS
preparation courses
Do you think you were/are successful on the preparation course(s)?
Yes

184

No

187

Unsure

39

Other

20

The main criteria for success mentioned by those responding positively to thisitem and adding their comments (130 students) were: perceived improvementin English proficiency level (26 responses) or skills (14 responses) andincreased familiarity with the test (33 positive responses). This result suggests
an interesting balance in students’ perceptions of success between
improvement in their target language proficiency and increased test
familiarity.
But (see the summary figures in Figure 6.12) a closer analysis of thereasons given by the 133 students elaborating their negative answers on
preparation course success reveals the candidates focusing mainly on
problems of their own rather than with the course itself.

113

6 The IELTS impact study: main messages
Figure 6.12: Reasons given by candidates for lack of success on
preparation courses
ReasonNo.

Did not work hard enough

29

Target language (TL) problems: vocabulary (8), grammar (2), reading (2),listening (2), writing (4), speaking (4), spelling (1)23

Did not have enough study time, had other demands on time

17

General TL level perceived as not good enough

16

Perceived personal characteristics: lack of concentration (4), lack of confidence (5),inconsistency (3), age (1), over use of L1 (1), poor problem-solving (1)15

Not enough practice (8), test techniques (3) given on course

11

Course too short

10

Did not achieve required IELTS band

6

Test time pressure

5

Course too academic

1

Test topics

1

Bad test

1

Significant among the explanations given by the participants was: the questionof time, including the inability of a significant number to devote enough timeto preparation for the test; the time span of the courses themselves; and thetime pressures in the test itself, a factor we shall return to. Perceived English
language proficiency problems are also a significant factor here. And it seems
that, although participants are unlikely to see their preparation course as
successful if they do not subsequently achieve their required IELTS bandscore, they tend not to blame the test itself for the situation. The next sectionof this chapter investigates further the impacts of the IELTS on students andteachers.
Figure 6.13 summarises preparation course teachers’ perceptions of thesuccess of their IELTS courses compared with other courses they teach.
While only 9% of the 83 teachers felt that their IELTS courses were lesssuccessful than other courses they taught, note the reasons given for theirpositive views:• clear course goals and focus (21)• high student motivation (16) (an important issue discussed further when
we investigate IELTS impact on candidates)• clearcut student achievement potential (12), and• course validity (11) in terms of student target language needs, topics andskills.

114

IELTS washback on preparation courses
Figure 6.13 Teacher perception of IELTS preparation course success
9%

39%

52%

More successful

As successful

Less successful

The negative points raised by the teachers concerning their IELTS preparation
courses form an interesting counterpoint to the positive perceptions. Nineteenof the teachers, for example, were concerned about the narrowness of theirstudents’ focus while on IELTS preparation courses, six about pressure on thestudents, and six about the validity of the IELTS Writing test, also a point
pursued further under IELTS washback.
The picture appears mixed on IELTS preparation courses. The teacher data
so far on the influence of IELTS on course content and methods areinteresting, emphasising once again the complexity of test washback, and theobvious intervening strength of teacher and teaching variables. The tendency
towards eclectic methodological approaches to language teaching on IELTS
preparation may again be emerging here (see above). We may recall one of the
responses in the Phase 3 teacher profile at 6.7 above, where 68% of theparticipants claim that they have not received training in the preparation of
students for IELTS. Five of the 33 teachers who responded to the final teacher
questionnaire item offering the opportunity to ‘note anything else you wish to
say about your IELTS preparation course’ refer to the need for more training
in IELTS preparation teaching. In one case, supported by views expressed atinterview by five face-to-face contacts (see below), training and technicalsupport for IELTS preparation centres were seen as an antidote to centres that‘provide training just for money and focus only on the passing rate of the test
takers’. The teacher concerned then suggested that, in order to ‘realise theoriginal concept of IELTS’, key centres should be provided with ‘more
technical support so as to have them play an even more influential effect on115

6 The IELTS impact study: main messages
the others and lead the training on to the right track’. The next section of thechapter will pursue this point further, the teacher training variable being oneof those identified by Alderson (e.g. 1993, 1995) and leading him to express
the view that: ‘in short, it is at least as much the teacher who brings about
washback, be it positive or negative, as it is the test’ (2004a:x).
The study’s face-to-face contacts emphasise the need to improve IELTS
results at centres running courses for the test, for example: by identifying anddisseminating examples of good preparation course practice, through closer
monitoring of IELTS centre preparation courses and by more seminars for
IELTS teachers. At a language centre student focus group (August 2002) it
was suggested that more self-study materials were needed for IELTS
candidates. But a research student interview (June 2002) with a participant
with first-hand knowledge of the increasing numbers of online IELTS
preparation websites in China warned that these, though often apparently
informed by specialists on the test, tended to focus on test tips and problem
avoidance strategies rather than communicative language skills development.

Impact of IELTS on candidates and IELTS preparationteachers

Perceptions of test fairness
The focus now is on IELTS impact study Research Question 4: What is theimpact of IELTS on the participants who have taken the test? This is in many
ways a key question. It is central to the study of washback and impact,
including as it is intended to not only language learning and teaching matters,
but also matters pertaining to test fairness. The question is, as discussed in
Chapters 1 and 4, crucial to the validation of high-stakes tests. In this section,
therefore, we shall cover perceptions of test fairness, likes and dislikes,pressures, test difficulty and test performance levels. As in the previous
sections, these factors will be investigated as they apply to teachers as wellas test candidates, thus also responding to Research Question 6: What is theimpact of the IELTS test on the teachers preparing candidates to take the test?
Phase 3 IELTS impact study candidate participants who had already taken
IELTS were asked whether they thought IELTS was a fair way to test theirproficiency in English. Figure 6.14 summarises the responses (of the 190
concerned) with the option to explain why/why not.

116

IELTS washback on preparation courses
Figure 6.14 IELTS takers’ perceptions of the fairness of the test
Do you think IELTS is a fair way to test your proficiency inEnglish? (N=190)
YES

72%

NO

28%

If No, why not?
1 opposition to all tests
2 pressure, especially of time
3 topics
4 rating of writing and speaking
5 no grammar test

The 72% : 28% split on perceived test fairness may be considered a rather
positive response, especially when we predict people’s expected response to aquestion on their perceptions of test fairness in general. Following through tothe test takers’ specific objections to the test, we find the most frequent of the
49 follow-up responders (some making more than one point) was opposition
to all tests, an interesting finding and perhaps somewhat unexpected. Amongthe 25 comments indicating that tests in general were seen as unfair were thefollowing:• ‘Any test is unfair as they’re tested for a day while they have done a lot
before.’• ‘It just depends on one test.’• ‘Because it is a test, it is different from doing it at home.’• ‘It is a test – some students can get a good mark in the test but are notable to use it in real life.’• ‘I usually cannot perform well on exam day.’• ‘Because sometimes it depends on your fate.’
But notice, too, how, in a way typical of open-ended responses, some students
claim a feeling that all tests are unfair, yet for reasons which probably do notreally apply to all tests:• ‘All tests are unfair because the candidate may not do well due to theirfear, confusing questions, time pressure and health problems.’
Some of the responses seem to be focusing on the pressure of high-stakes tests
per se, others on the time factor in IELTS specifically, for example:• ‘Performance is reduced under pressure conditions.’• ‘Not fair to test speaking proficiency in a 10-minute test.’

1176 The IELTS impact study: main messages
The three remaining rank-ordered participant categories of doubt over IELTS
fairness in Figure 6.14, namely test topics, the rating of speaking and writing,and the absence of a grammar test, are language oriented and, like the first
three objections, will be investigated further.
Teacher Questionnaire trialling had indicated that teachers found a straightquestion on IELTS fairness rather too broad. The related item on the Teacher
Questionnaire thus invited teachers to compare their students’ results on the
IELTS test with their own assessment of the students’ language proficiency
across the four macro-skills. Figure 6.15 summarises the results.
Figure 6.15 Teachers’ comparisons of their students’ IELTS test results
and their own assessment of their students’ language ability
Overall%Reading%

Listening%

Writing%

Speaking
%

Same

70

69

64

61

64

Some higher/some lower

23

15

16

23

16

Consistently lower

2

8

10

8

10

Consistently higher

5

8

10

8

10

The overall ratings in the table may be seen, in a way, as the teachers’ estimateof test fairness. If so, they are close to the students’ view as indicated in Figure
6.14. The least satisfactory match between teachers’ and IELTS assessments
appears to be for the writing module (see below).

Test taker IELTS likes and dislikes
In the Student Questionnaire, candidates who had already taken the IELTS test
were asked for their likes and dislikes in the test, these items seen as related tothe item on test fairness and supporting reasons (see above). The items were
open-ended, and analysed, like other open-ended items, using key words (seeChapters 4 and 5). Figure 6.16 summarises the top likes and dislikes, thecomparisons between the two being quite revealing.
Several of the positive perceptions of the test (including fairness,
comprehensiveness) have been combined into a ‘validity’ category. Three ofthe test modules appear as both likes and dislikes, but Reading (referring to the
IELTS Reading test module) is notably absent from the ‘likes’ column, only
to appear as a major ‘dislike’. This invites further investigation of the IELTS
Reading module below. Yet again the time pressure aspect of the IELTS test
re-appears as a major perceived problem.

118

Test anxiety
Figure 6.16 Comparison of test taker IELTS likes and dislikes
LIKES (132 post-IELTS students responding)n
1 VALIDITY [fair (17), 4-skills/
41
comprehensiveness (15), recognition (7),language and study skills (2)]DISLIKES (138 post-IELTS students
responding)n1

TIME PRESSURE

50

2 SPEAKING

17

2

READING

41

3 STRUCTURE, ORGANISATION,FORMAT

16

3

LISTENING

18

4 WRITING

15

4

WRITING

16

5 INCENTIVE, CHALLENGE,
INTEREST, VARIETY

145‘COMPLICATED QUESTIONS’

9

6 LISTENING

13

6

LANGUAGE DIFFICULTY

87SPEAKING

7

Test anxiety
There have been indications from the data already analysed on test fairness
and test likes and dislikes that the IELTS test does indeed cause anxiety. Thatis also one of Alderson’s givens:
We know that high-stakes tests – tests that have important consequences
for individuals and institutions – will have more impact than low-stakes
tests, although it is not always clear how to identify and define the natureof those stakes, since what is a trivial consequence for one person may be
an important matter for another (2004a:ix-x).

The test takers themselves responded to a Likert scale item ‘Did you worry
about taking the IELTS test?’ as in Figure 6.17.
Figure 6.17 IELTS test taker anxiety
Did you worry about taking the IELTS test?
Very much

Very little

No.

%

78

41

58

31

36

19

18

9

It is very clear that IELTS causes anxiety, 72% of the 190 responding post-test
participants claiming to have been worried or very worried by the test.
Applying the response-prediction check to this matter, however, one might119

6 The IELTS impact study: main messages
expect that range of professed anxiety about most of the high-stakes tests we
encounter in our lives. And the figures of 19% and 9% respectively ofparticipants who claim to have been not very or very little worried actuallyseem, totalling as they do 28%, quite high.
Of course, in test impact and validation terms, the concern must be whether
a test is causing levels of anxiety that could distort results, the validity ofinferences made according to test scores and thus the test’s fitness for use. To
investigate this matter further from test taker point of view and to triangulate
evidence, we can refer to the Language Learning Questionnaire items also
included in the IELTS impact study Student Questionnaire (see Chapter 4). Of
particular interest should be the section on test-taking attitudes and strategies,
the final item of which states After a test, I usually feel that I have done as wellas my knowledge and ability deserve. This is informative on candidates’ testtaking in general, to set against the perceptions of their performance of IELTS
candidates who had already taken the test. Figures 6.18 and 6.19 provide therequired data analyses.
Figure 6.18 IELTS candidate perceptions of their test
performance in general‘After a test, I feel that I have done as well as my knowledge andability deserve’No.

%

Always

75

14

Often

237

35

Rarely

181

34

Never

37

7

Figure 6.19 Post-IELTS test taker perception of IELTS performance(N=190)Do you feel that you performed to the best of your ability in the [IELTS] test?

Yes 46%

No 54%

What affected your performance?

1 A lot1
Timepressure(n)2
Unfamiliarity
of topics(n)3
Difficultyof questions(n)4
Fear of
tests
(n)

5
Difficulty oflanguage
(n)

91

48

33

26

20

2 Quite a lot

55

58

70

44

56

3 Not a lot

25

51

68

71

78

4 Not at all

10

22

10

32

27

120

Test anxiety
The two tables suggest the following about IELTS test anxieties:• The IELTS candidate population represents a mixture of optimistic andrather pessimistic views of own general test performance i.e. 49%
positive, as in Figure 6.18.
• This compares with the fairly similar 46% of those who had already taken
the IELTS test who felt that they had performed to the best of their ability
in the test (see Figures 6.18 and 6.19).
• Time pressure (see above) figures once again as a major cause of IELTS
test anxiety, 81% of the test takers rating it as affecting their performancea lot or quite a lot (Figure 6.19).
• Topic unfamiliarity again emerges as an issue of IELTS test concern for
59% of the IELTS test takers (Figure 6.19).
• Difficulty of questions affects the performance of 57% of the test takers,
which, if it relates to the category ‘complicated questions’ mentioned as
an IELTS ‘dislike’ in Figure 6.16, may be a matter warranting attention in
the continuing validation processes for IELTS.
• General test anxiety (40% rating this as affecting performance a lot orquite a lot) and language level difficulty (42%) are significant but lessdominant anxiety sources for the post-IELTS candidates.
The Phase 3 IELTS impact study teachers remind us of the frequently
perceived relationship between anxiety and motivation. Figure 6.20 presents
teacher responses to two items, the first asking their view on IELTS anxiety,the second on test motivation.
Figure 6.20 IELTS preparation course teacher (N=83) perceptions of
IELTS impact on their students
Does the IELTS test cause stress for your students?
Yes

44

No

27

Don’t know

11

Does the IELTS test provide motivation for your students?
Yes

70

No

8

It will be recalled from the discussion above of teachers’ views on the successof their IELTS preparation courses that ‘high student motivation’ wasconsidered a major positive factor in this. But, of course, the complexity anddynamism of both constructs is clear, as Green tellingly comments:121

6 The IELTS impact study: main messages
If it can be agreed that raising the stakes for participants increases their
extrinsic motivation, this is not regarded by all as a benefit for teaching
and learning. As with other sources of motivation and anxiety (Alderson
and Wall, 1993; Skehan, 1989; Spolsky, 1989), it seems likely that
individuals will be differently affected and that benefits for one
educational context may be deleterious to another. Indeed, the same
effects that are judged to be beneficial by some observers may be judged
to be damaging by others (2003:26).

It will be seen in Chapter 8 that the complex issue of motivation in language
learning is being pursued in a further Cambridge ESOL impact study research
project.

Test module difficulty and validity
As Figure 6.16 indicates, the fact that IELTS tests all four macro-skills,
Listening, Reading, Writing and Speaking, is seen by candidates as asignificant positive point. Figure 6.16 also shows that there is a difference inthe perceived levels of difficulty across the skills modules. This is confirmed
by both candidate and teacher questionnaire responses to closely-related items
inviting them to rank the IELTS skill modules in order of difficulty. Figure
6.21 summarises these results, indicating that both candidates and IELTS
preparation teachers have very similar perceptions on the relative difficultiesof the IELTS skills modules.
Table 6.21 Student and teacher perceptions of IELTS
module difficulty
Most difficult
IELTS Module? (%)
Students

Teachers

Reading49

45

Writing24

26

Listening

18

20

Speaking

99

The Reading module is seen as clearly the most difficult of the four IELTS test
modules across our candidate and preparation course teacher participants. The
point is pursued further, but it is of interest at this stage to note (from Figure
6.14 above) that the Reading test does not appear in the top five reasons givenby the 28% of IELTS candidates who feel IELTS is unfair. This is despite thefact that the Reading module is the second most disliked element in IELTS,
according to Figure 6.16.

122

Test anxiety
This variation across the perceived difficulty of the skill modules may tosome extent be reflected in the organisation of IELTS preparation lessons.
Reading activities (see Figure 6.22) are considered by the candidates as
occupying an average of 28% of course time, rather more than the almost
equal proportions for Listening, Speaking and Writing.
Figure 6.22 Chart of students’ perceived module difficulty andpreparation course timings (%)
28

17

20

19

7

8

Time

49

24

18

9

40

60

80

Listening

Speaking

GrammarDifficulty020

Reading

Writing

100

120

Vocabulary

The observed lessons, however, suggested that it is not always straightforward
to identify a classroom activity with a particular macro-skill. Integrated-skill
activities seem common. Group oral brainstorming or video note-taking work
as a preliminary to a writing topic, for example, may be categorised differently
by different students (and teachers).
The inter-relationships between perceived difficulties emerging from thequestionnaire data seemed to invite further investigation through second-level
analysis (see Chapter 4). One such analysis was intended to discover whether
there was a correlation between the perceived most difficult test skill, reading,and other factors perceived as affecting candidates’ performance, in particular,time, which was also frequently mentioned (see previous) as a significant
cause of worry for candidates. Figure 6.23 emphasises the dominance of theReading test module as the most difficult according to test takers and of timepressure as the most prominent problem with the Reading test, as it is,relatively, with the other tests, especially with the Listening test.
Figure 6.23 Relationship between perceived skill difficulty and otherfactors perceived as affecting candidate test performance
Difficultyof language

Difficulty ofquestions

Unfamiliarity
of topics

Timepressure

Fear of
tests

Others

Total

Listening

4

7

6

16

4

1

38

Reading

13

20

28

51

14

2

128

Writing

10

10

19

26

8

0

73

Speaking

2

4

6

9

3

1

25

123

6 The IELTS impact study: main messages
Figure 6.24 represents an attempt at a further consolidation among
candidates claiming to worry ‘very much’ about the test and suggesting that
the Reading module was a factor in their worry (i.e. their most or second mostdifficult skill) and also considering that they did not perform to the best of
their ability in the IELTS test. It was found that the perception of suchcandidates of factors that affected their performance a lot was again
dominated by the time factor.
Figure 6.24 Consolidation between perception of skill difficulty andperception of other factors affecting candidate performance45

Listening
Reading
Writing
Speaking

40

Opinion %

35
30
25
20
15
10
5
0

Difﬁculty oflanguage

Difﬁculty ofquestions

Unfamiliarity
of topics

Timepressure

Fear of
tests

Others

Factors

The perception of the Reading module as more difficult than the other IELTS
test modules, and of time pressure as affecting all modules, may be seen as test
validity concerns of the theory-based kind. One might ask, for example,
whether the reading expected of post-IELTS test takers in their target higher
education domain is more ‘difficult’ than their Writing, Listening and Speaking
activities in that domain. Such a question would be part of a set of questions
about how the IELTS test modules compare with the skills, macro and micro,that they purport to be testing. Should the Reading module focus more on
expeditious reading skills (e.g. skimming, search reading, scanning) rather thanor as well as narrower, text part specific micro-skills? And is time such apressure on those skills in the real higher educational life that candidates arepreparing to face? Candidate responses in Figure 6.25 to the Student
Questionnaire item on the occurrence of particular reading micro-skills in the
IELTS preparation lessons (see also 6.8 above), suggest good coverage ofexpeditious reading skills.

124

Test anxiety
Figure 6.25 IELTS students’ views on occurrence of reading skills on
their preparation courses
Reading skill

Yes

No

Analysing text structure and organisation.

74

13

Not sure
12

Interpreting statistics/graphs/diagrams.

74

15

12

Reading texts to predict test questions and tasks.

69

19

11

Learning quick and efficient ways of reading texts in English.

73

13

14

Reading articles, reports, books in your specialist subject area.

44

43

13

Using English-only dictionaries to complete reading tasks.

32

57

11

Reading quickly to get the main idea of a text.

77

13

10

It is interesting to note at this point that test users’ perceptions of the impactof time pressure on their performance, especially in reading, are notnecessarily supported by results from internal validation analyses whichmonitor the presence and effect of test speededness. Level of speededness istypically evaluated in the language testing profession on the basis of ‘omitted
responses’, i.e. test items which candidates fail to answer. According to this:
1. 80% of candidates ought to be able to respond to almost every item ineach section.
2. 75% of the items in a section ought to be completed by almost the entire
group.
By these criteria, analysis of test response data shows that the IELTS Reading
tests are not in fact speeded, even though candidates may perceive them assuch. Relevant points from face-to-face impact study data were examined,
collected, it will be remembered from Chapter 4, from a selection of thecentres visited for Phase 3 of the study. In the section of the face-to-face datareport headed Skill module validity, a range of related matters are raised.
A centre director and a teacher from a university English language centre(interviewed in June 2002) represented a frequently mentioned view, that it is,
by definition, difficult to test reading and writing skills directly in ways thatauthentically relate them to real academic reading and writing activities. In thetarget academic study domain, these tend to involve long, multi-sourced texts,
handled receptively and productively, and heavy in references and statistics. A
focus group of 15 IELTS candidates from five countries at the same university
questioned why reading and writing should be tested under time constraints
when in real higher education practice, students were able, to a large extent, tomake their own time. But the student group appeared less in favour of target
domain authenticity when its members suggested that there should be choicesof topics in the IELTS Writing test because a single topic might be‘unsuitable’.

125

6 The IELTS impact study: main messages
Members of a teacher focus group at the language centre of a different
university in November 2002 agreed that on balance the time pressure in the
IELTS Reading module was still excessive, and suggested that the Reading
test as it stands should be allocated 1 hour 20 minutes rather than 1 hour. The
same group recommended 11⁄2 hours rather than the current 1 hour for the
Writing module on the grounds, again, that the timing was both too tight andunrepresentative of most of the writing students were called on to do in theiractual academic life.
For some post-IELTS international students, the heavy (immediatelyrequired) reading and writing load on entry to university or college proved amajor difficulty. Teachers at the June 2002 interviews and in the November
2002 focus group agreed. That the IELTS Reading and Writing modules wereperceived as the two most difficult was thus to be expected. Some of theteachers in the face-to-face sessions and the focus group, however, wondered
why some very difficult tasks in the IELTS Reading module demanded carefulreading and analysis yet under time pressure. Were these there, they asked, todiscriminate for the ‘Band 8/9 candidates’? Were they, perhaps, the kinds ofitems referred to above among IELTS candidate dislikes, as ‘complicatedquestions’?
The complexity of the validity notion and the difficulty of designing anddelivering reading tasks on which test taker scores could be interpreted as avalid reflection of their competence in the skills concerned (as they occur inthe target language domain) appeared to be accepted in the face-to-face
discussions. A November 2002 focus group questioned the authenticity of
IELTS academic writing module task A, where candidates are asked to writea report ‘based on materials found in a table or diagram’ (IELTS Handbook,2003). A teacher from another centre (September 2002) wondered whether
this task was in fact ‘interpretative’ (as claimed in the IELTS Handbook) sincecandidates appeared mainly to be asked to describe. A teacher, at a third centre(also November 2002) asked why the data labels in the same tasks were often
so obviously non-authentic. Some teachers (at September 2002 interviews at amajor IELTS centre) were concerned that the Reading test tasks could become
formulaic, predictable and thus vulnerable to slick test techniques training.
A call for specialised subject modules was made by two language school
focus groups, one of 10 students from seven countries (July 2002) and one of
12 students from eight (November 2002). Six subject-specific Reading andWriting modules had been offered, of course, in the IELTS predecessor test, theELTS (see Chapter 4) and three such discipline areas (physical sciences andtechnology; life and medical sciences; arts and social sciences) by the IELTS
test itself from 1989 to 1995. However, the matching of subject-specific subtests to receiving institution courses was by no means a clear-cut matter(Charge and Taylor 1997:374). Research into the potential effectiveness of aone-module approach (see Clapham 1996) indicated that this would not126Test anxiety
discriminate against candidates, whatever their field of study. Thus, in April
1995 the subject-specific IELTS modules were replaced by single academic
Reading and Writing modules. Davies (forthcoming) discusses in some detailthe issues and considerations involved in this decision.
Teachers from more than one centre (in September and November 2002
interviews) felt that the IELTS test attempts, rightly, to assess micro-skills,
including those with a study-skill orientation, for example evaluating evidence(see Figure 6.9) as well as language performance. In fact, IELTS is notnormally considered to test study skills in the conventional sense such as
library use or referencing, but it is of interest that some IELTS preparation
course teachers think that it does. Note also that ‘study skills’ in the Common
European Framework interpretation mean ‘ability to make effective use of thelearning opportunities created by teaching situations’ and include ‘ability to
use available materials for independent learning’ (2001:107). Teachers at
almost all centres visited referred to the need to help some pre-IELTS students
with skills (which the teachers sometimes referred to as ‘study skills’) likely
to enable them to cope with the more independent study modes of the target
receiving institutions.

Test performance
IELTS impact study figures on band scores suggest that, when they wereattending their preparation courses, participant candidates had not yetachieved the band scores required by their target institutions, and were not yet
expecting to achieve them. The mode bands of the relevant sub-groups of thePhase 3 study’s student populations were as follows:• band 6 for the scores achieved by those who had already taken the test
• band 6.5 for the bands stated by the students as required, and• band 6 as the band expected.
The table in Figure 6.26 compares the performance of impact study candidates
who had taken the test with IELTS candidates in general.
Figure 6.26 IELTS impact study and IELTS general population
band scores
OverallBand score

Reading

Listening

Writing

Speaking

Phase 3 Study Mean

6.0

5.7

5.9

5.7

6.1

All IELTS Academic Module
Candidates (2002)

5.86

5.79

5.79

5.64

5.94

All IELTS GT General Training
Module Candidates (2002)

5.65

5.43

5.48

5.64

5.79

127

6 The IELTS impact study: main messages
Main points emerging from the table are:
1. That the band scores of the impact study and the general IELTS
populations are close, with the impact study participants scoring slightly
higher across all modules apart from Reading (which, of course, they
have identified as their most difficult module).
2. That impact study test taker candidate band scores for the Reading andWriting modules are slightly lower than for the other modules (again inline with their view of the difficulty of these two modules); their Reading
and Writing scores are also lower than their overall band score, although
all-IELTS mean band scores for Reading are equal third of the five
Academic Module bands (but just the lowest for General Training Module
candidates).
Phase 3 of the impact study did not, it will be recalled (see Chapter 3), haveas one of its research questions the investigation of IELTS scores. Routine
Cambridge ESOL analyses already include performance analyses withreference to Candidate Information Sheet (CIS) data. Given a system
combining routine test validation operations such as those carried out by the
Research and Validation Group (see Chapters 1 and 4) with validation-related
research projects, such as IELTS impact studies and IELTS funded research(see also Chapter 1), it would seem more appropriate to cover wholepopulation studies of relationships between candidate baseline data and test
performance through the routine Research and Validation Group systems (seeWeir and Milanovic 2003 Chapter 2). Closer investigation of particular test
taker characteristics or test washback effects and performance may well be
pursued through narrower washback studies (see Chapter 8).
However, impact studies, with their broad coverage of research questions
and multi-faceted data sources, would appear to add rich data to those used inroutine research and validation operations in answer to such questions.

Impacts on receiving institution administrators
Although a receiving institution questionnaire (see Chapter 4 and Appendix E)
had been developed, validated and was used at Phase 3 meetings with IELTS
administrators, most of the administrative impact data for the study werecollected through interviews. Fuller further impact inquiry is suggested forlater phases in the study of IELTS impact (see Chapter 8).
The Phase 3 administrator interviews covered areas similar to thosespecified in the questionnaires, that is: participant background andinstitutional role(s); IELTS entry cut-off levels and decisions; perceptions of
the test’s validity (including bias) and reliability; the four modules; IELTS
pressures and other impacts; candidate language difficulties; preparation

128Impacts on receiving institution administrators
courses; fees; test information. It will be noted that many of the information
areas in the receiving institution questionnaire are shared with the other IELTS
impact study questionnaires, especially the student and teacher instruments.
A university IELTS administrator interviewed for the study noted that thebroad range of responsibilities in her own brief included the following: test
information, application processing, written, phone and online; test schedules,
centres and rooms; test administration, invigilation and security; examiner
contacts, local marking, fee payments; overseas student advisory support andcertification; liaison with academic departments, and university administrationand finance; co-ordination with Cambridge ESOL and British Council; test
monitoring and staff training. It was clear that useful institutional views onsuch matters were held not only by full-time administrators but also by IELTS
candidates and teachers.

Receiving institution IELTS cut-off points
This key IELTS issue is covered in depth by Banerjee (2003) in her doctoral
research, which is a further example of the range of research approaches
employed in the continuous validation and revision of the IELTS test.
In Phase 3 of the study of IELTS impact, interviews and focus groups werearranged with people who had administrative responsibilities for IELTS. The
contacts took place between June and November 2002, at six IELTS centresin the UK, including two universities, and four language centres or schools. As
already mentioned (see Chapter 4) IELTS administration data were alsocollected through external contacts made by the Cambridge ESOL UK
Development Manager.
Administrators and teachers at a UK university (June 2002) noted variable
IELTS cut-offs across departments and under-graduate vs graduate course
levels. The following pattern was typical:• arts and humanities IELTS cut-off band score of 6 at under-graduate level
vs 6–7 for post-graduate programmes• 6.5 for both levels in the sciences and technology• social sciences: 6.5 to 7
• business: 7 (postgraduate only).
The group commented on tensions between a university’s pressure for higher
cut-off scores to maintain or raise standards and departmental requests tolower cut-off band scores in order to admit students with particularly strong
non-linguistic strengths.
A focus group at a second university (November 2002) identified 6.5 as theuniversity cut-off norm but with departments ultimately deciding their own
IELTS cut-offs, with the flexibility to select according to the claims of129

6 The IELTS impact study: main messages
individual applicants. Teachers in the focus group felt that post-graduate
courses would be particularly difficult for students who had not achieved atleast an IELTS 6 band score. There were reports from our interviews with
university administrators, however, that some institutions were willing toaccept students with bands below six. The IELTS band 5 ‘Modest user’descriptor refers to a ‘partial command of the language, coping with overall
meaning in most situations, though is likely to make many mistakes. Shouldbe able to handle basic communication in own field’. The band 4, ‘Limited
user’, is described as having a basic competence which is ‘limited to familiar
situations’, as having ‘frequent problems in understanding and expression’and being unable ‘to use complex language’.
The mode required band of the IELTS impact study Phase 3 student
population, it will be recalled from the previous section, was also 6.5(compared with expected and already achieved bands of 6). At an IELTS
managers’ and administrators’ meeting (January 2002) the question was raised
whether the test discriminated proficiency levels finely enough at bands 6–7,
given the prominence of receiving institution admission cut-off points aroundthese levels. The November 2002 university focus group, however, felt that
the IELTS appeared already to have been ‘designed for’ students at bands 5–7.
Administrators, as well as language teachers, tended to link the question of
IELTS cut-off points with IELTS preparation course provision. The feelingfrom teacher and administrator interviews at a language school (September
2002) was that students should not enter IELTS preparation courses unlessthey were already at least IELTS band 6 level. It was difficult, according toteachers at a university language centre (November 2002), to provide forstudents at bands 4 and 6 on the same course; students with lower proficiency
levels should, the teachers felt, enter ‘general English’ courses. Some of their
IELTS candidates were assessed as needing intensive remedial grammar andvocabulary courses before their IELTS course, although such students weresometimes unwilling to take these (‘non-IELTS’) courses even though they
were offered by the university.
At the other end of the proficiency scale were reports, from two of thelanguage schools visited, of first-language or ambilingual speakers of English
required, unnecessarily it was perceived, to take the IELTS test. The preparationof such candidates for the test thus often consisted merely of familiarisingthemselves with its format through IELTS specimen materials packs. All the
language centres contacted face-to-face referred to an increasing demand for
IELTS preparation courses as candidacies for both academic and generaltraining modules rose. Given the significant managerial and financial
implications of official IELTS centre status, the opening of new IELTS centres
to meet growing demand, it was emphasised, must be carried out in a fair andsystematic manner.
The IELTS partners now publish IELTS Band Score Requirements (the130Conclusions
2004 edition was compiled by IELTS Australia) providing ‘a sample oforganisations recognising IELTS and their required band scores’ (2004:2).
This names institutions, course types and minimum band score requirements,
although ‘as a guide only’ (op. cit:1).

IELTS administration
The question of IELTS test availability was raised at several of the IELTS
impact study Phase 3 interviews (in July, September and November 2002), thegeneral belief at that time being that students should be able to take the testwhen they wished. A change from IELTS tests arranged by centres on demand
to fixed-date scheduling was implemented in September 2003 to enhance
security, with test papers distributed nearer to test dates and with the use of
unique test versions. This change appeared to have caused much discussion
among IELTS test centres because of the various impacts, for example, room
bookings, administrative and examiner staff commitments, the reduction atsome centres from five to four sittings per month. Also related to test
scheduling was the two-week results turnaround for the IELTS, considered by
one administrator interviewed as an important justification for the IELTS fee.
Some IELTS impact study candidate participants disliked the compulsory
three-month gap before they were permitted to re-take the test. An
administrators’ meeting (January 2002), presumably reflecting the urgency formany candidates of institutional qualifying deadlines, suggested thatcandidates should be able to accumulate band scores on the test skill modules
over a specified period of time rather than having to take all modules at thesame sitting. This would offer the opportunity for those who did not achieve
the required band score in one module to focus on that skill in a retaken test
three months later.
A fairly general administrative point raised at several of the face-to-facemeetings with IELTS candidates, teachers and administrators was that IELTS
information was not as transparent or as user friendly as it might have been. It
seems clear that user expectations for more in the way of information, not only
on practical matters but also on research and validation processes andoutcomes, are rising, in tune, perhaps, with the times (see also Chapter 8).
IELTS handbooks, annual reports and websites (also Chapter 8) might thus beexpected to become more comprehensive and open. This is indeed what has
been happening over recent years as IELTS has grown in use and recognition.

Conclusions
This chapter has presented key messages from IELTS impact study Phase 3
data, collected using the instruments described in Chapter 4, through the1316 The IELTS impact study: main messages
systems outlined in Chapter 5 and attempting to answer the research questions
presented in Chapter 3. The focus here has been on: the profile of the impact
study candidate and teacher participants, IELTS washback on preparation
content, methods, skills, activities and materials; perceptions of IELTS test
fairness, pressures, likes and dislikes; a summary of impact study face-to-facedata on test module difficulty and validity; and receiving institution views on
various administrative impacts of the IELTS test.
The analyses above of impact study data from candidates, teachers andadministrators suggest that, in general, the IELTS test is perceived as:• a reasonable direct communicative performance test of all four macroskills• hard but mainly fair, considered appropriate for use with the candidates inthe study who were heading for under-graduate and post-graduate studies• using authentic and mainly appropriate content, on topics most of which
seem relevant to the target communicative activities, covering a suitable
range of micro-skills• having Reading and Writing modules generally considered the moredifficult of the four modules, partly because of time constraints often
perceived as severe and not necessarily authentic to the target domain
activities• a high-stakes test, causing anxiety though not to an out-of-the ordinary
extent, motivating both candidates and teachers, although sometimes tothe extent of demands for narrower, test-based preparation courses• encouraging some of these courses to motivate and satisfy both students
and teachers because of their focus and the range of relevant activities,
skills and materials inspired by the content of the IELTS test itself.
These are the main conclusions from the data and findings summarised in thischapter.
The chapter has covered selected key areas of inquiry in Phase 3 of the
IELTS study. Cambridge ESOL will now decide on measures to take accountof the findings of the study in the continuing validation of the IELTS test. It is
likely that the Research and Validation Group would advise the Examinations
and Assessment Group, Operations Group, Business Support Group,
appropriate senior management members and IELTS partners of mattersrequiring discussion and decision in the interests of continuing IELTS
validation work. The role of the impact study is to describe and try to explain
rather than to recommend. Chapter 8 continues the discussion of theimplications of the IELTS Phase 3 study within its Cambridge ESOL context.

132

7

The PL2000 Impact Study:
main messages

Following the presentation of main messages from the study of IELTS impact
in Chapter 6, Chapter 7 now summarises some of the key findings from thePL2000 Impact Study, its main messages for stakeholders emerging from
responses to the research questions defined in Chapter 3:• What washback is the PL2000 having on the pedagogy, materials, andmedia for language teaching and learning?
• What washback are changes in language teaching and learning pedagogy,
materials, and media having on the performance and attitudes of thestudents and the teachers?
• What washback is the PL2000 having on language evaluation andassessment?
• What impacts is the PL2000 having on educational managers, including
heads of schools?
• What impacts is the PL2000 having on support for teacher in-service andtest-awareness programmes, and resource centre development?

PL2000 Impact Study profile
Having agreed the purposes, approaches, outcomes and resources needed, anddefined the research questions and action plan for the PL2000 Impact Study(see Chapter 3), then agreed data collection instrumentation and approaches(see Chapter 4) and management (Chapter 5), an Impact Study population was
decided, as follows:• an opportunity sample of seven case study schools with region and school
type strata, in the north, central region, and south of Italy and representing
middle and high school levels, with some contact, also, at elementarylevel• to be seen on two visits (not including the January to March 2001 pilot
visits) to each of the case study schools, at the beginning and towards theend of the school year.
The table in Figure 7.1 summarises PL2000 Impact Study participation datacollection on school pilot visits.

133

7 The PL2000 Impact Study: main messages
Figure 7. 1 Participation on PL2000 Impact Study pilot visits
Participation on pilot visits to casestudy schools (January – March 2001)

Class videos

Headinterviews

Teacherinterviews

Parentinterviews

Technical institute

1

1

3

3

High school

–

1

3

–

Middle school

2*

–

–

–

* sound inadequate for analysis

The table in Figure 7.2 indicates the data collection, participation andscheduling on the main visits to the case study schools.
Figure 7.2 Data collection and participation on PL2000 Impact Study
October 2001 and April 2002 visits
SchoolsClass
videosInterviews, focus groups,
written contact

Types / Dates

10/01 4/02

Heads

Teachers

10/01 4/02 10/01 3-4/02

Questionnaires
Parents

Students

10/01 4/02

10/01 4/02

Teachers
4/02

Elementary

2

2

1

–

–

1

2

–

**

**

2

Comprehensive

1

1

1

–

1

3

2

–

23

24

2

Middle (1)

2

2

–

–

2*

2

–

–

43

42

Middle (2)

1

1

1

1

–

–

6

2

13 13****High school (1)

1

1

–

–

–

4

–

–

18

5

1

High school (2)

1

2

1

1

2

7

2

2

14

16

2

High school (3)

–

–

–

–

3(1)***

3

–

–

–

–

Technical
Institute

1

1

1

1

2

1

1

1

7

10

1

9

10

5

3

11

21

13

5

118

110

11

*
**
***
****

session not videoed
questionnaires not administered because pupils considered too young
private teacher interviewed in addition to three high school teachers
student questionnaires for the two visits were from different students, though from the same classes

Figure 7.3 summarises participations in the PL2000 Impact Study.
7.3 PL2000 Impact Study total participations

TOTALS

Lessons
videoedHeadinterviews

20

10

Teachers
Parentinterviewed etc* interviews
38

21

Student
questionnaires

Teacherquestionnaires

228

11

* these included some teachers of other PL2000 foreign languages (French, German, Spanish)

In the structure of the PL2000 Impact Study Main Report (Hawkey 2003:2–3),
answers to the research questions are sought in the data analysis chapter,
which is structured as follows:
134

2
1

Washback on the PL2000 classroom• washback on the PL2000 classroom– communicative approach to language teaching (CALT)– pedagogy and ethos– lesson planning and management– function and form
– media and modes– books• impacts on attitudes– from the students– from the teachers– from the heads– from the parents• washback on assessment– from the students, the teachers and the classroom– from the school heads and parents– Cambridge exams• washback on performance– PL2000 impacts on exam candidacies and performance• impacts on management and support– management– teacher professional support.
The sections below will follow the pattern of the main report in summarising
the main messages of the Cambridge ESOL PL2000 Impact Study.

Washback on the PL2000 classroom
The Communicative Approach to Language Teaching
The first PL2000 Impact Study research area was washback from the project
on foreign language learning, pedagogy, materials, and media.
The study elicited relevant data from several sources:• its classroom video analyses (see Appendix D)• the Student and Teacher Questionnaires (Appendices G and H
respectively)• teacher interviews (Appendix I)• school head and parent interviews• information which some of the case study school teachers provided inwriting direct to the Impact Study co-ordinator
1357 The PL2000 Impact Study: main messages• UCLES EFL : Bell International PL2000 teacher essay competition
entries.
The first point emerging from such data is that the PL2000 appeared to havetransmitted successfully the message of its aim and objectives in favour of thecommunicative approach to language teaching (CALT). The PL2000, it will
be recalled, targeted ‘the acquisition of a pragmatic-communicativecompetence according to the guidelines of the material contained in the
Common European Framework of Reference for Languages’, and‘communicative competence in reading, written and oral interaction andproduction’ (see Chapter 3). We note immediately that the PL2000 did notoffer new curricula but rather referred to the Common European Frameworkfor Languages and its levels as targets for the Italian school cycles (A1
Breakthrough level for primary school; A2 Waystage for middle school; andB1 Threshold for high school exit).
The case study teachers talked freely and knowledgeably about the CALT.
Typical views, that is views exemplifying comments on the CALT and, in
combination, encapsulating the overall views of the case study teachers on thecommunicative approach, are cited here. They are taken from thequestionnaires completed by the 11 teachers of our case study classes and theircolleagues, and of the 37 teachers participating in teacher interviews, focus
groups or written contacts. The words given in single quotation marks are
direct quotations from the teacher interviews or questionnaires; those without
are taken from the teacher interview summaries:• Teachers are now more aware of the CALT and teach with more
confidence.
• English teaching now concentrates on students’ communication needs;
new students speak the language much better and are more confident
doing so (both from a comprehensive school teacher, October 2001).
• PL2000 changes approaches to emphasise communication, listening andspeaking in particular.
• ‘My attitude has changed a lot; I speak English in class more often than
before. I cover all the four language abilities, listening included, and I
tend to use more enjoyable activities’ (both from a middle school teacher,
April 2002).
• Italian language teachers favour the CALT; previous FL teaching aims
and approaches emphasised gap-filling and reading, not listening or
speaking (two native English speaker high school teachers, October2001).
• Teaching approaches are communicative, so the students communicate inEnglish, not ‘in an Italian way’, which is noticeably different from nonPL2000 students (high school teacher, April 2002).
• ‘CLT (Communicative Language Teaching) is the best way but it is hard’,
especially as teachers should adapt approaches to the needs of different
136

Washback on the PL2000 classroom
students, motivated as well as unmotivated; CALT theory and practicediffer (three high school teachers April 2002).
• ‘New emphasis on communication, on students doing things with the TL,
and learning how to learn, through which foreign language doesn’t
matter’ (high school teacher of German, April 2002).
A further sign of positive PL2000 impact was the agreement of all 11 teachers
from the case study schools who completed the teacher questionnaire in April
2002 that ‘communication skills relevant to students’ language needs’ was a
Project objective which had been achieved ‘very well’ or ‘well’ in their schools.
The teachers’ analysis, in their questionnaires, of activities that wereprominent in their lessons suggested a fairly balanced coverage of the four
skills. The following student activities were all selected by nine or 10 of the
11 case study teachers as ‘very frequently’ or ‘quite often’ occurring in theirlessons:
•
•
•
•
•
•

pair discussions
listening to the teacher talking to the whole classlistening and note-taking
reading books or other materials
writing notes, letters
doing practice exams.

This compares interestingly with the case study student views on their classactivities. The activities perceived by the students as occurring ‘frequently’ or‘quite often’ in their classes were, in order of frequency of selection as
indicated in Figure 7.4:
Figure 7.4 Student perceptions of frequency of activities in theirEnglish classes at school (N=118)
Activities in classFrequently

Quite
Often

Sometimes Never

Individual students
1. listening to the teacher talking to the whole class

35

20

9

2

2

25

24

17

1

reading texts (from books or other materials)

3. writing notes, letters or compositions

18

27

20

2

4. reading then writing answers to questions

22

23

20

2

5. discussions with whole class

20

24

17

5

Two classes (total 42 students) negotiating
block votes*
1. listening to the teacher talking to the whole class2√

reading texts (from books or other materials)

3. writing notes, letters or compositions

√
√√

√

√

√

√

4. reading then writing answers to questions
5. discussions with whole class√√* in two of the case-study classes, the teacher and students worked together to produce joint
responses to some of the student questionnaire items.

137

7 The PL2000 Impact Study: main messages
Note that the students appeared to regard the speaking activities as somewhat
less prominent in their PL2000 classes than their teachers did. There may be amessage here (see Hawkey, forthcoming) on the difference in perceptions
between learners and teacher of the shape and elements of a classroom lesson.
But the students also felt that their English speaking skills were the mostimproved over the school year. Of the 161 responses to the questionnaire item
concerned in April 2002 (some students specified more than one ‘mostimproved’ skill) the following selections were made:Figure 7.5 Student questionnaire responses on the most improved
skill(s) over the 2001/2002 school year
Reading
29

Listening

Writing

Speaking

25

51

56

A significant majority of students perceived their productive skills as
improving more than their receptive skills. Yet both the case study teachersand, even more strongly, the students, saw ‘the teacher talking to the wholeclass’ as the dominant language classroom activity (see further below).
Cross-checking with the entries to the 2001 UCLES EFL : Bell International
PL2000 teacher essay competition entries, on the topic ‘What the Progetto
Lingue 2000 means to me as a teacher’, we find that the competitors appear to
support both the content and tone of our other data on the CALT. One candidate
calls her chosen methodology ‘functional-notional-communicative’ and enjoys
her students’ very motivating realisation ‘that they could express their ideas
and wishes in English’. Another sees the PL2000 approach as offering ‘a riskfree environment’ and ‘improved teacher attitude and interesting materials’.
The same teacher, however, warns against the ‘students-need-to-talk
syndrome’, that is ‘an excessive emphasis on oral-aural skills realised in pairor group work with the exclusion of individual tasks’.

Pedagogy and ethos
Here, we refer primarily to the ways in which the teachers (and the learners)sought to achieve the communicative objectives signposted by the PL2000.
The most direct evidence on PL2000 classroom pedagogy was gleaned
from the 20 sessions of closely observed and video-recorded classroom
language lessons in our case study schools, analysed (see Chapter 5) using theclassroom observation analysis instrument (Appendix D) shared with Phase 3of the IELTS impact study. The PL2000 Impact Study Main Report suggeststhat ‘there is good evidence in almost all the lessons analysed’ for the study
that the CALT did indeed motivate ‘teachers to seek teaching/learning

138

Washback on the PL2000 classroom
activities that were relevant to the communicative needs of their students’(2003:32). Examples of these from the video-recorded lessons, in rising orderof the ages of the students concerned, were:• middle school, Common European Framework (CEF) Level A2 students,
invited by their teacher not only to exchange personal information with
their friends, in pairs, groups and in informal presentations to the wholeclass, but also to amuse their colleagues (and themselves), as 13-year-olds
often like to do, with flights of imagination and humour, including, in theobserved lesson concerned, stories of intelligent, tennis-playing tropicalfish, and living as tramps• high school CEF Level B2 students (16–18 years old) working in pairs tocompose e-mails to their English teacher suggesting the kinds of English
language learning activities they considered would meet their needs ontheir course• CEF Level C1 (18-year-old) students, working in small groups, discussing
and writing an action plan for the multimedia promotion of a local
product.
Yet, as suggested in Chapter 6, the idea of increased communicative
opportunities for learners may appear to be contradicted by the fairly frequent
reference in the classroom observation video analyses to teacher
communicative dominance. In 11 of the 20 case study lessons analysed andobserved in October 2001 and in April 2002, there is reference to teachersinterrupting or supplying words and ideas to fill silences, limiting time andopportunity for learners to learn and feel their way towards coping with
communication in a foreign language, thereby perhaps reducing learning
impetus.
The data under our ‘pedagogy and ethos’ heading here are clearly relevant
to both the first and second research questions of the Impact Study, namely:
What washback is the PL2000 having on the pedagogy, materials, andmedia for language teaching and learning?
What washback are changes in language teaching and learning pedagogy,
materials, and media having on the performance and attitudes of the students
and the teachers?
The data discussed here also connect with the next issue, that of theplanning and facilitation of lessons.

Lesson planning and management
There are comments in the classroom analyses which suggest that restrictions
on learner communicating time may be the result of lesson planning andmanagement problems as well as teacher reluctance not to intervene. Stated or139

7 The PL2000 Impact Study: main messages
implied in comments on six of the 20 analysed lessons was the suggestion that
their perception of the CALT might have induced in some teachers a changed
view of the importance of lesson planning. In observed lessons where there
were signs of somewhat unproductive disorganisation, it appeared that theteacher might have had an appropriate main idea for a lesson (as in examples
such as a relevant role play or simulation, a spidergram on a currrent topic, asurvey questionnaire) but without necessarily considering the procedures,tasks or activities that might best ensure that learners (and teacher) were able
to make the most communicatively of the lesson planned round the main idea.
Where planning for useful communication by learners appeared to belacking, problems such as the following tended to arise:• teacher dominating talking time, thus too little opportunity for students tocreate their own interactions (in 10 of the lessons)• students left inactive and, possibly, bored (9)• lesson fragmentation into too many short episodes caused by changes ofpedagogic direction in a sometimes vain attempt to find paths (unmappedin the lesson plan) to more effective communication by learners, and/or,
relatedly, stop-start episodes, repetition of activities and frequent resort tofairly random question and answer sessions (6)• an absence of learning round-ups, reinforcement, consolidation (4)• reduced educational value added (4)• long and sometimes confusing task organisation explanations (3).
The penultimate problem above, perhaps less self-explanatory than theothers in the list, seemed sometimes to emerge in the form of mundane andpredictable comment and topic references inserted by teachers to or about thestudents, when communicative lessons had lost their way through lack ofadequate planning.
In eight of the 20 PL2000 Impact Study language teacher interviews andfocus group discussions, it was agreed that lesson planning, far from being lessimportant once teaching approaches moved towards an emphasis on learnercentred communicative opportunity, was still crucial or had become evenmore so. An experienced English teacher responsible for the co-ordination of
PL2000 activities at her high school, summed up this view with her remark
that, given the communicative construct and the aim of facilitating activities
in line with students’ communicative needs, particularly careful lesson
planning is needed to create the conditions for real communication. Related
points made by other teachers were:• Classroom and lesson management are harder and more time-consumingwith the CALT. With the emphasis on speaking, it is easy to lose controlof the lesson (A2 teacher, April 2002).
• ‘We’re not really scientific in planning our lessons; we’re not exact in our140Washback on the PL2000 classroom
lesson planning or our evaluation’. Lesson preparation is about ‘what wedo and say’ (middle school teacher of A2 level students, April 2002).
• Team teaching is encouraged but successful teamwork and sharing of
responsibility needs careful planning (two English mother tongue teachersof B2 level students, October 2001).
One of the essay competition candidates suggested that one washback of thePL2000 on teachers was ‘[f]orcing us to view the planning and assessment
stages of the process as extremely important parts of our teaching, as steps
which require far more serious consideration than they have ever had’.
There was strong evidence from the Impact Study all round, in fact, thatlesson planning and management were a crucial aspect of the implementationof the communicative teaching and learning approaches advocated by the
PL2000. Both lesson planning and lesson management thus appeared to invite
prominent attention on language teacher professional programmes set up inconnection with the PL2000 and related initiatives.

Function and form
Also relevant to the research question on the washback of the PL2000 on
language teaching and learning, were data on the balance between language
functions, the communication of meaning, fluency and appropriacy, andlanguage forms, correct language usage, and accuracy. These, of course,
represent continua conventionally referred to when the communicative and thestructural approaches to language teaching are compared (see, for example,
Cook, 2003:31–35).
A tendency was noted in the observed PL2000 Impact Study lessons forteachers to correct some pronunciation, grammar and vocabulary errors(though not others) while students were in the process of communicating. Thismay be acceptable, of course, in the interests of maximising communicative
opportunity, as discussed above. But inconsistency in error correction wasnoted with regard to the communicative impact of errors, the stage ofcommunication, and the level and confidence of performance of the students
concerned. There was also little evidence of systematic consolidation,although this may have come in subsequent lessons. The occasional andfleeting treatment of error does, of course, carry the risk that students’ oral
communication may become inaccurate even to the point of impeding
meaning. There is comment in 10 of the 20 PL2000 Impact Study classroom
lesson analyses on the apparent lack of a consistent error correction approachor on the prominence of unexpected errors.
Yet Figure 7.6, summarising teacher and student questionnaire data for
April 2002, indicates that grammar and vocabulary practice activities,
normally associated with a focus on form and accuracy, as well as function,141

7 The PL2000 Impact Study: main messages
figured quite prominently in the students’ perceptions of their English
language lessons.
Figure 7.6 Perceived prominence of grammar and vocabulary
exercises in the case study school lessons
Individual students
Grammar exercises
Frequently
24

Vocabulary exercises

Quite often

Sometimes

Never

Frequently

Quite often

Sometimes

Never

19

20

2

9

24

22

9

Two classes (42 students) negotiating block votes
Frequently

Quite often

Frequently

Quite often√√√√
Case study teachers

Grammar exercises
Frequently
3

Vocabulary exercises

Quite often

Sometimes

Never

Frequently

Quite often

Sometimes

Never

3

5

0

3

6

2

0

The teacher interviews suggest conscious attempts to balance fluency andaccuracy work at all levels, 10 of the sessions including comment on the roleof grammar in the teachers’ classes. A B2 level case study teacher who
appeared very ‘communicative’ in the theory and practice of the pedagogy in
his observed lesson, felt nevertheless that, on his and other English courses at
his school, ‘[t]he students are getting quality instruction in language grammar,
syntax and structure’. A group of high school teachers agreed: ‘CLT
(communicative language teaching) does not mean no attention to grammar;
all approaches are acceptable as long as they improve students’communicative competence’. The same teachers six months later confirmedthat ‘CLT has to be combined with more structural approaches’. A middle
school teacher summed up the form/function relationship as follows:
There is a need for a balance between allowing students to make mistakes
in English communication, but not accepting everything; some correctionof grammar and lexis errors is necessary; if you don’t know the code,
you’re in trouble.

Once again the message from the essay competition entries confirms ourobservation on the accuracy/fluency question. One candidate noted therelatively poor level of listening comprehension of students who had beenlearning their English through traditional methods without really using thelanguage, adding that they might nevertheless have only a basic grammaticalcompetence, and a very limited vocabulary. Another candidate suggested thebrainstorming of lexis as a remedy to vocabulary problems, a third the use of
CD-ROM stories to improve weak understanding and pronunciation.
142

Washback on the PL2000 classroom
The whole accuracy/fluency question remained crucial to the achievementof the PL2000’s aim to enable students to meet their needs as specified by theappropriate CEF proficiency level statements. These, remember, include
A1–C2 level can-do statements for both grammatical accuracy andvocabulary control (ibid. 2001:112–114).

Materials, media, modes
The PL2000 project descriptions referred to the integration of information
technology into the teaching and the ‘use of diversified materials in place of
textbooks’. Eight of the 23 English language teaching textbooks specified by
the case study schools on the school profile form had the name of an external
exam in their titles (e.g. FCE Gold, PET Practice Plus), such tests being
among those approved by the Ministry for the external certification of students
on PL2000 courses. The remaining textbook selections in the Impact Study
data indicated a preference for books taking a fairly communicative approach.
However, teachers tended to use additional materials from a variety of othersources rather than a single coursebook, for example cut-out photographs,
self-designed spider-grams, information-gap hand-outs, audio-cassettes, wall
charts and, in four of the 20 lessons, external exam practice cassettes, papers
or photographs.
The following are the comments on textbooks from the teacher interviews.
Being teachers, our case study interviewees and focus group members tended totalk about language teaching books in terms of their pedagogical context of use:
• New and improved textbook materials are better designed forcommunicative activities such as letter writing, written dialogues … A
textbook may be used as the main guide to a lesson plan, with the teacher
adding material (often from other textbooks), usually because one
textbook does not contain enough practice materials relevant to particular
students’ needs for the exams (from an interview with two middle school
teachers of A2 level students, April 2002).
• Textbooks are lacking in up-to-date ideas re students’ actual target
language activities; ‘so I use videos, BBC, photocopies, examples ofAustralian and US culture (not only UK), pop song lyrics’ (middle school
A2 class teacher, April 2002).
• The good new CALT textbooks can be useful in providing ideas, starting
points and material; teachers must pick, from a range of books, what issuitable for their students. … Old coursebooks were artificial (culturally
for example); new books use everyday language and real situations (C1technical institute teacher, April 2002).
• CALT coursebooks and workbooks have useful exercises and activities,
including attention to grammar (PL2000 German teacher, April 2002).

143

7 The PL2000 Impact Study: main messages• Past exam papers, videos and selected coursebooks are used on theexternal certification courses … Past papers are useful but, because they
are international, lack a national (Italian) cultural relevance (case study
high school teacher and PL2000 course organiser, April 2002).
Though clearly an element in a teacher’s pedagogical resources, information
technology appeared as a separate project recommendation in the PL2000
guideline document, which referred to ‘the integration of information
technology into the teaching and use of diversified materials in place of
textbooks’. Of the 20 lessons video-recorded, observed and analysed for the
PL Impact Study, five, from two of the case study schools, both in B2 classes,
used information technological support, including the following:• personal and school computers used for student >< student and student>< teacher e-mails• computer laboratory use for word-processing; Internet picture-viewing
and discussion; video viewing and discussion; videocam/videoconferencing language practice; students’ own test construction.
More traditional audio-visual support materials in the observed classes
included:• sample listening test cassettes for external exams (seen in use at A2 level)• jigsaw listening practice cassettes (A2)• cassettes of chants and stories (professionally recorded as support
materials for a textbook) designed for use at elementary level (A1).
In the teacher interview reports, references to IT use included bothconventional and newer technologies, and often, as the PL2000 guidelines
suggested, made the connection with other language learning and teachingmeans, for example:• ‘The computer is a perfect lingua vortex. With a few modifications andadministrative systematising, [our computer lab] could become aninternational model cyber-seminar room’ (B2 level teacher at a highschool, October 2001).
• ‘Setting Internet search tasks and creating CDs with students are usefulcommunicative activities’ (teacher at a high school focus group, April
2002).
• ‘We try for the optimal use of multimedia, e.g. computers, video, theInternet’ (B2 level teacher, April 2002).
Like other language teaching media, of course, the use of information
technology needs the appropriate rationale, planning and management.
Certain of the uses of computers in the observed lessons seemed somewhat
less authentic and practical than the same activity might have been without
their use. Nevertheless, most of the IT examples in the observed lessons
144

PL2000 impacts and washback on attitudes
suggested positive washback from this area of PL2000 recommendation andsupport.

PL2000 impacts and washback on attitudes
The quest for information on the impacts of the PL2000 on the attitudes of key
participants in the project sought answers to one research question inparticular, namely: What washback are changes in language teaching andlearning pedagogy, materials, and media having on the performance andattitudes of the students and the teachers?

From the students
The nature of our case study students’ motivation for learning English wasinvestigated in both the October 2001 and the April 2002 student
questionnaires. In October, the students (n=118) responding to an open-ended
item ‘What are your opinions about learning English?’ focused mainly on theirreasons for learning the language, with:• 51 references to the status of English as an international language• 48 to its use in work
• 36 to their ‘future’ (which, one would assume, may well also beconnected with their professions)• fewer than 10 mentions each for: travel, university study, and a liking forthe language or its culture.
Figure 7.7 summarises responses to the April 2002 student questionnaire item,on students’ ‘main reason for learning English’.
Figure 7.7 Students’ main reasons for studying English (from Student
Questionnaire, April 2002, n=110)
Reason selected*

Numbers of selections

To get a better job

49

To communicate with more people

48

For travel overseas

37

Because I like it

25* Some students selected more than one of the reasons given; all selections were counted.

These figures again indicate the perceived importance of the English language
as a means of international communication and of enhancing professional
opportunity. There may also be an indication of rather more integrative
motivations, in the ‘because I like it’ and in the overseas travel figurescollected on the April 2002 case study school visits.
145

7 The PL2000 Impact Study: main messages
In their October and April questionnaires, the students were also asked torate the frequency of their activities in English outside class. Figure 7.8
summarises the responses.
Table 7.8 Summary of Student Questionnaire responses on the use ofEnglish outside class.
Out of class
Activities

Responses in
October 2001

Responses in
April 2002

Never

Almostnever

Occasionally Often

Reading books

43

16

24

Reading newspapers

38

30

Reading magazines

33

31

Reading newspapers,
magazines–

Writing letters
Radio

Never Almost Occasionally
never

Often

13

28

16

55

8

22

2

–

–

–

–

14

11

–

–

–

–

–

–

–

52

28

21

7

32

36

12

14

11

19

41

27

38

23

15

20

35

22

23

19

TV

13

26

36

20

17

19

38

16

Movies

40

26

22

8

29

41

26

13

Going to shows

37

27

15

3

57

18

8

3

Using email

22

34

16

18

46

21

22

20

Using the Internet

30

19

13

35

20

25

21

44

Talking with visitors

30

20

27

8

30

30

39

8

Apart from the general picture of relatively limited activity in English outsideclass for most students, there are some indications from the questionnaire
responses on the October 2001 and April 2002 visits, that students wereincreasingly often participating in English in:• reading books• using the Internet, and• writing letters.
These tentative indications recall a comment by a teacher at interview in April
2002: ‘Students read and write less nowadays in their real lives; language
teaching should reflect their actual reading and writing activities’.
More inferences will be made about the students’ motivation and attitudes
below, when their perceptions of their progress in English over the
2001/2002 school year are analysed. As suggested in Chapter 1, effects of aprogramme or test on student (and teacher) attitudes appear to span thewashback/impact divide. Perhaps effects on attitudes in learning andteaching contexts belong to washback, while broader attitude changes might146

PL2000 impacts and washback on attitudes
more appropriately be attributed to impact. But the distinctions in this area
are by no means certain.

From the teachers
In general, the above case study student data on motivations and attitudes inrelation to the PL2000 were supported by the perceptions of the 11 case study
teachers, as expressed in their April 2002 questionnaires, for example:• More motivation for students and teachers (elementary school Frenchteacher).
• ‘More interesting: children and teachers improve their use of the foreign
language’ (elementary school English teacher).
• ‘More motivation, more involved in the lessons, more variety of lexis’(middle school teacher).
• ‘We prepare students, in a limited number, of course, good students. They
are more stimulated, they are more involved in their activities’ (middle
school teacher).
• ‘Students are gaining motivation, extra hours, interest for languages’(high school teacher).
• ‘Students’ performance improves especially orally; students show moreinterest in different languages’ (high school teacher).
• ‘Students are extremely enthusiastic; their speaking/listening/reading
abilities have greatly improved’ (middle school teacher).
The case study teachers also appeared positive on the impact of the PL2000 onthemselves. In addition to the apparently enthusiastic ways in which they
responded to the methodological approaches implied by the project (seeabove), the following points from the teacher interviews suggest that PL2000
had a healthy impact on teacher as well as student attitudes. Note therelationships inferred by the teachers between motivation, clearcut aims andnew approaches:• PL2000 means that ‘teachers know where they are going’; previously FL
teaching objectives were vague, the emphasis on the academic aspects ofliterature, with the basics of language forgotten.
• Teaching is now more rewarding because you can see the results.
• PL2000 emphasis on communication is appropriate and ‘more challenging
for the teachers’ (all from the high school teachers’ focus group, April
2002).
• PL2000 appeals even to language teachers not involved in it; someteachers talk of backwash from PL2000 to non-PL2000 courses (middle
school English teachers, October 2001).

147

7 The PL2000 Impact Study: main messages
There were, of course, extra pressures from the PL2000, as from any such
radical educational reform. The following three less positive comments fromthe teacher interviews are indications of this, again to be taken up below.
• Language teachers are sometimes de-motivated by the extra
administrative work entailed by PL2000.
• There may be resistance from other subject teachers to the scheduling ofPL2000 afternoon classes (high school teachers’ focus group, October2001).
• Difficult to motivate all the students in a mixed-ability class; ‘It is
important for a teacher to stay with the same students over more than a
year’ (middle school teacher, April 2002).
The overall impression conveyed by the PL2000 teachers contacted throughour study was that they were well motivated by the Progetto and keen to
pursue its aims and processes in a professional way. It should be mentioned
here that the attitudes of the case study teachers to the PL2000 Impact Study
itself was unfailingly positive and helpful.

From the heads and the parents
This section moves the data analysis and findings on to the next researchquestion, namely: What impacts is the PL2000 having on educational
managers, including heads of schools? Given that case study school heads
referred regularly to the attitudes of parents to the Progetto, these are also
covered here. Note also the evidence of developing views as the interviews
enable the comparison of opinions over the longitudinal span of the PL Impact
Study, across the March 2001 (pilot visit), October 2001 and April 2002
interviews.
The heads of the seven PL2000 Impact Study case study schools took an
appropriately broad view of the significance of the PL2000, with a focus on
the project’s impact as well as its washback, to use the terms as defined in
Chapter 1. The summaries of typical comments here are taken from 10
interviews with the case study school heads, conducted in Italian, withsimultaneous interpretation in English and subsequent notes made using thevideo-recordings and discussions with a bilingual interpreter.
• Very positive [impact] since the project highlights the EU and the needfor foreign languages for communication, and an increase in students’
enthusiasm, pride in their foreign languages, willingness to interact inthem, and understanding of their importance (elementary school head,
October 2001).
• Foreign languages are important as borders and barriers have fallen andcommunication is facilitated between European and non-European

148

PL2000 impacts and washback on attitudes

•
•
•
•
•
•
•
•

•
•

countries. The students have more opportunities of communicating; aknowledge of foreign languages is fundamentally important for students
completing secondary education as well as for continuing education at
higher levels.
The impact of PL2000 has been extremely positive, participation in it
being necessary to reinforce the position of the school locally and more
widely.
PL2000 also helps the school to achieve a greater degree of openness.
The parents seem happy and understand the objectives of PL2000
language teaching (all four from the head of a comprehensive middle
school, October 2001).
PL2000 is in line with pressure from parents who want to enrich thestudents’ educational experience.
PL2000 impact is positive in terms of motivation for the students. Thus itis good for the school, but time is needed before final judgement can bemade (from the head of a second middle school, October 2001).
There are now more students (and parents) wanting to participate in
PL2000 courses, numbers rising from 31 to 50 over the year (same head,
April 2002).
Teachers are enthusiastic, prepared to work beyond the objectives theschool sets; the school’s and PL2000’s objectives are the same! (head of atechnical institute, March 2001).
The training has improved teachers’ English, given them different kinds
of English (for different purposes), brought conferences, films,
discussions and debates in English, made them better teachers. So the PL
has given the teachers the opportunity to broaden their knowledge, use
better materials, made them more aware as teachers (same head April,
2002).
PL is accepted at all school levels in this region. The students are now used
to PL methods, so progress quickly (high school head, October 2001).
The PL is great from the teacher’s point of view given the training andsatisfaction they get out of this kind of project. School will haveconferences and meetings open to locals, to discuss projects of this kindand get suggestions from them (same head, April 2002).

The heads also refer to some typical difficulties with the PL2000, as an
educational reform:• Good parent/school relations, but, of course, it is the most motivatedfamilies who send students to the afternoon language learning activities,
about 10–15%. Not necessarily the best students, but the mid to highability ones. The students really needing help do not come; their families
tend not to trust the schools (head, middle school, October 2001).

149

7 The PL2000 Impact Study: main messages• The school is determined to expand and enrich the PL2000 programmes
in English and French to attract students (and parents) who are not sohighly motivated as the initial groups (same head, April 2002).
The PL2000 Impact Study held interviews or focus groups involving 21
parents with children at elementary, middle and high school levels. The
parents concerned spoke both of the impacts of the PL2000 on their children
and on themselves. Their comments were overwhelmingly positive, whichmight, of course, be expected since it was more likely that parents withpositive views would be called upon and accept invitations from the schoolheads to be interviewed for the Impact Study. Parents’ comments focused
mainly on the way English is or should be taught and learned, or on the broadvalues of competence in the language and on English as a tool for children’s
academic or working future.

PL2000 impacts on assessment
From the students, the teachers and the classroom
This section responds to research question 3, namely, What washback is thePL2000 having on language evaluation and assessment?
One of the key PL2000 innovations was that ‘[i]n addition to the internal
evaluation which is an obligatory part of the school system, the project will
permit the introduction of external certification by virtue of a protocol ofintent ratified by the Ministry and examination boards recognised
internationally in the four languages studied (English, French, German andSpanish)’. The rationale for this radical change was that ‘[e]xternal
certification represents an added value in quality of ‘transparent credit’ which
may facilitate the re-orientation of students in their educational programmes(the transition from one academic level to another or from a formal system of
instruction to professional education) constituting a negotiable certification in
the professional world and across national borders’. (Ministry of Education,
Italy, 1999, translated.)
The PL2000 Impact Study collected evidence and views on the impact ofexternal (and internal) exams, and teaching/testing relationships.
The Student Questionnaire returns in April 2002 are fairly clearcut on how
students felt about external exams, as Figure 7.9 indicates, a strong majorityof the case study class students suggesting that the prospect of external exams
at the end of their PL2000 courses motivated them to work harder.

150

PL2000 impacts on assessment
Figure 7.9 Summary Student Questionnaire responses on attitudes toexternal exams
My main feeling about taking external exams in English is they
are more difficult thanschool exams

help me to work harder

are for listening andspeaking

are necessary to enter
university

12

54

13

16

Responding to their questionnaire item on the achievement at their schools of
the PL2000 objective of the ‘use of external language exams at recognised
Council of Europe levels’, 10 of the 11 case study teachers rated this objectiveas ‘very well’ (5) or ‘well achieved’ (5).
The case study class observations raise the question of the relationshipbetween external exams and the teaching/learning activities that occurred incourses at the end of which an external language was to be taken, in otherwords external exam washback (see Chapter 1 and the discussion of the issue
in connection with the IELTS test in Chapter 6).
The case study classroom observations may help us answer this complex
question. The communicative tasks and activities in a lesson taken by a
PL2000 class of A2 level middle school students in April 2002 aresummarised in Figure 7.10 (page 152). The lesson described here using theform of classroom observation analysis discussed in Chapter 4, is typical of
PL2000 lessons implemented with a clear awareness of the aims of both thePL2000 and the target external exam, yet with some resistance to too direct a
relationship between the test and the learning.
The target exam of the course was the Cambridge ESOL Key English Test
(KET) a Common European Framework (CEF) level A2 exam, at the upper of
the two ‘Basic User’ levels. In the CEF global scales, A2 describes a user who
can ‘communicate in simple and routine tasks requiring a simple and direct
exchange of information on familiar and routine matters’ (2001:24). The KET
exam consists of three papers: Reading and Writing; Listening; and Speaking.
The KET Speaking test (the closest to the activities analysed in the observed
lesson summarised in Figure 7.10) assesses candidates’ ability to interact inconversational English with an examiner and with another candidate. Students
need to be able to answer and ask questions about themselves and talk freely
about their likes and dislikes. The Speaking test is normally taken in pairs(www.cambridgeesol.org/exams).

151

7 The PL2000 Impact Study: main messages
Figure 7.10 PL2000 Impact Study lesson taken by A2 students in
April 2002; observation analysis
No. Episode1

Timing Activity, participation, materials Comment(mins)

Questionnaire
7completion practice:task preparation

3

2

Pair interviews

103

Report-writing

2

8Pre-prepared worksheets given out. T uses target language (TL)
for classroom language, with
Teacher (T) explains thequestionnaire form and format, and a little L1.
how to use for pair ‘survey’ work. T encourages students to use
TL.
Examples given of A and B
student sheets, A a questionnaire
sheet on reading habits, B a selfevaluation questionnaire on thesame topic.
Students study sheets silently.
T monitors and helps.
Students work in pairs, A students
asking Qs and completingquestionnaires,
B students answering questions on
their reading habits.

Students use L1 a lot in pairs.
Quite a lot of silences.
T uses TL mainly whenmonitoring.
Some good pair interactions.

T has to explain quite basic
T explains report-writing task:
words e.g. ‘good’.
A students to write report ofquestionnaire findings,
B students to write a selfevaluation of own reading habits.
Report-writing begins (completing
pre-prepared information frames).
T monitors.

4

Report back9

A students read their reports aloudto class.
B students read out their self6 students get to present.
evaluations.
T adds questions to individual
reporting students.

5T >< student
Oral question andanswer4

T asks questions of individual
students re. the questionnaires andself-evaluations.

Most students questioned.

Students seem rather more confident than in October 2001? T has positive and sympathetic attitude.
Perhaps helps and intervenes too much. Not much intensive student communication?

The relationship between the CEF level, the KET exam and the A2 class lesson
would seem positive here. The learners were participating in several
communicative activities, listening, speaking, writing and reading, which they mostprobably considered relevant to their own communicative needs and life
aspirations. These activities also coincided with their CEF target level and therequirements of the external exam due at the end of their course. But the exam was
not mentioned, and the materials for the lesson were by no means just practice tests.

152

PL2000 impacts on assessment
Yet PL2000 Impact Study data do also indicate a tendency for external
exams to be regarded as, in a sense, the syllabus and the target criterion for a
PL2000 course. It was commonplace during the study to hear school heads,
teachers and students refer to ‘the PET (KET or FCE) course’; or to see
curriculum specifications expressed in exam preparation terms. Making surethat students are prepared for the formats and conventions of external exams
is, of course, an important and natural part of a course leading to an exam, as
discussed with reference to IELTS above. School language courses that are
simply exam cramming, however, causing ‘curricular alignment’, that is anarrowing of curricula to fit closely and exclusively with the demands of a test(see Hamp-Lyons in Chapter 1), would not be in the spirit of the needs andtask-based communicative approaches suggested by the PL2000. It was
encouraging in this respect that 17 out of 20 of our observed lessons were notexplicitly exam-oriented, but appeared rather to be attempting to involve thelearners in communicative activities appropriate to their English language
levels, needs and wants. A middle school teacher noted in his April 2002
interview that the washback of a test is positive if the test and the teaching
have common aims. But the overlap relationship can be complex, we alreadyknow. Teachers’ perceptions of the route to the achievement of teaching aims
may vary.
Balancing the demands of an exam with the communicative needs oflearners remains a key washback issue, one on which the data from ourteachers and head of school interviews revealed further insights, many ofwhich could inform teacher professional support programmes.
At the teacher interviews and focus groups, the issue of external exams wasraised frequently, often with regard to the status and effect of external andother exams, and to the teaching/testing relationship. The responses typified
here reveal a mature and generally positive view of external foreign language
exams:• ‘Teachers were surprised at first, at the idea of external certification. They
didn’t think it suited their teaching. But external certification is importantas it means that students can be checked for their level of competence at
any stage, from A1 to C2. Teachers are used to the idea now.’ (Teacher ofGerman and PL2000 official, February 2001)
From the teacher and her English first language colleague teaching at C1 leveltowards the Cambridge ESOL Certificate in Advanced English (CAE) exam,
comments (in October 2001) bring a useful perspective to the teaching/exam
relationship question:• Exams lead in the right direction, use lively, up-to-date everyday
language, in line with PL2000.
• ‘External certification increases student motivation.’153

7 The PL2000 Impact Study: main messages
But, at the same time cautioning:• Course targets and lengths are set according to the final, external test.
• ‘Students know the exam aims better than PL2000 aims.’
A case study teacher at a middle school, interviewed twice, expressed thefollowing enlightened views on external exam impact and the teaching/testing
relationship:• There is a close teaching/testing relationship but teaching should never be‘just for the test’.
• External exams are an important motivation through their international
credibility but should not be emphasised too much.
• A test must not be a threat to the students (October 2001).
At the end of the school year, in April 2001, the same teacher had this to say:
• Don’t teach just for a test; get ‘the test objectives and the student learning
objectives to work together’ in a programme; an external test should be‘motivation but not a threat’.
• Where a test represents real-life communication, it is good; this is notalways the case, though, e.g. tests with tasks from contexts (e.g. roadsigns) students are not familiar with … Target language letter-writing
may not be a realistic communicative activity for students who write only
notes in real life.
• Exam preparation is important but teachers are given the freedom toapproach this in their own ways.
Then, accepting, perhaps, the added validity and reliability that should be
expected from external certification, the teacher added:• It is harder for teachers to evaluate students ‘scientifically’ with large
classes, mixed abilities and some less able students.
A case study high school teacher responsible for organising the PL2000
programme at her school wrote, in March 2002, of the instrumental motivationfor external exams that had to be taken into account:• ‘Parents encourage students to take external certification, as giving usefulcredits at the end of the year and facilitating university entrance orEnglish course exemption.’
In April 2002, the same teacher, in her interview, raised the important questionof the other tests on PL2000 courses with an external final exam:• Placement, progress and mock tests are important on a course leading to
an external exam.
• Next year it is planned to make the external tests less of an end inthemselves; ‘remember that it takes time for the levels of the CEF to havetheir full influence on planning and teaching’.
154

From the school heads and the parents
A teacher at the same school agreed, in March 2002, with the use of external
exams:• ‘Earning a benchmark of proficiency gives students something to strive
for and motivates language acquisition.’
From the classroom observations, the students and the teachers, we gained
important insights into the influences of external exams on learning andteaching. The message so far appeared to be that external certification was
welcomed by the learners and their teachers, and that much thought was going
into ensuring that the washback was positive because the right teaching/testing
relationships were being developed. Implied in the comments cited above, as innearly all the discussions of PL2000 aims and their achievement, was a good fit
between PL2000 English language communicative targets, the Common
European Framework levels and the external exams. PL2000 levels were, it will
be recalled, expressed in explicitly CEF level terms. And Cambridge ESOL
would claim both theory-based and empirical evidence for ‘a coherent link’(Taylor 2004:2) between their Main Suite of exams and the CEF, as establishedthrough the ALTE Can Do Project (Jones and Hirtzel 2001).
Growing interest in 2002 in the European Language Portfolio (ELP),
which, with its ‘format in which language learning and interculturalexperience of the most diverse kinds can be recorded and formally recognised’(Council of Europe 2001:5) was consolidating the links between PL2000/CEF
levels and the recognised external exams. Chapter 8 reports further on this in
the aftermath of the PL2000.

From the school heads and the parents
For the Heads of the Impact Study schools, the selection of exams from theapproved external examination boards had implications, both educational andmanagerial, of course. From our ten interviews with the heads, we gained
insights into both.
On the March 2001 PL2000 Impact Study pilot visit to his high school, one
head provided the study with a thought-provoking view of testing. He was, hesaid, ‘trying to teach students to view the exams as a kind of research into theirown knowledge and ability, a sort of self-evaluation rather than a check linked
to promotion or other rewards’. The case study school heads also commented
on the positive effect of good Cambridge ESOL exam results on performancein state exams, the apparent improved student confidence and motivation
resulting from their taking external certification through PL2000.
The focus of almost all the comments on external exams from our 11
interview and focus groups with 21 parents was on the greater internationalrecognition of such exams. These were seen as ‘good for their work in Italy
and abroad’, or offering ‘wider work opportunities in the EU and beyond’, tocite typical parent interview responses.
155

7 The PL2000 Impact Study: main messages

PL2000 washback on exam candidacies andperformance
We now return to the research question: What washback are changes inlanguage teaching and learning pedagogy, materials, and media having on theperformance and attitudes of the students and the teachers? The focus here isevidence of impact on performance from both a broad and a case study
specific perspective.

Exam candidacies
It is clear that the PL2000, with its advocacy of the use of the tests ofapproved examination boards to provide the added value of internationallyrecognised certification to students’ foreign language qualifications, waslikely to increase the number of external foreign language exams taken in
Italy. Figure 7.11, giving official Cambridge ESOL PL2000 exam and total
candidate numbers from 1998/99 to 2001/02, showed this indeed to be thecase.
Figure 7.11 PL2000: all Cambridge exam entries 1998/2002
80000

PL2000

70000

All

60000
50000
40000
30000
20000
10000
0
1998/99

1999/00

2000/01

2001/02

Cambridge exam candidacies for Young Learners English Tests (YLE), KET,
PET, FCE, CAE and CPE, rose by 54% in the 1999/2000 academic year, by
71% in 2000/01 and by 36% in 2001/02, to a 2002 total of 71,953. In 2002,
PL2000 entries accounted for 68% of entries at A1 level, 81% at A2, 64% at
B1, the level with the most Cambridge exam candidates, and 49% at B2 level.
The PL2000 candidatures at C1 and C2 levels were still few, presumably
because most school age students did not take exams at that high CEF level156

PL2000 washback on exam candidacies and performance(although several students in our case study group did take the CAE exam at
C1 level).

Exam performances
It should eventually be possible to see how learning English on PL2000
courses actually affected student exam results in the long term. The PL2000,
which ended as a formal Ministry project in 2003, certainly provided
interesting insights into learning processes and attitudes as students moved
from one level of foreign language proficiency towards another. In Figure
7.12, the pass rates at national level are compared across the main Cambridge
exams for PL2000, Italy non-PL2000 candidates, and for candidates from therest of the world, that is countries other than Italy where the Cambridge exams
are taken. The figures indicate the kinds of patterns of change that might be
expected at the early stages of educational reform programmes.
Figure 7.12 PL2000, Italy non-PL2000 and rest of the world pass rates(%) for 2001/2%
85

PL200080

Italy non-PL2000

75

rest of world

70
65
60

KET

PET

FCE

At A2 level, the pass rate for PL2000 candidates in the 2001/2002 academic
year was 70%, just below the rest of the world average (71%), but not yet as
high as the A2 Italy non-PL2000 figure of 79%. This latter figure may be highbecause it embraces a wider A2 level age range and includes more learners
taking private tuition. Checking the PL vs. non-PL pass rates for the two
preceding years (1999/2000 and 2000/01) we note that the pattern was thesame, with PL2000 KET (A2) pass rates below both the Italy non-PL and restof the world figures. In fact, the 2001/02 rates were a little more encouraging
as the gap between PL2000 and the rest of the world had narrowed to only 1%.
At B1 level, the PL2000 pass rate was 82%, a little higher than the world
average (81%) and the same as Italy non-PL, thus an improvement on
2000/01. The FCE results for 2001/02 showed a higher pass rate for PL2000

157

7 The PL2000 Impact Study: main messages
students than non-PL or the rest of the world, this for the third year running.
The CAE (PL2000 126, all Italy 1686) and CPE (PL2000 3, all Italy 741)
results are not included in Figure 7.13 because the PL2000 numbers are too
low for valid national or international comparisons to be made.
The table in Figure 7.13 presents performance data for the 60 students in
our case study group who had already taken an external Cambridge exam at
the end of the academic year during which we had become acquainted withthem. Thirty-one from the case study schools, had moved on from A1 to A2level to take KET; 25 had moved from B1 level (PET) to B2, to take FCE, andsix had advanced from B2 level to take CAE at C1 level. The pass rate patterns
were encouraging, although it is no surprise at all to note that they vary
according to school. QPT data (see Chapters 3,4,5 above) tended to confirm
Impact Study students at the CEF levels at which their schools had placed
them.
Figure 7.13 Levels and external exam performance in the PL Impact
Study case study group
ID

SchoolStart level

cycle

End exam

Gradeslevel

Numbers of candidates
Merit

Pass

Narrow Fail

Fail

A

MiddleYLE (A1)

KET (A2)

3

14

2

4

B

MiddleYLE (A1)

KET (A2)

1

1

2

A

B

C

D

3

Pass

E
Fail

C

High

PET (B1)

FCE (B2)

6

11

–

–

–

D

High

PET (B1)

FCE (B2)

–

–

7

–

1

E

High

FCE (B2)

CAE (C1)

–

–

2

1

2

PL2000 impacts on management and support
This section covers two further PL Impact Study research questions, namely:
What impacts is the PL2000 having on educational managers, including headsof schools? and What impacts is the PL2000 having on support for teacher inservice and test-awareness programmes, resource centre development, andtextbook design?
We have heard already, in the quotations from the school head interviews,of several of the PL2000 impacts on management. These included (seeprevious): the added motivation the Project appeared to bring to students,
teachers and parents; the additional professional training and resource support
it provided for language teachers; and the increased importance it brought tosome schools as regional language teaching resource centres.
One of the principles of the PL2000 which was mainly administrative, andto which we have not yet referred was ‘the division of classes into learning
158

PL2000 impacts on management and support
groups of 12–15 students homogeneous in terms of initial competence,
interests or shared specific educational needs’ (Ministry of Education, Italy
1999). Most of the classes we saw which were specifically under the PL2000,
and thus, as was often the case, held in the afternoons, were of groups ofaround the indicated number of students. We saw such classes in one of the
middle schools, both the high schools and at the technical institute. There wasevidence, too, that the schools were taking advantage of the PL2000’s‘specification of a number of hours over several years for every school cycle,
thus a choice of level by individual schools of the number of programme hours
per year and of the modality of such programmes (extensive, intensive or
mixed mode)’ (ibid).
As far as the homogeneity of the groups was concerned, management andteachers were aware of the relevant PL2000 objective on group size. However,this was sometimes irrelevant, given the relatively small overall numbers
involved (30 students altogether requesting PL2000 afternoon classes at oneof the middle schools, 18 at the beginning of the year at a high school,dwindling to fewer than 10 by the end). Yet most of the PL2000 groups wesaw seemed reasonably homogeneous in terms of English language
competence level, and attempts were being made to sort and place students atthe beginning of the school year.
One of the case study high school teachers, who was also the school’s
PL2000 course organiser, described her school’s PL2000 course structurethus:… short early afternoon courses (20–30 hrs.) to offer classes to a larger
number of students. More or less homogenous groups of about 10–15
students per class, according to CEF parameters.

The same teacher, like two others in the case study group, referred to thepossible conflict between the idea of groups selected for homogeneity of levelof competence and the comprehensive school principle:
I had also thought of organising curricular classes [i.e non-PL2000]
dividing students according to their initial language competence (thisseems to be one of the aims of PL2000), but this seems very difficult to doin Italian schools, where class groups are based by law on the union of
mixed ability students. When you cannot always have homogeneousgroups, you have to plan to cope with mixed abilities and give extra
teaching for the weaker students.

The additional work load of teachers organising PL2000 programmes could
clearly be quite heavy. One of our case study teachers suggests that a ‘major
defect of PL2000 courses is both administrative organization and coordination
which all rely on one person, already engaged in full time work’. She
described graphically the work of a PL2000 official:159

7 The PL2000 Impact Study: main messages
The responsibilities of PL2000 referenti [officials] are considerable,
including: the administration of PL course schedules, placements andcertification; the collection and analyses of performance data; thedevelopment and use of resource centres.

One of the reasons why such efforts were worthwhile, of course, was the
Ministry funding available for the running of PL2000 courses, for teacher
professional support, and for student exam enrolments.

Teacher professsional support
Returning to our research questions, we are reminded that impact on thevarious possible forms of support for PL2000 teachers was included, in thequestion: What washback and impact is the PL2000 having on support forteacher in-service and test-awareness programmes, resource centre
development, and textbook design?
A good range of information and views on professional support emerge
from the PL2000 Impact Study data, first of all from the teacher interviews
and focus groups:• The PL2000 regional resource centre does a lot of planning, holds
teachers’ meetings, distributes materials, liaises with other centres, takes
part in Europrojects, and has a website (PL2000 official and teacher ofGerman at a technical institute, March 2001).
• Teacher meetings help through mutual advice, problem-solving as a team,the selection and use of equipment, books, test development and use.
There is no peer observation, though.
• A young teacher learns little by little, should steal ideas from moreexperienced colleagues, and make use of all facilities and support (bothfrom a new middle school teacher, April 2002).
• PL2000 has led to improvements in IT facilities and more, though notenough, in-service training for teachers (two high school teachers,
October 2001).
• Preparing tests is time consuming and needs special professional
competence; Italian teachers are always preparing tests, but should
undergo more training (high school teacher, written communication,
March 2002).
• The project is useful for teachers because it offers the chance of extra
professional training and qualifications (technical institute head, April
2002).
• The teachers, and Cambridge, have worked hard for the PL. Some newteachers, weaker in English language level and culture resisted a bit, but

160

PL2000 impacts on management and support
once they were trained and once they understood why the school was
supporting the PL, they were more positive (high school head, October2001).
The additional need for professional support for teachers required by the
PL2000 also had an impact on the selected examination boards. In the UCLES
Progetto Lingue Full Report for January 2000–2001, the establishment of theProgetto is described as having led to a change in both the focus and the scaleof the Cambridge seminar programme to ‘reflect the needs of state-school
teachers’. To meet the increased demand for seminars in support of thePL2000, Cambridge ESOL set up a team of 25 experienced and trained
presenters across Italy. Print and web format Cambridge support materials
have also been developed in Italian for teachers, parents and students.
Cambridge ESOL also appointed a part time Progetto Lingue 2000 Coordinator, based in Rome.
A fuller account of the PL2000 Impact Study data and analyses appears in
Hawkey (2003a).
Chapters 6 and 7 have presented the main findings of the IELTS and
PL2000 impact studies. The two chapters should serve both to exemplify the
implementation and outcomes of such studies, and to convey their main
impact messages on the IELTS test and the PL2000.

161

8

Related developments, further
research and lessons learned

Chapters 6 and 7 have presented key messages from IELTS and PL2000
impact study data. These messages represented some of the answers to theresearch questions presented in Chapter 3 above, using the instruments
described in Chapter 4, delivered through the systems outlined in Chapter 5.
Chapters 3–7 all deal with the constructs and approaches to their investigation
which were discussed in Chapters 1 and 2.
Now in Chapter 8, the intention is to reach some kind of closure by:• tracing further developments, research and planning related to the twostudies• suggesting some lessons to be learned, in terms of impact study
development, validation, implementation and models.

IELTS impact study developments, research andplanning
Research implied in previous IELTS impact study phases
IELTS impact study Phases 1, 2 and 3 as described in this book were neither
conceived nor implemented as an autonomous, monolithic inquiry, but rather
as part of a system of continuing and iterative research intended to help ensure
the test’s continuing validity. The study investigated the washback of the teston learning and learners, teaching and teachers, and the impact of the test on
stakeholders beyond the teaching/learning context including educational
administrators, candidate selectors, testers, writers, decision-makers. As theanalysis in Chapter 6 indicates, data from this phase of IELTS impact study
help us to understand and consider means of improving language teaching,learning, methods, materials, activities and attitudes with regard to IELTS. In
addition to insights into improved test washback elements such as these, ourstudy of test impact has helped us to understand reactions to andrecommendations on IELTS test administration and candidate selection.
True to the acknowledged iterative nature of high-stakes impact research,
and as indicated in the original project design for IELTS impact studies

162

IELTS impact study developments, research and planningdescribed in Chapter 4, there were certain focus areas in the original long term
research design (see Chapters 3 and 4) which are still to be covered:• pursuit through receiving institute questionnaire of IELTS impact on
administrators; this area of inquiry has so far been handled mainly
through face-to-face rather than questionnaire data
• additional inquiry, perhaps using the trial instrumentation alreadydeveloped, into receiving institution subject lecturer views on IELTS• iterative research into the existing IELTS impact study research questions
but with General Training Module takers; the Phase 3 data were primarily
from Academic Module test takers.
Other key targets for new research could be IELTS-related studies linking test
impact and test performance, taking as criteria for test washback measurement
matters raised by the findings of the impact study phases described in thisbook (see Chapters 3, 4 and 6), for example:• given criticisms in the study of the limited time available for the IELTS
Reading module, exploring, through sample test-task scores under
different timings, relationships between performance and test time
• given evidence of the need for further investigation of the validity of
IELTS reading and writing tasks, research involving observation and/or
the views of receiving institution subject lecturers, into the nature of
reading and writing activities on a range of higher education courses. This
could be linked, perhaps, to– an investigation of candidate reading and writing performances on
different reading and writing test tasks– further investigation of candidate test and target domain performancein relationship to their test-preparation courses (see Green, 2003 on
writing) across the four skills.

Role of the Research and Validation Group
Such further research under the IELTS impact study would feed into theroutine monitoring and evaluation of the IELTS undertaken by the Cambridge
ESOL Research and Validation Group, whose systems cover analyses andresearch projects for all Cambridge ESOL exams, including IELTS. It will berecalled, from Chapter 4, that it was from this group that initiatives for the
IELTS impact studies first came. The group’s role is, after all, to provide
quality assurance for Cambridge ESOL exams at every stage of the assessment
process through routine operational analyses of the administration cycle of all
exams, i.e. exam production, exam conduct, marking/grading, and post-examevaluation (see Chapter 1). The Group continues to support instrumental

163

8 Related developments, further research and lessons learned
research through small-scale projects designed to inform operational activities
and through longer-term language assessment research such as the phased
IELTS impact study, to meet validation targets relevant to Cambridge ESOL’s
objectives and future developments. Detailed research and analysis of bothtest material and test takers’ performance are carried out to ensure that not
only does IELTS provide accurate information for the institutions thatrecognise it, but that the tests are fair to test takers whatever their nationality,
first language and gender.

The IELTS funded-research programme and Research Notes
Areas of interest for external research purposes under the joint-funding andsupport of IELTS Australia and the British Council, and co-ordinated with theCambridge ESOL framework for research and validation were identified by
the IELTS Research Committee and summarised in Research Notes 16 (May2004) as follows:••••

the use of IELTS and IELTS scores
IELTS and its impact on the teaching/learning environment
perceptions of and attitudes to IELTS among stakeholder groups
individuals and small groups taking/using IELTS (2004:22).

Chapter 1 referred to IELTS-related research activities rounds 1–7 of thefunded IELTS research agenda between 1995 and 2001 (see Research Notes
8, May 2002), a significant proportion of which covered test impact issues.
The research topics since rounds 1–7 of the funded research programme,
indicate a continuing washback and impact interest. Recent topics with thisfocus, and indeed a shared focus on areas also investigated to a greater or
lesser extent in the study analysed in this volume, have included (see rounds
8–10, 2002–2004):• candidate identity, learning and performance, with specific reference tothe affective and academic impact of IELTS on successful IELTS
students (Rea-Dickins, Kiely and Yu)• the relationships between IELTS and preparation courses for it, including
ethnographic study of classroom instruction (Mickan), and therelationship between teacher background and classroom instruction on an
IELTS preparation programme (Swan and Gibson)• the impact of IELTS on receiving institutions, for example, tertiary
decision-maker attitudes to English language tests (Smith and Haslett);the use of IELTS for university selection (O’Loughlin), and IELTS as apredictor of academic language performance (Ingram, Bayliss and Paul)• perceptions on the IELTS skills modules, for example the Speaking test
(Brown; Hampton and Huang Chung; O’Sullivan, Weir and Horai;

164

IELTS impact study developments, research and planning
O’Sullivan and Yang Lu; Read and Nation; Seedhouse and Egbert) andthe Writing test, features of written language production (Franceschina
and Banerjee), impact of computer vs pen-and-paper versions (Weir andO’Sullivan).
All these research topics focus on the effects of the tests on stakeholders, on
their attitudes or on their actions, or on the complex of variables (for example,
here, the backgrounds of preparation course teachers) intervening between an
aspect of the test and one of its impacts (see Chapter 1).
It is the aim of the UCLES/Cambridge University Press Studies in
Language Testing (SILT) series, to which this volume also belongs, todisseminate knowledge, research and experience with regard to testdevelopment, validation and use. SILT volume 19 IELTS Collected Papers(Taylor and Falvey, forthcoming) gathers together a set of 10 IELTS-funded
research projects on the Speaking and the Writing tests. As well as evaluating
the wide range of methodologies used in these projects, the volume reviews
the importance of their research findings on the Speaking and Writing
modules of the IELTS test in the past five years. In Part 1 of the volume (onthe IELTS Speaking module) the research studies investigate impact
relationships such as interviewer styles and gender, with candidate
performance (Brown; Brown and Hill; Merrylees; O’Loughlin). The Part 2
projects investigate relationships between target domain and IELTS
academic writing tasks (Moore and Morton); training and rater judgments(Rignall and Furneaux); quantity and presentation and performance(O’Loughlin and Wigglesworth), bias analysis feedback and rater
performance (O’Sullivan and Rignall) and analyses of linguistic output(Kennedy and Thorp; Mayor, Hewings and Swann).
The messages from such a volume emphasise:• the importance of disseminating impact-related studies on a high-stakes
test such as IELTS in the context of continuing and iterative research• the close relationships of broader, multi-faceted research such as that
described in this volume, with smaller-scale studies such as those
supported by the IELTS funded-research initiative, and the need tointegrate the two as far as possible• the superordinate role of the routine test validation work carried out byCambridge ESOL with its focus on test taker performance across thewhole IELTS population, and the desirability of relating all other IELTS
impact study systematically to this.
We can already see, from the references here (and in preceding chapters)
the important role played by Cambridge ESOL’s Research Notes(www.cambridgeesol.org/rs_notes) in the dissemination of test development,165

8 Related developments, further research and lessons learned
administration and validation information. The November 2004 issue of thejournal, for example, which was devoted to IELTS, articles covered: IELTS :CEF level comparability; paper- and computer-based versions; the suitabilityof the General Training (GT) modules for 16–17-year-olds; Writing test
revision Phase 4 (see Chapter 4); frequently asked questions, and test
performance data for 2003.
Several points of relevance to the discussions in this book arise from theselection of articles in the issue. The paper on the IELTS GT module is headed
IELTS Impact: a study on the accessibility of IELTS GT Modules to 16–17
year old candidates (Smith 2004). The study ‘required a comparison between
materials used in classes preparing students for entry to upper secondary andthose used in the General Training Modules’ (2004:6). The instrument used topursue this purpose was the Instrument for the Analysis of Textbook Materials(IATM, see Chapters 3, 4 and 6), developed for and used in IELTS impact
study Phase 3. It will be also recalled from Chapter 4 that the instrumentation
developed in Phases 1 and 2 of the study, was intended for use in as many
future studies as they were felt appropriate for.
The Research Notes 18, 2004 paper on IELTS performance data
(Blackhurst 2004:18–21) presents mean module score data for the whole 2003
IELTS population and for a range of sub-populations (for example by
Academic and GT module, frequent language backgrounds, nationalities).
This is precisely the kind of routine validation-oriented performance analysis
that is best handled by the Research and Validation Group rather than bysmaller scale studies. This point was made in Chapter 6, it will be recalled, toexplain why the various aspects of IELTS washback on candidate
performance was not a major objective of Phase 3 of the IELTS study, with its
test taker sample of 572 participants. But medium and smaller-scale impact
studies may be useful in following up particular leads provided by the largescale data. Green, for example, reports in Research Notes 16 (2004) on studies
of score gains on the IELTS Writing test (see also below).

IELTS website
The IELTS website (www.ielts.org) contains details of and links to bothcurrent and completed IELTS research awards, and summaries of the mostrecent candidate and test data. The stated aim of the IELTS partners, in
keeping with the high-stakes nature of the test, is ‘to provide appropriate
public access to both comprehensive test data and research findings’.
The very significant increase in IELTS candidature over the past few years(see Chapter 6) reflects the increase in recognition of IELTS as ‘abenchmarking tool for English language proficiency’ (www.ielts.org:1). To
date, education institutions, professional bodies, government departments and166

PL2000: further impact research and related developments
employers in approximately 40 countries use the test as one of their entry
requirements. The last few years have seen a parallel increase in demand for
IELTS preparation services. As a result of this development, the IELTS test
partners, the British Council, IDP: IELTS Australia and the University ofCambridge ESOL Examinations, have introduced a Website designed forteachers of IELTS preparation classes. This includes the following features:
•
•
•
•
•

sample tasks
tips for teachers and students
dos and don’ts
ready-made activities for use in preparation classes
skills’ development activities.

The contents and approaches of this site may be seen as a further example ofthe strength of test washback. The test is influencing the website directly andthe website is likely to have washback on its users, mainly teachers and test
takers, and on IELTS preparation courses.
The findings of Phase 3 of the IELTS impact study have already been
disseminated at various conferences and seminars. These include IATEFL
annual conferences, York (March 2002) and Liverpool (April 2004); a
Cambridge ESOL staff seminar (July 2004); the EFL Testing and Evaluation
Conference, Athens (October 2004); the 18th IDP International Conference inSydney (October 2004); the British Council Going Global Conference,
Edinburgh, (December 2004), and the ALTE second International Conference,
Berlin, (May 2005). Publications have included Saville and Hawkey (2004),
Hawkey (2001, 2003b).

PL2000: further impact research and related
developments
Research pursuing issues from the PL2000 Impact Study
The PL2000 Impact Study described had always been envisaged as part ofcontinuing research into the effects of changes in foreign language teaching,learning and testing in Italy, with particular reference to Cambridge ESOL
exams. As with research into the impacts of the IELTS test, impact is seen
here as part of Cambridge ESOL’s four-element test development focus, tooptimise, continuously and iteratively, the validity, reliability, impact andpracticality (VRIP), in fact the usefulness, of its tests. The PL2000 Impact
Study, like IELTS impact study Phase 3, has thus led into related studies.

167

8 Related developments, further research and lessons learned

The Florence Language Learning Gain Project
In order to attempt to investigate further the vital yet elusive matter of highstakes test washback on language performance, language learning growth andchange, Cambridge ESOL, in co-operation with the British Institute of
Florence, initiated the Florence Language Learning Gain Project (FLLGP).
This second-phase study took as its sample populations English language
learners in young learner and adult classes at the British Institute. Some of theparticipants had the Cambridge ESOL Preliminary English Test (PET) at B1
level, others the First Certificate in English (FCE) at B2 level as their target
exams. Some had no immediate external examination target.
Participant learner groups and individuals would be compared, in terms ofvariables such as competence level, age, stage, motivation, high-stakes orinternal final exams, and language gain. All groups were pre-tested using theCommunicat English language placement test, a computer adaptive language
test designed to provide prompt and economical assessment across language
skill areas, and available in six languages. The participant learners were alsopre-profiled through the Language Learning Questionnaire (LLQ), ‘Can Do’statements in Italian, and Student and Teacher Questionnaires adapted from theparallel instruments used in the PL2000 Impact Study. Teacher interview andlog data, and classroom video analyses at the beginning, in the middle and at
the end of the study year (2002–2003), samples of English language workthroughout the year, and final exam (external and internal) performance data,
are being analysed for insights into factors affecting language learning andperformance. The issue of learner motivations, in their multi-faceted anddynamic forms, are a key focus of the research. The data analyses are being
conducted in collaboration with Professor Jim Purpura and doctoral student
Michael Perrone of the Teachers College, Columbia University, New York.

The Siena University–Cambridge ESOL collaboration
project
The PL2000 did not, of course, cover higher educational institutions, nor,
therefore, did the Impact Study. However, it is intended that students who
studied some of their English under the PL2000 should be a focus ofcollaborative research between Cambridge ESOL and the University of Siena
Language Centre (Centro Linguistico dell’Ateneo (CLA)). The focus of suchresearch would again be learning gain and performance data collection, related
to the question of which CEF level and which exams are appropriate forstudents learning and using English as a foreign language during their
university studies. Areas for possible research questions in the collaborative
study, especially in the light of the increasing acknowledgement of the impactof the Common European Framework of Reference for Languages: learning,teaching and assessment (2001) (see previous), could be:168PL2000: further impact research and related developments• How do CEF aims compare with school and university views of foreign
language needs and target levels? Are foreign language target proficiency
levels changing for various learner groups and cohorts?
• Is CEF Level B1 an appropriate achievement level for students by thetime they leave school or enter university? Should universities turn away
those who have not reached B1? What is the right entry level for foreign
language specialists?
• Can the CEF help ensure high validity high stakes exams? Is the use ofuniversity-developed exams a viable alternative to external exams?
The collaboration will use insights from Alison Duguid’s Anatomy of aContext: English Language Teaching in Italy (2001), which describes auniversity response to higher education reforms in Italy, in particular theinternational certification (idoneità internazionale) project at Siena
University. This introduced the Cambridge ESOL PET exam as a B1 English
language qualification for university degree programmes.
The recent Campus 1 project, which involves 71 Italian universities, meansthat over half of all Italian universities are enrolling students for Cambridge
exams either directly (through 26 university centres) or indirectly. The
Cambridge ESOL–University of Siena Centro Linguistico dell’Ateneo (CLA)
collaboration would make use of Cambridge ESOL impact study experienceand instrumentation to enhance CLA language course teacher data collection,
including further classroom video recording and analyses.

Related PL2000 developments
A report in the EL Gazette (Vint, June 2004; 1) noted significant impacts fromthe PL2000. The ‘overall consensus’ was that the Project achieved ‘a great
deal of success in moving language learning in state schools towards a morelanguage-oriented approach, with syllabuses less dominated by literature orgrammar and lessons increasingly conducted in the L2’. External
examinations such as those of Cambridge ESOL encouraged schools to set
targets according to CEF language ‘Can Do’ levels (A1 for primary, A2 formiddle schools, B1 for high school). More than 400,000 state school students
were reported to have taken external certificates in one of the four foreign
languages since January 2000, 350,000 or more of these in English. The EL
Gazette also reported that most certificates were at levels A2 and B1, but thata large proportion also took oral exams only at below A2 in primary andmiddle schools, rather than exams covering all the four macro-skills, this inline perhaps with the concept of ‘partial skills’ permitted by the Progetto. An
increasing number of candidates was reported to be taking four-skill
examinations at B2 level (FCE).
The Italian Ministry of Education had always stressed that external examboards were not brought into the PL2000 in order to evaluate the success of169

8 Related developments, further research and lessons learned
Italian language teachers. Regional inspectors, however, are said to havepointed out that the pass rates for PL2000 course students in four-skill exams,
particularly at B1 and B2 levels, suggest teacher success in helping Italian
students towards appropriate CEF levels.
The PL2000 officially terminated in December 2002 but the number ofstate school students taking external exam certificates has continued toincrease, rising to 140,000 students in the academic year 2002/03 withindications of further increases in 2003/04. The EL Gazette articles suggested
that the reason for this was probably that ‘parents are prepared to pay for theexams if school funds are not available, and that most exam boards havecontinued with special prices and support activity’ (2004:1). Figure 8.1 shows
the trend.
Figure 8.1 Total Cambridge ESOL exam enrolments in
Italy 1999–2003
ITALY

1999/00

2000/01% Incr.

2001/02% Incr.

2002/03% Incr.

Non-PL2000

23,97825,251

5

28,59413

33,85018

Progetto Lingue

8,058

29,647268

45,491

53

52,46515

Total

32,03654,89871

74,08535

86,31517

Post-PL2000, the Ministry of Education is supporting the preparation ofstate-school students for the required exit levels as part of the normal
curriculum, ‘taking greater stock of the CEF objectives’ (ibid). From
September 2004, regional local education authorities were able to offer
language certification to primary school teachers as part of a general teacher
training programme.
As noted in the PL2000 Impact Study report (and see Chapters 1 and 7),
impact has tended to be two-way, the external exams approved under the
project influencing teaching, learning and performance, and the project
influencing the external exams. A number of changes were reported in theexams on offer since the start of PL2000. Trinity International offers moreskills with its International Skills exam, and Cambridge-ESOL modular
certificates through its Certificates in English Language Skills exam (CELS).
Primary school teachers would also have the option of taking an exam basedon teaching methodology at B1 level (the new Cambridge ESOL Teaching
Knowledge Test [TKT]), as well as more traditional teaching certificates(from Trinity International, UCLES and City and Guilds).
The PL2000 had emphasised the importance of ensuring that external
examinations are selected according to how well they assess agreed foreign
language targets or syllabus. The Progetto also advised that courses should be170

PL2000: further impact research and related developments
seen as helping learners towards a CEF level rather than a particular exam(see, for example, evidence from the Impact Study in Chapter 7). In thisconnection, the Council of Europe has produced a draft manual Relating
Language Examinations to the Common European Framework of Reference
for Languages: Learning, Teaching, Assessment – Manual (Council ofEurope, 2003). This should also help schools and teachers respond to the key
message of the PL2000 Impact Study, to think first about a syllabus in linewith the CEF and its exemplification in the European Language Portfolio(ELP) ‘to help them monitor progress and understand competency objectives’(Vint 2004). Decisions can then be taken on appropriate language learning
methods, materials and activities, before the selection of an appropriate exam
for external certification.
Teacher support seminars run by the external exam providers (see Chapter7) have continued after PL2000, now with a stronger ELP focus (there are now
at least 11 different versions of the portfolio available in Italy). And on theperformance front, it is noted (Vint 2004) that A2 (KET) pass rates have
improved considerably since the PL2000, perhaps ‘as a result of increased
support attention’ (for example, teachers’ seminars and special guides in
Italian).

Information dissemination
Cambridge ESOL has disseminated information on the PL2000 Impact Study
through its formal reports to the Ministry of Education and throughpresentations at conferences and seminars. In Italy, for example, these haveincluded British Council conferences in Venice (March 2001), and Florence,(February–March 2002); and TESOL Italy conferences in Rome,(November–December 2001 and November 2002). Details of the Impact
Study were also presented outside Italy, for example at the Association of
Language Testers in Europe (ALTE) European Year of Languages
Conference, Barcelona (July 2001), at the ALTE meeting in Salamanca, Spain(November 2002), and at the 36th Annual Conference of the International
Association of Teachers of English as a Foreign Language (IATEFL) in York(March 2002). PL2000 Impact Study publications include a chapter in thevolume European Languages in a Global Context (2004) in the
UCLES/Cambridge University Press Studies in Language Testing series, andin the EL Gazette (Vint 2004). This dissemination process fulfils one of theoriginal aims of the PL2000 Impact Study, to ‘provide a focus for thepresentation, to other EU countries, of the Italian model for state-sector
adoption of the Common European Framework of Reference for language’(see Chapters 3 and 6).

171

8 Related developments, further research and lessons learned

Impact studies: lessons to be learned
The design and implementation of the two studies of impact presented in thisbook have suggested both theoretical and practical lessons which could informfuture similar studies. Most of these lessons will already have been suggested
in preceding chapters, but they can usefully be summarised here.

Define and adhere to the purpose in context
It is important to have established, agreed and to reiterate regularly the
purpose, aims and institutional context of an impact study. Phases 1 to 3 of the
IELTS impact study, and the PL2000 Impact Study were fundamentally
Cambridge ESOL exam validation exercises. They were to be carried out
within continuing and iterative Cambridge ESOL systems of test validation,
pursuing data and findings on the impact (including the washback) of
Cambridge language tests, impact being a fundamental quality of a test or
examination, along with its validity, reliability and practicality. Impact study
implementers should never lose sight of the wood of the impact study purposeas they pick their way through the trees of study data collection and analysis.
The ownership of the two studies covered in this book was clear, their
purposes agreed at senior management level. It was nevertheless always
important, both at project design and at implementation stages, to recall andrestate the objectives of and target stakeholders for all instruments, interviews,
focus groups and data analyses. This helped ensure that they remained ontarget and that the resulting data were constantly checked for their relevanceto impact study purpose. The research questions should guide all datacollection. If instrumentation development or data analysis ever appear to lose
direction, it will normally clarify matters to re-pose the relevant researchquestion.

Define and maintain the research structure
Impact studies should be based on sound principles of research design andproject management. In our two studies, the use of an academic researchpattern of organisation was found useful.
First a study was envisaged in terms of context and purpose, aim andobjectives, research questions, research design/action plan and hypotheses.
This helped to focus the pre-implementation stage, both in terms of theorybased validity in the research design, and an action plan, which was a practicalstructure assigning roles and responsibilities, including the allocation of
resources.
Then, for the implementation phase, the choice of appropriate research typeis made, leading inevitably to the use of both quantitative and qualitative172

Impact studies: lessons to be learned
approaches to data collection. The techniques of wide-scale surveys andlimited case studies can both provide valuable evidence in answer to a study’s
research questions. The research should try for symmetry and control but
accepting that studies carried out in real time and places have their own value,
despite possible inconvenience and the possibility of some patchy data. It
seems crucial for an impact study to include observation as well as stakeholder
response data. Since impact studies are, by definition, making comparisons, it
is important that baseline data collection is included in the research design andimplementation.
Timescale is an important dimension in impact studies. It may be necessary
to conduct a study project in phases over a significant period of time. Likeother aspects of test validation, these phases should form part of a cyclical anditerative approach, where evidence collected along the way feeds into theplanning of subsequent phases.

Validate data collection instruments through agreed steps
and appropriate language test validation operations
It is important to start from and sustain the view that impact data collection
approaches are, in many cases, like language tests and must be validated. The
techniques for the design and development of data collection means should be
agreed in advance (though, of course, subject to modification and addition
where events really dictate this). Approaches such as brainstorming, response
prediction, expert opinion, report and review, revision, piloting, trialling andsystematic revision and rationalisation should be planned into thedevelopment stage. Language test validation techniques used in the process ofdata collection instrument validation could include, depending on instrument
format, descriptive analyses; factor analysis; convergent–divergent, multitrait, multi-method validation; triangulation within and across instruments anddata collection modes.

Plan and implement the most efficient applicable data
management systems
The evidence from an impact study, probably in the form of various types ofdata, will need to be adequately stored, for retrieval, analysis and reporting
purposes. Implications for the type and quantity of analyses to be carried out
need to be carefully considered from the start of the impact study design
process and in formulating the action plan. These considerations can affect thedesign of the instruments. Data collection and management systems shouldtake account of the practical constraints of the implementers of and theparticipants in a study. But the most advanced possible technology should be
favoured within those constraints.
173

8 Related developments, further research and lessons learned

Establish and sustain good practice
The conduct of an impact study should always adhere to good practice, interms, for example, of: informed consent, protection of participants,
confidentiality and anonymity, accountability, reciprocity and intellectual
property and data protection legislation. These are matters where up-to-date
official sources of information need to be consulted regularly.

Use ideas and instrumentation, adapted, across research
projects
Impact studies tend to be demanding both of time and effort. It would be rare
for a study implementation team, however well developed its action plan, toconsider itself adequately staffed. Instrumentation and items were shared,
modified as required, by the IELTS Phase 3 and the PL2000 studies. We havealso seen that related projects, for example in the IELTS funded research
programme, have used Phase 3 instruments. This policy of instrument sharing
is to be recommended wherever feasible.

Towards revised models for test validation and the place of
impact study in them
Certain constant themes will have been noted in this book on the matter ofsystematic approaches to impact studies within an institutional context. These
have included:• the view of impact study as part of routine test validation research• the study of impact, like the test validation process, as continuing anditerative• an emphasis on good practice in the planning and conduct of impact
studies• the need to seek ever-greater coherence and rigour in models and systems
for test specification, design, development, validation, production
administration and use.
Significant developments and achievements in Cambridge ESOL systems andmodels for test development, validation and use have been traced in preceding
chapters, leading to a model of test research, development, implemention andrevision based on the principles of validity, reliability, impact and practicality(VRIP). These principles and the systems to implement them were arrived atthrough internal and external collaboration, including:• long-term theoretical and empirical investigation:– by Cambridge ESOL’s Research and Validation Group, in co-operation
with the Examinations and Assessment Group, responsible for174

Impact studies: lessons to be learned
professional aspects of the development, production and on-going
servicing of all Cambridge ESOL exams, and the Operations Group
which handles, among other matters, exam materials, commissioning
and pretesting, candidacy entry and results processing, marking, andscript management.
• collaboration with partners:– in the case of the IELTS test, the British Council and IDP Education:
IELTS Australia, both fully informed of Cambridge ESOL testdevelopment and validation systems and models and very much
involved in IELTS validation research and dissemination, in particularthrough the funded-research programme– the Association of Language Testers in Europe (ALTE) in thedevelopment and finalisation of the Code of Practice, which, it will be
remembered, lays a strong emphasis on provider responsibility for exam
development, scoring, result interpretation, fairness and transparency• consultation with key outside language testing figures including:– Bachman, in particular on the relationship between VRIP and Bachman
and Palmer’s 1996 six test usefulness qualities, reliability, constructvalidity, authenticity, interactiveness, impact and practicality, and on
test comparability (Bachman et al 1995)– Weir (2004), with his important socio-cognitive framework forvalidating tests, with test validity the superordinate category to theorybased, context, scoring, concurrent and consequential validities– Alderson, as in Phase 1 of the study of IELTS impact and at the ALTE
Barcelona Conference (Alderson in Milanovic and Weir (eds) 2004.
– Kunnan (2000) and Purpura (1996b, 2001), as in their contributions tothe study of IELTS impact.
Such research, practical investigation, collaboration and consultation has led
to the development of a range of frameworks and models designed to informthe test development, validation and management process. Among theframeworks and models encountered in this book are:• the taxonomy of Cambridge ESOL stakeholders as they relate to test
constructs, formats, conditions and assessment criteria, see Figure 1.4(and Saville 2003, Saville and Hawkey 2004, Taylor 1999)• the standard current Cambridge ESOL model of the test development orrevision process (Figure 1.5), accounting for phases in the test cycle fromperceived need for a new or revised test, through planning, design,development (trialling, analysis, evaluation and review), operation, further
monitoring, review and evaluation through to further test revision or
renewal• representations used above (e.g. Figures 1.1, 1.2) to indicate the place androle of impact study in test development and validation.
175

8 Related developments, further research and lessons learned
This book should already have shown the important role played by the studyof impact in test development, validation, production and administration
models and systems.
Models and systems themselves, of course, are also subject to monitoring,
revision and refinement. Recent discussion at Cambridge ESOL indicatestrends towards renewed models. The direction of these trends appears to be
influenced by the growing expectation, noted (for example in Chapter 1, see
Saville and Hawkey, 2004) of good value, good practice and accountability.
Test validity is expected to be strongly theory- and evidence-based (see theWeir 2004 framework) and there appears to be a stronger than everexpectation of tightly logical argument in support of validity claims. Relevant
recent influences in this direction appear to be:• the concept of interpretive argument proposed by Kane (1992) and Kane,
Crooks and Cohen (1999), emphasising clarity of argumentation,
coherence of argument, and the plausibility of assumptions• the ‘standards approach to validation’ of Chapelle et al (2004), which‘directs validation researchers to gather kinds of evidence’ which ‘areneeded to evaluate the intended interpretation of test scores’ to be guidedby ‘developing a set of propositions that support the proposedinterpretations’• the work of Mislevy et al (2002, 2003) on evidence-centred assessment
design (ECD), like the ‘evidence-based’ label hardly revolutionary, but as
Saville (June 2004) says, ‘a way of introducing greater systematicity inthe design and development’ of language tests• the method of Toulmin, favouring analyses as a process of breaking anargument into its different parts to make judgements on how well thedifferent parts work together; for example in terms of claim (‘aconclusion whose merits we are seeking to establish’); data (‘information
on which the claim is based’), a warrant (a statement which ‘provides
legitimacy of the step in the argument and refers to the claim’); thebacking (evidence from theory, research, evidence from validation
operations (2003:90-96), ‘without which the warrants themselves would
possess neither authority nor currency’), and rebuttals (‘exceptional
conditions which might be capable of defeating or rebutting the warranted
conclusion’)• Bachman (2004), with his characteristic emphasis on assessment use as
well as assessment validation argument, suggesting a model where aclaim, that is an inference that we make according to data from test taker
responses, is justified on the basis of a warrant, where the backing is
positive evidence (from theory, research, evidence from validation
operations) or, refuted on the basis of a rebuttal if such evidence issignificantly negative.

176

Conclusion
Saville (2004) claims that the Cambridge ESOL research and validation
framework ‘is being recast’ taking these features into account. The intention
is to adopt a process model and theoretical framework approach ‘to develop a
more coherent programme of test development, validation and research tosupport claims about the validity of the interpretation of our examinations
results and their uses’. This, Saville adds, would enable ESOL ‘to set out our
CLAIMS relating to the usefulness of the test for its intended purpose, explain
why each claim is appropriate by giving REASONS and justifications; provide
adequate EVIDENCE to support the claims and the reasoning’.
It should be noted that, as this book has confirmed, Cambridge ESOL
already employ models and systems making reasoned claims on test
usefulness and validity based on evidence from the multi-faceted sources
some of which are listed above. The continuing and iterative nature ofCambridge ESOL models for the key steps in the test cycle, however, requiresthat such models and systems are themselves revisited regularly.

Conclusion
In Chapter 8 further research and other post-impact study developments havebeen discussed, lessons to be learned for the design and implementation of
impact studies suggested, based on the theory and the case studies describedin the book. In both the current and possible future frameworks and models of
test validation noted, there is clearly a place for the study of test impact.
Chapter 8 brings closure to the logic of the book. It has travelled a routefrom the constructs of impact and related concepts in Chapter 1, approaches totheir investigation in Chapter 2, through impact study research design in
Chapter 3, the histories of the development and validation of the datacollection instruments described in Chapter 4 and the practicalities of theirimplementation in Chapter 5. Chapters 6 and 7 presented key messages fromthe IELTS and PL2000 impact studies.

177

Appendices

179

APPENDIX A:
Student Questionnaire

180

Student Questionnaire

181

Appendix A

182

Student Questionnaire

183

Appendix A

184

Student Questionnaire

185

Appendix A

186

Student Questionnaire

187

Appendix A

188

Student Questionnaire

189

Appendix A

190

Student Questionnaire

191

APPENDIX B:
Teacher Questionnaire (for teachers preparing students
for IELTS)

192

Teacher Questionnaire

193

Appendix B

194

Teacher Questionnaire

195

Appendix B

196

Teacher Questionnaire

197

APPENDIX C:
Instrument for the Analysis of Textbook Materials (IATM)

198

Instrument for the Analysis of Textbook Materials (IATM)

199

Appendix C

200

Instrument for the Analysis of Textbook Materials (IATM)

201

Appendix C

202

Instrument for the Analysis of Textbook Materials (IATM)

203

Appendix C

204

Instrument for the Analysis of Textbook Materials (IATM)

205

APPENDIX D:
Impact Study Lesson Observation Analysis Form

206

APPENDIX E:
Receiving Institution Questionnaire

207

Appendix E

208

Study of the Impact of International English Language Testing System (IELTS)

209

Appendix E

210

APPENDIX F:
Background information form for PL2000 Impact Study
schools211

APPENDIX G:
PL2000 Impact Study questionnaire for students

212

APPENDIX H:
PL2000 Impact Study questionnaire for teachers

213

APPENDIX I:
Semi-structured interview/focus group topics for
PL2000 Impact Study Italy visit 14–21 October 2001

214

References

Agar, M (1986) Speaking of Ethnography, Beverly Hills: Sage Publications.
Airasian, P (1988) Measurement driven instruction: A closer look. Socio-culturalIssues in English for Academic Purposes, London: Macmillan.
Alderson, J (1986) Innovations in language testing, in Portal, M (ed.) Innovations in
Language Testing, Windsor: NFER Nelson, 93–105.
Alderson, J (1995) Ideas for research into impact, washback and IELTS, paper
submitted to the University of Cambridge Local Examinations Syndicate.
Alderson, J (2004a) Introduction, in Cheng, L and Watanabe, Y (eds).
Alderson, J (2004b) The shape of things to come: will it be the normal distribution?
in Milanovic, M and Weir, C (eds).
Alderson, J and Banerjee, J (1996) How might impact study instruments bevalidated? Paper commissioned by the University of Cambridge Local
Examinations Syndicate (UCLES) as part of the IELTS impact study.
Alderson, J and Buck, G (1993) Standards in Testing: a study of the practice of UK
examination boards in EFL/ESL testing, Language Testing 10, 1–26.
Alderson, J and Clapham, C (1993) Examining the ELTS Test: an Account of theFirst Stage of the ELTS Revision Project, IELTS Research Report 2, London,
Cambridge and Canberra: British Council, UCLES, IDP Australia.
Alderson, J and Hamp-Lyons, L (1996) TOEFL preparation courses: a study of
washback, Language Testing 13 (3), 280–297.
Alderson, J and Hughes, A (eds) (1981) Issues in Language Testing, ELT Documents
111, Oxford: Pergamon Press.
Alderson, J and North, B (eds) (1991) Language Testing in the 1990s, London:
Macmillan.
Alderson, J and Wall, D (1993) Does washback exist? Applied Linguistics 14,
115–129.
Anderson, S and Gansneder, B (1995) Using electronic mail surveys and computermonitored data for studying computer-mediated communication systems, Social
Sciences Computer Review, 13 (1).
Andrews, S (1995) Washback or washout? The relationship between examination
reform and curriculum innovation, in Nunan, D, Berry, V and Berry, R (eds)
Bringing about Change in Language Education, Hong Kong: University of Hong
Kong.
Andrews, S (2004) Washback and curriculum innovation, in Cheng, L and Watanabe,
Y (eds).
Archer, T and Kendall, L (2003) Headteachers’ main concerns, Annual Trends in
Education Digest 12, Slough: NFER News, online.
Association of Language Testers in Europe (1994) Code of Practice, Cambridge:
UCLES.
Association of Language Testers in Europe (1995) Development and descriptive
checklists for tasks and examinations. Members of ALTE.
215

References
Association of Language Testers in Europe (1998) ALTE Handbook of Language
Examinations and Examination Systems, Cambridge: UCLES.
Bachman, L (1990) Fundamental Considerations in Language Testing, Oxford:
Oxford University Press.
Bachman, L (2005) Building and supporting a case for test use, Language
Assessment Quarterly 2(1), 1–34.
Bachman, L, Cushing, S and Purpura, J (1993) Development of a questionnaire item
bank to explore test-taker characteristics, Cambridge: UCLES interim internalreport.
Bachman, L, Davidson, F, Ryan, K and Choi, I-C (1995) An investigation into thecomparability of two tests of English as a foreign language, Cambridge:
Cambridge University Press/UCLES.
Bachman, L and Palmer, A (1996) Language Testing in Practice, Oxford: Oxford
University Press.
Baker, C (1997) Survey Methods in Researching Language and Education, in
Hornberger and Corson (eds) Encyclopedia of Language and Education, Volume
8: Research Methods in Language and Education, Dordrecht: Kluwer Academic
Publishers.
Banerjee, J (2003) Interpreting and using proficiency test scores, unpublished PhD
thesis, University of Lancaster.
Banerjee, J (1996) The design of the classroom observation instrument, Cambridge:
UCLES report.
Blackhurst, A (2004) IELTS test performance data 2003, Research Notes 18, 18–21.
Bogdan, R and Biklen, S (1982) Qualitative Research for Education: An introduction
to theory and methods, Boston: Allyn and Bacon.
Bonkowski, F (1996) Instrument for the assessment of teaching materials,
unpublished manuscript, Lancaster University.
Bracey, G (1987) Measurement-driven instruction: catch phrase, dangerous practice,
Phi Delta Kappa 68 (9).
Brown, A (2002) An examination of candidate discourse in the revised IELTS
speaking test, British Council/IDP Australia IELTS Research Report.
Brown, A (forthcoming) An investigation of raters’ orientation in awarding scores in
the IELTS oral interview, in Taylor, L and Falvey, P (eds).
Brown, A and Hill, K (forthcoming) Interviewer style and candidate performance in
the IELTS oral interview, in Taylor, L and Falvey, P (eds).
Bunton, D (1998) Linguistic and textual problems in Ph.D and M.Phil theses: ananalysis of genre moves and metatext, unpublished PhD thesis, University ofHong Kong.
Chapelle, C, Enright, M and Jamieson, J (2004) Issues in developing a TOEFL
validity argument, draft paper presented at LTRC, California, March 2004.
Chapman, D and Snyder, C (2000) Can high stakes national testing improve
instruction? Reexamining conventional wisdom, International Journal of
Educational Development 20 (6), 457–474.
Charge, N and Taylor, L (1997) Recent Developments in IELTS, ELT Journal 51 (4),374–380.
Cheng, L (1997) How does washback influence teaching? Implications for Hong
Kong, Language and Education 11, 38–54.
Cheng, L (1998) The washback effect of public examination change on classroom

216

References
teaching. An impact study of the 1996 Hong Kong Certificate of Education in
English on classroom teaching of English in Hong Kong secondary schools,
unpublished PhD, University of Hong Kong.
Cheng, L and Watanabe, Y (eds) (2004) Washback in Language Testing: ResearchContexts and Methods, New Jersey: Lawrence Erlbaum Associates.
Cheng, L and Curtis, A (2004) Washback or Backwash: A review of the impact of
testing on teaching and learning, in Cheng and Watanabe (eds) (2004).
Clapham, C (1996) The development of IELTS: A study of the effect of backgroundknowledge on reading comprehension, Studies in Language Testing 4, Cambridge:
UCLES, Cambridge University Press.
Connell, J, Kubisch, A, Schorr, L and Weiss, C (eds) (1995) New Approaches to
Evaluating Community Initiatives: Concepts, methods, and contexts, Washington
DC: The Aspen Institute.
Cook, T and Reichardt, C (1979) Qualitative and quantitative methods in evaluation
research, Beverly Hills: Sage Publications.
Cook, V (2003) Introduction: the changing L1 in the L2 user’s mind, in Cook, V(ed.) Effects of the Second Language on the First, Clevedon: Multilingual
Matters.
Council of Europe (2001) Common European Framework of Reference for
Languages: Learning, teaching, assessment, Cambridge: Cambridge University
Press.
Council of Europe (2003) Relating language Examinations to the Common European
Framework of Reference for Languages: Learning, teaching, assessment. Manual.
Draft.
Criper, C and Davies, A (1988) ELTS Validation Project Report, ELTS Research
Report 1 (i).
Cronbach, L (1982) Designing Evaluations of Educational and Social Programs, San
Francisco: Jossey-Bass.
Davies, A (1996) Ironising the myth of linguicism, Journal of Multilingual andMulticultural Development 17 (6), 485–96.
Davies, A (1997) Demands of being professional in language testing, Language
Testing 14 (3) 328–339.
Davies, A (forthcoming) Assessing Academic English: testing English proficiency
1950–2005 – the IELTS solution, Studies in Language Testing 23, Cambridge:
UCLES, Cambridge University Press.
Davis, K (1995) Qualitative theory and methods in applied linguistics research,
TESOL Quarterly 29, 3.
Davies, A, Brown, A, Elder, C, Hill, K, Lumley, T and McNamara, T (1999)
Dictionary of language testing, Cambridge: Cambridge University Press/UCLES.
Denzin, N (1978) Research Act: Theoretical Introduction to Sociological Methods,
New York: McGraw-Hill.
Department for International Development (DFID) (1998) Glossary of terms, internal
DFID document.
Duff, P (2002) Beyond generalizability: contextualization, complexity and credibility
in applied linguistics research, presentation to the AAL / ILTA colloquium, Salt
Lake City, April 6, 2002.
Duguid, A (2001) Anatomy of a Context : English Language Teaching in Italy,
London: Granville.

217

References
Farley Templeton, J (1996) The Focus Group: A Strategic Guide to Organizing,
Conducting and Analyzing the Focus Group Interview, New York: McGraw-Hill.
Fitz-Gibbon, C and Morris, L (1987) How to Analyse Data, Newbury Park: Sage
Publications.
Franceschina, F and Banerjee, J (2004) Documenting features of written language
production typical at different IELTS bands levels, British Council/ IDP Australia
IELTS Research Report.
Frankfort-Nachmias, C and Nachmias, D (1996) Research Methods in the Social
Sciences, London: Arnold.
Gardiner, K (1999) Analysis of IELTS impact test-taker background questionnaire,
Cambridge: UCLES IELTS EFL validation report.
Glaser, B and Strauss, A (1967) The Discovery of Grounded Theory: Strategies for
Qualitative Research, Chicago: Aldine Publishers.
Gunter, B (2001) Media Research Methods: measuring audiences, reactions, impact,
London: Sage Publications.
Green, A (2003a) Test impact and English for academic purposes: a comparative
study in backwash between IELTS preparation and university pre-sessionalcourses, unpublished PhD thesis, Centre for Research in Testing, Evaluation andCurriculum in ELT, University of Surrey, Roehampton.
Green, A (2003b) Monitoring score gain on the IELTS Academic Writing module in
EAP programmes of varying duration: Phase 2 report, Cambridge: UCLES
internal report.
Green, A (2004) Making the grade: score gains on the IELTS Writing test, Research
Notes 16, 9–13.
Green, A and Hawkey, R (2004) Test Washback and Impact, Modern English
Teacher 13 (4), 66–70.
Hamp-Lyons, L (1997) Washback, impact and validity: ethical concerns, Language
Testing 14 (3), 295–303.
Hamp-Lyons, L (1998) Ethical test preparation practice: the case of the TOEFL,TESOL Quarterly 32 (2), 329–337.
Hamp-Lyons, L (2000) Social, professional and individual responsibility in language
testing, System 28, 579–591.
Hampton, C and Huang Chung (2003) What makes a good IELTS speaking test?
Perceptions of candidates and examiners, British Council/ IDP Australia IELTS
Research Report.
Hawkey, R (2000) Draft Survey Questions and Recommendations on Specifications
for the Institutional Survey Instrument, consultancy report to the British Council.
Hawkey, R (2001) Phase 2 Focus Groups: Analysis relating Participant Comment toother Current IELTS Research Projects, consultancy report to the British Council.
Hawkey, R (2003a) The Cambridge ESOL Progetto Lingue 2000 Impact Study: Fullreport to UCLES.
Hawkey, R (2003b) IELTS Impact Study: Phase 3: Report to UCLES.
Hawkey, R (2005) A Modular Approach to Testing English Language Skills,
Cambridge: Cambridge University Press/UCLES.
Hawkey, R (2006) Teacher and learner perceptions of language learning activity,
English Language Teaching Journal 60 (3).
Hawkey, R and Green, A (2000) Comments on the Test takers’ Background
Questionnaires, Cambridge: Reports to UCLES.

218

References
Hawkey, R, Thompson, S and Turner, R (forthcoming) The development of a video
database for language educational projects.
Hayes, B and Read, J (2004) IELTS test preparation in New Zealand: preparing
students for the IELTS academic module, in Cheng, L and Watanabe, Y (eds).
Heath, S (1982) Ethnography in education: defining the essentials, in Gilmore, P and
Glatthorn, A (eds) Children in and out of School: Ethnography and education,
Washington DC: Centre for Applied Linguistics.
Henerson, M, Lyons Morris, L and Fitz-Gibbon, C (1987) How to Measure Attitudes,
Newbury Park: Sage Publications.
Henning, G (1987) A Guide to Language Testing, Boston, Massachusetts: Heinle andHeinle.
Herington, R (1996) Test-takers’ background questionnaire design characteristics:
Report to UCLES.
Horak, T (1996) IELTS Impact Study Project, unpublished MA assignment,
Lancaster University.
Hopkins, D (1985) A teacher's guide to classroom research, Philadelphia: Open
University Press.
Hughes, A (2003) Testing for Language Teachers, Cambridge: Cambridge University
Press.
Ingram, D, Bayliss, A and Paul, A (2004) IELTS as a predictor of academic language
performance, British Council/ IDP Australia IELTS Research Report.
International English Language Testing System, Australia (2004) IELTS Band Score
Requirements, IELTS Australia.
International Language Testing Association (ILTA) (1997) Draft code of practice forforeign/second language testing, Lancaster: ILTA.
Jones, N (2001) Reliability in UCLES Examinations, Cambridge: UCLES internalreport.
Jones, N and Hirtzel, M (2001) Appendix D: The ALTE ‘Can do’ statements, inCouncil of Europe Framework of Reference for Languages.
Kane, M (1992) An argument-based approach to validity, Psychological Bulletin 112,
527–535
Kane, M, Crooks, T and Cohen, A (1999) Validating measures of performance,
Educational Measurement: Issues and Practices 18 (2), 5–17.
Kelly, G (1955) The Psychology of Personal Constructs, New York: Norton.
Kennedy, C and Thorp, D (forthcoming) Investigation of linguistic output of
Academic Writing Task 2, in Taylor, L and Falvey, P (eds).
Knapp, L (2003) Summary comments from UK centres, Cambridge: UCLES internalreport.
Kunnan, A (2000) Reviewing the instruments (scales) for impact of the IELTS study,
Cambridge: Report to UCLES.
Larsen-Freeman, D and Long, M (1991) An Introduction to Second Language
Acquisition Research, London: Longman.
Lazaraton, A (1995) Qualitative research in applied linguistics: a progress report,
TESOL Quarterly 29, 3.
Lazaraton, A (2004) Quantitative research methods in language test development andvalidation, in Milanovic, M and Weir, C (eds).
Levine, T (2002) Stability and change in curriculum evaluation, Studies in
Educational Evaluation 28, 1–33.

219

References
Light, R, Singer, J and Willett, J (1990) By Design: Planning research on higher
education, Cambridge Mass: Harvard University Press.
Litwin, M (1995) How to Measure Survey Reliability and Validity, Thousand Oaks:
Sage Publications.
Luxon, A (1996) ODA ELT Project: Nicaragua – A Case Study in ProjectManagement, in Encuentro, University of Alcala de Henares.
MacDonald, R (1974) Evaluation and the control of education, in MacDonald, B and
Walker, R (eds) Innovation, Evaluation research and the Problem of Control,
CARE, University of East Anglia.
May, N and Sigsworth, A (1983) Teacher-outsider partnerships in the observation ofclassrooms, in Murphy, R and Torrance, H (eds).
Mayor, B, Hewings, A and Swann, J (2000) An analysis of the linguistic features of
output from IELTS academic writing tasks 1 and 2, British Council/ IDP
Australia IELTS Research Report.
McNamara, T (1996) Measuring Second Language Performance, Harlow:
Longman.
McNamara, T (2000) Language Testing, Oxford: Oxford University Press.
Mehrens, W (1998) Consequences of Assessment: What is the Evidence? Education
Policy Analysis Archives 6 Number 13: http://epaa.asu.edu/epaa/v6n13.html
Mehta, R and Sivadas, E (1995) Comparing response rates and response content inmail versus electronic mail surveys, Journal of the Market Research Society 37(4) 429–439.
Merrylees, B (1997) A monitoring program of examiner performance in IELTS
Australia centres, British Council/ IDP Australia IELTS Research Report.
Merrylees, B (1998) An impact study of two IELTS user groups: immigration andsecondary, British Council/ IDP Australia IELTS Research Report.
Messick, M (1996) Validity and washback in language testing, Language Testing 13(3), 241–256.
Messick, S (1989) Validity, in Linn (ed.) Educational Measurement, New York:
ACE/Macmillan.
Mickan, P (1998) A study of the response validity of the IELTS writing test – Stage
2, British Council/ IDP Australia IELTS Research Report.
Mickan, P (2004) An ethnographic study of classroom instruction in an IELTS
preparation program, British Council/ IDP Australia IELTS Research Report.
Milanovic, M and Saville, N (1992) Principles of Good Practice for UCLES
Examinations: a discussion paper, Cambridge: UCLES internal report.
Milanovic, M and Saville, N (1996) Considering the impact of Cambridge EFL
examinations, Cambridge: UCLES internal report.
Milanovic, M and Weir, C (eds) (2004) European Language Testing in a Global
Context: Proceedings of the ALTE Barcelona Conference July 2001, Cambridge:
UCLES, Cambridge University Press.
Miller, G (1997) Building Bridges: the possibility of analytic dialogue between
ethnography, conversation analysis and Foucault, in Silverman (ed.) (2001).
Miller, N (2002) Questionnaires, in The Handbook for Economic Lecturers,
University of Bristol: The Higher Education Academy.
Ministry of Education, Italy (1999) Progetto Lingue in Comunicazione di Servizio,
23 settembre 1999, under Circolari ministeriali e comunicazione di servizio at
www.istruzione.it/argomenti/autonomia/documenti

220

References
Mislevy, R, Steinberg, L and Almond, R (2002) Design & analysis in task-based
language assessment, Language Assessment 19, 477–496.
Mislevy, R, Steinberg, L and Almond, R (2003) On the structure of educational
assessments, Measurement: Interdisciplinary Research & Perspectives 1(1), 3–62.
Moore, T and Morton, J (forthcoming) Authenticity in the IELTS Academic Writing
module: a comparative study of Task 2 items and university assignments, in
Taylor, L and Falvey, P (eds).
Morrison, D (1998) The Search for a Method: Focus Groups and the Development ofMass Communication, Luton: University of Luton Press.
Munby, J (1978) Communicative Syllabus Design, Cambridge: Cambridge University
Press.
Murphy, R and Torrance, H (eds) (1987) Evaluating Education: Issues and methods,
London: Paul Chapman.
O’Dwyer, J (forthcoming) The management of a process of curriculum development:a longitudinal case study of the use of formative evaluation for curriculum
renewal and development in a large English language teaching institution,
unpublished PhD thesis, University of Roehampton.
O’Loughlin, K (forthcoming) The impact of gender in the IELTS oral interview, in
Taylor, L and Falvey, P (eds).
O’Loughlin, K and Wigglesworth, G (forthcoming) Task design in Academic Writing
Task 1: the effect of quantity and manner of presentation of information on
candidate writing, in Taylor, L and Falvey, P (eds).
O’Sullivan, B, Weir, C and Horai, T (2003) Exploring difficulty in speaking tasks: an
intra-task perspective, British Council/ IDP Australia IELTS Research Report.
O’Sullivan, B and Rignall, M (2000) Assessing the value of bias analysis feedback toraters for the IELTS Writing module, British Council/ IDP Australia IELTS
Research Report.
O’Sullivan, B and Yu, L (2002) An empirical study on examiner deviation from theset interlocutor frames in the IELTS Speaking Test, British Council/ IDP Australia
IELTS Research Report.
Oxford, R (1990) Language Learning Strategies, New York: Newbury House.
Parlett, M and Hamilton, D (1972) Evaluation as Illumination: a new approach to thestudy of innovatory programmes, in Glass, G (ed.) Evaluation Studies Review
Annual, Vol. 1, Beverly Hills: Sage Publications.
Pavians, M, Bertocchi, D, Hofmannova, M and Kazianka, M (2001) Introducing
CLIL Content and Language Integrated Learning. Guides for teachers andschools to use foreign languages in content teaching, edited in five languages byGisella Langé, Milan: Ministry of Education Regional Direction Lombardy.
Patton, M (1987) How to use Qualitative Methods in Evaluation, London: Sage
Publications.
Popham, W (1987) Two-plus decades of educational objectives, International
Journal of Educational Research 11 (1), 31–41.
Popham, W (1993) Measurement-driven instruction as a ‘quick-fix’ reform stratagy,
Measurement and Evaluation in Counseling and Development 26, 31–34.
Purpura, J (1994) The development and construct validation of an instrument
designed to investigate selected cognitive background characteristics of test takers,
unpublished doctoral qualifying paper, Dept of TESL/Applied Linguistics, UCLA.
Purpura, J (1996a) Modeling the relationships between test takers’ reported cognitive221

References
and meta-cognitive strategy use and performance on language tests, unpublisheddoctoral thesis, UCLA.
Purpura, J (1996b) What is the relationship between students’ learning strategies andtheir performance on language tests? unpublished paper given at UCLES seminar,
Cambridge, June 1996.
Purpura, J (1999) Learner Strategy Use and Performance on Language Tests:
A structural equation modelling approach, Cambridge: UCLES, Cambridge
University Press.
Purpura, J (2001) A proposal for continued research on the UCLES Language
Learning Questionnaire item bank, Cambridge: paper to UCLES.
Purpura, J (2004) Validating questionnaires to examine personal factors in L2 test
performance, in Milanovic, M and Weir, C (eds).
Ramirez, A (1999) Assessment-Driven Reform: The Emperor still has no clothes,Phi Delta Kappan 81 (3), 204–208.
Rea-Dickins, P (1997) So why do we need relationships with stakeholders inlanguage testing? A view from the UK, Language Testing 14 (3), 304–314.
Rea-Dickins, P, Kiely, R and Yu (2003) Student identity, learning and progression:
with specific reference to the affective and academic impact of ‘successful’
IELTS students, British Council/ IDP Australia IELTS Research Report.
Read, J and Nation, P (2002) An investigation of the lexical dimension of the IELTS
speaking test, British Council/ IDP Australia IELTS Research Report.
Reichardt, C and Cook, T (eds) (1979) Qualitative and Quantitative Methods in
Evaluation Research, Beverly Hills: Sage Publications.
Rignall, M and Furneaux, C (1997) Speaking, London: Longman.
Roberts, J (1993) Evaluating the impacts of teacher research, System 21 (1), 1–9.
Roy, S (1998) A general framework for evaluating educational programmes,
in McKay, V and Treffgarne, C (eds.) Evaluating Impact, London: Department
for International Development, 69–74.
Satterly, D (1989) Assessment in Schools, Oxford: Basil Blackwell.
Saville, N (2001) Investigating the impact of international language examinations.
Research Notes 2.
Saville, N (2003) The process of test development and revision within UCLES EFL,
in Weir, C and Milanovic, M (eds).
Saville, N (2004) The ESOL test development and validation strategy, Cambridge:
Cambridge ESOL internal report.
Saville, N and Hawkey, R (2004) The IELTS Impact Study: Investigating washback
on teaching materials, in Cheng, L and Watanabe, Y (eds).
Saxe, L and Fine, M (1979) Expanding our view of control groups in evaluations,in Datta, L and Perfoff, R (eds) Improving Evaluations, Beverly Hills: Sage
Publications.
Scriven, M (1967) The methodology of evaluation, in Tyler, R, Gagne, M and
Scriven, M (eds) Perspectives of Curriculum Evaluation, AERA Monograph
Series on Curriculum Evaluation, 1, 39–83, Chicago: Rand McNally.
Seedhouse, P and Egbert, M (2004) The international organisation of the speaking
test, British Council/ IDP Australia IELTS Research Report.
Selwyn, N and Robson, K (1998) Using e-mail as a research tool, Social Research
Update 21, Department of Sociology: University of Surrey.
Shaw, S (2002) IELTS Writing: revising assesment criteria, Research Notes 9.

222

References
Shaw, S (2004) IELTS Writing: revising assessment criteria and scales (Phase 3),
Research Notes 16, 3–7.
Shohamy, E (1997) Testing methods, testing consequences: Are they ethical? Arethey fair? Language Testing 14 (3), 340–349.
Shohamy, E (1999) Language Testing: Impact, in Spolsky, B (ed.).
Shohamy, E (2001) The Power of Tests: A Critical Perspective on the Use of
Language Tests, Harlow: Pearson Education.
Silverman, D (ed.) (2001) Qualitative Research: Theory, Method and Practice,London: Sage Publications.
Skehan, P (1989) Individual Differences in Second Language Learning, London:
Edward Arnold.
Smith, H and Haslett, S (2003) Attitudes of tertiary key decision makers towards
English tests: A New Zealand case study, British Council/ IDP Australia IELTS
Research Report.
Smith, J (2004) IELTS Impact: a study on the accessibility of IELTS GT Modules to
16–17 year old candidates, Research Notes 18, 6–8.
Snodgrass, J (1977) The Numbers Game, Oxford: Oxford University Press.
Spolsky, B (1989) Conditions for second language learning: Introduction to a
general theory, Oxford: Oxford University Press.
Spolsky, B (1995) Measured Words: The Development of Objective Language
Testing, Oxford: Oxford University Press.
Spolsky, B (ed.) (1999) Concise Encyclopedia of Educational Linguistics, Oxford:
Pergamon.
Swan, A and Gibson, C (2004) The significance of socio-linguistic backgrounds ofteachers of IELTS preparation courses in selected Malaysian institutions, British
Council/ IDP Australia IELTS Research Report.
Taylor, L (1999) Constituency matters: responsibilities and relationships in our
testing community, paper delivered to the Language Testing Forum, University of
Edinburgh, 19–21 November 1999.
Taylor, L (2001) Revising the IELTS Speaking Test, Research Notes 4.
Taylor, L (2001) Revising the IELTS Speaking Test: developments in test format andtask design, Research Notes 5.
Taylor, L (2001) Revising the IELTS Speaking Test: retraining IELTS examiners
worldwide, Research Notes 6.
Taylor, L (2004) IELTS, Cambridge ESOL examinations and the Common European
Framework, Research Notes 18.
Taylor, L and Falvey, P (eds) (forthcoming) IELTS Collected Papers: Research inspeaking and writing assessment, Cambridge: UCLES, Cambridge University
Press.
Templeton, J Farley (1996) The Focus Group: A Strategic Guide to Organizing,
Conducting and Analyzing the Focus Group Interview, New York:
McGraw-Hill.
Tindall, D (2001) Review of Giugni, M, McAdam, A and Tilly, C (eds) How Social
Movements Matter, Minnesota: University of Minnesota Press, Canadian Journalof Sociology 26 (4), 678–682
Toulmin, P (2003) The uses of argument, Cambridge: Cambridge University Press.
Turner, R, Hawkey, R and Thompson, S (forthcoming) Developing a video databasefor classroom observation research.

223

References
University of Cambridge ESOL Examinations (2005) Teaching Knowledge Test
Handbook.
University of Cambridge ESOL Examinations, British Council, IDP Education inAustralia and IELTS Australia (2003) IELTS Annual Review.
University of Cambridge Local Examinations Syndicate (1999) The IELTS
Handbook, UCLES, The British Council, IDP Education, IELTS Australia.
Varghese, N (1998) Evaluation vs Impact studies, in McKay, V and Treffgarne, C(eds) Evaluating Impact, London: Department for International Development,
47–54.
Vint, L (2004) PL2000, an Italian government initiative to improve language learning
brings home the bacon: examining the results, EL Gazette, 293.
Wall, D (1997) Impact and Washback in Language Testing, in Clapham, C andCorson, D (eds) Encyclopaedia of Language and Education, Volume 7: Language
Testing and Assessment, Dordrecht: Kluwer Academic Publishers, 334–343.
Wall, D and Alderson, J (1993) Examining Washback: the Sri Lankan Impact Study,
Language Testing 10, 41–70.
Watanabe, Y (1996) Investigating washback in Japanese EFL classrooms: Problemsof methodology, in Wigglesworth, G and Elder, C (eds) The language testing
circle: From inception to washback, Melbourne: Applied Linguistics Associationof Australia.
Watanabe, Y (2000) Washback Effects of the English Section of Japanese University
Entrance Examinations on Instruction in Pre-College Level EFL, Language
Testing Update 27, 42–27.
Watanabe, Y (2004) Methodology in washback studies, in Cheng, L and Watanabe,
Y (eds).
Weir, C (2004) Language Testing and Validity Evidence, Basingstoke: Palgrave.
Weir, C and Milanovic, M (eds) (2003) Continuity and innovation: Revising theCambridge Proficiency in English Examination 1913–2002, Cambridge: UCLES,
Cambridge University Press.
Weir, C and Roberts, J (1994) Evaluation in ELT, Oxford: Blackwell.
Weir, C and O’Sullivan, B (2002) Does the computer make a difference? Reactionsof candidates to a CBT versus traditional hand-written form of IELTS Writing
component: effects and impact, British Council/ IDP Australia IELTS Research
Report.
Weiss, C (1998) Evaluation, New Jersey: Prentice Hall.
Winetroube, S (1997) The design of the teachers’ attitude questionnaires, Cambridge:
UCLES internal report.
Yue, W (1997) An investigation of textbook materials designed to prepare students
for the IELTS test: A study of washback, unpublished Master thesis, Lancaster
University.

224IndexA

Academic Module (IELTS) 126, 127, 128,
163, 166
accountability 91, 95
accuracy/fluency question 141–43
action plans 48, 49, 51, 53, 76, 172
administrators 7, 14, 15, 17, 22, 28, 46, 61,
71–72, 92, 93, 105, 130, 132, 162, 163
admissions officers 12, 71, 73, 74
fixed date scheduling 131
information availability 131
interview data 128–29
officials 5, 28, 51, 76, 79, 80
Receiving Institution Questionnairedevelopment 74–75
security 72, 129, 131
test availability 131
Agar, M 3
aims/objectives, of impact studies
context and purpose 42, 172
definitions 42–43
IELTS statement of aims 31, 41, 42, 43
PL2000 aims and objectives 27, 32, 41, 42,
43–45, 76
see also impact studies
Airasian, P 17
Alderson, J 7, 8, 9, 12, 13, 15, 21, 36, 37, 54,
57, 58, 59, 60, 61, 63, 64, 65, 69, 106,
107, 116, 119, 175
Alliance Française 42
ALTE see Association of Language Testers in
Europeanalysis of variance (ANOVA) 39
see also inferential statistics
Anatomy of a Context: English Language
Teaching in Italy (Duguid) 44, 169
Anderson, S 88
Andrews, S 8, 13, 59
Annual Review (IELTS 2003) 100
anonymity 91, 95, 174
approaches see experimental research;
impact research, approaches; qualitative/
quantitative approaches
Archer, T 88
assessment 3, 4, 10, 23, 24, 25, 42, 44, 45, 47,
67, 76, 89, 90, 118, 132, 133, 135, 150,
163, 164

see also PL2000 Impact Study, findings
Association of Language Testers in Europe(ALTE)Can Do Project 155
Code of Practice 16, 18, 175
European Year of Languages conference(July 2001) 171
International Conference, Berlin, Germany(May 2005) 167
Meeting, Salamanca, Spain (November
2002) 171
attitudes 22, 27, 56, 162, 165
administrators 128–31
parents 150
receiving institutions 164
school heads/managers 148–50
students 145–47, 150–53
teachers 147–48
test-taker likes and dislikes 118–19
authenticity 10, 69, 125, 126

B

Bachman, L 7, 10, 12, 18, 24, 56, 61, 62, 67,
175, 176
backwash/washback no distinction 8
Baker, C 35
band scores 32, 114
accumulation of 131
cut-off points 35, 72, 128, 129–30
test performance 127–28
see also Common European Framework(CEF) of Reference of Languages
Banerjee, J 36, 54, 57, 58, 60, 65, 69, 74, 79,
109, 129, 165
baseline data 22, 78, 85, 102, 104, 106
Bayliss, A 164
Bell International 83, 91, 136, 138
bias 9, 72, 88, 128, 165
Blackhurst, A 166
Bonkowski, F 68
Bracey, G 17
brainstorming 64–65, 66, 69, 77, 78, 123, 142
British Council
dissemination of information to 171
Going Global Conference (December 2004)
167
Linguistics Seminar (September 1995) 63

225

Indexpartner organisation 6, 25, 82, 175
British Institute, Florence 98, 168
Brown, A 164, 165
Buck, G 15
Bunton, D 41
Business English Certificates (BEC) 20
Business Support Group (Cambridge ESOL)132

C

CAE see Certificate in Advanced English
CALT see Communicative Approach toLanguage Teaching
Cambridge ESOL
Candidate Information Sheets (CIS) 20, 61,
102, 128
examinations see individual examinationsgroups see Business Support; Examinations
and Assessment; Operations; Researchand Validation
revision process model see revision processrole 5, 26–28, 42
validity, reliability, impact, practicality,
principles see VRIP principles
Cambridge-TOEFL Comparability Study(1987-1989) 18
Campus 1 Project, Italy 169
Can Do levels 155, 168, 169
candidate (IELTS) profile
fields of study 101–2
regional backgrounds 99–100
see also Cambridge ESOL, Candidate
Information Sheets; Test-Taker
Background Questionnaire
candidature (IELTS), growth of 18, 156,
166–67case studies 23, 30, 35, 173
CEF levels see Common European Framework(CEF) of Reference for Languages
CELS see Certificates in English Language
Skillscentre selection see pre-survey (IELTS)
Centro Linguistico dell’Ateneo (CLA,
University of Siena) 168–69
Certificate in Advanced English (CAE) 27,
42, 153, 156, 157, 158
Certificate of Proficiency in English (CPE) 20,
156, 158
Certificates in English Language Skills (CELS)
20, 170
Chapelle, C 176
Chapman, D 13
Charge, N 24, 126
Cheng, L 8, 12, 23, 33, 106, 109
City and Guilds 170
Civil Service examinations (India) 10

226

Clapham, C 21, 22, 48, 60, 126
class size 103–4, 158–60
classroom observation 112, 151–52
see also video recordings
Classroom Observation Analysis Form 62,
69–70, 81
Classroom Observation Instrument 62, 69–70,
74, 77, 81, 90, 112, 135, 138–39
CLT see Communicative Approach toLanguage Teachingcodes of practice 16, 17–18, 175
Cohen, A 176
COLT see Communicative Orientation to
Language Teaching
Common European Framework (CEF) of
Reference for Languages 26, 44, 47, 127,
136, 139, 143, 151–52, 155, 156, 168–71
Communicat placement test 168
Communicative Approach to Language
Teaching (CALT)/Communicative
Language Teaching (CLT) 108, 110
communicative competence 132, 136, 142
communicative construct see constructcommunicative needs of students 138–39
communicative opportunity 110, 112, 139,
140, 141, 153
comunicative activities 109, 111, 112, 132,
137–38, 143, 144, 152, 153, 154data sources 135–36
lesson planning 139–41
skills coverage 112, 137–38
teacher views of 136–37
textbook materials 143–44
Communicative Orientation to Language
Teaching (COLT) classroom observation
instrument 69, 81
computer adaptive versus pencil and paper
tests 83, 90, 91, 95, 165, 166
computers 78, 83, 90–91, 95, 98, 144, 165
see also email surveys; information
technology
confidentiality 91, 93, 95, 174
Connell, J 3
consent, informed 91–96
consequential validity 9, 17, 18, 19, 175
constructcommunicative language teaching 12, 13,
17, 48, 52, 140
operationalisation (IELTS) 62–63, 67–72
operationalisation (PL2000) 76–77, 79–82
under- and over-representation 58
validity 9, 10, 18, 19, 58, 175 see also test
use/usefulness, IELTS impact study
findings
washback 2, 13, 16–17, 28, 36, 111–12
Cook, T 30

Index
Cook, V 141
Council of Europe 151, 171
see also Common European Framework(CEF) of Reference for Languages
countries
candidate 99–100
destination 103
coursebooks 68–69, 87, 110–11, 143–44
covariance analysis 39
CPE see Certificate of Proficiency in English
Criper, C 21
critical language testing
consequential validity 18
ethics, and validity 16–18
positive washback, demand for 16–17
public accountability 15–16
Cronbach, L 3, 30
Crooks, T 176
curricular alignment 12, 153
curriculum 4, 11Curtis, A 8, 12
Cushing, S 56, 61, 67
cut-off bands/points/scores 35, 72, 128,
129–31Ddata analysis strategies
analysis of variance 39
clustering 39
commonalities, finding 39
comparison 39
counting 38
data analysis steps 37–40
descriptions 38
drawing final inferences 38, 40
face-to-face data 74–75, 97, 98, 99, 116,
125, 126, 163
factor analysis 38–39
Item Response Theory 55–56
modelling 40
patterns of generalisation 37, 39
rival interpretations, ruling out 38, 39–40
telling the story 40, 54
see also descriptive statistics; inferential
statistics
data management see impact studies, data
management
data protection 174
Davies, A 10, 12, 17, 21, 127
Davis, K 37, 39
demographic data collection instrument 61
Department for International Development(DFID) 2
descriptive statistics/analyses 97, 173
central tendency 39
frequency of 65, 67

kurtosis 56, 65, 67
mean 56, 65, 67, 125, 128
skew 56, 65, 67
standard deviation 56, 65, 67
variation 38, 39, 123
deviant cases 39
Duff, P 31, 33, 34, 35
Duguid, A 44, 169

E

EFL Testing and Evaluation Conference(October 2004) 167
Egbert, M 164
EL Gazette 169, 170, 171
ELTS see English Language Testing Services
email surveys 88–89, 90–91
enabling skills see language, micro-skills
English for Academic Purposes (EAP) 52
English Language Testing Services (ELTS)
21–22, 24, 60
EQS for Windows 65
errors 141
ethical issues 91
see also validity, ethical
European Language Portfolio (ELP) 155, 171
evaluation studies
evaluation, definitions of 3–4
evolving curriculum concept 4
formative/summative evaluation 2–3
illuminative evaluation 3–4
insider/outsider evaluator roles 5–6
monitoring 4–5
process and product 2–3, 5
qualitative versus quantitative approach
29–30
evidence 9, 19, 21, 37, 173
evidence-centred assessment design (ECD)
176
evolving curriculum concept 4
examinations see individual examinations
Examinations and Assessment Group 174–75
experimental researchcontrolled experiments, rarity of 33–34
local contexts 34
qualitative data, generalisations from 34
sampling 34
external certification 5, 26, 42, 44, 143,
150–55F

face-to-face data 74–75, 97, 98, 99, 116, 125,
126, 163
factor analysis see inferential statistics
Falvey, P 165
Farley Templeton, J 70, 82
FCE see First Certificate in English

227

Indexfeedback systems 20
Fine, M 30
First Certificate in English (FCE) 27, 42, 57,
156, 157, 158, 168
Fitz-Gibbon, C 58, 77
Florence Language Learning Gain Project(FLLGP) 98, 168
focus groups 32, 33, 38, 57, 61, 62, 97,
125–26, 129–30
formative/summative evaluation 2–3
four main project areas, specification of
60–61
frameworks see validation, models
Franceschina, F 165
Frankfort-Nachmias, C 88
frequency see descriptive statistics/analyses
Furneaux, C 165

G

Gansneder, B 88
Gardiner, K 57, 67
General Training Module (IELTS) 128, 163,
166
generalisability 34, 35
see also qualitative/quantitative approaches
Gibson, C 164
Goethe Institut 42
good practicecodes of practice 16, 17–18, 175
impact studies, role of 21
test usefulness qualities 18–19
grammar 55, 108, 114, 117, 118, 130, 169
and accuracy/fluency question 141–43
grammar-translation methods 44
Green, A 7, 9, 12, 13, 16–17, 52, 66, 67, 69,
70, 74, 75, 79, 81, 107, 109, 111, 121–22,
166
Gunter, B 70, 82

H

Hamilton, D 3
Hamp-Lyons, L 7, 8, 10, 12, 16, 59, 60, 61,
69, 107, 153
Hampton, C 164
Hargreaves, P 44, 83
Haslett, S 164
Hawkey, R 9, 12, 16, 32, 47, 48, 66, 67, 69,
70, 73, 74, 75, 81, 97, 134, 138, 167, 175,
176
Hawthorne Effect 83
Hayes, B 69, 74
Heath, S 30
Henerson, M 77
Henning, G 55, 56, 57
Herington, R 67
Hewings, A 165

228

high-stakes tests
areas for research 162–63
consequential validity 9, 18 see also
validity
ethics 16, 17–18
sequence of washback/impact 11–12, 168,175
test anxiety 119–12
test fairness 63, 116–18
Hill, K 165
homogeneity, of learning groups 158–60
Hong Kong Certificate Examinations inEnglish 106, 109
Hopkins, D 32
Horak, T 67, 68, 109
How might impact study instruments bevalidated? (Alderson & Banerjee) 36,
54, 60
Hughes, A 8, 56
hypotheses 51–53
alternative 52
IELTS 51, 52
null hypothesis 52
PL2000 51
qualitative/quantitative approaches 51–52

I

IATEFL annual conferences 167, 171
Ideas for research into impact, washback and
IELTS (Alderson) 36, 59–60
IDP Education Australia: IELTS Australia 6,25, 167, 175
IDP International Conference (October 2004)
167
IELTS Annual Review 2003 100
IELTS Band Score Requirements 130–31
IELTS Collected Papers (SILT volume 19)
165
IELTS funded-research programme 6, 25, 128,
164–66, 174
IELTS Handbook 1997/1998 16
IELTS Impact: a study on the accessibility of
IELTS GT Modules to 16-17 year old
candidates 166
IELTS Impact Study, findings
Academic Module 126, 127, 128, 163, 166
candidate profile 99–102
conclusions 131–32
construct 111–12, 121 see also construct
cut-off points 129–31 see also cut-off
bands/points/scores
General Training Module 128, 163, 166
preparation courses 102–18
receiving institution administrators 128–31
teacher perceptions 118, 121–24
test anxiety 119–22

Indextest fairness 116–18 see also test fairness
test module difficulty 122–27
test module validity 119, 124–27 see also
validity
test performance 127–28 see also test
performance
test skills modules see language
test taker likes and dislikes 118–19
IELTS Impact Study Report (April 1996) 63
IELTS Impact Study Report (May 1996) 64
impactcomplexity 13, 14, 19, 23, 31, 33, 36, 38,
40, 59, 106, 108, 115, 121, 151
construct see construct
in educational research 2–6
expanded view of 7–8, 11, 60
sequence of washback/impact 11–12, 168,175
and validity 9–10
and washback 7–8
see also VRIP principles
impact research, approaches
data collection methods 32–33
instrumentation and data, pre- and postvalidation of 36–37
longitudinal studies 31, 32–33
objectives, methods and contexts 31–33
see also experimental research; qualitative/
quantitative approaches
impact studies
aims, objectives (IELTS) 31, 41, 42, 43
aims, objectives (PL2000) 27, 32, 41, 42,
43–45, 76
data collection instrument validation 173
data management systems 173
frameworks and models see validation,modelsgood practice 174
ideas and instrument sharing 174
limitations on scope of 19, 23
location of, in programme and testdevelopment 10–12, 175
meetings 51, 63, 64, 76, 78, 128
pre- and post-validation 23, 36–37
purpose in context 172
research structure 172–73
inferential statistics 97
analysis of variance (ANOVA) 39
covariance analysis 39
factor analysis 38–39, 57, 65
internal consistency 67
Item Response Theory (IRT) 55–56
Kuder Richardson 20 formula 67
regression analysis 39
reliability estimates 67
information technology (IT) 144–45

informed consent 91–96
Ingram, D 164
insider 5–6
Instituto Cervantes 42
instrument delivery, and data return
IELTS 87–89
PL2000 90–91
instrument development and validation process(IELTS)
brainstorming and review 64–65
contextualisation 59–61
cycles of procedure 62–75
operationalisation 62–63, 67–72
participants, demands on 73
piloting and trialling 65, 67–72
post-trial validation 65–66, 67–72
prediction 63–64
rationalisation of instrument use 66, 73–75
receiving institution administrators 74–75
subject teachers 74–75
instrument development and validation process(PL2000)
baseline data 78
brainstorming and review 78
case study school teachers, information from83
contextualisation 61–62
data collection instruments 77–82
operationalisation 76–77, 79–82
piloting and trialling 79–82
prediction 78
principles and prompts for 75–76
team meetings 78
validation 79–82
Instrument for the Analysis of Textbook
Materials (IATM) 57, 66, 68–69, 74, 87,110, 166
instrument versus test validationinter-evaluator reliability 57
item difficulty analysis 55
Item Response Theory 55–56 see alsoinferential statistics
Language Learning Questionnaire
56–57
see also Language Learning
Questionnaire multi-trait, multi-method
validation 58
questionnaires 55, 57
test-retest reliability 57–58
intellectual property 174
inter-evaluator reliability 57
International Language Testing Association(ILTA) 16
International Skills exam (Trinity International)
170
internet 88–89

229

Indexinterviews 22, 29, 32, 33, 52, 97, 136
administrators, data from 128–29
PL2000 participation data 133–34
survey research 35
validation of 55–58, 70–71, 82
see also face-to-face data; Structured
Interview and Focus Group form
item analysis 55, 66, 69
Item Response Theory (IRT) 55–56
see also inferential statistics

J

Jones, N 18, 155K

Kane, M 176
Kelly, G 52
Kendall, L 88
Kennedy, C 165
KET see Key English Test
Key English Test (KET) 42, 151–52, 156,
157, 158, 171
Kiely, R 164
Kiesler, S 88
Knapp, L 75
Kuder Richardson 20 formula 67
see also inferential statistics
Kunnan, A 66, 74, 75, 175
kurtosis see descriptive statistics

L

Lancaster Testing Research Group (LTRG)
63, 64
Lancaster University 60, 61, 63, 64, 69
languageaccuracy 55, 141–43
appropriacy 44, 55, 109, 132, 141, 170
comunicative activities 109, 111, 112, 132,
137–38, 143, 144, 152, 153, 154discourse 13
fluency 141–43
form and function 141–43
functions 15
grammar see grammar
macro-skills (four skills) 69, 70, 109, 111,
112, 113, 114, 116, 118, 119, 122, 123,
124, 125, 126, 132, 137, 138, 147, 163,
167–69micro-skills (enabling skills) 63, 64, 69,
108, 110, 111, 112, 124, 125, 127, 132
test skills modules 107, 109, 122, 123, 124,
125, 126, 127, 132, 164, 170
vocabulary/lexis 141–43
Language Learning Questionnaire (LLQ) 32,
39, 56–57, 58, 73, 78, 83, 91, 95, 120,
168

230

language testing for specific purposes 20
Larsen-Freeman, D 29
Lazaraton, A 29, 30–31
levelseducational 101, 129–30
language see Common European
Framework (CEF) of Reference for
Languages
Levine, T 4
Light, R 45
likes/dislikes, test-taker 118–19
see also IELTS Impact Study, findings
Litwin, M 57
Long, M 29
longitudinal studies 31, 32–33
Luxon, A 69
Lyons Morris, L 77

M

MacDonald, B 3
mail surveys 88, 89
Main Suite exams (ESOL) 23, 42, 51, 86
materials 109–10, 143–45
materials analysis, instrument for see
Instrument for the Analysis of Textbook
Materials (IATM)
May, N 24, 64
Mayor, B 165
McNamara, T 7, 10, 19, 55
measurement driven instruction 13, 17
Mehrens, W 9
Mehta, R 88
Merrylees, B 165
Messick, S 9, 10, 19
methods
impact research 31–33
teaching 106–9, 115, 147–48
Mickan, P 164
Microsoft
Access 65, 97
Excel 97, 98
Milanovic, M 14, 15, 59, 60, 128, 175
Miller, G 35
Miller, N 89
Ministry of Education (Italy) 1, 26–27, 32, 42,
43, 44, 61, 76, 93, 143, 150, 157, 169–70
Mislevy, R 176
models see validation, models
monitoring 4–5, 22, 23–24, 27, 42
Moore, T 165
Morris, L 58
Morrison, D 70, 82
Morton, J 165
motivation 9, 25, 33, 35, 39, 56, 78, 91, 99,
114, 121–22, 145, 146, 147–48, 149, 153,
154, 157, 158, 168

Index
Munby, J 68
Murphy, R 4

N

Nachmias, D 88
Nicaraguan Impact Study 69
Notes on Meeting at UCLES, IELTS Washback
Project (November 1995) 64

Oobjectives see aims/objectives, of impact
studies
observation see classroom observation
O’Dwyer, J 5
officials 5, 28, 51, 76, 79, 80
see also administrators
O’Loughlin, K 164, 165
operationalisation
IELTS 62–63, 67–72
PL2000 76–77, 79–82
Operations Group (Cambridge ESOL) 132,
175
O’Sullivan, B 165
out-of-class activities 146–47
outsider 5–6

PPalmer, A 7, 10, 12, 24, 175
parents 134, 150, 155, 170
Parlett, M 3
Patton, M 34
Paul, A 164
pedagogy 138–39
performance see test performance
Perrone, M 168
PET see Preliminary English Test
piloting 65, 67–72, 79–82
PL2000 Impact Study, findings
assessment see also assessment‘curricular alignment’ 153
instrumental motivation 154
KET, and CEF levels 151–52, 155parent views 155
participation data 133–34
school head views 155
student attitudes 150–51
teacher views 153–54
teaching/learning activities 151–52
attitudes
parents 150
school heads, managers 148–50
student motivation 145–47
teacher motivation 147–48
classroom
accuracy/fluency question 141–43

CALT 135–38
lesson planning and management 139–41
materials, media, modes 143–45
pedagogy and ethos 138–39
exam candidacies 156–57
management and support 158–61
participation data 133–34
test performance 157–58 see also test
performance
Popham, W 13, 17
positive washback 13, 16–17, 111–12
practicality see VRIP principles
pre-survey (IELTS) 49, 85–87
baseline data collection 85
centres breakdown 85–86
course length 86
data collection 88, 89
IELTS courses, clarification of concept of86
nationality groupings 87
student numbers 86
textbooks in use 87
Preliminary English Test (PET) 27, 42, 153,
158, 168, 169
preparation courses (IELTS)
activities 109, 110–11
class sizes 103–4
communicative opportunities 110–12
content, actual 109–12
content/methodology, teacher perception of
106–9
course satisfaction 113–16
coursebooks 110–11
courses 102–4
destination countries 103
episodes 140
IELTS pre-survey 85–87
IELTS research programme 25
increasing demand for 130
lesson management 139–41
lesson planning 139–41
materials, supplementary 109–10
teacher profile 104–5
team teaching 141
Principles of Good Practice for UCLES
Examinations (Milanovic & Saville) 15
Progetto Lingue Full Report (January 20002001) 161
project plans 49, 51
Proposal for work to develop instruments tostudy the washback and impact of IELTS(Alderson) 63
protection, of participants 91, 95
Purpura, J 32, 56, 57, 61, 66, 67, 168, 175

231

IndexQ

qualitative/quantitative approaches
case studies 35
centre selection (sampling) 33–34
choice between 29–31
data analysis strategies 37–39
distance data collection 32
ethnographic research 30–31
ethnomethodological studies 35
experimental research 33–34
generalisability 35
hypotheses 51–52
local contexts 35
longitudinal research 31, 32–33
multiple-method approaches 30
post-trial validation 66
qualitative data collection 32
research questions 31–32
survey research 35–36
questionnaires 22, 32, 52, 97, 136
design of 61
survey research 35–36
validation of 55–57, 67–69, 71–72, 79–81
see also email surveys; Language Learning
Questionnaire; Receiving Institution
Questionnaire; Student Questionnaire;
Teacher Questionnaire; Test-Taker
Background Questionnaire
Quick Placement Test (QPT) (Oxford
University) 57, 78, 83
computer-based adaptive version 90–91, 95

Rradio 110, 146
Ramirez, A 17
raters 57
Rea-Dickins, P 14, 164
Read, J 69, 74
Reading Module
IELTS, revisions 24time pressures 123–25, 126, 163
receiving institution administrators see
administrators
Receiving Institution Questionnaire (IELTS)
71–72, 128receiving institutions 11, 15, 22, 23, 25, 28,
35, 46, 47, 61, 70, 71–72, 75, 99, 128–29,
132, 163, 164, 168–69
reciprocity, IELTS-PL2000 91, 96
regression analysis 39
Reichardt, C 30
Reid, J 69, 74
reliability
reliability estimates 67 see also VRIP
principles
consistency 37, 54, 55, 56, 91, 128, 172

232

inter-evaluator, inter-rater 57
test-retest 57–58
researchareas for new 162–63
CEF levels 169–71
design 48–53, 172–73
ESOL exam enrolments 170
experimental 33–34
funded research program see IELTS fundedresearch programme
IELTS website 166–67
information, dissemination of 167, 171
iterative nature of 11, 19–20, 23–24, 25, 30,
75, 162–63, 173, 174
questions 171
data from responses to 25–26
general 45–46
IELTS 46–47, 51, 52, 110, 116
PL2000 23, 47, 52, 76, 133, 135, 139,
145, 150, 158
receiving institutions, impact on 164
Research and Validation Group see
Research and Validation Group
Research Notes, role and content of 165–66
Siena University-Cambridge ESOL
collaboration project 168–69
SILT Studies 165
skills modules, perceptions of 164–65
teacher support seminars 171
two-way impact 170
VRIP test development focus 167
Research and Validation Group (Cambridge
ESOL) 25, 92, 96, 128, 132, 163–64,
166, 174–75
Research Notes 24, 164, 165–66
Revision Committee, for ELTS 21
revision process
Cambridge ESOL model of 20–22
IELTS, revisions 23–24, 60
Rignall, M 165
Roberts, J 34
Robson, K 88
Roy, S 2

S

Sanzo, R 76
satisfaction (preparation courses)
eclectic methodological approach 115
improved results, need for 116
success
candidate views 113–14
teacher perceptions 114–15
teacher factor, in washback 116
training and support issues 115–16
Saville, N 9, 10, 14, 15, 16, 18, 19, 21, 22, 42,
43, 48, 60, 69, 167, 175, 176, 177

IndexSaxe, L 30
scales 57, 64, 66, 77
see also Common European Framework(CEF) of Reference for Languages
school heads/managers
assessment, views on 155
attitudes 148–50
School Profile form (PL2000) 61, 77, 79, 90
Scriven, M 3
security 129, 131
Seedhouse, P 164
Selwyn, N 88
Shaw, S 24
Shohamy, E 7, 17, 19
Siena University see Centro Linguisticodell’Ateneo
Sigsworth, A 64
SILT see Studies in Language Testing
Silverman, D 35
Singer, J 45
Sivadas, E 88
skew see descriptive statistics
skills 25, 118
activity frequencies, student perceptions of
137–38band scores 127–28
IELTS findings 109, 110–12
listening 63
macro-skills (the four skills) see language
micro-skills see language
most improved 138
reading 63, 123–25, 126, 163
research programme 25, 164–65
skill module validity 125–27
skills coverage 112, 137–38 see also
Communicative Approach to Language
Teaching
speaking 24, 165
test module difficulty 122–27
writing 24, 126, 163, 165
see also language
Smith, H 164
Smith, J 166
Snodgrass, J 52
Snyder, C 13
Speaking Module
IELTS, revisions 24
and SILT 19 165
specialised subject modules, call for 126–27
see also English Language Testing Services
Spolsky, B 16
Sri Lankan Impact Study 60, 69, 106
stakeholders
research activities 25–26
taxonomy of 14–15, 18, 175
standard deviation see descriptive statistics

Statistical Package for Social Sciences (SPSS)
65, 97Structured Interview and Focus Group form
62, 70–71, 77, 82
Student Questionnaire (IELTS) 65, 67, 73–74,
92–93, 118, 124Student Questionnaire (PL2000) 61, 77,
79–80, 90, 135, 145–46, 150–51
Studies in Language Testing (SILT) Volume 15
19–20
Studies in Language Testing (SILT) Volume 16
20
Studies in Language Testing (SILT) Volume 17
20
Studies in Language Testing (SILT) Volume 19
165
study skills 127
subject teachers 72
summative/formative evaluation 2–3
surveys 33, 35, 88–89, 90–91
see also pre-survey (IELTS)
Swan, A 164
Swann, J 165

T

Target Language Use (TLU)
negative washback 12–13
target domain authenticity 125
target language construct overlap (Green’s
model) 13, 16–17, 111–12
tasks see test task validity
Taylor, L 4, 15, 24, 126, 155, 165, 175
teacher professional support 160–61, 171
teacher profile 104–5
teacher qualifications 44
Teacher Questionnaire (IELTS) 58, 65, 68, 74,
103, 106–9, 118
Teacher Questionnaire (PL2000) 62, 77,
80–81, 90, 135, 213teachers, subject 72
see also attitudes; Communicative
Approach to Language Teaching; IELTS
Impact Study, findings; instrument
development and validation process(PL2000); preparation courses; research;satisfaction
teaching
content 106–13
methods 106–9, 115, 147–48
qualifications 44
see also preparation courses (IELTS)
Teaching Knowledge Test (TKT) 170
television 110, 146
TESOL Italy conferences 171
test anxiety
levels of 119–20

233

Indexmotivation 121–22
results 120–21
test availability 131
test development process see revision process
test difficulty
perceived skill difficulty 122–26
specialist subject modules, call for 126–27
study skills, testing of 127
target domain authenticity 125
time pressure 123–24, 125test fairness 63, 67, 116–18
test fees 170
test format 13, 15, 67
Test of English as a Foreign Language(TOEFL) 60, 61, 68, 69, 86
Test of English for International
Communication (TOEIC) 86
test performance 19, 22, 127–28, 157–58
test recognition 131, 155
test revision see revision process
Test-Taker Background Questionnaire (TTBQ)
57–58, 65–66, 67, 73
test task validity 22, 126, 163
test use/usefulness 1, 2, 7, 9, 10, 14, 15, 16,
17, 18, 19, 22, 23, 24, 32, 43, 45, 51, 61,
72, 99, 120, 132, 151, 155, 156, 164, 166,
167, 168–69, 174–77
textbooks see coursebooks
texts
topics 69, 71, 114, 117, 120, 121, 123, 125,132
types 69, 108
Thompson, S 97
Thorp, D 165
time pressures 114, 117, 123–24, 125, 126,
163
Tindall, D 91
TOEFL see Test of English as a Foreign
Language
TOEIC see Test of English for International
Communication
Torrance, H 4
Toulmin, P 176
trialling 65, 67–72, 79–82
triangulation 31, 33, 34, 58, 67, 70, 75, 80, 82,
83, 107
Trinity International Examinations Board 42,
170
Turner, R 96, 97
two-way impact 5, 28, 170

U

UCLES EFL : Bell International PL2000
teacher essay competition 83, 91, 136,
138
UCLES Progetto Lingue Full Report (January
2000-20001) 161

234

use/usefulness see test use/usefulnessVvalidationargument, analysis of 176
assessment use 176
coherence, rigour, revising of 174–77
convergent-divergent 57, 66, 173
documentation 27
evidence-centred assessment design 176
framework see validation, models
interpretive argument 176
item analysis 55, 66
Item Response Theory 55–56 see alsoinferential statistics
models (not CFER) 5, 6, 10, 13, 19–21, 24,
40, 48, 49, 58, 111, 162, 164, 174–77
multi-method 58, 173
multi-trait 58, 173
pre- and post-test validation intent 23
routine research 174
socio-cognitive 19, 57, 175
standards approach 176
see also good practice; instrument
development and validation process;instrument versus test validation
Validation Group (UCLES EFL) 66
validity
concurrent 19, 175
consequential 9, 17, 18, 19, 175
context 19, 175
ethical 16–18
revised models of 174–77
scoring 19, 175
skill module validity 28
survey research 35
theory-based 19, 175, 176
Varghese, N 3
video-data management system 97–98, 112
video recordings 90, 112, 138–39, 168
Vint, L 171
vocabulary 141–43
VRIP principles (validity, reliability, impact,
practicality)
Cambridge ESOL research and validation
programme 25–26
exam targets 18
implementation of 174–75
interrelated nature of 19
research 167
test usefulness 10, 18–19, 175

W

Wall, D 7, 12, 13, 60, 69, 106, 107
Walsh, J 88
washbackcomplexity see impact, complexity

Indexcurricular alignment 12
intensity of 23
intentional 13
intentional and unintentional 12–14
multi-faceted nature of 13–14, 30, 31, 33,
106
negative 12–13
neutral 12, 16
positive 12–13, 16–17, 111–12
and target language construct overlap 13,
16–17, 111–12
two-way 28
see also impact
Washback in Language Testing: ResearchContexts and Methods (Cheng &Watanabe) 33, 34
Watanabe, Y 23, 30, 33, 34, 37–38, 39–40,
59, 107website, IELTS 166–67
Weir, C 19, 40, 109, 128, 165, 175, 176
Weiss, C 2, 3, 4, 5, 14, 29, 33, 35, 37, 42, 43,
45, 48, 51, 70, 71, 83, 91
Wigglesworth, G 165
Willett, J 45
Winetroube, S 109
Writing Module
difficulty of 126
IELTS, revisions 24
and SILT 19 165
writing tasks, validity of 163

Y

Young Learners English (YLE) Tests 156
Yu, L 164, 165
Yue, W 69

235