ResearchNotes

Editorial Notes
Welcome to issue 17 of Research Notes, our quarterly publication reporting on
matters relating to research, test development and validation within Cambridge

Contents

ESOL.

Editorial Notes

1

A Common Solution to a

2

The theme of this issue is language testing in a European context, focussing on
the work of the Association of Language Testers in Europe (ALTE) of which
Cambridge ESOL is a founding member. This issue will describe the history andactivities of ALTE, concentrating on some of the many projects being undertaken

Common European Challenge:

by ALTE members which impact on many thousands of language learners in

The work of ALTE

Europe and further afield.
Test Equivalence and Construct

6

Compatibility across Languages
Development of an Electronic

12

European Language Portfolio
Automated Writing Assessment:

The introductory article by Barbara Stevens outlines the constitution and workof ALTE, concentrating on how its members work together to develop ways ofdescribing and comparing their language examinations and to establish common
levels of proficiency to aid the transferability and portability of language
qualifications. She reports on a number of research projects including the13

ALTE Framework Project and work on the ALTE Code of Practice and Quality
Assurance.

a review of four conceptual models

Peter Hardcastle reports on a project to investigate three European language
Studies in Language Testing18– Volume 18

tests of Reading at Threshold level (B1) in terms of describing the equivalenceof the tests themselves and the constructs they test by comparing their task types

Conference Reports

19

Other News23

and formats. This is an ambitious project and an important aspect of the workof ALTE members. Peter also describes the challenges faced by users of theCommon European Framework of Reference when applying such a tool forcomparing tests across languages and having to balance its theoretical andCall for proposals for

24

ALTE 2nd International Conference,
Berlin, May 2005

practical aspects in the process.
In the following article Simon Fenn describes how an ALTE and EAQUALS
electronic language learning portfolio is being developed with the help ofCambridge ESOL. This project will enable language learners to record their aims
and successes and is one way in which ALTE/EAQUALS members can motivate
language learners to record and celebrate their achievements. Staying with howtechnology is influencing language learning and testing, Stuart Shaw considers
how writing assessment can be automated and reviews four computerised systems
for marking essays and short-answer questions. This area is a key focus for
Cambridge ESOL at present and could radically alter the rating process in future.
We also include the Editors’ notes from the latest Studies in Language Testing
volume, number 18, which contains the key papers presented at the first ALTE

24

Conference held in Barcelona in 2001. This is followed by conference reports
focussing on reading, writing and English for Academic Purposes attended by staff,

The URL for reading/downloading single articles or
issues of Research Notes is:together with a report of the Young Learners Symposium recently hosted byCambridge ESOL.

www.CambridgeESOL.org/rs_notes

We end this issue with a range of Other News items and a call for proposals for

The URL for subscribing to Research Notes is:

the second ALTE conference taking place in Berlin in May 2005. We look forward

www.CambridgeESOL.org/rs_notes/inform.cfm

to this stimulating event and hope to see you there.

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

1

A Common Solution to a Common European Challenge:
The work of ALTEBARBARA STEVENS, EUROPEAN PROJECTS OFFICER, RESEARCH AND VALIDATION GROUP

Introduction
With the breaking down of international barriers between
European states and increasing opportunities for the members of
the workforce to move from one country to another, the need forthe transferability and portability of language qualifications is
unmistakable. First and foremost, employers and employees alikeneed to know what language qualifications gained in various
countries mean – what the holder of a given certificate canactually be expected to do – and how to make meaningfulcomparisons between qualifications gained in different states of a
now enlarged European Union. Employers need to know which
language qualification it is realistic to demand when advertising a
post, and employees have an interest in being able to rate theirown present level of expertise and future training needs.
Consequently, the members of ALTE, the Association of Language
Testers in Europe, have been working together to devise a means ofdescribing and comparing their language examinations to establish
common levels of proficiency in order to promote transnational
portability and recognition of language certification in Europe.

What is ALTE?

from member countries of the European Union and European
Economic Area. However, ALTE also admits Associate Members aswell as Observers (from outside the EU), the main difference beingthat the Associate Members do not have voting rights within theformal arrangements of the EEIG. The use of the word “economic”
in the title of the grouping can be misleading. The ALTE EEIG is not
a commercial organisation and does not have commercial gain as
the group’s objective. In fact, in the regulations for establishing an
EEIG it is made clear that an EEIG itself cannot be formed with theobject of making a profit. In EEIGs, like ALTE, where non-profit
making organisations are involved, their contribution is generally
seen in terms of “the services and skills they can provide”.
ALTE can thus rightly be regarded as an “interest grouping” which
carries out particular activities for the benefit of the members and awider public audience. In the case of ALTE, the member institutions
are almost all publicly-funded organisations such as ministry
departments, cultural institutions, universities, and research
institutes, or other not-for-profit organisations with charitable status.
Furthermore, ALTE became an International Non Governmental
Organisation with participatory status in the Council of Europe in
2003. Participatory status is granted to organisations which areparticularly representative in their field of competence at a broader

ALTE is an association of providers of European foreign language

European level and are capable of contributing to and participating

examinations, which includes many of the major international

actively in Council of Europe deliberations and activities.

bodies in the field of language testing (see list of members on
page 5). It was founded in 1990 by the University of Salamanca
and Cambridge ESOL (then UCLES EFL), and in 2004 now has

ALTE Research Projects

26 full members and 2 associate members, with membership

Since the foundation of the organisation the members of ALTE

growing continually. ALTE members come from a wide variety of

have been working together to develop a framework for describing

organisations, including universities, ministries, state and private

and comparing their language examinations and for establishinglanguage training institutions. Some represent languages spoken

common levels of proficiency. This goal is being pursued by means

widely in the world, others the so-called less widely spokenof an ongoing ‘ALTE Framework Project’. The first stage of thislanguages. As such there is a wide variety of interests within the

project was achieved in 1991 with the production of the first

association, and members are able to learn from each other.

version of an ALTE Handbook which contained descriptions of

ALTE is constituted as an EEIG – European Economic Interest

the examinations offered by all the ALTE members. All of theGrouping – under European Law and this has determined certain

examinations included were described using the same format,

aspects of the way it works. ALTE was set up, like other EEIGs, toand details given of recognition of educational or workplacecreate a legal entity based on EU law to facilitate and encourage

qualifications, the relevant importance attached to different skills,

cross-border co-operation. When setting up the association, it was

the testing focus of each paper, and the number and types offound to be difficult for some potential members to enter into

questions used.

associations under foreign rules or jurisdictions and so the use ofprocedures under European law was thought appropriate. An EEIG,Can Do Statements

in fact, may be set up in any one of the EU Member States, andA central platform of this long-term research project is constituted

operate in any part of the EU but it can also enter intoby the development of the ALTE Can Do Statements. The aim ofarrangements with organisations outside the EU, although these

the Can Do project being ‘to develop and validate a set oforganisations cannot themselves become full members of an EEIG.

performance related scales, describing what learners can

ALTE’s rules on membership eligibility derive from this restriction:

actually do in the foreign language’ (Jones and Hirtzel 2001:244.)

full membership of the association is only open to organisations

The Can Do system, comprising approximately 400 statements

2|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

organised into three general areas: Social and Tourist, Work, andof linking assessments to the CEFR; encourage increased

Study, gives a picture of what typical language users can do in atransparency on the part of examination providers; encourage theparticular language, at a particular level, in one of the skill areas

development of both formal and informal national and(Listening/Speaking, Writing, Reading.) Table 1 illustrates someinternational networks of institutions and experts’ (COE website.)typical Can Do statements for different proficiency levels (A1 to

ALTE fully supports this initiative; indeed, some ALTE members areC2) in each of the skill and general areas.

already working on preparing case studies for the manual, while

Importantly the statements are available in over 16 of theothers have given feedback on the pilot manual.

European languages represented in ALTE. For instance, Catalan,Danish, Dutch, English, Finnish, French, German, Greek, Italian,

Developing Common Standards and Joint Projects

Norwegian, Portuguese, Spanish and Swedish and ‘as language-

In addition to establishing a framework of levels on which theneutral descriptions of levels of language proficiency theyexaminations can be placed, agreement must be reached on theconstitute a frame of reference to which different language exams

standards to which these examinations are produced. ALTE

at different levels can potentially be related’ (Jones and Hirtzel

members share this concern for establishing common standards for

2001:245.) The statements were developed through a rigorous

all stages of the language testing process: that is, for test

process involving leading experts on language assessment anddevelopment, task and item writing, test administration, marking

applied linguistics and data were collected from 10,000 language

and grading, reporting of test results, test analysis and reporting oflearners throughout Europe. The initial calibration of the statements

findings and striving for fairness in the treatment of all candidates

was followed by work on establishing a link between the Can Do

who take their exams. In sharing their knowledge and expertise on

scales and the ALTE examinations as well as to the Common

these issues, they have established a Code of Practice and adopted

European Framework of Reference for Languages (CEFR) levels.

a Quality Assurance approach based on awareness raising and self-

With the expansion of the membership of ALTE, the Frameworkassessment; thereby, recognising the varied linguistic, educational

is being revised and updated as more data becomes available

and cultural contexts within which the examinations are being

across the range of languages of the ALTE members. In fact,

developed and used. It is of enormous value to the thousands ofPeter Hardcastle’s article examines the issues involved fortest takers that the tests developed under the ALTE banner are of

ALTE members in comparing tests and the calibration of tests

good quality and that ALTE members are working towards best

across languages.

practice. The results of ALTE’s work in this area are freely available
on the ALTE website www.alte.org, for instance, in the form of the

Aligning Examinations to the Common European Framework

ALTE Code of Practice and Quality Management Checklists.

In addition, ALTE is currently represented on a Council of Europe

Another of ALTE’s major aims is to collaborate on joint projects

authoring group which is working on the production of a Pilot

and in the exchange of ideas and know-how. There are numerous

Manual for Aligning Language Examinations to the Common

instances of such co-operation and space does not permit mention

European Framework of Reference (2003). The Language Policyof them all here. However, some examples may serve to indicate

Division of the Council of Europe in Strasbourg is developing thisthe amount of work carried out. For instance, several ALTE

in order to assist member states, as well as national andrepresentatives formed part of the Working Group set up tointernational providers of examinations, in relating their certificates

represent various professional interests in the development of theand diplomas to the CEFR. According to the Council of Europe,

Common European Framework of Reference document. Moreover,‘the Manual aims to contribute to competence building in the area

ALTE co-sponsored the development of the Council of Europe

Table 1: Example Can Do statements

Writing(Work)

Reading(Study)Listening/Speaking(Social & Tourist)

C2

CAN make full and accurate notes on all
routine meetings.

CAN make full and effective use of dictionaries
for productive and receptive purposes.

CAN keep up casual conversations for an
extended period of time.

C1

CAN draft a set of straightforward instructions.

CAN assess appropriacy of source material
quickly and reliably.

CAN show visitors round and give a detailed
description of a place.

B2

CAN write a non-routine letter where this isrestricted to matters of fact.

CAN scan texts for relevant information, andgrasp main topic of text.

CAN ask for advice and understand the answer,
provided this is given in everyday language.

B1

CAN write a simple routine request to acolleague.

CAN assess whether a textbook or article is within
the required topic area.

CAN go to a counter service shop and ask formost of what (s)he wants.

A2

CAN leave a simple message giving information.

CAN understand simple visuals on familiar topics.

CAN express an opinion about food.

A1

CAN write a simple routine request to acolleague, such as ‘Can I have 20X please?’

CAN read basic notices and instructions.

CAN ask simple questions of a factual natureand understand answers expressed in simple
language.

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

3

Vantage level specifications, written by Dr John Trim and Professorof the meeting, where a wider local audience is invited to comeJan van Ek (van Ek and Trim 2001) and ALTE members wereand find out more about the work of ALTE, has proved very

involved in a project to provide feedback to the authors. Members

successful. At the last such conference day in Krakow in April

commissioned expert reviewers, in their organisations or countries,

2004, presentations were made, by among others, representatives

to write reports on Vantage level. A series of regional workshops

from the European Commission and the Council of Europe, to analso took place in various member countries, at which the reports

audience of over 100 people, on the Common European

were discussed and additional feedback given to Dr Trim, who

Framework of Reference in the Polish context; limitations of theattended each workshop. ALTE made the results freely available toFramework in developing comparable examinations and tests; thethe Language Policy Division of the Council of Europe.

manual for relating examinations to the CEFR as well as Europe

ALTE has taken part in the Council of Europe’s European
Language Portfolio project and has produced, together withand Languages: The Lingua action of the Socrates programme.
Many of the outcomes of ALTE work are publicly available.

EAQUALS (European Association for Quality Language Services)

For instance, in 2001 an international public conference, whicha version of the language portfolio for adult learners. This portfolio

was attended by over 300 delegates, was organised by ALTE in(accredited model 06.2000) is now available in various versions

Barcelona to mark the European Year of Languages. The

including Croatian, English, French, German, Italian andproceedings of this conference were published in 2004 as Studies

Romanian. A new version in Basque and Spanish is also inin Language Testing Vol.18. The second major international

preparation and will be published shortly. ALTE and EAQUALS are

ALTE conference will be held in Berlin from 19–21 May 2005.

currently collaborating with the Council of Europe on producing

The call for papers is included in this issue on page 24.

an electronic version of the language portfolio (see Simon Fenn’s
article).
Co-operation among the ALTE members has produced significant

Conclusion

new ALTE products. Organisations from countries in eastern Europe

ALTE members’ shared concern for high standards in language

have become interested in developing new testing systems andassessment and for fair treatment of the candidates who take their

ALTE members from Estonia, Hungary, Latvia and Lithuania worked

exams has provided a focus for working together and a key

together on a Lingua project (submitted in 2003) to extend various

objective of the Association. ALTE members are responsible forpreviously developed ALTE LINGUA products in the field of test

testing millions of individuals all over the world, thoughdevelopment to Estonian, Hungarian, Latvian and Lithuanian:predominantly in Europe, and ALTE is thus in direct contact with•

A Glossary of Testing Terms to be used by test developers andenormous numbers of stakeholders: the staff and colleagues

teachers (LINGUA reference: 94–09/1801/UK–III)

working on the exams themselves; professional consultants•

Can Do Statements to be used by language learners, users,
teachers and testers (94–03/1629/UK–III)••

working on producing and administering the exams, such as item
writers and examiners; teachers and other educationalists,
including course writers and materials producers; government

Guidelines for Test Item Writers to be used by those producing

bodies; test takers and their sponsors (parents, employers); users ofmaterials for tests (93–09/1326/UK–III)

test results such as employers and university departments, as well

Checklists for Test Evaluation to be used by test developers

as many others. ALTE’ s approach has been and remains an attempt(93–09/1326/UK–III).

to identify the roles and responsibilities of key stakeholders in the
ALTE institutions and to apply a system of quality assurance with

A further LINGUA product, the Breakthrough Level

flexibility according to the specific features of each organisation(37098–CP–1–96–1–GR–LINGUA–LD) was also extended to(i.e. the different ALTE Members and their stakeholder groups).

Estonian, Latvian and Lithuanian.

In seeking to establish standards, it is not the aim to make all

A set of descriptors for Breakthrough level was, in fact, first

ALTE Members conform to the same models of assessment for all

developed by a group of ALTE members (from Finland, Greece,

24 languages represented, and it is important to recognise theIreland, Norway and Sweden) who felt that there was a need invaried linguistic, educational and cultural contexts within which

their languages for a set of descriptors at a lower level (below thethe examinations are being developed and used. An appropriate

Waystage Level). The group therefore worked on a possible

balance is required between the need to guarantee professional

approach to a Breakthrough level, which is now being used as

standards to users, and the need to take into account the differing

one of the main reference documents for producing a full set oforganisational features of the ALTE institutions and the contexts in

Council of Europe Breakthrough descriptors, a project which

which their exams are used. Standards are not imposed fromALTE is now concerned with.

“outside” but are established through the mechanism of the systemitself and the procedures for monitoring standards are based on

ALTE Conferences

awareness raising and self-assessment in the first instance with peer
monitoring introduced at a later stage to confirm that the minimum

Members of ALTE hold 3-day meetings twice a year to discuss

standards are being met. The aim for all ALTE Members is toprojects and plan further work, with each meeting being hosted

continue to share expertise and aim at best practice models

and chaired by a different member organisation. A development,

through an on-going process of development.

which first took place in 2000, to include a conference day as part

4|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

Cambridge ESOL, as one of ALTE’s founding partners, takes

the two institutions formalised their collaboration by creating

seriously its commitment to improving its own and others’

the Norwegian Language Testing Consortium.

language tests through the work of organisations such as ALTE.
Recently Cambridge ESOL staff enjoyed an internal seminar led by•

Universidad de Salamanca, Spain – in charge of developing,

Barbara Stevens and Peter Hardcastle which informed staff clearlymarking, setting and evaluating the tests leading to the Institutoof the membership and role of ALTE and what challenges lie ahead

Cervantes Diplomas de Español como Lengua Extranjera

for this organisation.

(DELE).
•

Universidad de Lisboa, Portugal – Departamento de Língua e
Cultura Portuguesa (DLCP) is the Department of the University

ALTE Membersof Lisbon where Portuguese as a Foreign and Second Language•

Alliance Française, France•

Eusko Jaurlaritza – Department of Education, Universities andis taught.
•

Research in the Basque Government.

Universita per Stranieri di Perugia, Italy – University faculty of
Italian Language and Culture and special research centres.

•

Centre de Langues, Luxembourg•

Centre International d’Etudes Pedagogiques, France – public

Foreign Language of the University of Athens forms part of theinstitute of the French Ministry of Education.

Interdepartmental Programme for the Teaching of Greek as a

•

•

•

•

a Foreign Language Certificate.

University of Cambridge ESOL Examinations (Cambridge ESOL)
has charitable status as an institution of the University ofNetherlands – National Institute for Educational Measurement.

Cambridge.

Danish Language Testing Consortium – The Ministry for•

Education and the University of Jyväskylä developed the

Copenhagen, Studieskolen in Copenhagen.

National Certificate as a joint non-profit project.

Generalitat de Catalunya – The Direcció General de Política

•interdisciplinary research and teaching centre of the

Policy in Catalonia.

Jagiellonian University in Kraków.

Instituto Cervantes, Spain – state organisation responsible for•

Languages and Literatures of the Faculty of Arts, University of
Ljubljana.

Extranjera (DELE).
Idegennyelvi Továbbképzõ Központ (ITK) (Centre for•

Philology of the University of Vilnius.

non-profit institution under the auspices of Eötvös Lóránd
•the form of a limited company under German charity law.

supervision of the Ministry of Justice deals with all issues•

adults who are learning Welsh as a second language.

Qualification Centre, Estonia) – the executive organisation oforganises national exams.
Stockholms Universitet, Sweden – Centre for Research on

Associate Members•

for Education. The Department of Scandinavian Languages in

University of Sofia “St Klimint Ohridski” Department of
Language Learning, Bulgaria

Bilingualism produces the National Test of Swedish for Adult
Immigrants (Sfi-provet) at the request of the National Agency

Welsh Joint Education Committee, Wales – WJEC are atpresent responsible for providing examinations specifically for

Eksami- ja Kvalifikatsioonikeskus (National Examination and
the Ministry of Education and National School Board which

WBT Weiterbildungs-Testsyteme, Germany – The WBT(Further Education Test Systems) is an institution organised in

Republic of Latvia, Naturalisation Board – under thepertaining to the citizenship of Latvia.

Vilnius University Department of Lithuanian Studies,
Lithuania – Department of Lithuanian Studies, Faculty of

Advanced Language Learning, Hungary) – independent,
University (ELTE).

Univerza v Ljubljana Slovenia – The Centre for Slovene as a
Second/Foreign Language in the Department of Slavonicthe management of the Diplomas de Español como Lengua•

Uniwersytet Jagiellonski – The Polish Institute is an

Lingüística (DGPL) is the General Directorate of Linguistic••

University of Jyväskylä, Finland – The National Board of

Refugees, Immigrants and Integration, The University of

Goethe-Institut, Germany•

University of Cambridge ESOL Examinations, UK – The

CITOgroep Instituut voor Toetsontwikkeling, Arnhem, the••University of Athens, Greece – the School of Greek as a

Foreign Language.

Certificaat Nederlands als Vreemde Taal, Leuven –government-subsidised non-profit organisation for the Dutch as

•

•

•

Russian Language Testing Consortium: Moscow University and
St Petersburg University

co-operation with the Universities of Gothenburg, Linköping,
Lund, Malmö, Umeå and Uppsala is responsible for producing,•analysing and developing TISUS (Test In Swedish for University

References and further reading

Studies).

ALTE: http://www.alte.org

Studieforbundet Folkeuniversitetet Universitetet i Bergen

ALTE (2004) Multilingual Glossary of Language Testing Terms (Latvian)Riga: LR Naturalizacijas parvalde.

Norsk Spraktest – Studieforbundet Folkeuniversitetet and the
University of Bergen have been working together under theterms of a collaboration agreement since 1990. In spring 2000,

ALTE (2004) Materials for the Guidance of Test Item Writers (Latvian)Riga: LR Naturalizacijas parvalde.

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

5

Council of Europe http://www.coe.int
Council of Europe (2003) Preliminary Pilot Manual for Relating Language
Examinations to the Common European Framework of Reference for
Languages: Learning, Teaching, Assessment, Strasbourg: Language
Policy Division.
Jones, N and Hirtzel, M (2001) Appendix D, in the Common European
Framework of Reference for Languages: Learning, Teaching, Assessment.
Council of Europe, Cambridge: Cambridge University Press.

Milanovic, M and Weir, C (Eds) (2004) European Language Testing in a
Global Context, Proceedings of the ALTE Barcelona Conference,
Studies in Language Testing Vol.18, Cambridge: Cambridge University
Press.
van Ek, J A and Trim, J L M (2001) Vantage, Cambridge: Cambridge
University Press.

Test Equivalence and Construct Compatibility acrossLanguages
PETER HARDCASTLE, ALTE VALIDATION OFFICER, RESEARCH AND VALIDATION GROUP

Introduction
Cambridge ESOL is currently involved in validation studies of threesuites of European language tests in conjunction with its European
partners within ALTE: the University of Salamanca/the Cervantes
Institute, The Universitá per Stranieri of Perugia, and the Goethe
Institute. All partners either already produce or intend to produce
a series of tests aligned to at least five levels of the Common
European Framework of Reference for Languages (CEFR) (Councilof Europe 2001), from A2 Waystage to C2 Mastery. This article
reports on an investigation into the constructs, task types andformats of three European language tests of reading at CEFR
Threshold Level B1. The tests involved in this study are:•the Certificati di Conoscenza della Lingua Italiana (CELI);
levels 1–5••

design and calibration in the context of individual languages.
Within ALTE, over the last decade and before, much effort had
already been made to develop strategies for comparing tests
across languages and the contribution made by this thinking to
the development of the CEFR was substantial. The Common
European Framework, recognising the importance of this work(see Appendix D of the CEFR – The ALTE ‘Can Do’ Statements),
aims to be complete enough to constitute an adequately descriptivemodel of cross-linguistic language proficiency, yet concise enoughto be a practical tool of real use to language teachers and test
designers. In a way, however, it is caught between a rock and ahard place, in that its theoretical completeness and hence its
descriptive complexity tend to function in indirect proportion to its
potential for practical application. Clearly, the framework should
encourage, not deter, everyday use, serving as a specialists’ andthe Diplomas de Español como Lengua Extranjera (DELE);

non-specialists’ vade mecum in the language teaching and language

levels inicial, intermedio, superior

assessment professions. Furthermore (and most importantly for this

The Goethe Institute’s suite of German tests, including Start

project), it should be demonstrably and overtly applicable toDeutsch 1, Start Deutsch 2, Zertifikat Deutsch (and Zertifikat

language testing procedures and language test development. To thisDeutsch für Jugendliche), Zentrale Mittelstufenprüfung (ZMP).

end ALTE is developing a Manual and a Code of Practice whose

The intention of the project is to link these tests to the CEFR,
using established Cambridge ESOL validation procedures,
including the application of IRT (Rasch) item calibration andanchoring techniques.
The project sets out to study the equivalence of individual tests
over a given time frame examining issues of stability, consistency
and reliability, as well as examining test equivalence acrosslanguages; for example, did the German B1 level test function at
the same CEFR level as the Italian and Spanish B1 level tests in thesummer 2003 administrations?

What is the CEFR?

aims are to enable language testers to conceptualise test alignment
within the Framework and to consider ways in which justifiableclaims, relating tests to CEFR levels, can be made (ALTE 2001,
Council of Europe 2003).
As well as its declared potential as a framework ‘for the
specification of the content of test and examinations’ (Council ofEurope 2001:18), the CEFR establishes a framework for thedescription of language proficiency to be applied as benchmark
criteria for the definition of proficiency levels across languages;
i.e. a B1 threshold level qualification in French is qualitatively
equivalent to a B1 in German or Lithuanian etc. It sets out toprovide ‘a common set of proficiency statements (which) will
facilitate comparisons of objectives, levels, materials, tests andVery broadly speaking, the CEFR is a set of band descriptors/

achievement in different systems and situations’ (ibid. page 16).

Can Do statements (around 400), which set out to describe

It is an implicational scale employing band descriptors which

language proficiency in terms general enough to be relevant to all

identify the salient linguistic features employed by users of theEuropean languages, yet specific enough to provide a basis forlanguage at any one of six posited levels of proficiency. The scale

generating thought and discussion about test specifications,is cumulative and hierarchical, in that descriptors of competences

6|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

operationalised at lower levels are viewed as prerequisites for
effective performance at the higher levels. In the words of BrianNorth (2004), one of the CEFR’s main authors, ‘the descriptor

Issues related to theory-based validity•

scales can be used for setting objectives, for self-assessment, forteacher assessment and for relating assessment results to common•

frequently used in the descriptors. Such terms are subjective

This paper focuses on some of the assessment-related aspects ofand may be interpreted differently from one user to the next.

the Framework.
•

established hierarchical model of cognition or of language

A number of issues relating to the CEFR, its robustness,development, such as the paradigm developed by Bloom

completeness and the applicability of its band descriptors and(1956). (Note, however, that the ALTE Can Do statements andCan Do statements continue to be raised (Weir 2004). Similarly,

the CEFR display an interesting degree of congruence with thisAlderson et al. (2004) (also known as the Dutch Constructs Group)

early paradigm, at least in terms of the verbs chosen to express

have pointed out that, although the CEFR claims to be a useful tool

instrument for this purpose and needs considerable elaboration
before it can be of significant applicability in this area. Theselimitations impact on the feasibility of this Cambridge ESOL/ALTE
project and fall broadly into two categories, relating to context
validity and theory-based validity (see Weir 2004).

The terms employed in the Framework are descriptive andtaxonomical rather than developmental and do not reflect an

Issues with the CEFR

examinations (ibid. page 19), the Framework is an incomplete

Use of relativistic or value-laden terminology. Words like
simple, straightforward, short, slow (speech delivery) etc arereference points’.

for use in the content specification of language tests andCognitive load (cognitive processing) is an important componentof task design as yet inadequately considered by the CEFR.

the target abilities.)•

It has been suggested that the CEFR does not fit with any
current model of language acquisition or theory of language
development (Fulcher 2004), though the frameworkspecifically does not set out to apply any such model andincorporates a disclaimer to this effect (ibid. page 18, 2.3.2).

It is not the intention of this article to address these issues at any
length, but it is important for any user of the Framework to be

Issues relating to context and context validity•

Task types are not specified by the framework and the degree
to which this impacts on test design and interpretation issignificant.

•Channel needs to be specified in the descriptors. It is mucheasier to arrange a meeting face-to-face with a friend than tonegotiate a job interview by telephone, yet ability to performin both of these very different situations may be defined by thesame performance descriptor or Can Do statement.

•

The context in which the task is to be performed is largely
unspecified by the framework. It does not adequately take
into account theories of cognitive processing, socio-cognitive
development and the contextual features of language use(Weir 2004) which lead to deficiencies in its application
as a descriptive/comparative tool and to inadequate
consideration of elements of authenticity (situational as well asinteractional).

••

aware of these limitations when attempting to employ the
Framework for practical ends in language assessment. Without
recognition of the limitations, as well as the strengths, of the CEFR
it would not be possible to develop a well-founded set of basereferences into a fully operational structure for the meaningfulcomparison of tests.

Cross-language comparability
Cross-language comparability of language tests has to be one of thetoughest challenges currently facing the developers of the CEFR andthose ALTE members attempting to implement it. Although work has
been going on in this domain for several years, much researchremains to be done. For the Cambridge ESOL/ALTE project thiswork is fundamental, as the basic credibility of the CEFR rests on
the assumption that comparability of language tests across the levels
posited by the Framework can be established by empirical
procedures, defined by a set of universal performance descriptors
and validated using IRT (Rasch) anchoring techniques. By setting up

Can Dos in one language may involve the application of morethis project, Cambridge ESOL and its ALTE partners have establishedsophisticated language skills than in another language, calling

an agenda for research into cross-language test alignment based on

into question the feasibility of applying a common set ofdata collection and analysis. This approach will allow for additionaldescriptors to different languages. This is a major issue for thecalibration to take place for each set of language tests in their own

project and is explored further below.

right (German, Italian and Spanish) using classical and IRT methods.

How well can they do it? The level of performance associatedIn addition, the question of alignment of the tests to the relevantwith the descriptor is not normally specified or quantified.

CEFR levels will be addressed more systematically than in the past.

It is possible to operationalise descriptors at various levels,

The CEFR performance descriptors, largely in the form of Can Do

depending on the complexity of the language used, forstatements (cf ALTE 2002), have been translated into all of theexample, Can make and respond to invitations, suggestions,languages with which this study is currently concerned, but doubt

apologies is a set of functions which could activate complex

has been expressed as to whether the translated equivalents convey

linguistic exponents, but which could also be performed using

precisely the same meaning.

simpler language at A2 level, for which it is one of the Can Do
specifications.

The concept of Can Do in the context of linguistic performancedescriptors is one which does not lend itself well to concise and

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

7

accurate translation into other European languages. The notion ofwhich the Cambridge ESOL/ALTE project has found to be useful inability is subdivided into more semantic categories in otherbroadly considering test equivalence. The number of quantitative

European languages than it is in English; e.g., two in Frenchand qualitative fields which could be included in such an(savoir/pouvoir), two in German (können/wissen), three in Russian

instrument are limited only by the imagination of the designer and(уÜÕâì/×ÝÐâì/Üоçì) resulting in sometimes lengthy translations ofby the practicalities involved in applying the framework to thethe Can Do concept, of the order:

analysis of test construct, format and data. The elements included••

Especificaciones de capacidad lingüística (“puede hacer”)

are based on informed judgements and on practical experience(Spanish)and are by no means complete. The three tables containKann-Beschreibungen, Kompetenzbeschreibungen (orFähigkeitsbeschreibungen) (German)•

Indicazioni de capacitá (Italian)•

Capacités de faire (French)

All of the above are less succinct and less exact than the English
original.

information under the following headings for one language test
each for Italian, Spanish and German:

Key to categories in the comparability tables
1. The question or section number in the test.
2. A brief description of the task type.
3. Task format; e.g. multiple choice (MCQ), multiple matching,

Similarly, the exponents of the performance descriptors, when
expressed in lexico-grammatical terms, may fall into very different
categories of difficulty, further complicating the issue of crosslanguage test calibration based on Can Do statements and/or
performance descriptors. For example, the ability to carry outtransactions to obtain goods or services (Can ask people forthings and give people things – level A1) is operationalised at amuch higher level of linguistic skill in English when grammaticallycomplex issues of modality and conditionality may arise, than it
is in, say, Russian or Arabic where such grammatical complexities

gap-filling, live interview.
4. Number of items in the task.
5. Relevant descriptor refers to the page of the CEFR manual
where an appropriate performance descriptor or Can Do
statement can be found. Where two descriptors are identified
they are both applicable. Where descriptors are identified at
two different levels, this should be viewed as a potential
difficulty for alignment to the Framework.
6. Text/discourse type – an open-ended description of the

do not necessarily occur in the linguistic realisation of this

discourse category into which the text/target language

function.

behaviour is best fitted.

Such incompatibilities are many and varied and present aformidable obstacle to a framework of reference attempting tocompartmentalise, describe and calibrate linguistic ability
independently of the language in which such abilities are

7. PRIM: production, reception, interaction, mediation (oral or
written).
8. Skill: reading, writing, speaking or listening.

expressed. The comparability tables proposed overleaf hope to go

9. Sub-skill is an open-ended sub-division of skill.

some of the way towards resolving these issues.

10. General strategy: components of strategic competence which
are mainly paralinguistic or metacognitive behaviours
associated with performance on the task (e.g. skimming,

Developing Comparability Tables

scanning, deducing meaning from context).

In response to the CEFR’s descriptive framework and in spite of its

11. Construct statement – users of the table are not expected tolimitations, this article attempts to construct the modest beginnings

enter the full statement into this field, but rather to indicate

of a simplified set of tabulated reference points for the comparison

whether a construct statement is available for the informationof European language tests, incorporating the elements identifiedof stakeholders and where this might be found. Developers

above as omissions from the original Framework. The tests undershould be encouraged to produce construct statements forstudy are threshold level (B1) measures of reading (written

each test and sub-test in order that stakeholders might obtain

reception) in Spanish, German and Italian, but this concept ofa full understanding of what is expected of candidates whenhorizontal comparability is, of course, applicable to other language

taking the test, and why.

tests and at other levels of proficiency. The tabulation is a
simplification of test qualities and could be supplemented by manyother comparative categories (functional, notional, linguistic,
statistical etc.) all of which ultimately form parts of a moreexhaustive comparative framework which will need to be
produced.
The tables presented overleaf are an attempt to establish the

12. Register – various categories, including formal/informal,
familiar, legal, journalistic etc, or a combination of thesecategories.
13. Task time allocation – Approximate or exact time allocated tothe task.
14. Linguistic load – the expected level of difficulty (lexical,

comparability of three language tests in tabular form. They serve

grammatical) of the language used (actively or passively) in the

only as examples as to how test comparability across languages

task. For tests of spoken language or listening tests, additional

may be conceptualised. As with the CEFR itself, a disclaimer in this

aspects of difficulty might be included (speed of delivery,

context is more than appropriate: the tables incorporate elements

accent etc). This category has to be judged by largely intuitive

8|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

criteria. In itself it is, of course, subjective, but when viewed in

One of the main absences relating to all three tests is that of the

conjunction with other categories it may help to identify

Rasch calibration results (column 20) on which this project is

discrepancies or anomalies. (Clearly, a more precise scale will

currently working. These results will shortly be entered andhave to be developed in association with this field. Such a

reviewed along with the other test properties relevant to calibration

scale could be six-point Likert-type scale, compatible with theand comparability.

six levels of the CEFR and could be developed along the linesof the work of Van Ek and Trim (1998, 1999, 2001), referring
specifically to their level-specific grammatical summaries andto the lexical indices.)
15. Cognitive load – non-linguistic elements of information

Conclusion
Although at an early stage of development this basis for thecomparison of tests across languages shows promise, especiallywhen used in conjunction with the CEFR and the Rasch anchoringprocessing which may affect the way the task is performed.

techniques used by Cambridge ESOL. More work needs to be

An example is a writing task which uses an information

done, more categories (both quantitative and qualitative) need totransfer input prompt requiring non-linguistic interpretationalbe established, but as a rough guide to test equivalence the tables

skills and an element of world-knowledge, on which adequate

used in this study are already proving useful in establishingcompletion of the task may depend. Such a question would

tentative equivalence among European language tests.

have a high cognitive load. Again, this is a category whose
values must be established by intuitive criteria, based on
experience. A more refined scale than the one used here
(low-medium-high) is under development.
16. Lexical load – a quantitative category which considers theapproximate number of lexical items to be processed per
answer (the lexical input: response ratio). This identifies thenumber of words which must be read/heard/understood/written
for each point earned. In the tests investigated, variation in thisarea is surprisingly large. Care must be taken when word
counts for lexically agglutinative languages (e.g. German) are
interpreted.
17. Situational authenticity – a judgement of the degree to which
the task is representative of a real-life activity. Again, a morerefined scale than the one used here (low-medium-high) is
under development.
18. Interactional authenticity – a judgement of the extent to which
the task is concomitant with a real-life language processing
task. A task could have high situational authenticity but low
interactional authenticity, or indeed vice-versa (see Bachman1990).
19. Rasch difficulty range (target) using Cambridge ESOL scaled
logits. Raw logit scores could equally well be used, provided
that they were related to a global difficulty scale establishedempirically by Rasch calibration techniques.
20. Rasch difficulty range (established) – the difficulty of task andtest after calibration.
A comparison of the three tables (even though they refer only to
one skill in only three of the dozens of European languages tests
currently available) begins to address some of the issues outlined

References and further reading
Alderson, J C, Figueras, N, Kuijper, H, Nold, G, Takala, S, Tardieu, C(2004) Specification for Item Development and Classification within
the CEF; Reading and Listening (English, French and German): The
Dutch CEF Construct Project, paper presented at a workshop to discussresearch into and with the Common European Framework, Universityof Amsterdam.
ALTE (2001) Principles of Good Practice for ALTE exams, Revised Draft,
Oct 2001, ALTE.
ALTE (2002) The ALTE Can Do Project, Association of Language Testers in
Europe.
Bachman, L (1990) Fundamental Considerations in Language Testing,
Oxford: Oxford University Press.
Bloom, B S and Krathwohl, D R (1956) Taxonomy of Educational
Objectives: The Classification of Educational Goals, by a committee ofcollege and university examiners. Handbook I: Cognitive Domain,
New York: Longmans, See: http://faculty.washington.edu/krumme/
guides/bloom.html
Council of Europe (2001) Common European Framework of Reference for
Languages: Learning, Teaching, Assessment, Cambridge: CUP.
Council of Europe (2003) Preliminary Pilot Manual for Relating Language
Examinations to the Common European Framework of Reference for
Languages: Learning, Teaching, Assessment, Strasbourg: Language
Policy Division.
Fulcher, G (2004) Are Europe’s tests being built on an ‘unsafe’ framework?
Education Guardian. Retrieved from:http://education.guardian.co.uk/tefl/story/0,5500,1170569,00.html
North, B (2004) Europe’s framework promotes language discussion,
not directives. Education Guardian. Retrieved from:http://education.guardian.co.uk/tefl/story/0,5500,1191130,00.html
Taylor, L (2004) Issues of Test Comparability, Research Notes 15,
Cambridge: Cambridge ESOL.

earlier in this article. The categories in the table are not exhaustive

van Ek, J A and Trim J L M (1998) Threshold 1990, Cambridge:
Cambridge University Press.

and it would not be difficult to include an additional twenty— (1999) Waystage 1990, Cambridge: Cambridge University Press.

categories and remain incomplete. Clearly, however, for such an— (2001) Vantage, Cambridge: Cambridge University Press.

instrument to be useful it must trade off the comprehensive against
the practical and reach a useful compromise. From a review of theinformation in the tables a degree of compatibility among the threelanguage tests can be observed, though blank spaces in the gridindicate that information remains to be collected and inserted.

Weir, C J (2004) Limitations of the Common European Framework ofReference (CEFR) in Developing Comparable Examinations and Tests,
paper delivered at ALTE conference, Kraków, April 2004.
— (forthcoming) Language Testing and Validity Evidence, Basingstoke:
Palgrave Macmillan.

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

9TABLE 1: TEST COMPARABILITY TABLE FOR READING COMPREHENSION TASK TYPES IN DELE
Test: DELE Level 1 (Nivel Inicial)

CEFR Level: Threshold (B1)

Language: Spanish

Target Skill/s: Reading

Total Time: 2 hrs

1

2

3

4

5

6

7

8

9

10

Question

Task Type

FormatNo.
items

Relevant
descriptor (CEFR)

Text/discourse type

PRIM
(o/w)

Skill

Sub-skill

General strategy

1

Reading comp.
passage

MCQ(4 options)5p70 B1(ii)Tourist information(descriptive/
informational)

r (w)

r

Extracting main
point from text

Intensive readingfor info.

2

Adverts(c. 30 words)

Matching(5 with 7)

5

p70 A2(i) &B1(ii)

Accommodation
information(informational)

r (w)

r

Assessing validityof text/related
statement

Skimming/
scanning

3

Adverts(c. 60 words)

MCQ(4 options)5

p70 A2(i) &B1(ii)

Publicity(suasive)

r (w)

r

?

Skimming

4

Adverts(c. 60 words)

4 texts –matching
exercise10

p70 A2(i) &B1(ii)

Publicity/personalannouncements,
recipes etc.

r (w)

r

?

Skimming

5

Sentences(c. 30 words)

Sentencecompletion by
matching

10?

Tourist information(descriptive/
informational)

r (w)

r

Grammaticalcompetence(pronouns)

Identifying
grammaticalfeatures

TABLE 2: TEST COMPARABILITY TABLE FOR READING COMPREHENSION TASK TYPES IN CELI
Test: CELI (Certificato de Conoscenza della Lingua Italiana) Level 2

CEFR Level: Threshold (B1)

Language: Italian

Target Skill/s: Reading

Total Time: 2 hrs

1

2

3

4

5

6

7

8

9

10

Question

Task Type

FormatNo.
items

Relevant
descriptor (CEFR)

Text/discourse type

PRIM
(o/w)

Skill

Sub-skill

General strategy

A1

7 short texts

MCQ(3 options)
1 item per text

7

p70 B1

Informationalr (w)reading

Extracting main
point from text

Identifying
main points(skimmimg)

A2

Reading text(c. 300 words)

10 questions(true/false)

10p70 B1Comparativer (w)reading

Assessing validityof text/related
statement

Evaluating
validity of a
proposition

A3

Sentencecompletion

MCQ(4 options)5?

Short individual
sentencesr (w)reading?

Intensive
reading

A4

Selective
deletion cloze

MCQ(3 options)10

p69 B1(i)150 word passager (w)reading?

Intensive
reading

A5

Sentencecompletion

gap filling(usingpronouns)

5?

Short individual
sentencesr/p

reading/
grammarbasedskillsGrammaticalcompetence(pronouns)?

TABLE 3: TEST COMPARABILITY TABLE FOR READING COMPREHENSION TASK TYPES IN ZERT
Test: ZERT (Zertificat Deutsch)

CEFR Level: Threshold (B1)

Language: German

Target Skill/s: Reading

Total Time: 90 minutes

1

2

3

4

5

6

7

8

9

10

Question

Task
Type

FormatNo.
items

Relevant
descriptor (CEFR)

Text/discourse
type

PRIM
(o/w)

Skill

Sub-skill

General strategy

1

5 short(130 word)
Readingpassages

Matching

5
answers(choicefrom 10)p70 B1(ii) &B1(iv)

Tourist information(Informational)r (w)reading

Extracting main
point from text

Identifying
main point

2

Extensive
Reading passage(400 words)

MCQ

5p70 B1(i) & (iv)

Discussionr (w)reading

Assessing validityof text/related
statement

3

Adverts (12)Matching

10(choicefrom 12)p70 B1(ii),
A2(i), B1(iv)Suasion,informationr (w)reading?

4

Selective
deletion cloze

MCQ(3 options)10?

Informationalr (w)reading?
(with lang.
use)

5

Selective
deletion cloze

Matching

10(choicefrom 16)?

Information,
suasion(publicity letter)r (w)reading?
(with lang.
use)

10

|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

Scanning

General Construct Statement: To be developed

11

12

13

14

15

16

17

18

19

20

Constructstatement

Register

Task time
allocation

Linguistic
load

Cognitive
load

Lexicalload

Situational
authenticity

Interactional
authenticity

Rasch difficulty
range (target)

Rasch difficulty
range (actual)

To bedeveloped

Formal25 mins

Fairly
high

Low

100:1

High

Low

48–68

To be established

Informal/publicity

25 mins

Low

Low

100:1

High

Low

48–68

Informal/publicity

20 mins

Low

Low

90:1

High

Low

48–68

Informal/publicity

20 mins

Low

Low

70:1

High

Low

48–68

Formal15 mins

Fairly
high

Low

25:1

Low

Low

48–68

General Construct Statement: To be developed

11

12

13

14

15

16

17

18

19

20

Constructstatement

Register

Task time
allocation

Linguistic
load

Cognitive
load

Lexicalload

Situational
authenticity

Interactional
authenticity

Rasch difficulty
range (target)

Rasch difficulty
range (actual)

To bedeveloped

Formal/informaljournalistic

20 mins

Fairlyhigh

Medium150:1

MediumMediumto Low

48–68

To be establishedSemiformal

15 mins

Fairlyhigh

Medium30:1

MediumLow

48–68

Semiformal

5 mins

Low

Low

10:1

Low

Low

48–68

Semiformal
journalistic

10 mins

Medium

Low

15:1

Low

Low

48–68

Formal5 mins

Low

Low

10:1

Low

Low

48–68

General Construct Statement: To be developed

11

12

13

14

15

16

17

18

19

20

Constructstatement

Register

Task time
allocation

Linguistic
load

Cognitive
load

Lexicalload

Situational
authenticity

Interactional
authenticity

Rasch difficulty
range (target)

Rasch difficulty
range (actual)

To bedeveloped

Journalistic

20 mins

Fairlyhigh

Medium130:1

MediumLow

48–68

To be established

Journalistic

15 mins

Fairlyhigh

Medium

100:1

High

Low

48–68

Informal/publicity

5 mins

Low

Low

80:1

High

Low

48–68

Informal/personal

10 mins

MediumMedium20:1

MediumLow

48–68

Informal/personal

10 mins

MediumMedium20:1

MediumLow

48–68

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

11

Development of an Electronic European Language Portfolio
SIMON FENN, ESOL PROJECTS OFFICE, OPERATIONS GROUP

The European Language Portfolio (ELP) was developed and piloted

determine and record their language learning level by

from 1998 until 2000 by the Language Policy Division of thecompleting self assessment grids, the levels of which map toCouncil of Europe, Strasbourg. It was launched on a pan-European

the levels in the CEFR. Being able to do this on screen, rather

level during the European Year of Languages, 2001 as a tool tothan on paper, is more effective for language learners as they

support the development of plurilingualism and pluriculturalism.

are able to control how many grids are available according to

The ELP is based on the six level system of the Council ofthe number of languages the learner wishes to record. Cutting

Europe’s Common European Framework of Reference fordown on the ‘white space’ of the paper ELP is seen as a key

Languages (A1, A2, B1, B2, C1, C2) and includes the overall grid

objective in the development of the eELP.

of descriptors by level and skill from the CEFR. It consists of the•following elements:•

Language passport‘An updateable overview of your experience in and ability withdifferent languages. It records formal qualifications andawarded.
•Language biography•

Content areas, explaining how the language learner should
complete the Language Passport.

‘A record of your personal language learning history whichhelps you to evaluate your learning objectives, and reflect on•

The ALTE framework of examinations and grids to fill in
showing courses taken in EAQUALS schools.

diplomas, and self-assessments.’•

The Certificates and Diplomas section allows the language
learner to record the details of any language examinations

The other main area of the eELP is the Language Biography.

your language learning and inter-cultural experiences.’

While the Language Passport concentrates on recording language

Dossierlearners’ achievements, the Language Biography concentrates‘A collection of pieces of work and certificates chosen by youon why language learners are learning a particular language andto document and illustrate your language skills, experiences

their aims and aspirations, in practical terms, for learning theand achievements.’

language. The Biography helps language learners to structure

Language portfolios are used by language learners to record andreflect on their language learning achievements and aspirations
and to demonstrate to others their language skills and theirknowledge of other cultures. The ALTE/EAQUALS validated
European Language Portfolio for adults was introduced in 2002 infour languages French, German, English and Italian and has been

their language learning experiences in order that they can assess
how best to progress their language learning. The Biography
will consist of the following areas, essentially a replication of thepaper ELP but enhanced to make best use of the electronic
medium:•further developed in Croatian, Romanian, Basque and Spanish.

languages. Once again the electronic format is more user

Now, moving beyond this paper-based ELP, ALTE and EAQUALS,

friendly so instead of presenting new language learners with ain collaboration with Council of Europe, have begun development

large and rather intimidating blank grid, the language learnerof an electronic version, known as the ALTE/EAQUALS electronic

can add in new aims, as and when required.

European Language Portfolio or eELP. The eELP will be adownloadable application which will be installed locally on

My Language Learning Aims allows language learners to add
specific aims which they wish to achieve in particular•

Similar sections will exist to allow language learners to record
their Language Learning History, Significant Linguistic andcomputers. It will be built using Java, a programming language
which allows the application to run on a variety of platforms andIntercultural Experiences and Current Language Learning

operating systems. Anyone wishing to use the new eELP will be

Priorities allowing language learners to build up a wide-

able to download it from a website. The eELP will be available freeranging and detailed biography of any language of which theyof charge but users will be required to register their details before

have any experience.

they are able to download and install the software. Once installed,•

The Dossier will provide language learners with functionality to

the application will support multiple users, an important

add pieces of work or details of achievements, aimed atconsideration if the eELP is to be used on shared computers indemonstrating and acting as evidence of language learning

language schools or other organisations. Users will be able to login

achievements. Where the piece of work exists in an electronic

to the application and access a number of functional and content-

format, such as an audio file or Word document, it can bebased areas.

stored in the eELP. Where the work or achievement cannot be

The Language Passport contains a number of areas, some ofstored electronically, for example where the language learner

which are listed below:

wants to reference a certificate or diploma, details can be stored•

The Profile of Language Skills allows language learners to

but not the item itself (although the item could be scanned).

12|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

In addition, the eELP will have a Roadmap, which is essentially an

EAQUALS. The new product will be available to users to download

online help facility, guiding language learners through the different

in January 2005, following successful validation and testing. The

areas of the eELP.

eELP will initially support English and French as the two languages

The project to build the eELP is being supported by theof administration, although future versions will support additional

Cambridge ESOL Projects Office. The eELP must conform to thelanguages. Other suggested future enhancements include making

Principles and Guidelines for European Language Portfolio models

the eELP interact with web based functionality which would mean

and be validated by the Validation Committee of the Council ofthat language learners would be able to share their achievements

Europe, the next meeting of which is in November 2004, meaning

more easily.

tight deadlines for the development of this product. Detailed design

Cambridge ESOL supports the eELP project as it recognises theof the new application started in June 2004, following a high-level

wider benefits to individuals and societies that access to, andrequirements gathering phase. The detailed design phase will beachievement in, language learning opportunities bring.

followed by an intensive build phase where developers willproduce the new eELP, which will then be tested by ALTE and

For further information please see the ALTE website: www.alte.org
or contact the Secretariat: stevens.b@ucles.org.uk

Automated Writing Assessment: a review of four conceptualmodelsSTUART D SHAW, RESEARCH AND VALIDATION GROUP

Introduction

Automated assessment

With the advent of a burgeoning testing candidature, theTraditionally, open-ended items have been widely perceived to be

increased demand placed upon large-scale assessment

unsuitable for machine marking because of the difficulty of

programmes in evaluating responses to direct tests of writing ishandling the multiform ways in which credit worthy responses may

widely recognised. Direct tests of writing – essay writing designed

be expressed. The extent of successful automatic marking of free

to measure writing ability, along with open-ended questions

text answers would seem to presuppose, at least in part, arequiring short answers – have long been highly valued

sophisticated level of performance in automated natural language

components of effective assessment programmes.

understanding. More recently, however, advances in Natural

UCLES is investigating the feasibility of automatically

Language Processing (NLP) techniques have revealed potential for

grading essays using a computer. In this context an essay is

automatic assessment of free text responses keyed into a computer

simply a textual response to a question typed into a computer

without the need of having to create system software that fully

including any number of words from a few sentences upward.

comprehend the responses. In essence, NLP is the application of

The aim of these studies is to investigate a number of possible

computational methods to analyse characteristics of electronic files

approaches to the automatic assessment of natural languageof text (or speech). NLP applications utilise tools such as syntactic

essays. Human assigned grades represent an overall

parsers which analyse the discourse structure/organisation of ajudgement of the quality of language form, structure and content

response (Marcu 2000) and lexical similarity measures whichin the essay. The research question raised by such studies

analyse word-use of a text (Salton 1989).

is whether automatic techniques can simulate humanjudgement.
Clearly, the practical implications of either automated orPerceptions of automated assessment

partially-automated rating are obvious – not simply in terms of

Despite understandable scepticism regarding the potential value ofincreasing the speed of turning around results or the money saved

automated essay assessment (Wresch 1993), some reactions havebut also in terms of the automatic generation of valuable

been decidedly favourable. Apart from being cost effective,

diagnostic feedback which could be of use to learners, teachers

computerised scoring is unfailingly consistent, highly objective andand testers alike.

almost wholly impartial (Schwartz 1998). There are those,

UCLES’ primary interest in automated writing evaluation of

however, who regard the concept of computer-assisted evaluation‘free’ or extended text was engendered by the development andas being incompatible with current notions of communicative

implementation of computerised systems capable of assessing

writing proficiency, which stress – amongst other things – theessays automatically. This article provides a review of thewriter’s ability to communicate, or engage, a specific readership.

theoretical models of four of the most prominent implemented

It is the view of some critics that, unlike human raters, computerssystems, and attempts to appraise their respective strengths andare incapable of differentiating between inspirational and creatively

weaknesses.

exceptional essays and their technically correct but ordinary

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

13

counterparts (DeLoughry 1995; Mitchell 1998). This recognised
shortcoming originates, it would seem, from the presumption that
automated scoring emphasises linguistic rules and grammaticalconventions at the expense of less tangible or demonstrable
qualities, such as textual clarity and overall coherence. In
accordance with this view, computers may be able to investigate
writing for the presence or the absence of certain words, phrases or
structures, but they cannot be expected to appreciate a writer’scommunicative purpose in the same way that human raters can.
Even the developers of automated systems readily acknowledge
that – although such systems can be useful tools in assessment –they are unable to adequately replace effective writing assessors.

Four conceptual models for automated essay
assessment
1. Project Essay Grader (PEG) can trace its beginnings to work
undertaken during the 1960s by Page (1966) and is widely
regarded as the first and most enduring implementation of
automated writing evaluation. It depends, principally, on the
linguistic features of an essay;

Project Essay Grader model
Acknowledging the enormous rating load placed upon teachers,
trainers and large-scale assessment programmes in evaluating
student essays, Page developed an automated essay rating system.
Beginning with an established corpus of pre-rated student
compositions, by experimenting with a combination of
automatically extractable textual features, Page applied multiple
linear regression techniques to ascertain an optimal permutation ofweighted features that most accurately predicted the teachers’
ratings. The system was then able to rate – by adopting the same
set of identified weighted features – other student essays. In theirdevelopmental work, Page and his research associates adopted two
explanatory terms:‘Trins were the intrinsic variables of interest – fluency, diction,
grammar, punctuation, and many others. We had no direct
measures of these, so began with substitutes: Proxes wereapproximations, or possible correlates, of these trins. All thecomputer variables (the actual counts in the essays) were proxes.
For example, the trin of fluency was correlated with the prox ofthe number of words.’ (Page 1994:130)

2. E-rater (or Essay-rater) is a system developed by the Education

Multiple regression techniques employed by the PEG model are

Testing Service (ETS) (Burstein et al. 1998a; Burstein andused in the computation, derived from the proxes, of an algorithm

Kukich 1998a; Burstein, Leacock and Swartz 2001). The

to predict a rating for each essay.

system – which uses shallow parsing techniques to identify
syntactic and discourse features – employs a hybrid approachof combining both statistical tools and linguistic featuresderived by using NLP techniques to model the decision of ahuman assessor. An additional but related strand of research
activity currently being undertaken by ETS Technologies is aninvestigation into the feasibility of automating the rating of
short answer content-based questions. The C-rater prototype,again utilising NLP technology, is being evaluated for its
effectiveness at producing ‘credit/no credit’ ratings;
3. Latent Semantic Analysis (LSA) often referred to as the ‘bag ofwords’ approach. LSA was developed by Landauer (Universityof Colorado) and is based on word document co-occurrence
statistics represented as a matrix, which is subsequently
decomposed and then subjected to a dimensionality reduction
technique. Latent Semantic Analysis is a theory and method forextracting and representing the contextual-usage meaning ofwords by statistical computations applied to a large corpus of
text. LSA is a machine-learning technology for simulating themeaning of words, phrases and stretches of textual discourseand effectively ignores essay linguistic and structure features i.e.
the LSA technique evaluates content via the choice of words
and does not take into account any syntactic information;

Evaluation of PEG
Page’s system demonstrated an inter-rater multiple correlational
index with ‘human’ scores of 0.78 – nearly as high as the claimed
0.85 correlation between two or more human raters. Despite its
undoubted early success, initial versions of PEG received merely
partial recognition amongst members of the educational community
primarily due to the fact that it only addressed indirect measuresof writing skill (because of the computational difficulty of
implementing more direct measures). Indirect measures of writing
ability provided students with the potential for cheating as it waspossible to affect the rating by artificially inflating scores throughthe use of various strategies (for example writing lengthiercompositions) and other unfair methods. Indirect measures could
not capture certain rhetorical, organisational and stylistic writing
features which meant that PEG was incapable of generating
diagnostic information, a type of feedback valued by both teachersand students.
In 1995, PEG underwent a series of radical changes designed toembrace more direct measures of writing ability. Despite somesuccess, the principal research challenge must continue to be oneof identifying and automatically extracting more direct measures ofwriting quality.

4. Text Categorisation Techniques (TCT) – developed by Larkey(University of Massachusetts, 1998) – utilise a combination ofmodified key words and linguistic features. Text categorisation

E-Rater modelis the process of grouping text documents into one or more

E-rater employs a combination of statistical and NLP techniques topredefined categories based on their content. Several machine

extract linguistic features of the essays to be rated. It has been usedlearning methods and statistical classifications have been

to rate GMAT (Graduate Management Admissions Test) and TWE

applied to text categorisation including Bayesian classifiers,(Test of Written English) essays for prospective tertiary-level

nearest neighbour classifiers, decision trees, neural networks

students.

and support vector machines.

14|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

Essay-rating specifications for the Analytical Writing Assessment

portion of the GMAT specify a set of general writing assessment
characteristics: topic content, syntactic variety and organisation ofideas. A pilot version of the computerised Analytical Writing
Assessment enabled the generation of data necessary for apreliminary automated essay rating investigation to be undertaken.
Researchers at ETS identified and defined over 100 automaticallyextractable essay features and computerised algorithms were
employed to extract values for every feature from each essay.
Stepwise linear regression methods were applied to subsets offeatures in order to ascertain optimised rating models – models
which were predictive of ratings awarded by ‘expert’ assessors.
E-rater comprises modules that automatically extract essay
lexical content, discourse structure and syntactic (grammaticalstructure) information. Multiple linear regression techniques aresubsequently employed to suggest a rating for the essay, based
upon the very features extracted by the model.
Subsequent refinements to the system have been implemented by
ETS (Burstein et al. 1998a, 1998b; Burstein, Leacock and Swartz
2001). Shallow parsing techniques, employed ostensibly to identify
syntactic and discourse features, are fundamental to the model.
Content is checked by vectors of weighted content words. An essaythat remains focused i.e. stays ‘on-track’, is coherent as evidenced
by use of discourse structures and demonstrates good lexical
resource and varied syntactic structure is awarded a higher score.
Additionally, ETS have researched the feasibility of automating
the rating of short answer content-based questions and havedeveloped an automated rating prototype called C-rater which uses
NLP technology.

Evaluation of E-rater
E-rater has been evaluated by Burstein et al. (1998a) who haveobserved levels of agreement with human raters of between
87% and 94% – on a par with that found between human raters.
Empirically founded, E-rater is ‘trained’ through exposure to
examples of ‘gold standard’ essays (previously rated by humanreaders) – the samples being selected according to essay prompt
so as to ensure they exemplify a comprehensive range ofavailable scores. Although empirically based, E-rater is not‘blindly’ empirical in the sense that it could readily adopt
indiscriminate selection criteria for extracting various features.
However, available extractable features do not always correspondequally well nor do they contribute uniformly to the features thathuman raters are trained to address when rating. E-rater featuresare, as a consequence, required not only to be predictive ofhuman ratings, but also to demonstrate a degree of logical

Latent Semantic Analysis model
LSA seeks to look beneath the surface lexical content of an essay
to quantify its deeper semantic content. The fundamental
principle underpinning LSA is that the aggregate of all the contexts
in which words or lexical chunks occur provides a huge complex
of simultaneous equations that determine the similarity of
meaning of various words and portions of text to each other.
Words and their various contexts are represented by a large
two-dimensional matrix. A word is represented as a row whilst
a column represents larger portions of context (sentences,
paragraphs) in which the words are to be found. By employing
a matrix algebra technique known as Singular Value
Decomposition (SVD) – deconstructing the matrix into threecomponent matrices and then subjecting the matrices to adimensionality reduction technique – new relationships between
words are revealed and already existing relationships arechanged in order for their true importance to be more accurately
represented.
In order to rate an essay, LSA first constructs an essay matrix.
The matrix is subsequently transformed by the SVD algebraic
technique to replicate approximately the matrix using the reduced
dimensional matrices constructed for the essay topic domain
semantic space. Each word and sentence is represented in LSA as adiscrete point in a multi-dimensional semantic space (the semantic
space typically consists of human rated essays). Similarity of
meaning estimates between any two words are derived from theirrelative positions in this semantic space. Vectors are thencalculated from data contained within the student’s essay and theseessay vectors and all essays contained within the semantic space
are compared against each other. Finally, the award for the rated
essay with the lowest cosine value in relation to the essay to be
rated is assigned.
The Intelligent Essay Assessor (IEA) – developed by Knowledge
Analysis Technologies (KAT), Colorado (Foltz, Laham and Landauer,
2003) – uses LSA for rating essay responses and is used to makecomparisons between students’ responses and ‘exemplar’ responses
through computation of the ‘distance’ between their corresponding
vector projections (Graesser et al. 2000). IEA has been subject to avariety of testing scenarios: essay comparison with pre-scoredratings, comparison with gold standard essays (Wolfe et al. 1998),
to sections of the original text, or to sub-components of texts or
essays (Foltz 1996; Foltz, Britt and Perfetti 1996).

Evaluation of LSA

correspondence to the various characteristics that human raters

Foltz (1996) observes that LSA rating performance is almost as

are trained to consider. These characteristics are made explicit

reliable as human assessors. Percentages for adjacent agreementin the available rater training documentation and/or thewith human raters fall in the range 85%–91% on GMAT essays.

accompanying rubrics.
Results from an exploratory, small-scale C-Rater study with a

The LSA technique evaluates content via the choice of words
and is unable to account for syntactic information. In this sense it

university virtual learning programme have been extremely

is a ‘bag of words’ approach and as such can be tricked as thepromising with C-rater achieving over 80% agreement with thefollowing quote graphically demonstrates:

score awarded by an expert. This area of research has the potential
appropriate for evaluating user-constructed responses in ‘on-line’‘It has no way of knowing the difference between The Germans
bombed the British cities and The British bombed the German
cities … The machine would grade both these sentences theassessment applications.

same way.’ (Charles Perfetti 1998)

to evolve into an automated scoring application that would be

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

15

Nor can LSA adequately cope with phenomena considered

vector of weighted feature values (i.e. linguistic properties believeddifficult for NLP systems, like negation, attachment, binding,relevant), ordered and categorised in some canonical way. It is

predication, modification, scope ambiguities and so on. Attempts

then possible to compute a cosine or similar distance measure

have been made to enhance the performance of LSA by

between the training exemplars and the input text to be rated.

incorporating certain syntactic and semantic information: adding apart-of-speech to the given word (Wiemar-Hastings and Zipitria

Evaluation of TCT2001) or adding a part-of-speech to the previous word (Kanejiya,

K nearest neighbour results derived from preliminary work

Kumar and Prasad 2003). Such enhancements have not, however,undertaken by UCLES using information extraction and retrieval

produced a significant improvement over the basic method.

techniques to mark General Certificate of Secondary Education(GCSE) biology answers (described below) reveal that on average

Text Categorisation Technique model

the number of times an automatic rating matches the examiner’s
rating is 67 in 100 i.e. 67%.

From one perspective, LSA is simply a way of categorising a text.

Larkey (1998) – in his linear regression trials – automatically

Other researchers (Rosé et al. 2003; Larkey 1998) have employed

rated general opinion compositions (in which style constituted thetext-categorisation (or classification) for the rating of qualitative

main assessment criterion). The system performed well: thephysics questions and law responses (where the text is expected toproportion of compositions automatically rated exactly the same ascomprise a legal argument). While the accuracy of such systems

those rated by humans was of the order of 0.6. (0.97 for ratings

may not be able to exceed that of hand-crafted systems, they

within +/– one grade of human ratings).

nevertheless have the advantage of being automaticallycustomisable to new domains requiring no other expert knowledge
than that of a human assessor.
Basing his model on Text Categorisation Techniques, Larkey’s
approach to automatic assessment is to:Comparison of automatic essay assessment
models
The table below depicts – in summary form – several
characteristics of the four automated writing assessment models‘… train binary classifiers to distinguish ‘good’ from ‘bad’ essays,
and use the scores output by the classifiers to rank essays andassign grades to them’. (1998:90)

described and shows reported model performances (in comparison

TCT depends, primarily, on the analysis of the occurrence ofeach of the four models. As is the case for each of the models

to human assessors), shared variation between model-human andhuman-human ratings, and the particular approaches employed byparticular words in documents. The approach makes use of

discussed here, essays are evaluated against a set of ‘gold standard’

Bayesian independent classifiers (Maron 1961) for assigning

scripts i.e. a benchmark set of human rated essays.

probability indices to documents, the computation of membership
likelihood estimates and the adoption of k nearest neighbour

Table 1: Comparative performance of automated assessment models

techniques (Mitchell 1997).

Model

Performance* %** correlation

Shared variation(as a %)

Approach

E-rater

87–94*

87–94

hybrid – linguistic features(NLP) + document structurefeatures

LSA

85–91*

85–91“bag of words” – ignores
document linguistic +structure features

TCT

0.60–0.80**

36–64

combination of modified key
words + linguistic features

PEG

0.39–0.78**

15–61

linguistic features

K nearest neighbour techniques start with a set or range of
exemplars representing the main possible outcomes (k is deduced
through training the automated system on a set of human rated
essays). When this particular part of the training phase has been
completed, new examples requiring analysis are matched with theestablished exemplars (or combinations of them) and the closest
match determines the appropriate response. In regard to essay
rating, the ‘analyses’ are ratings and the rating of the nearest
matching exemplar is assigned to the input text to be rated.
New input text (examples) are decomposed into a set offeatures (words, grammatical relations, synonym sets and so on).
By beginning with word tokens, say, it is possible to approximate
a crude rating method for identifying keywords in responses.

The most sensible way of interpreting the correlation coefficients

A weight – known as the ‘tf-idf’ measure – is assigned to eachshown in the second column is to convert them into overlap

content word which appears in the training set. The ‘tf-idf’

between the two measures. To compute the overlap, the square ofmeasure is term frequency (the number of times the term or feature

the correlation coefficient is taken. This enables us to see how

appears in the example) multiplied by inverse document

much of the variance in one measure can be accounted for by thefrequency, i.e. 1/(the number of times term or feature appears in allother. To the degree that the two measures correlate, they shareexamples). Clearly, terms which do not distinguish well among

variance. Thus PEG performance accounts for between 15% andexamples carry less weight. It is possible, therefore, to assign

61% of the variations between PEG and human ratings, whilst TCT

higher weights to words that are associated with (in)correct

accounts for between 36% and 64%. It would appear that E-rater

responses, and less weight to words that occur in almost all

tends to perform marginally better than LSA and very much better

responses. Exemplars and any new input can be represented as athan either TCT or PEG.

16|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

Automatic marking of short textual answers
using information extraction
A highly regarded feature of many of UCLES examinations(including Cambridge ESOL tests) is the widespread use ofquestions that require – from candidates – one or two sentences as

In the final analysis, however, human raters will always be
required to work alongside automated systems, for example inpreparing exemplar responses and to rate samples of student essays
before the assessment systems described here are able to complete
the task.

a response. Any system that could either partially or completely
automate the valid marking of short, free text answers would,

References and further reading

therefore, be of great value. Until comparatively recently this has

Burstein, J and Kukich, K, (1998) Computer analysis of essays, inProceedings of the NCME Symposium on Automated Scoring, April
1998, Montreal, Canada.

been considered either impossible or impractical. However, recent
innovations in computational linguistics together with anincreasing emergence of computers in the classroom, havetriggered a number of assessment organisations to explore thepossibility of automatic marking and its application to high or
low-stakes tests. UCLES is one such organisation. The Interactive
Technologies and Assessment Learning Unit (ITAL) is funding athree-year study at Oxford University which began in summer
2002. The project employs information extraction and retrieval
techniques to mark GCSE biology answers using an automatic
short answer assessor called Automark 3.
Automark 3 has been developed by Intelligent Assessment
Technologies (Mitchell et al. 2002) and employs information
extraction techniques in the sense that the content of a correct
response is specified in the form of a number of mark scheme
templates. The stretch of text to be rated is fed into a parser (in thiscase the Link Grammar parser (Sleator and Temperley 1991, 1993))
and the resulting parsed text is then compared to the alreadydefined templates or mark scheme. Mitchell et al. (2002) claim
about 95% agreement with human markers in blind testing.
Callear, Jerrams-Smith and Soh (2001) at the University ofPortsmouth also use pattern-matching techniques to mark shortanswers in programming languages, psychology and biology –related fields.
Preliminary findings demonstrate that information extraction
techniques can be successfully employed for the task of marking
GCSE biology scripts. It has also been shown that a relatively naïve
text classification method can rate better than a simple baseline
grading technique. There are still many refinements to theapproach that can be usefully made: the final aim is to attempt toapproach the accuracy of the information extraction method but
using completely automatic machine learning techniques.

Conclusion
The potential for using automatic essay scoring in operational
testing environments reduces both the time and the significant
costs associated with having multiple human raters manuallyassess essay responses. The agreement between two human raters,
and between the conceptual models presented in this article anda human rater is very favourable. Automated essay scoring would,
therefore, appear to be a very feasible solution toward theintroduction of more writing assessments in certain high-stakes
standardised tests, and in a lower stakes context, for example forthe purposes of classroom instruction. Moreover, the increased

Burstein, J, Kukich, K, Wolff, S, Chi, Lu and Chodorow, M (1998a)
Enriching automated scoring using discourse marking, paper presented
at the Workshop on Discourse Relations and Discourse Marking
conducted at the Annual Meeting of the Association of Computational
Linguistics, Montreal, Canada.
Burstein, J, Kukich, K, Wolff, S, Chi, Lu, Chodorow, M, Braden-Harder, L
and Harris, M D (1998b) Automated scoring using a hybrid feature
identification technique, in Proceedings of the Annual Meeting of the
Association of Computational Linguistics, August 1998, Montreal,
Canada.
Burstein, J, Leacock, C and Swartz, R (2001) Automated evaluation of
essays and short answers, paper presented at 5th International
Computer Assisted Assessment Conference, Loughborough University.
Callear, D, Jerrams-Smith, J, and Soh,V (2001) CAA of short non-MCQ
answers, in Proceedings of the 5th International CAA conference,
Loughborough. Retrieved from:http://www.lboro.ac.uk/service/ltd/flicaa/conf2001/pdfs/k3.pdf.
DeLoughry, T J (1995) Duke professor pushes concept of grading essays
by computer, Chronicle of Higher Education, ppA24–25.
Foltz, P W (1996) Latent semantic analysis for text-based research,
Behavioral Research Methods, Instruments and Computers, 28,
197–202.
Foltz, P W, Britt, M A and Perfetti, C A (1996) Reasoning from multiple
texts: An automatic analysis of readers’ situation models, inProceedings of the 18th Annual Cognitive Science Conference,
110–115. Hillsdale, NJ: Lawrence Erlbaum Associates.
Foltz, P W, Laham, D and Landauer, T K (2003) Automated essay scoring:
Applications to Educational technology. http://wwwpsych.nmsu.edu/~pfoltz/reprints/Edmedia99.html. Reprint.
Graesser, A C, Wiemer-Hastings, P, Wiemer-Hastings, K, Harter, D (2000)
Person N and the Tutoring Research Group. Using latent semantic
analysis to evaluate the contributions of students in AutoTutor,
Interactive Learning Environments, 8(2), 87–109.
Kanejiya, D, Kumar, A and Prasad, S (2003) Automatic evaluation of
students’ answers using syntactically enhanced LSA, in Building
Educational Applications Using Natural Language Processing,Proceedings of the HLT-NAACL 2003 Workshop, 53–60, Association ofComputational Linguistics.
Larkey, L S (1998) Automatic essay grading using text categorization
techniques, in Proceedings of the Twenty First Annual International
ACM SIGIR Conference on Research and Development in Information
Retrieval, Melbourne, Australia, 90–95.
Marcu, D (2000) The Theory and Practice of Discourse Parsing andSummarization, The MIT Press.
Maron, M E (1961) Automatic indexing: An experimental enquiry, Journalof the Association for Computing Machinery, 8, 404–417.

making a greater quantity and range of assessment andMitchell, J S (1998) Commentary: SAT don’t get you in, Education Weekon the Web. Retrieved from:http://www.edweek.org/ew/current/3mitch.h17

instructional materials available ‘online’.

Mitchell, T (1997) Machine Learning, Maidenhead: McGraw-Hill.

availability of these technologies may well provide incentives for

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

17

Mitchell, T, Russell, T, Broomhead, P and Aldridge, N (2002) Towards
robust computerised marking of free-text responses, in Proceedings of
6th International Computer Aided Assessment Conference,
Loughborough.

Schwartz, A E (1998) Graded by machine, Washington Post, p.107.

Page, E B (1966) The imminence of grading essays by computer,
Phi Delta Kappan, January, 238–243.

Sleator, D K and Temperley, D (1991) Parsing English with a link
grammar, Technical Report. October CMU-CS-91–196.

—(1994) Computer grading of student prose, using modern concepts andsoftware, Journal of Experimental Education, 62, 127–142.

—(1993) Parsing with a link grammar, in Proceedings of Third
International Workshop on Parsing Technologies, Tilburg, theNetherlands, August 10–13.

Perfetti, C (1998) Teachers of Tomorrow? Retrieved from:http://www.wired.com/news/technology/0,1282,16009,00.html
Qian, D D and Harley, B (1998) The role of depth knowledge in adult
ESL learners’ comprehension of academic texts, paper presented at theConference of the Canadian Association for Applied Linguistics on
Trends in Second language Teaching and Learning, Ottawa, Canada.
Rosé, C P, Roque, A, Bhembe, D and VanLehn, K (2003) A hybrid
text classification approach for analysis of student essays, in
Building Educational Applications Using Natural Language Processing,
68–75.

Salton, G (1989) Automatic Text Processing: The Transformation, Analysis,
and Retrieval of Information by Computer, Reading, MA: AddisonWesley.

Wiemar-Hastings, P and Zipitria, I (2001) Rules for syntax, vectors forsemantics, in Proceedings of 23rd Annual Conference of the Cognitive
Science Society, Mahwah, N J: Laurence Erlbaum Associates.
Wolfe, M, Schreiner, M E, Rehder, B, Laham, D, Foltz, P W, Kintsch, W
and Landauer, T K (1998) Learning from text: Matching readers andtexts by latent semantic analysis, Discourse Processes, 25(2 and 3),
309–336.
Wresch, W (1993) The imminence of grading essays by computer –25 years later, Computers and Composition, 10, 46–58.

Studies in Language Testing – Volume 18
Volume 18 in the Studies in Language Testing series is entitledconclusions are certainly controversial, particularly his criticism ofEuropean Language Testing in a Global Context. It contains a

ALTE and several other organisations. These conclusions would not

small subset of the conference papers presented at the firstgo unchallenged by many ALTE members, not least because he

ALTE conference held in July 2001 in Barcelona, Spain.

misrepresents the nature of the association and how it operates.

2001 was designated as the European Year of Languages andKunnan’s paper discusses the qualities of test fairness andthe ALTE conference was held in celebration of this event.

reflects his longstanding concerns with the issues involved in this

The volume is divided into three parts. The first, with twoarea. The framework he presents is of great value to the field of

papers, one written by Charles Alderson and the other by

Language Testing and Kunnan has contributed significantly to theAntony Kunnan, has a focus on more general issues in Language

ongoing debate on the qualities of test fairness within ALTE.
The second part of the volume presents a number of research

Testing.
Alderson looks at some key issues in the field: he considers

studies. Anne Lazaraton focuses on the use of qualitative research“the shape of things to come” and asks if it will be the “normal

methods in the development and validation of language tests.

distribution”. Using this pun to structure his paper, he focuses on

Lazaraton is a pioneer of qualitative research in language testing

two aspects of language testing; the first relates to the technicaland her involvement dates back to the late eighties when suchaspects of the subject (issues of validity, reliability, impact etc.);

approaches were not yet widely used in the field. It is in part due

the second relates to ethical and practical concerns.

to her efforts that researchers are now more willing to embrace

Most of his paper chimes well with current thinking on theapproaches that can provide access to the rich and deep data oftechnical aspects and, as he admits, much of what he presents isqualitative research. Readers are encouraged to look at an earlier

not new and is uncontroversial. Within the European context hevolume in the Studies in Language Testing series written byrefers to the influential work of the Council of Europe, especially

Lazaraton (A Qualitative Approach to the Validation of Oral

the Common European Framework of Reference and the European

Language Tests Volume 14).

Language Portfolio; he describes a number of other European

Vivien Berry and Jo Lewkowicz focus on the important issue of

projects, such as DIALANG and the national examination reformcompulsory language assessment for graduating students in Hongproject in Hungary, and he praises various aspects of the work of

Kong. Their paper considers alternatives to using a language test

ALTE (e.g. for its Code of Practice, for organising usefulalone for this purpose and looks at the applicability of variations

conferences, for encouraging exchange of expertise among its

on the portfolio concept. Jim Purpura’s work on the validation ofmembers, and for raising the profile of language testing in Europe.)

questionnaires, which addresses the interaction of personal factors

In focusing on the political dimension, however, he positions

and second language test performance, represents an interesting

himself as devil’s advocate and sets out to be provocative –and challenging dimension of validation in language testing.

perhaps deliberately introducing a “negative skew” into his

An earlier volume by Purpura in the Studies in Language Testing

discussion. As always his contribution is stimulating and hisseries (Learner strategy use and performance on language tests:

18|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

A structural equation modelling approach Volume 8) looks in morestudies into the dimensions of oral assessment will be published in

depth at the development of questionnaires to determine personaldue course.

factors and a methodology that can be used to investigate theirinteractions with test performance.
Annie Brown reports on an exploratory study investigating the

Finally, Sari Luoma’s paper looks at self-assessment in thecontext of DIALANG. The DIALANG project, also referred to in
Alderson’s paper, has been one of the key initiatives of thesalience of legibility as a factor in rater judgements. This isEuropean Commission in relation to language testing. As such it

particularly relevant as we move towards greater use of computers

has benefited from significant funding and generated muchin language testing. Such a move is, of course, fraught with issues,

research potential.

not least of which is the one of legibility that Brown addresses

The last two parts of the volume cover aspects of work inhere. In the context of IELTS and other Cambridge ESOL tests, suchprogress. On the one hand, Joe Shiels and Wolfgang Mackiewicz

research is currently being conducted in Cambridge and has beensummarise aspects of the ongoing work of the Council of Europe

reported in Research Notes.

and the European Union in relation to language policy. On the

Barry O’Sullivan’s paper attempts to model the factors affecting

other, a number of researchers bring us up-to-date with test

oral test performance, an area of particular significance in large-

development work largely, though not exclusively, in the context ofscale assessment. The paper is part of ongoing research

ALTE. These papers provide the reader with an overview of what iscommissioned by Cambridge ESOL and a collection of researchgoing on in a number of European countries.

Conference Reports
We report below on a range of conferences attended by Cambridge ESOL staff and consultants: Reading Worlds (Lynda Taylor and Diana
Fried-Booth), Writing Developments in Higher Education and BALEAP PIM on Phraseology and Metaphor in EAP (Stuart Shaw) and Lynda
Taylor reports on a Young Learners Symposium that Cambridge ESOL hosted recently.

‘Reading Worlds’ – British Council Conference
on the teaching of literature

to the discussion sessions focusing on assessment and evaluation
which took place on Wednesday 7 April.
In the morning Dr Lynda Taylor took part in the Assessment

The 19th Oxford Conference on the teaching of literature –Panel chaired by Alan Pulverness. Other panel members were:organised by the British Council Seminars programme – was

Julia Bell (novelist and lecturer in Creative Writing at Birkbeck

entitled ‘Reading Worlds’ and was held at Corpus Christi College

College, University of London); Isaac Quist (Curriculum Areain Oxford from 2nd to 8th April 2004. Thirty-nine participants

Manager for the International Baccalaureate Organisation);

attended the seminar from the following countries: Ireland,

Noel Cassidy (secondary English teacher, and a senior examiner for

Denmark, Sweden, Norway, Spain, Portugal, Germany,

A/AS level English for both OCR and CIE). A range of questions on

Switzerland, Austria, Slovenia, Slovakia, Romania, Macedonia,

assessment issues had been gathered from participants in advance

Russia, Kyrgyzstan, Malta, Cyprus, Morocco, Lebanon, India,

and these were submitted to the panel for comment and

Bangladesh, Australia, USA, Venezuela, and Brazil. Most werediscussion; questions related to: the purpose of assessment;

school teachers or university lecturers, and all had a strong

the suitability of different testing/assessment formats; defining

professional and personal interest in language and literature.

the construct of reading; alternative assessment approaches

The Oxford Conference has influenced discussion and debate
on literature teaching for nearly 20 years. Participants haveopportunities to meet and hear some of the most eminent(e.g. portfolios); issues of validity, reliability, practicality andimpact; the tension between assessing language and content.
In the afternoon Diana Fried-Booth, an experienced ELT

academics, critics and writers in the UK, as well as to take part inconsultant and Chair of the CPE Paper 2 set texts, presented a

discussions about a range of pedagogic and literary trends andsession as part of the Exam Board Showcase. Her presentation dealt

issues. In bringing together the insights and experiences ofwith the rationale for including set texts in FCE and CPE which areparticipants and speakers in a strongly international context, thea long-established feature of the Cambridge ESOL examinations.

Oxford conference offers an unrivalled event. The 2004

She referred to the tension in the assessment of a candidate’s

programme featured two interrelated professional tracks, each

response to literature in an examination which focuses on assessing

exploring two important facets in literature study and teaching.

a candidate’s ability to communicate. Reference was made to the

The first looked at the ‘politics’ of contemporary canon-building

General and Task Specific Markschemes in looking at the criteria

and how reputations are built. The second examined assessing

which underpin assessment. The factors which have to be taken

the value of literature teaching. The importance of readers,

into account in selecting appropriate texts for an internationaland reading as a creative activity, linked both tracks.

candidature were also considered. A brief historical overview from

Representatives from Cambridge ESOL were invited to contribute1914 onwards was provided, in order to reflect the shift that has

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

19

taken place over the years in the choice of texts, particularly in the•

What kinds of questions are being raised in different contexts

last eight years. The presentation continued with a review of theand what kinds of answers – theoretical, pedagogical,

advantages to be gained by language teachers and students ininstitutional – are emerging?

studying a set text, and the ways in which language teachers can•Given different historical, linguistic and cultural contexts,

exploit the potential of literature. In conclusion the presentation

to what extent can research findings and pedagogicallooked at possible choices of different genres for the future andprogrammes developed in one context be usefully adopted

invited teachers to send in suggestions they might have for texts toto others?

the Cambridge ESOL website or to the appropriate Subject Officer.
In a future issue of Research Notes we hope to discuss at greater

The conference provided an opportunity for Cambridge ESOL tolength the role of the literary set texts option in the Cambridge

contribute to the on-going debate on the assessment of academic

examinations.

writing skills together with an overview of current EAP teachingpractice and pedagogy. Additionally this event gave us the chance

WDHE (Writing Developments in Higher
Education) 2004: Valuing and Evaluating
Writing in Higher Education

to correct any misconceptions about our range of examinations,specifically the IELTS Academic Writing module. It also enabled us
to evaluate how well Cambridge ESOL Writing Tests reflect current
developments in EAP, and specifically developments in teaching

The WDHE initiative was launched in September 1994 by the

English to the target candidature, namely, candidates intending toUniversity of Northumbria in response to widely expressed

study in an English-speaking Higher Education environment.

concerns about the standards of students’ written communication
skills. Previous conferences have brought together practitioners andresearchers from a wide range of countries, institutions andresearch areas to discuss the forms that writing developments cantake within Higher Education.
The WDHE 2004 conference at Sheffield Hallam University from
11–12 May focused on how (and how much) students’ writing isvalued, assessed, and evaluated, as well as celebrating the10th conference and a decade of significant development.
Specific themes and issues included:
•

•

•

•

topic of Phraseology and Metaphor in EAP.
David Oakey and Jeannette Littlemore (English for International
Students Unit, University of Birmingham) opened the meeting withan overview of these two areas of language which cause difficulty
for university learners of English. David and Jeannette considered

their transfer to the workplace

different conceptual approaches to identifying phraseology andAcademic writing – how it can be valued and how space canmetaphor, methodological approaches to observing and identifying

be created to accommodate writing development in an alreadyphraseological and metaphorical realisations generally and then incrowded curriculum

specific terms, the use of phraseology and metaphor by academic

Assessment strategies, methods and criteria – how Higher

International perspectives – how writing is valued andevaluated in different national contexts•

Birmingham on 19 June 2004. The one-day event focused on thefor their courses can be developed in a way which encouragesof their own writing and its development•

A BALEAP (British Association of Lecturers in English for Academic
Purposes) Professional Issues Meeting was held at the University of

Writing for employability – how the writing skills students need

Education assessment practices can support students’ valuing•

BALEAP Professional Issues Meeting:Phraseology and Metaphor in EAP

discourse communities, and pedagogical applications of researchwhereby students can be helped to understand and use
phraseology and metaphor in EAP contexts. Areas of overlap
between phraseology and metaphor raised issues that wereaddressed by other speakers during the day.
In his presentation – Consolidating collocational competence –New technologies – how they can be used to help students

Peter Howarth (University of Leeds) attempted a survey of current

develop their academic writing (and how they can also help

methods for assisting learners to remedy their collocational errors.

staff support students)

Collocational, or phraseological, competence, he argued, isStudent and staff diversity – how the needs of students andperhaps one of the highest levels of linguistic proficiency that

staff can be met

learners can attain. While, he contended, for many purposes

Staff development and university structures – how staff acrossabsolute native-speaker naturalness is not always required (or evendifferent disciplines can help students develop their skills andpossible) there are some learners who aim at high levels of

how this can be facilitated.

achievement in this area for occupational purposes and this is anarea receiving great attention in EFL and EAP coursebooks and in

Academic writing is increasingly the focus of debates in many

examinations such as Cambridge ESOL’s Certificate of Proficiency incountries and the questions being raised are of importance toEnglish. The view presented here was that collocations themselves

Cambridge ESOL – particularly with regard to the academic writing

cannot be taught; instead, teachers should focus on raisingdimension of IELTS. Questions the conference presentations

awareness of how the phenomenon of phraseology operates in theattempted to answer included:

language, especially its connection with formal style, and should•

Are the issues that are encountered similar in different

encourage students to make use of the range of reference materials

contexts? How do they differ?

available.

20|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

John Skelton and Andrew Shanks (Department of Primary Carereported on a series of experiments whose aim was to assess theand General Practice, University of Birmingham) consideredefficiency of mnemonic strategies that could make the task ofmetaphoric expressions in two health-care settings. They focused

memorising phrases a little easier for students. In addition, he also

on differences between health professional and service-users inpresented evidence of beneficial effects of the given strategies with

their use of metaphoric expressions in two separate contexts:

regard to comprehension, register awareness and cultural

doctor-patient consultations in general practice and focus groups

awareness.
This event illustrated the range of research being undertaken on

consisting of mental health service users, community psychiatricnurses and general practitioners. For both studies, implications formetaphor and phraseology in EAP settings and provided

training were explored.

Cambridge ESOL with insights into how their exams are viewed

Almut Koester (University of Birmingham) presented on teachingby the EAP community in relation to these areas. Incorporating

idioms in spoken academic discourse, arguing that idioms play an

collocational knowledge into teaching and testing, particularly

important role in this spoken genre and often present a particular

at higher proficiency levels, is clearly a challenging enterprise.

problem for non-native speakers; not only because they involve

We continue our research into describing written and spoken

metaphors and multi-word units, but also because of the indirect

proficiency using corpora in order to explore this area to ensure

and implied meanings and functions they convey. Somethat all Cambridge exams remain relevant to the needs of test

preliminary attempts to teach idioms from a functional perspective

takers and users and form the cutting edge of language testing

to international students were reported and evaluated.

design and implementation.

Julia Lockheart (Goldsmiths College, University of London) andHarriet Edwards (Royal College of Art) presented a series of visuals,
which tutors and students employ metaphors in their teaching andCambridge ESOL’s Young Learners Symposium– May 2004

learning practices around writing in the arts/humanities context.

It is over 10 years since Cambridge ESOL began work on a suite

both diagrammatic and pictorial in nature, that reflect ways in

Willie Henderson (University of Birmingham) explored how the
work of applied linguists reveals the significance of metaphor inof language proficiency tests for younger learners of English.
The Young Learners English (YLE) Tests at three levels – Starters,

Economics and how economic methodologists also look at theMovers and Flyers – are designed for 7 to 12 year-olds and aim tosignificance of metaphor in Economics. In his presentation heachieve the following:attempted to ascertain what, if anything, can they learn from each•sample relevant and meaningful language use•

measure accurately and fairly

association between high and low levels of social-cultural•

promote effective learning and teaching

integration amongst international students at English-medium•

other.
Svenja Adolphs (University of Nottingham) looked at theoffer a positive first impression and experience of international

universities and the acquisition of recurrent lexical clusters.

tests.

Whilst it is widely held that exposure to language plays a

The original design and development of our YLE tests involved

significant part in the acquisition process especially with regard to

an extensive review of the available research literature in a number

the acquisition of individual lexical items, grammatical structures,of relevant fields, as well as consultation with a wide range ofand discourse competencies the influence that exposure has on thestakeholders, including members of the academic, pedagogic andacquisition of formulaic sequences in language use has been lesspublishing communities. Following successful trialling in various

well demonstrated. Svenja’s presentation explored how

parts of the world, the tests became operational in 1997. Interest insocial/cultural integration (and the related exposure) affects thethe YLE tests has grown steadily since then; by the end of 2003

acquisition of formulaic sequences.

annual candidature had reached almost 300,000, with large

Philip King (University of Birmingham) gave an interesting
seminar entitled Cross-cultural Wiring in which he assessed writing

numbers of candidates in China, Spain, Argentina and Italy.
Since 1997 the operational test performance of the YLE Testsby a Greek student working at MA level. The writing showed a

has been routinely monitored within Cambridge ESOL; an annual

number of problems and exhibited unusual patterns of lexical

YLE Examination Report is also produced for external publicationphrases, in some cases under the influence of the student’s firstto provide test users with an overview of test performance during alanguage. The presentation explored the profiles of some of thesegiven year. These ongoing monitoring and evaluation activities are

expressions in both languages and aimed to show how despite

an essential part of the validation process for any test; they provide

superficial similarities between some expressions in both

evidence that a test is functioning as intended, and they can help

languages, the collocational boundaries are not coterminous.

to identify features of the test which may need to be reviewed and

The event closed with a presentation by Frank Boers

revised at a later stage. Regular consultation with test users is a

(Erasmushogeschool Brussels, Belgium) entiltled Phrase-wisepriority and reflects the importance Cambridge ESOL attaches toEnglish: some rhyme and reason. Mastery of semi-fixed lexical

evaluating the washback and impact of its tests, especially thosephrases (idioms, collocations, etc.) is believed to facilitate L2

for young learners. In 2002 – five years after their introduction –fluency and helping learners notice useful expressions in the

we began a formal review process with a view to making someauthentic L2 language they are exposed to adds a valuable

minor changes to the tests from 2007. Findings from some of thedimension to L2 language instruction. In his presentation Frankconsultation exercises and the internal studies of test performance

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

21

carried out under this review have been reported in previous issues

management). This set the scene for four 90-minute discussionof Research Notes (see, in particular, issues 7 and 15).

sessions during the symposium which attempted to explore theCambridge ESOL has a long tradition of engaging with members

cognitive, social, pedagogic and assessment dimensions ofof the various ‘stakeholder’ communities who use or have an

language teaching and learning among children. Each session wasinterest in our tests, especially those who are practitioners in thechaired by a member of the Cambridge ESOL staff and wasworld of language teaching/learning and those who work in an

structured around a set of focus questions derived from theacademic research context. Regular consultation with specialists

papers in the delegate volume. Throughout our discussions wefrom these groups is an important element in our approach tofound ourselves returning to some of the fundamental concepts

developing and revising all of our tests (see, for example, theof complex systems theory – dynamism, variation,

published account of the CPE Revision Project, 2003). At ainterconnectedness and creativity – as they apply in the context

meeting of the internal YLE Steering Group in 2002, we discussedof language learning and teaching among children. To close

how we might engage more effectively with the growing

the symposium, Melanie Williams – an experienced teacher

community of individuals working throughout the world in the

trainer and consultant specialising in young learner education –field of young learner teaching, learning and assessment. It was theskilfully drew together the various threads of our discussions tolate Dr Peter Hargreaves, then Chief Executive of our organisation,reflect the key elements and identify possible directions for the

who first suggested that we organise an event to which we could

future.

invite teachers, trainers, academic researchers, materials writers

Although somewhat experimental in its content and format,

and language testers with an active interest in young learners of

the YL Symposium proved stimulating and fruitful from a number

English and other foreign languages. By bringing together inof different perspectives. First of all, it offered a rare opportunity

Cambridge a group of specialists with professional and personal

for members of the YL academic, pedagogic and assessment

interest in young language learners, he believed we might create acommunity to meet one another and spend time together; thecontext where knowledge and experience could be shared, current

sharing of knowledge/experience and the building of professional

issues of interest/concern could be discussed, and new professional

relationships between individuals and groups went on not only

relationships established.

during the plenaries and discussion sessions, but also over the

The conceptualisation of such an event took time to evolve andcoffee/lunch breaks and during the formal dinner on the Friday

in the end we decided to plan an invited symposium on theevening. Even though professional colleagues meet on thegeneral theme of “Issues in the teaching, learning and assessment

international conference circuit, there is rarely much time to sitof L2 young learners”. Rather than organise a traditional,

down and discuss issues of mutual interest in any depth. Secondly,

conference-style event, we chose a slightly more experimental

the event offered a valuable opportunity for the community toapproach: invited participants were commissioned to submit ashare and debate matters of policy and practice in the teaching,paper relating to one of four main themes; the submitted papers

learning and assessment of young language learners. It soon

were then compiled into a delegate volume which participants

became clear that absolute principles and definitive conclusions

were asked to read prior to the symposium. We hoped that thiswere hard to reach, largely because of the many varied andapproach would encourage a stimulating debate of some key

complex environments around the world in which teaching/issues and concerns, and would allow plenty of time forlearning takes place. However, this simply confirms the need forinteraction during the actual symposium meeting. Finally, after

more research to be undertaken in the field and for a continuingmany months of planning, Cambridge ESOL’s YL Symposium tookprofessional debate about what constitutes best practice withinplace on 28/29 May 2004 at Downing College in Cambridge

specific contexts. Finally, from Cambridge ESOL’s perspective we

University. Around 20 external delegates were able to attend:

welcomed the chance to consult with some key stakeholders in thesome represent key universities and other organisations in the UK;

YL field, to explain our approach to young learner assessment, andmany live and work in Europe (France, Spain, Portugal, Hungary,to hear the issues which concern them. The views expressed will

Cyprus, Greece); and some live and/or have experience of working

be invaluable in informing our own thinking and activity as test

in other parts of the world (e.g. Canada, India, China). The external

providers.

delegates were joined by a similar number of Cambridge ESOL
staff, many of whom work directly on our YLE Tests.

As a result of the symposium, plans are already in hand forincreased collaboration between Cambridge ESOL andTo open the symposium, Lynne Cameron (Professor of Applied

individuals/institutions, both in the context of the YLE Review and

Linguistics in the School of Education at the University of Leeds)

more broadly. One key objective over the next few months will bedelivered an introductory plenary talk on the theme of ‘Thea project to develop the draft papers prepared for the symposium

learning child in the language classroom’. In a fascinating andinto an edited volume in the Studies in Language Testing series.

stimulating paper she offered a new metaphor for thinking about

We hope that this will be a valuable contribution to the field, notyoung learner issues by drawing on ‘complex systems theory’ –just for academics and researchers but also for teachers, teacher-a conceptual framework which several other academic disciplines

trainers and others working in the area of YL teaching, learning

have found useful (e.g. biological sciences, economics,

and assessment.

22|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

Other News
Launch of English UK

flexible assessment of ESOL learners in the UK, whilst minimising

The two leading associations in UK English language teaching –the workload of teachers and administrators. They reflect the use of

ARELS (Association of Recognised English Language services) andEnglish in everyday life within the UK and are fully based on the

BASELT (British Association of State English Language Teaching) –Core Curriculum.
The new Skills for Life Certificates are:have joined forces to form a new association called English UK.
This is an important development in language teaching and•available at three levels – Entry 1, Entry 2 and Entry 3

creates a major new force to promote the UK as an English•

externally assessed by trained and experienced examiners

language study destination for students worldwide.
The launch of English UK took place on 12 May 2004 in•

London and was followed by a Parliamentary Reception at theHouses of Parliament. The day featured a number of interesting•

Dr Michael Milanovic, Chief Executive of Cambridge ESOL,
spoke about innovations in language testing and represented
Cambridge ESOL at the Parliamentary Reception. He is
pictured below with Simon Lebus, Chief Executive of the
UCLES Group and Charles Clarke, UK Secretary of State for
Education.

accredited by the QCA and funded by the Learning and Skills
Councilpresentations by experts in the field, including representatives
from English UK and the Department for Education and Skills.

available on demand to suit the learner and the learning
provider•available from March 2005 to support programmescommencing September 2004.
Recognising that the individual’s language skills can varyquite significantly in their development, the new Certificatesallow learners to enter for different modes (Reading, Writing, andcombined Speaking and Listening) at different levels. These
modes reflect the requirements of the Core Curriculum and will
enable candidates to build up a portfolio of achievement overtime.
At each level, learners who achieve a pass will be awarded
certificates from Cambridge ESOL. After Entry 3, candidates canprogress onto other qualifications. Cambridge ESOL offer
qualifications at Level 1 (FCE, CELS Vantage, BEC Vantage),
Level 2 (CAE, CELS Higher, BEC Higher) and Level 3 (CPE).
The Certificates are designed for adult (16+) speakers whose
home language is not English, who are living or trying to settle
in the UK. It is expected that the learners’ educational andemployment backgrounds will be diverse, as well as their
aspirations, literacy levels and language learning skills. This
diversity will be reflected in the range of material selected for use
in these tests and in the task types which candidates will need to

From left: Michael Milanovic, Chief Executive of Cambridge ESOL, andSimon Lebus, Chief Executive of the UCLES Group, meet Charles Clarke,
UK Secretary of State for Education, at the Parliamentary reception.

complete.
Further information on the tests, along with classroom materials
and notes on the preparation of candidates, will be available on the
Cambridge ESOL Teaching Resources website from Autumn 2004.

As Cambridge ESOL shares English UK’s commitment toensuring quality in English language teaching, it was proud toPlease visit the Skills for Life website:
www.CambridgeESOL.org/SFL/

sponsor the launch and reception.
To find out more visit the English UK website:www.englishuk.com

Teaching Knowledge Test (TKT)
Cambridge ESOL is developing a new test for teachers of English

New Certificates in ESOL Skills for Life

which focuses on the core teaching knowledge needed by teachers

Skills for Life is the UK’s national strategy for improving adultof primary, secondary or adult learners, anywhere in the world.

literacy, numeracy and ESOL skills. From September 2004,

TKT is suitable for teachers with a level of English at Level B1 of

all courses and qualifications for adult learners of English in the

the Common European Framework (PET level) or higher. The test

UK state sector must be based on the Adult ESOL Core Curriculum.

will be available in 2005 and is currently undergoing extensive

In response to these requirements, Cambridge ESOL is offering

trialling throughout the world. Look out for articles on TKT in

three new Certificates in ESOL Skills for Life, which provide

future issues of Research Notes.

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4|

23

Changes to Cambridge ESOL’s provision for candidates withdyslexia

longer offer separate marking as a provision for candidates with

An earlier issue of Research Notes (11 – February 2003) reported

the field and with the policy/practice of other assessment providers.

dyslexia. This change in policy is consistent with expert advice in
The ‘extra-time’ provision is generally acknowledged to be theon the special arrangement provisions for a range of candidates,
including dyslexic candidates. For Upper Main Suite Writing

most appropriate provision for candidates with dyslexia and thispapers (FCE, CAE and CPE), our provisions for candidates withwill continue, as will the other special arrangements which aredyslexia currently include the option of separate marking. As part

available.

of our commitment to keep all aspects of our Special Arrangements
policy and practice under constant and careful review, Cambridge

2003 Grade Statistics now available

ESOL has conduced research into the appropriacy of this provision.

Have you ever wondered how candidates score worldwide on

The number of theoretical and empirical studies in the area of

their Cambridge ESOL examinations? On our website you can finddyslexia has grown considerably in recent years and Cambridge

annual grade statistics for a wide range of examinations showing

ESOL’s own studies have revealed that ‘separate marking’ of writing

the average percentages in each grade for all sessions by country

is not an appropriate special arrangement in language-focused(please note these statistics do not include absent, partial absent orassessment (even though it is common practice in content-focusedungraded candidates).
The latest statistics, for the 2003 calendar year, can be found at:assessment).
For this reason, from December 2004, Cambridge ESOL will nowww.cambridgeesol.org/stats/2003/index.cfm

Call for proposals for ALTE 2nd International Conference,
Berlin, May 2005
The Association of Language Testers in Europe (ALTE) is holding•

Issues related to the testing of less widely spoken languages

its second major international conference from 19–21 May 2005•

IT/Distance Learningat the DBB Forum, Berlin, Germany.
The aim of the conference is to provide a forum for the

Ethics and Diversity

discussion of issues associated with:•‘Language Assessment in a Multilingual Context: Attaining
Standards, Sustaining Diversity’
The relationship of linguistic diversity to the testing of•

languages in Europe•

The interaction of the above areas with developments in•

Language and citizenship evaluation•

Language testing and immigration

Transparency and Diversity•

diversity and testing in a global context.
This event is open to the public and ALTE invites papers on thefollowing themes and topics of discussion:

Quality and Diversity

Relating examinations to the Common European Frameworkof Reference•

Use of the Council of Europe Pilot Manual for Relating
Examinations to the Common European Framework

If you would like to consider submitting a paper proposal,
please visit the ALTE website (www.alte.org) where you will•

Quality of examinations and Codes of Practicefind the submission form and accompanying guidelines.

•

Setting standards in testing: minimum standards

All paper proposals must be submitted electronically by•

Research on impact•

Testing language for specific purposes•State certification projects•

The testing of young learners: children of primary and middle
school age

24|

RESEARCH NOTES : I S S U E 1 7 / AU G U S T 2 0 0 4

Thursday 30 September 2004 via the ALTE website. The
acceptance of individual proposals will be announced in
January 2005.