bluuebunny/crossref_metadata_2025 huggingface dataset:
Student affective reactions to oral language tests
Authors: Scott Mary Lee
Published: March 2008

Many factors influence student reactions to tests, including format, length, time constraints, testing environment, familiarity with test format, perceptions of test validity, and student anxiety. The present study assessed the affective reactions of native Brazilian students to different oral EFL test formats (group and pair) in an achievement testing situation. Each student participated in both the group and the pair test formats and completed an affect questionnaire after each test. A principal factor analysis of student responses to the questionnaire revealed two major factors. The first appeared to involve cognitive judgements about the validity of the test and how well students liked the test. The other factor seemed to be emotive, and involved student reports of the anxiety they felt both before and after the test. Similarly, a qualitative analysis of student responses to open-ended items on the questionnaire found that students were mostly concerned about their emotional state, test content, and time constraints in the testing situation, while comments about test administrators accounted for only a small percentage of the total responses. A multivariate analysis of variance based on the results of the factor analysis showed no significant difference among student reactions to the different test formats.
A latent trait method for measuring a dimension in second language proficiency
Authors: Adams Raymond J. , Griffin Patrick E. , Martin Lyn
Published: March 2008

In this paper it is argued that classical test theory and correlational techniques such as factor analysis have been unable to deal with many of the measurement problems that arise in the language area. As an alternative, the Partial Credit model, a latent trait model for the analysis of data scored in ordered categories, is used as the basis for the construction and analysis of an oral interview test of English as a second language. The test is designed to measure a dimension of oral proficiency that can loosely be called grammatical competence. The advantages of applying such a model are mentioned along with a discussion of its implications for future research and testing methodology in the language area.
A comparison of ethnic, sex and age bias in the predictive validity of English language aptitude tests: some Israeli data
Authors: Zeidner Moshe
Published: March 2008

The primary aim of this study was to test for ethnic, sex, and age bias in the pre dictive validity of English language aptitude test scores, routinely employed for student selection and placement in the Israeli scene. The analyses were based on the English language aptitude test scores of 824 full time Jewish students studying at a major Northern Israeli university. For data analytic purposes, students were partitioned by ethnicity (Western/Oriental), sex (males/females), and age group membership (18-21, 22-25, 26-29, 30+). English language aptitude test scores, as predictors, evidenced both slope and intercept bias by ethnicity, tending signif icantly to overpredict the first year cumulative grade point average of Oriental students and underpredict that of their Western counterparts. Furthermore, test scores were observed to evidence a small degree of intercept bias by gender, tend ing slightly to overpredict the first year grade point average of males and under predict that of females. Also, test scores are shown to be slightly less valid predic tors of criterion performance for students in the oldest age group (30+ ) when compared to students in the remaining age groups. On the whole, the data are consistent with previous pointing to intercept bias in aptitude test scores by gender and ethnicity. Furthermore, the data are consistent with some previous research attesting to the differential predictive validity of aptitude test scores by age group. The implications of the study for college admission testing policy and bias research are discussed and explicated.
A construct definition study of a standardized ESL vocabulary test
Authors: Perkins Kyle , Linnville Steven E.
Published: March 2008

This paper reports a construct definition study of a standardized English as a second language (ESL) vocabulary test. The test, which contained 20 stimulus word items and 17 context word items, was administered to two different samples of ESL students at different proficiency levels and to a sample of native speakers of English. The item responses for the 40 items were used as the dependent variable in a stepwise multiple regression, construct specification equation and 12 variables (measures of the stimulus words' and keyed responses' fluency, distribution, abstractness, length, affective impact, and prototypicality) were used as predictors. Only six of the predictors accounted for a significant amount of variance in the dependent variable, and the number and arrangement of significant predictors varied across proficiency strata, indicating the utility of incorporating the examinees' stages of learning into construct validation studies.

Construct definition studies such as the current one permit the statement of explicit predictions and a means to falsify construct theory, which is a perceived need in educational and psychological measurement.

The paper concludes with an agenda for future research based on the results of the present study.
Validation of listening comprehension tests using item response theory
Authors: de Jong John H.A.L. , Glas C.A.W.
Published: March 2008

This paper presents evidence for the construct validity of foreign language listening comprehension tests obtained from native speaker data and target population data on the tests. Data are analysed using the Rasch model and item parameters estimated using conditional maximum likelihood estimation. A statistical test is used to identify causes for lack of fit of the data with the Rasch model, and an iterative item deletion procedure is proposed to improve Rasch homogeneity of tests. The measure of fit of items in the analysis of item responses from the target population is shown to be associated with native speakers' results on these items. It is suggested that, as the underlying trait of tests for foreign language listening comprehension, literal understanding of aurally perceived language is preferable to interpretative understanding.
The short-context technique: an alternative for testing reading comprehension
Authors: Jafarpur Abdoljavad
Published: March 2008

Traditional reading tests have been subject to mounting criticism of various kinds. This article discusses some of these criticisms and the ways in which an alternative approach - namely, the short-context technique - avoids these defects. It then reports on an attempt to construct a short-context test and the examination of its validity. The results show sufficient evidence substantiating the short-context test as an authentic measure of reading ability.
An algorithmic approach to prescriptive assessment in English as a Second Language
Authors: Griffin Patrick E. , Adams Raymond J. , Martin Lyn , Tomlinson Barry
Published: March 2008

This paper discusses the development of an interview test of proficiency in the 0 to 1 + range on the Australian Second Language Proficiency Rating Scale. Items were written toward 29 specified objectives using specially developed algorithms. These were then trialled on 253 students from the Adult Migrant Education Program in Melbourne. The data were analyzed using the Rasch Partial Credit model and trial forms were brought onto a common scale (ITESL scale). Subsequently 20 items were selected and organized into three over lapping forms of 10 items each. Satisfactory reliabilities and validities were established.
Tailored cloze: improved with classical item analysis techniques
Authors: Brown James Dean
Published: March 2008

This study investigates the possibility that the reliability and validity of a cloze procedure can be improved by applying traditional item analysis and selection techniques. Students in a single level (n = 89) at the Guangzhou English Language Centre (People's Republic of China) were chosen for this study because previous experience and research had indicated that cloze tests gene rally produce low reliability and validity coefficients in samples wherein the range of ESL proficiencies is limited. This turned out to be the case when a 399-word every seventh word deletion cloze passage with 50 items was administered to this group. The study was designed such that 250 of the poten tial items in this passage could be piloted and item analyzed. The results of the item analysis were used to select the 'best' items on the basis of item facility and discrimination indices. The resulting 50 item 'tailored cloze' was then read ministered to the same group and the results for the original version of the cloze test were compared to those for the tailored version. These results indicate statistically significant and meaningful improvements in test quality due to the revision. The dispersion of scores, reliability and validity were all substantially improved by the item analysis and selection processes. The article concludes with discussion of the implications of these findings.
College students' attitudes towards written versus oral tests of English as a Foreign Language
Authors: Zeidner Moshe , Bensoussan Marsha
Published: March 2008

The major aim of the present research is twofold: (a) to compare students' atti tudes towards oral versus written English language test; and (b) to examine the nature of the relationship between students' affective dispositions toward lan guage tests and level of test performance. The study is based on a sample of 170 students at Haifa University enrolled in the advanced reading course of English as a foreign language. Students responded to an examinee feedback inventory specifically designed to gauge examinees' perceptions of key variables related to mode of test presentation (oral vs. written examinations). In addition, back ground information was collected on students' gender, sociolinguistic back ground, father's education, the university psychometric entrance examination (including the English proficiency test score), and average grade in department of study. On the whole, students preferred written over oral tests and rated the former more favourably along a variety of dimensions. Written tests were per ceived to be more pleasant, valuable, fair, less anxiety evoking and more reflec tive of students' comprehension of the English text than oral tests. Oral tests, on the other hand, were viewed to be more interesting to take than written tests. No meaningful relationship, however, was observed between students' atti tudes and test performance.
New procedures for validating proficiency tests of ESP? Theory and practice
Authors: Alderson J. Charles
Published: March 2008

This paper reports on work in progress on an international project to revise the English Language Testing Service (ELTS) test, currently jointly produced and administered by the British Council and the University of Cambridge Local Examinations Syndicate. The project is trying out what it believes are somewhat new approaches to the content validation of the revised test, and both the rationale for this and the details of the procedures for test validation are set out here. It remains to be seen, however, whether such procedures will result in improved construct, concurrent or predictive validity.
Meanings and implications of the principle of local independence
Authors: Henning Grant
Published: March 2008

This paper reviews a variety of definitions of local (conditional) independence presented in the testing literature. An attempt is made to clarify and differen tiate among conflicting conceptualizations of this fundamental measurement principle. In particular, it is argued that local independence, unidimension ality, and noninvasiveness are important but distinct concepts that may, but need not necessarily, overlap. Methods of testing for the presence of local independence in several of its conceptualizations are also presented.
'Naive' native speakers and judgements of oral proficiency in Spanish
Authors: Barnwell David
Published: March 2008

The notion of the 'native speaker' occupies an important place in any theory of foreign language proficiency. In the case of the ACTFL oral interview, on several occasions the 'native speaker' is invoked as a hypothetical interlocutor for candidates. However, no body of research evidence exists as a foundation for such a use of the concept of 'native speaker'. In an effort to discover whether native speakers do really react to differing levels of proficiency in the way predicted for them in the ACTFL scale, a study was carried out in Barcelona, Spain. A group of linguistically 'naive' native speakers evaluated American students' performances on taped ACTFL oral interviews in Spanish. Apart from the fact that there was a considerable divergence in how they viewed particular candidates' exhibited proficiency, it was striking that the native speakers were consistently more strict in their evaluations than was an ACTFL-trained rater.
Expertise in evaluating second language compositions
Authors: Cumming Alister
Published: March 2008

This study (1) assesses whether raters implicitly distinguish students' writing expertise and second language proficiency while evaluating ESL compositions holistically and (2) seeks to describe the decision-making behaviours used by experienced and inexperienced raters in this process. The performance of 7 novice and 6 expert ESL teachers was assessed while they evaluated 12 compositions written by adult students with differing levels of ESL proficiency (intermediate and advanced) and writing expertise (average and professionally experienced writers) in their mother tongues. Multivariate analyses of rating scores indicated that both groups' evaluations distinguished students' second language proficiency and writing skills as separate, non-interacting factors. Descriptive analyses of the raters' concurrent verbal reports revealed 28 common decision-making behaviours, many of which varied significantly in use between novice and expert groups. Implications discuss biases in holistic evaluations of second language compositions, aspects of expertise in this skill, and potential uses of this research for the training of composition raters and student-teachers.
Cloze method: what difference does it make?
Authors: Chapelle Carol A. , Abraham Roberta G.
Published: March 2008

Considerable evidence suggests that cloze techniques can create tests which measure aspects of students' second language competence. However, it remains unclear how variations in the cloze procedure affect measurement. This study compared results obtained from cloze passages constructed from the same text using four different procedures: fixed-ratio, rational, (rational) multiple choice, and C-test. The four procedures produced tests similar in reliabilities but distinct in levels of difficulty and patterns of correlations with other tests. These results are discussed in view of theoretically-based expecta tions for convergent and discriminate relationships of the four cloze tests with other tests.
The testing of listening comprehension: an introspective study1
Authors: Buck Gary
Published: March 2008

The present research uses the verbal report methodology to examine how listening tests work, and how processes not normally accessible through quan titative research methods influence test performance. Six introspectees pro vided data on four main areas of interest: the influence of the short-answer test method on the measurement of listening comprehension; whether test items can measure 'higher-level' cognitive processes; whether test items can measure how well listeners monitor the appropriacy of their interpretation; and how question preview influences comprehension and test performance. The interview protocols provide a great deal of data relevant to these and related issues, the presentation and interpretation of which is the main purpose of this paper. The protocols also indicate a serious dilemma for language testers in that listening comprehension involves far more than the application of linguistic knowledge to produce a propositional representation of a text; rather it is an inferential process in which listeners attempt to construct an interpretation which is meaningful in the light of their own assessment of the situation, knowledge and experience. Thus there are often no clear, objective criteria against which to judge the appropriacy of any one interpretation. The implications of this are discussed.
Development and validation of a scale to measure test-wiseness in EFL/ESL reading test takers
Authors: Allan Alastair
Published: March 2008

Researchers investigating how EFL/ESL students take reading tests typically note that testees use a combination of prior knowledge, analysis of the text and accompanying questions, and test-taking skills. Test-wiseness (TW) is an important source of test content/construct invalidity since it examines students' ability to answer correctly by exploiting weaknesses in test design. The purpose of the study reported here was to design a valid and reliable test of TW in EFL/ESL students. A 33-item multiple-choice instrument with four subscales was constructed and trialled on several groups of ESL students. The test was validated by analysis both of item form and content and of testees' explanations of their answers. Statistical item analyses suggest that the test has reasonable internal consistency and that two subscales may have psychological reality. The findings indicate that students are differentially skilled in test taking and that the scores of some learners may be influenced by skills which are not the focus of the test, thus invalidating their results. The test of test-wiseness in ESL students (TOTWESL) described here can be used to diagnose inexperienced or underachieving language test takers, and (by exposure and sensitization) to confront their weaknesses.
Accommodative questions in oral proficiency interviews
Authors: Ross Steven
Published: March 2008

The processes involved in defining oral proficiency in second languages have to date not involved detailed analysis of the discourse characteristic of oral proficiency interviews. This study considers the phenomenon of variation in questions posed by interviewers at key junctures in the interview process. Based on variable rule analyses of 16 full-length oral proficiency interviews, it is argued that perceptions of oral proficiency are reflected in the extent of accommodation in interviewer questioning, and that the extent of accom modation may provide a powerful factor in determining oral proficiency as well as a criterion for interviewer training.
Validation of the C-test amongst Hungarian EFL learners
Authors: Dörnyei Zoltán , Katona Lucy
Published: March 2008

This article reports on the results of a research programme carried out to validate the C-test amongst Hungarian EFL learners. One hundred and two university English majors were administered four different language tests (including an oral interview) to form a General Language Proficiency measure against which the C-test was evaluated. Various analyses were made, partly to replicate the results of the earlier studies to see to what extent these could be generalized, and partly to shed light on controversial issues. The same C-test was then administered to four groups of secondary school pupils (N=53) to examine whether the findings amongst university students were also true in another proficiency range. The results of the programme confirmed that the C-test is a reliable and valid instrument, and detailed information was obtained about issues such as text difficulty and text appropriateness, the role of content and structure words, and the use of different scoring methods.
Examining washback: the Sri Lankan Impact Study
Authors: Wall Dianne , Alderson J. Charles
Published: March 2008

Language tests are frequently criticized for having negative impact on teaching — so-called 'negative washback'. Some writers believe that it is possible to bring about positive change in language teaching by changing tests. However, neither positive nor negative washback on classrooms has been established empirically by observation of classrooms. This article seeks to redress this situation by reporting on an innovative study of the impact of a new English examination in Sri Lanka on language teaching. Although impact is demonstrated on the content of teaching, no evidence was found for any influence of the test on how teachers taught. It is argued that studies of washback need to relate teachers' attitudes to and understanding of exams to observations of classrooms in order to understand why teachers teach the way they do, and why tests might not have the impact that is frequently asserted. It is concluded that the supposition of washback as currently formulated is an oversimplified account of the relationship between tests and teaching and it is suggested that the complexity of that relationship, and of curricular innovation more generally, needs further exploration.
Surrogate indices for item information functions in criterion-referenced language testing
Authors: Hudson Thom
Published: March 2008

Item analysis of criterion-referenced tests (CRTs) presents several practical problems. Traditional item discrimination indices may be of limited informa tiveness if score distributions are narrow. When no prior defined mastery group is available application of the CRT item difference index is not possible. In settings which have relatively small numbers of examinees, item-response theory (IRT) methods will not yield stable estimates. Like wise, in many language programs either IRT computer programs are unavailable or the results of IRT analysis will be uninformative to those involved in test development. This study examines the relationship of three item discrimination indices and the biserial correlation to IRT item informa tion functions (IIFs) in order to provide testers with information which will be useful in contexts in which IRT analysis is inappropriate. Three indices which indicate item discrimination at the cut-score are compared to IRT results on data from three types of language test data. The indices are the phi-coefficient (Φ), the B-index and the agreement statistic. The three types of language tests are 1) an ESL reading placement test, 2) an ESL reading achievement test and 3) an EFL multiple-choice reading cloze test. Implica tions and cautions for CRT development and analysis are presented.
Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction
Authors: Wigglesworth Gillian
Published: March 2008

In this article, multifaceted Rasch techniques are used to investigate inter- and intrarater reliability in the assessment of two alternate versions of an oral interaction test - a direct version and a semi-direct version. Following the rating of trial administration of both versions of the test, an analysis of the bias of each individual rater in respect to test type and particular criteria used in rating the test was undertaken. The results of these analyses were then presented to the rater. The raters then assessed a further administration of the test and their bias with respect to this administration was analysed. The results of the two bias analyses were compared to determine whether rater performance had improved as a result of the feedback. There was some evidence that performance had improved.
A comparative item analysis study of a language testing instrument
Authors: Reynolds Trudy , Perkins Kyle , Brutten Sheila
Published: March 2008

The goals of this study were threefold: (1) to provide a comparative analysis of five different item analysis indices using both IRT and non-IRT indices; (2) to describe the characteristics of flagged items; and (3) to investigate the appropriateness of logistic regression as an item analysis technique for further studies. The performance of five item analysis indices was examined in this study; p-value, point-biserial correlation, Rasch b parameter, Rasch χ2 fit and χ2 from the logistic regression model. A factor analysis indicated that the first factor was a difficulty factor and included the Rasch b and p-value indices, whereas the second factor represented the discrimination power of an item and consisted of the Rasch χ2 fit, χ2 from the logistic regression and the point-biserial correlation. There was a tendency for difficult items to exhibit more unusual response patterns. The χ 2 from the logistic regression fit was found to be an appropriate index when a researcher is attempting to detect items that function differentially and when the subject pool consists of varying ability groups.
Assessing spontaneous language abilities of aphasic speakers
Authors: Edwards Susan , Knott Raymond
Published: March 2008

Aphasia is an acquired language disorder subsequent to brain damage in which aspects of a speaker's language are impaired in varying combinations and degrees. Although, traditionally, these language impairments have been assessed by tests consisting of tasks such as naming, repetition, sentence construction and recitation of serial speech, there is increasing interest in developing detailed and systematic descriptive frameworks whereby spon taneous conversational or monologic speech can be quantified. In this article we report on a three-year research project in which we are developing such a framework capable of revealing relevant and detailed linguistic features (focusing on lexical and grammatical features) of aphasic speech. Spon taneous speech samples are collected from aphasic and normal speakers in dyadic conversational settings and from monologic picture description tasks. The samples are transcribed and lexical, phrasal and clausal elements identified, coded and quantified. We report on our elicitation, coding and quantification procedures, give some preliminary results as illustration and discuss whether this method of assessment reveals characteristics of aphasic versus normal speech otherwise undisclosed by more traditional methods of testing.
The validity of direct versus semi- direct oral tests
Authors: Shohamy Elana
Published: March 2008

The article reports the results of a study which examined the validity of two oral language tests - one direct and one semi-direct. While concurrent validity of the two types of tests (as obtained via correlations in a number of earlier studies) was high, a variety of qualitative analyses indicated that the two tests differed in a number of aspects. Differences were found in the elicitation tasks (in the number and types of functions and topics) and in the language samples obtained (in communicative strategies and in the discourse features). Conclusions are drawn regarding the need to validate tests from multiple perspectives.
The language used to perform cognitive operations during full- immersion maths tasks
Authors: Cohen Andrew D.
Published: March 2008

Thirty-two students were selected from the third to sixth grade at a Spanish full-immersion school in St Paul. A team of five investigators collected data from the pupils over a five-month period: 1) verbal report; 2) questionnaire data about the pupils' abilities, attitudes and preferences with regard to thinking in Spanish and using Spanish as a vehicle for communication with peers and adults; 3) insights from classroom observation regarding language use patterns in the process of doing the particular task; and 4) background information on the selected learners, including achievement test scores available from the school office, the learners' school grades, sociolinguistic information on their exposure to Spanish language out of class, and so forth.

The findings revealed that, for the immersion students under study, English seemed at times to play a more prominent role in their internal language environment than Spanish. In responding to both numerical and verbal problems in maths, students reported favouring English in their cognitive processing and were also observed to be doing so. They read the problem in Spanish but would shift to English immediately or as soon as they had some conceptual difficulty. These findings may provide some contribu tion to the gaps that have been noted in the spoken and written output of immersion pupils.
Effects of training on raters of ESL compositions
Authors: Weigle Sara Cushing
Published: March 2008

Several effects of training on composition raters have been hypothesized but not investigated empirically. This article presents an analysis of the verbal protocols of four inexperienced raters of ESL placement compositions scoring the same essays both before and after rater training. The verbal protocols show that training clarified the intended scoring criteria for raters, modified their expectations of student writing and provided a reference group of other raters with which raters could compare themselves, although agreement with peers was not an over-riding concern. These results are generally in accordance with hypothesized effects of rater training.
Evaluations of foreign accent in extemporaneous and read material
Authors: Munro Murray J. , Derwing Tracey M.
Published: March 2008

Previous research has yielded apparently contradictory evidence about whether the utterances of second language learners are likely to be perceived as more foreign accented when the speech material has been read or produced extemporaneously. One difficulty with some previous studies is that different test material was used under the two speaking conditions. It is possible that, in some studies, utterances read by L2 learners were evaluated as more accented than extemporaneous speech, not because speakers were unable to implement knowledge about individual segments or prosodic features when under pressure to read in the L2 (as has been supposed), but because of a higher frequency of errors or even a reduction in fluency as a result of less familiarity with vocabulary or syntactic structures, or a preponderance of relatively difficult phones in the reading material. In this study, native Mandarin speakers produced extemporaneous narratives and then read transcriptions of their own utterances. An analysis of accentedness ratings from native English judges revealed no advantage for the speakers in the extemporaneous speaking condition. However, evidence was obtained in support of the hypothesis that familiarity with particular non-native speech samples and/or speakers may lead to perceptions of greater foreign- accentedness.
The effect of rater variables in the development of an occupation-specific language performance test
Authors: Brown Anne
Published: March 2008

There has been little research into the effect of raters' background on assessments made in language tests. This article explores the effect that the rater's occupational and linguistic background has on assessments made in an occupation-specific oral language test, the Japanese Language Test for Tour Guides. Assessments of 51 test candidates made by 33 such assessors, including native and near-native speakers of Japanese with backgrounds either in teaching Japanese as a foreign language or in tour guiding in Japanese, were compared in order to determine what effect background has on assessments made on both linguistic and 'real-world' criteria. Multifaceted Rasch analysis was used, as this allows a range of facets to be modelled; for the purposes of this research we were interested in the facet 'raters', and in particular how the scale was applied for each of the assessment categories. These were then compared across the different types of rater. While differences were found in the assessments according to rater type, these were minor and did not point to the unsuitability of any group as a whole; there were no significant differences between the different types of rater in terms of the overall grade awarded. However, there were significant differences in ratings awarded for some individual criteria, and the fact that these reveal different perceptions of what constitutes good performances has obvious implications not only for the selection and training of raters for occupation- specific language performance tests but also for the development of assess ment procedures.
The dimensionality of a placement test from several analytical perspectives
Authors: Blais Jean-Guy , Laurier Michel D.
Published: March 2008

In order to find the most appropriate group for students enrolling in FSL classes at the postsecondary level, a placement test has been designed. The purpose of this test is to measure general proficiency in the language. Using IRT procedures, an adaptive version has been developed. The test consists of three parts: comprehension of a short paragraph, selection of the appropriate statement in a situation and a 'fill-the-gap' part. Different approaches to assess the unidimensionality are used: structural equation approach (LIS REL), factor-analysis approach (TESTFACT), nonparametric approach (Stout's procedure) and IRT-based approach (Bejar's procedure). These approaches are complementary. The subtests generally meet the unidi mensionality assumption for IRT although they measure more than one ability.
The effect of prior knowledge on EAP listening-test performance
Authors: Jensen Christine , Hansen Christa
Published: March 2008

In this study we looked at whether prior study of a lecture topic enhanced performance on the lecture subtests of a content-based listening test. We postulated that listening proficiency level would affect whether prior knowl edge could be accessed and used: we hypothesized that listeners would need a high proficiency level to be able to use their prior knowledge. To examine these issues, we analyzed the data from 11 lecture subtests from the Test of Listening for Academic Purposes (T-LAP). We used a two-step multiple regression analysis to determine 1) if level of listening proficiency affects whether prior knowledge can be accessed and 2) if prior knowledge is a significant factor in performance on lecture subtests. Our findings did not support the hypothesis that high proficiency listeners who have indicated prior study of a topic will perform better on lecture comprehension than listening skills alone would predict. Although prior knowledge was a significant main effect for five of the 11 lectures, the effect size was trivial. However, the effect is more likely to be present in technical lectures than it is in non-technical lectures. These results indicate that we need to look at what features differentiate technical from non-technical lectures. Lecture dis course features, delivery style, and listening tasks promise to be fruitful areas for future research.
Begging the questionnaire: instrument effect on readers' responses to a self-report checklist
Authors: Allan Alastair I.C.G.
Published: March 2008

This study investigates the validity of having English as a second language (ESL) test-takers select from a checklist the strategies they believe they use when answering multiple-choice reading comprehension questions. Five groups of tertiary-level ESL male and female students studied the same reading passage and questions, and four groups each sequentially used a different version of a self-report checklist of reading strategies. The checklist was manipulated to establish whether it exercised an effect on users. The most popular strategy, established empirically, was first repositioned on, then removed from, the checklist. It was observed that students then selected an alternative from the modified list rather than volunteer in an open category information about how they had arrived at their answers. A similar finding was noted when the next most popular strategy was also removed from the list. The fifth group responded without the aid of the checklist, and were asked to explain their answering strategies in their own words. This group tended to use words and phrases different from those on the checklist. It was concluded that self-report checklists exercise an instrument effect on users' behaviour and thus can invalidate the data collected. Implications are suggested for test-item validation.
Aptitude from an information-processing perspective
Authors: McLaughlin Barry
Published: March 2008

For some years now I have been wrestling with the question of aptitude from within an information-processing perspective. In this article I will briefly outline the approach I take, examine how aptitude is conceptualized in this framework and discuss one possible component of L2 aptitude: working memory.
Working for washback: a review of the washback concept in language testing
Authors: Bailey Kathleen M.
Published: March 2008

This article is a literature review which seeks to answer four questions: 1) What is washback? 2) How does washback work? 3) How can we promote positive washback? 4) How can we investigate washback? Building on the 'Washback Hypothesis' proposed by Alderson and Wall, and suggestions from Hughes, the article proposes a model that identifies participants, processes and products which may influence, or be influenced by, washback. Strategies for investigating wash back are also discussed.
TOEFL preparation courses: a study of washback
Authors: Alderson J. Charles , Hamp-Lyons Liz
Published: March 2008

'Washback' (sometimes also known as 'backwash') is the influence that writers on language testing, syllabus design and language teaching believe a test will have on the teaching that precedes it. Much has been written about the influence of testing on teaching. To date, however, little empirical evidence is available to support the assertions of either positive or negative washback.

The English proficiency test TOEFL (Test of English as a Foreign Language: ETS) is among the best-known examinations in the field of ELT. The study reported in this article set out to investigate common claims that the TOEFL exerts an undesirable influence on language teaching. Our data consist of interviews with teachers and students, and observations made of two sorts of classes: 'normal' language proficiency classes, and parallel classes intended for students preparing to take the TOEFL. We observed both TOEFL preparation classes and non-TOEFL preparation classes by the same teachers in order to be able to separate washback from the TOEFL from any possible effect of individual teacher style.

This study leads us to suggest that simple forms of washback hypotheses are too naive: influences on what happens in class are much more complex than unexamined beliefs about washback allow. The results of the study suggest the need for more complex hypotheses about washback.
C-Tests in the context of reduced redundancy testing: an appraisal
Authors: Klein-Braley Christine
Published: March 2008

In language testing, the concept of reduced redundancy has been a fruitful approach for the development of major test procedures. The way in which exam inees perform under conditions of 'noise' is believed to provide evidence for the level of their current status in overall or general language proficiency. This article reports an investigation comparing the empirical performance of C-Tests with other representatives of the 'family' of reduced redundancy tests - classical cloze, cloze-elide, multiple-choice cloze. The criterion for empirical validity is DELTA, the Duisburg English Language Test for Advanced Students. Overall, the C-Test emerges as the most economical and reliable procedure, it has the highest empirical validity and is shown to be the best representative of the general factor in the bat tery.
The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational I settings
Authors: McNamara T.F. , Lumley Tom
Published: March 2008

The increasing demand for performance assessment of speaking skills in second languages has led to logistic complications, for example, the delivery of tests in overseas locations. One solution to the problem has been to train native speaker interlocutors to carry out a series of oral interactions with the candidate, with assessment from audiorecordings of the test session postponed and conducted cen trally by a small team of trained raters. This technique is currently used in two large-scale occupationally related ESP tests administered internationally on behalf of the Australian government. But these procedures raise questions about the effect of such facets of the assessment situation as interlocutor variables and the quality of the audiotape recording. Recent developments in multifaceted Rasch measure ment have significantly broadened the possibilities for investigation of these issues. The research presented in this article investigates potential problems associated with the above approach to the offshore testing of speaking skills. Data from audiotape-based assessments of 70 offshore candidates from two administrations of the Occupational English Test, an advanced-level ESP test for health pro fessionals, are considered. In addition to multiple ratings of candidate perform ance, each recording is rated for perceptions of the competence of the interlocutor, the rapport established between the candidate and the interlocutor, and the audi bility of the interaction. These aspects of the assessment situation are treated as facets in a series of multifaceted Rasch analyses of the data.

The results of the analysis reveal the effects of interlocutor variability and audio tape quality on ratings. The article concludes with an evaluation of the overall feasibility of the procedure, and implications for test administration are considered. The study is also a further demonstration of the application of multifaceted Rasch measurement in performance assessment settings.
The political dimension of English language testing in Australia
Authors: Hawthorne Lesleyanne
Published: March 2008

The past decade has coincided with the increasing use of English language testing in Australia for nonlinguistic purposes, frequently in contexts governed by signifi cant political pressures. This article explores two recent examples: the access: test (Australian Assessment of Communicative English Skills), a form of ESL testing designed actively to regulate the flow of skilled migrants to Australia; and the step test (Special Test of English Proficiency), a test ostensibly designed to assess linguistic competence, but in fact structured to play a central role in the determi nation of asylum seekers' residential status. This article explores the pragmatic considerations which can give rise to such test development. Its conclusion is that macropolitical pressures can have a profound impact on test design, administration and outcomes, and as such warrant detailed ethical consideration from applied lin guists.
What does test bias have to do with fairness?
Authors: Elder Cathie
Published: March 2008

This article reports on the issue of bias in school examinations as it affects langu ages other than English (LOTE) learners from different Ll backgrounds. Statisti cal methods for detecting test bias claim to be ethically neutral in that they do no more than demonstrate whether or not there are systematic discrepancies in test performance across groups. They leave unanswered the question of whether the criterion adopted as benchmark for group comparisons is fair. The determination of whether a test, or parts of a test, are biased depends ultimately on how the test construct is defined (and on who is defining it). What may appear to be an instance of bias or distortion in the measurement process may, on closer analysis, turn out to be an indication of real differences in the ability which the test aims to measure. This issue is discussed with reference to a scheme introduced in the Victorian State of Australia to compensate foreign language learners for alleged bias in favour of those with a home background in the target language on the Year 12 Victorian Certificate of Education (VCE) examinations in LOTE.
Covert language assessment in academic writing
Authors: Norton Bonny , Starfield Sue
Published: March 2008

In this article 'covert' language assessment refers to the implicit assessment of non-native speaking and writing that takes place when the focus of the assessment is on an examinee's understanding of an academic subject such as geography, history, biology or physics, and not on the examinee's language proficiency as such. Drawing on a larger study of assessment practices at the University of the Witwatersrand in Johannesburg, South Africa, the authors demonstrate that there was in some cases a lack of clarity in the extent to which proficiency in written English was taken into account in the assessment of the academic writing of second language students at the university. They argue that if universities are to be accountable to students, the criteria used in the assessment of assignments and examinations should be made explicit. More specifically, second language students need to know to what extent their performance on academic writing tasks is determined by the quality of their ideas and to what extent it is determined by the quality of their written expression - notwithstanding the complex relationship between them. Recommendations are made for the assessment of students in ter tiary institutions with multilingual student populations.
Washback, impact and validity: ethical concerns
Authors: Hamp-Lyons Liz
Published: March 2008

For many years it was asserted that language tests had a negative impact on teach ing and thereby on learning/learners. This impact has become widely known in language testing as 'washback' ('backwash'). A theory of washback is emerging and its domain being delineated; in this article the theory of washback is linked with the broader concept of 'impact' in educational measurement, and thence to the recent debate on construct validity associated with Messick.