bluuebunny/crossref_metadata_2025 huggingface dataset:
Authentic language tests: where from and where to?
Authors: Shohamy Elana , Reves Thea
Published: March 2008

In the first part of the paper the development of language tests towards authenticity is surveyed. The advantages and shortcomings of indirect vs direct (authentic) types are analysed. Whereas indirect tests were more efficient and lent themselves to psychometric analyses, they did not tap real-life language. On the other hand, direct tests based on real interactions aim at capturing real language and its variations in use. In the second part of the paper two problems involved in authentic tests are addressed: a) The difficulty of applying appropriate psychometric measures, caused by the complexity of tapping the whole construct of authentic language, and b) the large number of 'test' variables which interfere with the authenticity of the language produced and reduce it to authentic 'test language'.
A self-rating scale of English difficulty: Rasch scalar analysis of items and rating categories
Authors: Davidson Fred , Henning Grant
Published: March 2008

This study has sought to demonstrate the utility of Rasch Model scalar analysis when applied to self-ratings of ability/difficulty associated with component skills of English as a second language. Eleven skill areas were rated for difficulty on a seven-point Likert-type scale by 228 ESL students at the University of California. Following appropriate tests of unidimen sionality, both skill area items and rating categories were calibrated for difficulty, examined for fit to the Rasch Model, and plotted to provide visual representation of the nature of the item characteristic curves. Specific suggestions were made for the improvement of the rating cate gories of the self-rating scale, and skill areas most susceptible to self- rating error were identified. It was concluded that scalar analysis of the kind considered here is feasible with self-rating data, and that other rating scale procedures such as those employed. to rate proficiency in foreign language speaking or writing would probably benefit from similar scalar analyses.
The effect of students' academic discipline on their performance on ESP reading tests
Authors: Alderson J. Charles , Urquhart A.H.
Published: March 2008

This paper briefly reviews two earlier studies of the effect of student background knowledge on reading comprehension, then goes on to report a third study into the same effect. The paper draws the conclusion that the test results obtained in the study can be accounted for in terms of an interaction between background knowledge and linguistic proficiency. The paper closes with some observations on the broad concepts 'linguistic proficiency' and 'background knowledge'.
Models of second language competence: a structural equation approach
Authors: Sang F. , Schmitz B. , Vollmer H.J. , Baumert J. , Roeder P.M.
Published: March 2008

Starting with a critical discussion of the consensus reached concerning the structure of second language competence, the unitary competence hypothesis as advocated especially by Oller (1976) is then confronted with new evidence supporting a multidimensional model of foreign language ability. In order to question the plausibility of the general (unitary) factor hypothesis, its central claim, that of universal validity, is put to the test. This claim implies at least two assumptions: a) the invariance of the general factor, regardless of differences in the populations studied, and b) the invariance of this factor irrespective of the conditions under which language acquisition takes place. By showing divergent structures in subgroups of high and low first language ability, the first assumption would be falsified. By demonstrating specific influences of two teaching styles on different language components, the second assumption could be disproved.

The analyses are based on data from a sample of 14 000 seventh grade students and their teachers from 427 secondary schools ('Gymnasien') being highly representative for West Germany and West Berlin. The hypotheses were tested using confirmatory factor analysis (LISREL). The results obtained show the appropriateness of the three factor model proposed (ELEMENTARY, COMPLEX, COMMUNICATIVE), which is theoretically meaningful and best fits the data. Moreover, the expected effects of first language ability grouping and of teaching strategies on the structure of second language competence clearly emerge. In spite of the limitations of the data set analysed, these results can all be interpreted as strong and further evidence against a single factor model (and its educational implications) and in favour of a multiple factor model as outlined above. Finally, didactical consequences of these findings are discussed.
Are English language aptitude tests biased towards culturally different minority groups? Some Israeli findings
Authors: Zeidner Moshe
Published: March 2008

The major aim of this study was to examine the validity and crosscultural gen eralizability of the test bias contention as applied to English language aptitude tests, routinely employed for student selection and placement functions within the Israeli academic scene. The bias analysis was based on the English Language Aptitude Test scores of n = 1778 Jewish and n = 1017 Arab student candidates applying for admissions to a major campus in northern Israel, who were administered the test as part of the scholastic aptitude test battery used for college admissions purposes. The psychometric properties of the English Language Aptitude Test were compared for Jewish and Arab subgroups, respectively, via a variety of internal (e.g. factor loading, reliability, standard error of measurement, etc.) and external (e.g. predictive validity, homogeneity of regression, standard error of estimate, etc.) criteria. The test scores show sig nificant, but slight, intercept bias, tending to overpredict the overall first-year GPA of Arab student candidates. Also, test scores appear to be somewhat less reliable measures for Arab compared to Jewish student candidates. On the whole, however, our data provide only marginal evidence for differential construct or predictive validity of English Language Aptitude Test scores as a function of cultural group membership, thus lending a greater amount of gen erality and crosscultural validity to the findings of much of previous research negating the cultural bias hypothesis.
A history of the Test of Written English: the developmental year
Authors: Stansfield Charles
Published: March 2008

[This article describes in detail the development of a major new departure in an important language test.]
Second language performance testing: the Ontario Test of ESL as an example
Authors: Wesche Marjorie Bingham
Published: March 2008

A recently developed post-admissions testing battery for Ontario colleges and universities is discussed in the light of the multiple issues raised by a performance-based approach to second language testing. In this battery, listening, speaking, reading and writing skills are tested through integrated texts and tasks which simulate academic language use. The test components include a general academic section plus a discipline-related thematic section, of which examinees take either the Social Sciences or Science and Technology module. The battery yields placement and diagnostic information about examinee readiness to undertake full time or part time academic studies in English.
An alternative to multiple choice vocabulary tests
Authors: Meara Paul , Buxton Barbara
Published: March 2008

This paper reports a preliminary evaluation of the Y/N technique for producing tests of vocabulary knowledge. The results obtained suggest advantages over the more traditional multiple choice format for testing vocabulary.
Student major field and text content: interactive effects on reading comprehension in the Test of English as a Foreign Language
Authors: Hale Gordon A.
Published: March 2008

It was hypothesized that a student's academic discipline would interact with the text content in determining performance on the reading passages of the Test of English as a Foreign Language (TOEFL). To test this hypothesis, the present study examined performance on the reading passages in TOEFL forms used in four operational test administrations. The results supported the hypothesis, as students in the two key major-field groups, the humanities/social sciences and the biological/physical sciences, performed better on passages related to their own groups than on other passages. The effect was significant for three of the four test forms. The effect was relatively small in each case, however, as expressed in terms of points on the TOEFL scale, perhaps because TOEFL reading passages are drawn from general readings rather than specialized text books. Thus, it is important to distinguish between the statistical significance and the practical significance of the effect.
Field independence : a source of language test variance?
Authors: Chapelle Carol
Published: March 2008

Recent language testing research investigates factors other than language profi ciency that may be responsible for variance in language test performance. There is some evidence indicating field independence may be one variable responsible for introducing systematic error into language test scores. This paper reports research investigating the relationship between field independence and lan guage measures. Results indicate differential relationships of field indepen dence with cloze, dictation, and multiple-choice language tests. The relative strengths of these relationships also differ for native speakers in regular English classes, native speakers in remedial English classes, and nonnative speakers. Directions for further research are suggested.
Task and ability analysis as a basis for examining content and construct comparability in two EFL proficiency test batteries
Authors: Bachman Lyle F. , Kunnan Swathi Vanniaraian Antony , Lynch Brian
Published: March 2008

This paper describes the content analysis undertaken as part of the Cambridge- TOEFL comparability study, the purpose of which is to examine the compara bility of two EFL test batteries.
Validation of a tape-mediated ACTFL/ILR-scale based test of Chinese speaking proficiency
Authors: Clark John L. D.
Published: March 2008

This paper reports the results of a validation study of the 'semi-direct' Chinese Speaking Test (CST), involving a direct comparison of examinee performance on the CST with performance on the 'live' ACTFL/ILR interview.
Person dimensionality in language test validation
Authors: Lynch Brian , Davidson Fred , Henning Grant
Published: March 2008

In an attempt to extend our knowledge about potential implications of person dimensionality for language test validation, the study reported here was designed: (1) to identify person similarities in a language proficiency testing data set; (2) to attempt to relate such person dimensions to demographic information on the persons so as to provide appropriate labels for the dimensions; and (3) to investigate differential item functioning for the person dimensions identified and labelled. This final focus of the study was intended to reveal whether test items were functioning in an equivalent and valid manner for all person dimensions and, if not, whether there were any discernible patterns of item types that were functioning differentially for particular person dimensions.
Self-assessment of language proficiency: rationale and applications
Authors: Oscarson Mats
Published: March 2008

During the last 10 years there has been a surge of interest in methods for self- assessment of foreign language proficiency. Work has been undertaken in many parts of the world and several reports on both theoretical and practical issues have been published recently. This paper outlines the justification for adopting self-assessment principles in language teaching and learning, makes references to research and sources supporting the statements made, and also presents some ideas and materials that illustrate developments taking place.
The construct validation of self-ratings of communicative language ability
Authors: Bachman Lyle F. , Palmer Adrian S.
Published: March 2008

The trait structure of an experimental self-rating test of communicative language ability was investigated through the use of MTMM design and CFA procedure. The language abilities we attempted to measure comprised three main traits: grammatical competence, pragmatic competence, and socio linguistic competence. Three different question types were used. The subjects were 116 non-native English speakers from the Salt Lake City area. The results of this study indicate that self-ratings can be reliable and valid measures of communicative language abilities. The obtained reliabilities were much higher than had been expected, and all the self-rating measures had strong loadings on a general factor. In addition, some measures proved to be reasonably good indicators of specific language abilities. The measures of grammatical com petence appear to be better indicators of this trait than are the measures of pragmatic and sociolinguistic competence. Of the three question types used, the most effective appears to be that which asked about subjects' perceived diffi culty with various aspects of the language. The least effective question type was the so-called can-do question.
Background and culture as factors in EFL reading comprehension
Authors: Chihara Tetsuro , Sakurai Toshiko , Oller John W.
Published: March 2008

An experimental comparison was made between two original passages in English written for ESL/EFL consumers. Since the subjects tested were 159 Japanese women, unfamiliar, non-Japanese elements were changed to conform more to the expectations of the readers tested. Only a few terms were changed in each of two passages: e.g. names of persons and places; in one instance kissing was changed to hugging. All subjects completed two cloze tests: one over one of the passages in its original form and the other over the other in the modified form. A possible order effect was controlled by counter balancing, and transfer across tests was minimized both by counterbalancing and by using cloze tests over two distinct passages. The hypothesis that such minor changes in textual elements would result in a significantly better performance on cloze tests based on the modified texts was sustained. It is suggested that teachers and materials producers might want to take such factors into consideration.
Baddeley's Grammatical Reasoning Test: active versus passive processing differences re-examined
Authors: Silver N. Clayton , Phelps Glenn R. , Dunlap William P.
Published: March 2008

In Baddeley's grammatical reasoning test it is assumed that passive voice takes longer to process than active and that negative sentences take longer than posi tive ; however, such differences are confounded with sentence length. The present study equated sentence length to examine these issues. A factorial design was used with: (1) two methods of presentation (computer vs. paper and pencil); (2) using the words 'leads' vs. 'precedes'; (3) variable (3-6 word) vs. equal length (6-word) sentences; (4) two replications of each test. When sentences are of unequal length, there was a significant active-passive main effect; however, equal length sentences produced no significant active-passive main effect. Instead, there was an interaction such that there was an active- passive difference for positive sentences, but this difference was not significant for negative sentences.
Developing homogeneous TOEFL scales by multidimensional scaling
Authors: Oltman Philip K. , Stricker Lawrence J.
Published: March 2008

A recent multidimensional scaling analysis of item response data for the TOEFL identified clusters of items in the test sections and suggested that these clusters might be more homogeneous and more distinct than their parent sections, and hence better suited for diagnostic use. The present study explored the feasibility and value of using such cluster scores. The original analysis was based on all the item responses (choosing one of the four alternatives, or omitting or not reaching an item). That analysis was repeated, this time using the traditional scoring of item responses as correct or incorrect. Scores were then obtained for the clusters identified in the new analysis. The new dimen sions and clusters were similar to the old ones. The corresponding scores for the clusters and the test sections did not differ in their internal-consistency reliabilities and intercorrelations for the total sample, but diverged inconsis tently for high- and low-scoring examinees.
Test-takers' reactions to a placement test
Authors: Bradshaw Jenny
Published: March 2008

Reactions to tests can have effects on test scores, motivation and relationships, and these reactions can conceivably be affected by various aspects of tests and the testing situation. In the present study, questionnaires were used to elicit reactions on various dimensions from Italian and Spanish teenagers to test items used as part of placement procedures. The data indicated that a C-test was the most negatively rated on most dimensions by all groups, and that this reaction seemed to relate most strongly to the perceived difficulty of the test. Reactions of lower-scoring students were more negative than those of higher- scoring, although the differences were not so great on the more evaluative dimensions. No gender differences were identified, and there were few differ ences between reactions of Italian and Spanish students, or the most and least nervous groups. Data on the effects of a longer time limit and first language instructions was inconclusive.
Towards a valid measurement of monitored knowledge
Authors: DeKeyser R.
Published: March 2008

This paper provides a number of reflections on the difficulties involved in measuring monitored knowledge of second language grammar, and describes the construction and validation of a prototype test for measuring this knowl edge. It is argued that a fill-in-the-blanks format is to be preferred over multi ple choice, grammaticality judgement or error correction tasks. Problems in defining the structures to be tested are discussed, and a pragmatic solution is proposed. The psychometric characteristics of three versions of the prototype test are described, based on the results obtained for 62 learners of French as a second language in their last year of high school in Dutch-speaking Belgium.
Self-assessment of second language ability: the role of response effects
Authors: Kathy Heilenman L.
Published: March 2008

The role of response effects (tendencies to respond to factors other than item content) in the self-assessment of second language ability was investigated through a split-ballot procedure using positively- and negatively-worded ques tions and graded (i.e. level-specific questions). Results indicate that both an acquiescence effect (a tendency to respond positively regardless of item con tent) and overestimation were present. Although both effects were present at all levels of subjects, they were most evident for less experienced learners. Discussion focuses on explanation of these effects and implications for both test construction and pedagogy.
Using standardized achievement and oral proficiency tests for self-assessment purposes: the DLIFLC study
Authors: Blanche Patrick
Published: March 2008

While there seems to be enough empirical evidence to substantiate the claim that language learners have the ability to evaluate their own performance, many researchers still feel that self-assessments are rarely as reliable or valid as standard test results or teachers' estimates of their students' linguistic com petence. This skepticism may well be justified, because a large number of self- assessments are based almost entirely on appraisal questionnaires and the use of such instruments without extensive training or preparation is bound to pro duce incorrect evaluations. In the DLIFLC study, however, the experimenter used 'conventional' examinations both for testing purposes and for self- assessment purposes. This method yielded interesting results. It also had limitations, but it generally looked promising.
Validation of listening comprehension tests: the effect of text and question type
Authors: Shohamy Elana , Inbar Ofra
Published: March 2008

The paper reports on a study which investigated the effect of both texts and question types on test taker's scores on listening comprehension tests. Listening comprehension tests were administered to 150 EFL learners in their last year of secondary school. The listening stimuli consisted of three text types, a newsbroadcast, a lecturette, and a consultative dialogue, varying in the degree of oral features they contained. The consultative dialogue was the most orally-oriented version, followed by the lecturette and followed then by the newsbroadcast which constituted the most literate version of the three text types. Test takers listened to two different versions about the two topics and answered identical questions to enable comparison of performance on the different text types. The questions were classified into global and local types, according to the strategies utilized for text processing. A third category, trivial questions type, referred to items relating to trivial factual details, such as numerical figures and names. Results indicated that different types of texts located at different points on the oral/literate continuum resulted in different test scores, so that the more 'listenable' texts were easier. In terms of the ques tion types, the results showed that subjects performed better on items referring to local cues than on items referring to global cues. This was observed across topics as well as across text types and across students' levels. Trivial questions were seen to affect performance differently. Implications of the results for the selection of texts and tasks on listening comprehension tests are drawn to arrive at highly construct valid listening comprehension tests.
A comparison of two methods for detecting differential I item functioning in an ESL placement test
Authors: Sasaki Miyuki
Published: March 2008

This paper compares two approximation techniques for detecting differential item functioning (DIF) in an English as a second language (ESL) placement test when the group sizes are too small to use other possible methods (e.g., the three parameter item response theory method). An application of the Angoff delta- plot method (Angoff and Ford, 1973) utilizing the one parameter Rasch model adopted in Chen and Henning (1985), and Scheuneman's chi-square method (Scheuneman, 1979) were chosen because they are among the few methods appropriate for a sample size smaller than 100. Two linguistically and culturally diverse groups (Chinese and Spanish speaking) served as the subjects of this study. The results reveal that there was only marginal overlap between DIF items detected by Chen and Henning's method and Scheuneman's method; the former detected fewer DIF items with less variety than the latter. Moreover, Chen and Henning's method tended to detect easier items with smaller differ ences in p-value between the two groups whereas Scheuneman's method tended to detect items with the opposite features. Implications of these results are discussed.
The reliability ritual
Authors: Schils E.D.J. , van der Poel M.G.M. , Weltens B.
Published: March 2008

A critical appraisal is presented of some of the misconceptions surrounding the notion of test reliability which appear to be quite pervasive in some linguistic research circles. First, it is demonstrated that observed group differences which prove significant in the adequate significance tests, are in no way discredited by low reliability coefficients. Second, it is argued that test reliability should be determined a priori, i.e. before a test is actually used in an investigation. Post-hoc computations of test reliability are redundant and may even be misleading. Third, the results of computer simulations are presented which show that reliability coefficients such as Cronbach's alpha depend to a large degree on the heterogeneity of the subject sample and the range of item dif ficulties. This imposes serious limitations on the usefulness of Cronbach's alpha as a reliability measure, especially when computed post-hoc. As a scalability measure Cronbach's alpha seems fundamentally inappropriate.
Relationships among IRT item discrimination and item fit indices in criterion-referenced language testing
Authors: Hudson Thom
Published: March 2008

This study investigates relationships among the IRT one-parameter fit statistics, the two-parameter slope parameter and traditional biserial correla tions in terms of the role these indices play in criterion-referenced language test construction. It discusses the assumptions of the two models and how these assumptions can affect criterion-referenced test construction and interpreta tion. The study then specifically examines how the indices interrelate as indices of item discrimination. Examinees in Mexico, Saudi Arabia and Japan were administered one of two forms of a functional test (Form A n = 430, k = 94: Form B n = 400, k = 95). The data were analysed using the two IRT models and the results were compared. The results indicate strong relationships among biserial correlation, two-parameter slope, and one-parameter infit and outfit. These results indicate the need to employ the two-parameter model when con ditions allow, and to take item discrimination and item difficulty indices into account when conditions do not. Further implications for interpreting the strong relationships between the indices are discussed.
An investigation of a criterion- referenced test using G-theory, and factor and cluster analyses
Authors: Kunnan Antony John
Published: March 2008

There has been relatively little research on analytical procedures for examin ing the dependability and validity of criterion-referenced tests especially when compared to similar investigations for norm-referenced ESL or EFL tests. This study used three analytical procedures, namely, G-theory, factor and cluster analyses, to investigate the dependability and validity of a criterion-referenced test developed at the University of California, Los Angeles in 1989.

Dependability estimates showed that test scores are not equally depend able for all placement groups and are rather undependable for two out of the four placement groups. Factor analysis of test scores for the placement groups showed that though two-factor solutions were the best solutions for the different groups, there were differences in the way the subtests loaded in the different groups, with progressively fewer subtests loading on the second factor as ability increased. This finding led to the extension study with cluster analysis which showed that a number of students might have been differently placed if subtest scores were used to place them.
Translation as a language testing procedure: does it work?
Authors: Buck Gary
Published: March 2008

This article looks at translation from L2 to L1 as a language testing procedure. Two studies are presented. The first examines the reliability and validity of a translation reading test under conditions where minimal, or no effort is made to ensure rater reliability. Seven raters, with no instructions or training, rated translations by 121 Japanese junior college students. All estimates of reliability proved acceptable. Criterion validity was examined by correlating the translations with other measures of passage comprehension, and were found to be satisfactory. The second study reports the inclusion of two tests of translation in a multitrait-multimethod validation study. They showed satisfactory reliability as estimated by four different methods, and examination of the correlation matrix indicated that the translation tests had acceptable construct validity with almost no method effect. It is argued that these results are generalizable to many other translation tests in other situations, and hence the widespread rejection of translation as a language testing procedure by teachers and testers is probably not warranted on psychometric grounds. However, it is further argued that translation would often have an undesirable washback effect on classroom practice, and thus ought to be used with extreme care.
Examination of the relationships among TSE, TWE and TOEFL scores
Authors: DeMauro G.
Published: March 2008

Several analyses are presented on the relationships among the Test of Spoken English (TSE), the Test of Written English (TWE), and the Test of English as a Foreign Language (TOEFL). The multivariate prediction of each of these tests from the scores on the others is very accurate. Factor analysis reveals two prominent factors. One accounts mainly for oral proficiencies measured by the TSE, and the other accounts for the abilities measured by the TOEFL and the TWE. Scores on the TOEFL listening comprehension, TSE Grammar, TSE overall comprehensibility and the TWE share considerable variance with both factors. The factors may be related to different cognitive skills required to answer different types of questions.
Standards in testing: a study of the practice of UK examination boards in EFL/ESL testing
Authors: Alderson J. Charles , Buck Gary
Published: March 2008

It is often asserted that the UK tradition in language testing is different from the North American tradition. Whereas in North America it is considered normal practice to examine the psychometric properties of tests, in the UK more emphasis is thought to be placed on test content. However, there has been no systematic study of this stereotypical view. The research reported in this article aimed to investigate the issue by examining the practices of UK EFL/ESL examination boards. A survey was carried out in two phases, using an open-ended instrument in phase one and a detailed questionnaire in phase two. The results, presented and discussed in some detail, show that no accepted standards exist which UK boards adhere to, that current practice and procedures followed by the boards vary greatly, but that information appears to exist within some boards which could attest to the quality of their examinations. However, that information is not publicly or easily available. The article argues that the time is ripe for the development of a set of standards boards should follow, and for a public discussion of those procedures that would be most appropriate to ensure that standards were met.
Teaching EFL students to use a test- taking strategy
Authors: Amer Aly Anwar
Published: March 2008

This study investigated the effect of teaching a test-taking strategy to EFL students on their performance on EFL tests. Eighty-two 7th-grade students participated in the study. The experimental group (n = 40) was taught the components of a test-taking strategy: to read the instructions carefully, to schedule their time appropriately, to make use of clue words in the questions, to delay answering difficult questions, and to review their work in order to check their answers. The results showed that the experimental group outperformed the control group (n = 42). The findings suggest that there is a positive relationship between test-performance and skill in test-taking. The findings also suggest support for training EFL classroom teachers to provide instruction in test-taking strategy
The prediction of TOEFL reading item difficulty: implications for construct validity
Authors: Freedle Roy , Kostin Irene
Published: March 2008

The purpose of this study is to predict the difficulty of a large sample ( n = 213) of TOEFL reading comprehension items. A related purpose was to examine whether text and text-by-item interaction variables play a significant role in predicting item difficulty. It was argued that evidence favouring the construct validity of multiple-choice reading test formats requires signincant contributions from these particular predictor variables. Details of item predictability and construct validity were explored by evaluating two hypoth eses : 1) that multiple-choice reading comprehension tests are sensitive to 12 categories of sentential and/or discourse variables found to influence compre hension processes in the experimental literature; and 2) that many of these categories of variables identified in the first hypothesis contribute significant independent variance in predicting item difficulty. For the first hypothesis, correlational analyses confirmed the importance of 11 out of the 12 categor ies, while stepwise regression analyses, accounting for up to 58% of the variance, provided some support for the second hypothesis. The pattern of predictors showed that text and text-by-item variables accounted for most of the variance, thereby providing evidence favouring the construct validity of the TOEFL reading items.
The notion of subskills in reading comprehension tests: an EAP example
Authors: Lumley Tom
Published: March 2008

It has been suggested that reading ability can be divided into various subskills, and this notion is common in ESL teaching and testing. It has, however, also been argued (Alderson and Lukmani, 1989) that teachers are unable to reach agreement about the reading subskills which may be tested by particular reading test items. This study begins by examining the place of subskills in ESL syllabus and test design, with particular attention to the enduring influence of the work of Munby (1978). The issue of teachers' perceptions of subskills and their difficulty, as represented in reading comprehension tests, is discussed. A framework is put forward for negotiat ing agreement between teachers about subskills tested by reading compre hension test items. Using this framework, very substantial agreement between a group of five experienced teachers of EAP is shown to be achieved in matching subskills to individual test items in the reading section of a test of EAP, as well as in judging the difficulty of these subskills. After brief discussion of the use of Rasch IRT in analysis of reading comprehension test items, the teachers' consensus regarding subskill difficulty level is compared to the Rasch analysis of item difficulty, and the significant correlation found gives some empirical validation to the teachers' perceptions. Implications of the findings for analysis of test content, and for teaching, are considered.
The role of test-taker feedback in the test development process: test-takers' reactions to a tape-mediated test of proficiency in spoken Japanese
Authors: Brown Annie
Published: March 2008

This study explores how test-taker reactions in a specific purpose testing context may vary according to characteristics of the test-taker. Such reactions are of interest both theoretically to the researcher and practically to the test developer, who is concerned to ensure that the test is fair and appropriate for all candidates and acceptable to the range of test-takers. The reactions may also be of use in the improvement of test items (as a supplement to item analysis) and test rubrics.

The article reports on the use of test-taker feedback in the development of the occupational foreign language test (Japanese), a tape-mediated test of spoken Japanese for the tourism and hospitality industry. Some 53 trial subjects completed a post-test questionnaire, providing reactions to the test as a whole, to task types, and to individual test items. Relationships are investigated between responses and a number of test-taker characteristics, including gender, type of course undertaken (general or specific purpose), amount of study of the language, time spent in Japan and relevant occupa tional experience. Responses are also considered for items and person shown by Rasch IRT analysis as indicating significant misfit. Aspects of the content, construct and face validity of the test are considered in the light of the analysis. The role of feedback in the revision of test items, in the writing of test rubrics and in the development of the test-user handbook is also discussed.
Rating scales and native speaker performance on a communicatively oriented EAP test
Authors: Hamilton Jan , Lopes Marilyn , McNamara Tim , Sheridan Eileen
Published: March 2008

Explicit or implicit references to the performance of native speakers are to be found in rating scale descriptors in communicatively oriented tests. But the use of the native speaker as a reference point derives from a precommunica tive tradition, and the performance of native speakers on cognitively demanding communicative tests has not been carefully investigated. This article reports on three studies of the performance of native speakers of varying educational backgrounds on a test of reading and writing skills in English for academic purposes contexts. The results show that reference to native speaker performance in rating scales is unwarranted, and help us to understand the nature of the skills being measured in such tests.
Automated assembly of pre-equated language proficiency tests
Authors: Henning Grant , Johnson Pansy J. , Boutin Albert J. , Rice H. Ruth
Published: March 2008

This study examines the effectiveness of an automated language proficiency test assembly system introduced at the Lackland Air Force Base Defense Language Institute English Language Center in several stages over the past year. Since it is desirable that the recently recalibrated item bank and newly implemented test assembly hardware and software be effective in the rapid generation of equivalent test forms, the criteria of effectiveness adopted for this study have been the established classical standards of test equivalence. Thus, the focus of the study is on the investigation of the equivalence of mean score difficulty, total score variance and intercorrelation covariance across ECL test forms. An experimental comparison of four randomly selected, computer-generated ECL test forms was made employing a representative sample of 338 ELC examinees. Results indicated that a high level of test-form equivalence and uniformly high internal consistency evidence exist across ECL test forms, exceeding the levels present in any known commer cially available language tests. It was concluded that the automated system meets requisite quality standards as implemented.
The appropriacy of psychometric measurement models for testing second language listening comprehension
Authors: Buck Gary
Published: March 2008

This article examines the assumptions made by classical and item response (IRT) measurement models, especially the assumption of unidimensionality, and compares these with current theories of listening comprehension, which commonly explain listening in cognitive information-processing terms as a multidimensional interaction between a number of linguistic and non- linguistic skills. On the basis of this, it is hypothesized that second language listening tests will not meet the assumptions made by commonly used, norm- referenced measurement models. This hypothesis is examined by means of the verbal report methodology. Using a structured interview procedure, six L2 listening test-takers are asked to introspect while taking a short-answer listening comprehension test. The resulting protocols are then analysed in order to ascertain the knowledges, skills and abilities which influenced item performance. Results indicate that items typically require a variety of skills for successful performance, and that these usually differ from one test-taker to another. It is argued that this constitutes a violation of the assumption of unidimensionality made by norm-referenced measurement models currently in use.
Evaluating a placement test
Authors: Wall Dianne , Clapham Caroline , Alderson J. Charles
Published: March 2008

The nature and validation of placement tests is rarely discussed in the language testing literature, yet placement tests are probably one of the commonest forms of tests used within institutions which are not designed by individual teachers and which are used to make decisions across the institution rather than within individual classes. Questions to be asked in the validation and evaluation of any placement test include the following: Does the placement test correctly identify those students who most need English and study skills classes? Do the students who take the test feel that their language has been accurately measured? Is the content of the test appro priate to the uses made of the tests? Is the test reliable? This paper reports on an attempt to validate an institutional placement test at Lancaster University. After presenting the results of the study, the paper comments both on the validity and reliability of the test, and on the wider issues that influence how validation studies of placement tests can be carried out.
Predicting item difficulty in a reading comprehension test with an artificial neural network
Authors: Perkins Kyle , Gupta Lalit , Tammana Ravi
Published: March 2008

This article reports the results of using a three-layer backpropagation artificial neural network to predict item difficulty in a reading comprehension test. Two network structures were developed: one with the sigmoid function in the output processing unit and the other without the sigmoid function in the output processing unit. The dataset which consisted of a table of coded test items and corresponding item difficulties was partitioned into a training set and a test set in order to train and test the neural networks. To demonstrate the consistency of the neural networks in predicting item difficulty, the training and testing runs were repeated four times starting with a new set of initial weights. Additionally, the training and testing runs were repeated by switching the training set and the test set. The mean squared error values between the actual and predicted item difficulty demonstrated the consistency of the neural networks in predicting item difficulty for the multiple training and testing runs. Significant correlations were obtained between the actual and predicted item difficulties and the Kruskal-Wallis test indicated no significant difference in the ranks of actual and predicted values.
An oral interview procedure for assessing second language abilities in children
Authors: Carpenter Kathie , Fujii Noriko , Kataoka Hiroko
Published: March 2008

There has been a recent proliferation of Japanese language immersion schools in the USA, reflecting a surge of interest in developing Japanese language abilities in ever-younger children. Efforts to develop and evaluate programmes, though, are hindered by the absence of assessment instruments for measuring progress and competence in young children. In this article we present the goals, design and pilot-testing results of a new oral interview procedure for eliciting a representative sample of spontaneous Japanese language abilities from children aged 5-10. The test consists of six subsets and makes use of realia, role playing, information gap activities and naturalistic conversation, all designed to comprise an oral interview that is devel opmentally appropriate and comparable across children of different levels in different programmes, while overcoming some of the special problems of children's reluctance to interact as equal conversational partners with unfamiliar adults. Pilot results show that the procedure elicits a language sample that is superior in quality and quantity to other existing Japanese language assessment instruments for children.
Towards a common ability scale for Hong Kong English secondary-school forms
Authors: Coniam David
Published: March 2008

This article describes the construction of a common scale which attempts to span the English language ability range of students in the Hong Kong secondary-school system. The purpose of the scale is principally to give English language teachers a reference point - a 'snapshot' - of their students' ability level so that they may access materials in a testing database at points of ability which are appropriate to different groups of students. The testing database - the TeleNex test bank - has the objective of providing English language teachers in Hong Kong secondary schools with professional help and support with their school-based testing. The article describes how tests with common items were designed for a range of secondary-school forms in Hong Kong and how the common ability scale was derived from the resulting bank of items. The article suggests that, while the sale has been established as a reference point for the test bank, it may well have more general use within the Hong Kong educational system.
Lexical density in candidate output on direct and semi-direct versions of an oral proficiency test
Authors: O'Loughlin Kieran
Published: March 2008

This article examines the effects of test format and task type on candidate output in direct and semi-direct versions of the oral interaction subtest of the access: test (the Australian Assessment of Communicative English Skills). Specifically, it contrasts the degree of 'orality' versus 'literacy' characterizing the audiotaped samples collected for the study from the perspective of lexical density, which provides a measure of the relationship between lexical and grammatical items in spoken or written discourse. The findings are discussed in relation to the degree of interactiveness and other factors which appear to influence lexical density. Finally, the results are examined in relation to the validity and interchangeability of the two kinds of oral proficiency tests.
Language-specific prediction in foreign language learning
Authors: MacWhinney Brian
Published: March 2008

Language educators have often suggested that some students may be relatively better at certain types of languages. However, the ways in which target language structures interact with individual differences in language learners has never been seriously investigated. If there are such interactions, we would expect to find certain learning patterns in which our normal expectations for language learning outcomes are reversed. This paper reviews a variety of factors that might lead us to expect such reversals. Psychological and neurological evidence points towards a wide variety of individual differences in language learning mechanisms. Crosslinguistic psy cholinguistic analysis indicates that different target languages offer a wide variety of learning challenges. On the level of orthography, languages with huge inventories of non-Roman characters and symbols offer the greatest challenge. In phonology, all languages require a remapping from L1 struc tures. Lexical learning is most challenging for languages with few cognates to L1. Learners with strong access to the phonological rehearsal loop may be most successful for those languages. In the areas of syntax and morphology, some languages tend to favour learners who establish systematic structural paradigms, whereas other languages may be more easily learned through rote learning processes.
Prognostication and language aptitude testing, 1925-62
Authors: Spolsky Bernard
Published: March 2008

During the 1930s, efforts were made in the USA and elsewhere to develop prognostic tests that would justify decisions to exclude unqualified students from high school foreign language classes. In the USA, after the second world war, USA government language programmes supported research in the assessment of language aptitude to improve selection techniques. While a study by one group of psychologists failed, later work by John B. Carroll and colleagues led to the development of usable language aptitude tests, and contributed to the understanding of the nature of language aptitude.