bluuebunny/crossref_metadata_2025 huggingface dataset:
Beyond Dyadic Exchange: Functional Interdependence and Sub-unit Power
Authors: Astley W. Graham , Zajac Edward J. , Astley W. Graham , Zajac Edward J.
Published: March 2008

This study critically examines the relevance of exchange theory in explaining the power of organizational sub-units. The study argues that sub-unit power is generated not by the balance of exchange dependencies between sub-units, but by workflow interdependencies created by the organization's division of labour. The findings support the arguments of the study. The paper concludes that dyadic power relationships between organizational sub-units can only be fully understood in terms of their location in organization-wide systems of functional interdependence.
Effects of Dependence and Tasks on Unit Design and Efficiency
Authors: Gresov Christopher
Published: March 2008

This paper explores the ramifications of external dependence for work-unit design and efficiency. Previous studies at the work-unit level have been largely concerned with task elements alone as the most important contingency for design. Two rival formulations of a dependence-contingent model are derived and contrasted with the more familiar task-contingent propositions. Tests using data from 230 employment security agency units show that, if task is controlled, both horizontal and vertical dependence are important predictors of more organic work-unit designs. In addition, failure to relate design to horizontal dependence is related to lower efficiency.
Environmental Disturbances and Organizational Transitions and Transformations: Some Alternative Models
Authors: Laughlin Richard C.
Published: March 2008

This paper is built on the premise that organizational change can only be understood by tracing the process, track or pathway a disturbance/kick/jolt takes through an organization. Its dominant concern is to build alternative processual models of organizational transitions and transformations which occur following different environmental disturbances. Four models of organizational change are developed (models of 'rebuttal', 'reorientation', 'colonization' and 'evolution') derived from the literature on organizational change and developed using some aspects of Habermas' critical theory. These models are intentionally embryonic and skeletal, requiring detailed case studies to amplify their nature. As a start to this amplification, aspects of two case studies are presented which illustrate respectively 'reorientation' and 'colonization' change pathways. Finally, given the rudimentary stage in our understanding of the dynamics of change, pointers to future research are given in the conclusion.
Statistical aspects of reliability in language testing
Authors: Krzanowski Wojtek J. , Woods Anthony J.
Published: March 2008

Reliability is a concept central to testing. Whatever measure of reliability is used, its value will always have to be estimated from a sample and will therefore be subject to sampling variation. The magnitude of sampling error involved in measuring reliability from a sample should be indicated by quoting the standard error of a reliability estimate or by giving a confidence interval for its true value. This paper considers several simple ANOVA models which can be used to define and estimate reliability coefficients. It discusses the statistical properties of commonly used measures of reliability and explains how to test hypotheses about their true values.
Advantages of latent trait measurement in language testing
Authors: Henning Grant
Published: March 2008

Solutions to a number of important problems in language test develop ment not resolved by classical measurement theory are to be found in latent trait measurement theory. An ESL reading comprehension test is analysed first using classical measurement procedures and then with Rasch Model latent trait procedures. The advantages gained through use of the latter are demonstrated and discussed.
A survey of research on the C-Test1
Authors: Klein-Braley Christine , Raatz Ulrich
Published: March 2008

The use of cloze tests is beset with problems. The C-Test represents an attempt to develop a measure of general language competence which will avoid these problems. A large number of studies, involving children and adults learning a variety of languages, point to its being a reliable and valid measure of overall language ability. Construct validation has now begun. The principal usefulness of the C-Test is seen to be in selection and placement procedures. It is essential that the test should not be used to make significant decisions without prior statistical evaluation.
Does the testing method make a difference? The case of reading comprehension
Authors: Shohamy Elana
Published: March 2008

The paper reports results of a study which examined the effect that various methods used for testing reading comprehension have on the measurement of that trait. The methods investigated in this research were multiple-choice and open-ended questions, each presented in L1 and L2, on the same L2 texts. Results indicated that each of the testing methods produced different degrees of difficulty for the test takers, and that each of the variables - method, text and language - had a significant effect on students' scores in reading comprehension. These effects were strongest on low-level students. Recommendations are made with regard to the use of multiple procedures for testing reading comprehension as well as to the need for further research towards finding the most valid method for testing that trait.
A criterion-referenced measurement approach to ESL achievement testing
Authors: Hudson Thom , Lynch Brian
Published: March 2008

The distinction between norm-referenced measurement (NRM) and criterion-referenced measurement (CRM) has become recognized as an issue in second-language testing. Traditional methods used to determine the reliability and validity of a test, as well as to analyse items for test improvement have been based on NRM principles. These traditional methods are not entirely appropriate for criterion-referenced tests designed to measure course achievement. This study presents approaches to test development item analysis, reliability and validity based on CRM principles. These CRM approaches are discussed and compared with NRM approaches in terms of the types of decisions which result from either approach. The study was conducted using data from an ESL achievement testing project currently in progress at UCLA. The results indicate that CRM approaches provide information not available through NRM methods.
The limits of authenticity in language testi ng
Authors: Spolsky Bernard
Published: March 2008

Setting authenticity as a criterion raises important pragmatic and ethical questions in language testing. Lack of authenticity in the material or method used in a test weakens the generalizability of results. Any language test is by its very nature inauthentic, abnormal language behaviour, for the task is not to give so much as to display knowledge. With examinees who do not know or who are unwilling to play by the rules of the game the results of formal tests will not be an accurate and valid account of their knowledge. Only part of this difficulty can be overcome by authentic-seeming tasks. Observation of authentic behaviour (even allowing for the observer's paradox) is another partial solution. Long, patient and sympathetic observation by observers who care to help seems the only full solution.
Better theory for better tests?
Authors: Raatz Ulrich
Published: March 2008

So-called authentic language tests may not have recognizable separate items. Alternatively such tests may involve items which are dependent on each other, in a fixed order, and which cannot be replaced by alternatives. In neither case can classical test analysis be used at the item level. The use of the Rasch Model is also inappropriate. However, where total test score is the result of adding scores on a number of different, independent but equivalent parts, it is possible to estimate test reliability and to carry out item analysis on the basis of parts. In order to examine the homogeneity of test parts, the unidimensionality of the total score and the question of scale level, the CLA Model may be used. An example is provided of the use of the CLA Model in relation to a German C-Test.
Item response theory
Authors: Woods Anthony , Baker Rosemary
Published: March 2008

A perennial problem for language testers is the need to construct and select test items with 'good' properties. The difficulty lies in the need to assess the properties of items by trying them out on a sample of subjects whose abilities, in turn, it ought to be possible to measure by observing their response to the items. This paper discusses the more important concepts of item response theory (IRT) - a technique, or set of tech niques, developed over the last 25 years, mainly by psychometricians. (An application of IRT was discussed in a recent issue of this journal (Henning, (1984).) Basic concepts are introduced and their implications considered by concentrating on the simplest IRT tool, the Rasch (1960) Model.
Item response theory and the assumption of unidimensionality for language tests
Authors: Henning Grant , Hudson Thom , Turner Jean
Published: March 2008

Considerable controversy has arisen around the assumption of unidimen sionality underlying the application of latent trait models of measurement. The intent of the present paper is to provide a clearer articulation of the unidimensionality assumption and to investigate the robustness and appli cability of a particular unidimensional model, the Rasch Model, for use with language proficiency tests that consist of batteries of subtests in a variety of skill areas and that are applied in the testing of the abilities of students from diverse educational, linguistic and cultural backgrounds. Results of the analysis of response data from the administration of a 150- item, five-subskill ESL proficiency/placement examination to 312 entering university students indicated that unidimensionality constraints were not violated.
Linguistic and cultural bias in language proficiency tests
Authors: Zheng Chen  , Henning Grant
Published: March 2008

The extent to which language proficiency/placement tests may be biased for or against examinees from particular language or cultural groups has never to our knowledge been the focus of empirical research. The purpose of the present study has been to examine the English as a Second Language Placement Examination (ESLPE) employed at the University of California, to determine the nature, direction and extent of bias present for members of two linguistically and culturally diverse subgroups of the sample of examinees. By comparison of the response patterns of 34 native speakers of Spanish and 77 native speakers of Chinese from among a total sample of 312 students tested with one form of the test, it was possible to identify test items exhibiting bias in their respective skill domains. Included is a discussion of the nature, direction, extent and implications of the bias detected.
Assessing the psycholinguistic abilities of young children
Authors: Le Normand Marie-Therese , Simon Anne-Marie , Chevrie-Muller Claude
Published: March 2008

This report concerns three tools in an attempt to provide an objective and flexible means of screening speech and language between the ages of three and four years;1) a composite and standardized neuropsycho linguistic battery; 2) a psycholinguistic assessment in a conversational play session (a bath situation);3) a questionnaire addressed to parents.
Academic demands related to listening skills
Authors: Powers Donald E.
Published: March 2008

A literature review was conducted in order to identify various parameters underlying listening comprehension. The results of this review were used as a basis for a survey of faculty in six graduate fields as well as undergraduate English faculty. The purpose of the survey was to a) obtain faculty perceptions of the importance to academic success of various listening skills and activities, b) assess the degree to which both native and non-native speakers experience difficulties with these skills or activities, and c) determine faculty views of alternative means of evaluating these skills.

Faculty perceived some listening skills as more important than others for academic success. These included nine skills in particular that were related primarily to various aspects of lecture content (e.g. identifying major ideas and relationships among them). As might be expected, faculty perceived that non- native students experience more difficulty than native students with all listening activities, and that non-native students have disproportionately greater difficulty with some activities, such as following lectures given at different speeds and comprehending or deducing the meaning of important vocabulary. With respect to measuring listening comprehension, some general approaches and specific item types were judged to be more appropriate than others. These included tasks that entail answering questions involving the recall of details as well as those involving inference or deductions.

The results of the survey are used to suggest further research on the construct validity of the Listening Comprehension section of the TOEFL.
A multiple choice for language testers
Authors: Spolsky Bernard
Published: March 2008

["Recent years", Spolsky writes, "have marked the triumph of functional, communicative tests...but there remain basic problems." This article focuses on one of these problems: the 'submersion of individual identities' in overall rating scales and goals for foreign language instruction.]
Test validation and cognitive psychology: some methodological considerations
Authors: Grotjahn Rüdiger
Published: March 2008

[This article focuses on attempts to go beyond traditional methods of construct validation to introspective validation, with particular reference to the C-Test.]
Text banking and test design
Authors: Theunissen T.J.J.M.
Published: March 2008

Within the context of item response theory it is possible to computerize test design. Recent developments to do so in an optimal way, using methods from the field of Operations Research, are briefly recapitulated. It is shown that the same methods apply where the test necessarily consists of sub-tests, as is often the case in reading comprehension tests, for example.
Calibrating graded assessments: Rasch partial credit analysis of performance in writing
Authors: Pollitt Alastair , Hutchinson Carolyn
Published: March 2008

Language tests have until now followed one or other of two strategies, focusing either on the difficulty of the questions or on the quality of the performance, but never on both. This article describes the use of the partial credit form of the Rasch model in the analysis and calibration of a set of writing tasks. Qualitative assessments of components of writing competence are rescaled to take into account the empirical difficulty of each grade in each assessment, in order to pro vide more generalisable measures of writing ability.

To exploit this kind of test analysis in assessing language production, it is necessary that the tasks be carefully controlled and that the assessment scales and criteria be adapted to suit the specific demands of each task. The analysis shows the pattern of grade difficulties across task competents and this pattern is dis cussed in some detail. For these tasks at least, it appears that the difficulty of minimally engaging with a task is highly task specific whereas the difficulty of achieving competence in the task depends more on competence in the compo nents of writing skill.
Measuring pronunciation accuracy using computerized techniques
Authors: Major Roy C.
Published: March 2008

Techniques for measuring pronunciation accuracy have lagged far behind measuring competence in other areas. It is argued that pronunciation is an important aspect of second language proficiency and just as it is important to have reliable measures of competence in such areas as listening comprehension, grammar, reading and writing, so it is important to have reliable measures of pronunciation accuracy. This paper discusses several computerized techniques which measure pronunciation accuracy. They have the advantage over other methods because they rigorously control for such factors as sequence effects and length of utterance. These techniques are applicable in a number of areas, including basic research, measuring proficiency gains in students, comparing different methods and measuring comprehensibility and language attitudes.
Operationalizing uncertainty in language testing: an argument in favour of content validity
Authors: Davies Alan
Published: March 2008

It is argued that five variables associated with language proficiency are all subject to unreliability. They are: the native speaker, the cut-off, the criterion score, the test and the language. The native speaker exists only in an idealized form. The cut-off has at best a local definition and at worst is hugely subjective. The criterion used for validation is itself often less reliable than its predictor. Tests are difficult, perhaps impossible to replicate and therefore to generalise from. Language itself balances the general and the particular (this is the 'language': 'a language' distinction). In all these areas the object of description is elusive and fugitive. Language testing has these problems to an even greater extent because measurement is by its nature prone to error. The elusiveness of precision in language testing is discussed in relation to three major issues of linguistic analysis: universality, combining power and the incorporation of extra language data, and results from a recent validation study of the English Language Testing Service (ELTS) Test are quoted to exemplify that relation ship. It is suggested that the test has attempted overspecification and concluded that language testing should accept the centrality of uncertainty and make it explicit.
The influence of test and sample dimensionality on latent trait person ability and item difficulty calibrations
Authors: Henning Grant
Published: March 2008

The present study was designed to test the effects of violation of the unidimen sionality assumption on Rasch Model estimates of item difficulty and person ability. Also considered was the sensitivity of the Bejar Method, Rasch Model fit statistics, classical internal consistency estimation and principal components analysis in detecting the nature and extent of violations of unidimensionality.

For the study of test item dimensionality, use was made of a simulated testing situation involving a two-dimensional 60-item test administered to an illustra tive 120-person sample. For investigation of person sample dimensionality, the simulation involved use of a 120-item test with an illustrative 60-person sample.

Results clearly suggested that violations of item unidimensionality produced distorted estimates of person ability, and violations of person unidimen sionality produced distorted estimates of item difficulty. The Bejar Method was found to be sensitive to such distortions, and results of applying the Bejar Method along with internal consistency estimation and principal components analysis were mutually confirmatory.
Collaborative assessment by masters' candidates in a tutor based system
Authors: Haughton Gillies , Dickinson Leslie
Published: March 2008

Moray House has a long and distinguished record in the promotion and development of the idea of self-directed learning and consequently of self- assessment. This paper begins by examining the beliefs which underlie that commitment and the reasons which can be adduced in support of those beliefs. It then considers one of the major problems which such a commitment presents, the tension between self-assessment and institutional assessment leading to certification. A scheme which goes some of the way toward reconciling these tensions is described and its advantages and shortcomings are commented upon. Finally proposals for future development of the scheme are briefly considered.
The development of a test of Dutch as a second language: the validity of self- assessment by inexperienced subjects
Authors: Janssen-van Dieten Anne-Mieke
Published: March 2008

This article consists of a detailed comparison between a test of Dutch as a second language for use in language courses for adult learners, and a parallel version of that test in self-assessment format. No consistent significant relationship between performance on the two could be demonstrated; neither was it possible to show a significant tendency for ability to self-assess reliably to increase at high proficiency levels.
The relation of multiple-choice cloze items to the Test of English as a Foreign Language
Authors: Hale Gordon A. , Stansfield Charles W. , Rock Donald A. , Hicks Marilyn M. , Butler Frances A. , Oller John W.
Published: March 2008

Four categories of multiple-choice (MC) cloze items were examined in relation to the TOEFL. The object was to assess the factor structure of the TOEFL and the potential of distinguishing MC cloze items aimed at reading comprehension (defined in terms of textual constraints ranging across clauses) as contrasted with knowledge of grammar (short-range surface syntax and morphology) or vocabulary. Since it is impossible in principle to distinguish such skills absolutely at any given point in a text, a compromise was to identify items whose difficulty seemed to be based primarily on one level of processing and secondarily on another. The pivotal category was reading comprehension. In all, 50 MC cloze items over three texts were used in four subsets: ones for which reading comprehension seemed to be the primary source of difficulty, and (1) grammar secondary or (2) vocabulary secondary (nine and 14 items respec tively) ; and ones for which either (3) grammar or (4) vocabulary was the main source of difficulty and reading comprehension secondary (15 and 12 items). Results were analysed separately for each of nine language groups, with a total of 11,290 subjects in all. Factor analysis of the TOEFL suggested two factors related to (a) the Listening Comprehension section, and (b) the nonlistening subsections. The data did not clearly reveal the expected differential relations between the MC cloze categories and subsections of the TOEFL, though tendencies were apparent and analyses on the whole revealed substantial reliability and validity for the MC cloze items.
Testing meaning construction: can we do it fairly?
Authors: Sarig Gissi
Published: March 2008

Variations in readers' content schemata explain why different readers may construct different meanings for a given text and still be in the right. When aiming to reflect this meaning relatively in meaning construction tests, test developers may face theoretical, ethical and practical questions: Are any limits on the free interpretation of a text justifiable? If so, what are these limits? If these limits are set, how can they be reflected in a fair, objective and feasible reading test? Following principles emerging from work by Van Dijk and Kintsch (1983) and Alderson and Short (1981), a meaning consensus criterion answer (MCCA) is suggested as a basis for a relative meaning reading com prehension test. The MCCA is derived from analyses of model answers of a sample of readers from diverse professional backgrounds and levels of expertise. Thus, the MCCA represents both essential, full-consensus compo nents of text meaning, as well as partial, but still considerable, consensus com ponents. It is recommended that the MCCA be used as a basis for item scoring in order to ensure a more feasible objective, yet more relative and, therefore, an unbiased tool for the testing of meaning construction. The paper includes a discussion of the theoretical rationale for the MCCA, and a detailed and an exemplified report on the procedures the MCCA involves. A discussion of MCCA reliability, interconsistency, discrimination power and score meaning follows, and suggestions for future research are made.
Latent structure analysis of the Test of English as a Foreign Language
Authors: Boldt Robert F.
Published: March 2008

Equating studies support the use of IRT methods for TOEFL equating, but there remains a discrepancy between IRT assumptions and diversity of the population served. The model assumes that a single latent proficiency variable underlies item performance, but the TOEFL candidate population is diverse, perhaps containing many groups with separate latent variables. This study attempted to identify such groups.

The basic and very suprising outcome of this study was that a single factor accurately accounted for the proportions of joint item successes; proportions of joint item success were proportional to the products of item difficulties. These results indicate that latent group effects are small. The search for groups included assessing the relative importance of two factors in accounting for examinees' item successes, but the balance of importance of the two factors was rather smoothly distributed, with much overlap between different national origins, and between different language backgrounds. Neither of these variables accounted for more than 25% of the variation of importance of the factors.

The present study supports equating with item response theory and suggests the use of a restrictive assumption of proportionality of item response curves. This single parameter model (not Rasch) could lead to simplified statistical procedures.
The underlying factor structure of L2 close test performance in Francophone, university-level students: causal modelling as an approach to construct validation
Authors: Turner Carolyn E.
Published: March 2008

This study begins to address the question of cloze test construct validity through causal modelling. It investigates the underlying factor structure of L2 cloze test performance as explained by a theoretical model including the following hypothetical constructs: cloze-taking ability; knowledge of language; content domain; and knowledge of contextual constraints. Eight cloze tests reflecting the posited factors were constructed and administered to 182 Francophone, university-level students. The factors were examined separately and in combination through a model building process which included model fitting and model comparison. A model composed of three orthogonal factors was confirmed and accepted as the best explanation of the data. The results indicate that cloze performance is dependent upon language factors (a second language factor or a first language factor) and nonlinguistic specific knowledge related to cloze-taking ability that crosses over linguistic boundaries. Cloze has been considered as an overall L2 proficiency measure. This study empirically demonstrates that factors other than language are significantly contributing to cloze test performance. It also demonstrates the potential of a causal modelling approach.
Item Response Theory and the validation of an ESP test for health professionals
Authors: McNamara T.F.
Published: March 2008

This paper uses a discussion of the role of Rasch model IRT in the validation of two sub-tests of the Occupational English Test to argue for the usefulness of IRT as a tool in the exploration of test constructs and also to consider the implications of the empirical analysis presented for the validity of communica tive language tests involving the skills of speaking and writing.
Short-cut estimators of criterion-referenced test consistency
Authors: Brown James Dean
Published: March 2008

The purpose of this paper is to present relatively easy-to-calculate estimates for the consistency of criterion-referenced tests (CRTs). Four techniques are demonstrated using data from a CRT administered in the intermediate ESL reading classes in the English Language Institute at the University of Hawaii at Manoa. First, the threshold loss agreement approach is represented by the agreement coefficient (Subkoviak 1980) and kappa coefficient (Cohen 1960). Until recently, these coefficients could only be applied in test-retest CRT situa tions. Alternative and simpler methods (suggested by Subkoviak 1988) are given for estimating these coefficients from the results of a single CRT administration. Second, the squared-error loss agreement approach is repre sented by the phi(lambda) dependability index, for which simplified calcula tions, derived from Brennan (1980), are given. Finally, the domain score dependability approach is represented by a short-cut method for estimating a general purpose phi coefficient. This short-cut estimate is derived from Brennan's 1983 and 1984 discussions of the phi coefficient.
The prehistory of TOEFL
Authors: Spolsky Bernard
Published: March 2008

This article explores the origins of the TOEFL, with its distinctive charac teristics, in a 1961 conference, and seeks thereby to gain an understanding of how developments in language testing theory - and perhaps theory in general - are blended with the requirements and possibilities of real-life implementation.
Oral examinations: an historical note
Authors: Spolsky Bernard
Published: March 2008

A report written by J.O. Roach in 1945 for the University of Cambridge Local Examinations Syndicate appears to have been one of the first discussions in print of the problems of reliability and validity in the testing of oral proficiency in a second language. It describes a number of experiments in improving consistency in examiner's rating by joint examining and by the use of recorded samples, considers the problems of establishing valid standards in subjective testing, and proposes the development of what is now the standard form of listening comprehension test.
Cloze item types and second language comprehension
Authors: Jonz Jon
Published: March 2008

This paper describes the further analysis of one portion of the data reported in Jonz (1989). Four 50-item fixed-ratio cloze tests were administered to 238 nonnative university freshman. Half of the subjects were randomly assigned to take the cloze tests in their naturally occurring order; the other half of the subjects were randomly assigned to take the tests after the original sequence of the sentences in the passage had been thoroughly scrambled. Additionally, half of the subjects were allowed to read the entire correctly sequenced unaltered passage prior to completing the cloze tests. After each item on the two normal-order tests had been assigned to one of five item types according to a scheme introduced in Bachman (1985) and further developed in Jonz (1990b), the patterns of response of each of the item types on all four cloze tests were then studied. The data analysis revealed that under the prior-knowledge condition, textually cohesive cloze-test items showed a significant sensitivity to sentence scrambling. The paper concludes that (1) intersentential ties are particularly salient in the comprehension processes of nonnative speakers (cf. Jonz, 1987), and that (2) in specific respect of nonnative-speaker responses, fixed-ratio cloze tests are significantly sensitive to textual variations and continuities at levels well beyond local phrase structure. That is to say, attained scores on standard fixed-ratio cloze procedure reflect to a significant extent a nonnative-speaker examinee's utilization of text-level linguistic structures and discourse comprehension processes.
An exploratory study into the construct validity of a reading comprehension test: triangulation of data sources
Authors: Anderson Neil J. , Bachman Lyle , Perkins Kyle , Cohen Andrew
Published: March 2008

Recent research in reading comprehension has focused on the processes of reading, while recent thinking in language testing has recognized the impor tance of gathering information on test taking processes as part of construct validation. And while there is a growing body of research on test-taking strate gies in language testing, as well as research into the relationship between item content and item performance, no research to date has attempted to examine the relationships among all three - test taking strategies, item content and item performance. This study thus serves as a methodological exploration in the use of information from both think-aloud protocols and more commonly used types of information on test content and test performance in the investigation of construct validity.
Dictation as a measure of Japanese proficiency
Authors: Kaga Mariko
Published: March 2008

While a number of researchers have reported positive and high reliability and validity for dictation as a measure of language proficiency in English and French as second languages, not much research on dictation has been con ducted with other languages, particularly where there is a close correspondence between pronounciation and orthography. Stansfield (1977) found that dicta tion was only a moderately valid measure for Spanish as a second language. The study reported here examined whether a form of modified dictation called 'graduated dictation' would be able to discriminate between different language proficiency levels in students learning Japanese at an American university. ANOVA showed that there were significant differences between proficiency levels. Results also suggested that dictation would be an effective measure of language proficiency even if a target language has a very close relationship between its pronounciation and orthography.
Test dimensionality: IRT analysis of an ESP listening test1
Authors: McNamara T.F.
Published: March 2008

The use of Item Response Theory (IRT) in the examination of the qualities of language tests is a comparatively recent development, and one that has proved controversial. Oller (1983) contains no reference to IRT in a wide-ranging col lection. By contrast, IRT, particularly involving the family of models deriving from the work of Rasch (1960 [1980]), has featured in a number of studies since the early 1980s. Although Rasch IRT has sometimes been discussed primarily as a tool for improved investigation of the reliability of tests (Skehan, 1989), its potential for investigating aspects of the validity of language tests has also been demonstrated (McNamara, 1990a). The application of Rasch IRT in this latter role has in some cases met with objections based on what are claimed to be its unsatisfactory theoretical assumptions, in particular the so-called 'unidimensionality' assumption (Hamp-Lyons, 1989). In this paper, this issue will be discussed in the context of the analysis of data from an ESP listening test for health professionals, part of a larger test, the Occupational English Test (OET), developed on behalf of the Australian Government (McNamara, 1989b). The paper is in three sections. First, there is a brief description of the listening subtest of the OET. Second, the appropriateness of the use of Rasch IRT in language testing research is discussed. Third, the use of Rasch IRT in the validation of the listening subtest of the OET is reported. In this part of the paper, the issue of unidimensionality is considered in the context of analysis of data from the two parts of this test.
Dimensionality and construct validity of language tests
Authors: Henning Grant
Published: March 2008

At times 'unidimensional' tests have been characterized as being unduly constrained in what they can measure, and ultimately as lacking construct validity for communicative language assessment since communicative lan guage ability is held to be 'multidimensional' in nature. This simulation study considers the effects on statistical measures of test dimensionality that result from systematic sampling variation in both a single and a double-trait assessment model. Results are offered as evidence that there are both psychological and psychometric states of test dimensionality that are distinct, and that psychometric unidimensionality is not at all exclusive of psycho logical multidimensionality. Moreover, it is argued that psychometric uni dimensionality may often be rather more reflective than detractive of construct validity, even when complex multidimensional constructs underlie performance on the test.
Crossvalidation of item response curve models using TOEFL data
Authors: Boldt Robert F.
Published: March 2008

In a previous research study, factor analyses for TOEFL item phi-coefficients yielded a single factor for each section. This result conforms to the assumption that, within TOEFL sections, the item response curves are proportional. This assumption, called PIRC for 'proportional item response curve', could serve as a basis for simpler application methods than are now being used. Use of the assumption entails estimating fewer parameters and simpler computations than with current methods. Other anticipated benefits are reduced chances of error in calibrating items and smaller equating samples, which would in turn allow pretest evaluation of substantially more items for possible operational use.

To evaluate the model further, a crossvalidation study was undertaken in which PIRC was used to predict item scores of selected examinees on selected items. The three-parameter logist model and a modified Rasch (one-parameter) model were used as well. The study also compared pre dicted half-test scores with actual scores for each method. These comparisons were made using varying sampling sizes to calibrate the items and calculate scores for the examinees. Surprisingly, the models' accuracies of prediction were approximately the same, and the estimation sample size appeared to make little difference.
Using dictionaries to estimate vocabulary size: essential, but rarely followed, procedures
Authors: Nation Paul
Published: March 2008

This article describes the steps that need to be followed when sampling from a dictionary to make a test of vocabulary size. Because these steps have not been followed, most published estimates of vocabulary size are misleading.
Assessing different components of reading comprehension: fact or fiction?
Authors: Rost Detlef H.
Published: March 2008

The starting point of this study was the debate about the structure of L1 reading comprehension. Kalb, Rabenstein and Rost (1979a; 1979b) claim to assess several different subskills of reading comprehension with the test battery Lesen und Verstehen - Diagnose (Reading and Comprehension- Assessment) (RuC-A) they designed specially for this purpose. To check this, 220 second-grade German elementary-school pupils were tested with the RuC-A. The results show medium-to-high correlations for the eight subtests (0.56 ≤ r ≤ 0.85), and some correlation coefficients are nearly as high as the corresponding subtest reliabilities. Corrected for attenuation, they reach 0.66 ≤ rk ≤ 0.99. Depending on the theoretical perspective, factor analyses (with the reliabilities of the subtests as communalities) yield either one broad factor, 'General reading competence' (accounting for 85% of the nonchance variance) or, at most, two factors, 'inferential reading comprehension' and 'vocabulary' (accounting unrotated for 85% and 6%, rotated for 55% and 36% of the reliable variance). As in other comparable L1 reading compre hension tests, RuC-A apparently cannot measure several clearly distinguish able components of reading comprehension. A reliable and valid diagnosis of typical L1 reading comprehension profiles is not possible.
What are the characteristics of natural cloze tests?
Authors: Brown James Dean
Published: March 2008

This study investigates the characteristics of natural cloze tests. Natural cloze tests are defined here as cloze procedures developed without intercession based on the test developer's knowledge and intuitions about passage difficulty, suitable topics, etc. (i.e., the criteria which are often used to select a cloze passage appropriate for a particular group of students). Fifty reading passages were randomly selected from an American public library. Each passage was made into a 30-item cloze test (every twelfth word deletion). The subjects were 2298 EFL students from 18 colleges and universities in Japan. Each student completed one of the 30-item cloze tests. The 50 cloze tests were randomly administered across all of the subjects so that any variations in statistical characteristics could be assumed to be due to other than sampling differences. The students also took a 10-item cloze test that was common to all students.

The 50 cloze tests were compared in terms of descriptive, reliability and validity testing characteristics. The results indicate that natural cloze tests are not necessarily well-centred, reliable and valid. A typical natural cloze is described, but considerable variations were also found in the characteristics of these cloze tests (with many of them having skewed distributions and/or poor reliability). The implications for cloze test construction and use are discussed.
Modified scoring, traditional item analysis and Sato's caution index used to investigate the reading recall protocol
Authors: Deville Craig , Chalhoub-Deville Micheline
Published: March 2008

The written recall protocol is increasingly being used in second language reading research as a measure of comprehension. Although the recall protocol is an essay-like instrument, the total score derived is based entirely on summing the discrete propositions correctly recalled. Consequently, item and reliability analyses comparable to those run on multiple-choice tests can and should be performed on recall protocols and on other integrative measures. The purpose of this study is to demonstrate how modified scoring, item and internal consistency analyses, along with Sato's caution indices, can be used to evaluate the quality of the recall protocol as a reading comprehen sion measure. Issues concerning the assumptions underlying classical local independence are discussed with regard to the reading recall protocol and other integrative measures. Results indicate that the procedures can be applied and can yield interpretable results. These results need to be replicated using other texts and other weighting systems. Only when recall protocols are routinely subjected to item and reliability analyses comparable to those performed on other measures, can the instrument be considered a viable alternative.
The assessment of language impairment in elderly bilinguals and second language speakers in Australia
Authors: Baker Rosemary
Published: March 2008

The pattern of migration to Australia over the last sixty years has been such that by the year 2001, almost 25% of the population aged 65 or over will have a first language other than English (Rowland, 1991a). Given that there is an age-related increase in the incidence of neurologically based language impairment, the need for appropriate language assessment procedures for elderly bilinguals and speakers of English as a second language is becoming increasingly apparent to health professionals in Australia. The design of such procedures presents particular challenges, however, and it is the purpose of this article (1) to set these out and (2) to consider ways in which they might be met.
The effects of note-taking on listening comprehension in the Test of English as a Foreign Language
Authors: Hale Gordon A. , Courtney Rosalea
Published: March 2008

This study examined the effects of taking notes in the portion of the TOEFL listening comprehension section that contains short monologues, or 'mini talks'. These effects were assessed in experimental testing sessions with students in intensive English language programmes and with undergraduate and graduate international students. A multiple-choice questionnaire sur veyed the students' reactions to the opportunity to take notes and their previous note-taking experiences.

Allowing students to take notes had little effect on their performance, and urging students to take notes significantly impaired their performance. These effects were observed even for students who reported being in the habit of taking many notes or reported having had classroom instruction in note taking. Apparently, then, little benefit is gained by taking notes in the context of the present TOEFL minitalks, perhaps because they are designed to assess listening comprehension with minimal demand placed on memory. Responses to the questionnaire aid in understanding the results and provide useful general information about the students' note-taking experiences and habits.
'Sprogtest': a smart test (or how to develop a reliable and anonymous EFL reading test)
Authors: Dollerup Cay , Glahn Esther , Hansen Carsten Rosenberg
Published: March 2008

This article deals with a Danish English-language reading proficiency test. It is dubbed 'Sprogtest' which means 'language test'. Since English textbooks are used extensively in Denmark, the test is offered to freshman students in order to diagnose weaknesses which may impede the undergraduates' academic careers. The test has to be anonymous and convincing to those who use it. In order to facilitate the immediate assessment of what parts can be transferred and used in other language areas, the article discusses the test construction, development and improvement in detail.
Quantity and quality in speaking test performance
Authors: Douglas Dan
Published: March 2008

Scores on speaking tests are used as evidence of both learner language ability and the second language acquisition process, and most speaking tests include scoring rubrics to help ensure that ratings are reliable and reflect a theoretical construct of speaking ability. Nevertheless, it may be that similar ratings on a speaking test in fact represent qualitatively different learner performances. Such a situation would mean that interpretations of ability or acquisition process based on such test scores may not be valid. The purpose of this article is to investigate the hypothesis that similar quantitative scores on a speaking test represent qualitatively different performances. The results of the study raise a number of issues for further investigation.
Modelling relationships among some test-taker characteristics and performance on EFL tests: an approach to construct validation
Authors: John Kunnan Antony
Published: March 2008

Construct validation has seen two proposals recently: the use of construct representation and nomothetic span variables through structural modelling and the concept of population generalizability. This study investigated the influence of two major test-taker characteristics (TTCs), social milieu or cultural background and exposure or previous instruction, on test perform ance (TP) in tests of English as a foreign language (EFL) for two native language groups, the non-Indo-European (NIE) and the Indo-European (IE). Data from the Cambridge-TOEFL comparability study (Bachman et al., 1991; N = 1448) from eight sites in eight countries was used. The instruments were 1) a 45-item Likert scale background questionnaire which captured the test-taker characteristics; and 2) the First Certificate in English, administered by the University of Cambridge Local Examinations Syndicate, the TOEFL and the SPEAK, administered by the Educational Testing Service, Princeton, and the Test of English Writing. Modelling of the TTCs and the TP factors generally supported an equal influence factors model (where the factors have equal status) and an intervening factors type model (where the factors are not equal in status and one factor is an intervening factor) for both the NIE and IE groups.