title: Weaving CEFR Scales into Rubrics for EFL Writing Assessment in Indonesia
Creators: name: Hoinbala, Festif Rudolf
Publisher: name: University of Muenster, Institutional Repository MIAMI
Year: 2024
Subjects: subject: 420 English and Old English (Anglo-Saxon)
subject: 370 Education

description: Diese Studie untersuchte die Erstellung einer geeigneten Bewertungsskala, die aus den CEFR-Skalen gewebt wurde und zur EFL-Schreibbewertung in Indonesien passt. Zu diesem Zweck wurden die „Can-Do-Aussagen“ in den CEFR-Deskriptoren an die entwickelte Bewertungsskala angepasst. Diese Studie ergab drei Hauptergebnisse mit einem gemischten qualitativen und quantitativen Ansatz. Das erste war das Modell einer Bewertungsskala für die schriftliche Beurteilung, die von den CEFR-Skalen übernommen wurde und aus Inhalts-, Organisations-, Grammatik-, Wortschatz- und Mechanikkomponenten besteht. Die Standardisierung der Komponenten, das Niveau und die Gewichtung der entwickelten Bewertungsskala basierten auf dem EFL-Schreibbewertungskontext. Darüber hinaus kam es nach Verwendung der entwickelten Bewertungsskala zu einer signifikanten Verbesserung von einem moderaten auf ein starkes Niveau der Inter-Bewerter-Zuverlässigkeit, und die Übereinstimmung zwischen den Bewertern in Bezug auf die Intra-Bewerter-Zuverlässigkeit war stark. Schließlich waren die Reaktionen der Bewerter insgesamt positiv, als es um die Umsetzung der Bewerterschulung ging.
description: This study investigated the production of a proper rating scale weaved from the CEFR scales that fit the EFL writing assessment in Indonesia. For this purpose, the “can-do statements” in the CEFR descriptors were adapted to the developed rating scale. This study had three main findings using a mixed qualitative and quantitative approach. The first was the model of a rating scale for writing assessment adopted from the CEFR scales, which consists of content, organization, grammar, vocabulary, and mechanics components. The standardization of the components, the level, and the weight of the developed rating scale were based on the EFL writing assessment context. Further, a significant improvement occurred from a moderate to a strong level in the inter-rater reliability after using the developed rating scale, and the agreement among raters in intra-rater reliability was strong. Finally, the raters’ responses were overall positive when discussing the implementation of the rater training.
description: ACKNOWLEDGEMENTS ..... i
ABSTRACT ..... iii
TABLE OF CONTENT ..... vi
LIST OF TABLES ..... xiii
LIST OF FIGURES ..... xvi
LIST OF APPENDICES ..... xix
Chapter One - Introduction ..... 1
1.1. The background to the research ..... 1
1.2. Aims and Objectives ..... 4
1.3. Research Questions ..... 4
1.4. Overview of Common European Framework of Reference and its Implementation ..... 5
1.4.1. The judgments on the Implementation of the CEFR ..... 13
1.4.2. Highlights on the use of Common European Framework of Reference in Asia ..... 16
1.4.3. The Need for Standard and Reference Framework for English Language Learning, Teaching and Assessment in Indonesian Context: Is the CEFR an option? ..... 19
1.5. The overview of the thesis ..... 23
Chapter Two - Writing Assessment in EFL Classroom ..... 25
2.1. Introduction ..... 25
2.2. Assessment in the teaching-learning process ..... 25
2.2.1. Assessment concepts: Definitions and problems ..... 25
2.2.2. Issues in assessment literacy ..... 27
2.3. Forms of assessment and the implications in the teaching-learning process ..... 31
2.3.1. Diagnostic assessment ..... 32
2.3.2. Formative assessment (Assessment for Learning) ..... 34
2.3.3. Summative assessment (Assessment of Learning) ..... 37
2.3.4. Norm-Referenced Assessment vs. Criterion-Referenced Assessment ..... 40
2.3.5. Traditional Assessment vs. Alternative or Authentic Assessment ..... 46
2.4. Classroom writing assessment ..... 51
2.4.1. The nature of writing ability ..... 52
2.4.2. Writing as a performance assessment ..... 60
2.4.3. The quality of effective writing assessment ..... 66
2.4.4. Aspects of writing skills ..... 73
2.5. Trends in EFL writing assessment research in Indonesia: A brief overview ..... 76
2.5.1. Issues in the EFL teaching in Indonesia ..... 76
2.5.2. Research trends in EFL writing assessment in Indonesia ..... 79
2.6. Summary ..... 84
Chapter Three - Rating Scales, Rater Variability, Rater Training, and Rater Reliability in Writing Assessment ..... 85
3.1. Introduction ..... 85
3.2. Rating scales and scoring procedures in writing assessment ..... 86
3.2.1. Definitions and concepts of rating scales ..... 86
3.2.2. Types of rating scales ..... 89
3.3. Designing and developing rating scales for writing assessment ..... 98
3.3.1. Steps in developing rating scales ..... 98
3.3.2. Developing rating scales in line with the CEFR ..... 114
3.4. Rater variability and rater training ..... 120
3.5. Rater reliability of scoring rubrics (Review of some empirical studies) ..... 124
3.6. Summary of the Chapter ..... 128
Chapter Four - Methodology ..... 129
4.1.Overview ..... 129
4.2. Research Questions ..... 129
4.4. Participants and Settings ..... 131
4.4.1. EFL Students‘ writing samples ..... 131
4.4.2. Experts' judgments for rating scale development ..... 132
4.4.3. External raters for benchmarking ..... 132
4.4.4. Raters in the rater training ..... 132
4.5. Instrument design and development ..... 134
4.5.1. Test ..... 134
4.5.2. Rating Scale ..... 135
4.5.3. Checklist of Evaluation ..... 135
4.5.4. Observation ..... 136
4.5.5. Questionnaire ..... 137
4.6. Research Procedure ..... 138
4.6.1. Rating scale initial development ..... 138
4.6.2. Inter-rater and intra-rater reliability ..... 139
4.6.3. Rater training sessions – Rating scale final development ..... 140
4.6.4. Rater‘s Responses on rater training session ..... 144
4.7. Data Analysis ..... 144
Chapter Five - Findings and Discussion 1 – The Model of Rating Scale Weaved from the CEFR Descriptors ..... 145
5.1. Introduction ..... 145
5.2. Determining the type of rating scales and the criteria ..... 145
5.3. Weaving the CEFR descriptors to the developed rating scale ..... 151
5.3.1 Content criterion ..... 153
5.3.2. Organization criterion ..... 155
5.3.3. Grammar Criterion ..... 157
5.4.4. Vocabulary criterion ..... 159
5.3.5. Mechanics criterion ..... 161
5.4. The first draft of the rating scale developed from the CEFR descriptors ..... 163
5.5. Expert judgments –The second draft of the developed rating scale ..... 167
5.6. Raters‘ feedback and recommendations on rater training sessions – The final draft of the developed rating scale ..... 175
5.6.1. First-day workshop – Familiarization to the CEF scales and the developed rating scale ..... 176
5.6.2. Rater Training session 1 ..... 177
5.6.3. Rater training session 2 ..... 180
5.6.4. Rater training session 3 ..... 183
5.6.5. Rater training session 4 ..... 184
5.6.7. Final workshop – Finalization of descriptors wording ..... 186
5.7. Summary ..... 191
Chapter Six - Findings and Discussion 2 – Raters Reliability in Assessing EFL Students’ Writing ..... 192
6.1. Introduction ..... 192
6.2. The inter-rater reliability using raters‘ existing rating scales (initial-rating) ..... 192
6.3. The development of raters‘ reliability during the rater training sessions ..... 197
6.3.1. The inter-rater reliability of the content criterion on rater training session ..... 199
6.3.2. The inter-rater reliability of the organization criterion on rater training session ..... 200
6.3.3. The inter-rater reliability of the grammar criterion on rater training session ..... 201
6.3.4. The inter-rater reliability of the vocabulary criterion on rater training session ..... 202
6.3.5. The inter-rater reliability of the mechanics criterion on rater training session ..... 203
6.4. The inter-rater reliability using the developed rating scale adopted from the CEFR scales (final-rating) ..... 204
6.4.1. The inter-rater reliability between raters in content criterion ..... 209
6.4.2. The inter-rater reliability between raters in organization criterion ..... 210
6.4.3. The inter-rater reliability between raters in grammar criterion ..... 210
6.4.4. The inter-rater reliability between raters in vocabulary criterion ..... 211
6.4.5. The inter-rater reliability between raters in mechanics criterion ..... 212
6.5. The intra-rater reliability using the developed rating scale adopted from the CEFR scales ..... 213
6.6. Summary of the chapter ..... 215
Chapter Seven - Findings and Discussion 3 – Raters’ Responses on the Implementation of Rater Training ..... 216
7.1. Introduction ..... 216
7.2. Raters‘ backgrounds and profiles ..... 216
7.3. Questionnaire results ..... 217
7.3.1 Question 1 ..... 218
7.3.2. Question 2 ..... 220
7.3.3. Question 3 ..... 221
7.3.4. Question 4 ..... 223
7.3.5. Question 5 ..... 225
7.3.6. Question 6 ..... 226
7.3.7. Question 7 ..... 227
7.3.8. Question 8 ..... 229
7.3.9. Question 9 ..... 230
7.3.10. Question 10 ..... 232
7.4. Discussion - Raters‘ responses on the implementation of rating training on the developed rating scale adopted from the CEF scales ..... 233
7.5. Summary of the chapter ..... 238
Chapter Eight - Conclusions, Limitations, and Implication ..... 239
8.1. Introduction ..... 239
8.2. Summary of main findings ..... 240
8.3. Implications for assessment practice and policy development ..... 243
8.4. Limitations of the study ..... 248
8.5. Recommendations for future research ..... 251
8.6. Concluding remarks ..... 252
BIBLIOGRAPHY ..... 253
APPENDICES ..... 282