Comparabilité entre modalités d’évaluation TIC et papier-crayon : cas de productions écrites en français en cinquième secondaire au Québec

Writing skills are no longer limited to those developed in performing paper-and-pencil tasks. Word processing is ubiquitous and becomes indispensable in writing authentic situations. In Messick (1989) perspective, writing assessment that is based on handwritten documents raises authenticity and validity problems. For instance, some studies have revealed that students who use to write with a computer perform better in word processing test conditions than on handwriting conditions (Russell and Haney, 2000). We need to better understand the impact of using a computer for a test as opposed to administering paper-and-pencil test: Is the writing process modified? Are the testing conditions equivalent? Should spell and grammar check software be disabled? Is the assessment process different when the reader has to rate in different modes of presentation - handwritten essays or word processed essays that are either printed or displayed on the screen? Do both assessment methods lead to equivalent results? The purpose of this study is to compare Grade 11 Quebec students’ writing assessments based on essays written with a computer versus handwritten essays. The conceptual framework is built on the concepts of Writing process and Writing assessment. The approach is both quantitative and qualitative. Two writing tasks previously used in the Grade 11MELS French writing exam were used to administer a test with a computer and a paper-and-pencil test. The sample consisted of 127 students from two schools. In the first school, students are enrolled in a regular program with access to ICTs and the spell and grammar check software was disabled. In the second school, ICT use is fully integrated in teaching and learning. In this group, the spell and grammar check software was available. Three writers were observed in the computer mode and two in the handwritten mode. Observations on the writing process were collected in three ways - verbalization and video camera for both modes, screen capture added in computerized mode. In addition, to collect data on the assessment process, the three teachers were observed during the rating task by two means (think-aloud and video camera) in different modes of presentation: two assessed a set of paper essays, one a set of paper that were displayed on the screen and three a set of manuscript essays. Qualitative observations on the writing process and the assessment process were collected and transcribed. Quantitative data consisted of scores given by the three raters using the MELS correction grid. 254 essays (127 handwritten, 77 word-processed and printed, 50 word-processed and displayed on the screen), were analyzed. First, the analysis of the qualitative data on the writing process revealed that some strategies from different writing sub-processes involved in word processing are used exclusively or more frequently when the writer uses a computer. Their application means moving more frequently between different sub-processes when the writer uses the word processor. Therefore, the writing process is more recursive with word processing. Then, the analysis of the quantitative data on assessment process revealed that : 1) the teacher not only identifies errors and problems in text; he also identifies positive features in the essay. This is important because previous frameworks focused on the errors detection; 2) The difference in the mode of presentation does not influence rater’s strategies in the detection of relevant features.  Finally, as far as the comparability of scores is concerned, the statistical analysis revealed that: 1) Both the way an essay is produced (word processing or handwriting) and the type of program influence the outcome; 2) Enabling the spell and grammar check software seems to has a small impact whereas disabling it has an important impact.

Keyphrases: Technology and assessment, Writing assessment, Writing process, Text rating process, Computerized assessment, Equivalence of scores