title: Investigating test-takers' cognitive processes while completing an
      integrated reading-to-write task
title: evidence from eye-tracking, stimulated recall and questionnaire
titleType: Subtitle
Creators: name: Wang, Pucheng
Publisher: name: Lancaster University
Year: 2018

description: The TBEM-8 (Test for Business English Majors, band 8) is a
      newly-developed, nationwide test of business English proficiency
      administered to business English undergraduates in China at the end of
      their final year. One notable feature of the test is that it includes a
      reading-to-write task in which test-takers read texts in English and
      Chinese and then use this information to write an essay on a
      business-related topic. Although the test has been operational for several
      years, there is currently little validity evidence to support claims about
      the cognitive processing which takes place during this reading-to-write
      task. This presents a threat to the quality of inferences drawn from test
      scores.
      
      The present research examined test-takers’ cognitive processes while
      completing the TBEM-8 reading-to-write task, aiming to gain further
      insights into cognitive processing on this integrated task type. Two
      separate studies were conducted. In Study I, 16 participants completed
      this task while their eye movements were tracked by a Tobii TX300
      eye-tracker. These eye traces then formed the stimuli for a stimulated
      recall session to elicit cognitive processes; in Study II, another 172
      participants responded to a reading-to-write process questionnaire after
      completing the task. This questionnaire was developed by Chan (2013) and
      adapted for the TBEM-8 reading-to-write task. A pilot study was also
      conducted to finalise the main study questionnaire, in which 40 items were
      grouped to reflect the cognitive processes that writers are hypothesised
      to undergo.
      
      The results showed that test-takers engaged in a wide range of cognitive
      processes specified in Shaw and Weir’s (2007) model of writing and
      Spivey’s (1990, 1997, 2001) discourse synthesis model during task
      completion, thus justifying the current use of it in the TBEM-8 test. Text
      interpretation and selecting were the two most frequently reported
      processes according to participants’ stimulated recalls, and
      macro-planning and translating were the two least reported processes. A
      high level of agreement was found in participants’ responses to the
      reading-to-write process questionnaire, with more than 70 percent of
      participants choosing either “agree” or “strongly agree” in 28 items, and
      only four items achieving an agreement rate below 60 percent.
      
      The correlation analysis between the use of cognitive
      processes/eye-tracking measures and test-takers’ performance on the TBEM-8
      reading-to-write task yielded no statistically significant results(at the
      0.05 or 0.01 levels), except for a moderate positive correlation (ρ=.499,
      p=.049) between the participants’ max visit duration on Source 5 (key
      concepts and expressions) and their reading-to-write performance, and one
      (ρ=.432, p=.098) between the counts of text interpretation-2 process
      (reading source materials) and the task performance if the p-value was set
      to 0.1.
      
      This study demonstrated the usefulness of combining eye-tracking,
      stimulated recall and questionnaire methods for generating insights into
      the complexity of cognitive processing on an integrated reading-to-write
      task. Findings from the analysis of all sources of data were triangulated
      and discussed, providing a solid basis for the conclusions drawn about
      test-takers’ cognitive processing during task completion. Also, a model of
      reading-to-write process was proposed to illustrate how different
      categories of cognitive processes examined in this study interact with
      each other for successful task completion.