04 Version 1 . 0 : September 2012 Aggregating fixation data across trials of different durations

When individual time series (i.e., trials) have differing durations, the researcher must decide how to aggregate data at time points where some trials do not have data. Some researchers argue that since there is no data for those trials, they should be excluded. Other researchers include those trials and “fill in” or “pad” the data with sensible values. Here we consider the specific case of gaze data from “visual world paradigm” studies and use Monte Carlo simulation to generate simplified gaze data to examine the consequences of different aggregation approaches. The results provide a concrete demonstration that excluding trials with no data is a form of selection bias that systematically distorts results. Unbiased data aggregation requires that the “denominator” (i.e., number of trials) remain the same for each time point in the analysis. LCDL Technical Report 2012.04 Aggregating trials of different durations 2 Aggregating fixation data across trials of different durations In a typical “visual world paradigm” (VWP) eye tracking study of spoken language comprehension, participants view an array of images or words on a computer screen, and hear a linguistic stimulus relevant to one or more of the elements of the array. The trial-level data are strongly constrained by the physical mechanics of the oculomotor system, so these data are aggregated over trials by condition and participant (or item) in order to reduce the oculomotor contributions and extract the underlying cognitive contributions. One of the most basic and oft-ignored issues in analysis of eye tracking (and other time series) data is how to aggregate data across trials of different durations. Typically, a trial ends when the participant responds, which naturally leads to some trials that are shorter than others. It is hypothetically possible to restrict the analysis to the time window before any trials have terminated, but this is generally impractical because the time course of processing usually evolves over a substantially longer time window than the single shortest response time. So, when computing fixation proportions at later time points, should terminated trials be included or not? Three approaches are currently in use: (1) for each time bin, include all trials and count post-response frames as non-object fixations (i.e., the participant is done fixating all objects from this trial), (2) include all trials and count post-response frames as target fixations (i.e., if the participant selected the correct object, then consider all subsequent fixations to be on that object; note that, typically, any trials on which the participant made an incorrect response are excluded from analysis), (3) include only trials that are currently on-going and ignore any terminated trials since there is no data for those trials. It is important to note that researchers rarely even report which of these aggregation methods they used, so a formal survey of the literature is not possible. In the domain of applied regression, the challenges of missing data are well-known, as are various approaches for handling them (e.g., Gelman & Hill, 2007, Chapter 25), but regression techniques have only recently penetrated into the world of eye tracking and psycholinguistics (e.g., Baayen et al., 2008; Barr, 2008; Mirman, Dixon, & Magnuson, 2008). Critically, aggregating data over trials of different lengths differs from other cases of missing data that may be more familiar in psycholinguistics. Focusing on eye tracking studies, data points may be missing for essentially random reasons such as equipment failures, blinks, etc., but these would affect all time points and conditions equally. In contrast, trial termination times are intrinsically related to cognitive processing – more difficult trials tend to last longer. Thus, the selective elimination of completed trials represents a form of selection bias. To our knowledge, the consequences of choosing one aggregation method over another have not been examined concretely. There are two points concerning the relative merits of the three methods of data aggregation described above. First, the two methods that include all trials are fundamentally the same – they will capture the same data and merely depict those data differently, in the same way that probability distribution curves and cumulative distribution curves depict the same underlying data in somewhat different ways. As we will show, depending on the researcher’s goals, one visualization method or the other may be more effective or appropriate. Second, ignoring terminated trials represents a form of selection bias that will distort the data. This is because trials do not terminate at random, so as the time series progresses through the time window, the data move further and further from the complete, unbiased set of trials to a biased subset of only trials that required additional processing time. This bias will operate both between conditions (i.e., more trials from a condition with difficult stimuli than from a condition with easy stimuli) and within conditions (i.e., more of the trials that were difficult than that were easy within a condition). We will demonstrate both of these points using Monte Carlo simulation. LCDL Technical Report 2012.04 Aggregating trials of different durations 3 Methods To examine the effect of different aggregation methods, one must know the true pattern of underlying data, so we used a Monte Carlo simulation procedure to generate simplified eye tracking data. The simulations were designed to model a simple spoken word recognition experiment with two conditions: “Easy” and “Hard”, where response times are about 400ms slower in the Hard condition, as might result from a manipulation of word frequency, cohort density, or other lexical variables (e.g., Magnuson, Dixon, Tanenhaus, & Aslin, 2007). Response times were sampled randomly from the gamma distributions shown in the left panel of Figure 1 (shape = 4, scale = 1 for both distributions; to convert raw gamma distribution values to the RT range, they were multiplied by 200 and 800 (“Easy”) or 1200 (“Hard”) was added). For the target fixation analysis, we simulated the simplest case: participants make exactly one 500ms fixation on the target just before clicking on it (i.e., there was a target fixation for the 500ms interval ending with that trial’s RT). This is schematically shown in the right panel of Figure 1 for a random subset of 50 trials (25 from each of the two conditions). Figure 1. Left panel: RT distributions from Monte Carlo simulations of “Easy” and “Hard” conditions. Right panel: Time periods of target fixation for a subset of 50 trials (25 from each condition). For the competitor fixation analysis, 75% of the way through each trial there was a 50% probability of a 500ms fixation to the “Competitor” (i.e., related) distractor and a 25% probability of a 500ms fixation to the “Unrelated” distractor. For each analysis, 50,000 trials were simulated (25,000 for each condition) and the results aggregated according to the three different methods described above: (a) consider all post-response data as a fixation somewhere other than the critical objects (“non-object fixation”), (b) consider all post-response data as target fixation, or (c) consider only on-going trials. Note that for the competitor fixation analysis, methods (a) and (b) are equivalent (because the target is, by definition, not a competitor object) so they will not be distinguished in the presentation of the results. In VWP experiments, researchers typically exclude some trials from analysis (e.g., error responses). For LCDL Technical Report 2012.04 Aggregating trials of different durations 4 simplicity, we did not exclude any trials from this analysis. The analyses were conducted in R version 2.13 (R Development Core Team, 2011).