The Presence and Impact of Order Effects in Airline Pilot Competency Assessments

This study investigated whether competency assessments of professional Airline Pilots are subject to order effects such as primacy or recency effects. 18 examiners participated in a web-based experiment where they evaluated crew performance in three short video-recordings of a flight crew performing in a simulator. The scenarios, depicting three levels of competency, were randomly presented in either an improving or deteriorating order. A final, overall evaluation was made following the three scenarios. Results indicate no recency effects, although weak statistical signs of systematic differences in gradings between presentation orders exists. A low number of participants combined with high variations stipulates a cautious conclusion that order effects are likely to have been present and may skew the appraisal of true performance. In many professions, performance and competency assessments are natural and integrated parts of working life. This is especially true when it comes to professional airline pilots, who are assessed on their competence on a minimum of three occasions annually (European Commision, 2012). Regardless whether the purpose of workplace competency assessments is for safety assurance, career path development, licensing requirements etc, it is nonetheless a reasonable desire that assessments are fair, accurate and reliable. However, assessments carried out by individual assessors are fallible and subject to errors such as observational inaccuracy and cognitive biases (Morgeson & Campion, 1997). Performance evaluations are complex cognitive tasks, consolidating several mental computations. These processes comprise of observing, directing attention, detecting and selecting relevant behaviours, categorizing them in the working memory and then make comparisons towards prevalent professional norms and standards (Tavares & Eva, 2013). As long as the evaluation process takes place, these cognitive operations are performed for a multitude of stimuli simultaneously. However, with increasing complexity follows an increasing risk that assessors revert to mental shortcuts and heuristic principles that may limit or bias the evaluations (Tavares & Eva, 2013). Even in settings with predefined competency standards such as within medical education and flight crew performance assessments, previous studies recognises that there are indeed considerable variances in the way assessors appraise job performance and competency (Gontar & Hoermann, 2016; Kogan, Conforti, Bernabeo, Iobst, & Holmboe, 2011; Roth, Mavin, & Munro, 2014). There are many factors that explain assessors’ varying judgements such as frame of reference (Kogan et al., 2011), attention and observational errors (Tavares & Eva, 2013) or pre-occupancy with controlling simulations or training scenarios (Baker & Dismukes, 2002). Among the variability of cognitive errors that may occur is the tendency amongst assessors to compare with, and become biased by recently observed performance (Yeates, O'Neill, Mann, & Eva, 2012). The sequence in which a number of personality traits are described have proved to have an impact of the overall evaluation of a personality (Asch, 1946). Later studies have dealt with order and sequence effects in job interviews (Farr, 1973), memory research (Greene, 1986) and competency assessments (Murphy, Balzer, Lockhart, & Eisenman, 1985; Yeates et al., 2012). A number of various terms and concepts have been employed to illustrate these effects. In the context of competency and performance assessments these have