Human vs . Computer : Scoring a Test of Emotional Awareness

Emotional intelligence is the ability to perceive, understand, use, and mange emotions (Mayer et al, 2003). One key aspect of emotional intelligence is emotional awareness: the ability to describe feelings in oneself and others (Lane & Schwartz, 1987). The Levels of Emotional Awareness Scale (LEAS; Lane et al., 1990) includes 20 open-ended items. For each, participants read a short scenario that involves themselves and another person. Then they describe how they and the other person would feel. These written responses are scored based upon the type and number of emotion words used. Traditionally, the LEAS is scored by hand. However, hand scoring is time-consuming. Therefore, Program for Open-Ended Scoring (POES) was created to score the LEAS (Leaf, 2003). The purpose of this study was to compare the validity of hand scoring and the eight POES scoring methods, by correlating these scores with the 141-item MayerSalovey-Caruso Emotional Intelligence Test (MSCEIT; Mayer et al., 2003). A total of 356 undergraduates completed the LEAS and MSCEIT online. All correlations were significant, thus supporting the validity of the LEAS. HighestN and HighestNUnique had the highest correlations, making these methods a viable alternative to hand scoring. The new AllSumUnique method had a higher correlation than the AllSum method, and the new HighestNUnique method had a higher correlation than the HighestN method, thus indicating that duplicate words add little to our knowledge of someone’s emotional awareness. The new CountCat method counts the number of distinct categories used. It deserves further exploration, perhaps with revised categories that more precisely examine the richness of one’s emotional descriptions. Introduction Emotional intelligence is the ability to perceive, understand, and manage emotions and to use emotions to facilitate thinking (Mayer, Salovey, & Caruso, 2004). One key aspect of emotional intelligence is emotional awareness. Emotional awareness is a cognitive ability that allows individuals to identify and describe feelings in themselves and others (Lane & Schwartz, 1987). Emotional awareness is important because recognizing emotion and understanding emotion is something that we do in our day to day functioning. Moreover, emotional awareness may be a prerequisite to learning higher order emotional skills. The Levels of Emotional Awareness Scale (LEAS; Lane, Quinlan, Schwartz, Walker, & Zeitlin, 1990) is an openended measure of emotional awareness. The LEAS contains 20 emotionally evocative scenarios that involve the self and another person. For each scenario, participants describe how they would feel and how the other person would feel. Participants’ scores on the LEAS depend upon the emotion words they use. Participants get higher scores when they use a larger number of emotion words and more specific emotion words. The LEAS can be scored by hand or by computer. Scoring a LEAS response by hand requires expert subjective judgments based upon the meaning of words, the context in which they appear, and the presence of synonyms. Learning to make these subjective judgments takes approximately 10 hours (Barchard, Bajgar, Leaf, & Lane, 2010). Once a person is fully trained, doing the scoring itself is also time-consuming. Even the most experienced hand scorer will take about 10 minutes to score each participant (Barchard et al., 2010). Thus, scoring 100 participants (a reasonable sample size for most psychological studies) would take about 17 hours. Computer scoring of the LEAS is also possible. Automated scoring was suggested when the LEAS was first developed in 1990 (Lane, Quinlan, Schwartz, Walker, & Zeitlin, 1990) but it wasn’t until 2003 that a computer program was developed to score the LEAS automatically (Leaf, 2003). Subsequent research has demonstrated that the initial Program Open-Ended Scoring (POES) scoring methods had strong correlations with hand scoring, high internal consistency, and validity that was comparable to or higher than the validity of hand scoring (Barchard et al., 2010). Recently, a new version of POES has been developed (Ermini Leaf & Barchard, 2013). The AllSum and HighestN techniques remain the same as in the previous version of POES. However, the remaining scoring methods have been revised to be more flexible. First, the AllSum-AllinOne and HighestN-AllinOne methods were replaced by the AllSumUnique and HighestNUnique methods, which allow scores to be calculated at the subpart and item level, a feature that was not available in past versions. Second, the 334 and 3345 methods have been replaced with the UniqueMaximums method. Finrally, POES 2.0. 1 incorporates three new scoring methods (CountWords, CountCat, and CountFreq). The purpose of this study is to compare the validity of hand scoring and the eight POES 2.0. 1 scoring methods of the LEAS. To do so, we correlated the LEAS scoring methods with the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT; Mayer, Salovey, Caruso & Sitarenios, 2003). Method Participants A total of 356 university students (204 females, 152 males) participated in this study in return for course credit. They ranged from 18 to 50 years (mean 19.95, SD 3.53). Participants identified themselves as follows: Caucasian (57.9%), Hispanic (12.4%), Asian (12.4%), African American (7.0%), Pacific Islander (6.2%), Native American (0.3%), and other (3.9%). Measures The Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT; Mayer, Salovey, Caruso & Sitarenios, 2003) is a 141-