A set of tasks to test interpretation and use of emoji by children

In light of the difficulties children with an autism spectrum disorder (ASD) face when reading facial expressions, as well as their tendency to prefer online interaction to face-to-face communication, a study was planned to investigate whether children with ASD interpret and use emoji differently from typically developing children. Three tasks were developed to test how children group, interpret, and apply emoji. Due to setbacks in recruiting a test group in the limited time frame, an alternative criterion for division into two test groups was selected post hoc. Participants were divided according to their self-reported frequency of emoji use. Various differences were found and will be discussed, along with recommendations for future continuation of this research with participants with ASD. Introduction Autism Spectrum Disorders (ASD) form a group of developmental conditions associated with trouble with social skills. People with ASD generally find expressing their own emotions difficult, as well as interpreting the emotions displayed by others (Baron-Cohen, 1993). Subtleties like tone of voice and sarcasm may be lost on people with ASD. They also have been shown to have trouble interpreting facial expressions from illustrations (Celani, 1999). As the label suggest, people with ASD fall on a spectrum. When someone with ASD has an IQ of 70 or greater, they are said to have high-functioning (HF) autism (Carpenter, 2009). Recent studies have shown that children with HF ASD are equally good at recognising the basic emotions of happiness, sadness, anger, surprise, fear, and disgust in photographs as typically-developing children (Castelli, 2005). These studies were conducted with photographs designed for testing recognition of basic emotions. There are currently no studies investigating if children with HF ASD are able to recognise more complex emotions from photographs, and if they can recognise emotions from different pictures, like icons. People with ASD tend to be more comfortable in online communications (Benford, 2009). Digital devices such as tablets have also been shown to help people with ASD in social interactions (Hourcade, 2013). Software has been developed with the goal of training individuals with ASD on how to behave in social situations. These programs showed good results, with the best results being attained by games that involved a roleplay component that mirrored natural social interaction (Mitchell, 2007). With recent advances in technology, research has also been done to show how new technologies can be used in interventions for people with ASD. An example of this was the use of virtual reality environments designed to train children with ASD in interactions (Ke, 2013). Today, digital communication is ubiquitous. Whenever we want to send a quick message to anyone we do so through text messaging. However, text messaging lacks the interactive depth that face-to-face communication offers with added dimensions like body language. People without social developmental disorders are able to infer whether a text message has a positive or negative emotional valence. Communication partners used strategies like verbosity and punctuation to express negative affect, which was understood by both partners (Hancock, 2007). Aside from these strategies, emoji can also be used to help the reader understand the emotion behind the message. Emoji are a collection of icons representing a simplified human face with exaggerated emotional display. These pictures are often used in online communications to help convey emotion with text, something that is harder to accomplish without images (Garrison, 2011). The difficulty with interpretation of emotions from facial expressions found in people with ASD may have some implications for online communications. This paper will investigate whether children with HF ASD have trouble interpreting emoji, in the same way they have trouble interpreting emotions from regular facial expressions. Research questions Originally this paper aimed at answering 2 research questions: in comparison with typically developed peers, to what degree are children with HF ASD (1) able to interpret the use of emoji? ; and (2) able to pick a fitting emoji when given a sample conversation? The idea was to recruit an equal number of participants at primary schools with and without ASD. However, only 5 participants with an ASD diagnosis were found within the time frame of this study, versus 44 participants without ASD. This ratio made the between-group comparison necessary for answering research questions (1) and (2) virtually impossible. With this in mind, it was decided to add a third research question exhibiting a better fit with the available data: (3) Does frequency of emoji-use have an effect on emoji recognition and interpretation? A difference in emoji recognition and interpretation between frequent and infrequent emoji users could show whether emoji use relies rather on learned versus innate mechanisms. If infrequent users interpret emoji very differently from frequent users, this would indicate a tendency towards learning; after all, it shows that infrequent users haven’t used emoji enough to have acquired a consensus on their meaning. Conversely, if no difference is found it would point towards emoji interpretation relying more on innate mechanisms. Methods and test materials This study was approved by the ethics board of the Faculty of Social Sciences at Leiden University in the Netherlands on the 2nd of June 2017. Participants were recruited from elementary schools in the western part of the Netherlands. Recruitment took place from September through November 2018. In total, 49 participants agreed to participate, 5 of which had an ASD diagnosis. The participants’ ages ranged from 10 to 15, with most participants being 12 years old. 45% of the participants were boys, 55% were girls. The tasks that are described below were developed by the researcher in Qualtrics. This allowed for a test that could be completed on various devices, as well as an easy way to manage responses. Responses were coded to separate the groups based on ASD diagnosis. Data and consent forms will be saved at the faculty of social sciences at Leiden University for 5 years after the study. The tasks consisted of two main parts. The first part asked participants for their age, gender, and their experience with emoji. The second part contained the main tasks. The main tasks were a grouping task, an interpretation task, and an application task. Each part will be described in detail below. In the first part, participants were asked some questions about their own use of emoji. We asked participants how often they used emoji, how clear they thought others’ emoji use is, and whether they thought others understood their use of emoji. Participants rated their answers on a 5-point scale (see figure 1). Figure 1. Example question. Participants were asked how often they use emoji and had to rate their answer on a 5-point scale, with the low end being labelled “never” and the top end “very often. ” Next, participants did three different tasks: a grouping task, an interpretation task, and an application task. Originally, the first two tasks were designed to test research question (1), the third task to test research question (2). With each question, the differences between groups were analysed. For the grouping task, participants were shown a word for one of four basic emotions, “happy,” “angry,” ”sad,” or “surprised. ” They then had to choose as many emoji as they wanted from a group of 10 that they thought fit the word. See figure 2 for an sample of these questions. We will focus on differences between groups, rather than trying to decide on the ‘correct’ use of emoji a priori and using this as a benchmark for measuring participants’ ‘performance’. In other words, our view is that there is never a ‘right’ answer to the task, but it should be used to measure differences in average answers between groups. Figure 2. Sample of the grouping task. Participants are asked to pick all happy faces from a set of 10. Some emoji clearly depict happy or sad faces, while others are more ambiguous. In the interpretation task participants were shown an emoji (target emoji). They were asked to match 1 of 4 emoji representing the basic emotions of happiness, sadness, anger, and surprise to the target emoji. Then they were also asked which word they thought best described the test emoji. This is to control for the participant selecting a word not describing the chosen emoji, thus possibly misunderstanding the chosen emoji in the first place. See figure 3 for a sample of this task. Figure 3. Sample of the interpretation task. The emoji matching and word matching questions were presented on separate pages. They are shown in one image for clarity. The word matching task asks participants which word they think matches the target emoji, with options for sad, happy, surprised, and angry. The application task involved text conversations written by the researcher. In each text conversation a space was marked. Participants were asked to fill in this space with one of six provided emoji. Each set of 6 emoji contained 2 with a positive emotional valence, 2 with a negative emotional valence, and 2 ambiguous emoji. The text conversations were about subjects a participant could encounter in daily life, like losing a pencil case. The conversations were written in a way that multiple emoji answers could be appropriate. See figure 4 for a sample of this task. Figure 4. Sample of the application task. The text conversation reads: “I’ve lost my pencil case, is it at your place? ” “I’ll have a look. ” “I haven’t found it. ” The participant was asked to choose which emoji they would put in the red circle. To determine which emoji would be investigated the tasks were first given to a group of 8 adults. This group was recruited among friends and family of the researcher. Based on their an