E m o t i o n a l A c t i v i t i e s i n C o m p u t e r- m e d i a t e dCommunication: Facial Electromyography Response
and Emoticon Use during Instant Messaging
Keywords:
Emoticon, Computer-mediated Communication, Instant Messaging, Facial Electromyography, Facial
Muscle Activity

Lei TANG, Chiba University

Abstract
In this study, we use electromyography data of the orbicularis oculi and zygomaticus major as
indices of positive emotions to analyze facial muscle activities in computer-mediatedcommunication. We find a difference in frequency of facial muscle activities, influence fromstimuli, and other aspects under varied instructions on emoticon use. Results suggest thatrichness of nonverbal methods is a key factor that affects emotional activities in computermediated communication. Although we collected data from two counties, we could not find any
significant difference between them. Instead, we found that participants from the two countries
have some patterns of emoticon use in common. We believe that, in certain situations emoticon
use and facial activities in computer-mediated communication can show commonality between
countries on a certain level.

1

CW6_A8350_02_D03.indd

1

2016/09/29

13:25:52

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG

used as a tool for inferring affective states in

1. Introduction

media-related studies; it has proven to haveInterpersonal communication involves not only

advantages over other methods. For example,

information from words or messages but alsofEMG has been used in studies of emotionalfrom facial expressions and gestures. However,

responses to television commercials; results

whencomputer-mediatedindicate that it can be more accurate and sensitive

communication (CMC) by smart phone or PC,

weperform

than self-report measures (Hazlett & Hazlett,most of the time, these nonverbal behaviors

1999). A study of user experience with mobile

cannot be communicated (Perry, 2010). To dophones proved that subtle emotional changes

this, users employ some new and simple, but

could be detected by fEMG measurementseasy-to-use methods. Emoticons are one of these(Mahlke & Minge, 2006). Other researchmethods. They can be used in most computerindicatesand smart phone software. Although emoticons

unconscious stimuli can also be detected byhave been discussed from various perspectives infEMG measurements (Dimberg & Thunberg,

previous studies, we still do not know how they2000). These advantages can be very useful forcorrelate with our facial expressions. One of the

CMC study, because when performing CMC,

purposes of our study is to confirm whether thefacial activities are less obvious and more difficult

use of emoticon has influence on facial expression.

to recognize than in face-to-face situations. Under

Similar to the facial expressions that we might

the circumstances, self-report or visual methods

make during a phone call, our facial expressions

like Facial Action Coding System (FACS) might

still manifest even when we cannot see eachnot be as efficient as fEMG measurements.

thatfacialactivities

causedbyother in CMC. Nowadays, in the information

On the other hand, discriminating among

society, the innate methods like facial expression

different emotions is an important application ofand computer mediated methods like emoticon

fEMG measurements, too (van Boxtel, 2010).

will be both functional in many occasions in our

Explanations for these connections between

communications. We assumed the emoticon use

elementary emotions and specific facial muscle

might have influence on facial expressions in aactions can be found in a study based on FACS

direct or indirect way. The other purpose of our(Cohn

study is to confirm whether CMC users frommeasurements to record specific emotional

different countries have different way of using

responses is feasible (Larsen et al, 2003).

et

al,

2006).

Thus,

using

fEMG

nonverbal methods. As the result, correlation

Another reason for using fEMG in CMC study

between emoticon use and facial expressions isis about the emoticon use in instant messaging

confirmed. However, we found more similarity(IM) and e-mail. According to Xu et al. (2007),

than difference in emotional expression between

emoticons can help form an impression of thecountries in CMC.

sender’s personality and emotional state. This
opinion is supported by study of gender

2. Material and methods

differences in impressions about emoticon use(Constantin et al, 2002). Furthermore, otherFacial electromyography (fEMG) has beenstudies suggest that the use of emoticons is not
2

CW6_A8350_02_D03.indd

2

2016/09/29

13:25:52

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016

only for enjoyment but can also provide users

China (16 females and 12 males). Experimentswith rich information (Huang et al, 2008;

were

Thompsen & Fougler, 1996). Therefore, forparticipants were all local residents; none had

people who are used to CMC, emoticons might

long-term overseas experience in the otherbe a reliable method for sending and reading

country. All the participants had had prioremotional cues. Possibly, the emotional activities

experience of CMC and emoticon use.

conducted

in

both

countries,

andappear in this process can be detected by fEMG

3.2. Stimuli and procedure

measurements.
Similar to facial expressions, emoticons can be

Participants were asked to perform thisclassified by the emotions they convey. However,experiment in pairs. Each member of a pair sat

emoticon has features fundamentally differ fromsimultaneously in different rooms, and eachfacial expression. According to recent studies,room was equipped with an Internet-accessible

emoticons with positive meanings like “happy,”

computer. Participants were asked to watch a“humorous,” or “enjoyable” are used more thanshort video (as stimuli for positive emotional

negative ones (Huffaker & Calvert, 2005; Derks

activities) with Japanese or Chinese subtitles andet al, 2008); even the expressions of teasing orthen have a chat about their experiences related

sarcasm carries positive meaning in most cases

to that video with the participant in the other(Wolf,

2000).

Therefore,

for

CMC

study,

room for 10 minutes. These videos were allon

positivecomedic from the situation comedy named

emotional activities seems necessary. In many

Friends described some simple stories which our

facial activity studies using fEMG, the orbicularis

participants might also have experienced in theiroculi and zygomaticus major were indices ofdaily life. From nine videos used in the pilot

positive emotions (Kimura et al, 2013; Lang et al,

study, we selected three because their fEMG data

1993; de Wied et al, 2006; de Wied et al, 2009).

was found to be relatively obvious and stable. The

The activities of the zygomaticus major are

video-chat procedure was repeated three times in

especially reported to be sensitive to positive

the experiment, and we changed the three videos’emotions (Witvliet & Vrana, 1995). We conducted

playback order each time to avoid bias caused bya small-scale experiment as a pilot study; collectedinterests or experiential differences.

conducting

research

focusedfEMG data during chats on IM and ensured

In the first video-chat procedure (Part 1), no

activities of the two muscles could be correctly

particular instruction about emoticon use wasrecorded.

given (the “no-instruction” condition), andparticipants could use emoticons as they wished.

3. Main study

For the second procedure (Part 2), participantsreceived instructions to use emoticons more than

3.1. Participants

three times (the “amplified-use instruction”

Our participants comprised 52 college students

condition); however, when or where to use them(age range of 21～25 with 0.693 as standard

was decided by the participants. In the third

deviation). 24 of them were from Japan (12procedure

females and 12 males), and 28 of them were from

instructions to inhibit emoticon use (the

(Part

3),

participantsreceived

3

CW6_A8350_02_D03.indd

3

2016/09/29

13:25:53

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG“inhibited-use instruction” condition). The fEMG

were with extremely complex meanings which

activities of these three parts were recorded bymight bring unpredictable influence to data

Mwatch type 101 (WADA Aircraft Technology)

analysis. Thus, we had to prevent the use of someand two pairs of Ag/AgCl surface electrodes (4methods and control the emoticons to ensure no

mm in diameter). Figure 1 showed an image of

strange or particularly complicated ones could be

the attached electrodes for each muscle. The

used.

electrodes were attached one pair each for

For the convenience of waveform observation

zygomaticus major (lower left in Figure 1) andand calculation, we transferred all fEMG data toorbicularis oculi (upper right in Figure 1). The

integrated EMG (iEMG) data (300 ms time

fEMG activity was amplified 8000 times andconstant). And to make our data comparable

filtered (50 Hz to 500 Hz). Data collection was

betweenconducted at a sampling rate of 1000 Hz. We alsotransferred into ％iEMG data, which wereused screen capture software to record all actions

proportional to an adequate baseline value

on computers to check every occurrence of(baseline ％iEMG value =100％). At some level,

emoticon use afterward.

thisindividuals,

standardization

all

also

iEMG

data

enabledwasdirectcomparison of the two muscles’ activities.
Because our experiment was basically conducted
on computers, there should have been lessdistribution and influence from low-frequency
artifacts. However, to ensure data’s comparability;
six samples showed unusually high values were
removed, leaving 22 from Japan (12 females and
10 males), and 24 from China (14 females and 10
males) as the total number of participants.

4. Data analysis and results

Figure 1. An Image of the Attached Surface
Electrodes for Each Muscle

4.1. Overview of fEMG data
We used LINE in Japan, and WECHAT in

All our participants used more than three times

China. We purposely used two different kinds ofin Part 2, and most of them used emoticons in

IM software because they were each widely used

Part 1. In general, more emoticons were used inin one country, but not in the other. We wanted toPart 2. Therefore, we believe that emoticon use

obtain data reflected situations close to CMC in

was well controlled. Beyond that, most emoticons

daily life. There were many kinds of emoticons(431 times in 485) had positive meanings such as

and other similar methods such as “Stamp” in“happy,” “exciting,” and “satisfaction,” etc. Other

LINE and “Biaoqing” in WECHAT. Theseemoticons used to show “sarcasm,” “confusion,”

methods were widely used by many people inand “apology” also showed some positive

both countries. However, not all of them could bemeanings.
No significant difference in ％iEMG data wasused from both countries, and some of them
4

CW6_A8350_02_D03.indd

4

2016/09/29

13:25:53

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016

found among general (undifferentiated as to Part

times that emoticon use received higher values

1, 2, or 3) videos or chats by group comparison.

than mean ％iEMG values; however, 229 times it

And after observation of the fEMG curve, we

received lower values. Nearly half of emoticon

noticed that stronger fluctuations appeared while

use indicated lower values than mean ％iEMG

watching videos. Comparative results of meanvalues.

％iEMG values revealed a significant difference

We also analyzed data of receiving emoticon

between general chats and videos (t = −2.390, df =side. When emoticons were received with

550, p<. 05).

messages, participants could see messages andIn the main study we found no significant

emoticons simultaneously even without obvious

correlation between the number of emoticons

eye movements. Thus, it was difficult to selectused and ％iEMG data from the zygomaticus

the moment when only emoticons were watched.

major (r = -0.05, p>. 05) or the orbicularis oculi (rHowever, we noticed that sometimes emoticons= 0.03, p>. 05). Neither did we find significant

were sent without messages (46 times in total).

difference in ％iEMG values of chats among

Under the circumstance, the effect caused byinstructions. Thus, we assume that it is not a

watching

simple mechanism that more emoticon use cause

calculated the mean ％iEMG values in time, i.e.,stronger facial muscle activities.

from the moment emoticons were received to theemoticonscanbetestified.

We

start of typing messages for reply. Except for a

4.2. The influence from emoticon use to facial

few times (3 times in total) that two muscles

expressions

showed different results, same tendency was

4.2.1. 
The effect caused by emoticons asshowed by two muscles in most of time. Theredirect stimuli

were 19 times that higher values than mean％iEMG values appeared; however, 24 times it

First, we have to verify the possibility thatemoticons themselves could be direct stimuli.

received lower values.

Emoticons are graphical icons with emotional

Based on the data showed above, we cannot

information; thus, it is possible that emotional

claim that as direct stimuli, emoticons promote

activity may be caused by watching them.

emotional activities when participants wereWe calculated the mean ％iEMG values inwatching them. Additionally, as mentioned in 3.1,

time, i.e., from the participants’ opening of the

we could not find significant correlation between

emoticon list to their click on an emoticon andthe number of emoticons used and ％iEMG data.

compared these data with the mean ％iEMG

Thus, emoticon stimuli are at least not thevalues of the corresponding chats. When the list

greatest factors related to facial muscle activities.

was opened, it blocked participant’s view ofreading messages. The ％iEMG data showed

4.2.2. The effect caused by video stimuli

information more related to the effect caused by

In main study, significant responses by the two

watching emoticons but not messages. Under themuscles appeared when watching videos. Thus,

circumstance, the effect of emoticons as direct

we can consider the video stimuli were functional.

stimuli could be testified. The data from two

We believe they could have stronger influence on

muscles showed the same result. There were 256

facial activates than emoticons as stimuli.
5

CW6_A8350_02_D03.indd

5

2016/09/29

13:25:53

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG

However, it is necessary to confirm whether the

Table 1 r-Values of Mean %iEMG in Every Part

influence from video stimuli continued to befunctional in chat parts.
Because videos and chats were on different
timelines, we could not directly compare ％iEMG
value in each part of the experiment. We usedmean ％iEMG value instead and calculated thesimple linear correlation of the mean ％iEMG
value between videos and chats. Table 1 presents
the results.
Statistical significance of correlation between
video and chat parts was found in almost all cases(r = 0.36 ～ 0.72). It emerged from both muscles

differ much among instructions. Therefore, it isand both sets (Japan and China) of participants.

highly possible that the differences in r-valus

Thus, we can presume that positive emotional

relates to the difference of communication

activities caused by stimuli did not disappear

methods.

after watching videos. These emotions and the

4.2.3. 
Difference in facial muscle activities

subsequent facial activities also continued toamong instructions

emerge from chats. Beyond that, this correlation
was lower in Part 3 than in Parts 1 and 2 in bothIn our experiment, participants could not see

countries. Higher correlation revealed that, ineach other. They could not conduct self-

Chats 1 and 2, the emotional activities ourpresentation or impression management by facial

participants had were closer to the ones directly

expressions. Thus, we can assume that if facialcaused by video stimuli. Thus, we can consider aactivities occurred, especially the obvious ones,

possibility that the emotions directly or indirectly

most of them were caused by their emotionalcaused by video stimuli, could be expressed

activities at those moments. As mentioned above,

more easily when emoticons were allowed to be

the emotional activities in chats were affected byused.

video stimuli on a certain level, and no significant

However, this was not the only indication wedifference in ％iEMG data was found among theobtained. We also noticed that although in mostthree videos. Thus, if there were no othercases r-values were higher in Part 2 than in Part

important factors existed, the facial muscle

1; the difference between Parts 1 and 2, and Part

activities should not show significant difference

3 was far more significant. As mentioned in 4.2.1,

among instructions. However, differences were

the number of emoticons used is not the main

found in the frequency of facial muscle activities.

reason for facial activities. Perhaps this relates

When performing calculations, we wanted tomore to the richness of nonverbal methods.

selectBecause using emoticons was the only way ofelectromyography curve, and then calculate theirmakingournumber and the time gaps between them. Most

experiment, the richness of information would

relatively obvious peaks can be confirmed bynonverbal

communicationinrelatively

obvious

peaks

from

the

6

CW6_A8350_02_D03.indd

6

2016/09/29

13:25:53

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016

observing the electromyography; however, toby definition. To avoid influence from rare cases,

select these peaks in a more explicit, objective,

we compared the result of processing with theand unified way, we did not use the observationelectromyography curve. Most selected peaks

way as a main method. Instead, we set a series ofwere obvious enough to be confirmed frommathematical definitions and programmed themobservation. At least, every obvious peak

to ensure the peaks that we wanted could be

confirmed from the electromyography curve wasautomatically selected. We assumed the ％iEMG

selected by this method. Therefore, we believe

value of n (ms) was a peak we wanted. We set thethat the peaks were well defined. Figure 2 is an

definition as follows:

example of ％iEMG curve in 5 s (％iEMG range(1) ％ iEMG (n) = max ％iEMG value in 1001 ms

of 100％～300％). We can confirm an obvious
peak at about 2.1 s from it.

from (n − 500) to (n + 500)(2) Max ％iEMG value appeared only once in

To analysis the frequency, we used the numberof peaks to calculate how many times peaks

1001 ms from (n − 500) to (n + 500)(3) ％ iEMG (n) > mean (％iEMG) value of theappeared in 1 minute (times/min value). As
Table 2 shows, the times/min values were higher

chat
(4) ％
 iEMG (n − 50) > ％iEMG (n − 100) >and closer to the values of video parts in Chats 1％iEMG (n − 150) > ％iEMG (n − 200) >…>and 2 than in Chat 3. To confirm influence from％iEMG (n − 400) > ％iEMG (n − 450) >instruction and country factors, we conducted％iEMG (n − 500)

analysis of variance (ANOVAs), finding the effect(5) ％iEMG (n + 50) >％iEMG (n + 100) >from instruction significant on the zygomaticus％iEMG (n + 150) > ％iEMG (n + 200) >…>major (F(2, 88) = 14.442, p<. 01) and the％iEMG (n + 400) > ％iEMG (n + 450) >orbicularis oculi (F(2, 88) = 6.768, p<. 01). No％iEMG (n + 500)

significant effect of country factor was found on

This definition does not seem strict enough;the zygomaticus major (F(1, 44) = 1.066, p>. 05)

however, it was sufficiently functional for ouror on the orbicularis oculi (F(1, 44) = 0.20, p>. 05).

purposes according to the results. Because weAfter multiple comparisons by Scheffe, significant

used ％iEMG data in this selection, most curves

differences between Chats 1 and 3 and between

were smooth, and only relatively dramatic muscle

Chats 2 and 3 were found in both muscles.

activities could cause obvious peaks. Thus, forHowever, no significant difference was found

any obvious peak to appear and disappear in as

between Chats 1 and 2.

short a time as under 1001 ms was nearly

As for time gaps in chats, these values were an

impossible. We also tested using longer oraverage of approximately 27 to 40 s. Significant

shorter time definitions, such as 501 ms and 2001

facial muscle activities appearing at this frequency

ms, but obtained no better results. Beyond that,are acceptable results. No extremely short time％iEMG data was accurate to six decimal places,

gap was found in each sample’s data. Thus, we

thus the appearance of a repeated value in a shortbelieve that data of the peaks selected from chatstime should be very rare. Although considered a

are reliable.

very rare case, if several peaks appear in 1001

The data above suggest that when emoticons

ms, part of the values we wanted can be excluded

could be used, the frequency of emotional
7

CW6_A8350_02_D03.indd

7

2016/09/29

13:25:53

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG

Figure 2. An Example of %iEMG Curve in 5 s
Table 2 Mean Value of Times/Min Value and Time Gaps

activities was higher and closer to the condition

by group comparison on zygomaticus major (t =of the videos. Thus, at least, it is highly possible

0.054, df = 44, p>. 05), nor on orbicularis oculi (t =that the instructions (whether emoticons could

0.115, df = 44, p>. 05). And none of the analyses

be used or not) have influence on emotional

we made in 4.2 showed any difference between

expressions, and the use of emoticons might be acountries. Therefore, it is highly possible thatkey factor caused these differences.

when strong emotional stimuli appear in CMC,
users’ facial activities show commonality on a

4.3. The commonality of emotional activities

certain level. Although there were differences in

between countries in CMC

timing and intensity of muscle activities among

4.3.1. Comparison of data from the two

individuals, none of them could show any patterncountries

varied between countries.

To confirm whether significant differences inSecondly, we compared data of emoticon use

emoticon use and facial activities exist between

between the two countries. Emoticon use is not athe two countries, we conducted calculation andphysiological method like facial expression.

comparison. However, we obtained no data

Thus,

showed such differences.

commonality in using them. However, we

thereshouldbenophysiologicalFirst, we analyzed data of facial activities fromobtained no data showed significant difference

both countries. We could not find significant

between the two countries. We confirmed 9 times

difference in ％iEMG data between two countriesof emoticon use on average from Japanese
8

CW6_A8350_02_D03.indd

8

2016/09/29

13:25:53

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016

samples, and 11.96 times from Chinese samples.

emotional activities in CMC, We need to know

Chinese participants used more emoticons on

the context in which emoticons were used. We

average; however, the numbers were very close.

confirmed the messages used before and afterNo significant difference in the numbers ofeach emoticon use, tried to discover whether

emoticon used could be found by grouppatterns existed in these messages.

comparison between the two countries (t = 1.382,

We found three patterns appeared many times

df = 44, p>. 05). We also conducted analysis about

in both countries. And as mentioned in 4.2.1,

the meaning of emoticons used in chats. We

there were also emoticons used alone without

confirmed 8.5 times of positive meaning on

messages. We consider it as an independent

average from Japanese samples, and 10.2 times

pattern, too. There were 402 times of emoticonfrom Chinese samples. No significant difference

use related to them. One of the patterns appeared

could be found by group comparison (t = 1.382, df

mostly= 44, p>. 05). The emoticons showed otherexperiences. Especially when they had a chatmeaning like “sarcasm” and “confusion” wereabout their particular experiences other people

also found in both countries. Thus, we consider

might not know. There were 195 times ofthat the commonality may not only exist in facial

emoticon use related to this pattern. Under the

expressions, but also in emoticon use.

circumstance, most emoticons were used to show

whenparticipants

sharedtheirpositive emotions; however, some of them were

4.3.2. The context in which emoticons werewith meanings of “puzzled”, “shy”, “cautious”, etc.

used

The second pattern was confirmed on the occasion

If we want to confirm the commonality ofthat participants had same opinions with each

Table 3 Patterns of Emoticon Use

9

CW6_A8350_02_D03.indd

9

2016/09/29

13:25:54

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG

other. Especially when they chatted “me, too”,

obtained lower values. In only 16 times, higher

emoticons were used very often. There were 82

values emerged when emoticons were used alone

times of emoticon use related to this pattern. It

without messages. In only 14 times, we obtained

added emotional information into the chats, andhigher values in other usages of emoticons. Data

made the conversation more dramatic than justof the occasion when receiving emoticons

saying approval. The third pattern was confirmed

showed a similar result, in 231 times (126 times

when participant made a joke in a brief reply. Allin pattern 1, 46 times in pattern 2, and 59 times in

the replies confirmed were under 8 words after

pattern 3), higher values emerged when the firsttranslated into English. Emoticons with meanings

three patterns appeared. In 125 times, weof “exciting” and “sarcasm” were found manyobtained lower values. In only 19 times, higher

times in those replies. There were 79 times ofvalues emerged when emoticons were used

emoticon use related to this pattern. The

without messages. In only 9 times, we obtained

emoticons used alone without messages werehigher values in other usages of emoticons.

confirmed 46 times. We confirmed more

Because the result of comparison differed very

appearance of these four patterns in Chinese

little between two muscles, in order to give asamples. However, if we consider the number ofmore accessible image, data showed above was

Chinese participants was bigger than Japanese,

calculated as the average times of data from two

the appearances of these patterns in two countries

muscles. The result revealed that stronger facial

are possibly closer than data showed. Thus, we

activities emerged in first three patterns. It isbelieve participants from the two countries havetestified by group comparison of the ％iEMG

these patterns in common. There were also somevalues on zygomaticus major between the firstother usages of emoticons (83 times in total)

three patterns and last two ones when sending

could not be categorized or only appeared a few(Japan: t = 30.563, df = 173.618, p<. 05; China: t =times, like emoticons used in greetings. The

37.423, df = 245.727, p<. 05) and when receiving

overall result was showed in Table 3 (％iEMG

messages (Japan: t = 33.563, df = 129.875, p<. 05;

range of 111％～517％).

China: t = 31.696, df = 282.590, p<. 05). The

We calculated the mean ％iEMG values indifference is testified on orbicularis oculi, tooeach time of sending (time from the beginning of(when sending, Japan: t = 34.587, df = 165.073,

typing messages to the click on the sendingp<. 05; China: t = 28.207, df = 280.399, p<. 05;

button) and receiving emoticons (time from thewhen receiving, Japan: t = 11.385, df = 196, p<. 05;

moment when messages were received to the

China: t = 38.264, df = 227.721, p<. 05). We also

start of typing messages for reply) of thesecompared the time data of the first three patterns

patterns. These values were compared with mean

with the frequency data (time when peaks％iEMG values of the chats in which the patterns

appeared) mentioned in 4.2.3. We found at leastappeared. As the result of comparison in situation

one time of peak appearance in 117 times of theseof sending emoticons, in 255 times (157 times inpatterns. Thus, we believe relatively strong facial

pattern 1, 56 times in pattern 2 and 42 times in

activities did emerge in these patterns.

pattern 3), higher values emerged when the first

Because these patterns (especially the first

three patterns appeared. In 101 times, we

three patterns) were found in chats of both
10

CW6_A8350_02_D03.indd

10

2016/09/29

13:25:54

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016

countries, to some extent, it explained why weoculi (t = 0.087, df = 22, p>. 05). Data above

could not obtain any data showing significant

indicated no evidence of influence from fatigue ordifference between countries. In the three

concentration. Neither could we find evidence

patterns, more obvious emotional activities wereshowing such possibilities by electromyography

found from data of both countries. Thus, thecurves observation or getting such report fromcontext in which emotional activities improved

our participants. Although other variables might

showed commonality at some level.

exist, our data still showed much evidence
supporting our hypothesis.

4.4. Result of the additional experiment
5. Discussion

To confirm possible influence from factors
varying with time axis change, like fatigue orconcentration, a small-scale experiment was

The analyses above lead us to a high probability

conducted as an adjunct. It was basically thethat richness of nonverbal methods is a factor

same as the main experiment; only the order ofgreatly influencing emotional activities. Although

no-instruction and inhibited-use instruction was

we found many differences among instructions,swapped. Correlation of ％iEMG data between

most were between Chat 3 and Chats 1 and 2.

video and chat showed lower values under

The difference we found between Chats 1 and 2

inhibited-use instruction on both muscles. To be

was very small. This result suggests that the

specific, on zygomaticus major, the r-values were

number of emoticons used were not strong

3.30 under inhibited-use instruction, 4.53 andenough to be an important factor in emotional

5.32 when emoticons were allowed to be used.

activities.

And on orbicularis oculi, they were 5.25 and 4.87

Underno-instructionandamplified-use

when emoticons were allowed to be used;

instruction, ％iEMG data showed a higher

significance of correlation could not be foundcorrelation of facial muscle activities between

under inhibited-use instruction (r = 3.58, p>. 05).

video and chat parts. We believe the emotional

Data of frequency showed lower values underactivities caused by video stimuli did not simply

inhibited-use

Significant

disappear in chat parts. And they could be

difference of times/min value was found by

expressed more easily when emoticons weregroup comparison between the inhibited-use

allowed to be used (4.2.2). This hypothesis isconstruction and no-instruction on zygomaticus

based on not only our results but also previous

major (t = 1.933, df = 22, p<. 05); and on orbicularis

studies. According to Walther et al. (2001), whenoculi (t = 2.126, df = 22, p<. 05). It was also foundemoticons and text were used together, positive

between the inhibited-use construction andemotion could be communicated more easily, andamplified-use instruction on zygomaticus major

CMC interpretation improved. Other research(t = 2.692, df = 22, p<. 05); and on orbicularis

provided detailed information for us. Affective

instruction,too.

oculi (t = 2.571, df = 22, p<. 05). No Significant

evaluation of emoticons seems to be an automaticdifference was found between no-instruction andprocess that has been analyzed by Event Related

amplified-use instruction, not on zygomaticus

Potentials measurements (Comesañaa et al,

major (t = 0.103, df = 22, p>. 05) nor on orbicularis2013). In contrast, lack of a nonverbal method
11

CW6_A8350_02_D03.indd

11

2016/09/29

13:25:54

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG

caused a more complicated situation whenemoticon is more efficient, and emotional signal

emoticon use was inhibited. Words and messages

recognition is eased. When emoticons could be

can also convey emotional information, however,used, participants’ emotions could be expressedin a verbal way. When participants could not use

easily, and then new emotions could appear one

emoticons in Chat 3, part of the emotional

after another. However, during the time when

expressions had to be verbally transferred. Words

these emotions did not appear or when they did

and messages include complicated information.

not appear obviously enough, the emotional

This complication will cause difficulty inactivities emerged might not be stronger than

discriminating emotional information from other

under inhibited-use instruction. We believe it istypes. As mentioned in previous studies, the lack

the reason why the ％iEMG value was not

on nonverbal cues makes the miscommunicationstronger in average than under inhibited-use

of emotion more likely (Perry, 2010; Byron,

instruction. It is not a simple mechanism that2008). Thus, compared with using verbal andmore emoticon use caused stronger facial

nonverbal methods together, this process ofactivities. We believe it is more related to thetransferring nonverbal signals into verbal ones

efficiency of affective delivery and the process of

could be less efficient. We believe when messages

transformation mentioned above. Of course, notcould be used with a nonverbal method

all emotional activities were transformed into(emoticons), a more direct, less processed way of

verbal or nonverbal information. This depends onemotional

the participants’ judgments. In our experiment,

expression

appearedinourexperiment.

because strong emotional stimuli (video stimuli)

During videos, our participants had no need toexisted, the subjective demand of emotional

exchange information with others; thus, there

expression should be enhanced. Thus, thewas a natural condition with no process ofchance of using emoticons became a moretransformation from emotions to words, messages,

valuable and effective way of delivering emotionalemoticons, or any other information. Under the

information. In contrast, under inhibited-use

circumstances, emotional expressions in facial

instruction, the inhibitory effects were also

activities were direct and efficient. We found thatenhanced and affected participants stronger thanwhen emoticons and messages could be usedin other situations. Under the circumstances, thetogether, the frequency of facial muscle activities

complementary effect of verbal and nonverbal

was higher and closer to the frequency of thosemethods disappeared; low efficiency of emotional

during videos (4.2.3). We also mentioned in 4.1

signal recognition could influence their judgments.

that no significant difference in ％iEMG data wasIf it was sensed by participants, part of thefound among the three instructions by group

emotional expressions might even be canceled.

comparison. Therefore, the differences caused

We believe this is also one of the reasons why

by emoticon use exist in frequency, but cannot be

difference

simply reflected by the comparison of mean

instructions.

infrequency

appearedamong％iEMG value in general. Compared with words

When our participants used emoticons inand messages, emoticons are closer to nonverbal

context with some patterns, stronger emotional

methods; thus, transformation from emotion toactivities appeared much more than under other
12

CW6_A8350_02_D03.indd

12

2016/09/29

13:25:54

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016

circumstances (4.3.2). Because this tendency

methods could be used together, the richness of

was observed from both sending and receiving

information improved. We believe it brought

occasions, we believe the improvement ofefficiency and smoothness to affective delivery.

affective delivery was a mutual effect on both

Furthermore, we obtained no data indicatingsides. The ％iEMG values of emoticon use in thesignificant differences of facial activities or

three patterns were recalculated by using theemoticon use between countries. We even foundmethod introduced in 4.2.1 (mean ％iEMG

certain patterns of emoticon use from twovalues in time from the participants’ opening ofcountries in common. Although emoticon use is

the emoticon list to their click on an emoticon).

not an innate method of nonverbal communication,

In 191 times, emoticon use received higher

it has functional similarity to facial expressions

values than mean ％iEMG values; in 165 times it(Comesañaa et al, 2013). And some social factors—

received lower values. Based on this result, wee.g., education level—do not significantly influence

still cannot claim that as direct stimuli, emoticons

facial activities (Cacioppo et al, 1992). We believe

promote emotional activities when participantsthat under strong emotional stimuli, emoticon use

were watching them. However, when emoticons

can show features similar to facial activities.

and messages were used together in context

6. Conclusions

mentioned above, evident improvement ofemotional activities was confirmed. We selected
some chats (150 times) with messages ofAs the conclusion of this study, we consider it

experience sharing, agreement expressing andis highly possible that richness of nonverbal

jokes randomly from Chat 3. Then the mean

methods is a key factor affecting emotional％iEMG values during sending and receiving

activities in CMC. We also believe that in certain

messages were calculated with the methods

situations emoticon use and facial activities inmentioned in 4.3.2. We compared these values

CMC can show features in common between

with the ones of Chats 1 and 2 introduced in 4.3.2.

different countries on a certain level. However,Significant difference of ％iEMG values on

because this study is based on samples ofzygomaticus major was found in sending

Japanese and Chinese college students, results

occasions (t = 11.198, df = 498.346, p<. 05) andmight not generalize to users’ conditions in otherreceiving occasions (t = 13.232, df = 463.282,

counties. It is a limitation of this study. As

p<. 05). Significant difference were found on

mentioned above, some other nonverbal methods

orbicularis oculi in sending occasions (t = 9.859,

such as “Stamp” in LINE could not be included in

df = 482.693, p<. 05), but not in receiving occasions

this study. It is not because they are not important.

(t = 1.032, df = 504, p>. 05). However, the mean

Some of them are widely used and may have theirvalue of data in Chats 1 and 2 (2.78) was found

own features as nonverbal methods. We believe it

higher than in Chats 3 (2.56) when receiving.

is necessary to find a more appropriate way to doThus, possibly, the obvious improvement ofresearch on these methods in future. We also

affective delivery mentioned above could not berealized that not all people can be influenced bycaused by using emoticons or messages

emoticon use on the same level, some people

independently. When verbal and nonverbal

who are not used to CMC or using emoticons
13

CW6_A8350_02_D03.indd

13

2016/09/29

13:25:54

Emotional Activities in Computer-mediated Communication: Facial Electromyography Response and Emoticon Use during Instant MessagingTANG

may show different result. We believe theof masked affective priming with emoticons.

discussion of more detailed information likeComputers in Human Behavior, Vol. 29(3), 588-595.
・Constantin, C., Kalyanaraman, S., Stavrositu, C., &above is necessary in future study.
In our study, we used fEMG measurements toWagoner, N. (2002). Impression Formation 11threcord and analyze emotional activities emerged

Pacific-Asia Conference on Information Systems

in CMC. It made the use of two nonverbal

Effects in Moderated Chatrooms: An Experimental

methods intercomparable and with calculating

Study of Gender Differences. Paper presented at thepracticability.

88th annual meeting of the National Communication

Detailed

comparisons

wereconducted among different instructions, patterns

Association, New Orleans, LA.

and specific occasions. We believe this method

・Derks, D., Bos, A. E. R., & von Grumbkow, J. (2008).

can provide accuracy and logicality to CMC

Emoticons in computer-mediated communication:

study. As mentioned in the beginning, in theSocial motives and social context. CyberPsychology &information society, the innate methods andBehavior, Vol. 11(1), 99-101.

computer mediated ones will be both functional

・Derks, D., Fischer, A. H., & Bos, A. E. R. (2008). The

in many occasions. People are getting used torole of emotion in computer-mediated communication:fusing and combining these methods so that they

A review. Computers in Human Behavior, Vol. 24(3),can take full advantage of using them. We believe766-785.

our study provided a feasible method of analyzing・Dimberg, U., Thunberg, M. & Elmehed, K. (2000).

the interaction between these methods. And we

Unconscious Facial Reactions to Emotional Facial

wish our works can inspire more research in thisExpressions. Psychological Science, Vol. 11(1), 86-9.
・de Wied, M., Boxtel, A. V., Zaalberg, R., Goudena,

area.

P.P., & Matthys, W. (2006). Facial EMG responses

References

to dynamic emotional facial expressions in boys

・Byron, K. (2008). Carrying too heavy a load? The

with disruptive behavior disorders. Journal ofcommunication and miscommunication of emotion

Psychiatric Research, Vol. 40, 112-121.
・de Wied, M., Boxtel, A. V., Posthumus, J. .A.,by email. Academy of Management Review, Vol.
33(2), 309-327.

Goudena, P. P., Matthys, W. (2009) Facial EMG

・Cacioppo, J. T., Bush, L., Tassinary, L. G. (1992).

and heart rate responses to emotion-inducing film

Microexpressive Facial Actions as a Function ofclips in boys with disruptive behavior disorders.

Affective Stimuli: Replication and Extension,

Psychophysiology, Vol. 46(5), 996-1004.
・Hazlett, R. L. & Hazlett, S. Y. (1999). Emotional

Personality and Social Psychology Bulletin, Vol.
18(5), 515-526.

response to television commercials: Facial EMG

・Cohn, J., Ambadar, Z. & Ekman, P. (2006). Observer-

vs. self-report. Journal of Advertising Research, Vol

based measurement of facial expression with the39(2), 7-23.
・Huang, A. H., Yen, D. C., & Zhang, X. (2008).

Facial Action Coding System. The handbook of
emotionelicitation

andassessment,

OxfordExploring the effects of emoticons. Information &University Press Series in Affective Science.

Management, 45(7), 466-473.

・Comesañaa, M., Soaresa, A. P., Pereab, M., Piñeirod,・Huffaker, D. A. & Calvert, S. L. (2005). Gender,

A. P., Fragac, I., Pinheiroe, A. (2013). ERP correlates

identity, and language use in teenage blogs. Journal

14

CW6_A8350_02_D03.indd

14

2016/09/29

13:25:54

Journal of Socio-Informatics Vol. 9 No. 1 Sep. 2016of Computer-Mediated Communication, Vol. 10(2).

pictographs and quoting on flaming in electronic・Kimura, K., Ehara, S., Katayama, J. (2013). The

mail. Computers in Human Behavior, Vol. 12, pp.

subjective experience of positive emotions and225-243.
・van Boxtel, A. (2010). Facial EMG as a tool for

facial muscle activity. Japanese ournal of Researchon Emotions, Vol. 20(3), 96-104.

inferring affective states. In Proceedings

・Lang, P. J., Greenwald, M. K., Bradley, M. M. &ofmeasuring behavior, 104-108. Wageningen: Noldus

Hamm, A. O. (1993). Looking at pictures:

Information Technology.
・Walther, J. B., & D’Addario, K. P. (2001). The

Affective, facial, visceral, and behavioral reactions.
Psychophysiology, Vol. 30, 261-273.

impacts of emoticons on message interpretation in・Larsen, J.T., Norris, C.J., Cacioppo, J.T. (2003).

computer-mediated communication. Social Science

Effects of positive and negative affect on

Computer Review, Vol. 19(3), 324-347.

electromyographic activity over zygomaticus major・W itvliet, C. V., & Vrana, S. R. (1995).

and corrugator supercilii. Psychophysiology, Vol.

Psychophysiological responses as indices of40(5): 776-85.

af fective dimensions. Psychophysiology, Vol.

・Mahlke, S. & Minge, M. (2006). Emotions and32, 436-443.

EMG measures of facial muscles in interactive・Wolf, A. (2000). Emotional expression online:

contexts. Cognition and Emotion, Vol. 6, 169-200.

Gender differences in emoticon use. CyberPsychology

・Perry, M. (2010). Face to face versus computer-& Behavior, Vol. 3(5), 827-33.
・Xu, L. L., Yi, C., & Xu, Y. J. (2007). Emotional

mediated communication: couples satisfaction andexperience

acrossconditions.

Universityofexpression online: The impact of task, relationship

Kentucky Master’s Theses. Paper 66.

and personality perception on emoticon usage in・Thompsen, P. A., & Fougler, D. A. (1996). Effects of

instant messenger. PACIS 2007 Proceedings. 79.

15

CW6_A8350_02_D03.indd

15

2016/09/29

13:25:54