# The effect of subliminal facial expression on perspective adoption during
## language comprehension in Japanese

\author{
小波津豪 ${ }^{1}$ ，赤嶺奨 ${ }^{2}$, 大城彩佳 ${ }^{3}$ ，新国佳祐 ${ }^{4}$ ，里麻奈美 ${ }^{3}$ \\ ${ }^{1}$ University of the Basque Country, ${ }^{2}$ Max Planck Institute for Psycholinguistics, ${ }^{3}$ 沖䋲国際大学, ${ }^{4}$ 新潟青陵大学 \\ tkohatsu001@ikasle.ehu.eus
}

\begin{abstract}
This study explores the link between the perception of positive/negative facial expressions and the broadening/narrowing of cognitive processing by examining whether brief exposure to facial expressions affects participants' tendency to adopt an agent or observer perspective when comprehending null-subject sentences in Japanese. We found no influence of positive facial expressions on perspective adoption, supporting the hypothesis that perceiving positive emotions broadens cognition. We also found that negative facial expressions facilitated the adoption of an actor's perspective, supporting the hypothesis that perceiving negative emotions narrows cognition. The study discusses the effects of subliminal facial priming on cognitive processing and subsequent perspective adoption during language comprehension.
\end{abstract}

## 1 Introduction

During the processing of a sentence, language comprehenders mentally represent the event described in the sentence from a particular perspective (Brunyé, et al., 2009; Sato \& Bergen, 2013). For example, Brunyé et al. (2009) showed that comprehending the pronouns you or $I$ in a sentence (e.g., you are / I am slicing the tomato) primed an actor's perspective (i.e., as if the comprehender is conducting the action), while the pronoun he (e.g., he is slicing the tomato) primed an observer's perspective (i.e., as if the comprehender is observing another's action). In line with Brunyé, et al.'s results, Sato and Bergen (2013) found that no perspective adoption was necessary when agent/observer were not linguistically specified, as in Japanese nullsubject sentences. Their participants saw three sentences: a context sentence; a sentence describing an event in


which the grammatical subject was overtly stated (e.g., You are / He is making an apple pie); and a target sentence with one of three types of grammatical subject (e.g., You are / He is / $\phi$ cutting an apple). They then saw an image of the event depicted from either the actor's or the observer's perspective, and judged whether the image matched what they had read. A reaction time analysis showed that while second-person pronouns primed actor perspective and third-person pronouns primed observer perspective, null-subject sentences led to no significant difference in reaction times to the actor's and observer's perspective images. Interestingly, Niikuni, et al. (2021) found that people with a high sense of agency ${ }^{i}$ (SoA) showed no preference in perspective adoption, consistent with Sato and Bergen's results, but that people with low SoA showed a preference for the observer's perspective ${ }^{\mathrm{ii}}$. Taken together, these studies suggest the possibility that perspective adoption during language comprehension is affected by not only by linguistic cues but also by psychological factors, such as SoA.

Much research on the interaction between emotion and cognitive processing has stated that positive emotions broaden thought, action, and processing while negative emotions narrow them down (e.g., The broaden-andbuild theory of positive emotions in Fredrickson, 2001; Fredrickson \& Branigan, 2005; Schwarz, 2002). This has some empirical support from research on the distribution of attention to a visual field (Fujimura \& Suzuki, 2010) and language processing (Hernandez-Guitierrez et al., 2022; Yano et al., 2018). For instance, Fujimura and Suzuki (2010) found higher accuracy rates on sad-face recognition when the sad face appeared in the central

ii SoA was evaluated on Asai et al.'s (2009) "SoA scale," where a high score indicates unstable SoA and a low score, stable SoA.
visual field rather than in a peripheral field, although they found no such difference for happy-face recognition. Moreover, Yano and Koizumi (2018) found the larger electric activities in a happy mood group for semantic reversal anomalies which required multipleprocessing (world knowledge, syntactic, and semantic processing) compared to a sad mood group.

## 2 The present research

This study uses a sentence-picture verification task with visual priming to investigate the effects of emotional facial expressions on perspective adoption during the comprehension of null-subject sentences in Japanese. It builds on the findings of the studies reviewed above, which all suggest the possibility that observing another person's facial expressions can interact with a language comprehender's perspective adoption. It also draws on social psychology research showing that social information perceived below the level of conscious awareness may activate knowledge that shapes subsequence behavior or various types of decision-making (e.g., Winkielman \& Berridge, 2004; Ferguson and Bargh, 2004, for review). Thus, a brief exposure to another's facial expressions may influence our cognitive processing. More specifically, seeing a positive face would broaden subsequent linguistic processing while seeing a negative face would narrow it. In fact, Akamine, et al., (2022) found that subliminal priming with positive/negative facial expressions facilitated the processing of Japanese null-subject sentences with up/down vertical spatial meanings (e.g., Hashigo-o nobotteimasu '(Somebody is) climbing the ladder'; Hashigo-o kudatteimasu '(Somebody is) going down the ladder'). The current study looks for a similar subliminal priming effect of emotional facial expressions on perspective adoption.

We made two predictions based on the studies of Fredrickson (2001) and Schwarz (2002), which suggest a link between positive/negative emotions and the broadening/narrowing of processing. First, we predicted that a brief exposure to a positive facial expression will make comprehenders engage in heuristic processing, which will enable them to adopt both actor's and observer's perspectives when comprehending Japanese null-subject sentences. Second, we predict that a negative facial expression will make comprehenders engage in analytic processing, which will hinder them from adopting both perspectives, and instead lead them to take one or the other perspective.

## 3 Experiment

## 3.1 Participants

Forty-five native speakers of Japanese, all students at Okinawa International University (Female $=30$, Mage $=$ 21.6, $\mathrm{SD}=1.62$ ), participated in the study.

## 3. 2 Materials

Prime Pictures (Faces): For each trial, participants were first briefly primed by seeing an image of a male's or a female's face $(N=6)$ with either a smile, a neutral expression or a frown for $17 \mathrm{~ms}$ (ATR-promotions, 2006).

Sentence Materials: The sentence materials comprise 72 transitive, past-tense, null-subject Japanese sentences: 36 targets and 36 fillers. Of each set of 36 sentences, 18 describe positive events, as in (1), while the other 18 describe negative events, as in (2).

(1) Target positive sentence

Houseki-o migai-ta tokorodesu.

jewelry-ACC polish-past just

" $\phi$ just polished jewelry"

(2) Target negative sentence Fuusen-o wat-ta tokorodesu. balloon-ACC pop-past just " $\phi$ just popped a balloon"

Target Pictures (Events): The picture materials were drawn specifically for this experiment. For each target and filler sentence, a pair of pictures was created, one from an actor's perspective and another from an observer's perspective. For the actor's perspective $(N=36)$, the pictures showed actions depicted from the participant's viewpoint; that is, as if the participant were conducting the action (Fig. 1a). For the observer perspective $(N=36)$, the pictured actions were depicted from the participant's viewpoint, as if the participant were watching another's action (Fig. 1b).

Figure 1. Popping a balloon from (a) actor's perspective on the left and (b) observer's perspective on the right.

## 3. 3 Procedure

The experiment was conducted with each participant individually in a quiet room, using Psytoolkit (Stoet, 2010, 2017). Participants completed nine practice trials, repeating them if necessary until they reached $100 \%$ accuracy. In the main experiment, each trial started with a fixation cross for $3000 \mathrm{~ms}$, followed by the prime image, which was either the smile, the neutral, or the frown face, presented for $17 \mathrm{~ms}$; the prime was followed by an image of a puzzle for $183 \mathrm{~ms}$. To ensure that participants saw the brief representation of a facial prime, they were told not to blink until they saw the puzzle image. Next, a nullsubject sentence was presented for $3000 \mathrm{~ms}$. The sentence was followed by a fixation cross for $1500 \mathrm{~ms}$, which was succeeded by the picture depicting the sentence action from either the actor's or the observer's perspective. Participants judged whether the picture correctly described the prior sentence by pressing a key on the keyboard: the F key for incorrect or the $\mathrm{K}$ key for correct, having been instructed to respond as quickly and accurately as possible (Fig. 2). As soon as the participant pressed one of the keys for the response, the next trial began.

Figure 2. A trial for an actor's perspective in the smile condition.

## 3. 4 Results

Using the lme4 (Bates, et al., 2015) and the lmerTest (Kuznetsova, et al., 2017) packages in the R programming language ( $\mathrm{R}$ Core Team, 2019), linear mixed-effects model analyses with participants and items as random factors (Baayen, et al., 2008) were conducted. The dependent variable was reaction times (RTs) to the target pictures. Face (frown/neutral/smile) and Perspective (observer/actor) were included as fixed effects with interactions between the factors. The accuracy of the response was included as an additional fixed factor (Accuracy) in the model, without interactions with Face and Perspective.

One participant's data were excluded because the accuracy rate for the target trials was $<75 \%(72.2 \%)$; all the other participants showed an accuracy rate of $>83 \%$. In addition, one trial in which the RT was $>7000 \mathrm{~ms}$ was excluded as an outlier; all remaining RTs were $<5000 \mathrm{~ms}$. Furthermore, for each participant, trials with RTs more than 2.5 SDs away from that individual's mean RT for target trials were excluded. In total, $5.4 \%$ of the target data were excluded from the analysis.

In the mixed model analyses, Face conditions (frown $/$ neutral $/$ smile) were treatment-coded with the neutral condition as the reference level, and Perspective conditions (observer/subjective) were deviation-coded.

Figure 3 shows the mean RTs for the target pictures. The analysis revealed no significant main effect of Perspective $(\beta=-55.4, \mathrm{SE}=66.3, \mathrm{t}=-1.44, \mathrm{p}=.15)$ or of Smile-Face $(\beta=-24.7, \mathrm{SE}=24.1, \mathrm{t}=1.03, \mathrm{p}=.30)$, and no significant interaction between Smile-Face and Perspective $(\beta=32.9, \mathrm{SE}=48.2, \mathrm{t}=0.68, \mathrm{p}=.50)$. In contrast, the main effect of Frown-Face was significant ( $\beta$ $=-60.7, \mathrm{SE}=24.2, \mathrm{t}=-2.51, \mathrm{p}<.05)$, and there was a significant interaction effect between Frown-Face and Perspective $(\beta=103.0, S E=48.4, t=2.13, p<.05)$.

Figure 3. Mean RTs for each experimental condition. Error bars represent the standard error of the mean.

In addition to the RT analysis, we conducted a logistic mixed-effect model analysis for the accuracy of the responses. The analysis revealed no significant main effect of Face or Perspective, and no interaction between them $(p s>.20)$. Table 1 shows the mean accuracy rate for each Face and Perspective conditions.

Table 1: Mean accuracy rate for each experimental condition.

## 4 Discussion

Based on the broaden-and-build theory of positive emotions (Fredrickson, 2001), this study investigated the effects of subliminal facial expressions on perspective adoption during language comprehension in Japanese. In line with Sato and Bergen (2013), the results showed that no particular perspective was adopted in Japanese nullsubject sentences. This finding bears out our first prediction that perceiving positive facial positive facial expressions broadens cognition, allowing comprehenders to adopt either the actor or the observer perspective. We also found that brief exposure to negative facial expressions did influence perspective adoption. Specifically, the Frown-Face $\times$ Perspective interaction was significant in the RT analysis, indicating that the (non-significant) tendency to respond faster to observer- perspective pictures than to actor-perspective ones in the neutral-face condition is reversed in the frown-face condition (see Figure 3). Thus, we can conclude that the brief exposure to negative faces encouraged the participants to adopt an actor perspective rather than an observer perspective when comprehending null-subject sentences, which supports our second prediction. This finding suggests that a negative face might narrow cognitive processing, preventing comprehenders from taking both perspectives but influencing them to take one particular perspective, namely, the actor's perspective.

Furthermore, the main effect of the Frown-Face prime was significant, indicating that the RTs were significantly faster in the negative-face condition than in the neutral condition. However, the finding of no significant main effect of the Smile-Face prime indicates no significant difference in RTs between the positive-face and neutral conditions. The rapid responses in the negative-face condition might possibly be due to a higher arousal level driven by the perception of a negative-face expression compared to neutral and positive-facial expressions (Robinson, et al., 2004).

## Acknow ledgment

This study is supported by KAKENHI \#19H01263

(PI: Manami Sato).

## References

Akamine, S., Kohatsu, T., Niikuni, K., Schafer, A. J., \& Sato, M. (2022). Emotions in language processing:

Affective priming in embodied cognition. In proceedings of the 39th Annual Meeting of Japanese Cognitive Science Society, 326-332.

Asai, T., Takano, K., Sugimori, E., \& Tanno, Y. (2009). Development of the sense of agency scale and its factor structure. Japanese Journal of Psychology, 80, 414-421.

Baayen, R. H., Davidson, D. J., \& Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390-412.

Brunyé, T. T., Ditman, T., Mahoney, C. R., Augustyn, J. S., \& Taylor, H. A. (2009). When you and I share perspectives: Pronouns modulate perspective taking
during narrative comprehension. Psychological Science, 20(1), 27-32.

Ferguson, M. J., \& Bargh, J. A. (2004). How social perception can automatically influence behavior. Trends in Cognitive Sciences, 8(1), 33-39. https://doi.org/10.1016/j.tics.2003.11.004

Fredrickson, B. L. (2001). The role of positive emotions in positive psychology: The broaden-and-build theory of positive emotions. American Psychologist, 56(3), 218-226.

Fredrickson, B. L., \& Branigan, C. (2005). Positive emotions broaden the scope of attention and thoughtaction repertoires. Cognition \& Emotion, 19(3), 313332.

Fujimura, T., \& Suzuki, N. (2010). Recognition of dynamic facial expressions in peripheral and central vision. The Japanese Journal of Psychology, 81(4), 348-355.

Kuznetsova, A., Brockhoff, P. B., \& Christensen, R. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13). https://doi.org/10.18637/jss.v082.i13

Moore, J. W. (2016). What is the sense of agency and why does it matter? Frontiers in Psychology, 7, 1272. https://doi.org/10.3389/fpsyg.2016.01272

Niikuni, K., Sato, M., \& Muramoto, T. (2021) Individual differences in sense of agency and perspective adoption in comprehending Japanese null-subject sentences. The Japanese Journal of Psychology, 92(2), 89-99.

R Core Team (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www. R-project.org/ (February 6, 2020)

Robinson, M. D., Storbeck, J., Meier, B. P., \& Kirkeby, B. S. (2004). Watch out! That could be dangerous: Valence-arousal interactions in evaluative processing. Personality and Social Psychology Bulletin, 30(11), 1472-1484.

Sato, M., \& Bergen, B. K. (2013). The case of the missing pronouns: Does mentally simulated perspective play a functional role in the comprehension of person? Cognition, 127(3), 361374.

Schwarz, N. (2002). Situated cognition and the wisdom of feelings: Cognitive tuning. In L. F. Barrett \& P. Salovey (Eds.), The wisdom in feeling: Psychological processes in emotional intelligence (pp. 144-166). The Guilford Press.

Stoet, G. (2010). PsyToolkit—A software package for programming psychological experiments using Linux. Behavior Research Methods, 42(4), 10961104. https://doi.org/10.1016/j.cognition.2004.04.003

Stoet, G. (2017). PsyToolkit: A novel web-based method for running online questionnaires and reaction-time experiments. Teaching of Psychology, 44(1), 24-31.

Winkielman, P., \& Berridge, K. C. (2004). Unconscious emotion. Current Directions in Psychological Science, 13(3), 120-123.

Yano, M., Suzuki, Y., \& Koizumi, M. (2018). The effect of emotional state on the processing of morphosyntactic and semantic reversal anomalies in Japanese: Evidence from event-related brain potentials. Journal of Psycholinguistic Research, 47(1), 261-277.