<figure><img src="image_1.png" /></figure>

HAL
open science

# The Effect of Expression Geometry and Facial Identity on the Expression Aftereffect

Miao Song, Qian Qian, Shinomori Keizo

<figure><img src="image_2.png" /></figure>

## To cite this version:

Miao Song, Qian Qian, Shinomori Keizo. The Effect of Expression Geometry and Facial Identity on the Expression Aftereffect. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.124-129, 10.1007/978-3-319-68121-4_13. hal-01820898

HAL Id: hal-01820898

https://inria.hal.science/hal-01820898

Submitted on 22 Jun 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The effect of Expression Geometry and Facial Identity on the Expression Aftereffect

Miao Song<fnref n="1" />*, Qian Qian<fnref n="2" />, and Shinomori Keizo<fnref n="3" />

**Abstract.** Few studies have systematically examined the effect of expression geometry and facial identity on this cross-identity expression aftereffect. This issue is, however, critical for understanding the nature of expression adaptation. We measured expression aftereffects using a cross-identity/cross-expression geometry factorial design. The results show that expression aftereffect would reduce if adaptor and tests have different identities or expression geometries, and that identity-independent expression aftereffect is relatively more robust to variance in expression geometry in comparison to identity-dependent expression aftereffect. Based on these results, we discussed their psychophysical and physiological implication for understanding the nature of identity-dependent and identity-independent expression representations.

**Keywords:** Facial Expression, Visual Adaptation, Neural Representation.

## 1 Introduction

Adaptation is a universal phenomenon in the human visual system, which refers to that the previous sensory experiences would affect the subsequent perception and response properties of visual system [1]. For instance, an observer adapts to a smiling face for a minute, and then looks at a middle expression within a series of ambiguous expression images morphing between smiling and angry expressions, this middle expression is tended to be perceived as an angry expression. This illusion is called expression aftereffect induced by the visual adaptation [2]. As visual adaptation can isolate and/or temporarily reduce the contribution of specific neural populations, visual adaptation is a time-honored tool for researchers to investigate the neural representation of human sensory system.

Previous study has found expression aftereffect in different person condition was weaker than that in same person condition, which indicated that there are two different neural representations in facial expression system, i.e., identity-dependent neural

<footnote n="1">¹ Shanghai Maritime University, 1550 Haigang Ave, Shanghai, 201306 China
miaosong@shmtu.edu.cn</footnote>

<footnote n="2">² Kunming University of Science and Technology, Kunming, 650500 China
qianqian_yn@126.com</footnote>

<footnote n="3">³ Kochi University of Technology, Kami-shi, 782-8502 Japan
shinomorikeizo@kochi-tech.ac.jp</footnote>

<footnote>*Corresponding Author</footnote>
representation and identity-independent neural representation [3]. However, two issues should be further considered. First, although variance in identity impaired expression aftereffect, when adapting faces and test faces are of different person, not only facial identity but also expression geometry is also changed. It raises a possibility that the reduction of expression aftereffect in different person condition is simply induced by the difference of expression geometry instead of the difference of face identity between adaptor and test faces. Second, as the similarity between the adaptor and the test is an important factor to modulate the aftereffect in the low-level visual adaptation, it is necessary to examine whether the similarity of facial identities also affects the magnitude of expression aftereffect.

To clarify these two issues and systematically examine the effects of expression geometry and facial identity on the expression aftereffect, five experimental conditions were tested in current study. The interaction of facial identity and expression geometry were considered in 2X2 combinations to constitute four conditions (same or different: facial identity and expression geometry between adaptor and test face): same identity/same geometry, same identity/ different geometry, different identity/same geometry, different identity/ different geometry. In fifth condition, to examine the effect of identity similarity on the expression aftereffect, we used the computer-morphing technology to generate the artificial faces of different similarity, and tested on the same ambiguous images.

## 2 Method

There were five adapting conditions in the experiment, in which test faces were always the same among conditions, but the identity and/or the expression geometry of adaptor were manipulated. The five conditions are respectively termed as SI/SG, SI/DG, DI/SG, DI/DG, and DIDS, where the first two letters indicates whether adaptor and test face are of the same identity (SI: Same Identity) or not (DI: Different Identity), and last two letters indicates whether the expression geometry of adaptor are the same with that of test face (SG: Same Geometry) or not (DG: Different Geometry). The last DIDS refers to the condition in which the expression aftereffect is measured with the face adaptor of different similarity.

### 2.1 Subject

The subjects were 8 paid students with normal or corrected-to-normal vision (5 from Kochi University of Technology and 3 from Shanghai Maritime University, mean age: 20.2, SD=3.7). All 8 subjects participated SI/SG, SI/DG, DI/SG, DI/DG conditions, and 5 subjects from Kochi University of Technology participated the DIDS condition.

### 2.2 Stimuli and Apparatus

The face stimuli were selected from the affiliated image set of the Facial Action Coding System [4] and the Cohn-Kanade AU-Coded Facial Expression Database [5],
which are coded by Face Action Coding System and enables us to select same expres-

<figure><img src="image_4.png" /></figure>

sion configuration for different photographic subjects.

<caption>**Fig. 1.** The adaptors of surprise-disgusted expression pair used in four conditions (a), and experimental procedure (b).</caption>

In SI/SG, SI/DG, DI/SG, and DI/DG conditions, happy, angry, surprised and disgusted expressions constituted two expression pairs, i.e., happy-angry and surprised-disgusted expression pairs. Illustrated by the case of surprised-disgusted expression pair, we selected two female photographic subjects (F01 and F02) depicting two expression configurations (C01 and C02) of surprise and disgusted expressions, resulting in four combinations of identity and expression configuration (F01 with C01, F01 with C02, F02 with C01, and F02 and C02) (see Fig. 1a). The same method was used to select the adaptors of happy-angry expression pair, except that the two photographic subjects showing happy and angry expressions were male.

Using two expression images of the same photographic subject, we morphed a series of test faces using Microsoft Azure Morph 5.0 for surprised-disgusted and happy-angry expression pairs, respectively, with the nine middle ambiguous images, which varied from 30% to 70%, served as test stimuli.

The test faces in SI/SG, SI/DG, DI/SG, and DI/DG conditions were always the ambiguous expression images morphed between F01 with surprised expression (C01) and F01 with the disgusted expression (C01). We used the same test faces but different adaptors in the four experimental conditions. For instance, In the same identity/same configuration condition, the same images used to construct the test faces were used as the adaptors (i.e., F01 with surprised expression C01, and F01 with disgusted expression C01). In different identity/different configuration condition, the adaptors were the face images with different identities and expression configuration that differed from test faces (i.e., F02 with surprise expression C02, and F02 with disgusted expression C02).

In DIDS condition, we used computer-morphing technology to create the similar faces, because the morphing faces resembled two original faces when the features of two face are blended together. The similar faces were created by morphing the test face between F01 with G01 and F02 with G01 on smiling and angry expression pair
using the Fanta morphing software. We created the nine similar faces with identity
proportion of test face from zero to 100% in steps of 10% in terms of the scale of the
morphing software, with zero refer to that the created adapting face is dissimilar to the
test face, while one denote the adapting face is absolutely same with test face, we
select those face with proportion equal with 30%, 50% serves as adapting face. The
reason not to select the face images with similarity strength greater than 50% as
adapting face, because these faces are too similar with test face so that subject may
consider them as the same person rather than similar person.

## 2.3 Procedure

The experiment consisted of 10 blocks, each for one of eight experimental conditions (2 expression pair x 5 experimental conditions), performed in an order that was randomized across subjects. Each block includes 2 adapting images and each adapting image was presented 20 times in 9 test images, resulting in a total of 360 trials for each block. The trials for different test in a block were also randomized. The duration for one block is approximately 40 minutes. The subject participated one block every other day and finished all experiments within 20 days.

Each subject was tested individually. They learned the experimental task through oral instruction and a short training session. In each trial, adaptor and test image were sequentially presented in the center of screen (Fig. 1b). After the presentation of test image, subjects perform a two-alternative forced choice (2-AFC) task to classify presented image into one of two categories (i.e., two expression images used to create the morphed series). The subject was instructed to attend to the face stimuli but no fixation point was given. This is to prevent subjects from overly attending to local facial features near fixation point. The duration of adapt stimuli and test stimuli were determined to 4000 and 200 ms, respectively, with 100ms noise mask between adaptation and test stage to minimize the possible apparent motion effects.

3 Result

All adapting conditions generated significant aftereffect (Fig. 2), confirming the expression aftereffect reported in the previous literatures [2][6]. A two-way analysis of variance (ANOVA) has been performed on SI/SG, SI/DG, DI/SG, and DI/DG conditions. There was a significant main effect for adapting condition (F(3,18)=10.19, P<0.001), indicating that the expression aftereffect were significantly different for four different adapting conditions. There was no significant main effect for expression pairs (F(1,36)=0.11, p=0.742) and no interaction between expression pair and adapting condition, thus, the data from different expression pair was merged for the further analysis.

The size of expression aftereffect in DI/SG condition was weak relative to that in SI/SG condition (t(17)=6.57, p<0.001), indicating that the reduction of expression aftereffect across identity is held even when the adaptor and test face has the same expression geometry. This result suggests that only variance in expression geometry cannot well explain the reduction of expression aftereffect, thus confirming the inter-
ference from identity system to the expression system. On other hand, the size of expression aftereffect in SI/DG condition was weaker than that in SI/SG condition (t(17)=5.84, p<0.001). In contrast, the aftereffect size in DI/SG condition was approximately the same with that in DI/DG condition (t(17)=0.19, p=0.851). These are new findings suggesting that variance in expression geometry impairs the identity dependent expression aftereffect, but not the identity independent expression aftereffect.

For DIDS condition, the result reveals that identity similarity between the adapting images and test could significantly increase the expression aftereffect. The adapting effects by 30% and 50% similar faces are 10.4% (S.E.M. =2.3%) and 22.1% (S.E.M. = 3.9%), and the adapting effect by the 50% similar faces is significantly stronger than that by the 30% similar faces (t8 = 2.843, p < 0.02). It is worth noting that the adapting effect by the similar faces is still weaker than that in the SI/SG adapting condition (100% similar faces), but stronger than that in the DI/SG adapting condition (0% similar faces) (Fig. 2b). It suggests that the expression aftereffect would increase as the function of identity strength, thus confirming that the expression aftereffect is identity-dependent.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** The aftereffect sizes in SI/SG, SI/DG, DI/SG, DI/DG (a) and DIDS(b) conditions. Error bars denote SEM.</figcaption></figure>

# 4 Discussion

We firstly found that the expression aftereffect is still much reduced when adaptors and test face have the identical expression geometry. Such finding is consistent with Ellamil et al.'s work [7], which used the artificial faces of anger and surprise expression as adaptors and found the reduction of adaptation effect when adaptor and test faces have same expression morphing prototype but different facial texture and contour. Together, our and Ellamil et al.'s observation suggests that the reduced aftereffect cannot be simply attributed to the dissimilarity of expression geometry between adaptor and test faces, it does reflect a functional interference from identity system on expression system. This observation consolidates Fox's proposal of the identity dependent neural representation. We next found that identity similarity between the
adapting images and tests could significantly increase the expression aftereffect, thus indicating that the expression aftereffect is identity-dependent in specific range of identity strength.

Our data fits with the hypothesis of partially overlapping neural representations for the perception of expression and identity [8][9]. These two systems depend on the different facial component to extract the information and process the expression. As the structural reference hypothesis stated [10], the face structure information is not only important in the identity discrimination, but also used by observers as a reference to compute and recognize expressions. The identity dependent expression system may more rely on the facial shape and/or structure information (e.g., local edge, facial contour) to perform the expression geometry analysis, while the identity independent expression system seems to depend on the abstract emotional information. It would be interesting for the future researches to locate the cortex of these two different neural representations.

## 5 Acknowledgement

This work is supported by the National Natural Science Foundation of China under Grants No. 61403251, and Shanghai Municipal Natural Science Foundation under Grants No. 14ZR1419300.

## References

1. Mather, G., Verstraten, F., Anstis, S.: The motion aftereffect: A modern perspective. Cambridge, Mass: MIT Press (1998)

2. Webster, M. A., Kaping, D., Mizokami, Y., and Duhamel, P.: Adaptation to natural facial categories. Nature, 428, 557--561 (2004)

3. Fox, C. J., Barton, J. J. S.: What is adapted in face adaptation? The neural representations of expression in the human visual system. Brain Res. 1127, 80--89 (2007)

4. Ekman, P, Friesen, W. V., Hager, J. C.: Facial Action Coding System: The Manual on CD ROM. A Human Face, Salt Lake City(2002)

5. Kanade, T., Cohn, J. F., Tian, Y.: Comprehensive database for facial expression analysis. Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France, 46--53(2000)

6. Butler, A., Oruc, I., Fox, C. J., and Barton J. J. S.: Factors contributing to the adaptation aftereffects of facial expression. Brain Res. 1191, pp. 116--126(2008)

7. Ellamil M., Susskind J. M., Anderson A. K.: Examinations of identity invariance in facial expression adaptation. Cogn. Affect. Behav Neurosci., 8(3), 273--281(2008)

8. Baudouin J. Y., Gilibert D., Sansone S., and Tiberghien G., When the smile is a cue to familiarity. Memory, pp. 285--292(2008)

9. Calder, A.J., Young, A.W.: Understanding the recognition of facial identity and facial expression. Nat. Rev. Neurosci. 6, 641--651(2005)

10. Ganel, T., Goshen-Gottstein, Y.: Effects of familiarity on the perceptual integrality of the identity and expression of faces: The parallel route hypothesis revisited. Q. J. Exp. Psychol.: Hum. Percept. Perform. 30(3), 583--597(2004)