Morphing Identity: Exploring Self-Other Identity Continuum
through Interpersonal Facial Morphing Experience
Kye Shimizu

Santa NaruseSony Computer Science Laboratories, Inc.
Tokyo, Japan

Sony Computer Science Laboratories, Inc.
Tokyo, Japan

Jun Nishida

Shunichi KasaharaSony Computer Science Laboratories, Inc.
University of Maryland, College Park
Japan/United StatesSony Computer Science Laboratories, Inc.
Okinawa Institute of Science and Technology
Graduate University
Japan

Interactive & real-time morphing“it’s still my face”

Camera“we have a joint face”“I acquired mypartner’s face”

self

Display
0.5

self

other

other

0.2

morphing level

self

other

Figure 1: (a) We developed Morphing Identity, an interactive system that blends facial images of two users in real-time. Users’facial images are gradually blended from image of self to that of partner, while both users can see and interact with each other.
(b) We installed this system in several exhibitions to conduct a qualitative study to explore interpersonal experiences andquantitative analysis to provide further insights into the boundary between self and others.

ABSTRACT
We explored continuous changes in self-other identity by designing an interpersonal facial morphing experience where the facial
images of two users are blended and then swapped over time. Bothusers’ facial images are displayed side by side, with each user controlling their own morphing facial images, allowing us to create
and investigate a multifaceted interpersonal experience. To explore
this with diverse social relationships, we conducted qualitative andquantitative investigations through public exhibitions. We foundthat there is a window of self-identifcation as well as a variety
of interpersonal experiences in the facial morphing process. Fromthese insights, we synthesized a Self-Other Continuum represented
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
CHI ’23, April 23–28, 2023, Hamburg, Germany© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9421-5/23/04. . . $15.00
https://doi.org/10.1145/3544548.3580853

by a sense of agency and facial identity. This continuum has implications in terms of the social and subjective aspects of interpersonalcommunication, which enables further scenario design and could
complement fndings from research on interactive devices for remote communication.

CCS CONCEPTS• Human-centered computing → Interaction devices; Interaction
paradigms.

KEYWORDS
face, morphing, enfacement illusion, self-other identity
ACM Reference Format:
Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara. 2023.
Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience. In Proceedings of the 2023 CHI
Conference on Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/3544548.3580853

CHI ’23, April 23–28, 2023, Hamburg, Germany

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara
Social interaction
through morphed identity

Real-time
facial signal

Recompose

Recompose

Real-time
facial signal

Decompose

Decompose

Staticfacial identity

Staticfacial identity

Morphed facial identity

Figure 2: Our system achieves facial identity morphing by decomposing facial signals and facial identity from the user’s face
and then recomposing them, allowing for social interactions through morphed identities.

1

INTRODUCTION

With recent advancements in wearables, haptics, and virtual reality
techniques, developing interactive systems that can share or blend
human embodied experiences across multiple users has become
an increasingly emerging research interest in the HCI community.
These allow a user to embody a partner’s physical attributes in realtime (e.g. swap visual perspectives [44], copy muscle activities [68],and blend body movements [31] between users). It is also known
that embodying diferent physical attributes can change the user’s
bodily sensations, such as the perception of size or distance andeven self-identifcation (e.g., [7, 30, 53, 90]).
Recently, another bodily attribute important in shaping our social interaction has also been explored, that is facial signals, thechangeable aspects of faces such as facial expressions, gaze, mouth
movements, and so on [9, 36, 39, 69, 112]. These play a signifcantrole in shaping our emotional states [26, 27]; thus, HCI researchers
have employed these fndings into interaction design that can be applied in daily life, such as for regulating one’s emotional states using
a digital mirror [101, 111] and enhancing relationships during videocommunication [51, 76, 89] by manipulating facial expressions.
In addition to these facial signals, faces also contain a facial
identity, an important aspect that defnes the boundary between
self and other. The psychology community has previously exploredthis identity boundary by transferring the sense of ownership to adiferent person’s face by means of a visual and tactile stimulation
technique, where a user sees an image of someone’s face on a screen
while receiving tactile stimulation that is synchronized to both theuser’s face and the face appearing on the screen [29, 91].
While the technique has lead to signifcant fndings, such as how
a user’s implicit bias against a person with a diferent skin tone canbe positively changed by transferring the ownership of the face [29]through the use of visuo-tactile interactions, due to the nature of
the technique (i.e., it requires a certain amount of time to induce
the transfer of ownership and the user’s body must be fxed), thesesystems do not allow users to have conversations or communicate
using non-verbal communication methods (i.e., facial expressions,body actions), which play a critical role in creating a sense of motor
agency. While these studies have provided valuable insights throughcontrolled lab studies, taking social context and interaction into
account would also be benefcial for providing further interaction
design implications for face-changing applications.

2

OUR APPROACH: INTERPERSONAL FACIALIDENTITY MORPHING IN REAL-TIME

In this research, we focus on building a system that morphs twousers’ facial identities together in real-time, while preserving theirdiverse social interactions and contexts through the communicationof their facial signals.
To this end, we propose Morphing Identity, an approach that initially decomposes facial signals and identities from facial images, as
shown in Figure 2. By generating interpolation between two static
facial images and real-time image animation processing, the user
moves an interpolated facial image that represents an interpolated
facial identity with his/her own facial signals.
With our Morphing Identity system, the user can see his/her
own face side by side with their partner’s face. The left side of thescreen shows the moving face of the user as it gradually morphs
into the partner’s face. On the right side of the screen, the user
can see the partner’s face freely moving as it gradually morphs
into the user’s face. Our system can decompose and recompose
the relationship between one’s own real-time movements (facialsignals) and facial identity. This allows us to create a multifaceted
relationship in interpersonal communication where the user is able
to become the partner’s face and maneuver it, while at the sametime, the partner is able to move the user’s face.
We carried out both quantitative and qualitative analysis toinvestigate the changes in interpersonal experience as well as toexplore the boundaries of self-other identity, which could be used
for further scenario design, described in a later section (Figure 1 - b).
Since we are particularly interested in exploring how such a system
can change the interaction between two users and its applicationto actual scenarios, we conducted a feld study at public exhibitions
where we could invite dozens of people with diverse backgrounds
and relationships, such as family, friends, and couples [78]. We
found that diferent interpersonal experiences emerged during theprocess of interpersonal facial morphing, as well as areas of selfidentifcation across the steps of morphing. From these insights,
we synthesized the Self-Other Continuum, which is represented bya sense of agency and facial identity. This paper also presents thetechnical pipeline involved in the creation of the facial morphingsystem, which can be installed at exhibitions, and discusses its
technical challenges and further opportunities.

Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience

Lastly, note that our objective is to explore potential interaction
design spaces under an interpersonal facial morphing setting byinvestigating boundaries (Section 6) and subjective experiences(Section 7) in the interactive and continuous changes of identity.
This paper addresses this unexplored setting along with the scenario
design for the frst time to identify possible factors that can befurther examined by the community.

3

RELATED WORKCHI ’23, April 23–28, 2023, Hamburg, Germany

in VR, fostering saving behavior through showing one’s future
self [40]. Suzuki et al. developed FaceShare [89], which shares facial expressions, such as smiling, from one user to another during
a video conference, demonstrating that it enhanced a feeling ofcloseness and the conversation fow among the users.
These have demonstrated the signifcant potential of sharing orchanging facial signals to modify self-perception, subjectivity, andeven behavior, and our research extends these fndings to explore
the role of sharing facial identities in an interpersonal setting.

Our work is built primarily on the areas of computer-mediated communication, enfacement illusion, and facial morphing techniques.

3.1

Interactive systems for sharing/changing
experience

A number of interactive systems have been proposed for sharing embodied experiences among users to facilitate motor-skill acquisition
as well as embodying diferent physical attributes. These are often
achieved by using a pair of wearable cameras and head-mounted
displays (HMD) [19, 44–47, 49, 50], as well as haptic devices thatshare haptic sensation from the upper-limbs [55, 66, 79], wrist [32,41, 96, 97], hands [6, 12, 38, 63, 70], muscle activity [38, 67, 68],
tactile sensations [21, 61], and whole body sensations [73, 75] fromone person to another.
The efect of using these techniques has been also examinedin virtual reality (VR) environments to understand the changes inuser subjectivity, such as in their sense of agency through having ashared virtual body [31, 98] or size/distance perception and evenself-identifcation by embodying a diferent person’s body (e.g., thatof children [7, 53, 90, 102], with a diferent skin tone [8, 56, 72], or
one appearing as a taller [109] or smaller [30] avatar).
It has been shown that these can also enhance interpersonal
feelings such as the sense of togetherness, trust, presence, andpartner’s afect through the sharing/blending of physical experiences [6, 13, 52, 64].

3.2

Face, self-identity, and morphing

In addition to these body components, it is also noted that our
faces play a critical role in shaping not only interpersonal feelings but also social interaction [9, 39, 69], one’s own emotional
states [101], relationships [76], and behavior [40], which should
also be considered for this type of interactive system.
Researchers have conducted a series of face-changing and facemorphing studies. Canonical examples use the technique of the
Rubber Hand Illusion (RHI) [10], in which a user feels that a dummy
rubber hand becomes a part of their body by receiving synchronous
visual and tactile stimulation to both a dummy and their real body.
This has been extended to induce a sense of ownership over a different person’s face, which resulted in changing self-facial identityand attribution [71, 82, 99].
HCI researchers have also worked to incorporate these fndings
into designing interactive devices to support users in regulating
afective states in a daily life setting. Examples include a wearable
projection mapping onto a user’s face [4], a glasses-type device that
dispenses teardrops to induce a feeling of sadness [110], a mirrortype face morphing device for emotion regulation [101, 111], andeven an avatar with age-progressed renderings of a user’s face

3.3

Interpersonal self-other face boundaries

There have been several attempts to investigate changes in identity
in an interpersonal setting in the psychology community, since it
is known that how we identify the faces of others is adaptive andhighly dependent on our context and experience [62].
For instance, with synchronous visuo-tactile stimulation on
the face, participants accepted faces that contained a signifcantly
higher percentage of the other’s face when their own face wasstimulated [91, 99]. This change in self-other boundaries in the face,called enfacement illusion, has been reported to occur with active
touch [93], visuomotor synchronization [28, 81], and synchronization of the heartbeat and vision (Cardio-Visual Integration) [80], andit has been shown to correlate with whole-body embodiment [23],interoceptive sensitivity [94] and interpersonal somatosensory processing [18]. These types of bodily illusions take place based on themechanisms by which our brains attempt to solve conficts between
sensory and sensorimotor information [5, 100], also based on computational principles of movement [108]. These mutual fusions ofself and other face recognition have been reported to correlate withinterpersonal relationship factors such as empathic traits, physicalattractiveness [82], and positive interpersonal interactions [14].
More importantly, the changes in self-other boundaries are expected to increase the positive chemistry with others by overcoming
the boundaries between in-group and out-group categories [29], in
fact, the similarity between individuals is considered to be a foundation for social cognition [59]. Under the enfacement experimentparadigm, researchers have reported that the interpersonal enfacement efect resulted in participants judging others to be closer tothe self physically and in terms of personality [58], or reported
the potential of increasing closeness and attraction towards an
opposite-gender other [35, 65, 76]. These prior researches may indicate that a change in self-other boundaries through multisensory
integration of face images (though not only face images) can afectinterpersonal social cognition [22].
In light of the technological capabilities for facial image intervention in communication spaces including virtual reality and videocommunication, investigating the impact and possibilities of interpersonal face fusion under various contexts is an important agenda
for HCI. To this end, to further elevate the face morphing paradigm, which has previously been seen in laboratory experiments,
we built a real-time interactive face morphing system with stateof-the-art video processing to enable exploration within various
social contexts.

CHI ’23, April 23–28, 2023, Hamburg, Germany

3.4

Real-time face morphing

In recent years, face morphing methods using machine learningtechnology have been proposed [103]. Generative adversarial networks (GANs) such as the StyleGAN2 series [43] are able to generateincredibly photo-realistic face images that are represented as a latent vector, and the vector distance relates to how we perceive andrecognize generated images [85]. Moreover, not only for generating
facial images, GAN projection methods allow us to tie images tolatent vectors in latent space [2, 77]. This enables the generation ofa series of interpolated images between a source and target image,e.g., between two users.
Although interpolated images can be generated, the images are
static without facial signals. To recompose the facial movements insync with the user’s facial movements, a facial reenactment technique is utilized, that is, a technique for transferring the emotion
and pose of a source face to a target face while preserving the identity of the target face [33, 86]. Facial reenactment has also beenused to reduce the network bandwidth by transmitting only face
signals (feature points) and key frame images, and then recomposing a realistic face image [1, 106]. This approach is closest to our
proposal in terms of creating realistic face images using one’s own
face as a key frame. By combining these algorithms, we built a facemorphing system that can generate arbitrary interpolation while
preserving the facial movements of the users.

4

IMPLEMENTATION

Our system consists of 1) a real-time interpersonal face morphingalgorithm, 2) a server-client data streaming network, and 3) user
I/O interfaces.

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara

latent space, 2) generating a morphing sequence by interpolating
the latent vector from the user’s face to the destination face, 3)
performing real-time facial reenactment and alignment with the key
frame image from the morphing sequence while applying dynamic
control to proceed with morphing (Figure 3 - b).
4.1.1 Projecting user’s face onto the latent space. We employed apre-trained model based on StyleGAN2 [43] that was trained with
the Flickr-Faces-HQ Dataset (FFHQ) [42]. This model generates
realistic face images at a resolution of up to 1024×1024 pixels, which
are generated from a latent vector (18 × 512 matrix of real values,
512-dimensional vectors are input to 18 layers). By manipulating thelatent vector, StyleGAN2 can generate a corresponding facial image.
For instance, a vector interpolated from two latent vectors cangenerate an interpolated facial image between two corresponding
two facial images. We leverage this principle to generate the facemorphing images between two users.
To create a GAN-generated facial image from an actual facial
image of a user, a latent vector corresponding to the input facial
image of the user is estimated through a projection process, embedding an image into the latent space of a pre-trained GAN model 1 .
The quality of the generated image is correlated to the model and
GAN utilized for inferencing. Though some architectures of models
can infer faster due to their more limited number of parametersand model size, they produce lower-quality images. For feasibility in terms of time and experience at the exhibitions, StyleGAN2
was used for its balance of quality of facial imagery and inferencetime for the projection process. To make sure that participants
recognized a facial image as themselves at the beginning of theexperience, the image in the frst 10% of the morphing process
also included a geometry-matched blended image, combining agenerated image and the actual captured facial image.
4.1.2 Generating Morphing Sequence. Face morphing can then be
performed by mixing the latent vectors of two face images with
� ′ f = � f + � (� t − � f ). Here, � f and � t represent the latentvector of the original facial image (from) and the destination facial
image (to).
Morphed latent vector � f′ from � f to � t can be obtained by changing � . Utilizing the latent vector generated in the frst step, thesystem generates a sequence of smooth morphing images from theface of the user to the destination face with 100 interpolated steps(Figure 4 - 1).

Figure 3: (a) Overview of the device setup, (b) process of generating and animating morphing faces from a facial imageof a camera input.

4.1

Real-time interpersonal face morphing

Here, we describe the technical pipeline of the Morphing Identity
system. The pipeline consists of 1) projecting the user’s face onto the

4.1.3 Real-time composition with facial reenactment. To allow interaction between users while the facial morphing proceeds, face
reenactment technology was integrated to enable the transfer of a
facial expression and head pose of a source input to be transferred
to a target face while preserving the target facial identity.
Our system performs real-time composition with a facial reenactment process utilizing the First Order Motion Model [86]. A face
image selected from the morphing sequence is animated on thebasis of the motion appearing in the real-time video of the user.
At the beginning of interpersonal face morphing, the source image for facial reenactment is the user’s projected face (Figure 4 A’), i.e., the user’s facial image is animated on the basis of theirown facial motion. Then, the source image for facial reenactment
1 Example: https://github.com/rolux/stylegan2encoderMorphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience
1
A

A

A

A’

B’

A’

C’

A’

B

4.3
D

2

Reenactment
source image

E

Reenactment
target image++

Animatedface image

Face extraction

Video composition

F

Video composition

F

Figure 4: (1) Process of generating morphed static facial identity. (2) Real-time facial signals processed by video composition with real-time facial reenactment.

is gradually morphed towards the destination face (Figure 4 - B’
or C’ or D’). To enable dynamic movement of the user’s face, thesystem extracts the region for facial reenactment by consideringface alignment (Figure 4 - E). The synthesized facial image andcorresponding image region are synchronized and composited together for the fnal display output to the users. (Figure 4 - F). In
our implementation, the latency of the real-time composition isapprox. 100 ms. All delays caused by the camera capturing, image
composition, and display output add up to approx. 300 ms as theend-to-end latency, which was measured using a high-speed video
camera. It is noted that previous studies generally agree that a critical range of visual-delay range for preserving the sense of agency
is 200–300 ms [60, 84, 105, 107]. Thus, our system provides fastenough video feedback to users.
Throughout the pipeline for interpersonal face morphing, theuser’s facial image appearing on the display is gradually mixed
and then swapped with the other user’s face over time, while bothusers are able to move the facial image in real-time. As the facial
image is gradually updated, the visual change can be difcult to be
detected since the visual transient is suppressed [87]. This allows
us to investigate how users recognize facial identities rather thandetecting visual changes.

4.2

equipped with two GPUs (NVIDIA RTX 3090) and an i9-9900K CPU,and the server computer is equipped with one GPU (NVIDIA RTX
3090). The two computers are connected via a 5-Gbps network.

C

D’

Face reenactment

CHI ’23, April 23–28, 2023, Hamburg, Germany

System architecture

To facilitate a real-time interactive experience as well as heavy
computational processing, we implemented our system with twocomputers: 1) a main computer to manage the interactive experience including face reenactment rendering and user interface
control, and 2) a server computer to organize heavy computational
processes, including projecting the user face into the latent space,and creating the GAN generated images. The main computer isUser input and output

The exhibition consisted of two semi-open white booths that weremirrored against each other, integrated with digital mirror devices,a foot pedal for input in the experiment, an LCD monitor, and acamera attached to the inner wall, allowing a clear view of each individual in front of the monitor from the camera (Figure 3 - a). Insteadof showing a single facial image, we opted to use a side-by-side
layout, which provided the opportunity for the participant to seeboth faces at the same time and experience diverse contexts in theiragency and identity, allowing us to explore complex interpersonal
facial morphing experiences. Because of the semi-open structure of
the booths, users were also able to openly communicate with one
another while the morphing experience took place.
Both the display and the camera on either side of the booth wereconnected to the main computer. The camera was used for tracking
the facial movements of each user, acquiring facial images for the
GAN projection to generate facial morphing images. The display
showed the instructions and facial images of the two users side byside for interpersonal face morphing. As the generated GAN image
takes background imagery into account when generating imagery,two 4000K LED bars were equipped on either side of the monitor toprovide an adequate lighting environment for the camera as well asa uniform lighting background for the user image.This is importantfor maintaining the quality of the image taken at the very beginning
to avoid the artifacts in the generated images. The foot pedal, which
worked as keyboard input, was used by the participants to respondto consent questioning as well as to provide input during the facial
morphing experience.

5

EXPLORING SELF-OTHER FACIALIDENTITY THROUGH EXHIBITIONS

Our research objective in the exhibition was twofold; frst, to conduct a quantitative investigation regarding how much a person
recognizes a facial image as their own, while gradually morphing
towards another face, and second, to conduct a qualitative investigation regarding the interpersonal experience during the facemorphing process with diverse social relationships and contexts.

5.1

Booth design

The exhibition space was designed with the following three design
considerations: 1) allowing interpersonal communication in a public
space by designing the installation booth as a half-open box space(Figure 5 - a) to avoid disturbance by social anxiety such as frombeing observed by outsiders, 2) allowing participants of all ages
to experience the exhibition by preparing adjustable seats that ft
into the camera frame, video instructions, written explanations aswell as having well-trained museum staf on-site and 3) providing
expectations of the interpersonal experience from the form factor,
which appeared like a photo booth in a symmetrical confguration.

CHI ’23, April 23–28, 2023, Hamburg, Germanya

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi KasaharaMorphing rateb

1.0 (other)Entering
booth

Captureface images

Askconsent

Self boundaryrate
0.0 (self)

Start morphing
1

2

3

Note

Foot switch release
5

4

6

7

For as long as you think

In this exhibition, anonymized data will be recorded

the image looks like “yourself"

The data will only be used for academic researchkeep pressing the foot switch

If you agree, please press on the foot switch

Press the foot switch to Start

Figure 5: (a) Exhibition setting, showing participants lining up to participate, with the well-trained exhibition staf in yellow
overlooking the situation. (b) A sequence of interpersonal face morphing experience.

5.2

Sequence

Two exhibit participants were frst instructed to sit in the adjustableseats (Figure 5 - b1). When the system recognized the participant’s
face, the display showed a mirror image from the camera with anoverlay graphic to guide the face to a designated position so as tocapture it from the front while it was stable (Figure 5 - b2).
After the participant’s face was captured, the system cropped thefacial areas from the camera image and utilized the server computer
to search for a latent vector corresponding to the facial image by
projecting the image into the latent space of the GAN. During theapprox. 15 seconds taken for this computation, the system showeda written statement for getting consent, stating, "Taken photos
will not be stored; anonymized data will be recorded and used forresearch purposes only," and then asked the participant to press thefoot switch if they agreed to give consent (Figure 5 - b3).
After 100 morphed images were generated, the display showedeach participant’s real-time moving faces (i.e, 100 percent themselves). The participants were instructed, "For as long as you thinkthat the image looks like “yourself," keep pressing the foot switch,"and they were also told to "Press the foot switch to start" to start thesequence (Figure 5 - b4),which was important for ensuring that theparticipants saw the face image from the beginning of the sequence.
The real-time face morphing started to proceed with a morphing
speed of (2%/1 sec). During the face morphing, the display showedboth faces of the participants in side by side. Throughout the experience, the participants could freely see their own faces and theother person’s face, and also have conversations. The participants
were expected to release the foot switch when they no longer recognized that the displayed face as their own face (Figure 5 - b5). The
face morphing proceeded until the faces were completely swapped(Figure 5 - b6), and the displayed face then began to return towards
the original face in the opposite direction as well. Finally, after themorphing was completed, the faces on the screen had returned totheir original form, and the display indicated the end of the experience (Figure 5 - b7). All participants were allowed to use glasses tohave corrected vision. Throughout the experience, the two participants engaged in real-time interactive face morphing while having
a conversation and viewing realistic facial expressions.

5.3

Data acquisition

In the exhibition experience, we acquired two types of data: 1) the
self-face recognition boundary as input using the foot switch, and 2)

the metadata recognized from the facial images of both participants.
These data were recorded in an anonymized manner and securely
managed only for this research purpose only. For analysis of theexperiment and to preserve the privacy of the participants, we asked
each participant for their consent through an explanation panel andfoot switch. Underage participants were asked for parental consent
before partaking in the experience.
To analyze the age and gender of each participant, we used thedata from commercial facial image recognition software (Azure
Face API 2 ) as an alternative method to direct input so as to obtain of demographic information from participants and provide asmooth experience for both the active participants and those waiting. It is also noted that there are still limitations regarding thequality of the analysis as well as the risk of bias towards specifc
demographics [16]. The participant facial-imagery data was stored
as a latent vector, in favor of facial imagery, to protect the privacyof the participants. This study was approved by our Institutional
Review Board (22-F-0002).

6

QUANTITATIVE ANALYSIS

Our primary interest in this analysis was to measure the boundary of self-face recognition across the steps of morphing toward
the other face. We analyzed the participants’ responses obtained
throughout the exhibition from the foot switch from our collected,
9269 pairs of response data throughout the exhibition. Each pieceof collected data included the self-face boundary at a particularmorphing rate (0 - 100 %), software-recognized metadata (genderand age range), anonymized ID, and partner’s anonymized ID.
As planned, our exhibition was participated by an arbitrary
pairs of participants with various social relationships, from pairs of
strangers to families. These variations might have also infuenced
the variance in our analysis. One may consider that the morphingrate (�� : 0 - 100 %) is a relative metric that could potentially diverge
based on how much two faces are visually similar or diferent. To
measure the self-face boundary between arbitrary pairs of faces,
instead of the morphing rate �� , we used the distance of the latent vector (� � ) which can be calculated by multiplying the ��and the distance between two latent vectors of two faces (� f1 and� f2 ). Here, we defne the self boundary as � � = �� ∗ |� f1 − � f2 |.
In the following section, we describe � � (distance of the latent
2 Microsoft

services/face

Azure

Face

API:https://azure.microsoft.com/products/cognitive-b
Reported self face
Distance of latent vector (dw)

CHI ’23, April 23–28, 2023, Hamburg, GermanycdDistance of latent vector (dw)

Reported self faceaFrequency of self-face boundary

Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience

Distance of latent vector (dw)

Figure 6: (a) A histogram of the reported boundary value. The solid line depicts the kernel density estimation of the distribution(N = 3411). The dotted line represents the peak in the histogram. (b) Cumulative distribution of reported boundary value. The
dotted line represents the point of halfway 50% recognition of the self-face. (c) Cumulative distribution of reported boundary
value by recognized gender and (d) the range of age.
vector) as the self-boundary value, and �� (0 - 100 %) as the selfboundary rate in the morphing steps. Since both the value rangeof the distance of the latent vectors and the rate of morphing areindeed similar, we also note that we describe the rate of morphingin terms of morphing level (0 - 100%) in the later part of this paper
to facilitate an intuitive understanding.

6.1

Data preparation

Due to the nature of this public exhibition in a public space, it would
be difcult to completely guarantee that participants would understand the instructions accurately and respond to the experience as
we expected. To analyze data collected in unpredictable situations,
we carefully observed the behavior of the participants and defned
the criteria for the data exclusion as follows.
• estimated age < 10: Younger (kids) participants would notunderstand the instructions for the foot switch correctly.
• count of foot switch > 5: To flter out data from some participants who pressed the foot switch multiple times without
any intention.
• self-face boundary rate > 70%: The face image on thedisplay was clearly not the user’s face, but the foot switch
was accidentally pressed.
• self-face boundary rate < 1%: The participant did not understand how to use the foot switch.
• invalid captured face: For the case where the facial imageof the participant was not correctly captured, for example,
due to incorrect cropping or obstacles appearing in the frame.
On the basis of these criteria, we excluded 5858 pieces of data
(63.2 %) before analyzing the collected data. Even with a higher
exclusion rate, this amount of data is sufcient for refecting on thenature of the public exhibition in public space. Thus, we used 3411
pieces of data for our analysis.

6.2

ResultFigure 6 - (a) shows a histogram of the self-face boundary rate, i.e.
when the person recognized that the displayed facial image wasno longer his/her face in the face morphing starting from his/her
own face and morphing into the partner’s face. A kernel density

estimation of the histogram is also displayed in Figure 6 - (a) as
a solid line. To generalize the data obtained for analysis, we havetransformed the obtained histograms into a cumulative distributionfunction (Fig. 6 - b), calculated as a percentage that aggregated all
the collected data. As we can observe, a peak (in Fig. 6 - a) andshape drop (in Fig. 6 - b) in the self-face boundary rate appeared at
18.2 in � � , the distance of the latent vector. In other words, there
was a clear margin for self-face recognition along with the facemorphing into the other face. Indeed, the corresponding � � where
the halfway 50% recognition point was 23.4, was at a morphinglevel of approx. 20 %.
We also found diferences in the self-face boundary in terms ofgender and age based on the recognized metadata of the participants.
Figure 6 - (c, d) shows graphs of the cumulative distribution function
with diferent demographics. Since a violation of normality was
found in the self-boundary value across the age group and gender(Shapiro-Wilk Test), we used non-parametric methods to evaluate
the signifcance in statistical comparisons.
For the diference in gender, a Wilcoxon Rank Sum test showedthat there was a statistically signifcant diference between gender(male and female) in self-boundary value in terms of � � , (� = 9.40,
p < 0.001). Indeed, the median values were 34.9 for males (N = 1491)
and 28.9 for females (N = 1920). For the age range, a Kruskal-Wallis
H test showed that there was a statistically signifcant diferencein � � among the age ranges, � = 77.0, p < 0.01, with a mean selfboundary value (� � ) of 29.6 for 10s (N = 438), 29.1 for 20s (N = 1839),
34.8 for 30s (N = 852) and 39.9 for 40s (N = 231). A Steel–Dwass
posthoc analysis revealed that there were signifcant diferences
between 10s-30s, 10s-40s, 20s-30s, and 20s-40s (p=0.001) and a trendof diference between 30s-40 (p=0.054).
Taken together, our main fndings from the quantitative analysis
on the self-face boundary were that 1) there is a window of selfidentifcation in the face morphing steps and 2) the windows ofself-identifcation would difer with the demographics.

6.3

Discussion

6.3.1 Window of self-identification. The results for the boundaryof self-face recognition allowed us to analyze how the participants

CHI ’23, April 23–28, 2023, Hamburg, Germany

recognized faces in the images displayed during the experience.
In particular, our analysis revealed that when the morphing level
was within 20% (with a latent vector distance of 23.4), participantstended to recognize the displayed face as their own, even thoughmorphing changes were applied. On the other hand, if more morphing was applied, the face was not recognized as their own. Furthermore, our analysis may suggest that when the morphing levelis 80% or higher, the face displayed on the opposite side should
be within the boundary of self-face recognition, where it would
be recognized as the participant’s own face even if it is moved by
the other person. This means that participants would experiencethat their faces were swapped and that the other person is moving
the swapped face and vice versa. This is also mentioned in thequalitative observations in a later section.
6.3.2 Comparison to previous literature. Here we compare ourresults with the results from the previous studies on the enfacement illusion to further understand the potential mechanismof enfacement and identity [3, 15, 92]. Enfacement illusion experiments have reported that the window of self-identifcation for static
facial images can be changed before and after the visuo-tactile stimulation. While many experiments reported a boundary value ofaround 40-60% in self-other judgments [82, 91, 99] as a morphing
percentage, our results indicated diferent trends compared withthese previous reports. Here, we discuss the potential factors thatinduce these diferences.
Firstly, in our morphing identity experience, participants judged
the self-identifcation on the basis of their real-time facial signals (facial movements), while the enfacement illusion employed asense of ownership over the static facial images [28, 81, 82, 91, 99].
While the real-time and congruent visuomotor feedback of the facial
movements would preserve a sense of agency that may facilitate
self-attribution [36, 37, 48], at the same time, it would also provide
more visual cues for discriminating between faces [24]. Thus, theidentity recognized through real-time motion could be perceived
diferently from that recognized by observing a still image. Thisdiference may also provide an interesting experimental setting as
a paradigm for investigating the self-boundary.
Secondly, the enfacement illusion experiments used a pair ofsame-gender faces when morphing [81, 82, 91, 99], whereas arbitrary gender combinations were provided in our study. Thus,
the window of self-identifcation found in the previous studies
could be larger than that of our results. To overcome measurement
consistency, we introduced the GAN latent vector distance [43] as a
metric for the self-face boundary. It has been shown that the latentvector distance can be larger between diferent genders and races
than between the same-gender and race [83, 88] and that the vector
distance is correlated with the distance required to identify theface of the other [85]. Using this metric would allow us to compare
distances between any pair of faces using a common distance scalerather than a relative scale of morphing percentage.
Lastly, the answering criteria for self-identifcation could
be a potential factor. In the enfacement illusion experiment, theparticipants are asked to answer binary categorization questions
such as "rather me or other" in Two-alternative forced choice [81,91], visual analog scale [15, 82], or a key press with "look more
like self than other" as comparison manner[28, 99]. In comparison,

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara

in our study, the participants were directly asked whether "it isme" (asking about self-attribution), which may have resulted in thecriteria being stricter.
Due to these aspects, our results cannot be directly comparedwith previous enfacement illusion studies; however, our study could
suggest new ways to further understand the potential mechanismof enfacement and identity.
6.3.3 Race and gender. As an additional discussion, previous research has also shown that latent vector distances can vary depending on the race and gender of individuals [83, 88]. Our quantitative
study specifcally focused on the distance between two arbitrary
faces, and did not include a comprehensive analysis of racial diferences; however, in fact, we statistically confrmed that the averagelatent distance between two participants was larger for face pairs ofdiferent genders than pairs of the same gender. A Mann-Whitney
test indicated that the latent distance was greater between diferent
genders (Mdn = 105.84, SD. = 12.80, n = 1358) than between thesame gender (Mdn = 93.93, SD. = 14.15, n = 1010), U = 526890, p< 0.0001. Note that this latent distance indicates the distance between two faces, which corresponds to a 100% morphing rate. In
addition, the categorical perception of gender could also infuence
the recognition of the self-boundary [17]. Further incorporatingthese control factors could enhance the feasibility and efect of theinteraction design in our scenarios.

7

QUALITATIVE FEEDBACK

To further gain insight into the interpersonal experience between apair of participants, a series of semi-structured interview sessions
were organized on-site. Interviewees were chosen from participants
who were able to complete the experience without having any
interruptions and failures and were willing to partake in the survey(31 pairs totaling 62 participants; 36 participants reported "family"
status, 7 chose "in-a-relationship" status, 14 chose "friends" status,and 5 chose "other" status). As the interview research scope looked
into the interpersonal changes, participants who experienced theexhibition alone were excluded from the interview.

7.1

Data preparation

The data corpus consisted of audio recordings of the semi-structured
interview sessions, feld notes taken by the authors, and surveys on
our questionnaire completed by participants. All audio recordings(63 recordings with an avg. of 2.4 minutes) and our feld notes
were transcribed and analyzed using a deductive thematic analysis
focusing on both semantic and latent interpretation [11], which
is widely used for deriving key themes across various datasets,
and is increasingly used in exhibition settings [20, 54, 74]. Initialcoding was carried out by one of the authors, who interviewed andreviewed the data with the other authors to extract themes andstructures from the qualitative data. We then integrated the insightfrom the qualitative data and quantitative data to yield outcomesfrom our overall experiment.

7.2

Questionnaire and comments

Participants were asked to answer the following 7-point Likert
questions after completing the experience (� 1−3 ; 1: disagree, 7:
agree, � 4 ; 1: decreased, 4: no change, 7: increased): (� 1 ) Did youMorphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experienceenjoy partaking in the experience?; (� 2 ) Do you think your own facehas changed?; (� 3 ) Do you think your partner’s face has changed?;(� 4 ) Was there any change in your interpersonal feelings such asincreased closeness with the partner?
Disagree (1)

2

3

4

5

6

(7) Agree

2

3

4

5

6

(7) Increased

I enjoyed partaking inthis experience
I felt my own face had changed
I felt my partner's face
had changed
Decreased (1)Was there any change in yourinterpersonal feeling such as
closeness?
average(×), median( | ), IQR, outlier(�) and total range of variation

Figure 7: Subjective reporting on (1) the overall experience,(2,3) the changes in self and partner’s face, and (4) an interpersonal feeling.
Figure 7 shows the results of the Likert questionnaire. First, theparticipants reported that they generally enjoyed partaking in theexperience at the exhibition (range=3-7, median=6), which suggeststhat the setup was an enjoyable experience for most participants
overall. When asked why, the participants mentioned: "the experience was weird but fun"(P38 and P44), "[it was] interesting tosee a mixed face"(P43), "it was intriguing to explore the boundary
between the sense of my face and the other over time"(P1), and itwas "fun to become my son’s face"(P17, the mother of P18). They
also commented that it reminded them of their family ties, such as"[I] realized that our faces are comprised of that of my parents"(P23,
who tried the experience with his sister(P24)) and "I could imagine the face if there had been one more child [brother]"(P11 andhis mother(P12)). Similarly, P10, the mother of P9 who was in his
teens, stated after the experience, "I was happy to see and have aconversation with my son when he was a bit older looking." Theparticipants also felt entertained when controlling their partners’
faces, such as "I see her face every day but never control her face.
So it was strange, totally strange and interesting"(P49).
On the contrary, some participants mentioned that the experience was neither convincing nor entertaining. When asked why,
they mentioned the diference in race (P39, a white female, who experienced the exhibition with an Asian female(P38)) and in gender(P52, a male, who experienced it with a female(P53)).
Secondly, the participants reported the degree of the change intheir own faces throughout the experience (median=7, range=5-7)
as well as the change in their partner’s faces (median=7, range=5-7)
but with a slightly lower average rating and higher deviation. In
fact, some participants clearly mentioned this trend, for instance, "It
was actually a bit difcult to notice whether my partner’s face had
clearly changed"(P3) and "I felt awkward when I realized my face
had changed. On the other hand, I did not feel the same way when
I saw my partner’s face had changed"(P4, who was a total stranger
to P3). P45 added, "When I realized, my wife’s face had completelychanged." At the beginning of the experience (at the low morphinglevel), many participants mentioned that they were unsure of achange occurring in their own faces; for instance, P55 remarked,

CHI ’23, April 23–28, 2023, Hamburg, Germany"I did not realize there were some changes until the middle [stageof the morphing level]," and both P40 and P57 mentioned, "WhenI explicitly noticed it, the face already looked like someone otherthan me, but at the same time, it somehow felt familiar."
Lastly, the participants stated that they felt slightly positive
when asked about the changes in their interpersonal feeling (median=4, range=1-7). P38 stated, "It felt entertaining when controllingthe head orientation of my face, which is slightly morphed into
my partner’s face, resulting in an increasing sense of closeness."
P60 also added, "It actually increased my awareness towards myhusband’s face(P61), enhancing my feeling of closeness." P13, whois the mother of P12, stated that "I realized my son’s face is very similar to mine, and this fnding increased the sense of closeness." P52
and P53, who are in a relationship and reported that their feeling ofcloseness has increased, commented, "I could see the potential faceof our future generation." P42 and P43, who are female friends, alsomentioned that their feeling of closeness had increased because
they enjoyed the experience together.
The participants, who reported that there was no change, pointed
out that it was because they have already established good relationships, such as friends (P54 and P55) or spouses (P48 and P49). P47(a male friend of P46) and P30 (a male, who is in a relationship with
P29) also stated that they felt no change because their partner’s
faces became their own faces, which was not something expectedor pleasant for them.

Figure 8: Participants reacted to the interactive morphingexperience in various ways.

7.3

Interaction observation

Figure 8 shows the various reactions of the participants captured
in the middle of the morphing experience. As shown in Figure
8 - (a, b), one of the most common reactions we observed fromparticipants was their expression indicating a weird and strange
feeling. As seen in the comment feedback as well, the participants
were very new to this situation where faces were morphed into
those of another while they could still control them. As depicted in
Figure 8 - (c), many participants were surprised and embarrassed
when they realized that their facial image was blended with that ofa partner, mentioning that it was worth trying since it provided afeeling of shared attention and became a conversation topic with
their partner. In addition, some participants enjoyed controlling
their facial images by actively moving their mouths, eyebrows,
and heads, as shown in Figure 8 - (d). Interestingly, a participant(female), who was experiencing the exhibition with her son, stated"Stop playing with my face!" when their morphing level was at
the highest (almost completely swapped). Several participants also

CHI ’23, April 23–28, 2023, Hamburg, Germany

Morphing level

Experiencein Identity

Low

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara
20%

Unaware of the changes

Not me but some of me

Swapped face w/ agency

During this window, the participants
felt their facial identities still belonged
to themselves, even if they were extended to that of their partners

P40, 57: "When I explicitly noticed, theface already looked like someone
other than me, but at the same time,somehow it felt familiar"

The participants enjoyed controllingthe swapped images (Fig. 8d, P49) as
well as owning a face of their families(P17) or a different gender (P50)

Same face, shared narrative

Own face on other’s face

Provided a shared experience & evenfostered conversations about family
connections & future generations

Some participants still felt their identities in the facial images controlled bytheir partners: it was “strange” (P49)or “embarrassing” (Fig. 8c)from the quantitative analysis

e.g. P55 remarked "I did not realize
there were some changes until the
middle [stage of the morphing level](similar comments by P40, 57)

Experiencein Agency

High(comments by P10, 11, 13, 23, 52, 53)

summarysummarysummary

Participants’ facial identities wereextended, yet preserved

Participants experienced a joint facial
identity, creating a shared context

Participants’ facial identities &
feelings of agency were intertwined

P38: "It felt entertaining when controlling [...] my face, which had slightly
morphed into my partner’s face"

P53: "Still, not only the front view but
also each facial component was synchronized. This was an interesting experience for me."P49: "I enjoyed controlling my partner’s face". P45: "I could fully manipulate a face & its expression, which wascompletely different from my face."summary

Participants’ feeling of agency was preserved across all stages

Figure 9: Summary of the experiences in identity and agency plotted based on the morphing level.
tried to identify the diferences from the original facial images whileconversing with their partners (Figure 8 - e). We also observed thateven a pair of children (friends), who had experienced the system,
showed strong interest in the morphing images (Figure 8 - f).

8

SYNTHESIS OF OUR FINDINGS: SELF-OTHERCONTINUUM

In this section, we review on our fndings, to synthesize a Self-Other
Continuum through the interpersonal facial morphing experience.

8.1

Summary of quantitative and qualitative
fndings

Here, we frst summarize the quantitative analysis, comments, andobservations from the perspective of participants’ facial identityand agency, grouped by reported morphing levels at the time.
As revealed in the quantitative analysis, we found that the participants had a certain window in the morphing level, where they
were unaware of the changes in their facial identity, even if
it was morphed to that of their partners, and they were able tointeractively send facial signals to each other. In fact, this was also
supported by the comment feedback, as P40, P55, and P57 stated.
Regarding their sense of agency, as stated by P38, we observed thatthe participants felt that they were controlling their own faces.
It was also indicated that when the morphing level progressedbeyond the edge of self-identity (20%), the participants started feeling that there were mixed faces that felt like new but werestill comprised of their own facial identities, which was also

clearly mentioned by P40 and P57. It is also worth noting that, at thesame time, this created an interesting opportunity for both users
to see faces that are similar to each other based on the projectionsof the two users’ facial identities, creating a shared context that
fostered conversation on the topic of family connections and even
their future generations, as P10, P11, P13, P23, P52, and P53 pointed
out. Regarding their sense of agency, as noted by P53 when theirfaces were mixed ("Still, not only the front view but also each facialcomponent was synchronized. This was an interesting experience
for me."), they still felt a sense of control over their overall anddetailed facial components during this stage.
At the stage of a high morphing level, where their facial components were almost swapped, the participants started mentioning
the enjoyment and curiosity that they felt to not only controllingthe facial image that had morphed into their partner’s face (pointed
out by P49 and as depicted in Figure 8 - d) but also embodying
a new face such as that of their family (P17) or a diferent gender (P50) on their own. Interestingly, some participants also statedthat they felt weird or embarrassed when they saw that their faces
were controlled by their partners, as depicted in Figure 8 - c). Even
though the participant had no agency over the facial image, which
had morphed into that of the participant but was controlled by thepartner, the participant still felt his/her identity over it andfelt disgusted (P49). Regarding their sense of agency, as quotedby P45 ("I enjoyed controlling my partner’s face") and P49 ("I could
fully manipulate the face and its expression, which is completely
diferent from my face"), they were still able to feel that they wereMorphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience

Morphing level
Identity exp.

CHI ’23, April 23–28, 2023, Hamburg, Germany

Low

High

Extended facial identity

Joint facial identity

Intertwined facial identity

Users
SoA

FI

FI

SoA

SoA

SoA

SoA

SoA

FI

FI

e.g. Creating shared narrative
through joint facial identity

e.g. Surrogating & embodying one’s facial identity

MorphedimagesStatic identity
imagesPotential
scenarioe.g. Increasing feeling ofcloseness through implicit
morphing experience

Figure 10: Self-Other Continuum: consists of three concepts of facial identity; 1) Extended Facial Identity: where self-identity isimplicitly extended to that of a partner while feeling their own facial identities attached to each user, 2) Joint Facial Identity:
where their facial identities are mixed, but each user still feels an essence of each identity, and 3) Intertwined Facial Identity:
where their facial identities are swapped, with each user controlling a facial image with the other user’s identity morphed intoit. (SoA: sense of agency, FI: facial identity)
in control of the facial images that had morphed into those of theirpartners. We found this an interesting situation in that their sense ofagency and identity over the morphed facial images were separated
but intertwined with each other.
Throughout the entire exhibition process, it is notable that we
did not receive any comments that pointed out a decrease in the
feeling of control at any facial identity stage. Overall, while thissuggests that the participants felt a sense of agency over the facial
images across all diferent morphing levels, we observed there were
three potential boundaries in the facial identity during theinterpersonal morphing experience when the measured self-identity
window, comment feedback, and observations were grouped andplotted based on the reported morphing level, as depicted in Figure 9.
We then discuss further implications along with potential scenariosin the next section.
From our fndings, we then synthesized a continuum that represents the experiences with interpersonal face morphing, as depicted in Figure 10. It consists of three concepts of facial identity:
1) extended facial identity, 2) joint facial identity, and 3) intertwined facial identity, grouped based on how users recognize
the facial identity and the feeling of agency over the facial images.
To better illustrate the design implications, we give details on eachregion in the continuum, along with potential scenarios.

8.2

Extended facial identity

The left region on the Self-Other Continuum depicts the state of "extended facial identity" where self-identity is implicitly extended
to that of a partner while feeling their own facial identity.

In addition, users can feel a full sense of agency fully. Withextended self-identity being congruent with the agency, it would bepossible to design a scenario where a user can attach an extended
facial identity to his/her partner undetected in a daily life setting.
8.2.1 Potential scenario: increasing closeness through implicit morphing experience. As comments from the participants suggested,
the morphing experience at this stage increased the feeling of facial similarity (P3, P49, P62), and the participants envisioned moreefective changes to interpersonal feelings through more gradualand longer usage of the morphing, rather than having it progressin 100-second intervals as provided in the exhibitions. Previous
studies showed that facial resemblance enhances the feelings oftrustworthiness, likability, and attraction [35, 65, 76]. Our system
might be able to achieve this in a more implicit and interactive
manner that can be embedded in daily life settings while retaining
the user’s facial identity, such as increasing the sense of closenessbetween friends during a video call or between a store staf member
and a customer during online shopping, or decreasing anxiety whena student or a patient are talking to a teacher or a medical doctor
during a remote class or a clinical examination.

8.3

Joint facial identity

The middle region on the Self-Other Continuum depicts the joint
facial identity, where users feel their facial identities are mixed,
and feel a sense of agency over the intermediate facial identity, i.e., both users do not fully feel the mixed faces are their own
faces. However, as pointed out by the participants through thecomments regarding family connections, they can still feel the

CHI ’23, April 23–28, 2023, Hamburg, Germany

essence of each facial identity in the joint faces and can initiate
and perform social interactions with a complete sense of agency.
This would provide an opportunity to have a common experienceand narrative by interacting concurrently through a joint facial
identity, enhancing the feeling of togetherness among users.
8.3.1 Potential scenario: creating shared narrative through the joint
identity. When multiple users are working together as a team andperforming joint action [104], it is known that they tend to feel ahigher sense of agency in the case that the task outcome is positive(success), providing a stronger feeling of achievement and motivation to individuals, with the same principle applying to the case ofa negative outcome (failure) [95]. This principle seen in joint action
could also be applied to this joint facial identity setting where two
or more users need to work together as one representative virtualavatar (such as [31, 98]), which would allow each user to minimize
feelings of social responsibility when the outcome is negative andmaximize feelings of success when the outcome is positive sinceeach user still feels a part of the facial identity in the joint face.
Scenarios include a situation where a new employee is working
together with a senior instructor at a virtual online store so thatthey are able to share a sense of responsibility as well as achievement at the same time or a situation where only one person isallowed to give a speech from a company while sharing the feelingof representing the company with teammates.

8.4

Intertwined facial identity

Lastly, the right region on the Self-Other Continuum depicts an intertwined facial identity where the facial identities are swapped,with each user controlling a facial image with the other user’s
identity morphed onto it, i.e., each user still feels a sense ofagency over the face that belongs to another identity. This intertwined situation causes the attribution of self-control and self-facial
identity to become conficted, which can be described as: “I am moving the partner’s face” but "My partner is also moving my face.” Asseen in our qualitative observation, this could lead to benefcial use
cases of embodying or surrogating one’s identity when two users
have a consensus. On the other hand, these conficts could lead toviolating one’s facial identity without consensus, inducing distress.
Our observations suggest that the experience of swapping faces
can be highly bipolar, with individuals reporting both enjoyment
and disgust. Since we believe that the direction of this polarity islikely infuenced by factors such as personality, nonverbal, andnon-categorical relationships between individuals, the complexity of these factors makes it difcult to fully analyze using only
the qualitative data obtained in our study. As such, we would notassume that there is a one-size-fts-all approach for designing experiences involving facial swapping; instead, we highlight the fact of
the bipolar experiences as a design guideline for potential scenarios.
8.4.1 Potential scenario: surrogating identity to a partner user. Previous research has demonstrated that embodying diferent bodies
changes users’ implicit attitude and self-identifcation [57], resulting in better performance in social interactions [109]. This intertwined facial identity experience also has the potential to change
our subjectivity through owning a partner’s facial identity in daily

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara

life. For instance, one user might be able to acquire a trusted partner’s facial identity (e.g., such as those with more social authority
like business superiors, or those who are more physical facial attractiveness) to have more confdence when speaking to a large
audience and to give a better impression and speech performance.

9 FUTURE WORK
9.1 Extended "other"-identity
Under our current scope, we focused on investigating an extended
window of self-identity, which provided insight to boundaries ofself-identity with the facial morphing experience that enables one
to augment their facial identity in order to increase their feelings ofcloseness. In addition to this, it would also be possible to reveal anextended window of "other-identity" that represents how one user
perceives changes in another user’s facial identity. The qualitative
feedback suggested that there could be a wider window of the"other-identity", as the participants reported a slightly lower scoreon the degree of facial changes when it comes to their partner’s
faces. Revealing this factor would append further knowledge tothe continuum, allowing us to open up new design spaces in facialcommunication systems, as well as future applications.

9.2

More factors on self-identity

Though our focus was to explore potential interaction design spaces
in an interpersonal facial morphing setting, our approach also suggests new research directions, such as introducing baseline conditions. These include how factors in the facial signal transmission(e.g., video latency, quality of facial signal mapping, morphing
speed, manipulation of latent vector values, physical attributes
such as age, gender and race, and social relationship statuses) could
afect the morphing experience from the perspective of identityand agency. Investigating these factors in a more controlled setting would provide a better understanding of the emergence ofself-identity and application scenarios in various facial communication formats, such as video conferencing, virtual reality, or evenin-person with augmented reality.

9.3

Face composition technologies

Moreover, moving around with diferent face postures is an essential part of the experience, as users are able to interact withhow their face is being morphed in real-time. However, as there
are limitations to how the facial image is generated, only a frontal
image was generated, making full head movements and gestures diffcult for the system to synchronize. Recent research utilizes GANs
to generate facial imagery to be used as UV textures for facial
reenactment [25] and 3D face reconstruction and generation [34].
Additionally, the end-to-end latency of our system (300 ms) is righton the upper limit of the critical range for preserving or disrupting
agency over movements [60, 84, 105, 107]. A more fully generated
image of the head and minimized system latency could provide amore immersive experience to participants communicating throughnon-verbal head gestures.

Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience

10

CONCLUSION

Through the designing of an interpersonal facial morphing experience where the facial images of two users are blended and thenswapped over time, we investigated continuous changes in selfother identifcation, as well as various interpersonal experiences.
Through a public exhibition, we conducted a qualitative and quantitative analysis to examine this with diverse social relationships. We
revealed that the process of facial morphing includes an extended
window of self-identifcation as well as a variety of interpersonal
experiences. We then synthesized a Self-Other Continuum, which
is symbolized by a sense of agency and facial identity. This continuum ofers implications on social and subjective components ofinterpersonal communication, enabling additional scenario designand complementing research fndings on interactive technologies
for remote communication.

ACKNOWLEDGMENTS
This work was supported by the JST Moonshot R&D Program(Grant Number JPMJMS2013). Any opinions, fndings, conclusions,or recommendations expressed in this material are those of theauthors and do not necessarily refect the views of any funding
agencies. We are grateful for the feedback from our reviewers. We
thank The National Museum of Emerging Science and Innovationin Tokyo, Japan for providing an opportunity for the exhibition.

REFERENCES[1] 2020. NVIDIA Maxine Video Conferencing Platform. https://developer.nvidia.
com/maxine. Accessed: 2021-2-22.
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019. Image2StyleGAN: How to
Embed Images Into the StyleGAN Latent Space?. In 2019 IEEE/CVF International
Conference on Computer Vision (ICCV). 4431–4440.
[3] Matthew A J Apps, Ana Tajadura-Jiménez, Marty Sereno, Olaf Blanke, andManos Tsakiris. 2015. Plasticity in unimodal and multimodal brain areas refects
multisensory changes in self-face identifcation. Cereb. Cortex 25, 1 (Jan. 2015),
46–55.
[4] Nobumichi Asai, Hiroto Kuwahara, and Paul Lacroix. 2015. OMOTE / REALTIME FACE TRACKING & PROJECTION MAPPING. https://archive.aec.at/prix/
showmode/49140/. Accessed: 2022-9-10.
[5] Elena Azañón, Luigi Tamè, Angelo Maravita, Sally A Linkenauger, Elisa R Ferrè,
Ana Tajadura-Jiménez, and Matthew R Longo. 2016. Multimodal Contributions
to Body Representation. Multisensory Research 29, 6-7 (Jan. 2016), 635–661.
[6] Jeremy N Bailenson, Nick Yee, Scott Brave, Dan Merget, and David Koslow. 2007.
Virtual Interpersonal Touch: Expressing and Recognizing Emotions throughHaptic Devices. Hum.-Comput. Interact. 22, 3 (Aug. 2007), 325–353.
[7] Domna Banakou, Raphaela Groten, and Mel Slater. 2013. Illusory ownership ofa virtual child body causes overestimation of object sizes and implicit attitudechanges. Proceedings of the National Academy of Sciences 110, 31 (2013), 12846–
12851.
[8] Domna Banakou, Parasuram D Hanumanthu, and Mel Slater. 2016. Virtual
Embodiment of White People in a Black Virtual Body Leads to a Sustained
Reduction in Their Implicit Racial Bias. Front. Hum. Neurosci. 10 (Nov. 2016),
642–12.
[9] Sarah-Jayne Blakemore, Joel Winston, and Uta Frith. 2004. Social cognitive
neuroscience: where are we heading? Trends Cogn. Sci. 8, 5 (May 2004), 216–222.
[10] Matthew Botvinick and Jonathan Cohen. 1998. Rubber hands ‘feel’ touch thateyes see. Nature 391, 6669 (Feb. 1998), 756–756.
[11] Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology.
Qual. Res. Psychol. 3, 2 (Jan. 2006), 77–101.
[12] Scott Brave and Andrew Dahley. 1997. inTouch: A Medium for Haptic Interpersonal Communication. In CHI ’97 Extended Abstracts on Human Factors in
Computing Systems (CHI EA ’97). ACM, New York, NY, USA, 363–364.
[13] Scott Brave, Cliford Nass, and Erenee Sirinian. 2001. Force-Feedback incomputer-mediated communication. In HCI. 145–149.
[14] Ilaria Bufalari, Bigna Lenggenhager, Giuseppina Porciello, Brittany
Serra Holmes, and Salvatore M Aglioti. 2014. Enfacing others but only
if they are nice to you. Front. Behav. Neurosci. 8 (March 2014), 102.

CHI ’23, April 23–28, 2023, Hamburg, Germany[15] Ilaria Bufalari, Anna Laura Sforza, Francesco Di Russo, Lucia Mannetti, andSalvatore Maria Aglioti. 2019. Malleability of the self: electrophysiologicalcorrelates of the enfacement illusion. Sci. Rep. 9, 1 (Feb. 2019), 1682.
[16] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classifcation. In Proceedings of the 1st
Conference on Fairness, Accountability and Transparency (Proceedings of Machine
Learning Research, Vol. 81), Sorelle A Friedler and Christo Wilson (Eds.). PMLR,
77–91.
[17] S Campanella, A Chrysochoos, and R Bruyer. 2001. Categorical perception of
facial gender information: Behavioural evidence and the face-space metaphor.
Vis. cogn. 8, 2 (March 2001), 237–262.
[18] Flavia Cardini, Ana Tajadura-Jiménez, Andrea Serino, and Manos Tsakiris. 2013.
It feels like it’s me: interpersonal multisensory stimulation enhances visual
remapping of touch from other to self. J. Exp. Psychol. Hum. Percept. Perform.
39, 3 (June 2013), 630–637.
[19] Wei-Chao Chen, Herman Towles, Lars Nyland, Greg Welch, and Henry Fuchs.
2000. Toward a compelling sensation of telepresence: demonstrating a portal toa distant (static) ofce. In Proceedings of the conference on Visualization ’00 (Salt
Lake City, Utah, USA) (VIS ’00). IEEE Computer Society Press, Washington, DC,
USA, 327–333.
[20] Ned Cooper, Tifanie Horne, Gillian R Hayes, Courtney Heldreth, Michal Lahav,
Jess Holbrook, and Lauren Wilcox. 2022. A Systematic Review and Thematic
Analysis of Community-Collaborative Approaches to Computing Research.
In Proceedings of the 2022 CHI Conference on Human Factors in Computing
Systems (New Orleans, LA, USA) (CHI ’22, Article 73). Association for Computing
Machinery, New York, NY, USA, 1–18.
[21] Frédérique de Vignemont, Henrik H Ehrsson, and Patrick Haggard. 2005. Bodily
illusions modulate tactile perception. Curr. Biol. 15, 14 (July 2005), 1286–1290.
[22] Diane Deroualle and Christophe Lopez. 2014. Toward a vestibular contribution
to social cognition. , 16 pages.
[23] Martin Dobricki and Betty J Mohler. 2015. Self-Identifcation With Another’s
Body Alters Self-Other Face Distinction. Perception 44, 7 (2015), 814–820.
[24] Katharina Dobs, Isabelle Bülthof, and Johannes Schultz. 2018. Use and Usefulness of Dynamic Face Stimuli for Face Perception Studies-a Review of Behavioral
Findings and Methodology. Front. Psychol. 9 (Aug. 2018), 1355.
[25] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. 2020.
HeadGAN: One-shot neural head synthesis and editing. (Dec. 2020), 14398–
14407. arXiv:2012.08261 [cs.CV][26] Paul Ekman and Wallace V Friesen. 1978. Facial Action Coding System. Environmental Psychology & Nonverbal Behavior (1978).
[27] Paul Ekman, Wallace V Friesen, and Joseph C Hager. 2002. Facial Action Coding
System. Human Face.
[28] Alejandro J Estudillo and Markus Bindemann. 2017. Can gaze-contingent mirrorfeedback from unfamiliar faces alter self-recognition? Q. J. Exp. Psychol. 70, 5(May 2017), 944–958.
[29] Chiara Fini, Flavia Cardini, Ana Tajadura-Jiménez, Andrea Serino, and Manos
Tsakiris. 2013. Embodying an outgroup: the role of racial bias and the efect ofmultisensory processing in somatosensory remapping. Front. Behav. Neurosci. 7(Nov. 2013), 165.
[30] Daniel Freeman, Nicole Evans, Rachel Lister, Angus Antley, Graham Dunn, and
Mel Slater. 2014. Height, social comparison, and paranoia: An immersive virtual
reality experimental study. Psychiatry Res. 218, 3 (2014), 348–352.
[31] Rebecca Fribourg, Nami Ogawa, Ludovic Hoyet, Ferran Argelaguet, Takuji
Narumi, Michitaka Hirose, and Anatole Lecuyer. 2021. Virtual Co-Embodiment:
Evaluation of the Sense of Agency While Sharing the Control of a Virtual Body
Among Two Individuals. IEEE Trans. Vis. Comput. Graph. 27, 10 (Oct. 2021),
4023–4038.
[32] G Ganesh, A Takagi, R Osu, T Yoshioka, M Kawato, and E Burdet. 2014. Two is
better than one: physical interactions improve motor performance in humans.
Sci. Rep. 4 (Jan. 2014), 3824.
[33] Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormaehlen, Patrick
Perez, and Christian Theobalt. 2014. Automatic Face Reenactment. In 2014 IEEE
Conference on Computer Vision and Pattern Recognition. 4217–4224.
[34] Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. 2019.
GANFIT: Generative adversarial network ftting for high fdelity 3D face reconstruction. In 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR) (Long Beach, CA, USA). IEEE, 1155–1164.
[35] Trang Giang, Raoul Bell, and Axel Buchner. 2012. Does facial resemblanceenhance cooperation? PLoS One 7, 10 (Oct. 2012), e47809.
[36] Mar Gonzalez-Franco, Anthony Steed, Steve Hoogendyk, and Eyal Ofek. 2020.
Using Facial Animation to Increase the Enfacement Illusion and Avatar SelfIdentifcation. IEEE Trans. Vis. Comput. Graph. 26, 5 (May 2020), 2023–2029.
[37] C Martin Grewe, Tuo Liu, Andrea Hildebrandt, and Stefan Zachow. 2022. The
Open Virtual Mirror Framework for enfacement illusions : Enhancing the senseof agency with avatars that imitate facial expressions. Behav. Res. Methods (May2022).
[38] Shin Hanagata and Yasuaki Kakehi. 2018. Paralogue: A Remote Conversation
System Using a Hand Avatar which Postures are Controlled with Electrical

CHI ’23, April 23–28, 2023, Hamburg, Germany

Muscle Stimulation. In Proceedings of the 9th Augmented Human International
Conference (Seoul, Republic of Korea) (AH ’18, Article 35). Association for Computing Machinery, New York, NY, USA, 1–3.
[39] J V Haxby, E A Hofman, and M I Gobbini. 2000. The distributed human neuralsystem for face perception. Trends Cogn. Sci. 4, 6 (June 2000), 223–233.
[40] Hal E Hershfeld, Daniel G Goldstein, William F Sharpe, Jesse Fox, Leo Yeykelis,
Laura L Carstensen, and Jeremy N Bailenson. 2011. INCREASING SAVING
BEHAVIOR THROUGH AGE-PROGRESSED RENDERINGS OF THE FUTURE
SELF. J. Mark. Res. 48 (Nov. 2011), S23–S37.
[41] Ekaterina Ivanova, Axel Krause, Marie Schalicke, Franziska Schellhardt, Natalie
Jankowski, Josy Achner, Henning Schmidt, Michael Joebges, and Jorg Kruger.
2017. Let’s do this together: Bi-Manu-Interact, a novel device for studying humanhaptic interactive behavior. In 2017 International Conference on Rehabilitation
Robotics (ICORR). IEEE, London, 708–713.
[42] Tero Karras, Samuli Laine, and Timo Aila. 2021. A Style-Based Generator
Architecture for Generative Adversarial Networks. IEEE Trans. Pattern Anal.
Mach. Intell. 43, 12 (Dec. 2021), 4217–4228.
[43] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, andTimo Aila. 2019. Analyzing and Improving the Image Quality of StyleGAN.
[44] Shunichi Kasahara, Mitsuhito Ando, Kiyoshi Suganuma, and Jun Rekimoto.
2016. Parallel Eyes: Exploring Human Capability and Behaviors with ParalleledFirst Person View Sharing. In Proceedings of the 2016 CHI Conference on Human
Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association
for Computing Machinery, New York, NY, USA, 1561–1572.
[45] Shunichi Kasahara and Jun Rekimoto. 2014. JackIn: integrating frst-personview with out-of-body vision generation for human-human augmentation. In
Proceedings of the 5th Augmented Human International Conference (Kobe, Japan)(AH ’14, Article 46). Association for Computing Machinery, New York, NY, USA,
1–8.
[46] Shunichi Kasahara and Jun Rekimoto. 2015. JackIn head: immersive visual
telepresence system with omnidirectional wearable camera for remote collaboration. In Proceedings of the 21st ACM Symposium on Virtual Reality Software andTechnology (Beijing, China) (VRST ’15). Association for Computing Machinery,
New York, NY, USA, 217–225.
[47] Hiroki Kawasaki, Hiroyuki Iizuka, Shin Okamoto, Hideyuki Ando, and Taro
Maeda. 2010. Collaboration and skill transmission by frst-person perspective
view sharing system. In 19th International Symposium in Robot and Human
Interactive Communication. IEEE, Viareggio, Italy, 125–131.
[48] Elena Kokkinara and Rachel McDonnell. 2015. Animation realism afects perceived character appeal of a self-virtual face. In Proceedings of the 8th ACM
SIGGRAPH Conference on Motion in Games (Paris, France) (MIG ’15). Association
for Computing Machinery, New York, NY, USA, 221–226.
[49] Ryohei Komiyama, Takashi Miyaki, and Jun Rekimoto. 2017. JackIn space:designing a seamless transition between frst and third person view for efective telepresence collaborations. In Proceedings of the 8th Augmented Human
International Conference (Silicon Valley, California, USA) (AH ’17, Article 14).
Association for Computing Machinery, New York, NY, USA, 1–9.
[50] Keitaro Kurosaki, Hiroki Kawasaki, Daisuke Kondo, Hiroyuki Iizuka, Hideyuki
Ando, and Taro Maeda. 2011. Skill transmission for hand positioning task
through view-sharing system. In Proceedings of the 2nd Augmented Human
International Conference on - AH ’11. ACM Press, Tokyo, Japan, 1–4.
[51] Jason Lawrence, Danb Goldman, Supreeth Achar, Gregory Major Blascovich,
Joseph G Desloge, Tommy Fortes, Eric M Gomez, Sascha Häberling, Hugues
Hoppe, Andy Huibers, Claude Knaus, Brian Kuschak, Ricardo Martin-Brualla,
Harris Nover, Andrew Ian Russell, Steven M Seitz, and Kevin Tong. 2021. Projectstarline: a high-fdelity telepresence system. ACM Trans. Graph. 40, 6 (Dec.
2021), 1–16.
[52] Daniel Leithinger, Sean Follmer, Alex Olwal, and Hiroshi Ishii. 2014. Physicaltelepresence: shape capture and display for embodied, computer-mediated remote collaboration. In Proceedings of the 27th annual ACM symposium on Userinterface software and technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 461–470.
[53] Sally A Linkenauger, Markus Leyrer, Heinrich H Bülthof, and Betty J Mohler.
2013. Welcome to Wonderland: The Infuence of the Size and Shape of a Virtual
Hand On the Perceived Size and Shape of Virtual Objects. PLoS One 8, 7 (2013).
[54] Duri Long, Takeria Blunt, and Brian Magerko. 2021. Co-Designing AI Literacy
Exhibits for Informal Learning Spaces. Proc. ACM Hum.-Comput. Interact. 5,
CSCW2 (Oct. 2021), 1–35.
[55] Azumi Maekawa, Shota Takahashi, Mhd Yamen Saraiji, Sohei Wakisaka, Hiroyasu Iwata, and Masahiko Inami. 2019. Naviarm: Augmenting the Learning ofMotor Skills using a Backpack-type Robotic Arm System. In Proceedings of the10th Augmented Human International Conference 2019 on - AH2019. ACM Press,
Reims, France, 1–8.
[56] Lara Maister, Natalie Sebanz, Günther Knoblich, and Manos Tsakiris. 2013.
Experiencing ownership over a dark-skinned body reduces implicit racial bias.
Cognition 128, 2 (Aug. 2013), 170–178.
[57] Lara Maister, Mel Slater, Maria V Sanchez-Vives, and Manos Tsakiris. 2015.
Changing bodies changes minds: owning another body afects social cognition.

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara

Trends Cogn. Sci. 19, 1 (Jan. 2015), 6–12.
[58] Mara Mazzurega, Francesco Pavani, Maria Paola Paladino, and Thomas W
Schubert. 2011. Self-other bodily merging in the context of synchronous but
arbitrary-related multisensory inputs. Exp. Brain Res. 213, 2-3 (Sept. 2011),
213–221.
[59] Andrew N Meltzof. 2007. ’Like me’: a foundation for social cognition. Dev. Sci.
10, 1 (Jan. 2007), 126–134.
[60] Fritz Menzer, Anna Brooks, Pär Halje, Christof Faller, Martin Vetterli, and Olaf
Blanke. 2010. Feeling in control of your footsteps: Conscious gait monitoring
and the auditory consequences of footsteps. Cogn. Neurosci. 1, 3 (Sept. 2010),
184–192.
[61] Kouta Minamizawa, Yasuaki Kakehi, Masashi Nakatani, Soichiro Mihara, andSusumu Tachi. 2012. TECHTILE toolkit: a prototyping tool for design andeducation of haptic media. In Proceedings of the 2012 Virtual Reality International
Conference (Laval, France) (VRIC ’12, Article 26). Association for Computing
Machinery, New York, NY, USA, 1–2.
[62] Farshad Moradi, Christof Koch, and Shinsuke Shimojo. 2005. Face adaptation
depends on seeing the face. Neuron 45, 1 (Jan. 2005), 169–175.
[63] Hideyuki Nakanishi, Kazuaki Tanaka, and Yuya Wada. 2014. Remote handshaking: touch enhances video-mediated social telepresence. In Proceedings of the
32nd annual ACM conference on Human factors in computing systems - CHI ’14.
ACM Press, Toronto, Ontario, Canada, 2143–2152.
[64] Hideyuki Nakanishi, Kazuaki Tanaka, and Yuya Wada. 2014. Remote handshaking: touch enhances video-mediated social telepresence. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario,
Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA,2143–2152.
[65] Tamami Nakano and Takuto Yamamoto. 2022. You trust a face like yours.
Humanities and Social Sciences Communications 9, 1 (July 2022), 1–6.
[66] Samuel Navas Medrano, Max Pfeifer, and Christian Kray. 2020. Remote Deictic Communication: Simulating Deictic Pointing Gestures across Distances
Using Electro Muscle Stimulation. International Journal of Human–Computer
Interaction 36, 19 (Nov. 2020), 1867–1882.
[67] Jun Nishida, Shunichi Kasahara, and Kenji Suzuki. 2017. Wired muscle. In ACM
SIGGRAPH 2017 Emerging Technologies. ACM Press, New York, New York, USA,
1–2.
[68] Jun Nishida and Kenji Suzuki. 2017. bioSync: A Paired Wearable Device for
Blending Kinesthetic Experience. In Proceedings of the 2017 CHI Conferenceon Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17).
Association for Computing Machinery, New York, NY, USA, 3316–3327.
[69] Naomi Nota, James P Trujillo, and Judith Holler. 2021. Facial Signals and Social
Actions in Multimodal Face-to-Face Interaction. Brain Sci 11, 8 (July 2021).
[70] Shannon O’Brien and Florian Mueller. 2006. Holding hands over a distance technology probes in an intimate, mobile context. OZCHI (2006), 293.
[71] Maria-Paola Paladino, Mara Mazzurega, Francesco Pavani, and Thomas W Schubert. 2010. Synchronous multisensory stimulation blurs self-other boundaries.
Psychol. Sci. 21, 9 (Sept. 2010), 1202–1207.
[72] Tabitha C Peck, Sofa Seinfeld, Salvatore M Aglioti, and Mel Slater. 2013. Putting
yourself in the skin of a black avatar reduces implicit racial bias. Conscious.
Cogn. 22, 3 (Sept. 2013), 779–787.
[73] Valeria I Petkova and H Henrik Ehrsson. 2008. If I Were You: Perceptual Illusionof Body Swapping. PLoS One 3, 12 (2008), 1–9.
[74] Daniela Petrelli and Sinead O’Brien. 2018. Phone vs. Tangible in museums: A
comparative study. In CHI ’18 Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems. ACM.
[75] Bertrand Philippe, Daniel Gonzalez-Franco, Arthur Pointeau, and Christian
Cherene. 2014. The Machine to be Another - Embodied Telepresence using
human performers. In Prix Ars Electronica.
[76] Virginie Quintard, Stéphane Joufre, Maria-Paola Paladino, and Cédric A Bouquet. 2020. Getting closer: Synchronous interpersonal multisensory stimulation
increases closeness and attraction toward an opposite-sex other in female participants. Conscious. Cogn. 77 (Jan. 2020), 102849.
[77] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav
Shapiro, and Daniel Cohen-Or. 2021. Encoding in Style: a StyleGAN Encoder for
Image-to-Image Translation. In 2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR) (Nashville, TN, USA). IEEE.
[78] Karin Ryding. 2020. The Silent Conversation: Designing for Introspection andSocial Play in Art Museums. In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for
Computing Machinery, New York, NY, USA, 1–10.
[79] Mhd Yamen Saraiji, Tomoya Sasaki, Reo Matsumura, Kouta Minamizawa, andMasahiko Inami. 2018. Fusion: Full Body Surrogacy for Collaborative Communication. In ACM SIGGRAPH 2018 Emerging Technologies (SIGGRAPH ’18). ACM,
New York, NY, USA, 7:1–7:2.
[80] Alejandra Sel, Ruben T Azevedo, and Manos Tsakiris. 2017. Heartfelt Self: CardioVisual Integration Afects Self-Face Recognition and Interoceptive Cortical
Processing. Cereb. Cortex 27, 11 (Nov. 2017), 5144–5155.

Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience[81] Andrea Serino, Anna Laura Sforza, Noriaki Kanayama, Michiel van Elk, Mariia
Kaliuzhna, Bruno Herbelin, and Olaf Blanke. 2015. Tuning of temporo-occipital
activity by frontal oscillations during virtual mirror exposure causes erroneous
self-recognition. Eur. J. Neurosci. 42, 8 (Oct. 2015), 2515–2526.
[82] Anna Sforza, Ilaria Bufalari, Patrick Haggard, and Salvatore Maria Aglioti. 2010.
My face in yours: Visuo-tactile facial stimulation infuences sense of identity.
Soc. Neurosci. 5, 2 (2010), 148–162.
[83] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. 2020. InterFaceGAN:
Interpreting the Disentangled Face Representation Learned by GANs. (May2020). arXiv:2005.09635 [cs.CV][84] Sotaro Shimada, Kensuke Fukuda, and Kazuo Hiraki. 2009. Rubber hand illusion
under delayed visual feedback. PLoS One 4, 7 (July 2009), e6185.
[85] Kye Shimizu, Naoto Ienaga, Kazuma Takada, Maki Sugimoto, and Shunichi
Kasahara. 2022. Human Latent Metrics: Perceptual and Cognitive Response
Correlates to Distance in GAN Latent Space for Facial Images. In ACM Symposium on Applied Perception 2022 (TBC, USA) (SAP ’22, Article 3). Association for
Computing Machinery, New York, NY, USA, 1–10.
[86] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, andNicu Sebe. 2019. First Order Motion Model for Image Animation. In Conference
on Neural Information Processing Systems (NeurIPS).
[87] D J Simons, S L Franconeri, and R L Reimer. 2000. Change blindness in theabsence of a visual disruption. Perception 29, 10 (2000), 1143–1154.
[88] Chandan Singh, Guha Balakrishnan, and Pietro Perona. 2021. Matched sample selection with GANs for mitigating attribute confounding. (March 2021).
arXiv:2103.13455 [cs.CV][89] Keita Suzuki, Masanori Yokoyama, Shigeo Yoshida, Takayoshi Mochizuki, Tomohiro Yamada, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2017.
FaceShare: Mirroring with Pseudo-Smile Enriches Video Chat Communications.
In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery,
New York, NY, USA, 5313–5317.
[90] Ana Tajadura-Jiménez, Domna Banakou, Nadia Bianchi-Berthouze, and Mel
Slater. 2017. Embodiment in a Child-Like Talking Virtual Body Infuences Object
Size Perception, Self-Identifcation, and Subsequent Real Speaking. Sci. Rep. 7, 1
(2017), 9637.
[91] Ana Tajadura-Jiménez, Stephanie Grehl, and Manos Tsakiris. 2012. The other inme: interpersonal multisensory stimulation changes the mental representationof the self. PLoS One 7, 7 (July 2012), e40682.
[92] Ana Tajadura-Jiménez, Matthew R Longo, Rosie Coleman, and Manos Tsakiris.
2012. The person in the mirror: using the enfacement illusion to investigate theexperiential structure of self-identifcation. Conscious. Cogn. 21, 4 (Dec. 2012),
1725–1738.
[93] Ana Tajadura-Jiménez, Ludovica Lorusso, and Manos Tsakiris. 2013. Active andpassive-touch during interpersonal multisensory stimulation change self-otherboundaries. Conscious. Cogn. 22, 4 (Dec. 2013), 1352–1360.
[94] Ana Tajadura-Jiménez and Manos Tsakiris. 2014. Balancing the “inner” and the“outer” self: interoceptive sensitivity modulates self-other boundaries. J. Exp.
Psychol. Gen. 143, 2 (April 2014), 736–744.
[95] Daisuke Tajima, Jun Nishida, Pedro Lopes, and Shunichi Kasahara. 2022. Whose
Touch is This?: Understanding the Agency Trade-Of Between User-Driven
Touch vs. Computer-Driven Touch. ACM Trans. Comput.-Hum. Interact. 29, 3(Jan. 2022), 1–27.
[96] Atsushi Takagi, Gowrishankar Ganesh, Toshinori Yoshioka, Mitsuo Kawato, andEtienne Burdet. 2017. Physically interacting individuals estimate the partner’s
goal to enhance their movements. Nature Human Behaviour 1, 3 (March 2017),
1–6.
[97] Atsushi Takagi, Masaya Hirashima, Daichi Nozaki, and Etienne Burdet. 2019.
Individuals physically interacting in a group rapidly coordinate their movement
by estimating the collective goal. Elife 8 (Feb. 2019), e41328.
[98] Ryo Takizawa, Takayoshi Hagiwara, Adrien Verhulst, Masaaki Fukuoka, Michiteru Kitazaki, and Maki Sugimoto. 2021. Dynamic Shared Limbs: An Adaptive
Shared Body Control Method Using EMG Sensors. In Augmented Humans Conference 2021. ACM, Rovaniemi Finland, 10–18.
[99] Manos Tsakiris. 2008. Looking for myself: current multisensory input alters
self-face recognition. PLoS One 3, 12 (Dec. 2008), e4040.
[100] Manos Tsakiris, Matthew R Longo, and Patrick Haggard. 2010. Having a bodyversus moving your body: neural signatures of agency and body-ownership.
Neuropsychologia 48, 9 (July 2010), 2740–2749.
[101] Hitomi Tsujita and Jun Rekimoto. 2011. Smiling makes us happier: enhancing
positive mood and communication with smile-encouraging digital appliances. In
Proceedings of the 13th international conference on Ubiquitous computing (Beijing,
China) (UbiComp ’11). Association for Computing Machinery, New York, NY,
USA, 1–10.
[102] Björn van der Hoort, Arvid Guterstam, and H Henrik Ehrsson. 2011. Being
Barbie: The Size of One’s Own Body Determines the Perceived Size of the World.
PLoS One 6, 5 (2011), 1–10.
[103] Sushma Venkatesh, Haoyu Zhang, Raghavendra Ramachandra, Kiran Raja, Naser
Damer, and Christoph Busch. 2020. Can GAN Generated Morphs Threaten Face

CHI ’23, April 23–28, 2023, Hamburg, Germany

Recognition Systems Equally as Landmark Based Morphs? - Vulnerability and
Detection. In 2020 8th International Workshop on Biometrics and Forensics (IWBF).
1–6.
[104] Cordula Vesper, Ekaterina Abramova, Judith Bütepage, Francesca Ciardo, Benjamin Crossey, Alfred Efenberg, Dayana Hristova, April Karlinsky, Luke
McEllin, Sari R R Nijssen, Laura Schmitz, and Basil Wahn. 2016. Joint Action: Mental Representations, Shared Information and General Mechanisms for
Coordinating with Others. Front. Psychol. 7 (2016), 2039.
[105] Thomas Waltemate, Irene Senna, Felix Hülsmann, Marieke Rohde, Stefan Kopp,
Marc Ernst, and Mario Botsch. 2016. The impact of latency on perceptual judgments and motor performance in closed-loop interaction in virtual reality. In
Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST ’16). Association for Computing Machinery,
New York, NY, USA, 27–35.
[106] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2020. One-Shot Free-View
Neural Talking-Head Synthesis for Video Conferencing.
[107] Wen Wen. 2019. Does delay in feedback diminish sense of agency? A review.
Conscious. Cogn. 73 (Aug. 2019), 102759.
[108] D M Wolpert and Z Ghahramani. 2000. Computational principles of movement
neuroscience. Nat. Neurosci. 3 Suppl (Nov. 2000), 1212–1217.
[109] Nick Yee and Jeremy Bailenson. 2007. The Proteus Efect: The Efect of Transformed Self-Representation on Behavior. Hum. Commun. Res. 33, 3 (2007),
271–290.
[110] Shigeo Yoshida, Takuji Narumi, Tomohiro Tanikawa, Hideaki Kuzuoka, and Michitaka Hirose. 2021. Teardrop Glasses: Pseudo Tears Induce Sadness in You andThose Around You. In Proceedings of the 2021 CHI Conference on Human Factors
in Computing Systems (Yokohama, Japan) (CHI ’21, Article 508). Association for
Computing Machinery, New York, NY, USA, 1–12.
[111] Shigeo Yoshida, Tomohiro Tanikawa, Sho Sakurai, Michitaka Hirose, and Takuji
Narumi. 2013. Manipulation of an emotional experience by real-time deformed
facial feedback. In Proceedings of the 4th Augmented Human International Conference (Stuttgart, Germany) (AH ’13). Association for Computing Machinery,
New York, NY, USA, 35–42.
[112] Cha Zhang, Qin Cai, Philip Chou, Zhengyou Zhang, and Ricardo Martin-Brualla.
2013. Viewport: A Distributed, Immersive Teleconferencing System with Infrared Dot Pattern. IEEE Multimedia 20, 1 (Jan. 2013), 17–27.