International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2017)
T. Huang (Editor)

PosterCan Face Swapping Technology Facilitate
Mental Imagery Training?
Haruka Matsumura1 , Hironori Watanabe2 , Tai Chih Chen2 , Takafumi Taketomi1 , Yasuhide Yoshitake2 ,Alexandor Plopski1 , Christian Sandor1 and Hirokazu Kato1
1 Nara Institute of Science and Technology, Japan
2 National Institute of Fitness and Sports in KANOYA, Japan

Abstract
In this research, we conducted a preliminary study to investigate the effectiveness of face swapping technology for mental
imagery training. To confirm its effectiveness, we used transcranial magnetic stimulation for measuring motor evoked potential(MEP) as brain excitability during mental imagery training. In the experiment, we used three motions: wrist dorsiflexion as
an easy-to-perform motion, and pen spinning and baoding balls rotating motions as difficult motions. In each target motion,
we compared MEPs when watching own motion video, another person’s motion video, and another person’s motion video
with the face swapped with own face. The results showed that there was a difference between MEPs in difficult motion video
observations. Watching another person’s motion video with face swapping showed higher MEP than simply watching anotherperson’s video.
CCS Concepts•Computing methodologies → Mixed / augmented reality; Image processing;

1. Introduction
Mental imagery training is a motor learning method that utilizes
imagination in order to learn a particular motion. For example, insports training, athletes usually observe an expert’s motion to improve their performance during training. In the neurophysiology
research field, it is known that brain activity can be observed froma person watching a video of an athlete performing a motion, even
if the said person is not performing the actual motion [RC04]. For
this reason, mental imagery training has been known to facilitate
motor performance. In addition, it has been reported that observing
one’s own video produces stronger brain activity than observing
another person’s motion video [KBRF10]. From these physiological backgrounds, we make the hypothesis that "it becomes easy
to imagine one’s own motion by observing another person’s motion video but with the face swapped with one’s own face, andas such, face swapping facilitates athlete’s motor learning." On
the other hand, in the computer vision research field, face swapping technologies have been proposed to generate a natural faceswapping result [BKD∗ 08, DSJ∗ 11]. Recently, this technology has
been widely used in the mixed reality application domain such as
SNOW [SNO].
In this study, we investigate an application of face swapping ina mental imagery learning scenario. In order to confirm the effectiveness of face swapping, we conducted a preliminary experiment.
In the experiment, we measured motor evoked potential (MEP) asa brain activity in the motor cortex by using transcranial magnetic
c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

stimulation (TMS) method while watching videos. Differences ofMEPs are measured between own video, another person’s video,
and another person’s video with face swapping. In addition, wealso investigated a method for quantitatively evaluating the qualityof face swapping results. To the best of our knowledge, no researchhas been done yet on evaluating the quality of face swapping resultsquantitatively.
2. Face Swapping Algorithm
The purpose of the preliminary study is to confirm the effectivenessof face swapping for mental imagery training. Therefore, we use a
simple configuration; a person sits in front of a camera, and thenhe/she performs a finger or arm motion. For this reason, in theexperiment, the face swapped video is generated using a simple
algorithm. The face swapping algorithm consists of an initialization
process and a sequential process.
Initialization Process: In the initialization process, face part location differences d i between a source image (trainee’s face image)
Isrc and a target video (expert’s face image) Idst are calculated asfollows.
d i = A f it p i + b f it − q i(1)where A f it and b f it are affine transformation parameters to alignfaces in Isrc and Idst . p i and q i are facial landmark locations in Isrc
and Idst , respectively.

H. Matsumura et al. / Can Face Swapping Technology Facilitate Mental Imagery Training?
own motion(a) wrist dorsiflexion(b) pen spinning(c) baoding balls rotating

Persentage of MEP peak-to-peak value
normalized by control value [%]

300.0

250.0

200.0

ʶ4.1%

ʶ3.7%
ʶ8.3%

150.0

100.0

0.0
wrist dorsiflexion

Sequential Process: In this process, facial landmark detection[KS14], facial landmark location compensation, and image warping and composition are repeatedly executed. In the facial landmark compensation process, detected landmark locations are compensated using d i as follows.

s j is a scaling factor, which is calculated as follows.
 0
w j /w j0sj =00
h j /h jface swapping motion

ʶ16.9%

50.0

Figure 1: Observed motions.

q̄q0i = s j d i + q̄qi

another person's motion

(2)

(3)

where w j and h j are bounding box width and height of a face j, andsuperscript represents the target video. After this compensation
process, the source image Isrc is warped into the target video Idst .
In order to warp the source image, first, a face area is divided into
small regions using Delaunay triangulation for face landmarks. Isrc
is then warped into Idst using affine transformation calculated fromthe corresponded regions. Finally, warped regions are compositedusing poisson image blending [PGB03].
3. Experiment
We conducted a user study to confirm the effectiveness of faceswapping for mental imagery training. In the experiment, we measured MEPs using TMS while imagining a target motion by observing a target motion video. We used three motions as shown inFig. 1. Wrist dorsiflexion motion can be performed by all participants. Therefore, we measured MEPs for own video, another person’s video and face swapped video. On the other hand, pen spinning and baoding ball rotating motions cannot be performed by allparticipants. Therefore, we used another person’s video and faceswapped video only. The participant’s own video, another person’svideo, and face swapped video were randomly shown to participants. It should be noted that we used mirrored videos to reduce
the effect of mental rotation. The number of participants for eachmeasurement are 6 for wrist dorsiflexion, 3 for pen spinning, and4 for baoding balls rotating. MEP was obtained as a peak-to-peakvalue from averaged waveform of 15 trials. It should be noted that
we also measured a normal MEP in relaxed state. The normal MEP
is used as a control data, and then obtained MEPs are normalized by
the normal MEP. Participants tried to imagine each target motion
without actual motion by themselves.
Fig. 2 shows results of average MEPs in each motion. The results confirmed that average MEPs of face swapped video werehigher than that of another person’s motion video. Specifically,

pen spinningbaoding balls rotating

Figure 2: Average MEPs for each video observation.
there were large differences between another person’s video observation and face swapped video observation in pen spinning andbaoding balls rotating cases. From these results, it can be considered that the face is important to get good imagination in difficult
motions. In addition, according to the interview, we could confirm that the relationship between MEP values and user’s sensation.
Thus, we believe that TMS can be used to evaluate face swapping
algorithms quantitatively.
4. Conclusion
In this research, we conducted a preliminary user study to confirm
the effectiveness of face swapping in mental imagery training. We
measured MEP using TMS while watching videos from recorded
motions. From the results of the experiment, we consider that faceswapping is effective to get higher MEP. However, some participants feel unnatural sensation in the face swapped video, and their
MEPs did not increase from another person’s video. In the future,
we are planning to confirm the effectiveness of face swapping withmore participants. In addition, we are planning to develop a faceswapping algorithm that can generate self-recognizable image.
Acknowledgements
This work was supported by the MIC/SCOPE #162107006.
References[BKD∗ 08] B ITOUK D., K UMAR N., D HILLON S., B ELHUMEUR P.,
NAYAR S. K.: Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics 27, 3 (2008), 39. 1[DSJ∗ 11] DALE K., S UNKAVALLI K., J OHNSON M. K., V LASIC D.,
M ATUSIK W., P FISTER H.: Video face replacement. ACM Transactions
on Graphics 30, 6 (2011), 130. 1[KBRF10] K EYES H., B RADY N., R EILLY R. B., F OXE J. J.: My faceor yours? event-related potential correlates of self-face processing. Brainand cognition 72, 2 (2010), 244–254. 1[KS14] K AZEMI V., S ULLIVAN J.: One millisecond face alignment withan ensemble of regression trees. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (2014), pp. 1867–1874. 2[PGB03] P ÉREZ P., G ANGNET M., B LAKE A.: Poisson image editing.
ACM Transactions on Graphics 22, 3 (2003), 313–318. 2[RC04] R IZZOLATTI G., C RAIGHERO L.: The mirror-neuron system.
Annu. Rev. Neurosci. 27 (2004), 169–192. 1[SNO]

SNOW. http://snow.me/. last checked: 2017-09-19. 1

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.