Style Translation Filter to Change Attribute of Motion
Akihiko Yamaguchi, Shiori Sato, Kentaro Takemura, Jun Takamatsu, and Tsukasa Ogasawara
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5, Takayama, Ikoma, Nara 630-0192, JAPAN{akihiko-y, j-taka, ogasawar}@is.naist.jp
Abstract—In this paper, we propose a style translation filter
that changes the attribute (style) of the motion coming fromthe actors’ ages, genders, and so on. Using this filter, we candiversify the motions. Specifically, this filter is modeled by theGaussian process regression that estimates the difference ofpose (joint angles) between a neutral motion and the motionof a target attribute. In learning this filter, a key technique isto find pairs of corresponding posed from the sample motions.
We solve this problem by employing the Multifactor GaussianProcess Model (MGPM) proposed by Wang et al. [1]. In theexperiments, we constructed multiple style translation filtersfrom several attributes of walking motions, such as genders,ages, and emotions. The obtained filters were applied to somekinds of testing motions, such as walking, jumping, kicking, anddancing. The acquired motions were verified by a questionnaire
study; the most of their attributes were changed to the filters’
target attributes.

I. INTRODUCTION
Motion capture system is widely used as a tool for
generating motions of robots and CG (Computer Graphics)
characters because of its ease of utilization. However, inorder to generate variety of motions, we need to capture
many motions. Even in capturing a same kind of motions,
there is variation; for instance, walking motions of a childand an elder person are different. Rather than capturing
a number of motions, creating new motions from a small
number of captured motions is useful.
For such a purpose, methods to synthesize a new motion
from sample motions are studied [2]∼[6]. Broadly speaking,there are two approaches: (1) dividing motions into several
segments and producing a new motion as a sequence of thesegments (e.g. the Motion Graphs by Kovar et al. [2]), and(2) constructing a generative model from sample motions
and generating a new motion with manipulated parametersof the model (e.g. the Style Machine by Brand et al. [3]). On
the other hand, though there are few researches, it is useful
to directly model an attribute of the motions coming fromthe actors’ ages, genders, and so on, which enables us todiversify an input motion easily.
Therefore, in this paper, we propose a method to directlymodel such an attribute of motions. Specifically, we propose
the style translation filter that changes the attribute (style)of an input motion. This filter is constructed from thesample motions consisting of neutral motions and motionsof the attribute; the filter models their difference. Rather
than modeling the entire motions and their difference, we
simply model the difference between corresponding poses in
the motions. Here, a pose means the joint angles of the wholebody at a frame. In addition, the difference of speed causedby the attribute is also estimated. For an input motion, frameby frame, we calculate the difference of pose and output thesum of the input pose and the difference, then by adjusting
the cycle of the motion, a new motion is obtained.
This filter assumes that among the different kinds ofmotions, the differences of pose caused by an attribute havea similar tendency. For instance, an elder person bends at
waist during a motion, which can be seen in many kindsof motions. Under this assumption, the style translation
filter trained with a few motions is applicable to the otherkinds of motions widely. Thus, we consider that this filter
is applicable to a variety of motions that are not used toconstruct the filter; we verify that through the experiments.
A key technique in making this filter is to calculate
the differences of pose; namely, we need to find pairs ofcorresponding poses between the neutral motion and the
motion of the attribute. We refer to this problem as aphase matching problem. In order to solve this problem, we
employ the Multifactor Gaussian Process Model (MGPM)[1]. The calculated differences of pose are modeled using
the Gaussian process regression [7].
In the experiments, we construct multiple style translation
filters from motions performed by professional actors andactresses. The modeled attributes include genders, ages, andemotions; these attributes are virtually performed by theactors and the actresses. The obtained filters are applied toseveral kinds of testing motions, such as walking, running,jumping, kicking, picking, and dancing, which are verified bya questionnaire study. The experimental results demonstratethat the proposed style translation filter can convert the
attribute of a motion as the filter have been designed.
The rest of this paper is organized as follows. Section II
describes the related works. Section III proposes the styletranslation filter. Section IV describes about the experiments.
Section V concludes this paper.
II. RELATED WORK
Iwai et al. proposed a method to extract differences offeature between the dancing motions of Japanese and Latin
American [4]; the extracted features can be used for synthesizing a new motion. However, the features are extracted
manually, which is difficult to use in our scenario.
Hsu et al. proposed a style translation model that isconstructed from two motions of the same kind [5]. Thismethod models a translation using a dynamical system.

III. STYLE TRANSLATION FILTER
The style translation filter models an attribute of a motion.
Specifically, the filter is a regression model that inputs a pose,
the joint angles of the whole body at a frame, and outputs a
difference of pose between a pose in a neutral motion and apose in a motion of the attribute. We introduce MGPM [1]for calculating the differences. The calculated differences ofpose are modeled using the Gaussian process regression [7].
In this section, we first describe how to learn filter from thesample motions, and then, we describe how to apply the filter
to an input motion.
A. Learning Filter
Style translation filter is learned from a set of sample
motions. Each style translation filter is constructed for atarget attribute. We assume that each motion is labeled bya kind of motion, such as walking and jumping, and by an
attribute of motion including “neutral”. For each pair of kindand attribute, several motions are prepared in order to remove
the individuality and the experimental noise.
The learning stage consists of the following steps (Fig. 1):(1) Calculating the mean of the neutral motions and themean of the motions of the target attribute by using
MGPM respectively.
(2) Calculating the differences of pose between the twomean motions frame by frame.
Step (1) and (2) are performed for each kind of motion. Thus,
we obtain data for training the regression model. Then,

Learning Filter
1. Generate the mean motions
Input a multiple motion
g3
g2
g1

Extract the style parameters
g3
g2
g1

Generate the mean motions
g3
g2
g1
Base mean

Base 1 Base 2

Base nTarget 1 Taget 2

Target m

Base mean
Target mean

Target mean

2. Calculate the difference vector

Unification

g3
g2
g1

Joint Angle

However, the translation uses a linear time-invariant model;
our method can represent much more complex differences.
Inamura et al. proposed a method to model motions with
the Hidden Markov Models (HMMs), and synthesize a new
motion by interpolating or extrapolating the parameters of
HMMs [6]. In the CG field, Brand et al. proposed the Style
Machine which is a kind of HMM [3]. In the style machine,a variable to represent a style is introduced, which is a
generative model, so can produce a new motion by setting
a new value for the style variable. Wang et al. proposed
the Multifactor Gaussian Process Models (MGPM), where
the kernel includes style parameters that encode the style ofmotions [1]. MGPM is also a generative model, which can
synthesize a new motion.
These approaches are considered to be difficult in changing
the attribute of a motion that is not contained in the sample
motions. For instance, when we have some walking motionsof several genders as the sample motions, changing theattribute of gender of a kicking motion may produce a
strange motion, since these methods do not model an attributedirectly. Our proposed filter directly models an attribute ina frame-by-frame manner, which makes the filter applicable
to an input motion other than the sample motions used totrain the filter. In addition, the proposed filter estimates thedifference to be near zero for an input pose that is far fromthose in the sample motions, which hardly produce a strange
motion.

Target mean
Difference
vector

Base
mean

Time

Difference
vectorBase mean

3. Model the difference vector
Difference vector

Base mean

GPR

Fig. 1.

Learning the style translation filter.

(3) Training the regression model to approximate the difference from a pose in a neutral motion.
In the rest of this section, we describe the detail oflearning.
1) Sample Motions: Here, we define the set of sample
motions used for learning. We denote the set as M. Eachmotion m ∈ M has a label gm ∈ G denoting the kind of
motion, and a label sm ∈ {B, A} denoting the attribute of
motion. G denotes the set of kinds, B denotes the neutral
attribute (B comes from Base), and A denotes the target
attribute. Each motion m consists of a sequence of statesm = {ym1 , . . . , ymTm } where Tm denotes the number offrames. Each state ym consists of the translational velocityof the body link vm , the joint angles (pose) qm including therotation of the body link, and the joint angular velocity ω m ;that is ymt = (vmt , qmt , ω mt ). The rotation of the body link
and the rotation of each joint are encoded by Exponential
Map [8].
2) Calculating Mean Motion: In calculating the differences, we need to find pairs of corresponding posed fromthe sample motions, i.e. the phase matching problem. To
solve this, we first model a set of motions of kind g byusing the Multifactor Gaussian Process Model (MGPM).
These motions are categorized into two sets according to
the attribute sm . In each set of motions, its mean motion is
generated by using MGPM.
MGPM is a kind of Gaussian process regression modelthat inputs multiple latent variables and outputs an observabletarget variable. In our case, the latent variables are aninternal state x which is time-variant and a style parameterξ which is time-invariant; the target variable is a pose y.
The style parameter ξ is a low-dimensional vector whose
dimensionality is chosen for each data set. The internal
state x is constrained by a Circle Dynamics Model (CDM).
Namely, the internal state xt at time t is given by
xt = (cos θt , sin θt ),θt = θ0 + t∆θ,(1)ξ Bg = avr{ξ m | m ∈ M, gm = g, sm = B},(2)

ξAg = avr{ξ m | m ∈ M, gm = g, sm = A},(3)

Applying Filter
1. Output the difference vector for each pose of the input motion
GPR

Time

{∆qAgt | ∆qAgt = qmAg t − qmBg t , t = 1, . . . , T∆mg }.

(4)

Corresponding poses in the mean of neutral motions are usedin training the filter: {qmBg t | t = 1, . . . , T∆mg }. If multiple
kinds of motions are available as the sample motions, werepeat the same calculation for each motion kind g, and unify
the results. Let {∆qA } denote the differences of pose, and{qB } denote the corresponding poses in the mean of neutral
motions.
4) Training Filter: We construct the style translation filterby training a Gaussian process regression model with {∆qA }and {qB } where the input is a pose qB and the output is acorresponding difference ∆qA . Let f A denote the regression
model which predicts the difference for a given pose as
follows: ∆qA = f A (qB ). As the kernel of model, we employ
a Gaussian kernel:(µkq − q0 k2 )kd (q, q0 ) = exp λ −
+ b,
(5)
2
where d denotes the dimensionality of pose. The parametersof each kernel λ, µ, and b are trained by a scaled conjugate
gradient method.
In addition, the ratio of the average step-sizes is defined inorder to model the translation of motion speed. Specifically,
the step-size ratio is given as follows:

GPR

Pose

Time

GPR

Joint Angle

2. Add the difference vector to the input motion
Target style

Difference
vectorInput motion
Time

Fig. 2.

where avr denotes a function that averages the elements ofa set.
For each ξBg and ξAg , we produce a motion by MGPM
where we use the same sequence of internal states generated
by CDM of the same θ0 and ∆θ. Let mBg and mAg denotethe obtained mean motions respectively, and let T∆mg denotethe number of frames in each motion.
3) Calculating Differences of Pose: The differences ofpose are calculated from the mean motions frame by frame
as follows:

Difference
vectorJoint Angle

Difference

Input
motionJoint Angle

where θ0 denotes an initial phase, and ∆θ denotes a stepsize of phase. For each motion m, the style parameter ξ mand the parameters of CDM θ0m , ∆θm are assigned; they
are trained from the sample motions so that the likelihood is
maximized.
Next, we produce mean motions by using MGPM withaveraged style parameters. Specifically, we first average
the style parameters of the neutral motions and the styleparameters of the motions of the target attribute as follows:

Applying the style translation filter.

each frame, the difference is calculated for the pose, then is
added to the pose with a strength factor α (Fig. 2).
Let {ymin t | ymin t = (vmin t , qmin t , ω min t ), t =1, . . . , Tmin } denote the input motion. For t-th frame, thefilter is applied for the pose:
qmout t = qmin t + αf A (qmin t ),(9)where α > 0 is the strength factor to adjust the strength of
the filter (its typical value is 1). The other elements of theoutput motion are the same as those of the input motion;
the output motion can be denoted as {ymout t | ymout t =(vmin t , qmout t , ω min t ), t = 1, . . . , Tmin }. The step-size ratioδθA is applied by changing the FPS (frame per second) of
the motion.
IV. EXPERIMENTS:
MODELING THE ATTRIBUTE OF ACTING∆θB = avr{∆θm | m ∈ M, sm = B},(6)

We construct multiple style translation filters from motions
acted by professional actors and actresses. As the sample
motions for learning, we use walking motions whose attributes include genders, ages, and emotions; these attributes
are virtually performed by the actors and the actresses. Each
attribute is modeled by a separate filter. The obtained filters
are applied to some testing motions, which are verified by aquestionnaire study.
In order to obtain the motions, we use an inertial motion
capture system MVN made by Xsens co. A raw motion
is captured in 120 Hz which is down-sampled to 30 Hz
for training the filters. Each state ymt consists of the 3dimensional translational velocity of the body link vmt , the63-dimensional joint angles (pose) qmt including the rotationof the body link, and the 63-dimensional joint angularvelocity ω mt ; that is, ymt is a 129-dimensional vector.

∆θA = avr{∆θm | m ∈ M, sm = A},(7)

A. Training FiltersδθA = ∆θA /∆θB .

(8)

Every kind of the sample motions is walking, and there
are seven attributes: neutral, masculine, feminine, elderish,
childish, sad-looking, and happy-looking. The neutral motion
is performed as a natural walking by the actor or theactress. Each motion of these attributes is performed by two
actors and two actresses; they are professional, have acting

B. Applying Filter
We describe how to apply the style translation filter to an
input motion. Here, we assume that the attribute of the input
motion is neutral. The filter is applied frame by frame; forTable IS TEP - SIZES OF PHASE OF THE MEAN MOTIONS .
Attribute
NeutralMasculine
Feminine
ElderishStep-size [rad]
0.17
0.17
0.17
0.12

AttributeChildish
Sad-looking
Happy-looking

t=5

t=10

t=15

t=20

t=25

Step-size [rad]
0.19
0.11
0.18

experience more than eight years, and are aged between 23
and 36.
These motions are modeled by MGPM1 . Table I shows
the step-sizes ∆θ of phase of the mean motions. We canfind that the step-sizes of the elderish motion and the sadlooking motion are two-thirds of that of the neutral motion.
Fig. 3 shows the acquired mean motions of the attributes
where the same sequence of internal state is used in order tocompare the corresponding poses. Let us compare the motionof each attribute with the neutral motion. In the masculine
motion (Fig. 3(b)), the stride is longer, each arm opens along
the coronal plane, and the motion looks duckfooted. In thefeminine motion (Fig. 3(c)), the stride is shorter, each arm
is along the body, and the motion looks pigeon-toed. In
the elderish motion (Fig. 3(d)), the character bends at waist,swings the arms shortly, and bends the knees throughout
walking. In the childish motion (Fig. 3(e)), the character
opens the arms along the coronal plane, swings the arms
widely, and walks bending a knee widely. In the sad-looking
motion (Fig. 3(f)), the character bends the neck, swings thearms shortly, and the stride is shorter. In the happy-looking
motion (Fig. 3(g)), the character swings the arms widely, andthe stride is longer.
From these motions, we calculate the differences of pose,and train six filters. Each filter models an attribute; we referto each of them as the masculine, the feminine, the elderish,
the childish, the sad-looking, and the happy-looking filter
respectively. As the implementation of the Gaussian process
regression, we use the NETLAB toolbox for Matlab2 .

(a) The mean of neutral motions.

(b) The mean of masculine motions.

(c) The mean of feminine motions.

(d) The mean of elderish motions.

(e) The mean of childish motions.

(f) The mean of sad-looking motions.

B. Applying Filters
In this experiment, we apply the trained six filters to six
kinds of testing motions: walking, running, jumping, kicking,picking, and dancing. Note that the testing walking motion
is not included in the sample motions. Each motion is input
to the filters independently.
We acquired 36 motions; some of them are shown in Fig. 4
to 7. Fig. 4 shows an input walking motion, the output of the
masculine filter, and the output of the feminine filter. This
input motion was performed by a female subject. In eachfilter, the strength factor α was set to be 1.5. Comparing
the input and each output at the 30th frame, we find thatby the masculine filter, the pose changed to duckfooted one
1 In Section III, the algorithm is defined for a single attribute. In order tohandle multiple attributes, we use MGPM in two stages; first, the motions ofeach attribute are modeled by MGPM and their mean motion is generated,
then, the mean motions of all attributes are modeled by MGPM.
2 http://www1.aston.ac.uk/eas/research/groups/ncrg/resources/netlab/(g) The mean of happy-looking motions.
Fig. 3. Snapshots of the mean walking motions of the attributes; t denotes
a frame index.

and each arm opened along the coronal plane. On the other
hand, by the feminine filter, the pose changed to pigeon-toed.
Thus, these filters have a generalization ability to walking
motions other than the sample motions. Note that the sameconclusions are obtained in using the other filters.
Fig. 5 shows an input picking motion, the output of theelderish filter, and the output of the childish filter. This input
motion was performed by a female subject. In each filter,
the strength factor α was set to be 1.5. Comparing the input
and each output at the 40th frame, we find that by theelderish filter, the character bent at waist. By the childish

t=20

t=25

t=30

t=35

t=40

t=30

t=45

t=60

t=75

t=90

(a) Input motion.

(a) Input motion.

(b) Output motion of the masculine filter.

(b) Output motion of the sad-looking filter.
Fig. 6. Snapshots of the input and filtered kicking motions; t denotes aframe index.
t=40

t=50

t=60

t=70

t=80

(c) Output motion of the feminine filter.
Fig. 4. Snapshots of the input and filtered walking motions; t denotes aframe index.
(a) Input motion.
t=10

t=30

t=50

t=70

t=90

(a) Input motion.

(b) Filtered motion: translated to the happy style.
Fig. 7. Snapshots of the input and filtered dancing motions; t denotes aframe index.

(b) Output motion of the elderish filter.

(c) Output motion of the childish filter.
Fig. 5. Snapshots of the input and filtered picking motions; t denotes aframe index.

filter, the character swung the arms widely. The reason of
the difference of height at picking is that in this experiment,
we do not constrain the endeffector’s position.
Fig. 6 shows an input kicking motion, and the output of
the sad-looking filter. This input motion was performed by amale subject. In this filter, the strength factor α was set to be
1.5. Comparing the input and the output at the 60th frame,
we find that by the sad-looking filter, the character bent theneck and swung the leg shortly.

Fig. 7 shows an input dancing motion, and the output of
the happy-looking filter. The input motion was performed bya female subject. In this filter, the strength factor α was set
to be 1. Comparing the input and the output, we find thatby the happy-looking filter, the character swung the arms
widely and the steps was slightly big. From these results,
the applicability to various kinds of motions is verified.
C. Questionnaire StudyIn this section, we conduct a questionnaire study of the
6 input and the 36 output motions of the previous section.
The subjects consist of 10 males and 10 females who
are aged between 23 to 25. In this experiment, we ask
each subject to (1) watch the movie of each motion, then(2) rate the movie about gender (masculine/feminine), age(elderish/childish), and emotion (sad-looking/happy-looking)in five levels respectively.
Fig. 8 and 9 show the results of the questionnaire of
the 42 motions where each score is rated between −1 to+1 in five levels. In each graph, each point denotes themean of score over the subjects; in Fig. 8, the points areplotted on the gender-age graph, and in Fig. 9, the points

InputMale
FemaleElderly
Child
Sad
Happy

P

J
W

R

D
K

0.5

W

W

J

J

K
P

Age

K

D
K

R

R W

W
P

R

P
K

R
D

Walk
Run
Jump
Kick
Pick
Dance

W

P

D

D

K

J

J

0

J

W
R
J
K
P
D

D

D

Happy 1

Input
Male
FemaleElderly
Child
Sad
Happy

D
W
D

W

P

J

P

R

D

0.5

D

K

J

W

R

R
K

J

P
P

R

0

R
J

R

Walk
Run
Jump
Kick
Pick
Dance

P
J

P

J

K
K

P

W

R
K
W

W

−0.5

W
R
J
K
P
D

K
D

EmotionChild 1

−0.5
R

R
D
P

Elderly −1

−1

Male

D

K

J

P
J

W

−0.5

K

Sad −10

0.5

1FemaleGender

W

−1

−0.5

0

0.5

Male

1

FemaleGenderFig. 8. Evaluation of the degree of gender and age for input and filteredmotions.

Fig. 9. Evaluation of the degree of gender and emotion for input andfiltered motions.

are plotted on the gender-emotion graph. By comparing thepoints of the output motions with those of the input motions,
we find that the most of their attributes were changed to
the filters’ target attributes. In addition, by using an onetailed t-test, 34 out of 36 output motions differs significantly
from the corresponding input motions (p < 0.05). The
two motions that do not have significant differences are thejumping motion translated by the masculine filter and therunning motion translated by the happy-looking filter. The
reason is considered as follows: the input jumping motion
originally had a rate of gender close to masculine. About
the running motion, it is considered that in such a dynamic
motion, recognizing the effect of the happy-looking filter
was difficult. Nevertheless, our style translation filters could
generally change the attribute of the motion as the filter had
been designed.
Finally, we discuss about the side effects of the filters; thatis to say, the effects of the filter to the attributes other than thetarget attribute. From Fig. 8 and 9, we discover the following:(i) the elderish filter changed the attribute of gender towardmasculine (Fig. 8), (ii) the childish filter changed the attributeof emotion toward happy (Fig. 9), and (iii) the filters ofgender and emotion did not change the attribute of age(Fig. 8, 9). About (i) and (ii), there may be a certain relation
between being elder and being masculine, or between being
childish and being happy-looking. For instance, we may havean image of children as being happy.

the phase matching problem. The calculated differences ofpose were modeled by the Gaussian process regression [7].
In order to verify the filter, we conducted experiments
where we constructed multiple style translation filters frommotions acted by professional actors and actresses. Specifically, we modeled six attributes: genders (masculine, feminine), ages (childish, elderish), and emotions (sad-looking,happy-looking). These filters are applied to the testing motions that consist of walking, running, jumping, kicking,picking, and dancing motions. From an observation analysis
and a questionnaire study, we found that each filter changed
the attribute of the input motion as the filter had been
designed.

V. CONCLUSION
In this paper, we proposed the style translation filter that
directly models the attribute of motions. This filter changes
the attribute of an input motion, resulting in diversifying the
motions. Specifically, the filter estimates a difference causedby the attribute for each pose of the input motion, where apose means the joint angles of the whole body at a frame. In
learning the filter, we employed MGPM [1] in order to solve

ACKNOWLEDGMENT
The authors are deeply grateful to the actors and theactresses of Gekidan Enjin and Nakano Gekidan for theircooperation in our experiments.
R EFERENCES[1] J. M. Wang, D. J. Fleet, and A. Hertzmann, “Multifactor gaussian
process models for style-content separation,” in the Twenty-FourthInternational Conference on Machine Learning (ICML 2007), 2007,
pp. 975–982.
[2] L. Kovar, M. Gleicher, and F. Pighin, “Motion graphs,” in the 29th
annual conference on Computer graphics and interactive techniques
SIGGRAPH, vol. 21, 2002, pp. 473–482.
[3] M. Brand and A. Hertzmann, “Style machines,” in Proceedings of
SIGGRAPH 2000, 2000, pp. 183–192.
[4] D. Iwai, T. Felipe, N. Nagata, and S. Inokouchi, “Identification of
motion features affecting perceived rhythmic sense of virtual charactersthrough comparison of latin american and japanese dances.” Informationand Media Technologies, vol. 65, no. 2, pp. 203–210, 2011.
[5] E. Hsu, K. Pulli, and J. Popović, “Style translation for human motion,”in Proceedings of the 2005 ACM SIGGRAPH, 2005, pp. 1082–1089.
[6] T. Inamura and T. Shibata, “Interpolation and extrapolation of motion
patterns in the proto-symbol space,” in International Conference on
Neural Information Processing, 2007.
[7] C. M. Bishop, Pattern Recognition and Machine Learning. Springer,
2006.
[8] F. S. Grassia, “Practical parameterization of rotations using the exponential map,” Journal of Graphics Tools, vol. 3, pp. 29–48, March 1998.