GANimator: Neural Motion Synthesis from a Single Sequence
PEIZHUO LI, ETH Zurich, Switzerland
KFIR ABERMAN, Google Research, USA
ZIHAN ZHANG, The University of Chicago, USA
RANA HANOCKA, The University of Chicago, USA
OLGA SORKINE-HORNUNG, ETH Zurich, Switzerland

Fig. 1. Given a single training motion sequence of skeleton with arbitrary topology, our method learns to synthesize novel motion sequences.
We present GANimator, a generative model that learns to synthesize novel
motions from a single, short motion sequence. GANimator generates motions
that resemble the core elements of the original motion, while simultaneously
synthesizing novel and diverse movements. Existing data-driven techniques
for motion synthesis require a large motion dataset which contains the desired and specific skeletal structure. By contrast, GANimator only requires
training on a single motion sequence, enabling novel motion synthesis for avariety of skeletal structures e.g., bipeds, quadropeds, hexapeds, and more.
Our framework contains a series of generative and adversarial neural networks, each responsible for generating motions in a specific frame rate. The
framework progressively learns to synthesize motion from random noise,enabling hierarchical control over the generated motion content across varying levels of detail. We show a number of applications, including crowd
simulation, key-frame editing, style transfer, and interactive control, which
all learn from a single input sequence. Code and data for this paper are athttps://peizhuoli.github.io/ganimator.
CCS Concepts: • Computing methodologies → Motion processing; Machine learning approaches.
Additional Key Words and Phrases: neural motion processing, motion synthesis

Authors’ addresses: Peizhuo Li, ETH Zurich, Switzerland, peizli@ethz.ch; Kfir Aberman, Google Research, USA, kfiraberman@gmail.com; Zihan Zhang, The University of
Chicago, USA, zzhang18@uchicago.edu; Rana Hanocka, The University of Chicago,
USA, ranahanocka@gmail.com; Olga Sorkine-Hornung, ETH Zurich, Switzerland,sorkine@inf.ethz.ch.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
© 2022 Copyright held by the owner/author(s).
0730-0301/2022/7-ART138
https://doi.org/10.1145/3528223.3530157

ACM Reference Format:
Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga SorkineHornung. 2022. GANimator: Neural Motion Synthesis from a Single Sequence. ACM Trans. Graph. 41, 4, Article 138 (July 2022), 12 pages. https://doi.org/10.1145/3528223.3530157

1

INTRODUCTION

Generating realistic and diverse human motion is a long-standing
objective in computer graphics. Motion modeling and synthesis
commonly uses a probabilistic model to capture limited local variations [Li et al. 2002] or utilizes a large motion dataset obtainedby motion capture (mocap) [Holden et al. 2016]. Capturing datawith a mocap system is costly both during stage-setting and in
post-process (e.g., involving manual data clean-up). Motion datasets
are often limited, i.e., they lack the desired skeletal structure, body
proportions, or styles. Therefore, utilizing motion datasets often
requires non-trivial processing such as retargeting, which may potentially degrade or introduce errors in the original captured motion.
Moreover, there are no extensive datasets which contain imaginary
creatures or non-standard animals (such as the hexapedal crab in
Figure 1), which limits existing data-driven techniques.
In this work, we develop a framework that is capable of generating
diverse and realistic motions using only a single training sequence.
Our strategy greatly simplifies the data collection process while stillallowing the framework to create realistic variations and faithfully
capture the details of the individual motion sequence. Inspired by
the SinGAN model for image synthesis [Shaham et al. 2019], our key
idea is to leverage the information within a single motion sequencein order to exploit the rich data over multiple temporal and spatial
scales. The generation process is divided into several levels, similar to progressive training schemes for images [Karras et al. 2018;
Shaham et al. 2019].

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

138:2• Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung

Our framework is an effective tool for generating novel motion
which is not exactly present in the single given training sequence.
GANimator can synthesize long, diverse, and high-quality motion
sequences using only a short motion sequence as input. We showthat for various types of motions and characters, we generate sequences that still resemble the core elements of the original motion,
while simultaneously synthesizing novel and diverse movements.
It should be noted that the synthesized motion is not copied verbatim from the original sequence. Namely, patches from the original
motion do not appear in the synthesized motion.
We demonstrate the utility of the GANimator framework to generate both conditional and unconditional motion sequences that arereminiscent of the input motion. GANimator can unconditionally(i.e., driven by noise) generate motions for simulating crowds andmotion editing/mixing. In addition, GANimator is able to produce
controllable motion sequences, which are conditioned on a usergiven input (e.g., trajectory position). We are able to inject global
position and rotation of the root joint as an input for interactively
controlling the trajectory of the synthesized motion. GANimator isalso able to perform key-frame editing, which generates high-quality
interpolations between user-modified poses. Lastly, GANimator enables motion style transfer – synthesizing the style of the input
motion onto a different motion.
Our system relies on skeleton-aware operators [Aberman et al.
2020a] as a backbone for our neural motion synthesis framework.
The skeleton-aware layers provides the platform for applying convolutions over a fixed skeleton topology. Since our network trains on
a single motion sequence, we automatically adjust the operators toadhere to the structure of the input skeleton. Therefore, our system
is able to train on a wide variety of skeletal structures, e.g., bipeds,
quadropeds, hexapeds, and more. Further, for ground-inhabiting
creatures, we incorporate a self-supervised foot contact label. This
ensures proper placement of the feet on the ground plane and avoids
notorious foot sliding artifacts. We demonstrate the effectiveness of
GANimator in handling a wide variety of skeletal structures and motions, and its applicability in various motion editing and synthesis
tasks.
Achieving desirable results in this constrained scenario is highly
challenging, with existing techniques commonly producing undesirable results that fall into one of two extremes. In the first extreme,
generated results span the breadth of poses contained in the original motion sequence, but are jittery and incoherent. In the second
extreme, results are smooth, but lack variety and coverage of the
motion elements contained in the original sequence. Our proposed
technique strikes a favorable balance between these extremes, synthesizing high-quality, novel, and varied motion sequences. Our
framework produces desirable motion sequences that contain all theoriginal motion elements, while still achieving diverse and smooth
motion sequences.

2

RELATED WORK

We review various relevant works on motion generation, focusing
on human and other character animation. For an in-depth survey ofthis extensive body of literature we refer the readers to the surveys
in [Geijtenbeek et al. 2011; Mourot et al. 2021].

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

Statistical modeling of visual and motion data. Representing andgenerating complex visual objects and phenomena of stochastic
nature, such as stone and wood textures or breathing and walking
cycles with natural variations, is one of the fundamental tasks incomputer graphics. In one of the pioneer works in this area, Perlin [1985] introduced the famous Perlin noise function that canbe used for image synthesis with variations. This technique waslater applied to procedural animation synthesis [Perlin and Goldberg 1996] to achieve random variability. Instead of using additivenoise, statistical approaches [Bowden 2000; Brand and Hertzmann
2000; Chai and Hodgins 2007; Pullen and Bregler 2000] propose
using probabilistic models such as kernel-based distribution andhidden Markov models, enabling synthesis with variations via direct
sampling of the model. To capture both local and global structure,
MotionTexture [Li et al. 2002] clusters a large training dataset into
textons, a linear dynamics model capable of capturing local variation, and models the transition probability between textons. Though
the goal resembles ours, this data-hungry statistical model does not
fit in our single-motion setting. We perform an in-depth comparison
to MotionTexture regarding the ability to model both global and local variations when data is limited in Sec. 4.2. A similar idea of using
two-level statistical models is employed in multiple works [Lee et al.
2002; Tanco and Hilton 2000] for novel motion synthesis. Other statistical models based on Gaussian processes learn a compact latent
space [Grochow et al. 2004; Levine et al. 2012; Wang et al. 2007]
or facilitate an interactive authoring process [Ikemoto et al. 2009]
from a small set of examples for multiple applications, including
inverse kinematics, motion editing and task-guided character control. Lau et al. [2009] use a similar Bayesian approach to model thenatural variation in human motion from several examples. However,probabilistic approaches tend to capture limited variations in local
frames and require a large dataset for a better understanding of
global structure variations.
Interpolation and blending of examples. Another approach forexample-based motion generation is explicit blending and concatenation of existing examples. Early works [Mizuguchi et al. 2001;
Park et al. 2002; Rose et al. 1998, 1996; Wiley and Hahn 1997]use various interpolation and warping techniques for this purpose.
Matching-based works [Arikan and Forsyth 2002; Kovar and Gleicher 2004; Pullen and Bregler 2002] generate motion by matchinguser-specified constraints in the existing dataset and interpolating
the retrieved clips. Kovar et al. [2002] explicitly model the structureof a corpus of motion capture data using an automatically constructed directed graph (the motion graph). This approach enables
flexible and controllable motion generation and is widely used in interactive applications like computer games. Min et al. [2012] propose
a mixture of statistical and graph-based models to decouple variations in the global composition of motions or actions, and the local
movement variations. One significant constraint of blending-based
methods is that the diversity of the generated results is limited tothe dataset, since it is composed of interpolations and combinationsof the dataset. Further works [Heck and Gleicher 2007; Safonova
and Hodgins 2007; Zhao et al. 2009] focus on the efficiency androbustness of the algorithm but are still inherently limited in the variety of the generated results. Motion matching [Büttner and Clavet

GANimator: Neural Motion Synthesis from a Single Sequence •2015] directly finds the best match in the mocap dataset given theuser-provided constraints, bypassing the construction of a graph toachieve better realism. However, it generally requires task-specifictuning and is unable to generate novel motion that are unseen in
the dataset.
Physics-based motion synthesis. Another loosely related field is
physically-based motion generation, where the generation process runs a controller in a physics simulator, unlike kinematics approaches that directly deal with joints’ transformations. Agrawal
et al. [2013] optimize a procedural controller given a motion example to achieve physical plausibility while ensuring diversity ingenerated results. However, such controllers are hand-crafted forspecific tasks, such as jumping and walking, and are thus difficult to generalize to different motion data. Results in [Wei et al.
2011; Ye and Liu 2010] show that combining physical constraints
and statistical priors helps generate physically realistic animations
and reactions to external forces, but the richness of motion is still
restricted to the learned prior. With the evolution of deep reinforcement learning, control policies on different bodies including bipedlocomotion [Heess et al. 2017; Peng et al. 2017] and quadurpeds [Luoet al. 2020] can be learned from scratch without reference. It is also
possible to achieve high quality motion by learning from reference
animation [Peng et al. 2018]. Lee et al. [2021] propose to learn aparameterized family of control policies from a single clip of a single movement, e.g., jumping, kicking, backflipping, and are able togenerate novel motions for a different environment, target task, andcharacter parameterization. As in other works, the learned policy islimited to several predefined tasks.
Neural motion generation. Taylor and Hinton [2009] made initialattempts to model motion style with neural networks by restricted
Boltzmann machines. Holden et al. [2016; 2015] applied convolutional neural networks (CNN) to motion data for learning a motion manifold and motion editing. Concurrently, Fragkiadaki et al.
[2015] chose to use recurrent neural networks (RNN) for motion
modeling. RNN based works also succeed in short-term motion
prediction [Fragkiadaki et al. 2015; Pavllo et al. 2018], interactive
motion generation [Lee et al. 2018] and music-driven motion synthesis [Aristidou et al. 2021]. Zhou et al. [2018] tackle the problem of
error accumulation in long-term random generation by alternating
the network’s output and ground truth as the input of RNN during
training. This method, called acRNN, is able to generate long andstable motion similar to the training set. However, like many otherdeep learning based methods, it struggles when only a short training
sequence is provided, whereas we are able to address the limited
data problem. We compare our method to acRNN in Sec. 4.2. Holden
et al. [2017] propose phase-functioned neural networks (PFNN) for
locomotion generation and introduce phase to neural networks.
Similar ideas are used in quadruped motion generation by Zhang
et al. [2018]. Starke et al. [2020] extend phase to local joints to copewith more complex motion generation. Henter et al. [2020] propose
another generative model for motion based on normalizing flow.
Neural networks succeed in a variety of motion generation tasks:
motion retargeting [Aberman et al. 2020a, 2019; Villegas et al. 2018],
motion style transfer [Aberman et al. 2020b; Mason et al. 2022],
key-frame based motion generation [Harvey et al. 2020], motion

138:3

matching [Holden et al. 2020] and animation layering [Starke et al.
2021]. It is worth noting that the success of deep learning methods
hinges upon large and comprehensive mocap datasets. However,acquiring a dataset involves costly capturing steps and nontrivial
post-processing, as discussed in Section 1. In contrast, our method
achieves comparable results by learning from a single input motion
sequence.

3

METHOD

We propose a generative model that can learn from a single motion
sequence. Our approach is inspired by recent works in the image domain that use progressive generation [Karras et al. 2018] and works
that propose to train deep networks on a single example [Shaham
et al. 2019; Shocher et al. 2019]. We next describe the main building
blocks of our hierarchical framework, motion representation, andthe training procedure.

3.1

Motion representation

We represent a motion sequence by a temporal set of 𝑇 poses thatconsists of root joint displacements O ∈ R𝑇 ×3 and joint rotations
R ∈ R𝑇 ×𝐽 𝑄 , where 𝐽 is the number of joints and 𝑄 is the number ofrotation features. The rotations are defined in the coordinate frameof their parent in the kinematic chain, and represented by the 6D
rotation features (𝑄 = 6) proposed by Zhou et al. [2019], whichyields the best result among other representations for our task (seeablation study in Section 4.4).
To mitigate common foot sliding artifacts, we incorporate footcontact labels in our representation. In particular, we concatenate to
the feature axis 𝐶 ·𝑇 binary values L ∈ {0, 1}𝑇 ×𝐶 , which correspondto the contact labels of the foot joints, F , of the specific creature.
For example, for humanoids, we use F = {left heel, left toe, rightheel, right toe}. For each joint 𝑗 ∈ F and frame 𝑡 ∈ {1, . . . ,𝑇 }, the𝑡 𝑗-th label is calculated via
L𝑡 𝑗 = 1[∥FKS ([R, O])𝑡 𝑗 ∥ 2 < 𝜖],where ∥FKS ([R, O])𝑡 𝑗 ∥ 2 denotes the magnitude of the velocity ofjoint 𝑗 in frame 𝑡 retrieved by a forward kinematics (FK) operator.
FK applies the rotation and root joint displacements on the skeletonS, and 1[𝑉 ] is an indicator function that returns 1 if 𝑉 is true and 0
otherwise.
To simplify the notation, we denote the metric space of the concatenated features by M𝑇 ≡ R𝑇 ×(𝐽 𝑄+𝐶+3) . In addition, we denotethe input motion features by T ≡ [R, O, L] ∈ M𝑇 , and its correponding downsampled versions by T𝑖 ∈ M𝑇𝑖 .

3.2

Progressive motion synthesis architecture

Our motion generation framework is illustrated in Fig 2. It consistsof 𝑆 coarse-to-fine generative adversarial networks (GANs) [Goodfellow et al. 2014], each of which is responsible to generate motion
𝑆 . We denote thesequences with a specific number of frames {𝑇𝑖 }𝑖=1
𝑆
𝑆 , respectively.
generators and discriminators by {𝐺𝑖 }𝑖=1 and {𝐷𝑖 }𝑖=1
The first level is purely generative, namely, 𝐺 1 maps a random
noise 𝑧 1 ∈ M𝑇1 into a coarse motion sequence
Q1 = 𝐺 1 (𝑧 1 ),(1)

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

138:4• Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung

Fig. 2. Our progressive motion synthesis architecture. Starting with random noise that synthesize a coarse motion sequence through a generative network,
our framework progressively upsamples the motion until it reaches to the finest temporal resolution. Each level receives the output of the previous level(except for the first level) and a random noise as output an upsampled version of the input sequence. We use adversarial training, such that each generated
result is fed into a discriminator at a corresponding level.

where Q1 ∈ M𝑇𝑖 . Then, the generators in the finer levels 𝐺𝑖 (2 ≤𝑖 ≤ 𝑆) progressively upsample Q1 via
Q𝑖 = 𝐺𝑖 (Q𝑖−1, 𝑧𝑖 ),(2)

where in each level the sequence is upsampled by a fixed scaling
factor 𝐹 > 1. The process is repeated until the finest output sequence
Q𝑆 ∈ M𝑇𝑆 is generated by 𝐺𝑆 .
Note that 𝑧𝑖 ∈ M𝑇𝑖 has an i.i.d normal distribution ∼ N (0, 𝜎𝑖 )along the temporal axis while being shared along the channel axis,
and we found that 𝜎𝑖 is highly correlated with the magnitude of thehigh-frequency details generated by 𝐺𝑖 , thus, we select1𝜎𝑖 =∥↑ T𝑖−1 − T𝑖 ∥ 22,(3)
𝑍𝑖
where 𝑍𝑖 = 𝑇𝑖 (𝑄 𝐽 + 𝐶 + 3) is the number of entries in T𝑖 , and ↑is a linear upsampler with a scaling factor 𝐹 > 1. In all of ourexperiments we select 𝐹 = 4/3 and 𝑆 = 7.
3.2.1

Generator. Our generator 𝐺𝑖 contains a fully convolutional neural network 𝑔𝑖 (·) that has a few skeleton-aware convolution layers [Aberman et al. 2020a] followed by non-linear layers (see Appendix A). Since the main role of the network is to add missing
high-frequency details, we use a residual structure [He et al. 2016],
hence for 2 ≤ 𝑖 ≤ 𝑆, we get(4)

Discriminator. While discriminators in classic GAN architectures
output a single scalar indicating whether the input is classified as“real” or “fake”, such a structure in the case of a single sequencein the training data leads to mode collapse, since our generator
trivially overfits the sequence. In order to prevent overfitting welimit the receptive field of the discriminator by employing a PatchGAN [Isola et al. 2017; Li and Wand 2016] classifier, which calculates
a confidence value to each input patch. The final output of ourdiscriminator is the average value of all the per-patch confidence
values, predicted by the network.
Skeleton-aware operators. We employ skeleton-aware convolutions [Aberman et al. 2020a] as a fundamental building block in
our framework. Skeleton-aware operators require a fixed skeleton
topology that is defined by a set of joints (vertices) and an adjacency
ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

3.2.2

Loss functions.

Adversarial Loss. We train level 𝑖 with the WGAN-GP [2017] loss:
Ladv

=
+

Network components.

𝐺𝑖 (Q𝑖−1, 𝑧𝑖 ) = 𝑔𝑖 (↑ Q𝑖−1 + 𝑧𝑖 )+ ↑ Q𝑖−1 .

list (edges). Since our network operates on a single sequence we
adapt the topology to adhere the input sequence. This enables operation on any skeleton topology, and does not require retargeting
the input motion to a specific skeletal structure. To incorporate thefoot contact labels into the skeleton-aware representation, we treat
each label as a virtual joint connected to its corresponding joint
rotations vertex. In addition, due to the high correlation between
end-effectors and global positions, we add a connection between thecontact labels vertices to the vertex of the global displacements O,
where we treat the latter as another virtual joint that is connectedto the neighbors of the root joint in the kinematic chain.

EQ𝑖 ∼P𝑔𝑖 [𝐷𝑖 (Q𝑖 )] − 𝐷𝑖 (T𝑖 )2
𝜆gp EQ̂∼P∥∇𝐷𝑖 (Q̂𝑖 )∥ 2 −1 ,

(5)

𝑔ˆ𝑖

where P𝑔𝑖 is defined as the distributions of our generated samples
in the 𝑖th level, ∇ is the gradient operator and P𝑔ˆ𝑖 is the distributionof the linear interpolations Q̂𝑖 = 𝜆Q𝑖 + (1 − 𝜆)T𝑖 with 𝜆 as a uniformly distributed variable in [0, 1]. The gradient penalty term in (6)enforces Lipschitz continuity so the Wasserstein distance between
generated and training distribution can be well approximated [Gulrajani et al. 2017].
Reconstruction Loss. To ensure that the network generates variations of all the different temporal patches, and does not collapse
to generation of a specific subset of movements, we require the
network to reconstruct the input motion from a set of pre-defined
𝑆 , namely, 𝐺 (↑ T , 𝑧 ∗ ) should approximate thenoise signals {𝑧𝑖∗ }𝑖=1𝑖
𝑖−1 𝑖single training example T𝑖 at level 𝑖. To encourage the system to doso, we define a reconstruction loss
Lrec = ∥𝐺𝑖 (↑ T𝑖−1, 𝑧𝑖∗ ) − T𝑖 ∥ 1 .

(6)

Note that the noise models the variation of generated results, butduring reconstruction, we do not expect any variation. To this end,
we fix 𝑧 1∗ as a pre-generated noise for the first level and set 𝑧𝑖∗ = 0
for the other levels.

GANimator: Neural Motion Synthesis from a Single Sequence •138:5

with the loss term in Eq. (8) using the Adam optimizer [Kingma
and Ba 2014]. We used different training sequences, whose length
ranges from 140 frames to 800 frames at 30fps. It contains both artistcreated animation and motion capture data from Mixamo [2021] andTruebones [2022]. The training time is proportional to the training
sequence length, e.g., it takes about 4 hours to train our network
on a human animation sequence with around 600 frames (15,000
iterations per level).
Fig. 3. Generator structure. Our residual generator receives the sum of anoise vector and the upsampled result from previous level (in the first level
it receives only noise). It predicts the missing high-frequency details, whichis added to the input animation via skip connections.

Contact Consistency Loss. As accurate foot contact is one of the
major factors of motion quality, we predict the foot contact labels
in our framework and use an IK post-process to ensure the contact.
Since the joint contact labels L are integrated into our motion representation M, the skeleton-aware networks can directly operate
on M and learn to predict the contact label as part of the motion.
We noticed that the implicit learning of contact labels can cause artifacts in the transition between activated and non-activated contact
labels. Thus, we propose a new loss that encourages a consistency
between contact label and feet velocity. We require that in every
frame either the contact label or the foot velocity will be minimized,
via
𝑇
1 XX
∥FKS (R, O)𝑡 𝑗 ∥ 22 ·𝑠(L𝑡 𝑗 ),(7)
Lcon =
𝑇 |F | 𝑗 ∈ F 𝑡 =1
where 𝑠(𝑥) = 1/[1+exp(5−10𝑥)] is the transformed sigmoid function.
We demonstrate the effectiveness of this loss in the ablation study(Section 4.4).
3.2.3

Training. Our full loss used for training summarizes as:
L = 𝜆adv Ladv + 𝜆rec Lrec + 𝜆con Lcon .

(8)

Although each level can be trained separately, the generated samples of generators in the low levels may be over-blurred due to low
temporal resolution and the smoothing effect applied by convolutional kernel. To improve the robustness and quality of the results,
we combine every 2 consecutive levels as a block and train theframework block by block. A similar technique is also used by Hinzet al. [2021].
For a detailed description of the layers in each component andthe specific values of the hyper-parameters, we refer the reader toAppendix A.

4

EXPERIMENTS AND EVALUATION

We evaluate our results, compare them to other motion generation
techniques and demonstrate the effectiveness of various components
in our framework through an ablation study. Please refer to the
supplementary video for the qualitative evaluation.

4.1

Implementation details

Our GANimator framework is implemented in PyTorch [Paszke
et al. 2019], and the experiments are performed on NVIDIA GeForce
RTX 2080 Ti GPU. We optimize the parameters of our network

4.2

Novel motion synthesis

We demonstrate the ability of motion sequence extrapolation in
Fig. 4 and compare our method to the recent acRNN work [Zhouet al. 2018] and the classical statistical model MotionTexture [Liet al. 2002]. We quantitatively compare these methods with metrics
dedicated to a single training example in Section 4.3.
Since our network is fully convolutional, we can generate highquality motion sequences of arbitrary length given a single training
example. As a simple application, we can easily generate a crowd
animation, as shown in Fig. 5 and the accompanying video.
When only a single training sequence is provided, acRNN can
only generate a limited number of frames before converging to a
constant pose, because the lack of data leads to an overfitted modelthat is not robust to perturbation and error accumulation in RNNs,
while our fully convolutional framework does not suffer from thisissue.
MotionTexture [Li et al. 2002] automatically clusters the patchesof the training dataset into several textons and models the transition probability between textons, where each texton represents
the variation of a small segment of similar motions. MotionTexture
relies on similar but not identical patches in the dataset for modeling local variations. It constructs the global transition probability
between textons based on the frequencies of consecutive relationships of corresponding patches in the dataset. However, when onlya single training sequence is provided, we need to manually divide
the sequence into patches and specify the transition between textons. When splitting the training sequences into several textons
and manually permuting them to create global structure variations,the transitions between textons can be unnatural (see Fig. 4). It is
possible to learn a single texton for the whole training sequence
to achieve better quality, but the model creates almost zero local
variations due to limited data.

4.3

Evaluation

We discuss quantitative metrics for novel motion synthesis froma single training sequence and compare our method and existing
motion generation techniques.
Coverage. An important quantitative measurement for the qualityof our model is the coverage of the training example. Since there is
only one training example, we measure the coverage on all possible
𝐿𝑇 −𝐿+1
temporal windows W(T, 𝐿) = T𝑖 : 𝑖+𝐿−1 𝑖=1
of a given length
𝐿, where T𝑖 : 𝑗 denotes the sequence of frames 𝑖 to 𝑗 of the training
example T, and 𝐿𝑇 is the total length of T. Given a generated result
Q, we label a temporal window T𝑤 ∈ W(T, 𝐿𝑐 ) as covered if its
distance measure to the nearest neighbor in Q is smaller than anempirically chosen threshold 𝜖. The coverage of animation Q on T
ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

138:6• Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung

Training sequence

Ours

MotionTexture [2002]

acRNN [2018]
Fig. 4. We train our framework, MotionTexture [2002] and acRNN [2018] on the Gangnam style dancing sequence with 371 frames and use them to synthesize
a new sequence with 600 frames. The magnitude of velocity of the right hand is visualized with a heatmap (white - low, green - average, red - high). It can beseen that our method generates global structure variation, the poses and transitions look natural (see supplementary video) and visually similar to the training
sequence. For MotionTexture [2002], we manually pick a path between all the trained textons to generate results with similar structure. However, it can beseen that such a process result in unnatural transitions visualized by the large hand velocity and bad foot contact. The result of acRNN [2018] converges to astatic pose very quickly due to insufficient data and thus the velocity of the hand gradually vanishes.

NN(Q1, Q2 ) denotes the distance of the nearest neighbor of theanimation sequence Q1 in Q2 . It is crucial to use an appropriate
distance measure here. In our setting, we choose the Frobenius normon the local joint rotation matrices:1∥Q1 − Q𝑤 ∥ 2𝐹 ,min
(10)
NN(Q1, Q2 ) =𝐿1 Q𝑤 ∈W(Q2 ,𝐿1 )where 𝐿1 is the length of Q1 . We use joint rotations and not positions
since our model creates local variations so that location deviations
accumulate along the kinematics chain, which would cause locationbased high distance measures on visually similar patches.
We choose 𝐿𝑐 = 30, capturing local patch length of 1 second.
Similarly, the coverage of a model G(·) on T is defined by
Fig. 5. Crowd animation. Our framework trained on a single crab dancing
sequence can synthesize various novel motions that can be used to simulate
crowd and augment data for various purposes.

is defined as
Cov(Q, T) =X
1
1 [NN(T𝑤 , Q) < 𝜖] .
|W(T, 𝐿𝑐 )| T ∈W(T,𝐿 )𝑤𝑐

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

(9)

Cov(G, T) = E𝑧 Cov(G(𝑧), T).

(11)

Global diversity. To quantitatively measure the global structure
diversity against a single training example, we propose to measure
the distance between patched nearest neighbors (PNN). The idea is
to divide the generated animation into several segments, where each

GANimator: Neural Motion Synthesis from a Single Sequence •138:7

Table 1. Quantitative comparison to existing motion generation techniques.

Fig. 6. Patched Nearest Neighbor metric. The generated animation (toprow) is divided into several segments, where the length of each segment is
at least 𝑇min . Then each segment is assigned to its nearest neighbor in the
training animation (bottom row) as visualised by the color bars.

segment is no shorter than a threshold 𝑇min , and find a segmentation that minimizes the average per-frame nearest neighbor cost as
illustrated in Figure 6. For each frame 𝑖 in the generated animation
Q we match it with frame 𝑙𝑖 in the training animation T, such thatbetween every neighboring points in the set of discontinuous points{𝑖 | 𝑙𝑖 ̸= 𝑙𝑖−1 + 1} is at least 𝑇min . This is because every discontinuous
point corresponds to the starting point of a new segment. We call
𝐿 a segmentation on Q. When large global
such an assignment {𝑙𝑖 }𝑖=1
structure variation is present, it is difficult to find a close nearest
neighbor for large 𝑇min . The patched nearest neighbor is defined by,minimizing over all possible segmentations,
LPNN = min𝐿1X{𝑙𝑖 } 𝐿 𝑖=1∥Q𝑖 − T𝑙𝑖 ∥ 2𝐹 ,(12)

where Q𝑖 denotes frame 𝑖 in 𝑄 and T𝑙𝑖 denotes frame 𝑙𝑖 in T. The
PNN can be solved efficiently with dynamic programming, similar
to MotionTexture [2002]; we refer the reader to Appendix B for moredetails. In all our experiments, we choose 𝑇min = 30, corresponding
to one second of the animation.
Local diversity. Our framework synthesizes animations carrying
similar but diverse visual content compared to the training sequence.
We measure the local frame diversity by comparing every local
window Q𝑤 ∈ W(Q, 𝐿𝑑 ) of length 𝐿𝑑 to its nearest neighbor in the
training sequence T:
Llocal =X
1
NN(Q𝑤 , T).
|W(Q, 𝐿𝑑 )| Q ∈W(Q,𝐿 )
𝑤

(13)

𝑑

Similar to the definition of coverage, we use the Frobenius norm
over local joint rotation matrices. We choose 𝐿𝑑 = 15 to capture local
differences between the generated result and the training example.
We quantitatively compare our results to MotionTexture [Li et al.
2002] and acRNN [Zhou et al. 2018] using the metrics above as reported in Table 1. Since we use the nearest neighbor cost against thetraining sequence to measure diversity, high diversity score does notnecessarily imply plausible results: MotionTexture creates unnatural transitions that are not part of the training sequence, and acRNN
converges to a pose that does not exist in the training sequencealso leading to a high diversity score. It can be seen that acRNN has
limited coverage due to its convergence to a static pose, while ourmethod generates motions that cover the training sequence well.
MotionTexture trained as a single texton overfits to the training
sequence, creating little variation on both local and global scale.

MotionTexture [2002]
MotionTexture (Single)
acRNN [2018]
Ours

Coverage ↑Global Diversity

Local Diversity

84.6%
100%
11.6%
97.2%

1.03
0.21
5.63
1.29

1.04
0.33
6.69
1.19

Table 2. Quantitative comparison for ablation study.
Rec. Loss ↓W/o contact consist. loss
Euler angle
Quaternion
Full appraoch

4.04
23.6
9.25
2.85

Meanwhile, our model strikes a good balance between generating
plausible motions and maintaining diversity.

4.4

Ablation study

We evaluate the impact of motion representation, temporal receptive field, and the foot contact consistency loss on our performance.
The results are reported in Table 2 and demonstrated in the supplementary video.
Reconstruction loss. In this experiment, we discard the reconstruction loss and retrain our model. The supplementary video showsthat the quality of the motion is degraded as a result. The reconstruction loss ensures that an anchor point in the latent space can be used
to reconstruct the training sequence perfectly. Since our framework
generates novel variations of the training sequence, this anchor
point helps to stabilize the generated results. For the same reason,
the reconstruction loss reflects the quality of generated results ofour framework and we report the impact of different components
on the quality of results in Table 2.
Contact consistency loss. In this experiment we discard the contact consistency loss Lcon and retrain our model. Our framework
predicts the foot contact label that can be used to fix sliding artifacts in a post-process. Although it can be learned implicitly, i.e.,
without the contact consistency loss, the generated result contains
inconsistent global positions and contact labels, leading to unnatural leaning and transitions of contact status after the post-process.
The accompanying video shows that the contact consistency loss
promotes consistency between predicted animation and contact
labels, providing a robust fix to sliding artifacts.
Rotation representation. In this experiment we use three different representations of joint rotations to train our networks: Euler
angles, quaternions, and 6D representation [Zhou et al. 2019]. The
results are shown in the accompanying video. It can be seen that the
network struggles to generate reasonable results with Euler angles
because of the extreme non-linearity. Quaternions yield more stable
results compared to Euler angles, but the double-cover problem and

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

138:8• Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung

Table 3. Coverage measurement of motion mixing.
Training Set

Coverage of Seq. 1

Coverage of Seq. 2

Only Seq. 1
Only Seq. 2
All Seq.

100.0%
2.4%
100.0%

1.5%
100%
97.8%

the non-linearity still cause sudden undesired changes and degrade
the realism of the generated motion.
Temporal receptive field. A key component of generating diverse
and realistic motion with global variation is the choice of the temporal receptive field, which we explore in this experiment. We demonstrate the impact by training our framework on a clip of the YMCA
dance in the accompanying video. We observe that when a large
temporal receptive field is chosen, the network memorizes the global
structure, creating a static pose in the middle of generation without
being able to permute the dance sequence. When the temporal receptive field is small, the network observes only limited information
about the context and generates jittering results. The network cangenerate smooth and plausible permutations of the dance when asuitable receptive field is picked.

5

Training sequence 1

Training sequence 2

Closer to sequence 1

APPLICATIONSIn this section we showcase the utility of pure generation and userguided generation based on a single training example in different
applications, including motion mixing, style transfer, key-frame
editing, and interactive trajectory control.
Motion mixing. In addition to novel motion variation synthesis
based on a single example animation, our framework can be also
𝑁 ,trained on several animations. Given 𝑁 training animations {T𝑘 }𝑘=1
we can train a single generative network on all 𝑁 examples by
replacing the reconstruction loss in Eq. (6) with
′
Lrec
=

𝑁
1 X
∥𝐺𝑖 (↑ T𝑘𝑖−1, 𝑧𝑖𝑘∗ ) − T𝑘𝑖 ∥ 1 .
𝑁 𝑘=1

Fig. 7. The model is trained with two sequences. The first sequence (left)
contains relative static motion, the second sequence (right) contains larger
movement. We visualize the skeletal animation of our generated result (blue)
and its patched nearest neighbor (green) in corresponding sequences. It canbe seen that our result contains the content from both training sequence.

(14)

We train our model on two animation sequences of an elephant,
shown in Fig. 7 and the video. We quantitatively measure the coverage of novel motion synthesis on two training animations in Table 3.
It can be seen that the generated result of mixed training covers all
the training examples well, and the synthesized output naturally
fuses the two training sequences.
Style transfer. Our framework can perform motion style transfer
using an input animation sequence whose style is applied to thecontent of another input animation. We exploit the hierarchicalcontrol of the generated content at different levels of detail. Since
style is often expressed in relatively high frequencies, we use thedownsampling of content input T𝐶 to control the generation of
the neural network trained on style input T𝑆 for the style transfer
task. Namely, we downsample the content animation T𝐶 to thecorresponding coarsest resolution T𝐶1 and use it to replace the outputof the first level of the network trained on T𝑆 . We demonstrate ourresult in Fig. 8 and the video. It can be seen that, with a single

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

Closer to sequence 2

example, we successfully transfer the proud style, yielding a proud
walk sharing the same pace as the content input. Due to the factthat the output is generated by multi-scale patches learned fromthe style input T𝑆 , the content of T𝐶 is required to be similar to thecontent of T𝑆 , in order to generate high-quality results (e.g., both of
the inputs are walking).
Key-frame editing. Our framework can also be applied to keyframe editing (Fig. 9). We train our network on the input animation
T. The user can then manually edit some frames by changing theirposes at the coarsest level T1 , and the network produces smooth,realistic, and highly-detailed transitions between the modified keyframes, including unseen but plausible content.
Conditional generation. Our framework can generate motion whileaccounting for user-specified constraints on the motion of selectedjoints. Formally, given the constrained joints C, the user-specified
constraints C are given by Q C = {Q 𝑗 , 𝑗 ∈ C, where Q 𝑗 denotes the
motion of joint 𝑗. In the case of motion generation with trajectory
control, we set C = {root position, root orientation}.

GANimator: Neural Motion Synthesis from a Single Sequence •138:9

Fig. 10. Training of conditional generation. To construct the input for theconditional generator, we concatenate the constraints (e.g., root joint movement) sampled from a pre-trained generator to the upsampled result generated with the same condition from the previous level. We again concatenate
the constraints to the result produced by the generator and feed it to the discriminator. The discriminator ensures the correlation between constraints
and the generated result.

Style (proud)

Content

Aberman [2020b]

Ours

Fig. 8. We synthesize the proud style from the style input onto the content
input. It can be seen that our result contains the same content as the content
input while express the proud style, e.g. higher elbow position on walking.
We also show the result from [Aberman et al. 2020b] for comparison.

Input

Ours

Edited Key-frames
Fig. 9. We manually edit three key-frames (blue) in the input sequenceby changing the poses. We visualize the result of one editing, where thecharacter is made to face the audience. It can be seen that our model
follows the editing and generates a plausible result. Please refer to theaccompanying video for a complete result.

The key component of enabling conditional generation is enforcing correlation between the generated motion and the constraints.
The training process of the conditional generation is described inFig. 10. As constraints are defined as part of the motion, we use
the concatenation trick, denoted by 𝑐𝑜𝑛𝑐𝑎𝑡(Q, C), during the training
and inference time, where 𝑐𝑜𝑛𝑐𝑎𝑡(Q, C) is the result of replacing the
motion of constrained joints Q C in motion Q by the constraints C.
For the evaluation of the 𝑖-th level, given the constraints at corresponding frame rate C𝑖 , we use 𝑐𝑜𝑛𝑐𝑎𝑡(↑ Q𝑖−1, C𝑖 ) as the input of thegenerator instead of the upsampled result ↑ Q𝑖−1 from the previous
level. For training, we randomly generate C𝑖 by sampling from a
pre-trained generator mentioned above, taking the corresponding

Fig. 11. Interactive Generation. (a) Our conditional generation framework
can be conceptually simplified as a multi-layer convolutional generator thattakes user-specified constraints e.g., root joint movement, and random noise
as input to generate an animation. (b) When new constraints are given, we
concatenate them with the existing constraints and noise as the input for
the generator. In the generated result, the frames that are outside of thereceptive field of new constraints remain the same (blue area). The frames
within the receptive field of new constraints are changed and are used tocreate a smooth transition between existing and new constraints (dark cyan).
The frames complying with new constraints are generated (cyan). (c) During
a generation, we only need to keep the frames within the receptive field of
the dark cyan area, denoted by activated colors.

part of the motion and downsampling it to the corresponding frame
rate. We employ the concatenation trick again on the output of thegenerator and let Q𝑖 = 𝑐𝑜𝑛𝑐𝑎𝑡(𝐺𝑖 (𝑐𝑜𝑛𝑐𝑎𝑡(↑ Q𝑖−1, C𝑖 ), 𝑧𝑖 ), C𝑖 ) as theoutput of this stage and feed it into the discriminator. The discriminator rules out motions that do not belong to the plausible motion
distribution and forces the network to generate motions complying
with the given constraints.

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

138:10• Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung

Interactive generation. Conditional generation is particularly interesting for interactive applications, such as video games, where
the motion of a character needs to be generated online in accordance
to joystick controller inputs, for example. We demonstrate the process for exploiting a pre-trained conditional generation frameworkfor such interactive application in Fig. 11. Unlike RNNs that arestraightforward to use for interactive or real-time generation, convolutional neural networks generally require modifications, such as
causal convolution [Oord et al. 2016]. We take a different approachby exploiting the limited receptive field of convolution.
Let G(C, 𝑍 ) be the multi-level conditional generation result with
𝑆 usedconstraints C, where 𝑍 is the set of random noise {𝑧𝑖 }𝑖=1
during generation. In the interactive setting, we assume there areexisting constraints C1 and the corresponding random noise 𝑍 1 ,which generates the result Q1 = G(C1, 𝑍 1 ), as demonstrated inFig. 11(a). Given the new constraints C2 and corresponding noise𝑍 2 , we concatenate them with C1 and 𝑍 1 along the temporal axis,
denoted by C = 𝑒𝑥𝑡(C1, C2 ) and 𝑍 = 𝑒𝑥𝑡(𝑍 1, 𝑍 2 ), and generate a new
result Q = G(C, 𝑍 ). Denote the temporal receptive field of G(·) by 𝑅
and the lengths of constraints C1 and C2 by 𝐿1 and 𝐿2 , respectively.
Note that Q1 : 𝐿1 −𝑟 and Q11 : 𝐿1 −𝑟 are equal (blue area in Fig. 11(b)),
where Q𝑖 : 𝑗 denotes frame 𝑖 to 𝑗 of motion Q and 𝑟 = ⌈𝑅/2⌉ is thehalved receptive field. The sequence Q𝐿1 −𝑟 +1 : 𝐿1 is different from
Q𝐿1 1 −𝑟 +1 : 𝐿1 and creates a smooth transition to the new constraints(dark cyan area in Fig. 11(b)). The remaining part Q𝐿1 +1 : 𝐿1 +𝐿2 isthe newly generated result complying with the new constraints C2(cyan area in Fig. 11(b)).
Therefore, given the new constraints, we only need to run the
convolutional network on the concatenation of the last 2𝑟 frames ofthe existing constraints and the new constraints, as demonstrated by
the activated area in Fig. 11(c), and withhold the last 𝑟 frames fromdisplaying on the screen. The full method for interactive generation
is summarized in Algorithm 1. We demonstrate interactive trajectory
control in Fig. 12 and in the accompanying video.
Algorithm 1 Interactive Generation
C1 ← initial constraints
𝑍 1 ← inital generated noise𝑟 ← halved receptive field of G(·)
Q ← G(C1, 𝑍 1 )Display Q1 : end−𝑟
while C2 ← new constraints do
𝑍 2 ← generated noiseC ← 𝑒𝑥𝑡(C1, C2 )𝑍 ← 𝑒𝑥𝑡(𝑍 1, 𝑍 2 )Q ← G(C, 𝑍 )Display Q𝑟 +1 : end−𝑟
C1 ← Cend−2𝑟 +1 : end
𝑍 1 ← 𝑍 end−2𝑟 +1 : end
end while

6

DISCUSSION AND CONCLUSION

In this work we presented a neural motion synthesis approach thatleverages the power of neural networks to exploit the information
ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

Input motion

Interactive generation

Fig. 12. We demonstrate the interactive trajectory control on humanoidand a hexapod crab. It can be seen that our model can cope with diverse
input trajectories despite the single training sequence.

available within a single motion sequence. We demonstrated theutility of our framework on a variety of applications, including
synthesizing longer motion sequences of the same essence but withplausible variations, controlled motion synthesis, key-frame editing
and interpolation, and style transfer. Despite the fact that motion
data is irregular, we presented a neural representation and systemfor effectively learning to synthesize motion. Key to our technique
is a combination of skeleton-aware convolutional operators, which
serve as a backbone for a progressive motion synthesis framework.
While our current framework enables interactive control of thesynthesized motion trajectory, the motion essence itself has to belearned offline, in advance. In addition, our current system’s physical plausibility is limited to simple skeletal kinematics and footcontact handling. It would be interesting to incorporate higher level
physics to enable synthesis of interactions and motions in complex
environments. An interesting future direction is to explore online
motion learning through sparse demonstrations, for example using
continual learning. It is conceivable that our method could be used
for data augmentation to benefit training of elaborate frameworks
that require large datasets. It is also possible to explore applicationsof our ideas in non-skeletal settings, such as facial and other nonskeletal rigs. Our current approach does not involve the interplayof motion and shape deformation, although the latter is also an essential part of realistic character animation (e.g., geometric groundGANimator: Neural Motion Synthesis from a Single Sequence •contact for creatures like snake demonstrated in the supplementary
video), and we are keen on exploring this intersection in future
research.

ACKNOWLEDGMENTS
We thank Andreas Aristidou and Sigal Raab for their valuable inputs
that helped to improve this work. This work is supported in partby the European Research Council (ERC) Consolidator grant no.
101003104.

REFERENCES
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or,
and Baoquan Chen. 2020a. Skeleton-aware networks for deep motion retargeting.
ACM Transactions on Graphics (TOG) 39, 4 (2020), 62–1.
Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen.
2020b. Unpaired motion style transfer from video to animation. ACM Transactions
on Graphics (TOG) 39, 4 (2020), 64–1.
Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or.
2019. Learning Character-Agnostic Motion for Motion Retargeting in 2D. ACM
Trans. Graph. 38, 4 (2019), 75.
Adobe Systems Inc. 2021. Mixamo. https://www.mixamo.com Accessed: 2021-12-25.
Shailen Agrawal, Shuo Shen, and Michiel van de Panne. 2013. Diverse motion variations for physics-based character animation. In Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 37–44.
Okan Arikan and David A Forsyth. 2002. Interactive motion generation from examples.
ACM Transactions on Graphics (TOG) 21, 3 (2002), 483–490.
Andreas Aristidou, Anastasios Yiannakidis, Kfir Aberman, Daniel Cohen-Or, Ariel
Shamir, and Yiorgos Chrysanthou. 2021. Rhythm is a Dancer: Music-Driven Motion
Synthesis with Global Structure. arXiv preprint arXiv:2111.12159 (2021).
Richard Bowden. 2000. Learning statistical models of human motion. In IEEE Workshop
on Human Modeling, Analysis and Synthesis, CVPR, Vol. 2000. Citeseer.
Matthew Brand and Aaron Hertzmann. 2000. Style machines. In Proceedings of the 27th
annual conference on Computer graphics and interactive techniques. 183–192.
Michael Büttner and Simon Clavet. 2015. Motion Matching-The Road to Next Gen
Animation. Proc. of Nucl. ai 2015 (2015). https://www.youtube.com/watch?v=z_
wpgHFSWss&t=658s
Jinxiang Chai and Jessica K Hodgins. 2007. Constraint-based motion optimization using
a statistical dynamic model. In ACM SIGGRAPH 2007 papers. 8–es.
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proceedings of the IEEE International
Conference on Computer Vision. 4346–4354.
Thomas Geijtenbeek, Nicolas Pronost, Arjan Egges, and Mark H. Overmars. 2011.
Interactive Character Animation using Simulated Physics. In Eurographics 2011 State of the Art Reports, N. John and B. Wyvill (Eds.). The Eurographics Association.
https://doi.org/10.2312/EG2011/stars/127-149
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets.
Advances in neural information processing systems 27 (2014).
Keith Grochow, Steven L Martin, Aaron Hertzmann, and Zoran Popović. 2004. Stylebased inverse kinematics. In ACM SIGGRAPH 2004 Papers. 522–531.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron
Courville. 2017. Improved training of wasserstein GANs. In Proceedings of the
31st International Conference on Neural Information Processing Systems. 5769–5779.
Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. 2020. Robust
motion in-betweening. ACM Transactions on Graphics (TOG) 39, 4 (2020), 60–1.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learningfor image recognition. In Proceedings of the IEEE conference on computer vision andpattern recognition. 770–778.
Rachel Heck and Michael Gleicher. 2007. Parametric motion graphs. In Proceedings ofthe 2007 symposium on Interactive 3D graphics and games. 129–136.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne,
Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. 2017. Emergence of locomotion
behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).
Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. 2020. Moglow: Probabilistic
and controllable motion synthesis using normalising flows. ACM Transactions on
Graphics (TOG) 39, 6 (2020), 1–14.
Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. 2021. Improved
techniques for training single-image gans. In Proceedings of the IEEE/CVF WinterConference on Applications of Computer Vision. 1300–1309.
Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020. Learned
motion matching. ACM Transactions on Graphics (TOG) 39, 4 (2020), 53–1.

138:11

Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-functioned neural networks
for character control. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–13.
Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework forcharacter motion synthesis and editing. ACM Transactions on Graphics (TOG) 35, 4(2016), 1–11.
Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015. Learning motion
manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 Technical Briefs.
1–4.
Leslie Ikemoto, Okan Arikan, and David Forsyth. 2009. Generalizing motion edits withgaussian processes. ACM Transactions on Graphics (TOG) 28, 1 (2009), 1–12.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-imagetranslation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growingof GANs for Improved Quality, Stability, and Variation. In International Conference
on Learning Representations.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980 (2014).
Lucas Kovar and Michael Gleicher. 2004. Automated extraction and parameterizationof motions in large data sets. ACM Transactions on Graphics (ToG) 23, 3 (2004),
559–568.
Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion Graphs. In Proceedingsof the 29th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’02). Association for Computing Machinery, New York, NY, USA, 473–482.
https://doi.org/10.1145/566570.566605
Manfred Lau, Ziv Bar-Joseph, and James Kuffner. 2009. Modeling spatial and temporalvariation in motion data. ACM Transactions on Graphics (TOG) 28, 5 (2009), 1–10.
Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard. 2002.
Interactive control of avatars animated with human motion data. In Proceedings ofthe 29th annual conference on Computer graphics and interactive techniques. 491–500.
Kyungho Lee, Seyoung Lee, and Jehee Lee. 2018. Interactive character animation bylearning multi-objective control. ACM Transactions on Graphics (TOG) 37, 6 (2018),
1–10.
Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. 2021. Learning a family of
motor skills from a single motion clip. ACM Transactions on Graphics (TOG) 40, 4(2021), 1–13.
Sergey Levine, Jack M Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. 2012.
Continuous character control with low-dimensional embeddings. ACM Transactions
on Graphics (TOG) 31, 4 (2012), 1–10.
Chuan Li and Michael Wand. 2016. Precomputed real-time texture synthesis withmarkovian generative adversarial networks. In European conference on computer
vision. Springer, 702–716.
Yan Li, Tianshu Wang, and Heung-Yeung Shum. 2002. Motion texture: a two-levelstatistical model for character motion synthesis. In Proceedings of the 29th annual
conference on Computer graphics and interactive techniques. 465–472.
Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen.
2020. Carl: Controllable agent with reinforcement learning for quadruped locomotion. ACM Transactions on Graphics (TOG) 39, 4 (2020), 38–1.
Ian Mason, Sebastian Starke, and Taku Komura. 2022. Real-Time Style Modelling of
Human Locomotion via Feature-Wise Transformations and Local Motion Phases.
arXiv preprint arXiv:2201.04439 (2022).
Jianyuan Min and Jinxiang Chai. 2012. Motion graphs++ a compact generative model
for semantic motion analysis and synthesis. ACM Transactions on Graphics (TOG)
31, 6 (2012), 1–12.
Mark Mizuguchi, John Buchanan, and Tom Calvert. 2001. Data driven motion transitions
for interactive games.. In Eurographics (Short Presentations).
Lucas Mourot, Ludovic Hoyet, François Le Clerc, François Schnitzler, and Pierre Hellier.
2021. A Survey on Deep Learning for Skeleton-Based Human Animation. Computer
Graphics Forum (2021).
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex
Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet:
A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016).
Sang Il Park, Hyun Joon Shin, and Sung Yong Shin. 2002. On-line locomotion generation
based on motion blending. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics
symposium on Computer animation. 105–111.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan
Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library.
In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle,
A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates,
Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-stylehigh-performance-deep-learning-library.pdf
Dario Pavllo, David Grangier, and Michael Auli. 2018. Quaternet: A quaternion-based
recurrent model for human motion. arXiv preprint arXiv:1805.06485 (2018).

ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

138:12• Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. Deepmimic:
Example-guided deep reinforcement learning of physics-based character skills. ACM
Transactions on Graphics (TOG) 37, 4 (2018), 1–14.
Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. 2017. Deeploco:
Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM
Transactions on Graphics (TOG) 36, 4 (2017), 1–13.
Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3 (1985),
287–296.
Ken Perlin and Athomas Goldberg. 1996. Improv: A system for scripting interactive
actors in virtual worlds. In Proceedings of the 23rd annual conference on Computer
graphics and interactive techniques. 205–216.
Katherine Pullen and Christoph Bregler. 2000. Animating by multi-level sampling. In
Proceedings Computer Animation 2000. IEEE, 36–42.
Katherine Pullen and Christoph Bregler. 2002. Motion capture assisted animation:
Texturing and synthesis. In Proceedings of the 29th annual conference on Computer
graphics and interactive techniques. 501–508.
Charles Rose, Michael F Cohen, and Bobby Bodenheimer. 1998. Verbs and adverbs:
Multidimensional motion interpolation. IEEE Computer Graphics and Applications
18, 5 (1998), 32–40.
Charles Rose, Brian Guenter, Bobby Bodenheimer, and Michael F Cohen. 1996. Efficient
generation of motion transitions using spacetime constraints. In Proceedings of the
23rd annual conference on Computer graphics and interactive techniques. 147–154.
Alla Safonova and Jessica K Hodgins. 2007. Construction and optimal search of interpolated motion graphs. In ACM SIGGRAPH 2007 papers. 106–es.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. SinGAN: Learning a generative model from a single natural image. In Proceedings of the IEEE/CVF International
Conference on Computer Vision. 4570–4580.
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. 2019. Ingan: Capturing andretargeting the" dna" of a natural image. In Proceedings of the IEEE/CVF International
Conference on Computer Vision. 4492–4501.
Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. 2020. Local motion phases
for learning multi-contact character movements. ACM Transactions on Graphics(TOG) 39, 4 (2020), 54–1.
Sebastian Starke, Yiwei Zhao, Fabio Zinno, and Taku Komura. 2021. Neural animation
layering for synthesizing martial arts movements. ACM Transactions on Graphics(TOG) 40, 4 (2021), 1–16.
Luis Molina Tanco and Adrian Hilton. 2000. Realistic synthesis of novel human movements from a database of motion capture examples. In Proceedings Workshop on
Human Motion. IEEE, 137–142.
Graham W Taylor and Geoffrey E Hinton. 2009. Factored conditional restricted Boltzmann machines for modeling motion style. In Proceedings of the 26th annual international conference on machine learning. 1025–1032.
Truebones Motions Animation Studios. 2022. Truebones. https://truebones.gumroad.
com/ Accessed: 2022-1-15.
Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee. 2018. Neural kinematic
networks for unsupervised motion retargetting. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition. 8639–8648.
Jack M Wang, David J Fleet, and Aaron Hertzmann. 2007. Gaussian process dynamical
models for human motion. IEEE transactions on pattern analysis and machine
intelligence 30, 2 (2007), 283–298.
Xiaolin Wei, Jianyuan Min, and Jinxiang Chai. 2011. Physically valid statistical models
for human motion generation. ACM Transactions on Graphics (TOG) 30, 3 (2011),
1–10.
Douglas J Wiley and James K Hahn. 1997. Interpolation synthesis of articulated figure
motion. IEEE Computer Graphics and Applications 17, 6 (1997), 39–45.
Yuting Ye and C Karen Liu. 2010. Synthesis of responsive motion using a dynamic
model. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 555–562.
He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. 2018. Mode-adaptive neuralnetworks for quadruped motion control. ACM Transactions on Graphics (TOG) 37, 4(2018), 1–11.
Liming Zhao, Aline Normoyle, Sanjeev Khanna, and Alla Safonova. 2009. Automaticconstruction of a minimum size motion graph. In Proceedings of the 2009 ACM
SIGGRAPH/Eurographics symposium on Computer animation. 27–35.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. 2019. On the continuityof rotation representations in neural networks. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition. 5745–5753.
Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. 2018. AutoConditioned Recurrent Networks for Extended Complex Human Motion Synthesis.
In International Conference on Learning Representations.

A

NETWORK ARCHITECTURESIn this section, we describe the details for the network architectures.
Table 4 describes the architecture for our generator and discriminator network for a single level, where Conv and LReLU denote
ACM Trans. Graph., Vol. 41, No. 4, Article 138. Publication date: July 2022.

Name

Layersin/out channels

Generator

Conv + LReLU
Conv + LReLU
Conv + LReLU
Conv

𝐹 0 /𝐹 0
𝐹 0 /2𝐹 0
2𝐹 0 /2𝐹 0
2𝐹 0 /𝐹 0

Discriminator

Conv + LReLU
Conv + LReLU
Conv + LReLU
Conv

𝐹 0 /𝐹 0
𝐹 0 /2𝐹 0
2𝐹 0 /2𝐹 0
2𝐹 0 /1

Table 4. Network Architectures

skeleton-aware convolution [Aberman et al. 2020a] and leaky ReLU
activation respectively. All the convolution layers use reflectedpadding, kernel size 5 and neighbor distance 2. For simplicity, we
denote the number of input animation features 𝐽𝑄 + 3 + |F | by 𝐹 0 .
In our experiments, we use 𝜆adv = 1, 𝜆rec = 50, 𝜆con = 5 and
𝜆gp = 1.

B

SOLVING PATCHED NEAREST NEIGHBOR

In this section, we describe how to solve the patched nearest neighbor (PNN). Given the generated animation Q of length 𝐿𝑄 and thetraining animation T of length 𝐿𝑇 , we search for the correspond𝐿𝑄
ing segmentation {𝑙𝑖 }𝑖=1
for each frame in Q, such that the minimum distance of any two points in the discontinuous point set{𝑖 | 𝑙𝑖 ̸= 𝑙𝑖−1 + 1} is no less than 𝑇min .
Let 𝐷(𝑖) denotes the PNN for first 𝑖 frames, with boundary condition 𝐷(0) = 0. We can solve other 𝐷(𝑖) by
𝐷(𝑖)𝐽 (𝑖), 𝐾(𝑖)=min

𝐷(𝑗) + cost(𝑗, 𝑖, 𝑘) (15)arg min𝐷(𝑗) + cost(𝑗, 𝑖, 𝑘) (16)0≤ 𝑗 ≤𝑖−𝑇min ,1≤𝑘 ≤𝐿𝑇 −𝑖+𝑗=

0≤ 𝑗 ≤𝑖−𝑇min ,1≤𝑘 ≤𝐿𝑇 −𝑖+𝑗cost(𝑗, 𝑖, 𝑘)

=

𝑖−𝑗
X

∥Q 𝑗+𝑚 − T𝑘+𝑚 ∥ 22,

(17)

𝑚=1

where cost(𝑗, 𝑖, 𝑘) denotes the distance between Q 𝑗+1:𝑖 and T𝑘+1:𝑘+𝑖−𝑗 .
After solving 𝐷(𝑖), 𝐽 (𝑖), 𝐾(𝑖), the label {𝑙𝑖 } can be backtraced by Algorithm 2.
Algorithm 2 Backtracing Label𝑝 ← 𝐿𝑄
while 𝑝 > 0 do𝑙 𝐽 (𝑝)+1:𝑝 ← 𝐾(𝑝) + 1, 𝐾(𝑝) + 2, · · · , 𝐾(𝑝) + 𝑝 − 𝐽 (𝑝)𝑝 ← 𝐽 (𝑝)end while
2 𝐿 ).
The cost(𝑗, 𝑖, 𝑘) can be precomputed with time complexity 𝑂(𝐿𝑄𝑇
The dynamic programming for solving 𝐷(𝑖) also runs with time com2 𝐿 ). The PNN cost L
plexity 𝑂(𝐿𝑄
PNN is given by 𝐷(𝐿𝑄 )/𝐿𝑄 .
𝑇