Composable Semi-parametric Modelling for Long-range Motion Generation
Authors: Jingwei Xu
Huazhe Xu
Bingbing Ni
Xiaokang Yang
Trevor Darrell
Venue:  (2020)
Keywords: Semi-parametric
Long-range
Motion Generation

Abstract: Learning diverse and natural behaviors is one of the longstanding goal for creating intelligent characters in the animated world. In this paper, we propose ``COmposable Semi-parametric MOdelling'' (COSMO), a method for generating long range diverse and distinctive behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by combining the complementary strengths of both non-parametric techniques and parametric ones. Given the starting and ending state, a memory bank is used to retrieve motion references that are provided as source material to a deep network. The synthesis is performed by a deep network that controls the style of the provided motion material and modifies it to become natural. On skeleton datasets with diverse motion, we show that the proposed method outperforms existing parametric and non-parametric baselines. We also demonstrate the generated sequences are useful as subgoals for actual physical execution in the animated world. 

Under review as a conference paper at ICLR 2020
COMPOSABLE SEMI-PARAMETRIC MODELLING FOR LONG-RANGE MOTION GENERATION
Anonymous authors Paper under double-hlind review
ABSTRACT
Learning diverse and natural behaviors is one of the longstanding goal for creating intelligent characters in the animated world. In this paper, we propose "COmpos- able Semi- parametric MOdelling" (COSMO), method for generating long range diverse and distinctive behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by combining the complementary strengths of both non-pan para-aerrttic techniques and parametric ones. Given the start- ing and ending state, a memory bank is used to retrieve motion references that are provided as source material to deep network. The synthesis is performed by the deep network that controls the style of the provided motion material and modifies it to become natural. On skeleton datasets with diverse motion, we show that the proposed method outperforms existing parametric and non-parametric baselines. We also demonstrate the generated sequences are useful as subgoals for actual physical execution in the animated world. Please refer to our project page for more synthesised results.
INTRODUCTION
When faced with specific goal in another location, humans can effortlessly find multiple distinctive trajectories and control their body to approach the goal with diverse and natural behaviours. However, we are still at the early stage for such sophisticated controlling of simulated characters. Recent reinforcement learning approaches (Peng et al., 2018) struggle to generate diverse motion. Other methods like imitation learning (Ye & Alterovitz 2017| Alcotti & Caselli 2006 Lawitzky et al., 2012) generalize badly to large scale demonstration data. The heart of this challenge lies in that real-world human behaviours are inherently multi-modal distributed. Direct behaviour learning is difficult without access the explicit distribution of motion states.
In this paper, we take a step toward generating long-range, diverse and physically plausible motion sequences given starting and ending states, rather than learning a policy for physical simulator. Meanwhile, we expect the model could generate novel behaviour, i.e., unseen motion in training set. This has several valuable applications: (1) Synthesised vivid motion for animation production without excessive human labor. (2) Generated novel behaviours for player customization of action skills in video games. (3) Interpolated sequences as subgoals for policy training through reinforcement learning (Peng et al.] 2018).
To fulfil the above requirements, we need address the following difficulties: (1) How to guarantee both diversity and naturalness, which is usually trade-off in the domain of generation (Srivastava et al. 2015)? (2) How to achieve long-range behaviour synthesis, which stuck by error accumulation problem in many temporal modelling tasks (Denton & Fergus, 2018)? ? (3) How to generate unseen behaviours without loss of diversity or naturalness, which is hardly addressed in previous researches? As shown in Fig [ the current two main branches of motion generation methods, i.e., parametric and non-parametric, are yet to deal with these difficulties properly. Parametric (Goodfellow et al.) (014) (e.g., GAN) methods could not maintain the reality/naturalness of generated sequences, and diversity (e.g., VAE) is hard to preserve. Non-parametric methods (Haarbach et al. 2018) involving motion clip copy/paste or blending has less superiority in smooth transition based diverse reference sequences.

---

Under review as a conference paper at ICLR 2020
Parametric
Random Noise (0.1)
Direct Mapping
Visually Unnatural
Modal Collapse
Pon-Parametric
Reference Motion
Copy Paste Blending
Short-range
Unsmooth
Seri-Paramettiic
Reference
Edit Composition
Motion
Long-range
Natural & Diverse
Figure I: Comparison of different motion generation schemes. Three rows from top to bottom corre- spond to generated results of parametric, non-parana con-parmmetri and pami-paraettric modelling respectively, Parametric model like GAN is prone to generate unnatural motion (twisted body in top row). Mean- while the modal collapse is another longstanding issue for parametric model. Non-parametric model like simply copy and paste is infeasible to get visually smooth and Long-range motion (second row). On the contrary, Our proposed semi-parametric model i.e., COSMO, is able to generate long-range motion sequences with diverse behaviours.
In this work, we propose COmposable Semi-parametrics MOdelling (COSMO), which is a method that can leverage the large spectrum of motion skills from unlabeled data in a semi-parametric manner. Our method combines the complementary strengths of both non-parametric techniques and parametric ones. First, to avoid mode collapse the sequences, we initially sample reference subsequences from held-out reference set to encourage multi-modal behaviors in the generated sequence, which is mainly inspired from non-parametrie methods (Haarbach et al., 2018). Second, propose sepe-sup disentanglement model for extracting the content and style from each reference subsequence respectively. Here content refers to the characteristic of state at each time step, e.g., moving speed, direction, and general gesture of upper body, while style refers to the long-range motion pattern which keeps relative constant across whole subsequence. An embedded latent space is constructed where the learned style vectors could be combined obtain novel style feature. new subsequence is obtained by composing content and style freely from different reference subsequences. Considering that this operation only produces a single subsequence, we refer this step as local motion composition. Finally, for all new sequences, we compose them along the temporal direction in sequential order. However, from hodgepodge reference segments, is non-trivial to generate natural and meaningful behaviors. To guarantee naturalness at long-range scale, then propose to use goal conditioned bi-directional interpolation for modeling the long-range nature needed in the task. This step is regarded as global motion composition, which covers the whole temporal scale (often more than 100 steps). On two human motion datasets, we show that the proposed method outperforms existing parametric and non-parametrie baselines. We also demonstrate the generated sequences are useful as subgoals for actual physical execution in the animated world.
Our paper makes three contributions. First, we proposed COmposable Semi-parametris MOtion generation (COSMO), a method that can generate the required sequences in a semi semi-parametric, composed way. Second, we conducted experiments on CMU Mocap dataset and SFU Mocap dataset and shows our method outperforms strong baselines such as VAEs and GANs significantly. Third, we proved empirically that the generated sequences can serve as subgoals learn an actual policy.
METHOD
2.1 SEMI-PARAMETRIC MODELLING FOR MOTION GENERATION
Given starting and ending states (denoted as So and sy respectively) as inputs, goal is synthesize intermediate states that have natural transition and diverse behaviors. As illustrated in Fig[,N refer- ence subsequences (denoted R R 1, Nr) are sampled and fed into local motion composition
Phtpp://noccppcs.cmmuddd http://moccpp.sssff..rr
2

---

Under review as a conference paper at ICLR 2020
Reference
New Style Motion
Edited Reference!
Edited Reference i+1
Input Route
AALLABBARM
Metion
1614
Sag No
Reference
Gap
Pifffittl
Bi-directional Modelling
Reference 2
Local Mation
Concatenate
Style Feature
Feature
Edited Reference
Reference Motion Set
MUAIA WAY
MUMA WN
Reference N₂
Reference Motion Search
Local Motion Composition
Global Motion Composition
Figure 2: Proposed framework for motion generation. Step I: We search for reference subsequences within held-out set. Step 2: A novel subsequence is then generated based reference with local motion composition. Step 3: All edited subsequences are connected together temporal order with global motion composition.
sub-module otoc. Invalid subsequences are first filtered out with predefined requirements and please refer to appendix [6] for detailed description. &ioe is used for altering the motion style of reference subsequence R based on another sequence R imput or directly sampling a style feature from learned latent space: R &loc RR', R.), where is optional input. With this in hand, we compose 1= along temporal direction smoothly. Concretely, we generate new clip R 1 9gto (R RR) which bridges the gap between R and R The final composed sequence is obtained through concatenation along the temporal direction for all subsequences and generated clips.
edited subsequences [R' ] Ni
Why semi- panamerareetiic modelling? Regarding both motion category and transition dynamics (from current next state), human behaviors are intrinsically multi-modal distribution (Holden et al. 2017). Parametric models (e.g., VAE (Kingma & Welling 2014) or GAN (Goodfellow et a (014)) assume prior distribution to be uni-modal and predetined. which is not suitable to handle real world motion sequent
sequences [Kingma & Welling 2014 Arora & Zhang 2017). Non Non-parametrie model does not require explicit prior distribution but not able to deal with congrrange and highly diverse motion properly, mainly restricted by modelling capability. We also conduct targeted comparison experiments in Section [ Compared parametric methods, semi semi-earman parameeraitiic modelling naturally guarantees motion diversity by introducing reference sequence also maintaining large degree of freedom for generation. Moreover, -arrmmraretrticc modelling does not require access to model of data distribution. implies that our method is also directly applicable for unseen motion sequence.
Constrained by starting and ending states with unchangeable locations, sample reference subsequences for generation. To construct complete motion sequence, we execute a manually designed procedure for searching reference subsequences. For detailed description, please refer to appendix
feasible randomly
2.2 LOCAL MOTION COMPOSITION WITH CONTENT/STYLE DISENTANGLEMENT
Local motion composition stands for generating relative short-range of motion clip based on reference subsequences as inputs. Content and style features are learned jointly, whose free-form composition is used for synthesising new motion sequence. We first describe the motion style modelling part.
Motion Style Modelling. Style is considered as motion information which stays the same throughout the whole subsequence. To this end, we extract style pattern by conducting feature fusion along the temporal direction. More specifically, for sequence with T' frames, we gradually fuse all frames into one constant feature with C, channels (denoted as h2). We adopt 1D convolution along the temporal direction for each operation. Three convolution layers stacked together with kernel size of (T/2, T /4, T/8) respectively, where large kernel size facilitates capturing motion style throughout the whole subsequence.
3

---

Under review as a conference paper at ICLR 2020
To get better generalization ability for unseen reference or starting/ending states, propose to construct an embedding space for style pattern modelling. More specifically, a set of embedding vectors (denoted as H, [h ===1) are learned together with above convolution layers (denoted as Patyy). The corresponding outputs (denoted as A. E R² of Vsty a family of coefficients for weighted sum of H. This is mainly inspired by the intuition that complex human activity could be considered combination of several simple actions. When trained with sufficiently large motion data, the model tends to learn basic style patterns for human motion. Compared to directly outputting feature vector, this helps better generalize to unseen motion during testing. The final style feature (denoted as f, E RC*) is generated as follows:
f Pasty R H
(2.1)
Motion Content Modelling. In this part, we focus on motion dynamics reflected by each single state. To this end, ID convolution layer with kernel size of 3, i.e., covering consecutive 3 states, is adopted for content modelling. The output channel of this layer is designed less than original motion dimension for extracting the most relevant information. This is empirically set to 5 as design choice. Meanwhile, velocity, position for root joint and step pattern for foot joint are fed as inputs. We denote all the above factors f. with T steps and C. channels. During local motion composition, style feature f, is combined with content feature f. through concatenating along the channel axis:
f = vent (R"), R = Dire (f,, f.),
(2.2)
where f, is tiled I'' times to match with f. and thee outputs reconstructed R three follows stacked three layer ID convolution operation with kernel size of 3. Finally, reconstruction loss is used to learn the style and content feature jointly as follows:
R rec
1
(2.3)
Nn
2.3 GLOBAL MOTION COMPOSITION VIA GOAL CONDITIONED BI-DIRECTIONAL MODELLING
Global motion composition is connecting edited short-range subsequences into a completed and long-range one. Generating smooth and natural transition between R' and R is critical to obtain a long-range sequence. The major challenge comes from large states variation between R and R Inspired by recent work of Bi-L.STM modelling (Ma & Hovy, 2016), bi-directional motion composition is utilized for this part. More specifically, motion states are first mapped to higher dimensional space with Penc and Pdec. which is defined as follows:
f Penee(r (1)). F. -Pdec
(2.4)
Both Senc and (dee are ID convolution layers. Then we predict the possible states both directions with fJ, where higher dimension (256) provides sparser space enabling better composition results. For forward prediction, we take - as inputs to obtain forward states fia+1 Mcanwhile, f++1 are fed as inputs treated as goal conditions. We conduct this procedure in recurrent manner with P-step prediction in total. Predicted pia+1 is concatenated with last four states forming next step inputs. Backward direction takes exactly the reverse procedure, i.e., p are treated as inputs while fi.s are as goal:
pis+1 p+1
Prre (fpp 115 p+1 = Popree (ppp:p -rr,
(2.5)
where frre and Pbpre share the same architecture.. Moreover, we utilize another model to produce coefficients for the weighted sum of outputs. In this way, we are able to construct more flexible latent space to facilitate composition:
= Peom (Pcom(piiti (pi:t+1 pi+1s
p.-p,,) p), f,. cfti3t1 + (1 -
(2.6)
Poom is one layer ID convolution with sigmoid function for producing More specifically
introduce and linearly map it to á E (- and + 0.1 * (ccono((P5tt) rf=)))) During training, we randomly select a motion clip with length of 10 + P. First as well as last 5 states are fed as inputs to get intermediate P states. Reconstruction loss is used for training:
C
p SP - 1lte - -Pedec (f
(2.7)
4

---

Under review as a conference paper at ICLR 2020
Compared with goal-conditioned single forward prediction model, the starting and ending states are treated equally in our model, which avoids drifting away from the ending state. Meanwhile, provide corresponding empirical results in Section 4. After global motion composition, final composed sequence is directly obtained by concatenating all edited subsequences (local part) as well as transition clips (global part) along the temporal direction.
We use tensorflow (Abadi et al. 2015) to implement all our models. All activation function used in our model is ReLU operation. During training, learning rate is set le-4 and optimized with Adam optimizer (Kingma & Bal 2015). 31 and 32 are set as 0.9 and 0.999 respectively. All models are trained with 30 epochs in total. Note that the local and global motion composition submodule do not need joint training. They are used jointly during testing.
3 RELATED WORK
Motion Interpolation. Given start and end states, this task aims to synthesize intermediate states
which smoothly translate between them (Urtasun et al. 2008). j. For video interpolation (Liu et al.) 2017) where start and end states two
2017 Li et al. 2019 Meyer et al., 2015: Niklaus et al.
consecutive frames, the final result is expected to increase frame rate of original video to higher value. Previous researches often utilize phase dynamics (Meyer et al. 2015), flow based feature (Liu ct al. 2017) and other motion information (Niklaus et al.) (017) to facilitate this task. Our work is different from this branch of work because there exists large motion gap between start and end states in our settings. Another branch of work is video completion (Cai et al. 2018 Li et al. 2019 Wexler jet al. 2007). receives two nonconsecutive frames as input and aims to fill the motion gap between start and end states. Cai et al. (2018) firstly attempts to solve this task and more specifically, propose to select out a rational path in the latent space with BFGS (Byrd et al. 1995) algorithm. Li et al. (2019) incorporates the 3D convolution layers and LSTM network into unified model, which tries to automatically find the optimal results for intermediate frames. Despite much progress has been made in this filed, the high dimensional data (i.e., video frames) severely restricts video completion within simple and seen motion categories. However, we do not limit the start and end states belonging to the same sequence. Meanwhile, we expect the interpolated sequence as diverse as possible meanwhile with natural transition between synthesised states. This has not been deeply addressed in previous motion completion works (Xia al. 2019 Lee et al. 2019). As a potential downstream application. our model could be used to construct motion planning (Myers) 1983) algorithm. Compared to gral-driven RL (Kulkarni et al. 2016 Lee et al. 2019), our model gets rid of requirements hard to achieve, i.e., known dynamics of agent, which is more general and applicable to more motion planning scenarios.
Temporal Data Prediction. There have been many researches focusing on temporal data prediction. This task aims to infer the future possible states conditioned on history states as input. Video predic- tion (Srivastava et al., 2015) takes a major part in this field, which involves pixel-wise forecasting for every following frames. Previous methods (Denton & Fergus 2018 Babacizadch et al. 2018 Denton & Birodkar 2017: Finn et al. 2016) have made it to produce high quality prediction
bouncing
digits (Srivastava et al. 2015), ), robot motion (Finn et al. 2016) and semantic map (Jin et al. 2017)
However, recent work (Xu al. 2018a) still encounters much difficulties in forecasting complex movements involving articulated subject (Wichers et al. 2018). Moreover, these works are prone to
have motion blur (Finn et al. 2016)) and error accumulation (Denton & Fergus) 2018) problems. Out of above concerns it is more appropriate for us to formulate our method as interpolation instead of prediction model. In terms of lower dimension data, human pose (Sun al.) 2019 Zhang et al. (019)) as well as path trajectory (Xu et al. (0188) are also hot spots in this task. Several works adopt proba- bilistic Bayesian model (Bhattacharyya al., 2019) to dig out latent factors which influence future dynamics. Another branch of researches (Gupta et al. 2018 Xu et al.] 2018b) utilize deep recursive model (e.g., LSTM Hochreiter Schmidhuber 1997) ) to extract critical feature for prediction. Low dimension data enables all these works could easily scale to multi-target settings (Zhao et al., 2019). However, prediction model loses its power when encountering out of distribution data (e.g., unseen motion category in test set). As comparison, unseen motion facilitated by semi comi-parametri modelling, which gets rid of explicit representation for future dynamics.
interpolation model could naturally generalize to
5

---

Under review as a conference paper at ICLR 2020
Generative Semi-parametric Modelling, For generative task such image synthesis (Qi et al. 2018), translation (Wang et al. 2019) and inpainting [Iskakov] 2018). parameetii modelling achieves considerably more realistic visual quality and better style consistency between source and target images. Semi-parametric modelling is also utilized in reinforcement learning (Kulkarni et al. 2016) for navigation task (Eysenbach et al.). More specifically, it searches rational path on replay buffer, which enables agents to solve sparse reward tasks over one hundred steps. Wang et al. (2017) combines semi-parametric modelling and VAE for large-scaled motion affordance. In this work we use semi- semi-paramette parametric modelling for two main reasons: (1) it could substantially increase the length of interpolated sequence through acting as example guidance, (2) it enables our model to synthesise visually appealing results even with out of distribution inputs. Meanwhile, we would like to highlight that to our best knowledge this is the first attempt to use semi-parametric modelling on motion interpolation task.
Motion Generation in Computer Graphics. In the context of computer graphics, there a branch of researches (Kovar & Gleicher 2003 Park et al. [002 Levine et al. 2012 Tan & Tai 2012 Holden ct al. 2016) which also concentrate on motion generation, i.e., obtaining a continuous trajectory from discrete set of poses. HHarbbcch et al. (2018) provides comprehensive study this topic. analyzes the characteristics of higher order rigid body motion interpolation methods. Our works shares similar target with this branch of work. However,
would like to emphasise that these works (Tan & Tail 2012] Kovar & Gleicher| 20033] Park et al. (2002) are in parallel with ours and have completely different research routine on this task. More specifically, graphics methods focus on finding a optimal and explicit mathematical solution regardless of input motion sequences. Different from them, our work is data-driven and encourages both reality and diversity of interpolated results.
Imitation Learning, Imitation learning (Pomerleau 1989 Ye & Alterovitz 2017 Lawitzky et al.,
2012) is commonly adopted as standard method in the domain of robotics and many other areas. Behavioral Cloning (Osa et al. 2018) is one of the underlying approaches that utilize a demonstrations as supervisory signal. The most relevant work to ours is (Peng et al., 2018) which leverages reinforcement learning to imitate natural motions. However, our method does not imitate specific motion trajectories but to generate natural and diverse actions that are reasonable between given states. Our method can be further differentiated from past literature in two aspects: (1) our method is orthogonal to imitation learning because interpolates states rather than predicts actions when states given. More importantly, (2) the proposed method is able to produce between unseen states while imitation learning focuses on imitating and capturing the demonstrated distribution.
4 EXPERIMENTS
4.1 EVALUATION SETTINGS
We use CMU Mocap P and SFU Mocap datasets for evaluation. Both datasets contain diverse daily human motion sequences which are suitable for training. Considering that the original sequences possess different length, we get reference subsequence a sliding window manner, where T 120 and P === 40. For both datasets the dimension of state is 63, which is 21 joints with 3D coordinates. Here we would like to emphasise that we keep a held-out reference set (denoted as DR)) from training data (denoted Dr) for further testing, to demonstrate that our model could generate novel behaviour never seen during training. More specifically, DR is used for sampling novel motion and starting/ending states during testing. Note that the following all experiments are with SFU Mocap datasets. For the visual results on CMU Mocap datasets, please refer to our project page.
For local motion composition, we compare our model with two strong parametric baselines: VAE (Kingma & Welling) 2014) and GAN (Goodfellow et al. 2014). For fair comparison. we train both models with all data, i.e., Dr and DR. Meanwhile we follow Yan et al. (2018) and [Bar- soum et al. (2018) about the hyper-parameter setting of both models. But the input dimension is adjusted match our data. For global motion composition, we compare our model with temporal prediction baselines: (1) Baselinel: fpre without last 5 states as goal condition, (2) Baseline2: fpre with last states as goal condition, (3) Baseline3: both 4 fre and Pbpre but without last/first
"http://noccp.cs.ccm..ee http://moocup.esscfu..a
6

---

Under review as a conference paper at ICLR 2020
tsne For embedded style feature
train set
40
0.32
GAN
COSMO
20
0.50
0.22
20
0.15
-00
10%
30%
50%
100%
-40
-20
20
Figure 3: Motion diversity evaluation.
Figure 4: Style feature visualization
COSMO
GAN
VAE
Figure 5: Comparison of motion naturalness with VAE and GAN based models. Best view in color.
5 states as goal condition. All three baselines are trained with the same hyper--arameee setting as our global composition model.
4.2 EVALUATION FOR LOCAL MOTION COMPOSITION
Is generated motion diverse enough? We compare our model with parametric model (i.e., VAE (Kingma & Welling) 2014) and GAN (Goodfellow et al.) 2014)) in evaluation of motion diversity. For the majority of generative models, the diversity of generated result is upper bounded by training data Barratt Sharma (2018). Motivated by this, we compare motion diversity with VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al. 2014) based models under different percentages data used for training. As illustrated in Fig 10% 30,, 50% and 100% training data used respectively. After training converged, joints with a higher value indicating more diverse. We can see that both parametric models keep relative constant motion diversity which is comparable with training data used less than 30%, but largely inferior to the diversity of full training data. On the contrary, with an increasing percentage of data used for training, our model achieves higher motion diversity, which mainly benefits from the general semi parameeraetticc modelling framework.
calculate the averaged standard diviation of all
generated motion visually natural? For evaluation of visual naturalness, we provide generated motion results and compare with VAE (Kingma & Welling 2014) and GAN (Goodfellow et al.) 2014). As shown Fig 5 from top to bottom generated motion sequences correspond to our model (COSMO),
GAN (Goodfellow et al. 2014) and VAE (Kingma & Welling 2014), respectively. Note that all three sequences are with a length 440 time steps. Our model (COSMO) generate the sequence with three reference subsequences. While both GAN (Goodfellow et al. 2014) and VAE (Kingma & Welling 2014) models directly produce the motion a recurrent manner. The GAN based model fails synthesise normal walking sequence with large pose distortion. The VAE based model able to generate a visually natural walking sequence facilitated by KL loss during training but struggles produce diverse motion behaviour. Different from all these parametric models, Fig
7

---

Under review as a conference paper at ICLR 2020
Reference Mation
Target Style Motion
Edited Reference Motion
Sequence
Sequence
Figure 6: Visualization of local composed reference sequence.
0.4
Baselinel Baseline2 Baseline3
Train MSE
foot height
0.2
Start Interpolation
End
Interpolation
0.0
200
300
30
40
50
60
70
80
90
(A) MSE for global composition
(B) Foot height variation
Figure 7: Evaluation of global composition in terms training MSE and foot height variation.
shows that our semi-parametric modelling based model achieves natural state transition throughout the whole sequence, meanwhile provides natural and diverse motion (i.e., walking-tunnn turning- turning-walking) -walking) behaviours during generation. We encourage readers refer our project page for more video demonstrations.
Do proposed model learn a meaningful style space? Recall that for representation learning style feature, we construct embedded space instead of directly generating style feature. In this way, we expect our model able to map novel style pattern into learned embedded space. For validation we randomly select 10% training data then extract corresponding style feature (denoted as Strain) with Psty Regarding the held-out set (denoted as Syef) we conduct subsequences. We visualize the distribution of both Strn and Sre/ with t-SNE (van der Maaten & Hinton 2008) a -wo-dimensional plane. As shown in Fig 4 the purple dots indicate training data while green dots stand for reference data. We evenly across the plane. Meanwhile, the majority of style feature of reference subsequences is of style covered by that of training data. Within the learned style space, embedded layer acts bases where novel style can be approximated by the combination of these bases.
same operation for all reference
that the style feature of training data spread
Can COSMO compose two reference subsequences into novel one? Part of the generation diversity of our model results from free composition of reference subsequences. As shown in Fig we provide two edited sequence examples which possess the general motion style from one subsequence, but detailed motion pattern from another one. Taking the second sequence (bottom row Fig (6) for example, the target style motion shows a spinning motion with both hands raised up (style), while the reference motion is a regular walking sequence. We can notice that the final edited reference motion (third column, bottom row) fully captures the style upper body meanwhile maintains the walking pattern from reference sequence. Moreover, both top and bottom rows use the same reference motion, but with different style as inputs, our model can still produce highly diverse behaviours. For the space restriction of paper writing, we provide more examples whose style features are sampled from constructed embedded space. We encourage readers to refer to our project page for more video demonstrations.
8

---

Under review as a conference paper at ICLR 2020
Figure 8: States transition visualization for evaluation of smoothness.
End
Start
Long-range Motion Generation
Figure 9: Left: Motion generation given the same starting and ending states. Right: Long-range Motion generation with characteristic routes.
4.3 EVALUATION FOR GLOBAL MOTION COMPOSITION
Do COSMO learn better motion transition than basic prediction models? Different from pre- diction model, COSMO produces motion transition given starting or ending states from different sequences, i.e., there is no ground truth for evaluation. To this end, we provide training error for evaluation of motion transition. As shown in Fig A)), we can see that our model (COSMO) out- performs other methods by large margin. Base achieves lowest accuracy caused by no ending states are provided during composition. Baseline2 and Baseline3 models perform better than basel model, which indicates that both bi-directional and coal-conditional modelling scheme facilitates motion composition by large margin. Our global composition sub-module combines the strength of both Baseline2 and Baseline3 models, which evenly utilize the information from starting and ending states. However, merely considering the training eтror is not sufficient. Next, we further evaluate the performance of our model given two different sequences.
Can COSMO guarantee smooth transition between two different sequences? Fig demonstrates motion transition results given starting and ending states from different reference subsequences, respectively. Note that for all sequences shown in Fig 8 the starting well as ending states are from held-out reference set. We can observe that our model is able generate smooth and natural transition when starting and ending states are similar. Moreover, when encountered large motion difference, e.g., from walking to greeting, turning back with relatively large degree, our model still makes it to generate visually natural transition sequence. We encourage readers to refer to our project page for more video demonstrations.
Is global motion composition merely linear interpolation? One possible trivial solution learned by our transition model is to simply linear interpolation between starting and ending states. However, common human motion generally involves non-linear trajectories. Linear interpolation is prone to produce unnatural motion which is easy to detect by human eyes. To valid that whether our model conducts linear interpolation between sequences, record the height variation of the right foot

---

Under review as a conference paper at ICLR 2020
Synthesired Aemonstration
0.8
Peturn
0.6
Learned Polic"
Iteration 4000
5000
Figure 10: Generated results as demonstration guidance for imitation learning.
in composed motion sequence. Meanwhile, manually rotate the second sequence w.r.t the final state of the first sequence to show that our model is robust to a large range direction difference between sequences. As shown in Fig A), two black arrows indicate the starting and ending steps for composition. Here we present multiple curves which correspond to different rotation angle mentioned earlier. All recorded curves are highly non-linear but smooth between starting and ending points. Moreover, our model adaptively changes foot height with different rotation configurations, which indicates smooth and natural motion for motion composition.
Visualization of final composed sequences. Combining local and global motion together, we are able to generate final sequences. Recall that our model constrained by given starting and ending states based on motion interpolation. To this end, we present three composed sequences with a length of 480 time steps, i.e., three edited reference sequences (length of 120) and four generated clips for global motion composition. As shown in left part of Fig 9 starting from the same state, we are able to generate long-range and visually natural motion boosted by the local and global motion composition. Meanwhile, we are able to generate diverse behaviour (shown as complex hand and foot motion) facilitated semi-parametric modelling. We encourage readers to refer to our project page for more video demonstrations.
4.4 APPLICATION
In this section, we present several downstream applications related our motion generation model. The first one is diverse motion generation under fixed rout constrain. The second one is demonstration motion guidance for imitation learning (Peng al. 2018).
Diverse motion generation. As shown in the right part of Fig 9 we manually design four routes for motion generation. We can see that our semi-parametric model is able to produce intermediate motion states clearly following the predefined route. Note that the longest motion sequence indicating character "R" over 1500 time steps. We encourage readers to refer to our project page for corresponding video demonstrations.
Expert demonstration guidance for imitation learning. Under a simulated environment with gravity constrain (Peng et al. 2018), unnatural motion violating physical law (e.g., severe joint twisting) hard for a simulated agent to follow. To further show our model produces realistic motion. generated results the left part is demonstration synthesised by COSMO (top) while the bottom one is learned policy with|Peng al. (2018). We can see that the learned motion succeeds follow the synthesised one. The right part is return curve during training, which also shows that our generated motion is realistic enough for the guidance of imitation learning. We encourage readers refer our project page for corresponding video demonstrations.
used for demonstration guidance of imitation learning. As shown in Fig 10
5 CONCLUSION
In this work, we propose to generate long-range motion in a reference motion subsequence from the held-out set and change the motion style with local motion composition scheme. We then compose all reference subsequences with the proposed global motion composition scheme. Given the same starting and ending states, the proposed model is able to
way. We first sample
10

---

Under review as a conference paper at ICLR 2020
generate oongrrange, diverse and natural motion sequences 1000 time steps without loss of visual quality.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenherg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- scale machine learning on heterogencous systems, 2015. URL htt.:htttt://eeeeeeeeeeeee flow. .wwoorr.. Software available from tensorflow.org.
Jacopo Alcotti and Stefano Caselli. Grasp recognition virtual reality robot pregrasp planning by demon- stration. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., pp. 2801-2806. IEFE, 2006.
Sunjeev Arora and Yi Zhang. Do gans actually Iearn the distribution? an empirical study. CoRR, abs/ Abss/106.008244 2017.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018.
Shane T. Barratt and Rishi Sharma. A note on the inception score. CoRR, abs/ 00993,, 2018.
Emad Barsoum, John Kender, and Zicheng Liu. HP-GAN: probabilistic human motion prediction via GAN. In CVPR Workshops, pp. 1418-1427, 2018.
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Bayesian prediction of future street scenes using synthetic likelihoods. In ICLR, 2019.
Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM J. Scientific Computing, 190-)119,, 1995.
Haoye Cai, Chunyan Bai, Yu-Wing Tai, and Chi-Keung Tang. Deep video generation, prediction and completion of human action sequences. In ECCV, pp. 374-390, 2018.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In ICML, pp. 1182-1199, 1191, 2018.
Emily L. Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In
NIPS, pp. 4414 4423, 2017.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. CoRR.
Chelsea Finn, Ian J. Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NIPS, pp. 64-72, 2016.
Ian J. Goodfellow, Jean Pouget-Abass Mehdi Mirza, Bing Xu, David Marde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp.
2014.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: socially acceptable trajectories with generative adversarial networks. In CVPR, pp. 2255-2264. 2018.
Adrian Haarbach, Tolga Birdal, and Slobodan Ilic. Survey of higher order rigid body motion interpolation methods for keyframe animation and continuous-time trajectory estimation. In 3DV, pp. 381-389, 2018
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Daniel Holden, Jun Saito, and Taku Komura. deep learning framework for character motion synthesis and editing. ACM Trans. Graph., 35(4):138:1-1388 2016.
Daniel Holden, Taku Komura, and Jun Saito. Phase-functioned neural networks for character control. ACM Trans. Graph., 36(4))42:1-42.) 2017.
Karim Iskakov. Seri-parametric image inpainting. CoRR, wwt/1800.008.... 2018.
11

---

Under review as a conference paper at ICLR 2020
Xiaojie Jin, Huaxin Xiao, Xiaohui Shen, Jimei Yang. Zhe Lin, Yunpeng Chen, Zequn Jie, Jiashi Feng. and Shuicheng Yan. Predicting seene parsing and motion dynamics in the future. In NIPS, pp. 6915-69199999 6924, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Lucas Kovar and Michael Gleicher. Flexible automatic motion blending with registration curves. In SIG- GRAPH/Euwgruphics Symposium, pp. 214-224, 2003.
Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NIPS, pp. 9675-3683, 2016.
Martin Lawitzky, Jose Ramon Medina, Dongheui Lee, and Sandra Hirche. Feedback motion planning and learning from demonstration in physical robotic assistance: differences and synergies. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.
IEEE, 2012.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S. Hu, and Joseph J. Lim, Composing complex skills by learning transition policies. In ICLR, 2019.
Sergey Levine, Jack Wang, Alexis Haraux. Zoran Popovic, and Vladien Koltun. Continuous character control with cww-dimensional embeddings. ACM Trans. Graph., (1())288: (10(4)2::122:11100 1-2::10, 2012.
Yunpeng Li, Dominik Roblek, and Marco Tagliasacchi. From here there: Video inbetweening using direct 3d convolutions. CoRR, abs/1905. 10240, 2019.
Ziwei Liu, Raymond A. Ych, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In ICCV, pp. 4473-4481, 2017.
Xuezhe Ma and Eduard H. Hovy. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisticcs, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, 2016. URL httpp:: wwwww/ org/ wnw.hhtthhg..o.
P16-1100
Simone Meyer, Oliver Wang. Henning Zimmer, Max Grosse, and Alexander Sorkine- Horinne. Phase-based frame interpolation for video. In CVPR, pp. 1410- 1410,118,2200 2015.
Donald R. Myers. Robot Motion: Planning and Control edited by michael brady M.I.T. press, cambridge ma, usa, 1983 (E33.95). Robotica, 1(2):109, 1983.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. In ICCV, pp. 261-270, 2017.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2)11111) 179, 2018.
Sang II Park, Hyun Joon Shin, and Sung Yong Shin. On-line locomotion generation based on motion blending. In SICGRAPH/Eiworrgphicc Symposium, pp. 105-111, 2002.
Xue Bin Peng, Pieter Abheel, Sergey Levine, and Michiel de Panne. Deepmimic: Example-guided deep reinforeement learning of physics-hased character skills, ACM Transactions on Graphics (TOG), 143) 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances neural information processing systems, pp. 305-313, 1989.
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi-parametric image synthesis. In CVPR, pp- 1808-88166 2018.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of vidco representations using Istms. ICMI, pp. 843-852, 2015.
Chen Sun, Per Karlsson, Jiajun Wu, Joshua B. Tenenbaum, and Kevin Murphy. Stochastic prediction of multi-agent interactions from partial observations. In ICLR, 2019.
Charlic Irawan Tan and Wen-Kai Tai. Characteristics preserving racer animation: data-driven race path
synthesis in formation space. Journal of Visualization and Computer Animation, 23(3-4):115-223, 2012.
Raquel Urtasun, David J. Fleet, Andreas Geiger, Jovan Popovic, Trevor Darrell, and Neil D. Lawrence. Topolleicallly-consrainee latent variahle models. In ICMI., pp. 1080-1087, 2008.
12

---

Under review as a conference paper at ICLR 2020
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. In Journal of Machine Learning Research, 2008.
Miao Wang, Guo- Ye Yang. Ruilong Li, Runze Liang, Song-Har Hai Zhang, Peter M. Hall, and Shi-Min Hu. Example-guided style consistent image synthesis from semantic labeling. CoRR, 2019.
Xiaolong Wang, Rohit Girdhar, and Abhinav Gupta. Binge watching: Scaling affordance learning from sitcoms. In CVPR, pp. 3366-33 3375, 2017.
Yonatan Wexler. Eli Shechtman, and Michal Irani. Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell., 29(3):463-477 2007.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision. ICML, pp. 6033 6041, 2018.
Guiyu Xia, Huaijiang Sun, Qingshan Liu, and Renlong Hang. Learning-based sphere nonlinear interpolation motion synthesis. IEEE Trans. Industrial Informatics, 1565):2927-2977, 2019. doi: 10.1109/T1..019 2894113, URL ott.
wttp://....1111.1..1111111 2019. 2000:.......11111.1...1.1.1.00..
Jingwei Xu, Bingbing Ni, Xiaokang Yang. Video prediction selective sampling. NIPS, pp. 1712-1722. 2018a.
Yanyu Xu, Zhixin Piao, and Shenghua Gao. Encoding crowd interaction with deep neural network pedestrian trajectory prediction. In CVPR, pp. 5275 9775-2244, 2018b.
Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. MT-VAE: learning motion transformations to generate multimodal human dynamics. In ECCV, 2018,
Gu Ye and Ron Alterovitz. guided motion planning. In Robotics research, pp. 29. 291-307. Springer, 2017.
Jason Y. Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting human dynamics from video. CoRR, abs/1908.04781, 2019.
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. CoRR, 2019.
13

---

Under review as a conference paper at ICLR 2020
APPENDIX
Requirements for reference sequence R', Our model does not restrict the length of RA, neither any specific/preddffiee starting or ending states of it. However, there are still restrictions for R`:
Physically reachable. In this work we do not have access to the dynamics of other objects, which should be excluded from candidate references involved. For example, our model does not intend for synthesising transition sequence from running lifting. The latter be- haviour needs dynamic information (bell) to model. Therefore, our experiments reference subsequence is first restricted to single character motion without external interaction.
Moving on flat ground. Besides physically reachable, we also restrict the character moving on flat ground. Because in our task there is no predefined ground information available for modelling. To get rid of unexpected composition results, we filter out these sequences recorded from uneven terrain.
Searching procedure for reference subsequences
Step O: Calculate overall root locations for all reference subsequences. We denote corre-
sponding minimum/maximum value as tmintt respectively.
Step I: Given So and S,, extract corresponding root locations (lo and 4,) respectively, Here root is defined by the ground projection of hip joint. Then sample NR + mid locations, which forms NR + 2 pairs, i.e. [4,,4], 41,..,,,,,, [N 42. Distance defined by each pair dm is bounded, i.e., mmnn dm mmaz-
Step 2: For each location pair, we search reference subsequence for similar translation defined by dm with maximum tolerance oN. We conduct local motion composition for searched reference subsequence (in Section 2.2).
Step 3: We sequentially conduct goal conditioned bi-directional motion interpolation for two consecutive reference subsequences (in Section 2.3). As preprocessing procedure, the overall location and direction of latter sequence is adjusted w.r.t former one for interpolation. The final sequence is obtained through concatenating synthesised clips along temporal axis.
14


Reviews:
[{"review_id": "rJxVtJsQcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at taking techniques from motion interpolation into the regime where one is able to generate longer range motion sequences, in the domain of physically plausible computer animation of characters. In the way that the authors have set up the problem, an initial database seeds the search for plausible transitions between two given poses. So, the technique being proposed must address how to keep physical realism (the long-standing question of \"dynamics filtering\" along the lines of Yamane, Katsu, and Yoshihiko Nakamura. \"Dynamics filter-concept and implementation of online motion generator for human figures.\" IEEE transactions on robotics and automation 19.3 (2003): 421-432. Of course the problem here also needs to address \"style\" which needs different models). \n\nThe authors propose an approach, building on recent neural network architectures including GANs and VAEs, which combines an embedding into a latent space to capture the style+content and then a Bi-LSTM model to make P-step predictions to extract the interpolating poses. The architecture is not very well explained and could be presented much better. However, the constituent elements are fairly standard ones. The learning of the embedding space is based on minimising a reconstruction loss and adapting parameters of a network that takes convolutions over time windows and stacks different channels. The LSTM training is also based on prior work in that domain. \n\nThe authors try to show with experiments that the proposed model is doing better both on diversity and accuracy in standard motion capture datasets. However, the baselines used, especially for the accuracy comparison, are more along the lines of an ablation study than a genuine comparison with alternate methods.\n\nThis is coupled with the fact that the authors do not seem to have engaged with a fairly established literature on modelling such sequences, e.g., \nBrand, M., & Hertzmann, A. (2000, July). Style machines. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (pp. 183-192). ACM Press/Addison-Wesley Publishing Co..\nLi, Y., Wang, T., & Shum, H. Y. (2002, July). Motion texture: a two-level statistical model for character motion synthesis. In ACM transactions on graphics (ToG) (Vol. 21, No. 3, pp. 465-472). ACM.\n\nThese papers also solve the same problem the authors have set out to solve and arguably do pretty well. How well do they compare to the authors' approach and in what ways have they built further? It would be helpful to understand this with quantitative evidence. \n\nAs it stands, this comes across as a report on a preliminary experiment. Indeed, figure 10 is just a single learning curve without much more interpretation and analysis. The paper would be much stronger if situated better with respect to other established work and also supported by more systematic empirical experiments.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1x7nbEP5S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary:\nThe paper proposed ''composable semi-parametric modeling'' for generating long-range diverse and distinctive behaviors to achieve a specific goal location. The non-parametric part is a memory bank that is used to retrieve motion patterns from source materials. The parametric part contains several deep neural networks which are to compose the retrieved materials for high quality and smooth motion generation. The overall idea is novel in the sense that they aim to combine the strength of the non-parametric method (with rich pattern and diversities) and the parametric method (powerfull ability to generate coherent results). The proposed ideas are evaluated on two datasets and outperform compared approaches qualitatively.\n\n 2. I am borderline to this paper but prone to weakly reject this paper.\n\n   (1) My major concern about this paper is the lack of reasoning about each component. \n        - local motion composition: How does the design help encode both style and content information in the outputs? Since reconstruction loss is utilized, the optimal should be to learn an identity mapping. How does the author avoid this to happen? The paper referred to the style embedding features H_s, but it is hard to understand how this is associated with the architecture. Also, the usefulness of local motion composition is not well illustrated in the experiments part. \n\n    -global motion composition: How the network is supervised to compose different clips together? It seems that the authors again adopt reconstruction loss, how the loss penalize inconsistent predictions between clips is not well explained.\n      \n   (2) The overall presentation of this paper is hard to follow. I spent a lot of time on understanding the notations and corresponding networks. For example, the input representation is not well stated and the author referred to \\phi_loc without any indication in Fig, 2. It's very hard to understand the details of the method. I would like to suggest that the author improves the writing especially for Section 2.\n \n(3) Does the style motion shared by all sub-sequences? \n\n(4) Figure 5 shows a sequence. however, the COSMO result seems not natural for me since the velocity in the sequence changes. \n\nI would like to change my rating if my questions are well addressed in the rebuttal. Overall, this is an interesting idea.\n   ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1l9gbTwqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of generating long-range, diverse and natural looking motion sequence between initial and end states, and proposes to use a semi-parametric approach consisting of local and global models. Specifically, first the proposed approach extracts local motion feature from a reference subsequence and style feature from another, and then generates a new motion sequence. Then, global motion composition is done to interpolated generated local subsequences by bi-directional composition. In experimental validation, the approach outperforms two baselines (GAN and VAE). \n\nThe proposed approach seems interesting and relatively novel, and technically sound. Also using disentangled representation of style and content is well-motivated. In addition, its generated motions look natural and visually appealing. However, the paper needs more thorough experimental validation to demonstrate its effectiveness better. First,  it was not quite clear which specific baselines were exploited. Are they simple VAE and GAN, or HP-GAN (Barsoum et al. (2018)) and MT-VAE (Yan et al. (2018)) ? And no animated sample of baselines was provided in the demo website. Additionally, in the global composition results, there is no other baseline except its own variants. Finally, even thought the provided figures and animated demos look appealing, it may be required to provide more qualitative results such as user study. \n\nWriting is fine but there were many typos and it may need more proofreading. Also it might need to add more detail in the presentation.\n\nDetailed comments:\n- Why is GAN better than the proposed model with 10% of training data in Fig. 3? Do you have any explanation about it?\n- The paper uses separate training of local and global models. But how about joint training of them?\n- Difference between style and content is sometimes not clear, for example, rightmost one in the last sample in the demo website. Could you add more constraint or model specification to induce more disentangled representation?\n- It is not quite clear if standard deviation is a good metric for motion diversity metric since visual diversity might not be directly correlated to standard deviation of joints (for example, some joints might be more important than others for inducing diversity.) \n\nTypos:\np4: lone-range one -> long-range one\np5: (Gupta et al.) -> (Gupta et al., 2018), (Zhao et al.) -> (Zhao et al., 2019), (Wang et al.) -> (Wang et al., 2019) \np6: Our works -> Our work\np7: Fig. 3: standard diviation -> standard deviation \np8: Recall that we representation -> Recall that we do representation?\np9: As shown in Fig 7(A) -> As shown in Fig 7(B)\np10: rout constrain -> route constraint, gravity constrain -> gravity constraint\n\n\nIn sum, I think the paper is at the borderline but it could be improved and better by having more through experimental validation and more detailed presentation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents a semi-parametric approach to motion synthesis. The reviewers expressed concerns about the presentation, the relationship to existing work, and the scope of the results. After the authors' responses and revision, concerns remain. The AC also notes that the submission is 10 pages long. The AC recommends rejecting the submission.
Decision: Reject