Title: COMPOSABLE SEMI-PARAMETRIC MODELLING FOR LONG-RANGE MOTION GENERATION Anonymous

Abstract: Learning diverse and natural behaviors is one of the longstanding goal for creating intelligent characters in the animated world. In this paper, we propose "COmposable Semi-parametric MOdelling" (COSMO), a method for generating long range diverse and distinctive behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by combining the complementary strengths of both non-parametric techniques and parametric ones. Given the starting and ending state, a memory bank is used to retrieve motion references that are provided as source material to a deep network. The synthesis is performed by the deep network that controls the style of the provided motion material and modifies it to become natural. On skeleton datasets with diverse motion, we show that the proposed method outperforms existing parametric and non-parametric baselines. We also demonstrate the generated sequences are useful as subgoals for actual physical execution in the animated world. Please refer to our project page 1 for more synthesised results.

1. INTRODUCTION:

When faced with a specific goal in another location, humans can effortlessly find multiple distinctive trajectories and control their body to approach the goal with diverse and natural behaviours. However, we are still at the early stage for such sophisticated controlling of simulated characters. Recent reinforcement learning approaches (Peng et al., 2018) struggle to generate diverse motion. Other methods like imitation learning (Ye & Alterovitz, 2017; Aleotti & Caselli, 2006; Lawitzky et al., 2012) generalize badly to large scale demonstration data. The heart of this challenge lies in that real-world human behaviours are inherently multi-modal distributed. Direct behaviour learning is difficult without access to the explicit distribution of motion states.

In this paper, we take a step toward generating long-range, diverse and physically plausible motion sequences given starting and ending states, rather than learning a policy for a physical simulator. Meanwhile, we expect the model could generate novel behaviour, i.e., unseen motion in training set. This has several valuable applications: (1) Synthesised vivid motion for animation production without excessive human labor. (2) Generated novel behaviours for player customization of action skills in video games. (3) Interpolated sequences as subgoals for policy training through reinforcement learning (Peng et al., 2018) .

To fulfil the above requirements, we need to address the following difficulties: (1) How to guarantee both diversity and naturalness, which is usually trade-off in the domain of generation (Srivastava et al., 2015) ? (2) How to achieve long-range behaviour synthesis, which is stuck by error accumulation problem in many temporal modelling tasks (Denton & Fergus, 2018) ? (3) How to generate unseen behaviours without loss of diversity or naturalness, which is hardly addressed in previous researches? As shown in Fig 1 , the current two main branches of motion generation methods, i.e., parametric and non-parametric, are yet to deal with these difficulties properly. Parametric (Goodfellow et al., 2014 ) (e.g., GAN) methods could not maintain the reality/naturalness of generated sequences, and diversity (e.g., VAE) is hard to preserve. Non-parametric methods (Haarbach et al., 2018) involving motion clip copy/paste or blending has less superiority in smooth transition based on diverse reference sequences. In this work, we propose COmposable Semi-parametric MOdelling (COSMO), which is a method that can leverage the large spectrum of motion skills from unlabeled data in a semi-parametric manner. Our method combines the complementary strengths of both non-parametric techniques and parametric ones. First, to avoid mode collapse in the sequences, we initially sample reference subsequences from a held-out reference set to encourage multi-modal behaviors in the generated sequence, which is mainly inspired from non-parametric methods (Haarbach et al., 2018) . Second, we propose a self-supervised disentanglement model for extracting the content and style from each reference subsequence respectively. Here content refers to the characteristic of state at each time step, e.g., moving speed, direction, and general gesture of upper body, while style refers to the long-range motion pattern which keeps relative constant across whole subsequence. An embedded latent space is constructed where the learned style vectors could be combined to obtain novel style feature. A new subsequence is obtained by composing content and style freely from different reference subsequences.

Considering that this operation only produces a single subsequence, we refer this step as local motion composition. Finally, for all new sequences, we compose them along the temporal direction in a sequential order. However, from a hodgepodge of reference segments, it is non-trivial to generate natural and meaningful behaviors. To guarantee naturalness at long-range scale, we then propose to use goal conditioned bi-directional interpolation for modeling the long-range nature needed in the task. This step is regarded as global motion composition, which covers the whole temporal scale (often more than 100 steps). On two human motion datasets, we show that the proposed method outperforms existing parametric and non-parametric baselines. We also demonstrate the generated sequences are useful as subgoals for actual physical execution in the animated world.

Our paper makes three contributions. First, we proposed COmposable Semi-parametric MOtion generation (COSMO), a method that can generate the required sequences in a semi-parametric, composed way. Second, we conducted experiments on CMU Mocap dataset2 and SFU Mocap dataset3 and shows our method outperforms strong baselines such as VAEs and GANs significantly. Third, we proved empirically that the generated sequences can serve as subgoals to learn an actual policy. sub-module φ loc . Invalid subsequences are first filtered out with predefined requirements and please refer to appendix 6 for detailed description. φ loc is used for altering the motion style of reference subsequence R i based on another sequence R as input or directly sampling a style feature from learned latent space: Ri = φ loc (R i , R), where R is optional input. With this in hand, we compose edited subsequences { Ri } N R i=1 along temporal direction smoothly. Concretely, we generate a new clip Ri-1:i = φ glo ( Ri-1 , Ri ) which bridges the gap between Ri-1 and Ri . The final composed sequence is obtained through concatenation along the temporal direction for all subsequences and generated clips.

2. METHOD:

Why semi-parametric modelling? Regarding both motion category and transition dynamics (from current to next state), human behaviors are intrinsically multi-modal distribution (Holden et al., 2017) . Parametric models (e.g., VAE (Kingma & Welling, 2014) or GAN (Goodfellow et al., 2014) ) assume prior distribution to be uni-modal and predefined, which is not suitable to handle real world motion sequences (Kingma & Welling, 2014; Arora & Zhang, 2017) . Non-parametric model does not require explicit prior distribution but not able to deal with long-range and highly diverse motion properly, mainly restricted by modelling capability. We also conduct targeted comparison experiments in Section 4. Compared to parametric methods, semi-parametric modelling naturally guarantees motion diversity by introducing reference sequence also maintaining a large degree of freedom for generation. Moreover, semi-parametric modelling does not require access to or a model of data distribution. It implies that our method is also directly applicable for unseen motion sequence.

Constrained by starting and ending states with unchangeable locations, it is not feasible to randomly sample reference subsequences for generation. To construct a complete motion sequence, we execute a manually designed procedure for searching reference subsequences. For detailed description, please refer to appendix 6

2.2. LOCAL MOTION COMPOSITION WITH CONTENT/STYLE DISENTANGLEMENT:

Local motion composition stands for generating a relative short-range of motion clip based on reference subsequences as inputs. Content and style features are learned jointly, whose free-form composition is used for synthesising new motion sequence. We first describe the motion style modelling part.

Motion Style Modelling.:

Style is considered as motion information which stays the same throughout the whole subsequence. To this end, we extract style pattern by conducting feature fusion along the temporal direction. More specifically, for a sequence with T frames, we gradually fuse all frames into one constant feature with C s channels (denoted as h s ). We adopt 1D convolution along the temporal direction for each operation. Three convolution layers are stacked together with a kernel size of (T /2, T /4, T /8) respectively, where large kernel size facilitates capturing motion style throughout the whole subsequence.

To get better generalization ability for unseen reference or starting/ending states, we propose to construct an embedding space for style pattern modelling. More specifically, a set of embedding vectors (denoted as H s = {h i s } M i=1 ) are learned together with above convolution layers (denoted as ψ sty ). The corresponding outputs (denoted as A s ∈ R M ) of ψ sty are a family of coefficients for a weighted sum of H s . This is mainly inspired by the intuition that complex human activity could be considered as a combination of several simple actions. When trained with sufficiently large motion data, the model tends to learn basic style patterns for human motion. Compared to directly outputting a feature vector, this helps better generalize to unseen motion during testing. The final style feature (denoted as f s ∈ R Cs ) is generated as follows:

f i s = ψ sty (R i )H s .

(2.1)

Motion Content Modelling. In this part, we focus on motion dynamics reflected by each single state. To this end, 1D convolution layer with kernel size of 3, i.e., covering consecutive 3 states, is adopted for content modelling. The output channel of this layer is designed less than original motion dimension for extracting the most relevant information. This is empirically set to 5 as a design choice. Meanwhile, velocity, position for root joint and step pattern for foot joint are fed as inputs. We denote all the above factors as f c with T steps and C c channels. During local motion composition, style feature f s is combined with content feature f c through concatenating along the channel axis:

f i c = ψ cnt (R i ), Ri = ψ rec (f i c , f i s ), (2.2)

where f s is tiled T times to match with f c and ψ rec outputs reconstructed Ri . ψ rec follows stacked three layer 1D convolution operation with kernel size of 3. Finally, reconstruction loss is used to learn the style and content feature jointly as follows:

L R rec = 1 N R Σ N R i=1 || Ri -R i || 2 2 . (2.3) 2.3 GLOBAL MOTION COMPOSITION VIA GOAL CONDITIONED BI-DIRECTIONAL MODELLING

Global motion composition is connecting edited short-range subsequences into a completed and long-range one. Generating smooth and natural transition between Ri and Ri+1 is critical to obtain a long-range sequence. The major challenge comes from large states variation between Ri and Ri+1 . Inspired by recent work of Bi-LSTM modelling (Ma & Hovy, 2016) , bi-directional motion composition is utilized for this part. More specifically, motion states are first mapped to a higher dimensional space with ϕ enc and ϕ dec , which is defined as follows:

f i t = ϕ enc (r i t ), ri t = ϕ dec (f i t ).

(2.4)

Both ϕ enc and ϕ dec are 1D convolution layers. Then we predict the possible states at both directions with f i t , where higher dimension (256) provides a sparser space enabling better composition results. For forward prediction, we take f i T -4:T as inputs to obtain forward states f i:i+1 1 . Meanwhile, f i+1 1:5 are fed as inputs treated as goal conditions. We conduct this procedure in a recurrent manner with P -step prediction in total. Predicted f i:i+1 1 is concatenated with last four states forming next step inputs. Backward direction takes exactly the reverse procedure, i.e., f i+1 T -4:T are treated as inputs while f i 1:5 are as goal:

f i:i+1 p+1 = ϕ f pre (f i p-4:p , f i+1 1:5 ), f i+1:i p+1 = ϕ bpre (f i+1 p-4:p , f i T -4:T ),

(2.5) where ϕ f pre and ϕ bpre share the same architecture.. Moreover, we utilize another model to produce coefficients for the weighted sum of outputs. In this way, we are able to construct more flexible latent space to facilitate composition:

c = ϕ com (f i:i+1 p , f i+1:i P -p , p), fp = cf i:i+1 p + (1 -c)f i+1:i P -p . (2.6)

ϕ com is one layer 1D convolution with sigmoid function for producing c. More specifically we introduce p and linearly map it to p ∈ (-1, 1) and c = p + 0.1 * σ(conv([f i:i+1 p , f i+1:i P -p ])). During training, we randomly select a motion clip with a length of 10 + P . First as well as last 5 states are fed as inputs to get intermediate P states. Reconstruction loss is used for training:

L f rec = 1 P Σ P t=1 ||r t -ϕ dec ( f t )|| 2 2 .

(2.7)

Compared with goal-conditioned single forward prediction model, the starting and ending states are treated equally in our model, which avoids drifting away from the ending state. Meanwhile, we provide corresponding empirical results in Section 4. After global motion composition, final composed sequence is directly obtained by concatenating all edited subsequences (local part) as well as transition clips (global part) along the temporal direction.

We use tensorflow (Abadi et al., 2015) to implement all our models. All activation function used in our model is ReLU operation. During training, learning rate is set to 1e-4 and optimized with Adam optimizer (Kingma & Ba, 2015) . β 1 and β 2 are set as 0.9 and 0.999 respectively. All models are trained with 30 epochs in total. Note that the local and global motion composition submodule do not need joint training. They are used jointly during testing.

3. RELATED WORK:

Motion Interpolation. Given start and end states, this task aims to synthesize intermediate states which smoothly translate between them (Urtasun et al., 2008) . For video interpolation (Liu et al., 2017; Li et al., 2019; Meyer et al., 2015; Niklaus et al., 2017) where start and end states are two consecutive frames, the final result is expected to increase frame rate of original video to a higher value. Previous researches often utilize phase dynamics (Meyer et al., 2015) , flow based feature (Liu et al., 2017) and other motion information (Niklaus et al., 2017) to facilitate this task. Our work is different from this branch of work because there exists large motion gap between start and end states in our settings. Another branch of work is video completion (Cai et al., 2018; Li et al., 2019; Wexler et al., 2007) . It receives two nonconsecutive frames as input and aims to fill the motion gap between start and end states. Cai et al. (2018) firstly attempts to solve this task and more specifically, propose to select out a rational path in the latent space with BFGS (Byrd et al., 1995) algorithm. Li et al. (2019) incorporates the 3D convolution layers and LSTM network into a unified model, which tries to automatically find the optimal results for intermediate frames. Despite much progress has been made in this filed, the high dimensional data (i.e., video frames) severely restricts video completion within simple and seen motion categories. However, we do not limit the start and end states belonging to the same sequence. Meanwhile, we expect the interpolated sequence as diverse as possible meanwhile with natural transition between synthesised states. This has not been deeply addressed in previous motion completion works (Xia et al., 2019; Lee et al., 2019) . As a potential downstream application, our model could be used to construct motion planning (Myers, 1983) algorithm. Compared to goal-driven RL (Kulkarni et al., 2016; Lee et al., 2019) , our model gets rid of requirements hard to achieve, i.e., known dynamics of agent, which is more general and applicable to more motion planning scenarios.

Temporal Data Prediction. There have been many researches focusing on temporal data prediction. This task aims to infer the future possible states conditioned on history states as input. Video prediction (Srivastava et al., 2015) takes a major part in this field, which involves pixel-wise forecasting for every following frames. Previous methods (Denton & Fergus, 2018; Babaeizadeh et al., 2018; Denton & Birodkar, 2017; Finn et al., 2016) have made it to produce high quality prediction on bouncing digits (Srivastava et al., 2015) , robot motion (Finn et al., 2016) and semantic map (Jin et al., 2017) . However, recent work (Xu et al., 2018a) still encounters much difficulties in forecasting complex movements involving articulated subject (Wichers et al., 2018) . Moreover, these works are prone to have motion blur (Finn et al., 2016) and error accumulation (Denton & Fergus, 2018) problems. Out of above concerns it is more appropriate for us to formulate our method as interpolation instead of prediction model. In terms of lower dimension data, human pose (Sun et al., 2019; Zhang et al., 2019) as well as path trajectory (Xu et al., 2018b) are also hot spots in this task. Several works adopt probabilistic Bayesian model (Bhattacharyya et al., 2019) to dig out latent factors which influence future dynamics. Another branch of researches (Gupta et al., 2018; Xu et al., 2018b ) utilize deep recursive model (e.g., LSTM Hochreiter & Schmidhuber (1997) ) to extract critical feature for prediction. Low dimension data enables all these works could easily scale to multi-target settings (Zhao et al., 2019) . However, prediction model loses its power when encountering out of distribution data (e.g., unseen motion category in test set). As comparison, our interpolation model could naturally generalize to unseen motion facilitated by semi-parametric modelling, which gets rid of explicit representation for future dynamics.

Generative Semi-parametric Modelling. For generative task such as image synthesis (Qi et al., 2018) , translation (Wang et al., 2019) and inpainting (Iskakov, 2018) , semi-parametric modelling achieves considerably more realistic visual quality and better style consistency between source and target images. Semi-parametric modelling is also utilized in reinforcement learning (Kulkarni et al., 2016) for navigation task (Eysenbach et al.) . More specifically, it searches rational path on replay buffer, which enables agents to solve sparse reward tasks over one hundred steps. Wang et al. (2017) combines semi-parametric modelling and VAE for large-scaled motion affordance. In this work we use semi-parametric modelling for two main reasons: (1) it could substantially increase the length of interpolated sequence through acting as example guidance, ( 2) it enables our model to synthesise visually appealing results even with out of distribution inputs. Meanwhile, we would like to highlight that to our best knowledge this is the first attempt to use semi-parametric modelling on motion interpolation task.

Motion Generation in Computer Graphics. In the context of computer graphics, there is a branch of researches (Kovar & Gleicher, 2003; Park et al., 2002; Levine et al., 2012; Tan & Tai, 2012; Holden et al., 2016) which also concentrate on motion generation, i.e., obtaining a continuous trajectory from a discrete set of poses. Haarbach et al. (2018) provides a comprehensive study on this topic. It analyzes the characteristics of higher order rigid body motion interpolation methods. Our works shares similar target with this branch of work. However, we would like to emphasise that these works (Tan & Tai, 2012; Kovar & Gleicher, 2003; Park et al., 2002) are in parallel with ours and have completely different research routine on this task. More specifically, graphics methods focus on finding a optimal and explicit mathematical solution regardless of input motion sequences. Different from them, our work is data-driven and encourages both reality and diversity of interpolated results.

Imitation Learning. Imitation learning (Pomerleau, 1989; Ye & Alterovitz, 2017; Lawitzky et al., 2012) is commonly adopted as a standard method in the domain of robotics and many other areas.

Behavioral Cloning (Osa et al., 2018) is one of the underlying approaches that utilize a demonstrations as supervisory signal. The most relevant work to ours is (Peng et al., 2018) which leverages reinforcement learning to imitate natural motions. However, our method does not imitate specific motion trajectories but to generate natural and diverse actions that are reasonable between given states. Our method can be further differentiated from past literature in two aspects: (1) our method is orthogonal to imitation learning because it interpolates states rather than predicts actions when states are given. More importantly, (2) the proposed method is able to produce between unseen states while imitation learning focuses on imitating and capturing the demonstrated distribution.

4. EXPERIMENTS:



4.1. EVALUATION SETTINGS:

We use CMU Mocap4 and SFU Mocap5 datasets for evaluation. Both datasets contain diverse daily human motion sequences which are suitable for training. Considering that the original sequences possess different length, we get reference subsequence in a sliding window manner, where T = 120 and P = 40. For both datasets the dimension of state is 63, which is 21 joints with 3D coordinates.

Here we would like to emphasise that we keep a held-out reference set (denoted as D R ) from training data (denoted as D T ) for further testing, to demonstrate that our model could generate novel behaviour never seen during training. More specifically, D R is used for sampling novel motion and starting/ending states during testing. Note that the following all experiments are with SFU Mocap datasets. For the visual results on CMU Mocap datasets, please refer to our project page.

For local motion composition, we compare our model with two strong parametric baselines: VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al., 2014) . For fair comparison, we train both models with all data, i.e., D T and D R . Meanwhile we follow Yan et al. (2018) and Barsoum et al. (2018) about the hyper-parameter setting of both models. But the input dimension is adjusted to match our data. For global motion composition, we compare our model with temporal prediction baselines: (1) Baseline1: ϕ f pre without last 5 states as goal condition, (2) Baseline2: ϕ f pre with last 5 states as goal condition, (3) Baseline3: both ϕ f pre and ϕ bpre but without last/first 

4.2. EVALUATION FOR LOCAL MOTION COMPOSITION:

Is generated motion diverse enough? We compare our model with parametric model (i.e., VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al., 2014) ) in evaluation of motion diversity. For the majority of generative models, the diversity of generated result is upper bounded by training data Barratt & Sharma (2018) . Motivated by this, we compare motion diversity with VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al., 2014) After training converged, we calculate the averaged standard diviation of all joints with a higher value indicating more diverse. We can see that both parametric models keep relative constant motion diversity which is comparable with training data used less than 30%, but largely inferior to the diversity of full training data. On the contrary, with an increasing percentage of data used for training, our model achieves higher motion diversity, which mainly benefits from the general semi-parametric modelling framework.

Is generated motion visually natural? For evaluation of visual naturalness, we provide generated motion results and compare with VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al., 2014) .

As shown Fig 5 , from top to bottom generated motion sequences correspond to our model (COSMO), GAN (Goodfellow et al., 2014) and VAE (Kingma & Welling, 2014) , respectively. Note that all three sequences are with a length of 440 time steps. Our model (COSMO) generate the sequence with three reference subsequences. While both GAN (Goodfellow et al., 2014) shows that our semi-parametric modelling based model achieves natural state transition throughout the whole sequence, meanwhile provides natural and diverse motion (i.e., walking-turning-walking) behaviours during generation. We encourage readers to refer to our project page for more video demonstrations.

Do proposed model learn a meaningful style space? Recall that for representation learning of style feature, we construct an embedded space instead of directly generating style feature. In this way, we expect our model able to map a novel style pattern into learned embedded space. For validation we randomly select 10% training data then extract corresponding style feature (denoted as S train ) with ϕ sty . Regarding the held-out set (denoted as S ref ) we conduct the same operation for all reference subsequences. We visualize the distribution of both S trn and S ref with t-SNE (van der Maaten & Hinton, 2008) in a two-dimensional plane. As shown in Fig 4 , the purple dots indicate training data while green dots stand for reference data. We can see that the style feature of training data spread evenly across the plane. Meanwhile, the majority of style of feature of reference subsequences is covered by that of training data. Within the learned style space, embedded layer acts as a set of style bases where novel style can be approximated by the combination of these bases.

Can COSMO compose two reference subsequences into a novel one? Part of the generation diversity of our model results from free composition of reference subsequences. As shown in Fig 6 , we provide two edited sequence examples which possess the general motion style from one subsequence, but detailed motion pattern from another one. Taking the second sequence (bottom row in Fig 6 ) for example, the target style motion shows a spinning motion with both hands raised up (style), while the reference motion is a regular walking sequence. We can notice that the final edited reference motion (third column, bottom row) fully captures the style of upper body meanwhile maintains the walking pattern from reference sequence. Moreover, both top and bottom rows use the same reference motion, but with different style as inputs, our model can still produce highly diverse behaviours. For the space restriction of paper writing, we provide more examples whose new style features are sampled from constructed embedded space. We encourage readers to refer to our project page for more video demonstrations. We can observe that our model is able to generate smooth and natural transition when starting and ending states are similar. Moreover, when encountered large motion difference, e.g., from walking to greeting, turning back with a relatively large degree, our model still makes it to generate visually natural transition sequence. We encourage readers to refer to our project page for more video demonstrations.

Is global motion composition merely linear interpolation? One possible trivial solution learned by our transition model is to simply linear interpolation between starting and ending states. However, common human motion generally involves non-linear trajectories. Linear interpolation is prone to produce unnatural motion which is easy to detect by human eyes. To valid that whether our model conducts linear interpolation between two sequences, we record the height variation of the right foot generate long-range, diverse and natural motion sequences over 1000 time steps without loss of visual quality.

:

https://sites.google.com/view/cosmo-supp/home

http://mocap.cs.cmu.edu/

http://mocap.cs.sfu.ca/

http://mocap.cs.cmu.edu/

http://mocap.cs.cfu.ca/

Figures and Tables:

Figure 1: Comparison of different motion generation schemes. Three rows from top to bottom correspond to generated results of parametric, non-parametric and semi-parametric modelling respectively. Parametric model like GAN is prone to generate unnatural motion (twisted body in top row). Meanwhile the modal collapse is another longstanding issue for parametric model. Non-parametric model like simply copy and paste is infeasible to get visually smooth and long-range motion (second row). On the contrary, Our proposed semi-parametric model i.e., COSMO, is able to generate long-range motion sequences with diverse behaviours.

Figure 2: Proposed framework for motion generation. Step 1: We search for reference subsequences within a held-out set. Step 2: A novel subsequence is then generated based on reference one with local motion composition. Step 3: All edited subsequences are connected together in temporal order with global motion composition.

Figure 3: Motion diversity evaluation.Figure 4: Style feature visualization

Figure 3: Motion diversity evaluation.Figure 4: Style feature visualization

Figure 6: Visualization of local composed reference sequence.

Figure 7: Evaluation of global composition in terms of training MSE and foot height variation.

Figure 8: States transition visualization for evaluation of smoothness.

Figure 9: Left: Motion generation given the same starting and ending states. Right: Long-range Motion generation with characteristic routes.

(denoted as s 0 and s L respectively) as inputs, our goal is to synthesize intermediate states that have natural transition and diverse behaviors. As illustrated in Fig 2, N R reference subsequences (denoted as R i , i = 1, .., N R ) are sampled and fed into local motion composition

Table content: <table><tr><td>2.1 SEMI-PARAMETRIC MODELLING FOR MOTION GENERATION Reference Motion Set …… Reference 1 Reference 2 Reference N R Reference Motion Search Reference i New Style Motion Local Motion Composition Edited Reference i Global Motion Composition Edited Reference i+1 Local Motion Feature Style Feature Concatenate Edited Reference i Motion Gap Given starting and ending states Input Route Bi-directional Modelling</td></tr></table>

based models under different percentages of data used for training. As illustrated in Fig 3, 10%, 30%, 50% and 100% training data are used respectively.

Table content: <table/>

Peer review (iclr2020):
1. Summary: The paper proposed **composable semi-parametric modeling** for generating long-range diverse and distinctive behaviors to achieve a specific goal location. The non-parametric part is a memory bank that is used to retrieve motion patterns from source materials. The parametric part contains several deep neural networks which are to compose the retrieved materials for high quality and smooth motion generation. The overall idea is novel in the sense that they aim to combine the strength of the non-parametric method (with rich pattern and diversities) and the parametric method (powerfull ability to generate coherent results). The proposed ideas are evaluated on two datasets and outperform compared approaches qualitatively. 2. I am borderline to this paper but prone to weakly reject this paper. (1) My major concern about this paper is the lack of reasoning about each component. - local motion composition: How does the design help encode both style and content information in the outputs? Since reconstruction loss is utilized, the optimal should be to learn an identity mapping. How does the author avoid this to happen? The paper referred to the style embedding features H_s, but it is hard to understand how this is associated with the architecture. Also, the usefulness of local motion composition is not well illustrated in the experiments part. -global motion composition: How the network is supervised to compose different clips together? It seems that the authors again adopt reconstruction loss, how the loss penalize inconsistent predictions between clips is not well explained. (2) The overall presentation of this paper is hard to follow. I spent a lot of time on understanding the notations and corresponding networks. For example, the input representation is not well stated and the author referred to phi_loc without any indication in Fig, 2. It*s very hard to understand the details of the method. I would like to suggest that the author improves the writing especially for Section 2. (3) Does the style motion shared by all sub-sequences? (4) Figure 5 shows a sequence. however, the COSMO result seems not natural for me since the velocity in the sequence changes. I would like to change my rating if my questions are well addressed in the rebuttal. Overall, this is an interesting idea.