Single Motion Diffusion
Authors: Sigal Raab
Inbal Leibovitch
Guy Tevet
Moab Arar
Amit Haim Bermano
Daniel Cohen-Or
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep Learning
Motion synthesis
Animation
Single Instance Learning
Generative models
Primary area: generative models
TL;DR: We present a model designed to learn the internal motifs of a single motion sequence with arbitrary topology and synthesize diverse motions that are faithful to the learned motifs.

Abstract: Synthesizing realistic animations of humans, animals, and even imaginary creatures, has long been a goal for artists and computer graphics professionals. Compared to the imaging domain, which is rich with large available datasets, the number of data instances for the motion domain is limited, particularly for the animation of animals and exotic creatures (e.g., dragons), which have unique skeletons and motion patterns. In this work, we introduce SinMDM, a Single Motion Diffusion Model. It is designed to learn the internal motifs of a single motion sequence with arbitrary topology and synthesize a variety of motions of arbitrary length that remain faithful to the learned motifs. We harness the power of diffusion models and present a denoising network explicitly designed for the task of learning from a single input motion. SinMDM is crafted as a lightweight architecture, which avoids overfitting by using a shallow network with local attention layers that narrow the receptive field and encourage motion diversity. Our work applies to multiple contexts, including spatial and temporal in-betweening, motion expansion, style transfer, and crowd animation. Our results show that SinMDM outperforms existing methods both qualitatively and quantitatively. Moreover, while prior network-based approaches require additional training for different applications, SinMDM supports these applications during inference. Our project page, which includes links to the code and trained models, is accessible at https://sinmdm.github.io/SinMDM-page.

Published as conference paper ICLR 2024
SINGLE MOTION DIFFUSION
Sigal Raab", Inbal Leibovitch", Guy Tevet, Moab Arar, Amit H. Bermano and Daniel Cohen-Or Tel Aviv University, Israel {s ssigal. raab, inbal. leibovitch) @ 1 com
ABSTRACT
Synthesizing realistic animations of humans, animals, and even imaginary creatures, has long been a goal for artists and computer graphics professionals. Compared to the imaging domain, which is rich with large available datasets, the number of data instances for the motion domain is limited, particularly for the animation of animals and exotic creatures (e.g., dragons), which have unique skeletons and motion patterns. In this work, we introduce SinMDM, Single Motion Diffusion Model. It is designed to learn the internal motifs of a single motion sequence with arbitrary topology and synthesize a variety of motions of arbitrary length that remain faithful to the learned motifs. We harness the power of diffusion models and present denoising network explicitly designed for the task of learning from a single imput motion. SinMDDM is as a lightweight architecture, which avoids overfitting by using a shallow network with local attention layers that narrow the receptive field and encourage mo- tion diversity. Our work applies multiple contexts, including spatial and temporal en-betweening, motion expansion, style transfer. and crowd animation. Our results show that SinMDM outperforms existing methods both qualitatively and quantita- tively. Moreover, while prior sawork-based approaches require additional training for different applications. SinMDM supports these applications during inference. Our project page, which includes links to the code and trained models, is accessible at https: / rmmdm. . git..b. ic/ MMDM-page
1 INTRODUCTION
Animation of 3D characters is a long pursued task in computer graphics with many applications, from the big screen to virtual reality headsets. It is notoriously known as a time-consuming task done by expert artists. In recent years. neural models have offered faster and less expensive tools for modeling motion (Holden et al.]
2016: Petrovich et al., 2022] Raab et al. 2023). In particular, the very recent adaptation of diffusion models into the motion domain provides unprecedented results in both quality
and diversity (Tevet et al. 2023 Kim et al. 2022).
These data-driven methods typically require large amounts of data for training. However, motion data is quite scarce. Moreover, for non-human skeletons, it barely exists. The few available datasets contain humanoids only, with fixed topology and bone ratio. Animators often customize skeleton per character (human, animal, or magical creature), for which common data-driven techniques are irrelevant.
In this work, we present a Single Motion Diffusion Model, dubbed SinMDM, that trains on a single motion input sequence. Our model enables modeling motions of arbitrary skeletal topology, which often have no more than one animation sequence to learn from. An arbitrary topology refers to the agnostic nature of the architecture towards skeletal structure. Each motion sequence (of any topology) requires its own trained model. SinMDM can synthesize variety of variable-length motion sequences that retain the core motion elements of the input and can handle complex and non-trivial skeletons. For example, our model can generate a diverse clan based on one flying dragon or one hopping ostrich.
In the motion domain, learning from a single instance has been explored by Li et al. (2022) using GANs (Goodfellow et al., 2014) and by Li et al. (023) using Motion-grapii (Kovar et al. 2002). We advocate diffusion models (Ho et al. (0220) for single input learning, as gradual denoising enhances descriptive capability and avoids mode-collapse using a rather lightweight scheme that, compared the prior art, generates high-quality motions. Furthermore, we demonstrate that diffusion models can be effectively utilized with limited data, challenging the notion that they solely rely on large datasets.
To learn local motion sequences, the receptive field must be small enough, analogously to the use of patch-based discriminators (Isola et al., 2017] [Li & Wand, 2016) in GAN-hased techniques. The use of narrow receptive field (Fig. 2 promotes diversity and reduces overfitting. We show the importance of narrow receptive fields our ablation studies.
Most motion diffusion models use transformers (Vaswani et al. 2017). However, vanilla transformers are not suitable for learning a single sequence, as their receptive field encompasses the entire motion.
"qqul contribution.

---

Published as conference paper ICLR 2024
Figure 1: SinMDM learns the internal motion motifs from single motion sequence with arbitrary topology and synthesizes motions that are faithful to the learned core motif's of the input sequence. Left: girl exercising while walking. Right: a breakdancing dragon. Left to right: breakdance uprock, breakdance freeze, and breakdance flair.
A similar challenge arises with UNets (Rouneberger et al., 2015), commonly used in image diffusion models. Its depth, along with global attention layers, creates a receptive field covering the whole motion.
SinMDM leverages the concept of narrow receptive fields and introduces motion architecture specif- ically designed with this concept. It combines a shallow UNet model adapted for motion with a QnA (Arar et al. 2022) local attention mechanism, instead of global attention. As result, SinMDM outpertorms prior art both quantitatively and qualitatively, and demonstrates high efficiency with shorter training time and less memory consumption. Imputed to its lightweight architecture, SinMDM can be trained on single mid-range GPU.
We present many use cases of SinMDM. While neural-network -based prior arts require designated train- ing per application, ours are applied at inference time, with no need to re-train. Moreover, applications that require different dedicated algorithms in prior art, are here grouped together special cases of the same technique, significantly simplifying their use. One of the applications we present is Motion Composition, where a given motion sequence is composed jointly with a synthesized one, either tem- porally spatially. Its special cases include in-betweening and motion expansion. Another application that we present Harmonization, along with its special case, style transfer. Here, a reference motion modified to match the learned motion motifs. It should be emphasized that implementing style transfer using denoising model is non-trivial task, and enabling it through motion harmonization is unique. We further present two more applications: long sequence generation and crowd animation.
In our work, we suggest two comprehensive benchmarks for single-motion evaluation. The first is built upon the artistically crafted MIXAMO (202T) dataset, utilizing metrics that do not require additional Ceature-extracting model. The second is based on the HumanML33 (2022) dataset and enables metrics that use latent features, such as SiFID (Shaham et al. 2019). We show that our model outperforms current works both benchmarks.
Works with limited fidelity or diversity may excel in one metric, but struggle in others. Thus, rather than favoring models that shine on specific criterion, we use the Harmonic Mean metric, which balances given scores and quantifies their weighted performance.
2 RELATED WORK
Single-Instance Learning The goal of single-instance generation to learn an unconditional gener- ative model from a single instance, capturing patch-level statistics for generating diverse content. The instance type varies by input domain, with a majority in imaging. The first works on this topic are SinGAN (Shaham al. 2019) and InGAN (Shocher al.) 2019). SinGAN uses patch-based discrim- inator (Isola et al. 2017 Li& Wand 2016) and an image pyramid to hierarchically generate diverse re- sults. InGAN uses conditional GAN to solve the same problem using geometry transformation. More recent approaches include ExSinGAN (Zhang al. 2021c), which trains multiple modular GANs model the distribution of structure, semantics, and texture, and ConSinGAN (Hinz et al.] (021). which trains several stages sequentially and improves SinGAN. Many works in the imaging domain follow and improve these works (Asano al., (2020) Granot et al. 2022 Chen al. 2021 Lin et al. 2020
Sun & Liu 2020. Sushko et 2021 Yoo & Chen 202 Zhang et al. 20226 Zheng et al. (021))
Similar to Wang et al. (2022) and Nikankin et al. (2023) avoid the pyramid structure and use a
UNet with limited depth. This is not directly applicable the motion domain. Unlike images with a
regularized 2D spatial structure, motions consist of non- non-rreularrizdd skeletal joints with a temporal axis and fewer degrees of freedom. Kulikov et al. (2023) construct a multi-scale diffusion process from down-sampled versions of the training image and their blurry versions.
Several works have been done in other domains, e.g., shapes (Wu & Zheng 2022) and 3D scenes (Son jet al. 2023). In the motion domain, Ganimator (Li et al. 2022) follows SinGAN and uses a GAN architecture, with a patch-based discriminator and a temporal pyramid. GenMM (Li al. 2023) follows
2

---

Published as conference paper ICLR 2024
2022). ) and employs non non-parametric patch nearest-neighbor methods, implemented with motion-graphs Kovar et al. 2002), and yielding higher-quality outputs with significantly reduced generation time. Yet, nearest-neighhor methods have limited generalization capabilities, and hence are mainly suitable for tasks involving "copying" parts of the input. In this respect, learning-base -haiin methods, like Ganimator, offer more applicability as shown in their styl-transserr task.
GPNN (Granot et al.
Diffusion Models Diffusion models have been adapted from thermodynamics (Sohl-Dickstein et al. 2015] Song & Ermon, 2020) to the imaging Ho et al. 2020; Song et al., 2020a). A generated image can be further controlled by classifier (Dhariwal & Nichol 2021) or classifer--rre (Ho & Sali-
mans 2022) guidance. Local editing of images may be viewed as inpainting problem, in which a portion of the image is held constant while the model denoises the remaining part (Song et al. 2020b; Saharia et al. 2022). In our work, we adapt this technique for spatial and temporal motion composition.
In the motion domain, several works (Zhang et al.
2022a Kim et al., 2022) introduce diffusion- -aase 2023). which utilizes a lightweight network, uses a
synthesis. The prominent one is MDM (Tevet et al. transformer rather than the common UNet and predicts motion rather than noise. Like MDM, SinMDM presents a lightweight architecture and predicts motion rather than noise. Unlike MDM, our work uses a QnA-based UNet, as the receptive field of a transformer is the full motion, inducing over-fitting.
Motion Synthesis Models
Most of motion synthesis models focus on specific tasks, conditioned on
some limiting factors, which can be high-level guidance such as action (Petrovich et al.) |021] Guo
at... 2020)
[0000/Cervantes Cervantes al. 2022) text (Tevet et al., [022] ZZhng al.
202 Petrovich et al. 2022
Ahuja & Morency] 2019; Guo et al. [2022; Bhhatacharya et [2021] Tevet et al.
2023), can be parts of a motion such as motion prefix (Aksan et 2019 Barsoum et al. 2018 Habibie et al., 2017; Yuan &
Kitani 2020;
Zhang et al.) [202 Hernandez al.
2019) or in-betweening (Harvey et al. 2020) Duan
ct al.
2021 Kautmann et al. [2020] Harvey & Pal
2018).
motion retargeting or style transfer (Holden
et al.
2017 Villegas et al., 2018) Aberman et al.
2019
2020a by. and even music (Aristidou et al.
2022
Sun et al.] 2020 Li et al. L02T]Leee al. (0018). Fewer models are fully unconditioned (Holden
ct al.
2016 Raab et al. [223] Starke et al.] (2022).
The architecture of synthesis models can be roughly divided into recurrent (Habibie et al., 2017 Ghor-
bani et al.) 2020), autoencoder based (Maheshwari et al. 2022 Jaang & Lee (020), GAN based (De-
gardin et al. [202]] Wang et al. [2020] Yan et al. 2019 Yu et al.
2020), normalizing flows based (Henter
et al. 2020). and more recently, neural field based (He et al.
2022) and diffusion based (Tevet et al.
2023
Zhang et al. 2022a Kim et al. 2022 Shafir et al. 2023).
Ourwork belongs to the latter category.
3
PRELIMINARY
In this work, we present SinMDM, a novel framework that learns the internal motion motifs of a single motion of arbitrary topology and generates a variety of synthesized motions that retain the core motion elements of the input sequence. At the crux of our approach lays a denoising diffusion probabilistic model-DDie - models offer generalization without being susceptible to mode collapse.
(Ho et al. 2020). See Appendix [or a recap of DDPM. Our premise is that diffusion
We present a lightweight model, efficient in time and space and simple in architecture. This achieved through the gradual denoising process, which enhances the model's descriptive capability. Our gener- ative network is UNet (Ronneberger al. 2013) whose global attention layers are replaced by local QnA layers (Arar et al. (022)..
In the rest of this section we describe our motion representation. Next, we describe our method and de- sign choices (Sec./4p, 4p. explore various applications enabled by SinMDM (Sec. (5). detail the experiments conducted to validate our approach (Sec. (.. and summarise with conclusions (Sec. (7). The readers are encouraged to watch the supplementary video to get a full impression of our results.
Motion representation A motion sequence is represented by its dynamic and static features, D and S. respectively. The former differ at each temporal frame (e.g.. rotation angles), while the latter is temporally fixed (e.g.. bone lengths). D and S can be combined into global 3D pose sequences using forward kinematics (FK). In our research, we focus on synthesizing the dynamic features, leaving the static features intact. That is, we predict dynamics for a fixed skeleton topology with fixed bone lengths. For simplicity, we use the term motion synthesis for the generation of dynamic features only.
Let N denote the number of frames in a motion sequence, and F denote the length of the features describing single frame. In the HumanML3D dataset, a frame is redundantly represented with the root position and joint positions, angles, velocities, and foot contact (Guo et al., (022). For the other datasets used in this work, frame is represented by joint angles, root positions, and foot contact labels.
We represent the dynamic features of a motion by a tensor D E RNx XFF Naturally, the convolution for this representation is ID, convolving on the temporal dimension (of size N) and holding F features. More motion representation details can be found in Appendix
3

---

Published as conference paper ICLR 2024
000
Cony Norm Shn And Sea
BLU Cotv Norm
Narrow Receptive Field
UNet
ResBlock
Pa (x t)
QnA
O0O
To allow training on a single motion, our denoising network designed such that its overall receptive field covers only portion of the input sequence. This effectively allows the network to simultaneously learn from multiple local temporal motion segments. Our denoiser predicts the input sequence from noisy one. zi ... 2 is motion of N frames at diffusion step t. Right: Our network is a shallow UNet, enhanced with a QnA local attention layer.
Figure 2: Left:
4 GENERATIVE NETWORK
Our goal is to construct a model that can generate a variety of synthesized motions that retain the core motion motif's of single learned input sequence. More formally, we would like to construct the generative network Po (Eq. 2 Appendix(A) that synthesizes a motion &g from a noised motion zt.
Unlike traditional Lingl-insttane methods that utilize down-sampled instance pyramids for coarse-to- fine learning, our model employs a straightforward architecture without requiring any pyramids.
Our key insight is that internal motifss are learned more effectively with limited receptive field (Fig. 2 Left). We design SinMDM, novel generative architecture, accordingly. Our model QnA-based degenerated UNet (Fig. 2 Right). UNets (Ronneberger et al., 2013) are commonly used by imaging diffusion models (Nichol et al. 2022). However, training a UNet on single-input leads to overfitting due to its large receptive field, yielding synthesized sequences that closely resemble the input.
Our first design choice in mitigating this issue is to decrease the depth of the UNet and thereby limit the receptive field width. However, this step alone is not enough, since standard UNets employ global attention layers, resulting in a receptive field that encompasses the entire sequence. A possible solution would be using local attention in non-overlapping windows, like in ViT (Dosovitskiy et al., (021). Nonetheless, non-interleaving windows tend to limit the cross-window interaction, compromising the model's performance. Our solution to use QnA (Arar et al. 2022). a state-of-the-ar shift-invariant local attention layer, that aggregates the input locally in an overlapping manner, much like convolutions, but with the expressive power of attention. The key idea of QnA is to introduce learned queries, shared by all windows, allowing fast and efficient implementation. Specifically, QnA enables local attention with a temporally narrow receptive field. Our QnA-based UÑet is the first to be used in the motion domain, where we plug QnA layers instead of the global attention layers of a vanilla UNet. In this context, a vanilla UNet refers to the UNet utilized early imaging diffusion works (Ho et al.) 2020).
QnA is substantially more efficient than global attention in space and time, and our model benefits from this advantage
byproduct. A detailed description of the QnA layers is available in Appendix [D]
In Sec. 6 we validate these design choices. We show the effectiveness of a narrow receptive field, and justify the usage of QnA layers and the choice of a UNet rather than a transformer. In Appendix C we provide comprehensive list of the hyperparameters that can be used to reconstruct our results.
5 APPLICATIONS
Single-motion learning using diffusion models enables various applications. All our applications are applied at inference time, with no need to re-train. Unlike SinMDM, Ganimator 2022 requires spe- cialized training for most of its applications, and GenMM (2023), limited to replications hence cannot handle complex applications like harmonization. In the following, we show Motion Composition (Sec. 5.I] where a given motion sequence is composed jointly with synthesized one, either temporally or spatially. Its special cases include in in-berweening, motion expansion, trajectory control, and joints control. With our Motion Harmonization (Sec. 5.2). a reference input motion is altered to align with the learned motion motifs. We illustrate one important special case, style transfer. Lastly, we show how straightforward use (Sec. [5.3) of SinMDM enables one shot long motion generation and crowd animation. The applications presented here are also demonstrated in our supplementary video.
5.1 MOTION COMPOSITION
Given arbitrary reference motion y unseen by our model, and region of interest (ROI) mask (Avrahami et al., 2022), our goal is to synthesize a new motion åo such that the regions of interest 2o O m are synthesized from random noise, while the complementary area remains as close as possible to the given motion y, i.e., y (1 - m) x 20 @ (1 - m), where @ is diement-wisc multiplication. The model should output coherent motion sequence, where the transition between given and synthesized
4

---

Published as conference paper ICLR 2024
time
time
Figure 3: Temporal composition motion expansion. Motion pairs show varied synthesis from a single input. The input, distinguishable by its faded color, temporally expanded. Note that the parts given input are identical in both sequences, while the synthesized parts differ. Left: synthesize a suffix given a temporal prefix. Right: synthesize prefix and suffix, given the middle part.
Palka
Pa
coo
Fract
DOo
As
Composition
DUD
A--1
OOO
%
Compostion
Frames
Spalial Com compooatoon
HP
LP
y5
X
Composition
Figure 4: Motion composition. Parts from refer- ence motion y, are composed with the synthesized motion &o. according composition map.
Figure 5: Harmonization. During inference, we inject guidance from input yo by adding its low frequencies ! in each step t.
parts is seamless. Moreover, the reference motion can be an arbitrary one, on which our model has not been trained. When the reference motion is very different from the learned motion, blending between the becomes less smooth. To mitigate this issue, we change the ROI mask such that the borders between the given and the synthesized motion segments are linearly interpolated, as depicted in Fig. 4 We fix the motion segments that need to remain unchanged and sample the parts that need to be filled in. Each step of the iterative inference process (described in Appendix A) slightly changed, such that parts of y are assigned into åg according to the indices of the mask. That is, &g O (1-m) = (1 (1-m).
Temporal composition use cases: in-betweening, motion expansion Temporal composition is the action of filling in selected frame sequences. In- tn-betweeining (Harvey et al. 2020). depicted in Fig. is special of temporal composition, where the filled-in part at the temporal interior of the sequence, and the reference y not necessarily from the same distribution as the learned motion. Another special case of temporal composition is motion expansion, the motion domain's 's equivalent of image outpainting (Yu et al. 2019 Lin et al. 2021 Teterwak et al. 2019), where the model generates content that resides beyond edges of reference motion sequence. In the case of motion expansion, the ROI mask zeroed in the center frames, and assigned ones in the outer regions. See Fig.
Spatial composition use cases: trajectory control, joints control Motion composition can be applied spatially, by assigning selected joint indices to the ROI mask. In Fig. 7 we illustrate control over the upper body, where the motion of the upper body determined by reference motion and assigned to the target motion. The model synthesizes the rest of the joints yielding a motion with the given sequence in the upper body, and with the learned motif's in the lower body. A composition can be both spatial and temporal, and all it takes is an ROI mask where several frame sequences are zeroed, i.e. taken from the reference motion, and in the complementary part, several joints are zeroed (see Fig.
5.2 MOTION HARMONIZATION
Given synthesized motion sequence To. would like to integrate portion of an unseen motion, y. into The portion of y can be either temporal, i.e., several frames, or spatial, i.e., several joints, or both. As visualized in Fig. 5 SinMDM overrides a window in 2o with the desired portion of and denotes the outcome yo. Next, Yo is harmonized such that it matches the core motion elements learned by our model, using a linear low-pass filter dN as suggested by Choi et (2021). Let s/-1 and yt-1
5

---

Published as conference paper ICLR 2024
time
time
Figure 6: Temporal composition In-betweening. Both rows show results for the same reference input, introducing diverse outputs. The beginning and the end (in faded tones) given, hence identical in both rows. The center each motion is synthesized, hence the dif- ference between rows.
Figure Spatial composition. Top: "warm- up" unseen reference motion. Bottom: composed motion. The learned motion "walk in circle". In the composed result, the top body part performs a warm-up activity, and the bottom walks in curved path.
Content
"Happy" style
'Crouched' style
Result
Result
Figure 9: Style transfer. Top: one unseen content ap- plied to both styles. Left: "happy" style learned by the network, with the harmonized result below. Right: a "crouched"" style. Note the character in both results is us- ing the exact step rhythm and size as in the content motion.
Figure 10: Crowd animation. Groups of jaguars, horses, and ostriches. In each group, no motion is like the other, and yet they are all learned from single mo- tion sequence.
denote the noised version of motions pe (x () and yo. respectively. The high-frequency details of s are added to the www-freueeccy of 3t-1 via #T-1 @x(y)-1) 2/-1 (2(2)-1).
Note difference between harmonization and motion composition: Both assign parts of an unseen sequence into a synthesized motion However, harmonization changes the assigned part such that it matches the learned distribution, while composition aims keep it unchanged.
Style transfer We implement style transfer (Fig. 9 special case of harmonization, where we use all of y stead of a portion of it, hence fully overriding zg. We use model-learned style motion and unseen content mo- tion y. After harmonization, the result combines y content with z's style.
5.3
APPLICATIONS
Long motion. "a person walking back, turning, and walking back again". Learned 10-second sequence and synthesized to 60 seconds.
Figure 8:
In this section, present applications that may require unique methods in prior research but pose no such challenge when employing our model.
Long motion sequences Our model can synthesize variable-length motions, even very long ones, with additional training. Imputed to its small receptive field, the model can hallucinate sequence as long as requested. An example one-minute animation is introduced in Fig.
6

---

Published as conference paper ICLR 2024
Crowd animation Although trained on single sequence, during inference SinMDM can generate a diverse crowd, each sampled from a different Gaussian noise æT (N 1), as shown in Fig. 10
6
EXPERIMENTS
Our experiments are held on data from the HumanML3D (2022). Mixamo (2021). and Truebones Zoo (2022) datasets, and on an artist-created animation, using an NVIDIA GeForce RTX 2080 Ti GPU.
6.1 BENCHMARKS
We test our framework on two benchmarks, based on the HLmanML33D and the Mixamo datasets. These two datasets are different in many aspects. The data in HumanML3D fits the SMPL (Loper et al.) 2015) topology, and its users normally use SMPL's mean body definition. In contrast, Mixamo provides 70 characters, each possessing their unique bone lengths and some possessing unique topologies. In addition, the motions in the Mixamo dataset are more diverse and more dynamic.
6.2 METRICS
For the Mixamo benchmark, we use the metries introduced in Ganimator (Li et al. these metrics are based on the values of motion features (e.g.. rotation angles) while the usage of deep features the current best practice (Zhang et al. 2018). Given HumanMLL3D's capability for deep feature calculation, we utilize it to present our results specifically on these features.
2022).
However,
We define a good score being either high low depending on the specific metric's preference for higher or lower values. Note that a balance of good scores across all metrics is better than excelling in just few. In particular, good diversity scores coupled with poor fidelity indicate deviation from the input, while good fidelity scores paired with bad diversity suggest overfitting.
An ideal outcome is a combination of good values for all metrics. For models with mixed scores, a better-scoring model is the one whose scores are more balanced. To this end, we follow established literature (Rijsbergen 1979 Chinchor 1992) and suggest the Harmonic Mean metric, which is widely used in Machine Learning for this purpose (Taha & Hanbury 2015). We compute it as follows: first, we normalize the scores for each metric. Normalization is between zero to the metric's maximum value. If the maximum is not known, we select the 90% percentile of the computed scores. For metries where lower is better, we subtract the score from the maximum value. Note that a negative value is therefore
valid. We compute the Harmonic Mean via HM = E/ +...+ ) where E is the number of metries in a table and s; is the normalized score of metric i. Additionally, our radar plots (Fig, [I] provide visual representation of SinMDM's dominance for both benchmarks, considering all metries.
Metries on the Mixamo benchmark Our comparison with prior art (Li et al. 2022 the Mixamo dataset, which is also used by these works. We use metrics suggested by Li et al. 2022).
2023 is held on
This group of metrics is applied to the motion itself, and not deep features. It consists of (a) coverage, which is the rate of temporal windows in the input motion xg that are reproduced by the synthesized one, (b) global diversity, measuring the distance between tess(2n) (2)) and zo. where tess(-) is a tessellation that minimizes the L2 distance to the input sequence, and (c) local diversity, which is the average distance between windows in the synthesized motion in and their nearest neighbors in the input one.
However, these metrics lack important qualities which we address here: (d) inter diversity, the diversity between synthesized motions. We define intra diversity to be the diversity between sub-windows inter- nal to a motion and define (c) intra diversity diff. which is the difference between the intra diversity of the synthesized motions and that of the input motion.
Additionally, we measure the following time-space efficiency values: (f) number of network parameters, (g) number of iterations, (h) iteration time, and (i) total running time (the product of the last two). A higher score is better for metrics (a)-(d), and a lower one is better for (e)-(i).
Metrics on the HumanMI.3) benchmark We use this benchmark to measure metrics that are ap- plied on deep features, obtained with a motion encoder by Guo et al. (2022). The computed metrics are (a) SiFID (Shaham et al. 2019). which measures the distance between the distribution of sib-wwssows in the learned motion and a synthesized one, (b) inter diversity, which is the LPIPS distance (Zhang) ct al. 2018) between various motions synthesized out of one input, and (c) intra diversity diff. which the difference between the intra diversity of the synthesized motions and that of the input motion, where intra diversity is the I.PIS distance between sub-winn uub-windows in one synthesized motion. For metrics (a) and (c) lower is better, and for metric (b) higher better.
6.3 QUANTITATIVE RESULTS
We use the following color scheme in our tables: regular metrics are in grey, the Harmonic Mean metric, which weighs all scores, black, and the best scores are in bold.
7

---

Published as conference paper ICLR 2024
Coverage
SIFID
Tota Time
Global Div.
SIMMUM
MDM
Genimator
MDM on crops
GenMM
Ganimator
NParams
Aocal DIV.
Intra D.A. Dift
Inter Div.
Dift.
(a) Mixamo benchmark.
(b) HumanML3D benchmark.
Figure 11: Benchmark results, depicted as radar plots. Our results present balanced combination of high scores. The displayed scores were first converted 'higher is better' and normalized (Sec. 6.2).
Tab. D and Fig. Ia wC compare SinMDM with prior works and show that it outperforms them. All the metrics are computed separately on each benchmark motion and then averaged. The metrics that measure time computed benchmark motion number 9 only.
To evaluate our performance vs. another motion diffusion model, we compare it with two variations of the MDM (Tevet et al., (023) framework. The first is a vanilla MDM, trained on single-motion. The second is a variation MDM in which we extract short crops out of the angle-motion sequence and train an MDM on them. Note that the second variation holds a narrow receptive field.
The comparison is conducted results are shown in Tab. 2 and Fig. TIb MDM yields complete overfit, thus its SiFID and intra-diversity scores are perfect (indicating similarity to the input motion), but its inter-diversity scores are low. The overfit of MDM is caused by the global attention it uses. On the other hand, the quantitative results for the second MDM variation indicate divergence from the input motion motifs. These quantitative results are supported by the qualitative results in our supplementary video.
the HumanML3D dataset with metrics based deep features. The
Finally, we reinforce our quantitative results in a user study in which users evaluate model superiority based on diver- sity, fidelity, and quality. In the study, we compare our model vs. other works. Each pair of models is com- pared over 8 different motions, and each such comparison is judged by 10 distinct users. The results (Fig- [2] show that our model is significantly preferred by the users. Screen- shots from our user study are provided in Appendix
Divensity
83.0%
Ours Ganimator
Eddlity
65.0%)
Quality
76.2%
Curs Diversity MPM Inwined
Rddlity
100.00 0%
on crops
Quality
96 2%
Figure 12: User study. The dashed line marks 50%.
6.4 QUALITATIVE RESULTS
Our supp video reflects the quality of our results. It presents multiple synthesized motions, as well as comparison to other works. Additionally, Fig- [3 depicts SinMDM current work. Other works exhibit mode collapse, overfitting, or jittery motion, while SinMDM demonstrates a coherent motion.
6.5 ABLATION
We examine architectural variations on the HumanML3D benchmark and present the results in Tab. First, as many motion diffusion models favor a transformer
UNet (Tevet et al. 2023 Kim et al. 2022), , we measure the scores for a QnA-based transformer (row 1). To refrain from overfitting we apply QnA layers within the transformer as we do with the UNet. In addition, to promote diversity and permit the rearrangement of motion components,
employ relative temporal positional embeddings (Shaw et al. 2018 Press et al. 2022, Su et al.) 2021) instead the existing global ones. However, QnA- based transformer attains a bad SiFID score, indicating poor fidelity to the input motion.
Table Mixamo benchmark. SinMDM demonstrates a significant advantage the harmonic mean metric. Compared to Ganimator, it leads in all metrics but one. Recall that GenMM requires no training time/space, reflected in zero values at the right-hand side of the table. However, its inability to generalize results in inferior scores for most diversity metrics, leading to lower harmonic mean.
Coverage
Global Div.
Local
Inter,
Intra Div.
#Param. (M)
#lter. (K)
Iter.
Tot. Time (h)
Harmon.
Div.
Diff.
Time (s)
Mean
Ganimator
94.3
1.24
0.09
0.13
21.7
60
(15x4)
0.36
6.0
-0.51
GenMM
98.7
0.4
0.35
0.13
0.05
0
0
0
0
0.56
SinMDM (Ours)
94.3
1.42
1.00
0.13
0.03
5.26
60
0.09
1.5
0.84
8

---

Published as conference paper ICLR 2024
Table 2: HumanML3D benchmark. Compar- ison with two variants of the diffusion model MDM. The first exhibits overfitting and the sec- ond lacks fidelity to input. Our model attains good and balanced scores in all metrics.
Table 3: Ablation the HumanML33 bench- marks. Row 1: QnA-based transformer. Rows 2,3: comparing receptive field widths. Row 1 exhibits divergence and Row 2 indicates overfit. Abbr.: r.f. - receptive field, d -> depth.
SiFID.
Inter Div.
Intra Div. Diff.*
Harmon, Mean
SIFID
Inter Intra Div.
Hammon. Mean
Div.
Diff.
MDM 2023
0.01
0.14
0.05
Transforner w/ QnA
5.99
1.74
0.57
0.47
MDM crops
13:94
1.64
1.83
-0.46
UNet w/ QuA
Ganimator
11.01
0.87
0.86
0.15
wide r.f. (d=3)
0.69
0,20
0.34
0.28
SinMDM (Ours)
1.87
0.73
0.40
0.60
narrow (d=1) (Ours)
1.87
0.73
0.40
0.59
GT
Ganimator
MDM
GenMM
on craps
time
SinMOM
time
SinMDM
(a) Mixamo dataset. Motion punch to elbow. Sin- MDM outperforms Ganimator, which experiences mode collapse, and is comparable in quality to the patch nearest-neighhor method GenMM.
(b) HumanML.3 dataset. MDM exhibits overfit, and MDM trained on crops exhibits jittery motion, egg., when transferring from standing jumping without bending the knees before and after.
Figure 13: Qualitative comparison.
We continue by confirming that narrow receptive field produces good results while wider one induces overfit (rows 2.3). In order do so, we examine a fixed architecture (QnA-based UNet) with two receptive field widths. We control the width by tweaking the depth of the UNet. Indeed we observe that the model with the wide receptive field overfits (replicates) the input motion, as its inter-diversity is bad while its SiFID and intra-diversity are good.
Note that due to the mixed scores (that indicate either overfit or divergence), the usage of the Harmonic Mean metric is essential as it allows for the assessment of the combined strength of all scores. In addition, Appendix F we show an ablation study of the smoothness fidelity score.
7 CONCLUSIOON
We have explored the use of diffusion models on single motion sequence synthesis and designed a motion denoising transformer with a narrow receptive field. Training on single motions is particularly useful in motion domains, where the number of data instances is scarce. Particularly, for animals and imaginary creatures, which have unique skeletons and distinctive motion motifs. The motion of such creatures cannot be captured easily nor learned from the human motion data available.
Our experiments across datasets show that our lightweight diffusion-basee method significantly outper- forms current work both quantitatively and qualitatively. Moreover, our approach allows the synthesis of particularly long motions and enables variety of motion manipulation tasks, including spatial and temporal in- tn-betwenning, motion expansion, harmonization, style transfer, and crowd animation.
The innate limitation of our method, common to single-instance models in all domains, is the limited ability to synthesize nut-of-distribution. Another limitation, also common to all single-instance models, is the inability to set generated sub-motions dance moves). This can be addressed by enlarging the receptive field (at the cost of lower diversity). The main limitation of our approach, due to the iterative nature of diffusion models, is the relatively long inference time.
specific order, when such order matters (c.g., certain
Finally, our work shows that diffusion models can learn from limited data, contrary to their reputation for requiring large amounts of data. Nevertheless, in the future, we aim to address the single input limi- tations, by possibly learning from available motion data of creatures with rather compatible skeletons.
9

---

Published as conference paper ICLR 2024
8 ACKNOWLEDGMENTS
We are grateful to Panayiotis Charalambous, Andreas Aristidou and Brian Gordon for reviewing ear- lier versions of the manuscript. This research was supported in part by the Israel Science Foundation (grants no. 2492/20 and 3441/21), Len Blavatnik and the Blavatnik family foundation, and the Tel Aviv University Innovation Laboratories (TILabs).
REFERENCES
Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Learning character- agnostic motion for motion retargeting in 2d. ACM Transactions on Graphics (TOG), 38(4):75, 2019.
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Dorkine-Hornung Daniel Cohen-Or, and Baoquan Chen. Skeleton-aware networks for deep motion retargeting. ACM Transactions on Graphics (TOG), 39(4);22-1, 2020a.
Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. Unpaired motion style transfer from video to animation. ACM Transactions on Graphies (TOG), 39(4):64 1, 2020b.
Adobe Systems Inc. Mixamo, 2021. URL https://www.mixamo com
Accessed: 2021-12-25.
Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose fore- casting. In 2019 International Conference on 3D Vision (3DV), pp. 719-728, Washington, DC, USA, 2019. IEEE, IEEE Computer Society.
Emre Aksan, Manuel Kaufmann, and Otmar Hilliges. Structured prediction helps 3d human motion modelling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7144- 7153, Washington, DC, USA, 2019. IEEE Computer Society.
Moab Arar, Ariel Shamir, and Amit H Bermano. Learned queries for efficient local attention. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10841- 10852, Washington, DC, USA, 2022. IEEE Computer Society.
A Aristidou, A Yiannakidis, K Aberman, D Cohen-Or Or, A Shamir, and Y Chrysanthou. Rhythm is a dancer. Music-driven motion synthesis with global structure. IEEE Transactions Visualization and Computer Graphics, 2022.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, what we can learn from single image. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. OpenReview.net. URL htt.s: httphttts///opeeeeeett hwtp///ppnrree..eeoeeeee. net forum? d= esx6EYvr.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12208-18218, Washington, DC, USA, 2022. IEEE Computer Society.
Emad Barsoum, John Kender, and Zicheng Liu, Hp-gan: Probabilistic 3d human motion prediction via gan. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 1418-1427, Washington, DC, USA, 2018. IEEE Computer Society.
Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 1-10, Washington, DC, USA, 2021. IEEE, IEEE Computer Society.
Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations for variable length human motion generation. In European Conference on Computer Vision, pp. 356-372. Springer, 2022.
Jinshu Chen, Qihui Xu, Qi Kang, and MengChu Zhou. Mogan: Morphologic-structuree generrtttteeeeee learning from single image, 2021.
generative
N Chinchor. Muc-4 evaluation metrics in proc. of the fourth message understanding conference 22-29, 1992.
10

---

Published as conference paper ICLR 2024
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. in 2021 ieee. In CVF International Conference on Computer Vision (ICCV), pp. 14347-14356, Washington, DC, USA, 2021. IEEE Com- puter Society.
Bruno Degardin, João Neves, Vasco Lopes, João Brito, Ehsan Yaghoubi, and Hugo Proença. Gener- ative adversarial graph convolutional networks for human action synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1150-1159, Los Alamitos, CA. USA, 2022. IEEE Computer Society.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, net/
May 3-7, 2021, OpenReview.net, 2021. OpenReview.net. URL / Eorum? id=Yi DeeYicbFNNTT |
Yinglin Duan, Tianyang Shi, Zhengxia Zou, Yenan Lin, Zhehui Qian, Bohan Zhang, and Yi Yuan. Single-shot motion completion with transformer, 2021.
Saeed Ghorbani, Calden Wloka, Ali Etemad, Marcus A. Brubaker, and Nikolaus F. Troje. Probabilis- tic character motion synthesis using a hierarchical deep latent variable model. Computer Graphics Forum, 1, 2020. ISSN 1467-8659. doi: 10.1111/cgg/ 14116.
Ian Goodfellow, Jean Pouget- Abbdet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information process- ing systems, 27, 2014.
Brian Gordon, Sigal Raab, Guy Azov, Raja Giryes, and Daniel Cohen-Or. Flex:
Extrinsic parameters- free multi-view 3d human motion reconstruction. In European Conference on Computer Vision, pp. 176-196, Berlin/Heidelbergg Germany, 2022. Springer, Springer International Publishing.
Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani. Drop the gan: In defense of patches nearest neighbors as single image generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33460-13469 Washington, DC, USA, 2022. IEEE Computer Society.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2021,2029, New York, NY, USA, 2020. ACM.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 5152-5161, Washington, DC, USA, 2022. IEEE Computer Society.
Ikhsanul Habibie, Daniel Holden, Jonathan Schwarz, Joe Yearsley, and Taku Komura. A recurrent variational autoencoder for human motion synthesis. In 28th British Machine Vision Conference, UK, 2017. BMVC.
Félix G Harvey and Christopher Pal. Recurrent transition networks for character locomotion. In SIG- GRAPH Asia 2018 Technical Briefs, pp. 1-4, ACM, New York, NY, USA, 2018
Félix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in- betweening. ACM Transactions on Graphics (TOG), 39(4):60-1, 1, 2020.
Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. Nemf: Neural motion fields for kinematic animation. In NeurlPS, 2022.
Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow, Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions Graphics 396);1111 2020.
Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer. Human motion prediction via spatio- temporal inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7134-7143 Washington, DC, USA, 2019. IEBE Computer Society.
11

---

Published as conference paper ICLR 2024
Tobias Hinz, Matthew Fisher, Oliver Wang. and Stefan Wermter. Improved techniques for training single-image gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1300-1309, Washington, DC, USA, 2021. IEEE Computer Society.
Jonathan Ho and Tim Salimans. Classifier-free free diffusion guidance, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 2020.
Daniel Holden, Jun Saito, and Taku Komura. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG), 35(4):1-11, 2016.
Daniel Holden, Ikhsanul Habibie, Ikuo Kusajima, and Taku Komura. Fast neural style transfer for motion data. IEEE computer graphics and applications, 37(4):42-49, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with condi- tional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1133 1134, Washington, DC, USA, 2017. IEEE Computer Society.
Deok-Kyeond Jang and Sung-Hee Lee. Constructing human motion manifold with sequential networks. In Computer Graphics Forum, volume 39, pp. 314-324, Hoboken, NJ, USA, 2020. Wiley Online Library, The Eurographics Association and John Wiley and Sons Ltd.
Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In Computer Vision and Pattern Recognition (CVPR), 2019.
Manuel Kaufmann, Emre Aksan, Jie Song, Fabrizio Pece, Remo Ziegler, and Otmar Hilliges. Convo- lutional autoencoders for human motion infilling. In 2020 International Conference on 3D Vision (3DV), pp. 918 927, Washington, DC, USA, 2020. IEEE, IEEE Computer Society.
Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language-basee motion synthesis & editing, 2022.
Muhammed Kocabas, Nikos Athanasiou, and Michael Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5253-5263, Washington, DC, USA, 2020. IEEE Computer Society.
Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques (SIGGRAPH 2002)., pp. 473 482. ACM, New York, NY, USA, 2002.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denois- ing diffusion model. In International Conference on Machine Learning, pp. 17920- PMLR, 2023. PMLR, PMLR.
Juheon Lee, Seohyun Kim, and Kyogu Lee. Listen to dance: Music-driven choreography generation using autoregressive mecoer-deccodr network, 2018.
Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative ad- versarial networks. In European conference computer vision, pp. 702-716, Berlin/Heidellerg, Germany, 2016. Springer, Springer International Publishing.
Peizhuo Li, Kfrr Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine- Hornung. Ganimator: Neu- ral motion synthesis from single sequence. ACM Transactions on Graphics (TOG), 41(4))138, 2022.
Ruilong Li. Shan Yang. David A Ross, and Angjoo Kanazawa. Learn to dance with aist++: Music conditioned 3d dance generation, 2021.
Weiyu Li, Xuelin Chen, Peizhuo Li, Olga Sorkine- Hornung, and Baoquan Chen. Example- based motion synthesis via generative motion matching. ACM Transactions on Graphics (TOG), 2023.
Hsin-Ying Chieh Hubert Lin, Hsin- Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, and Ming-HHuaa Yang. Infinity- gan: Towards infinite-pixel image synthesis. In International Conference on Learning Representa- tions, OpenReview.net, 2021. OpenReview.net.
Jianxin Lin, Yingxue Pang, Yingee Xia, Zhibo Chen, and Jiebo Luo. TuiGAN: Learning versatile image-to-image translation with two unpaired images. In European Conference on Computer Vision, pp. 18-35, Berlin/Heideeherg, Germany, 2020. Springer, Springer International Publishing.
12

---

Published as conference paper ICLR 2024
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Shubh Maheshwari, Debtanu Gupta, and Ravi Kiran Sarvadevabhatla, Mugl: Large scale multi person conditional action generation with locomotion. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 257-265, Los Alamitos, CA, USA, 2022. IEEE Computer Society.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and edit- ing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference Muchine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16784-17680 16804, PMLR, 2022. PMLR. URL https: htops://ppcceedding.. m m.r.press/ve v 62/nicho// 22a. html
Yaniv Nikankin, Niv Haim, and Michal Irani. Sinfusion: Training diffusion models on single image video. In International Conference on Machine Learning, pp. 26199 26199-26214, PMLR, 2023. PMLR,
PMLR.
Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché-Bue, Emily B. Fox, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 68-80. 2019. URL
werpogssneeriiie heurips... wwtpcc/ppp.////////// 9/hah/hhhh//// htmml l
341 7516675f4cee 9109909000000 www.://///c..ceeeeaa.eooooo
Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer VAE. In International Conference on Computer Vision (ICCV), pp. 10985-10995, Washington, DC, USA, October 2021. IEEE Computer Society.
Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV), Berlin/Heidelberg, Ger- many, 2022. Springer International Publishing.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables in- put length extrapolation. In International Conference on Learning Representations, OpenReview.net, 2022. OpenReview.net. URL
net wtt.:e//oorommmaedeeeeeee
Jia Qin, Youyi Zheng, and Kun Zhou, Motion in in-betweening via two-stage transformers. ACM Trans- actions on Graphics (TOG), 41(6):1-16, 2022.
Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfrr Aberman, Olga Sorkine- and Daniel Cohen- Or. Modi: Unconditional motion synthesis from diverse data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13873-13883, Washington, DC, USA, 2023. IEEE Computer Society.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022.
CJ Rijsbergen. Information retrieval. online book http://www. des. gla. ac. uk/keith. Preface. html, 1979.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Berlin/Heidelbergg Germany, 2015. Springer, Springer Interna- tional Publishing.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, New York, NY, USA, 2022. ACM.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Benmano. Human motion diffusion as a generative prior, 2023.
13

---

Published as conference paper ICLR 2024
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from a single natural image, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4570-4580, Washington, DC, USA, 2019. IEEE Computer Society.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of NAACL-HLT,, pp. 464-468 2018.
Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Ingan: Capturing and retargeting the" dna" of natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4492-449445 -901,, Washington, DC, USA, 2019. IEEE Computer Society.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. pp. 2265, PMLR, 2015. PMLR, PMLR.
Minjung Son, Jeong Joon Park. Leonidas Guibas, and Gordon Wetzstein. Singraf: Learning 3d gen- erative radiance field for a single scene. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. ety.
Washington, DC, USA, 2023. IEEE Computer Soci-
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, OpenReview.net, 2020a. OpenReview.ned.
Yang Song and Stefano Ermon. Improved techniques for training score- generative models. Ad- vances in neural information processing systems, 33:12438 12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, OpenReview.net, 2020b. OpenReview.net.
Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan S Kankanhalli, Weidong Geng, and Xiangdong Li. Deepdance: music-to-dance motion choreography with adversarial learning. IEEE Transactions on Multimedia, 23:497-509,200
Wenyue Sun and Bao- Di Liu. Esingan: Enhanced single-image gan using pixel attention mechanism for image super-resolution. In 2020 15th IEEE International Conference on Signal Processing (ICSP), volume 1, pp. 181-186, Washington, DC, USA, 2020. IEEE, IEEE Computer Society.
Vadim Sushko, Dan Zhang, Juergen Gall, and Anna Khoreva. Generating novel scene compositions from single images and videos, 2021.
Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging, 15(1):1-28 2015.
Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10521-10530, Washington, DC, USA, 2019. IEEE Computer Society.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In Computer Vision-ECCV 2022: 17th European Confer- ence. Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pp. 358-374, Berlin/Heidelberg, Germany, 2022. Springer, Springer International Publishing.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Hu- man motion diffusion model. In The Eleventh International Conference on Learning Represen- tations, OpenReview.net, 2023. OpenReview.net. URL https:/ http://ppnneeweeweeeeeee net.ileww.eee/eoee forum?
1 dd=SJ1kSy02 fwu!
Truebones Motions Animation Studios. Truebones, 2022. BRRLatpp://rruuabsss..ot com/ Accessed: 2022-1-15.
14

---

Published as conference paper ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ruben Villegas, Jimei Yang, Duygu Ceylan, and Honglak Lee. Neural kinematic networks for unsuper- vised motion retargetting In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8639-8648, Washington, DC, USA, 2018. IEEE Computer Society.
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Sindiffusion: Learning a diffusion model from a single natural image, 2022.
Zhenyi Wang, Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, and Changyou Chen. Learning diverse stochastic human- action generators by learning smooth latent transitions. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1 12288, Washington, DC, USA, 2020. AAAI Press.
Rundi Wu and Changxi Zheng. Learning to generate 3d shapes from a single example, 2022.
Sijie Yan, Zhizhong Li, Yuanjun Xiong. Huahan Yan, and Dahua Lin. Convolutional sequence genera- tion for bbased action synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4402,440,, Washington, DC, USA, 2019. IEEE Computer Society.
Jihyeong Yoo and Qifeng Chen. Sinir: Efficient general image manipulation with single image re- construction. International Conference on Machine Learning, pp. 12040-12055 PMLR, 2021. PMLR, PMLR.
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpaint- ing with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4471-4480, Washington, DC, USA. 2019. IEEE Computer Society.
Ping Yu, Yang Zhao, Chunyuan Li, Junsong Yuan, and Changyou Chen. Structure-aware human-action generation. In European Conference on Computer Vision, pp. 18 18-34, Berlin/Heidelberg, Germany, 2020. Springer, Springer International Publishing.
Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In Eu- ropean Conference Computer Vision, pp. 346-364, Berlin/Heidelherg, Germany, 2020. Springer, Springer International Publishing.
Jia-Qi Zhang, Xiang Xu, Zhi-Meng Shen, Ze-Huan Huang, Yang Zhao, Yan-Pei Cao, Pengfei Wan, and Miao Wang. Write-an-animation High-level text-based animation editing with character-scene interaction. In Computer Graphics Forum, volume 40, pp. 217-228, Hoboken, NJ, USA, 2021a. Wiley Online Library, The Eurographics Association and John Wiley and Sons Ltd.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong. Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model, 2022a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 386-595, Washington, DC, USA, 2018, IEEE Computer Society.
Yan Zhang, Michael J Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3372-3382, Washington, DC, USA, 2021b. IEEE Computer Society.
ZiCheng Zhang, CongYing Han, and TianDe Guo. Exsingan: Learning an explainable generative model from a single image. 2021c.
Zicheng Zhang, Yinglu Liu, Congying Han, Hailin Shi, Tiande Guo, and Bowen Zhou Petsgan: Rethinking priors for single image generation. In Thirty-Sixth AAAI Conference on Artificial In- telligence, AAAL, pp. 3408 3416, Washington, DC, USA, 2022b. AAAI Press. URL https: 7/ojs. aaai. org/ index.idxxx/////
Zilong Zheng, Jianwen Xie, and Ping Li. Patchwise generative convnet: Training energy-based mod- cls from a single natural image for internal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2961-2970, Washington. DC, USA, 2021. IEEE Computer Society.
15

---

Published as conference paper ICLR 2024
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation repre- sentations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, Washington, DC, USA, 2019. IEEE Computer Society.
APPENDIX
This Appendix adds details on top the ones given in the main paper. While the main paper stands on its own, the details given here may shed more light.
In Appendix A we provide concise recap of Denoising Diffusion Prohabilistic Models (DDPM). Ap- pendix [ describes the dynamic features predicted by our network, and and details the hyperparametens used by it. In Appendix D we describe the QNA framework, and in Appendix E we depict screenshots from
study. Lastly, Appendix [F] presents study about motion smoothness, its correlation receptive field width, a metric measure it, and ways ameliorate
A DENOISING DIFFUSION PROBABILISTI MODELS (DDPM) RECAP
DDPMs (Ho et al. 2020) have become the de-facto leading generative networks technique. While they have primarily dominated the imaging domain (Dhariwal & Nichol 2021). recent works have success- fully applied this approach in the motion domain (Tevet et al. 2023; Zhang et al. 2022a). Denoising networks learn to convert unstructured noise to samples from given distribution, through an iterative process of progressively removing small amounts of Gaussian noise.
Given input motion sequence Tp. apply Markov noising process of T steps, (2:) such that
((11/111) = N (Voitt- (1 a)))),
(1)
where aq E (0,1) are constant hyper-paramette parameters. When is small enough, we can approximate ØT No0 7).
We apply unconditional motion synthesis that models 2y as the reversed diffusion process of gradually cleaning IT. using generative network pg. Following Tevet al. (2023) choose to predict the input motion, denoted Zig (Ramesh al. (022)) rather than predicting e,, hence
ig = po( t)-
(2)
We apply the widespread diffusion loss, via
Lsimple Exsp1,111 pe 1)
(3)
Synthesis at inference time is applied through a series of iterations, starting with pure noise Tr- In each iteration, a clean version of the current sample is predicted using generator pe- This predicted clean sample do is then "re-noised" create the next sample 2t-1, with the process being repeated until 0
is reached.
B MOTTON REPRESENTATION ADDITIONAL DETAILS
This section completes the Motion Representation section in the main paper.
this section, we describe the dynamic features predicted by our network. Recall that N denotes the number of frames in the sequence, and F denotes the length of the features of all joints together in a single motion frame.
Let J denote the number of skeletal joints, and let Q denote the number of rotational features, where rotational features may be Buler angles, quaternions, 6D rotations, etc. Let C denote the number of joints that are prone to contact the ground. Clearly, a human, a spider, and a snake possess different values of C.
When using the HumanML3D (uo et al. 2022) dataset, adhere its representation, in which a single pose is defined by
- (7 1z f2 nd "" E RF.
where j E R is the root angular velocity along the Y-axis. f9 E R are root linear velocities on the XZ.plaaee and rV R is the root height. jr E R3J. jo E RR3J and jr E TJ are the local joint
16

---

Published as conference paper ICLR 2024
Table 4: Our choice of hyperparameters, given with the same names used in the code.
Name
Value
UNet related
num_channels
256
channel_mult
num num_res-blocks blocks
kernelssize
use-scalse_shiee norm
True
use_checkpoint
True
use _te_attention
True
use_qna
True
QnA related
head_dim
32
num num_heads
Diffusion related
diffusion steps
1000
noise -scheuuee
cosine
Training related
batch_size
64
dropout
0.5
Ir method
ExponentialLR
Ir gagmmaa
0.99998
num steps
60000
padding mode
zeros
warmup_steps
weight
positions, velocities, and rotations with respect to the root, and of E IRd are binary features denoting the foot contact labels for four foot joints (two for each leg).
When using data from other datasets, we adhere to the representation used by Li et al. 2022). so we can conduct fair comparison with their results. Their representation consists of 3D root location, rotation angle for each joint, and foot contact labels. Altogether, for general representation D N.F we have got F = 3 JQ +
The rotations in both representations are defined in the coordinate frame of their parent in the kinematic chain, and are represented by the 6D rotation features (Q 6) Zhou et al. 2019), which yields the best result in many works (Qin et al. 2022 Petrovich et al. 2021
A growing number of works use foot contact labels Gordon et al. 2022 Raab et al. 2023) to mitigate common foot sliding artifacts, Let C denote the set of joints that contact the ground In the subject whose motion is being learned such that C = |C]. The foot contact labels are represented by L E {0, 1]Nx xc
When dataset provides foot contact label information (Guo al. (202)).
use it as is. When dataset does not provide them, we calculate it as done by ei al. (2022), via
Vj C, N]:
LT IIIAA, S))* <],,
(4)
where (KD,, SI denotes the velocity of joint j in frame retrieved by forward kinematics operator, and 1[:] is indicator function.
HYPERPARAMETERS AND TRAINING DETAILS
Tab. 4 we detail the values of the hyperparameters that have been used to produce the results shown in this work. Our models have been trained on an NVIDIA GeForce RTX 2080 Ti GPU.
D QNA RECAP
QnA layers (Aar et al. 2022) are fundamental component in our suggested architecture. In this sec- tion, we provide an overview of underlying implementation and illustrate it in Fig. 14 particular,
17

---

Published as conference paper ICLR 2024
Murti Actontion
Malt-lleed Attention
Lood Apprcation
Geaig Windon
Tovara
(a) Convolution
(b) SASA
(c) QnA
Figure 14: QnA overview (extracted from the QnA paper). Left: Local layers may utilize various approaches to overlapping windows. (a) Convolutions apply aggregation by learning shared weighted filters. (b) SASA (2019) combines window tokens through self-attention. (c) QnA use shared learned queries across windows, maintaining the expressive power of attention while achieving linear space complexity.
Reshat
83.0
2.5
81.0
80.5
25
50
100
1000
500
Params (M)
GFLOPs
Throughput (img/sec)
Figure 15: QnA demonstrates better accuracy-efficiency trade-off compared to state-of-the-art baselines (extracted from the QnA paper).
QnA an efficient attention-based layer, which operates in shift-invariant manner. For every k-size window, the output calculated using the self-attention mechanism which is commonly used the transformer architecture (Vaswani et al. (2017). The self-attention is calculated by first projecting the input features into keys K =WW,, values V XWv, and queries Q XWQ via three linear projection matrices W. Wy, Wq RR DxD, Then, the output of the self-attentic operation is defined by:
SA(X) Attention (Q. V Softmax (QK*) /VD) V.
(5)
Instead of performing the pricey query-key operation, QnA detours from extracting the queries from the window itself and directly learns them for the whole-training data (see Fig. [4c). Learning the queries preserves the expressive capability of the self-attention mechanism and enables an efficient implementation that relies on simple and fast operations. In particular, a single query á is learned, and the attention is applied locally for every k-size window. Therefore, the output at entry becomes:
Attention (q, KN.) Vx..
(6)
where M; is the set k-neighbourhood of frame i.
QnA exhibits sate-of-the-an accuracy-efficiency trade-off, as depicted in Fig.
18

---

Published as conference paper ICLR 2024
Table 5: Smoothness across varying receptive field widths, the Mixamo benchmark. Each row shows an architecture with an increasing receptive field width, with the right column displaying the smoothness error. The top row refers our best-performing architecture (same ones as in Tab.
Coverage
Global Div.
Local Div.
t
D
Intra Div. Diff.
+
Diff. (x le-5) +
Smoothness
no. layers=1
no. res blocks (baseline)
94.3
1.42
1.00
0.13
0.03
9.8
layers=1
97.9
1.22
0.83
0.13
0.03
2.9
no. res blocks =
layers=2
99.4
0.99
0.68
0.12
0.04
1.9
no. res blocks =
Table Smoothness error across varying Asmooth values, on the Mixamo benchmark. The left column shows increasing values of Asmooth. and the right column displays the smoothness error, which decreases as Asmooth increases. The top row, where Asmooth = O, corresponds to our best-performing architecture to date. Introducing the smoothness loss paves the way for new best-performing architec- ture, as using Aemooth = attains better smoothness with no degradation at the other metrics.
Global Div.
Local Div.
Inter Div.
Intra Div. Diff.
Smoothness Diff. (x le-5)
Asmonth
Coverage
O (baseline)
94.3
1.42
1.00
0.13
0.03
9.8
0.1
94.1
1.43
1.02
0.13
0.04
7.7
94.0
1.43
1.02
0.13
0.05
4.8
10
92.8
1.48
1.10
0.13
0.07
2.6
100
89.4
1.50
1.17
0.12
0.12
1.5
E
USER STUDY SCREENSHOTS
Our user study displays several video clips on each screen, requesting the select the that is more suitable to the examined attribute, which is either quality, fidelity, or diversity. Screenshots from a representative video for each attribute are shown in Fig. [6
F SMOOTHNESS STUDY
We target our generated motions to closely match the input motion's smoothness. Smoothness varies from motion motion, for example, slow walk is characterized by very smooth and flowing motions, while fast dance often exhibits abrupt transitions. Next, we present metric to measure the smoothness fidelity. Then, we show that the smoothness етгог is strongly related to the receptive field width, and lastly, present way improve the smoothness score while keeping receptive field narrow.
Metric To assess the fidelity of our generated motions to the smoothness of the input motion, we introduce a metric that quantifies the difference between the ground truth smoothness and that of the generated motions. The metric
employ is the acceleration error of each joint, first proposed by Kanazawa al. (2019) and subsequently adopted by many others (Kocabas et al. 2020 Gordon al. 2022). The acceleration at joint j is measured using
Acc; = E,eei...N FK (n-1)n 2F + (x
where FK(-)" denotes the location of joint in frame retrieved using a forward kinematics operator, and is motion (given synthesized). The acceleration error is measured using
Lemooth | (Gce((G(zz) Acc, (x)
(8)
where G(-) the inference process and æ the input motion on which our model has been trained.
19

---

Published as conference paper ICLR 2024
Varying receptive field study In Tab. 5 compare smoothness scores for varying receptive field widths. Recall, that a UNet layer is composed of varying number residual blocks and an attention layer. First we present our baseline, which is the architecture presented in the main paper. Recall that our hest-performing model uses UNet of depth 1, with one residual block within it. Adding one more residual block a UNet of depth (row #2) widens the receptive field, and results in better smoothness error. Using a UNet of depth 2 results in an even wider receptive field, and a better smoothness error. We conclude that widening the receptive field decreases the smoothness error. Regrettably, expanding the receptive field leads to a significant decline in the quality of the generated motion, indicated by the diversity metrics, hence it is not practical way to get better smoothness fidelity.
Next, suggest method to enhance smoothness while maintaining favorable metric scores.
Loss study To get better smoothness without increasing the receptive field, we have used the smooth- ness term loss, denoted Lsmooth- Accordingly, our loss is changed to
C = + Asmooth smooth.
Table 6 displays our metric scores on the Mixamo benchmark, for varying values of Asmooth. The table shows that increasing Xemooth decreases the smoothness error. Up to a value of Asmooth the other metric scores remain comparable to the original ones, and for larger values of Asmooth we see a degradation in the coverage and the intra diversity diff. We conclude that training with a value of Asmooth would provide the best results.
20

---

Published as conference paper ICLR 2024
[VC-4500 Which modon
amooh
(a) Quality.
motion mollh the ground Inlth, where the order the motton mothh coes not master
(b) Fidelity.
Hil
JVC-6532 We ch ant of sdeed retioon a mire diveral Civere genane be dfferr from anchher
EI
Diversity.
Figure 16: Screenshots from our user study. Note that each human figure in the screenshot is played as video.
21


Reviews:
[{"review_id": "GqgWLdjKgR", "reviewer": "Reviewer_ep7Y", "summary": "The paper proposes a motion diffusion model (SinMDM) that utilizes a QnA-based UNet architecture in the motion domain. The model incorporates QnA layers, which enable local attention with a temporally narrow receptive field, resulting in improved efficiency in space and time. The use of QnA layers allows for fast and efficient implementation, enhancing the model's performance compared to global attention layers. The paper validates the design choices through experiments and provides a comprehensive list of hyperparameters for reproducibility. The paper demonstrates various applications of single-motion learning using diffusion models. These applications include motion composition, motion harmonization, and straightforward use for long motion generation and crowd animation. The authors showcase the effectiveness of the proposed model in these applications, highlighting its ability to handle complex tasks like harmonization and style transfer.", "questions": "- Are there motion representations other than the adopted one from HumanML3D tested for the proposed method? How do you expect the motion representation could affect the generation performance?\n \n- In what circumstances does the method fail?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The proposed method is the first work that applies diffusion to single motion learning. The paper is well-written and details are clearly presented. The methodology is well-designed in the way to reduce the range of receptive field to prevent acquiring global context and overfitting. The results in the paper and the supp video are of high quality. Applications are interesting and making use of the diffusion framework. The authors also presents a user study that demonstrates the superiority of the proposed model in terms of diversity, fidelity, and quality.", "weaknesses": "- As restricted by the source of data, the results of some editing applications are somehow not that impressive. Similar results can be achieved by simple manipulations without resorting to a powerful diffusion model. E.g. the spatial composition looks no more than cutting-and-pasting the upper/lower body motion, and the style transfer application is like temporally synchronising the pace of the steps of the happy/crouched style video to the content.\n\n- Although a solid work, most techniques used in the methodology are from existing papers, including diffusion for motion, and single motion generation, and the contribution to the community is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iLbrSgByO9", "reviewer": "Reviewer_qd2e", "summary": "The paper introduces SinMDM, a Single Motion Diffusion Model based on the motivation that the number of data for the motion domain is limited. SinMDM learns from a single motion sequence, generating motions true to original motifs, while avoiding overfitting through a lightweight, attention-focused architecture (narrow respective field). It demonstrates various motion tasks, including spatial and temporal composition, as well as Harmonization (style transfer). SinMDM shows high efficiency and good performance.", "questions": "1. As the receptive field is narrowed, how to guarantee temporal smoothness? As shown in Table 3, using a wide receptive field gives better FID.   \n\n2. Comparison with Ganimator in HumanML3D?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The usage of the Diffusion model in the single-motion generation is interesting. \n\n2. Also, this paper demonstrates various interesting applications of motion synthesis with efficiency and good quality. \n\n3. In order to overcome overfitting and promote diversity, SinMDM proposes to use the shallow QnA module to limit the receptive field and relative temporal positional embeddings.", "weaknesses": "1. Single-motion generation task and using Diffusion to do single-instance generation is not new. Some concepts have been explored in previous papers. For example, Sinfusion[1] has pointed out that the receptive field needs to be reduced for single-instance generation task. The only difference is that they use ConvNext while SinMDM uses QnA module. \n\n2. Also, for the Harmonization, the way of using guidance injection is from ILVR [2]. \n\nMaybe, could you elaborate on this, and emphasize the contribution of this work?  \n\n[1] Nikankin, Yaniv, Niv Haim, and Michal Irani. \"Sinfusion: Training diffusion models on a single image or video.\" ICML 2023.\n\n[2] Choi, Jooyoung, et al. \"Ilvr: Conditioning method for denoising diffusion probabilistic models.\"ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3itq0x57kK", "reviewer": "Reviewer_gTdv", "summary": "Paper introduces a generative diffusion model for motion synthesis. The model is a UNet that operates on motion data with a specialized efficient local attention layer (QnA) in place of global attention. The key intuition of this work is restricting the capacity (receptive field) of the underlying UNet to improve generalization and avoid mode collapse. Results are provided on several tasks, including motion composition, harmonization and generation, and seem to be state-of-the-art.", "questions": "- What exactly is meant by \"single input UNet\" in 4.? \n- Why is the UNet the default option here? Wouldn't LSTMs / WaveNet / TCNN would be a better option?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper is well-written and is easy to follow. Supplementary material is of high quality.\n- The key technical novelty - to reduce receptive field - is simple and is easy to implement, and makes sense in the limited data regime.\n- Qualitatively results look impressive. User study and benchmarks also confirm performance of the method is state-of-the-art.", "weaknesses": "- Authors propose a solution that fixes a specialized version of UNet (presumably similar to the one used for image generation) - by introducing local attention mechanism. I understand that it might be a way to repurpose an existing architecture developed for a different task, but it does seem like there are exist a wide range of existing architectures (1d convnets / wavenets), which exhibit the same property by construction. It is a bit unclear why is the vanilla UNet considered in the first place?\n- (Novelty, minor): diffusion models have been used for motion synthesis, the QnA has been used before, although in different context.\nMisc:\n- afaik vanilla UNet [Ronneberger'2015] does not have attention layers, do you imply the unet similar to that used in stable diffusion implementation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TQIJkaupU8", "reviewer": "Reviewer_BL7j", "summary": "The paper proposes \"Single Motion Diffusion\" to learn a generative model of skeletal motions from small training data. The key architectural ideas are 1. shallow U-Net and 2. QnA attention model for local receptive fields. The paper demonstrates applications in the temporal composition (temporal in-painting for in-betweening and motion expansion), spatial composition (spatial in-painting by constraining a part of the skeletal tree with an unseen motion), harmonization (mix low-frequency unseen motion during denoising), and style transfer using harmonization. The method is compared numerically against Ganimator [Li et al. 2022] and GenMM [Li et al. 2023] for the coverage (the rate of generated motions reproducing the input), global diversity, local diversity, inter-diversity, intra-diversity difference, and computational efficiency metrics (number of parameters and total running time). The method is also compared against MDM trained on a single clip and MDM trained on a cropped clip to simulate the narrow receptive field regarding SiFID, inter-diversity, and intra-diversity differences. The numerical evaluations are backed up by the user studies.", "questions": "(Minor comment)\nFrom the abstract:\n> learn the internal motifs of a single motion sequence with arbitrary topology\n\nI initially thought \"arbitrary topology\" meant cases like generating the dragon motions from the model trained on human skeletons. I now know that this just means that the architecture itself is agnostic to the skeletal structure but the trained model must be applied to the same skeletal structure. I hope there is a way to phrase this better to reduce the confusion.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The core takeaway is reducing the temporal receptive field by tweaking existing architectures (reducing the U-Net depth and using the QnA transformer) allows the training of generative motion models with a very small amount of data. While the idea is simple, the technique shows promising results on important animation tasks.\n\nThorough numerical evaluations and user studies support the claim. The visual quality is also fine but with some caveats (see Weakness).\n\nThe technique is reproducible with the provided code, data, and pretrained models.", "weaknesses": "The flip side of the simplicity of the technique is that the technical novelty may not be so significant.\n\nIt seems like there is a trade-off to reducing the temporal receptive field. The generated results often show abrupt transitions, such as unnaturally fast turns. I believe this is due to the limited receptive field not seeing the full segment of some longer actions. Can authors discuss this? This may indicate that we need more metrics to measure the visual quality of motions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)